nextcraft/deploy/README.md

Deploying Nextcraft — the admin runbook (v0.8)

TL;DR — AI-native outcome school + marketplace — graduates prove what they can build, not what they can write. This file explains how to stand the stack up on any Debian-based server: pick one of three install channels, run nextcraft deploy --service, and verify with the checklist; troubleshooting tables and evidence sit lower in the file.

Production topology since v0.3.6: one port, one process. The ai-service (uvicorn, :8420) serves both the API and the exported web UI, so the site works behind any proxy that can reach exactly one backend port — e.g. the existing HAProxy frontend for nextcraft.coreci.dev.

browser ──HTTPS──▶ HAProxy :443 ──▶ 10.0.0.105:8420 (uvicorn)
                                        ├── /v1/*  /health  /docs  → API routes
                                        └── everything else        → apps/web/out

No CORS, no mixed content, no second port. Same-origin pages fetch the API with relative URLs (NEXT_PUBLIC_AI_SERVICE_URL=self).

Pick your channel (v0.8)

nextcraft installs on any Debian-based Linux server through three channels. Every channel ends at the same place: nextcraft doctor + verify green on a running single-port stack. nextcraft guide <topic> carries the same recipes offline inside the CLI.

Channel	First install	Upgrades	Best for
docker	docker run (image: API + UI pre-loaded)	docker pull + recreate	immutable, no host toolchain
deb	dpkg -i + nextcraft deploy --service	new .deb + deploy --service	apt-shaped fleets
binary	one-liner install + deploy --service	one-liner re-run + deploy --service	the v0.4 default path

Channel 1: docker (the image channel)

The image lives in the Gitea registry — every release pushes git.coreci.dev/coreci/nextcraft:<tag> + :latest. It carries the whole stack with all dependencies pre-loaded (web export baked, venv baked, CLI embedded): no node, no pnpm, no repo clone on the host.

# First install (any Debian-based host with docker):
docker run -d --name nextcraft \
  -p 8420:8420 \
  -v nextcraft-state:/home/nextcraft/.nextcraft \
  git.coreci.dev/coreci/nextcraft:<release-tag>

curl -fsS http://localhost:8420/health     # {"status":"ok",...}
curl -fsS http://localhost:8420/           # HTML — the learner landing page

Point your proxy at :8420 (HAProxy notes below). Durable state (SQLite, sandbox workdirs, uploads) lives in the nextcraft-state volume — it survives upgrades and recreation.

The honest limit — learner sandbox builds: the image runs unprivileged by default and docker's default seccomp profile blocks the user namespaces the sandbox fabric needs. The boot log says so loudly (SANDBOX FABRIC DEGRADED), and env creation returns 503 with the same remediation text. Options:

• run fully privileged where the host allows unprivileged userns: docker run ... --security-opt seccomp=unconfined --security-opt apparmor=unconfined
• or run the stack binary-side on the host (channel 3) — the fabric works natively there.

In-container CLI (the image embeds it at /usr/local/bin/nextcraft): doctor and verify work (container-aware: export mount, venv import, live mount probe); guide carries the runbook; bootstrap/deploy/dev refuse honestly — the image is immutable.

Upgrades:

docker pull git.coreci.dev/coreci/nextcraft:<new-tag>
docker rm -f nextcraft
docker run -d --name nextcraft -p 8420:8420 \
  -v nextcraft-state:/home/nextcraft/.nextcraft \
  git.coreci.dev/coreci/nextcraft:<new-tag>

The state volume carries data across recreates. NEVER docker exec nextcraft deploy — the immutable-image refusal is correct behavior. [Backed by: the release-lane smoke gate (five assertions per release) + cli:test:e2e:docker + the v0.7.4 live image probe.]

Channel 2: deb (the apt-shaped channel)

Every release ships nextcraft_<ver>_<arch>.deb (+ .sha256) for amd64 and arm64. Contents: the self-contained CLI binary + docs; the postinst runs an ADVISORY doctor — missing Recommends are reported, never fatal.

# First install:
curl -fsSL -o nextcraft.deb \
  "https://git.coreci.dev/coreci/nextcraft/releases/download/<tag>/nextcraft_<ver>_amd64.deb"
dpkg -i nextcraft.deb          # advisory doctor prints what deploy needs
apt install nodejs python3 git util-linux    # the Recommends (deploy stack)
nextcraft doctor
nextcraft deploy --service

Upgrades: dpkg -i nextcraft_<new-ver>_<arch>.deb then nextcraft deploy --service (idempotent). arm64 hosts: use the arm64.deb. [Backed by: the release-lane deb smoke (fresh debian:bookworm install + advisory postinst proof) + cli:test:e2e:deb.]

Channel 3: binary (the v0.4 one-liner — unchanged)

# First install (linux x86_64 + aarch64/arm64 — the installer resolves
# your arch):
curl -fsSL https://git.coreci.dev/coreci/nextcraft/raw/main/scripts/install.sh | sh
nextcraft doctor      # optional preflight (node, pnpm, python3, git, unshare)
nextcraft deploy --service

Upgrades: re-run the one-liner (idempotent — latest release binary, sha256 + version-stamp gates) + nextcraft deploy --service. [Backed by: the fresh-clone E2E (cli:test:e2e, v0.4) + install-script fixture tests (tamper rejection, arch-resolve, both-arch matrix).]

Upgrade matrix (summary)

Channel	Upgrade	State
docker	docker pull + recreate the container	persists in the nextcraft-state volume
deb	dpkg -i new deb + nextcraft deploy --service	~/.nextcraft/ (host)
binary	install one-liner re-run + deploy --service	~/.nextcraft/ (host)

nextcraft deploy --service is the SAME command for deb + binary: it pulls origin/main (or --ref <tag>), rebuilds the export, restarts, and re-verifies. Inside docker these commands refuse honestly (image immutable).

The one-command deploy (v0.4.6 — channels 2/3)

nextcraft deploy --service

That is the whole upgrade path AND the whole first-install path for the host-side channels. In order it will:

1. clone or update the repo (default ~/nextcraft; --dir <path> to pick another location, --ref <tag> to pin a release)
2. pnpm install — the toolchain is discovered, not assumed: node/pnpm behind nvm, PNPM_HOME, or corepack are found even when a non-interactive shell can't see them (the v0.4.5-era outage was exactly this: the box HAD pnpm, the deploy steps couldn't)
3. ai-service venv (scripts/bootstrap.sh)
4. build the web export (NEXT_PUBLIC_AI_SERVICE_URL=self pnpm build) — always rebuilt; a stale or missing export is what serves the API's JSON 404 at the domain root
5. restart — stops any old daemon, starts fresh
6. verify — venv, env, export presence, and the live mount probe: with the service running, GET :8420/ must return HTML, not the API's JSON 404. The deploy fails loudly if the site isn't actually served.

--service additionally installs a systemd unit (generated with the real clone + binary paths, no hand-editing) so the stack survives reboots and crashes (Restart=always, enabled at install). Under systemd the daemon commands are not used — systemctl is the supervisor:

systemctl status nextcraft-dev
systemctl stop nextcraft-dev      # or: nextcraft stop
journalctl -u nextcraft-dev -f     # or: nextcraft log -f (daemon mode)

Without --service, deploy uses the dev -d daemon (pidfile + log under ~/.nextcraft/run/) — fine for dev boxes, not boot-persistent.

Manual mode (the underlying commands)

The deploy command composes the same steps you could run by hand:

cd /path/to/nextcraft
git pull --ff-only
pnpm install
NEXT_PUBLIC_AI_SERVICE_URL=self pnpm build
nextcraft stop && nextcraft dev -d      # or: systemctl restart nextcraft-dev
nextcraft verify                        # live mount probe included

nextcraft dev -d auto-detects apps/web/out and exports AI_WEB_STATIC_DIR to the server, so the UI is served with zero .env editing. State lives in ~/.nextcraft/:

Path	Contents
~/.nextcraft/data/nextcraft.db	SQLite (traces/grades/variants/defenses/identity)
~/.nextcraft/sandboxes/	sandbox workdirs (snapshot restore)
~/.nextcraft/run/<clone-hash>/dev.log	daemon log (daemon mode)
~/.nextcraft/run/<clone-hash>/dev.pid	daemon pidfile (daemon mode)

Nothing durable is written inside the repo (v0.3.6 founder directive).

Why static export

The web app is mock-data-driven with no server-side runtime needs, so pnpm build emits plain HTML/JS/CSS to apps/web/out (output: 'export'). uvicorn serves those files — there is no Node web server in production. Cost: web changes need a pnpm build + service restart (no hot reload) — nextcraft deploy does both.

HAProxy notes

No config change is required — keep the single backend to :8420. Two timeouts matter for streaming endpoints (SSE chat, LLM responses):

backend nextcraft_api
  server api1 10.0.0.105:8420
  timeout server 300s     # matches the service's 300s LLM read budget
  timeout tunnel 3600s    # long-lived streams

The in-sandbox telemetry capture agents do NOT traverse HAProxy: they dial ws://127.0.0.1:8420 on the host's loopback directly.

Fresh start on old boxes (v0.3.6 migration)

The pre-v0.3.6 defaults kept the SQLite DB and sandbox workdirs inside the repo. On upgrade, those old locations are simply abandoned (fresh start per founder decision) — delete them at leisure:

rm -rf apps/ai-service/ai_service/data apps/ai-service/sandboxes

Orphaned sandbox workdirs under the NEW ~/.nextcraft/sandboxes/ are reaped automatically on service startup (a-1 startup reaper).

v0.5 note: pre-v0.5 databases (fresh start per the deploy directive)

The deployed box runs a FRESH ~/.nextcraft state (v0.3.6 directive), so nothing below applies there. For any other box carrying a pre-v0.5 ~/.nextcraft/data/nextcraft.db: v0.5 adds two variant_record columns (environment, test_command). The variant store backfills them automatically on first open (idempotent ALTER TABLE; pre-v0.5 rows read as build-kind, test commands fall back to pytest at the API seam), so an in-place upgrade is safe — no schema migration step is required. Alternatively, follow the fresh-start convention and move the old DB aside (mv ~/.nextcraft/data/nextcraft.db{,.pre-v05.bak}); it is regenerated on boot.

Verification checklist

• curl -fsS https://nextcraft.coreci.dev/health → {"status":"ok",...}
• curl -fsS https://nextcraft.coreci.dev/ → HTML (the learner landing page)
• nextcraft verify → all checks pass (including the live mount probe)
• An unknown path returns the site's 404 page with status 404
• systemctl status nextcraft-dev shows active (or nextcraft log -f shows the uvicorn access log in daemon mode)
• After a reboot: the service comes back by itself (systemd mode)

Troubleshooting (v0.8 extension — the four tables)

Site serves JSON 404 at /

The web export is missing or the mount is unset. Binary/deb: nextcraft deploy --service rebuilds the export and re-verifies. Docker: the image bakes the export at /opt/nextcraft/web-out — check nothing overrides AI_WEB_STATIC_DIR. [Backed by: the live mount probe in verify — the check that caught every prior outage.]

Port 8420 busy

Binary/deb: nextcraft stop (or systemctl stop nextcraft-dev). Docker: the -p map is yours — fix the conflict or change the host side of the map.

pnpm not found during deploy

The toolchain discovery probes PATH, nvm, PNPM_HOME, corepack. If still missing: corepack enable pnpm. Docker channel: not applicable (the toolchain is baked).

Sandbox degrade notice (docker channel)

Expected unprivileged behavior — the fabric needs user namespaces and docker's default seccomp blocks them. The boot notice and the 503 payload name the two options (privileged flags / host-side stack); see Channel 1 above for the exact flags. NOT a bug; a 201 there would be a bug.

Checksum mismatch on install

Do NOT run the binary. Delete it, re-download, verify again; report if it persists. The installer hard-stops by design (sha256 + version-stamp gates). [Backed by: install-script tamper-rejection fixture tests.]

Runner / CI questions

The runner fleet lives on CT 113 (deploy/ci/ is the committed fleet-as-code — compose + configs + prune cron). Every push runs the node+python lanes; every release is machine-made by the tag-triggered workflow (release.yaml). A red Actions tab on a tag = a red release by design (the smoke gate).

Evidence appendix (which test or live probe backs each recipe)

Recipe	Backing
docker first-install + degrade + refusals	release-lane smoke gate (5 assertions, every release) + cli:test:e2e:docker + the v0.7.4 live image probe (HEALTH/GUIDE/REFUSAL all green)
deb install + advisory doctor	release-lane deb smoke (fresh debian:bookworm per release) + cli:test:e2e:deb
binary install (both arches)	cli:test install-script fixtures: checksum tamper rejection, version-stamp gate, arch-resolve, both-arch matrix
one-command deploy + live mount probe	the v0.4 fresh-clone E2E (cli:test:e2e) + the deployed live box (nextcraft.coreci.dev 200 since v0.4.6)
honest degrade text (boot notice = 503 payload)	tests/sandbox/test_capability_degrade.py (G-208 no-drift pinned by test)
guide ↔ runbook parity	the guide's channel sections mirror this file's Channels 1-3 (kept in agreement by review at P5 ship; guide tests pin the channel markers)
release machine-made (assets + image)	every release from v0.7.1 onward: binary + checksums + debs attached + image pushed by .gitea/workflows/release.yaml