Deploying Nextcraft — the admin runbook (v0.8)
TL;DR — AI-native outcome school + marketplace — graduates prove what they can build, not what they can write. This file explains how to stand the stack up on any Debian-based server: pick one of three install channels, run nextcraft deploy --service, and verify with the checklist; troubleshooting tables and evidence sit lower in the file.
Production topology since v0.3.6: one port, one process. The ai-service
(uvicorn, :8420) serves both the API and the exported web UI, so the
site works behind any proxy that can reach exactly one backend port — e.g.
the existing HAProxy frontend for nextcraft.coreci.dev.
browser ──HTTPS──▶ HAProxy :443 ──▶ 10.0.0.105:8420 (uvicorn)
├── /v1/* /health /docs → API routes
└── everything else → apps/web/out
No CORS, no mixed content, no second port. Same-origin pages fetch the API
with relative URLs (NEXT_PUBLIC_AI_SERVICE_URL=self).
Pick your channel (v0.8)
nextcraft installs on any Debian-based Linux server through three channels.
Every channel ends at the same place: nextcraft doctor + verify green on
a running single-port stack. nextcraft guide <topic> carries the same
recipes offline inside the CLI.
| Channel | First install | Upgrades | Best for |
|---|---|---|---|
| docker | docker run (image: API + UI pre-loaded) |
docker pull + recreate |
immutable, no host toolchain |
| deb | dpkg -i + nextcraft deploy --service |
new .deb + deploy --service |
apt-shaped fleets |
| binary | one-liner install + deploy --service |
one-liner re-run + deploy --service |
the v0.4 default path |
Channel 1: docker (the image channel)
The image lives in the Gitea registry — every release pushes
git.coreci.dev/coreci/nextcraft:<tag> + :latest. It carries the whole
stack with all dependencies pre-loaded (web export baked, venv baked,
CLI embedded): no node, no pnpm, no repo clone on the host.
# First install (any Debian-based host with docker):
docker run -d --name nextcraft \
-p 8420:8420 \
-v nextcraft-state:/home/nextcraft/.nextcraft \
git.coreci.dev/coreci/nextcraft:<release-tag>
curl -fsS http://localhost:8420/health # {"status":"ok",...}
curl -fsS http://localhost:8420/ # HTML — the learner landing page
Point your proxy at :8420 (HAProxy notes below). Durable state (SQLite,
sandbox workdirs, uploads) lives in the nextcraft-state volume — it
survives upgrades and recreation.
The honest limit — learner sandbox builds: the image runs unprivileged
by default and docker's default seccomp profile blocks the user namespaces
the sandbox fabric needs. The boot log says so loudly (SANDBOX FABRIC DEGRADED), and env creation returns 503 with the same remediation text.
Options:
- run fully privileged where the host allows unprivileged userns:
docker run ... --security-opt seccomp=unconfined --security-opt apparmor=unconfined - or run the stack binary-side on the host (channel 3) — the fabric works natively there.
In-container CLI (the image embeds it at /usr/local/bin/nextcraft):
doctor and verify work (container-aware: export mount, venv import,
live mount probe); guide carries the runbook; bootstrap/deploy/dev
refuse honestly — the image is immutable.
Upgrades:
docker pull git.coreci.dev/coreci/nextcraft:<new-tag>
docker rm -f nextcraft
docker run -d --name nextcraft -p 8420:8420 \
-v nextcraft-state:/home/nextcraft/.nextcraft \
git.coreci.dev/coreci/nextcraft:<new-tag>
The state volume carries data across recreates. NEVER docker exec nextcraft deploy — the immutable-image refusal is correct behavior.
[Backed by: the release-lane smoke gate (five assertions per release) +
cli:test:e2e:docker + the v0.7.4 live image probe.]
Channel 2: deb (the apt-shaped channel)
Every release ships nextcraft_<ver>_<arch>.deb (+ .sha256) for amd64
and arm64. Contents: the self-contained CLI binary + docs; the postinst
runs an ADVISORY doctor — missing Recommends are reported, never fatal.
# First install:
curl -fsSL -o nextcraft.deb \
"https://git.coreci.dev/coreci/nextcraft/releases/download/<tag>/nextcraft_<ver>_amd64.deb"
dpkg -i nextcraft.deb # advisory doctor prints what deploy needs
apt install nodejs python3 git util-linux # the Recommends (deploy stack)
nextcraft doctor
nextcraft deploy --service
Upgrades: dpkg -i nextcraft_<new-ver>_<arch>.deb then nextcraft deploy --service (idempotent). arm64 hosts: use the arm64.deb.
[Backed by: the release-lane deb smoke (fresh debian:bookworm install +
advisory postinst proof) + cli:test:e2e:deb.]
Channel 3: binary (the v0.4 one-liner — unchanged)
# First install (linux x86_64 + aarch64/arm64 — the installer resolves
# your arch):
curl -fsSL https://git.coreci.dev/coreci/nextcraft/raw/main/scripts/install.sh | sh
nextcraft doctor # optional preflight (node, pnpm, python3, git, unshare)
nextcraft deploy --service
Upgrades: re-run the one-liner (idempotent — latest release binary,
sha256 + version-stamp gates) + nextcraft deploy --service.
[Backed by: the fresh-clone E2E (cli:test:e2e, v0.4) + install-script
fixture tests (tamper rejection, arch-resolve, both-arch matrix).]
Upgrade matrix (summary)
| Channel | Upgrade | State |
|---|---|---|
| docker | docker pull + recreate the container |
persists in the nextcraft-state volume |
| deb | dpkg -i new deb + nextcraft deploy --service |
~/.nextcraft/ (host) |
| binary | install one-liner re-run + deploy --service |
~/.nextcraft/ (host) |
nextcraft deploy --service is the SAME command for deb + binary: it pulls
origin/main (or --ref <tag>), rebuilds the export, restarts, and
re-verifies. Inside docker these commands refuse honestly (image immutable).
The one-command deploy (v0.4.6 — channels 2/3)
nextcraft deploy --service
That is the whole upgrade path AND the whole first-install path for the host-side channels. In order it will:
- clone or update the repo (default
~/nextcraft;--dir <path>to pick another location,--ref <tag>to pin a release) pnpm install— the toolchain is discovered, not assumed: node/pnpm behind nvm, PNPM_HOME, or corepack are found even when a non-interactive shell can't see them (the v0.4.5-era outage was exactly this: the box HAD pnpm, the deploy steps couldn't)- ai-service venv (
scripts/bootstrap.sh) - build the web export (
NEXT_PUBLIC_AI_SERVICE_URL=self pnpm build) — always rebuilt; a stale or missing export is what serves the API's JSON 404 at the domain root - restart — stops any old daemon, starts fresh
- verify — venv, env, export presence, and the live mount probe:
with the service running,
GET :8420/must return HTML, not the API's JSON 404. The deploy fails loudly if the site isn't actually served.
--service additionally installs a systemd unit (generated with the
real clone + binary paths, no hand-editing) so the stack survives reboots
and crashes (Restart=always, enabled at install). Under systemd the
daemon commands are not used — systemctl is the supervisor:
systemctl status nextcraft-dev
systemctl stop nextcraft-dev # or: nextcraft stop
journalctl -u nextcraft-dev -f # or: nextcraft log -f (daemon mode)
Without --service, deploy uses the dev -d daemon (pidfile + log under
~/.nextcraft/run/) — fine for dev boxes, not boot-persistent.
Manual mode (the underlying commands)
The deploy command composes the same steps you could run by hand:
cd /path/to/nextcraft
git pull --ff-only
pnpm install
NEXT_PUBLIC_AI_SERVICE_URL=self pnpm build
nextcraft stop && nextcraft dev -d # or: systemctl restart nextcraft-dev
nextcraft verify # live mount probe included
nextcraft dev -d auto-detects apps/web/out and exports
AI_WEB_STATIC_DIR to the server, so the UI is served with zero .env
editing. State lives in ~/.nextcraft/:
| Path | Contents |
|---|---|
~/.nextcraft/data/nextcraft.db |
SQLite (traces/grades/variants/defenses/identity) |
~/.nextcraft/sandboxes/ |
sandbox workdirs (snapshot restore) |
~/.nextcraft/run/<clone-hash>/dev.log |
daemon log (daemon mode) |
~/.nextcraft/run/<clone-hash>/dev.pid |
daemon pidfile (daemon mode) |
Nothing durable is written inside the repo (v0.3.6 founder directive).
Why static export
The web app is mock-data-driven with no server-side runtime needs, so
pnpm build emits plain HTML/JS/CSS to apps/web/out (output: 'export').
uvicorn serves those files — there is no Node web server in production.
Cost: web changes need a pnpm build + service restart (no hot reload) —
nextcraft deploy does both.
HAProxy notes
No config change is required — keep the single backend to :8420. Two timeouts matter for streaming endpoints (SSE chat, LLM responses):
backend nextcraft_api
server api1 10.0.0.105:8420
timeout server 300s # matches the service's 300s LLM read budget
timeout tunnel 3600s # long-lived streams
The in-sandbox telemetry capture agents do NOT traverse HAProxy: they dial
ws://127.0.0.1:8420 on the host's loopback directly.
Fresh start on old boxes (v0.3.6 migration)
The pre-v0.3.6 defaults kept the SQLite DB and sandbox workdirs inside the repo. On upgrade, those old locations are simply abandoned (fresh start per founder decision) — delete them at leisure:
rm -rf apps/ai-service/ai_service/data apps/ai-service/sandboxes
Orphaned sandbox workdirs under the NEW ~/.nextcraft/sandboxes/ are
reaped automatically on service startup (a-1 startup reaper).
v0.5 note: pre-v0.5 databases (fresh start per the deploy directive)
The deployed box runs a FRESH ~/.nextcraft state (v0.3.6 directive), so
nothing below applies there. For any other box carrying a pre-v0.5
~/.nextcraft/data/nextcraft.db: v0.5 adds two variant_record columns
(environment, test_command). The variant store backfills them
automatically on first open (idempotent ALTER TABLE; pre-v0.5 rows read
as build-kind, test commands fall back to pytest at the API seam), so
an in-place upgrade is safe — no schema migration step is required.
Alternatively, follow the fresh-start convention and move the old DB
aside (mv ~/.nextcraft/data/nextcraft.db{,.pre-v05.bak}); it
is regenerated on boot.
Verification checklist
curl -fsS https://nextcraft.coreci.dev/health→{"status":"ok",...}curl -fsS https://nextcraft.coreci.dev/→ HTML (the learner landing page)nextcraft verify→ all checks pass (including the live mount probe)- An unknown path returns the site's 404 page with status 404
systemctl status nextcraft-devshows active (ornextcraft log -fshows the uvicorn access log in daemon mode)- After a reboot: the service comes back by itself (systemd mode)
Troubleshooting (v0.8 extension — the four tables)
Site serves JSON 404 at /
The web export is missing or the mount is unset. Binary/deb: nextcraft deploy --service rebuilds the export and re-verifies. Docker: the image
bakes the export at /opt/nextcraft/web-out — check nothing overrides
AI_WEB_STATIC_DIR. [Backed by: the live mount probe in verify — the
check that caught every prior outage.]
Port 8420 busy
Binary/deb: nextcraft stop (or systemctl stop nextcraft-dev). Docker:
the -p map is yours — fix the conflict or change the host side of the map.
pnpm not found during deploy
The toolchain discovery probes PATH, nvm, PNPM_HOME, corepack. If still
missing: corepack enable pnpm. Docker channel: not applicable (the
toolchain is baked).
Sandbox degrade notice (docker channel)
Expected unprivileged behavior — the fabric needs user namespaces and docker's default seccomp blocks them. The boot notice and the 503 payload name the two options (privileged flags / host-side stack); see Channel 1 above for the exact flags. NOT a bug; a 201 there would be a bug.
Checksum mismatch on install
Do NOT run the binary. Delete it, re-download, verify again; report if it persists. The installer hard-stops by design (sha256 + version-stamp gates). [Backed by: install-script tamper-rejection fixture tests.]
Runner / CI questions
The runner fleet lives on CT 113 (deploy/ci/ is the committed
fleet-as-code — compose + configs + prune cron). Every push runs the
node+python lanes; every release is machine-made by the tag-triggered
workflow (release.yaml). A red Actions tab on a tag = a red release by
design (the smoke gate).
Evidence appendix (which test or live probe backs each recipe)
| Recipe | Backing |
|---|---|
| docker first-install + degrade + refusals | release-lane smoke gate (5 assertions, every release) + cli:test:e2e:docker + the v0.7.4 live image probe (HEALTH/GUIDE/REFUSAL all green) |
| deb install + advisory doctor | release-lane deb smoke (fresh debian:bookworm per release) + cli:test:e2e:deb |
| binary install (both arches) | cli:test install-script fixtures: checksum tamper rejection, version-stamp gate, arch-resolve, both-arch matrix |
| one-command deploy + live mount probe | the v0.4 fresh-clone E2E (cli:test:e2e) + the deployed live box (nextcraft.coreci.dev 200 since v0.4.6) |
| honest degrade text (boot notice = 503 payload) | tests/sandbox/test_capability_degrade.py (G-208 no-drift pinned by test) |
| guide ↔ runbook parity | the guide's channel sections mirror this file's Channels 1-3 (kept in agreement by review at P5 ship; guide tests pin the channel markers) |
| release machine-made (assets + image) | every release from v0.7.1 onward: binary + checksums + debs attached + image pushed by .gitea/workflows/release.yaml |