nextcraft/deploy/README.md

# Deploying Nextcraft — the admin runbook (v0.8)

**TL;DR** — AI-native outcome school + marketplace — graduates prove what they can build, not what they can write. This file explains how to stand the stack up on any Debian-based server: pick one of three install channels, run `nextcraft deploy --service`, and verify with the checklist; troubleshooting tables and evidence sit lower in the file.

Production topology since v0.3.6: **one port, one process**. The ai-service
(uvicorn, `:8420`) serves **both** the API and the exported web UI, so the
site works behind any proxy that can reach exactly one backend port — e.g.
the existing HAProxy frontend for `nextcraft.coreci.dev`.

```
browser ──HTTPS──▶ HAProxy :443 ──▶ 10.0.0.105:8420 (uvicorn)
                                        ├── /v1/*  /health  /docs  → API routes
                                        └── everything else        → apps/web/out
```

No CORS, no mixed content, no second port. Same-origin pages fetch the API
with relative URLs (`NEXT_PUBLIC_AI_SERVICE_URL=self`).

## Pick your channel (v0.8)

nextcraft installs on any Debian-based Linux server through three channels.
Every channel ends at the same place: `nextcraft doctor` + `verify` green on
a running single-port stack. `nextcraft guide <topic>` carries the same
recipes offline inside the CLI.

| Channel | First install | Upgrades | Best for |
|---------|---------------|----------|----------|
| **docker** | `docker run` (image: API + UI pre-loaded) | `docker pull` + recreate | immutable, no host toolchain |
| **deb** | `dpkg -i` + `nextcraft deploy --service` | new `.deb` + `deploy --service` | apt-shaped fleets |
| **binary** | one-liner install + `deploy --service` | one-liner re-run + `deploy --service` | the v0.4 default path |

## Channel 1: docker (the image channel)

The image lives in the Gitea registry — every release pushes
`git.coreci.dev/coreci/nextcraft:<tag>` + `:latest`. It carries the whole
stack with all dependencies pre-loaded (web export baked, venv baked,
CLI embedded): no node, no pnpm, no repo clone on the host.

```sh
# First install (any Debian-based host with docker):
docker run -d --name nextcraft \
  -p 8420:8420 \
  -v nextcraft-state:/home/nextcraft/.nextcraft \
  git.coreci.dev/coreci/nextcraft:<release-tag>

curl -fsS http://localhost:8420/health     # {"status":"ok",...}
curl -fsS http://localhost:8420/           # HTML — the learner landing page
```

Point your proxy at `:8420` (HAProxy notes below). Durable state (SQLite,
sandbox workdirs, uploads) lives in the `nextcraft-state` volume — it
survives upgrades and recreation.

**The honest limit — learner sandbox builds:** the image runs unprivileged
by default and docker's default seccomp profile blocks the user namespaces
the sandbox fabric needs. The boot log says so loudly (`SANDBOX FABRIC
DEGRADED`), and env creation returns 503 with the same remediation text.
Options:

- run fully privileged where the host allows unprivileged userns:
  `docker run ... --security-opt seccomp=unconfined --security-opt apparmor=unconfined`
- or run the stack binary-side on the host (channel 3) — the fabric works
  natively there.

**In-container CLI** (the image embeds it at `/usr/local/bin/nextcraft`):
`doctor` and `verify` work (container-aware: export mount, venv import,
live mount probe); `guide` carries the runbook; `bootstrap`/`deploy`/`dev`
refuse honestly — the image is immutable.

**Upgrades:**

```sh
docker pull git.coreci.dev/coreci/nextcraft:<new-tag>
docker rm -f nextcraft
docker run -d --name nextcraft -p 8420:8420 \
  -v nextcraft-state:/home/nextcraft/.nextcraft \
  git.coreci.dev/coreci/nextcraft:<new-tag>
```

The state volume carries data across recreates. NEVER `docker exec
nextcraft deploy` — the immutable-image refusal is correct behavior.
[Backed by: the release-lane smoke gate (five assertions per release) +
`cli:test:e2e:docker` + the v0.7.4 live image probe.]

## Channel 2: deb (the apt-shaped channel)

Every release ships `nextcraft_<ver>_<arch>.deb` (+ `.sha256`) for amd64
and arm64. Contents: the self-contained CLI binary + docs; the postinst
runs an ADVISORY doctor — missing Recommends are reported, never fatal.

```sh
# First install:
curl -fsSL -o nextcraft.deb \
  "https://git.coreci.dev/coreci/nextcraft/releases/download/<tag>/nextcraft_<ver>_amd64.deb"
dpkg -i nextcraft.deb          # advisory doctor prints what deploy needs
apt install nodejs python3 git util-linux    # the Recommends (deploy stack)
nextcraft doctor
nextcraft deploy --service
```

**Upgrades:** `dpkg -i nextcraft_<new-ver>_<arch>.deb` then `nextcraft
deploy --service` (idempotent). arm64 hosts: use the `arm64.deb`.
[Backed by: the release-lane deb smoke (fresh debian:bookworm install +
advisory postinst proof) + `cli:test:e2e:deb`.]

## Channel 3: binary (the v0.4 one-liner — unchanged)

```sh
# First install (linux x86_64 + aarch64/arm64 — the installer resolves
# your arch):
curl -fsSL https://git.coreci.dev/coreci/nextcraft/raw/main/scripts/install.sh | sh
nextcraft doctor      # optional preflight (node, pnpm, python3, git, unshare)
nextcraft deploy --service
```

**Upgrades:** re-run the one-liner (idempotent — latest release binary,
sha256 + version-stamp gates) + `nextcraft deploy --service`.
[Backed by: the fresh-clone E2E (`cli:test:e2e`, v0.4) + install-script
fixture tests (tamper rejection, arch-resolve, both-arch matrix).]

## Upgrade matrix (summary)

| Channel | Upgrade | State |
|---------|---------|-------|
| docker | `docker pull` + recreate the container | persists in the `nextcraft-state` volume |
| deb | `dpkg -i` new deb + `nextcraft deploy --service` | `~/.nextcraft/` (host) |
| binary | install one-liner re-run + `deploy --service` | `~/.nextcraft/` (host) |

`nextcraft deploy --service` is the SAME command for deb + binary: it pulls
`origin/main` (or `--ref <tag>`), rebuilds the export, restarts, and
re-verifies. Inside docker these commands refuse honestly (image immutable).

## The one-command deploy (v0.4.6 — channels 2/3)

```sh
nextcraft deploy --service
```

That is the whole upgrade path AND the whole first-install path for the
host-side channels. In order it will:

1. **clone or update** the repo (default `~/nextcraft`; `--dir <path>` to
   pick another location, `--ref <tag>` to pin a release)
2. **`pnpm install`** — the toolchain is *discovered*, not assumed:
   node/pnpm behind nvm, PNPM_HOME, or corepack are found even when a
   non-interactive shell can't see them (the v0.4.5-era outage was exactly
   this: the box HAD pnpm, the deploy steps couldn't)
3. **ai-service venv** (`scripts/bootstrap.sh`)
4. **build the web export** (`NEXT_PUBLIC_AI_SERVICE_URL=self pnpm build`)
   — always rebuilt; a stale or missing export is what serves the API's
   JSON 404 at the domain root
5. **restart** — stops any old daemon, starts fresh
6. **verify** — venv, env, export presence, and the **live mount probe**:
   with the service running, `GET :8420/` must return HTML, not the API's
   JSON 404. The deploy fails loudly if the site isn't actually served.

`--service` additionally **installs a systemd unit** (generated with the
real clone + binary paths, no hand-editing) so the stack survives reboots
and crashes (`Restart=always`, `enable`d at install). Under systemd the
daemon commands are not used — `systemctl` is the supervisor:

```sh
systemctl status nextcraft-dev
systemctl stop nextcraft-dev      # or: nextcraft stop
journalctl -u nextcraft-dev -f     # or: nextcraft log -f (daemon mode)
```

Without `--service`, deploy uses the `dev -d` daemon (pidfile + log under
`~/.nextcraft/run/`) — fine for dev boxes, not boot-persistent.

## Manual mode (the underlying commands)

The deploy command composes the same steps you could run by hand:

```sh
cd /path/to/nextcraft
git pull --ff-only
pnpm install
NEXT_PUBLIC_AI_SERVICE_URL=self pnpm build
nextcraft stop && nextcraft dev -d      # or: systemctl restart nextcraft-dev
nextcraft verify                        # live mount probe included
```

`nextcraft dev -d` auto-detects `apps/web/out` and exports
`AI_WEB_STATIC_DIR` to the server, so the UI is served with zero `.env`
editing. State lives in `~/.nextcraft/`:

| Path | Contents |
|------|----------|
| `~/.nextcraft/data/nextcraft.db` | SQLite (traces/grades/variants/defenses/identity) |
| `~/.nextcraft/sandboxes/` | sandbox workdirs (snapshot restore) |
| `~/.nextcraft/run/<clone-hash>/dev.log` | daemon log (daemon mode) |
| `~/.nextcraft/run/<clone-hash>/dev.pid` | daemon pidfile (daemon mode) |

Nothing durable is written inside the repo (v0.3.6 founder directive).

## Why static export

The web app is mock-data-driven with no server-side runtime needs, so
`pnpm build` emits plain HTML/JS/CSS to `apps/web/out` (`output: 'export'`).
uvicorn serves those files — there is no Node web server in production.
Cost: web changes need a `pnpm build` + service restart (no hot reload) —
`nextcraft deploy` does both.

## HAProxy notes

No config change is required — keep the single backend to :8420. Two
timeouts matter for streaming endpoints (SSE chat, LLM responses):

```
backend nextcraft_api
  server api1 10.0.0.105:8420
  timeout server 300s     # matches the service's 300s LLM read budget
  timeout tunnel 3600s    # long-lived streams
```

The in-sandbox telemetry capture agents do NOT traverse HAProxy: they dial
`ws://127.0.0.1:8420` on the host's loopback directly.

## Fresh start on old boxes (v0.3.6 migration)

The pre-v0.3.6 defaults kept the SQLite DB and sandbox workdirs **inside
the repo**. On upgrade, those old locations are simply abandoned (fresh
start per founder decision) — delete them at leisure:

```sh
rm -rf apps/ai-service/ai_service/data apps/ai-service/sandboxes
```

Orphaned sandbox workdirs under the NEW `~/.nextcraft/sandboxes/` are
reaped automatically on service startup (a-1 startup reaper).

## v0.5 note: pre-v0.5 databases (fresh start per the deploy directive)

The deployed box runs a FRESH `~/.nextcraft` state (v0.3.6 directive), so
nothing below applies there. For any other box carrying a pre-v0.5
`~/.nextcraft/data/nextcraft.db`: v0.5 adds two `variant_record` columns
(`environment`, `test_command`). The variant store backfills them
automatically on first open (idempotent `ALTER TABLE`; pre-v0.5 rows read
as build-kind, test commands fall back to `pytest` at the API seam), so
an in-place upgrade is safe — no schema migration step is required.
Alternatively, follow the fresh-start convention and move the old DB
aside (`mv ~/.nextcraft/data/nextcraft.db{,.pre-v05.bak}`); it
is regenerated on boot.

## Verification checklist

- `curl -fsS https://nextcraft.coreci.dev/health` → `{"status":"ok",...}`
- `curl -fsS https://nextcraft.coreci.dev/` → HTML (the learner landing page)
- `nextcraft verify` → all checks pass (including the live mount probe)
- An unknown path returns the site's 404 page with status 404
- `systemctl status nextcraft-dev` shows active (or `nextcraft log -f`
  shows the uvicorn access log in daemon mode)
- After a reboot: the service comes back by itself (systemd mode)

## Troubleshooting (v0.8 extension — the four tables)

### Site serves JSON 404 at `/`

The web export is missing or the mount is unset. Binary/deb: `nextcraft
deploy --service` rebuilds the export and re-verifies. Docker: the image
bakes the export at `/opt/nextcraft/web-out` — check nothing overrides
`AI_WEB_STATIC_DIR`. [Backed by: the live mount probe in `verify` — the
check that caught every prior outage.]

### Port 8420 busy

Binary/deb: `nextcraft stop` (or `systemctl stop nextcraft-dev`). Docker:
the `-p` map is yours — fix the conflict or change the host side of the map.

### `pnpm not found` during deploy

The toolchain discovery probes PATH, nvm, PNPM_HOME, corepack. If still
missing: `corepack enable pnpm`. Docker channel: not applicable (the
toolchain is baked).

### Sandbox degrade notice (docker channel)

Expected unprivileged behavior — the fabric needs user namespaces and
docker's default seccomp blocks them. The boot notice and the 503 payload
name the two options (privileged flags / host-side stack); see Channel 1
above for the exact flags. NOT a bug; a 201 there would be a bug.

### Checksum mismatch on install

Do NOT run the binary. Delete it, re-download, verify again; report if it
persists. The installer hard-stops by design (sha256 + version-stamp
gates). [Backed by: install-script tamper-rejection fixture tests.]

### Runner / CI questions

The runner fleet lives on CT 113 (`deploy/ci/` is the committed
fleet-as-code — compose + configs + prune cron). Every push runs the
node+python lanes; every release is machine-made by the tag-triggered
workflow (release.yaml). A red Actions tab on a tag = a red release by
design (the smoke gate).

## Evidence appendix (which test or live probe backs each recipe)

| Recipe | Backing |
|--------|---------|
| docker first-install + degrade + refusals | release-lane smoke gate (5 assertions, every release) + `cli:test:e2e:docker` + the v0.7.4 live image probe (HEALTH/GUIDE/REFUSAL all green) |
| deb install + advisory doctor | release-lane deb smoke (fresh debian:bookworm per release) + `cli:test:e2e:deb` |
| binary install (both arches) | `cli:test` install-script fixtures: checksum tamper rejection, version-stamp gate, arch-resolve, both-arch matrix |
| one-command deploy + live mount probe | the v0.4 fresh-clone E2E (`cli:test:e2e`) + the deployed live box (nextcraft.coreci.dev 200 since v0.4.6) |
| honest degrade text (boot notice = 503 payload) | `tests/sandbox/test_capability_degrade.py` (G-208 no-drift pinned by test) |
| guide ↔ runbook parity | the guide's channel sections mirror this file's Channels 1-3 (kept in agreement by review at P5 ship; `guide` tests pin the channel markers) |
| release machine-made (assets + image) | every release from v0.7.1 onward: binary + checksums + debs attached + image pushed by `.gitea/workflows/release.yaml` |