nextcraft/.ciagent/GRILL.md

Nextcraft v0.3 — GRILL.md (Adversarial Review Verdict)

Stage: GRILL, Phase 0 pre-execution · Verdict: GO-WITH-CHANGES · Confidence: 0.72

Summary

The credential-pipeline architecture (telemetry → trace → grade → defense) is sound and correctly sequenced. Three plan claims did NOT survive contact with this box and were correct before execution. The central problem: the plan overstated sandbox resource-limit enforcement and deferred KYC without closing the resulting no-auth local abuse vector. Fixed via binding decisions G-1..G-6 + scope cuts CUT-1/CUT-2 — no redesign required.

Per-Axis Findings

Axis	Verdict	Rationale
Feasibility	CONCERN	Core unshare userns/mount/pid/net isolation probe-verified (uid=0 in-ns, network isolated, writes contained). But rlimit enforcement is partial: RLIMIT_NPROC scopes to the real host uid (5 sandboxes share one pids budget) and no disk-quota tool exists on the box.
Over-scoping	CONCERN	39 tasks across 5 new subsystems + learner-surface rewrite for a solo founder. Voice-real-path and the interactive terminal relay are separable from the pipeline proof → cut (CUT-1, CUT-2).
Architecture risk	CONCERN	SandboxBackend/TraceStore protocols are the right seams. Overclaimed "limits enforced" + WS ingest "drop-oldest on unbounded growth" contradicted the at-least-once grading guarantee.
Phase sequencing	PASS	P1 sandbox → P2 telemetry → P3 grading → P4 variants → P5 voice → P6 integration is a correct dependency DAG.
Verification honesty	FAIL (fixed)	Must-Haves asserted "resource limits enforced + observable" (CPU/memory/disk/time quotas) the named mechanism cannot satisfy; probe tests would pass while the guarantee was false. Corrected by G-1.
Cost/quota	CONCERN	LLM/voice mock-gated (good). No-auth sandbox creation + unbounded disk + shared NPROC let one learner starve others at zero cost → closed by G-5.
Milestone honesty	CONCERN (fixed)	Release note disclosed KYC deferral + IDE-only but was silent on partial resource-limit enforcement → G-6.
Security	FAIL (fixed)	POST /v1/sandboxes {learner_id} client-supplied over localhost CORS let any local process mint sandboxes/flood/exhaust shared resources → G-5 abuse control ships despite KYC deferral.
Operability	CONCERN (advisory)	In-memory sandbox registry loses handles on restart (orphaned namespaces) → a-1 startup reaper.

Binding Decisions (applied to PLAN.md/ARCHITECTURE-adjacent docs/REQUIREMENTS.md/ROADMAP.md/PROJECT.md)

• G-1 (BINDING) — Resource-limit claims match the deliverable mechanism. Memory (RLIMIT_AS) + CPU (RLIMIT_CPU) + single-file (RLIMIT_FSIZE) + wall-clock reaper are kernel-enforced; per-sandbox pids and hard disk quota are NOT kernel-enforceable without cgroup delegation/sudo → documented as accepted v0.3 risk. Applied to REQ-3-002, PLAN P1 Must-Haves + Task 1-2-02, ROADMAP P1 criteria.
• G-2 (BINDING) — Disk cap via manager workdir-size sweep. AI_SANDBOX_MAX_WORKDIR_MB (default 512MB); sweep snapshots+destroys over-cap sandboxes and logs an integrity signal; closes the unbounded-dd hole. Applied to PLAN Task 1-2-01 + 1-2-02(e) + P1 Must-Haves.
• G-3 (BINDING) — Telemetry flood control WITHOUT silent drop. Bounded queue; on overflow or >AI_TELEMETRY_MAX_EVENTS_PER_TASK (default 50k) → WS close 1008 + trace marked INCOMPLETE_FLOODED (Proctor signal). Silent drop-oldest forbidden (corrupts grading). Applied to PLAN Task 2-2-02 + P2 Must-Haves.
• G-4 (BINDING) — Grader refuses incomplete/gapped traces. grade() gates on TraceStore.gaps() + INCOMPLETE_FLOODED → returns verdict=UNGRADABLE_TRACE_INCOMPLETE; no credential from a gapped trace. Applied to PLAN Task 3-2-01 + P3 Must-Haves.
• G-5 (BINDING) — No-auth abuse control at MVP scale. Per-learner sandbox cap + global create-rate cap (429) + server-side learner_id allowlist (403) so the unauthenticated surface can't exhaust shared NPROC/disk. Ships WITH the milestone even though KYC is deferred. Applied to PLAN Task 1-3-01 + PROJECT A-110.
• G-6 (BINDING) — Disclose partial enforcement in the P7 release note. Item (e): which limits are kernel-enforced vs best-effort, and that full enforcement is deferred to the post-MVP containerd backend. Applied to PLAN P7 release-note honesty block.

Scope Cuts (accepted — preserve the end-to-end credential pipeline)

• CUT-1 (G-7) — Real server STT/TTS (OpenAIAudioProvider) deferred to v0.4. Voice is mock-first (D-030); the /audio/* real path can never run in CI and was the least-verifiable surface. v0.3 proves the full defense dialogue + integrity-signal pipeline over mock + browser-native fallback; the VoiceProvider protocol is the future drop-in seam. Applied to PLAN Phase 5 Goal + Task 5-1-01/5-4-01 + P5 Must-Haves.
• CUT-2 (G-8) — Interactive xterm.js shell relay deferred to v0.4. The credential pipeline needs process events (Run/Test + file edits), not a live keystroke-level shell — the most fragile real-time piece, unverifiable without a real terminal. The build panel becomes Run/Test buttons + read-only exec output render; @xterm/* is NOT a v0.3 dependency. Applied to PLAN Env-facts, P6 Goal, Task 6-2-01/6-2-02/6-3-01, P6 Must-Haves, MVP/UX sections.
• Variants (Phase 4) and the Examiner agent dialogue KEPT — both are on the credential critical path (anti-collusion + the defense dialogue).

Advisory (applied)

• a-1 Startup reaper: on lifespan boot, scan AI_SANDBOX_DIR, reap workdirs whose recorded pid is dead, log a warning. → Task 1-2-01.
• a-2 RLIMIT_FSIZE (~50MB) as a cheap partial single-file disk guard in the spawner's preexec_fn. → Task 1-2-01 + 1-2-02(c).
• a-3 SQLite PRAGMA journal_mode=WAL + synchronous=NORMAL at engine creation (avoids database is locked under concurrent ingest + grader reads). → Task 2-1-02.
• a-4 Grading prompt note: treat high edit/command churn with no test-progress as a process-quality negative (softens digest-gaming naivety). → Task 3-2-01.
• a-5 Variant fairness envelope: two variants of one template must compute digests within the template's expected feature envelope ("same bar" is testable). → Task 4-2-01 Verify.

Outcome

GO — all six binding decisions and both scope cuts applied to PLAN.md / REQUIREMENTS.md / ROADMAP.md / PROJECT.md before Phase 1 execution. No axis requires escalation (all resolvable at confidence ≥ 0.85). The milestone no longer claims resource enforcement it cannot deliver, and the no-auth abuse vector is closed at MVP scale.

---

Nextcraft v0.4 — GRILL.md (Adversarial Review Verdict)

Stage: GRILL, Phase 0 pre-execution · Verdict: GO-WITH-CHANGES · Confidence: 0.83

Summary

The distribution milestone is small, founder-directed (D-016, confidence 0.99), and additive (zero changes to the running credential pipeline). The plan's central risk: Node SEA was probe-verified as a flag, not as a working build — the v0.3 lesson (A-101: probe the mechanism, not the existence) applies. Second gap: a binary whose --version lies (stale package.json) would poison the "ongoing binaries" contract. Third: sed-based JSON parsing in install.sh is a fragility + integrity risk. Fourth: "ongoing binaries" has no enforcement mechanism beyond prose. All four closed by binding decisions G-101..G-104 below. No scope cuts required — the milestone is already minimal.

Per-Axis Findings

Axis	Verdict	Rationale
Business case	PASS	Founder directive explicit + recorded (D-016). Evidence of need: live Gitea probe shows latest release v0.2.8 with ZERO assets; bootstrap requires repo archaeology (scripts found only via package.json spelunking).
Scope	PASS	5 REQs, 3 execution phases, one focused surface (apps/cli + scripts). Smallest milestone yet. macOS arm64 already cut (D-036, unverifiable here).
Feasibility	CONCERN (fixed)	SEA flag exists on node v24.15.0, but no end-to-end SEA binary was built during RESEARCH. postject availability assumed (npx postject — needs npm registry reachability, unproven). Zipapp fallback requires python3 on target — an honest-degradation ladder, not a silent downgrade. → G-101.
Honest versioning	CONCERN (fixed)	--version from package.json would print a stale hardcoded version inside a per-release binary — breaks upgrade detection + the one-liner's re-run-to-upgrade promise. → G-102.
Install integrity	CONCERN (fixed)	sed/grep JSON parsing is brittle; a parse failure must never fall through to installing an unverified artifact. Exact asset-name matching + hard-degrade to source instructions. → G-103.
Sequencing	PASS	P1 CLI (source-runnable) → P2 binary+pipeline → P3 docs+E2E matches dependency order; each phase ships independently.
Cost/quota	PASS	Zero new paid infra; binaries built on-box; Gitea releases free. Dev-only esbuild dep.
Risks	CONCERN (fixed)	Top 3: SEA end-to-end (→ G-101 live probe FIRST in P2), npm registry reachability for esbuild (→ proven by P1's pnpm install must-have), Gitea asset-upload token scope (→ live-proven at the v0.3.2 ship itself).
Adoption/operability	PASS	Consumer = founder + future pilots; one command replaces README archaeology. Rollback trivial (rm ~/.local/bin/nextcraft). No server changes.

Binding Decisions (applied to PLAN.md)

• G-101 (BINDING) — SEA live-build probe is the FIRST P2 action. Task 2-1-01 builds a real binary before anything depends on it; the build script encodes the fallback ladder explicitly (SEA → zipapp with "requires python3" honesty). If SEA fails on this box, zipapp becomes primary with the docs stating the requirement — no silent claim of node-less operation.
• G-102 (BINDING) — Version stamping at build time. build-binary accepts the shipping tag and stamps it into the bundle (NEXTCRAFT_VERSION replace); --version prints it; install E2E asserts the installed binary reports the tag it was downloaded from. A binary may never report a version it was not built as.
• G-103 (BINDING) — Install-script integrity hard-degrade. install.sh matches assets by EXACT name (nextcraft-linux-x64, nextcraft-linux-x64.sha256); any parse/lookup/download failure degrades to source-bootstrap instructions (exit 0) — never installs unverified or name-approximate artifacts. Checksum mismatch = hard stop, exit 1, explicit do-not-run message. dash-safe POSIX sh, no jq.
• G-104 (BINDING) — Ongoing-binaries enforcement. Every ship from v0.3.2 onward MUST run scripts/release-assets.sh <tag> after tag+merge (best-effort, non-blocking, release_pending escalation on failure — but attempted + logged every release). The final-phase audit gate includes "milestone release carries both assets" as a check. This makes the founder's "ongoing binaries" directive a pipeline property, not prose.

Escalations

None. All four concerns resolved at confidence ≥ 0.85. No axis requires founder escalation (directive already explicit).

Outcome

GO — G-101..G-104 applied to PLAN.md before Phase 1 execution. The milestone claims only what its probes prove, and the ongoing-binaries contract has an enforcement mechanism.