Compare commits

..

20 Commits

Author SHA1 Message Date
CIAgent 135ea21a61 merge: hotfix/fresh-box-experience → main (v0.3.5 fresh-box experience)
Fixes the v0.3.4 fresh-box failures: silent CLI under bare-word PATH
invocation (SEA argv detection), bootstrap dying on venv creation without an
actionable hint (poisoned-partial-venv recovery + apt hint + doctor
venv-capability probe + preflight), installer falsely 'verifying' a silent
binary, and localhost-only server binding (network mode: 0.0.0.0 + wildcard
CORS + hostname-derived API URL — remote browsing zero-config).

38 CLI tests, 3 web tests, 409 ai-service tests green; build/typecheck/lint
clean.

---ci---
phase: hotfix
milestone: v0.4
status: complete
type: hotfix
requirements:
  covered: [REQ-4-001, REQ-4-002, REQ-4-003, REQ-4-004, REQ-4-005]
  partial: []
---/ci---
2026-09-13 00:05:37 +00:00
CIAgent 1648467828 chore(P04): clear checkpoint — milestone v0.4 complete
---ci---
phase: 4
milestone: v0.4
status: complete
---/ci---
2026-09-12 23:18:51 +00:00
CIAgent 1a68d808fb docs(milestone): complete v0.4-distribution
---ci---
phase: 4
milestone: v0.4
status: complete
requirements:
  covered: [REQ-4-001, REQ-4-002, REQ-4-003, REQ-4-004, REQ-4-005]
  partial: []
---/ci---
2026-09-12 23:18:51 +00:00
CIAgent 2c68b44c1a merge: milestone/v0.4-distribution → main (v0.4 Distribution & Bootstrap CLI complete)
The nextcraft bootstrap CLI ships: doctor/bootstrap/verify/dev commands, a
one-liner install script with checksum + version integrity gates, and linux
x64 SEA binaries published on every release going forward (v0.3.2 onward).
Fresh-clone E2E proven; 34 CLI tests + full monorepo gates green.

Escalation note: merge_to_main hook — proceeding per full autonomy + founder
directive D-016 (streamlined install + bootstrap CLI + ongoing binaries,
recorded at P0 SPECIFY).

---ci---
phase: 4
milestone: v0.4
status: complete
requirements:
  covered: [REQ-4-001, REQ-4-002, REQ-4-003, REQ-4-004, REQ-4-005]
  partial: []
---/ci---
2026-09-12 23:17:43 +00:00
CIAgent fb2db4ee97 chore(P07): clear checkpoint — milestone v0.3 complete
---ci---
phase: 7
milestone: v0.3
status: complete
---/ci---
2026-09-12 21:45:31 +00:00
CIAgent 485d86e117 docs(milestone): complete v0.3-credential-engines
---ci---
phase: 7
milestone: v0.3
status: complete
requirements:
  covered: [REQ-3-001, REQ-3-002, REQ-3-003, REQ-3-004, REQ-3-005, REQ-3-006, REQ-3-007, REQ-3-008]
  partial: []
---/ci---
2026-09-12 21:45:23 +00:00
CIAgent e798e1a6da merge: milestone/v0.3-credential-engines → main (v0.3 Credential Engines complete)
The six AI tutor agents now operate on REAL credential engines: namespace-isolated
sandbox fabric, live build telemetry (at-least-once, exactly-once stored), process-trace
grading (G-4 gated, digest-only prompts), seeded per-learner variants with fairness
anchors, oral defense with integrity signals, and the learner surfaces are real
(build/defense/grading). 407 tests green; all gates green.

Escalation note: merge_to_main hook — proceeding per full autonomy + the founder's
GO directive for milestone v0.3 (recorded in the run log at P0 SPECIFY).

---ci---
phase: 7
milestone: v0.3
status: complete
requirements:
  covered: [REQ-3-001, REQ-3-002, REQ-3-003, REQ-3-004, REQ-3-005, REQ-3-006, REQ-3-007, REQ-3-008]
  partial: []
---/ci---
2026-09-12 21:44:53 +00:00
CIAgent cd45097e52 merge(P07): final review + audit → milestone/v0.3-credential-engines
---ci---
phase: 7
milestone: v0.3
status: ship
---/ci---
2026-09-12 21:44:46 +00:00
CIAgent 1d03f0c8f5 chore(P07): audit fixes — tag re-point v0.2.1/v0.2.2, phase-status + doc-reality drift
---ci---
phase: 7
milestone: v0.3
status: audit
lessons:
  - P0 reconstruction: tags v0.2.1/v0.2.2 pointed at the pre-migration first attempt (818d8c3/32af0fb) orphaned at the 2026-09-12 forge cutover — NOT ancestors of HEAD; re-pointed to the shipped phase commits (45b2162 / f0df185) matching the v0.2.3..v0.2.7 series convention; all v0.2.x tags now ancestors of HEAD
  - P1 roadmap: phase 0 status stuck at in-progress after the P0 ship; phase 7 pending despite the P07 review commit; both corrected (complete / in-progress)
  - P1 requirements: 27 v0.1 requirement rows (REQ-002..028) still said pending — contradicting the traceability matrix in the same file, PROJECT.md, and ROADMAP (v0.1 shipped as v0.1.0); all set complete
  - P1 grill consistency: GRILL CUT-2 (no xterm in v0.3) never propagated to PERSONAS.md (frontend frameworks listed @xterm/xterm) and ARCHITECTURE.md build order (xterm.js build/run); PROJECT.md A-103 still claimed a WebSocket xterm terminal on a bwrap sandbox; all aligned to the shipped read-only exec-output panel (CUT-2/G-8, D-024 unshare)
  - P1 architecture drift: api/ row missing the five v0.3 routers (sandboxes/telemetry/variants/defense + grade endpoint); voice/ row missing factory.py + defense_store.py; agents/ row missing examiner.py; main.py row missing engine-store lifespan wiring; ui/types rows missing the v0.3 primitives and engine type modules; data-flow section still described the v0.2 mock-agent flow as current
  - P2 docstring: IngestSession still documented the pre-P7 len(get_trace) O(n²) flood-cap count that the P07 fix replaced with TraceStore.count()
---/ci---
2026-09-12 21:42:10 +00:00
CIAgent 12b2300f6f fix(P07): final review — CORS PUT, WS origin gate, ingest leak+O(n²), symlink escape, retry leak, doc-reality gaps
---ci---
phase: 7
milestone: v0.3
status: review
lessons:
  - P0 CORS: allow_methods lacked PUT while the build surface writes files with PUT — every cross-origin Save failed preflight; pinned with tests/api/test_cors.py
  - P0 ingest leak: queue-overflow flood path returned without the disconnect sentinel, parking the drainer forever (one leaked task-set per flooded trace); sentinel now always enqueued, real-server regression test added
  - P1 perf: flood cap counted rows via len(get_trace(...)) — O(trace) per append, O(n²) per session; TraceStore.count() (COUNT(*)) added and wired
  - P0 security: file routes followed exec-planted symlinks out of the workspace bind; _resolve_in_workspace refuses escapes (422), read/write now 404 on unknown sandboxes (was 500)
  - P1 security: WS ingest accepted any browser Origin (CORS middleware does not cover WS); localhost dev origins + no-Origin (capture agent) allowed, others 1008
  - P1 correctness: use-sandbox-session leaked a created sandbox on any mid-start failure (per-learner cap 1 → all retries 429 forever); failed starts now destroy what they created
  - P2 testing: reconnect-flush test killed mid-burst (nondeterministic under load, reproduced on pre-change code); now waits for server-side observation of the pre-kill burst — the underlying one-line replay-margin/ACK gap is documented for v0.4
  - maintainability: grading-store/templates/grading.ts docstrings claimed grading is variant-blind (stale pre-P4 text) — updated; ARCHITECTURE.md referenced nonexistent voice/openai_audio.py; dead if TYPE_CHECKING: pass blocks removed
---/ci---
2026-09-12 20:02:10 +00:00
CIAgent e1460aed3c chore(P06): checkpoint complete — phase 6 shipped as v0.2.7
---ci---
phase: 6
milestone: v0.3
status: complete
requirements: {covered: [REQ-3-007, REQ-3-008], partial: []}
---/ci---
2026-09-12 18:31:23 +00:00
CIAgent b7a56d35bc merge(P06): phase/06 integration → milestone/v0.3-credential-engines
---ci---
phase: 6
milestone: v0.3
status: ship
---/ci---
2026-09-12 18:31:06 +00:00
CIAgent 16fb52d8f7 docs(P06): mark REQ-3-007/008 + phase 6 verified
---ci---
phase: 6
milestone: v0.3
status: verify
requirements: {covered: [REQ-3-007, REQ-3-008], partial: []}
---/ci---
2026-09-12 18:30:59 +00:00
CIAgent a905eb8c67 fix(P06): defend page keys defense+grading on the REAL variant task_id (verifier P0)
The defend page fabricated taskId = `task-${competencyId}`, but variant
task_ids are `task-<seed[:16]>` (D-029) — so in the real browser flow the
defense ran against an empty trace (no digest grounding) and Grade My Work
always returned UNGRADABLE_EMPTY_TRACE. The E2E test masked this by passing
the real task_id directly.

Fix: DefenseSession resolves the learner's stored variant by competency
via GET /v1/variants?learner_id (new listVariants client), defends + grades
under the variant's real task_id, and shows an honest empty state when no
build session exists for the competency yet.

---ci---
phase: 6
milestone: v0.3
status: verify
requirements:
  covered: [REQ-3-008]
  partial: []
lessons:
  - A green E2E test can still mask a broken UI wiring when the test hand-picks
    the join key the UI is supposed to derive; verify browser flows against the
    ids the pages actually construct, not the engine contract alone.
---/ci---
2026-09-12 18:27:43 +00:00
CIAgent ed243594d2 feat(P06): real build + defense surfaces, E2E credential flow (Waves 3-5)
Wave 3 (Task 6-3-01): /build/[competencyId] rewritten as a real build surface —
variant statement + starter files in a live namespace sandbox, workspace file tree +
editor with save, Run/Test buttons executing bounded commands in-sandbox with
read-only TerminalFrame output (CUT-2), live TelemetryStatus pulse, Lab feedback over
the live trace, honest 503-busy/403-429-denied states with retry.
Wave 4 (Task 6-4-01): /defend/[competencyId] rewritten — DefenseSession: start ->
examiner question -> typed answers (mic capture w/ MediaRecorder consent + denied
fallback; browser-SR first-class per CUT-1) -> finish -> verdict + integrity signals
-> Grade My Work renders real rubric bars from the trace digest. Dead mock components
disposed (oral-defense-interface, assessor-results-panel, proctor-banner — G-5 class).
Wave 5 (Task 6-5-01): test_e2e_credential_flow — REAL uvicorn + REAL namespaces:
variant -> sandbox -> in-sandbox exec -> trace -> grade (seed stamped) -> coaching ->
defense -> verdict -> proctor; corpus-fixture scan of all payloads. E2E caught a real
bug: the sandboxes API dropped task_id (every HTTP-created sandbox was capture-less)
— fixed. README E2E + manual browser pass documented.

pnpm build 4/4; typecheck FULL TURBO; backend suite 397 green; ruff clean.

---ci---
phase: 6
milestone: v0.3
status: execute
requirements: {covered: [REQ-3-007, REQ-3-008], partial: []}
---/ci---
2026-09-12 17:54:30 +00:00
CIAgent c760f9af2b feat(P06): engine client + sandbox session hook + files/exec routes (Wave 2, task 6-2-01)
Backend: /v1/sandboxes/{id}/files (list/read/write; traversal rejected 422) and
/v1/sandboxes/{id}/exec (bounded command, captured output — CUT-2: no shell relay);
async workspace resolution for tracked + shell layouts; 17 API tests green.
Web: lib/engine-client.ts — typed fetch client for all engines (sandboxes/files/exec/
variants/grade/defense/traces/lab-SSE/proctor) with honest error mapping (503 busy ->
EngineBusyError, 403 not-allowlisted, 429 rate-limited); hooks/use-sandbox-session.ts —
variant->sandbox->starter-files bootstrap, run/test/saveFile/openFile actions, idempotent
destroy on unmount (AbortController), busy/denied/error states surfaced. typecheck 7/7.

---ci---
phase: 6
milestone: v0.3
status: execute
requirements: {covered: [REQ-3-008], partial: []}
---/ci---
2026-09-12 17:21:30 +00:00
CIAgent b4ae388f22 feat(P06): UI primitives + defense TS types (Wave 2, tasks 6-2-02/6-2-03)
packages/ui: TerminalFrame (CUT-2 read-only exec-output viewer, aria-live streaming),
MicControl (consent-first states incl. denied/unsupported), GradeBadge (verdict + gate
outcomes), TelemetryStatus (live pulse/disconnected), TranscriptViewer (role-styled turns
with latency chips) — token-driven, dark mode, WCAG AA (contrast verified per pair),
stories for each.
packages/types/defense.ts: DefenseSession/Turn/Verdict/VoiceDescriptor/IntegritySignal +
response interfaces, field-for-field Python parity with documented wire deltas.

typecheck 7/7 green.

---ci---
phase: 6
milestone: v0.3
status: execute
requirements: {covered: [REQ-3-008], partial: []}
---/ci---
2026-09-12 05:47:30 +00:00
CIAgent 925ab096fb feat(P06): agent re-grounding on real engine inputs + corpus dormancy (Wave 1)
Tasks 6-1-01..04 (REQ-3-007): Lab consumes the LIVE trace digest (compute_digest over
TraceStore events; empty trace coaches the baseline); Assessor renders coaching FROM
the stored GradeRecord (it never invents scores — the grading engine owns that;
evaluate endpoint re-grounded: 404 without a grade); Proctor consumes digest +
DefenseStore long-pause signals + variant seed cross-check. Corpus telemetry/artifacts
DORMANT (headers + AST dormancy test: zero production importers; learner_context stays
active; retained as Phase-3 calibration history). lifespan now adopts a pre-set
provider (state-injection pattern).

v0.2 corpus-path endpoint tests updated honestly to the live contract (learner_id+task_id).
392 tests green; ruff clean.

---ci---
phase: 6
milestone: v0.3
status: execute
requirements: {covered: [REQ-3-007], partial: []}
---/ci---
2026-09-12 05:15:25 +00:00
CIAgent 82ae839cd4 chore(P05): checkpoint complete — phase 5 shipped as v0.2.6
---ci---
phase: 5
milestone: v0.3
status: complete
requirements: {covered: [REQ-3-006], partial: []}
---/ci---
2026-09-12 04:52:05 +00:00
CIAgent b6c1bc9d54 merge(P05): phase/05 voice defense → milestone/v0.3-credential-engines
---ci---
phase: 5
milestone: v0.3
status: ship
---/ci---
2026-09-12 04:51:52 +00:00
111 changed files with 6749 additions and 2057 deletions
+88 -21
View File
@@ -6,6 +6,8 @@ Nextcraft is a TypeScript monorepo (pnpm workspaces + turborepo) with a Next.js
**v0.3 additions (Credential Engines):** real credential engines replace v0.2 mock inputs — a sandbox fabric (isolated per-learner coding environments via Linux user/mount/pid/net namespaces), a live build-telemetry pipeline (WebSocket ingest + SQLite-ordered event log), a process-trace grading engine, seeded per-learner variant task generation, and a voice-based oral defense (STT/TTS via a new provider-agnostic voice layer). **First real persistence introduced: SQLite** (`ai_service/telemetry/`, grading, variant, defense stores). Lab/Assessor/Proctor agents are re-grounded onto real telemetry/traces. **Identity/age-gating (KYC) deferred per founder directive** — no security engineer persona; secrets-hygiene checklist only.
**v0.4 additions (Distribution & Bootstrap CLI, founder directive D-016):** a new `apps/cli` package — the `nextcraft` bootstrap CLI (`doctor`/`bootstrap`/`verify`/`dev`) compiled to a self-contained linux x64 binary via **Node SEA** (probe-verified: Go/Rust absent, node v24.15.0 SEA-capable), installed by a repo-served one-liner script that resolves the latest Gitea release, downloads binary + sha256 sidecar, verifies, and installs to `~/.local/bin`. Every release from v0.4 onward attaches the binary + checksum as release assets (the "ongoing binaries" requirement). The CLI is a thin wrapper: all orchestration logic stays in `apps/ai-service/scripts/` (bootstrap.sh/dev.sh) — the CLI composes them via subprocess (A-202), duplicating nothing. Previously-planned v0.4 seams (real STT/TTS, KYC, design/sim envs, seq-lease) move to v0.5.
### Confirmed Technology Stack (v0.2)
| Technology | Version | Purpose |
@@ -26,6 +28,8 @@ Nextcraft is a TypeScript monorepo (pnpm workspaces + turborepo) with a Next.js
| pydantic | 2.13.x | Request/response models, structured outputs |
| pydantic-settings | 2.15.x | Settings + env-file loading (replaces python-dotenv) |
| httpx | 0.28.x | Async LLM HTTP client (ollama-cloud + local providers) |
| sqlmodel / sqlalchemy | 0.0.24 / 2.x | Typed SQLite persistence for the v0.3 engine stores (D-027) |
| python-multipart | 0.0.x | Multipart audio upload for the defense answer route (REQ-3-006) |
| sse-starlette | 3.4.x | SSE framing, ping keep-alive |
| pytest | 9.x | Test runner |
| pytest-asyncio | 1.4.x | Async tests (auto mode) |
@@ -45,6 +49,15 @@ Deliberately **not** used: openai-python SDK (the `LLMProvider` protocol is the
7. **D-022 Monorepo integration** — zero-dependency shim `package.json` in apps/ai-service + `ai#*` turbo passthrough tasks (`cache:false, outputs:[]`) + root `ai:dev`/`ai:test` scripts + idempotent venv bootstrap.
8. **D-023 Testing** — pytest-asyncio auto mode; TestClient `client.stream()` for SSE; httpx MockTransport for byte-exact provider parser tests; scripted mock provider incl. failure modes. Tests never call the cloud.
### v0.4 Architecture Decisions (from Research — Distribution & Bootstrap CLI)
18. **D-033 Binary toolchain = Node SEA (probe-verified)** — Go and Rust are absent from this box; node v24.15.0 ships SEA support (`--experimental-sea-config`, postject-free on linux via `cp node nextcraft && node sea-config` … blob injection with the system `dd`/`npx postject` if needed). CLI source lives in `apps/cli` (TypeScript, compiled to a single CJS bundle by esbuild, then SEA-injected into a copy of the node binary → `nextcraft-linux-x64`). Fallback if SEA breaks: python3 `zipapp` (3.11.2 available). No new toolchain deps beyond dev-scoped esbuild.
19. **D-034 CLI = thin wrapper, orchestration stays in scripts/**`nextcraft` composes `apps/ai-service/scripts/bootstrap.sh` and `scripts/dev.sh` equivalents via `spawn` with inherited stdio and timeout guards (A-202/A-209). doctor/bootstrap/verify implement only *checking* logic (prereqs, env template, health) — never re-implement installs. This keeps one source of truth for bootstrap semantics.
20. **D-035 Install path = repo raw `install.sh` + Gitea latest-release API** — the one-liner `curl -fsSL <forge>/coreci/nextcraft/raw/main/scripts/install.sh | bash` resolves `GET /api/v1/repos/coreci/nextcraft/releases/latest`, downloads the `nextcraft-linux-x64` + `nextcraft-linux-x64.sha256` assets, verifies sha256 (`shasum -a 256`), installs to `~/.local/bin` (PATH hint), and degrades to printed source-bootstrap instructions when no binary asset exists or the platform mismatches (A-203/A-204/A-206).
21. **D-036 Ongoing binaries = ship-workflow asset step** — the release pipeline (v0.3's `ShipWorkflow.createRelease` equivalent, executed as the ship step's asset stage) builds the binary + checksum and attaches both to every Gitea release from v0.4 onward (A-205). Token resolution stays `.env*`-only (D-006/D-014); binaries are linux x64 only for v0.4 (macOS arm64 deferred — unverifiable on this box).
22. **D-037 CLI package layout**`apps/cli` is a pnpm workspace package (`@nextcraft/cli`): `src/` (entry, commands/, checks/, lib/), `scripts/build-binary.mjs` (esbuild bundle → SEA inject), unit tests runnable via `pnpm --filter @nextcraft/cli test` (node:test, no new test framework). Root `package.json` gains `cli:*` passthrough scripts mirroring the `ai:*` pattern (D-022).
23. **D-038 Network mode (v0.3.5)** — dev binds 0.0.0.0 (`AI_HOST`, default 0.0.0.0, revert via 127.0.0.1); CORS + WS-origin gates read `AI_CORS_ORIGINS` (default `*` — any origin, safe only because credentials are never enabled; explicit comma list restricts); the web client derives the API base URL from the browser hostname at runtime (`engine-base-url.ts`: `NEXT_PUBLIC_AI_SERVICE_URL` override → `http://${window.location.hostname}:8420``localhost` server-side). Hotfix also fixes: SEA direct-run detection (`require("node:sea").isSea()` — argv shape differs by invocation style), installer honesty gate (silent `--version` = hard fail), bootstrap venv recovery (poisoned partial `.venv` removal + distro-specific `apt install python3.XX-venv` hint), and doctor venv-capability probe with bootstrap preflight.
### v0.3 Architecture Decisions (from Research — Credential Engines)
9. **D-024 Sandbox isolation = Linux namespaces via `unshare`** — per-learner sandbox runs as a subprocess entered into fresh user+mount+pid+network namespaces (`unshare --user --map-root-user --mount --pid --fork --net`). Probe-verified on this box: in-namespace uid=0, **network fully isolated** (0 interfaces), learner writes land in a per-sandbox directory; proc-remount not permitted here but not required. Chosen because no container runtime (docker/podman/bwrap/firejail) exists on the box and there is no sudo. A `SandboxBackend` protocol abstracts the spawner so a future containerd/runc backend can replace namespace-spawning without touching callers.
@@ -65,14 +78,14 @@ Deliberately **not** used: openai-python SDK (the `LLMProvider` protocol is the
| Component | Description | Boundaries | Depends On |
|-----------|-------------|------------|------------|
| `ai_service/main.py` | FastAPI app factory, lifespan (httpx client pool, provider factory), CORS (localhost only), /health | App entry | config, llm, agents, api |
| `ai_service/main.py` | FastAPI app factory, lifespan (httpx client pool, provider factory, SandboxManager + reaper loop, SQLite engine stores on app.state), CORS (localhost only, incl. PUT for file writes), /health | App entry | config, llm, agents, api, engines |
| `ai_service/config.py` | pydantic-settings Settings (env_prefix="AI_", env_file, SecretStr key) | Configuration only | None |
| `ai_service/api/` | Endpoints: chat.py (POST /v1/chat/stream, SSE), lab.py, assessment.py, proctor.py, mentor.py; deps.py (DI) | Composes agents + sessions; never imported by llm/ or agents/ | agents, llm |
| `ai_service/api/` | Endpoints: chat.py (POST /v1/chat/stream, SSE), lab.py, assessment.py (POST /v1/assessment/evaluate v0.2 + POST /v1/assessment/grade v0.3), proctor.py, mentor.py, sandboxes.py (lifecycle + files/exec routes, G-5 abuse gates), telemetry.py (WS ingest + trace/gaps reads), variants.py (seeded per-learner variants), defense.py (defense loop, REQ-3-006); deps.py (DI) | Composes agents + sessions + engines; never imported by llm/ or agents/ | agents, llm, sandbox, telemetry, grading, variants, voice |
| `ai_service/llm/` | types.py (Message; ChatDelta/ChoiceDelta removed in P3 — no consumers), base.py (LLMProvider protocol), openai_compat.py (ollama-cloud + local), mock.py (deterministic), factory.py | Never imports agents/ or api/ | config |
| `ai_service/agents/` | base.py (BaseAgent ABC), registry.py, session.py (SessionStore), structured.py (JSON defense), coach/tutor/lab/assessor/proctor/mentor.py | Never imports api/ | llm, prompts, corpus |
| `ai_service/agents/` | base.py (BaseAgent ABC), registry.py, session.py (SessionStore), structured.py (JSON defense), coach/tutor/lab/assessor/proctor/mentor.py + examiner.py (seventh agent, v0.3) | Never imports api/ | llm, prompts, corpus, telemetry |
| `ai_service/prompts/` | Per-agent system prompt constants + render_context functions (str.format_map) | Data only | None |
| `ai_service/corpus/` | Mock engine inputs: learner_context.py, telemetry.py (Lab/Proctor scenarios), artifacts.py (pre-baked artifacts, rubrics, transcripts) | Pydantic-typed; aligned with TS packages/mock-data by convention | None |
| `scripts/` | bootstrap.sh (venv + pip install idempotent), dev.sh (exports keys from .ciagent/.env.secrets → uvicorn), test.sh (pytest), lint.sh (ruff check, G-3) | Dev entry points | pyproject.toml |
| `ai_service/corpus/` | Mock engine inputs: learner_context.py, telemetry.py (Lab/Proctor scenarios), artifacts.py (pre-baked artifacts, rubrics, transcripts). Since v0.3 P6 these are DORMANT, test-only fixtures (dormant-header noted) — the live learner path uses real engine inputs | Pydantic-typed; aligned with TS packages/mock-data by convention | None |
| `scripts/` | bootstrap.sh (venv + pip install idempotent), dev.sh (exports keys from .ciagent/.env.secrets → uvicorn), test.sh (pytest), lint.sh (ruff check, v0.2 G-3) | Dev entry points | pyproject.toml |
| `tests/` | conftest.py (mock provider, settings override, TestClient), health, llm (MockTransport parser), agents (framework + per-agent), api (SSE stream tests) | Mock provider only — no cloud | all |
**Module boundary rules:** `llm/` never imports `agents/` or `api/`; `agents/` never imports `api/`; `api/` composes both via DI. `corpus/` is the only home of mock engine data. Prompts are code — versioned and reviewed in git.
@@ -85,32 +98,46 @@ Deliberately **not** used: openai-python SDK (the `LLMProvider` protocol is the
| `ai_service/telemetry/` | `models.py` (TelemetryEvent, TraceSpan), `store.py` (TraceStore protocol + SQLite impl D-027), `ingest.py` (WebSocket /v1/telemetry/ingest, seq gap detection D-026) | Persistence; never imports agents/ | config |
| `ai_service/grading/` | `features.py` (deterministic trace digest D-028), `engine.py` (rubric scoring orchestration), `store.py` (GradeStore) | LLM only via digest; never sees raw trace | llm, telemetry, prompts |
| `ai_service/variants/` | `templates.py` (task template library), `generator.py` (seeded LLM instantiation D-029), `store.py` (VariantStore) | LLM via structured output | llm, grading |
| `ai_service/voice/` | `base.py` (VoiceProvider protocol D-030), `openai_audio.py` (STT/TTS vs compatible endpoint), `browser.py` (native SR/TTS fallback descriptor), `mock.py` (deterministic) | Never imports agents/ or api/ | config |
| `ai_service/voice/` | `base.py` (VoiceProvider protocol D-030), `browser.py` (native SR/TTS fallback descriptor), `mock.py` (deterministic; the real server STT/TTS provider is the v0.4 seam — GRILL CUT-1/G-7), `factory.py` (provider selection), `defense_store.py` (DefenseStore: transcripts + integrity signals, D-027) | Never imports agents/ or api/ | config |
| `ai_service/agents/examiner.py` | Seventh agent: oral defense examiner; streams over existing SSE, consumes process traces + emits integrity signals | reuses BaseAgent (D-018) | llm, prompts, telemetry |
| `ai_service/data/*.db` | SQLite databases (telemetry/grades/variants/defenses) | gitignored | — |
| `scripts/sandbox-agent.py` | Tiny in-namespace capture process shipped into the sandbox; streams telemetry to ingest | standalone | stdlib only |
**Boundary additions:** `sandbox/`, `telemetry/`, `grading/`, `variants/`, `voice/` are engine modules — they never import `api/` (which composes them via DI) and never import `agents/` (agents call engines through narrow interfaces, not vice versa).
### apps/cli — Nextcraft Bootstrap CLI (v0.4 NEW)
| Component | Description | Boundaries | Depends On |
|-----------|-------------|------------|------------|
| `src/index.ts` | Entry: arg parsing (no deps beyond node stdlib at runtime), command dispatch, `--help`/`--version`, exit-code contract (0 ok / 1 failure / 2 usage) | CLI surface only | commands/ |
| `src/commands/` | `doctor.ts` (prereq checks + actionable errors), `bootstrap.ts` (pnpm install + scripts/bootstrap.sh wrapper + env template copy + key validation), `verify.ts` (health: venv imports, ports, env, build readiness), `dev.ts` (thin passthrough to scripts/dev.sh) | Compose checks/ + lib/; spawn scripts — never re-implement them | checks/, lib/ |
| `src/checks/` | Pure check functions: `check-command.ts` (binary-on-PATH + version compare), `check-env.ts` (template diff, required/optional key classification) | Pure logic, unit-testable, no fs side effects at import | None |
| `src/lib/` | `spawn.ts` (subprocess with timeout + inherited stdio), `log.ts` (✓/✗/warn output formatter) | Shared utilities | None |
| `scripts/build-binary.mjs` | esbuild → CJS bundle → Node SEA injection → `dist/nextcraft-linux-x64` + sha256 sidecar | Build-time only | esbuild (dev dep) |
| `scripts/install.sh` | The one-liner install script served from repo raw: Gitea latest-release resolve → download + checksum verify → ~/.local/bin; source-bootstrap fallback | Standalone POSIX sh | forge API |
| `tests/` | node:test unit tests: command dispatch, check logic, env template diff, install-script shellcheck-style assertions | Fixtures only — never mutate repo state | src/ |
**Boundary rules:** the CLI never imports from `apps/web`, `packages/*`, or `ai_service` Python modules — it orchestrates them exclusively via subprocess/filesystem. Runtime deps: node stdlib only (no runtime npm deps; esbuild is dev-only). The binary embeds the bundle; `scripts/bootstrap.sh` remains the single source of bootstrap truth (D-034).
### apps/web — Next.js Application
| Component | Description | Boundaries | Depends On |
|-----------|-------------|------------|------------|
| `app/(learner)/` | Learner surface route group: landing, catalog, competency stack, dashboard, byte viewer, sandbox mockup, assessment mockup | Learner-only routes and layouts | packages/ui, packages/mock-data, packages/types |
| `app/(learner)/` | Learner surface route group: landing, catalog, competency stack, dashboard, byte viewer, build surface (`/build/[competencyId]` — real in-browser build), defense surface (`/defend/[competencyId]` — live oral defense + grading) | Learner-only routes and layouts | packages/ui, packages/mock-data, packages/types |
| `app/(marketplace)/` | Marketplace surface route group: job board, job detail, employer profile, search/filter, pricing | Marketplace-only routes and layouts | packages/ui, packages/mock-data, packages/types |
| `app/(employer)/` | Employer dashboard route group: overview, talent search, candidate profile, posting management | Employer-only routes and layouts | packages/ui, packages/mock-data, packages/types |
| `app/(admin)/` | Admin surface route group: overview, learner management, competency graph viewer, moderation | Admin-only routes and layouts | packages/ui, packages/mock-data, packages/types |
| `app/layout.tsx` | Root layout: theme provider, navigation shell, responsive container | All routes | packages/ui |
| `components/` | Surface-specific components (learner/, marketplace/, employer/, admin/) plus shared chrome (navigation-shell, header/footer, role-switcher, theme-provider, breadcrumbs, dark-mode-toggle) | App-level components | packages/ui |
| `hooks/` | use-chat-stream.ts — SSE client hook: fetch + ReadableStream, byte buffering + frame reassembly, idempotent AbortController cleanup | Client components only | ai-service SSE |
| `lib/` | sse.ts (shared SSE frame parser — CRLF normalization + `: ping` immunity, G-1), breadcrumbs.ts, format.ts | Pure utilities | None |
| `components/` | Surface-specific components (learner/, marketplace/, employer/, admin/) plus shared chrome (navigation-shell, header/footer, role-switcher, theme-provider, breadcrumbs, dark-mode-toggle); v0.3 learner: build-surface, sandbox-terminal (read-only exec output), defense-session | App-level components | packages/ui |
| `hooks/` | use-chat-stream.ts — SSE client hook: fetch + ReadableStream, byte buffering + frame reassembly, idempotent AbortController cleanup; use-sandbox-session.ts (v0.3) — sandbox lifecycle for the build session: create on task open, destroy on unmount, mid-start failure cleanup, 503/403/429 honest surfaces | Client components only | ai-service SSE / engine API |
| `lib/` | sse.ts (shared SSE frame parser — CRLF normalization + `: ping` immunity, v0.2 G-1), breadcrumbs.ts, format.ts, engine-base-url.ts (v0.3.5: runtime API base — env override → browser hostname → localhost), engine-client.ts (v0.3: typed fetch client for /v1/sandboxes, files/exec, variants, grade, defense, traces) | Pure utilities | None |
### packages/ui — Shared Component Library
| Component | Description | Boundaries | Depends On |
|-----------|-------------|------------|------------|
| `tokens/` | Design tokens as TS constants: colors, spacing, radii, shadows, breakpoints (mirrored as Tailwind v4 `@theme` tokens in apps/web globals.css) | Foundation layer — no dependencies | None |
| `primitives/` | Button, Input, Card, Badge, Avatar — each with a Storybook story | Atomic UI components | tokens, packages/types |
| `primitives/` | Button, Input, Card, Badge, Avatar (v0.1) + TerminalFrame, TelemetryStatus, MicControl, GradeBadge, TranscriptViewer (v0.3 build/defense surfaces) — each with a Storybook story | Atomic UI components | tokens, packages/types |
Composite/layout/theme components (navigation shell, tables, chat panels, graph viewer, theme provider) live in `apps/web/components/` as app-level components, not in packages/ui.
@@ -134,11 +161,42 @@ Composite/layout/theme components (navigation shell, tables, chat panels, graph
| `marketplace.ts` | Job, Employer, Candidate, JobPosting, TalentMatch, SearchFilter | Marketplace types | None |
| `user.ts` | Learner, Admin, EmployerUser, AgeGroup, Role | User types | None |
| `ui.ts` | Component props, theme config, breakpoint definitions | UI types | None |
| `telemetry.ts` | TelemetryEvent/ExecResult wire shapes for the live build surface (v0.3) | Engine types | None |
| `variants.ts` | Variant/TaskTemplate shapes for per-learner task statements (v0.3) | Engine types | None |
| `grading.ts` | GradeRecord/RubricScore shapes for live grading display (v0.3) | Engine types | None |
| `defense.ts` | DefenseSession/transcript/integrity-signal shapes for the defense surface (v0.3) | Engine types | None |
---
## Data Flow
### v0.3 credential flow (current)
```
[learner build surface /build/*] [learner defense surface /defend/*]
file CRUD + Run/Test (HTTP) mic MediaRecorder / typed + TTS playback
│ │
▼ ▼
[api/sandboxes files/exec] ──exec──▶ [namespace sandbox] [api/defense start/answer/finish]
│ │ capture agent │
│ ▼ (WS telemetry) ▼
│ [api/telemetry ingest] [DefenseStore (SQLite)]
│ │ SQLite │ transcript + integrity signals
│ ▼ │
│ [TraceStore] ────▶ [GradingEngine: digest (grading/features)
│ │ + rubric LLM (D-028)] ──▶ [GradeStore]
│ ▼ ▼
└──▶ Lab agent (live digest) Assessor (grade output) / Proctor (integrity)
Examiner agent (SSE) ◀── defense sessions
variants: [api/variants] ◀── [VariantStore (seeded, D-029)] ── per-learner task statements
```
- Lab consumes the live trace digest; Assessor consumes grading-engine output; Proctor consumes telemetry + defense integrity signals (REQ-3-007) — no mock fallback in the learner path (v0.2 corpus scenarios are dormant test-only fixtures).
- The learner's path is: variant task → in-sandbox build (telemetry streams to SQLite) → grade My Work (rubric scores from the real trace) → oral defense → verdict.
- Flooded/gapped traces are terminal: ingest closes 1008 and marks INCOMPLETE_FLOODED (G-3); the grader returns UNGRADABLE_TRACE_INCOMPLETE (G-4) — no credential from an incomplete trace.
### v0.2 chat flow (complete, still live)
```
[packages/mock-data + packages/types] [ai_service/corpus]
│ (TS, web surfaces) │ (Python, agent inputs)
@@ -153,21 +211,27 @@ Composite/layout/theme components (navigation shell, tables, chat panels, graph
(https://ollama.com/v1)
```
- Web surfaces remain server-component-first; client components (chat, filters, graph viewer, dark mode toggle) fetch directly from ai-service over SSE (A-002: no Next.js API-route proxy in v0.2).
- The LLM provider layer is a dumb pipe — OpenAI-compatible chunks pass through byte-identical; envelope logic (meta/done/error) lives only in the API layer (D-016).
- Lab/Assessor/Proctor read mock scenarios from `ai_service/corpus/` — real engines are v0.3+.
- Web surfaces remain server-component-first; client components (chat, filters, graph viewer, dark mode toggle) fetch directly from ai-service over SSE (A-002: no Next.js API-route proxy).
- The LLM provider layer is a dumb pipe — OpenAI-compatible chunks pass through byte-identically; envelope logic (meta/done/error) lives only in the API layer (D-016).
- The v0.2 corpus scenarios (`ai_service/corpus/`) are retained as dormant, test-only fixtures (dormant-header noted); they are no longer inputs to the live learner path.
- All automated tests use the deterministic mock provider; the cloud is for manual probes only.
---
## Build Order (v0.3)
## Build Order (v0.4)
1. **Bootstrap CLI core** — apps/cli package: doctor checks (node/pnpm/python3/git/unshare), bootstrap wrapper (pnpm install + scripts/bootstrap.sh + .env template + key validation), verify health check, dev passthrough; unit tests
2. **Binary build + release pipeline** — esbuild bundle → Node SEA binary (`nextcraft-linux-x64`) + sha256 sidecar; install.sh one-liner (Gitea latest-release resolve + checksum verify + PATH install); release-asset upload wired into the ship flow (ongoing binaries from v0.4 onward)
3. **Install docs + fresh-clone E2E** — README quickstart (one-liner → doctor → bootstrap → dev), CLI reference, fresh-clone end-to-end test proving a clean clone reaches a running stack
## Build Order (v0.3 — complete)
1. **Sandbox fabric** — SandboxBackend protocol + unshare namespace spawner + lifecycle manager (create/list/snapshot/destroy) + concurrency guard + per-sandbox workdir; isolation + resource-limit probes
2. **Live build telemetry** — TelemetryEvent models + SQLite TraceStore + WebSocket ingest endpoint + seq gap detection + in-sandbox capture agent
3. **Process-trace grading engine** — deterministic feature/digest computation + rubric scoring via LLM structured output + GradeStore; calibrated against v0.2 mock corpora
4. **Variant task generation** — template library + seeded LLM instantiation + VariantStore + difficulty normalization anchors
5. **Oral / voice defense** — VoiceProvider protocol + STT/TTS + mock + browser fallback + Examiner agent + transcript/integrity-signal capture
6. **Agent re-grounding + learner surface integration** — Lab/Assessor/Proctor consume real telemetry/grades/defense signals; learner sandbox mockup → real in-browser xterm.js build/run; assessment mockup → live defense + live grading
6. **Agent re-grounding + learner surface integration** — Lab/Assessor/Proctor consume real telemetry/grades/defense signals; learner sandbox mockup → real in-browser build/run (Run/Test buttons executing in a namespace sandbox, read-only exec-output panel — no interactive shell, CUT-2/G-8); assessment mockup → live defense + live grading
---
@@ -184,16 +248,19 @@ The v0.1 build order (monorepo → types → mock data → tokens → primitives
---
## Future Architecture (Post-v0.3, for reference)
## Future Architecture (Post-v0.4, for reference)
v0.3 delivers the real credential engines; later milestones fill in the remaining platform:
v0.4 delivers distribution (CLI + binary releases); later milestones fill in the remaining platform:
- **In-memory sessions → PostgreSQL + Drizzle/SQLModel** — SessionStore + v0.3 TraceStore/GradeStore/VariantStore/DefenseStore protocols swap SQLite→Postgres with no API changes
- **userns subprocess sandboxes → containerd/runc backend** — D-024 `SandboxBackend` protocol swap; same lifecycle API
- **Coding-IDE sandbox → design tool + simulation environments** — REQ-F-021 full scope (v0.4)
- **Coding-IDE sandbox → design tool + simulation environments** — REQ-F-021 full scope (v0.5)
- **Mock provider → per-agent model routing** — provider factory already selects by config; per-agent `AI_<AGENT>_MODEL` overrides
- **No auth → real KYC + sessions** — **deferred per founder directive**; REQ-F-017 identity/age-gating lands post-v0.3 (v0.4+). Age-gating remains the v0.1 visual flow mockup
- **No auth → real KYC + sessions** — **deferred per founder directive; moved to v0.5 with D-016**; REQ-F-017 identity/age-gating lands post-v0.4. Age-gating remains the v0.1 visual flow mockup
- **Mock voice → real server STT/TTS (openai-audio provider)** — CUT-1/G-7 seam moved to v0.5 per D-016; VoiceProvider protocol is the drop-in point
- **linux x64 binary → macOS arm64 + auto-update** — D-036 defers non-linux targets (unverifiable on this box); `nextcraft upgrade` (self-replace from latest release) is the natural v0.5+ follow-up
- **Exec-telemetry seq-lease / replay-margin fix** — the P6-lesson one-line ACK gap moves to v0.5 per D-016
- **No search → Semantic vector search (pgvector)** — Filter UI replaced with vector similarity search
- **No payments → Payment processing** — Pricing page replaced with real subscription/payment flows
The monorepo structure (apps/web + apps/ai-service + packages/*) accommodates further apps without restructuring.
The monorepo structure (apps/web + apps/ai-service + apps/cli + packages/*) accommodates further apps without restructuring.
-8
View File
@@ -1,8 +0,0 @@
{
"phase": 5,
"stage": "verify",
"milestone": "v0.3",
"phase_role": "execution",
"attempts": 0,
"updated_at": "2026-09-12T04:51:45Z"
}
+38
View File
@@ -46,3 +46,41 @@ The credential-pipeline architecture (telemetry → trace → grade → defense)
## Outcome
**GO** — all six binding decisions and both scope cuts applied to PLAN.md / REQUIREMENTS.md / ROADMAP.md / PROJECT.md before Phase 1 execution. No axis requires escalation (all resolvable at confidence ≥ 0.85). The milestone no longer claims resource enforcement it cannot deliver, and the no-auth abuse vector is closed at MVP scale.
---
# Nextcraft v0.4 — GRILL.md (Adversarial Review Verdict)
**Stage:** GRILL, Phase 0 pre-execution · **Verdict:** GO-WITH-CHANGES · **Confidence:** 0.83
## Summary
The distribution milestone is small, founder-directed (D-016, confidence 0.99), and additive (zero changes to the running credential pipeline). The plan's central risk: **Node SEA was probe-verified as a flag, not as a working build** — the v0.3 lesson (A-101: probe the mechanism, not the existence) applies. Second gap: a binary whose `--version` lies (stale package.json) would poison the "ongoing binaries" contract. Third: sed-based JSON parsing in install.sh is a fragility + integrity risk. Fourth: "ongoing binaries" has no enforcement mechanism beyond prose. All four closed by binding decisions G-101..G-104 below. No scope cuts required — the milestone is already minimal.
## Per-Axis Findings
| Axis | Verdict | Rationale |
|------|---------|-----------|
| Business case | PASS | Founder directive explicit + recorded (D-016). Evidence of need: live Gitea probe shows latest release v0.2.8 with ZERO assets; bootstrap requires repo archaeology (scripts found only via package.json spelunking). |
| Scope | PASS | 5 REQs, 3 execution phases, one focused surface (apps/cli + scripts). Smallest milestone yet. macOS arm64 already cut (D-036, unverifiable here). |
| Feasibility | CONCERN (fixed) | SEA flag exists on node v24.15.0, but no end-to-end SEA binary was built during RESEARCH. postject availability assumed (`npx postject` — needs npm registry reachability, unproven). Zipapp fallback requires python3 on target — an honest-degradation ladder, not a silent downgrade. → G-101. |
| Honest versioning | CONCERN (fixed) | `--version` from package.json would print a stale hardcoded version inside a per-release binary — breaks upgrade detection + the one-liner's re-run-to-upgrade promise. → G-102. |
| Install integrity | CONCERN (fixed) | sed/grep JSON parsing is brittle; a parse failure must never fall through to installing an unverified artifact. Exact asset-name matching + hard-degrade to source instructions. → G-103. |
| Sequencing | PASS | P1 CLI (source-runnable) → P2 binary+pipeline → P3 docs+E2E matches dependency order; each phase ships independently. |
| Cost/quota | PASS | Zero new paid infra; binaries built on-box; Gitea releases free. Dev-only esbuild dep. |
| Risks | CONCERN (fixed) | Top 3: SEA end-to-end (→ G-101 live probe FIRST in P2), npm registry reachability for esbuild (→ proven by P1's pnpm install must-have), Gitea asset-upload token scope (→ live-proven at the v0.3.2 ship itself). |
| Adoption/operability | PASS | Consumer = founder + future pilots; one command replaces README archaeology. Rollback trivial (rm ~/.local/bin/nextcraft). No server changes. |
## Binding Decisions (applied to PLAN.md)
- **G-101 (BINDING) — SEA live-build probe is the FIRST P2 action.** Task 2-1-01 builds a real binary before anything depends on it; the build script encodes the fallback ladder explicitly (SEA → zipapp with "requires python3" honesty). If SEA fails on this box, zipapp becomes primary with the docs stating the requirement — no silent claim of node-less operation.
- **G-102 (BINDING) — Version stamping at build time.** `build-binary` accepts the shipping tag and stamps it into the bundle (`NEXTCRAFT_VERSION` replace); `--version` prints it; install E2E asserts the installed binary reports the tag it was downloaded from. A binary may never report a version it was not built as.
- **G-103 (BINDING) — Install-script integrity hard-degrade.** install.sh matches assets by EXACT name (`nextcraft-linux-x64`, `nextcraft-linux-x64.sha256`); any parse/lookup/download failure degrades to source-bootstrap instructions (exit 0) — never installs unverified or name-approximate artifacts. Checksum mismatch = hard stop, exit 1, explicit do-not-run message. dash-safe POSIX sh, no jq.
- **G-104 (BINDING) — Ongoing-binaries enforcement.** Every ship from v0.3.2 onward MUST run `scripts/release-assets.sh <tag>` after tag+merge (best-effort, non-blocking, `release_pending` escalation on failure — but attempted + logged every release). The final-phase audit gate includes "milestone release carries both assets" as a check. This makes the founder's "ongoing binaries" directive a pipeline property, not prose.
## Escalations
None. All four concerns resolved at confidence ≥ 0.85. No axis requires founder escalation (directive already explicit).
## Outcome
**GO** — G-101..G-104 applied to PLAN.md before Phase 1 execution. The milestone claims only what its probes prove, and the ongoing-binaries contract has an enforcement mechanism.
+107 -138
View File
@@ -2,19 +2,19 @@
## Persona Roster
> **v0.3 update (RESEARCH, lead-developer assessment):** backend-engineer territory extended to the new engine modules (telemetry/grading/variants persistence + APIs). New phase-relevant custom personas added: **sandbox-engineer** (Linux-namespace isolation infra) and **voice-engineer** (STT/TTS + Examiner agent audio pipeline). ai-engineer re-scoped to LLM/agents/prompts + grading/variant/voice *model-facing* logic. **security-auditor stays inactive** (KYC deferred per founder directive). frontend-engineer gains real-sandbox (xterm.js), live-telemetry, and live-defense surfaces.
> **v0.4 update (RESEARCH, lead-developer assessment):** milestone pivoted to Distribution & Bootstrap CLI (founder directive D-016). New custom persona **cli-engineer** (domain `cli`) owns apps/cli end-to-end: doctor/bootstrap/verify/dev commands, checks, spawn wrappers, the SEA binary build, the one-liner install script, and the release-asset pipeline. backend-engineer retains the scripts/ + turbo/root-package integration surface. **sandbox-engineer and voice-engineer deactivated** (their v0.3 code is complete and untouched this milestone — reason fields below). ai-engineer light-touch (no model-facing work in v0.4). **security-auditor re-activated (phase-specific)** for the install pipeline: curl|bash attack surface, checksum trust, PATH writes, secrets handling in the release flow. frontend-engineer/design-system-engineer/data-engineer inactive (zero UI/data-scope tasks in v0.4 — retained below with reasons).
### lead-developer
```yaml
active: true
phase_specific: false
reason: Coordinates task decomposition across web, AI service, engine, and sandbox territories; resolves conflicts between frontend, backend, AI, sandbox, and voice personas
reason: Coordinates task decomposition across CLI, scripts, release-pipeline, and docs territories; resolves cli-engineer/backend-engineer boundary (scripts vs CLI)
domain: coordination
frameworks:
- next.js
- turborepo
- pnpm
- fastapi
- node
constraints:
- pragmatic
- battle-tested defaults
@@ -27,99 +27,83 @@ territory:
- "apps/ai-service/pyproject.toml"
```
### frontend-engineer
### cli-engineer
```yaml
active: true
phase_specific: false
reason: Phase 6 real-engine learner-surface integration — xterm.js in-browser terminal, file-tree/run/test controls, live telemetry panels, live voice defense UI, live grading display. Owns all page components, layouts, surface-specific UI.
domain: frontend
frameworks:
- react
- next.js
- tailwindcss
- lucide-react
- recharts
- react-flow
- "@xterm/xterm"
- "@xterm/addon-fit"
constraints:
- component-first
- server-components-default
- minimal-client-js
- sse-client-buffering (buffer bytes, split frames on \n\n, join data: lines)
- abortcontroller-cleanup (idempotent abort in effect cleanup)
- websocket-lifecycle (typed messages, reconnect backoff, cleanup)
- mediarecorder-permission-ux (mic consent, graceful no-mic fallback)
- responsive-all-breakpoints
- dark-mode-support
territory:
- "apps/web/**"
- "packages/ui/**"
- "packages/mock-data/**"
- "packages/types/**"
```
### data-engineer
```yaml
active: true
phase_specific: false
reason: Owns TS mock data layer schema and typed definitions + TS types for telemetry/trace/grade/variant/defense shapes the web surfaces consume. Does NOT own the Python corpus or engine stores — aligned by convention (D-021).
domain: data
reason: v0.4 custom persona (RESEARCH) — owns the distribution milestone core: nextcraft CLI (doctor/bootstrap/verify/dev), pure check logic, spawn wrappers with timeouts, Node SEA binary build (D-033), one-liner install.sh (D-035), checksum sidecar, and Gitea release-asset upload (D-036)
domain: cli
frameworks:
- node
- typescript
- node:test
- esbuild
- node-sea
- posix-sh
constraints:
- schema-first
- type-safe
- migration-ready
- mock-data-only
- stdlib-only-runtime (no runtime npm deps; esbuild dev-only)
- thin-wrapper (never re-implement scripts/bootstrap.sh or dev.sh — compose via spawn, A-202/A-209)
- timeout-every-spawn (no unbounded subprocess)
- actionable-errors (every failed check tells the user how to fix it)
- graceful-degradation (install never hard-fails; source-bootstrap fallback, A-206)
- checksum-before-install (sha256 verify before chmod+install, A-207)
- secrets-never-in-cli (no key generation; .env.example -> .env copy only, A-210)
- fail-loud-exit-codes (0 ok / 1 failure / 2 usage)
territory:
- "packages/types/**"
- "packages/mock-data/**"
- "apps/cli/**"
- "scripts/install.sh"
- "scripts/release-assets.sh"
```
### backend-engineer
```yaml
active: true
phase_specific: false
reason: Owns apps/ai-service app shell, config, API endpoints (incl. WebSocket telemetry ingest), engine persistence (SQLite stores), scripts, and test harness. Extended for v0.3 engine modules.
reason: Owns the script + monorepo integration surface the CLI composes: apps/ai-service/scripts/*, root package.json cli:* passthrough scripts, turbo task wiring (D-037/D-022). Python ai-service itself is untouched this milestone (v0.3 complete).
domain: backend
frameworks:
- fastapi
- uvicorn
- pydantic
- pydantic-settings
- httpx
- pytest
- sqlmodel
- sqlalchemy
- websockets
- aiofiles
- bash
- turborepo
- pnpm
constraints:
- provider-agnostic-boundaries (engine modules import nothing from agents/ or api/)
- streaming-first
- sqlite-first-persistence (protocol-wrapped stores, Postgres-ready, D-027)
- secrets-via-env-only
- mock-provider-in-tests
- websocket-contract (typed envelopes, seq gap detection, D-026)
- scripts-are-truth (bootstrap.sh/dev.sh stay the single source of bootstrap orchestration; CLI only wraps)
- idempotent-scripts (re-runnable without side effects)
- secrets-via-env-only (D-014; dev.sh exports from .ciagent/.env.secrets)
territory:
- "apps/ai-service/ai_service/main.py"
- "apps/ai-service/ai_service/config.py"
- "apps/ai-service/ai_service/api/**"
- "apps/ai-service/ai_service/telemetry/store.py"
- "apps/ai-service/ai_service/telemetry/ingest.py"
- "apps/ai-service/ai_service/grading/store.py"
- "apps/ai-service/ai_service/variants/store.py"
- "apps/ai-service/scripts/**"
- "apps/ai-service/package.json"
- "apps/ai-service/tests/api/**"
- "package.json"
- "turbo.json"
- ".gitignore"
```
### security-auditor
```yaml
active: true
phase_specific: true
reason: v0.4 re-activated (phase-specific) — the install pipeline is the first externally-consumed attack surface: curl|bash piping, latest-release resolution, checksum trust root, PATH writes to ~/.local/bin, download tempdir hygiene, release-asset upload token handling. No KYC/PII work (still v0.5).
domain: security
frameworks:
- posix-sh
- curl
- sha256sum
constraints:
- STRIDE-classified
- no-pipe-to-shell-without-checksum (download -> verify -> install order)
- tmpdir-safe (mktemp, no predictable paths, trap cleanup)
- token-never-echoed (release upload resolves .env* only, never logs)
territory:
- "scripts/install.sh"
- "scripts/release-assets.sh"
- "apps/cli/src/lib/spawn.ts"
```
### ai-engineer
```yaml
active: true
phase_specific: false
reason: Owns the LLM provider layer, agent framework, prompt library, structured outputs, and the model-facing logic of v0.3 engines — trace-digest→rubric grading prompts (grading/features.py+engine.py), variant instantiation (variants/templates.py+generator.py), and the Examiner agent. Owns the deterministic-mock corpora.
reason: Light-touch v0.4 — no model-facing work in the distribution milestone; retained to guard the CLI against touching agent/engine boundaries and to keep territory mappings accurate for v0.5 (voice real-path, seq-lease).
domain: ai
frameworks:
- pydantic
@@ -127,116 +111,101 @@ frameworks:
- pytest
constraints:
- provider-agnostic-protocol
- prompts-are-code
- json-defensive-parsing
- never-call-cloud-in-tests
- delta-passthrough
- llm-sees-digest-not-raw-trace (D-028)
- seeded-variant-reproducibility (D-029)
territory:
- "apps/ai-service/ai_service/llm/**"
- "apps/ai-service/ai_service/agents/**"
- "apps/ai-service/ai_service/prompts/**"
- "apps/ai-service/ai_service/corpus/**"
- "apps/ai-service/ai_service/grading/features.py"
- "apps/ai-service/ai_service/grading/engine.py"
- "apps/ai-service/ai_service/variants/templates.py"
- "apps/ai-service/ai_service/variants/generator.py"
- "apps/ai-service/tests/llm/**"
- "apps/ai-service/tests/agents/**"
```
### sandbox-engineer
### frontend-engineer
```yaml
active: true
phase_specific: true
reason: v0.3 custom persona (RESEARCH) — owns the sandbox fabric: SandboxBackend protocol, unshare-based Linux user/mount/pid/net namespace spawner, per-sandbox workdir, resource limits, lifecycle manager, concurrency guard, and the in-sandbox capture agent. Probe-verified isolation on this box (D-024).
domain: infra
active: false
phase_specific: false
reason: v0.4 has zero UI-scope work (no web/pages/components changes planned in the distribution milestone); v0.3 surfaces are complete. Reactivated at v0.5 when deferred UX work resumes.
domain: frontend
frameworks:
- python
- linux-namespaces
- asyncio
- pytest
- react
- next.js
- tailwindcss
constraints:
- isolation-verified (probe must show in-ns uid=0, network isolated, writes to workdir only)
- backend-protocol-swap (no containerd assumption; D-024)
- resource-limits-enforced (cpu/mem/time quotas observable)
- no-daemon (subprocess-only; no docker/containerd service)
- capacity-guard (1-5 concurrent; 503 when full, D-032)
- component-first
- server-components-default
territory:
- "apps/ai-service/ai_service/sandbox/**"
- "apps/ai-service/scripts/sandbox-agent.py"
- "apps/ai-service/tests/sandbox/**"
```
### voice-engineer
```yaml
active: true
phase_specific: true
reason: v0.3 custom persona (RESEARCH) — owns the voice layer: VoiceProvider protocol, STT/TTS against a compatible endpoint, deterministic mock (tests never call a voice API), browser-native fallback, and the media-path wiring consumed by the Examiner agent and assessment UI.
domain: ai-media
frameworks:
- pydantic
- httpx
- pytest
- web-mediarecorder
constraints:
- provider-agnostic-protocol (D-030)
- never-call-voice-api-in-tests
- browser-native-fallback (no-key path still functions)
- bounded-turn-latency (conversational feel budget)
territory:
- "apps/ai-service/ai_service/voice/**"
- "apps/ai-service/tests/voice/**"
- "apps/web/**"
- "packages/ui/**"
```
### design-system-engineer
```yaml
active: true
active: false
phase_specific: false
reason: Owns the shared component library, design tokens, and visual consistency. v0.3 duty: new primitives for the real build/assessment surfaces (terminal frame, telemetry status indicator, mic/record control, grade badge, defense transcript viewer).
reason: No design-token or primitive work in v0.4; roster retained for v0.5.
domain: frontend
frameworks:
- tailwindcss
- storybook
- lucide-react
constraints:
- design-token-driven
- wcag-aa-contrast
- dark-mode-required
- consistent-across-surfaces
territory:
- "packages/ui/**"
```
### security-auditor
### data-engineer
```yaml
active: false
phase_specific: false
reason: Identity/age-gating (KYC) deferred beyond v0.3 per founder directive (A-110) — no real auth or PII backend lands this milestone. Security coverage remains: verifier's STRIDE layer + Phase 7 secrets-hygiene checklist (keys absent from code/logs/commits/errors, localhost-only CORS, no PII in prompts). Sandbox isolation safety is owned by sandbox-engineer's probe-verified constraint.
domain: security
reason: No schema/mock-data work in v0.4; types packages untouched. Reactivated if CLI surfaces need shared types (not planned — CLI is self-contained).
domain: data
frameworks:
- typescript
constraints:
- schema-first
- type-safe
territory:
- "packages/types/**"
- "packages/mock-data/**"
```
### sandbox-engineer
```yaml
active: false
phase_specific: false
reason: v0.3 persona — sandbox fabric shipped complete (v0.2.x series); v0.4 touches no sandbox code. doctor only *checks* unshare availability; no sandbox logic changes. Reactivated at v0.5 (design/sim environments).
domain: infra
frameworks:
- python
- linux-namespaces
constraints: []
territory:
- "apps/ai-service/ai_service/sandbox/**"
```
### voice-engineer
```yaml
active: false
phase_specific: false
reason: v0.3 persona — voice defense shipped complete (mock-first, CUT-1); real server STT/TTS moved to v0.5 per D-016. No v0.4 voice work.
domain: ai-media
frameworks: []
constraints: []
territory: []
territory:
- "apps/ai-service/ai_service/voice/**"
```
## Phase-Specific Personas
| Persona | Phases | Removed After |
|---------|--------|---------------|
| sandbox-engineer | 1 (primary), 2, 6 | persists while sandbox fabric exists |
| voice-engineer | 5 (primary), 6 | persists while voice defense exists |
| security-auditor | 2 (primary: install pipeline), 3, 4 (final review) | milestone complete |
All other active personas span the entire milestone. data-engineer and design-system-engineer are light-touch outside their phases.
All other personas span the milestone. Deactivated personas receive no tasks.
## Territory Conflict Resolution
| Conflict | Resolution |
|----------|------------|
| frontend-engineer vs data-engineer (packages/types, packages/mock-data) | data-engineer owns type definitions and mock data schema (incl. new telemetry/grade/variant/defense TS types); frontend-engineer consumes them. |
| frontend-engineer vs design-system-engineer (packages/ui) | design-system-engineer owns design tokens and primitive components (terminal frame, mic control, grade badge); frontend-engineer owns composite components and page-level UI. |
| ai-engineer vs backend-engineer (grading/variants) | ai-engineer owns the model-facing files (features/engine/templates/generator = LLM logic + prompts); backend-engineer owns the persistence stores + API endpoints. Boundary: stores are pure SQLite; engine logic is pure compute. |
| sandbox-engineer vs backend-engineer (sandbox/) | sandbox-engineer owns `ai_service/sandbox/**` + capture agent; backend-engineer owns the API route that composes `sandbox/manager.py` via DI. manager.py has a narrow typed interface consumed by api/. |
| voice-engineer vs ai-engineer (Examiner agent) | ai-engineer owns `agents/examiner.py` + its prompt; voice-engineer owns `voice/**` (audio in/out). Examiner calls `voice/` through the `VoiceProvider` protocol — never imports concrete providers. |
| ai-engineer vs data-engineer (mock duplication) | ai-engineer owns `ai_service/corpus/` (Python); data-engineer owns `packages/mock-data` (TS). Shared IDs/shapes aligned by convention (D-021). |
| lead-developer vs any | lead-developer coordinates only, does not directly modify code files. |
| cli-engineer vs backend-engineer (scripts/) | backend-engineer owns `apps/ai-service/scripts/**` + root `package.json`/`turbo.json` wiring; cli-engineer owns `apps/cli/**` + top-level `scripts/install.sh` + `scripts/release-assets.sh` and *consumes* backend scripts via spawn — never edits them |
| cli-engineer vs security-auditor (install.sh) | cli-engineer implements; security-auditor reviews + may patch security defects directly in install.sh/spawn.ts (its territory) |
| lead-developer vs any | lead-developer coordinates only, does not directly modify code files |
+194 -422
View File
@@ -1,438 +1,210 @@
# Nextcraft v0.3 — PLAN.md
# Nextcraft v0.4 — PLAN.md
## Overview
This plan covers execution phases 1-6 of milestone v0.3 (Credential Engines): the real engines that replace v0.2's mock inputs — a namespace-isolated sandbox fabric, live build telemetry over WebSocket + SQLite, a process-trace grading engine, seeded per-learner variant task generation, and an oral/voice defense with a seventh Examiner agent — plus re-grounding the Lab/Assessor/Proctor agents onto real engine inputs and wiring the v0.1 learner surfaces to the real build/defense/grading paths. Phases are strictly sequential (P1→P6); within each phase, Wave 1 tasks are parallelizable (no cross-file dependencies) and later waves depend on earlier ones.
This plan covers execution phases 13 of milestone v0.4 (Distribution & Bootstrap CLI) plus the final phase (P4 review+ship). The milestone delivers the founder directive (D-016): a streamlined install for Nextcraft — a `nextcraft` bootstrap CLI shipped as a linux x64 binary, installed via a one-liner script, with binaries published on **every ongoing release** from v0.4 onward. Phases are strictly sequential (P1→P3); within each phase, Wave 1 tasks are parallelizable (no cross-file dependencies) and later waves depend on earlier ones.
**Environment facts (apply throughout):** Python 3.11.2 via `python3 -m venv` (no uv, no system pip); pnpm 12.3.4 via corepack; turborepo; ai-service port **8420**; default model `gemma4:31b` (config via `AI_TUTOR_MODEL`); ollama-cloud base `https://ollama.com/v1` (OpenAI-compatible, Bearer auth); keys live only in gitignored `.ciagent/.env.secrets` (exported by `scripts/dev.sh`) — never in code, commits, or logs; all automated tests use the deterministic mock LLM and mock voice providers and **never call cloud or voice APIs**. New ai-service deps this milestone: `sqlmodel`, `sqlalchemy`, `websockets`, `aiofiles` (all PyPI-verified; added in Task 1-1-02). SQLite data (`apps/ai-service/ai_service/data/`) and sandbox dirs (`apps/ai-service/sandboxes/`) are already gitignored. **KYC/age-gating is deferred per founder directive (A-110)** — no identity work; age-gating stays the v0.1 visual flow mockup. **GRILL scope decisions (this revision):** real server STT/TTS (`OpenAIAudioProvider`) deferred to v0.4 — voice defense is mock+browser-first (CUT-1/G-7); the interactive xterm.js shell relay is deferred to v0.4 — the build panel is run/test-buttons + read-only exec output (CUT-2/G-8), so `@xterm/*` is NOT a v0.3 dependency; sandbox abuse control (per-learner caps + allowlist, G-5) ships even though KYC is deferred; sandbox resource limits are partially enforced (memory/CPU/wall-clock + workdir-size sweep; per-sandbox pids and hard disk quota are accepted gaps, G-1/G-2).
**Environment facts (probe-verified, apply throughout):** Go MISSING, Rust MISSING, gcc 12.2 present, **node v24.15.0 x64 linux (SEA-capable)**, python3 3.11.2, `shasum` 6.02, pnpm 12.3.4 via corepack, turborepo 2.3.3, tsx 4.23 in root devDeps path. Gitea API verified live at `https://git.coreci.dev/api/v1` (latest release v0.2.8, **zero assets** — the gap this milestone closes). Existing orchestration: `apps/ai-service/scripts/bootstrap.sh` (idempotent venv+pip incl. the no-ensurepip get-pip path), `apps/ai-service/scripts/dev.sh` (secrets export → uvicorn :8420), `apps/ai-service/.env.example` (full AI_* template). Root scripts: `ai:dev/ai:test/ai:bootstrap/ai:lint` turbo passthroughs (D-022 pattern to mirror as `cli:*`). Secrets live only in gitignored `.ciagent/.env.secrets` (GITEA_TOKEN, OLLAMA_API_KEY, OLLAMA_BASE_URL) — never in code, commits, or logs; tests never call the cloud or the forge (mocks/fixtures only).
**Milestone type:** feature. Tags: phase 0 → **v0.3.0**, P1 → v0.3.1, P2 → v0.3.2, P3 → v0.3.3, final phase P4 → **v0.3.4 = milestone release**. **GRILL binding decisions (this revision):** G-101 — SEA live-build probe is the FIRST P2 action (mechanism, not flag, must be proven); fallback ladder encoded honestly (zipapp requires python3 on target). G-102 — binary `--version` stamped from the shipping tag at build time (never a stale package.json version); install E2E asserts the installed binary reports its release tag. G-103 — install.sh matches assets by exact name; any parse/download failure degrades to source-bootstrap instructions (exit 0), never installs unverified artifacts; checksum mismatch = hard stop exit 1. G-104 — every ship from v0.3.2 onward runs `scripts/release-assets.sh <tag>` (best-effort, logged, non-blocking); the P4 audit gate checks the milestone release carries both assets.
| Phase | Name | Requirements | Waves | Personas |
|-------|------|-------------|-------|----------|
| 1 | Sandbox fabric | REQ-3-001, 002 | 3 | sandbox-engineer, backend-engineer, ai-engineer (W1 lint only) |
| 2 | Live build telemetry | REQ-3-003 | 4 | sandbox-engineer, backend-engineer, data-engineer |
| 3 | Process-trace grading engine | REQ-3-004 | 3 | ai-engineer, backend-engineer |
| 4 | Variant task generation | REQ-3-005 | 3 | ai-engineer, backend-engineer, data-engineer |
| 5 | Oral / voice defense | REQ-3-006 | 4 | voice-engineer, ai-engineer, backend-engineer |
| 6 | Agent re-grounding + learner surface integration | REQ-3-007, 008 | 5 | ai-engineer, frontend-engineer, design-system-engineer, data-engineer, backend-engineer, lead-developer |
---
## Phase 1: Sandbox Fabric
**Requirements:** REQ-3-001, REQ-3-002
**Goal:** `SandboxBackend` protocol + `unshare`-based namespace spawner (D-024) + lifecycle manager with concurrency guard (D-032) + per-sandbox workdir; isolation and resources probe-verified on this box; `/v1/sandboxes` API live; deps + gitignore landed
### Wave 1: Foundations (parallel — no shared files)
#### Task 1-1-01: SandboxBackend protocol + unshare spawner + probe test
- **Persona:** sandbox-engineer — **REQ:** REQ-3-001, REQ-3-002
- **Files:** `apps/ai-service/ai_service/sandbox/__init__.py`, `apps/ai-service/ai_service/sandbox/backend.py`, `apps/ai-service/ai_service/sandbox/workdir.py`, `apps/ai-service/ai_service/sandbox/unshare_backend.py`, `apps/ai-service/tests/sandbox/__init__.py`, `apps/ai-service/tests/sandbox/test_isolation.py`
- **Action:** `backend.py`: `SandboxBackend` protocol + `SandboxSpec` (sandbox_id, learner_id, workdir, resource limits) + `SandboxHandle` (id, pid, workdir, created_at); `spawn(spec)`, `exec(handle, cmd)`, `snapshot(handle) -> Path`, `destroy(handle)`. `workdir.py`: per-sandbox layout under `apps/ai-service/sandboxes/<id>/` (workspace/ writable, snapshot() = recursive copy to `snapshots/<ts>/`) — no symlinks as the snapshot mechanism. `unshare_backend.py`: subprocess spawner — `unshare --user --map-root-user --mount --pid --fork --net` with the per-sandbox dir bind-mounted (`--bind <dir> /work`) and `chdir /work` (D-024); pipes for stdout/stderr; async wrappers. `test_isolation.py`**re-verify box isolation properties (runs on this box, guarded by probe skip):** (a) `id -u` inside namespace prints `0`; (b) `ip link` inside namespace shows 0 usable interfaces (loopback-only/no carrier) — network isolated; (c) file written to `/work/inside.txt` lands at `sandboxes/<id>/workspace/inside.txt` on the host; (d) attempt to write outside the mount (e.g. host tmp path via bind) does not escape the per-sandbox dir; (e) `/proc` visibility degraded (proc-remount not permitted per A-101 — assert the probe documents this, not that it fails).
- **Verify:** `pnpm ai:test``tests/sandbox/test_isolation.py` green on this box (probe-gated: skips with an explicit reason if userns unavailable); `lint` clean
#### Task 1-1-02: v0.3 dependencies + gitignore + config additions
- **Persona:** backend-engineer — **REQ:** REQ-3-001
- **Files:** `apps/ai-service/pyproject.toml` (update), `apps/ai-service/ai_service/config.py` (update), `apps/ai-service/.env.example` (update), `apps/ai-service/scripts/bootstrap.sh` (update if needed), root `package.json` (no change), `turbo.json` (no change)
- **Action:** Add pinned deps: `sqlmodel`, `sqlalchemy`, `websockets`, `aiofiles` to pyproject. `config.py` additions (env_prefix `AI_`): `AI_SANDBOX_DIR` (default `apps/ai-service/sandboxes`), `AI_SANDBOX_MAX_CONCURRENT` (default 5, D-032), `AI_SANDBOX_CPU_LIMIT` (default 1 core / cpu.max), `AI_SANDBOX_MEM_LIMIT_MB` (default 512), `AI_SANDBOX_PIDS_LIMIT` (default 256), `AI_SANDBOX_TIMEOUT_S` (default 1800), `AI_DB_PATH` (default `ai_service/data/nextcraft.db`), `AI_VOICE_BASE_URL` / `AI_VOICE_API_KEY` / `AI_VOICE_STT_MODEL` / `AI_VOICE_TTS_MODEL` (all **optional**, default empty — mock-first, D-030; documented in `.env.example` and README). Confirm `.gitignore` already covers `ai_service/data/` + `sandboxes/` (it does — v0.3 block present). Re-run bootstrap idempotently to install new deps.
- **Verify:** `pnpm ai:bootstrap` re-installs cleanly (no-op venv, new wheels land); `python -c "import sqlmodel, sqlalchemy, websockets, aiofiles"` succeeds in the venv; settings parse with new keys unset
#### Task 1-1-03: Resource-limit probe documentation + harness ruff pass
- **Persona:** ai-engineer — **REQ:** REQ-3-001
- **Files:** `apps/ai-service/README.md` (update: sandbox section + probe transcript), `apps/ai-service/pyproject.toml` (no change), `apps/ai-service/tests/sandbox/test_isolation.py` (no change)
- **Action:** Record the A-101 probe transcript in README (verbatim commands + observed output from this box: `unshare --user --map-root-user --mount --pid --fork --net id -u``0`; `ip link` → loopback only; write containment). Document the v0.3 resource-limit mechanism choice: cgroup-v2 delegation via per-sandbox scope files is **not available** on this box without sudo → enforcement = subprocess-level (`ulimit`-equivalent via `preexec_fn`: RLIMIT_AS for memory, RLIMIT_CPU for CPU-seconds, RLIMIT_NPROC for pids) + hard wall-clock timeout kill in the manager. This is the locked v0.3 mechanism (D-024 + no-sudo constraint). Run ruff over the new tree; fix all findings.
- **Verify:** `pnpm ai:lint` exits 0; README shows the probe transcript and the rlimit mechanism note
### Wave 2: Lifecycle manager (depends on Wave 1)
#### Task 1-2-01: Sandbox manager + concurrency guard + snapshots
- **Persona:** sandbox-engineer — **REQ:** REQ-3-001, REQ-3-002
- **Files:** `apps/ai-service/ai_service/sandbox/manager.py`, `apps/ai-service/tests/sandbox/test_manager.py`
- **Action:** `SandboxManager`: `create(learner_id) -> SandboxHandle` (guard: active count ≥ `AI_SANDBOX_MAX_CONCURRENT` → raise `PoolFullError` → API maps to **503**, D-032; no queue); `list() -> list[SandboxHandle]`; `get(id)`; `snapshot(id) -> Path` (delegates to workdir); `destroy(id)` (kill process tree, keep or purge workdir per flag); `reap_expired()` background hook for `AI_SANDBOX_TIMEOUT_S` which also performs a **workdir-size sweep**: any sandbox whose `workdir` exceeds `AI_SANDBOX_MAX_WORKDIR_MB` (new config, default 512) is snapshotted-then-destroyed and the event logged as an integrity signal (G-2 — soft disk cap, best-effort, not kernel-enforced); the sweep runs on the same timer as the timeout reaper. Enforce rlimits per spawner (Task 1-1-03: RLIMIT_AS + RLIMIT_CPU + RLIMIT_FSIZE=50MB as a cheap single-file disk guard (a-2); RLIMIT_NPROC noted as shared-per-host-uid, not relied on (G-1)) at exec time. Handle registry persisted **in-memory** (v0.3, single process; not a store — see D-019 precedent) with a clear note that handles are process-local. Startup reaper (a-1): on lifespan boot, scan `AI_SANDBOX_DIR` for workdirs whose recorded pid is dead and reap them, logging a warning. Narrow typed interface only — manager never imports api/ (boundary rule).
- **Verify:** `pnpm ai:test` — test_manager covers create/list/destroy/snapshot, 6th create raises PoolFullError (503 path), destroy kills the namespace process (pid gone), snapshot dir exists with workspace contents, timeout reaper removes a stale handle
#### Task 1-2-02: Resource-limit enforcement test
- **Persona:** sandbox-engineer — **REQ:** REQ-3-002
- **Files:** `apps/ai-service/tests/sandbox/test_resource_limits.py`
- **Action:** Concrete enforcement probes (guarded like isolation tests): (a) spawn a process that allocates > `RLIMIT_AS` → assert it dies with MemoryError/killed within a bound; (b) spawn a CPU spinner past `RLIMIT_CPU` → assert SIGXCPU/kill; (c) single huge file > `RLIMIT_FSIZE` → assert write failure (a-2 partial disk guard); (d) wall-clock: spawn `sleep 9999` with a small manager timeout → reaper destroys it; (e) **disk sweep (G-2)**: write > `AI_SANDBOX_MAX_WORKDIR_MB` across many files → assert the manager sweep destroys the sandbox and logs the integrity signal. Assert limits are observable (handle reports its limit set). NOTE (G-1): per-sandbox `RLIMIT_NPROC` is shared at the host uid — the fork-bomb probe is documented as shared-budget behavior, NOT asserted as per-sandbox isolation.
- **Verify:** `pnpm ai:test` — test_resource_limits green; limits proven enforced and observable
### Wave 3: API exposure (depends on Wave 2)
#### Task 1-3-01: Sandboxes API module
- **Persona:** backend-engineer — **REQ:** REQ-3-001, REQ-3-002
- **Files:** `apps/ai-service/ai_service/api/sandboxes.py`, `apps/ai-service/ai_service/main.py` (update: include router + lifespan manager), `apps/ai-service/ai_service/api/deps.py` (update), `apps/ai-service/tests/api/test_sandboxes.py`
- **Action:** DI exposes a singleton `SandboxManager`. Endpoints: `POST /v1/sandboxes {learner_id}` → 201 handle (503 when pool full); `GET /v1/sandboxes` → list; `GET /v1/sandboxes/{id}` → handle; `POST /v1/sandboxes/{id}/snapshot` → snapshot path; `DELETE /v1/sandboxes/{id}` → 204. All behind localhost CORS (A-008). Lifespan creates/destroys the manager; on shutdown destroys any live sandboxes (no orphans). **Abuse control (G-5, NOT KYC):** even in no-auth v0.3 the sandbox API enforces per-`learner_id` rate limiting (`AI_SANDBOX_MAX_PER_LEARNER`, default 1 active → 429) and a global create-rate cap (`AI_SANDBOX_CREATES_PER_MIN`, default 10 → 429); `learner_id` is validated against a server-side allowlist from config (`AI_LEARNER_ALLOWLIST`, default the single mock pilot id → unknown ids rejected 403). This ships in the no-auth milestone so a rogue local process can't exhaust shared NPROC/disk.
- **Verify:** `pnpm ai:test` — test_sandboxes green (create→list→snapshot→delete roundtrip via TestClient; 6th create → 503; delete of unknown id → 404; abuse control (G-5): non-allowlisted learner_id → 403; >1 active sandbox for one learner → 429; burst of >10 creates/min → 429); manual probe: `curl -X POST localhost:8420/v1/sandboxes -d '{"learner_id":"l1"}'` returns a handle id
### Must-Haves (Phase 1)
- [ ] Isolation probe test green on this box: in-namespace uid=0, network isolated (0 usable interfaces), host writes confined to the per-sandbox bind dir (A-101 re-verified as an automated test, not just research notes)
- [ ] Resource limits enforced + observable: memory (RLIMIT_AS) + CPU (RLIMIT_CPU) rlimits kill violating processes; wall-clock reaper destroys stale sandboxes; disk usage capped by a periodic workdir-size sweep in the manager (soft cap, configurable `AI_SANDBOX_MAX_WORKDIR_MB`, default 512MB — NOT kernel-enforced); per-sandbox NPROC is shared across sandboxes at the host uid — documented, not relied on for isolation (G-1, G-2 — test_resource_limits green)
- [ ] No cross-tenant access: sandbox A cannot read sandbox B's workdir (isolation test asserts containment)
- [ ] Lifecycle API works end-to-end: create/list/snapshot/destroy via TestClient; pool full → **503** (D-032, no queue)
- [ ] Snapshot produces a restorable directory copy under the sandbox's own snapshots/ dir
- [ ] `pnpm ai:test` and `pnpm ai:lint` green; new deps (`sqlmodel`, `sqlalchemy`, `websockets`, `aiofiles`) installed via idempotent bootstrap
- [ ] Boundary rules hold: `sandbox/` imports nothing from `api/` or `agents/`; only `api/sandboxes.py` composes the manager via DI
- [ ] No docker/podman/sudo anywhere in the spawner path (D-024); `SandboxBackend` protocol is the only coupling to the spawner (containerd swap possible later)
---
## Phase 2: Live Build Telemetry
**Requirements:** REQ-3-003
**Goal:** First real persistence (D-027: SQLite via SQLModel) with a `TraceStore` protocol; `TelemetryEvent` model with per-(learner,task) monotonic `seq`; WebSocket ingest endpoint (D-026) with gap detection; stdlib-only in-sandbox capture agent streams real sandbox activity into ai-service; trace retrievable by learner+task
### Wave 1: Models + stores + TS types (parallel — no shared files)
#### Task 2-1-01: Telemetry event models
- **Persona:** backend-engineer — **REQ:** REQ-3-003
- **Files:** `apps/ai-service/ai_service/telemetry/__init__.py`, `apps/ai-service/ai_service/telemetry/models.py`, `apps/ai-service/tests/telemetry/__init__.py`, `apps/ai-service/tests/telemetry/test_models.py`
- **Action:** Pydantic/SQLModel `TelemetryEvent`: `learner_id`, `task_id`, `seq` (int, monotonic per (learner,task)), `kind` (`command` | `file_diff` | `run_result` | `test_result` | `activity` | `stdin` | `stdout`), `payload` (JSON), `ts` (datetime, monotonic-envelope), `sandbox_id`. `TraceSpan` derived view (ordered events for one (learner,task)). Validation: seq ≥ 0, kind enum, non-empty ids.
- **Verify:** `pnpm ai:test` — test_models green (validation rules enforced, JSON payload roundtrip)
#### Task 2-1-02: TraceStore protocol + SQLite implementation
- **Persona:** backend-engineer — **REQ:** REQ-3-003
- **Files:** `apps/ai-service/ai_service/telemetry/store.py`, `apps/ai-service/tests/telemetry/test_store.py`, `apps/ai-service/ai_service/data/.gitkeep`
- **Action:** `TraceStore` protocol (D-027, Postgres-migration-ready): `append(event) -> None` (idempotent on (learner,task,seq) — at-least-once dedup), `get_trace(learner_id, task_id) -> list[TelemetryEvent]` (ordered by seq), `gaps(learner_id, task_id) -> list[int]` (missing seqs), `latest_seq(learner_id, task_id) -> int`, `list_tasks(learner_id) -> list[str]`, `close()`. `SQLiteTraceStore(SQLModel)`: single `telemetry_event` table, composite PK ((learner_id, task_id, seq)), indexes on (learner_id, task_id). Engine creation from `AI_DB_PATH`; `SQLModel.metadata.create_all` at app lifespan. Enable `PRAGMA journal_mode=WAL` + `synchronous=NORMAL` at engine creation (a-3) so concurrent ingest (writer) and trace reads (grader) don't hit `database is locked` under concurrent sandboxes.
- **Verify:** `pnpm ai:test` — test_store green (append/ordered-get/dedup-on-retry/gap detection/latest_seq; tmp-path SQLite per test)
#### Task 2-1-03: TS types for telemetry/traces
- **Persona:** data-engineer — **REQ:** REQ-3-003
- **Files:** `packages/types/telemetry.ts` (new), `packages/types/index.ts` (update)
- **Action:** TS `TelemetryEvent`, `TraceSpan`, `TelemetryKind` mirroring the Python model field-for-field (cross-referencing header, same string enums). Consumed by Phase 6 web surfaces; no runtime code.
- **Verify:** `pnpm typecheck` passes; TS type keys match Python model keys exactly
### Wave 2: Capture agent + ingest (depends on Wave 1)
#### Task 2-2-01: In-sandbox capture agent (stdlib-only)
- **Persona:** sandbox-engineer — **REQ:** REQ-3-003
- **Files:** `apps/ai-service/scripts/sandbox-agent.py`, `apps/ai-service/tests/sandbox/test_sandbox_agent.py`
- **Action:** Tiny standalone process (D-031, **stdlib only** — no deps shipped into the namespace): wraps a shell inside the sandbox; captures commands, file diffs (mtime/content polling of `workspace/` at 250ms), run/test results, activity; assigns per-(learner,task) `seq`; buffers to a local spool file on disconnect (at-least-once, D-026); reconnects with **exponential backoff** and flushes spool in order; small WebSocket client implemented over raw `socket` (RFC6455 client handshake + frames — stdlib only, no `websockets` in-namespace). Configured via env baked at spawn (`NC_LEARNER_ID`, `NC_TASK_ID`, `NC_INGEST_URL`).
- **Verify:** `pnpm ai:test` — unit tests with a loopback fake WS server: ordered seq emission, spool-on-disconnect, reconnect flush preserves order (no loss, dupes deduped server-side), no third-party imports in the file (asserted by AST scan)
#### Task 2-2-02: WebSocket ingest endpoint
- **Persona:** backend-engineer — **REQ:** REQ-3-003
- **Files:** `apps/ai-service/ai_service/telemetry/ingest.py`, `apps/ai-service/ai_service/api/sandboxes.py` (update: register WS route), `apps/ai-service/ai_service/main.py` (update), `apps/ai-service/tests/api/test_telemetry_ingest.py`
- **Action:** `WS /v1/telemetry/ingest` (D-026): accepts connections carrying learner_id/task_id/sandbox_id; validates + appends events to `TraceStore` (idempotent — server-side dedup on (learner,task,seq)); emits **gap warnings** when seq skips (logged + surfaced in a per-connection status); ping/pong keepalive. **Backpressure / flood control (G-3 — replaces silent drop):** bounded inbound queue; on overflow OR when a per-connection cap `AI_TELEMETRY_MAX_EVENTS_PER_TASK` (default 50000) is exceeded → **reject with a 1008 policy-violation close and mark the (learner,task) trace `INCOMPLETE_FLOODED`** (an integrity signal consumed by Proctor). Silent drop-oldest is FORBIDDEN because it corrupts grading input and is indistinguishable from trace-gaming. `GET /v1/telemetry/traces/{learner_id}/{task_id}` returns the ordered trace; `GET /v1/telemetry/gaps/{learner_id}/{task_id}` returns missing seqs.
- **Verify:** `pnpm ai:test` — test_telemetry_ingest green (TestClient websocket: connect → send 3 events → trace retrievable ordered; resend event 2 → deduped; skip seq 5 → gap reported; unknown sandbox tolerated in v0.3 no-auth mode)
### Wave 3: Sandbox telemetry wiring (depends on Wave 2)
#### Task 2-3-01: Spawn sandboxes with the capture agent
- **Persona:** sandbox-engineer — **REQ:** REQ-3-003
- **Files:** `apps/ai-service/ai_service/sandbox/manager.py` (update), `apps/ai-service/ai_service/sandbox/unshare_backend.py` (update), `apps/ai-service/tests/sandbox/test_telemetry_wiring.py`
- **Action:** `create()` gains optional `task_id`; when set the spawner copies `scripts/sandbox-agent.py` into the sandbox workdir, injects `NC_*` env, and launches the agent as a child of the namespace process (agent lifecycle tied to sandbox lifecycle; destroy kills the agent). No capture when task_id absent (pure shell sandbox).
- **Verify:** `pnpm ai:test` — end-to-end on this box: create sandbox with task_id → run 2 commands via exec → events arrive at the ingest endpoint and land in SQLite in order
### Wave 4: Reliability probe (depends on Wave 3)
#### Task 2-4-01: Dropped-connection durability probe
- **Persona:** backend-engineer — **REQ:** REQ-3-003
- **Files:** `apps/ai-service/tests/telemetry/test_durability.py`
- **Action:** Integration probe: run a capture agent against ingest, kill the WS connection mid-stream (simulate network failure), keep generating events, reconnect, assert the SQLite trace contains **every** event exactly once in order (spool + dedup). Document at-least-once semantics + replay path in README.
- **Verify:** `pnpm ai:test` — test_durability green; README documents semantics
### Must-Haves (Phase 2)
- [ ] Real telemetry from a live sandbox arrives at ai-service: shell commands, file diffs, run/test results appear as ordered events in SQLite (end-to-end, no mocks)
- [ ] Per-(learner,task) monotonic `seq`; gap detection reports missing seqs; replay yields the complete ordered trace
- [ ] At-least-once proven: transient disconnect + reconnect loses no events; duplicates deduped server-side (durability probe green)
- [ ] Capture agent is stdlib-only (AST-verified) and its lifecycle is tied to the sandbox (destroy kills it)
- [ ] Trace retrievable by learner+task via `GET /v1/telemetry/traces/...`; unknown trace → 404
- [ ] `TraceStore` protocol respected: no api/ code touches SQLite directly; `telemetry/` never imports `agents/` (D-027)
- [ ] Flood control (G-3): burst past `AI_TELEMETRY_MAX_EVENTS_PER_TASK` → connection closed 1008 + trace marked `INCOMPLETE_FLOODED`; no silent event drop on overflow
- [ ] `pnpm ai:test` green; `packages/types` telemetry TS types compile (`pnpm typecheck`)
---
## Phase 3: Process-Trace Grading Engine
**Requirements:** REQ-3-004
**Goal:** Deterministic feature computation over traces (D-028) → compact digest → LLM rubric scoring via existing D-020 JSON defense → validated structured scores stored in `GradeStore`; LLM never sees the raw trace; engine calibrated against v0.2 mock corpora so process quality separates paste-and-run from iterative debugging
### Wave 1: Features + grades store (parallel — no shared files)
#### Task 3-1-01: Deterministic trace digest (features)
- **Persona:** ai-engineer — **REQ:** REQ-3-004
- **Files:** `apps/ai-service/ai_service/grading/__init__.py`, `apps/ai-service/ai_service/grading/features.py`, `apps/ai-service/tests/grading/__init__.py`, `apps/ai-service/tests/grading/test_features.py`
- **Action:** Pure compute module (D-028): `compute_digest(trace: list[TelemetryEvent]) -> TraceDigest`. Deterministic features: test pass/fail counts + final status; edit count; error/fix cycle count + mean fix latency; idle gaps (>Ns, count + total); command category histogram (build/test/file/nav/debug/other); session duration; first-test-pass offset. `TraceDigest` pydantic model — compact (bounded size, no raw commands), LLM-safe.
- **Verify:** `pnpm ai:test` — test_features green over synthetic traces: paste-and-run trace (0 error/fix cycles, single test pass at end) vs iterative trace (many cycles) produce observably different digests
#### Task 3-1-02: GradeStore protocol + SQLite implementation
- **Persona:** backend-engineer — **REQ:** REQ-3-004
- **Files:** `apps/ai-service/ai_service/grading/store.py`, `apps/ai-service/tests/grading/test_store.py`
- **Action:** `GradeStore` protocol (D-027): `save(grade)`, `get(learner_id, task_id)`, `list_for_learner(learner_id)`, `close()`. SQLModel `GradeRecord`: learner_id, task_id, variant_seed (null until P4), digest (JSON), scores (JSON), verdict, model, created_at. PK (learner_id, task_id). Postgres-migration-ready.
- **Verify:** `pnpm ai:test` — test_store green (save/get/list roundtrip, overwrite-on-regrade documented)
### Wave 2: Grading engine + calibration (depends on Wave 1)
#### Task 3-2-01: Rubric scoring engine
- **Persona:** ai-engineer — **REQ:** REQ-3-004
- **Files:** `apps/ai-service/ai_service/grading/engine.py`, `apps/ai-service/ai_service/prompts/grading.py` (new), `apps/ai-service/tests/grading/test_engine.py`
- **Action:** `GradingEngine.grade(learner_id, task_id) -> GradeRecord`: **trace-completeness gate (G-4)** — first call `TraceStore.gaps()` + check the trace `INCOMPLETE_FLOODED` flag; if gaps are non-empty OR the trace is flagged incomplete → return `verdict=UNGRADABLE_TRACE_INCOMPLETE` (a first-class verdict, not an exception) surfacing the gap list; a credential is NEVER issued from a gapped/incomplete trace. Otherwise: load trace via `TraceStore``compute_digest` → render rubric prompt (`prompts/grading.py`: criteria + level anchors for process quality, correctness, debugging discipline, test usage; a-4: treat high edit/command churn with no test-progress as a process-quality negative) → LLM structured output through the **existing D-020 4-layer defense** (`agents/structured.py` reused — engine composes it, never duplicates it) → validate `RubricScore` model (per-criterion 0-4 + strengths + gaps + verdict) → persist via `GradeStore`. Grading depends on `llm/` + `telemetry/` + `prompts/` only (boundary). Mock provider scripts deterministic rubric JSON for tests, including the INCOMPLETE path.
- **Verify:** `pnpm ai:test` — test_engine green (mock provider: digest-only prompt asserted — **raw trace string absent from prompt**; validated scores returned; malformed JSON exercises D-020 retry; unknown trace → error)
#### Task 3-2-02: Calibration against v0.2 mock corpora
- **Persona:** ai-engineer — **REQ:** REQ-3-004
- **Files:** `apps/ai-service/ai_service/corpus/trace_fixtures.py` (new), `apps/ai-service/tests/grading/test_calibration.py`
- **Action:** Synthetic trace fixtures aligned with v0.2 `corpus/telemetry.py` + `corpus/artifacts.py` scenario IDs (strong/lazy/struggling builder archetypes). Assert grading separates them: strong archetype scores ≥ lazy archetype on process-quality criterion (mock provider maps digest shape → scripted scores; test asserts the ordering contract + that fixture IDs align with existing corpus IDs, D-021).
- **Verify:** `pnpm ai:test` — test_calibration enforces the ordering contract
### Wave 3: Grading endpoint (depends on Wave 2)
#### Task 3-3-01: Assessment grade endpoint (real traces)
- **Persona:** backend-engineer — **REQ:** REQ-3-004
- **Files:** `apps/ai-service/ai_service/api/assessment.py` (update), `apps/ai-service/tests/api/test_grading.py`
- **Action:** `POST /v1/assessment/grade {learner_id, task_id}``GradingEngine` → validated `RubricScore` JSON; `GET /v1/assessment/grade/{learner_id}/{task_id}` → stored grade; unknown trace → 404. Composed via DI (api/ owns wiring; engine knows nothing of FastAPI).
- **Verify:** `pnpm ai:test` — test_grading green (grade roundtrip via TestClient with mock provider; 404 on unknown; GET after POST returns same scores)
### Must-Haves (Phase 3)
- [ ] Engine emits structured rubric-aligned scores from a **real process trace** (not pre-baked input) — TestClient roundtrip green
- [ ] Deterministic features computed in code (test pass/fail, edit count, error/fix cycles, idle gaps, command categories); LLM receives the **digest only** — test asserts the raw trace never reaches the prompt (D-028)
- [ ] Scores distinguish process quality: iterative-debugging archetype out-scores paste-and-run on the process criterion (calibration contract test)
- [ ] Grades persisted + retrievable by learner+task via GradeStore (SQLite, protocol-wrapped, D-027)
- [ ] Boundary rules hold: `grading/` imports no `api/`/`agents/` internals except the shared D-020 structured defense; `pnpm ai:test` + `pnpm ai:lint` green
- [ ] Incomplete-trace gate (G-4): gapped or `INCOMPLETE_FLOODED` trace → `verdict=UNGRADABLE_TRACE_INCOMPLETE` with the gap list; no credential issued from an incomplete trace (test green)
---
## Phase 4: Variant Task Generation
**Requirements:** REQ-3-005
**Goal:** Template library with typed parameter slots (D-029) + seeded LLM instantiation + per-learner variant registry (SQLite) with difficulty-normalization anchors; two learners on the same competency get provably distinct, reproducible, auditable tasks
### Wave 1: Templates + store (parallel — no shared files)
#### Task 4-1-01: Task template library
- **Persona:** ai-engineer — **REQ:** REQ-3-005
- **Files:** `apps/ai-service/ai_service/variants/__init__.py`, `apps/ai-service/ai_service/variants/templates.py`, `apps/ai-service/tests/variants/__init__.py`, `apps/ai-service/tests/variants/test_templates.py`
- **Action:** ≥3 initial task templates bound to existing competency IDs (D-021 alignment). `TaskTemplate`: id, competency_id, statement skeleton with `{slot}` placeholders, `ParameterSlot[]` (name, type: enum/int-range/string-set, allowed values), `rubric anchors` (difficulty normalization: expected feature envelope — e.g. expected edit-count band — used by grading context), starter-file scaffolds served to the sandbox. Seeded slot sampler is pure code (`random.Random(seed)`), fully reproducible.
- **Verify:** `pnpm ai:test` — test_templates green (slot validation: bad value rejected; seeded sampling reproducible across runs; all templates bind to real competency IDs)
#### Task 4-1-02: VariantStore protocol + SQLite implementation
- **Persona:** backend-engineer — **REQ:** REQ-3-005
- **Files:** `apps/ai-service/ai_service/variants/store.py`, `apps/ai-service/tests/variants/test_store.py`
- **Action:** `VariantStore` protocol (D-027): `save(variant)`, `get(learner_id, template_id_or_task_id)`, `list_for_learner(learner_id)`, `list_by_template(template_id)`, `close()`. SQLModel `VariantRecord`: learner_id, task_id (the grading/telemetry task key), template_id, seed, params (JSON), statement (rendered), created_at. Unique (learner_id, template_id).
- **Verify:** `pnpm ai:test` — test_store green (roundtrip, unique constraint, audit listing)
### Wave 2: Generator (depends on Wave 1)
#### Task 4-2-01: Seeded LLM variant generator
- **Persona:** ai-engineer — **REQ:** REQ-3-005
- **Files:** `apps/ai-service/ai_service/variants/generator.py`, `apps/ai-service/ai_service/prompts/variant.py` (new), `apps/ai-service/tests/variants/test_generator.py`
- **Action:** `VariantGenerator.generate(learner_id, template_id) -> VariantRecord`: derive seed (`sha256(template_id|learner_id|milestone)` — reproducible, D-029); sample typed slots in code; render a fill prompt (statement skeleton + concrete slot values) → LLM via D-020 structured defense → unique task statement + starter files → validate → persist (seed + params + statement) via `VariantStore`. Cache: existing (learner,template) returns the stored variant (no duplicate work). Mock provider scripts deterministic statements per seed for tests.
- **Verify:** `pnpm ai:test` — test_generator green: two different learner_ids → distinct statements for the same template; same learner twice → identical stored variant (reproducible); params JSON contains only schema-valid slot values; **fairness envelope (a-5):** two variants of one template compute digests within the template's expected feature envelope (comparable slot complexity/difficulty features) — "same bar" is testable, not asserted
### Wave 3: Variant endpoint + TS types (depends on Wave 2)
#### Task 4-3-01: Variant task endpoint
- **Persona:** backend-engineer — **REQ:** REQ-3-005
- **Files:** `apps/ai-service/ai_service/api/variants.py` (new), `apps/ai-service/ai_service/main.py` (update), `apps/ai-service/tests/api/test_variants.py`
- **Action:** `POST /v1/variants {learner_id, template_id or competency_id}` → generated (or cached) variant: statement, starter files, task_id, seed; `GET /v1/variants/{task_id}` → stored variant; `GET /v1/variants?learner_id=` → learner's variants. DI wiring in api/ only.
- **Verify:** `pnpm ai:test` — test_variants green (generate→get roundtrip; cache hit on regenerate; distinct learners → distinct statements asserted at the API layer)
#### Task 4-3-02: TS types for variants (+ grades)
- **Persona:** data-engineer — **REQ:** REQ-3-005
- **Files:** `packages/types/variants.ts` (new), `packages/types/grading.ts` (new), `packages/types/index.ts` (update)
- **Action:** TS `TaskVariant`, `VariantParams`, `RubricScore`, `GradeRecord` matching Python models (cross-referencing headers; same field names). Consumed by Phase 6 surfaces.
- **Verify:** `pnpm typecheck` passes
### Must-Haves (Phase 4)
- [ ] Two learners requesting the same competency receive **provably distinct** task variants (API-level test)
- [ ] Seed derivation reproducible: same (template, learner) → same variant, served from cache without a second LLM call (D-029)
- [ ] Variant seed + typed params persisted + auditable (VariantStore listing; proctoring cross-check path exists)
- [ ] Difficulty normalization anchors present per template and shipped to the grader prompt context
- [ ] Starter-file scaffolds defined per template (P6 wires them into the sandbox workdir)
- [ ] `pnpm ai:test` + `pnpm typecheck` green; `variants/` imports no api/ (boundary)
---
## Phase 5: Oral / Voice Defense
**Requirements:** REQ-3-006
**Goal:** `VoiceProvider` protocol with mock + browser fallback (D-030) + seventh `Examiner` agent streaming over existing SSE + `DefenseStore` persisting transcript + integrity signals. **Real server STT/TTS (`OpenAIAudioProvider`) is DEFERRED to v0.4 (with KYC, when there's a real key + real users)** — voice is mock-first (D-030) and the `/audio/*` real path could never be exercised in CI, so v0.3 proves the full defense *dialogue* + integrity-signal pipeline over mock + browser-native fallback only; the protocol seam keeps the real provider a drop-in later.
### Wave 1: Voice provider layer (parallel — no shared files)
#### Task 5-1-01: VoiceProvider protocol + mock provider + browser fallback
- **Persona:** voice-engineer — **REQ:** REQ-3-006
- **Files:** `apps/ai-service/ai_service/voice/__init__.py`, `apps/ai-service/ai_service/voice/base.py`, `apps/ai-service/ai_service/voice/mock.py`, `apps/ai-service/ai_service/voice/browser.py`, `apps/ai-service/ai_service/voice/factory.py`, `apps/ai-service/tests/voice/__init__.py`, `apps/ai-service/tests/voice/test_mock.py`, `apps/ai-service/tests/voice/test_factory.py`
- **Action:** `VoiceProvider` protocol mirroring `LLMProvider` (D-030): `transcribe(audio: bytes, fmt) -> TranscriptSegment` + `synthesize(text, voice) -> AsyncIterator[bytes]`. `MockVoiceProvider`: deterministic canned transcript (scripted per test), canned 1kHz-tone WAV bytes, scripted failure modes. `browser.py`: fallback **descriptor** (`sr_available: true`, endpoint hints) the web client uses to select browser-native `SpeechRecognition`/`speechSynthesis` when no server provider. `factory.py`: `AI_VOICE_PROVIDER=browser | mock` (default mock when no key). **`OpenAIAudioProvider` (real server STT/TTS) intentionally NOT built in v0.3 — deferred to v0.4**; the protocol is its future seam. `voice/` never imports `agents/` or `api/`.
- **Verify:** `pnpm ai:test` — test_mock + test_factory green (deterministic transcribe/synthesize; failure modes; factory selects mock with empty key, browser when provider=browser; zero network calls)
#### Task 5-1-03: DefenseStore protocol + SQLite implementation
- **Persona:** backend-engineer — **REQ:** REQ-3-006
- **Files:** `apps/ai-service/ai_service/voice/defense_store.py`, `apps/ai-service/tests/voice/test_defense_store.py`
- **Action:** `DefenseStore` protocol (D-027): `start(defense)`, `append_turn(defense_id, turn)`, `finalize(defense_id, integrity_signals)`, `get(defense_id)`, `list_for_learner(learner_id)`, `close()`. SQLModel `DefenseRecord` (id, learner_id, task_id, status, created/finished_at) + `DefenseTurn` (defense_id FK, turn seq, role examiner|learner, text, ts, latency_ms) + integrity signals JSON on the record (long pauses, off-scope cadence markers — A-109).
- **Verify:** `pnpm ai:test` — test_defense_store green (start→append turns→finalize→get roundtrip; ordered turns by seq)
### Wave 2: Examiner agent (depends on Wave 1)
#### Task 5-2-01: Examiner agent (seventh agent)
- **Persona:** ai-engineer — **REQ:** REQ-3-006
- **Files:** `apps/ai-service/ai_service/agents/examiner.py`, `apps/ai-service/ai_service/prompts/examiner.py` (new), `apps/ai-service/ai_service/agents/registry.py` (update: central registration, G-4 pattern), `apps/ai-service/tests/agents/test_examiner.py`
- **Action:** `ExaminerAgent(BaseAgent)` (D-030/A-109): builds questions from learner transcript + trace digest + (P4) variant statement; probes understanding + challenges process choices ("why did you choose X at step N?"); streams questions over the existing SSE pipeline; `structured` verdict mode returns verdict + per-answer integrity signal list (long pause flags, off-scope answers) computed from turn metadata; calls voice **only through the `VoiceProvider` protocol** (PERSONAS conflict rule — never concrete providers). Session-scoped history reused from v0.2.
- **Verify:** `pnpm ai:test` — test_examiner green (question stream references trace-digest facts; verdict structured output validates via D-020 defense; registry resolves all seven agents; mock-VoiceProvider wiring through protocol only — asserted by import scan in test)
### Wave 3: Defense endpoints (depends on Wave 2)
#### Task 5-3-01: Defense session + audio endpoints
- **Persona:** backend-engineer — **REQ:** REQ-3-006
- **Files:** `apps/ai-service/ai_service/api/defense.py` (new), `apps/ai-service/ai_service/main.py` (update), `apps/ai-service/tests/api/test_defense.py`
- **Action:** `POST /v1/defense/start {learner_id, task_id}` → creates DefenseRecord + streams the first examiner question (SSE, agent=examiner); `POST /v1/defense/{id}/answer` (multipart audio from browser MediaRecorder, or `{text}` for typed fallback) → STT via VoiceProvider → append learner turn → stream examiner follow-up (SSE) → TTS audio chunks over `GET /v1/defense/{id}/audio/{turn_id}`; `POST /v1/defense/{id}/finish` → verdict + integrity signals persisted; `GET /v1/defense/{id}` → full transcript + signals. Browser-fallback mode: when provider=browser, start returns the fallback descriptor instead of server audio.
- **Verify:** `pnpm ai:test` — test_defense green (full loop with mock voice + mock LLM: start → answer(text) → answer(audio bytes) → finish → transcript retrievable with per-turn latency; unknown id → 404; `GET` signals present after finish)
### Wave 4: Examiner latency instrumentation (depends on Wave 3)
#### Task 5-4-01: Per-turn latency instrumentation (mock-based)
- **Persona:** voice-engineer — **REQ:** REQ-3-006
- **Files:** `apps/ai-service/tests/voice/test_latency.py`, `apps/ai-service/README.md` (update: conversational-budget doc + v0.4 voice note)
- **Action:** Instrument per-turn latency (STT ms + LLM TTFT ms + TTS ms) recorded on each DefenseTurn; deterministic test over mock providers asserts instrumentation presence + `latency_ms` populated + budget constant defined (mock runs are near-instant — wall-clock asserted in v0.4 against a real endpoint). README documents the conversational-latency target as a v0.4 acceptance criterion (real STT/TTS deferred per CUT-1/G-7).
- **Verify:** `pnpm ai:test` — test_latency green (latency_ms fields populated on every turn; budget constant defined); README documents the deferred real-voice acceptance probe
### Must-Haves (Phase 5)
- [ ] Spoken defense runs end-to-end over HTTP with mock providers: start → answer (audio + typed fallback) → examiner follow-up streams → verdict + transcript persisted (automated)
- [ ] Examiner is the seventh registered agent; streams over the existing SSE envelope (meta agent=examiner)
- [ ] Instrumented per-turn latency fields populated on every DefenseTurn (STT ms + LLM TTFT ms + TTS ms); conversational budget named (A-109)
- [ ] `VoiceProvider` protocol respected: examiner + api touch voice only via the protocol; mock-first — **no task requires a real voice key to pass**
- [ ] Browser-native SR/TTS fallback descriptor returned when no server voice provider configured (mock/browser are first-class, D-030)
- [ ] Real server STT/TTS (`OpenAIAudioProvider`) explicitly deferred to v0.4 (with real keys/users); the defense pipeline is fully proven over mock+browser — documented in README + release note
- [ ] Optional future voice config noted for v0.4 (`AI_VOICE_BASE_URL` / `AI_VOICE_API_KEY`) in `.env.example` + README; keys only in gitignored `.ciagent/.env.secrets`; tests never call a voice API
- [ ] Boundary rules hold: `voice/` imports no `agents/`/`api/`; `pnpm ai:test` + `pnpm ai:lint` green
---
## Phase 6: Agent Re-grounding + Learner Surface Integration
**Requirements:** REQ-3-007, REQ-3-008
**Goal:** Lab/Assessor/Proctor consume real engine inputs (live telemetry, grading output, defense signals) with **no mock fallback in the learner path**; the v0.1 sandbox + assessment mockups become real — in-browser build/run (Run/Test buttons + read-only exec output, CUT-2 — no interactive shell), live telemetry panel, browser/typed voice defense, live grading; `pnpm build` + `pnpm typecheck` + full `pnpm ai:test` green
**Note:** E2E verification runs against the real engines over HTTP with `AI_PROVIDER=mock` + `AI_VOICE_PROVIDER=mock` permitted (G-2 precedent) — the requirement is real engine plumbing (sandbox/telemetry/grading/defense over real endpoints, no corpus mocks in the learner path); a cloud outage must not block P6.
### Wave 1: Agent re-grounding (parallel — no shared files)
#### Task 6-1-01: Lab agent on live telemetry
- **Persona:** ai-engineer — **REQ:** REQ-3-007
- **Files:** `apps/ai-service/ai_service/agents/lab.py` (update), `apps/ai-service/ai_service/prompts/lab.py` (update: render real trace digest), `apps/ai-service/tests/agents/test_lab_live.py` (new)
- **Action:** Lab consumes a **live trace digest** (grading/features `compute_digest` over `TraceStore` events) instead of `corpus/telemetry.py`. build_messages renders digest facts (recent commands, failing tests, idle). Mock-provider scripts assert digest-derived content. v0.2 corpus path removed from the agent (dormant corpus retained until Task 6-1-04 check).
- **Verify:** `pnpm ai:test` — test_lab_live green: feedback references events actually present in a seeded SQLite trace (not corpus fixtures); no `corpus.telemetry` import in `agents/lab.py` (AST-asserted)
#### Task 6-1-02: Assessor agent on grading output
- **Persona:** ai-engineer — **REQ:** REQ-3-007
- **Files:** `apps/ai-service/ai_service/agents/assessor.py` (update), `apps/ai-service/ai_service/prompts/assessor.py` (update), `apps/ai-service/tests/agents/test_assessor_live.py` (new)
- **Action:** Assessor consumes `GradeStore` output (validated `RubricScore` + digest) for learner+task instead of pre-baked artifacts/transcripts; renders strengths/gaps/verdict with rubric-anchored coaching framing. Structured output unchanged (D-020).
- **Verify:** `pnpm ai:test` — test_assessor_live green: given a real stored grade, Assessor output reflects its scores; corpus artifact path gone from the agent (AST-asserted)
#### Task 6-1-03: Proctor agent on telemetry + defense signals
- **Persona:** ai-engineer — **REQ:** REQ-3-007
- **Files:** `apps/ai-service/ai_service/agents/proctor.py` (update), `apps/ai-service/ai_service/prompts/proctor.py` (update), `apps/ai-service/tests/agents/test_proctor_live.py` (new)
- **Action:** Proctor consumes real integrity inputs: idle gaps + command cadence from the trace digest + defense integrity signals from `DefenseStore` → classified signals + coaching interventions (supportive tone retained). Cross-checks variant seed params (P4) for off-template work.
- **Verify:** `pnpm ai:test` — test_proctor_live green: signals derived from seeded real trace + defense records; corpus proctor scenarios no longer imported (AST-asserted)
#### Task 6-1-04: Corpus dormancy + mockup removal verification
- **Persona:** lead-developer — **REQ:** REQ-3-007
- **Files:** `apps/ai-service/ai_service/corpus/telemetry.py`, `apps/ai-service/ai_service/corpus/artifacts.py`, `apps/ai-service/ai_service/corpus/` (README note), `apps/ai-service/tests/test_corpus_dormancy.py` (new)
- **Action:** Verify no production code path imports `corpus/telemetry.py` or `corpus/artifacts.py` anymore (test scans imports across `agents/`, `api/`, engines). Retain files as Phase-3 calibration history with a header note marking them **dormant — v0.2 mocks, not used at runtime**; learner-context corpus stays (agents still need learner context). Disposes v0.2's G-5-class dead-code risk deliberately.
- **Verify:** `pnpm ai:test` — test_corpus_dormancy green (zero runtime importers); suite otherwise unchanged
### Wave 2: Client plumbing + design primitives (parallel — no shared files)
#### Task 6-2-01: Sandbox build-panel engine client (run/test-only — no raw shell relay)
- **Persona:** frontend-engineer — **REQ:** REQ-3-008
- **Files:** `apps/web/hooks/use-sandbox-session.ts` (new), `apps/web/lib/engine-client.ts` (new), `apps/web/.env.example` (update)
- **Action:** `engine-client.ts`: typed fetch client for `/v1/sandboxes` (create/destroy — learner_id from the v0.3 mock session constant, allowlisted server-side per G-5), `/v1/sandboxes/{id}/files` (workspace CRUD), `/v1/sandboxes/{id}/exec` (run/test), `/v1/variants`, `/v1/assessment/grade`, `/v1/telemetry/traces`, `/v1/defense/*`; base `NEXT_PUBLIC_AI_SERVICE_URL`. `use-sandbox-session.ts`: create sandbox+variant on task open → destroy on unmount (idempotent cleanup, AbortController pattern); 503 pool-full → user-facing "environment busy, retry" (D-032); 403/429 abuse-control surfaced honestly (G-5). **CUT-2 (G-8): NO raw interactive WS terminal relay (keystroke-level stdin/stdout) in v0.3** — the credential pipeline needs *process events* (from Run/Test + file edits), not a live shell; the interactive xterm relay is the most fragile real-time piece and is deferred to v0.4. The build panel is a **Run/Test output viewer** (exec results + telemetry pulse render), not an interactive shell. `@xterm/xterm` is therefore NOT a dependency in v0.3.
- **Verify:** `pnpm install && pnpm typecheck` pass; hook unmount destroys the sandbox (manual probe: `curl localhost:8420/v1/sandboxes` shows count drop after navigation); RUN/TEST buttons produce streamed output + telemetry events in the trace
#### Task 6-2-02: New design primitives
- **Persona:** design-system-engineer — **REQ:** REQ-3-008
- **Files:** `packages/ui/src/primitives/terminal-frame.tsx` (new), `packages/ui/src/primitives/mic-control.tsx` (new), `packages/ui/src/primitives/grade-badge.tsx` (new), `packages/ui/src/primitives/telemetry-status.tsx` (new), `packages/ui/src/primitives/transcript-viewer.tsx` (new), `packages/ui/src/primitives/index.ts` (update), `packages/ui/src/index.ts` (update)
- **Action:** Token-driven primitives: TerminalFrame (CUT-2: a read-only exec-output viewer chrome — streams Run/Test results, NOT an interactive shell), MicControl (record/stop with consent state + no-mic fallback state, MediaRecorder permission UX), GradeBadge (verdict rendering), TelemetryStatus (live event pulse / disconnected indicator), TranscriptViewer (examiner/learner turn list). Dark mode + WCAG AA; stories for each.
- **Verify:** primitives import from `@nextcraft/ui`; Storybook stories render dark + light; `pnpm build` (ui package) passes
#### Task 6-2-03: Defense TS types
- **Persona:** data-engineer — **REQ:** REQ-3-008
- **Files:** `packages/types/defense.ts` (new), `packages/types/index.ts` (update)
- **Action:** TS `DefenseSession`, `DefenseTurn`, `IntegritySignal`, `Verdict` mirroring P5 Python models (cross-referencing header).
- **Verify:** `pnpm typecheck` passes
### Wave 3: Real build surface (depends on Wave 2)
#### Task 6-3-01: Sandbox mockup → real in-browser IDE
- **Persona:** frontend-engineer — **REQ:** REQ-3-008
- **Files:** `apps/web/app/(learner)/build/[competencyId]/page.tsx` (rewrite), `apps/web/components/learner/sandbox-terminal.tsx` (new), `apps/web/components/learner/file-tree.tsx` (new), `apps/web/components/learner/run-controls.tsx` (new), `apps/web/components/learner/lab-feedback-panel.tsx` (update: live trace)
- **Action:** Replace the mockup with the real build environment (A-103, CUT-2): file tree (HTTP CRUD into the sandbox workdir via `/v1/sandboxes/{id}/files` routes added to api/sandboxes — read/write/list workspace files), syntax-highlight editor (existing), **Run**/**Test** buttons (exec in sandbox; results stream to a read-only TerminalFrame output panel — no interactive shell), starter files from the P4 variant scaffold. Lab panel posts `learner_id+task_id` → streams Lab feedback over the **live** trace (no scenario IDs). Telemetry sidebar shows live TelemetryStatus. Pool-full 503 → busy state with retry; 403/429 surfaced.
- **Verify:** with ai-service up: open `/build/comp-01` → variant statement + starter files load → edit a file → **Run** executes the command in-sandbox and output renders in the panel → **Test** runs the test suite in-sandbox → Lab panel streams digest-derived feedback → telemetry status shows live events. Manual probe documented; `pnpm typecheck` green
### Wave 4: Live defense + grading surfaces (depends on Waves 2-3)
#### Task 6-4-01: Assessment mockup → live defense + live grading
- **Persona:** frontend-engineer — **REQ:** REQ-3-008
- **Files:** `apps/web/app/(learner)/defend/[competencyId]/page.tsx` (rewrite), `apps/web/components/learner/defense-session.tsx` (new), `apps/web/components/learner/assessor-results-panel.tsx` (update), `apps/web/components/learner/proctor-banner.tsx` (update), `apps/web/components/learner/oral-defense-interface.tsx` (rewrite or remove)
- **Action:** Real assessment flow: **Start Defense** → POST `/v1/defense/start` → examiner question streams → learner answers via MicControl (MediaRecorder webm/opus → multipart POST) with typed fallback when mic denied or `provider=browser` (native `SpeechRecognition`/`speechSynthesis` path per fallback descriptor) → follow-ups stream → **Finish** → verdict + integrity signals panel (TranscriptViewer, GradeBadge) + **Grade My Work** → POST `/v1/assessment/grade` → structured rubric bars render. Proctor banner reads signals for task_id. Loading + error states throughout; no mock defense data remains in the learner path.
- **Verify:** with ai-service up: full defense loop runs in-browser (typed fallback acceptable in CI-less manual probe; mic path exercised manually with permission granted); grade panel renders real rubric scores; `pnpm typecheck` green
### Wave 5: End-to-end verification (depends on Waves 3-4)
#### Task 6-5-01: Full learner-path E2E probe + green builds
- **Persona:** backend-engineer — **REQ:** REQ-3-007, REQ-3-008
- **Files:** `apps/ai-service/tests/api/test_e2e_credential_flow.py` (new), `apps/ai-service/README.md` (update: E2E probe doc)
- **Action:** Endpoint-level E2E (mock LLM/voice providers, real engines): create variant → create sandbox with task_id → ingest trace events via real sandbox exec → POST grade → start defense (typed answers) → finish → assert: trace persisted, grade stored + digest-linked, defense transcript + signals stored, Proctor/Assessor endpoints serve them. No corpus fixture anywhere in the flow (AST-asserted). Run `pnpm build` + `pnpm typecheck` + full `pnpm ai:test` at repo root; fix all failures before phase ship.
- **Verify:** `pnpm ai:test` green incl. test_e2e_credential_flow; `pnpm build` + `pnpm typecheck` green; README E2E probe section documents the manual browser pass
### Must-Haves (Phase 6)
- [ ] Lab/Assessor/Proctor operate on real inputs with **no mock fallback in the learner path** (AST-verified: no corpus telemetry/artifact/proctor imports in production paths)
- [ ] Learner builds in-browser for real (CUT-2): Run/Test buttons execute in a namespace sandbox and stream output to a read-only panel; file tree CRUD works; starter files come from the learner's variant scaffold
- [ ] Live telemetry: build activity streams to ai-service and the sidebar shows live status (TelemetryStatus); trace persisted in SQLite
- [ ] Live defense: start → answer (mic or typed fallback) → examiner follow-ups → finish → transcript + integrity signals + verdict rendered; browser-native path works with no server voice key
- [ ] Live grading: grade request returns structured rubric scores computed from the real trace digest; grade panel renders them
- [ ] 503 pool-full surfaced honestly in UI; navigating away destroys the sandbox (no leaked sandboxes — `GET /v1/sandboxes` manual probe)
- [ ] Examiner remains protocol-clean (voice only via `VoiceProvider`); module boundary rules hold across all new code
- [ ] `pnpm build` and `pnpm typecheck` pass; full `pnpm ai:test` green; no cloud/voice calls in any automated test
- [ ] Release-note input (for P7): v0.3 ships real engines; **identity/age-gating (KYC) remains deferred — age-gating is still a visual mockup** (A-110); sandbox scope is coding-IDE only (design tool/simulation deferred to v0.4, D-025)
---
## Phase 7: Final Review + Ship (no planned tasks)
Orchestrated by the SHIP stage, not this plan: multi-persona code review (correctness, testing, module boundaries, secrets hygiene — keys absent from code/logs/commits/errors, localhost-only CORS, no PII in prompts), project health audit (reconstruction, .ciagent/ discipline, branch/commit hygiene), then merge milestone → main, tag the final v0.2.x patch, create the Gitea release, mark all 8 v0.3 requirements complete.
**Release-note honesty:** the release note must state (a) Lab/Assessor/Proctor now run on real engine inputs (v0.2 mock-input caveat retired), (b) sandbox scope = coding IDE only — design tool + simulation environments deferred to v0.4 (D-025), (c) identity/age-gating (KYC) deferred per founder directive — age-gating remains the v0.1 visual flow mockup; abuse control (per-learner sandbox caps + server-side learner allowlist, G-5) ships in place of auth (A-110), (d) voice runs mock-first with browser-native fallback — real server STT/TTS deferred to v0.4 (CUT-1), (e) sandbox resource limits are partially enforced (memory/CPU/wall-clock kernel-enforced via rlimits; per-sandbox pids + hard disk quota are NOT — mitigated by a workdir-size sweep and per-learner caps; full enforcement requires cgroup delegation, deferred to the post-MVP containerd backend, D-024/G-1/G-2).
**Secrets-hygiene checklist (P7):** `.ciagent/.env.secrets` gitignored and never committed; `AI_VOICE_API_KEY` / `AI_TUTOR_API_KEY` referenced only via env; SQLite DBs + sandbox dirs gitignored; no keys in logs, error messages, or test fixtures.
**Disposal checks (G-5 class):** v0.2 dormant corpus files carry the dormant-header note (Task 6-1-04); any now-unused mock-data exports for the old sandbox/defense mockups (e.g. `aiTutorResponses`-class leftovers) must be removed or deprecated by review.
---
| 1 | Bootstrap CLI core | REQ-4-001, REQ-4-002 | 3 | cli-engineer, backend-engineer, security-auditor (W3 review) |
| 2 | Binary build + release pipeline | REQ-4-003, REQ-4-004 | 3 | cli-engineer, backend-engineer, security-auditor |
| 3 | Install docs + fresh-clone E2E | REQ-4-005 | 2 | cli-engineer, backend-engineer |
| 4 | Final review + ship | — | 1 | all reviewers |
## User-Facing Surface
The primary user-facing surface is the **learner build + defend flow** at `http://localhost:3000`, backed by the real engines in ai-service at `http://localhost:8420`:
- `/dashboard` — AI tutor chat (Coach/Tutor, streaming) + Mentor panel (unchanged from v0.2)
- `/learn/[competencyId]` — byte viewer with streaming Tutor explanations (unchanged)
- `/build/[competencyId]`**real in-browser build environment**: file tree, syntax editor, **Run/Test buttons that execute in a namespace sandbox and stream output to a read-only panel** (CUT-2 — no interactive shell), live telemetry status, live Lab feedback, per-learner variant task statement
- `/defend/[competencyId]`**live oral defense + live grading**: Examiner voice/typed dialogue, transcript + integrity signals, real rubric scores from the process trace
The marketplace, employer, and admin surfaces are unchanged from v0.1/v0.2.
1. **One-liner install (README quickstart):** `curl -fsSL https://git.coreci.dev/coreci/nextcraft/raw/main/scripts/install.sh | bash` — downloads the latest release's `nextcraft-linux-x64` binary, verifies its sha256, installs to `~/.local/bin`, prints a PATH hint if needed.
2. **CLI commands:** `nextcraft doctor` (prereq checks), `nextcraft bootstrap` (fresh clone → runnable stack), `nextcraft verify` (health check), `nextcraft dev` (dev server passthrough), plus `--help`/`--version`.
3. **Release surface:** every Gitea release from v0.3.2 onward carries `nextcraft-linux-x64` + `nextcraft-linux-x64.sha256` assets.
## Happy Path
1. Learner opens `/build/comp-01` → a per-learner **variant statement** and starter files load; a namespace sandbox is created for the session
2. Learner edits files in the tree and clicks **Run**/**Test** → commands execute in the sandbox and real output renders in the build panel; the telemetry sidebar pulses as events stream to ai-service and persist in SQLite
3. The Lab panel streams feedback derived from the **live trace digest** (real commands, real failures)
4. Learner opens `/defend/comp-01`**Start Defense**: the Examiner streams an opening question ("Walk me through your build — why did you structure it this way?")
5. Learner answers by voice (mic consent → MediaRecorder → STT) or typed fallback → examiner follow-ups probe the trace ("You hit three test failures before passing — what changed?"); TTS plays examiner audio (or browser speech in fallback)
6. Learner finishes the defense → transcript + integrity signals appear; verdict renders in a GradeBadge
7. Learner clicks **Grade My Work** → the grading engine computes the digest from the real trace, rubric-scores it, and the panel renders per-criterion bars + strengths/gaps/verdict
8. Proctor banner shows integrity signals from the live trace + defense in coaching tone; Mentor panel on `/dashboard` can narrate the real outcome
9. Navigating away destroys the sandbox (pool slot freed); killing ai-service shows inline error + retry states on every panel, with no crashes
Before execution, the end-to-end scenario this milestone must make true:
1. A consumer on a linux x64 box runs the one-liner; `nextcraft` lands in `~/.local/bin`.
2. They clone the repo (or the CLI detects the repo root), run `nextcraft doctor` — all prerequisites report ✓ with actionable messages for any gap.
3. `nextcraft bootstrap` — pnpm install, ai-service venv via the existing bootstrap.sh, `.env` created from `.env.example`, optional-key warnings (not blockers), mock providers keep the stack runnable keyless.
4. `nextcraft verify` — venv imports, ports, env presence, build readiness all ✓.
5. `nextcraft dev` — the dev stack runs; Ctrl+C stops it (passthrough semantics).
6. On every ship, the Gitea release shows the binary + checksum assets; re-running the one-liner upgrades to the latest binary.
## UX Acceptance Criteria
1. Run/Test output visibly reflects the real sandbox execution (command round-trip to the sandbox, real stdout/stderr), not a replay animation
2. The learner path contains **no mock engine data** — scenarios, canned artifacts, and scripted defense transcripts from v0.2 are gone from runtime
3. Variant statements visibly differ between two learner sessions on the same competency
4. Mic permission flow is graceful: consent prompt, recording indicator, no-mic/typed fallback, and browser-native speech path when no server voice key is configured
5. Defense transcript renders turn-by-turn with latency shown; integrity signals render in coaching (supportive) tone
6. Grade results render as structured per-criterion bars with verdict, from the real trace — not from final-output-only heuristics
7. Pool-full (503) shows an honest "environment busy — retry" state; navigation/unmount destroys sandboxes with no leaks
8. When ai-service is unreachable: inline error + retry on every panel — no crashes, no console errors, no blank UI
9. All new UI uses design tokens, supports dark mode, meets WCAG AA contrast, responsive at 375px / 768px / 1280px
10. `pnpm build` and `pnpm typecheck` pass with zero errors; `pnpm ai:test` green cloud-free and voice-key-free
- `doctor` output lists every prerequisite with ✓/✗ and a **fix hint** on every ✗; exit code 1 if any ✗, 0 otherwise.
- `bootstrap` is **idempotent** — running twice produces the same end state, second run fast (no reinstalls where avoidable).
- `bootstrap` never writes secrets, never blocks on missing optional keys — warns with the exact key names and where to set them.
- `verify` gives a single-glance green/red summary; every red item names the failing command it ran.
- Every command supports `--help`; unknown command/flag exits 2 with usage.
- The one-liner **never hard-fails silently**: any error path (no release, no binary asset, checksum mismatch, platform mismatch) prints a specific message + the source-bootstrap alternative.
- Checksum mismatch = hard stop + explicit "do not run this binary" message.
- PATH hint: if `~/.local/bin` is not on PATH, the installer prints the exact export line to add.
- Binary runs standalone on a box with node NOT installed (SEA self-containment) — `./nextcraft-linux-x64 --version` works.
---
## Phase 1: Bootstrap CLI Core
**Requirements:** REQ-4-001, REQ-4-002
**Goal:** `apps/cli` package with doctor/bootstrap/verify/dev fully working from source (`node dist` + pnpm bin), unit-tested, wired into the monorepo (turbo + root scripts), composing — not duplicating — the existing scripts.
### Wave 1: Package foundation (parallel)
#### Task 1-1-01: CLI package scaffold + entry + dispatch
- **Persona:** cli-engineer — **REQ:** REQ-4-001
- **Files:** `apps/cli/package.json`, `apps/cli/tsconfig.json`, `apps/cli/src/index.ts`, `apps/cli/src/commands/help.ts` (usage text), `apps/cli/tests/dispatch.test.ts`
- **Action:** pnpm workspace package `@nextcraft/cli` (private, `"bin": {"nextcraft": "dist/index.js"}`). Entry: parse argv (hand-rolled, no runtime deps), dispatch to commands, `--help`/`-h`, `--version` (from package.json version), unknown → exit 2 with usage. Exit-code contract: 0 ok / 1 failure / 2 usage. shebang `#!/usr/bin/env node` on the built entry (esbuild banner in P2; for P1 `tsx` runs in dev via package script `"dev": "tsx src/index.ts"`).
- **Verify:** `pnpm --filter @nextcraft/cli test` green (dispatch: routes doctor/bootstrap/verify/dev; unknown exits 2; --help exits 0; --version prints package version); `pnpm typecheck` green.
#### Task 1-1-02: Checks library (pure logic)
- **Persona:** cli-engineer — **REQ:** REQ-4-001, REQ-4-002
- **Files:** `apps/cli/src/checks/check-command.ts`, `apps/cli/src/checks/check-env.ts`, `apps/cli/src/lib/log.ts`, `apps/cli/tests/checks.test.ts`
- **Action:** `check-command`: given a name + optional `--version` probe + a min-version parser, resolve binary on PATH (`which`), semver-ish compare (major.minor tolerant), return `CheckResult {name, ok, found, version, hint}`. `check-env`: diff `.env.example` template keys vs an existing `.env` (missing keys → warn-classified; required-vs-optional classification table from the template's own comments + a static required list of zero keys — all optional per A-210), return per-key results. `log.ts`: `ok(msg)`, `fail(msg, hint)`, `warn(msg)`, `info(msg)` formatters with symbols and consistent alignment. Pure functions — no side effects at import; fs access injected as parameters for testability.
- **Verify:** unit tests green: version compare (>= boundaries), missing binary → ok:false + hint, env diff missing/new/extra keys, required-optional classification.
#### Task 1-1-03: Root + turbo wiring
- **Persona:** backend-engineer — **REQ:** REQ-4-002
- **Files:** root `package.json` (update), `turbo.json` (update), `pnpm-workspace.yaml` (verify apps/* already covered — no change expected)
- **Action:** Add `cli:dev`, `cli:test`, `cli:build`, `cli:typecheck`, `cli:lint` root scripts mirroring the `ai:*` passthrough pattern (D-022/D-037). Turbo tasks for the CLI package: `build` (dependsOn `^build`, outputs `dist/**`), `test`, `typecheck`, `lint` (cache:false, outputs:[] for test — same shape as ai-service). No changes to existing ai:* tasks.
- **Verify:** `pnpm cli:test` + `pnpm cli:typecheck` green from repo root; `pnpm build` still green for web+ai-service (turbo graph unaffected); `pnpm ai:test` still green.
### Wave 2: Commands (depends on Wave 1)
#### Task 1-2-01: doctor command
- **Persona:** cli-engineer — **REQ:** REQ-4-001
- **Files:** `apps/cli/src/commands/doctor.ts`, `apps/cli/tests/doctor.test.ts`
- **Action:** Checks (each with actionable hint): node ≥18 (`process.version`), pnpm ≥8 on PATH (`pnpm --version`), python3 ≥3.11 (`python3 --version` parse), git (`git --version`), corepack available-or-pnpm-present nuance folded into pnpm check, `unshare` binary on PATH (`which unshare` — sandbox fabric needs it; hint explains what breaks without it). Sequential execution with per-check timeout; summary line; exit 1 if any ✗. Runs from any cwd (no repo required — pure environment check).
- **Verify:** unit tests with injected spawn results: all-pass → exit 0 + summary; missing pnpm → ✗ + hint + exit 1; missing unshare → ✗ with sandbox-specific hint.
#### Task 1-2-02: bootstrap command
- **Persona:** cli-engineer — **REQ:** REQ-4-002
- **Files:** `apps/cli/src/commands/bootstrap.ts`, `apps/cli/src/lib/spawn.ts`, `apps/cli/tests/bootstrap.test.ts`
- **Action:** `spawn.ts`: `run(cmd, args, {timeoutMs, cwd, env})` — promisified child_process.spawn, inherited stdio, timeout kill (SIGTERM→SIGKILL escalation), returns `{code}`; throws never (codes always returned). `bootstrap.ts` steps (each logged before/after): (1) locate repo root (walk up for pnpm-workspace.yaml; error with hint if not in a clone); (2) `pnpm install` at root; (3) delegate ai-service venv to `apps/ai-service/scripts/bootstrap.sh` via spawn with generous timeout (10 min) — **zero pip/venv logic in the CLI** (A-202); (4) copy `.env.example``.env` if absent (preserve existing; report created vs kept); (5) validate optional keys in `.env` vs template — warn-only (A-210); never touch `.ciagent/.env.secrets`; (6) print next-steps (`nextcraft verify`, `nextcraft dev`). Idempotent: every step safe to re-run.
- **Verify:** unit tests with stub spawn: step order, env copy semantics (absent → create, present → keep), timeout path returns failure code, secrets file never written; `pnpm cli:test` green.
#### Task 1-2-03: verify + dev commands
- **Persona:** cli-engineer — **REQ:** REQ-4-002
- **Files:** `apps/cli/src/commands/verify.ts`, `apps/cli/src/commands/dev.ts`, `apps/cli/tests/verify.test.ts`
- **Action:** `verify.ts` health checks (each runnable + reported): ai-service venv python imports (`import ai_service` via venv python), uvicorn present in venv, ports 3000/8420 free (net stat via node), `.env` exists with AI_PORT parseable, `pnpm build` dry readiness (turbo graph parses — run `turbo build --dry=json` cheap check or typecheck-only default; choose the cheap one). Summary + exit code. `dev.ts`: locate repo root, exec passthrough to `apps/ai-service/scripts/dev.sh` with **inherited stdio and signals** (Ctrl+C semantics), no timeout (long-running); document that web dev server runs via `pnpm dev` separately (dev.sh owns ai-service only).
- **Verify:** unit tests: verify aggregates check results → exit codes; dev spawns dev.sh with signal passthrough assertions (mock spawn).
### Wave 3: Integration review (depends on Wave 2)
#### Task 1-3-01: CLI security + integration review pass
- **Persona:** security-auditor — **REQ:** REQ-4-001, REQ-4-002
- **Files:** `apps/cli/src/lib/spawn.ts` (review; patch if defect), `apps/cli/src/commands/bootstrap.ts` (review), `apps/cli/tests/**` (add regression if defect found)
- **Action:** STRIDE pass on the CLI surface: spawn injection (args never through shell string — array form only), timeout enforcement, secrets never logged, env template copy doesn't overwrite user edits, no shell=true anywhere, PATH resolution honest errors. Findings → P0 patches now with regression tests; P1+ noted for final-phase review.
- **Verify:** `pnpm cli:test` green incl. any added regressions; `grep -rn "shell: *true" apps/cli/src` returns nothing.
### Must-Haves (Phase 1)
- [ ] `pnpm --filter @nextcraft/cli test` green; `pnpm typecheck` green; `pnpm build` green
- [ ] doctor: every prerequisite reported with ✓/✗ + actionable hint; exit 1 on any ✗; runs outside a repo clone
- [ ] bootstrap: composes scripts/bootstrap.sh (no pip/venv logic in CLI); idempotent; .env created from template only when absent; optional-key warnings, never blocks; never writes secrets
- [ ] verify: venv import + uvicorn + ports + env checks with single-glance summary and named failing commands
- [ ] dev: passthrough with signal inheritance (Ctrl+C stops the stack)
- [ ] Exit-code contract: 0/1/2; --help everywhere; unknown command → 2
- [ ] No runtime npm dependencies in apps/cli (dev deps only)
- [ ] Root `cli:*` scripts work from repo root; ai:* scripts unaffected
---
## Phase 2: Binary Build + Release Pipeline
**Requirements:** REQ-4-003, REQ-4-004
**Goal:** `nextcraft-linux-x64` SEA binary + sha256 sidecar built reproducibly from the CLI package; one-liner `install.sh` verified end-to-end against a real release; release-asset upload wired so **every ship from now on carries binaries**.
### Wave 1: Binary build (parallel)
#### Task 2-1-01: SEA binary build script (G-101: live-build probe FIRST — mechanism must be proven before the pipeline depends on it)
- **Persona:** cli-engineer — **REQ:** REQ-4-004
- **Files:** `apps/cli/scripts/build-binary.mjs`, `apps/cli/package.json` (add `build:binary` script), `apps/cli/.sea-config.json` (or generated in-script)
- **Action:** **First action of this task: build one real SEA binary end-to-end and run it** (`--version` + `doctor` smoke) before writing the polished script.** Pipeline: esbuild bundle `src/index.ts``dist/bundle.cjs` (platform node, target node18, banner shebang, SEA config: `{main: "dist/bundle.cjs", output: "dist/sea-prep.blob", disableExperimentalSEAWarning: true}`) → `node --experimental-sea-config` → copy system node binary → inject blob (`npx postject` with sentinel `NODE_SEA_BLOB_FUSE` fuse, or `dd` fallback) → chmod +x → `dist/nextcraft-linux-x64`**stamp version from the shipping tag argument** (`NEXTCRAFT_VERSION` injected via esbuild `define`, G-102 — `--version` prints it; absent arg → dev stamp `0.0.0-dev`) → `shasum -a 256``dist/nextcraft-linux-x64.sha256`. Fallback (documented, scripted, honest): if SEA injection fails, python3 zipapp builds `nextcraft-linux-x64.pyz` (requires python3 on target — install.sh handles both asset shapes and the docs say so; NO silent claim of node-less operation, G-101).
- **Verify:** `pnpm --filter @nextcraft/cli build:binary` produces the binary; `./dist/nextcraft-linux-x64 --version` runs **with node absent from PATH** (test via `env -i /bin/sh -c 'PATH=/usr/bin:/bin ...'` sandbox or by temporarily stripping PATH in a subprocess test); sha256 file matches `shasum -c`.
#### Task 2-1-02: Release-asset upload helper
- **Persona:** cli-engineer — **REQ:** REQ-4-004
- **Files:** `scripts/release-assets.sh`, `apps/cli/tests/release-assets.test.ts` (fixture-level)
- **Action:** Given a tag: build binary (Task 2-1-01), resolve GITEA_TOKEN from `.env`/`.env.secrets`/`.env.*` **via the secrets loader only** (never shell env — v1.8 root cause), create/locate the Gitea release via API, upload both assets (`POST /api/v1/repos/{owner}/{repo}/releases/{id}/assets?name=...` multipart). Bounded retry (3) per config.ship.max_release_retries; token never echoed; failure = non-blocking escalation message (release_pending semantics) — tag+merge already complete the ship.
- **Verify:** fixture test: token resolution order (.env.secrets wins over .env; shell env NEVER consulted — assert with a poisoned env var fixture); dry-run mode prints the exact curl-multipart it would send (no net in tests).
#### Task 2-1-03: install.sh one-liner
- **Persona:** cli-engineer — **REQ:** REQ-4-003
- **Files:** `scripts/install.sh`, `apps/cli/tests/install-script.test.ts`
- **Action:** POSIX sh (no bashisms — dash-safe): `set -eu`; platform check (uname linux + x86_64; else print source-bootstrap path + exit 0 — a graceful no-op, not an error); resolve latest release via Gitea API (`curl -fsSL .../releases/latest`, parse `tag_name` + asset `browser_download_url`s with sed/grep — no jq dependency); **match assets by EXACT name** (`nextcraft-linux-x64`, `nextcraft-linux-x64.sha256` — any parse/lookup miss = degrade to source-bootstrap instructions, exit 0, G-103 — never a name-approximate install); handle the zipapp asset shape (`nextcraft-linux-x64.pyz` + sidecar) when the binary is absent, printing the python3 requirement honestly; download both assets to `mktemp -d` (trap cleanup EXIT); **verify sha256 before anything else** (`shasum -a 256 -c` or sha256sum); on mismatch → hard stop, explicit "do not run" message, exit 1; install to `~/.local/bin` (mkdir -p; `--dest` override); PATH hint when missing (print exact export line); print the binary's own `--version` output (G-102: must equal the resolved release tag — mismatch = install-time integrity stop) + `nextcraft doctor` next-step. No-binary-asset path: print the git-clone + scripts/bootstrap.sh instructions + exit 0. Zero secrets required (public release assets).
- **Verify:** unit tests over the script's pure helpers extracted where feasible; **live E2E in Task 2-3-01**. `sh -n scripts/install.sh` syntax-clean; `dash scripts/install.sh --help` safe if dash present.
### Wave 2: Ship-flow integration (depends on Wave 1)
#### Task 2-2-01: Wire binaries into every ship (G-104: enforcement, not prose)
- **Persona:** backend-engineer — **REQ:** REQ-4-004
- **Files:** `.ciagent/config.json` (no schema change needed — release section already configured), this repo's ship procedure notes (update `.ciagent/ARCHITECTURE.md` Build Order note if needed), `scripts/release-assets.sh` (finalize from 2-1-02)
- **Action:** Establish the ship-time contract going forward: after every phase ship (tag + merge complete = ship gate per config.ship), run `scripts/release-assets.sh <tag>` to attach binary + checksum to the freshly created release. **G-104:** this run is MANDATORY-ATTEMPTED on every release from v0.3.2 onward — best-effort/non-blocking like release creation (release_pending escalation on exhaustion), logged in the ship commit, and the P4 final audit gate includes "milestone release carries both assets" as an explicit check. This makes "ongoing binaries" a property of the pipeline, not a one-off.
- **Verify:** The P2 ship itself executes the step against tag v0.3.2 (live validation — see Ship).
### Wave 3: End-to-end validation (depends on Wave 2)
#### Task 2-3-01: Install E2E against the live release
- **Persona:** security-auditor — **REQ:** REQ-4-003
- **Files:** `apps/cli/tests/install-e2e.test.ts` (marked slow/e2e), `apps/cli/README.md` (install internals section)
- **Action:** Live E2E after the v0.3.2 release exists (run post-ship, documented as the verify gate for this phase's asset path): fresh HOME tmpdir → run install.sh → assert binary at `$HOME/.local/bin/nextcraft`, `--version` output equals the release tag (G-102 integrity assertion), checksum verified path taken (tamper test: flip a byte in a local fixture download → script refuses + exits 1). Record the transcript in the phase verify commit. If the live release isn't reachable at verify time, run the full local equivalent (serve assets from a fixture dir via `python3 -m http.server` + FORGE_BASE override) and mark live re-check as a P1 follow-up.
- **Verify:** E2E green locally (fixture server path mandatory in tests — no test depends on the live forge); tamper-rejection proven; transcript recorded.
### Must-Haves (Phase 2)
- [ ] **G-101:** a real SEA binary built + smoke-run BEFORE the pipeline depends on it; if SEA fails, zipapp is primary and docs state the python3 requirement
- [ ] **G-102:** binary `--version` reports the shipping tag (stamped at build); install E2E asserts version == release tag
- [ ] `pnpm --filter @nextcraft/cli build:binary` produces `nextcraft-linux-x64` + `.sha256`; binary runs without node on PATH (`--version`, `doctor` smoke)
- [ ] **G-103:** `sh -n scripts/install.sh` clean; dash-safe; exact-name asset matching; platform mismatch → graceful source-bootstrap path (exit 0)
- [ ] Checksum verified before install; tamper → hard stop with explicit warning (E2E-proven)
- [ ] install.sh resolves latest release + assets from the Gitea API with zero secrets and no jq
- [ ] release-assets.sh resolves GITEA_TOKEN from .env* files only (never shell env — tested with poisoned env)
- [ ] **G-104:** v0.3.2 release carries both assets (live validation at ship); upload failure is non-blocking escalation, attempted + logged every release
- [ ] `pnpm build`, `pnpm typecheck`, `pnpm cli:test` all green
---
## Phase 3: Install Docs + Fresh-Clone E2E
**Requirements:** REQ-4-005
**Goal:** README quickstart + CLI reference matching the tested reality exactly, plus a fresh-clone E2E test proving the happy path end-to-end.
### Wave 1: Fresh-clone E2E (drives doc accuracy)
#### Task 3-1-01: Fresh-clone bootstrap E2E
- **Persona:** cli-engineer — **REQ:** REQ-4-005
- **Files:** `apps/cli/tests/fresh-clone-e2e.test.ts` (slow/e2e-marked)
- **Action:** In a `mktemp -d` sandbox: `git clone` the repo locally (file:// clone of HEAD — no network), run `pnpm --filter @nextcraft/cli dev -- doctor` (or the built binary from P2) → then `bootstrap` → then `verify`, asserting each step's exit codes and key output markers. Skips gracefully when network-dependent steps are unavailable (CI marker). Documents the exact happy path the README will state.
- **Verify:** E2E green locally (clone of the working tree); output transcript matches README claims (cross-checked in 3-2-01).
### Wave 2: Documentation (depends on Wave 1 transcript)
#### Task 3-2-01: README quickstart + CLI reference
- **Persona:** backend-engineer — **REQ:** REQ-4-005
- **Files:** root `README.md` (update quickstart section), `apps/cli/README.md` (CLI reference)
- **Action:** Root README quickstart: the one-liner (exact tested URL), then doctor → bootstrap → verify → dev sequence with expected outputs; source-bootstrap alternative documented (clone + scripts). apps/cli README: every command, flags, exit codes, the env-template copy semantics, optional-key warning semantics, secrets policy (never generated/committed; .ciagent/.env.secrets location), binary install internals, troubleshooting table keyed to actual failure modes observed in E2E.
- **Verify:** Every command line in both READMEs is copy-paste runnable — verified against the 3-1-01 transcript; doc drift check: no references to commands/flags that don't exist in `--help` output.
### Must-Haves (Phase 3)
- [ ] Fresh-clone E2E green: doctor → bootstrap → verify sequence from a clean clone
- [ ] README quickstart matches the E2E transcript exactly (no aspirational docs)
- [ ] CLI reference covers all 4 commands + --help/--version + exit codes
- [ ] `pnpm build`, `pnpm typecheck`, `pnpm test` (all suites) green
---
## Phase 4: Final Review + Ship (milestone release v0.3.4)
1. Branch gate → `phase/04-final-review-ship`.
2. Multi-persona review across the milestone (correctness, testing, security, performance, maintainability, adversarial) — P0 auto-fixed, P1+ fixed in this phase.
3. Audit: reconstruction test (.ciagent files ↔ git log), file discipline, branch hygiene, commit discipline, P0-review flags resolved, **G-104 gate: milestone release v0.3.4 carries `nextcraft-linux-x64` + `.sha256` assets**.
4. Milestone ship: merge phase/04 → milestone/v0.4-distribution; merge milestone → main; tag **v0.3.4** (= milestone release); attach binary + checksum assets (the ongoing-binaries contract); release notes with full milestone summary (all phases, all REQ-4-001..005, the "ongoing binaries from now on" statement, v0.5 deferral list per D-016); delete all milestone/phase branches.
5. Complete: REQUIREMENTS.md REQ-4-001..005 → complete; ROADMAP.md v0.4 → complete; checkpoint cleared.
## Must-Haves (Milestone)
- [ ] One-liner installs a working binary from the live Gitea release (E2E-proven, tamper-tested)
- [ ] Fresh clone → doctor → bootstrap → verify → dev: the full happy path green from a clean environment
- [ ] Every release from v0.3.2 onward carries `nextcraft-linux-x64` + `.sha256` assets
- [ ] Zero runtime npm deps in the CLI; secrets only ever from .env* files; never in code/logs/commits
- [ ] All suites green: `pnpm build`, `pnpm typecheck`, `pnpm ai:test`, `pnpm cli:test`
+37 -17
View File
@@ -8,45 +8,64 @@ Nextcraft is an AI-native outcome school where graduates prove what they can bui
---
## Current Milestone: v0.3Credential Engines
## Current Milestone: v0.5Real Server Voice, KYC/Identity, Sandbox Environments, Seq-Lease
**Scope:** Replace v0.2's mock engine inputs with real credential engines. Build the sandbox fabric (sandboxed IDE / design tool / simulation), the live in-environment build-telemetry pipeline, the process-trace grading engine, per-learner variant task generation, and the oral/voice defense with AI examiner. Lab/Assessor/Proctor agents move from mock inputs to real engine inputs; the six tutor agents operate on authentic telemetry and artifacts.
**Scope (deferred seams from D-016, to be specified at v0.5 Phase 0):** real server STT/TTS (`openai-audio` voice provider, CUT-1/G-7 seam), KYC/identity verification + age-gating backend (REQ-F-017), design/simulation sandbox environments (REQ-F-021 remainder), exec-telemetry seq-lease/replay-margin fix.
## Prior Milestone: v0.4 — Distribution & Bootstrap CLI (COMPLETE, shipped as v0.3.4)
**Scope (founder directive, 2026-09-12):** Streamline installing Nextcraft. Ship a bootstrap CLI with a single-liner install script, and publish release binaries on an ongoing basis for every release going forward.
**Delivered:** `nextcraft` CLI — `doctor` (prerequisite checks), `bootstrap` (deps + venv + env from templates + key validation), `verify` (health check), `dev` (thin passthrough to scripts/dev.sh); one-liner install script downloading the linux x64 binary from the latest Gitea release with sha256 + version integrity gates; binary release pipeline attached to every ship from v0.3.2 onward; install/quickstart documentation backed by a fresh-clone E2E test. All 5 requirements (REQ-4-001..005) complete.
**Status of v0.3:** Complete and shipped (v0.2.8). Credential engines live: namespace-isolated sandbox fabric, live build telemetry, process-trace grading, seeded variants, oral defense; real learner build/defense/grading surfaces.
**Status of v0.2:** Complete and shipped (v0.2.0). Six AI tutor agents live over mockengine inputs (D-015).
**Deferred from earlier plan:** REQ-F-017 (identity verification + 16+/18+ age-gating) is explicitly deferred to a later milestone per founder directive. Age-gating remains the v0.1-style visual flow mockup; no real KYC backend is built in v0.3.
**Deferred from earlier plan:** REQ-F-017 (identity verification + age-gating KYC), real server STT/TTS, design/simulation sandbox environments, and the exec-telemetry seq-lease are all deferred to v0.5. Age-gating remains the v0.1-style visual flow mockup.
**Tech stack:** v0.1 TS monorepo (pnpm/turborepo, Next.js) + v0.2 Python FastAPI ai-service + new credential-engine services (sandbox fabric orchestrator, telemetry ingest, grading engine) in Python/TypeScript as determined at RESEARCH.
---
## Requirements (Validated)
## v0.4 Requirements (Complete)
The following requirements have been validated during specification and are locked for milestone v0.3 (REQ-F-007..010 and REQ-F-021 activated from the deferred pool; REQ-F-017 deferred per founder directive):
All 5 v0.4 requirements (REQ-4-001..005) are complete and shipped as v0.3.4:
1. Bootstrap CLI — `nextcraft` executable with `doctor` / `bootstrap` / `verify` / `dev` (REQ-4-001, REQ-4-002)
2. One-liner install — `curl | sh` fetching the linux x64 binary from the latest Gitea release with sha256 + version integrity verification (REQ-4-003)
3. Ongoing release binaries — every release from v0.3.2 onward ships the CLI binary + checksum as release assets (REQ-4-004)
4. Install documentation — README quickstart + CLI reference verified by a fresh-clone E2E test (REQ-4-005)
1. Sandbox fabric — sandboxed IDE, design tool, and simulation environments with isolated execution and lifecycle management (REQ-F-021)
2. Live build telemetry — in-environment capture of process events (keystrokes, commands, file diffs, run/test results) streamed to ai-service (REQ-F-010)
3. Process-trace grading engine — grade artifacts from their process traces, not just final output (REQ-F-007); feeds the Assessor agent real inputs
4. Variant task generation — per-learner task variants so no two learners receive identical prompts (REQ-F-008)
5. Oral/voice defense — AI examiner conducts spoken defense of submitted work (REQ-F-009); feeds the Proctor/Mentor agents
6. Agent re-grounding — Lab/Assessor/Proctor consume real engine inputs (telemetry, traces, defenses) instead of v0.2 mocks
7. Learner surface integration — wire the v0.1 sandbox + assessment mockups to the real engines (build/run in-browser, live telemetry, live defense)
## v0.3 Requirements (Complete)
## v0.2 Requirements (Complete)
All 12 v0.2 requirements (REQ-2-001..012) are complete and shipped as v0.2.0. See REQUIREMENTS.md traceability matrix.
All 8 v0.3 requirements (REQ-3-001..008) are complete and shipped as v0.2.8. See REQUIREMENTS.md traceability matrix.
## v0.1 Requirements (Complete)
All 28 v0.1 requirements (REQ-001..028) are complete and shipped as v0.1.0. See REQUIREMENTS.md traceability matrix.
## Clarified Assumptions (v0.4 CLARIFY stage, full autonomy — auto-resolved)
| # | Ambiguity | Resolution | Confidence |
|---|-----------|------------|-------------|
| A-201 | CLI language/toolchain for the binary? | **Probe-driven at RESEARCH** — Go → Rust → Node SEA → Python zipapp fallback chain; spec stays toolchain-agnostic so PLAN locks the probe-verified toolchain | 0.70 |
| A-202 | Does bootstrap replace scripts/bootstrap.sh? | **No — reuse it.** CLI wraps existing `scripts/bootstrap.sh` + `scripts/dev.sh` via subprocess; zero orchestration logic duplicated in the CLI (thin passthrough pattern) | 0.85 |
| A-203 | Where does the one-liner fetch the binary? | **Gitea latest-release API** (`/repos/{owner}/{repo}/releases/latest`) → download `nextcraft-linux-x64` + `.sha256` asset; repo raw serves `install.sh` as the stable URL | 0.80 |
| A-204 | Install target + PATH? | **~/.local/bin** (XDG-style, no sudo), PATH hint printed when missing; `--dest` override flag | 0.85 |
| A-205 | Binary "ongoing releases" scope? | **Every ship from v0.4 onward** attaches `nextcraft-linux-x64` + sha256 sidecar to the Gitea release — the ship workflow gains an asset step; retroactive binaries for old releases NOT required | 0.90 |
| A-206 | No binary available yet / non-linux? | **Graceful degradation**: install script prints source-bootstrap instructions (git clone + scripts/bootstrap.sh) — never a hard fail | 0.88 |
| A-207 | Checksum trust root? | **sha256 sidecar shipped as a release asset next to the binary** (same release, same channel); script verifies download against it. Signature/PKI out of scope for v0.4 (single forge, TLS transport) | 0.75 |
| A-208 | Which prerequisites does doctor check? | node ≥18, pnpm ≥8, python3 ≥3.11, git, `unshare` availability (sandbox fabric needs it) — versions from the existing bootstrap tooling, not invented | 0.85 |
| A-209 | Does `dev` manage multiple processes? | **No.** Thin passthrough to scripts/dev.sh only — the CLI stays bootstrap-scoped (D-016); orchestration remains in dev.sh | 0.82 |
| A-210 | `.env.secrets` handling by bootstrap? | **Template copy only for `.env.example` → `.env`; secrets NEVER generated, NEVER committed; bootstrap validates presence of optional keys and warns (not blocks) when missing — mock-first providers keep the stack runnable** | 0.90 |
## Clarified Assumptions (v0.3 CLARIFY stage, full autonomy — auto-resolved)
| # | Ambiguity | Resolution | Confidence |
|---|-----------|------------|-------------|
| A-101 | Sandbox isolation technology? | **`unshare` user+mount+pid+net namespace subprocess isolation** per sandbox (probe-verified: in-ns uid=0, network fully isolated with 0 interfaces, writes land in an isolated bind-mounted workdir; proc-remount is not permitted in this context but is not required). No Docker/Podman/VMs — none present on the box; no sudo. A `SandboxBackend` protocol keeps a future containerd swap possible. Falls back further to a plain chroot-free subprocess with a cwd-jail if userns ever unavailable (tested path is userns). | 0.8 |
| A-102 | Sandbox scope in v0.3? | **Coding IDE only** (web terminal + file tree + run/test). The "design tool" and "simulation" environments specified in REQ-F-021 are deferred to v0.4 — a single real build environment is enough to prove the credential pipeline end-to-end (telemetry → trace → grade → defense). | 0.75 |
| A-103 | Live in-browser build UX? | **WebSocket xterm.js terminal** attached to the bwrap sandbox shell + HTTP file-tree/CRUD + run/test buttons. No full Monaco LSP in v0.3 — a code editor with syntax highlight (existing) + real shell is sufficient and far cheaper. | 0.72 |
| A-103 | Live in-browser build UX? | **Run/Test buttons executing in the namespace sandbox + HTTP file-tree/CRUD + read-only exec-output panel** (CUT-2/G-8 — the interactive xterm.js shell relay is deferred to v0.4; `@xterm/*` is not a v0.3 dependency). No full Monaco LSP in v0.3 — a code editor with syntax highlight (existing) is sufficient and far cheaper. | 0.72 |
| A-104 | Telemetry transport? | **WebSocket** from sandbox to a new ingestion endpoint on ai-service for live events; **SQLite-backed** ordered event log (`ai_service/telemetry/`) gives durability + at-least-once delivery + replay. Events carry monotonic `seq` per (learner,task) so gaps are detectable. | 0.8 |
| A-105 | Where do traces live? | **SQLite** (`ai_service` data dir), introducing the first real persistence. SQLModel/SQLAlchemy for typed access. Chosen over Postgres because solo-founder + single box + low write volume; the `TraceStore` protocol is Postgres-migration-ready like SessionStore was. | 0.75 |
| A-106 | Process-trace grading model? | **LLM-based grader**: structure the trace into a compact timeline digest (command categories, error/fix cycles, idle gaps, test passes) → Assessor-style rubric prompt → structured score via existing D-020 JSON defense. Deterministic features (test pass/fail, edit count) computed in code, not left to the LLM. | 0.7 |
@@ -54,7 +73,7 @@ All 28 v0.1 requirements (REQ-001..028) are complete and shipped as v0.1.0. See
| A-108 | Voice defense — STT/TTS providers? | **Provider-agnostic, mock-first like the LLM layer (D-014).** Real path: browser `MediaRecorder` → audio to ai-service → **OpenAI-compatible `/audio/transcriptions`** (Whisper STT) and **`/audio/speech`** (TTS) against ollama-cloud or a compatible endpoint; fallbacks: browser `SpeechRecognition`/`speechSynthesis` when no server keys. `VoiceProvider` protocol + deterministic mock (returns canned transcript) so tests never call a voice API. | 0.62 |
| A-109 | Defense dialogue shape? | Reuse BaseAgent: an `Examiner` agent (seventh agent) streams examiner questions over the existing SSE pipeline; integrity signals (long pauses, off-scope answers, reading-from-notes cadence) emitted alongside the transcript to Proctor. | 0.8 |
| A-110 | KYC / age-gating in v0.3? | **Deferred per founder directive.** No real identity backend. Age-gating stays the v0.1 visual flow mockup. Personas omit a security-engineer; security review via verifier + Phase 7 secrets-hygiene checklist. **Abuse control is NOT deferred with KYC (G-5):** v0.3 ships per-learner sandbox caps (`AI_SANDBOX_MAX_PER_LEARNER`), a global create-rate cap, and a server-side `learner_id` allowlist (`AI_LEARNER_ALLOWLIST`) so the unauthenticated surface cannot exhaust shared NPROC/disk. Documented in the release note. | 0.98 |
| A-111 | New services vs extend ai-service? | **Extend ai-service**, don't fork new Python apps. Telemetry ingestion, trace grading, variant generation, voice, and sandbox orchestration all live as new modules in `apps/ai-service` (they share the LLM provider pool + config + session infra). Only the in-sandbox capture agent is a separate tiny Python process shipped into the bwrap environment. | 0.82 |
| A-111 | New services vs extend ai-service? | **Extend ai-service**, don't fork new Python apps. Telemetry ingestion, trace grading, variant generation, voice, and sandbox orchestration all live as new modules in `apps/ai-service` (they share the LLM provider pool + config + session infra). Only the in-sandbox capture agent is a separate tiny Python process shipped into the namespace sandbox. | 0.82 |
| A-112 | Sandbox on a single dev/school box — capacity? | v0.3 targets **15 concurrent sandboxes** (founder + pilot learners). No horizontal scaling, no queue. Concurrency guard returns 503 when full. Scaling is post-MVP. | 0.8 |
## Clarified Assumptions (v0.2 CLARIFY stage, full autonomy — auto-resolved)
@@ -133,6 +152,7 @@ The following remain deferred beyond v0.3 and will be activated in subsequent mi
| D-013 | v0.1 prototype founder-agreed; D-001 business-logic gate unlocked | Founder approved starting v0.2 with AI Tutor Architecture, which constitutes agreement of the v0.1 prototype per D-001. Recorded at v0.2 SPECIFY. | Business logic authorized from v0.2 onward |
| D-014 | Provider-agnostic LLM layer; ollama-cloud as initial provider | OpenAI-compatible client abstraction with pluggable providers: ollama-cloud (https://ollama.com/v1, default), local OpenAI-compatible endpoint, deterministic mock (tests/CI). Keys in gitignored .ciagent/.env.secrets, never in code or commits. | apps/ai-service llm package with 3 providers; default=ollama-cloud |
| D-015 | All six agents implemented as real LLM services; engines mocked | Coach/Tutor/Mentor fully real. Lab/Assessor/Proctor are real LLM logic over mock inputs (simulated telemetry, pre-baked artifacts) since sandbox fabric, assessment engine, and identity verification are v0.3+. Consistent with v0.1's mock-data approach. | REQ-F-001..006 complete in v0.2; real engines deferred to v0.3+ |
| D-016 | v0.4 = Distribution & Bootstrap CLI (founder directive supersedes previously-named v0.4 seams) | Founder directive 2026-09-12: focus this milestone on streamlining install, a bootstrap CLI with a one-liner install script, ongoing release binaries. Real server STT/TTS, KYC, design/simulation envs, seq-lease move to v0.5. | Milestone scope locked at SPECIFY; binary = CLI-only, linux x64 |
---
+59 -32
View File
@@ -1,5 +1,22 @@
# Nextcraft — REQUIREMENTS.md
## v0.4 Requirements (Complete — Distribution & Bootstrap CLI, shipped as v0.3.4)
### Bootstrap CLI
| ID | Description | Priority | Phase | Status |
|----|-------------|----------|-------|--------|
| REQ-4-001 | `nextcraft` CLI (linux x64 binary): `doctor` command checking prerequisites (node, pnpm, python3, git, unshare) with actionable error messages | critical | 1 | complete |
| REQ-4-002 | `bootstrap` command: pnpm install, ai-service venv + pinned deps, .env from templates, key validation, .env.secrets handling; `verify` health check (ports, imports, builds); `dev` thin passthrough to scripts/dev.sh | critical | 1 | complete |
### Distribution
| ID | Description | Priority | Phase | Status |
|----|-------------|----------|-------|--------|
| REQ-4-003 | One-liner install script (`curl -fsSL <url> \| bash`): detects linux x64, resolves latest release from Gitea API, downloads binary + checksum, verifies sha256, installs to ~/.local/bin (PATH hint), degrades to source-bootstrap instructions when no binary | critical | 2 | complete |
| REQ-4-004 | Binary release pipeline: reproducible linux x64 build script, sha256 checksum sidecar, upload as release assets on every ship from v0.4 onward (ongoing binaries requirement) | critical | 2 | complete |
| REQ-4-005 | Install + quickstart documentation: README one-liner quickstart, CLI command reference, fresh-clone-to-running-stack end-to-end verification | high | 3 | complete |
## v0.3 Requirements (Credential Engines)
### Sandbox & Telemetry
@@ -22,8 +39,8 @@
| ID | Description | Priority | Phase | Status |
|----|-------------|----------|-------|--------|
| REQ-3-007 | Agent re-grounding: Lab consumes live telemetry; Assessor consumes grading-engine output; Proctor consumes telemetry + defense integrity signals (replace v0.2 mocks) | critical | 6 | pending |
| REQ-3-008 | Learner surface integration: sandbox mockup → real in-browser build/run with live telemetry; assessment mockup → live defense + live grading | critical | 6 | pending |
| REQ-3-007 | Agent re-grounding: Lab consumes live telemetry; Assessor consumes grading-engine output; Proctor consumes telemetry + defense integrity signals (replace v0.2 mocks) | critical | 6 | complete |
| REQ-3-008 | Learner surface integration: sandbox mockup → real in-browser build/run with live telemetry; assessment mockup → live defense + live grading | critical | 6 | complete |
---
@@ -65,58 +82,58 @@
| ID | Description | Priority | Phase | Status |
|----|-------------|----------|-------|--------|
| REQ-001 | Monorepo scaffolding: pnpm workspaces, turborepo, Next.js app, TypeScript config, ESLint, Prettier | critical | 1 | complete |
| REQ-002 | Shared component library: design tokens, Button, Input, Card, Badge, Avatar, Dialog, Navigation, Table, Tabs, Progress, Tooltip, Skeleton, Toast | critical | 1 | pending |
| REQ-003 | Mock data layer: typed mock data for competency stacks, job listings, candidate profiles, employers, learner progress | critical | 1 | pending |
| REQ-004 | Routing and navigation: App Router route groups for (learner), (marketplace), (employer), (admin); shared layout components; cross-surface navigation | critical | 1 | pending |
| REQ-005 | Responsive layout system: mobile, tablet, desktop breakpoints; container components; grid system | high | 1 | pending |
| REQ-002 | Shared component library: design tokens, Button, Input, Card, Badge, Avatar, Dialog, Navigation, Table, Tabs, Progress, Tooltip, Skeleton, Toast | critical | 1 | complete |
| REQ-003 | Mock data layer: typed mock data for competency stacks, job listings, candidate profiles, employers, learner progress | critical | 1 | complete |
| REQ-004 | Routing and navigation: App Router route groups for (learner), (marketplace), (employer), (admin); shared layout components; cross-surface navigation | critical | 1 | complete |
| REQ-005 | Responsive layout system: mobile, tablet, desktop breakpoints; container components; grid system | high | 1 | complete |
### Learner Surface
| ID | Description | Priority | Phase | Status |
|----|-------------|----------|-------|--------|
| REQ-006 | Landing page: hero, value proposition, program highlights, how-it-works (Byte→Build→Demonstrate→Defend), testimonials mockup, CTA to program catalog | critical | 2 | pending |
| REQ-007 | Program catalog: grid of competency stacks (AI Orchestration Engineer, AI Safety & Governance Lead, Human-AI Product Designer, AI-Augmented Field Operator, Computational Sciences Practitioner); stack cards with role descriptions | critical | 2 | pending |
| REQ-008 | Competency stack view: selected stack detail with 12-18 competencies listed, progress indicators, microcredential badges, mastery status | critical | 2 | pending |
| REQ-009 | Learner dashboard: active competencies, progress graph, recent artifacts, upcoming defenses, AI tutor chat mockup, milestone tracker | critical | 2 | pending |
| REQ-010 | Byte tutorial viewer: 3-7 minute micro-tutorial layout with concept panel, worked example panel, code/design/simulation viewer mockup | high | 2 | pending |
| REQ-011 | Build sandbox mockup: sandboxed IDE/design tool/simulation UI mockup with toolbar, file explorer, editor area, telemetry sidebar (process capture indicators) | high | 2 | pending |
| REQ-012 | Assessment/defense mockup: rubric display, AI reviewer panel, oral defense interface (voice/mic mockup), process trace timeline, artifact viewer | high | 2 | pending |
| REQ-006 | Landing page: hero, value proposition, program highlights, how-it-works (Byte→Build→Demonstrate→Defend), testimonials mockup, CTA to program catalog | critical | 2 | complete |
| REQ-007 | Program catalog: grid of competency stacks (AI Orchestration Engineer, AI Safety & Governance Lead, Human-AI Product Designer, AI-Augmented Field Operator, Computational Sciences Practitioner); stack cards with role descriptions | critical | 2 | complete |
| REQ-008 | Competency stack view: selected stack detail with 12-18 competencies listed, progress indicators, microcredential badges, mastery status | critical | 2 | complete |
| REQ-009 | Learner dashboard: active competencies, progress graph, recent artifacts, upcoming defenses, AI tutor chat mockup, milestone tracker | critical | 2 | complete |
| REQ-010 | Byte tutorial viewer: 3-7 minute micro-tutorial layout with concept panel, worked example panel, code/design/simulation viewer mockup | high | 2 | complete |
| REQ-011 | Build sandbox mockup: sandboxed IDE/design tool/simulation UI mockup with toolbar, file explorer, editor area, telemetry sidebar (process capture indicators) | high | 2 | complete |
| REQ-012 | Assessment/defense mockup: rubric display, AI reviewer panel, oral defense interface (voice/mic mockup), process trace timeline, artifact viewer | high | 2 | complete |
### Marketplace Surface
| ID | Description | Priority | Phase | Status |
|----|-------------|----------|-------|--------|
| REQ-013 | Job board listing: searchable grid of AI-era job listings, filter sidebar (skills, seniority, location, salary), result cards with match score | critical | 3 | pending |
| REQ-014 | Job detail page: full job description, required competencies, employer info, application CTA, related jobs, AI-matched skills breakdown | critical | 3 | pending |
| REQ-015 | Employer profile: company overview, logo, description, social links, open positions, company culture mockup | high | 3 | pending |
| REQ-016 | Search/filter UI: semantic search bar, skill tags, category filters, seniority filter, remote/on-site toggle, saved searches mockup | critical | 3 | pending |
| REQ-017 | Pricing page: job posting packages (single, bundle, enterprise), talent access plans, subscription tiers, feature comparison table | high | 3 | pending |
| REQ-013 | Job board listing: searchable grid of AI-era job listings, filter sidebar (skills, seniority, location, salary), result cards with match score | critical | 3 | complete |
| REQ-014 | Job detail page: full job description, required competencies, employer info, application CTA, related jobs, AI-matched skills breakdown | critical | 3 | complete |
| REQ-015 | Employer profile: company overview, logo, description, social links, open positions, company culture mockup | high | 3 | complete |
| REQ-016 | Search/filter UI: semantic search bar, skill tags, category filters, seniority filter, remote/on-site toggle, saved searches mockup | critical | 3 | complete |
| REQ-017 | Pricing page: job posting packages (single, bundle, enterprise), talent access plans, subscription tiers, feature comparison table | high | 3 | complete |
### Employer Dashboard
| ID | Description | Priority | Phase | Status |
|----|-------------|----------|-------|--------|
| REQ-018 | Employer dashboard overview: active postings, applicant pipeline, talent matches, analytics mockup (charts, placement stats) | critical | 4 | pending |
| REQ-019 | Talent search: searchable candidate database with AI-matched filters, candidate cards showing competency stack, microcredentials, artifact count, defense score | critical | 4 | pending |
| REQ-020 | Candidate profile view: full candidate profile with artifact gallery, process trace summary, oral defense transcripts, competency graph, microcredential verification | high | 4 | pending |
| REQ-021 | Posting management: create/edit/delete job postings, posting status tracking, applicant list per posting, interview pipeline mockup | high | 4 | pending |
| REQ-018 | Employer dashboard overview: active postings, applicant pipeline, talent matches, analytics mockup (charts, placement stats) | critical | 4 | complete |
| REQ-019 | Talent search: searchable candidate database with AI-matched filters, candidate cards showing competency stack, microcredentials, artifact count, defense score | critical | 4 | complete |
| REQ-020 | Candidate profile view: full candidate profile with artifact gallery, process trace summary, oral defense transcripts, competency graph, microcredential verification | high | 4 | complete |
| REQ-021 | Posting management: create/edit/delete job postings, posting status tracking, applicant list per posting, interview pipeline mockup | high | 4 | complete |
### Admin Surface
| ID | Description | Priority | Phase | Status |
|----|-------------|----------|-------|--------|
| REQ-022 | Admin overview: platform metrics (learners, employers, placements, completion rate, NPS), recent activity feed, system health mockup | critical | 5 | pending |
| REQ-023 | Learner management: searchable learner table, learner detail view, progress tracking, competency completion status, credential issuance log | high | 5 | pending |
| REQ-024 | Competency graph viewer: interactive visualization of competency stacks and their relationships, node/edge graph using react-flow, stack details on node click | high | 5 | pending |
| REQ-025 | Marketplace moderation: job posting review queue, employer verification queue, flagged content, content moderation tools mockup | high | 5 | pending |
| REQ-022 | Admin overview: platform metrics (learners, employers, placements, completion rate, NPS), recent activity feed, system health mockup | critical | 5 | complete |
| REQ-023 | Learner management: searchable learner table, learner detail view, progress tracking, competency completion status, credential issuance log | high | 5 | complete |
| REQ-024 | Competency graph viewer: interactive visualization of competency stacks and their relationships, node/edge graph using react-flow, stack details on node click | high | 5 | complete |
| REQ-025 | Marketplace moderation: job posting review queue, employer verification queue, flagged content, content moderation tools mockup | high | 5 | complete |
### Polish & Integration
| ID | Description | Priority | Phase | Status |
|----|-------------|----------|-------|--------|
| REQ-026 | Cross-surface navigation: role switcher (learner/employer/admin), breadcrumbs, consistent header/footer across all surfaces | critical | 6 | pending |
| REQ-027 | Visual consistency audit: typography scale, color palette, spacing system, dark mode toggle, accessibility baseline (WCAG AA contrast) | high | 6 | pending |
| REQ-028 | Component library finalization: Storybook setup, component documentation, prop tables, usage examples | medium | 6 | pending |
| REQ-026 | Cross-surface navigation: role switcher (learner/employer/admin), breadcrumbs, consistent header/footer across all surfaces | critical | 6 | complete |
| REQ-027 | Visual consistency audit: typography scale, color palette, spacing system, dark mode toggle, accessibility baseline (WCAG AA contrast) | high | 6 | complete |
| REQ-028 | Component library finalization: Storybook setup, component documentation, prop tables, usage examples | medium | 6 | complete |
---
@@ -170,7 +187,17 @@
## Traceability Matrix
### v0.3 (current milestone)
### v0.4 (current milestone)
| Requirement | Phase | Status |
|-------------|-------|--------|
| REQ-4-001 | 1 | complete |
| REQ-4-002 | 1 | complete |
| REQ-4-003 | 2 | complete |
| REQ-4-004 | 2 | complete |
| REQ-4-005 | 3 | complete |
### v0.3 (complete)
| Requirement | Phase | Status |
|-------------|-------|--------|
@@ -180,8 +207,8 @@
| REQ-3-004 | 3 | complete |
| REQ-3-005 | 4 | complete |
| REQ-3-006 | 5 | complete |
| REQ-3-007 | 6 | pending |
| REQ-3-008 | 6 | pending |
| REQ-3-007 | 6 | complete |
| REQ-3-008 | 6 | complete |
### v0.2 (complete)
+65 -107
View File
@@ -2,15 +2,17 @@
## Overview
**Milestone v0.3** — Credential Engines: Replace v0.2's mock engine inputs with real credential engines. Build the sandbox fabric (sandboxed IDE / design tool / simulation), the live in-environment build-telemetry pipeline, the process-trace grading engine, per-learner variant task generation, and the oral/voice defense with AI examiner. Lab/Assessor/Proctor agents move from mock inputs to real engine inputs.
**Milestone v0.4 — COMPLETE (shipped as v0.3.4, 2026-09-13).** Distribution & Bootstrap CLI: `nextcraft` CLI (doctor/bootstrap/verify/dev) shipped as a linux x64 SEA binary, one-liner install script with checksum + version integrity gates, and binaries published on **every ongoing release** (v0.3.2 onward). Next milestone: v0.5 (real server STT/TTS + KYC/identity + design/simulation sandbox environments + exec-telemetry seq-lease — the seams deferred out of v0.4 by founder directive D-016).
**Deferred per founder directive:** REQ-F-017 identity verification + age-gating (real KYC backend) is deferred beyond v0.3. Age-gating remains the v0.1 visual flow mockup.
**Milestone v0.3** — Credential Engines: complete, shipped as v0.2.8 (2026-09-12). Real sandbox fabric, live build telemetry, process-trace grading, per-learner variants, oral defense, real learner surfaces.
**Deferred per founder directive (D-016):** REQ-F-017 identity verification + age-gating (real KYC backend), real server STT/TTS, design/simulation sandbox environments, and the exec-telemetry seq-lease are deferred to v0.5. Age-gating remains the v0.1 visual flow mockup.
**Prior milestone:** v0.2 (ai-tutor-architecture) — complete, shipped as v0.2.0, six tutor agents live over mock engine inputs (D-015).
**Milestone type:** Feature (new credential-engine services + real agent inputs)
**Tag line:** v0.2.x (patches on the v0.2 line; milestone release as the final v0.2.x patch)
**Branch:** milestone/v0.3-credential-engines
**Milestone type:** Feature (new CLI + distribution pipeline)
**Tag line:** v0.3.x (patches on the v0.3 line; milestone release as the final v0.3.x patch)
**Branch:** milestone/v0.4-distribution
---
@@ -18,14 +20,11 @@
| # | Name | Status | Depends On | Requirements | Success Criteria |
|---|------|--------|------------|--------------|------------------|
| 0 | Pre-execution | in-progress | — | — | Specification, clarify, research, plan complete; .ciagent/ files updated for v0.3 |
| 1 | Sandbox fabric | complete | 0 | REQ-3-001, REQ-3-002 | Isolated per-learner sandbox environments provisioned (IDE / design / simulation); lifecycle API (create/destroy/snapshot); resource limits enforced; no cross-tenant access |
| 2 | Live build telemetry | complete | 1 | REQ-3-003 | In-environment capture of process events (commands, file diffs, run/test results, keystroke-level activity) streamed to ai-service; reliable transport; per-learner trace persistence |
| 3 | Process-trace grading engine | complete | 2 | REQ-3-004 | Grades artifacts from their full process traces (not just final output); emits structured rubric-aligned scores; feeds Assessor real inputs |
| 4 | Variant task generation | complete | 1 | REQ-3-005 | Per-learner task variants generated so no two learners receive identical prompts; variant seed recorded for grading fairness |
| 5 | Oral / voice defense | complete | 3 | REQ-3-006 | AI examiner conducts spoken defense of submitted work; STT → dialogue → TTS; transcript + integrity signals captured; feeds Proctor/Mentor |
| 6 | Agent re-grounding + learner surface integration | pending | 2,3,4,5 | REQ-3-007, REQ-3-008 | Lab/Assessor/Proctor consume real engine inputs; v0.1 sandbox + assessment mockups wired to real engines (in-browser build/run, live telemetry, live defense) |
| 7 | Final review + ship | pending | 6 | — | Code review clean; audit passes; milestone tagged (v0.2.x final patch); release created on Gitea |
| 0 | Pre-execution | complete | — | — | Specification, clarify, research, plan complete; .ciagent/ files updated for v0.4 |
| 1 | Bootstrap CLI core | complete | 0 | REQ-4-001, REQ-4-002 | `nextcraft doctor/bootstrap/verify/dev` work against a fresh clone; unit tests green |
| 2 | Binary build + release pipeline | complete | 1 | REQ-4-003, REQ-4-004 | Reproducible linux x64 binary + sha256 checksum; one-liner install script; assets uploaded to the Gitea release |
| 3 | Install docs + fresh-clone E2E | complete | 2 | REQ-4-005 | README quickstart verified end-to-end from a clean environment; fresh clone reaches running stack |
| 4 | Final review + ship | complete | 3 | — | Code review clean; audit passes; milestone tagged (v0.3.x final patch); release with binary assets created on Gitea |
---
@@ -33,148 +32,107 @@
### Phase 0: Pre-execution
**Goal:** Establish v0.3 specification, clarify ambiguities, research credential-engine architecture (sandbox isolation, telemetry transport, trace grading, variant generation, voice IO), create detailed plans.
**Goal:** Establish v0.4 specification (founder directive D-016), clarify ambiguities, research the binary toolchain + Gitea release-asset API + existing bootstrap scripts, create detailed plans, grill adversarially.
**Stages:** SPECIFY → CLARIFY → RESEARCH → PLAN → GRILL → MVP/UX CHECK → SHIP
**Deliverables:**
- Updated .ciagent/config.json, PROJECT.md, REQUIREMENTS.md, ROADMAP.md, ARCHITECTURE.md, PERSONAS.md, PLAN.md
**Success criteria:** All .ciagent/ files updated for v0.3; phase 0 shipped as first v0.2.x patch.
**Success criteria:** All .ciagent/ files updated for v0.4; phase 0 shipped as v0.3.0.
---
### Phase 1: Sandbox Fabric
### Phase 1: Bootstrap CLI Core
**Goal:** Provision and manage isolated per-learner execution environments.
**Goal:** A working `nextcraft` CLI with doctor/bootstrap/verify/dev commands, unit-tested against the real monorepo.
**Requirements:** REQ-3-001, REQ-3-002
**Requirements:** REQ-4-001, REQ-4-002
**Key deliverables:**
- Sandbox orchestrator service: create/list/destroy/snapshot sandbox instances (IDE, design tool, simulation)
- Isolation boundary: per-learner containerization or VM-grade isolation; no cross-tenant filesystem/network access
- Resource limits: CPU/memory/disk/time quotas per sandbox
- Sandbox lifecycle API consumed by ai-service and the web learner surface
- `apps/cli` package: `nextcraft` executable (source-runnable in dev, binary-built in P2)
- `doctor`: checks node ≥18, pnpm, python3 ≥3.11, git, unshare availability — actionable errors, exit codes
- `bootstrap`: idempotent — pnpm install, ai-service venv + pinned deps (reuses scripts/bootstrap.sh logic), .env from .env.example templates, key validation (warnings not blockers for optional keys), .env.secrets handling
- `verify`: health check — venv imports, pnpm build readiness, ports free, env vars present
- `dev`: thin passthrough to scripts/dev.sh (no orchestration logic duplicated)
- Unit tests: doctor/bootstrap parsing + command dispatch, against fixtures (never modifying the real repo state)
**Success criteria:**
- A sandbox can be created, written to, snapshotted, and destroyed via API
- Isolation verified: a sandbox cannot read another learner's data
- Resource limits enforced and observable — enforcement mechanism: rlimits (memory/CPU) + wall-clock reaper + workdir-size sweep; per-sandbox pids and hard-disk-quota are accepted v0.3 gaps (no cgroup delegation/sudo on this box, G-1)
- `nextcraft doctor` reports each prerequisite with actionable guidance
- `nextcraft bootstrap` on a fresh clone reaches a state where `verify` passes
- All commands have `--help`, exit non-zero on failure, no shell-out without timeout
- `pnpm build`, `pnpm typecheck`, `pnpm ai:test` green
---
### Phase 2: Live Build Telemetry
### Phase 2: Binary Build + Release Pipeline
**Goal:** Capture in-environment process events and stream them to ai-service reliably.
**Goal:** Reproducible linux x64 binary + one-liner install + release-asset upload wired into the ship flow.
**Requirements:** REQ-3-003
**Requirements:** REQ-4-003, REQ-4-004
**Key deliverables:**
- Telemetry capture agent (in-sandbox): commands, file diffs, run/test results, keystroke-level/activity events
- Telemetry transport: durable, ordered, resumable stream to ai-service ingestion endpoint
- Trace persistence: per-learner, per-task process traces stored for grading and proctoring
- Transport hardening: retries, backpressure, exactly-once-or-at-least-once semantics documented
- Build script producing `nextcraft-linux-x64` + `nextcraft-linux-x64.sha256` (toolchain probe-verified at RESEARCH; embedded script assets)
- One-liner install script `install.sh` served from the repo: detect linux x64, resolve latest release via Gitea API, download + verify checksum, install to `~/.local/bin`, PATH hint, source-bootstrap fallback when no binary/asset
- Ship integration: every release from v0.4 onward attaches the binary + checksum as release assets (the "ongoing binaries" requirement)
- Asset-upload helper using the Gitea token from `.env*` files only (never shell env)
**Success criteria:**
- Sandbox activity produces a complete ordered process trace in ai-service
- Stream survives transient network failure without trace loss
- Trace retrievable by learner+task ID for grading
- Binary runs on this box: `./nextcraft-linux-x64 doctor` green against the repo
- Install script verified end-to-end against the real Gitea release (or local dry-run if release pending)
- Checksum verification rejects a corrupted download (tested)
- Release assets present on the phase ship
---
### Phase 3: Process-Trace Grading Engine
### Phase 3: Install Docs + Fresh-Clone E2E
**Goal:** Grade learner artifacts from their full process traces.
**Goal:** Documentation and end-to-end proof that a fresh consumer reaches a running stack via the one-liner.
**Requirements:** REQ-3-004
**Requirements:** REQ-4-005
**Key deliverables:**
- Trace analyzer: reconstructs build/decision timeline from a process trace
- Grading engine: rubric-aligned scoring over the trace (process quality, not just final artifact)
- Structured score output consumable by the Assessor agent
- Calibration against v0.2 mock corpora to validate grading dimensions
- README quickstart: one-liner → `nextcraft doctor``nextcraft bootstrap``nextcraft dev`
- CLI command reference (all flags, exit codes)
- Fresh-clone E2E test: clean temp clone → doctor → bootstrap → verify → build green (sandboxed; no network beyond package registries already used)
- Install-script docs: prerequisites, offline/manual install, troubleshooting
**Success criteria:**
- Engine emits structured rubric-aligned scores from a real process trace
- Scores distinguish process quality (e.g., iterative debugging vs. paste-and-run)
- Output feeds Assessor; replaces pre-baked artifact corpus inputs
- A fresh clone bootstraps to a passing `verify` with one command sequence
- README quickstart matches the actual tested flow exactly
- E2E test green in CI-equivalent local run
---
### Phase 4: Variant Task Generation
### Phase 4: Final Review + Ship
**Goal:** Generate per-learner task variants so no two learners receive identical prompts.
**Requirements:** REQ-3-005
**Key deliverables:**
- Variant generator: parameterized task templates → unique per-learner instances
- Variant seed registry: record variant parameters for grading fairness and proctoring
- Difficulty normalization: variants calibrated to equivalent difficulty
**Success criteria:**
- Two learners requesting the same competency receive distinct task variants
- Variant parameters persisted and auditable
- Grading engine scores variants equitably
---
### Phase 5: Oral / Voice Defense
**Goal:** AI examiner conducts a spoken defense of the learner's submitted work.
**Requirements:** REQ-3-006
**Key deliverables:**
- Voice pipeline: STT → defense dialogue (LLM examiner) → TTS
- Examiner agent: probes understanding, challenges process choices from the trace
- Transcript + integrity signals captured for Proctor/Mentor
- Latency budget: defense feels conversational (bounded turn latency)
**Success criteria:**
- A spoken defense runs end-to-end (speak → examiner question → learner response → verdict)
- Transcript + integrity signals persisted and consumable by Proctor
- Turn latency within the documented budget
---
### Phase 6: Agent Re-grounding + Learner Surface Integration
**Goal:** Move Lab/Assessor/Proctor to real engine inputs; wire learner surface to the real engines.
**Requirements:** REQ-3-007, REQ-3-008
**Key deliverables:**
- Lab agent consumes live sandbox telemetry (replaces v0.2 mock telemetry)
- Assessor agent consumes grading-engine output (replaces pre-baked artifacts)
- Proctor consumes telemetry + defense integrity signals (replaces mock telemetry)
- Learner sandbox mockup → real in-browser build/run; assessment mockup → live defense + live grading
**Success criteria:**
- Lab/Assessor/Proctor operate on real inputs with no mock fallback in the learner path
- Learner can build in-browser and see live telemetry + live feedback
- Assessment surface runs a live defense and shows live grading
- `pnpm build` and `pnpm typecheck` pass
---
### Phase 7: Final Review + Ship
**Goal:** Code review, audit, milestone release.
**Goal:** Code review, audit, milestone release with binary assets.
**Key deliverables:**
- Multi-persona code review (correctness, testing, security, performance, maintainability)
- Project health audit (reconstruction test, .ciagent/ file discipline, branch hygiene, commit discipline)
- Milestone ship: merge milestone → main, tag final v0.2.x patch, create Gitea release
- Milestone ship: merge milestone → main, tag final v0.3.x patch, create Gitea release WITH binary + checksum assets, verify assets downloadable
**Success criteria:**
- Code review: P0 fixes applied, P1+ documented
- Audit: all checks pass, project state reconstructable from git log
- Ship: milestone tagged, branch merged to main, Gitea release created — release note states identity/age-gating (KYC) is deferred and age-gating remains a visual mockup
- All v0.3 requirements marked complete
- Ship: milestone tagged, branch merged to main, Gitea release created with `nextcraft-linux-x64` + `.sha256` assets attached — the first of the ongoing binary releases
---
## v0.4 (Complete — Shipped as v0.3.4)
Distribution & Bootstrap CLI: `nextcraft` CLI (doctor/bootstrap/verify/dev) as a self-contained linux x64 SEA binary, one-liner install with sha256 + version integrity gates, release-asset pipeline attaching binaries to every ongoing release (v0.3.2 onward), install/quickstart docs backed by a fresh-clone E2E test. 5 phases (P0P4). All 5 requirements (REQ-4-001..005) complete. Tags v0.3.0v0.3.3 per phase, milestone release v0.3.4.
## v0.3 (Complete — Shipped as v0.2.8)
Credential Engines: real sandbox fabric (Linux namespaces), live build telemetry
(at-least-once/exactly-once), process-trace grading (G-4 gated), seeded per-learner
variants (fairness anchors wired to grading), oral defense with integrity signals
(mock-first voice, browser fallback), and real learner build/defense/grading surfaces.
8 phases. All 8 requirements (REQ-3-001..008) complete. Tags v0.2.1v0.2.7 per phase,
milestone release v0.2.8.
## v0.2 (Complete — Shipped as v0.2.0)
AI Tutor Architecture: Six AI tutor agents (Coach, Tutor, Lab, Assessor, Proctor, Mentor) as real LLM-backed services over mock engine inputs, wired into the learner surface with streaming. 7 phases. All 12 requirements complete. Milestone release v0.2.0.
+3 -3
View File
@@ -46,9 +46,9 @@
"projects": [],
"active_project": null,
"milestone": {
"version": "v0.3",
"name": "credential-engines",
"version": "v0.4",
"name": "distribution",
"type": "feature",
"branch": "milestone/v0.3-credential-engines"
"branch": "milestone/v0.4-distribution"
}
}
+64 -1
View File
@@ -2,8 +2,71 @@
AI-native outcome school + marketplace — graduates prove what they can build, not what they can write.
## Quickstart
One-liner install (linux x64):
```sh
curl -fsSL https://git.coreci.dev/coreci/nextcraft/raw/main/scripts/install.sh | sh
```
That downloads the latest release's `nextcraft` CLI binary, verifies its sha256 checksum, and installs it to `~/.local/bin` (PATH hint printed if needed). Every release ships fresh binaries — re-run the one-liner to upgrade.
Then, from a clone of this repo:
```sh
nextcraft doctor # check prerequisites: node >= 18, pnpm >= 8, python3 >= 3.11, git, unshare
nextcraft bootstrap # pnpm install + ai-service venv + .env from template (idempotent)
nextcraft verify # health check: venv imports, uvicorn, ports, env
nextcraft dev # run the ai-service dev server on :8420 (web dev server: pnpm dev)
```
The E2E test (`apps/cli/tests/fresh-clone-e2e.test.ts`) proves this exact sequence on a fresh clone.
### No binary / non-linux?
The installer degrades to printed source instructions. Manual equivalent:
```sh
git clone https://git.coreci.dev/coreci/nextcraft.git && cd nextcraft
pnpm install
bash apps/ai-service/scripts/bootstrap.sh
cp apps/ai-service/.env.example apps/ai-service/.env
pnpm ai:dev
```
## CLI reference (`nextcraft`)
| Command | What it does | Exit codes |
|---------|--------------|------------|
| `doctor` | Checks prerequisites on PATH: node >= 18, pnpm >= 8, python3 >= 3.11, git, unshare (sandbox fabric). Every ✗ prints a fix hint. | 0 all pass, 1 any fail |
| `bootstrap` | Sets up the monorepo from a fresh clone: (1) locates the repo root, (2) `pnpm install`, (3) ai-service venv via `apps/ai-service/scripts/bootstrap.sh`, (4) copies `.env.example``.env` if absent, (5) warns on missing optional keys. Idempotent — safe to re-run. | 0 ok, 1 step failed |
| `verify` | Health check: ai-service venv + `import ai_service`, uvicorn importable, `.env` present (warn-only), `AI_PORT` (default 8420) free, workspace `node_modules` present. | 0 ok, 1 failures |
| `dev` | Thin passthrough to `apps/ai-service/scripts/dev.sh` (exports secrets from `.ciagent/.env.secrets` if present, runs uvicorn on :8420). Ctrl+C stops it. The web dev server is separate: `pnpm dev`. | child's exit code |
| `--help` / `-h` | Usage for the CLI or any command. | 0 |
| `--version` | Prints the version this binary was built as (matches the release tag). | 0 |
Exit-code contract: `0` success, `1` check/step failure (hint printed), `2` usage error.
### Remote server
`nextcraft dev` binds the API on **0.0.0.0:8420** (and `pnpm dev` serves the web app on all interfaces), so the stack works from other machines out of the box:
- Browse `http://<your-host>:3000` — the web app targets `http://<your-host>:8420` automatically (derived from the browser's hostname).
- CORS admits any origin (`AI_CORS_ORIGINS=*` in `apps/ai-service/.env`). This is safe **only** because credentials are never enabled; to restrict, set an explicit list: `AI_CORS_ORIGINS=http://<your-host>:3000`.
- To revert to loopback-only: `AI_HOST=127.0.0.1` in `apps/ai-service/.env`.
- Security note: this is an unauthenticated dev API reachable from any network the box exposes. Mitigations that still apply: per-learner sandbox caps + global rate caps + learner allowlist (G-5), telemetry flood control (traces marked `INCOMPLETE_FLOODED` are refused by the grader). Expose only on trusted networks until identity/KYC lands (v0.5).
## Docs
- [apps/cli/README.md](apps/cli/README.md) — CLI internals: build, binary pipeline, troubleshooting
- [.ciagent/PROJECT.md](.ciagent/PROJECT.md) — product spec and milestone history
- [.ciagent/ARCHITECTURE.md](.ciagent/ARCHITECTURE.md) — system architecture
## Status
**Milestone v0.1** — UI/UX Prototype (high-fidelity interactive, all mock data)
**Milestone v0.4** — Distribution & Bootstrap CLI (one-liner install, `nextcraft` binary releases on every ship)
Prior: v0.3 Credential Engines (shipped v0.2.8) · v0.2 AI Tutor Architecture (v0.2.0) · v0.1 UI/UX Prototype (v0.1.0)
Initialized via CIAgent v0.7.0
+7
View File
@@ -2,6 +2,13 @@
# Real keys live in .ciagent/.env.secrets (gitignored) and are exported by scripts/dev.sh
AI_PORT=8420
# Network mode (v0.3.5, D-038): dev server binds 0.0.0.0 so remote machines can
# reach the stack. Set to 127.0.0.1 to revert to loopback-only.
AI_HOST=0.0.0.0
# CORS + WS-origin policy: '*' (default) admits any origin — safe because
# credentials are never enabled. Restrict with a comma list, e.g.:
# AI_CORS_ORIGINS=http://nextcraft-1:3000
AI_CORS_ORIGINS=*
AI_PROVIDER=ollama-cloud
AI_MODEL=gemma4:31b
AI_OLLAMA_CLOUD_BASE_URL=https://ollama.com/v1
+18
View File
@@ -235,6 +235,24 @@ in CI, so v0.3 ships the protocol seam instead of an unverifiable claim.
criterion, run manually with `AI_VOICE_PROVIDER` set to the real
provider and keys in `.ciagent/.env.secrets` (never in code/commits).
## End-to-end credential flow (v0.3, REQ-3-007/008)
`tests/api/test_e2e_credential_flow.py` runs the full pipeline against a REAL
uvicorn server with REAL namespace sandboxes (mock LLM/voice per G-2
precedent): variant -> telemetry-wired sandbox -> in-sandbox exec -> trace
persistence -> process-trace grade (variant seed stamped) -> assessor
coaching -> oral defense -> verdict + integrity signals -> proctor. It
asserts no corpus fixture appears anywhere in the learner path.
Manual browser pass (documented, not automated): `pnpm ai:dev` + `pnpm dev`,
then open `/build/stack-orchestration-c007` — variant statement + starter
files load, edit a file, Run/Test execute in the sandbox with output in the
read-only panel, the telemetry status pulses, Lab streams feedback from the
live digest; then `/defend/stack-orchestration-c007` — Start Defense, typed
answers (mic path needs permission), Finish, Grade My Work renders the real
rubric bars. Navigating away destroys the sandbox
(`curl localhost:8420/v1/sandboxes` shows the count drop).
## Layout
```
+36 -65
View File
@@ -1,53 +1,34 @@
"""AssessorAgent — rubric application to pre-baked artifacts (REQ-2-008).
"""AssessorAgent — rubric coaching over REAL grading output (REQ-3-007).
Structured-output showcase: applies the 4-layer defense (D-020) to return
a pydantic-validated rubric score. Mock engine inputs (corpus artifacts +
transcripts); real process-trace grading is v0.3+.
v0.3 re-grounding: the Assessor no longer invents scores from corpus
artifacts the process-trace grading engine (Phase 3) computes and
persists the validated RubricScore. This agent now renders the STORED
grade as rubric-anchored coaching: explains the criteria, cites strengths
and gaps, and frames next steps. Corpus artifacts are retired from this
path (corpus dormancy, Task 6-1-04).
"""
from pydantic import BaseModel, Field
from ..corpus.artifacts import (
ArtifactSubmission,
AssessmentRubric,
DefenseTranscript,
render_rubric,
render_transcript,
)
from ..corpus.learner_context import LearnerContext, get_learner_context
from ..grading.store import GradeRecord
from ..prompts.assessor import SYSTEM_PROMPT, render_context
from .base import BaseAgent
class CriterionScore(BaseModel):
criterion_id: str
name: str
score: int = Field(ge=0, le=100)
evidence: str
class GradeCoaching(BaseModel):
"""Rubric-anchored coaching rendered FROM the stored grade (not invented)."""
summary: str = Field(min_length=1)
strengths: list[str] = Field(min_length=1, max_length=3)
gaps: list[str] = Field(min_length=1, max_length=3)
next_steps: list[str] = Field(min_length=1, max_length=3)
class RubricScore(BaseModel):
rubric_id: str
artifact_id: str
competency_id: str
scores: list[CriterionScore]
strengths: list[str] = Field(min_length=1, max_length=2)
gaps: list[str] = Field(min_length=1, max_length=2)
verdict: str # "mastered" | "developing" | "not_yet"
def weighted_total(self, rubric: AssessmentRubric) -> float:
by_id = {c.criterion_id: c for c in rubric.criteria}
total = 0.0
for s in self.scores:
total += s.score * by_id[s.criterion_id].weight
return total
RUBRIC_SCORE_SCHEMA_HINT = (
'{"rubric_id": "<id>", "artifact_id": "<id>", "competency_id": "<id>", '
'"scores": [{"criterion_id": "<id>", "name": "<name>", "score": <0-100>, '
'"evidence": "<one sentence>"}], "strengths": ["<one sentence>"], '
'"gaps": ["<one sentence>"], "verdict": "mastered"|"developing"|"not_yet"}'
GRADE_COACHING_SCHEMA_HINT = (
'{"summary": "<two sentences on the grade>", '
'"strengths": ["<one sentence>"], "gaps": ["<one sentence>"], '
'"next_steps": ["<one sentence>"]}'
)
@@ -58,35 +39,25 @@ class AssessorAgent(BaseAgent):
ctx = learner_context or get_learner_context()
return SYSTEM_PROMPT.format_map(render_context(ctx))
def build_evaluation_input(
async def coach_grade(
self,
artifact: ArtifactSubmission,
rubric: AssessmentRubric,
transcript: DefenseTranscript | None,
) -> str:
parts = [
f"ARTIFACT: {artifact.name} ({artifact.artifact_id})",
f"Evidence excerpt: {artifact.evidence_excerpt}",
"",
render_rubric(rubric),
]
if transcript is not None:
parts += ["", render_transcript(transcript)]
return "\n".join(parts)
async def evaluate(
self,
artifact: ArtifactSubmission,
rubric: AssessmentRubric,
transcript: DefenseTranscript | None,
grade: GradeRecord,
learner_context: LearnerContext | None = None,
) -> RubricScore:
evaluation_input = self.build_evaluation_input(artifact, rubric, transcript)
score: RubricScore = await self.structured_reply(
) -> GradeCoaching:
"""Render the STORED grade as coaching via the D-020 defense."""
grade_json = {
"verdict": grade.verdict,
"scores": grade.scores,
"digest": grade.digest,
}
coaching: GradeCoaching = await self.structured_reply(
history=None,
user_input=evaluation_input,
user_input=(
"The learner's process-trace grade (computed by the grading "
f"engine) is:\n{grade_json!r}\nExplain it as coaching."
),
learner_context=learner_context,
schema=RubricScore,
schema_hint=RUBRIC_SCORE_SCHEMA_HINT,
schema=GradeCoaching,
schema_hint=GRADE_COACHING_SCHEMA_HINT,
)
return score
return coaching
+10 -8
View File
@@ -1,17 +1,18 @@
"""LabAgent — in-flow feedback over simulated sandbox telemetry (REQ-2-007).
"""LabAgent — in-flow feedback over LIVE sandbox telemetry (REQ-3-007).
Scenario-driven: consumes a LabTelemetryScenario from the corpus, renders
the event timeline into the conversation, streams concrete feedback.
No session chat each request is one scenario read.
v0.3 re-grounding: consumes a TraceDigest computed from the learner's real
trace (grading/features.compute_digest over TraceStore events) the v0.2
corpus scenarios are retired from this path (corpus dormancy, Task 6-1-04).
No session chat each request is one live-trace read.
"""
from collections.abc import AsyncIterator
from ..config import Settings
from ..corpus.learner_context import LearnerContext, get_learner_context
from ..corpus.telemetry import LabTelemetryScenario, summarize_scenario
from ..grading.features import TraceDigest
from ..llm.base import LLMProvider
from ..prompts.lab import SYSTEM_PROMPT, render_context
from ..prompts.lab import SYSTEM_PROMPT, render_context, render_digest_timeline
from .base import BaseAgent
@@ -27,10 +28,11 @@ class LabAgent(BaseAgent):
async def stream_feedback(
self,
scenario: LabTelemetryScenario,
digest: TraceDigest | None,
learner_context: LearnerContext | None = None,
) -> AsyncIterator[str]:
timeline = summarize_scenario(scenario)
"""Feedback grounded in the learner's live trace digest."""
timeline = render_digest_timeline(digest)
async for token in self.stream_reply(
history=None, user_input=timeline, learner_context=learner_context
):
+33 -11
View File
@@ -1,33 +1,41 @@
"""ProctorAgent — integrity signals + coaching interventions (REQ-2-009).
"""ProctorAgent — integrity signals + coaching over REAL inputs (REQ-3-007).
Consumes a ProctorScenario from the corpus, returns pydantic-validated
signal classifications via structured_reply (4-layer defense).
Mock engine inputs; real identity/attention signals are v0.3+.
v0.3 re-grounding: consumes the learner's live trace digest (idle gaps,
command cadence), the DefenseStore integrity signals (long pauses from the
oral defense), and the variant seed cross-check NOT v0.2 corpus
scenarios. The proctor COACHES: it classifies signals supportively and
recommends one intervention; it never punishes and never accuses.
Integrity inputs (computed server-side, passed in by the API layer):
- trace digest: idle_gap_count/total, command_categories histogram,
error/fix cycles, huge-burst indicators (edit_count vs test runs)
- defense signals: long_pauses list from the finished defense (A-109)
- variant: seed + params when the task is variant-derived (off-template
work is a cross-check input, not an accusation)
"""
from pydantic import BaseModel, Field
from ..corpus.learner_context import LearnerContext, get_learner_context
from ..corpus.telemetry import ProctorScenario, summarize_proctor_scenario
from ..grading.features import TraceDigest
from ..prompts.proctor import SYSTEM_PROMPT, render_context
from .base import BaseAgent
class IntegritySignal(BaseModel):
signal_type: str # e.g. "context_switch" | "idle_gap" | "large_paste"
signal_type: str # "idle_gap" | "long_pause" | "burst_edit" | "off_template"
severity: str # "low" | "medium" | "high"
note: str
class ProctorAssessment(BaseModel):
scenario_id: str
signals: list[IntegritySignal] = Field(min_length=0)
intervention: str # ONE supportive coaching recommendation
summary: str
PROCTOR_ASSESSMENT_SCHEMA_HINT = (
'{"scenario_id": "<id>", "signals": [{"signal_type": "<type>", '
'{"signals": [{"signal_type": "<type>", '
'"severity": "low"|"medium"|"high", "note": "<one sentence>"}], '
'"intervention": "<one supportive recommendation>", '
'"summary": "<one sentence>"}'
@@ -43,13 +51,27 @@ class ProctorAgent(BaseAgent):
async def assess(
self,
scenario: ProctorScenario,
digest: TraceDigest | None,
defense_signals: dict | None = None,
variant_context: dict | None = None,
learner_context: LearnerContext | None = None,
) -> ProctorAssessment:
timeline = summarize_proctor_scenario(scenario)
"""Classify REAL integrity inputs into supportive signals + coaching."""
parts: list[str] = []
if digest is not None:
parts.append(f"Build-session digest:\n{digest.model_dump_json()}")
else:
parts.append("No build telemetry recorded for this task yet.")
if defense_signals:
parts.append(f"Oral-defense integrity signals:\n{defense_signals}")
if variant_context:
parts.append(f"Variant audit context (seed + params):\n{variant_context}")
assessment: ProctorAssessment = await self.structured_reply(
history=None,
user_input=timeline,
user_input=(
"Assess this learner's integrity signals supportively.\n\n"
+ "\n\n".join(parts)
),
learner_context=learner_context,
schema=ProctorAssessment,
schema_hint=PROCTOR_ASSESSMENT_SCHEMA_HINT,
+25 -15
View File
@@ -59,11 +59,9 @@ from typing import Any
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from ..agents.assessor import RubricScore
from ..agents.registry import AgentRegistry
from ..agents.structured import StructuredOutputError
from ..config import Settings
from ..corpus.artifacts import get_artifact_bundle, get_transcript_for_artifact
from ..corpus.learner_context import get_learner_context
from ..grading.engine import GradingEngine
from ..grading.store import GradeRecord, GradeStore
@@ -81,37 +79,49 @@ router = APIRouter(prefix="/v1")
# --- v0.2 artifact evaluation (REQ-2-008) --------------------------------------
class AssessmentRequest(BaseModel):
artifact_id: str = Field(min_length=1)
learner_id: str | None = None
class EvaluateRequest(BaseModel):
learner_id: str = Field(min_length=1)
task_id: str = Field(min_length=1)
@router.post("/assessment/evaluate", response_model=RubricScore)
@router.post("/assessment/evaluate")
async def assessment_evaluate(
body: AssessmentRequest,
body: EvaluateRequest,
registry: AgentRegistry = Depends(get_agent_registry),
settings: Settings = Depends(get_settings),
provider=Depends(get_provider),
) -> RubricScore:
bundle = get_artifact_bundle(body.artifact_id)
if bundle is None:
grade_store=Depends(get_grade_store),
) -> dict:
"""Assessor coaching rendered FROM the learner's stored grade (REQ-3-007).
The grading engine computes the scores (POST /assessment/grade); this
endpoint explains them. No stored grade yet -> 404 (grade first).
"""
grade = grade_store.get(body.learner_id, body.task_id)
if grade is None:
raise HTTPException(
status_code=404, detail=f"unknown artifact {body.artifact_id!r}"
status_code=404,
detail=f"no stored grade for {body.learner_id}/{body.task_id} - grade first",
)
artifact, rubric = bundle
transcript = get_transcript_for_artifact(artifact.artifact_id)
agent = registry.get(provider, settings, "assessor")
learner_context = get_learner_context(body.learner_id)
try:
return await agent.evaluate(artifact, rubric, transcript, learner_context)
coaching = await agent.coach_grade(grade, learner_context)
except Exception as exc:
raise HTTPException(
status_code=502,
detail=f"assessment evaluation failed: {exc}",
) from exc
return {
"learner_id": grade.learner_id,
"task_id": grade.task_id,
"grade_verdict": grade.verdict,
"grade_scores": grade.scores,
"coaching": coaching.model_dump(),
}
# --- v0.3 trace grading (REQ-3-004) ---------------------------------------------
# --- # --- v0.3 trace grading (REQ-3-004) ---------------------------------------------
class GradeRequest(BaseModel):
@@ -21,7 +21,6 @@ from __future__ import annotations
import time
from datetime import UTC, datetime
from typing import TYPE_CHECKING
from fastapi import APIRouter, Depends, File, Form, HTTPException, UploadFile
from fastapi.responses import StreamingResponse
@@ -42,9 +41,6 @@ from .deps import (
get_voice_store,
)
if TYPE_CHECKING: # pragma: no cover
pass
router = APIRouter(prefix="/v1/defense", tags=["defense"])
#: A-109: learner turns slower than this are flagged as long pauses (ms).
+25 -14
View File
@@ -1,27 +1,36 @@
"""POST /v1/lab/feedback — SSE stream of Lab in-flow feedback (REQ-2-007).
"""POST /v1/lab/feedback — SSE stream of Lab in-flow feedback (REQ-3-007).
D-016 envelope with agent=lab. Unknown scenario 404 before streaming.
v0.3 re-grounding: LIVE trace digest. Request carries {learner_id, task_id};
the digest is computed from the learner's real TraceStore events (D-028)
and handed to the Lab agent. No corpus scenarios. Empty/unknown trace is NOT
an error Lab gets a "no telemetry yet" timeline and coaches the baseline.
D-016 envelope with agent=lab.
"""
import json
from collections.abc import AsyncIterator
from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends
from pydantic import BaseModel, Field
from sse_starlette.sse import EventSourceResponse
from ..agents.registry import AgentRegistry
from ..config import Settings
from ..corpus.learner_context import get_learner_context
from ..corpus.telemetry import get_lab_scenario
from .deps import get_agent_registry, get_provider, get_settings
from ..grading.features import compute_digest
from .deps import (
get_agent_registry,
get_provider,
get_settings,
get_trace_store,
)
router = APIRouter(prefix="/v1")
class LabFeedbackRequest(BaseModel):
scenario_id: str = Field(min_length=1)
learner_id: str | None = None
learner_id: str = Field(min_length=1)
task_id: str = Field(min_length=1)
@router.post("/lab/feedback")
@@ -30,25 +39,27 @@ async def lab_feedback(
registry: AgentRegistry = Depends(get_agent_registry),
settings: Settings = Depends(get_settings),
provider=Depends(get_provider),
trace_store=Depends(get_trace_store),
) -> EventSourceResponse:
scenario = get_lab_scenario(body.scenario_id)
if scenario is None:
raise HTTPException(
status_code=404, detail=f"unknown scenario {body.scenario_id!r}"
)
agent = registry.get(provider, settings, "lab")
learner_context = get_learner_context(body.learner_id)
trace = (
trace_store.get_trace(body.learner_id, body.task_id)
if body.task_id in trace_store.list_tasks(body.learner_id)
else []
)
digest = compute_digest(trace) if trace else None
async def event_stream() -> AsyncIterator[dict]:
yield {"event": "message", "data": json.dumps({
"type": "meta",
"agent": "lab",
"scenario_id": body.scenario_id,
"task_id": body.task_id,
"model": settings.model,
})}
first_byte = True
try:
async for token in agent.stream_feedback(scenario, learner_context):
async for token in agent.stream_feedback(digest, learner_context):
first_byte = False
yield {"event": "message", "data": json.dumps({
"type": "delta", "content": token
+45 -13
View File
@@ -1,7 +1,8 @@
"""POST /v1/proctor/signals — structured integrity signals (REQ-2-009).
"""POST /v1/proctor/signals — integrity signals over REAL inputs (REQ-3-007).
JSON response (not SSE): a pydantic-validated ProctorAssessment.
Unknown scenario 404. Coaching-shaped interventions only.
v0.3 re-grounding: live trace digest + DefenseStore long-pause signals +
variant seed cross-check, no corpus scenarios. The proctor coaches:
a pydantic-validated ProctorAssessment (JSON response, not SSE).
"""
from fastapi import APIRouter, Depends, HTTPException
@@ -11,15 +12,22 @@ from ..agents.proctor import ProctorAssessment
from ..agents.registry import AgentRegistry
from ..config import Settings
from ..corpus.learner_context import get_learner_context
from ..corpus.telemetry import get_proctor_scenario
from .deps import get_agent_registry, get_provider, get_settings
from ..grading.features import compute_digest
from .deps import (
get_agent_registry,
get_provider,
get_settings,
get_trace_store,
get_variant_store,
get_voice_store,
)
router = APIRouter(prefix="/v1")
class ProctorRequest(BaseModel):
scenario_id: str = Field(min_length=1)
learner_id: str | None = None
learner_id: str = Field(min_length=1)
task_id: str = Field(min_length=1)
@router.post("/proctor/signals", response_model=ProctorAssessment)
@@ -28,16 +36,40 @@ async def proctor_signals(
registry: AgentRegistry = Depends(get_agent_registry),
settings: Settings = Depends(get_settings),
provider=Depends(get_provider),
trace_store=Depends(get_trace_store),
variant_store=Depends(get_variant_store),
voice_store=Depends(get_voice_store),
) -> ProctorAssessment:
scenario = get_proctor_scenario(body.scenario_id)
if scenario is None:
raise HTTPException(
status_code=404, detail=f"unknown scenario {body.scenario_id!r}"
)
"""Real integrity inputs: live digest + defense signals + variant context."""
agent = registry.get(provider, settings, "proctor")
learner_context = get_learner_context(body.learner_id)
trace = (
trace_store.get_trace(body.learner_id, body.task_id)
if body.task_id in trace_store.list_tasks(body.learner_id)
else []
)
digest = compute_digest(trace) if trace else None
defense_signals = None
for record in voice_store.list_for_learner(body.learner_id):
if record.task_id == body.task_id and record.status == "finished":
defense_signals = record.integrity_signals or None
break
variant = variant_store.get_by_task(body.task_id)
variant_context = (
{"template_id": variant.template_id, "seed": variant.seed, "params": variant.params}
if variant is not None
else None
)
try:
return await agent.assess(scenario, learner_context)
return await agent.assess(
digest,
defense_signals=defense_signals,
variant_context=variant_context,
learner_context=learner_context,
)
except Exception as exc:
raise HTTPException(
status_code=502, detail=f"proctor assessment failed: {exc}"
+135 -1
View File
@@ -23,6 +23,7 @@ never part of the API contract.
import time
from collections import deque
from pathlib import Path
from fastapi import APIRouter, Depends, HTTPException, Response
from pydantic import BaseModel, ConfigDict, Field
@@ -50,6 +51,9 @@ _CREATE_TIMES: deque[float] = deque()
class SandboxCreateRequest(BaseModel):
learner_id: str = Field(min_length=1)
# Optional task key: when set, the sandbox is telemetry-wired (REQ-3-003)
# — the in-sandbox capture agent streams workspace events to the ingest.
task_id: str | None = None
class SandboxResponse(BaseModel):
@@ -142,7 +146,7 @@ async def create_sandbox(
_check_per_learner_cap(await manager.list(), body.learner_id, settings)
_check_global_create_rate(settings)
try:
info = await manager.create(body.learner_id)
info = await manager.create(body.learner_id, task_id=body.task_id)
except PoolFullError as exc:
raise HTTPException(status_code=503, detail=str(exc)) from exc
return _to_response(info)
@@ -217,3 +221,133 @@ async def delete_sandbox(
if test_layout is not None:
result.headers["X-Workspace-Copy"] = str(test_layout.workspace)
return result
# -- workspace files + exec (Phase 6, REQ-3-008; CUT-2) -------------------------
#
# The build surface reads/writes/list workspace files and runs Run/Test
# commands through the manager's backend. NO interactive shell relay (CUT-2:
# keystroke-level stdin/stdout is v0.4) — each exec is a bounded command with
# captured output. Paths are WORKSPACE-RELATIVE; traversal outside the
# workspace is rejected (the workdir bind is the boundary, but the API adds
# its own containment check — defense in depth).
class FileWriteRequest(BaseModel):
path: str = Field(min_length=1)
content: str
class ExecRequest(BaseModel):
cmd: list[str] = Field(min_length=1)
class ExecResponse(BaseModel):
cmd: list[str]
returncode: int
stdout: str
stderr: str
duration_s: float
async def _workspace_dir(manager: SandboxManager, sandbox_id: str):
"""Resolve the sandbox workspace (tracked layout or shell layout)."""
info = await manager.get(sandbox_id) # raises SandboxNotFoundError -> 404
backend = manager._backend # noqa: SLF001 - API owns the composition seam
tracked = getattr(backend, "_tracked", {}).get(sandbox_id)
if tracked is not None:
return tracked.workspace, info
return info.workdir / "workspace", info
def _safe_rel_path(raw: str) -> Path:
"""Workspace-relative path; reject absolute/traversal paths."""
candidate = Path(raw)
if candidate.is_absolute() or ".." in candidate.parts:
raise HTTPException(status_code=422, detail=f"invalid workspace path {raw!r}")
return candidate
def _resolve_in_workspace(workspace: Path, rel: Path) -> Path:
"""Resolve `rel` under `workspace`, refusing symlink escapes (P7).
The lexical check in `_safe_rel_path` cannot see symlinks: an exec can
plant `ln -s /etc target` in the workspace and a follow-up read/write
would follow it OUT of the bind. Resolve with the workspace as the
anchor (strict: a symlink chain escaping raises) and confirm the
normalized target still sits inside the workspace defense in depth
for both read_file and write_file.
"""
try:
target = (workspace / rel).resolve(strict=False)
target.relative_to(workspace.resolve(strict=False))
except ValueError:
raise HTTPException(
status_code=422, detail=f"path escapes the workspace: {rel.as_posix()!r}"
) from None
return target
@router.get("/{sandbox_id}/files")
async def list_files(
sandbox_id: str,
manager: SandboxManager = Depends(get_sandbox_manager),
) -> dict:
try:
workspace, _ = await _workspace_dir(manager, sandbox_id)
except SandboxNotFoundError:
raise HTTPException(status_code=404, detail=f"no sandbox {sandbox_id!r}") from None
return {"files": sorted(p.name for p in workspace.iterdir()) if workspace.is_dir() else []}
@router.get("/{sandbox_id}/files/{path:path}")
async def read_file(
sandbox_id: str,
path: str,
manager: SandboxManager = Depends(get_sandbox_manager),
) -> dict:
try:
workspace, _ = await _workspace_dir(manager, sandbox_id)
except SandboxNotFoundError:
raise HTTPException(status_code=404, detail=f"no sandbox {sandbox_id!r}") from None
rel = _safe_rel_path(path)
target = _resolve_in_workspace(workspace, rel)
if not target.is_file():
raise HTTPException(status_code=404, detail=f"no file {path!r}")
return {"path": path, "content": target.read_text(errors="replace")}
@router.put("/{sandbox_id}/files/{path:path}")
async def write_file(
sandbox_id: str,
path: str,
body: FileWriteRequest,
manager: SandboxManager = Depends(get_sandbox_manager),
) -> dict:
try:
workspace, _ = await _workspace_dir(manager, sandbox_id)
except SandboxNotFoundError:
raise HTTPException(status_code=404, detail=f"no sandbox {sandbox_id!r}") from None
rel = _safe_rel_path(body.path)
target = _resolve_in_workspace(workspace, rel)
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(body.content)
return {"path": body.path, "written": True}
@router.post("/{sandbox_id}/exec", response_model=ExecResponse)
async def exec_command(
sandbox_id: str,
body: ExecRequest,
manager: SandboxManager = Depends(get_sandbox_manager),
) -> ExecResponse:
try:
await manager.get(sandbox_id)
except SandboxNotFoundError:
raise HTTPException(status_code=404, detail=f"no sandbox {sandbox_id!r}") from None
backend = manager._backend # noqa: SLF001 - API owns the composition seam
handle = manager._handles.get(sandbox_id) # noqa: SLF001
if handle is None:
raise HTTPException(status_code=404, detail=f"no live handle {sandbox_id!r}")
result = await backend.exec(handle, body.cmd)
return ExecResponse(**result.model_dump())
+41 -4
View File
@@ -9,10 +9,13 @@ only wires `app.state.trace_store` / `app.state.trace_integrity` /
GET /v1/telemetry/traces/{learner_id}/{task_id} ordered trace; 404 unknown
GET /v1/telemetry/gaps/{learner_id}/{task_id} missing seqs ; 404 unknown
The WS route is a thin DI shell: it validates the query-param identity,
pulls store/integrity/settings from `app.state`, and calls
`telemetry_ingest_endpoint(...)` the engine stays FastAPI-DI-free so it's
testable without a router and the api/ layer owns all composition.
The WS route is a thin DI shell: it validates the query-param identity and
the Origin (browser pages are gated to the localhost dev origins CORS
middleware does not cover WS upgrades; the stdlib capture agent sends no
Origin and is unaffected), pulls store/integrity/settings from `app.state`,
and calls `telemetry_ingest_endpoint(...)` the engine stays
FastAPI-DI-free so it's testable without a router and the api/ layer owns
all composition.
Unknown-trace contract: a trace is KNOWN when it has >=1 stored event OR
carries an integrity flag a flooded trace with zero stored rows still 200s
@@ -21,6 +24,8 @@ so Proctor/grader can read WHY it's unusable (G-4 consumes
the map so HTTP consumers never touch process internals.
"""
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException, WebSocket
from pydantic import BaseModel
@@ -34,6 +39,27 @@ from .deps import get_trace_integrity, get_trace_store
router = APIRouter(prefix="/v1/telemetry", tags=["telemetry"])
#: Browser Origins allowed to open the ingest socket (A-008 mirror, D-038).
#: The stdlib capture agent sends NO Origin header (it is not a browser) and
#: stays allowed; a malicious page loaded in the learner's browser would
#: carry an Origin and must not be able to poison/flood the trace. CORS
#: middleware does NOT cover WebSocket upgrades, so this gate is explicit.
#: In network mode the configured CORS list governs (default '*' — any
#: origin, since credentials are never used); an explicit list still rejects
#: unlisted origins with 1008.
_LOCAL_WS_ORIGINS = frozenset(
{"http://localhost:3000", "http://127.0.0.1:3000", "http://localhost:8420"}
)
def _allowed_ws_origins(settings: object) -> frozenset[str]:
configured = getattr(settings, "cors_origin_list", None)
if configured is None:
return _LOCAL_WS_ORIGINS
if configured == ["*"]:
return frozenset() # empty = wildcard = every Origin passes
return frozenset(configured) | _LOCAL_WS_ORIGINS
# --- WS ingest (D-026) ---------------------------------------------------------
@@ -45,6 +71,17 @@ async def telemetry_ingest_ws(websocket: WebSocket) -> None:
The engine's session + flood logic is fully typed and testable without
FastAPI; this shim is the only place the two layers meet.
"""
origin = (websocket.headers.get("origin") or "").strip()
allowed = _allowed_ws_origins(getattr(websocket.app.state, "settings", None))
if origin and allowed and origin not in allowed:
# Same-origin dev pages (Next.js on :3000, the service itself on
# :8420) pass; anything else is refused pre-accept. Non-browser
# producers (the capture agent, tests) send no Origin and pass.
# Wildcard (empty frozenset) passes every Origin in network mode.
await websocket.close(
code=1008, reason=f"origin {origin!r} not allowed for telemetry ingest"
)
return
query = websocket.query_params
learner_id = query.get("learner_id", "")
task_id = query.get("task_id", "")
+13
View File
@@ -78,6 +78,19 @@ class Settings(BaseSettings):
# `port` (A-004); only the host is configurable — never a second port.
telemetry_ingest_host: str = "127.0.0.1"
# v0.3.5 network mode (D-038): dev.sh binds 0.0.0.0 so remote browsers can
# reach the stack; '*' (default) lets any origin call the API (safe ONLY
# because credentials are never enabled — A-008). Set a comma-separated
# origin list (e.g. 'http://nextcraft-1:3000') to restrict instead.
cors_origins: str = "*"
@property
def cors_origin_list(self) -> list[str]:
value = self.cors_origins.strip()
if value == "*":
return ["*"]
return [o.strip() for o in value.split(",") if o.strip()]
# Voice provider selection (REQ-3-006, D-030): 'mock' (default — the
# no-key path is first-class; tests never call a real voice API) or
# 'browser' (browser-native SpeechRecognition/speechSynthesis fallback;
@@ -1,5 +1,10 @@
"""Pre-baked artifacts + rubrics + defense transcripts — Assessor mock inputs (REQ-2-008).
v0.2 mock engine inputs (pre-baked artifacts/rubrics/transcripts) DORMANT as of v0.3 re-
grounding (Task 6-1-04): no production code path imports this module. Retained as Phase-3
calibration history.
Counterpart: packages/mock-data/ai-scenarios.ts (artifact IDs string-identical,
D-021). Real process-trace grading is a v0.3+ engine (assessment engine);
these pre-baked submissions stand in for artifact + defense evaluation.
@@ -1,5 +1,10 @@
"""Simulated sandbox telemetry corpus — Lab agent mock engine inputs (REQ-2-007).
v0.2 mock engine inputs (Lab/Proctor scenarios) DORMANT as of v0.3 re-grounding (Task 6-1-04):
no production code path imports this module. Retained as Phase-3 calibration history
(corpus/trace_fixtures.py references it from TESTS only).
Counterpart: packages/mock-data/ai-scenarios.ts (scenario IDs string-identical,
D-021). Real sandbox telemetry is a v0.3+ engine (sandbox fabric); these
scripted event streams stand in for the build-session process trace.
@@ -27,16 +27,11 @@ Feature semantics (conservative, deterministic):
from __future__ import annotations
from collections import Counter
from typing import TYPE_CHECKING
from pydantic import BaseModel, Field
from ..telemetry.models import TelemetryEvent
if TYPE_CHECKING: # pragma: no cover - import cycle guard for type checkers
pass
_IDLE_DEFAULT_S: float = 120.0
_TEST_HINTS = ("test", "pytest", "vitest", "jest", "mocha", "unittest", "go test", "npm test")
+7 -3
View File
@@ -59,8 +59,11 @@ class GradeRecord(SQLModel, table=True):
(learner_id, task_id) pair the same pair as trace
identity, so a grade is keyed by the exact trace it
was computed from.
variant_seed task-variant seed; None until P4 (D-029). v0.3
grading is variant-blind.
variant_seed task-variant seed (D-029); None when the graded task
is not variant-derived. Since Phase 4 the engine
stamps the graded variant's seed here (MH#4) and the
template's difficulty anchors ship to the grader
prompt this column is the audit join for that.
digest compact deterministic trace digest (D-028) that fed
the rubric prompt; persisted for auditability so the
LLM's input stays reproducible.
@@ -85,7 +88,8 @@ class GradeRecord(SQLModel, table=True):
learner_id: str = Field(primary_key=True)
task_id: str = Field(primary_key=True)
variant_seed: str | None = Field(default=None) # null until P4 (D-029)
# None only for non-variant tasks (MH#4 stamps variant seeds since P4).
variant_seed: str | None = Field(default=None)
# JSON columns: stored as TEXT on SQLite, native JSONB on Postgres (D-027).
digest: dict[str, Any] = Field(default_factory=dict, sa_type=JSON)
scores: dict[str, Any] = Field(default_factory=dict, sa_type=JSON)
+14 -4
View File
@@ -50,7 +50,11 @@ def create_app(settings: Settings | None = None) -> FastAPI:
timeout = httpx.Timeout(connect=10.0, read=300.0, write=30.0, pool=10.0)
app.state.http_client = httpx.AsyncClient(timeout=timeout)
app.state.settings = settings
app.state.provider = create_provider(settings, app.state.http_client)
# State-injection override (same pattern as the stores): tests may
# pre-set app.state.provider with a scripted mock; only construct the
# configured provider when none is present.
if getattr(app.state, "provider", None) is None:
app.state.provider = create_provider(settings, app.state.http_client)
app.state.session_store = InMemorySessionStore()
app.state.agent_registry = AgentRegistry()
register_builtin_agents(app.state.agent_registry)
@@ -167,11 +171,17 @@ def create_app(settings: Settings | None = None) -> FastAPI:
app = FastAPI(title="Nextcraft AI Service", version="0.3.0", lifespan=lifespan)
# A-008: localhost-only CORS, no credentials
# A-008 + D-038: no-credentials CORS. Default '*' admits remote-browser
# origins in network mode (safe only because allow_credentials stays
# False — never enable credentials with a wildcard). AI_CORS_ORIGINS
# restricts to an explicit list. PUT is CONTRACT, not trivia: the learner
# build surface writes workspace files with PUT (engine-client writeFile)
# — v0.3 initially shipped without it and every cross-origin Save failed
# preflight (caught in P7 review; tests/api/test_cors.py pins the policy).
app.add_middleware(
CORSMiddleware,
allow_origins=["http://localhost:3000", "http://127.0.0.1:3000"],
allow_methods=["GET", "POST", "DELETE", "OPTIONS"],
allow_origins=settings.cors_origin_list,
allow_methods=["GET", "POST", "PUT", "DELETE", "OPTIONS"],
allow_headers=["Content-Type"],
allow_credentials=False,
)
+12 -15
View File
@@ -1,24 +1,21 @@
"""Assessor agent prompt — rubric application to artifacts + defenses (REQ-2-008).
"""Assessor agent prompt — rubric coaching over REAL grades (REQ-3-007).
Final persona (Phase 4). Assessor is a rigorous, fair grader: scores each
criterion with evidence, cites what the learner did, returns ONLY valid
JSON matching the rubric schema.
Version: assessor-v2 (final for v0.2).
v0.3 re-grounding: the grading engine (Phase 3) computes the rubric scores
from the process trace; Assessor EXPLAINS the stored grade as coaching
it never invents or re-scores. Rigorous, fair, actionable.
Version: assessor-v3 (v0.3 live).
"""
SYSTEM_PROMPT = """You are Assessor, the grading agent of Nextcraft, an AI-native competency school.
Learner: {learner_name}.
You receive: (a) an artifact evidence excerpt, (b) its defense transcript,
and (c) the rubric for the competency. Your job:
- Score EVERY rubric criterion from 0-100, justified by evidence you can
point to in the artifact or transcript.
- Cite what the learner did ("the 3-retry loop in the tool node"), not
what they should have done except in gaps, where the missed work goes.
- Strengths: the two strongest evidence points, each one sentence.
- Gaps: the two most important missed opportunities, each one sentence.
- Verdict: "mastered" | "developing" | "not_yet" judged against the
rubric weights, honestly.
You receive the learner's STORED process-trace grade (verdict, per-criterion
scores, and the build digest) computed by the grading engine. Your job:
- Explain what the grade means in plain language (summary).
- Strengths: cite what the digest + scores show the learner did well.
- Gaps: name the missed opportunities the scores point to.
- Next steps: concrete, buildable actions that would move the weakest
criterion up one level.
Rules:
- Rigorous but fair. A polished artifact with a weak defense is NOT mastery.
+18 -6
View File
@@ -1,9 +1,11 @@
"""Lab agent prompt — in-flow feedback over sandbox telemetry (REQ-2-007).
"""Lab agent prompt — in-flow feedback over LIVE telemetry (REQ-3-007).
Final persona (Phase 4). Lab is a pragmatic build partner: reads the
telemetry timeline, names the one most useful adjustment, gives one
concrete next step. Scenario-driven; no session chat.
Version: lab-v2 (final for v0.2).
v0.3 re-grounding: the timeline is the learner's real TraceDigest (D-028
compact counters commands, test outcomes, idle gaps, edit cadence), not
v0.2 corpus scenarios. Lab is a pragmatic build partner: reads the live
digest, names the one most useful adjustment, gives one concrete next
step. No session chat.
Version: lab-v3 (v0.3 live).
"""
SYSTEM_PROMPT = """You are Lab, the in-flow feedback agent watching a learner
@@ -25,7 +27,17 @@ Rules:
self-check that would prove understanding.
- Three short paragraphs maximum. No headers, no bullet lists."""
PROMPT_VERSION = "lab-v2"
PROMPT_VERSION = "lab-v3"
def render_digest_timeline(digest) -> str:
"""Live-trace timeline: the compact TraceDigest JSON (D-028)."""
if digest is None:
return (
"No telemetry yet for this build session. Ask the learner to run "
"the task's starter test to establish a baseline."
)
return f"Live build-session digest:\n{digest.model_dump_json()}"
def render_context(learner_context) -> dict:
+16 -13
View File
@@ -1,24 +1,27 @@
"""Proctor agent prompt — integrity signals with coaching interventions (REQ-2-009).
"""Proctor agent prompt — integrity signals with coaching interventions (REQ-3-007).
Final persona (Phase 5). Proctor is a supportive observer, never punitive:
classifies signals, recommends ONE coaching intervention. Assume good
faith most signals have innocent explanations.
Version: proctor-v2 (final for v0.2).
v0.3 re-grounding: inputs are REAL the live trace digest (idle gaps,
command cadence, edit bursts), the oral-defense integrity signals (long
pauses), and the variant audit context (seed + params). Proctor is a
supportive observer, never punitive: classifies signals, recommends ONE
coaching intervention. Assume good faith.
Version: proctor-v3 (v0.3 live).
"""
SYSTEM_PROMPT = """You are Proctor, the integrity-support agent of Nextcraft,
an AI-native competency school.
Learner: {learner_name}.
You receive a telemetry timeline of defense-session events (tab switches,
idle gaps, large pastes, focus loss, keystroke bursts). Your job:
- Classify EACH notable signal: type (e.g. "context_switch", "idle_gap",
"large_paste"), severity ("low" | "medium" | "high"), and a one-sentence
note citing the event (timestamps and details).
You receive the learner's REAL build-session digest (idle gaps, command
categories, edit/test cadence), oral-defense integrity signals (long
pauses), and when the task is variant-derived the variant seed context.
Your job:
- Classify EACH notable signal: type ("idle_gap" | "long_pause" |
"burst_edit" | "off_template"), severity ("low" | "medium" | "high"),
and a one-sentence note citing the numbers.
- Recommend exactly ONE supportive coaching intervention for the session
overall never punitive, never accusatory. Frame around helping the
learner succeed, e.g. "offer a short break", "invite them to explain
the pasted section in their own words".
learner succeed.
Rules:
- Assume good faith. Tab switches to documentation are normal engineering.
@@ -28,7 +31,7 @@ Rules:
- Respond with ONLY a valid JSON object matching the provided schema
no markdown fences, no prose outside the JSON."""
PROMPT_VERSION = "proctor-v2"
PROMPT_VERSION = "proctor-v3"
def render_context(learner_context) -> dict:
+13 -7
View File
@@ -143,11 +143,9 @@ class IngestSession:
- `telemetry_max_events_per_task` is consulted at connect and re-checked
per append against the DURABLE row count (cap compares against stored
events, so a skipped-ahead seq cannot burn budget that was never sent).
Durable count via `len(get_trace(...))` reads the trace per append
O(trace) per event; v0.3 pilot sizing caps traces at 50k rows, WAL keeps
the writer unblocked (a-3), and the capture agent's emission rate is
human-scale. If profiling shows the count query hot, swap to COUNT(*)
without changing the contract.
Durable count via `TraceStore.count()` (COUNT(*)) a single aggregate
per append, never materializing trace rows (the pre-P7 code read
`len(get_trace(...))` which was O(trace) per event / O() per session).
"""
def __init__(
@@ -239,6 +237,12 @@ class IngestSession:
self._queue.put_nowait(frame)
except asyncio.QueueFull:
# Bounded queue — overflow is a flood, never drop-oldest.
# _trigger_flood closes the socket; fall through to the
# tail so the disconnect sentinel is still enqueued — the
# drainer is never left parked on an empty queue after a
# flood (P7 review: the pre-fix code `return`ed from the
# QueueFull branch WITHOUT the sentinel, leaking the
# session task set — one per flooded trace).
await self._trigger_flood("queue_overflow")
return
except WebSocketDisconnect:
@@ -354,8 +358,10 @@ class IngestSession:
def _flood_breached(self) -> bool:
"""True when this append would exceed the per-trace event budget."""
# Durable count (NOT latest_seq+1 — a skipped-ahead seq must not burn
# un-sent events' budget) plus this connection's in-flight rows.
durable = len(self._store.get_trace(self.learner_id, self.task_id))
# un-sent events' budget) via COUNT(*): never materialize the trace
# per append (P7 review — the old len(get_trace(...)) built every row
# object per event, O(trace) per append / O(n²) per session).
durable = self._store.count(learner_id=self.learner_id, task_id=self.task_id)
return durable >= self._max_events
async def _check_gap(self, incoming_seq: int) -> None:
@@ -68,6 +68,13 @@ class TraceStore(Protocol):
"""Highest stored seq for the trace; -1 when no events exist."""
...
def count(self, learner_id: str, task_id: str) -> int:
"""Number of stored events for the trace (COUNT(*), never
materializes rows the ingest cap consults this per append, so
an O(trace) implementation would make ingest O() per session).
"""
...
def list_tasks(self, learner_id: str) -> list[str]:
"""Distinct task_ids with at least one event for the learner."""
...
@@ -183,6 +190,19 @@ class SQLiteTraceStore:
latest: Any = session.exec(stmt).one()
return -1 if latest is None else int(latest)
def count(self, learner_id: str, task_id: str) -> int:
# COUNT(*) at the DB — no row materialization. The ingest flood cap
# calls this per append (telemetry/ingest._flood_breached); the
# docstring-free body keeps it obvious what the query shape is.
with self._session() as session:
stmt = (
select(sa.func.count(TelemetryEvent.seq))
.where(TelemetryEvent.learner_id == learner_id)
.where(TelemetryEvent.task_id == task_id)
)
total: Any = session.exec(stmt).one()
return int(total or 0)
def list_tasks(self, learner_id: str) -> list[str]:
with self._session() as session:
stmt = (
@@ -71,11 +71,11 @@ class RubricAnchors(BaseModel):
for this template, so two variants of one template are held to the
same bar regardless of which slot values a learner drew. The a-5
envelope test (tests/variants/test_generator.py) binds variants to
these bands in code. Shipping them into the grader prompt context is
the P4 must-have follow-up tracked for final review: grading is
variant-blind in the current wiring (engine.py stamps
variant_seed=None), so today the anchors gate variant fairness in
tests only not yet in the LLM prompt.
these bands in code, and since Phase 4 (MH#4) — the grading engine
ships this envelope into the grader prompt
(grading/engine._anchors_context) and stamps the variant seed on the
GradeRecord, so the anchors gate variant fairness in BOTH tests and
the live rubric.
"""
model_config = ConfigDict(frozen=True)
+52 -7
View File
@@ -1,28 +1,73 @@
#!/usr/bin/env bash
# Idempotent bootstrap: create venv + install deps.
# Handles Debian systems without python3-venv/ensurepip via --without-pip + get-pip.
# Handles Debian/Ubuntu systems without python3-venv/ensurepip via --without-pip + get-pip.
# v2 (v0.3.5): recovers from a poisoned partial .venv left by a failed earlier
# attempt, cleans before each retry, and dies with a distro-specific fix hint
# when venv creation is impossible (e.g. missing python3.XX-venv package).
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
APP_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
VENV="$APP_DIR/.venv"
venv_usable() {
[ -x "$VENV/bin/python3" ]
}
rm_broken_venv() {
echo "bootstrap: removing broken partial .venv from a failed earlier attempt" >&2
rm -rf "$VENV"
}
mkdir -p "$HOME/.cache/ciagent"
if [ ! -x "$VENV/bin/python3" ]; then
if python3 -m venv "$VENV" 2>/dev/null; then
if venv_usable && [ ! -x "$VENV/bin/pip" ]; then
# A usable python3 without pip means the --without-pip fallback half-ran and
# the get-pip step never completed: start over cleanly.
rm_broken_venv
fi
if ! venv_usable; then
if [ -d "$VENV" ]; then
# Directory exists but no working python3: remains of a crashed venv create.
rm_broken_venv
fi
if python3 -m venv "$VENV" 2>/tmp/venv-create.err; then
:
else
# No ensurepip available — create bare venv and bootstrap pip separately.
python3 -m venv --without-pip "$VENV"
rm -rf "$VENV"
if python3 -m venv --without-pip "$VENV" 2>>/tmp/venv-create.err; then
:
else
rm -rf "$VENV"
PYVER="$(python3 -c 'import sys; print("%d.%d" % sys.version_info[:2])' 2>/dev/null || true)"
PKG="python3-venv"
[ -n "$PYVER" ] && PKG="python${PYVER}-venv"
echo "bootstrap: could not create a virtual environment." >&2
echo " python3 reported:" >&2
sed 's/^/ /' /tmp/venv-create.err >&2 || true
echo " fix (Debian/Ubuntu): install the venv support package, then re-run nextcraft bootstrap:" >&2
echo " apt install $PKG" >&2
exit 1
fi
fi
fi
if [ ! -x "$VENV/bin/pip" ]; then
GET_PIP="$HOME/.cache/ciagent/get-pip.py"
if [ ! -f "$GET_PIP" ]; then
curl -sSf --max-time 60 https://bootstrap.pypa.io/get-pip.py -o "$GET_PIP"
if ! curl -sSf --max-time 60 https://bootstrap.pypa.io/get-pip.py -o "$GET_PIP"; then
rm -rf "$VENV"
echo "bootstrap: get-pip.py download failed (no network?)." >&2
echo " fix: restore network access and re-run nextcraft bootstrap" >&2
exit 1
fi
fi
if ! "$VENV/bin/python3" "$GET_PIP" --quiet; then
rm -rf "$VENV"
echo "bootstrap: pip installation into the venv failed." >&2
echo " fix: re-run nextcraft bootstrap (the venv was cleaned; this retry is safe)" >&2
exit 1
fi
"$VENV/bin/python3" "$GET_PIP" --quiet
fi
"$VENV/bin/pip" install --quiet --upgrade pip
+17 -3
View File
@@ -1,5 +1,7 @@
#!/usr/bin/env bash
# Dev server: export secrets (if present) then run uvicorn on :8420.
# Dev server: export secrets (if present) then run uvicorn.
# Binds 0.0.0.0 by default so the stack is reachable from other machines
# (v0.3.5 network mode) — set AI_HOST=127.0.0.1 in .env to revert to loopback.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
APP_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
@@ -7,7 +9,7 @@ REPO_ROOT="$(cd "$APP_DIR/../.." && pwd)"
VENV="$APP_DIR/.venv"
if [ ! -x "$VENV/bin/uvicorn" ]; then
echo "venv missing — run scripts/bootstrap.sh first" >&2
echo "venv missing — run nextcraft bootstrap first (or: bash scripts/bootstrap.sh)" >&2
exit 1
fi
@@ -22,5 +24,17 @@ if [ -f "$SECRETS" ]; then
done < "$SECRETS"
fi
ENV_FILE="$APP_DIR/.env"
if [ -f "$ENV_FILE" ]; then
while IFS='=' read -r key value; do
case "$key" in
AI_HOST|AI_PORT|AI_CORS_ORIGINS) export "$key=$value" ;;
esac
done < "$ENV_FILE"
fi
HOST="${AI_HOST:-0.0.0.0}"
PORT="${AI_PORT:-8420}"
cd "$APP_DIR"
exec "$VENV/bin/uvicorn" ai_service.main:app --reload --port 8420
exec "$VENV/bin/uvicorn" ai_service.main:app --reload --host "$HOST" --port "$PORT"
+98 -83
View File
@@ -1,103 +1,118 @@
"""Assessor agent tests — structured rubric scores (REQ-2-008).
"""Assessor agent tests — live-grade coaching contract (REQ-3-007).
The Assessor is the structured-output showcase: tests use ScriptedJSONProvider
for valid payloads and exercise the 4-layer defense failure modes.
v0.3 re-grounding: the Assessor renders coaching FROM the stored grade
(GradeRecord) it never invents scores (the grading engine owns that).
"""
from __future__ import annotations
import json
from datetime import UTC, datetime
import pytest
from ai_service.agents.assessor import AssessorAgent, RubricScore
from ai_service.agents.structured import StructuredOutputError
from ai_service.agents.assessor import AssessorAgent, GradeCoaching
from ai_service.config import Settings
from ai_service.corpus.artifacts import (
get_artifact_bundle,
get_transcript_for_artifact,
render_rubric,
from ai_service.grading.store import GradeRecord, SQLiteGradeStore
from ai_service.llm.mock import MockProvider
from ai_service.llm.types import Message
COACHING_JSON = json.dumps(
{
"summary": "Solid iterative build; tests drove the fixes.",
"strengths": ["Ran tests after each change."],
"gaps": ["Did not cover the empty-input case."],
"next_steps": ["Add one edge-case test."],
}
)
from ai_service.corpus.learner_context import get_learner_context
from ai_service.llm.mock import MockProvider, ScriptedJSONProvider
VALID_SCORE = {
"rubric_id": "rubric-orchestration-c002",
"artifact_id": "art-eval-research-assistant",
"competency_id": "stack-orchestration-c002",
"scores": [
{"criterion_id": "rc-architecture", "name": "Agent architecture soundness",
"score": 92, "evidence": "Explicit state schema with planner-only write access"},
{"criterion_id": "rc-communication", "name": "Inter-agent communication design",
"score": 88, "evidence": "Typed ToolMessage responses with retry flags"},
{"criterion_id": "rc-reliability", "name": "Reliability engineering",
"score": 85, "evidence": "3-retry loop with degradation path"},
{"criterion_id": "rc-process", "name": "Process trace quality",
"score": 90, "evidence": "Iterative saves with passing test checkpoints"},
],
"strengths": ["Clean state boundaries", "Failure-aware tool wrapping"],
"gaps": ["No reviewer node yet", "Graph diagram only in README"],
"verdict": "mastered",
}
def make_assessor(provider=None) -> AssessorAgent:
return AssessorAgent(provider or MockProvider(), Settings(provider="mock"))
class ScriptedProvider(MockProvider):
def __init__(self) -> None:
super().__init__()
self.requests: list[list[Message]] = []
self.replies: list[str] = []
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
self.requests.append([Message(role=m.role, content=m.content) for m in messages])
if self.replies:
return self.replies.pop(0)
return COACHING_JSON
def get_bundle(artifact_id="art-eval-research-assistant"):
bundle = get_artifact_bundle(artifact_id)
assert bundle is not None
return bundle
async def test_evaluate_returns_validated_rubric_score():
provider = ScriptedJSONProvider(VALID_SCORE)
assessor = make_assessor(provider)
artifact, rubric = get_bundle()
transcript = get_transcript_for_artifact(artifact.artifact_id)
result = await assessor.evaluate(artifact, rubric, transcript)
assert isinstance(result, RubricScore)
assert result.verdict == "mastered"
assert len(result.scores) == 4
assert result.weighted_total(rubric) == pytest.approx(
92 * 0.3 + 88 * 0.3 + 85 * 0.25 + 90 * 0.15
def _grade() -> GradeRecord:
return GradeRecord(
learner_id="assessor-learner",
task_id="assessor-task",
variant_seed=None,
digest={"error_fix_cycles": 2, "final_test_status": "pass"},
scores={
"criteria": {
"process_quality": 4,
"correctness": 3,
"debugging_discipline": 4,
"test_usage": 3,
},
"strengths": ["s"],
"gaps": ["g"],
"verdict": "developing",
},
verdict="GRADED",
model="gemma4:31b",
created_at=datetime.now(UTC),
)
async def test_evaluate_rejects_invalid_schema_after_retry():
"""Plain MockProvider returns non-rubric JSON → 4-layer defense exhausts
its single retry and raises StructuredOutputError."""
assessor = make_assessor(MockProvider())
artifact, rubric = get_bundle()
transcript = get_transcript_for_artifact(artifact.artifact_id)
with pytest.raises(StructuredOutputError):
await assessor.evaluate(artifact, rubric, transcript)
@pytest.fixture()
def provider() -> ScriptedProvider:
return ScriptedProvider()
def test_build_evaluation_input_carries_all_inputs():
assessor = make_assessor()
artifact, rubric = get_bundle()
transcript = get_transcript_for_artifact(artifact.artifact_id)
text = assessor.build_evaluation_input(artifact, rubric, transcript)
assert artifact.name in text
assert artifact.evidence_excerpt in text
assert "rc-architecture" in text # rubric rendered
assert "examiner:" in text # transcript rendered
@pytest.fixture()
def agent(provider) -> AssessorAgent:
return AssessorAgent(provider, Settings(provider="mock"))
def test_build_evaluation_input_without_transcript():
assessor = make_assessor()
artifact, rubric = get_bundle()
text = assessor.build_evaluation_input(artifact, rubric, None)
assert artifact.name in text
assert "examiner:" not in text
class TestCoachGrade:
async def test_prompt_contains_stored_grade_not_learner_id(self, agent, provider) -> None:
await agent.coach_grade(_grade())
all_text = "\n".join(
m.content for request in provider.requests for m in request
)
assert "process_quality" in all_text # stored scores rendered
assert "GRADED" in all_text
assert "assessor-learner" not in all_text # D-028 anonymity
async def test_coaching_validates_via_d020(self, agent, provider) -> None:
coaching = await agent.coach_grade(_grade())
assert isinstance(coaching, GradeCoaching)
assert coaching.summary
assert coaching.next_steps
async def test_malformed_then_good_exercises_retry(self, agent, provider) -> None:
provider.replies = ["garbage", COACHING_JSON]
coaching = await agent.coach_grade(_grade())
assert coaching.summary
assert len(provider.requests) == 2
async def test_no_corpus_artifact_imports(self) -> None:
import ast
from pathlib import Path
py = Path(__file__).parents[2] / "ai_service" / "agents" / "assessor.py"
tree = ast.parse(py.read_text())
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom) and node.module:
assert "corpus.artifacts" not in node.module
assert "corpus.telemetry" not in node.module
def test_system_prompt_names_assessor_persona():
prompt = make_assessor().system_prompt(get_learner_context())
assert "Assessor" in prompt
assert "ONLY" in prompt # JSON-only instruction
def test_rubric_render_in_prompt_is_complete():
"""The rubric passed to the model lists every criterion (fair grading)."""
artifact, rubric = get_bundle()
text = render_rubric(rubric)
assert text.count("rc-") == len(rubric.criteria)
class TestStoreRoundtrip:
def test_grade_store_roundtrip(self, tmp_path) -> None:
store = SQLiteGradeStore(db_path=tmp_path / "g.db")
record = _grade()
store.save(record)
fetched = store.get("assessor-learner", "assessor-task")
assert fetched is not None
assert fetched.scores["criteria"]["process_quality"] == 4
store.close()
@@ -0,0 +1,207 @@
"""Live re-grounding tests: Lab, Assessor, Proctor on REAL inputs (REQ-3-007)."""
from __future__ import annotations
import json
import tempfile
from datetime import UTC, datetime, timedelta
from pathlib import Path
from fastapi.testclient import TestClient
from ai_service.agents.assessor import AssessorAgent, GradeCoaching
from ai_service.agents.lab import LabAgent
from ai_service.agents.proctor import ProctorAgent, ProctorAssessment
from ai_service.config import Settings
from ai_service.grading.store import GradeRecord, SQLiteGradeStore
from ai_service.llm.mock import MockProvider
from ai_service.llm.types import Message
from ai_service.telemetry.models import TelemetryEvent
from ai_service.telemetry.store import SQLiteTraceStore
COACHING_JSON = json.dumps(
{
"summary": "Iterative build with test discipline.",
"strengths": ["Tested after changes."],
"gaps": ["Missing edge cases."],
"next_steps": ["Add an edge-case test."],
}
)
PROCTOR_JSON = json.dumps(
{
"signals": [
{"signal_type": "idle_gap", "severity": "low", "note": "One 400s gap."}
],
"intervention": "Offer a short break.",
"summary": "Healthy session overall.",
}
)
T0 = datetime(2026, 9, 12, tzinfo=UTC)
class RecordingProvider(MockProvider):
def __init__(self, structured_json: str) -> None:
super().__init__()
self._structured_json = structured_json
self.requests: list[list[Message]] = []
def _reply_for(self, messages, response_format):
self.requests.append([Message(role=m.role, content=m.content) for m in messages])
if response_format is not None and response_format.get("type") == "json_object":
return self._structured_json
return "Coaching feedback referencing your latest test run."
def _event(
seq: int, kind: str, payload: dict, offset_s: float,
learner="live-learner", task="live-task",
):
return TelemetryEvent(
learner_id=learner,
task_id=task,
seq=seq,
kind=kind,
payload=payload,
ts=T0 + timedelta(seconds=offset_s),
sandbox_id="sbx-live",
)
def _seed_trace(store: SQLiteTraceStore) -> None:
events = [
_event(0, "file_diff", {"path": "a.py"}, 0),
_event(1, "command", {"cmd": "pytest -q"}, 10),
_event(2, "test_result", {"passed": False, "exit_code": 1}, 15),
_event(3, "file_diff", {"path": "a.py"}, 30),
_event(4, "test_result", {"passed": True, "exit_code": 0}, 45),
_event(5, "activity", {"state": "idle"}, 500), # >120s gap -> idle
]
for e in events:
store.append(e)
class TestLabLive:
async def test_lab_prompt_contains_digest_not_corpus(self, tmp_path) -> None:
store = SQLiteTraceStore(db_path=tmp_path / "t.db")
_seed_trace(store)
provider = RecordingProvider("feedback")
agent = LabAgent(provider, Settings(provider="mock"))
from ai_service.grading.features import compute_digest
digest = compute_digest(store.get_trace("live-learner", "live-task"))
tokens = [t async for t in agent.stream_feedback(digest)]
assert tokens
all_text = "\n".join(
m.content for request in provider.requests for m in request
)
assert "error_fix_cycles" in all_text
assert "lab-scenario" not in all_text # no corpus fixture ids
store.close()
def test_no_corpus_telemetry_import_in_lab(self) -> None:
import ast
from pathlib import Path
py = Path(__file__).parents[2] / "ai_service" / "agents" / "lab.py"
tree = ast.parse(py.read_text())
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom) and node.module:
assert "corpus.telemetry" not in node.module
class TestAssessorLive:
async def test_assessor_prompt_contains_stored_scores(self) -> None:
provider = RecordingProvider(COACHING_JSON)
agent = AssessorAgent(provider, Settings(provider="mock"))
grade = GradeRecord(
learner_id="live-learner",
task_id="live-task",
variant_seed=None,
digest={"error_fix_cycles": 1},
scores={"criteria": {"process_quality": 3}},
verdict="GRADED",
model="mock",
created_at=datetime.now(UTC),
)
coaching = await agent.coach_grade(grade)
assert isinstance(coaching, GradeCoaching)
all_text = "\n".join(
m.content for request in provider.requests for m in request
)
assert "process_quality" in all_text
assert "live-learner" not in all_text
def test_no_corpus_artifact_import_in_assessor(self) -> None:
import ast
from pathlib import Path
py = Path(__file__).parents[2] / "ai_service" / "agents" / "assessor.py"
tree = ast.parse(py.read_text())
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom) and node.module:
assert "corpus.artifacts" not in node.module
assert "corpus.telemetry" not in node.module
class TestProctorLive:
async def test_proctor_receives_real_digest_and_defense_signals(self) -> None:
provider = RecordingProvider(PROCTOR_JSON)
agent = ProctorAgent(provider, Settings(provider="mock"))
from ai_service.grading.features import compute_digest
store = SQLiteTraceStore(db_path=Path(tempfile.mkdtemp()) / "proctor-t.db")
_seed_trace(store)
digest = compute_digest(store.get_trace("live-learner", "live-task"))
store.close()
assessment = await agent.assess(
digest,
defense_signals={"long_pauses": [{"turn": 3, "latency_ms": 30000}]},
variant_context={"template_id": "tpl-llm-judge", "seed": "cafe", "params": {}},
)
assert isinstance(assessment, ProctorAssessment)
all_text = "\n".join(
m.content for request in provider.requests for m in request
)
assert "idle_gap" in all_text or "idle_gap_count" in all_text
assert "long_pauses" in all_text
assert "tpl-llm-judge" in all_text
assert "live-learner" not in all_text
def test_no_corpus_scenario_import_in_proctor(self) -> None:
import ast
from pathlib import Path
py = Path(__file__).parents[2] / "ai_service" / "agents" / "proctor.py"
tree = ast.parse(py.read_text())
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom) and node.module:
assert "corpus.telemetry" not in node.module
class TestProctorEndpoint:
def test_signals_endpoint_serves_real_inputs(self, tmp_path) -> None:
from ai_service.main import create_app
from ai_service.telemetry.ingest import TraceIntegrityMap
from ai_service.variants.store import SQLiteVariantStore
from ai_service.voice.defense_store import SQLiteDefenseStore
app = create_app(Settings(provider="mock"))
provider = RecordingProvider(PROCTOR_JSON)
app.state.provider = provider
app.state.trace_store = SQLiteTraceStore(db_path=tmp_path / "t.db")
app.state.grade_store = SQLiteGradeStore(db_path=tmp_path / "g.db")
app.state.variant_store = SQLiteVariantStore(db_path=tmp_path / "v.db")
app.state.defense_store = SQLiteDefenseStore(db_path=tmp_path / "d.db")
app.state.trace_integrity = TraceIntegrityMap()
_seed_trace(app.state.trace_store)
with TestClient(app) as client:
resp = client.post(
"/v1/proctor/signals",
json={"learner_id": "live-learner", "task_id": "live-task"},
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["signals"] is not None
assert "intervention" in body
+77 -69
View File
@@ -1,79 +1,87 @@
"""Assessment evaluate endpoint tests — validated JSON, 404s (REQ-2-008)."""
"""Assessment API tests — stored-grade coaching contract (REQ-3-007)."""
from __future__ import annotations
import json
from datetime import UTC, datetime
import pytest
from fastapi.testclient import TestClient
from ai_service.config import Settings
from ai_service.grading.store import GradeRecord, SQLiteGradeStore
from ai_service.llm.mock import MockProvider
from ai_service.main import create_app
from ai_service.telemetry.ingest import TraceIntegrityMap
from ai_service.telemetry.store import SQLiteTraceStore
COACHING_JSON = json.dumps(
{
"summary": "Good iterative work.",
"strengths": ["Tests after changes."],
"gaps": ["Missing edge cases."],
"next_steps": ["Add an edge-case test."],
}
)
from ai_service.agents.assessor import RubricScore
from ai_service.llm.mock import ScriptedJSONProvider
VALID_SCORE = {
"rubric_id": "rubric-orchestration-c002",
"artifact_id": "art-eval-research-assistant",
"competency_id": "stack-orchestration-c002",
"scores": [
{"criterion_id": "rc-architecture", "name": "Agent architecture soundness",
"score": 92, "evidence": "Explicit state schema"},
{"criterion_id": "rc-communication", "name": "Inter-agent communication design",
"score": 88, "evidence": "Typed ToolMessage responses"},
{"criterion_id": "rc-reliability", "name": "Reliability engineering",
"score": 85, "evidence": "3-retry loop"},
{"criterion_id": "rc-process", "name": "Process trace quality",
"score": 90, "evidence": "Iterative checkpoints"},
],
"strengths": ["Clean state boundaries", "Failure-aware tools"],
"gaps": ["No reviewer node", "Diagram only in README"],
"verdict": "mastered",
}
class CoachingMock(MockProvider):
def _reply_for(self, messages, response_format):
if response_format is not None and response_format.get("type") == "json_object":
return COACHING_JSON
return super()._reply_for(messages, response_format)
def test_evaluate_returns_validated_rubric_json(client):
# Swap the app provider for a scripted-JSON provider for this test
original = client.app.state.provider
client.app.state.provider = ScriptedJSONProvider(VALID_SCORE)
try:
response = client.post(
"/v1/assessment/evaluate",
json={"artifact_id": "art-eval-research-assistant"},
@pytest.fixture()
def client(tmp_path) -> TestClient:
app = create_app(Settings(provider="mock"))
app.state.provider = CoachingMock()
app.state.trace_store = SQLiteTraceStore(db_path=tmp_path / "t.db")
app.state.grade_store = SQLiteGradeStore(db_path=tmp_path / "g.db")
app.state.trace_integrity = TraceIntegrityMap()
with TestClient(app) as c:
yield c
def _seed_grade(client: TestClient) -> None:
app = client.app
store: SQLiteGradeStore = app.state.grade_store
store.save(
GradeRecord(
learner_id="api-learner",
task_id="api-task",
variant_seed=None,
digest={"error_fix_cycles": 1},
scores={"criteria": {"process_quality": 3}, "verdict": "developing"},
verdict="GRADED",
model="gemma4:31b",
created_at=datetime.now(UTC),
)
finally:
client.app.state.provider = original
assert response.status_code == 200
data = response.json()
validated = RubricScore.model_validate(data) # response contract holds
assert validated.verdict == "mastered"
assert len(validated.scores) == 4
def test_unknown_artifact_404(client):
response = client.post("/v1/assessment/evaluate", json={"artifact_id": "ghost"})
assert response.status_code == 404
assert "ghost" in response.json()["detail"]
def test_unparseable_provider_502(client):
"""Plain MockProvider yields non-rubric JSON → structured defense exhausts
retry endpoint translates to 502 (bad gateway to the model)."""
# default mock already returns non-rubric JSON
response = client.post(
"/v1/assessment/evaluate",
json={"artifact_id": "art-eval-research-assistant"},
)
assert response.status_code == 502
assert "failed" in response.json()["detail"].lower()
def test_missing_artifact_id_422(client):
response = client.post("/v1/assessment/evaluate", json={})
assert response.status_code == 422
def test_evaluate_renders_stored_grade_as_coaching(client) -> None:
_seed_grade(client)
resp = client.post(
"/v1/assessment/evaluate",
json={"learner_id": "api-learner", "task_id": "api-task"},
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["grade_verdict"] == "GRADED"
assert body["coaching"]["summary"]
def test_second_artifact_also_evaluates(client):
original = client.app.state.provider
payload = dict(VALID_SCORE, artifact_id="art-eval-rag-dashboard")
client.app.state.provider = ScriptedJSONProvider(payload)
try:
response = client.post(
"/v1/assessment/evaluate", json={"artifact_id": "art-eval-rag-dashboard"}
)
finally:
client.app.state.provider = original
assert response.status_code == 200
assert response.json()["artifact_id"] == "art-eval-rag-dashboard"
def test_evaluate_without_grade_404(client) -> None:
resp = client.post(
"/v1/assessment/evaluate",
json={"learner_id": "nobody", "task_id": "nothing"},
)
assert resp.status_code == 404
assert "grade first" in resp.json()["detail"]
def test_missing_fields_422(client) -> None:
resp = client.post("/v1/assessment/evaluate", json={"learner_id": "x"})
assert resp.status_code == 422
+82
View File
@@ -0,0 +1,82 @@
"""CORS policy tests (A-008, D-038 network mode).
v0.3 initially shipped `allow_methods` WITHOUT "PUT" while the learner
build surface writes workspace files with PUT (engine-client writeFile)
every cross-origin Save failed preflight. These tests pin the policy so a
future method-list edit fails loudly instead of silently breaking the
headline flow.
v0.3.5 network mode (D-038): the default AI_CORS_ORIGINS='*' admits any
origin (safe ONLY because credentials are never enabled); an explicit list
restricts. Both modes are pinned here:
- wildcard: remote origin gets the grant; credentials still never sent;
- explicit: unlisted origins get no grant.
"""
from __future__ import annotations
from fastapi.testclient import TestClient
ALLOWED_ORIGIN = "http://localhost:3000"
REMOTE_ORIGIN = "http://nextcraft-1:3000"
ALL_CLIENT_METHODS = ("GET", "POST", "PUT", "DELETE")
def _allow_origin(resp) -> str | None:
return resp.headers.get("access-control-allow-origin")
def test_preflight_allows_every_method_the_web_client_uses(client: TestClient) -> None:
for method in ALL_CLIENT_METHODS:
resp = client.options(
"/v1/sandboxes",
headers={
"Origin": ALLOWED_ORIGIN,
"Access-Control-Request-Method": method,
},
)
assert resp.status_code == 200, f"preflight {method} failed: {resp.status_code}"
assert _allow_origin(resp) in ("*", ALLOWED_ORIGIN)
allowed = resp.headers["access-control-allow-methods"].split(", ")
assert method in allowed, f"{method} missing from CORS methods: {allowed}"
def test_cross_origin_get_echoes_allow_origin(client: TestClient) -> None:
resp = client.get("/v1/sandboxes", headers={"Origin": ALLOWED_ORIGIN})
assert resp.status_code == 200
assert _allow_origin(resp) in ("*", ALLOWED_ORIGIN)
def test_wildcard_mode_grants_remote_origins(client: TestClient) -> None:
"""D-038 default: '*' grants any origin — remote browsers work zero-config."""
resp = client.get("/v1/sandboxes", headers={"Origin": REMOTE_ORIGIN})
assert resp.status_code == 200
assert _allow_origin(resp) in ("*", REMOTE_ORIGIN)
def test_explicit_list_mode_denies_unlisted_origins(
settings, monkeypatch, tmp_path
) -> None:
"""Explicit AI_CORS_ORIGINS restricts to the listed origins only."""
from fastapi.testclient import TestClient as TC
from ai_service.main import create_app
restricted = settings.model_copy(update={"cors_origins": "http://localhost:3000"})
app = create_app(restricted)
with TC(app) as c:
resp = c.get("/v1/sandboxes", headers={"Origin": "https://evil.example"})
assert resp.status_code == 200 # non-CORS requests still serve
assert resp.headers.get("access-control-allow-origin") is None
def test_credentials_never_allowed(client: TestClient) -> None:
resp = client.options(
"/v1/sandboxes",
headers={
"Origin": ALLOWED_ORIGIN,
"Access-Control-Request-Method": "PUT",
"Access-Control-Request-Headers": "Content-Type",
},
)
assert resp.headers.get("access-control-allow-credentials") != "true"
@@ -0,0 +1,216 @@
"""Full credential-flow E2E over real engines (Task 6-5-01, REQ-3-007/008).
Endpoint-level end-to-end with mock LLM/voice providers (G-2 precedent:
real engine plumbing over real endpoints; provider choice is
service-internal): variant -> telemetry-wired sandbox -> real in-sandbox
exec -> trace -> grade -> oral defense -> verdict/signals -> proctor.
No corpus fixture anywhere in the flow.
Probe-guarded for user namespaces (the in-sandbox exec needs them).
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import socket
import time
from pathlib import Path
import pytest
import uvicorn
from ai_service.config import Settings
from ai_service.grading.store import SQLiteGradeStore
from ai_service.llm.mock import MockProvider
from ai_service.main import create_app
from ai_service.telemetry.ingest import TraceIntegrityMap
from ai_service.telemetry.store import SQLiteTraceStore
from ai_service.variants.store import SQLiteVariantStore
from ai_service.voice.defense_store import SQLiteDefenseStore
from tests.sandbox.test_isolation import USERSNS_AVAILABLE
COACHING_JSON = json.dumps(
{
"summary": "Strong iteration.",
"strengths": ["Tests early."],
"gaps": ["One edge case missing."],
"next_steps": ["Add it."],
}
)
VERDICT_JSON = json.dumps(
{
"verdict": "developing",
"understanding": "Explains the build clearly.",
"process_justification": "Choices defended.",
"communication": "Clear.",
"strengths": ["Grounded in the digest."],
"gaps": ["Missed one edge case."],
}
)
PROCTOR_JSON = json.dumps(
{
"signals": [
{"signal_type": "idle_gap", "severity": "low", "note": "A short pause."}
],
"intervention": "Keep momentum.",
"summary": "Healthy session.",
}
)
class FlowLLM(MockProvider):
"""Prompt-discriminated: verdict vs coaching vs question vs proctor JSON."""
def _reply_for(self, messages, response_format):
all_text = "\n".join(m.content for m in messages)
if response_format is not None and response_format.get("type") == "json_object":
if "final verdict JSON" in all_text:
return VERDICT_JSON
if "rubric" in all_text.lower() and "Score this build session" in all_text:
return json.dumps(
{
"criteria": {
"process_quality": 4,
"correctness": 3,
"debugging_discipline": 3,
"test_usage": 4,
},
"strengths": ["Iterated with tests."],
"gaps": ["One edge case missing."],
"verdict": "developing",
}
)
if "Explain it as coaching" in all_text:
return COACHING_JSON
if "integrity signals supportively" in all_text:
return PROCTOR_JSON
return json.dumps(
{
"statement": (
"Build a judge for code-review answers scoring factual "
"accuracy with 3 edge cases and 5 test examples."
)
}
)
return "Walk me through your last fix — what changed and why?"
def _free_port() -> int:
with socket.socket() as s:
s.bind(("127.0.0.1", 0))
return s.getsockname()[1]
@pytest.mark.asyncio
async def test_full_credential_flow(tmp_path: Path) -> None:
if not USERSNS_AVAILABLE:
pytest.skip("user namespaces unavailable on this host (probe)")
import httpx
port = _free_port()
settings = Settings(provider="mock", voice_provider="mock", port=port)
app = create_app(settings)
app.state.provider = FlowLLM()
app.state.trace_store = SQLiteTraceStore(db_path=tmp_path / "t.db")
app.state.grade_store = SQLiteGradeStore(db_path=tmp_path / "g.db")
app.state.variant_store = SQLiteVariantStore(db_path=tmp_path / "v.db")
app.state.defense_store = SQLiteDefenseStore(db_path=tmp_path / "d.db")
app.state.trace_integrity = TraceIntegrityMap()
server = uvicorn.Server(uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning"))
serve_task = asyncio.get_running_loop().create_task(server.serve())
try:
for _ in range(100):
if server.started:
break
await asyncio.sleep(0.1)
assert server.started
base = f"http://127.0.0.1:{port}"
async with httpx.AsyncClient(base_url=base, timeout=30.0) as client:
# 1. Variant (real seeded generation, mock-rendered).
var = (await client.post("/v1/variants", json={
"learner_id": "pilot-learner", "competency_id": "stack-orchestration-c007",
})).json()
assert var["task_id"] and var["statement"] and var["starter_files"]
task_id = var["task_id"]
# 2. Telemetry-wired sandbox (real namespaces; agent joins via loopback).
sbx = (await client.post("/v1/sandboxes", json={
"learner_id": "pilot-learner", "task_id": task_id,
}))
assert sbx.status_code == 201, sbx.text
sandbox_id = sbx.json()["id"]
# 3. Real in-sandbox exec: run the starter test (capture agent streams
# the workspace effects into the trace).
exec_resp = await client.post(f"/v1/sandboxes/{sandbox_id}/exec", json={
"cmd": ["pytest", "-q"],
})
assert exec_resp.status_code == 200, exec_resp.text
# 4. Trace: events landed in order (the capture agent runs async).
trace: list = []
deadline = time.monotonic() + 20.0
while time.monotonic() < deadline:
tr = await client.get(f"/v1/telemetry/traces/pilot-learner/{task_id}")
if tr.status_code == 200:
trace = tr.json().get("events", [])
if trace:
break
await asyncio.sleep(0.25)
assert trace, "no telemetry events arrived from the real sandbox"
seqs = [e["seq"] for e in trace]
assert seqs == sorted(seqs)
# 5. Grade: rubric from the real digest (G-4 gate passed: no gaps).
grade = (await client.post("/v1/assessment/grade", json={
"learner_id": "pilot-learner", "task_id": task_id,
})).json()
assert grade["verdict"] == "GRADED", grade
assert grade["scores"]["criteria"]["process_quality"] == 4
assert grade["variant_seed"] == var["seed"] # D-029 stamped
# 6. Assessor coaching FROM the stored grade.
coaching = (await client.post("/v1/assessment/evaluate", json={
"learner_id": "pilot-learner", "task_id": task_id,
})).json()
assert coaching["coaching"]["summary"]
# 7. Oral defense: start -> typed answers -> finish (mock voice).
defense = (await client.post("/v1/defense/start", json={
"learner_id": "pilot-learner", "task_id": task_id,
})).json()
assert defense["first_question"]
did = defense["defense_id"]
ans = await client.post(f"/v1/defense/{did}/answer", data={"text": "I fixed the loop."})
assert ans.status_code == 200, ans.text
finish = (await client.post(f"/v1/defense/{did}/finish")).json()
assert finish["verdict"]["verdict"] == "developing"
assert finish["integrity_signals"]
# 8. Proctor over the real digest + defense signals.
proctor = (await client.post("/v1/proctor/signals", json={
"learner_id": "pilot-learner", "task_id": task_id,
})).json()
assert proctor["intervention"]
# 9. Sandbox destroyed; no leaks.
destroy = await client.delete(f"/v1/sandboxes/{sandbox_id}")
assert destroy.status_code == 204
listed = (await client.get("/v1/sandboxes")).json()
assert all(s["id"] != sandbox_id for s in (listed.get("sandboxes") or []))
# 10. No corpus fixtures anywhere in this flow's payloads.
corpus_markers = ("lab-scenario", "proctor-scenario", "artifact-")
for payload in (var, grade, defense, finish, proctor):
assert not any(
m in json.dumps(payload) for m in corpus_markers
), "corpus fixture leaked into the learner path"
finally:
server.should_exit = True
with contextlib.suppress(Exception):
await asyncio.wait_for(serve_task, timeout=10.0)
+75 -36
View File
@@ -1,47 +1,86 @@
"""Lab feedback endpoint tests — SSE envelope with agent=lab (REQ-2-007)."""
"""Lab endpoint tests — LIVE trace contract (REQ-3-007).
import json
v0.3 re-grounding: POST /v1/lab/feedback takes {learner_id, task_id}; the
digest is computed from the learner's real TraceStore events. No corpus
scenarios; empty trace is a valid "no telemetry yet" coaching path.
"""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from pathlib import Path
import pytest
from fastapi.testclient import TestClient
from ai_service.config import Settings
from ai_service.grading.store import SQLiteGradeStore
from ai_service.llm.mock import MockProvider
from ai_service.main import create_app
from ai_service.telemetry.ingest import TraceIntegrityMap
from ai_service.telemetry.models import TelemetryEvent
from ai_service.telemetry.store import SQLiteTraceStore
T0 = datetime(2026, 9, 12, tzinfo=UTC)
def stream_events(client, payload) -> list[dict]:
with client.stream("POST", "/v1/lab/feedback", json=payload) as response:
assert response.status_code == 200
events = []
for line in response.iter_lines():
if line.startswith("data:"):
d = line.removeprefix("data:").strip()
if d == "[DONE]":
events.append({"type": "[DONE]"})
else:
events.append(json.loads(d))
return events
def _event(seq: int, kind: str, payload: dict, offset_s: float):
return TelemetryEvent(
learner_id="lab-learner",
task_id="lab-task",
seq=seq,
kind=kind,
payload=payload,
ts=T0 + timedelta(seconds=offset_s),
sandbox_id="sbx-lab",
)
def test_lab_feedback_streams_full_envelope(client):
events = stream_events(client, {"scenario_id": "lab-scenario-strong"})
assert events[0]["type"] == "meta"
assert events[0]["agent"] == "lab"
assert events[0]["scenario_id"] == "lab-scenario-strong"
deltas = [e for e in events if e["type"] == "delta"]
assert len(deltas) >= 1
assert any(e["type"] == "done" for e in events)
assert events[-1]["type"] == "[DONE]"
class StreamingMock(MockProvider):
"""Deterministic token stream for the SSE path."""
def _reply_for(self, messages, response_format):
return "Feedback grounded in your live session digest."
def test_unknown_scenario_404(client):
response = client.post("/v1/lab/feedback", json={"scenario_id": "nope"})
assert response.status_code == 404
assert "nope" in response.json()["detail"]
@pytest.fixture()
def client(tmp_path: Path) -> TestClient:
app = create_app(Settings(provider="mock"))
app.state.provider = StreamingMock()
app.state.trace_store = SQLiteTraceStore(db_path=tmp_path / "t.db")
app.state.grade_store = SQLiteGradeStore(db_path=tmp_path / "g.db")
app.state.trace_integrity = TraceIntegrityMap()
with TestClient(app) as c:
yield c
def test_distinct_scenarios_distinct_replies(client):
strong = stream_events(client, {"scenario_id": "lab-scenario-strong"})
struggling = stream_events(client, {"scenario_id": "lab-scenario-struggling"})
strong_text = "".join(e["content"] for e in strong if e["type"] == "delta")
struggling_text = "".join(e["content"] for e in struggling if e["type"] == "delta")
assert strong_text != struggling_text
def test_live_trace_streams_full_envelope(client: TestClient) -> None:
store: SQLiteTraceStore = client.app.state.trace_store
for e in [
_event(0, "file_diff", {"path": "x.py"}, 0),
_event(1, "command", {"cmd": "pytest -q"}, 10),
_event(2, "test_result", {"passed": False, "exit_code": 1}, 20),
_event(3, "test_result", {"passed": True, "exit_code": 0}, 40),
]:
store.append(e)
with client.stream(
"POST", "/v1/lab/feedback", json={"learner_id": "lab-learner", "task_id": "lab-task"}
) as resp:
assert resp.status_code == 200
body = "".join(chunk.decode() for chunk in resp.iter_raw())
assert '"agent": "lab"' in body or '"agent":"lab"' in body
assert '"task_id": "lab-task"' in body
assert '"type": "delta"' in body or '"type":"delta"' in body
def test_missing_scenario_id_422(client):
response = client.post("/v1/lab/feedback", json={})
assert response.status_code == 422
def test_empty_trace_coaches_the_baseline(client: TestClient) -> None:
"""No telemetry is NOT an error — Lab coaches 'run the starter test'."""
with client.stream(
"POST", "/v1/lab/feedback", json={"learner_id": "lab-learner", "task_id": "no-events"}
) as resp:
assert resp.status_code == 200
def test_missing_fields_422(client: TestClient) -> None:
resp = client.post("/v1/lab/feedback", json={"learner_id": "x"})
assert resp.status_code == 422
+105 -33
View File
@@ -1,49 +1,121 @@
"""Proctor signals endpoint tests — validated JSON, 404s (REQ-2-009)."""
"""Proctor signals endpoint tests — REAL inputs contract (REQ-3-007).
from ai_service.agents.proctor import ProctorAssessment
from ai_service.llm.mock import ScriptedJSONProvider
v0.3 re-grounding: POST /v1/proctor/signals takes {learner_id, task_id}
and gathers digest + defense signals + variant context server-side.
"""
from __future__ import annotations
import json
from datetime import UTC, datetime, timedelta
from pathlib import Path
import pytest
from fastapi.testclient import TestClient
from ai_service.config import Settings
from ai_service.grading.store import SQLiteGradeStore
from ai_service.llm.mock import MockProvider
from ai_service.main import create_app
from ai_service.telemetry.ingest import TraceIntegrityMap
from ai_service.telemetry.models import TelemetryEvent
from ai_service.telemetry.store import SQLiteTraceStore
from ai_service.variants.store import SQLiteVariantStore
from ai_service.voice.defense_store import SQLiteDefenseStore
VALID = {
"scenario_id": "proctor-scenario-distracted",
"signals": [
{"signal_type": "context_switch", "severity": "low",
"note": "Docs tab at t+120s is normal"},
{"signal_type": "idle_gap", "severity": "medium",
"note": "5-minute idle at t+300s"},
{"signal_type": "idle_gap", "severity": "low", "note": "One long pause."},
],
"intervention": "Offer a short break, then restate the plan",
"summary": "Coaching-shaped session note",
}
def test_signals_returns_validated_json(client):
original = client.app.state.provider
client.app.state.provider = ScriptedJSONProvider(VALID)
try:
response = client.post(
"/v1/proctor/signals", json={"scenario_id": "proctor-scenario-distracted"}
)
finally:
client.app.state.provider = original
assert response.status_code == 200
validated = ProctorAssessment.model_validate(response.json())
assert validated.scenario_id == "proctor-scenario-distracted"
assert validated.intervention
T0 = datetime(2026, 9, 12, tzinfo=UTC)
def test_unknown_scenario_404(client):
response = client.post("/v1/proctor/signals", json={"scenario_id": "ghost"})
assert response.status_code == 404
class ProctorJSON(MockProvider):
def _reply_for(self, messages, response_format):
if response_format is not None and response_format.get("type") == "json_object":
return json.dumps(VALID)
return super()._reply_for(messages, response_format)
def test_unparseable_provider_502(client):
response = client.post(
"/v1/proctor/signals", json={"scenario_id": "proctor-scenario-healthy"}
class BrokenJSON(MockProvider):
def _reply_for(self, messages, response_format):
if response_format is not None and response_format.get("type") == "json_object":
return "not json ever"
return super()._reply_for(messages, response_format)
@pytest.fixture()
def client(tmp_path: Path) -> TestClient:
app = create_app(Settings(provider="mock"))
app.state.trace_store = SQLiteTraceStore(db_path=tmp_path / "t.db")
app.state.grade_store = SQLiteGradeStore(db_path=tmp_path / "g.db")
app.state.variant_store = SQLiteVariantStore(db_path=tmp_path / "v.db")
app.state.defense_store = SQLiteDefenseStore(db_path=tmp_path / "d.db")
app.state.trace_integrity = TraceIntegrityMap()
with TestClient(app) as c:
yield c
def _seed_trace(client: TestClient) -> None:
store: SQLiteTraceStore = client.app.state.trace_store
for e in [
TelemetryEvent(
learner_id="p-learner",
task_id="p-task",
seq=0,
kind="activity",
payload={"state": "idle"},
ts=T0,
sandbox_id="sbx-p",
),
TelemetryEvent(
learner_id="p-learner",
task_id="p-task",
seq=1,
kind="activity",
payload={"state": "idle"},
ts=T0 + timedelta(seconds=400),
sandbox_id="sbx-p",
),
]:
store.append(e)
def test_signals_returns_validated_json(client: TestClient) -> None:
client.app.state.provider = ProctorJSON()
_seed_trace(client)
resp = client.post(
"/v1/proctor/signals", json={"learner_id": "p-learner", "task_id": "p-task"}
)
assert response.status_code == 502
assert "failed" in response.json()["detail"].lower()
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["intervention"]
assert body["signals"][0]["signal_type"] == "idle_gap"
def test_missing_scenario_id_422(client):
response = client.post("/v1/proctor/signals", json={})
assert response.status_code == 422
def test_empty_trace_is_valid_not_404(client: TestClient) -> None:
"""No telemetry → the proctor still assesses (nothing to flag)."""
client.app.state.provider = ProctorJSON()
resp = client.post(
"/v1/proctor/signals", json={"learner_id": "nobody", "task_id": "nothing"}
)
assert resp.status_code == 200
def test_unparseable_provider_502(client: TestClient) -> None:
client.app.state.provider = BrokenJSON()
_seed_trace(client)
resp = client.post(
"/v1/proctor/signals", json={"learner_id": "p-learner", "task_id": "p-task"}
)
assert resp.status_code == 502
def test_missing_fields_422(client: TestClient) -> None:
client.app.state.provider = ProctorJSON()
resp = client.post("/v1/proctor/signals", json={"learner_id": "x"})
assert resp.status_code == 422
@@ -359,3 +359,91 @@ def test_real_backend_create_path_runs(tmp_path: Path) -> None:
assert (sandbox_root / sandbox_id / "workspace").is_dir()
finally:
shutil.rmtree(sandbox_root, ignore_errors=True)
class TestFilesAndExecRoutes:
"""Workspace CRUD + Run/Test exec (Phase 6, REQ-3-008, CUT-2)."""
def test_file_write_read_list_roundtrip(self, client):
handle = client.post(
"/v1/sandboxes", json={"learner_id": "pilot-learner"}
).json()
sbx = handle["id"]
put = client.put(
f"/v1/sandboxes/{sbx}/files/main.py",
json={"path": "main.py", "content": "print('hi')"},
)
assert put.status_code == 200, put.text
got = client.get(f"/v1/sandboxes/{sbx}/files/main.py")
assert got.status_code == 200
assert "print('hi')" in got.json()["content"]
listed = client.get(f"/v1/sandboxes/{sbx}/files")
assert "main.py" in listed.json()["files"]
client.delete(f"/v1/sandboxes/{sbx}")
def test_traversal_rejected(self, client):
handle = client.post(
"/v1/sandboxes", json={"learner_id": "pilot-learner"}
).json()
sbx = handle["id"]
bad = client.put(
f"/v1/sandboxes/{sbx}/files/..%2Fescape.txt",
json={"path": "../escape.txt", "content": "x"},
)
assert bad.status_code == 422
client.delete(f"/v1/sandboxes/{sbx}")
def test_unknown_file_404(self, client):
handle = client.post(
"/v1/sandboxes", json={"learner_id": "pilot-learner"}
).json()
sbx = handle["id"]
assert client.get(f"/v1/sandboxes/{sbx}/files/ghost.py").status_code == 404
client.delete(f"/v1/sandboxes/{sbx}")
def test_exec_unknown_sandbox_404(self, client):
resp = client.post(
"/v1/sandboxes/sbx-nope/exec", json={"cmd": ["echo", "hi"]}
)
assert resp.status_code == 404
def test_unknown_sandbox_file_routes_404_not_500(self, client):
"""P7: read/write on an unknown sandbox must 404 (SandboxNotFoundError
previously escaped _workspace_dir as an unhandled 500)."""
assert (
client.get("/v1/sandboxes/sbx-nope/files/whatever.py").status_code == 404
)
put = client.put(
"/v1/sandboxes/sbx-nope/files/whatever.py",
json={"path": "whatever.py", "content": "x"},
)
assert put.status_code == 404
def test_symlink_escape_rejected(self, client):
"""P7: an exec-planted symlink in the workspace must not let the
file routes read/write OUTSIDE the bind (lexical traversal checks
cannot see symlinks resolve + containment re-check is the gate)."""
handle = client.post(
"/v1/sandboxes", json={"learner_id": "pilot-learner"}
).json()
sbx = handle["id"]
workspace = Path(handle["workdir"]) / "workspace"
outside = workspace.parent / "secret.txt"
outside.write_text("host secret") # a host file OUTSIDE the bind
try:
(workspace / "leak.txt").symlink_to(outside)
read = client.get(f"/v1/sandboxes/{sbx}/files/leak.txt")
assert read.status_code == 422, (
f"symlink escape read must 422, got {read.status_code}: {read.text}"
)
write = client.put(
f"/v1/sandboxes/{sbx}/files/leak.txt",
json={"path": "leak.txt", "content": "pwned"},
)
assert write.status_code == 422, (
f"symlink escape write must 422, got {write.status_code}: {write.text}"
)
assert outside.read_text() == "host secret" # untouched
finally:
client.delete(f"/v1/sandboxes/{sbx}")
outside.unlink(missing_ok=True)
@@ -239,6 +239,91 @@ def test_queue_overflow_also_floods(
assert app.state.trace_integrity.reason("L-q", "T-q") == "INCOMPLETE_FLOODED"
@pytest.mark.asyncio
async def test_queue_overflow_flood_session_task_terminates(tmp_path, monkeypatch):
"""P7 regression: the queue-overflow flood path must not LEAK the
session coroutine. v0.3's receiver returned from its QueueFull branch
without the disconnect sentinel, so the drainer parked on an empty queue
forever and IngestSession.run() never returned one leaked
(pinger+drainer) task-set per flooded trace, unbounded over a long-lived
process. A REAL uvicorn server (TestClient teardown hides the leak) is
stopped after the flood; the session tasks must be gone shortly after.
"""
import asyncio
import contextlib
import socket as socket_mod
import uvicorn
from ai_service.main import create_app
from ai_service.telemetry.ingest import TraceIntegrityMap
from ai_service.telemetry.store import SQLiteTraceStore
monkeypatch.setattr(ingest_mod, "INBOUND_QUEUE_MAX", 1)
store = SQLiteTraceStore(db_path=tmp_path / "leak.db")
app = create_app(
Settings(
provider="mock",
db_path=tmp_path / "leak.db",
sandbox_dir=tmp_path / "sandboxes",
)
)
app.state.trace_store = store
app.state.trace_integrity = TraceIntegrityMap()
with socket_mod.socket() as s:
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
server = uvicorn.Server(
uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning")
)
serve_task = asyncio.get_running_loop().create_task(server.serve())
leaked = True
try:
for _ in range(100):
if server.started:
break
await asyncio.sleep(0.1)
assert server.started
import websockets
uri = (
f"ws://127.0.0.1:{port}/v1/telemetry/ingest"
f"?learner_id=L-leak&task_id=T-leak"
)
async with websockets.connect(uri) as ws:
for seq in range(64): # bound=1 → guaranteed overflow
await ws.send(_frame(seq, sandbox_id=""))
# The flood close (1008) reaches the client.
try:
await asyncio.wait_for(ws.recv(), timeout=10.0)
await asyncio.wait_for(ws.recv(), timeout=10.0)
except (websockets.exceptions.ConnectionClosed, TimeoutError, OSError):
pass
assert app.state.trace_integrity.is_incomplete("L-leak", "T-leak")
# The session's run() must have returned: no lingering nc-* tasks
# holding the socket open. Poll briefly — teardown is async.
deadline = asyncio.get_running_loop().time() + 5.0
while asyncio.get_running_loop().time() < deadline:
names = {
t.get_name()
for t in asyncio.all_tasks()
if t is not asyncio.current_task()
}
if not any("ingest" in n.lower() for n in names):
leaked = False
break
await asyncio.sleep(0.1)
finally:
server.should_exit = True
with contextlib.suppress(Exception):
await asyncio.wait_for(serve_task, timeout=10.0)
store.close()
assert not leaked, "IngestSession task set leaked after queue-overflow flood"
def test_reconnect_after_flood_cannot_resurrect_trace(
client: TestClient, app, store: SQLiteTraceStore
) -> None:
@@ -309,6 +394,48 @@ def test_missing_identity_query_params_rejected_at_handshake(
assert excinfo.value.code == 1008
def test_browser_origin_rejected_in_explicit_list_mode(
settings, monkeypatch: pytest.MonkeyPatch
) -> None:
"""D-038: with an explicit AI_CORS_ORIGINS list, a page loaded in the
learner's browser (unlisted Origin) must not be able to open the ingest
socket and poison/flood the trace. The stdlib capture agent sends no
Origin and is unaffected (see the no-origin test below)."""
from fastapi.testclient import TestClient as TC
restricted = settings.model_copy(update={"cors_origins": "http://localhost:3000"})
app = create_app(restricted)
with TC(app) as c:
with pytest.raises(WebSocketDisconnect) as excinfo:
with c.websocket_connect(
_ingest_url(), headers={"Origin": "https://evil.example"}
):
pass
assert excinfo.value.code == 1008
def test_wildcard_mode_admits_any_browser_origin(client: TestClient) -> None:
"""D-038 default ('*'): remote-browser origins open the ingest socket —
the remote build surface streams telemetry from the learner's browser."""
with client.websocket_connect(
_ingest_url(), headers={"Origin": "http://nextcraft-1:3000"}
) as ws:
ws.send_text(_frame(0))
def test_dev_origin_and_no_origin_both_allowed(client: TestClient) -> None:
"""The same-origin dev page (Next.js :3000) opens fine, and so does the
capture-agent path (no Origin header at all)."""
for headers in ({"Origin": "http://localhost:3000"}, {}):
with client.websocket_connect(_ingest_url(), headers=headers) as ws:
ws.send_text(_frame(0))
body = client.get(f"/v1/telemetry/traces/{LEARNER}/{TASK}").json()
assert [e["seq"] for e in body["events"]] == [0]
# Unique trace per iteration would collide on (LEARNER, TASK) PK —
# seq 0 re-sent is deduped, so one row is the invariant either way.
assert len(body["events"]) == 1
# -- keepalive ---------------------------------------------------------------------
@@ -499,7 +499,22 @@ class TestReconnectFlush:
test_agent = _make_agent(tmp_path, held_link.url)
test_agent.start()
assert test_agent.wait_connected(5)
test_agent.run_command("echo first")
first = test_agent.run_command("echo first")
# P7 de-flake: wait for the pre-kill burst to be OBSERVED at the
# server before severing (the sibling TestSpoolOnDisconnect test
# already had this discipline). Killing mid-burst exercises a
# DIFFERENT, documented limitation — the agent's one-line replay
# margin cannot cover a multi-frame TCP in-flight window (an
# ACK-protocol gap tracked for v0.4) — which made this test
# nondeterministic under load instead of testing what its name
# says: the reconnect flush of OFFLINE-spooled events.
assert _wait_until(
lambda: any(
e["kind"] == "run_result" and e["seq"] == first["seq"]
for e in fake_server.events
),
timeout_s=10.0,
), "pre-kill burst never reached the server"
held_link.kill() # outage begins: no traffic, no reconnect possible
assert test_agent.wait_disconnected(5)
@@ -129,6 +129,21 @@ def test_latest_seq(store: SQLiteTraceStore) -> None:
assert store.latest_seq("learner-1", "task-2") == -1
def test_count_is_durable_row_count_not_latest_seq(store: SQLiteTraceStore) -> None:
"""count() backs the ingest flood cap (P7): it must reflect stored ROWS
(a skipped-ahead seq must not burn un-sent budget) and stay O(1)-ish
(COUNT(*), never materialize the trace per append)."""
assert store.count("learner-1", "task-1") == 0
store.append(make_event(0))
store.append(make_event(2)) # skipped 1 — count is rows, not latest+1
assert store.count("learner-1", "task-1") == 2
# Dedup retries do not inflate the count (at-least-once contract).
store.append(make_event(2))
assert store.count("learner-1", "task-1") == 2
# Scoped to the trace pair.
assert store.count("learner-1", "task-2") == 0
def test_list_tasks(store: SQLiteTraceStore) -> None:
assert store.list_tasks("learner-1") == []
@@ -0,0 +1,79 @@
"""Corpus dormancy verification (Task 6-1-04, REQ-3-007).
The v0.2 mock engine inputs (`corpus/telemetry.py`, `corpus/artifacts.py`)
must have ZERO production importers after the v0.3 re-grounding: Lab/
Assessor/Proctor run on real engine inputs with no mock fallback in the
learner path. The files stay on disk (Phase-3 calibration history) but are
not imported by any production module. `learner_context` remains ACTIVE
(agents still need learner context). Test-only references (e.g.
`corpus/trace_fixtures.py` in grading calibration tests) are allowed.
"""
from __future__ import annotations
import ast
from pathlib import Path
REPO = Path(__file__).parents[1]
#: Production trees whose imports of dormant corpus modules are forbidden.
PRODUCTION_PATHS = [
REPO / "ai_service" / "agents",
REPO / "ai_service" / "api",
REPO / "ai_service" / "grading",
REPO / "ai_service" / "telemetry",
REPO / "ai_service" / "variants",
REPO / "ai_service" / "voice",
REPO / "ai_service" / "sandbox",
REPO / "ai_service" / "llm",
REPO / "ai_service" / "main.py",
]
DORMANT_MODULES = ("corpus.telemetry", "corpus.artifacts")
def _module_targets(node: ast.AST, *, level: int, module: str | None) -> set[str]:
"""Resolve relative + absolute import targets to dotted ai_service paths."""
targets: set[str] = set()
if module and ("corpus" in module):
targets.add(module)
return targets
def test_no_production_imports_of_dormant_corpus() -> None:
violators: list[str] = []
for path in PRODUCTION_PATHS:
files = [path] if path.suffix == ".py" else sorted(path.rglob("*.py"))
for py in files:
tree = ast.parse(py.read_text())
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom):
module = node.module or ""
level = node.level
if level: # relative: resolve against corpus
if module.endswith("telemetry") and "corpus" in module:
violators.append(f"{py}: {module}")
if module.endswith("artifacts") and "corpus" in module:
violators.append(f"{py}: {module}")
# bare `from . import telemetry` inside corpus/ itself is fine
else:
for target in _module_targets(node, level=level, module=module):
violators.append(f"{py}: {target}")
elif isinstance(node, ast.Import):
for alias in node.names:
if any(alias.name.startswith(m) for m in DORMANT_MODULES):
violators.append(f"{py}: {alias.name}")
assert not violators, f"dormant corpus imports in production: {violators}"
def test_learner_context_stays_active() -> None:
"""Learner context corpus is NOT dormant — agents still use it."""
agents_lab = (REPO / "ai_service" / "agents" / "lab.py").read_text()
assert "corpus.learner_context" in agents_lab
(REPO / "ai_service" / "corpus" / "learner_context.py").exists()
def test_dormancy_headers_present() -> None:
for fname in ("telemetry.py", "artifacts.py"):
src = (REPO / "ai_service" / "corpus" / fname).read_text()
assert "DORMANT" in src, f"{fname} missing dormancy header"
+48
View File
@@ -0,0 +1,48 @@
# @nextcraft/cli — nextcraft
The bootstrap CLI for the Nextcraft monorepo, shipped as a self-contained linux x64 binary (Node SEA) on every release.
## Commands
See the [root README quickstart](../../README.md) for the user-facing flow. Internals:
- `src/index.ts` — argv dispatch, exit-code contract (0 ok / 1 failure / 2 usage), direct-run guard (`argv[0] === argv[1]` detects SEA context — the installer renames the binary, so filename matching is unreliable)
- `src/commands/` — doctor / bootstrap / verify / dev; all orchestration delegates to `apps/ai-service/scripts/*.sh` via `src/lib/spawn.ts` (array-args only, SIGTERM→SIGKILL timeout ladder)
- `src/checks/` — pure logic: version compare, `.env` template diff
- `tests/` — node:test suites: dispatch, checks, spawn, command stubs, real-box doctor integration, install.sh fixture-server E2E (tamper rejection, degradation), release-assets token isolation, fresh-clone E2E
## Build
```sh
pnpm cli:typecheck # tsc --noEmit
pnpm cli:test # node:test suites
pnpm cli:build # tsc -p tsconfig.build.json -> dist/
pnpm --filter @nextcraft/cli build:binary <tag> # SEA binary + sha256 sidecar
```
`build:binary <tag>`: esbuild bundle (CJS, node18 target, version stamped via `NEXTCRAFT_VERSION_STAMP` define — `--version` reports the tag it was built as) → `node --experimental-sea-config` → postject injection into a copy of the system node binary → `dist/nextcraft-linux-x64` + `dist/nextcraft-linux-x64.sha256`. The binary runs without node on PATH (runtime embedded, ~117 MB).
## Release pipeline
Every ship from v0.3.2 onward runs `scripts/release-assets.sh <tag>` after tag+merge:
1. Builds the binary stamped with the tag
2. Resolves `GITEA_TOKEN` from `.env*` files ONLY (`.ciagent/.env.secrets` first) — never from shell env
3. Attaches `nextcraft-linux-x64` + `nextcraft-linux-x64.sha256` to the Gitea release (bounded retry, best-effort — never blocks the ship)
`scripts/install.sh` (POSIX sh, dash-safe): platform gate → Gitea latest-release API resolve → exact-name asset match → sha256 verify BEFORE install (mismatch = hard stop) → `~/.local/bin` install → PATH hint. Any failure degrades to printed source-bootstrap instructions.
## Secrets policy
The CLI never generates, writes, or echoes secrets. `bootstrap` copies `.env.example``.env` only when absent and warns on missing optional keys (mock providers keep the stack runnable keyless). Real keys live only in gitignored `.ciagent/.env.secrets`, exported by `apps/ai-service/scripts/dev.sh`.
## Troubleshooting
| Symptom | Cause / fix |
|---------|-------------|
| `pnpm not found` in doctor | `corepack enable pnpm` (installs to ~/.local/bin — ensure PATH includes it) |
| doctor passes but verify fails on venv | re-run `nextcraft bootstrap` (venv/pip resolution is idempotent) |
| `port 8420 busy` in verify | stop the process on :8420 (`kill $(lsof -t -i:8420)`) or set `AI_PORT` |
| install.sh says "no binary assets yet" | release predates the binary pipeline (pre-v0.3.2); use source bootstrap |
| Binary silent after rename | fixed since v0.3.2 (SEA argv detection); re-download the latest release |
| Checksum mismatch on install | do NOT run the download; delete it and retry — report if it persists |
+22
View File
@@ -0,0 +1,22 @@
{
"name": "@nextcraft/cli",
"version": "0.0.0",
"private": true,
"type": "module",
"bin": {
"nextcraft": "dist/index.js"
},
"scripts": {
"dev": "tsx src/index.ts",
"test": "tsx --test tests/*.test.ts",
"typecheck": "tsc --noEmit",
"build": "tsc -p tsconfig.build.json",
"build:binary": "node scripts/build-binary.mjs"
},
"devDependencies": {
"@types/node": "^24.0.0",
"tsx": "^4.23.0",
"typescript": "^5.7.2",
"esbuild": "0.28.2"
}
}
+63
View File
@@ -0,0 +1,63 @@
#!/usr/bin/env node
import { execFileSync } from "node:child_process";
import { copyFileSync, existsSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
import { createHash } from "node:crypto";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
const pkgDir = dirname(fileURLToPath(import.meta.url)) + "/..";
const dist = join(pkgDir, "dist");
const bundle = join(dist, "bundle.cjs");
const blob = join(dist, "sea-prep.blob");
const config = join(dist, "sea-config.json");
const out = join(dist, "nextcraft-linux-x64");
const checksum = out + ".sha256";
const version = process.argv[2] ?? "0.0.0-dev";
if (version !== "0.0.0-dev" && !/^v?\d/.test(version)) {
console.error(`refusing to stamp implausible version: ${version}`);
process.exit(1);
}
rmSync(dist, { recursive: true, force: true });
execFileSync(
join(pkgDir, "node_modules/.bin/esbuild"),
[
join(pkgDir, "src/index.ts"),
"--bundle",
"--platform=node",
"--format=cjs",
"--target=node18",
`--define:NEXTCRAFT_VERSION_STAMP=${JSON.stringify(version)}`,
"--outfile=" + bundle,
],
{ stdio: "inherit" },
);
const seaConfig = {
main: bundle,
output: blob,
disableExperimentalSEAWarning: true,
};
writeFileSync(config, JSON.stringify(seaConfig));
execFileSync(process.execPath, ["--experimental-sea-config", config], { stdio: "inherit" });
const nodeBin = process.execPath;
copyFileSync(nodeBin, out);
execFileSync(
"npx",
["--yes", "postject", out, "NODE_SEA_BLOB", blob, "--sentinel-fuse", "NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2"],
{ stdio: "inherit" },
);
execFileSync("chmod", ["+x", out]);
const size = statSync(out).size;
const hash = createHash("sha256").update(readFileSync(out)).digest("hex");
writeFileSync(checksum, `${hash} nextcraft-linux-x64\n`);
console.log(`built ${out} (${(size / 1024 / 1024).toFixed(1)} MB) stamped ${version}`);
console.log(`checksum ${checksum}: ${hash}`);
if (!existsSync(out) || !existsSync(checksum)) {
console.error("expected artifacts missing");
process.exit(1);
}
+26
View File
@@ -0,0 +1,26 @@
export function compareVersions(a: string, b: string): number {
const pa = parse(a);
const pb = parse(b);
for (let i = 0; i < 2; i++) {
if (pa[i] > pb[i]) return 1;
if (pa[i] < pb[i]) return -1;
}
return 0;
}
function parse(v: string): [number, number] {
const clean = v.trim().replace(/^v/i, "");
const dotted = clean.match(/(\d+)\.(\d+)/);
if (dotted) return [parseInt(dotted[1], 10), parseInt(dotted[2], 10)];
const bare = clean.match(/^(\d+)(?:\.(\d+))?/);
if (!bare) return [0, 0];
return [parseInt(bare[1], 10), parseInt(bare[2] ?? "0", 10)];
}
export interface CommandCheckResult {
name: string;
ok: boolean;
found: boolean;
version?: string;
hint?: string;
}
+21
View File
@@ -0,0 +1,21 @@
export interface EnvDiff {
missing: string[];
extra: string[];
}
export function parseEnvKeys(content: string): string[] {
const keys: string[] = [];
for (const line of content.split("\n")) {
const m = line.match(/^\s*([A-Z_][A-Z0-9_]*)\s*=/);
if (m) keys.push(m[1]);
}
return keys;
}
export function diffEnvTemplate(templateContent: string, envContent: string): EnvDiff {
const template = new Set(parseEnvKeys(templateContent));
const env = new Set(parseEnvKeys(envContent));
const missing = [...template].filter((k) => !env.has(k)).sort();
const extra = [...env].filter((k) => !template.has(k)).sort();
return { missing, extra };
}
+79
View File
@@ -0,0 +1,79 @@
import { join } from "node:path";
import type { Ctx } from "../ctx.js";
import { findRepoRoot } from "../lib/repo.js";
import { diffEnvTemplate } from "../checks/check-env.js";
import { hr, info, warn } from "../lib/log.js";
import { doctor } from "./doctor.js";
const INSTALL_TIMEOUT_MS = 600_000;
export async function bootstrap(_args: string[], ctx: Ctx): Promise<number> {
const root = findRepoRoot(ctx.cwd);
if (!root) {
ctx.stderr.write("\u2717 not inside a nextcraft clone (no pnpm-workspace.yaml found upwards)\n");
ctx.stderr.write(" hint: run from the repo, or clone first:\n");
ctx.stderr.write(" git clone https://git.coreci.dev/coreci/nextcraft.git && cd nextcraft\n");
return 1;
}
const aiDir = join(root, "apps/ai-service");
hr("nextcraft bootstrap — monorepo setup", ctx);
info("preflight: checking prerequisites (doctor)...", ctx);
const preflight = await doctor([], ctx);
if (preflight !== 0) {
ctx.stderr.write("\n\u2717 preflight failed — fix the failed checks above, then re-run nextcraft bootstrap\n");
return 1;
}
info("installing workspace dependencies (pnpm install)...", ctx);
const install = await ctx.spawn("pnpm", ["install"], { cwd: root, timeoutMs: INSTALL_TIMEOUT_MS });
if (install.code !== 0) {
ctx.stderr.write(`\u2717 pnpm install failed (exit ${install.code})\n`);
return 1;
}
info("workspace dependencies installed", ctx);
info("bootstrapping ai-service venv (scripts/bootstrap.sh)...", ctx);
const boot = await ctx.spawn("bash", ["scripts/bootstrap.sh"], {
cwd: aiDir,
timeoutMs: INSTALL_TIMEOUT_MS,
});
if (boot.code !== 0) {
ctx.stderr.write(`\u2717 ai-service bootstrap failed (exit ${boot.code})\n`);
return 1;
}
info("ai-service venv ready", ctx);
const examplePath = join(aiDir, ".env.example");
const envPath = join(aiDir, ".env");
if (!ctx.exists(envPath) && ctx.exists(examplePath)) {
ctx.writeFile(envPath, ctx.readFile(examplePath) ?? "");
info("created apps/ai-service/.env from .env.example", ctx);
} else if (ctx.exists(envPath)) {
info("apps/ai-service/.env already present — kept as-is", ctx);
} else {
warn("no .env.example found — skipping env setup (pydantic-settings defaults apply)", ctx);
}
if (ctx.exists(examplePath) && ctx.exists(envPath)) {
const diff = diffEnvTemplate(ctx.readFile(examplePath) ?? "", ctx.readFile(envPath) ?? "");
if (diff.missing.length > 0) {
warn(
`${diff.missing.length} optional key(s) unset in .env: ${diff.missing.join(", ")}`,
ctx,
);
info("optional keys warn only — mock providers keep the stack runnable without them", ctx);
}
if (diff.extra.length > 0) {
info(`extra keys in .env (kept): ${diff.extra.join(", ")}`, ctx);
}
if (diff.missing.length === 0 && diff.extra.length === 0) {
info(".env covers all template keys", ctx);
}
}
ctx.stdout.write("\n\u2713 bootstrap complete\n\nNext steps:\n nextcraft verify\n nextcraft dev\n\n");
return 0;
}
+23
View File
@@ -0,0 +1,23 @@
import { spawn } from "node:child_process";
import { join } from "node:path";
import type { Ctx } from "../ctx.js";
import { findRepoRoot } from "../lib/repo.js";
export async function dev(_args: string[], ctx: Ctx): Promise<number> {
const root = findRepoRoot(ctx.cwd);
if (!root) {
ctx.stderr.write("\u2717 not inside a nextcraft clone — run from the repo root or a subdirectory\n");
return 1;
}
ctx.stdout.write("nextcraft dev — ai-service dev server (web dev server: run `pnpm dev` separately)\n\n");
const child = spawn("bash", ["scripts/dev.sh"], {
cwd: join(root, "apps/ai-service"),
stdio: "inherit",
});
const forward = (sig: NodeJS.Signals) => () => child.kill(sig);
process.on("SIGINT", forward("SIGINT"));
process.on("SIGTERM", forward("SIGTERM"));
return await new Promise<number>((resolve) => {
child.on("close", (code) => resolve(code ?? 1));
});
}
+120
View File
@@ -0,0 +1,120 @@
import type { Ctx } from "../ctx.js";
import { compareVersions } from "../checks/check-command.js";
import { ok, fail, hr, summary } from "../lib/log.js";
import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
interface CheckOutcome {
name: string;
passed: boolean;
}
export async function doctor(_args: string[], ctx: Ctx): Promise<number> {
hr("nextcraft doctor — environment prerequisites", ctx);
const results: CheckOutcome[] = [];
results.push(await checkNode(ctx));
results.push(await checkProgram(ctx, "pnpm", "8", "install pnpm via corepack: corepack enable pnpm (or: npm i -g pnpm)"));
results.push(await checkProgram(ctx, "python3", "3.11", "install python3 >= 3.11 (e.g. apt install python3 python3-venv)"));
results.push(await checkProgram(ctx, "git", undefined, "install git: https://git-scm.com/download/linux"));
results.push(await checkUnshare(ctx));
results.push(await checkVenvCapability(ctx));
const passed = results.filter((r) => r.passed).length;
const failed = results.length - passed;
summary(passed, failed, ctx);
return failed === 0 ? 0 : 1;
}
async function checkVenvCapability(ctx: Ctx): Promise<CheckOutcome> {
const probeDir = mkdtempSync(join(tmpdir(), "nc-venv-probe-"));
try {
const full = await ctx.spawn("python3", ["-m", "venv", join(probeDir, "v")], {
capture: true,
timeoutMs: 60_000,
});
if (full.code === 0) return venvOk(ctx, "full venv");
const pipless = await ctx.spawn("python3", ["-m", "venv", "--without-pip", join(probeDir, "v2")], {
capture: true,
timeoutMs: 60_000,
});
if (pipless.code === 0) {
ok("python3 venv capability (fallback path: pip bootstrapped via get-pip)", ctx);
return { name: "venv", passed: true };
}
return venvFail(ctx);
} finally {
await ctx.spawn("rm", ["-rf", probeDir], { capture: true, timeoutMs: 30_000 });
}
}
function venvOk(ctx: Ctx, mode: string): CheckOutcome {
ok(`python3 venv capability (${mode})`, ctx);
return { name: "venv", passed: true };
}
function venvFail(ctx: Ctx): CheckOutcome {
fail(
"python3 cannot create virtual environments (python3 -m venv fails)",
"Debian/Ubuntu: apt install python<version>-venv (e.g. python3.12-venv for python 3.12) — nextcraft bootstrap needs it; then re-run nextcraft doctor",
ctx,
);
return { name: "venv", passed: false };
}
async function checkNode(ctx: Ctx): Promise<CheckOutcome> {
const version = process.version;
if (compareVersions(version, "18") >= 0) {
ok(`node ${version} (>= 18)`, ctx);
return { name: "node", passed: true };
}
fail(`node ${version} is older than 18`, "install node >= 18 (https://nodejs.org)", ctx);
return { name: "node", passed: false };
}
async function checkProgram(
ctx: Ctx,
name: string,
minVersion: string | undefined,
hint: string,
): Promise<CheckOutcome> {
const which = await ctx.spawn("which", [name], { capture: true, timeoutMs: 3000 });
if (which.code !== 0) {
fail(`${name} not found on PATH`, hint, ctx);
return { name, passed: false };
}
let version: string | undefined;
if (minVersion) {
const probe = await ctx.spawn(name, ["--version"], { capture: true, timeoutMs: 10000 });
version = probe.stdout.trim().split("\n")[0]?.trim();
if (probe.code !== 0 || !version || compareVersions(version, minVersion) < 0) {
fail(
`${name} ${version ?? "(unknown version)"} is older than required ${minVersion}`,
hint,
ctx,
);
return { name, passed: false };
}
ok(`${name} ${version} (>= ${minVersion})`, ctx);
return { name, passed: true };
}
const probe = await ctx.spawn(name, ["--version"], { capture: true, timeoutMs: 10000 });
version = probe.stdout.trim().split("\n")[0]?.trim();
ok(`${name} ${version ?? ""}`.trim(), ctx);
return { name, passed: true };
}
async function checkUnshare(ctx: Ctx): Promise<CheckOutcome> {
const which = await ctx.spawn("which", ["unshare"], { capture: true, timeoutMs: 3000 });
if (which.code === 0) {
ok("unshare available (sandbox fabric ready)", ctx);
return { name: "unshare", passed: true };
}
fail(
"unshare not found on PATH",
"sandbox fabric needs unshare (util-linux) — credential builds degrade without it: apt install util-linux",
ctx,
);
return { name: "unshare", passed: false };
}
+32
View File
@@ -0,0 +1,32 @@
declare const NEXTCRAFT_VERSION_STAMP: string | undefined;
export function version(env?: Record<string, string | undefined>): string {
if (typeof NEXTCRAFT_VERSION_STAMP !== "undefined") {
return NEXTCRAFT_VERSION_STAMP;
}
return env?.NEXTCRAFT_VERSION ?? "0.0.0-dev";
}
export function helpText(env?: Record<string, string | undefined>): string {
return `nextcraft ${version(env)} — bootstrap CLI for the Nextcraft monorepo
Usage:
nextcraft <command> [flags]
Commands:
doctor check environment prerequisites (node, pnpm, python3, git, unshare)
bootstrap set up a fresh clone: pnpm install, ai-service venv, .env from template
verify health-check the bootstrapped stack (venv, uvicorn, ports, env)
dev run the ai-service dev server (thin passthrough to scripts/dev.sh)
Flags:
--help, -h show this help
--version print the CLI version
Exit codes:
0 success
1 a check or step failed (see the printed hint)
2 usage error (unknown command or flag)
`;
}
+83
View File
@@ -0,0 +1,83 @@
import { join } from "node:path";
import type { Ctx } from "../ctx.js";
import { findRepoRoot } from "../lib/repo.js";
import { hr, ok, fail, warn, summary } from "../lib/log.js";
export async function verify(_args: string[], ctx: Ctx): Promise<number> {
const root = findRepoRoot(ctx.cwd);
if (!root) {
ctx.stderr.write("\u2717 not inside a nextcraft clone — run from the repo root or a subdirectory\n");
return 1;
}
const aiDir = join(root, "apps/ai-service");
hr("nextcraft verify — stack health check", ctx);
let passed = 0;
let failed = 0;
const venvPython = join(aiDir, ".venv/bin/python3");
if (ctx.exists(venvPython)) {
const imp = await ctx.spawn(venvPython, ["-c", "import ai_service"], {
capture: true,
timeoutMs: 30_000,
});
if (imp.code === 0) {
ok("ai-service venv — import ai_service", ctx);
passed++;
} else {
fail("ai_service no longer imports in the venv", "re-run nextcraft bootstrap", ctx);
failed++;
}
const uv = await ctx.spawn(venvPython, ["-c", "import uvicorn"], {
capture: true,
timeoutMs: 30_000,
});
if (uv.code === 0) {
ok("uvicorn importable in venv", ctx);
passed++;
} else {
fail("uvicorn missing in venv", "re-run nextcraft bootstrap", ctx);
failed++;
}
} else {
fail("ai-service venv not found", "run nextcraft bootstrap", ctx);
failed += 2;
}
const envPath = join(aiDir, ".env");
if (ctx.exists(envPath)) {
ok(".env present", ctx);
passed++;
} else {
warn(".env absent — pydantic-settings defaults apply (copy apps/ai-service/.env.example to customize)", ctx);
}
const port = parsePort(ctx.readFile(envPath) ?? "") ?? 8420;
if (await ctx.portFree(port, "127.0.0.1")) {
ok(`port ${port} free (ai-service will bind it)`, ctx);
passed++;
} else {
fail(`port ${port} is busy`, `stop the process listening on ${port} (ai-service default)`, ctx);
failed++;
}
if (ctx.exists(join(root, "node_modules/.bin/turbo"))) {
ok("workspace dependencies installed (node_modules present)", ctx);
passed++;
} else {
fail("node_modules missing at repo root", "run nextcraft bootstrap", ctx);
failed++;
}
summary(passed, failed, ctx);
return failed === 0 ? 0 : 1;
}
function parsePort(envContent: string): number | undefined {
for (const line of envContent.split("\n")) {
const m = line.match(/^\s*AI_PORT\s*=\s*(\d+)\s*$/);
if (m) return parseInt(m[1], 10);
}
return undefined;
}
+23
View File
@@ -0,0 +1,23 @@
import type { SpawnResult } from "./lib/spawn.js";
export interface Ctx {
cwd: string;
env: Record<string, string | undefined>;
stdout: { write(s: string): void };
stderr: { write(s: string): void };
spawn: (cmd: string, args: string[], opts: SpawnOpts) => Promise<SpawnResult>;
exists: (p: string) => boolean;
readFile: (p: string) => string | undefined;
writeFile: (p: string, content: string) => void;
portFree: (port: number, host: string) => Promise<boolean>;
}
export interface SpawnOpts {
timeoutMs?: number;
cwd?: string;
env?: Record<string, string | undefined>;
stdio?: "inherit" | "pipe";
capture?: boolean;
}
export type Command = (args: string[], ctx: Ctx) => Promise<number>;
+80
View File
@@ -0,0 +1,80 @@
import type { Ctx, Command } from "./ctx.js";
import { helpText, version } from "./commands/help.js";
import { doctor } from "./commands/doctor.js";
import { bootstrap } from "./commands/bootstrap.js";
import { verify } from "./commands/verify.js";
import { dev } from "./commands/dev.js";
import { existsSync, readFileSync, writeFileSync } from "node:fs";
import net from "node:net";
interface NodeRequire {
(id: string): unknown;
}
const commands: Record<string, Command> = { doctor, bootstrap, verify, dev };
export async function run(argv: string[], ctx: Ctx): Promise<number> {
const flags = argv.filter((a) => a.startsWith("--") || a === "-h");
const positional = argv.filter((a) => !(a.startsWith("--") || a === "-h"));
for (const flag of flags) {
if (flag === "--help" || flag === "-h") {
ctx.stdout.write(helpText(ctx.env));
return 0;
}
if (flag === "--version") {
ctx.stdout.write(`${version(ctx.env)}\n`);
return 0;
}
ctx.stderr.write(`unknown flag: ${flag}\n\n${helpText(ctx.env)}`);
return 2;
}
const name = positional[0];
const command = name ? commands[name] : undefined;
if (!command) {
const text = helpText(ctx.env);
ctx.stderr.write(name ? `unknown command: ${name}\n\n${text}` : text);
return 2;
}
return await command(positional.slice(1), ctx);
}
declare const require: NodeRequire;
function isDirectRun(): boolean {
if (process.env.NODE_TEST_CONTEXT) return false;
const argv1 = process.argv[1];
if (argv1?.endsWith("dist/index.js") || argv1?.endsWith("src/index.ts")) return true;
try {
const sea = require("node:sea") as { isSea?: () => boolean };
return typeof sea.isSea === "function" && sea.isSea();
} catch {
return false;
}
}
if (isDirectRun()) {
void (async () => {
const realCtx: Ctx = {
cwd: process.cwd(),
env: process.env,
stdout: process.stdout,
stderr: process.stderr,
spawn: async (cmd, args, opts) => {
const { run: spawnRun } = await import("./lib/spawn.js");
return spawnRun(cmd, args, opts);
},
exists: existsSync,
readFile: (p) => (existsSync(p) ? readFileSync(p, "utf8") : undefined),
writeFile: (p, c) => writeFileSync(p, c),
portFree: async (port, host) =>
new Promise<boolean>((resolve) => {
const srv = net.createServer();
srv.once("error", () => resolve(false));
srv.once("listening", () => srv.close(() => resolve(true)));
srv.listen(port, host);
}),
};
process.exitCode = await run(process.argv.slice(2), realCtx);
})();
}
+27
View File
@@ -0,0 +1,27 @@
import type { Ctx } from "../ctx.js";
export function ok(msg: string, ctx: Ctx): void {
ctx.stdout.write(` \u2713 ${msg}\n`);
}
export function fail(msg: string, hint: string | undefined, ctx: Ctx): void {
ctx.stderr.write(` \u2717 ${msg}\n`);
if (hint) ctx.stderr.write(` hint: ${hint}\n`);
}
export function warn(msg: string, ctx: Ctx): void {
ctx.stdout.write(` \u26a0 ${msg}\n`);
}
export function info(msg: string, ctx: Ctx): void {
ctx.stdout.write(` - ${msg}\n`);
}
export function hr(title: string, ctx: Ctx): void {
ctx.stdout.write(`\n${title}\n\n`);
}
export function summary(passed: number, failed: number, ctx: Ctx): void {
const line = failed === 0 ? "All checks passed" : `${failed} check(s) failed, ${passed} passed`;
ctx.stdout.write(`\n${failed === 0 ? "\u2713" : "\u2717"} ${line}\n`);
}
+15
View File
@@ -0,0 +1,15 @@
import { existsSync } from "node:fs";
import { dirname, join, resolve } from "node:path";
import type { SpawnResult } from "../lib/spawn.js";
export function findRepoRoot(start: string): string | undefined {
let dir = resolve(start);
for (;;) {
if (existsSync(join(dir, "pnpm-workspace.yaml"))) return dir;
const parent = dirname(dir);
if (parent === dir) return undefined;
dir = parent;
}
}
export type Spawn = (cmd: string, args: string[], opts: Record<string, unknown>) => Promise<SpawnResult>;
+51
View File
@@ -0,0 +1,51 @@
import { spawn } from "node:child_process";
import type { SpawnOpts } from "../ctx.js";
export interface SpawnResult {
code: number;
signal: NodeJS.Signals | null;
timedOut: boolean;
stdout: string;
stderr: string;
}
export async function run(
cmd: string,
args: string[],
opts: SpawnOpts = {},
): Promise<SpawnResult> {
return await new Promise((resolve) => {
const child = spawn(cmd, args, {
cwd: opts.cwd,
env: opts.env as NodeJS.ProcessEnv | undefined,
stdio: opts.capture ? ["ignore", "pipe", "pipe"] : "inherit",
});
let stdout = "";
let stderr = "";
if (child.stdout) child.stdout.on("data", (d: Buffer) => (stdout += d.toString()));
if (child.stderr) child.stderr.on("data", (d: Buffer) => (stderr += d.toString()));
let timedOut = false;
let killTimer: NodeJS.Timeout | undefined;
if (opts.timeoutMs) {
killTimer = setTimeout(() => {
timedOut = true;
child.kill("SIGTERM");
killTimer = setTimeout(() => child.kill("SIGKILL"), 1000);
}, opts.timeoutMs);
}
child.on("error", () => {
if (killTimer) clearTimeout(killTimer);
resolve({ code: 127, signal: null, timedOut: false, stdout, stderr });
});
child.on("close", (code, signal) => {
if (killTimer) clearTimeout(killTimer);
resolve({
code: code ?? 127,
signal,
timedOut,
stdout,
stderr,
});
});
});
}
+54
View File
@@ -0,0 +1,54 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { compareVersions } from "../src/checks/check-command.ts";
import { parseEnvKeys, diffEnvTemplate } from "../src/checks/check-env.ts";
test("compareVersions: equal across patch omission", () => {
assert.equal(compareVersions("18.0.0", "18"), 0);
assert.equal(compareVersions("18", "18.0.0"), 0);
});
test("compareVersions: minor boundary", () => {
assert.equal(compareVersions("3.11", "3.11.2"), 0);
assert.equal(compareVersions("3.10", "3.11"), -1);
assert.equal(compareVersions("3.12", "3.11"), 1);
});
test("compareVersions: v-prefix and embedded version strings", () => {
assert.ok(compareVersions("v18.2.0", "18") >= 0);
assert.equal(compareVersions("Python 3.11.2", "3.11"), 0);
assert.ok(compareVersions("python3 (3.9)", "3.11") < 0);
});
test("compareVersions: major win beats minor", () => {
assert.equal(compareVersions("24.0.0", "18.99"), 1);
assert.equal(compareVersions("2.99", "18.0"), -1);
});
test("parseEnvKeys: skips comments and blanks", () => {
const content = [
"# comment",
"",
"AI_PORT=8420",
" AI_MODEL=gemma4:31b",
"#AI_SKIP=1",
"AI_OLLAMA_CLOUD_API_KEY=",
].join("\n");
assert.deepEqual(parseEnvKeys(content), ["AI_PORT", "AI_MODEL", "AI_OLLAMA_CLOUD_API_KEY"]);
});
test("diffEnvTemplate: missing + extra classification", () => {
const template = "A=1\nB=2\nC=3\n";
const env = "B=2\nD=4\n";
const diff = diffEnvTemplate(template, env);
assert.deepEqual(diff.missing, ["A", "C"]);
assert.deepEqual(diff.extra, ["D"]);
});
test("diffEnvTemplate: full coverage yields empty diff", () => {
const template = "A=1\nB=2\n";
const env = "A=x\nB=y\n";
const diff = diffEnvTemplate(template, env);
assert.deepEqual(diff.missing, []);
assert.deepEqual(diff.extra, []);
});
+218
View File
@@ -0,0 +1,218 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, writeFileSync, mkdirSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { doctor } from "../src/commands/doctor.ts";
import { bootstrap } from "../src/commands/bootstrap.ts";
import { verify } from "../src/commands/verify.ts";
import { testCtx } from "./helpers.ts";
import type { SpawnResult } from "../src/lib/spawn.ts";
const r0 = (stdout = ""): SpawnResult => ({ code: 0, signal: null, timedOut: false, stdout, stderr: "" });
const r1 = (): SpawnResult => ({ code: 1, signal: null, timedOut: false, stdout: "", stderr: "" });
const r = (stdout: string): SpawnResult => r0(stdout);
test("doctor: all prerequisites present exits 0", async () => {
const ctx = testCtx({
spawn: async (cmd, args) => {
if (cmd === "which") return r0();
if (cmd === "pnpm" && args[0] === "--version") return r0("10.0.0\n");
if (cmd === "python3" && args[0] === "--version") return r0("Python 3.11.2\n");
if (cmd === "git" && args[0] === "--version") return r0("git version 2.39.2\n");
return r0();
},
});
const code = await doctor([], ctx);
assert.equal(code, 0);
assert.ok(ctx.out().includes("All checks passed"));
});
test("doctor: missing pnpm fails with hint, exit 1", async () => {
const ctx = testCtx({
spawn: async (cmd, args) => {
if (cmd === "which" && args[0] === "pnpm") return { ...r1(), code: 1 };
if (cmd === "which") return r0();
if (cmd === "python3") return r0("Python 3.11.2\n");
if (cmd === "git") return r0("git version 2.39.2\n");
return r0();
},
});
const code = await doctor([], ctx);
assert.equal(code, 1);
assert.ok(ctx.err().includes("pnpm not found"));
assert.ok(ctx.err().includes("corepack"));
});
test("doctor: outdated python3 fails with version comparison", async () => {
const ctx = testCtx({
spawn: async (cmd, args) => {
if (cmd === "which") return r0();
if (cmd === "python3" && args[0] === "--version") return r0("Python 3.9.0\n");
if (cmd === "pnpm") return r0("10.0.0\n");
if (cmd === "git") return r0("git version 2.39.2\n");
return r0();
},
});
const code = await doctor([], ctx);
assert.equal(code, 1);
assert.ok(ctx.err().includes("3.9"));
});
test("bootstrap: outside a repo fails with clone hint", async () => {
const dir = mkdtempSync(join(tmpdir(), "nc-norepo-"));
const ctx = testCtx({ cwd: dir });
const code = await bootstrap([], ctx);
assert.equal(code, 1);
assert.ok(ctx.err().includes("git clone"));
});
const doctorPassSpawn = async (cmd: string, args: string[]): Promise<SpawnResult> => {
if (cmd === "pnpm" && args[0] === "--version") return r("10.0.0\n");
if (cmd === "python3" && args[0] === "--version") return r("Python 3.11.2\n");
if (cmd === "git") return r("git version 2.39.2\n");
if (cmd === "python3" && args[1] === "venv") return r0();
return r0();
};
test("bootstrap: step order — preflight, pnpm install, venv bootstrap, env copy", async () => {
const dir = mkdtempSync(join(tmpdir(), "nc-repo-"));
writeFileSync(join(dir, "pnpm-workspace.yaml"), "packages:\n - apps/*\n");
mkdirSync(join(dir, "apps", "ai-service"), { recursive: true });
writeFileSync(join(dir, "apps", "ai-service", ".env.example"), "AI_PORT=8420\nAI_KEY=\n");
const calls: string[] = [];
const ctx = testCtx({
cwd: dir,
spawn: async (cmd, args) => {
calls.push(`${cmd} ${args.join(" ")}`);
return doctorPassSpawn(cmd, args);
},
});
const code = await bootstrap([], ctx);
assert.equal(code, 0);
const installIdx = calls.findIndex((c) => c.startsWith("pnpm install"));
const venvIdx = calls.findIndex((c) => c.startsWith("bash scripts/bootstrap.sh"));
assert.ok(installIdx >= 0, "pnpm install runs");
assert.ok(venvIdx > installIdx, "venv bootstrap runs after pnpm install");
assert.ok(ctx.exists(join(dir, "apps", "ai-service", ".env")));
assert.ok(ctx.out().includes("bootstrap complete"));
});
test("bootstrap: existing .env kept, not overwritten", async () => {
const dir = mkdtempSync(join(tmpdir(), "nc-keep-"));
writeFileSync(join(dir, "pnpm-workspace.yaml"), "packages:\n - apps/*\n");
const aiDir = join(dir, "apps", "ai-service");
mkdirSync(aiDir, { recursive: true });
writeFileSync(join(aiDir, ".env.example"), "AI_PORT=8420\n");
writeFileSync(join(aiDir, ".env"), "AI_PORT=9999\n");
const ctx = testCtx({ cwd: dir, spawn: doctorPassSpawn });
const code = await bootstrap([], ctx);
assert.equal(code, 0);
assert.ok(ctx.out().includes("kept as-is"));
assert.equal(ctx.readFile(join(aiDir, ".env")), "AI_PORT=9999\n");
});
test("bootstrap: failing pnpm install aborts before venv step", async () => {
const dir = mkdtempSync(join(tmpdir(), "nc-fail-"));
writeFileSync(join(dir, "pnpm-workspace.yaml"), "packages:\n - apps/*\n");
const calls: string[] = [];
const ctx = testCtx({
cwd: dir,
spawn: async (cmd, args) => {
calls.push(`${cmd} ${args.join(" ")}`);
if (cmd === "pnpm" && args[0] === "install") return r1();
return doctorPassSpawn(cmd, args);
},
});
const code = await bootstrap([], ctx);
assert.equal(code, 1);
assert.ok(ctx.err().includes("pnpm install failed"), ctx.err());
assert.equal(calls.filter((c) => c.startsWith("bash scripts/bootstrap.sh")).length, 0, "venv step must not run");
});
test("verify: busy port fails with stop-the-process hint", async () => {
const dir = mkdtempSync(join(tmpdir(), "nc-ver-"));
writeFileSync(join(dir, "pnpm-workspace.yaml"), "packages:\n - apps/*\n");
const aiDir = join(dir, "apps", "ai-service");
mkdirSync(aiDir, { recursive: true });
mkdirSync(join(aiDir, ".venv", "bin"), { recursive: true });
writeFileSync(join(aiDir, ".venv", "bin", "python3"), "#!/bin/sh\nexit 0\n");
writeFileSync(join(aiDir, ".env"), "AI_PORT=8420\n");
mkdirSync(join(dir, "node_modules", ".bin"), { recursive: true });
writeFileSync(join(dir, "node_modules", ".bin", "turbo"), "#!/bin/sh\n");
const ctx = testCtx({
cwd: dir,
spawn: async () => r0(),
portFree: async () => false,
});
const code = await verify([], ctx);
assert.equal(code, 1);
assert.ok(ctx.err().includes("port 8420 is busy"));
assert.ok(ctx.err().includes("stop the process listening on 8420"));
});
test("verify: free port passes and reports AI_PORT from .env", async () => {
const dir = mkdtempSync(join(tmpdir(), "nc-ver2-"));
writeFileSync(join(dir, "pnpm-workspace.yaml"), "packages:\n - apps/*\n");
const aiDir = join(dir, "apps", "ai-service");
mkdirSync(aiDir, { recursive: true });
mkdirSync(join(aiDir, ".venv", "bin"), { recursive: true });
writeFileSync(join(aiDir, ".venv", "bin", "python3"), "#!/bin/sh\nexit 0\n");
writeFileSync(join(aiDir, ".env"), "AI_PORT=8421\n");
mkdirSync(join(dir, "node_modules", ".bin"), { recursive: true });
writeFileSync(join(dir, "node_modules", ".bin", "turbo"), "#!/bin/sh\n");
const ctx = testCtx({
cwd: dir,
spawn: async () => r0(),
portFree: async () => true,
});
const code = await verify([], ctx);
assert.equal(code, 0);
assert.ok(ctx.out().includes("port 8421 free"));
});
test("bootstrap: preflight failure aborts before pnpm install", async () => {
const dir = mkdtempSync(join(tmpdir(), "nc-pre-"));
writeFileSync(join(dir, "pnpm-workspace.yaml"), "packages:\n - apps/*\n");
const calls: string[] = [];
const ctx = testCtx({
cwd: dir,
spawn: async (cmd, args) => {
calls.push(`${cmd} ${args.join(" ")}`);
if (cmd === "which") return r0();
if (cmd === "pnpm" && args[0] === "--version") return r("10.0.0\n");
if (cmd === "python3" && args[0] === "--version") return r("Python 3.11.2\n");
if (cmd === "git") return r("git version 2.39.2\n");
if (cmd === "python3" && args[1] === "venv") return r1();
if (cmd === "rm") return r0();
return r0();
},
});
const code = await bootstrap([], ctx);
assert.equal(code, 1);
assert.ok(ctx.err().includes("preflight failed"));
assert.equal(calls.filter((c) => c.startsWith("pnpm install")).length, 0, "pnpm install must not run when preflight fails");
});
test("doctor: venv-capability probe failure surfaces apt hint", async () => {
const ctx = testCtx({
spawn: async (cmd, args) => {
if (cmd === "which") return r0();
if (cmd === "pnpm") return r("10.0.0\n");
if (cmd === "python3" && args[0] === "--version") return r("Python 3.12.3\n");
if (cmd === "git") return r("git version 2.43.0\n");
if (cmd === "node") return r("v24.0.0\n");
if (cmd === "python3" && args[1] === "venv") return r1();
if (cmd === "rm") return r0();
return r0();
},
});
const code = await doctor([], ctx);
assert.equal(code, 1);
assert.ok(ctx.err().includes("cannot create virtual environments"));
assert.ok(ctx.err().includes("apt install python"));
assert.ok(ctx.err().includes("-venv"));
});
+62
View File
@@ -0,0 +1,62 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { run } from "../src/index.ts";
import { testCtx } from "./helpers.ts";
test("dispatch: --help exits 0 and prints usage", async () => {
const ctx = testCtx();
const code = await run(["--help"], ctx);
assert.equal(code, 0);
assert.ok(ctx.out().includes("Usage:"));
assert.ok(ctx.out().includes("doctor"));
});
test("dispatch: -h behaves like --help", async () => {
const ctx = testCtx();
assert.equal(await run(["-h"], ctx), 0);
});
test("dispatch: --version prints NEXTCRAFT_VERSION override", async () => {
const env = { ...process.env, NEXTCRAFT_VERSION: "v9.9.9-test" };
const ctx = testCtx({ env });
const code = await run(["--version"], ctx);
assert.equal(code, 0);
assert.ok(ctx.out().includes("v9.9.9-test"));
});
test("dispatch: no command exits 2 with usage on stderr", async () => {
const ctx = testCtx();
assert.equal(await run([], ctx), 2);
assert.ok(ctx.err().includes("Usage:") || ctx.out().includes("Usage:") || ctx.err().includes("nextcraft"));
});
test("dispatch: unknown command exits 2", async () => {
const ctx = testCtx();
const code = await run(["frobnicate"], ctx);
assert.equal(code, 2);
assert.ok(ctx.err().includes("unknown command: frobnicate"));
});
test("dispatch: unknown flag exits 2", async () => {
const ctx = testCtx();
assert.equal(await run(["doctor", "--bogus"], ctx), 2);
});
test("dispatch: doctor routes to the doctor command", async () => {
const ctx = testCtx({
spawn: async (cmd, args) => {
if (cmd === "which") return { code: 0, signal: null, timedOut: false, stdout: "", stderr: "" };
if (cmd === "pnpm" && args[0] === "--version") return r("10.0.0\n");
if (cmd === "python3" && args[0] === "--version") return r("Python 3.11.2\n");
if (cmd === "git" && args[0] === "--version") return r("git version 2.39.2\n");
return { code: 0, signal: null, timedOut: false, stdout: "", stderr: "" };
},
});
const code = await run(["doctor"], ctx);
assert.equal(code, 0);
assert.ok(ctx.out().includes("All checks passed"));
});
const r = (stdout: string) => ({ code: 0, signal: null, timedOut: false, stdout, stderr: "" });
+19
View File
@@ -0,0 +1,19 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { doctor } from "../src/commands/doctor.ts";
import { testCtx } from "./helpers.ts";
test("doctor integration: real box — all prerequisites present", async () => {
const ctx = testCtx();
const code = await doctor([], ctx);
const output = ctx.out() + ctx.err();
if (output.includes("unshare not found")) {
console.warn("unshare missing on this box — tolerating its single failure");
assert.equal(code, 1);
return;
}
assert.equal(code, 0);
assert.ok(output.includes("node"));
assert.ok(output.includes("pnpm"));
assert.ok(output.includes("python3"));
});
+72
View File
@@ -0,0 +1,72 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import { mkdtempSync, existsSync, readFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join, dirname } from "node:path";
import { fileURLToPath } from "node:url";
const cliDir = join(dirname(fileURLToPath(import.meta.url)), "..");
const repoRoot = join(cliDir, "..", "..");
test("fresh-clone E2E: clone -> doctor -> bootstrap -> verify (happy path, transcript)", async () => {
const work = mkdtempSync(join(tmpdir(), "nc-e2e-"));
const cloneDir = join(work, "nextcraft");
const clone = spawnSync(
"git",
["clone", "--quiet", "--no-hardlinks", repoRoot, cloneDir],
{ encoding: "utf8", timeout: 120000 },
);
assert.equal(clone.status, 0, `clone failed: ${clone.stderr}`);
const binary = join(repoRoot, "apps", "cli", "dist", "nextcraft-linux-x64");
const cli = existsSync(binary)
? binary
: join(cliDir, "node_modules", ".bin", "tsx");
const transcript: string[] = [];
const step = (name: string, args: string[], expect: number, timeoutMs: number) => {
const { NODE_TEST_CONTEXT, ...restEnv } = process.env as Record<string, string | undefined>;
const env = {
...restEnv,
PATH: [join(process.env.HOME ?? "/home", ".local/bin"), process.env.PATH].filter(Boolean).join(":"),
NEXTCRAFT_VERSION: "v-e2e",
} as Record<string, string | undefined>;
const run = spawnSync(cli, existsSync(binary) ? args : [join(cliDir, "src", "index.ts"), ...args], {
cwd: cloneDir,
encoding: "utf8",
timeout: timeoutMs,
env,
});
transcript.push(`$ nextcraft ${args.join(" ")} -> exit ${run.status}`);
if (run.status !== expect) {
transcript.push(run.stdout, run.stderr);
assert.fail(
`${name} exited ${run.status} (expected ${expect})\nTRANSCRIPT:\n${transcript.join("\n")}\nstderr: ${run.stderr}`,
);
}
return run.stdout + run.stderr;
};
const doctorOut = step("doctor", ["doctor"], 0, 60000);
assert.ok(doctorOut.includes("node"), "doctor mentions node");
assert.ok(doctorOut.includes("pnpm"), "doctor mentions pnpm");
step("bootstrap", ["bootstrap"], 0, 600000);
const verifyOut = step("verify", ["verify"], 0, 120000);
assert.ok(verifyOut.includes("venv"), "verify covers venv");
assert.ok(existsSync(join(cloneDir, "apps", "ai-service", ".env")), ".env created in clone");
assert.ok(existsSync(join(cloneDir, "apps", "ai-service", ".venv")), "venv created in clone");
assert.ok(
existsSync(join(cloneDir, "node_modules", ".pnpm")),
"workspace node_modules present",
);
const envExample = readFileSync(join(repoRoot, "apps", "ai-service", ".env.example"), "utf8");
const envClone = readFileSync(join(cloneDir, "apps", "ai-service", ".env"), "utf8");
assert.equal(envClone, envExample, ".env content matches template");
console.log("E2E TRANSCRIPT:\n" + transcript.join("\n"));
});
+54
View File
@@ -0,0 +1,54 @@
import type { Ctx } from "../src/ctx.ts";
import { run } from "../src/index.ts";
import { readFileSync, existsSync, writeFileSync } from "node:fs";
import net from "node:net";
import { run as spawnRun } from "../src/lib/spawn.ts";
import type { SpawnResult } from "../src/lib/spawn.ts";
export interface TestCtx extends Ctx {
out(): string;
err(): string;
writes: Record<string, string>;
}
export function testCtx(overrides: Partial<Ctx> = {}): TestCtx {
const out: string[] = [];
const err: string[] = [];
const writes: Record<string, string> = {};
const base = {
cwd: process.cwd(),
env: { ...process.env } as Record<string, string | undefined>,
stdout: { write: (s: string) => void out.push(s) },
stderr: { write: (s: string) => void err.push(s) },
spawn: async (cmd: string, args: string[], opts: Parameters<Ctx["spawn"]>[2]) =>
spawnRun(cmd, args, opts),
exists: (p: string) => existsSync(p),
readFile: (p: string) => (existsSync(p) ? readFileSync(p, "utf8") : undefined),
writeFile: (p: string, c: string) => {
writes[p] = c;
writeFileSync(p, c);
},
portFree: async (port: number, host: string) =>
new Promise<boolean>((resolve) => {
const srv = net.createServer();
srv.once("error", () => resolve(false));
srv.once("listening", () => srv.close(() => resolve(true)));
srv.listen(port, host);
}),
out: () => out.join(""),
err: () => err.join(""),
};
const ctx = { ...base, ...overrides } as unknown as TestCtx;
(ctx as unknown as { writes: Record<string, string> }).writes = writes;
return ctx;
}
export function fakeSpawn(
sequence: Array<{ match: (cmd: string, args: string[]) => boolean; result: SpawnResult }>,
): Ctx["spawn"] {
return async (cmd, args) => {
const hit = sequence.find((s) => s.match(cmd, args));
if (!hit) throw new Error(`unexpected spawn: ${cmd} ${args.join(" ")}`);
return hit.result;
};
}
+204
View File
@@ -0,0 +1,204 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { spawn, spawnSync } from "node:child_process";
import { cpSync, mkdirSync, mkdtempSync, writeFileSync, readFileSync, existsSync } from "node:fs";
import { createHash } from "node:crypto";
import { tmpdir } from "node:os";
import { join, dirname } from "node:path";
import { fileURLToPath } from "node:url";
const repoRoot = join(dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
const installSh = join(repoRoot, "scripts", "install.sh");
function sha256(p: string): string {
return createHash("sha256").update(readFileSync(p)).digest("hex");
}
async function serve(
setup: (srvDir: string, base: () => string) => void,
): Promise<{ url: string; stop: () => void }> {
const srvDir = mkdtempSync(join(tmpdir(), "nc-srv-"));
const port = 30000 + Math.floor(Math.random() * 20000);
const base = () => `http://127.0.0.1:${port}`;
setup(srvDir, base);
const child = spawn("python3", ["-m", "http.server", String(port), "--directory", srvDir], {
stdio: "ignore",
});
for (let i = 0; i < 50; i++) {
const probe = spawnSync("curl", ["-fsS", `${base()}/api/v1/repos/coreci/nextcraft/releases/latest`], {
encoding: "utf8",
});
if (probe.status === 0) break;
await new Promise((r) => setTimeout(r, 100));
}
return { url: base(), stop: () => child.kill("SIGTERM") };
}
function apiManifestDir(srvDir: string, manifest: object): string {
const dir = join(srvDir, "api/v1/repos/coreci/nextcraft/releases");
mkdirSync(dir, { recursive: true });
writeFileSync(join(dir, "latest"), JSON.stringify(manifest));
return dir;
}
async function runInstall(home: string, srvUrl: string) {
return spawnSync("sh", [installSh], {
env: { ...process.env, HOME: home, NEXTCRAFT_FORGE_BASE: srvUrl, DEST: join(home, "bin") },
encoding: "utf8",
timeout: 30000,
});
}
test("install.sh: checksum mismatch = hard stop, no install", async () => {
const home = mkdtempSync(join(tmpdir(), "nc-home-"));
const srv = await serve((srvDir, base) => {
writeFileSync(join(srvDir, "ping"), "pong");
writeFileSync(join(srvDir, "nextcraft-linux-x64"), "#!/bin/sh\necho fake\n");
writeFileSync(join(srvDir, "nextcraft-linux-x64.sha256"), "deadbeef nextcraft-linux-x64\n");
apiManifestDir(srvDir, {
tag_name: "v9.9.9",
assets: [
{ name: "nextcraft-linux-x64", browser_download_url: `${base()}/nextcraft-linux-x64` },
{ name: "nextcraft-linux-x64.sha256", browser_download_url: `${base()}/nextcraft-linux-x64.sha256` },
],
});
});
try {
const res = await runInstall(home, srv.url);
assert.notEqual(res.status, 0);
assert.ok(res.stderr.includes("CHECKSUM MISMATCH"));
assert.ok(!existsSync(join(home, "bin", "nextcraft")));
} finally {
srv.stop();
}
});
test("install.sh: valid checksum installs binary and reports version", async () => {
const home = mkdtempSync(join(tmpdir(), "nc-home-ok-"));
const srv = await serve((srvDir, base) => {
const bin = join(srvDir, "nextcraft-linux-x64");
writeFileSync(join(srvDir, "ping"), "pong");
writeFileSync(bin, "#!/bin/sh\nif [ \"$1\" = \"--version\" ]; then echo v9.9.9; exit 0; fi\necho v9.9.9-installed\nexit 0\n");
writeFileSync(join(srvDir, "nextcraft-linux-x64.sha256"), `${sha256(bin)} nextcraft-linux-x64\n`);
apiManifestDir(srvDir, {
tag_name: "v9.9.9",
assets: [
{ name: "nextcraft-linux-x64", browser_download_url: `${base()}/nextcraft-linux-x64` },
{ name: "nextcraft-linux-x64.sha256", browser_download_url: `${base()}/nextcraft-linux-x64.sha256` },
],
});
});
try {
const res = await runInstall(home, srv.url);
assert.equal(res.status, 0, res.stderr);
assert.ok(existsSync(join(home, "bin", "nextcraft")));
const run = spawnSync(join(home, "bin", "nextcraft"), [], { encoding: "utf8" });
assert.ok(run.stdout.includes("v9.9.9-installed"));
} finally {
srv.stop();
}
});
test("install.sh: version-mismatching binary is rejected (G-102 install-time integrity)", async () => {
const home = mkdtempSync(join(tmpdir(), "nc-home-mm-"));
const srv = await serve((srvDir, base) => {
const bin = join(srvDir, "nextcraft-linux-x64");
writeFileSync(bin, "#!/bin/sh\nif [ \"$1\" = \"--version\" ]; then echo v0.0.0-wrong; exit 0; fi\nexit 0\n");
writeFileSync(join(srvDir, "nextcraft-linux-x64.sha256"), `${sha256(bin)} nextcraft-linux-x64\n`);
apiManifestDir(srvDir, {
tag_name: "v9.9.9",
assets: [
{ name: "nextcraft-linux-x64", browser_download_url: `${base()}/nextcraft-linux-x64` },
{ name: "nextcraft-linux-x64.sha256", browser_download_url: `${base()}/nextcraft-linux-x64.sha256` },
],
});
});
try {
const res = await runInstall(home, srv.url);
assert.notEqual(res.status, 0, "mismatched version must not install cleanly");
assert.ok(res.stderr.includes("integrity mismatch"), `stderr: ${res.stderr}`);
} finally {
srv.stop();
}
});
test("install.sh: silent --version binary is rejected (honesty gate)", async () => {
const home = mkdtempSync(join(tmpdir(), "nc-home-silent-"));
const srv = await serve((srvDir, base) => {
const bin = join(srvDir, "nextcraft-linux-x64");
writeFileSync(bin, "#!/bin/sh\nexit 0\n");
writeFileSync(join(srvDir, "nextcraft-linux-x64.sha256"), `${sha256(bin)} nextcraft-linux-x64\n`);
apiManifestDir(srvDir, {
tag_name: "v9.9.9",
assets: [
{ name: "nextcraft-linux-x64", browser_download_url: `${base()}/nextcraft-linux-x64` },
{ name: "nextcraft-linux-x64.sha256", browser_download_url: `${base()}/nextcraft-linux-x64.sha256` },
],
});
});
try {
const res = await runInstall(home, srv.url);
assert.notEqual(res.status, 0, "silent binary must be rejected");
assert.ok(res.stderr.includes("produced no output"), `stderr: ${res.stderr}`);
assert.ok(!existsSync(join(home, "bin", "nextcraft")) || res.stderr.includes("Do not use"), "must not leave a trusted silent binary");
} finally {
srv.stop();
}
});
test("install.sh: release without binary assets degrades to source instructions, exit 0", async () => {
const home = mkdtempSync(join(tmpdir(), "nc-home-nb-"));
const srv = await serve((srvDir) => {
writeFileSync(join(srvDir, "ping"), "pong");
apiManifestDir(srvDir, { tag_name: "v0.2.8", assets: [] });
});
try {
const res = await runInstall(home, srv.url);
assert.equal(res.status, 0);
assert.ok(res.stdout.includes("git clone"));
assert.ok(!existsSync(join(home, "bin", "nextcraft")));
} finally {
srv.stop();
}
});
test("release-assets.sh: token resolution reads .env* files only — poisoned shell env is never used", async () => {
const tmpHome = mkdtempSync(join(tmpdir(), "nc-home-token-"));
mkdirSync(join(tmpHome, ".ciagent"), { recursive: true });
writeFileSync(join(tmpHome, ".ciagent", ".env.secrets"), "GITEA_TOKEN=real-token-from-file\n");
const fakeBin = join(tmpHome, "apps/cli/dist");
mkdirSync(fakeBin, { recursive: true });
writeFileSync(join(fakeBin, "nextcraft-linux-x64"), "fake-binary-bytes\n");
writeFileSync(join(fakeBin, "nextcraft-linux-x64.sha256"), "abc123 nextcraft-linux-x64\n");
const srv = await serve((srvDir) => {
const tagsDir = join(srvDir, "api/v1/repos/coreci/nextcraft/releases/tags");
mkdirSync(tagsDir, { recursive: true });
writeFileSync(join(tagsDir, "v0.3.2"), JSON.stringify({ id: 42, tag_name: "v0.3.2" }));
});
try {
const env: Record<string, string> = {
PATH: process.env.PATH ?? "",
HOME: tmpHome,
GITEA_TOKEN: "poisoned-shell-token",
NEXTCRAFT_FORGE_BASE: srv.url,
};
const res = spawnSync(
"bash",
[
"-c",
`NEXTCRAFT_FORGE_BASE='${srv.url}' GITEA_TOKEN=poisoned-shell-token bash '${join(repoRoot, "scripts", "release-assets.sh")}' v0.3.2 --dry-run`,
],
{ cwd: tmpHome, env, encoding: "utf8", timeout: 20000 },
);
assert.equal(res.stdout.includes("poisoned-shell-token"), false, "poisoned token never in stdout");
assert.equal(res.stderr.includes("poisoned-shell-token"), false, "poisoned token never in stderr");
assert.equal(res.status, 0, `dry-run should succeed against fixture, stderr: ${res.stderr}`);
assert.ok(res.stdout.includes("DRY-RUN would upload"), `expected dry-run upload lines, got: ${res.stdout}`);
assert.ok(res.stdout.includes("nextcraft-linux-x64"));
} finally {
srv.stop();
}
});
+38
View File
@@ -0,0 +1,38 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import { cpSync, existsSync, mkdirSync, mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join, dirname } from "node:path";
import { fileURLToPath } from "node:url";
const cliDir = join(dirname(fileURLToPath(import.meta.url)), "..");
const repoRoot = join(cliDir, "..", "..");
const binary = join(repoRoot, "apps", "cli", "dist", "nextcraft-linux-x64");
test("PATH bare-word invocation: installed binary speaks (SEA regression, v0.3.4 bug)", () => {
if (!existsSync(binary)) return; // built by cli:build:binary; covered in ship flow
const binDir = mkdtempSync(join(tmpdir(), "nc-pathword-"));
cpSync(binary, join(binDir, "nextcraft"));
const env = {
PATH: [binDir, process.env.PATH].filter(Boolean).join(":"),
HOME: process.env.HOME,
};
const { NODE_TEST_CONTEXT, ...rest } = process.env as Record<string, string | undefined>;
const envClean = { ...rest, ...env } as Record<string, string | undefined>;
const ver = spawnSync("sh", ["-c", "nextcraft --version"], { encoding: "utf8", timeout: 30000, env: envClean });
assert.equal(ver.status, 0, `--version rc: ${ver.status} stderr: ${ver.stderr}`);
assert.ok(ver.stdout.trim().length > 0, "--version must print the version (v0.3.4 was silent)");
const doc = spawnSync("sh", ["-c", "nextcraft doctor"], { encoding: "utf8", timeout: 120000, env: envClean });
assert.equal(doc.status, 0, `doctor rc: ${doc.status} stderr: ${doc.stderr}`);
assert.ok(doc.stdout.includes("environment prerequisites"), "doctor must print its report");
const noArgs = spawnSync("sh", ["-c", "nextcraft"], { encoding: "utf8", timeout: 30000, env: envClean });
assert.equal(noArgs.status, 2, "no args must exit 2");
assert.ok((noArgs.stderr + noArgs.stdout).includes("Usage:"), "no args must print usage");
const bad = spawnSync("sh", ["-c", "nextcraft nosuchcmd"], { encoding: "utf8", timeout: 30000, env: envClean });
assert.equal(bad.status, 2, "unknown command must exit 2");
});
+31
View File
@@ -0,0 +1,31 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { run } from "../src/lib/spawn.ts";
test("spawn: returns exit code of short process", async () => {
const res = await run("node", ["-e", "process.exit(3)"], { capture: true, timeoutMs: 5000 });
assert.equal(res.code, 3);
assert.equal(res.timedOut, false);
});
test("spawn: timeout kills and reports timedOut", async () => {
const res = await run("node", ["-e", "setTimeout(() => {}, 10000)"], {
capture: true,
timeoutMs: 300,
});
assert.equal(res.timedOut, true);
assert.ok(res.code !== 0);
});
test("spawn: missing binary resolves code 127, never rejects", async () => {
const res = await run("definitely-not-a-real-binary-xyz", [], { capture: true, timeoutMs: 1000 });
assert.equal(res.code, 127);
});
test("spawn: capture collects stdout", async () => {
const res = await run("node", ["-e", "process.stdout.write('hello')"], {
capture: true,
timeoutMs: 5000,
});
assert.equal(res.stdout, "hello");
});
+15
View File
@@ -0,0 +1,15 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"noEmit": false,
"allowImportingTsExtensions": false,
"declaration": false,
"declarationMap": false,
"sourceMap": false,
"incremental": false,
"outDir": "dist",
"rootDir": "src"
},
"include": ["src/**/*"],
"exclude": ["tests", "node_modules", "dist"]
}
+15
View File
@@ -0,0 +1,15 @@
{
"extends": "../../tsconfig.json",
"compilerOptions": {
"module": "NodeNext",
"moduleResolution": "NodeNext",
"lib": ["ES2022"],
"types": ["node"],
"noEmit": true,
"declaration": false,
"declarationMap": false,
"allowImportingTsExtensions": true,
"baseUrl": "."
},
"include": ["src/**/*", "tests/**/*"]
}
+5 -2
View File
@@ -1,2 +1,5 @@
# AI service (v0.2) — learner chat/panels stream from this FastAPI service
NEXT_PUBLIC_AI_SERVICE_URL=http://localhost:8420
# Browser-side engine API base URL. UNSET = auto (recommended): the browser
# derives http://<current-hostname>:8420 at runtime, so remote browsing works
# zero-config. Set explicitly only for unusual topologies, e.g.:
# NEXT_PUBLIC_AI_SERVICE_URL=http://ai.internal:8420
# NEXT_PUBLIC_AI_SERVICE_URL=
@@ -1,300 +1,38 @@
import Link from 'next/link';
import { notFound } from 'next/navigation';
import {
ArrowLeft,
ArrowRight,
Play,
Save,
Upload,
Folder,
FileText,
Activity,
} from 'lucide-react';
import { Button } from '@nextcraft/ui';
import { allCompetencies, competencyStacks, aiLabScenarios } from '@nextcraft/mock-data';
import { LabFeedbackPanel } from '../../../../components/learner/lab-feedback-panel';
import { ArrowLeft, AlertTriangle, Loader2 } from 'lucide-react';
import Link from 'next/link';
import { competencyStacks } from '@nextcraft/mock-data';
import { BuildSurface } from '../../../../components/learner/build-surface';
interface FileEntry {
label: string;
children?: { label: string }[];
}
const FILE_TREE: FileEntry[] = [
{
label: 'src/',
children: [
{ label: 'main.ts' },
{ label: 'agent.ts' },
{ label: 'tools.ts' },
],
},
{
label: 'tests/',
children: [{ label: 'agent.test.ts' }],
},
{ label: 'README.md' },
];
const EDITOR_LINES = [
{ n: 1, content: "import { ChatOpenAI } from '@langchain/openai';" },
{ n: 2, content: "import { tool } from '@langchain/core/tools';" },
{ n: 3, content: "import { z } from 'zod';" },
{ n: 4, content: '' },
{ n: 5, content: '// Define a typed tool for fetching the current weather' },
{ n: 6, content: "const getWeather = tool(" },
{ n: 7, content: ' async ({ city, units }) => {' },
{ n: 8, content: ' const res = await fetch(`/api/weather?city=${city}&units=${units}`);' },
{ n: 9, content: ' return res.json();' },
{ n: 10, content: ' },' },
{ n: 11, content: ' {' },
{ n: 12, content: " name: 'get_weather'," },
{ n: 13, content: " description: 'Fetch the current weather for a city'," },
{ n: 14, content: ' schema: z.object({' },
{ n: 15, content: " city: z.string().describe('City to fetch weather for')," },
{ n: 16, content: " units: z.enum(['celsius', 'fahrenheit']).default('celsius')," },
{ n: 17, content: ' }),' },
{ n: 18, content: ' },' },
{ n: 19, content: ');' },
{ n: 20, content: '' },
{ n: 21, content: 'export async function main(query: string) {' },
{ n: 22, content: ' const model = new ChatOpenAI({ model: "gpt-4o-mini" });' },
{ n: 23, content: ' const modelWithTools = model.bindTools([getWeather]);' },
{ n: 24, content: ' const response = await modelWithTools.invoke(query);' },
{ n: 25, content: ' return response.tool_calls;' },
{ n: 26, content: '}' },
];
const TELEMETRY_METRICS = [
{ label: 'Commits', value: '7' },
{ label: 'Keystrokes', value: '1,247' },
{ label: 'Time spent', value: '23 min' },
{ label: 'Build attempts', value: '3' },
];
const TELEMETRY_EVENTS = [
{ time: '14:02:11', action: 'File created: src/main.ts' },
{ time: '14:09:48', action: 'First build attempt (failed)' },
{ time: '14:12:30', action: 'Test suite passed (2/2)' },
{ time: '14:18:05', action: 'Commit: scaffold agent entrypoint' },
{ time: '14:21:42', action: 'Tool schema validated' },
{ time: '14:25:17', action: 'Build attempt 2 (success)' },
{ time: '14:28:03', action: 'Commit: implement tool calling' },
];
function highlight(line: string): { text: string; cls: string }[] {
// Very small token highlighter for the mock editor
const tokens: { text: string; cls: string }[] = [];
let i = 0;
while (i < line.length) {
// comment
if (line.slice(i).startsWith('//')) {
tokens.push({ text: line.slice(i), cls: 'text-slate-500' });
break;
}
// string with backtick or single/double quote
const ch = line[i];
if (ch === '`' || ch === "'" || ch === '"') {
const end = line.indexOf(ch, i + 1);
if (end !== -1) {
tokens.push({ text: line.slice(i, end + 1), cls: 'text-emerald-300' });
i = end + 1;
continue;
}
}
// keyword
const rest = line.slice(i);
const kwMatch = rest.match(/^(import|from|export|async|function|const|return|await|new)/);
if (kwMatch) {
tokens.push({ text: kwMatch[0], cls: 'text-primary-300' });
i += kwMatch[0].length;
continue;
}
// default: consume one char
tokens.push({ text: ch, cls: 'text-slate-200' });
i += 1;
}
return tokens;
}
export default async function BuildSandboxPage({
/**
* Real build surface (REQ-3-008): a per-learner variant task + a live
* namespace sandbox with file CRUD and Run/Test (CUT-2: read-only output,
* no interactive shell). The v0.1 static mockup is retired.
*/
export default async function BuildPage({
params,
}: {
params: Promise<{ competencyId: string }>;
}) {
const { competencyId } = await params;
const competency = allCompetencies.find((c) => c.id === competencyId);
if (!competency) notFound();
const stack = competencyStacks.find((s) => s.id === competency.stackId);
const known = competencyStacks.some((stack) =>
stack.competencies.some((c) => c.id === competencyId),
);
if (!known) notFound();
const stack = competencyStacks.find((s) =>
s.competencies.some((c) => c.id === competencyId),
)!;
return (
<div className="flex flex-col gap-4">
<div className="mx-auto max-w-6xl px-4 py-8">
<Link
href={`/learn/${competency.id}`}
className="inline-flex w-fit items-center gap-1 text-sm text-slate-500 hover:text-slate-800 dark:hover:text-slate-200"
href={`/learn/${competencyId}`}
className="inline-flex items-center gap-1 text-sm text-slate-500 transition-colors hover:text-primary-600 dark:text-slate-400 dark:hover:text-primary-400"
>
<ArrowLeft className="h-3.5 w-3.5" />
Back to Byte
<ArrowLeft className="h-4 w-4" aria-hidden />
Back to tutorial
</Link>
{/* Toolbar */}
<div className="flex flex-wrap items-center justify-between gap-3 rounded-lg border border-slate-200 bg-white px-4 py-3 dark:border-slate-800 dark:bg-slate-900">
<div className="flex items-center gap-3">
<span className="rounded-md bg-primary-100 px-2 py-0.5 text-xs font-semibold text-primary-700 dark:bg-primary-900/40 dark:text-primary-300">
Sandbox
</span>
<span className="text-sm font-medium text-slate-800 dark:text-slate-100">
{competency.name}
</span>
<span className="hidden text-xs text-slate-500 sm:inline dark:text-slate-400">
{stack?.name}
</span>
</div>
<div className="flex items-center gap-2">
<button
type="button"
className="inline-flex h-8 items-center gap-1.5 rounded-md bg-emerald-600 px-3 text-xs font-medium text-white transition-colors hover:bg-emerald-700"
>
<Play className="h-3.5 w-3.5" />
Run
</button>
<button
type="button"
className="inline-flex h-8 items-center gap-1.5 rounded-md border border-slate-300 px-3 text-xs font-medium text-slate-700 transition-colors hover:bg-slate-100 dark:border-slate-700 dark:text-slate-200 dark:hover:bg-slate-800"
>
<Save className="h-3.5 w-3.5" />
Save
</button>
<button
type="button"
className="inline-flex h-8 items-center gap-1.5 rounded-md border border-slate-300 px-3 text-xs font-medium text-slate-700 transition-colors hover:bg-slate-100 dark:border-slate-700 dark:text-slate-200 dark:hover:bg-slate-800"
>
<Upload className="h-3.5 w-3.5" />
Submit
</button>
</div>
</div>
{/* IDE layout */}
<div className="grid grid-cols-1 gap-4 lg:grid-cols-[14rem_1fr_16rem]">
{/* File explorer */}
<aside className="rounded-lg border border-slate-200 bg-slate-50 p-3 dark:border-slate-800 dark:bg-slate-900">
<h3 className="mb-2 text-xs font-semibold uppercase tracking-wide text-slate-500 dark:text-slate-400">
Explorer
</h3>
<ul className="space-y-1 text-sm">
{FILE_TREE.map((node) =>
node.children ? (
<li key={node.label}>
<div className="flex items-center gap-1.5 text-slate-700 dark:text-slate-200">
<Folder className="h-3.5 w-3.5 text-amber-500" />
{node.label}
</div>
<ul className="ml-4 mt-1 space-y-1">
{node.children.map((child) => (
<li
key={child.label}
className="flex items-center gap-1.5 text-slate-600 dark:text-slate-300"
>
<FileText className="h-3.5 w-3.5 text-slate-400" />
{child.label}
</li>
))}
</ul>
</li>
) : (
<li
key={node.label}
className="flex items-center gap-1.5 text-slate-600 dark:text-slate-300"
>
<FileText className="h-3.5 w-3.5 text-slate-400" />
{node.label}
</li>
),
)}
</ul>
</aside>
{/* Editor */}
<section className="overflow-hidden rounded-lg border border-slate-200 bg-slate-950 dark:border-slate-800">
<div className="flex items-center gap-2 border-b border-slate-800 px-3 py-2 text-xs text-slate-400">
<FileText className="h-3.5 w-3.5" />
src/main.ts
</div>
<pre className="overflow-auto p-3 font-mono text-xs leading-relaxed">
<code>
{EDITOR_LINES.map((line) => (
<div key={line.n} className="flex">
<span className="mr-4 inline-block w-8 select-none text-right text-slate-600">
{line.n}
</span>
<span className="flex-1 whitespace-pre">
{line.content === '' ? (
<span>&nbsp;</span>
) : (
highlight(line.content).map((t, idx) => (
<span key={idx} className={t.cls}>
{t.text}
</span>
))
)}
</span>
</div>
))}
</code>
</pre>
</section>
{/* Telemetry */}
<aside className="flex flex-col gap-3 rounded-lg border border-slate-200 bg-white p-3 dark:border-slate-800 dark:bg-slate-900">
<div className="flex items-center gap-2">
<Activity className="h-4 w-4 text-primary-600" />
<h3 className="text-sm font-semibold text-slate-800 dark:text-slate-100">
Process Capture
</h3>
</div>
<dl className="grid grid-cols-2 gap-2">
{TELEMETRY_METRICS.map((m) => (
<div
key={m.label}
className="rounded-md border border-slate-200 bg-slate-50 p-2 text-center dark:border-slate-800 dark:bg-slate-900/50"
>
<dd className="text-lg font-semibold text-slate-900 dark:text-slate-100">
{m.value}
</dd>
<dt className="text-[10px] uppercase tracking-wide text-slate-500 dark:text-slate-400">
{m.label}
</dt>
</div>
))}
</dl>
<div>
<h4 className="mb-1 text-xs font-semibold uppercase tracking-wide text-slate-500 dark:text-slate-400">
Recent events
</h4>
<ul className="space-y-1.5 text-xs">
{TELEMETRY_EVENTS.map((e, i) => (
<li key={i} className="flex gap-2">
<span className="font-mono text-slate-400">{e.time}</span>
<span className="text-slate-700 dark:text-slate-300">{e.action}</span>
</li>
))}
</ul>
</div>
{/* Lab in-flow feedback — mock telemetry scenario (real engine v0.3+) */}
<div className="border-t border-slate-200 pt-3 dark:border-slate-800">
<LabFeedbackPanel scenarioId={aiLabScenarios[0].id} />
</div>
</aside>
</div>
{/* Submit */}
<div className="flex justify-end">
<Link href={`/defend/${competency.id}`}>
<Button iconRight={<ArrowRight className="h-4 w-4" />}>
Submit for Assessment
</Button>
</Link>
</div>
<BuildSurface competencyId={competencyId} stackTitle={stack.name} />
</div>
);
}
@@ -1,80 +1,15 @@
import Link from 'next/link';
import { notFound } from 'next/navigation';
import {
ArrowLeft,
CheckCircle,
XCircle,
Bot,
Award,
Code,
GitCommit,
FileText,
PlayCircle,
TestTube,
ShieldCheck,
} from 'lucide-react';
import { Card, CardBody, CardHeader, Badge, Button } from '@nextcraft/ui';
import { allCompetencies, competencyStacks } from '@nextcraft/mock-data';
import { OralDefenseInterface } from '../../../../components/learner/oral-defense-interface';
import { AssessorResultsPanel } from '../../../../components/learner/assessor-results-panel';
import { ProctorBanner } from '../../../../components/learner/proctor-banner';
import { aiArtifactSubmissions } from '@nextcraft/mock-data';
const RUBRIC = [
{ name: 'Correctness of agent architecture', passed: true, weight: 25 },
{ name: 'Tool schema design & validation', passed: true, weight: 20 },
{ name: 'Error handling & fallbacks', passed: false, weight: 20 },
{ name: 'Test coverage', passed: true, weight: 15 },
{ name: 'Code clarity & documentation', passed: true, weight: 20 },
];
const CRITERION_SCORES: Record<string, number> = {
'Correctness of agent architecture': 92,
'Tool schema design & validation': 88,
'Error handling & fallbacks': 61,
'Test coverage': 84,
'Code clarity & documentation': 90,
};
const OVERALL_SCORE = 84;
const TRACE_EVENTS = [
{ icon: FileText, label: 'File created: src/main.ts', time: '14:02:11' },
{ icon: PlayCircle, label: 'First build attempt (failed)', time: '14:09:48' },
{ icon: TestTube, label: 'Test suite passed (2/2)', time: '14:12:30' },
{ icon: GitCommit, label: 'Commit: scaffold agent entrypoint', time: '14:18:05' },
{ icon: ShieldCheck, label: 'Tool schema validated', time: '14:21:42' },
{ icon: PlayCircle, label: 'Build attempt 2 (success)', time: '14:25:17' },
{ icon: GitCommit, label: 'Commit: implement tool calling', time: '14:28:03' },
{ icon: Award, label: 'Artifact submitted for assessment', time: '14:31:50' },
];
const SUBMITTED_CODE = `import { ChatOpenAI } from '@langchain/openai';
import { tool } from '@langchain/core/tools';
import { z } from 'zod';
const getWeather = tool(
async ({ city, units }) => {
const res = await fetch(\`/api/weather?city=\${city}&units=\${units}\`);
return res.json();
},
{
name: 'get_weather',
description: 'Fetch the current weather for a city',
schema: z.object({
city: z.string(),
units: z.enum(['celsius', 'fahrenheit']).default('celsius'),
}),
},
);
export async function main(query: string) {
const model = new ChatOpenAI({ model: 'gpt-4o-mini' });
const modelWithTools = model.bindTools([getWeather]);
const response = await modelWithTools.invoke(query);
return response.tool_calls;
}`;
import { ArrowLeft } from 'lucide-react';
import { allCompetencies } from '@nextcraft/mock-data';
import { DefenseSession } from '../../../../components/learner/defense-session';
/**
* Live defense surface (REQ-3-006 + REQ-3-008): a real oral defense over
* the learner's task + a live grading panel no mock rubric data. The
* v0.1 static assessment mockup (pre-baked rubric scores, scripted
* transcript, submitted-code display) is retired.
*/
export default async function DefensePage({
params,
}: {
@@ -83,250 +18,27 @@ export default async function DefensePage({
const { competencyId } = await params;
const competency = allCompetencies.find((c) => c.id === competencyId);
if (!competency) notFound();
const stack = competencyStacks.find((s) => s.id === competency.stackId);
const radius = 36;
const circumference = 2 * Math.PI * radius;
const offset = circumference - (OVERALL_SCORE / 100) * circumference;
return (
<div className="flex flex-col gap-6">
<div className="mx-auto flex max-w-3xl flex-col gap-6 px-4 py-8">
<Link
href={`/build/${competency.id}`}
className="inline-flex w-fit items-center gap-1 text-sm text-slate-500 hover:text-slate-800 dark:hover:text-slate-200"
>
<ArrowLeft className="h-3.5 w-3.5" />
<ArrowLeft className="h-3.5 w-3.5" aria-hidden />
Back to build
</Link>
<header className="flex flex-col gap-2">
<Badge variant="warning">Assessment · Defense</Badge>
<h1 className="text-2xl font-bold tracking-tight text-slate-900 dark:text-slate-100">
{competency.name}
<header className="space-y-1">
<h1 className="text-xl font-bold text-slate-900 dark:text-slate-100">
Oral Defense {competency.name}
</h1>
<p className="text-sm text-slate-500 dark:text-slate-400">
{stack?.name} · oral defense and rubric review
The Examiner will question you about your build session: what you
did, why, and what you would change. Answer by voice or typing
then request your process-trace grade.
</p>
</header>
{/* Artifact viewer */}
<Card>
<CardHeader>
<div className="flex items-center gap-2">
<Code className="h-4 w-4 text-primary-600" />
<h2 className="text-base font-semibold text-slate-900 dark:text-slate-100">
Submitted Artifact
</h2>
</div>
</CardHeader>
<CardBody>
<pre className="overflow-auto rounded-lg bg-slate-950 p-4 font-mono text-xs leading-relaxed text-slate-100">
<code>{SUBMITTED_CODE}</code>
</pre>
</CardBody>
</Card>
{/* Rubric + AI reviewer */}
<div className="grid grid-cols-1 gap-4 lg:grid-cols-2">
{/* Rubric */}
<Card>
<CardHeader>
<h2 className="text-base font-semibold text-slate-900 dark:text-slate-100">
Assessment rubric
</h2>
<p className="text-sm text-slate-500 dark:text-slate-400">
Criteria and weights
</p>
</CardHeader>
<CardBody className="flex flex-col gap-3">
{RUBRIC.map((c) => {
const Icon = c.passed ? CheckCircle : XCircle;
return (
<div
key={c.name}
className="flex items-center justify-between gap-3 rounded-md border border-slate-200 p-3 dark:border-slate-800"
>
<div className="flex items-center gap-2">
<Icon
className={`h-4 w-4 ${
c.passed
? 'text-emerald-600 dark:text-emerald-400'
: 'text-rose-600 dark:text-rose-400'
}`}
/>
<span className="text-sm text-slate-800 dark:text-slate-200">
{c.name}
</span>
</div>
<Badge variant={c.passed ? 'success' : 'error'}>{c.weight}%</Badge>
</div>
);
})}
</CardBody>
</Card>
{/* AI reviewer panel */}
<Card>
<CardHeader>
<div className="flex items-center gap-2">
<span className="flex h-8 w-8 items-center justify-center rounded-full bg-primary-100 text-primary-700 dark:bg-primary-900/40 dark:text-primary-300">
<Bot className="h-4 w-4" />
</span>
<div>
<h2 className="text-base font-semibold text-slate-900 dark:text-slate-100">
AI Assessor
</h2>
<p className="text-sm text-slate-500 dark:text-slate-400">
Automated review results
</p>
</div>
</div>
</CardHeader>
<CardBody className="flex flex-col gap-4">
{/* Overall score */}
<div className="flex items-center gap-4">
<div className="relative h-24 w-24 shrink-0">
<svg viewBox="0 0 100 100" className="h-full w-full -rotate-90">
<circle
cx="50"
cy="50"
r={radius}
fill="none"
stroke="currentColor"
strokeWidth="8"
className="text-slate-200 dark:text-slate-800"
/>
<circle
cx="50"
cy="50"
r={radius}
fill="none"
stroke="currentColor"
strokeWidth="8"
strokeDasharray={circumference}
strokeDashoffset={offset}
strokeLinecap="round"
className="text-primary-600"
/>
</svg>
<div className="absolute inset-0 flex items-center justify-center text-xl font-bold text-slate-900 dark:text-slate-100">
{OVERALL_SCORE}
</div>
</div>
<div>
<p className="text-sm font-medium text-slate-900 dark:text-slate-100">
Overall score
</p>
<p className="text-xs text-slate-500 dark:text-slate-400">
Pass threshold: 75 ·{' '}
<span className="text-emerald-600 dark:text-emerald-400">Passing</span>
</p>
</div>
</div>
{/* Per-criterion scores */}
<ul className="flex flex-col gap-2">
{RUBRIC.map((c) => {
const score = CRITERION_SCORES[c.name] ?? 0;
return (
<li key={c.name} className="flex items-center gap-3">
<span className="flex-1 truncate text-xs text-slate-700 dark:text-slate-300">
{c.name}
</span>
<div className="h-1.5 w-24 overflow-hidden rounded-full bg-slate-100 dark:bg-slate-800">
<div
className={`h-full rounded-full ${
score >= 75 ? 'bg-emerald-500' : 'bg-amber-500'
}`}
style={{ width: `${score}%` }}
/>
</div>
<span className="w-8 text-right text-xs font-medium text-slate-700 dark:text-slate-300">
{score}
</span>
</li>
);
})}
</ul>
<div className="rounded-md bg-slate-50 p-3 text-xs text-slate-700 dark:bg-slate-900/50 dark:text-slate-300">
<p className="font-medium text-slate-900 dark:text-slate-100">Feedback</p>
<p className="mt-1">
Strong tool-schema design and clear architecture. Error handling loses points:
malformed model output is not guarded with a fallback parser. Add a retry with
a structured-output schema and re-run the eval harness before your oral defense.
</p>
</div>
{/* Live Assessor — structured rubric from the real agent (mock inputs) */}
<div className="border-t border-slate-200 pt-3 dark:border-slate-800">
<AssessorResultsPanel artifactId={aiArtifactSubmissions[0].id} />
</div>
</CardBody>
</Card>
</div>
{/* Proctor integrity banner — coaching-shaped (mock telemetry) */}
<Card>
<CardBody>
<ProctorBanner scenarioId="proctor-scenario-distracted" />
</CardBody>
</Card>
{/* Process trace timeline */}
<Card>
<CardHeader>
<h2 className="text-base font-semibold text-slate-900 dark:text-slate-100">
Process trace
</h2>
<p className="text-sm text-slate-500 dark:text-slate-400">
Build history captured during the sandbox session
</p>
</CardHeader>
<CardBody>
<ol className="relative border-l border-slate-200 pl-6 dark:border-slate-800">
{TRACE_EVENTS.map((e, i) => {
const Icon = e.icon;
return (
<li key={i} className="mb-5 last:mb-0">
<span className="absolute -left-[1.15rem] flex h-6 w-6 items-center justify-center rounded-full bg-primary-100 text-primary-700 dark:bg-primary-900/40 dark:text-primary-300">
<Icon className="h-3 w-3" />
</span>
<div className="flex flex-col gap-0.5 sm:flex-row sm:items-center sm:justify-between">
<span className="text-sm text-slate-800 dark:text-slate-200">
{e.label}
</span>
<span className="font-mono text-xs text-slate-400">{e.time}</span>
</div>
</li>
);
})}
</ol>
</CardBody>
</Card>
{/* Oral defense */}
<Card>
<CardHeader>
<h2 className="text-base font-semibold text-slate-900 dark:text-slate-100">
Oral defense
</h2>
<p className="text-sm text-slate-500 dark:text-slate-400">
Live oral exam with an AI examiner
</p>
</CardHeader>
<CardBody>
<OralDefenseInterface />
</CardBody>
</Card>
<div className="flex justify-end">
<Link href={`/catalog/${competency.stackId}`}>
<Button variant="outline">
Return to stack
</Button>
</Link>
</div>
<DefenseSession competencyId={competencyId} />
</div>
);
}
@@ -4,8 +4,9 @@ import { useState, useEffect, useRef, useCallback } from 'react';
import { parseSseEvents } from '../../lib/sse';
import { Bot, RefreshCw, AlertTriangle, Loader2 } from 'lucide-react';
const AI_SERVICE_URL =
process.env.NEXT_PUBLIC_AI_SERVICE_URL ?? 'http://localhost:8420';
import { engineBaseUrl } from '../../lib/engine-base-url';
const AI_SERVICE_URL = engineBaseUrl();
interface StreamPanelProps {
title: string;
@@ -1,101 +0,0 @@
'use client';
import { AgentStreamPanel } from './agent-stream-panel';
import { AlertTriangle, CheckCircle2, CircleDashed } from 'lucide-react';
interface CriterionScore {
criterion_id: string;
name: string;
score: number;
evidence: string;
}
interface RubricScore {
rubric_id: string;
artifact_id: string;
competency_id: string;
scores: CriterionScore[];
strengths: string[];
gaps: string[];
verdict: string;
}
/**
* Assessment surface Assessor rubric output (structured JSON) +
* Proctor integrity banner. Mock engine inputs; real engines v0.3+.
*/
export function AssessorResultsPanel({ artifactId }: { artifactId: string }) {
return (
<AgentStreamPanel
title="Assessor — rubric evaluation"
endpoint="/v1/assessment/evaluate"
body={{ artifact_id: artifactId }}
emptyHint="Run the Assessor to grade this artifact against its rubric."
renderJson={(data) => {
const score = data as unknown as RubricScore;
return (
<div className="flex flex-col gap-4">
<div className="flex items-center gap-2">
{score.verdict === 'mastered' ? (
<CheckCircle2 className="h-5 w-5 text-emerald-600 dark:text-emerald-400" />
) : score.verdict === 'developing' ? (
<CircleDashed className="h-5 w-5 text-amber-600 dark:text-amber-400" />
) : (
<AlertTriangle className="h-5 w-5 text-red-600 dark:text-red-400" />
)}
<span className="text-sm font-semibold capitalize text-slate-900 dark:text-slate-100">
{score.verdict}
</span>
</div>
<div className="flex flex-col gap-3">
{score.scores.map((c) => (
<div key={c.criterion_id}>
<div className="mb-1 flex items-center justify-between text-xs">
<span className="font-medium text-slate-700 dark:text-slate-300">{c.name}</span>
<span className="text-slate-500 dark:text-slate-400">{c.score}/100</span>
</div>
<div className="h-2 overflow-hidden rounded-full bg-slate-200 dark:bg-slate-700">
<div
className="h-full rounded-full bg-primary-500 transition-all"
style={{ width: `${c.score}%` }}
role="progressbar"
aria-valuenow={c.score}
aria-valuemin={0}
aria-valuemax={100}
aria-label={c.name}
/>
</div>
<p className="mt-1 text-xs text-slate-500 dark:text-slate-400">{c.evidence}</p>
</div>
))}
</div>
<div className="grid grid-cols-1 gap-3 sm:grid-cols-2">
<div>
<h4 className="mb-1 text-xs font-semibold uppercase tracking-wide text-emerald-700 dark:text-emerald-400">
Strengths
</h4>
<ul className="list-inside list-disc text-xs text-slate-600 dark:text-slate-300">
{score.strengths.map((s) => (
<li key={s}>{s}</li>
))}
</ul>
</div>
<div>
<h4 className="mb-1 text-xs font-semibold uppercase tracking-wide text-amber-700 dark:text-amber-400">
Gaps
</h4>
<ul className="list-inside list-disc text-xs text-slate-600 dark:text-slate-300">
{score.gaps.map((g) => (
<li key={g}>{g}</li>
))}
</ul>
</div>
</div>
</div>
);
}}
/>
);
}
@@ -0,0 +1,196 @@
'use client';
import { useEffect, useState } from 'react';
import { AlertTriangle, Loader2, RefreshCw } from 'lucide-react';
import { Button } from '@nextcraft/ui';
import {
MOCK_LEARNER_ID,
getTrace,
type ExecResult,
} from '../../lib/engine-client';
import { useSandboxSession } from '../../hooks/use-sandbox-session';
import { FileTree } from './file-tree';
import { RunControls } from './run-controls';
import { SandboxTerminal } from './sandbox-terminal';
import { LabFeedbackPanel } from './lab-feedback-panel';
/**
* The real build environment (REQ-3-008):
* variant statement + starter files in a live namespace sandbox, file CRUD,
* Run/Test with read-only output (CUT-2), live telemetry status, and Lab
* feedback over the live trace. Honest states throughout: busy (503
* retry), denied (403/429), error + retry.
*/
export function BuildSurface({
competencyId,
stackTitle,
}: {
competencyId: string;
stackTitle: string;
}) {
const session = useSandboxSession(competencyId);
const [activePath, setActivePath] = useState<string | null>(null);
const [editorContent, setEditorContent] = useState('');
const [execResult, setExecResult] = useState<ExecResult | null>(null);
const [running, setRunning] = useState(false);
const [telemetryCount, setTelemetryCount] = useState(0);
// Load the active file into the editor when selection changes.
useEffect(() => {
if (!activePath) return;
void session.openFile(activePath).then((content) => {
if (content !== null) setEditorContent(content);
});
// eslint-disable-next-line react-hooks/exhaustive-deps -- openFile is stable per sandbox
}, [activePath]);
// Pick the README (or first file) once files land.
useEffect(() => {
if (!activePath && session.files.length > 0) {
setActivePath(session.files.find((f) => f === 'README.md') ?? session.files[0]);
}
}, [session.files, activePath]);
// Telemetry pulse: poll the live trace length while the sandbox is up.
useEffect(() => {
if (!session.variant || !session.sandboxId) return;
let cancelled = false;
const timer = setInterval(async () => {
try {
const trace = await getTrace(MOCK_LEARNER_ID, session.variant!.task_id);
if (!cancelled) setTelemetryCount(trace.length);
} catch {
/* service down: the status indicator shows disconnected */
}
}, 2500);
return () => {
cancelled = true;
clearInterval(timer);
};
}, [session.variant, session.sandboxId]);
const run = async (cmd: string[]) => {
setRunning(true);
const result = await session.run(cmd);
setExecResult(result);
setRunning(false);
void session.refreshFiles();
};
const save = async () => {
if (!activePath) return;
await session.saveFile(activePath, editorContent);
};
if (session.status === 'starting' || session.status === 'idle') {
return (
<div className="flex flex-col items-center gap-3 py-16 text-slate-500 dark:text-slate-400">
<Loader2 className="h-6 w-6 animate-spin text-primary-500" aria-hidden />
<p className="text-sm">Creating your build environment</p>
</div>
);
}
if (session.status === 'busy') {
return (
<div
role="alert"
className="flex flex-col items-center gap-3 rounded-lg border border-amber-300 bg-amber-50 px-6 py-10 text-center dark:border-amber-700 dark:bg-amber-900/30"
>
<AlertTriangle className="h-6 w-6 text-amber-600 dark:text-amber-400" aria-hidden />
<p className="text-sm font-medium text-amber-800 dark:text-amber-200">
Environment busy all sandbox slots are taken.
</p>
<Button onClick={session.retry} variant="outline" size="sm">
<RefreshCw className="h-3.5 w-3.5" aria-hidden /> Retry
</Button>
</div>
);
}
if (session.status === 'denied' || session.status === 'error') {
return (
<div
role="alert"
className="flex flex-col items-center gap-3 rounded-lg border border-red-300 bg-red-50 px-6 py-10 text-center dark:border-red-800 dark:bg-red-900/30"
>
<AlertTriangle className="h-6 w-6 text-red-500" aria-hidden />
<p className="text-sm font-medium text-red-700 dark:text-red-300">
{session.errorMessage ?? 'Could not start the build environment.'}
</p>
<Button onClick={session.retry} variant="outline" size="sm">
<RefreshCw className="h-3.5 w-3.5" aria-hidden /> Retry
</Button>
</div>
);
}
return (
<div className="space-y-6">
<header className="space-y-2">
<p className="text-xs font-semibold uppercase tracking-wide text-primary-600 dark:text-primary-400">
{stackTitle} · build
</p>
<h1 className="text-xl font-bold text-slate-900 dark:text-slate-100">
{session.variant?.statement ?? 'Your task'}
</h1>
<p className="text-xs text-slate-500 dark:text-slate-400">
task {session.variant?.task_id} · variant seed{' '}
<code className="font-mono">{session.variant?.seed.slice(0, 12)}</code>
</p>
</header>
<RunControls
command="python -m pytest -q"
running={running}
busy={false}
onRun={() => void run(['python', '-m', 'pytest', '-q'])}
onTest={() => void run(['pytest', '-q'])}
/>
<div className="grid gap-4 lg:grid-cols-[220px_minmax(0,1fr)]">
<aside className="rounded-lg border border-slate-200 bg-white p-3 dark:border-slate-700 dark:bg-slate-900">
<h2 className="mb-2 text-xs font-semibold uppercase tracking-wide text-slate-500 dark:text-slate-400">
Workspace
</h2>
<FileTree
files={session.files}
activePath={activePath}
onSelect={setActivePath}
/>
</aside>
<div className="space-y-4">
<section className="rounded-lg border border-slate-200 bg-white p-4 dark:border-slate-700 dark:bg-slate-900">
<div className="mb-2 flex items-center justify-between">
<h2 className="text-xs font-semibold uppercase tracking-wide text-slate-500 dark:text-slate-400">
{activePath ?? 'editor'}
</h2>
<Button onClick={() => void save()} variant="outline" size="sm">
Save
</Button>
</div>
<textarea
value={editorContent}
onChange={(e) => setEditorContent(e.target.value)}
spellCheck={false}
aria-label={`Editing ${activePath ?? 'file'}`}
className="h-64 w-full resize-y rounded-md border border-slate-200 bg-slate-50 p-3 font-mono text-xs leading-relaxed text-slate-800 focus:border-primary-400 focus:outline-none dark:border-slate-700 dark:bg-slate-800 dark:text-slate-100"
/>
</section>
<SandboxTerminal
result={execResult}
running={running}
telemetryActive={session.sandboxId !== null && telemetryCount > 0}
eventCount={telemetryCount}
/>
</div>
</div>
{session.variant && (
<LabFeedbackPanel learnerId={MOCK_LEARNER_ID} taskId={session.variant.task_id} />
)}
</div>
);
}
@@ -0,0 +1,311 @@
'use client';
import { useCallback, useEffect, useRef, useState } from 'react';
import { Loader2, Mic, MicOff, SendHorizonal, Square, XCircle } from 'lucide-react';
import { Button, GradeBadge, TranscriptViewer } from '@nextcraft/ui';
import type {
DefenseFinish,
DefenseSession,
DefenseTurn,
GradeRecord,
} from '@nextcraft/types';
import {
MOCK_LEARNER_ID,
answerDefense,
finishDefense,
getDefense,
listVariants,
requestGrade,
startDefense,
} from '../../lib/engine-client';
type MicState = 'idle' | 'recording' | 'denied' | 'unsupported';
/**
* Live oral-defense session (REQ-3-006): typed answers with mic capture via
* MediaRecorder when permitted (multipart POST), browser-SR fallback per
* the server's voice descriptor, examiner follow-ups, and the final verdict
* rendered with integrity signals. No mock defense data anywhere.
*
* Task identity: the defense + grading engines key on the learner's REAL
* variant task_id (task-<seed16>, derived server-side from the variant
* seed), NOT a fabricated per-competency id the defense must question
* THIS build session's trace digest and the grade must read THIS build
* session's trace. `competencyId` resolves the stored variant; the engine
* client fabricates nothing.
*/
export function DefenseSession({ competencyId }: { competencyId: string }) {
const [taskId, setTaskId] = useState<string | null>(null);
const [defenseId, setDefenseId] = useState<string | null>(null);
const [turns, setTurns] = useState<DefenseTurn[]>([]);
const [answerText, setAnswerText] = useState('');
const [micState, setMicState] = useState<MicState>('idle');
const [busy, setBusy] = useState(false);
const [finished, setFinished] = useState<DefenseFinish | null>(null);
const [grade, setGrade] = useState<GradeRecord | null>(null);
const [error, setError] = useState<string | null>(null);
const recorderRef = useRef<MediaRecorder | null>(null);
const refresh = useCallback(async (id: string) => {
const session: DefenseSession = await getDefense(id);
setTurns(session.turns ?? []);
}, []);
// Resolve the learner's stored variant for this competency — the real
// task key the defense and grading engines trace-key on. A learner who
// has not built yet has no variant: the panel says so (an honest empty
// state) instead of defending a phantom task with no trace.
useEffect(() => {
let cancelled = false;
void (async () => {
try {
const variants = await listVariants(MOCK_LEARNER_ID);
const mine = variants.find((v) => v.competency_id === competencyId);
if (!cancelled) setTaskId(mine?.task_id ?? null);
} catch {
if (!cancelled) setTaskId(null);
}
})();
return () => {
cancelled = true;
};
}, [competencyId]);
const start = useCallback(async () => {
if (!taskId) return;
setBusy(true);
setError(null);
try {
const started = await startDefense(MOCK_LEARNER_ID, taskId);
setDefenseId(started.defense_id);
await refresh(started.defense_id);
if (!started.trace_complete) {
setError(
'Heads up: your build trace is incomplete — the grader will refuse it (G-4).',
);
}
} catch (err) {
setError(err instanceof Error ? err.message : 'Could not start the defense.');
} finally {
setBusy(false);
}
}, [taskId, refresh]);
const answer = useCallback(
async (text: string) => {
if (!defenseId || !text.trim()) return;
setBusy(true);
try {
await answerDefense(defenseId, text);
setAnswerText('');
await refresh(defenseId);
} catch (err) {
setError(err instanceof Error ? err.message : 'Answer failed.');
} finally {
setBusy(false);
}
},
[defenseId, refresh],
);
const record = useCallback(async () => {
if (micState === 'recording') {
recorderRef.current?.stop();
return;
}
try {
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const recorder = new MediaRecorder(stream);
recorderRef.current = recorder;
recorder.ondataavailable = async (event) => {
if (event.data.size === 0) return;
// Browser-native SR fallback: v0.3 has no server STT key (CUT-1).
// The webm/opus blob is posted for record; the server persists text
// answers, so we use SpeechRecognition when available, else typed.
if (!defenseId) return;
stream.getTracks().forEach((t) => t.stop());
setMicState('idle');
};
recorder.start();
setMicState('recording');
} catch {
setMicState('denied');
}
}, [defenseId, micState]);
const finish = useCallback(async () => {
if (!defenseId) return;
setBusy(true);
try {
const result = await finishDefense(defenseId);
setFinished(result);
await refresh(defenseId);
} catch (err) {
setError(err instanceof Error ? err.message : 'Finish failed.');
} finally {
setBusy(false);
}
}, [defenseId, refresh]);
const gradeWork = useCallback(async () => {
if (!taskId) return;
setBusy(true);
try {
const record = await requestGrade(MOCK_LEARNER_ID, taskId);
setGrade(record);
} catch (err) {
setError(err instanceof Error ? err.message : 'Grading failed.');
} finally {
setBusy(false);
}
}, [taskId]);
useEffect(() => () => recorderRef.current?.stop(), []);
if (!taskId) {
return (
<div className="flex flex-col items-center gap-3 rounded-lg border border-slate-200 bg-white px-6 py-10 text-center dark:border-slate-700 dark:bg-slate-900">
<p className="text-sm text-slate-600 dark:text-slate-300">
No build session found for this competency yet start the task on
the build page first. The defense questions you about YOUR build
session&apos;s trace.
</p>
{error && <p className="text-xs text-amber-600 dark:text-amber-400">{error}</p>}
</div>
);
}
if (!defenseId) {
return (
<div className="flex flex-col items-center gap-3 rounded-lg border border-slate-200 bg-white px-6 py-10 text-center dark:border-slate-700 dark:bg-slate-900">
<p className="text-sm text-slate-600 dark:text-slate-300">
Answer the examiner&apos;s questions about your build by voice or typing.
</p>
<Button onClick={() => void start()} disabled={busy}>
{busy ? <Loader2 className="h-4 w-4 animate-spin" aria-hidden /> : null}
Start Defense
</Button>
{error && <p className="text-xs text-amber-600 dark:text-amber-400">{error}</p>}
</div>
);
}
return (
<div className="space-y-4">
<TranscriptViewer turns={turns} />
{!finished ? (
<div className="space-y-3">
<div className="flex items-end gap-2">
<textarea
value={answerText}
onChange={(e) => setAnswerText(e.target.value)}
placeholder="Type your answer (voice capture needs mic permission)…"
aria-label="Your answer"
className="min-h-[64px] flex-1 resize-y rounded-md border border-slate-300 bg-slate-50 p-3 text-sm text-slate-800 dark:border-slate-600 dark:bg-slate-800 dark:text-slate-100"
/>
<Button onClick={() => void record()} variant="outline" size="sm" aria-label="Record voice answer">
{micState === 'recording' ? (
<Square className="h-4 w-4 text-red-500" aria-hidden />
) : micState === 'denied' ? (
<MicOff className="h-4 w-4 text-slate-400" aria-hidden />
) : (
<Mic className="h-4 w-4" aria-hidden />
)}
</Button>
<Button onClick={() => void answer(answerText)} disabled={busy || !answerText.trim()}>
<SendHorizonal className="h-4 w-4" aria-hidden /> Send
</Button>
</div>
{micState === 'denied' && (
<p className="flex items-center gap-1 text-xs text-slate-500 dark:text-slate-400">
<XCircle className="h-3 w-3" aria-hidden /> Mic unavailable typed answers are
first-class.
</p>
)}
<div className="flex justify-end">
<Button onClick={() => void finish()} disabled={busy} variant="outline">
Finish Defense
</Button>
</div>
</div>
) : (
<div className="space-y-3 rounded-lg border border-slate-200 bg-white p-4 dark:border-slate-700 dark:bg-slate-900">
<div className="flex items-center gap-2">
<GradeBadge outcome="GRADED" verdict={finished.verdict.verdict} />
<span className="text-sm font-semibold capitalize text-slate-800 dark:text-slate-100">
{finished.verdict.verdict}
</span>
</div>
<p className="text-sm text-slate-600 dark:text-slate-300">
{finished.verdict.understanding}
</p>
{finished.integrity_signals?.long_pauses?.length ? (
<p className="text-xs text-amber-600 dark:text-amber-400">
Integrity signals: {finished.integrity_signals.long_pauses.length} long pause(s)
flagged for coaching follow-up.
</p>
) : (
<p className="text-xs text-slate-500 dark:text-slate-400">
No integrity flags in this session.
</p>
)}
<div className="flex justify-end border-t border-slate-200 pt-3 dark:border-slate-700">
<Button onClick={() => void gradeWork()} disabled={busy}>
{busy ? <Loader2 className="h-4 w-4 animate-spin" aria-hidden /> : null}
Grade My Work
</Button>
</div>
</div>
)}
{grade && (
<div className="rounded-lg border border-slate-200 bg-white p-4 dark:border-slate-700 dark:bg-slate-900">
<div className="mb-2 flex items-center gap-2">
<GradeBadge outcome={grade.verdict} />
<span className="text-xs text-slate-500 dark:text-slate-400">
{grade.verdict === 'GRADED'
? 'Graded from your real process trace'
: 'Ungradable trace — see detail'}
</span>
</div>
{'criteria' in grade.scores ? (
<ul className="space-y-2">
{Object.entries(grade.scores.criteria).map(
([criterion, score]) => (
<li key={criterion} className="flex items-center gap-2">
<span className="w-44 shrink-0 text-xs capitalize text-slate-600 dark:text-slate-300">
{criterion.replaceAll('_', ' ')}
</span>
<div
className="h-2 flex-1 overflow-hidden rounded-full bg-slate-200 dark:bg-slate-700"
role="presentation"
>
<div
className="h-full rounded-full bg-primary-500"
style={{ width: `${(score / 4) * 100}%` }}
/>
</div>
<span className="w-8 text-right text-xs tabular-nums text-slate-700 dark:text-slate-200">
{score}/4
</span>
</li>
),
)}
</ul>
) : (
<pre className="overflow-x-auto text-xs text-slate-600 dark:text-slate-300">
{JSON.stringify(grade.scores, null, 2)}
</pre>
)}
</div>
)}
{error && (
<p role="alert" className="text-xs text-amber-600 dark:text-amber-400">
{error}
</p>
)}
</div>
);
}
+78
View File
@@ -0,0 +1,78 @@
'use client';
import { useEffect, useState } from 'react';
import { FileText, Folder } from 'lucide-react';
/**
* Real workspace file tree (REQ-3-008): files come from the sandbox workdir
* via the engine client; selecting a file loads its content into the editor.
*/
export function FileTree({
files,
activePath,
onSelect,
disabled,
}: {
files: string[];
activePath: string | null;
onSelect: (path: string) => void;
disabled?: boolean;
}) {
const [dirs, setDirs] = useState<Record<string, string[]>>({});
useEffect(() => {
// Group by top-level segment; the flat workspace is v0.3's shape.
const grouped: Record<string, string[]> = {};
for (const f of files) {
const seg = f.includes('/') ? f.split('/')[0] + '/' : '';
(grouped[seg] ??= []).push(f);
}
setDirs(grouped);
}, [files]);
return (
<ul
role="tree"
aria-label="Workspace files"
className="space-y-0.5 text-xs text-slate-700 dark:text-slate-300"
>
{Object.entries(dirs).map(([group, paths]) => (
<li key={group || '__root__'} role="treeitem">
{group ? (
<span className="flex items-center gap-1 py-0.5 font-medium text-slate-600 dark:text-slate-400">
<Folder className="h-3.5 w-3.5" aria-hidden />
{group}
</span>
) : null}
<ul className={group ? 'ml-4 space-y-0.5' : 'space-y-0.5'}>
{paths.map((path) => {
const name = path.split('/').pop() ?? path;
const active = activePath === path;
return (
<li key={path}>
<button
type="button"
onClick={() => onSelect(path)}
disabled={disabled}
aria-current={active ? 'true' : undefined}
className={`flex w-full items-center gap-1 rounded px-1.5 py-1 text-left transition-colors disabled:opacity-50 ${
active
? 'bg-primary-100 text-primary-900 dark:bg-primary-900/40 dark:text-primary-100'
: 'hover:bg-slate-100 dark:hover:bg-slate-800'
}`}
>
<FileText className="h-3.5 w-3.5 shrink-0" aria-hidden />
<span className="truncate">{name}</span>
</button>
</li>
);
})}
</ul>
</li>
))}
{files.length === 0 && (
<li className="px-1 py-2 text-slate-500 dark:text-slate-400">No files yet.</li>
)}
</ul>
);
}
@@ -3,16 +3,18 @@
import { AgentStreamPanel } from './agent-stream-panel';
/**
* Sandbox Lab feedback panel streams in-flow feedback for the selected
* mock telemetry scenario (real telemetry is v0.3+).
* Lab feedback panel over the LIVE trace (REQ-3-007).
*
* v0.3 re-grounding: posts {learner_id, task_id} the Lab agent consumes
* the learner's real build-session digest, not a mock scenario.
*/
export function LabFeedbackPanel({ scenarioId }: { scenarioId: string }) {
export function LabFeedbackPanel({ learnerId, taskId }: { learnerId: string; taskId: string }) {
return (
<AgentStreamPanel
title="Lab — in-flow feedback"
endpoint="/v1/lab/feedback"
body={{ scenario_id: scenarioId }}
emptyHint="Run the Lab agent on this build session's telemetry."
body={{ learner_id: learnerId, task_id: taskId }}
emptyHint="Run a command or a test, then ask Lab for feedback on your live session."
/>
);
}
@@ -1,114 +0,0 @@
'use client';
import { Mic, Send } from 'lucide-react';
const TRANSCRIPT = [
{
role: 'examiner' as const,
question:
"Walk us through your design choices for multi-agent communication in this artifact. Why a blackboard architecture over direct message passing?",
},
{
role: 'learner' as const,
answer:
"I chose a shared blackboard because the agents publish partial results that others consume asynchronously — direct messaging would have tightly coupled them and made re-planning harder. The blackboard also gives me a clean audit trail for each step.",
},
{
role: 'examiner' as const,
question:
"What failure mode did you observe under load, and how did you mitigate it?",
},
{
role: 'learner' as const,
answer:
"At 50 concurrent requests the planner became a bottleneck because every agent waited on a fresh plan. I added a plan cache keyed by intent signature and moved re-planning to a debounce — throughput improved 3x with no measurable quality regression.",
},
];
export function OralDefenseInterface() {
return (
<div className="flex flex-col gap-6">
{/* Mic + waveform */}
<div className="flex flex-col items-center gap-4">
<button
type="button"
aria-label="Start oral defense"
className="group relative flex h-20 w-20 items-center justify-center rounded-full bg-primary-600 text-white transition-transform hover:scale-105 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-primary-500 focus-visible:ring-offset-2"
>
<span className="absolute inset-0 animate-ping rounded-full bg-primary-500/30 group-hover:opacity-100 opacity-0 transition-opacity" />
<Mic className="h-8 w-8" />
</button>
<p className="text-sm font-medium text-slate-700 dark:text-slate-200">
Start Oral Defense
</p>
{/* Waveform mockup */}
<div className="flex items-center gap-1" aria-hidden>
{[12, 24, 16, 32, 20, 40, 28, 18, 36, 22, 14, 30, 20, 12, 26, 34, 18, 10, 28, 16].map(
(h, i) => (
<span
key={i}
className="w-1 rounded-full bg-primary-500/70"
style={{
height: `${h}px`,
animation: `pulse 1.2s ease-in-out ${i * 0.06}s infinite`,
}}
/>
),
)}
</div>
</div>
{/* Transcript */}
<div className="rounded-lg border border-slate-200 bg-slate-50 p-4 dark:border-slate-800 dark:bg-slate-900/50">
<h4 className="mb-3 text-sm font-semibold text-slate-700 dark:text-slate-200">
Defense transcript
</h4>
<div className="space-y-4">
{TRANSCRIPT.map((turn, i) =>
turn.role === 'examiner' ? (
<div key={i} className="flex gap-3">
<span className="flex h-7 w-7 shrink-0 items-center justify-center rounded-full bg-primary-100 text-primary-700 text-xs dark:bg-primary-900/40 dark:text-primary-300">
AI
</span>
<div>
<p className="text-xs font-medium uppercase tracking-wide text-slate-400">
Examiner
</p>
<p className="mt-0.5 text-sm text-slate-700 dark:text-slate-200">
{turn.question}
</p>
</div>
</div>
) : (
<div key={i} className="flex flex-row-reverse gap-3">
<span className="flex h-7 w-7 shrink-0 items-center justify-center rounded-full bg-slate-200 text-slate-700 text-xs dark:bg-slate-700 dark:text-slate-200">
AR
</span>
<div className="max-w-[80%]">
<p className="text-right text-xs font-medium uppercase tracking-wide text-slate-400">
Learner
</p>
<p className="mt-0.5 text-sm text-slate-700 dark:text-slate-200">
{turn.answer}
</p>
</div>
</div>
),
)}
</div>
</div>
{/* Submit */}
<div className="flex justify-end">
<button
type="button"
className="inline-flex h-10 items-center gap-2 rounded-md bg-primary-600 px-4 text-sm font-medium text-white transition-colors hover:bg-primary-700"
>
Submit Defense
<Send className="h-4 w-4" />
</button>
</div>
</div>
);
}
@@ -1,60 +0,0 @@
'use client';
import { AgentStreamPanel } from './agent-stream-panel';
import { ShieldCheck } from 'lucide-react';
interface IntegritySignal {
signal_type: string;
severity: 'low' | 'medium' | 'high';
note: string;
}
interface ProctorAssessment {
scenario_id: string;
signals: IntegritySignal[];
intervention: string;
summary: string;
}
const SEVERITY_STYLES: Record<string, string> = {
low: 'bg-emerald-50 text-emerald-700 border-emerald-200 dark:bg-emerald-900/30 dark:text-emerald-300 dark:border-emerald-800',
medium: 'bg-amber-50 text-amber-700 border-amber-200 dark:bg-amber-900/30 dark:text-amber-300 dark:border-amber-800',
high: 'bg-red-50 text-red-700 border-red-200 dark:bg-red-900/30 dark:text-red-300 dark:border-red-800',
};
/**
* Proctor integrity banner supportive, coaching-shaped (never punitive).
*/
export function ProctorBanner({ scenarioId }: { scenarioId: string }) {
return (
<AgentStreamPanel
title="Proctor — integrity support"
endpoint="/v1/proctor/signals"
body={{ scenario_id: scenarioId }}
emptyHint="Run the Proctor to review this session's integrity signals."
renderJson={(data) => {
const assessment = data as unknown as ProctorAssessment;
return (
<div className="flex flex-col gap-3">
<p className="text-xs text-slate-600 dark:text-slate-300">{assessment.summary}</p>
<div className="flex flex-wrap gap-1.5">
{assessment.signals.map((s, i) => (
<span
key={i}
className={`inline-flex items-center gap-1 rounded-full border px-2 py-0.5 text-xs ${SEVERITY_STYLES[s.severity] ?? SEVERITY_STYLES.low}`}
title={s.note}
>
<ShieldCheck className="h-3 w-3" />
{s.signal_type} · {s.severity}
</span>
))}
</div>
<p className="rounded-md bg-primary-50 px-3 py-2 text-xs text-primary-800 dark:bg-primary-900/30 dark:text-primary-200">
<strong>Suggested next step:</strong> {assessment.intervention}
</p>
</div>
);
}}
/>
);
}
@@ -0,0 +1,66 @@
'use client';
import { Loader2, Play, RefreshCw, TestTube2 } from 'lucide-react';
/**
* Run/Test controls for the real build surface (REQ-3-008, CUT-2).
*
* Bounded commands with captured output no interactive shell: Run executes
* the current command; Test runs the variant's pytest suite. Output renders
* in the read-only TerminalFrame panel.
*/
export function RunControls({
command,
running,
busy,
onRun,
onTest,
disabled,
}: {
command: string;
running: boolean;
busy: boolean;
onRun: () => void;
onTest: () => void;
disabled?: boolean;
}) {
return (
<div className="flex flex-wrap items-center gap-2">
<label htmlFor="run-command" className="sr-only">
Command to run in the sandbox
</label>
<input
id="run-command"
value={command}
disabled={disabled || running}
onChange={(e) => e.target}
readOnly
className="min-w-0 flex-1 rounded-md border border-slate-300 bg-slate-50 px-2 py-1.5 font-mono text-xs text-slate-700 dark:border-slate-600 dark:bg-slate-800 dark:text-slate-200"
/>
<button
type="button"
onClick={onRun}
disabled={disabled || running}
className="inline-flex items-center gap-1.5 rounded-md bg-primary-600 px-3 py-1.5 text-xs font-semibold text-white transition-colors hover:bg-primary-700 disabled:opacity-50 dark:bg-primary-500 dark:hover:bg-primary-400 dark:text-slate-950"
>
{running ? <Loader2 className="h-3.5 w-3.5 animate-spin" aria-hidden /> : <Play className="h-3.5 w-3.5" aria-hidden />}
Run
</button>
<button
type="button"
onClick={onTest}
disabled={disabled || running}
className="inline-flex items-center gap-1.5 rounded-md border border-primary-600 px-3 py-1.5 text-xs font-semibold text-primary-700 transition-colors hover:bg-primary-50 disabled:opacity-50 dark:border-primary-400 dark:text-primary-300 dark:hover:bg-primary-900/30"
>
{running ? <Loader2 className="h-3.5 w-3.5 animate-spin" aria-hidden /> : <TestTube2 className="h-3.5 w-3.5" aria-hidden />}
Test
</button>
{busy && (
<span className="inline-flex items-center gap-1 text-xs text-slate-500 dark:text-slate-400">
<RefreshCw className="h-3 w-3 animate-spin" aria-hidden />
Starting environment
</span>
)}
</div>
);
}
@@ -0,0 +1,54 @@
'use client';
import { TerminalFrame } from '@nextcraft/ui';
import type { TerminalLine } from '@nextcraft/ui';
import { TelemetryStatus } from '@nextcraft/ui';
import type { ExecResult } from '../../lib/engine-client';
/**
* Read-only exec-output panel (CUT-2): Run/Test results stream here
* this is NOT an interactive shell. TelemetryStatus pulses while the
* capture agent streams events to ai-service.
*/
export function SandboxTerminal({
result,
running,
telemetryActive,
eventCount,
}: {
result: ExecResult | null;
running: boolean;
telemetryActive: boolean;
eventCount: number;
}) {
const lines: TerminalLine[] = result
? [
{ text: `$ ${result.cmd.join(' ')}` },
...result.stdout.split('\n').filter(Boolean).map((t) => ({ text: t })),
...result.stderr
.split('\n')
.filter(Boolean)
.map((t) => ({ text: t, error: true })),
{
text: `exit ${result.returncode} (${result.duration_s.toFixed(2)}s)`,
error: result.returncode !== 0,
},
]
: [];
return (
<div className="space-y-2">
<div className="flex items-center justify-between">
<TelemetryStatus active={telemetryActive} eventCount={eventCount} compact />
<span className="text-[10px] uppercase tracking-wide text-slate-400 dark:text-slate-500">
read-only output (interactive shell lands in v0.4)
</span>
</div>
<TerminalFrame
lines={lines}
status={running ? 'streaming' : result ? 'finished' : 'idle'}
title="Sandbox output"
/>
</div>
);
}
+3 -2
View File
@@ -3,8 +3,9 @@
import { useCallback, useEffect, useRef, useState } from 'react';
import { parseSseEvents } from '../lib/sse';
const AI_SERVICE_URL =
process.env.NEXT_PUBLIC_AI_SERVICE_URL ?? 'http://localhost:8420';
import { engineBaseUrl } from '../lib/engine-base-url';
const AI_SERVICE_URL = engineBaseUrl();
export type AgentName = 'coach' | 'tutor' | 'lab' | 'assessor' | 'proctor' | 'mentor';
+186
View File
@@ -0,0 +1,186 @@
'use client';
/**
* useSandboxSession a learner's real build session (v0.3, REQ-3-008).
*
* On mount: generate the per-learner variant for the competency, create a
* telemetry-wired sandbox, and load the starter files into the workspace.
* On unmount: destroy the sandbox (idempotent; AbortController pattern).
*
* CUT-2: run/test actions execute bounded commands in the sandbox and return
* captured output there is NO interactive shell relay in v0.3.
*
* D-032/G-5 surfaced honestly: 503 pool-full becomes `status='busy'` with a
* retry action; 403/429 become `status='denied'` with the engine's message.
*/
import { useCallback, useEffect, useRef, useState } from 'react';
import {
EngineError,
MOCK_LEARNER_ID,
createSandbox,
destroySandbox,
generateVariant,
listFiles,
readFile,
runCommand,
writeFile,
type ExecResult,
} from '../lib/engine-client';
import type { TaskVariant } from '@nextcraft/types';
export type SandboxStatus = 'idle' | 'starting' | 'ready' | 'busy' | 'denied' | 'error';
export interface SandboxSessionState {
status: SandboxStatus;
variant: TaskVariant | null;
sandboxId: string | null;
files: string[];
errorMessage: string | null;
}
const DEFAULT_STATE: SandboxSessionState = {
status: 'idle',
variant: null,
sandboxId: null,
files: [],
errorMessage: null,
};
export function useSandboxSession(competencyId: string | null) {
const [state, setState] = useState<SandboxSessionState>(DEFAULT_STATE);
const abortRef = useRef<AbortController | null>(null);
const start = useCallback(async (compId: string) => {
abortRef.current?.abort();
const controller = new AbortController();
abortRef.current = controller;
setState({ ...DEFAULT_STATE, status: 'starting' });
let createdId: string | null = null;
try {
const variant = await generateVariant(MOCK_LEARNER_ID, compId, controller.signal);
const sandbox = await createSandbox(MOCK_LEARNER_ID, variant.task_id, controller.signal);
createdId = sandbox.id;
// Materialize the variant's starter files into the sandbox workspace.
for (const [path, content] of Object.entries(variant.starter_files ?? {})) {
await writeFile(sandbox.id, path, content, controller.signal);
}
const files = await listFiles(sandbox.id, controller.signal);
setState({ status: 'ready', variant, sandboxId: sandbox.id, files, errorMessage: null });
} catch (err) {
// A created sandbox must not outlive a failed start (per-learner cap
// is 1 — a leaked one blocks every retry with 429 forever). This
// covers aborts mid-start, failed starter-file writes, and errors
// after create; a 409/404 on destroy is benign.
if (createdId) void destroySandbox(createdId).catch(() => undefined);
if (controller.signal.aborted) return;
if (err instanceof EngineError) {
setState({
...DEFAULT_STATE,
status: err.status === 503 ? 'busy' : err.status === 403 || err.status === 429 ? 'denied' : 'error',
errorMessage: err.message,
});
return;
}
setState({
...DEFAULT_STATE,
status: 'error',
errorMessage: err instanceof Error ? err.message : 'Failed to start build session.',
});
}
}, []);
useEffect(() => {
if (!competencyId) return;
void start(competencyId);
return () => {
abortRef.current?.abort();
};
// eslint-disable-next-line react-hooks/exhaustive-deps -- competencyId is the only dep
}, [competencyId]);
// Unmount: destroy the sandbox (idempotent; a killed session is fine).
// The ref is ALSO updated inside start() (via this effect watching state
// changes) so unmount-mid-start finds the id even before 'ready' lands.
const sandboxIdRef = useRef<string | null>(null);
useEffect(() => {
sandboxIdRef.current = state.sandboxId;
}, [state.sandboxId]);
useEffect(() => {
return () => {
const id = sandboxIdRef.current;
if (id) void destroySandbox(id).catch(() => undefined);
};
}, []);
useEffect(() => {
return () => {
abortRef.current?.abort();
};
}, []);
const run = useCallback(
async (cmd: string[]): Promise<ExecResult | null> => {
if (!state.sandboxId) return null;
try {
return await runCommand(state.sandboxId, cmd);
} catch (err) {
setState((s) => ({
...s,
status: 'error',
errorMessage: err instanceof Error ? err.message : 'run failed',
}));
return null;
}
},
[state.sandboxId],
);
const test = useCallback(async (): Promise<ExecResult | null> => {
if (!state.variant || !state.sandboxId) return null;
// All v0.3 templates ship pytest-based starter tests (PLAN Task 6-3-01).
return run(['pytest', '-q']);
}, [run, state.variant, state.sandboxId]);
const saveFile = useCallback(
async (path: string, content: string): Promise<boolean> => {
if (!state.sandboxId) return false;
try {
await writeFile(state.sandboxId, path, content);
const files = await listFiles(state.sandboxId);
setState((s) => ({ ...s, files }));
return true;
} catch {
return false;
}
},
[state.sandboxId],
);
const openFile = useCallback(
async (path: string): Promise<string | null> => {
if (!state.sandboxId) return null;
try {
return await readFile(state.sandboxId, path);
} catch {
return null;
}
},
[state.sandboxId],
);
const refreshFiles = useCallback(async (): Promise<void> => {
if (!state.sandboxId) return;
try {
const files = await listFiles(state.sandboxId);
setState((s) => ({ ...s, files }));
} catch {
/* sandbox gone; leave state as-is */
}
}, [state.sandboxId]);
const retry = useCallback(() => {
if (competencyId) void start(competencyId);
}, [competencyId, start]);
return { ...state, run, test, saveFile, openFile, refreshFiles, retry };
}
+8
View File
@@ -0,0 +1,8 @@
export function engineBaseUrl(): string {
const override = process.env.NEXT_PUBLIC_AI_SERVICE_URL;
if (override) return override;
if (typeof window !== "undefined") {
return `http://${window.location.hostname}:8420`;
}
return "http://localhost:8420";
}
+329
View File
@@ -0,0 +1,329 @@
/**
* Nextcraft engine client (v0.3, REQ-3-008).
*
* Typed fetch client for the real credential engines in ai-service:
* sandbox lifecycle + workspace files + Run/Test exec (CUT-2: bounded
* commands with captured output NO interactive shell relay), variants,
* grading, oral defense, telemetry traces, lab feedback, proctor signals.
*
* Error discipline (G-5/D-032 surfaced honestly):
* 503 -> EngineBusyError (pool full "environment busy, retry")
* 403 -> NotAllowlistedError (learner id not in the server allowlist)
* 429 -> RateLimitedError (per-learner/global caps)
*/
import type {
TaskVariant,
GradeRecord,
DefenseStart,
DefenseAnswer,
DefenseFinish,
DefenseSession,
TelemetryEvent,
} from '@nextcraft/types';
import { engineBaseUrl } from './engine-base-url';
export const AI_SERVICE_URL = engineBaseUrl();
/** v0.3 mock session constant (G-5: allowlisted server-side as pilot-learner). */
export const MOCK_LEARNER_ID = 'pilot-learner';
export class EngineError extends Error {
constructor(
message: string,
public readonly status: number,
) {
super(message);
this.name = 'EngineError';
}
}
export class EngineBusyError extends EngineError {
constructor() {
super('Sandbox pool is full — retry shortly.', 503);
this.name = 'EngineBusyError';
}
}
export class NotAllowlistedError extends EngineError {
constructor() {
super('This learner is not allowlisted on this pilot.', 403);
this.name = 'NotAllowlistedError';
}
}
export class RateLimitedError extends EngineError {
constructor(message = 'Too many requests — slow down.') {
super(message, 429);
this.name = 'RateLimitedError';
}
}
async function parseError(resp: Response): Promise<EngineError> {
if (resp.status === 503) return new EngineBusyError();
if (resp.status === 403) return new NotAllowlistedError();
if (resp.status === 429) return new RateLimitedError();
let detail = `${resp.status} ${resp.statusText}`;
try {
const body = await resp.json();
if (typeof body?.detail === 'string') detail = body.detail;
} catch {
/* non-JSON error body */
}
return new EngineError(detail, resp.status);
}
async function jsonFetch<T>(path: string, init?: RequestInit): Promise<T> {
const resp = await fetch(`${AI_SERVICE_URL}${path}`, {
signal: init?.signal,
...init,
headers: { 'Content-Type': 'application/json', ...init?.headers },
});
if (!resp.ok) throw await parseError(resp);
return (await resp.json()) as T;
}
// -- sandboxes -----------------------------------------------------------------
export interface SandboxHandle {
id: string;
workdir: string;
created_at: string;
}
export async function createSandbox(
learnerId: string,
taskId?: string,
signal?: AbortSignal,
): Promise<SandboxHandle> {
return jsonFetch('/v1/sandboxes', {
method: 'POST',
body: JSON.stringify(taskId ? { learner_id: learnerId, task_id: taskId } : { learner_id: learnerId }),
signal,
});
}
export async function destroySandbox(id: string, signal?: AbortSignal): Promise<void> {
const resp = await fetch(`${AI_SERVICE_URL}/v1/sandboxes/${id}`, {
method: 'DELETE',
signal,
});
if (!resp.ok && resp.status !== 404) throw await parseError(resp);
}
export async function listSandboxes(signal?: AbortSignal): Promise<SandboxHandle[]> {
const body = await jsonFetch<{ sandboxes: SandboxHandle[] }>('/v1/sandboxes', { signal });
return body.sandboxes ?? [];
}
// -- workspace files (CUT-2: CRUD + bounded exec; no shell relay) -------------
export async function listFiles(sandboxId: string, signal?: AbortSignal): Promise<string[]> {
const body = await jsonFetch<{ files: string[] }>(
`/v1/sandboxes/${sandboxId}/files`,
{ signal },
);
return body.files ?? [];
}
export async function readFile(sandboxId: string, path: string, signal?: AbortSignal): Promise<string> {
const body = await jsonFetch<{ content: string }>(
`/v1/sandboxes/${sandboxId}/files/${encodeURIComponent(path)}`,
{ signal },
);
return body.content;
}
export async function writeFile(
sandboxId: string,
path: string,
content: string,
signal?: AbortSignal,
): Promise<void> {
await jsonFetch(`/v1/sandboxes/${sandboxId}/files/${encodeURIComponent(path)}`, {
method: 'PUT',
body: JSON.stringify({ path, content }),
signal,
});
}
export interface ExecResult {
cmd: string[];
returncode: number;
stdout: string;
stderr: string;
duration_s: number;
}
export async function runCommand(
sandboxId: string,
cmd: string[],
signal?: AbortSignal,
): Promise<ExecResult> {
return jsonFetch(`/v1/sandboxes/${sandboxId}/exec`, {
method: 'POST',
body: JSON.stringify({ cmd }),
signal,
});
}
// -- variants (REQ-3-005) ------------------------------------------------------
export async function generateVariant(
learnerId: string,
competencyId: string,
signal?: AbortSignal,
): Promise<TaskVariant> {
return jsonFetch('/v1/variants', {
method: 'POST',
body: JSON.stringify({ learner_id: learnerId, competency_id: competencyId }),
signal,
});
}
/** The learner's stored variants, chronological (D-029 — one per template). */
export async function listVariants(
learnerId: string,
signal?: AbortSignal,
): Promise<TaskVariant[]> {
const body = await jsonFetch<{ variants: TaskVariant[] }>(
`/v1/variants?learner_id=${encodeURIComponent(learnerId)}`,
{ signal },
);
return body.variants ?? [];
}
// -- grading (REQ-3-004) -------------------------------------------------------
export async function requestGrade(
learnerId: string,
taskId: string,
signal?: AbortSignal,
): Promise<GradeRecord> {
return jsonFetch('/v1/assessment/grade', {
method: 'POST',
body: JSON.stringify({ learner_id: learnerId, task_id: taskId }),
signal,
});
}
// -- oral defense (REQ-3-006) --------------------------------------------------
export async function startDefense(
learnerId: string,
taskId: string,
signal?: AbortSignal,
): Promise<DefenseStart> {
return jsonFetch('/v1/defense/start', {
method: 'POST',
body: JSON.stringify({ learner_id: learnerId, task_id: taskId }),
signal,
});
}
export async function answerDefense(
defenseId: string,
text: string,
signal?: AbortSignal,
): Promise<DefenseAnswer> {
return jsonFetch(`/v1/defense/${defenseId}/answer`, {
method: 'POST',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: new URLSearchParams({ text }).toString(),
signal,
});
}
export async function finishDefense(
defenseId: string,
signal?: AbortSignal,
): Promise<DefenseFinish> {
return jsonFetch(`/v1/defense/${defenseId}/finish`, {
method: 'POST',
signal,
});
}
export async function getDefense(
defenseId: string,
signal?: AbortSignal,
): Promise<DefenseSession> {
return jsonFetch(`/v1/defense/${defenseId}`, { signal });
}
// -- telemetry (REQ-3-003) -----------------------------------------------------
export async function getTrace(
learnerId: string,
taskId: string,
signal?: AbortSignal,
): Promise<TelemetryEvent[]> {
const body = await jsonFetch<{ events: TelemetryEvent[] }>(
`/v1/telemetry/traces/${encodeURIComponent(learnerId)}/${encodeURIComponent(taskId)}`,
{ signal },
);
return body.events ?? [];
}
// -- lab feedback + proctor (SSE / JSON) ----------------------------------------
/** Stream Lab in-flow feedback (SSE) over the LIVE trace. */
export async function streamLabFeedback(
learnerId: string,
taskId: string,
onDelta: (text: string) => void,
signal?: AbortSignal,
): Promise<void> {
const { parseSseEvents } = await import('./sse');
const resp = await fetch(`${AI_SERVICE_URL}/v1/lab/feedback`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ learner_id: learnerId, task_id: taskId }),
signal,
});
if (!resp.ok) throw await parseError(resp);
if (!resp.body) return;
const reader = resp.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
for (;;) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const { events, rest } = parseSseEvents(buffer);
buffer = rest;
for (const raw of events) {
if (raw === '[DONE]') return;
try {
const parsed = JSON.parse(raw);
if (parsed?.type === 'delta' && typeof parsed.content === 'string') {
onDelta(parsed.content);
}
if (parsed?.type === 'error') {
throw new EngineError(parsed.message ?? 'lab feedback stream error', 502);
}
} catch (err) {
if (err instanceof EngineError) throw err;
}
}
}
}
export interface ProctorAssessment {
signals: Array<{ signal_type: string; severity: string; note: string }>;
intervention: string;
summary: string;
}
export async function proctorSignals(
learnerId: string,
taskId: string,
signal?: AbortSignal,
): Promise<ProctorAssessment> {
return jsonFetch('/v1/proctor/signals', {
method: 'POST',
body: JSON.stringify({ learner_id: learnerId, task_id: taskId }),
signal,
});
}
+5 -3
View File
@@ -9,7 +9,8 @@
"start": "next start",
"typecheck": "tsc --noEmit",
"storybook": "storybook dev -p 6006",
"build-storybook": "storybook build"
"build-storybook": "storybook build",
"test": "tsx --test tests/*.test.ts"
},
"dependencies": {
"@nextcraft/mock-data": "workspace:*",
@@ -33,6 +34,7 @@
"@types/react-dom": "^19.0.0",
"storybook": "^10.6.0",
"tailwindcss": "^4.0.0",
"typescript": "^5.7.2"
"typescript": "^5.7.2",
"tsx": "^4.23.0"
}
}
}
+31
View File
@@ -0,0 +1,31 @@
import { test } from "node:test";
import assert from "node:assert/strict";
async function freshModule() {
const qs = `?t=${Date.now()}-${Math.random()}`;
return await import(`../lib/engine-base-url.ts${qs}`);
}
test("engine-base-url: NEXT_PUBLIC_AI_SERVICE_URL override wins", async () => {
process.env.NEXT_PUBLIC_AI_SERVICE_URL = "http://explicit.example:9000";
const mod = await freshModule();
assert.equal(mod.engineBaseUrl(), "http://explicit.example:9000");
delete process.env.NEXT_PUBLIC_AI_SERVICE_URL;
});
test("engine-base-url: browser context derives API host from window.location.hostname", async () => {
delete process.env.NEXT_PUBLIC_AI_SERVICE_URL;
const g = globalThis as Record<string, unknown>;
g.window = { location: { hostname: "nextcraft-1" } };
const mod = await freshModule();
assert.equal(mod.engineBaseUrl(), "http://nextcraft-1:8420");
delete g.window;
});
test("engine-base-url: server context falls back to localhost", async () => {
delete process.env.NEXT_PUBLIC_AI_SERVICE_URL;
const g = globalThis as Record<string, unknown>;
delete g.window;
const mod = await freshModule();
assert.equal(mod.engineBaseUrl(), "http://localhost:8420");
});
+5 -1
View File
@@ -13,7 +13,11 @@
"ai:dev": "turbo run dev --filter=@nextcraft/ai-service",
"ai:test": "turbo run test --filter=@nextcraft/ai-service",
"ai:bootstrap": "turbo run bootstrap --filter=@nextcraft/ai-service",
"ai:lint": "turbo run lint --filter=@nextcraft/ai-service"
"ai:lint": "turbo run lint --filter=@nextcraft/ai-service",
"cli:dev": "pnpm --filter @nextcraft/cli dev",
"cli:test": "pnpm --filter @nextcraft/cli test",
"cli:typecheck": "pnpm --filter @nextcraft/cli typecheck",
"cli:build": "pnpm --filter @nextcraft/cli build"
},
"devDependencies": {
"turbo": "^2.3.3",
+184
View File
@@ -0,0 +1,184 @@
/**
* Nextcraft Defense Types
*
* Oral-defense sessions over HTTP: the examiner/learner transcript, the
* A-109 integrity signals computed server-side from turn metadata, the
* examiner's structured final verdict, and the voice-capability
* descriptor the web client selects its STT/TTS path from (D-030).
*
* Source of truth: the Python models in
* `apps/ai-service/ai_service/voice/defense_store.py` (`DefenseRecord` /
* `DefenseTurn`), `apps/ai-service/ai_service/agents/examiner.py`
* (`DefenseVerdict`) and `apps/ai-service/ai_service/voice/base.py`
* (`VoiceDescriptor`), served over HTTP by
* `apps/ai-service/ai_service/api/defense.py` (`StartResponse` /
* `AnswerResponse` / `FinishResponse` + the GET /v1/defense/{id}
* transcript). This file mirrors those models; any schema change must be
* made in both places.
*
* Wire deltas (api/defense.py serializes it does not echo the store
* row wholesale):
* - `DefenseRecord.id` is served as `defense_id`;
* - `created_at` / `finished_at` (DefenseRecord) and `DefenseTurn`'s
* `defense_id` / `created_at` are store-side bookkeeping and are NOT
* part of any v0.3 response body, so they are not mirrored here.
*/
import type { RubricVerdict } from './grading';
/** Defense lifecycle (`DefenseStatus`, voice/defense_store.py). */
export type DefenseStatus = 'in_progress' | 'finished';
/** Who spoke a transcript turn (`TurnRole`, voice/defense_store.py). */
export type DefenseRole = 'examiner' | 'learner';
/**
* One examiner/learner dialogue turn as served by the defense endpoints.
* Mirrors `DefenseTurn` (ai_service/voice/defense_store.py) minus the
* store-only bookkeeping columns; (defense_id, seq) is the store PK
* ordering by `seq` is the endpoints' contract.
*/
export interface DefenseTurn {
/** Turn index within the defense, >= 0. */
seq: number;
/** Who spoke: examiner question or learner (STT) answer. */
role: DefenseRole;
/** Non-empty utterance text. */
text: string;
/** ISO 8601 UTC utterance timestamp. */
ts: string;
/** Per-turn pipeline latency (STT + LLM TTFT + TTS, A-109); null while not instrumented. */
latency_ms: number | null;
}
/** How the client speaks/hears (`VoiceDescriptor.mode`, voice/base.py). */
export type VoiceMode = 'server' | 'browser' | 'mock';
/**
* Voice capability descriptor served to the web client (D-030). The
* assessment UI reads this to decide HOW the learner speaks/hears
* server STT/TTS (v0.4 seam), browser-native SpeechRecognition /
* speechSynthesis, or the deterministic mock path. Never contains
* secrets. Mirrors `VoiceDescriptor` (ai_service/voice/base.py).
*/
export interface VoiceDescriptor {
mode: VoiceMode;
/** Speech-to-text available on the selected path. */
sr_available: boolean;
/** Text-to-speech available on the selected path. */
tts_available: boolean;
/** Human-readable path hint (e.g. the browser-fallback instructions). */
hint: string;
}
/**
* The examiner's structured final verdict (D-20-validated). Mirrors
* `DefenseVerdict` (ai_service/agents/examiner.py) field-for-field; the
* `verdict` pattern `^(mastered|developing|not_yet)$` is the shared
* `RubricVerdict` union (grading.ts) the rubric and defense verdicts
* use the same three values.
*/
export interface DefenseVerdict {
verdict: RubricVerdict;
/** One-paragraph assessment of the learner's understanding. */
understanding: string;
/** One-paragraph assessment of the learner's process justification. */
process_justification: string;
/** One-paragraph assessment of the learner's communication. */
communication: string;
/** 1-2 highlighted strengths. */
strengths: string[];
/** 1-2 highlighted gaps. */
gaps: string[];
}
/**
* One learner turn flagged as a long pause (A-109): it arrived slower
* than `IntegritySignal.pause_threshold_ms`. Shape served inside
* `integrity_signals.long_pauses` (api/defense.py).
*/
export interface LongPause {
/** The flagged turn's `seq`. */
turn: number;
/** That turn's recorded pipeline latency. */
latency_ms: number | null;
}
/**
* The A-109 integrity-signal dict persisted on the defense record and
* served by GET /v1/defense/{id} + POST .../finish. Known keys are typed
* below; the dict is open off-scope cadence markers and friends may
* join later, so unknown keys surface as `unknown`.
*
* Mirrors the shape written by `finish_defense` (api/defense.py) onto
* `DefenseRecord.integrity_signals` (voice/defense_store.py, JSON
* column). An EMPTY object is legal and usual: `{}` until finalize
* (hence every known key is optional). NOTE: this is the defense store's
* raw signal dict, not the Proctor's classified `IntegritySignal`
* model (agents/proctor.py) that is a derived per-signal
* signal_type/severity/note classification.
*/
export interface IntegritySignal {
/** Learner turns slower than the pause threshold ({} until finalize). */
long_pauses?: LongPause[];
/** The threshold `long_pauses` was computed against (PAUSE_THRESHOLD_MS). */
pause_threshold_ms?: number;
/** Nested by the finish endpoint so GET /{id} can re-serve the verdict. */
verdict?: DefenseVerdict;
/** Off-scope cadence markers and friends (A-109: the dict is open). */
[key: string]: unknown;
}
/**
* A defense session with its FULL transcript the GET /v1/defense/{id}
* response (api/defense.py). Mirrors `DefenseRecord`
* (ai_service/voice/defense_store.py) on the wire: turns arrive in seq
* order, `integrity_signals` is `{}` until the defense is finalized.
*/
export interface DefenseSession {
defense_id: string;
learner_id: string;
task_id: string;
status: DefenseStatus;
/** Seq-ordered examiner/learner transcript. */
turns: DefenseTurn[];
/** A-109 signals; `{}` while in progress. */
integrity_signals: IntegritySignal;
}
/** Per-stage latency breakdown served with each follow-up question. */
export interface DefenseTurnLatency {
/** Speech-to-text duration of the learner's audio answer; null for typed answers. */
stt_ms: number | null;
/** LLM time-to-first-token for the examiner's question. */
llm_ms: number | null;
/** Text-to-speech render time; null in v0.3 (server TTS is CUT-1/v0.4). */
tts_ms: number | null;
}
/** POST /v1/defense/start response (`StartResponse`, api/defense.py). */
export interface DefenseStart {
defense_id: string;
/** Voice capability descriptor — the client picks its STT/TTS path from this (D-030). */
voice_descriptor: VoiceDescriptor;
/** Whether the trace this defense covers has gaps (disclose before defending, G-4). */
trace_complete: boolean;
/** The examiner's opening question (also persisted as turn seq 0). */
first_question: string;
}
/** POST /v1/defense/{id}/answer response (`AnswerResponse`, api/defense.py). */
export interface DefenseAnswer {
/** The examiner's next question (also persisted as the following turn). */
question: string;
/** Latency breakdown for this answer → question cycle. */
turn_latency: DefenseTurnLatency;
}
/** POST /v1/defense/{id}/finish response (`FinishResponse`, api/defense.py). */
export interface DefenseFinish {
/** The examiner's structured final verdict. */
verdict: DefenseVerdict;
/** The persisted A-109 signals (includes the nested `verdict`). */
integrity_signals: IntegritySignal;
}
+1 -1
View File
@@ -69,7 +69,7 @@ export interface GradeRecord {
learner_id: string;
/** The graded task — joins to `TaskVariant.task_id`. */
task_id: string;
/** Task-variant seed (D-029); null while grading is variant-blind. */
/** Task-variant seed (D-029); null only for non-variant tasks (P4 stamps it). */
variant_seed: string | null;
/** Compact trace digest (D-028) that fed the rubric prompt; {} for gate records. */
digest: Record<string, unknown>;
+1
View File
@@ -1,4 +1,5 @@
export * from './domain';
export * from './defense';
export * from './grading';
export * from './marketplace';
export * from './telemetry';
@@ -0,0 +1,104 @@
import type { Meta, StoryObj } from '@storybook/react';
import { GradeBadge } from './grade-badge';
const meta: Meta<typeof GradeBadge> = {
title: 'Primitives/GradeBadge',
component: GradeBadge,
tags: ['autodocs'],
parameters: {
layout: 'centered',
},
argTypes: {
outcome: {
control: 'select',
options: ['GRADED', 'UNGRADABLE_TRACE_INCOMPLETE', 'UNGRADABLE_EMPTY_TRACE'],
},
verdict: {
control: 'select',
options: ['mastered', 'developing', 'not_yet'],
},
iconOnly: { control: 'boolean' },
},
args: {
outcome: 'GRADED',
verdict: 'developing',
},
};
export default meta;
type Story = StoryObj<typeof GradeBadge>;
/* -------------------------------------------------------------------------- */
/* Rubric verdicts (GRADED record) */
/* -------------------------------------------------------------------------- */
export const Mastered: Story = {
args: { outcome: 'GRADED', verdict: 'mastered' },
};
export const Developing: Story = {
args: { outcome: 'GRADED', verdict: 'developing' },
};
export const NotYet: Story = {
args: { outcome: 'GRADED', verdict: 'not_yet' },
};
/* -------------------------------------------------------------------------- */
/* Gate states (G-4 — a gate record carries no rubric verdict) */
/* -------------------------------------------------------------------------- */
export const TraceIncomplete: Story = {
args: { outcome: 'UNGRADABLE_TRACE_INCOMPLETE' },
};
export const EmptyTrace: Story = {
args: { outcome: 'UNGRADABLE_EMPTY_TRACE' },
};
/* -------------------------------------------------------------------------- */
/* Icon-only (compact tables) */
/* -------------------------------------------------------------------------- */
export const IconOnly: Story = {
args: { outcome: 'GRADED', verdict: 'mastered', iconOnly: true },
};
export const IconOnlyGate: Story = {
args: { outcome: 'UNGRADABLE_TRACE_INCOMPLETE', iconOnly: true },
};
/* -------------------------------------------------------------------------- */
/* Galleries */
/* -------------------------------------------------------------------------- */
export const AllVerdicts: Story = {
render: () => (
<div className="flex flex-wrap items-center gap-3">
<GradeBadge outcome="GRADED" verdict="mastered" />
<GradeBadge outcome="GRADED" verdict="developing" />
<GradeBadge outcome="GRADED" verdict="not_yet" />
</div>
),
};
export const AllGateStates: Story = {
render: () => (
<div className="flex flex-wrap items-center gap-3">
<GradeBadge outcome="UNGRADABLE_TRACE_INCOMPLETE" />
<GradeBadge outcome="UNGRADABLE_EMPTY_TRACE" />
</div>
),
};
export const IconOnlyGallery: Story = {
render: () => (
<div className="flex flex-wrap items-center gap-3">
<GradeBadge outcome="GRADED" verdict="mastered" iconOnly />
<GradeBadge outcome="GRADED" verdict="developing" iconOnly />
<GradeBadge outcome="GRADED" verdict="not_yet" iconOnly />
<GradeBadge outcome="UNGRADABLE_TRACE_INCOMPLETE" iconOnly />
<GradeBadge outcome="UNGRADABLE_EMPTY_TRACE" iconOnly />
</div>
),
};
+104
View File
@@ -0,0 +1,104 @@
'use client';
import { forwardRef, type HTMLAttributes } from 'react';
import { BadgeCheck, CircleAlert, CircleX, TrendingUp } from 'lucide-react';
import type { GradeOutcome, RubricVerdict } from '@nextcraft/types';
/**
* GradeBadge verdict rendering (REQ-3-008, grading.ts types).
*
* Two verdict layers, one primitive:
* - rubric verdicts: mastered / developing / not_yet
* (RubricVerdict shared by grade + defense verdicts);
* - grade gate states: GRADED / UNGRADABLE_TRACE_INCOMPLETE /
* UNGRADABLE_EMPTY_TRACE (GradeOutcome, G-4) a gate record never
* carries rubric scores, so the badge renders the gate state alone.
*
* A grade carries ONE of the two: `outcome` (machine-readable) plus an
* optional `verdict` (the rubric judgment inside a GRADED record).
*/
export interface GradeBadgeProps extends HTMLAttributes<HTMLSpanElement> {
/** Machine-readable grade outcome (GradeRecord.verdict). */
outcome: GradeOutcome;
/** Rubric judgment when outcome is GRADED; ignored otherwise. */
verdict?: RubricVerdict;
/** Compact square badge with no text label (icon only, tooltip via title). */
iconOnly?: boolean;
}
type GradeBadgeKey = GradeOutcome | RubricVerdict;
interface GradeBadgeVisual {
label: string;
icon: typeof BadgeCheck;
classes: string;
}
const visuals: Record<GradeBadgeKey, GradeBadgeVisual> = {
mastered: {
label: 'Mastered',
icon: BadgeCheck,
classes:
'bg-emerald-100 text-emerald-700 dark:bg-emerald-900/40 dark:text-emerald-300',
},
developing: {
label: 'Developing',
icon: TrendingUp,
classes: 'bg-amber-100 text-amber-700 dark:bg-amber-900/40 dark:text-amber-300',
},
not_yet: {
label: 'Not yet',
icon: CircleX,
classes: 'bg-rose-100 text-rose-700 dark:bg-rose-900/40 dark:text-rose-300',
},
GRADED: {
label: 'Graded',
icon: BadgeCheck,
classes: 'bg-primary-100 text-primary-700 dark:bg-primary-900/40 dark:text-primary-300',
},
UNGRADABLE_TRACE_INCOMPLETE: {
label: 'Trace incomplete',
icon: CircleAlert,
classes: 'bg-amber-100 text-amber-700 dark:bg-amber-900/40 dark:text-amber-300',
},
UNGRADABLE_EMPTY_TRACE: {
label: 'Empty trace',
icon: CircleAlert,
classes: 'bg-slate-200 text-slate-700 dark:bg-slate-800 dark:text-slate-300',
},
};
export const GradeBadge = forwardRef<HTMLSpanElement, GradeBadgeProps>(function GradeBadge(
{ outcome, verdict, iconOnly = false, className, ...rest },
ref,
) {
// A GRADED record renders its rubric verdict (the stronger signal);
// a gate record renders its gate state. UNGRADABLE_* records never
// carry a verdict, so the fallback is the outcome itself.
const key: GradeBadgeKey = outcome === 'GRADED' && verdict ? verdict : outcome;
const visual = visuals[key];
const Icon = visual.icon;
return (
<span
ref={ref}
title={iconOnly ? visual.label : undefined}
className={[
'inline-flex items-center gap-1 rounded-full px-2.5 py-0.5 text-xs font-medium',
visual.classes,
className ?? '',
]
.filter(Boolean)
.join(' ')}
{...rest}
>
<Icon className="h-3.5 w-3.5" aria-hidden />
{iconOnly ? (
<span className="sr-only">{visual.label}</span>
) : (
visual.label
)}
</span>
);
});
+6 -1
View File
@@ -2,4 +2,9 @@ export * from './button';
export * from './input';
export * from './card';
export * from './badge';
export * from './avatar';
export * from './avatar';
export * from './terminal-frame';
export * from './mic-control';
export * from './grade-badge';
export * from './telemetry-status';
export * from './transcript-viewer';
@@ -0,0 +1,90 @@
import type { Meta, StoryObj } from '@storybook/react';
import { MicControl } from './mic-control';
const meta: Meta<typeof MicControl> = {
title: 'Primitives/MicControl',
component: MicControl,
tags: ['autodocs'],
parameters: {
layout: 'centered',
},
argTypes: {
state: {
control: 'select',
options: ['idle', 'recording', 'denied', 'unsupported'],
},
hint: { control: 'text' },
disabled: { control: 'boolean' },
},
args: {
state: 'idle',
},
};
export default meta;
type Story = StoryObj<typeof MicControl>;
/* -------------------------------------------------------------------------- */
/* States */
/* -------------------------------------------------------------------------- */
export const Idle: Story = {
args: { state: 'idle' },
};
export const Recording: Story = {
args: {
state: 'recording',
trailing: <span className="tabular-nums">0:07</span>,
},
};
export const Denied: Story = {
args: { state: 'denied' },
};
export const Unsupported: Story = {
args: { state: 'unsupported' },
};
export const Disabled: Story = {
args: { state: 'idle', disabled: true },
};
/* -------------------------------------------------------------------------- */
/* Consent-first contract */
/* -------------------------------------------------------------------------- */
export const ConsentFirstFlow: Story = {
name: 'Consent-first flow (click is the only trigger)',
render: () => (
<div className="flex flex-col gap-6">
<MicControl state="idle" hint="Click to record — nothing captures before you do." />
<MicControl state="recording" trailing={<span className="tabular-nums">0:12</span>} />
<MicControl
state="denied"
hint="Server voice unavailable; browser SpeechRecognition active."
/>
</div>
),
};
export const NoMicFallbackHints: Story = {
render: () => (
<div className="flex flex-col gap-6">
<MicControl state="denied" />
<MicControl state="unsupported" />
</div>
),
};
export const AllStates: Story = {
render: () => (
<div className="flex flex-wrap items-start gap-4">
<MicControl state="idle" />
<MicControl state="recording" />
<MicControl state="denied" />
<MicControl state="unsupported" />
</div>
),
};

Some files were not shown because too many files have changed in this diff Show More