docs(P00): create phase plans — v0.3 credential engines

Vertical-slice plan: 6 execution phases, 20 waves, 39 tasks.
P1 sandbox fabric (unshare namespaces, D-024/032) · P2 telemetry (WS+SQLite, D-026/027)
P3 trace grading (digest+LLM, D-028) · P4 variants (seeded templates, D-029)
P5 voice defense (VoiceProvider, Examiner, D-030) · P6 re-grounding + learner integration.
MVP/UX sections included (User-Facing Surface, Happy Path, UX Acceptance Criteria).
Must-haves per phase; KYC deferred (A-110); mock-first voice (no blocking key).

---ci---
phase: 0
milestone: v0.3
status: plan
---/ci---
This commit is contained in:
CIAgent
2026-09-11 17:52:07 +00:00
parent ec897336a5
commit 044e4b8cc8
+328 -308
View File
@@ -1,422 +1,442 @@
# Nextcraft v0.2 — PLAN.md
# Nextcraft v0.3 — PLAN.md
## Overview
This plan covers execution phases 1-6 of milestone v0.2 (AI Tutor Architecture): the six AI tutor agents (Coach, Tutor, Lab, Assessor, Proctor, Mentor) as real LLM-backed services in a new `apps/ai-service` Python FastAPI application, wired into the existing v0.1 learner surface with streaming responses. Phases are strictly sequential (P1→P6); within each phase, Wave 1 tasks are parallelizable (no cross-file dependencies) and later waves depend on earlier ones.
This plan covers execution phases 1-6 of milestone v0.3 (Credential Engines): the real engines that replace v0.2's mock inputs — a namespace-isolated sandbox fabric, live build telemetry over WebSocket + SQLite, a process-trace grading engine, seeded per-learner variant task generation, and an oral/voice defense with a seventh Examiner agent — plus re-grounding the Lab/Assessor/Proctor agents onto real engine inputs and wiring the v0.1 learner surfaces to the real build/defense/grading paths. Phases are strictly sequential (P1→P6); within each phase, Wave 1 tasks are parallelizable (no cross-file dependencies) and later waves depend on earlier ones.
**Environment facts (apply throughout):** Python via `python3 -m venv` (no system pip, no uv); pnpm 12.3.4 via corepack; ai-service port **8420**; default model `gemma4:31b` (config via `AI_TUTOR_MODEL`); ollama-cloud base `https://ollama.com/v1` (OpenAI-compatible, Bearer auth); API keys live only in gitignored `.ciagent/.env.secrets`, exported by `scripts/dev.sh` — never in code, commits, or logs; all automated tests use the deterministic mock provider and **never call the cloud**.
**Environment facts (apply throughout):** Python 3.11.2 via `python3 -m venv` (no uv, no system pip); pnpm 12.3.4 via corepack; turborepo; ai-service port **8420**; default model `gemma4:31b` (config via `AI_TUTOR_MODEL`); ollama-cloud base `https://ollama.com/v1` (OpenAI-compatible, Bearer auth); keys live only in gitignored `.ciagent/.env.secrets` (exported by `scripts/dev.sh`) — never in code, commits, or logs; all automated tests use the deterministic mock LLM and mock voice providers and **never call cloud or voice APIs**. New ai-service deps this milestone: `sqlmodel`, `sqlalchemy`, `websockets`, `aiofiles` (all PyPI-verified; added in Task 1-1-02). New web deps: `@xterm/xterm`, `@xterm/addon-fit` (Task 6-1-01). SQLite data (`apps/ai-service/ai_service/data/`) and sandbox dirs (`apps/ai-service/sandboxes/`) are already gitignored. **KYC/age-gating is deferred per founder directive (A-110)** — no identity work appears anywhere in this plan; age-gating stays the v0.1 visual flow mockup.
| Phase | Name | Requirements | Waves | Personas |
|-------|------|-------------|-------|----------|
| 1 | AI service scaffolding | REQ-2-001, 002, 003 | 3 | backend-engineer, ai-engineer |
| 2 | Agent framework | REQ-2-004 | 2 | ai-engineer, backend-engineer |
| 3 | Coach + Tutor agents | REQ-2-005, 006 | 3 | ai-engineer, backend-engineer |
| 4 | Lab + Assessor agents | REQ-2-007, 008 | 4 | ai-engineer, backend-engineer, data-engineer |
| 5 | Proctor + Mentor agents | REQ-2-009, 010 | 3 | ai-engineer, backend-engineer |
| 6 | Learner surface integration | REQ-2-011, 012 | 3 | frontend-engineer, design-system-engineer |
| 1 | Sandbox fabric | REQ-3-001, 002 | 3 | sandbox-engineer, backend-engineer, ai-engineer (W1 lint only) |
| 2 | Live build telemetry | REQ-3-003 | 4 | sandbox-engineer, backend-engineer, data-engineer |
| 3 | Process-trace grading engine | REQ-3-004 | 3 | ai-engineer, backend-engineer |
| 4 | Variant task generation | REQ-3-005 | 3 | ai-engineer, backend-engineer, data-engineer |
| 5 | Oral / voice defense | REQ-3-006 | 4 | voice-engineer, ai-engineer, backend-engineer |
| 6 | Agent re-grounding + learner surface integration | REQ-3-007, 008 | 5 | ai-engineer, frontend-engineer, design-system-engineer, data-engineer, backend-engineer, lead-developer |
---
## Phase 1: AI Service Scaffolding
## Phase 1: Sandbox Fabric
**Requirements:** REQ-2-001, REQ-2-002, REQ-2-003
**Goal:** apps/ai-service runs under uvicorn, /health responds, provider-agnostic LLM layer with ollama-cloud/local/mock providers, SSE chat streaming verified, pytest suite green with mock provider, turbo integration wired
**Requirements:** REQ-3-001, REQ-3-002
**Goal:** `SandboxBackend` protocol + `unshare`-based namespace spawner (D-024) + lifecycle manager with concurrency guard (D-032) + per-sandbox workdir; isolation and resources probe-verified on this box; `/v1/sandboxes` API live; deps + gitignore landed
### Wave 1: Service shell + LLM core (parallel — no shared files)
### Wave 1: Foundations (parallel — no shared files)
#### Task 1-1-01: FastAPI app scaffolding
- **Persona:** backend-engineer — **REQ:** REQ-2-001
- **Files:** `apps/ai-service/pyproject.toml`, `apps/ai-service/ai_service/main.py`, `apps/ai-service/ai_service/config.py`, `apps/ai-service/ai_service/__init__.py`, `apps/ai-service/tests/conftest.py`, `apps/ai-service/tests/test_health.py`, `apps/ai-service/.env.example`, `apps/ai-service/README.md`
- **Action:** pydantic-settings `Settings` (env_prefix `AI_`, env_file, `SecretStr` key, port 8420, provider select, `AI_TUTOR_MODEL` default `gemma4:31b`). FastAPI app factory in `main.py` with lifespan stub (httpx client pool comes in Wave 2), CORS localhost-only (A-008), `GET /health`. pyproject with pinned deps (fastapi, uvicorn, pydantic, pydantic-settings, httpx, sse-starlette, pytest, pytest-asyncio) and dev extra. conftest: settings override + TestClient fixture. `.env.example` documents all `AI_*` vars; README documents venv setup and dev workflow.
- **Verify:** `scripts/bootstrap.sh && scripts/test.sh` — test_health passes; `curl localhost:8420/health` returns 200
#### Task 1-1-01: SandboxBackend protocol + unshare spawner + probe test
- **Persona:** sandbox-engineer — **REQ:** REQ-3-001, REQ-3-002
- **Files:** `apps/ai-service/ai_service/sandbox/__init__.py`, `apps/ai-service/ai_service/sandbox/backend.py`, `apps/ai-service/ai_service/sandbox/workdir.py`, `apps/ai-service/ai_service/sandbox/unshare_backend.py`, `apps/ai-service/tests/sandbox/__init__.py`, `apps/ai-service/tests/sandbox/test_isolation.py`
- **Action:** `backend.py`: `SandboxBackend` protocol + `SandboxSpec` (sandbox_id, learner_id, workdir, resource limits) + `SandboxHandle` (id, pid, workdir, created_at); `spawn(spec)`, `exec(handle, cmd)`, `snapshot(handle) -> Path`, `destroy(handle)`. `workdir.py`: per-sandbox layout under `apps/ai-service/sandboxes/<id>/` (workspace/ writable, snapshot() = recursive copy to `snapshots/<ts>/`) — no symlinks as the snapshot mechanism. `unshare_backend.py`: subprocess spawner — `unshare --user --map-root-user --mount --pid --fork --net` with the per-sandbox dir bind-mounted (`--bind <dir> /work`) and `chdir /work` (D-024); pipes for stdout/stderr; async wrappers. `test_isolation.py`**re-verify box isolation properties (runs on this box, guarded by probe skip):** (a) `id -u` inside namespace prints `0`; (b) `ip link` inside namespace shows 0 usable interfaces (loopback-only/no carrier) — network isolated; (c) file written to `/work/inside.txt` lands at `sandboxes/<id>/workspace/inside.txt` on the host; (d) attempt to write outside the mount (e.g. host tmp path via bind) does not escape the per-sandbox dir; (e) `/proc` visibility degraded (proc-remount not permitted per A-101 — assert the probe documents this, not that it fails).
- **Verify:** `pnpm ai:test``tests/sandbox/test_isolation.py` green on this box (probe-gated: skips with an explicit reason if userns unavailable); `lint` clean
#### Task 1-1-02: LLM types, protocol, mock provider
- **Persona:** ai-engineer — **REQ:** REQ-2-002
- **Files:** `apps/ai-service/ai_service/llm/types.py`, `apps/ai-service/ai_service/llm/base.py`, `apps/ai-service/ai_service/llm/mock.py`, `apps/ai-service/ai_service/llm/__init__.py`
- **Action:** `types.py`: pydantic `Message` (role/content), `ChatDelta` (OpenAI-compatible chunk shape). `base.py`: `LLMProvider` protocol — async `stream_chat(messages, model, response_format=None) -> AsyncIterator[ChatDelta]`; the provider is a dumb pipe, no envelope logic (D-016 keeps envelope in API layer). `mock.py`: deterministic scripted provider (hash-seeded token streams, scripted failure modes: connect error, mid-stream error, malformed JSON) for tests and CI.
- **Verify:** mock provider importable and deterministic; two identical calls yield identical streams
#### Task 1-1-02: v0.3 dependencies + gitignore + config additions
- **Persona:** backend-engineer — **REQ:** REQ-3-001
- **Files:** `apps/ai-service/pyproject.toml` (update), `apps/ai-service/ai_service/config.py` (update), `apps/ai-service/.env.example` (update), `apps/ai-service/scripts/bootstrap.sh` (update if needed), root `package.json` (no change), `turbo.json` (no change)
- **Action:** Add pinned deps: `sqlmodel`, `sqlalchemy`, `websockets`, `aiofiles` to pyproject. `config.py` additions (env_prefix `AI_`): `AI_SANDBOX_DIR` (default `apps/ai-service/sandboxes`), `AI_SANDBOX_MAX_CONCURRENT` (default 5, D-032), `AI_SANDBOX_CPU_LIMIT` (default 1 core / cpu.max), `AI_SANDBOX_MEM_LIMIT_MB` (default 512), `AI_SANDBOX_PIDS_LIMIT` (default 256), `AI_SANDBOX_TIMEOUT_S` (default 1800), `AI_DB_PATH` (default `ai_service/data/nextcraft.db`), `AI_VOICE_BASE_URL` / `AI_VOICE_API_KEY` / `AI_VOICE_STT_MODEL` / `AI_VOICE_TTS_MODEL` (all **optional**, default empty — mock-first, D-030; documented in `.env.example` and README). Confirm `.gitignore` already covers `ai_service/data/` + `sandboxes/` (it does — v0.3 block present). Re-run bootstrap idempotently to install new deps.
- **Verify:** `pnpm ai:bootstrap` re-installs cleanly (no-op venv, new wheels land); `python -c "import sqlmodel, sqlalchemy, websockets, aiofiles"` succeeds in the venv; settings parse with new keys unset
#### Task 1-1-03: Monorepo integration (shim + turbo + scripts)
- **Persona:** backend-engineer — **REQ:** REQ-2-001
- **Files:** `apps/ai-service/package.json`, `apps/ai-service/scripts/bootstrap.sh`, `apps/ai-service/scripts/dev.sh`, `apps/ai-service/scripts/test.sh`, `turbo.json` (update), `package.json` (root, update)
- **Action:** zero-dependency shim `package.json` in apps/ai-service with `dev`/`test`/`bootstrap` script entries. Turbo passthrough tasks `ai#dev`, `ai#test`, `ai#bootstrap` (`cache: false`, `outputs: []`). Root scripts `ai:dev`, `ai:test`, `ai:bootstrap`. `bootstrap.sh`: idempotent `python3 -m venv .venv` + pip install. `dev.sh`: exports keys from `.ciagent/.env.secrets` → uvicorn on 8420. `test.sh`: pytest via venv.
- **Verify:** `corepack pnpm install && pnpm ai:bootstrap && pnpm ai:test` runs pytest through turbo; re-running bootstrap is a no-op
#### Task 1-1-03: Resource-limit probe documentation + harness ruff pass
- **Persona:** ai-engineer — **REQ:** REQ-3-001
- **Files:** `apps/ai-service/README.md` (update: sandbox section + probe transcript), `apps/ai-service/pyproject.toml` (no change), `apps/ai-service/tests/sandbox/test_isolation.py` (no change)
- **Action:** Record the A-101 probe transcript in README (verbatim commands + observed output from this box: `unshare --user --map-root-user --mount --pid --fork --net id -u``0`; `ip link` → loopback only; write containment). Document the v0.3 resource-limit mechanism choice: cgroup-v2 delegation via per-sandbox scope files is **not available** on this box without sudo → enforcement = subprocess-level (`ulimit`-equivalent via `preexec_fn`: RLIMIT_AS for memory, RLIMIT_CPU for CPU-seconds, RLIMIT_NPROC for pids) + hard wall-clock timeout kill in the manager. This is the locked v0.3 mechanism (D-024 + no-sudo constraint). Run ruff over the new tree; fix all findings.
- **Verify:** `pnpm ai:lint` exits 0; README shows the probe transcript and the rlimit mechanism note
#### Task 1-1-04: Python lint (ruff, check-only)
- **Persona:** backend-engineer — **REQ:** REQ-2-001
- **Files:** `apps/ai-service/pyproject.toml` (update: `[tool.ruff]` config + `ruff` in dev extra), `apps/ai-service/scripts/lint.sh`, `package.json` (root, update), `turbo.json` (update)
- **Action:** Add `ruff` (check-only, no formatter) to the dev extra; `[tool.ruff]` with line-length 100, target py311. `scripts/lint.sh`: `.venv/bin/ruff check .` with repo-root path resolution via script-relative dirname (not CWD). Root script `ai:lint`, turbo passthrough `ai#lint` (`cache: false`, `outputs: []`). Run over the entire ai-service tree; fix all findings before phase ship (G-3).
- **Verify:** `pnpm ai:lint` exits 0 on the Phase 1 codebase
### Wave 2: Lifecycle manager (depends on Wave 1)
### Wave 2: Real providers + SSE endpoint (depends on Wave 1)
#### Task 1-2-01: Sandbox manager + concurrency guard + snapshots
- **Persona:** sandbox-engineer — **REQ:** REQ-3-001, REQ-3-002
- **Files:** `apps/ai-service/ai_service/sandbox/manager.py`, `apps/ai-service/tests/sandbox/test_manager.py`
- **Action:** `SandboxManager`: `create(learner_id) -> SandboxHandle` (guard: active count ≥ `AI_SANDBOX_MAX_CONCURRENT` → raise `PoolFullError` → API maps to **503**, D-032; no queue); `list() -> list[SandboxHandle]`; `get(id)`; `snapshot(id) -> Path` (delegates to workdir); `destroy(id)` (kill process tree, keep or purge workdir per flag); `reap_expired()` background hook for `AI_SANDBOX_TIMEOUT_S`. Enforce rlimits per spawner (Task 1-1-03) at exec time. Handle registry persisted **in-memory** (v0.3, single process; not a store — see D-019 precedent) with a clear note that handles are process-local. Narrow typed interface only — manager never imports api/ (boundary rule).
- **Verify:** `pnpm ai:test` — test_manager covers create/list/destroy/snapshot, 6th create raises PoolFullError (503 path), destroy kills the namespace process (pid gone), snapshot dir exists with workspace contents, timeout reaper removes a stale handle
#### Task 1-2-01: OpenAI-compatible provider + factory
- **Persona:** ai-engineer — **REQ:** REQ-2-002
- **Files:** `apps/ai-service/ai_service/llm/openai_compat.py`, `apps/ai-service/ai_service/llm/factory.py`
- **Action:** `openai_compat.py`: single `OpenAICompatProvider` for ollama-cloud (`https://ollama.com/v1`, Bearer) and local endpoints (base URL from settings); raw httpx against `/v1/chat/completions` with `stream: true`, byte-identical delta passthrough, tolerant of ollama-cloud quirks. Uses the lifespan-managed `httpx.AsyncClient` (10s connect / 300s read, D-017) — no openai SDK. `factory.py`: select provider from settings (`ollama-cloud` | `local` | `mock`).
- **Verify:** provider constructs from settings for all 3 names; manual probe against ollama-cloud streams tokens (documented in README, not a test)
#### Task 1-2-02: Resource-limit enforcement test
- **Persona:** sandbox-engineer — **REQ:** REQ-3-002
- **Files:** `apps/ai-service/tests/sandbox/test_resource_limits.py`
- **Action:** Concrete enforcement probes (guarded like isolation tests): (a) spawn a process that allocates > `RLIMIT_AS` → assert it dies with MemoryError/killed within a bound; (b) spawn a CPU spinner past `RLIMIT_CPU` → assert SIGXCPU/kill; (c) fork-bomb attempt beyond `RLIMIT_NPROC` → assert fork failures; (d) wall-clock: spawn `sleep 9999` with a small manager timeout → reaper destroys it. Assert limits are observable (handle reports its limit set).
- **Verify:** `pnpm ai:test` — test_resource_limits green; limits proven enforced and observable
#### Task 1-2-02: Lifespan wiring + SSE chat endpoint
- **Persona:** backend-engineer — **REQ:** REQ-2-003
- **Files:** `apps/ai-service/ai_service/main.py` (update), `apps/ai-service/ai_service/api/deps.py`, `apps/ai-service/ai_service/api/chat.py`, `apps/ai-service/ai_service/api/__init__.py`
- **Action:** Lifespan creates the shared `httpx.AsyncClient` and provider factory; deps.py provides provider via DI. `POST /v1/chat/stream` in chat.py implements the D-016 envelope: `meta` event (agent/session/model) flushed before first token → raw OpenAI chunks passed through as `data: {json}``done` event → `error` event before `[DONE]` on mid-stream failure; pre-first-byte failures return proper HTTP status codes. Headers `Cache-Control: no-cache`, `X-Accel-Buffering: no`; sse-starlette ping keep-alive.
- **Verify:** `curl -N -X POST localhost:8420/v1/chat/stream` with mock provider shows meta event, token deltas, done, `[DONE]`
### Wave 3: API exposure (depends on Wave 2)
### Wave 3: Provider + endpoint test suites (depends on Wave 2)
#### Task 1-3-01: LLM provider tests (byte-exact, no cloud)
- **Persona:** ai-engineer — **REQ:** REQ-2-002
- **Files:** `apps/ai-service/tests/llm/test_openai_compat.py`, `apps/ai-service/tests/llm/test_mock.py`, `apps/ai-service/tests/llm/__init__.py`
- **Action:** httpx `MockTransport` tests parsing byte-exact fixture streams (happy path, empty delta, `[DONE]`, malformed line, mid-stream disconnect). Mock provider tests: determinism, scripted failure modes, response_format echo.
- **Verify:** `pnpm ai:test` — llm suite green; zero network calls in tests
#### Task 1-3-02: SSE stream endpoint tests
- **Persona:** backend-engineer — **REQ:** REQ-2-003
- **Files:** `apps/ai-service/tests/api/test_chat_stream.py`, `apps/ai-service/tests/api/__init__.py`
- **Action:** TestClient `client.stream()` tests: meta-first ordering, delta passthrough, done + `[DONE]` sentinel, mid-stream error event, pre-first-byte failure → HTTP status, required headers. pytest-asyncio auto mode (D-023).
- **Verify:** `pnpm ai:test` — api suite green
#### Task 1-3-01: Sandboxes API module
- **Persona:** backend-engineer — **REQ:** REQ-3-001, REQ-3-002
- **Files:** `apps/ai-service/ai_service/api/sandboxes.py`, `apps/ai-service/ai_service/main.py` (update: include router + lifespan manager), `apps/ai-service/ai_service/api/deps.py` (update), `apps/ai-service/tests/api/test_sandboxes.py`
- **Action:** DI exposes a singleton `SandboxManager`. Endpoints: `POST /v1/sandboxes {learner_id}` → 201 handle (503 when pool full); `GET /v1/sandboxes` → list; `GET /v1/sandboxes/{id}` → handle; `POST /v1/sandboxes/{id}/snapshot` → snapshot path; `DELETE /v1/sandboxes/{id}` → 204. All behind localhost CORS (A-008). Lifespan creates/destroys the manager; on shutdown destroys any live sandboxes (no orphans).
- **Verify:** `pnpm ai:test` — test_sandboxes green (create→list→snapshot→delete roundtrip via TestClient; 6th create → 503; delete of unknown id → 404); manual probe: `curl -X POST localhost:8420/v1/sandboxes -d '{"learner_id":"l1"}'` returns a handle id
### Must-Haves (Phase 1)
- [ ] `scripts/bootstrap.sh` is idempotent; creates venv + installs deps without system pip
- [ ] `pnpm ai:dev` starts uvicorn; `curl localhost:8420/health` returns 200
- [ ] `pnpm ai:lint` exits 0 (ruff check over the ai-service tree) (G-3)
- [ ] `pnpm ai:test` runs the full pytest suite via turbo and passes (mock provider only — no network)
- [ ] SSE stream delivers tokens: meta event, incremental deltas, done, `[DONE]` observed via `curl -N`
- [ ] Mid-stream failure emits `error` event before `[DONE]`; pre-first-byte failure returns HTTP error status
- [ ] Provider factory resolves ollama-cloud / local / mock from settings; manual ollama-cloud probe documented in README
- [ ] `llm/` imports nothing from `agents/` or `api/` (boundary rule holds)
- [ ] Isolation probe test green on this box: in-namespace uid=0, network isolated (0 usable interfaces), host writes confined to the per-sandbox bind dir (A-101 re-verified as an automated test, not just research notes)
- [ ] Resource limits enforced + observable: memory/CPU/pids rlimits kill violating processes; wall-clock reaper destroys stale sandboxes (test_resource_limits green)
- [ ] No cross-tenant access: sandbox A cannot read sandbox B's workdir (isolation test asserts containment)
- [ ] Lifecycle API works end-to-end: create/list/snapshot/destroy via TestClient; pool full → **503** (D-032, no queue)
- [ ] Snapshot produces a restorable directory copy under the sandbox's own snapshots/ dir
- [ ] `pnpm ai:test` and `pnpm ai:lint` green; new deps (`sqlmodel`, `sqlalchemy`, `websockets`, `aiofiles`) installed via idempotent bootstrap
- [ ] Boundary rules hold: `sandbox/` imports nothing from `api/` or `agents/`; only `api/sandboxes.py` composes the manager via DI
- [ ] No docker/podman/sudo anywhere in the spawner path (D-024); `SandboxBackend` protocol is the only coupling to the spawner (containerd swap possible later)
---
## Phase 2: Agent Framework
## Phase 2: Live Build Telemetry
**Requirements:** REQ-2-004
**Goal:** Shared framework all six agents use: BaseAgent contract, session store, prompt library, registry, structured outputs — all tested against the mock provider
**Requirements:** REQ-3-003
**Goal:** First real persistence (D-027: SQLite via SQLModel) with a `TraceStore` protocol; `TelemetryEvent` model with per-(learner,task) monotonic `seq`; WebSocket ingest endpoint (D-026) with gap detection; stdlib-only in-sandbox capture agent streams real sandbox activity into ai-service; trace retrievable by learner+task
### Wave 1: Framework primitives (parallel — no shared files)
### Wave 1: Models + stores + TS types (parallel — no shared files)
#### Task 2-1-01: BaseAgent ABC
- **Persona:** ai-engineer — **REQ:** REQ-2-004
- **Files:** `apps/ai-service/ai_service/agents/base.py`, `apps/ai-service/tests/agents/test_base.py`, `apps/ai-service/ai_service/agents/__init__.py`, `apps/ai-service/tests/agents/__init__.py`
- **Action:** `BaseAgent` ABC (D-018): `name`, `system_prompt`, `build_messages(history, learner_context)`, `stream_reply(...) -> AsyncIterator[ChatDelta]` (delegates to provider), `structured_reply(...)` (delegates to structured module, landed Wave 2). Subclass contract tested with a stub agent + mock provider.
- **Verify:** `pnpm ai:test` — test_base green
#### Task 2-1-01: Telemetry event models
- **Persona:** backend-engineer — **REQ:** REQ-3-003
- **Files:** `apps/ai-service/ai_service/telemetry/__init__.py`, `apps/ai-service/ai_service/telemetry/models.py`, `apps/ai-service/tests/telemetry/__init__.py`, `apps/ai-service/tests/telemetry/test_models.py`
- **Action:** Pydantic/SQLModel `TelemetryEvent`: `learner_id`, `task_id`, `seq` (int, monotonic per (learner,task)), `kind` (`command` | `file_diff` | `run_result` | `test_result` | `activity` | `stdin` | `stdout`), `payload` (JSON), `ts` (datetime, monotonic-envelope), `sandbox_id`. `TraceSpan` derived view (ordered events for one (learner,task)). Validation: seq ≥ 0, kind enum, non-empty ids.
- **Verify:** `pnpm ai:test` — test_models green (validation rules enforced, JSON payload roundtrip)
#### Task 2-1-02: SessionStore protocol + in-memory implementation
- **Persona:** ai-engineer — **REQ:** REQ-2-004
- **Files:** `apps/ai-service/ai_service/agents/session.py`, `apps/ai-service/tests/test_session.py`
- **Action:** `SessionStore` protocol + `InMemorySessionStore` (D-019): asyncio.Lock-guarded dict, agent-scoped session keys, 20-message rolling window, 500-cap LRU eviction. Protocol shape is DB-migration-ready (A-003).
- **Verify:** test_session covers create/append/window-trim/LRU-eviction/agent scoping
#### Task 2-1-02: TraceStore protocol + SQLite implementation
- **Persona:** backend-engineer — **REQ:** REQ-3-003
- **Files:** `apps/ai-service/ai_service/telemetry/store.py`, `apps/ai-service/tests/telemetry/test_store.py`, `apps/ai-service/ai_service/data/.gitkeep`
- **Action:** `TraceStore` protocol (D-027, Postgres-migration-ready): `append(event) -> None` (idempotent on (learner,task,seq) — at-least-once dedup), `get_trace(learner_id, task_id) -> list[TelemetryEvent]` (ordered by seq), `gaps(learner_id, task_id) -> list[int]` (missing seqs), `latest_seq(learner_id, task_id) -> int`, `list_tasks(learner_id) -> list[str]`, `close()`. `SQLiteTraceStore(SQLModel)`: single `telemetry_event` table, composite PK ((learner_id, task_id, seq)), indexes on (learner_id, task_id). Engine creation from `AI_DB_PATH`; `SQLModel.metadata.create_all` at app lifespan.
- **Verify:** `pnpm ai:test` — test_store green (append/ordered-get/dedup-on-retry/gap detection/latest_seq; tmp-path SQLite per test)
#### Task 2-1-03: Prompt library scaffolding
- **Persona:** ai-engineer — **REQ:** REQ-2-004
- **Files:** `apps/ai-service/ai_service/prompts/coach.py`, `apps/ai-service/ai_service/prompts/tutor.py`, `apps/ai-service/ai_service/prompts/lab.py`, `apps/ai-service/ai_service/prompts/assessor.py`, `apps/ai-service/ai_service/prompts/proctor.py`, `apps/ai-service/ai_service/prompts/mentor.py`, `apps/ai-service/ai_service/prompts/__init__.py`
- **Action:** Per-agent module with a versioned `SYSTEM_PROMPT` constant + `render_context(learner_context) -> dict` using `str.format_map` for learner-context injection (D-018: prompts are code, versioned in git). Initial drafts for all six; final personas land in Phases 3-5.
- **Verify:** all six prompt modules import; render_context fills placeholders without KeyError
#### Task 2-1-03: TS types for telemetry/traces
- **Persona:** data-engineer — **REQ:** REQ-3-003
- **Files:** `packages/types/telemetry.ts` (new), `packages/types/index.ts` (update)
- **Action:** TS `TelemetryEvent`, `TraceSpan`, `TelemetryKind` mirroring the Python model field-for-field (cross-referencing header, same string enums). Consumed by Phase 6 web surfaces; no runtime code.
- **Verify:** `pnpm typecheck` passes; TS type keys match Python model keys exactly
#### Task 2-1-04: Learner context corpus
- **Persona:** ai-engineer — **REQ:** REQ-2-004
- **Files:** `apps/ai-service/ai_service/corpus/learner_context.py`, `apps/ai-service/ai_service/corpus/__init__.py`
- **Action:** Pydantic-typed learner context (active stack, competencies, progress, recent artifacts) mirroring TS `packages/mock-data` IDs per D-021 convention (cross-referencing header comment, identical `comp-*`/`stack-*` ID strings).
- **Verify:** context renders into prompt placeholders; IDs match packages/mock-data strings
### Wave 2: Capture agent + ingest (depends on Wave 1)
### Wave 2: Composition layers (depends on Wave 1)
#### Task 2-2-01: In-sandbox capture agent (stdlib-only)
- **Persona:** sandbox-engineer — **REQ:** REQ-3-003
- **Files:** `apps/ai-service/scripts/sandbox-agent.py`, `apps/ai-service/tests/sandbox/test_sandbox_agent.py`
- **Action:** Tiny standalone process (D-031, **stdlib only** — no deps shipped into the namespace): wraps a shell inside the sandbox; captures commands, file diffs (mtime/content polling of `workspace/` at 250ms), run/test results, activity; assigns per-(learner,task) `seq`; buffers to a local spool file on disconnect (at-least-once, D-026); reconnects with **exponential backoff** and flushes spool in order; small WebSocket client implemented over raw `socket` (RFC6455 client handshake + frames — stdlib only, no `websockets` in-namespace). Configured via env baked at spawn (`NC_LEARNER_ID`, `NC_TASK_ID`, `NC_INGEST_URL`).
- **Verify:** `pnpm ai:test` — unit tests with a loopback fake WS server: ordered seq emission, spool-on-disconnect, reconnect flush preserves order (no loss, dupes deduped server-side), no third-party imports in the file (asserted by AST scan)
#### Task 2-2-01: Structured output defense
- **Persona:** ai-engineer — **REQ:** REQ-2-004
- **Files:** `apps/ai-service/ai_service/agents/structured.py`, `apps/ai-service/tests/test_structured.py`
- **Action:** 4-layer defense (D-020): (1) `response_format` request with auto-degrade on provider 400; (2) prompt-embedded JSON schema; (3) parse: strip code fences → first balanced JSON object; (4) single bounded retry with validation-error feedback. Returns pydantic-validated model or raises `StructuredOutputError`.
- **Verify:** test_structured covers fenced/unfenced/invalid JSON, retry path, degrade path — all against mock provider
#### Task 2-2-02: WebSocket ingest endpoint
- **Persona:** backend-engineer — **REQ:** REQ-3-003
- **Files:** `apps/ai-service/ai_service/telemetry/ingest.py`, `apps/ai-service/ai_service/api/sandboxes.py` (update: register WS route), `apps/ai-service/ai_service/main.py` (update), `apps/ai-service/tests/api/test_telemetry_ingest.py`
- **Action:** `WS /v1/telemetry/ingest` (D-026): accepts connections carrying learner_id/task_id/sandbox_id; validates + appends events to `TraceStore` (idempotent — server-side dedup on (learner,task,seq)); emits **gap warnings** when seq skips (logged + surfaced in a per-connection status); backpressure = bounded inbound queue with drop-oldest only on unbounded growth (documented); ping/pong keepalive. `GET /v1/telemetry/traces/{learner_id}/{task_id}` returns the ordered trace; `GET /v1/telemetry/gaps/{learner_id}/{task_id}` returns missing seqs.
- **Verify:** `pnpm ai:test` — test_telemetry_ingest green (TestClient websocket: connect → send 3 events → trace retrievable ordered; resend event 2 → deduped; skip seq 5 → gap reported; unknown sandbox tolerated in v0.3 no-auth mode)
#### Task 2-2-02: Agent registry
- **Persona:** ai-engineer — **REQ:** REQ-2-004
- **Files:** `apps/ai-service/ai_service/agents/registry.py`, `apps/ai-service/tests/test_registry.py`
- **Action:** Explicit registry: name → agent factory map with `register(name, factory)` / `get(name)`; raises on unknown agent. Agents are registered in their own phases (P3-P5).
- **Verify:** test_registry: register/get round-trip, unknown-agent error, duplicate registration error
### Wave 3: Sandbox telemetry wiring (depends on Wave 2)
#### Task 2-2-03: Session + agent DI wiring into API layer
- **Persona:** backend-engineer — **REQ:** REQ-2-004
- **Files:** `apps/ai-service/ai_service/api/deps.py` (update), `apps/ai-service/ai_service/api/chat.py` (update)
- **Action:** deps.py exposes SessionStore and provider singletons via DI. chat.py persists turn history through the session store (agent-scoped) and includes session ID in the meta event. API composes agents via DI — agents never import api/.
- **Verify:** chat request appends to and replays windowed history; `pnpm ai:test` green
#### Task 2-3-01: Spawn sandboxes with the capture agent
- **Persona:** sandbox-engineer — **REQ:** REQ-3-003
- **Files:** `apps/ai-service/ai_service/sandbox/manager.py` (update), `apps/ai-service/ai_service/sandbox/unshare_backend.py` (update), `apps/ai-service/tests/sandbox/test_telemetry_wiring.py`
- **Action:** `create()` gains optional `task_id`; when set the spawner copies `scripts/sandbox-agent.py` into the sandbox workdir, injects `NC_*` env, and launches the agent as a child of the namespace process (agent lifecycle tied to sandbox lifecycle; destroy kills the agent). No capture when task_id absent (pure shell sandbox).
- **Verify:** `pnpm ai:test` — end-to-end on this box: create sandbox with task_id → run 2 commands via exec → events arrive at the ingest endpoint and land in SQLite in order
### Wave 4: Reliability probe (depends on Wave 3)
#### Task 2-4-01: Dropped-connection durability probe
- **Persona:** backend-engineer — **REQ:** REQ-3-003
- **Files:** `apps/ai-service/tests/telemetry/test_durability.py`
- **Action:** Integration probe: run a capture agent against ingest, kill the WS connection mid-stream (simulate network failure), keep generating events, reconnect, assert the SQLite trace contains **every** event exactly once in order (spool + dedup). Document at-least-once semantics + replay path in README.
- **Verify:** `pnpm ai:test` — test_durability green; README documents semantics
### Must-Haves (Phase 2)
- [ ] BaseAgent unit tests pass (stub agent streams via mock provider)
- [ ] Session store tested: create/append, 20-message window trim, 500-cap LRU eviction, agent-scoped keys
- [ ] Structured output parsing tested against mock provider: fence-strip, first-balanced-object, invalid JSON, one bounded retry, response_format auto-degrade
- [ ] Registry tested: register/get/unknown/duplicate
- [ ] Six prompt modules render learner context without errors
- [ ] Module boundaries hold: `agents/` never imports `api/`; `llm/` never imports `agents/` or `api/`
- [ ] Real telemetry from a live sandbox arrives at ai-service: shell commands, file diffs, run/test results appear as ordered events in SQLite (end-to-end, no mocks)
- [ ] Per-(learner,task) monotonic `seq`; gap detection reports missing seqs; replay yields the complete ordered trace
- [ ] At-least-once proven: transient disconnect + reconnect loses no events; duplicates deduped server-side (durability probe green)
- [ ] Capture agent is stdlib-only (AST-verified) and its lifecycle is tied to the sandbox (destroy kills it)
- [ ] Trace retrievable by learner+task via `GET /v1/telemetry/traces/...`; unknown trace → 404
- [ ] `TraceStore` protocol respected: no api/ code touches SQLite directly; `telemetry/` never imports `agents/` (D-027)
- [ ] `pnpm ai:test` green; `packages/types` telemetry TS types compile (`pnpm typecheck`)
---
## Phase 3: Coach + Tutor Agents
## Phase 3: Process-Trace Grading Engine
**Requirements:** REQ-2-005, REQ-2-006
**Goal:** Both learner-facing conversational agents fully implemented with distinct personas, registered, routed through the chat streaming endpoint
**Requirements:** REQ-3-004
**Goal:** Deterministic feature computation over traces (D-028) → compact digest → LLM rubric scoring via existing D-020 JSON defense → validated structured scores stored in `GradeStore`; LLM never sees the raw trace; engine calibrated against v0.2 mock corpora so process quality separates paste-and-run from iterative debugging
### Wave 1: Agent implementations (parallel — no shared files)
### Wave 1: Features + grades store (parallel — no shared files)
#### Task 3-1-01: Coach agent
- **Persona:** ai-engineer — **REQ:** REQ-2-005
- **Files:** `apps/ai-service/ai_service/prompts/coach.py` (finalize), `apps/ai-service/ai_service/agents/coach.py`, `apps/ai-service/tests/test_coach.py`
- **Action:** Final Coach persona: pacing guidance, motivation, retrieval practice prompts; system prompt injects learner context (active stack, progress). `CoachAgent(BaseAgent)` streams replies. Mock provider scripts a distinct coach-voice response for tests.
- **Verify:** test_coach: build_messages includes system prompt + windowed history; stream_reply yields deltas; on-persona content asserted against mock script
#### Task 3-1-01: Deterministic trace digest (features)
- **Persona:** ai-engineer — **REQ:** REQ-3-004
- **Files:** `apps/ai-service/ai_service/grading/__init__.py`, `apps/ai-service/ai_service/grading/features.py`, `apps/ai-service/tests/grading/__init__.py`, `apps/ai-service/tests/grading/test_features.py`
- **Action:** Pure compute module (D-028): `compute_digest(trace: list[TelemetryEvent]) -> TraceDigest`. Deterministic features: test pass/fail counts + final status; edit count; error/fix cycle count + mean fix latency; idle gaps (>Ns, count + total); command category histogram (build/test/file/nav/debug/other); session duration; first-test-pass offset. `TraceDigest` pydantic model — compact (bounded size, no raw commands), LLM-safe.
- **Verify:** `pnpm ai:test` — test_features green over synthetic traces: paste-and-run trace (0 error/fix cycles, single test pass at end) vs iterative trace (many cycles) produce observably different digests
#### Task 3-1-02: Tutor agent
- **Persona:** ai-engineer — **REQ:** REQ-2-006
- **Files:** `apps/ai-service/ai_service/prompts/tutor.py` (finalize), `apps/ai-service/ai_service/agents/tutor.py`, `apps/ai-service/tests/test_tutor.py`
- **Action:** Final Tutor persona: concept delivery, Socratic questioning, worked examples. `TutorAgent(BaseAgent)` streams replies; mock scripts a distinct tutor-voice response.
- **Verify:** test_tutor mirrors test_coach; Coach and Tutor mock outputs are observably distinct
#### Task 3-1-02: GradeStore protocol + SQLite implementation
- **Persona:** backend-engineer — **REQ:** REQ-3-004
- **Files:** `apps/ai-service/ai_service/grading/store.py`, `apps/ai-service/tests/grading/test_store.py`
- **Action:** `GradeStore` protocol (D-027): `save(grade)`, `get(learner_id, task_id)`, `list_for_learner(learner_id)`, `close()`. SQLModel `GradeRecord`: learner_id, task_id, variant_seed (null until P4), digest (JSON), scores (JSON), verdict, model, created_at. PK (learner_id, task_id). Postgres-migration-ready.
- **Verify:** `pnpm ai:test` test_store green (save/get/list roundtrip, overwrite-on-regrade documented)
### Wave 2: Registration + persona verification (depends on Wave 1)
### Wave 2: Grading engine + calibration (depends on Wave 1)
#### Task 3-2-01: Register Coach + Tutor; document cloud probe
- **Persona:** ai-engineer — **REQ:** REQ-2-005, REQ-2-006
- **Files:** `apps/ai-service/ai_service/agents/registry.py` (update), `apps/ai-service/tests/test_registry.py` (update), `apps/ai-service/README.md` (update)
- **Action:** Register both agents in the explicit registry. Extend test_registry to assert both resolve. Document the manual ollama-cloud persona probe in README (curl commands with `AI_PROVIDER=ollama-cloud`): Coach and Tutor produce distinct on-persona responses; tests remain cloud-free.
- **Verify:** `pnpm ai:test` green; manual probe against ollama-cloud shows distinct personas (documented, not automated)
#### Task 3-2-01: Rubric scoring engine
- **Persona:** ai-engineer — **REQ:** REQ-3-004
- **Files:** `apps/ai-service/ai_service/grading/engine.py`, `apps/ai-service/ai_service/prompts/grading.py` (new), `apps/ai-service/tests/grading/test_engine.py`
- **Action:** `GradingEngine.grade(learner_id, task_id) -> GradeRecord`: load trace via `TraceStore``compute_digest` → render rubric prompt (`prompts/grading.py`: criteria + level anchors for process quality, correctness, debugging discipline, test usage) → LLM structured output through the **existing D-020 4-layer defense** (`agents/structured.py` reused — engine composes it, never duplicates it) → validate `RubricScore` model (per-criterion 0-4 + strengths + gaps + verdict) → persist via `GradeStore`. Grading depends on `llm/` + `telemetry/` + `prompts/` only (boundary). Mock provider scripts deterministic rubric JSON for tests.
- **Verify:** `pnpm ai:test` — test_engine green (mock provider: digest-only prompt asserted — **raw trace string absent from prompt**; validated scores returned; malformed JSON exercises D-020 retry; unknown trace → error)
### Wave 3: Chat endpoint agent routing (depends on Wave 2)
#### Task 3-2-02: Calibration against v0.2 mock corpora
- **Persona:** ai-engineer — **REQ:** REQ-3-004
- **Files:** `apps/ai-service/ai_service/corpus/trace_fixtures.py` (new), `apps/ai-service/tests/grading/test_calibration.py`
- **Action:** Synthetic trace fixtures aligned with v0.2 `corpus/telemetry.py` + `corpus/artifacts.py` scenario IDs (strong/lazy/struggling builder archetypes). Assert grading separates them: strong archetype scores ≥ lazy archetype on process-quality criterion (mock provider maps digest shape → scripted scores; test asserts the ordering contract + that fixture IDs align with existing corpus IDs, D-021).
- **Verify:** `pnpm ai:test` — test_calibration enforces the ordering contract
#### Task 3-3-01: Agent routing on /v1/chat/stream
- **Persona:** backend-engineer — **REQ:** REQ-2-005, REQ-2-006
- **Files:** `apps/ai-service/ai_service/api/chat.py` (update), `apps/ai-service/ai_service/api/deps.py` (update), `apps/ai-service/tests/api/test_chat_stream.py` (update)
- **Action:** Chat request gains `agent` field (validated against the registry; unknown agent → 422). Endpoint resolves the agent via DI, persists to the agent-scoped session, meta event carries the agent name. No autonomous routing in v0.2 (A-007).
- **Verify:** TestClient tests: `agent=coach` and `agent=tutor` route correctly, session scoped per agent, unknown agent rejected
### Wave 3: Grading endpoint (depends on Wave 2)
#### Task 3-3-01: Assessment grade endpoint (real traces)
- **Persona:** backend-engineer — **REQ:** REQ-3-004
- **Files:** `apps/ai-service/ai_service/api/assessment.py` (update), `apps/ai-service/tests/api/test_grading.py`
- **Action:** `POST /v1/assessment/grade {learner_id, task_id}``GradingEngine` → validated `RubricScore` JSON; `GET /v1/assessment/grade/{learner_id}/{task_id}` → stored grade; unknown trace → 404. Composed via DI (api/ owns wiring; engine knows nothing of FastAPI).
- **Verify:** `pnpm ai:test` — test_grading green (grade roundtrip via TestClient with mock provider; 404 on unknown; GET after POST returns same scores)
### Must-Haves (Phase 3)
- [ ] Both agents produce distinct, on-persona responses (mock-asserted; manual ollama-cloud probe documented in README)
- [ ] Agent routing tested: coach/tutor resolve via registry; unknown agent returns 422
- [ ] Both agents exposed end-to-end via `POST /v1/chat/stream` with agent-scoped session history
- [ ] `pnpm ai:test` green; no cloud calls in tests
- [ ] Engine emits structured rubric-aligned scores from a **real process trace** (not pre-baked input) — TestClient roundtrip green
- [ ] Deterministic features computed in code (test pass/fail, edit count, error/fix cycles, idle gaps, command categories); LLM receives the **digest only** — test asserts the raw trace never reaches the prompt (D-028)
- [ ] Scores distinguish process quality: iterative-debugging archetype out-scores paste-and-run on the process criterion (calibration contract test)
- [ ] Grades persisted + retrievable by learner+task via GradeStore (SQLite, protocol-wrapped, D-027)
- [ ] Boundary rules hold: `grading/` imports no `api/`/`agents/` internals except the shared D-020 structured defense; `pnpm ai:test` + `pnpm ai:lint` green
---
## Phase 4: Lab + Assessor Agents
## Phase 4: Variant Task Generation
**Requirements:** REQ-2-007, REQ-2-008
**Goal:** Lab consumes simulated sandbox telemetry and streams in-flow feedback; Assessor applies rubrics to pre-baked artifacts and returns structured scores — both over mock engine inputs
**Requirements:** REQ-3-005
**Goal:** Template library with typed parameter slots (D-029) + seeded LLM instantiation + per-learner variant registry (SQLite) with difficulty-normalization anchors; two learners on the same competency get provably distinct, reproducible, auditable tasks
### Wave 1: Mock engine inputs (parallel — no shared files)
### Wave 1: Templates + store (parallel — no shared files)
#### Task 4-1-01: Simulated sandbox telemetry corpus
- **Persona:** ai-engineer — **REQ:** REQ-2-007
- **Files:** `apps/ai-service/ai_service/corpus/telemetry.py`
- **Action:** Pydantic-typed Lab telemetry scenarios: scripted build-session event streams (keystrokes, commits, test runs, errors, idle gaps) keyed by scenario ID, aligned with packages/mock-data IDs (D-021).
- **Verify:** scenarios import, validate, and are addressable by ID
#### Task 4-1-01: Task template library
- **Persona:** ai-engineer — **REQ:** REQ-3-005
- **Files:** `apps/ai-service/ai_service/variants/__init__.py`, `apps/ai-service/ai_service/variants/templates.py`, `apps/ai-service/tests/variants/__init__.py`, `apps/ai-service/tests/variants/test_templates.py`
- **Action:** ≥3 initial task templates bound to existing competency IDs (D-021 alignment). `TaskTemplate`: id, competency_id, statement skeleton with `{slot}` placeholders, `ParameterSlot[]` (name, type: enum/int-range/string-set, allowed values), `rubric anchors` (difficulty normalization: expected feature envelope — e.g. expected edit-count band — used by grading context), starter-file scaffolds served to the sandbox. Seeded slot sampler is pure code (`random.Random(seed)`), fully reproducible.
- **Verify:** `pnpm ai:test` — test_templates green (slot validation: bad value rejected; seeded sampling reproducible across runs; all templates bind to real competency IDs)
#### Task 4-1-02: Pre-baked artifacts + rubrics corpus
- **Persona:** ai-engineer — **REQ:** REQ-2-008
- **Files:** `apps/ai-service/ai_service/corpus/artifacts.py`
- **Action:** Pre-baked artifacts (code, design, simulation), assessment rubrics (criteria, levels, weights), and defense transcripts keyed by ID — the Assessor's mock inputs (real engines are v0.3+).
- **Verify:** rubric/artifact/transcript fixtures validate; IDs align with TS mock data
#### Task 4-1-02: VariantStore protocol + SQLite implementation
- **Persona:** backend-engineer — **REQ:** REQ-3-005
- **Files:** `apps/ai-service/ai_service/variants/store.py`, `apps/ai-service/tests/variants/test_store.py`
- **Action:** `VariantStore` protocol (D-027): `save(variant)`, `get(learner_id, template_id_or_task_id)`, `list_for_learner(learner_id)`, `list_by_template(template_id)`, `close()`. SQLModel `VariantRecord`: learner_id, task_id (the grading/telemetry task key), template_id, seed, params (JSON), statement (rendered), created_at. Unique (learner_id, template_id).
- **Verify:** `pnpm ai:test` — test_store green (roundtrip, unique constraint, audit listing)
#### Task 4-1-03: TS mock-data alignment for engine inputs
- **Persona:** data-engineer — **REQ:** REQ-2-012
- **Files:** `packages/mock-data/ai-scenarios.ts` (new), `packages/mock-data/index.ts` (update)
- **Action:** Export scenario/artifact ID constants + display metadata used by the Phase 6 learner panels, mirroring `ai_service/corpus/` IDs exactly (D-021). Data-engineer owns the TS side; headers cross-reference the Python corpus.
- **Verify:** `pnpm typecheck` passes; IDs string-equal to corpus IDs
### Wave 2: Generator (depends on Wave 1)
### Wave 2: Agent implementations (depends on Wave 1)
#### Task 4-2-01: Seeded LLM variant generator
- **Persona:** ai-engineer — **REQ:** REQ-3-005
- **Files:** `apps/ai-service/ai_service/variants/generator.py`, `apps/ai-service/ai_service/prompts/variant.py` (new), `apps/ai-service/tests/variants/test_generator.py`
- **Action:** `VariantGenerator.generate(learner_id, template_id) -> VariantRecord`: derive seed (`sha256(template_id|learner_id|milestone)` — reproducible, D-029); sample typed slots in code; render a fill prompt (statement skeleton + concrete slot values) → LLM via D-020 structured defense → unique task statement + starter files → validate → persist (seed + params + statement) via `VariantStore`. Cache: existing (learner,template) returns the stored variant (no duplicate work). Mock provider scripts deterministic statements per seed for tests.
- **Verify:** `pnpm ai:test` — test_generator green: two different learner_ids → distinct statements for the same template; same learner twice → identical stored variant (reproducible); params JSON contains only schema-valid slot values
#### Task 4-2-01: Lab agent
- **Persona:** ai-engineer — **REQ:** REQ-2-007
- **Files:** `apps/ai-service/ai_service/prompts/lab.py` (finalize), `apps/ai-service/ai_service/agents/lab.py`, `apps/ai-service/tests/test_lab.py`
- **Action:** `LabAgent(BaseAgent)` consumes a telemetry scenario, builds messages summarizing the event stream, streams concrete in-flow feedback (what happened, what to adjust, next step). No session chat — scenario-driven.
- **Verify:** test_lab: given a mock scenario, feedback references scenario events (mock-scripted assertions)
### Wave 3: Variant endpoint + TS types (depends on Wave 2)
#### Task 4-2-02: Assessor agent
- **Persona:** ai-engineer — **REQ:** REQ-2-008
- **Files:** `apps/ai-service/ai_service/prompts/assessor.py` (finalize), `apps/ai-service/ai_service/agents/assessor.py`, `apps/ai-service/tests/test_assessor.py`
- **Action:** `AssessorAgent(BaseAgent)` applies a rubric to an artifact + defense transcript via `structured_reply`, returning a pydantic-validated rubric score model (per-criterion scores, strengths, gaps, verdict).
- **Verify:** test_assessor: structured output validates against the rubric model; failure modes exercise the 4-layer defense
#### Task 4-3-01: Variant task endpoint
- **Persona:** backend-engineer — **REQ:** REQ-3-005
- **Files:** `apps/ai-service/ai_service/api/variants.py` (new), `apps/ai-service/ai_service/main.py` (update), `apps/ai-service/tests/api/test_variants.py`
- **Action:** `POST /v1/variants {learner_id, template_id or competency_id}` → generated (or cached) variant: statement, starter files, task_id, seed; `GET /v1/variants/{task_id}` stored variant; `GET /v1/variants?learner_id=` → learner's variants. DI wiring in api/ only.
- **Verify:** `pnpm ai:test` — test_variants green (generate→get roundtrip; cache hit on regenerate; distinct learners → distinct statements asserted at the API layer)
### Wave 3: Registration (depends on Wave 2)
#### Task 4-3-01: Register Lab + Assessor
- **Persona:** ai-engineer — **REQ:** REQ-2-007, REQ-2-008
- **Files:** `apps/ai-service/ai_service/agents/registry.py` (update), `apps/ai-service/tests/test_registry.py` (update)
- **Action:** Register both agents; extend registry tests.
- **Verify:** registry resolves coach/tutor/lab/assessor; `pnpm ai:test` green
### Wave 4: Endpoints (depends on Wave 3)
#### Task 4-4-01: Lab feedback endpoint
- **Persona:** backend-engineer — **REQ:** REQ-2-007
- **Files:** `apps/ai-service/ai_service/api/lab.py`, `apps/ai-service/tests/api/test_lab.py`
- **Action:** `POST /v1/lab/feedback` with scenario ID → resolves corpus scenario + Lab agent → SSE stream using the D-016 envelope (meta names agent=lab). Unknown scenario → 404.
- **Verify:** TestClient streams meta + deltas + done + `[DONE]`; unknown scenario 404
#### Task 4-4-02: Assessment evaluate endpoint
- **Persona:** backend-engineer — **REQ:** REQ-2-008
- **Files:** `apps/ai-service/ai_service/api/assessment.py`, `apps/ai-service/tests/api/test_assessment.py`
- **Action:** `POST /v1/assessment/evaluate` with artifact ID → resolves corpus artifact/rubric/transcript + Assessor agent → JSON response (validated rubric score model). Unknown artifact → 404.
- **Verify:** TestClient returns validated rubric JSON; unknown artifact 404; `pnpm ai:test` green
#### Task 4-3-02: TS types for variants (+ grades)
- **Persona:** data-engineer — **REQ:** REQ-3-005
- **Files:** `packages/types/variants.ts` (new), `packages/types/grading.ts` (new), `packages/types/index.ts` (update)
- **Action:** TS `TaskVariant`, `VariantParams`, `RubricScore`, `GradeRecord` matching Python models (cross-referencing headers; same field names). Consumed by Phase 6 surfaces.
- **Verify:** `pnpm typecheck` passes
### Must-Haves (Phase 4)
- [ ] Lab produces scenario-relevant in-flow feedback for mock telemetry scenarios (mock provider, tested)
- [ ] Assessor returns structured rubric scores (pydantic-validated JSON) for pre-baked artifacts/transcripts
- [ ] `POST /v1/lab/feedback` streams (meta → deltas → done → `[DONE]`); `POST /v1/assessment/evaluate` returns validated JSON
- [ ] Unknown scenario/artifact IDs return 404
- [ ] Corpus IDs align with packages/mock-data (D-021); `pnpm ai:test` and `pnpm typecheck` green
- [ ] Two learners requesting the same competency receive **provably distinct** task variants (API-level test)
- [ ] Seed derivation reproducible: same (template, learner) → same variant, served from cache without a second LLM call (D-029)
- [ ] Variant seed + typed params persisted + auditable (VariantStore listing; proctoring cross-check path exists)
- [ ] Difficulty normalization anchors present per template and shipped to the grader prompt context
- [ ] Starter-file scaffolds defined per template (P6 wires them into the sandbox workdir)
- [ ] `pnpm ai:test` + `pnpm typecheck` green; `variants/` imports no api/ (boundary)
---
## Phase 5: Proctor + Mentor Agents
## Phase 5: Oral / Voice Defense
**Requirements:** REQ-2-009, REQ-2-010
**Goal:** Proctor classifies integrity signals with coaching interventions from mock telemetry; Mentor generates long-horizon career narrative; both exposed via endpoints
**Requirements:** REQ-3-006
**Goal:** `VoiceProvider` protocol with mock-first STT/TTS (D-030) + seventh `Examiner` agent streaming over existing SSE + HTTP audio endpoints + `DefenseStore` persisting transcript + integrity signals — nothing blocks on a real voice key (mock + browser fallback are first-class)
### Wave 1: Proctor scenarios + Mentor agent (parallel — no shared files)
### Wave 1: Voice provider layer (parallel — no shared files)
#### Task 5-1-01: Proctor telemetry scenarios
- **Persona:** ai-engineer — **REQ:** REQ-2-009
- **Files:** `apps/ai-service/ai_service/corpus/telemetry.py` (update)
- **Action:** Add proctor scenarios: tab switches, idle time, paste events, focus loss — scripted integrity-relevant event sets keyed by scenario ID.
- **Verify:** proctor scenarios validate; distinguishable from lab scenarios by type
#### Task 5-1-01: VoiceProvider protocol + mock provider
- **Persona:** voice-engineer — **REQ:** REQ-3-006
- **Files:** `apps/ai-service/ai_service/voice/__init__.py`, `apps/ai-service/ai_service/voice/base.py`, `apps/ai-service/ai_service/voice/mock.py`, `apps/ai-service/tests/voice/__init__.py`, `apps/ai-service/tests/voice/test_mock.py`
- **Action:** `VoiceProvider` protocol mirroring `LLMProvider` (D-030): `transcribe(audio: bytes, fmt) -> TranscriptSegment` + `synthesize(text, voice) -> AsyncIterator[bytes]` (audio chunks). `MockVoiceProvider`: deterministic canned transcript (scripted per test), canned 1kHz-tone WAV bytes; scripted failure modes (provider down, empty audio). `voice/` never imports `agents/` or `api/`.
- **Verify:** `pnpm ai:test` — test_mock green (deterministic: two identical transcribe calls → identical segments; synthesize yields non-empty bytes; failure modes trigger)
#### Task 5-1-02: Mentor agent (+ registration)
- **Persona:** ai-engineer — **REQ:** REQ-2-010
- **Files:** `apps/ai-service/ai_service/prompts/mentor.py` (finalize), `apps/ai-service/ai_service/agents/mentor.py`, `apps/ai-service/tests/test_mentor.py`, `apps/ai-service/ai_service/agents/registry.py` (update)
- **Action:** `MentorAgent(BaseAgent)`: long-horizon career narrative — trajectory story, competency-stack progression guidance, market positioning — streaming, session-backed. Registered centrally in `registry.py` (single registration pattern, G-4).
- **Verify:** test_mentor: narrative references learner context (mock-scripted); registry resolves mentor
#### Task 5-1-02: OpenAI-compatible audio provider + browser fallback descriptor + factory
- **Persona:** voice-engineer — **REQ:** REQ-3-006
- **Files:** `apps/ai-service/ai_service/voice/openai_audio.py`, `apps/ai-service/ai_service/voice/browser.py`, `apps/ai-service/ai_service/voice/factory.py`, `apps/ai-service/tests/voice/test_openai_audio.py`, `apps/ai-service/tests/voice/test_factory.py`
- **Action:** `OpenAIAudioProvider`: STT = POST multipart to `{AI_VOICE_BASE_URL}/audio/transcriptions` (Whisper-compatible, Bearer `AI_VOICE_API_KEY`); TTS = POST to `/audio/speech` streaming mp3/pcm chunks; reuses lifespan httpx client. `browser.py`: fallback **descriptor** only (`sr_available: true`, endpoint hints) consumed by the web client to select browser-native `SpeechRecognition`/`speechSynthesis` when no server key configured. `factory.py`: `AI_VOICE_PROVIDER=openai-audio | browser | mock` (default: mock when no key — **mock-first**, nothing blocks on a real voice key).
- **Verify:** `pnpm ai:test` — httpx MockTransport tests: transcription multipart parse, TTS byte stream, 4xx/5xx error mapping, factory selects mock with empty key; zero network calls
### Wave 2: Proctor agent + Mentor endpoint (depends on Wave 1)
#### Task 5-1-03: DefenseStore protocol + SQLite implementation
- **Persona:** backend-engineer — **REQ:** REQ-3-006
- **Files:** `apps/ai-service/ai_service/voice/defense_store.py`, `apps/ai-service/tests/voice/test_defense_store.py`
- **Action:** `DefenseStore` protocol (D-027): `start(defense)`, `append_turn(defense_id, turn)`, `finalize(defense_id, integrity_signals)`, `get(defense_id)`, `list_for_learner(learner_id)`, `close()`. SQLModel `DefenseRecord` (id, learner_id, task_id, status, created/finished_at) + `DefenseTurn` (defense_id FK, turn seq, role examiner|learner, text, ts, latency_ms) + integrity signals JSON on the record (long pauses, off-scope cadence markers — A-109).
- **Verify:** `pnpm ai:test` — test_defense_store green (start→append turns→finalize→get roundtrip; ordered turns by seq)
#### Task 5-2-01: Proctor agent (+ registration)
- **Persona:** ai-engineer — **REQ:** REQ-2-009
- **Files:** `apps/ai-service/ai_service/prompts/proctor.py` (finalize), `apps/ai-service/ai_service/agents/proctor.py`, `apps/ai-service/tests/test_proctor.py`, `apps/ai-service/ai_service/agents/registry.py` (update)
- **Action:** `ProctorAgent(BaseAgent)`: consumes proctor scenario → `structured_reply` returns pydantic-validated signal classification (severity, signal type) + recommended coaching intervention (supportive, not punitive). Registered centrally in `registry.py` (single registration pattern, G-4).
- **Verify:** test_proctor: classified signals + interventions validate for each mock scenario; registry resolves all six agents
### Wave 2: Examiner agent (depends on Wave 1)
#### Task 5-2-02: Mentor narrative endpoint
- **Persona:** backend-engineer — **REQ:** REQ-2-010
- **Files:** `apps/ai-service/ai_service/api/mentor.py`, `apps/ai-service/tests/api/test_mentor.py`
- **Action:** `POST /v1/mentor/narrative` → Mentor agent → SSE stream with D-016 envelope, session-backed.
- **Verify:** TestClient streams meta (agent=mentor) → deltas → done → `[DONE]`
#### Task 5-2-01: Examiner agent (seventh agent)
- **Persona:** ai-engineer — **REQ:** REQ-3-006
- **Files:** `apps/ai-service/ai_service/agents/examiner.py`, `apps/ai-service/ai_service/prompts/examiner.py` (new), `apps/ai-service/ai_service/agents/registry.py` (update: central registration, G-4 pattern), `apps/ai-service/tests/agents/test_examiner.py`
- **Action:** `ExaminerAgent(BaseAgent)` (D-030/A-109): builds questions from learner transcript + trace digest + (P4) variant statement; probes understanding + challenges process choices ("why did you choose X at step N?"); streams questions over the existing SSE pipeline; `structured` verdict mode returns verdict + per-answer integrity signal list (long pause flags, off-scope answers) computed from turn metadata; calls voice **only through the `VoiceProvider` protocol** (PERSONAS conflict rule — never concrete providers). Session-scoped history reused from v0.2.
- **Verify:** `pnpm ai:test` — test_examiner green (question stream references trace-digest facts; verdict structured output validates via D-020 defense; registry resolves all seven agents; mock-VoiceProvider wiring through protocol only — asserted by import scan in test)
### Wave 3: Proctor endpoint (depends on Wave 2)
### Wave 3: Defense endpoints (depends on Wave 2)
#### Task 5-3-01: Proctor signals endpoint
- **Persona:** backend-engineer — **REQ:** REQ-2-009
- **Files:** `apps/ai-service/ai_service/api/proctor.py`, `apps/ai-service/tests/api/test_proctor.py`
- **Action:** `POST /v1/proctor/signals` with scenario ID → resolves corpus scenario + Proctor agent → JSON response (validated signals + interventions). Unknown scenario → 404.
- **Verify:** TestClient returns classified signals JSON; `pnpm ai:test` green — full suite (all six agents registered)
#### Task 5-3-01: Defense session + audio endpoints
- **Persona:** backend-engineer — **REQ:** REQ-3-006
- **Files:** `apps/ai-service/ai_service/api/defense.py` (new), `apps/ai-service/ai_service/main.py` (update), `apps/ai-service/tests/api/test_defense.py`
- **Action:** `POST /v1/defense/start {learner_id, task_id}` → creates DefenseRecord + streams the first examiner question (SSE, agent=examiner); `POST /v1/defense/{id}/answer` (multipart audio from browser MediaRecorder, or `{text}` for typed fallback) → STT via VoiceProvider → append learner turn → stream examiner follow-up (SSE) → TTS audio chunks over `GET /v1/defense/{id}/audio/{turn_id}`; `POST /v1/defense/{id}/finish` → verdict + integrity signals persisted; `GET /v1/defense/{id}` → full transcript + signals. Browser-fallback mode: when provider=browser, start returns the fallback descriptor instead of server audio.
- **Verify:** `pnpm ai:test` — test_defense green (full loop with mock voice + mock LLM: start → answer(text) → answer(audio bytes) → finish → transcript retrievable with per-turn latency; unknown id → 404; `GET` signals present after finish)
### Wave 4: Turn-latency probe (depends on Wave 3)
#### Task 5-4-01: Latency budget probe + README budget doc
- **Persona:** voice-engineer — **REQ:** REQ-3-006
- **Files:** `apps/ai-service/tests/voice/test_latency.py`, `apps/ai-service/README.md` (update: latency budget section + voice config docs)
- **Action:** Instrument per-turn latency (STT ms + LLM TTFT ms + TTS ms) recorded on each DefenseTurn. Deterministic timing probe over mock providers: total turn budget < target (documented; mock runs are near-instant, so the test asserts **instrumentation presence + budget field populated** rather than wall-clock). README documents the conversational-budget target and how to run a manual live probe with a real voice endpoint (`AI_VOICE_PROVIDER=openai-audio` + key in `.ciagent/.env.secrets` — explicitly optional, never in tests/commits).
- **Verify:** `pnpm ai:test` — test_latency green (latency_ms fields populated on every turn; budget constant defined); README documents the manual live probe as out-of-band
### Must-Haves (Phase 5)
- [ ] Proctor produces classified signals with recommended coaching interventions for each mock scenario (structured JSON, validated)
- [ ] Mentor produces coherent long-horizon career narrative (streaming, session-backed)
- [ ] `POST /v1/proctor/signals` returns validated JSON; `POST /v1/mentor/narrative` streams
- [ ] Registry resolves all six agents; full ai-service test suite green, cloud-free
- [ ] Spoken defense runs end-to-end over HTTP with mock providers: start → answer (audio + typed fallback) → examiner follow-up streams → verdict + transcript persisted (automated)
- [ ] Examiner is the seventh registered agent; streams over the existing SSE envelope (meta agent=examiner)
- [ ] Transcript + integrity signals persisted via DefenseStore and retrievable; per-turn latency instrumented (A-109)
- [ ] `VoiceProvider` protocol respected: examiner + api touch voice only via the protocol; mock-first — **no task requires a real voice key to pass**
- [ ] Browser-native SR/TTS fallback descriptor returned when no server key configured (mock/no-key path first-class, D-030)
- [ ] Optional voice config documented: `AI_VOICE_BASE_URL` / `AI_VOICE_API_KEY` / models in `.env.example` + README; keys only in gitignored `.ciagent/.env.secrets`; tests never call a voice API
- [ ] Boundary rules hold: `voice/` imports no `agents/`/`api/`; `pnpm ai:test` + `pnpm ai:lint` green
- [ ] Manual live-voice turn-latency probe documented in README (out-of-band, optional)
---
## Phase 6: Learner Surface Integration
## Phase 6: Agent Re-grounding + Learner Surface Integration
**Requirements:** REQ-2-011, REQ-2-012
**Goal:** v0.1 learner surfaces wired to the real ai-service: streaming chat with agent switcher, Lab/Assessor/Proctor/Mentor outputs surfaced, error/loading states, build + typecheck green
**Note (G-2):** End-to-end verification may run with `AI_PROVIDER=mock` as a fallback — the requirement is the real ai-service over HTTP (not canned client-side responses); provider choice is service-internal. This prevents an ollama-cloud outage from blocking P6 verification. Cloud persona probes remain separate (Task 3-2-01).
**Requirements:** REQ-3-007, REQ-3-008
**Goal:** Lab/Assessor/Proctor consume real engine inputs (live telemetry, grading output, defense signals) with **no mock fallback in the learner path**; the v0.1 sandbox + assessment mockups become real — in-browser xterm.js build/run, live telemetry panel, live voice/typed defense, live grading; `pnpm build` + `pnpm typecheck` + full `pnpm ai:test` green
**Note:** E2E verification runs against the real engines over HTTP with `AI_PROVIDER=mock` + `AI_VOICE_PROVIDER=mock` permitted (G-2 precedent) — the requirement is real engine plumbing (sandbox/telemetry/grading/defense over real endpoints, no corpus mocks in the learner path); a cloud outage must not block P6.
### Wave 1: Client plumbing + primitives (parallel — no shared files)
### Wave 1: Agent re-grounding (parallel — no shared files)
#### Task 6-1-01: useChatStream hook
- **Persona:** frontend-engineer — **REQ:** REQ-2-011
- **Files:** `apps/web/hooks/use-chat-stream.ts`, `apps/web/.env.example` (update)
- **Action:** `useChatStream(agent)` hook: `fetch` POST to `${NEXT_PUBLIC_AI_SERVICE_URL}/v1/chat/stream` (default `http://localhost:8420`, A-002 — no API-route proxy); consumes `ReadableStream` with byte buffering, frame split on `\n\n`, joined `data:` lines; **ignores frames containing no `data:` lines (sse-starlette `: ping` keep-alive comment frames)** — TestClient streams are too short to surface pings, but real cloud delta gaps emit them (G-1); handles meta / delta / done / error events and `[DONE]` sentinel; idempotent `AbortController.abort()` in effect cleanup; exposes `{messages, isStreaming, error, send, retry, abort}`. `.env.example` gains `NEXT_PUBLIC_AI_SERVICE_URL`.
- **Verify:** hook unit-tested or exercised via the chat UI; unmount mid-stream aborts cleanly (no state updates after unmount)
#### Task 6-1-01: Lab agent on live telemetry
- **Persona:** ai-engineer — **REQ:** REQ-3-007
- **Files:** `apps/ai-service/ai_service/agents/lab.py` (update), `apps/ai-service/ai_service/prompts/lab.py` (update: render real trace digest), `apps/ai-service/tests/agents/test_lab_live.py` (new)
- **Action:** Lab consumes a **live trace digest** (grading/features `compute_digest` over `TraceStore` events) instead of `corpus/telemetry.py`. build_messages renders digest facts (recent commands, failing tests, idle). Mock-provider scripts assert digest-derived content. v0.2 corpus path removed from the agent (dormant corpus retained until Task 6-1-04 check).
- **Verify:** `pnpm ai:test` — test_lab_live green: feedback references events actually present in a seeded SQLite trace (not corpus fixtures); no `corpus.telemetry` import in `agents/lab.py` (AST-asserted)
#### Task 6-1-02: Agent switcher + streaming primitives
- **Persona:** design-system-engineer — **REQ:** REQ-2-011
- **Files:** `packages/ui/src/primitives/agent-switcher.tsx`, `packages/ui/src/primitives/stream-status.tsx`, `packages/ui/src/primitives/toast.tsx`, `packages/ui/src/primitives/index.ts` (update), `packages/ui/src/index.ts` (update)
- **Action:** Token-driven primitives: AgentSwitcher (segmented coach/tutor control with active state), StreamStatus (idle/streaming/error indicator), Toast with error variant. Dark mode + WCAG AA contrast; exported from `@nextcraft/ui`.
- **Verify:** primitives import from `@nextcraft/ui`; storybook stories render (dark + light)
#### Task 6-1-02: Assessor agent on grading output
- **Persona:** ai-engineer — **REQ:** REQ-3-007
- **Files:** `apps/ai-service/ai_service/agents/assessor.py` (update), `apps/ai-service/ai_service/prompts/assessor.py` (update), `apps/ai-service/tests/agents/test_assessor_live.py` (new)
- **Action:** Assessor consumes `GradeStore` output (validated `RubricScore` + digest) for learner+task instead of pre-baked artifacts/transcripts; renders strengths/gaps/verdict with rubric-anchored coaching framing. Structured output unchanged (D-020).
- **Verify:** `pnpm ai:test` — test_assessor_live green: given a real stored grade, Assessor output reflects its scores; corpus artifact path gone from the agent (AST-asserted)
### Wave 2: Chat rewrite + Byte viewer panel (depends on Wave 1)
#### Task 6-1-03: Proctor agent on telemetry + defense signals
- **Persona:** ai-engineer — **REQ:** REQ-3-007
- **Files:** `apps/ai-service/ai_service/agents/proctor.py` (update), `apps/ai-service/ai_service/prompts/proctor.py` (update), `apps/ai-service/tests/agents/test_proctor_live.py` (new)
- **Action:** Proctor consumes real integrity inputs: idle gaps + command cadence from the trace digest + defense integrity signals from `DefenseStore` → classified signals + coaching interventions (supportive tone retained). Cross-checks variant seed params (P4) for off-template work.
- **Verify:** `pnpm ai:test` — test_proctor_live green: signals derived from seeded real trace + defense records; corpus proctor scenarios no longer imported (AST-asserted)
#### Task 6-2-01: Real streaming learner chat
- **Persona:** frontend-engineer — **REQ:** REQ-2-011
- **Files:** `apps/web/components/learner/ai-tutor-chat.tsx` (rewrite), `apps/web/components/learner/agent-switcher.tsx` (composition wrapper)
- **Action:** Replace the canned `aiTutorResponses` behavior with `useChatStream`: agent switcher (Coach/Tutor per A-007), token-by-token rendering, streaming cursor + loading state, error state with retry button when ai-service is down (A-010), suggested-action chips from the meta event. Seed welcome message stays static.
- **Verify:** with ai-service running, messages stream visibly token-by-token; with ai-service stopped, error state + retry appears (no crash, no console errors)
#### Task 6-1-04: Corpus dormancy + mockup removal verification
- **Persona:** lead-developer — **REQ:** REQ-3-007
- **Files:** `apps/ai-service/ai_service/corpus/telemetry.py`, `apps/ai-service/ai_service/corpus/artifacts.py`, `apps/ai-service/ai_service/corpus/` (README note), `apps/ai-service/tests/test_corpus_dormancy.py` (new)
- **Action:** Verify no production code path imports `corpus/telemetry.py` or `corpus/artifacts.py` anymore (test scans imports across `agents/`, `api/`, engines). Retain files as Phase-3 calibration history with a header note marking them **dormant — v0.2 mocks, not used at runtime**; learner-context corpus stays (agents still need learner context). Disposes v0.2's G-5-class dead-code risk deliberately.
- **Verify:** `pnpm ai:test` — test_corpus_dormancy green (zero runtime importers); suite otherwise unchanged
#### Task 6-2-02: Byte viewer Tutor panel
- **Persona:** frontend-engineer — **REQ:** REQ-2-011 (agent routing: byte viewer always uses Tutor, A-007)
- **Files:** `apps/web/app/(learner)/learn/[competencyId]/page.tsx` (update), `apps/web/components/learner/byte-tutor-panel.tsx` (new)
- **Action:** Byte viewer gains a Tutor explanation panel (byte viewer always uses Tutor, A-007): "Explain this byte" streams a Socratic concept walkthrough for the current competency via useChatStream (agent fixed to tutor).
- **Verify:** on a byte page, the panel streams a Tutor explanation; error state when service down
### Wave 2: Client plumbing + design primitives (parallel — no shared files)
### Wave 3: Lab / Assessment / Mentor panels (depends on Wave 1 hook; parallel — no shared files)
#### Task 6-2-01: Terminal + sandbox engine client
- **Persona:** frontend-engineer — **REQ:** REQ-3-008
- **Files:** `apps/web/package.json` (update: `@xterm/xterm`, `@xterm/addon-fit`), `pnpm-lock.yaml` (update), `apps/web/hooks/use-sandbox-session.ts` (new), `apps/web/lib/engine-client.ts` (new), `apps/web/.env.example` (update)
- **Action:** `engine-client.ts`: typed fetch/WS client for `/v1/sandboxes` (create/destroy — learner_id from the v0.3 mock session constant), `/v1/variants`, `/v1/assessment/grade`, `/v1/telemetry/traces`, `/v1/defense/*`; base `NEXT_PUBLIC_AI_SERVICE_URL`. `use-sandbox-session.ts`: create sandbox+variant on task open → wire a WebSocket terminal channel (xterm attach: keystrokes → exec stdin in sandbox, stdout → terminal render; implemented as a WS relay endpoint consumed from api/sandboxes — reused ingest WS pattern) → destroy on unmount (idempotent cleanup, AbortController pattern). 503 pool-full → user-facing "environment busy, retry" state (D-032 surfaced honestly).
- **Verify:** `pnpm install && pnpm typecheck` pass; hook unmount destroys the sandbox (manual probe: `curl localhost:8420/v1/sandboxes` shows count drop after navigation)
#### Task 6-3-01: Sandbox Lab feedback panel
- **Persona:** frontend-engineer — **REQ:** REQ-2-012
- **Files:** `apps/web/app/(learner)/build/[competencyId]/page.tsx` (update), `apps/web/components/learner/lab-feedback-panel.tsx` (new)
- **Action:** Sandbox telemetry sidebar gains a Lab feedback panel: posts the scenario ID (from `packages/mock-data/ai-scenarios`) to `/v1/lab/feedback`, streams in-flow feedback into the panel; loading + error states.
- **Verify:** sandbox page streams Lab feedback for the mock scenario; error state when service down
#### Task 6-2-02: New design primitives
- **Persona:** design-system-engineer — **REQ:** REQ-3-008
- **Files:** `packages/ui/src/primitives/terminal-frame.tsx` (new), `packages/ui/src/primitives/mic-control.tsx` (new), `packages/ui/src/primitives/grade-badge.tsx` (new), `packages/ui/src/primitives/telemetry-status.tsx` (new), `packages/ui/src/primitives/transcript-viewer.tsx` (new), `packages/ui/src/primitives/index.ts` (update), `packages/ui/src/index.ts` (update)
- **Action:** Token-driven primitives: TerminalFrame (chrome around xterm with connection status), MicControl (record/stop with consent state + no-mic fallback state, MediaRecorder permission UX), GradeBadge (verdict rendering), TelemetryStatus (live event pulse / disconnected indicator), TranscriptViewer (examiner/learner turn list). Dark mode + WCAG AA; stories for each.
- **Verify:** primitives import from `@nextcraft/ui`; Storybook stories render dark + light; `pnpm build` (ui package) passes
#### Task 6-3-02: Assessment Assessor + Proctor surfaces
- **Persona:** frontend-engineer — **REQ:** REQ-2-012
- **Files:** `apps/web/app/(learner)/defend/[competencyId]/page.tsx` (update), `apps/web/components/learner/assessor-results-panel.tsx` (new), `apps/web/components/learner/proctor-banner.tsx` (new)
- **Action:** Assessment mockup: AI reviewer panel calls `/v1/assessment/evaluate` with the artifact ID and renders the structured rubric scores (per-criterion bars, strengths, gaps, verdict); a Proctor integrity banner surfaces `/v1/proctor/signals` classifications with coaching tone. Loading skeletons + error states.
- **Verify:** defend page renders real Assessor rubric output + Proctor banner; error states when service down
#### Task 6-2-03: Defense TS types
- **Persona:** data-engineer — **REQ:** REQ-3-008
- **Files:** `packages/types/defense.ts` (new), `packages/types/index.ts` (update)
- **Action:** TS `DefenseSession`, `DefenseTurn`, `IntegritySignal`, `Verdict` mirroring P5 Python models (cross-referencing header).
- **Verify:** `pnpm typecheck` passes
#### Task 6-3-03: Dashboard Mentor panel
- **Persona:** frontend-engineer — **REQ:** REQ-2-012
- **Files:** `apps/web/app/(learner)/dashboard/page.tsx` (update), `apps/web/components/learner/mentor-panel.tsx` (new)
- **Action:** Learner dashboard gains a Mentor panel: streams career narrative from `/v1/mentor/narrative` (learner progress context), with regenerate button, loading + error states. Sits alongside the existing AI tutor chat.
- **Verify:** dashboard shows streaming Mentor narrative; error state when service down
### Wave 3: Real build surface (depends on Wave 2)
#### Task 6-3-01: Sandbox mockup → real in-browser IDE
- **Persona:** frontend-engineer — **REQ:** REQ-3-008
- **Files:** `apps/web/app/(learner)/build/[competencyId]/page.tsx` (rewrite), `apps/web/components/learner/sandbox-terminal.tsx` (new), `apps/web/components/learner/file-tree.tsx` (new), `apps/web/components/learner/run-controls.tsx` (new), `apps/web/components/learner/lab-feedback-panel.tsx` (update: live trace)
- **Action:** Replace the mockup with the real build environment (A-103): file tree (HTTP CRUD into the sandbox workdir via `/v1/sandboxes/{id}/files` routes added to api/sandboxes — read/write/list workspace files; reused by run/test), syntax-highlight editor (existing), xterm.js terminal on `use-sandbox-session`, **Run**/**Test** buttons (exec in sandbox via relay; results stream to terminal), starter files from the P4 variant scaffold. Lab panel now posts `learner_id+task_id` → streams Lab feedback over the **live** trace (no scenario IDs). Telemetry sidebar shows live TelemetryStatus. Pool-full 503 → busy state with retry.
- **Verify:** with ai-service up: open `/build/comp-01` → variant statement + starter files load → type `echo hello` in terminal → real output renders; Run executes tests in-sandbox; Lab panel streams digest-derived feedback; telemetry status shows live events. Manual probe documented; `pnpm typecheck` green
### Wave 4: Live defense + grading surfaces (depends on Waves 2-3)
#### Task 6-4-01: Assessment mockup → live defense + live grading
- **Persona:** frontend-engineer — **REQ:** REQ-3-008
- **Files:** `apps/web/app/(learner)/defend/[competencyId]/page.tsx` (rewrite), `apps/web/components/learner/defense-session.tsx` (new), `apps/web/components/learner/assessor-results-panel.tsx` (update), `apps/web/components/learner/proctor-banner.tsx` (update), `apps/web/components/learner/oral-defense-interface.tsx` (rewrite or remove)
- **Action:** Real assessment flow: **Start Defense** → POST `/v1/defense/start` → examiner question streams → learner answers via MicControl (MediaRecorder webm/opus → multipart POST) with typed fallback when mic denied or `provider=browser` (native `SpeechRecognition`/`speechSynthesis` path per fallback descriptor) → follow-ups stream → **Finish** → verdict + integrity signals panel (TranscriptViewer, GradeBadge) + **Grade My Work** → POST `/v1/assessment/grade` → structured rubric bars render. Proctor banner reads signals for task_id. Loading + error states throughout; no mock defense data remains in the learner path.
- **Verify:** with ai-service up: full defense loop runs in-browser (typed fallback acceptable in CI-less manual probe; mic path exercised manually with permission granted); grade panel renders real rubric scores; `pnpm typecheck` green
### Wave 5: End-to-end verification (depends on Waves 3-4)
#### Task 6-5-01: Full learner-path E2E probe + green builds
- **Persona:** backend-engineer — **REQ:** REQ-3-007, REQ-3-008
- **Files:** `apps/ai-service/tests/api/test_e2e_credential_flow.py` (new), `apps/ai-service/README.md` (update: E2E probe doc)
- **Action:** Endpoint-level E2E (mock LLM/voice providers, real engines): create variant → create sandbox with task_id → ingest trace events via real sandbox exec → POST grade → start defense (typed answers) → finish → assert: trace persisted, grade stored + digest-linked, defense transcript + signals stored, Proctor/Assessor endpoints serve them. No corpus fixture anywhere in the flow (AST-asserted). Run `pnpm build` + `pnpm typecheck` + full `pnpm ai:test` at repo root; fix all failures before phase ship.
- **Verify:** `pnpm ai:test` green incl. test_e2e_credential_flow; `pnpm build` + `pnpm typecheck` green; README E2E probe section documents the manual browser pass
### Must-Haves (Phase 6)
- [ ] With ai-service running: learner chat at http://localhost:3000/dashboard streams real responses token-by-token (mock provider fallback allowed per G-2 — service over HTTP is the requirement)
- [ ] Agent switcher flips Coach ↔ Tutor and the response persona changes accordingly
- [ ] Hook tolerates keep-alive comment frames (`: ping`, no data lines) during live streams (G-1)
- [ ] With ai-service stopped: all chat/panels show error states with retry — no crashes, no unhandled promise rejections, no console errors
- [ ] Byte viewer, sandbox, and assessment mockups surface Tutor/Lab/Assessor/Proctor outputs; dashboard shows Mentor narrative
- [ ] Unmounting/navigating mid-stream aborts cleanly (no post-unmount state updates)
- [ ] `pnpm build` and `pnpm typecheck` pass; `pnpm ai:test` still green
- [ ] Lab/Assessor/Proctor operate on real inputs with **no mock fallback in the learner path** (AST-verified: no corpus telemetry/artifact/proctor imports in production paths)
- [ ] Learner builds in-browser for real: xterm.js terminal executes in a namespace sandbox; file tree CRUD + Run/Test work; starter files come from the learner's variant scaffold
- [ ] Live telemetry: build activity streams to ai-service and the sidebar shows live status (TelemetryStatus); trace persisted in SQLite
- [ ] Live defense: start → answer (mic or typed fallback) → examiner follow-ups → finish → transcript + integrity signals + verdict rendered; browser-native path works with no server voice key
- [ ] Live grading: grade request returns structured rubric scores computed from the real trace digest; grade panel renders them
- [ ] 503 pool-full surfaced honestly in UI; navigating away destroys the sandbox (no leaked sandboxes — `GET /v1/sandboxes` manual probe)
- [ ] Examiner remains protocol-clean (voice only via `VoiceProvider`); module boundary rules hold across all new code
- [ ] `pnpm build` and `pnpm typecheck` pass; full `pnpm ai:test` green; no cloud/voice calls in any automated test
- [ ] Release-note input (for P7): v0.3 ships real engines; **identity/age-gating (KYC) remains deferred — age-gating is still a visual mockup** (A-110); sandbox scope is coding-IDE only (design tool/simulation deferred to v0.4, D-025)
---
## Phase 7: Final Review + Ship (no planned tasks)
Orchestrated by the SHIP stage, not this plan: multi-persona code review (correctness, testing, secrets hygiene — key absent from code/logs/commits/errors, localhost-only CORS, no PII in prompts), project health audit (reconstruction test, .ciagent/ discipline, branch/commit hygiene), then merge milestone → main, tag v0.2.0, create Gitea release, mark all 12 v0.2 requirements complete.
Orchestrated by the SHIP stage, not this plan: multi-persona code review (correctness, testing, module boundaries, secrets hygiene — keys absent from code/logs/commits/errors, localhost-only CORS, no PII in prompts), project health audit (reconstruction, .ciagent/ discipline, branch/commit hygiene), then merge milestone → main, tag the final v0.2.x patch, create the Gitea release, mark all 8 v0.3 requirements complete.
**Release-note honesty (G-5):** the v0.2.0 release note must explicitly state that Lab/Assessor/Proctor operate on mock engine inputs (real engines are v0.3+), per D-015.
**Release-note honesty:** the release note must state (a) Lab/Assessor/Proctor now run on real engine inputs (v0.2 mock-input caveat retired), (b) sandbox scope = coding IDE only — design tool + simulation environments deferred to v0.4 (D-025), (c) identity/age-gating (KYC) deferred per founder directive — age-gating remains the v0.1 visual flow mockup (A-110), (d) voice runs mock-first with browser-native fallback — a server STT/TTS key is optional and configured only via `.ciagent/.env.secrets`.
**Dead-code disposal (G-5):** review must dispose of `aiTutorResponses` (packages/mock-data/ai-tutor-responses.ts) — its only consumer is rewritten in Task 6-2-01; remove the export or mark it deprecated.
**Secrets-hygiene checklist (P7):** `.ciagent/.env.secrets` gitignored and never committed; `AI_VOICE_API_KEY` / `AI_TUTOR_API_KEY` referenced only via env; SQLite DBs + sandbox dirs gitignored; no keys in logs, error messages, or test fixtures.
**Disposal checks (G-5 class):** v0.2 dormant corpus files carry the dormant-header note (Task 6-1-04); any now-unused mock-data exports for the old sandbox/defense mockups (e.g. `aiTutorResponses`-class leftovers) must be removed or deprecated by review.
---
## User-Facing Surface
The primary user-facing surface is the **learner dashboard and learning flow** at `http://localhost:3000`, backed by the real ai-service at `http://localhost:8420`:
The primary user-facing surface is the **learner build + defend flow** at `http://localhost:3000`, backed by the real engines in ai-service at `http://localhost:8420`:
- `/dashboard` — AI tutor chat (Coach/Tutor switcher, streaming) + Mentor career-narrative panel
- `/learn/[competencyId]` — byte viewer with streaming Tutor explanations
- `/build/[competencyId]`sandbox with Lab in-flow feedback panel (mock telemetry)
- `/defend/[competencyId]`assessment with live Assessor rubric scores + Proctor integrity banner
- `/dashboard` — AI tutor chat (Coach/Tutor, streaming) + Mentor panel (unchanged from v0.2)
- `/learn/[competencyId]` — byte viewer with streaming Tutor explanations (unchanged)
- `/build/[competencyId]`**real in-browser IDE**: xterm.js terminal into a namespace sandbox, file tree, Run/Test, live telemetry status, live Lab feedback, per-learner variant task statement
- `/defend/[competencyId]`**live oral defense + live grading**: Examiner voice/typed dialogue, transcript + integrity signals, real rubric scores from the process trace
The marketplace, employer, and admin surfaces are unchanged from v0.1.
The marketplace, employer, and admin surfaces are unchanged from v0.1/v0.2.
## Happy Path
1. Learner opens `/dashboard` → chat shows welcome message; meta event confirms coach/model in the stream
2. Learner types "I'm stuck on multi-agent communication" → reply streams token-by-token with pacing guidance + a retrieval-practice prompt
3. Learner switches to **Tutor** → asks the same question → gets a Socratic concept walkthrough instead
4. Learner opens a byte tutorial → Tutor panel streams an explanation of the current competency
5. Learner opens the build sandbox → Lab panel streams feedback on the simulated telemetry scenario
6. Learner opens the defense mockup → Assessor panel shows structured rubric scores; Proctor banner shows integrity signals in coaching tone
7. Back on `/dashboard`, the Mentor panel streams a career narrative tied to the learner's progress
8. Learner kills ai-service (or it crashes) → next message shows an inline error state with **Retry**; restarting the service and retrying resumes streaming
1. Learner opens `/build/comp-01` → a per-learner **variant statement** and starter files load; a namespace sandbox is created for the session
2. Learner edits files in the tree and types real shell commands in the xterm.js terminal → output renders live; the telemetry sidebar pulses as events stream to ai-service and persist in SQLite
3. **Run/Test** executes in-sandbox; results stream into the terminal; the Lab panel streams feedback derived from the **live trace digest** (real commands, real failures)
4. Learner opens `/defend/comp-01`**Start Defense**: the Examiner streams an opening question ("Walk me through your build — why did you structure it this way?")
5. Learner answers by voice (mic consent → MediaRecorder → STT) or typed fallback → examiner follow-ups probe the trace ("You hit three test failures before passing — what changed?"); TTS plays examiner audio (or browser speech in fallback)
6. Learner finishes the defense → transcript + integrity signals appear; verdict renders in a GradeBadge
7. Learner clicks **Grade My Work** → the grading engine computes the digest from the real trace, rubric-scores it, and the panel renders per-criterion bars + strengths/gaps/verdict
8. Proctor banner shows integrity signals from the live trace + defense in coaching tone; Mentor panel on `/dashboard` can narrate the real outcome
9. Navigating away destroys the sandbox (pool slot freed); killing ai-service shows inline error + retry states on every panel, with no crashes
## UX Acceptance Criteria
1. Streaming is visibly incremental — tokens appear as they arrive, not as one blob
2. Agent switcher shows the active agent (Coach/Tutor) and the response persona visibly changes
3. Loading state during connection (streaming cursor / skeleton) before first token
4. When ai-service is unreachable: inline error state + retry action on every chat/panel — no crashes, no console errors, no blank UI
5. `[DONE]` reliably ends the stream (input re-enables, no stuck "typing" state)
6. Navigating away mid-stream aborts cleanly — no leaked requests or post-unmount updates
7. All new UI uses design tokens, supports dark mode, meets WCAG AA contrast
8. Responsive at 375px, 768px, 1280px
9. No hardcoded model names or URLs in UI code — all via `NEXT_PUBLIC_AI_SERVICE_URL` and server meta events
10. `pnpm build` and `pnpm typecheck` pass with zero errors
1. Terminal output is visibly live (per keystroke/command round-trip), not a replay animation
2. The learner path contains **no mock engine data** — scenarios, canned artifacts, and scripted defense transcripts from v0.2 are gone from runtime
3. Variant statements visibly differ between two learner sessions on the same competency
4. Mic permission flow is graceful: consent prompt, recording indicator, no-mic/typed fallback, and browser-native speech path when no server voice key is configured
5. Defense transcript renders turn-by-turn with latency shown; integrity signals render in coaching (supportive) tone
6. Grade results render as structured per-criterion bars with verdict, from the real trace — not from final-output-only heuristics
7. Pool-full (503) shows an honest "environment busy — retry" state; navigation/unmount destroys sandboxes with no leaks
8. When ai-service is unreachable: inline error + retry on every panel — no crashes, no console errors, no blank UI
9. All new UI uses design tokens, supports dark mode, meets WCAG AA contrast, responsive at 375px / 768px / 1280px
10. `pnpm build` and `pnpm typecheck` pass with zero errors; `pnpm ai:test` green cloud-free and voice-key-free