Four gaps found by independent verifier probing of the defense endpoints
(all Must-Have-relevant, all trivially fixed):
1. Browser-mode descriptor was dead code: BROWSER_FALLBACK_DESCRIPTOR
existed but start always returned mode='mock' even with
AI_VOICE_PROVIDER=browser (Must-Have #6 violated). start now derives
the descriptor from settings.voice_provider (D-030).
2. answer after finish returned 200 and appended turns to a sealed
transcript — the store explicitly assigns sequencing to the endpoints
(defense_store.py: 'turns after finalize are a sequencing bug for the
endpoints to prevent, task 5-3-01'); the endpoints didn't. Now 409.
3. Zero-byte audio upload crashed the mock provider (MockVoiceFailure ->
500); a real provider would 500 the same way. Empty upload is a client
error: 422 before any provider call (provider contract unchanged).
4. Verdict was NOT persisted (Must-Have #1 'verdict + transcript
persisted'): finish persisted only signals; GET after finish could not
re-serve the verdict. The verdict now nests in integrity_signals
(JSON-object dict per the DefenseStore.finalize contract).
3 regression tests added (empty-audio 422, post-finish 409, verdict
retrievable from GET; browser-descriptor test). Suite 386 green; ruff clean.
---ci---
phase: 5
milestone: v0.3
status: verify
requirements:
covered: [REQ-3-006]
partial: []
lessons:
- A descriptor that exists but is never served is indistinguishable from
dead code until you probe the configured mode end-to-end (factory tests
proved selection, not service).
- Store contracts that 'assign' sequencing to callers need an endpoint
test for the forbidden transition, or the assignment is decorative.
---/ci---
Task 5-2-01: prompts/examiner.py (Socratic oral-defense examiner; one question per
turn; grounded in TraceDigest + variant statement — never raw trace, never learner id,
D-028 mirror; rubric internals never revealed) + agents/examiner.py — ExaminerAgent
(next_question for the SSE pipeline; final_verdict -> DefenseVerdict via the D-020
defense). BOUNDARY: the examiner is a text agent and imports NO voice/ (STT/TTS belong
to the endpoints; integrity signals computed from turn metadata — A-109). Registry
registers all seven agents centrally (G-4); registry test updated six -> seven.
6 examiner tests (digest-grounded prompt w/o learner id; D-020 retry; 7-agent roster;
boundary import scan). Suite 367 green; ruff clean.
---ci---
phase: 5
milestone: v0.3
status: execute
requirements: {covered: [REQ-3-006], partial: []}
---/ci---
GradingEngine takes an optional VariantStore (constructor DI); when the graded
task_id joins to a stored variant: the template's difficulty anchors render into
the grader user turn ("Expected effort envelope" — same bar for every variant of
the template, a-5) and the variant seed is stamped on the GradeRecord (D-029).
Lifespan reordered: VariantStore builds before the engine and is passed in.
Anchors context carries only template id + anchor numbers — D-028 learner-anonymity
preserved (leak tests keep holding). Plain engine (no store) stays variant-blind;
non-variant tasks grade without the envelope.
3 new tests: variant task -> anchors + seed present in prompt/record;
non-variant task -> no envelope; plain engine -> variant_seed None.
Suite 327 green; ruff clean.
---ci---
phase: 4
milestone: v0.3
status: verify
requirements: {covered: [REQ-3-005], partial: []}
---/ci---
Trivial P0 fixes (this commit): two docstrings claimed things the wiring
does not do. variants/__init__.py said the package 'never imports agents/'
— false since Wave 2: generator.py holds the module-direct
agents.structured import (the sanctioned D-020 shared defense, same
exception as grading/engine.py); docstring now states the real boundary.
templates.py (header + RubricAnchors) claimed rubric anchors are 'used by
grading context' / 'shipped to the grader' — false in the current wiring:
GradingEngine is variant-blind (variant_seed=None; no variant lookup; no
anchor consumption; render_trace_digest takes only the digest). Docstrings
now tell the truth and name the follow-up.
Verification (four layers, evidence in the phase report):
- Structural: AST boundary audit clean — variants/ has zero api/fastapi
imports; only sanctioned agents.structured + llm/prompts/config/store.
- Behavioral: variants slice 41/41 green; full suite 324/324 green;
pnpm typecheck 7/7 green (forced, no cache). Live app probes confirmed
distinct learners -> distinct statements/seeds/task_ids at the API
level, cache hit = zero LLM calls, deterministic fallback (calls==2),
a-5 fairness envelope test green, sha256(template|learner|milestone)
seed derivation verified byte-exact against the spec formula.
- Security: secrets scan over the P04 diff (b52bef9..4acffac, 17 files
+2100/-1) clean — no key/token/password assignments, no URLs, no key
shapes. Prompt-injection surface bounded: the variant prompt carries
only template skeleton/title/id + seeded slot values — no learner id
or user-controlled content reaches the LLM. Empty learner_id -> 422.
- Quality: ruff clean; AI_MODEL env override verified live; tests are
mock-only (MockProvider family, zero network imports).
Must-Haves: 5 of 6 SATISFIED. NOT satisfied: MH#4 half — anchors are
present per template and a-5-testable, but NOT shipped to the grader
prompt context (non-trivial cross-module wiring: engine + prompt
signature + lifespan ordering — grading engine is built before the
variant store exists; reported as P1, not fixed here).
---ci---
phase: 4
milestone: v0.3
status: verify
requirements:
covered: [REQ-3-005]
partial: [REQ-3-005]
lessons:
- Docstrings that describe a must-have's target state ('shipped to the
grader prompt') read as done in review — verify wiring, not words:
grep the consumer side (grading/) before believing the producer side.
- main.py builds GradingEngine before variant_store exists; any P4/P6
anchor-shipment fix must reorder lifespan construction or inject the
variant store into the engine after the fact.
- Params distinctness is parametric (280-64908 combos per template) —
two learners CAN draw identical params (~15% at 10 learners on the
tightest template); distinctness is proven via seeds/task_ids/statement
embeddings, which is what the must-have actually requires.
---/ci---
Four layers all PASS:
- Structural: grading/ AST-audited — zero fastapi/api imports; the ONLY
agents/ dependency is the module-direct agents.structured import
(sanctioned D-020 shared defense, grep-auditable); no D-020 duplication
inside grading/ (no extract_json/parse_structured/retry logic — engine
composes agents/structured.py). api/assessment.py owns all FastAPI
wiring; engine knows nothing of HTTP (DI via deps.py + main lifespan).
- Behavioral: grading slice 65/65 green; full suite 283/283 green. Digest-
leak test green (planted SECRET-COMMAND-MARKER-7f3a absent from every
provider message; digest JSON present; learner/task ids absent too —
grading is learner-anonymous). G-4 tests assert provider.calls==0 at
BOTH engine and HTTP level (gapped, INCOMPLETE_FLOODED, empty, both-
signals cases). D-020 retry recovers a fenced wrong-shape first reply
(calls==2, validation error fed back); persistent failure raises
StructuredOutputError -> 502 with NOTHING persisted. Calibration
ordering contract green (strong>=lazy on process, strong>struggling on
correctness) + deterministic digest feature separation over the three
D-021-aligned archetypes. Two composed ad-hoc probes (WS ingest ->
grade over the real app) additionally proved the cross-surface loop:
(a) WS flood -> INCOMPLETE_FLOODED flag -> POST grade -> 200
UNGRADABLE_TRACE_INCOMPLETE, LLM never called, durable gate record
via GET; (b) WS ingest of a complete iterative trace -> POST grade ->
200 GRADED with digest computed IN CODE from the ingested events
(edit_count=2, error_fix_cycles=1, final pass), marker/file-path/ids
absent from the provider prompt through the full stack.
- Security: secrets scan over the phase diff (2474678..HEAD, 16 files
+2958/-5) clean — no key/token/password assignments, no env/secret
files touched (initial sk-regex hits were false positives inside
'task-calibration'/'task_id'). Prompt-injection surface bounded:
digest-only prompts; TraceDigest carries fixed numeric fields + small
histograms, no raw commands/contents/payloads; bounded at 361 bytes
even at 49,999 events (just under the G-3 cap). No PII reaches the LLM
at all — stronger than the required learner/task-id-only bound.
- Quality: ruff check . clean; config env-overridable (AI_DB_PATH/
AI_PROVIDER/A_MODEL override verified live); grading tests are
mock-only (no network imports; providers are MockProvider family);
GradeStore shares the D-027 store contract (WAL, tz-normalization,
detached rows, protocol-wrapped, upsert-latest-wins documented).
Must-Haves 1-6 (PLAN.md Phase 3): all SATISFIED. REQ-3-004 covered.
P1/P2 for final review (no P0s found, no code changes needed):
- P1 (documented PLAN deviation, ratify): PLAN 3-3-01 said 'unknown trace
-> 404' on POST; implementation returns 200 UNGRADABLE_EMPTY_TRACE for
a POST of an unknown/empty pair (persisted first-class gate record —
the engine cannot distinguish absent from empty), reserving 404 for GET
of a never-graded pair. Reasoned in api/assessment.py + test docstrings.
- P2: RubricScore requires 1-2 strengths/gaps (min_length=1) — a model
legitimately returning zero gaps burns a D-020 retry; deliberate
strictness, low impact given the prompt demands both lists.
- P2: calibration is mock-scripted ORDERING through the real pipeline,
not a live-LLM quality benchmark (honest scope documented in
test_calibration.py); live-model calibration remains future work.
- P2: compute_digest is not idempotent over duplicate events, but
unreachable — TraceStore dedups on (learner,task,seq) and the engine
reads via get_trace.
---ci---
phase: 3
milestone: v0.3
status: verify
requirements:
covered: [REQ-3-004]
partial: []
lessons:
- G-4 must stay gate-FIRST ordering in _grade(): integrity flag checked
before gaps() because gaps() returns [] for an empty trace — swapping
the order lets an INCOMPLETE_FLOODED-but-complete-row trace fall
through to the LLM path.
- Starlette TestClient WS session: the telemetry ingest protocol has NO
per-event ack (only gap_warning/event_rejected/close frames) — probing
the composed loop means send-then-verify-via-store, not read-ack.
- pnpm is absent from PATH on this box (P01 lesson persists); the
underlying venv pytest/ruff commands are what scripts/test.sh and
lint.sh exec — running them directly is equivalent verification.
- ingest and grading MUST share one TraceIntegrityMap instance via
app.state; a second map in the engine would silently miss every
INCOMPLETE_FLOODED mark (probe proved the shared-map loop works).
---/ci---
Verifier-found P0: UnshareBackend.destroy reaped agent → inner → helper as
asyncio subprocesses, but the inner entry is the `unshare --fork` PARENT —
its forked child (the `sleep` that is PID 1 of the sandbox pid/mnt/net ns)
reparents to host init and survives, holding the tmpfs + workspace bind for
the sleep duration (3600s). Every tracked-sandbox destroy leaked one
namespace process: ~30 orphaned `sleep 3600` observed after one suite run.
`--kill-child` does not reach the child under this flag combo (verified
empirically: the child still survives parent SIGTERM).
Fix: SIGKILL the ns-init's host pid (already tracked as `inner_pid` for
nsenter) in destroy(), after reaping the agent so it cannot flush into a
dead sandbox. Regression test creates a REAL task sandbox, asserts agent +
ns-init alive, destroys, and asserts both host pids are gone — fails on the
old code, passes with the fix. Full suite 218 green; ruff clean.
---ci---
phase: 2
milestone: v0.3
status: verify
requirements: {covered: [REQ-3-003], partial: []}
---/ci---
Task 2-3-01: create(learner_id, task_id) — telemetry sandboxes run a persistent
helper/inner namespace topology (offline inner ns; agent joins mount ns only and stays
online to reach the loopback ingest). Capture agent copied into the workspace (visible
in-ns at the bind), launched via sh -c with in-ns absolute paths (host cwd invalid after
the nsenter mount swap), stdin=DEVNULL daemonizes the agent (lifecycle tied to sandbox:
destroy reaps agent -> inner -> helper). Wire contract: frames strip URL-owned identity
(ingest extra=forbid anti-spoofing); spool keeps full events.
E2E test (real uvicorn on ephemeral port): exec in a live namespace sandbox -> events
arrive at WS ingest -> SQLite, ordered, sandbox-scoped. Pure-shell path (task_id=None)
asserts no capture agent. Full suite 216 green; ruff clean.
---ci---
phase: 2
milestone: v0.3
status: execute
requirements: {covered: [REQ-3-003], partial: []}
---/ci---
Four layers all PASS:
- Structural: sandbox/ AST-audited — zero imports of api/, agents/, llm/;
manager composed only via api/deps.py DI + main.py lifespan; no docker/
podman/sudo/containerd in the spawner path (D-024 holds).
- Behavioral: pnpm ai:test = 174/174 green; sandbox+api slice 40/40 green
with ALL probe-gated tests RAN (zero skips) — uid=0 in-ns, 0 network
interfaces, workspace-write lands on host, no escape into snapshots,
/proc-remount blocked, memory/CPU/FSIZE rlimits kill violators,
wall-clock reaper proven, G-2 size sweep destroys + preserves snapshot,
orphan startup reaper proven, 503/403/429 paths all live-verified via
a real uvicorn probe (curl roundtrip: 201→list→snapshot→403→429→204→404).
- Security: no secrets in sandbox/api code or logs (sk-test key in tests is
an anti-leak assertion fixture); CORS localhost-only, no credentials;
documented gaps (userns != DAC write-barrier; in-ns children reparent to
host PID 1 when supervisor is killed; per-sandbox NPROC shared at host
uid) are present in unshare_backend.py, test_resource_limits.py, README,
and are NOT relied on — RLIMIT_NPROC is never set, disk cap is the G-2
sweep not a kernel quota.
- Quality: pnpm ai:lint = ruff clean; pnpm ai:bootstrap idempotent (exit 0);
all config keys env-overridable (AI_SANDBOX_*); zero cloud calls in
sandbox/api tests (mock-first verified by grep).
Must-haves 1-8 (PLAN.md Phase 1): all SATISFIED.
---ci---
phase: 1
milestone: v0.3
status: verify
requirements:
covered: [REQ-3-001, REQ-3-002]
partial: []
lessons:
- util-linux 2.38 lacks unshare --bind; the in-namespace mount shim is the
correct swap — keep the rlimit-on-payload-only ordering (bind, then
ulimit, then exec) so unshare/mount stay unconstrained.
- tmp_path under /tmp breaks the backend (in-ns tmpfs shadows host /tmp);
repo-anchored tests/sandboxes/ is the load-bearing fixture choice.
- pnpm on this box needs 'corepack enable' before turbo can resolve the
package-manager binary; bootstrap env setup, not a repo defect.
---/ci---
CIAgent