Four layers all PASS:
- Structural: grading/ AST-audited — zero fastapi/api imports; the ONLY
agents/ dependency is the module-direct agents.structured import
(sanctioned D-020 shared defense, grep-auditable); no D-020 duplication
inside grading/ (no extract_json/parse_structured/retry logic — engine
composes agents/structured.py). api/assessment.py owns all FastAPI
wiring; engine knows nothing of HTTP (DI via deps.py + main lifespan).
- Behavioral: grading slice 65/65 green; full suite 283/283 green. Digest-
leak test green (planted SECRET-COMMAND-MARKER-7f3a absent from every
provider message; digest JSON present; learner/task ids absent too —
grading is learner-anonymous). G-4 tests assert provider.calls==0 at
BOTH engine and HTTP level (gapped, INCOMPLETE_FLOODED, empty, both-
signals cases). D-020 retry recovers a fenced wrong-shape first reply
(calls==2, validation error fed back); persistent failure raises
StructuredOutputError -> 502 with NOTHING persisted. Calibration
ordering contract green (strong>=lazy on process, strong>struggling on
correctness) + deterministic digest feature separation over the three
D-021-aligned archetypes. Two composed ad-hoc probes (WS ingest ->
grade over the real app) additionally proved the cross-surface loop:
(a) WS flood -> INCOMPLETE_FLOODED flag -> POST grade -> 200
UNGRADABLE_TRACE_INCOMPLETE, LLM never called, durable gate record
via GET; (b) WS ingest of a complete iterative trace -> POST grade ->
200 GRADED with digest computed IN CODE from the ingested events
(edit_count=2, error_fix_cycles=1, final pass), marker/file-path/ids
absent from the provider prompt through the full stack.
- Security: secrets scan over the phase diff (2474678..HEAD, 16 files
+2958/-5) clean — no key/token/password assignments, no env/secret
files touched (initial sk-regex hits were false positives inside
'task-calibration'/'task_id'). Prompt-injection surface bounded:
digest-only prompts; TraceDigest carries fixed numeric fields + small
histograms, no raw commands/contents/payloads; bounded at 361 bytes
even at 49,999 events (just under the G-3 cap). No PII reaches the LLM
at all — stronger than the required learner/task-id-only bound.
- Quality: ruff check . clean; config env-overridable (AI_DB_PATH/
AI_PROVIDER/A_MODEL override verified live); grading tests are
mock-only (no network imports; providers are MockProvider family);
GradeStore shares the D-027 store contract (WAL, tz-normalization,
detached rows, protocol-wrapped, upsert-latest-wins documented).
Must-Haves 1-6 (PLAN.md Phase 3): all SATISFIED. REQ-3-004 covered.
P1/P2 for final review (no P0s found, no code changes needed):
- P1 (documented PLAN deviation, ratify): PLAN 3-3-01 said 'unknown trace
-> 404' on POST; implementation returns 200 UNGRADABLE_EMPTY_TRACE for
a POST of an unknown/empty pair (persisted first-class gate record —
the engine cannot distinguish absent from empty), reserving 404 for GET
of a never-graded pair. Reasoned in api/assessment.py + test docstrings.
- P2: RubricScore requires 1-2 strengths/gaps (min_length=1) — a model
legitimately returning zero gaps burns a D-020 retry; deliberate
strictness, low impact given the prompt demands both lists.
- P2: calibration is mock-scripted ORDERING through the real pipeline,
not a live-LLM quality benchmark (honest scope documented in
test_calibration.py); live-model calibration remains future work.
- P2: compute_digest is not idempotent over duplicate events, but
unreachable — TraceStore dedups on (learner,task,seq) and the engine
reads via get_trace.
---ci---
phase: 3
milestone: v0.3
status: verify
requirements:
covered: [REQ-3-004]
partial: []
lessons:
- G-4 must stay gate-FIRST ordering in _grade(): integrity flag checked
before gaps() because gaps() returns [] for an empty trace — swapping
the order lets an INCOMPLETE_FLOODED-but-complete-row trace fall
through to the LLM path.
- Starlette TestClient WS session: the telemetry ingest protocol has NO
per-event ack (only gap_warning/event_rejected/close frames) — probing
the composed loop means send-then-verify-via-store, not read-ack.
- pnpm is absent from PATH on this box (P01 lesson persists); the
underlying venv pytest/ruff commands are what scripts/test.sh and
lint.sh exec — running them directly is equivalent verification.
- ingest and grading MUST share one TraceIntegrityMap instance via
app.state; a second map in the engine would silently miss every
INCOMPLETE_FLOODED mark (probe proved the shared-map loop works).
---/ci---
Verifier-found P0: UnshareBackend.destroy reaped agent → inner → helper as
asyncio subprocesses, but the inner entry is the `unshare --fork` PARENT —
its forked child (the `sleep` that is PID 1 of the sandbox pid/mnt/net ns)
reparents to host init and survives, holding the tmpfs + workspace bind for
the sleep duration (3600s). Every tracked-sandbox destroy leaked one
namespace process: ~30 orphaned `sleep 3600` observed after one suite run.
`--kill-child` does not reach the child under this flag combo (verified
empirically: the child still survives parent SIGTERM).
Fix: SIGKILL the ns-init's host pid (already tracked as `inner_pid` for
nsenter) in destroy(), after reaping the agent so it cannot flush into a
dead sandbox. Regression test creates a REAL task sandbox, asserts agent +
ns-init alive, destroys, and asserts both host pids are gone — fails on the
old code, passes with the fix. Full suite 218 green; ruff clean.
---ci---
phase: 2
milestone: v0.3
status: verify
requirements: {covered: [REQ-3-003], partial: []}
---/ci---
Task 2-3-01: create(learner_id, task_id) — telemetry sandboxes run a persistent
helper/inner namespace topology (offline inner ns; agent joins mount ns only and stays
online to reach the loopback ingest). Capture agent copied into the workspace (visible
in-ns at the bind), launched via sh -c with in-ns absolute paths (host cwd invalid after
the nsenter mount swap), stdin=DEVNULL daemonizes the agent (lifecycle tied to sandbox:
destroy reaps agent -> inner -> helper). Wire contract: frames strip URL-owned identity
(ingest extra=forbid anti-spoofing); spool keeps full events.
E2E test (real uvicorn on ephemeral port): exec in a live namespace sandbox -> events
arrive at WS ingest -> SQLite, ordered, sandbox-scoped. Pure-shell path (task_id=None)
asserts no capture agent. Full suite 216 green; ruff clean.
---ci---
phase: 2
milestone: v0.3
status: execute
requirements: {covered: [REQ-3-003], partial: []}
---/ci---
Four layers all PASS:
- Structural: sandbox/ AST-audited — zero imports of api/, agents/, llm/;
manager composed only via api/deps.py DI + main.py lifespan; no docker/
podman/sudo/containerd in the spawner path (D-024 holds).
- Behavioral: pnpm ai:test = 174/174 green; sandbox+api slice 40/40 green
with ALL probe-gated tests RAN (zero skips) — uid=0 in-ns, 0 network
interfaces, workspace-write lands on host, no escape into snapshots,
/proc-remount blocked, memory/CPU/FSIZE rlimits kill violators,
wall-clock reaper proven, G-2 size sweep destroys + preserves snapshot,
orphan startup reaper proven, 503/403/429 paths all live-verified via
a real uvicorn probe (curl roundtrip: 201→list→snapshot→403→429→204→404).
- Security: no secrets in sandbox/api code or logs (sk-test key in tests is
an anti-leak assertion fixture); CORS localhost-only, no credentials;
documented gaps (userns != DAC write-barrier; in-ns children reparent to
host PID 1 when supervisor is killed; per-sandbox NPROC shared at host
uid) are present in unshare_backend.py, test_resource_limits.py, README,
and are NOT relied on — RLIMIT_NPROC is never set, disk cap is the G-2
sweep not a kernel quota.
- Quality: pnpm ai:lint = ruff clean; pnpm ai:bootstrap idempotent (exit 0);
all config keys env-overridable (AI_SANDBOX_*); zero cloud calls in
sandbox/api tests (mock-first verified by grep).
Must-haves 1-8 (PLAN.md Phase 1): all SATISFIED.
---ci---
phase: 1
milestone: v0.3
status: verify
requirements:
covered: [REQ-3-001, REQ-3-002]
partial: []
lessons:
- util-linux 2.38 lacks unshare --bind; the in-namespace mount shim is the
correct swap — keep the rlimit-on-payload-only ordering (bind, then
ulimit, then exec) so unshare/mount stay unconstrained.
- tmp_path under /tmp breaks the backend (in-ns tmpfs shadows host /tmp);
repo-anchored tests/sandboxes/ is the load-bearing fixture choice.
- pnpm on this box needs 'corepack enable' before turbo can resolve the
package-manager binary; bootstrap env setup, not a repo defect.
---/ci---
CIAgent