T

CIAgent 0bde9cbf2e verify(P03): passed — process-trace grading engine verified end-to-end on this box

Four layers all PASS:
- Structural: grading/ AST-audited — zero fastapi/api imports; the ONLY
  agents/ dependency is the module-direct agents.structured import
  (sanctioned D-020 shared defense, grep-auditable); no D-020 duplication
  inside grading/ (no extract_json/parse_structured/retry logic — engine
  composes agents/structured.py). api/assessment.py owns all FastAPI
  wiring; engine knows nothing of HTTP (DI via deps.py + main lifespan).
- Behavioral: grading slice 65/65 green; full suite 283/283 green. Digest-
  leak test green (planted SECRET-COMMAND-MARKER-7f3a absent from every
  provider message; digest JSON present; learner/task ids absent too —
  grading is learner-anonymous). G-4 tests assert provider.calls==0 at
  BOTH engine and HTTP level (gapped, INCOMPLETE_FLOODED, empty, both-
  signals cases). D-020 retry recovers a fenced wrong-shape first reply
  (calls==2, validation error fed back); persistent failure raises
  StructuredOutputError -> 502 with NOTHING persisted. Calibration
  ordering contract green (strong>=lazy on process, strong>struggling on
  correctness) + deterministic digest feature separation over the three
  D-021-aligned archetypes. Two composed ad-hoc probes (WS ingest ->
  grade over the real app) additionally proved the cross-surface loop:
  (a) WS flood -> INCOMPLETE_FLOODED flag -> POST grade -> 200
  UNGRADABLE_TRACE_INCOMPLETE, LLM never called, durable gate record
  via GET; (b) WS ingest of a complete iterative trace -> POST grade ->
  200 GRADED with digest computed IN CODE from the ingested events
  (edit_count=2, error_fix_cycles=1, final pass), marker/file-path/ids
  absent from the provider prompt through the full stack.
- Security: secrets scan over the phase diff (2474678..HEAD, 16 files
  +2958/-5) clean — no key/token/password assignments, no env/secret
  files touched (initial sk-regex hits were false positives inside
  'task-calibration'/'task_id'). Prompt-injection surface bounded:
  digest-only prompts; TraceDigest carries fixed numeric fields + small
  histograms, no raw commands/contents/payloads; bounded at 361 bytes
  even at 49,999 events (just under the G-3 cap). No PII reaches the LLM
  at all — stronger than the required learner/task-id-only bound.
- Quality: ruff check . clean; config env-overridable (AI_DB_PATH/
  AI_PROVIDER/A_MODEL override verified live); grading tests are
  mock-only (no network imports; providers are MockProvider family);
  GradeStore shares the D-027 store contract (WAL, tz-normalization,
  detached rows, protocol-wrapped, upsert-latest-wins documented).

Must-Haves 1-6 (PLAN.md Phase 3): all SATISFIED. REQ-3-004 covered.

P1/P2 for final review (no P0s found, no code changes needed):
- P1 (documented PLAN deviation, ratify): PLAN 3-3-01 said 'unknown trace
  -> 404' on POST; implementation returns 200 UNGRADABLE_EMPTY_TRACE for
  a POST of an unknown/empty pair (persisted first-class gate record —
  the engine cannot distinguish absent from empty), reserving 404 for GET
  of a never-graded pair. Reasoned in api/assessment.py + test docstrings.
- P2: RubricScore requires 1-2 strengths/gaps (min_length=1) — a model
  legitimately returning zero gaps burns a D-020 retry; deliberate
  strictness, low impact given the prompt demands both lists.
- P2: calibration is mock-scripted ORDERING through the real pipeline,
  not a live-LLM quality benchmark (honest scope documented in
  test_calibration.py); live-model calibration remains future work.
- P2: compute_digest is not idempotent over duplicate events, but
  unreachable — TraceStore dedups on (learner,task,seq) and the engine
  reads via get_trace.

---ci---
phase: 3
milestone: v0.3
status: verify
requirements:
  covered: [REQ-3-004]
  partial: []
lessons:
  - G-4 must stay gate-FIRST ordering in _grade(): integrity flag checked
    before gaps() because gaps() returns [] for an empty trace — swapping
    the order lets an INCOMPLETE_FLOODED-but-complete-row trace fall
    through to the LLM path.
  - Starlette TestClient WS session: the telemetry ingest protocol has NO
    per-event ack (only gap_warning/event_rejected/close frames) — probing
    the composed loop means send-then-verify-via-store, not read-ack.
  - pnpm is absent from PATH on this box (P01 lesson persists); the
    underlying venv pytest/ruff commands are what scripts/test.sh and
    lint.sh exec — running them directly is equivalent verification.
  - ingest and grading MUST share one TraceIntegrityMap instance via
    app.state; a second map in the engine would silently miss every
    INCOMPLETE_FLOODED mark (probe proved the shared-map loop works).
---/ci---

2026-09-12 02:48:10 +00:00