From 9ff86f9cd0735285023bf508086bc85c7173da4c Mon Sep 17 00:00:00 2001 From: CIAgent Date: Sat, 12 Sep 2026 03:46:30 +0000 Subject: [PATCH] =?UTF-8?q?verify(P04):=20gaps=5Ffound=20=E2=80=94=20P0=20?= =?UTF-8?q?docstring=20fixes;=20anchor->grader=20shipment=20is=20a=20P1=20?= =?UTF-8?q?gap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Trivial P0 fixes (this commit): two docstrings claimed things the wiring does not do. variants/__init__.py said the package 'never imports agents/' — false since Wave 2: generator.py holds the module-direct agents.structured import (the sanctioned D-020 shared defense, same exception as grading/engine.py); docstring now states the real boundary. templates.py (header + RubricAnchors) claimed rubric anchors are 'used by grading context' / 'shipped to the grader' — false in the current wiring: GradingEngine is variant-blind (variant_seed=None; no variant lookup; no anchor consumption; render_trace_digest takes only the digest). Docstrings now tell the truth and name the follow-up. Verification (four layers, evidence in the phase report): - Structural: AST boundary audit clean — variants/ has zero api/fastapi imports; only sanctioned agents.structured + llm/prompts/config/store. - Behavioral: variants slice 41/41 green; full suite 324/324 green; pnpm typecheck 7/7 green (forced, no cache). Live app probes confirmed distinct learners -> distinct statements/seeds/task_ids at the API level, cache hit = zero LLM calls, deterministic fallback (calls==2), a-5 fairness envelope test green, sha256(template|learner|milestone) seed derivation verified byte-exact against the spec formula. - Security: secrets scan over the P04 diff (b52bef9..4acffac, 17 files +2100/-1) clean — no key/token/password assignments, no URLs, no key shapes. Prompt-injection surface bounded: the variant prompt carries only template skeleton/title/id + seeded slot values — no learner id or user-controlled content reaches the LLM. Empty learner_id -> 422. - Quality: ruff clean; AI_MODEL env override verified live; tests are mock-only (MockProvider family, zero network imports). Must-Haves: 5 of 6 SATISFIED. NOT satisfied: MH#4 half — anchors are present per template and a-5-testable, but NOT shipped to the grader prompt context (non-trivial cross-module wiring: engine + prompt signature + lifespan ordering — grading engine is built before the variant store exists; reported as P1, not fixed here). ---ci--- phase: 4 milestone: v0.3 status: verify requirements: covered: [REQ-3-005] partial: [REQ-3-005] lessons: - Docstrings that describe a must-have's target state ('shipped to the grader prompt') read as done in review — verify wiring, not words: grep the consumer side (grading/) before believing the producer side. - main.py builds GradingEngine before variant_store exists; any P4/P6 anchor-shipment fix must reorder lifespan construction or inject the variant store into the engine after the fact. - Params distinctness is parametric (280-64908 combos per template) — two learners CAN draw identical params (~15% at 10 learners on the tightest template); distinctness is proven via seeds/task_ids/statement embeddings, which is what the must-have actually requires. ---/ci--- --- .../ai_service/variants/__init__.py | 6 +++-- .../ai_service/variants/templates.py | 22 ++++++++++++------- 2 files changed, 18 insertions(+), 10 deletions(-) diff --git a/apps/ai-service/ai_service/variants/__init__.py b/apps/ai-service/ai_service/variants/__init__.py index 1cad88b..f32809c 100644 --- a/apps/ai-service/ai_service/variants/__init__.py +++ b/apps/ai-service/ai_service/variants/__init__.py @@ -1,8 +1,10 @@ """Per-learner variant task generation — templates, generator, VariantStore (REQ-3-005). Boundary rule (D-027): variants/ is an engine module — it never imports -api/ or agents/ (api/ composes the generator and store via DI; store.py -imports config only). +api/; its ONLY agents/ dependency is the module-direct +agents.structured import in generator.py (the sanctioned shared D-020 +structured defense, same exception as grading/engine.py). api/ composes +the generator and store via DI; store.py imports config only. CO-ORDINATION NOTE (ADD, don't REMOVE — same convention as grading/): This __init__.py is a minimal placeholder created by the VariantStore diff --git a/apps/ai-service/ai_service/variants/templates.py b/apps/ai-service/ai_service/variants/templates.py index 5706ef8..6280dec 100644 --- a/apps/ai-service/ai_service/variants/templates.py +++ b/apps/ai-service/ai_service/variants/templates.py @@ -2,9 +2,10 @@ A `TaskTemplate` binds a competency (D-021-aligned corpus ID), a statement skeleton with `{slot}` placeholders, typed `ParameterSlot`s, difficulty- -normalization rubric anchors (the expected feature envelope the grader -prompt receives as context — a-5 makes "same bar" testable), and starter- -file scaffolds served into the sandbox workdir (wired in P6). +normalization rubric anchors (the expected feature envelope that bounds +variant fairness in the a-5 envelope test — grader-prompt shipment is the +tracked P4 follow-up; grading is variant-blind today), and starter-file +scaffolds served into the sandbox workdir (wired in P6). Slot sampling is PURE CODE: `random.Random(seed)` over typed slots — fully reproducible for a given seed, independent of the LLM. The LLM only renders @@ -64,12 +65,17 @@ class ParameterSlot(BaseModel): class RubricAnchors(BaseModel): - """Difficulty-normalization anchors shipped to the grader as context. + """Difficulty-normalization anchors for the grader (a-5). - Expected FEATURE ENVELOPE (digest-space): the grading prompt receives - these so two variants of one template are held to the same bar — the - anchors bound what "comparable effort" looks like for this template - regardless of which slot values a learner drew (a-5). + Expected FEATURE ENVELOPE (digest-space): the expected effort band + for this template, so two variants of one template are held to the + same bar regardless of which slot values a learner drew. The a-5 + envelope test (tests/variants/test_generator.py) binds variants to + these bands in code. Shipping them into the grader prompt context is + the P4 must-have follow-up tracked for final review: grading is + variant-blind in the current wiring (engine.py stamps + variant_seed=None), so today the anchors gate variant fairness in + tests only — not yet in the LLM prompt. """ model_config = ConfigDict(frozen=True)