verify(P04): gaps_found — P0 docstring fixes; anchor->grader shipment is a P1 gap
Trivial P0 fixes (this commit): two docstrings claimed things the wiring
does not do. variants/__init__.py said the package 'never imports agents/'
— false since Wave 2: generator.py holds the module-direct
agents.structured import (the sanctioned D-020 shared defense, same
exception as grading/engine.py); docstring now states the real boundary.
templates.py (header + RubricAnchors) claimed rubric anchors are 'used by
grading context' / 'shipped to the grader' — false in the current wiring:
GradingEngine is variant-blind (variant_seed=None; no variant lookup; no
anchor consumption; render_trace_digest takes only the digest). Docstrings
now tell the truth and name the follow-up.
Verification (four layers, evidence in the phase report):
- Structural: AST boundary audit clean — variants/ has zero api/fastapi
imports; only sanctioned agents.structured + llm/prompts/config/store.
- Behavioral: variants slice 41/41 green; full suite 324/324 green;
pnpm typecheck 7/7 green (forced, no cache). Live app probes confirmed
distinct learners -> distinct statements/seeds/task_ids at the API
level, cache hit = zero LLM calls, deterministic fallback (calls==2),
a-5 fairness envelope test green, sha256(template|learner|milestone)
seed derivation verified byte-exact against the spec formula.
- Security: secrets scan over the P04 diff (b52bef9..4acffac, 17 files
+2100/-1) clean — no key/token/password assignments, no URLs, no key
shapes. Prompt-injection surface bounded: the variant prompt carries
only template skeleton/title/id + seeded slot values — no learner id
or user-controlled content reaches the LLM. Empty learner_id -> 422.
- Quality: ruff clean; AI_MODEL env override verified live; tests are
mock-only (MockProvider family, zero network imports).
Must-Haves: 5 of 6 SATISFIED. NOT satisfied: MH#4 half — anchors are
present per template and a-5-testable, but NOT shipped to the grader
prompt context (non-trivial cross-module wiring: engine + prompt
signature + lifespan ordering — grading engine is built before the
variant store exists; reported as P1, not fixed here).
---ci---
phase: 4
milestone: v0.3
status: verify
requirements:
covered: [REQ-3-005]
partial: [REQ-3-005]
lessons:
- Docstrings that describe a must-have's target state ('shipped to the
grader prompt') read as done in review — verify wiring, not words:
grep the consumer side (grading/) before believing the producer side.
- main.py builds GradingEngine before variant_store exists; any P4/P6
anchor-shipment fix must reorder lifespan construction or inject the
variant store into the engine after the fact.
- Params distinctness is parametric (280-64908 combos per template) —
two learners CAN draw identical params (~15% at 10 learners on the
tightest template); distinctness is proven via seeds/task_ids/statement
embeddings, which is what the must-have actually requires.
---/ci---
This commit is contained in:
@@ -1,8 +1,10 @@
|
||||
"""Per-learner variant task generation — templates, generator, VariantStore (REQ-3-005).
|
||||
|
||||
Boundary rule (D-027): variants/ is an engine module — it never imports
|
||||
api/ or agents/ (api/ composes the generator and store via DI; store.py
|
||||
imports config only).
|
||||
api/; its ONLY agents/ dependency is the module-direct
|
||||
agents.structured import in generator.py (the sanctioned shared D-020
|
||||
structured defense, same exception as grading/engine.py). api/ composes
|
||||
the generator and store via DI; store.py imports config only.
|
||||
|
||||
CO-ORDINATION NOTE (ADD, don't REMOVE — same convention as grading/):
|
||||
This __init__.py is a minimal placeholder created by the VariantStore
|
||||
|
||||
@@ -2,9 +2,10 @@
|
||||
|
||||
A `TaskTemplate` binds a competency (D-021-aligned corpus ID), a statement
|
||||
skeleton with `{slot}` placeholders, typed `ParameterSlot`s, difficulty-
|
||||
normalization rubric anchors (the expected feature envelope the grader
|
||||
prompt receives as context — a-5 makes "same bar" testable), and starter-
|
||||
file scaffolds served into the sandbox workdir (wired in P6).
|
||||
normalization rubric anchors (the expected feature envelope that bounds
|
||||
variant fairness in the a-5 envelope test — grader-prompt shipment is the
|
||||
tracked P4 follow-up; grading is variant-blind today), and starter-file
|
||||
scaffolds served into the sandbox workdir (wired in P6).
|
||||
|
||||
Slot sampling is PURE CODE: `random.Random(seed)` over typed slots — fully
|
||||
reproducible for a given seed, independent of the LLM. The LLM only renders
|
||||
@@ -64,12 +65,17 @@ class ParameterSlot(BaseModel):
|
||||
|
||||
|
||||
class RubricAnchors(BaseModel):
|
||||
"""Difficulty-normalization anchors shipped to the grader as context.
|
||||
"""Difficulty-normalization anchors for the grader (a-5).
|
||||
|
||||
Expected FEATURE ENVELOPE (digest-space): the grading prompt receives
|
||||
these so two variants of one template are held to the same bar — the
|
||||
anchors bound what "comparable effort" looks like for this template
|
||||
regardless of which slot values a learner drew (a-5).
|
||||
Expected FEATURE ENVELOPE (digest-space): the expected effort band
|
||||
for this template, so two variants of one template are held to the
|
||||
same bar regardless of which slot values a learner drew. The a-5
|
||||
envelope test (tests/variants/test_generator.py) binds variants to
|
||||
these bands in code. Shipping them into the grader prompt context is
|
||||
the P4 must-have follow-up tracked for final review: grading is
|
||||
variant-blind in the current wiring (engine.py stamps
|
||||
variant_seed=None), so today the anchors gate variant fairness in
|
||||
tests only — not yet in the LLM prompt.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
Reference in New Issue
Block a user