merge(P03): phase/03 trace grading → milestone/v0.3-credential-engines
---ci--- phase: 3 milestone: v0.3 status: ship ---/ci---
This commit is contained in:
CIAgent
@@ -1,8 +1,8 @@
|
||||
{
|
||||
"phase": 2,
|
||||
"stage": "complete",
|
||||
"phase": 3,
|
||||
"stage": "verify",
|
||||
"milestone": "v0.3",
|
||||
"phase_role": "execution",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-09-12T01:41:31Z"
|
||||
"updated_at": "2026-09-12T02:48:49Z"
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-3-004 | Process-trace grading engine: grade artifacts from their full process traces; rubric-aligned structured scores; feeds Assessor real inputs | critical | 3 | pending |
|
||||
| REQ-3-004 | Process-trace grading engine: grade artifacts from their full process traces; rubric-aligned structured scores; feeds Assessor real inputs | critical | 3 | complete |
|
||||
| REQ-3-005 | Variant task generation: per-learner task variants (no two learners get identical prompts); variant seed registry; difficulty normalization | high | 4 | pending |
|
||||
| REQ-3-006 | Oral/voice defense: AI examiner conducts spoken defense (STT → dialogue → TTS); transcript + integrity signals captured; feeds Proctor/Mentor | high | 5 | pending |
|
||||
|
||||
@@ -177,7 +177,7 @@
|
||||
| REQ-3-001 | 1 | complete |
|
||||
| REQ-3-002 | 1 | complete |
|
||||
| REQ-3-003 | 2 | complete |
|
||||
| REQ-3-004 | 3 | pending |
|
||||
| REQ-3-004 | 3 | complete |
|
||||
| REQ-3-005 | 4 | pending |
|
||||
| REQ-3-006 | 5 | pending |
|
||||
| REQ-3-007 | 6 | pending |
|
||||
|
||||
+1
-1
@@ -21,7 +21,7 @@
|
||||
| 0 | Pre-execution | in-progress | — | — | Specification, clarify, research, plan complete; .ciagent/ files updated for v0.3 |
|
||||
| 1 | Sandbox fabric | complete | 0 | REQ-3-001, REQ-3-002 | Isolated per-learner sandbox environments provisioned (IDE / design / simulation); lifecycle API (create/destroy/snapshot); resource limits enforced; no cross-tenant access |
|
||||
| 2 | Live build telemetry | complete | 1 | REQ-3-003 | In-environment capture of process events (commands, file diffs, run/test results, keystroke-level activity) streamed to ai-service; reliable transport; per-learner trace persistence |
|
||||
| 3 | Process-trace grading engine | pending | 2 | REQ-3-004 | Grades artifacts from their full process traces (not just final output); emits structured rubric-aligned scores; feeds Assessor real inputs |
|
||||
| 3 | Process-trace grading engine | complete | 2 | REQ-3-004 | Grades artifacts from their full process traces (not just final output); emits structured rubric-aligned scores; feeds Assessor real inputs |
|
||||
| 4 | Variant task generation | pending | 1 | REQ-3-005 | Per-learner task variants generated so no two learners receive identical prompts; variant seed recorded for grading fairness |
|
||||
| 5 | Oral / voice defense | pending | 3 | REQ-3-006 | AI examiner conducts spoken defense of submitted work; STT → dialogue → TTS; transcript + integrity signals captured; feeds Proctor/Mentor |
|
||||
| 6 | Agent re-grounding + learner surface integration | pending | 2,3,4,5 | REQ-3-007, REQ-3-008 | Lab/Assessor/Proctor consume real engine inputs; v0.1 sandbox + assessment mockups wired to real engines (in-browser build/run, live telemetry, live defense) |
|
||||
|
||||
@@ -1,22 +1,86 @@
|
||||
"""POST /v1/assessment/evaluate — structured rubric scores (REQ-2-008).
|
||||
"""/v1/assessment — rubric evaluation + trace grading endpoints (REQ-2-008, REQ-3-004).
|
||||
|
||||
JSON response (not SSE): a pydantic-validated RubricScore. Unknown
|
||||
artifact → 404. The Assessor's structured output IS the payload.
|
||||
Two endpoint families share this router:
|
||||
|
||||
POST /v1/assessment/evaluate (v0.2, REQ-2-008) — corpus
|
||||
artifact evaluation through
|
||||
the Assessor agent.
|
||||
POST /v1/assessment/grade (v0.3, REQ-3-004) — grade a
|
||||
REAL process trace through
|
||||
the GradingEngine.
|
||||
GET /v1/assessment/grade/{learner_id}/{task_id} — stored latest grade.
|
||||
|
||||
Grading status-code mapping (the engine's outcomes are CONTRACT, not errors):
|
||||
|
||||
GradeRecord(verdict=GRADED) → 200 — rubric scores +
|
||||
verdict (in scores.verdict)
|
||||
+ digest summary.
|
||||
GradeRecord(UNGRADABLE_TRACE_INCOMPLETE) → 200 — the ungradable
|
||||
record IS a valid result:
|
||||
the trace cannot be graded,
|
||||
and the gate surfaces WHY
|
||||
(scores.missing_seqs +
|
||||
scores.integrity_flag).
|
||||
Persisted like any grade.
|
||||
GradeRecord(UNGRADABLE_EMPTY_TRACE) → 200 — no events stored for
|
||||
the pair. This covers BOTH
|
||||
a known pair whose trace
|
||||
ended up empty AND a task
|
||||
that never had a trace at
|
||||
all: the engine cannot
|
||||
distinguish them (zero
|
||||
stored events is zero
|
||||
events), and grading an
|
||||
absent trace genuinely has
|
||||
the empty-trace outcome —
|
||||
a 404 here would erase the
|
||||
durable gate record the
|
||||
engine persists for the
|
||||
pair. PLAN's 404 applies to
|
||||
GET of a never-graded pair.
|
||||
StructuredOutputError → 502 — the trace was
|
||||
gradable but the provider
|
||||
failed the D-020 budget;
|
||||
provider failure (bad
|
||||
gateway to the model), same
|
||||
mapping as evaluate.
|
||||
GET of an unknown (never-graded) pair → 404.
|
||||
|
||||
DI (D-027/D-032 house pattern): engine + store arrive via deps.get_grading_engine
|
||||
/ get_grade_store from app.state; this module owns all FastAPI wiring — the
|
||||
engine knows nothing of HTTP. UNGRADABLE_* bodies are rendered by the same
|
||||
GradeResponse model as GRADED ones (a gate record's `scores` holds the gate
|
||||
detail instead of rubric scores), so consumers read ONE shape.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..agents.assessor import RubricScore
|
||||
from ..agents.registry import AgentRegistry
|
||||
from ..agents.structured import StructuredOutputError
|
||||
from ..config import Settings
|
||||
from ..corpus.artifacts import get_artifact_bundle, get_transcript_for_artifact
|
||||
from ..corpus.learner_context import get_learner_context
|
||||
from .deps import get_agent_registry, get_provider, get_settings
|
||||
from ..grading.engine import GradingEngine
|
||||
from ..grading.store import GradeRecord, GradeStore
|
||||
from .deps import (
|
||||
get_agent_registry,
|
||||
get_grade_store,
|
||||
get_grading_engine,
|
||||
get_provider,
|
||||
get_settings,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/v1")
|
||||
|
||||
|
||||
# --- v0.2 artifact evaluation (REQ-2-008) --------------------------------------
|
||||
|
||||
|
||||
class AssessmentRequest(BaseModel):
|
||||
artifact_id: str = Field(min_length=1)
|
||||
learner_id: str | None = None
|
||||
@@ -45,3 +109,72 @@ async def assessment_evaluate(
|
||||
status_code=502,
|
||||
detail=f"assessment evaluation failed: {exc}",
|
||||
) from exc
|
||||
|
||||
|
||||
# --- v0.3 trace grading (REQ-3-004) ---------------------------------------------
|
||||
|
||||
|
||||
class GradeRequest(BaseModel):
|
||||
learner_id: str = Field(min_length=1)
|
||||
task_id: str = Field(min_length=1)
|
||||
|
||||
|
||||
class GradeResponse(BaseModel):
|
||||
"""GradeRecord over HTTP — one shape for GRADED and UNGRADABLE_* alike.
|
||||
|
||||
`scores` holds the validated rubric (criteria 0-4, strengths, gaps,
|
||||
rubric verdict) for a GRADED record, or the gate detail
|
||||
({integrity_flag, missing_seqs}) for an UNGRADABLE_* record — never both.
|
||||
`digest` is the compact trace summary that fed the rubric prompt (empty
|
||||
for gate records: nothing was graded).
|
||||
"""
|
||||
|
||||
learner_id: str
|
||||
task_id: str
|
||||
variant_seed: str | None
|
||||
digest: dict[str, Any]
|
||||
scores: dict[str, Any]
|
||||
verdict: str
|
||||
model: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
def _grade_response(record: GradeRecord) -> GradeResponse:
|
||||
return GradeResponse.model_validate(record, from_attributes=True)
|
||||
|
||||
|
||||
@router.post("/assessment/grade", response_model=GradeResponse)
|
||||
async def assessment_grade(
|
||||
body: GradeRequest,
|
||||
engine: GradingEngine = Depends(get_grading_engine),
|
||||
) -> GradeResponse:
|
||||
"""Run the grading engine for one (learner_id, task_id) trace.
|
||||
|
||||
Gate outcomes (UNGRADABLE_*) are 200s — they are first-class results the
|
||||
engine persists, not failures. Only a provider that exhausts the D-020
|
||||
budget turns into a 502; nothing is persisted on that path.
|
||||
"""
|
||||
try:
|
||||
record = await engine.grade(body.learner_id, body.task_id)
|
||||
except StructuredOutputError as exc:
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"grading failed: {exc}",
|
||||
) from exc
|
||||
return _grade_response(record)
|
||||
|
||||
|
||||
@router.get("/assessment/grade/{learner_id}/{task_id}", response_model=GradeResponse)
|
||||
async def assessment_get_grade(
|
||||
learner_id: str,
|
||||
task_id: str,
|
||||
store: GradeStore = Depends(get_grade_store),
|
||||
) -> GradeResponse:
|
||||
"""Latest stored grade for the pair; 404 when none was ever stored."""
|
||||
record = store.get(learner_id, task_id)
|
||||
if record is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"no stored grade for {learner_id!r}/{task_id!r}",
|
||||
)
|
||||
return _grade_response(record)
|
||||
|
||||
@@ -5,6 +5,8 @@ from fastapi import Request
|
||||
from ..agents.registry import AgentRegistry
|
||||
from ..agents.session import SessionStore
|
||||
from ..config import Settings
|
||||
from ..grading.engine import GradingEngine
|
||||
from ..grading.store import GradeStore
|
||||
from ..llm.base import LLMProvider
|
||||
from ..sandbox.manager import SandboxManager
|
||||
from ..sandbox.workdir import SandboxDir
|
||||
@@ -43,3 +45,11 @@ def get_trace_store(request: Request) -> TraceStore:
|
||||
|
||||
def get_trace_integrity(request: Request) -> TraceIntegrityMap:
|
||||
return request.app.state.trace_integrity
|
||||
|
||||
|
||||
def get_grade_store(request: Request) -> GradeStore:
|
||||
return request.app.state.grade_store
|
||||
|
||||
|
||||
def get_grading_engine(request: Request) -> GradingEngine:
|
||||
return request.app.state.grading_engine
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
"""Synthetic trace fixtures for grading calibration (Task 3-2-02, REQ-3-004).
|
||||
|
||||
Three builder archetypes as REAL `TelemetryEvent` traces (the grading
|
||||
engine's native input — these are NOT the v0.2 corpus's simplified
|
||||
`{timestamp, kind, detail}` event shapes):
|
||||
|
||||
strong builder — iterative debugging: small edits, tests early,
|
||||
failed runs closed by targeted fixes, eventual pass.
|
||||
lazy builder — one large paste, a single late test run, pass.
|
||||
(v0.2 alignment: the `lab-scenario-flagged` paste-
|
||||
and-run archetype; D-021.)
|
||||
struggling builder — many edit/test cycles, failures never close,
|
||||
never reaches a pass.
|
||||
|
||||
ID convention (D-021 alignment, documented in each fixture):
|
||||
v0.2 corpus scenario IDs are `<domain>-scenario-<slug>` (`lab-scenario-strong`,
|
||||
`lab-scenario-struggling`, `lab-scenario-flagged`, `proctor-scenario-*` — see
|
||||
corpus/telemetry.py). Grading fixtures carry ids string-aligned to that
|
||||
convention:
|
||||
|
||||
fixture id = "<scenario id>::<archetype>-trace"
|
||||
learner id = "learner-003" (a member of the corpus learner-00N id space;
|
||||
learner-001/002 exist in learner_context.py)
|
||||
|
||||
so a fixture is greppable against its v0.2 scenario counterpart while staying
|
||||
a distinct id space (a grading trace is a real event stream, not the v0.2
|
||||
mock scenario timeline — same convention, richer event kind set).
|
||||
|
||||
Instance hygiene: fixtures store event SPECS (plain tuples) and materialize
|
||||
FRESH `TelemetryEvent` instances on every `fixture.events` access. SQLModel
|
||||
rows carry SQLAlchemy instance state once a session has flushed them —
|
||||
re-adding the SAME instance to another store is a silent no-op, which would
|
||||
poison sequential test runs (a fixture ingested by test N would vanish for
|
||||
test N+1). Materializing per access keeps every consumer independent.
|
||||
|
||||
These fixtures exist for the CALIBRATION CONTRACT (tests/grading/
|
||||
test_calibration.py): the mock provider scripts archetype-mapped scores and
|
||||
the test asserts the ORDERING the rubric must eventually enforce. They are
|
||||
NOT an LLM quality benchmark — see the test module docstring for the honest
|
||||
scope statement.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Final
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..telemetry.models import TelemetryEvent
|
||||
|
||||
_T0: Final = datetime(2026, 9, 12, 0, 0, 0, tzinfo=UTC)
|
||||
|
||||
#: Event spec: (seq, kind, payload, seconds-since-session-start).
|
||||
EventSpec = tuple[int, str, dict, float]
|
||||
|
||||
|
||||
class TraceFixture(BaseModel):
|
||||
"""One named synthetic trace + its v0.2 scenario alignment (D-021).
|
||||
|
||||
`event_specs` is the durable, session-state-free description; `events`
|
||||
materializes fresh TelemetryEvent rows from it on every access.
|
||||
"""
|
||||
|
||||
model_config = {"frozen": True}
|
||||
|
||||
fixture_id: str # "<v0.2 scenario id>::<archetype>-trace"
|
||||
archetype: str # "strong" | "lazy" | "struggling"
|
||||
aligned_scenario_id: str # the v0.2 corpus scenario this fixture mirrors
|
||||
competency_id: str # corpus competency id space (stack-orchestration-c00N)
|
||||
task_id: str # trace task id (grading operates on (learner, task))
|
||||
learner_id: str
|
||||
event_specs: tuple[EventSpec, ...] = Field(default=())
|
||||
|
||||
@property
|
||||
def events(self) -> list[TelemetryEvent]:
|
||||
"""FRESH TelemetryEvent instances — safe to ingest into any store.
|
||||
|
||||
Never cache these: an instance flushed by one SQLite session
|
||||
carries persistent identity, and re-appending it elsewhere no-ops.
|
||||
"""
|
||||
return [
|
||||
TelemetryEvent(
|
||||
learner_id=self.learner_id,
|
||||
task_id=self.task_id,
|
||||
seq=seq,
|
||||
kind=kind,
|
||||
payload=payload,
|
||||
ts=_T0 + timedelta(seconds=offset_s),
|
||||
sandbox_id="sbx-calibration",
|
||||
)
|
||||
for seq, kind, payload, offset_s in self.event_specs
|
||||
]
|
||||
|
||||
def description(self) -> str:
|
||||
return (
|
||||
f"{self.fixture_id} (archetype={self.archetype}, aligned="
|
||||
f"{self.aligned_scenario_id}, competency={self.competency_id})"
|
||||
)
|
||||
|
||||
|
||||
class _Builder:
|
||||
"""Seq-accurate event-spec builder for one (learner, task) pair."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.specs: list[EventSpec] = []
|
||||
self._seq = 0
|
||||
|
||||
def add(self, kind: str, payload: dict | None, at: float) -> None:
|
||||
self.specs.append((self._seq, kind, payload or {}, at))
|
||||
self._seq += 1
|
||||
|
||||
|
||||
def _strong_builder_specs() -> list[EventSpec]:
|
||||
"""Iterative debugging: tests early, tight edit→test loops, eventual pass.
|
||||
|
||||
Mirrors `lab-scenario-strong` (v0.2: keystrokes → file_save → run_tests →
|
||||
test_pass, c002) at full telemetry fidelity — every failed cycle is
|
||||
closed by a targeted edit followed by a re-run that passes.
|
||||
"""
|
||||
b = _Builder()
|
||||
b.add("activity", {"state": "starting"}, 0)
|
||||
b.add("file_diff", {"path": "planner.py", "added": 14}, 95) # small edit
|
||||
b.add("command", {"cmd": "pytest -q tests/test_planner.py"}, 120) # tests EARLY
|
||||
b.add("test_result", {"passed": True, "exit_code": 0}, 126)
|
||||
b.add("file_diff", {"path": "tool_node.py", "added": 22}, 180)
|
||||
b.add("command", {"cmd": "pytest -q"}, 275)
|
||||
b.add("test_result", {"passed": False, "exit_code": 1}, 281) # honest failure
|
||||
b.add("file_diff", {"path": "tool_node.py", "added": 6, "removed": 2}, 340) # targeted fix
|
||||
b.add("command", {"cmd": "pytest -q"}, 430)
|
||||
b.add("test_result", {"passed": True, "exit_code": 0}, 436) # cycle CLOSED
|
||||
b.add("command", {"cmd": "git commit -m 'tool node with retries'"}, 500)
|
||||
return b.specs
|
||||
|
||||
|
||||
def _lazy_builder_specs() -> list[EventSpec]:
|
||||
"""Paste-and-run: one large paste, a single LATE test run, instant pass.
|
||||
|
||||
Mirrors `lab-scenario-flagged` (v0.2: paste of 2,400 chars → run_tests →
|
||||
instant 6/6 pass, c003) at full telemetry fidelity — zero iteration, zero
|
||||
verification during construction, one terminal test run only.
|
||||
"""
|
||||
b = _Builder()
|
||||
b.add("activity", {"state": "starting"}, 0)
|
||||
b.add("file_diff", {"path": "eval.py", "added": 240, "removed": 0}, 30) # one bulk paste
|
||||
b.add("file_diff", {"path": "README.md", "added": 12}, 40)
|
||||
b.add("command", {"cmd": "npm run build"}, 45)
|
||||
b.add("run_result", {"exit_code": 0, "ok": True}, 60)
|
||||
b.add("command", {"cmd": "pytest -q"}, 520) # single LATE test run
|
||||
b.add("test_result", {"passed": True, "exit_code": 0}, 540) # instant pass
|
||||
return b.specs
|
||||
|
||||
|
||||
def _struggling_builder_specs() -> list[EventSpec]:
|
||||
"""Many cycles, none close: repeated failures, no eventual pass.
|
||||
|
||||
Mirrors `lab-scenario-struggling` (v0.2: repeated identical ImportErrors,
|
||||
idle gaps, no checkpoint, c002) at full telemetry fidelity — edits happen
|
||||
between failures, but the same failure recurs; no pass is ever reached.
|
||||
"""
|
||||
b = _Builder()
|
||||
b.add("activity", {"state": "starting"}, 0)
|
||||
b.add("file_diff", {"path": "main.py", "added": 40}, 20)
|
||||
b.add("command", {"cmd": "pytest -q"}, 210)
|
||||
b.add("test_result", {"passed": False, "exit_code": 1}, 215) # ImportError
|
||||
b.add("file_diff", {"path": "main.py", "added": 8, "removed": 3}, 300)
|
||||
b.add("command", {"cmd": "pytest -q"}, 520)
|
||||
b.add("test_result", {"passed": False, "exit_code": 1}, 525) # SAME error
|
||||
b.add("file_diff", {"path": "main.py", "added": 5}, 610)
|
||||
b.add("command", {"cmd": "pytest -q"}, 960)
|
||||
b.add("test_result", {"passed": False, "exit_code": 1}, 965) # STILL failing
|
||||
b.add("activity", {"state": "idle"}, 1500) # long idle
|
||||
b.add("activity", {"state": "idle"}, 2200)
|
||||
return b.specs
|
||||
|
||||
|
||||
#: Calibration learner — a member of the corpus learner-00N id space (D-021;
|
||||
#: learner-001/002 live in corpus/learner_context.py; grading fixtures use
|
||||
#: a third id so calibration traces never collide with mock-context reads).
|
||||
CALIBRATION_LEARNER_ID: Final = "learner-003"
|
||||
|
||||
|
||||
STRONG_BUILDER: Final = TraceFixture(
|
||||
fixture_id="lab-scenario-strong::strong-trace",
|
||||
archetype="strong",
|
||||
aligned_scenario_id="lab-scenario-strong",
|
||||
competency_id="stack-orchestration-c002",
|
||||
task_id="task-calibration-strong",
|
||||
learner_id=CALIBRATION_LEARNER_ID,
|
||||
event_specs=tuple(_strong_builder_specs()),
|
||||
)
|
||||
|
||||
LAZY_BUILDER: Final = TraceFixture(
|
||||
# v0.2's paste-and-run archetype is the "flagged" lab scenario (D-021):
|
||||
# large paste → instant test pass. "lazy builder" is that behavior
|
||||
# without the proctor flag; the alignment is behavioral, documented here.
|
||||
fixture_id="lab-scenario-flagged::lazy-trace",
|
||||
archetype="lazy",
|
||||
aligned_scenario_id="lab-scenario-flagged",
|
||||
competency_id="stack-orchestration-c003",
|
||||
task_id="task-calibration-lazy",
|
||||
learner_id=CALIBRATION_LEARNER_ID,
|
||||
event_specs=tuple(_lazy_builder_specs()),
|
||||
)
|
||||
|
||||
STRUGGLING_BUILDER: Final = TraceFixture(
|
||||
fixture_id="lab-scenario-struggling::struggling-trace",
|
||||
archetype="struggling",
|
||||
aligned_scenario_id="lab-scenario-struggling",
|
||||
competency_id="stack-orchestration-c002",
|
||||
task_id="task-calibration-struggling",
|
||||
learner_id=CALIBRATION_LEARNER_ID,
|
||||
event_specs=tuple(_struggling_builder_specs()),
|
||||
)
|
||||
|
||||
TRACE_FIXTURES: Final[dict[str, TraceFixture]] = {
|
||||
f.fixture_id: f
|
||||
for f in (STRONG_BUILDER, LAZY_BUILDER, STRUGGLING_BUILDER)
|
||||
}
|
||||
|
||||
|
||||
def get_trace_fixture(fixture_id: str) -> TraceFixture | None:
|
||||
return TRACE_FIXTURES.get(fixture_id)
|
||||
|
||||
|
||||
def digest_of(fixture: TraceFixture):
|
||||
"""Compute the digest for a fixture (pure compute; test-side helper)."""
|
||||
from ..grading.features import compute_digest
|
||||
|
||||
return compute_digest(fixture.events)
|
||||
@@ -0,0 +1,27 @@
|
||||
"""Process-trace grading — trace digest, rubric engine, GradeStore (REQ-3-004).
|
||||
|
||||
Boundary rule (D-027): grading/ is an engine module — it never imports
|
||||
api/; the single sanctioned agents/ dependency is the shared D-020
|
||||
structured defense (agents/structured.py), imported module-direct in
|
||||
engine.py (see its docstring for why). features.py imports telemetry/;
|
||||
store.py imports config only; engine.py composes llm/ + telemetry/ +
|
||||
prompts/ + agents.structured.
|
||||
|
||||
Wave status: features.py (TraceDigest, compute_digest) + store.py
|
||||
(GradeRecord, GradeStore, SQLiteGradeStore) landed in Wave 1 (3-1-01 +
|
||||
3-1-02); engine.py (GradingEngine, RubricScore) is Wave 2 (3-2-01).
|
||||
"""
|
||||
|
||||
from .engine import GradingEngine, RubricScore
|
||||
from .features import TraceDigest, compute_digest
|
||||
from .store import GradeRecord, GradeStore, SQLiteGradeStore
|
||||
|
||||
__all__ = [
|
||||
"GradeRecord",
|
||||
"GradeStore",
|
||||
"GradingEngine",
|
||||
"RubricScore",
|
||||
"SQLiteGradeStore",
|
||||
"TraceDigest",
|
||||
"compute_digest",
|
||||
]
|
||||
@@ -0,0 +1,268 @@
|
||||
"""GradingEngine — rubric scoring over real process traces (REQ-3-004).
|
||||
|
||||
The Wave-2 composition of the grading stack:
|
||||
|
||||
trace completeness gate (G-4, FIRST — nothing is sent to any LLM
|
||||
when the gate trips) → TraceStore.get_trace → compute_digest (D-028)
|
||||
→ prompts.grading.render_trace_digest → D-020 4-layer structured
|
||||
defense (agents/structured.py — REUSED, composed, never duplicated)
|
||||
→ RubricScore validation → GradeStore persistence → GradeRecord.
|
||||
|
||||
Gate verdicts are FIRST-CLASS RESULTS, not exceptions (G-4 is binding):
|
||||
UNGRADABLE_TRACE_INCOMPLETE — seq gaps in the store OR the trace is
|
||||
flagged by TraceIntegrityMap (INCOMPLETE_FLOODED). The gap list /
|
||||
flag reason is surfaced in `scores` for API rendering, and the
|
||||
record is PERSISTED like any grade so a learner sees why no
|
||||
credential can be issued for this trace — the gate outcome is
|
||||
durable and auditable, not a transient error string.
|
||||
UNGRADABLE_EMPTY_TRACE — no events stored for the pair.
|
||||
On either verdict `scores.criteria` is empty and the LLM is never called.
|
||||
|
||||
DI (D-027/D-032 house pattern): the engine receives trace_store,
|
||||
grade_store, integrity and provider through the constructor and knows
|
||||
NOTHING of FastAPI — api/ composes it (Task 3-3-01). `model` is injected
|
||||
alongside the provider so tests script the mock against the production
|
||||
wiring without touching Settings.
|
||||
|
||||
Boundary (D-027): grading/ imports llm/ (provider protocol + Message
|
||||
type), telemetry/ (store + integrity map), prompts/ (rubric text) and
|
||||
ONLY agents.structured — the sanctioned shared D-020 defense. We import
|
||||
the MODULE directly (`from ..agents.structured import structured_completion`)
|
||||
rather than the `agents` package, mirroring how agents/base.py consumes
|
||||
it (same direct-module import): that keeps the dependency surface to
|
||||
exactly the two names the engine needs (structured_completion,
|
||||
StructuredOutputError) and avoids executing agents/__init__ re-exports
|
||||
(BaseAgent, registry, session store) that grading has no business
|
||||
loading — a side-effect-hygiene choice that keeps this import line
|
||||
grep-auditable as "the one agents dependency". grading/ never imports api/.
|
||||
|
||||
RubricScore placement (documented decision): the validated output model
|
||||
lives HERE, not in prompts/. The pydantic model is the engine's return
|
||||
CONTRACT (the shape GradeStore.scores must hold), while prompts/grading.py
|
||||
is pure prompt text + its mirror schema HINT string — the same split as
|
||||
agents/assessor.py (model + hint) but with the model owned by the engine
|
||||
module that validates it. Prompt files hold text, per the prompts/ house
|
||||
style; engine files hold typed contracts.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
from typing import Final
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from ..agents.structured import StructuredOutputError, structured_completion
|
||||
from ..llm.base import LLMProvider
|
||||
from ..llm.types import Message
|
||||
from ..prompts.grading import (
|
||||
RUBRIC_CRITERIA,
|
||||
RUBRIC_SCORE_SCHEMA_HINT,
|
||||
SYSTEM_PROMPT,
|
||||
render_trace_digest,
|
||||
)
|
||||
from ..telemetry.ingest import TraceIntegrityMap
|
||||
from ..telemetry.store import TraceStore
|
||||
from .features import compute_digest
|
||||
from .store import GradeRecord, GradeStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: First-class gate verdicts (G-4). GradeRecord.verdict values; non-empty
|
||||
#: by store contract. Rubric verdicts (mastered/developing/not_yet) ride in
|
||||
#: `scores.verdict` — `record.verdict` stays the machine-readable outcome.
|
||||
VERDICT_UNGRADABLE_INCOMPLETE: Final = "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
VERDICT_UNGRADABLE_EMPTY: Final = "UNGRADABLE_EMPTY_TRACE"
|
||||
#: record.verdict for a successfully LLM-graded trace (the rubric verdict
|
||||
#: travels inside scores); keeps verdict non-empty for every persisted row.
|
||||
VERDICT_GRADED: Final = "GRADED"
|
||||
|
||||
#: Gate-detail keys surfaced in GradeRecord.scores (tests assert on these).
|
||||
_INTEGRITY_FLAG_KEY: Final = "integrity_flag"
|
||||
_MISSING_SEQS_KEY: Final = "missing_seqs"
|
||||
|
||||
|
||||
class RubricScore(BaseModel):
|
||||
"""Validated LLM output: per-criterion 0-4 scores + strengths + gaps + verdict.
|
||||
|
||||
The D-020 schema for the grader: `structured_completion` parses the
|
||||
model reply into THIS shape (layer 3), retrying once with the
|
||||
validation error fed back (layer 4). Exact criteria set + 0-4 ranges
|
||||
are enforced here, so the scores dict persisted to GradeStore is
|
||||
always rubric-shaped no matter what the model produced.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
criteria: dict[str, int]
|
||||
strengths: list[str] = Field(min_length=1, max_length=2)
|
||||
gaps: list[str] = Field(min_length=1, max_length=2)
|
||||
verdict: str # "mastered" | "developing" | "not_yet"
|
||||
|
||||
@field_validator("criteria")
|
||||
@classmethod
|
||||
def _criteria_rubric_shaped(cls, value: dict[str, int]) -> dict[str, int]:
|
||||
"""Exact criteria keys (no extras, no omissions) and 0-4 scores."""
|
||||
expected = set(RUBRIC_CRITERIA)
|
||||
got = set(value)
|
||||
if got != expected:
|
||||
raise ValueError(
|
||||
f"criteria keys must be exactly {sorted(expected)}, got {sorted(got)}"
|
||||
)
|
||||
for key, score in value.items():
|
||||
if not 0 <= score <= 4:
|
||||
raise ValueError(f"criterion {key!r} must be within 0-4, got {score}")
|
||||
return value
|
||||
|
||||
@field_validator("verdict")
|
||||
@classmethod
|
||||
def _verdict_known(cls, value: str) -> str:
|
||||
allowed = {"mastered", "developing", "not_yet"}
|
||||
if value not in allowed:
|
||||
raise ValueError(f"verdict must be one of {sorted(allowed)}, got {value!r}")
|
||||
return value
|
||||
|
||||
|
||||
class GradingEngine:
|
||||
"""Scores a (learner_id, task_id) trace into a persisted GradeRecord."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
trace_store: TraceStore,
|
||||
grade_store: GradeStore,
|
||||
integrity: TraceIntegrityMap,
|
||||
provider: LLMProvider,
|
||||
*,
|
||||
model: str = "gemma4:31b",
|
||||
) -> None:
|
||||
self._trace_store = trace_store
|
||||
self._grade_store = grade_store
|
||||
self._integrity = integrity
|
||||
self._provider = provider
|
||||
self._model = model
|
||||
|
||||
async def grade(self, learner_id: str, task_id: str) -> GradeRecord:
|
||||
"""Grade one trace; persist latest-state (GradeStore upserts); return it.
|
||||
|
||||
Gate FIRST (G-4): the LLM is only ever reached from the fully
|
||||
guarded path — no gate state can be masked by an LLM error.
|
||||
"""
|
||||
record = await self._grade(learner_id, task_id)
|
||||
self._grade_store.save(record)
|
||||
return record
|
||||
|
||||
# ------------------------------------------------------------------ core
|
||||
|
||||
async def _grade(self, learner_id: str, task_id: str) -> GradeRecord:
|
||||
# -- G-4 gate FIRST: integrity flag OR seq gaps. Ordering matters:
|
||||
# gaps() returns [] for an EMPTY trace, so the empty check below is
|
||||
# reachable only when no rows exist at all; a gapped or flooded
|
||||
# trace can never fall through to the LLM path.
|
||||
if self._integrity.is_incomplete(learner_id, task_id):
|
||||
reason = self._integrity.reason(learner_id, task_id) or "unknown"
|
||||
gaps = self._trace_store.gaps(learner_id, task_id)
|
||||
logger.info(
|
||||
"grade gate (G-4): %s/%s integrity-flagged (%s) — ungradable",
|
||||
learner_id,
|
||||
task_id,
|
||||
reason,
|
||||
)
|
||||
return self._ungradable(
|
||||
learner_id,
|
||||
task_id,
|
||||
detail={_INTEGRITY_FLAG_KEY: reason, _MISSING_SEQS_KEY: gaps},
|
||||
verdict=VERDICT_UNGRADABLE_INCOMPLETE,
|
||||
)
|
||||
gaps = self._trace_store.gaps(learner_id, task_id)
|
||||
if gaps:
|
||||
logger.info(
|
||||
"grade gate (G-4): %s/%s seq gaps %s — ungradable",
|
||||
learner_id,
|
||||
task_id,
|
||||
gaps,
|
||||
)
|
||||
return self._ungradable(
|
||||
learner_id,
|
||||
task_id,
|
||||
detail={_INTEGRITY_FLAG_KEY: None, _MISSING_SEQS_KEY: gaps},
|
||||
verdict=VERDICT_UNGRADABLE_INCOMPLETE,
|
||||
)
|
||||
|
||||
trace = self._trace_store.get_trace(learner_id, task_id)
|
||||
if not trace:
|
||||
logger.info(
|
||||
"grade gate: %s/%s empty trace — ungradable", learner_id, task_id
|
||||
)
|
||||
return self._ungradable(
|
||||
learner_id,
|
||||
task_id,
|
||||
detail={_INTEGRITY_FLAG_KEY: None, _MISSING_SEQS_KEY: []},
|
||||
verdict=VERDICT_UNGRADABLE_EMPTY,
|
||||
)
|
||||
|
||||
# -- Guarded path: digest (D-028) → prompt → D-020 4-layer defense.
|
||||
digest = compute_digest(trace)
|
||||
messages = [
|
||||
Message(role="system", content=SYSTEM_PROMPT),
|
||||
Message(role="user", content=render_trace_digest(digest)),
|
||||
]
|
||||
try:
|
||||
rubric = await structured_completion(
|
||||
self._provider,
|
||||
messages,
|
||||
model=self._model,
|
||||
schema=RubricScore,
|
||||
schema_hint=RUBRIC_SCORE_SCHEMA_HINT,
|
||||
)
|
||||
except StructuredOutputError as exc:
|
||||
# The trace was gradable but the model failed to produce valid
|
||||
# JSON within the D-020 budget (two attempts). Raise — the API
|
||||
# layer maps this to a 502 (assessor precedent). Persisting a
|
||||
# fabricated or partial grade here would violate the no-silent-
|
||||
# fallback rule: no credential-worthy record without a validated
|
||||
# RubricScore.
|
||||
raise StructuredOutputError(f"grading LLM failed for {task_id}: {exc}") from exc
|
||||
|
||||
logger.debug(
|
||||
"graded %s/%s: %s (model=%s)",
|
||||
learner_id,
|
||||
task_id,
|
||||
rubric.verdict,
|
||||
self._model,
|
||||
)
|
||||
return GradeRecord(
|
||||
learner_id=learner_id,
|
||||
task_id=task_id,
|
||||
variant_seed=None, # null until P4 (D-029)
|
||||
digest=digest.model_dump(),
|
||||
scores=rubric.model_dump(),
|
||||
verdict=VERDICT_GRADED,
|
||||
model=self._model,
|
||||
created_at=datetime.now(tz=UTC),
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------ gate record
|
||||
|
||||
@staticmethod
|
||||
def _ungradable(
|
||||
learner_id: str,
|
||||
task_id: str,
|
||||
*,
|
||||
detail: dict,
|
||||
verdict: str,
|
||||
) -> GradeRecord:
|
||||
"""Build a gate record: no digest (nothing was graded), gate detail
|
||||
surfaced in `scores` (the store allows an empty scores dict, but
|
||||
G-4 requires the gap list / flag reason surfaced — the detail IS the
|
||||
verdict's payload), model="none" (no LLM was involved; provenance
|
||||
stays honest).
|
||||
"""
|
||||
return GradeRecord(
|
||||
learner_id=learner_id,
|
||||
task_id=task_id,
|
||||
variant_seed=None,
|
||||
digest={},
|
||||
scores=detail,
|
||||
verdict=verdict,
|
||||
model="none",
|
||||
created_at=datetime.now(tz=UTC),
|
||||
)
|
||||
@@ -0,0 +1,251 @@
|
||||
"""Deterministic process-trace digest (D-028, REQ-3-004).
|
||||
|
||||
Pure compute — no LLM, no I/O. `compute_digest` reduces an ordered
|
||||
TelemetryEvent trace to a compact, bounded `TraceDigest` that is safe to
|
||||
embed in a grading prompt:
|
||||
|
||||
- FIXED fields + small histograms only; NO raw commands, NO file contents,
|
||||
NO payloads — the raw trace NEVER reaches the LLM (D-028), which also
|
||||
bounds the prompt-injection surface.
|
||||
- Tolerant to both live trace mixes: daemon-topology traces carry
|
||||
`activity` + `file_diff` kinds (workspace watcher), while REPL-driven
|
||||
traces carry `command`/`stdin`/`stdout`/`run_result`/`test_result`
|
||||
(P2 verification P1). Features derive from whatever kinds are present and
|
||||
never crash on absent kinds.
|
||||
|
||||
Feature semantics (conservative, deterministic):
|
||||
- test pass/fail counts + final status derive from `test_result` payloads
|
||||
when present, falling back to `run_result` exit codes (0 = pass).
|
||||
- an error/fix CYCLE = a failing run/test followed by >= 1 edit and then a
|
||||
later run/test (pass or fail) — the next observed result closes the cycle.
|
||||
- idle gaps = wall-clock gaps between consecutive events exceeding
|
||||
`idle_threshold_s` (default 120s): count + total seconds.
|
||||
- command category histogram classifies `command`-kind payloads: build /
|
||||
test / file / nav / debug / other.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..telemetry.models import TelemetryEvent
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover - import cycle guard for type checkers
|
||||
pass
|
||||
|
||||
|
||||
_IDLE_DEFAULT_S: float = 120.0
|
||||
|
||||
_TEST_HINTS = ("test", "pytest", "vitest", "jest", "mocha", "unittest", "go test", "npm test")
|
||||
_BUILD_HINTS = ("make", "npm run build", "pip install", "pnpm", "cargo build", "gcc", "tsc")
|
||||
_DEBUG_HINTS = ("gdb", "pdb", "print(", "debug", "strace", "ltrace", "curl", "ping")
|
||||
_NAV_HINTS = ("ls", "cd", "pwd", "cat ", "grep ", "find", "rg ", "tree", "head", "tail", "less")
|
||||
_FILE_HINTS = ("mv ", "cp ", "rm ", "mkdir", "touch", "chmod", "nano", "vim", "sed -i", "tee ")
|
||||
|
||||
|
||||
class TraceDigest(BaseModel):
|
||||
"""Compact, bounded, LLM-safe summary of a process trace (D-028).
|
||||
|
||||
Fixed fields + small histograms. Serializes well under 4 KB; contains no
|
||||
raw commands, file contents, or event payloads.
|
||||
"""
|
||||
|
||||
model_config = {"frozen": True}
|
||||
|
||||
event_count: int = Field(ge=0)
|
||||
session_duration_s: float = Field(ge=0.0)
|
||||
edit_count: int = Field(ge=0)
|
||||
command_count: int = Field(ge=0)
|
||||
run_count: int = Field(ge=0)
|
||||
test_pass_count: int = Field(ge=0)
|
||||
test_fail_count: int = Field(ge=0)
|
||||
final_test_status: str = Field(pattern="^(pass|fail|none)$")
|
||||
first_test_pass_offset_s: float | None = None
|
||||
error_fix_cycles: int = Field(ge=0)
|
||||
mean_fix_latency_s: float | None = None
|
||||
idle_gap_count: int = Field(ge=0)
|
||||
idle_gap_total_s: float = Field(ge=0.0)
|
||||
command_categories: dict[str, int] = Field(default_factory=dict)
|
||||
kind_histogram: dict[str, int] = Field(default_factory=dict)
|
||||
|
||||
|
||||
def _event_pass_status(event: TelemetryEvent) -> bool | None:
|
||||
"""True (pass) / False (fail) / None (not a result event) for one event."""
|
||||
payload = event.payload or {}
|
||||
if event.kind == "test_result":
|
||||
if "passed" in payload:
|
||||
return bool(payload["passed"])
|
||||
if "exit_code" in payload:
|
||||
return int(payload["exit_code"]) == 0
|
||||
if "status" in payload:
|
||||
return str(payload["status"]).lower() in ("pass", "passed", "ok", "success")
|
||||
return None
|
||||
if event.kind == "run_result":
|
||||
if "exit_code" in payload:
|
||||
return int(payload["exit_code"]) == 0
|
||||
if "ok" in payload:
|
||||
return bool(payload["ok"])
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _classify_command(text: str) -> str:
|
||||
lowered = text.lower()
|
||||
if any(h in lowered for h in _TEST_HINTS):
|
||||
return "test"
|
||||
if any(h in lowered for h in _BUILD_HINTS):
|
||||
return "build"
|
||||
if any(h in lowered for h in _DEBUG_HINTS):
|
||||
return "debug"
|
||||
if any(h in lowered for h in _NAV_HINTS):
|
||||
return "nav"
|
||||
if any(h in lowered for h in _FILE_HINTS):
|
||||
return "file"
|
||||
return "other"
|
||||
|
||||
|
||||
def _command_text(event: TelemetryEvent) -> str:
|
||||
payload = event.payload or {}
|
||||
return str(payload.get("cmd") or payload.get("command") or payload.get("line") or "")
|
||||
|
||||
|
||||
def compute_digest(
|
||||
trace: list[TelemetryEvent], *, idle_threshold_s: float = _IDLE_DEFAULT_S
|
||||
) -> TraceDigest:
|
||||
"""Reduce an ordered trace to a bounded digest. Never raises on odd input."""
|
||||
events = sorted(trace, key=lambda e: (e.seq, e.ts))
|
||||
if not events:
|
||||
return TraceDigest(
|
||||
event_count=0,
|
||||
session_duration_s=0.0,
|
||||
edit_count=0,
|
||||
command_count=0,
|
||||
run_count=0,
|
||||
test_pass_count=0,
|
||||
test_fail_count=0,
|
||||
final_test_status="none",
|
||||
first_test_pass_offset_s=None,
|
||||
error_fix_cycles=0,
|
||||
mean_fix_latency_s=None,
|
||||
idle_gap_count=0,
|
||||
idle_gap_total_s=0.0,
|
||||
command_categories={},
|
||||
kind_histogram={},
|
||||
)
|
||||
|
||||
kind_histogram = Counter(e.kind for e in events)
|
||||
start_ts = events[0].ts
|
||||
end_ts = events[-1].ts
|
||||
duration = max(0.0, (end_ts - start_ts).total_seconds())
|
||||
|
||||
edit_count = kind_histogram.get("file_diff", 0)
|
||||
command_count = kind_histogram.get("command", 0)
|
||||
run_count = kind_histogram.get("run_result", 0)
|
||||
|
||||
# Tests: prefer test_result events; fall back to run_result exit codes.
|
||||
test_statuses: list[tuple[TelemetryEvent, bool]] = []
|
||||
for e in events:
|
||||
if e.kind == "test_result":
|
||||
ok = _event_pass_status(e)
|
||||
if ok is not None:
|
||||
test_statuses.append((e, ok))
|
||||
if not test_statuses:
|
||||
for e in events:
|
||||
if e.kind == "run_result":
|
||||
ok = _event_pass_status(e)
|
||||
if ok is not None:
|
||||
test_statuses.append((e, ok))
|
||||
|
||||
test_pass_count = sum(1 for _, ok in test_statuses if ok)
|
||||
test_fail_count = len(test_statuses) - test_pass_count
|
||||
if not test_statuses:
|
||||
final_test_status = "none"
|
||||
else:
|
||||
final_test_status = "pass" if test_statuses[-1][1] else "fail"
|
||||
first_pass = next((e for e, ok in test_statuses if ok), None)
|
||||
first_pass_offset = (
|
||||
max(0.0, (first_pass.ts - start_ts).total_seconds()) if first_pass is not None else None
|
||||
)
|
||||
|
||||
# Error/fix cycles: a failing result starts a pending cycle; the NEXT
|
||||
# observed result closes it (regardless of outcome) — a fix attempt that
|
||||
# fails again is itself another iteration of debugging, so it closes the
|
||||
# previous cycle and opens a new one. Edits since the fail mark the
|
||||
# close as a genuine fix attempt; latency = first edit -> closing result.
|
||||
cycles = 0
|
||||
fix_latencies: list[float] = []
|
||||
pending_fail_ts: float | None = None # seconds since start
|
||||
edits_since_fail = 0
|
||||
first_edit_ts: float | None = None
|
||||
for e in events:
|
||||
t = max(0.0, (e.ts - start_ts).total_seconds())
|
||||
if e.kind == "file_diff":
|
||||
if pending_fail_ts is not None:
|
||||
if edits_since_fail == 0:
|
||||
first_edit_ts = t
|
||||
edits_since_fail += 1
|
||||
continue
|
||||
ok = _event_pass_status(e)
|
||||
if ok is None:
|
||||
continue
|
||||
if ok is False:
|
||||
if pending_fail_ts is not None and edits_since_fail > 0 and first_edit_ts is not None:
|
||||
# failed fix attempt: closes the previous cycle, opens a new one
|
||||
cycles += 1
|
||||
fix_latencies.append(t - first_edit_ts)
|
||||
pending_fail_ts = t
|
||||
edits_since_fail = 0
|
||||
first_edit_ts = None
|
||||
continue
|
||||
if ok is True and pending_fail_ts is not None:
|
||||
if edits_since_fail > 0 and first_edit_ts is not None:
|
||||
cycles += 1
|
||||
fix_latencies.append(t - first_edit_ts)
|
||||
pending_fail_ts = None
|
||||
edits_since_fail = 0
|
||||
first_edit_ts = None
|
||||
|
||||
mean_fix_latency = (
|
||||
sum(fix_latencies) / len(fix_latencies) if fix_latencies else None
|
||||
)
|
||||
|
||||
# Idle gaps between consecutive events.
|
||||
idle_gap_count = 0
|
||||
idle_gap_total = 0.0
|
||||
prev_ts = None
|
||||
for e in events:
|
||||
if prev_ts is not None:
|
||||
gap = (e.ts - prev_ts).total_seconds()
|
||||
if gap > idle_threshold_s:
|
||||
idle_gap_count += 1
|
||||
idle_gap_total += gap
|
||||
prev_ts = e.ts
|
||||
|
||||
# Command category histogram (command-kind events only).
|
||||
categories: Counter[str] = Counter()
|
||||
for e in events:
|
||||
if e.kind == "command":
|
||||
categories[_classify_command(_command_text(e))] += 1
|
||||
|
||||
return TraceDigest(
|
||||
event_count=len(events),
|
||||
session_duration_s=round(duration, 3),
|
||||
edit_count=edit_count,
|
||||
command_count=command_count,
|
||||
run_count=run_count,
|
||||
test_pass_count=test_pass_count,
|
||||
test_fail_count=test_fail_count,
|
||||
final_test_status=final_test_status,
|
||||
first_test_pass_offset_s=(
|
||||
round(first_pass_offset, 3) if first_pass_offset is not None else None
|
||||
),
|
||||
error_fix_cycles=cycles,
|
||||
mean_fix_latency_s=(round(mean_fix_latency, 3) if mean_fix_latency is not None else None),
|
||||
idle_gap_count=idle_gap_count,
|
||||
idle_gap_total_s=round(idle_gap_total, 3),
|
||||
command_categories=dict(sorted(categories.items())),
|
||||
kind_histogram=dict(sorted(kind_histogram.items())),
|
||||
)
|
||||
@@ -0,0 +1,233 @@
|
||||
"""GradeStore — grade persistence protocol + SQLite implementation (REQ-3-004, D-027).
|
||||
|
||||
Postgres-migration-ready (D-027): the protocol is the only surface the
|
||||
grading engine and API layers touch; swapping SQLiteGradeStore for a
|
||||
Postgres-backed implementation must not change call sites. The
|
||||
`grade_record` table uses only portable column types (str / JSON /
|
||||
datetime), so the same SQLModel schema stands up unchanged on Postgres.
|
||||
|
||||
Upsert, NOT append: (learner_id, task_id) is the grade identity — one row
|
||||
per learner per task holding the LATEST grade. `save` overwrites the whole
|
||||
row when the pair already exists, so a regrade replaces scores, verdict,
|
||||
created_at, digest, model and variant_seed wholesale. That is deliberately
|
||||
the opposite of TraceStore.append's dedup-keep-first contract: a trace is an
|
||||
append-only event log, a grade is latest-state, so the engine can re-grade
|
||||
a task idempotently as its rubric or input evolves.
|
||||
|
||||
Concurrency (a-3): the engine enables WAL + synchronous=NORMAL and a busy
|
||||
timeout at connection time, so a regrade writer and API readers do not hit
|
||||
`database is locked` on the single-box pilot.
|
||||
|
||||
`created_at` contract: callers stamp UTC (datetime.now(UTC)); SQLite stores
|
||||
it naive and the read paths re-label it tz-aware UTC (same boundary
|
||||
normalization as TelemetryEvent.ts, so the contract holds on any backend).
|
||||
|
||||
Boundary (D-027): `grading/` never imports `agents/` / `api/`; this module
|
||||
imports config only.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy import JSON, Index
|
||||
from sqlalchemy.orm import validates
|
||||
from sqlmodel import Field, Session, SQLModel, create_engine, select
|
||||
|
||||
from ..config import Settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GradeRecord(SQLModel, table=True):
|
||||
"""A persisted grade; (learner_id, task_id) is the PK — latest wins.
|
||||
|
||||
Written by the grading engine (one save per grade attempt), read by the
|
||||
API layer through the GradeStore protocol. Constraint enforcement
|
||||
mirrors TelemetryEvent: sqlmodel 0.0.42's metaclass drops pydantic
|
||||
constraints on table models, so SQLAlchemy `@validates` hooks enforce
|
||||
instead and the column types stay Postgres-ready (D-027).
|
||||
|
||||
Field contract:
|
||||
learner_id — non-empty learner identifier (same id space as traces).
|
||||
task_id — non-empty task identifier; grade identity is the
|
||||
(learner_id, task_id) pair — the same pair as trace
|
||||
identity, so a grade is keyed by the exact trace it
|
||||
was computed from.
|
||||
variant_seed — task-variant seed; None until P4 (D-029). v0.3
|
||||
grading is variant-blind.
|
||||
digest — compact deterministic trace digest (D-028) that fed
|
||||
the rubric prompt; persisted for auditability so the
|
||||
LLM's input stays reproducible.
|
||||
scores — validated rubric scores (per-criterion 0-4,
|
||||
strengths, gaps); JSON dict. An empty dict is legal
|
||||
(e.g. an UNGRADABLE_TRACE_INCOMPLETE record carries a
|
||||
verdict but no scores).
|
||||
verdict — first-class verdict string (rubric verdict or
|
||||
UNGRADABLE_TRACE_INCOMPLETE); non-empty.
|
||||
model — provider model that produced the scores (provenance).
|
||||
created_at — UTC grade timestamp; a regrade replaces it (latest
|
||||
save wins).
|
||||
"""
|
||||
|
||||
__tablename__ = "grade_record"
|
||||
# The composite PK covers (learner_id, task_id) point lookups; this
|
||||
# secondary index covers list_for_learner ordered by created_at without
|
||||
# a sort step (Postgres migration target D-027).
|
||||
__table_args__ = (
|
||||
Index("ix_grade_record_learner_created", "learner_id", "created_at"),
|
||||
)
|
||||
|
||||
learner_id: str = Field(primary_key=True)
|
||||
task_id: str = Field(primary_key=True)
|
||||
variant_seed: str | None = Field(default=None) # null until P4 (D-029)
|
||||
# JSON columns: stored as TEXT on SQLite, native JSONB on Postgres (D-027).
|
||||
digest: dict[str, Any] = Field(default_factory=dict, sa_type=JSON)
|
||||
scores: dict[str, Any] = Field(default_factory=dict, sa_type=JSON)
|
||||
verdict: str
|
||||
model: str
|
||||
created_at: datetime
|
||||
|
||||
@validates("learner_id", "task_id")
|
||||
def _ids_non_empty(self, key: str, value: str) -> str:
|
||||
if not value:
|
||||
raise ValueError(f"{key} must be a non-empty identifier")
|
||||
return value
|
||||
|
||||
@validates("verdict")
|
||||
def _verdict_non_empty(self, key: str, value: str) -> str:
|
||||
if not value:
|
||||
raise ValueError(f"{key} must be a non-empty verdict string")
|
||||
return value
|
||||
|
||||
|
||||
class GradeStore(Protocol):
|
||||
"""Persistence contract for latest-state grades per (learner_id, task_id).
|
||||
|
||||
Implemented by SQLiteGradeStore (v0.3, D-027); a Postgres implementation
|
||||
must satisfy the same surface.
|
||||
"""
|
||||
|
||||
def save(self, grade: GradeRecord) -> None:
|
||||
"""Persist a grade. UPSERT on (learner_id, task_id): a regrade with
|
||||
the same pair REPLACES the stored row wholesale — the latest grade
|
||||
wins. NOT append-only; contrast TraceStore.append, which is
|
||||
dedup-keep-first for at-least-once ingest.
|
||||
"""
|
||||
...
|
||||
|
||||
def get(self, learner_id: str, task_id: str) -> GradeRecord | None:
|
||||
"""Latest stored grade for the pair; None when none exists.
|
||||
|
||||
Detached from any DB session — safe to pass across layers.
|
||||
"""
|
||||
...
|
||||
|
||||
def list_for_learner(self, learner_id: str) -> list[GradeRecord]:
|
||||
"""All stored grades for the learner, ordered by created_at
|
||||
ascending (chronological). Empty list when the learner has none.
|
||||
"""
|
||||
...
|
||||
|
||||
def close(self) -> None:
|
||||
"""Release DB connections. Store must not be used after close."""
|
||||
...
|
||||
|
||||
|
||||
def _sqlite_connect(dbapi_connection: sqlite3.Connection, _: object) -> None:
|
||||
"""Per-connection pragma setup (a-3). Mirrors telemetry/store.py.
|
||||
|
||||
journal_mode=WAL — readers never block the single writer.
|
||||
synchronous=NORMAL — safe in WAL mode, avoids full fsync-per-commit.
|
||||
busy_timeout=5000 — retry briefly under contention instead of
|
||||
`OperationalError: database is locked`.
|
||||
"""
|
||||
cursor = dbapi_connection.cursor()
|
||||
cursor.execute("PRAGMA journal_mode=WAL")
|
||||
cursor.execute("PRAGMA synchronous=NORMAL")
|
||||
cursor.execute("PRAGMA busy_timeout=5000")
|
||||
cursor.close()
|
||||
|
||||
|
||||
def _as_utc(ts: datetime) -> datetime:
|
||||
"""Timestamps travel as naive datetime on SQLite, tz-aware elsewhere.
|
||||
|
||||
SQLite (via SQLModel) drops tzinfo; Postgres TIMESTAMP WITH TIME ZONE
|
||||
keeps it. Normalizing on the read path makes the store's contract
|
||||
tz-aware UTC regardless of the backend (D-027).
|
||||
"""
|
||||
if ts.tzinfo is None:
|
||||
return ts.replace(tzinfo=UTC) # naive UTC read-side: label as UTC
|
||||
return ts.astimezone(UTC)
|
||||
|
||||
|
||||
class SQLiteGradeStore:
|
||||
"""SQLite-backed GradeStore (SQLModel). Second protocol-wrapped store
|
||||
of the D-027 family (first: SQLiteTraceStore).
|
||||
"""
|
||||
|
||||
def __init__(self, db_path: Path | None = None) -> None:
|
||||
self._db_path: Path = db_path if db_path is not None else Settings().db_path
|
||||
self._engine = create_engine(f"sqlite:///{self._db_path}")
|
||||
sa.event.listen(self._engine, "connect", _sqlite_connect)
|
||||
SQLModel.metadata.create_all(self._engine)
|
||||
|
||||
@contextmanager
|
||||
def _session(self) -> Iterator[Session]:
|
||||
# expire_on_commit=False: identical session behavior to
|
||||
# SQLiteTraceStore. save() discards the merged instance and the read
|
||||
# paths never commit, but a uniform flag across the D-027 stores
|
||||
# keeps their detachment guarantees from diverging.
|
||||
with Session(self._engine, expire_on_commit=False) as session:
|
||||
yield session
|
||||
|
||||
def save(self, grade: GradeRecord) -> None:
|
||||
# `merge` = SELECT-by-PK then UPDATE or INSERT — exactly the upsert
|
||||
# contract. The trace store deliberately avoids merge (its append is
|
||||
# dedup-keep-first); here latest-wins IS the contract, so merge is
|
||||
# the right tool. The caller's object is never attached to the
|
||||
# session and stays usable (unexpired) after save.
|
||||
with self._session() as session:
|
||||
session.merge(grade)
|
||||
session.commit()
|
||||
logger.debug(
|
||||
"grade saved (regrade overwrites): %s/%s verdict=%s model=%s",
|
||||
grade.learner_id,
|
||||
grade.task_id,
|
||||
grade.verdict,
|
||||
grade.model,
|
||||
)
|
||||
|
||||
def get(self, learner_id: str, task_id: str) -> GradeRecord | None:
|
||||
with self._session() as session:
|
||||
record = session.get(GradeRecord, (learner_id, task_id))
|
||||
if record is None:
|
||||
return None
|
||||
record.created_at = _as_utc(record.created_at)
|
||||
# Detach from the session: callers must not depend on
|
||||
# open-session ORM magic (lazy loads fail once it closes).
|
||||
session.expunge(record)
|
||||
return record
|
||||
|
||||
def list_for_learner(self, learner_id: str) -> list[GradeRecord]:
|
||||
with self._session() as session:
|
||||
stmt = (
|
||||
select(GradeRecord)
|
||||
.where(GradeRecord.learner_id == learner_id)
|
||||
# Chronological; task_id is a deterministic tie-break for
|
||||
# grades stamped within the same instant.
|
||||
.order_by(GradeRecord.created_at, GradeRecord.task_id)
|
||||
)
|
||||
results = session.exec(stmt).all()
|
||||
for row in results:
|
||||
row.created_at = _as_utc(row.created_at)
|
||||
session.expunge(row)
|
||||
return list(results)
|
||||
|
||||
def close(self) -> None:
|
||||
self._engine.dispose()
|
||||
@@ -21,6 +21,8 @@ from .api import (
|
||||
telemetry_router,
|
||||
)
|
||||
from .config import Settings
|
||||
from .grading.engine import GradingEngine
|
||||
from .grading.store import SQLiteGradeStore
|
||||
from .llm import create_provider
|
||||
from .sandbox import SandboxManager, UnshareBackend
|
||||
from .telemetry.ingest import TraceIntegrityMap
|
||||
@@ -71,6 +73,27 @@ def create_app(settings: Settings | None = None) -> FastAPI:
|
||||
if getattr(app.state, "trace_integrity", None) is None:
|
||||
app.state.trace_integrity = TraceIntegrityMap()
|
||||
|
||||
# Grading persistence + engine (REQ-3-004): GradeStore from the same
|
||||
# SQLite file as traces (D-027), one GradingEngine singleton wired
|
||||
# through app.state — the engine receives its stores via constructor
|
||||
# DI and knows nothing of FastAPI (api/ owns composition). Tests may
|
||||
# pre-set app.state.grade_store / app.state.grading_engine (the same
|
||||
# state-injection override as sandbox_manager/trace_store) to swap
|
||||
# either; the lifespan adopts a pre-set store but NEVER rebuilds a
|
||||
# pre-set engine (its provider binding is part of the test fixture).
|
||||
grade_store = getattr(app.state, "grade_store", None)
|
||||
if grade_store is None:
|
||||
grade_store = SQLiteGradeStore(db_path=settings.db_path)
|
||||
app.state.grade_store = grade_store
|
||||
if getattr(app.state, "grading_engine", None) is None:
|
||||
app.state.grading_engine = GradingEngine(
|
||||
trace_store,
|
||||
grade_store,
|
||||
app.state.trace_integrity,
|
||||
app.state.provider,
|
||||
model=settings.model,
|
||||
)
|
||||
|
||||
async def _reaper_loop() -> None:
|
||||
# Wall-clock timeout + G-2 workdir-size sweep, one pass per tick.
|
||||
while True:
|
||||
@@ -91,6 +114,7 @@ def create_app(settings: Settings | None = None) -> FastAPI:
|
||||
# everything live; workdirs stay on disk for snapshot restore.
|
||||
await manager.destroy_all()
|
||||
trace_store.close()
|
||||
grade_store.close()
|
||||
await app.state.http_client.aclose()
|
||||
|
||||
app = FastAPI(title="Nextcraft AI Service", version="0.3.0", lifespan=lifespan)
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
"""Grading rubric prompt — criteria, level anchors, digest render (REQ-3-004).
|
||||
|
||||
The grading prompt is deliberately learner-anonymous and trace-bare: the
|
||||
model receives ONLY the fixed rubric text and the compact numeric digest
|
||||
(TraceDigest JSON, D-028) — never a raw command, file path, payload
|
||||
string, learner id, or task id. Everything variable the LLM sees is
|
||||
deterministic counters, which both bounds the prompt-injection surface
|
||||
and makes "no raw trace reaches the prompt" assert-able in tests (plant
|
||||
a distinctive marker in a command payload; assert it absent from every
|
||||
message the provider received).
|
||||
|
||||
Rubric (four criteria, each scored 0-4 — the ids are the validated
|
||||
RubricScore keys enforced by grading/engine.py):
|
||||
process_quality — iterative building in small, verified steps.
|
||||
correctness — where the session ended (test/run outcomes).
|
||||
debugging_discipline — how failures were handled.
|
||||
test_usage — when and how often tests were run.
|
||||
|
||||
Advisory a-4 (embedded in the process_quality anchors): high edit/command
|
||||
churn with NO test progress is a process-quality NEGATIVE — churn is not
|
||||
work. A session with many edits/commands whose test state never moves is
|
||||
thrashing, not iterating, and must score low on process quality.
|
||||
|
||||
House-style deviation, documented: unlike the tutor prompts, this module
|
||||
has no SYSTEM_PROMPT placeholders and no render_context(learner_context)
|
||||
— grading is context-free by design (learner anonymity; the digest is the
|
||||
only variable input). Runtime imports are TYPE_CHECKING-only so this
|
||||
module stays pure text and can never import-cycle with grading/engine.py
|
||||
(engine imports this module; if this module imported grading.* at runtime
|
||||
while grading/__init__ pulls engine, the package init would deadlock on a
|
||||
partially-initialized module).
|
||||
|
||||
Version: grader-v1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Final
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover - typing only; keeps this module pure text
|
||||
from ..grading.features import TraceDigest
|
||||
|
||||
PROMPT_VERSION = "grader-v1"
|
||||
|
||||
#: Canonical criterion ids. The engine validates RubricScore criteria keys
|
||||
#: against this tuple; the schema hint and anchors below speak the same ids.
|
||||
RUBRIC_CRITERIA: Final[tuple[str, ...]] = (
|
||||
"process_quality",
|
||||
"correctness",
|
||||
"debugging_discipline",
|
||||
"test_usage",
|
||||
)
|
||||
|
||||
#: Sentinel line the engine's user turn is rendered around. Tests (and the
|
||||
#: calibration mock) split on it to locate the digest JSON in the prompt.
|
||||
DIGEST_MARKER: Final = "PROCESS TRACE DIGEST (JSON):"
|
||||
|
||||
SYSTEM_PROMPT = """You are the Grader of Nextcraft, an AI-native competency school.
|
||||
You score a learner's build session from a compact numeric digest of their
|
||||
process trace. You NEVER see the raw trace — commands, file contents, and
|
||||
payloads do not exist on your side; every number you need is in the digest.
|
||||
|
||||
Rubric — score each criterion 0-4:
|
||||
|
||||
process_quality — iterative building in small, verified steps.
|
||||
4: tight edit→test loops throughout; small verified increments; healthy pacing.
|
||||
3: steady small edits with regular runs; progress mostly verified.
|
||||
2: some iteration, but large unverified leaps or long idle stretches.
|
||||
1: a single bulk change (e.g. one large paste) then a single run; no iteration.
|
||||
0: no meaningful work visible.
|
||||
ADVISORY: high edit/command churn with NO test progress (no runs, no
|
||||
movement in pass counts) is a process-quality NEGATIVE — churn is not
|
||||
work. Cap such a session at 1 on this criterion no matter how many
|
||||
edits or commands were counted.
|
||||
|
||||
correctness — where the session ended up.
|
||||
4: final test status pass, with tests passing early and consistently.
|
||||
3: final pass, reached through fail→fix→pass cycles that closed.
|
||||
2: final pass, but preceded by a long unresolved failure streak.
|
||||
1: final fail, but partial passes observed along the way.
|
||||
0: final fail, or no test/run evidence at all.
|
||||
|
||||
debugging_discipline — how failures were handled.
|
||||
4: every failure cycle closes; targeted fixes with low mean fix latency.
|
||||
3: most fail→edit→re-run cycles close with a pass.
|
||||
2: failures followed by edits, but cycles rarely close.
|
||||
1: repeated failures with no targeted edits between runs (flailing).
|
||||
0: failures with no fix attempts at all.
|
||||
|
||||
test_usage — when and how often tests were run.
|
||||
4: tests run early (small first-pass offset) and throughout the session.
|
||||
3: regular test runs interleaved with edits.
|
||||
2: sparse tests; long stretches of unverified edits.
|
||||
1: a single late test run only.
|
||||
0: no test or run evidence.
|
||||
|
||||
Rules:
|
||||
- Judge STRICTLY from the digest numbers; cite the fields you used.
|
||||
- Strengths: the two strongest digest observations, one sentence each.
|
||||
- Gaps: the two most important missed opportunities, one sentence each
|
||||
(a clean session names its next-level improvement instead).
|
||||
- Be rigorous but fair: a session that ends green was not necessarily
|
||||
well built, and a struggling session that never passed may still show
|
||||
real debugging discipline.
|
||||
- Respond with ONLY a valid JSON object matching the provided schema —
|
||||
no markdown fences, no prose outside the JSON."""
|
||||
|
||||
RUBRIC_SCORE_SCHEMA_HINT = (
|
||||
'{"criteria": {"process_quality": <0-4 int>, "correctness": <0-4 int>, '
|
||||
'"debugging_discipline": <0-4 int>, "test_usage": <0-4 int>}, '
|
||||
'"strengths": ["<one sentence>"], "gaps": ["<one sentence>"], '
|
||||
'"verdict": "mastered" | "developing" | "not_yet"}'
|
||||
)
|
||||
|
||||
|
||||
def render_trace_digest(digest: TraceDigest) -> str:
|
||||
"""Render the grader's user turn: a marker line + the digest JSON — nothing else.
|
||||
|
||||
This is the ONLY per-session content that ever reaches the LLM (D-028):
|
||||
the engine composes [system: SYSTEM_PROMPT, user: render_trace_digest(digest)]
|
||||
and the D-020 defense appends its generic schema instruction to this
|
||||
user turn at request time. No learner id, task id, or raw trace material
|
||||
is injected — assert-able by tests.
|
||||
"""
|
||||
return (
|
||||
"Score this build session against the rubric.\n"
|
||||
f"{DIGEST_MARKER}\n{digest.model_dump_json()}"
|
||||
)
|
||||
@@ -0,0 +1,397 @@
|
||||
"""Assessment grade endpoint tests — grading engine over HTTP (Task 3-3-01).
|
||||
|
||||
Contract under test (api/assessment.py, REQ-3-004):
|
||||
|
||||
POST /v1/assessment/grade {learner_id, task_id}
|
||||
GRADED → 200, rubric scores + digest summary
|
||||
UNGRADABLE_TRACE_INCOMPLETE → 200, gate record (missing_seqs /
|
||||
integrity_flag surfaced in scores)
|
||||
UNGRADABLE_EMPTY_TRACE → 200, gate record (documented choice: an
|
||||
unknown task is ALSO an empty pair; the
|
||||
engine cannot distinguish, and the gate
|
||||
outcome is a durable first-class result)
|
||||
StructuredOutputError → 502 (provider exhausted the D-020 budget)
|
||||
GET /v1/assessment/grade/{learner_id}/{task_id}
|
||||
stored latest grade → 200 (same scores as the POST)
|
||||
never-graded pair → 404
|
||||
|
||||
Wiring: per-test tmp-path SQLite stores + a pre-set GradingEngine
|
||||
(state-injection override — the lifespan adopts trace_store/grade_store/
|
||||
trace_integrity/grading_engine from app.state instead of constructing
|
||||
them; same pattern as test_telemetry_ingest.py / test_sandboxes.py). The
|
||||
engine binds a scripted provider so each test controls the LLM exactly,
|
||||
and counts calls so gate tests can assert the LLM was never reached
|
||||
(G-4 holds through the whole HTTP stack).
|
||||
|
||||
Zero network: providers are MockProvider subclasses only (conftest rule).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from ai_service.config import Settings
|
||||
from ai_service.corpus.trace_fixtures import STRONG_BUILDER
|
||||
from ai_service.grading.engine import GradingEngine
|
||||
from ai_service.grading.store import SQLiteGradeStore
|
||||
from ai_service.llm.mock import MockProvider, ScriptedJSONProvider
|
||||
from ai_service.main import create_app
|
||||
from ai_service.telemetry.ingest import TraceIntegrityMap
|
||||
from ai_service.telemetry.models import TelemetryEvent
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
|
||||
LEARNER = "grade-learner"
|
||||
TASK = "task-grade-1"
|
||||
T0 = datetime(2026, 9, 12, 3, 0, 0, tzinfo=UTC)
|
||||
|
||||
#: Canonical well-formed rubric payload (matches grading.engine.RubricScore).
|
||||
RUBRIC_PAYLOAD: dict = {
|
||||
"criteria": {
|
||||
"process_quality": 4,
|
||||
"correctness": 4,
|
||||
"debugging_discipline": 3,
|
||||
"test_usage": 4,
|
||||
},
|
||||
"strengths": ["tight edit-test loops throughout"],
|
||||
"gaps": ["final commit discipline loose"],
|
||||
"verdict": "mastered",
|
||||
}
|
||||
|
||||
|
||||
class CountingScriptedProvider(ScriptedJSONProvider):
|
||||
"""ScriptedJSONProvider that counts chat() calls.
|
||||
|
||||
Gate tests assert calls == 0 (the LLM is never reached through the
|
||||
whole HTTP stack — G-4); happy-path tests sanity-check calls >= 1.
|
||||
"""
|
||||
|
||||
def __init__(self, payload: dict) -> None:
|
||||
super().__init__(payload)
|
||||
self.calls = 0
|
||||
|
||||
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
self.calls += 1
|
||||
return await super().chat(
|
||||
messages, model=model, temperature=temperature, response_format=response_format
|
||||
)
|
||||
|
||||
|
||||
def _event(seq: int, kind: str, payload: dict | None = None, offset_s: float = 0.0):
|
||||
return TelemetryEvent(
|
||||
learner_id=LEARNER,
|
||||
task_id=TASK,
|
||||
seq=seq,
|
||||
kind=kind,
|
||||
payload=payload or {},
|
||||
ts=T0 + timedelta(seconds=offset_s),
|
||||
sandbox_id="sbx-grade",
|
||||
)
|
||||
|
||||
|
||||
def _complete_trace() -> list[TelemetryEvent]:
|
||||
"""Contiguous seq 0..6 — a gradeable trace (gap-free, unflagged)."""
|
||||
return [
|
||||
_event(0, "activity", {"state": "starting"}, 0.0),
|
||||
_event(1, "file_diff", {"path": "a.py", "added": 12}, 10.0),
|
||||
_event(2, "command", {"cmd": "pytest -q"}, 20.0),
|
||||
_event(3, "test_result", {"passed": False, "exit_code": 1}, 25.0),
|
||||
_event(4, "file_diff", {"path": "a.py", "added": 4, "removed": 2}, 40.0),
|
||||
_event(5, "command", {"cmd": "pytest -q"}, 60.0),
|
||||
_event(6, "test_result", {"passed": True, "exit_code": 0}, 65.0),
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def trace_store(tmp_path: Path) -> Iterator[SQLiteTraceStore]:
|
||||
store = SQLiteTraceStore(db_path=tmp_path / "traces.db")
|
||||
yield store
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def grade_store(tmp_path: Path) -> Iterator[SQLiteGradeStore]:
|
||||
store = SQLiteGradeStore(db_path=tmp_path / "grades.db")
|
||||
yield store
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def integrity() -> TraceIntegrityMap:
|
||||
return TraceIntegrityMap()
|
||||
|
||||
|
||||
def _make_client(
|
||||
tmp_path: Path,
|
||||
trace_store: SQLiteTraceStore,
|
||||
grade_store: SQLiteGradeStore,
|
||||
integrity: TraceIntegrityMap,
|
||||
provider: MockProvider,
|
||||
) -> TestClient:
|
||||
"""App + TestClient with stores, integrity map and a pre-set engine.
|
||||
|
||||
The lifespan adopts every pre-set service (state-injection override);
|
||||
the engine binds OUR provider, so the fixture — not Settings — scripts
|
||||
the LLM. Cloud-free guard: the provider must be a MockProvider family
|
||||
member (conftest rule, enforced here because this module builds its
|
||||
own client rather than consuming the conftest one).
|
||||
"""
|
||||
assert isinstance(provider, MockProvider)
|
||||
settings = Settings(
|
||||
provider="mock",
|
||||
db_path=tmp_path / "grading-test.db",
|
||||
sandbox_dir=tmp_path / "sandboxes",
|
||||
)
|
||||
app = create_app(settings)
|
||||
app.state.trace_store = trace_store
|
||||
app.state.grade_store = grade_store
|
||||
app.state.trace_integrity = integrity
|
||||
app.state.grading_engine = GradingEngine(
|
||||
trace_store, grade_store, integrity, provider, model="gemma4:31b"
|
||||
)
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _seed(store: SQLiteTraceStore, events: list[TelemetryEvent]) -> None:
|
||||
for e in events:
|
||||
store.append(e)
|
||||
|
||||
|
||||
def _post(client: TestClient, learner: str = LEARNER, task: str = TASK):
|
||||
return client.post("/v1/assessment/grade", json={"learner_id": learner, "task_id": task})
|
||||
|
||||
|
||||
def _get(client: TestClient, learner: str = LEARNER, task: str = TASK):
|
||||
return client.get(f"/v1/assessment/grade/{learner}/{task}")
|
||||
|
||||
|
||||
# -- happy path: complete trace → GRADED -----------------------------------------
|
||||
|
||||
|
||||
class TestGraded:
|
||||
def test_post_complete_trace_returns_rubric_scores_and_digest(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
_seed(trace_store, _complete_trace())
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _post(client)
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["verdict"] == "GRADED"
|
||||
assert body["learner_id"] == LEARNER
|
||||
assert body["task_id"] == TASK
|
||||
assert body["scores"] == RUBRIC_PAYLOAD # rubric verdict rides in scores
|
||||
assert body["scores"]["verdict"] == "mastered"
|
||||
assert body["variant_seed"] is None # null until P4 (D-029)
|
||||
assert body["model"] == "gemma4:31b" # provenance travels
|
||||
assert provider.calls == 1 # happy path: exactly one LLM call
|
||||
# digest summary rides along (D-028 reproducible input)
|
||||
assert body["digest"]["event_count"] == 7
|
||||
assert body["digest"]["final_test_status"] == "pass"
|
||||
|
||||
def test_corpus_fixture_trace_grades(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
"""The strong-builder corpus fixture (real calibration trace) is
|
||||
gradeable over HTTP end to end."""
|
||||
_seed(trace_store, STRONG_BUILDER.events)
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _post(
|
||||
client, learner=STRONG_BUILDER.learner_id, task=STRONG_BUILDER.task_id
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["verdict"] == "GRADED"
|
||||
assert body["digest"]["event_count"] == len(STRONG_BUILDER.event_specs)
|
||||
|
||||
def test_get_after_post_returns_same_stored_scores(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
_seed(trace_store, _complete_trace())
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
posted = _post(client)
|
||||
assert posted.status_code == 200
|
||||
|
||||
fetched = _get(client)
|
||||
assert fetched.status_code == 200
|
||||
stored = fetched.json()
|
||||
|
||||
assert stored["scores"] == posted.json()["scores"]
|
||||
assert stored["verdict"] == "GRADED"
|
||||
assert stored["digest"] == posted.json()["digest"]
|
||||
assert stored["created_at"] == posted.json()["created_at"]
|
||||
|
||||
def test_post_regrade_upserts_get_returns_latest(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
"""POST twice → one row holding the LATEST grade (GradeStore upsert)."""
|
||||
_seed(trace_store, _complete_trace())
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
first = _post(client)
|
||||
assert first.status_code == 200
|
||||
assert first.json()["scores"]["verdict"] == "mastered"
|
||||
|
||||
# the same provider now scripts a different (weaker) rubric
|
||||
updated = {
|
||||
**RUBRIC_PAYLOAD,
|
||||
"criteria": {**RUBRIC_PAYLOAD["criteria"], "process_quality": 1},
|
||||
"verdict": "not_yet",
|
||||
}
|
||||
provider.payload = updated
|
||||
second = _post(client)
|
||||
assert second.status_code == 200
|
||||
assert second.json()["scores"] == updated
|
||||
|
||||
# GET returns the LATEST grade, not the first
|
||||
fetched = _get(client)
|
||||
assert fetched.json()["scores"] == updated
|
||||
assert fetched.json()["scores"]["verdict"] == "not_yet"
|
||||
|
||||
# one row, latest wins (upsert, not append)
|
||||
grades = grade_store.list_for_learner(LEARNER)
|
||||
assert len(grades) == 1
|
||||
assert grades[0].scores == updated
|
||||
|
||||
|
||||
# -- G-4 gate outcomes over HTTP: 200 with the ungradable record -------------------
|
||||
|
||||
|
||||
class TestGateOutcomes:
|
||||
def test_gapped_trace_200_ungradable_incomplete_with_missing_seqs(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
_seed(trace_store, [e for e in _complete_trace() if e.seq != 2]) # seq 2 missing
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _post(client)
|
||||
|
||||
assert response.status_code == 200 # ungradable IS a valid result — not 5xx
|
||||
body = response.json()
|
||||
assert body["verdict"] == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
assert body["scores"]["missing_seqs"] == [2]
|
||||
assert body["scores"]["integrity_flag"] is None
|
||||
assert body["digest"] == {} # nothing was graded
|
||||
assert body["model"] == "none" # no LLM involved (honest provenance)
|
||||
assert provider.calls == 0, "LLM was called despite a gapped trace (G-4)"
|
||||
|
||||
def test_flooded_trace_200_ungradable_incomplete_with_integrity_reason(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
"""Integrity-flagged trace (G-3 INCOMPLETE_FLOODED) surfaces the flag
|
||||
reason in scores — the same verdict as gaps, different detail."""
|
||||
_seed(trace_store, _complete_trace())
|
||||
integrity.mark(LEARNER, TASK, "INCOMPLETE_FLOODED")
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _post(client)
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["verdict"] == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
assert body["scores"]["integrity_flag"] == "INCOMPLETE_FLOODED"
|
||||
assert body["scores"]["missing_seqs"] == [] # rows complete but untrusted
|
||||
assert provider.calls == 0, "LLM was called despite an integrity flag (G-4)"
|
||||
|
||||
def test_empty_trace_200_ungradable_empty(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
"""Zero events for the pair (including a task that never had a
|
||||
trace) → UNGRADABLE_EMPTY_TRACE, persisted like any grade."""
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _post(client, learner="ghost-learner", task="never-started")
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["verdict"] == "UNGRADABLE_EMPTY_TRACE"
|
||||
assert body["scores"] == {"integrity_flag": None, "missing_seqs": []}
|
||||
assert body["digest"] == {}
|
||||
assert body["model"] == "none"
|
||||
assert provider.calls == 0
|
||||
|
||||
# the gate record is durable: GET now returns it (no longer 404)
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
fetched = _get(client, learner="ghost-learner", task="never-started")
|
||||
assert fetched.status_code == 200
|
||||
assert fetched.json()["verdict"] == "UNGRADABLE_EMPTY_TRACE"
|
||||
|
||||
|
||||
# -- provider failure → 502 --------------------------------------------------------
|
||||
|
||||
|
||||
class TestProviderFailure:
|
||||
def test_persistent_malformed_llm_output_502(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
"""Stock MockProvider: its json_object reply is wrong-shaped, so the
|
||||
D-020 defense exhausts both attempts → endpoint maps to 502 and
|
||||
nothing is persisted."""
|
||||
_seed(trace_store, _complete_trace())
|
||||
provider = MockProvider() # always malformed for the rubric schema
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _post(client)
|
||||
|
||||
assert response.status_code == 502
|
||||
assert "grading failed" in response.json()["detail"]
|
||||
# no fabricated/partial record was persisted
|
||||
assert grade_store.get(LEARNER, TASK) is None
|
||||
|
||||
def test_502_leaves_earlier_grade_intact(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
"""A failed regrade must not clobber the previously stored grade."""
|
||||
_seed(trace_store, _complete_trace())
|
||||
scripted = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, scripted) as client:
|
||||
assert _post(client).status_code == 200
|
||||
|
||||
# regrade attempt hits a provider that now always fails
|
||||
broken = MockProvider()
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, broken) as client:
|
||||
assert _post(client).status_code == 502
|
||||
fetched = _get(client)
|
||||
|
||||
assert fetched.status_code == 200 # earlier grade still readable
|
||||
assert fetched.json()["scores"] == RUBRIC_PAYLOAD
|
||||
|
||||
|
||||
# -- stored-grade reads ------------------------------------------------------------
|
||||
|
||||
|
||||
class TestStoredGradeReads:
|
||||
def test_get_unknown_pair_404(self, tmp_path, trace_store, grade_store, integrity):
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _get(client, learner="nobody", task="never-graded")
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "no stored grade" in response.json()["detail"]
|
||||
|
||||
def test_missing_body_fields_422(self, tmp_path, trace_store, grade_store, integrity):
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
assert client.post("/v1/assessment/grade", json={}).status_code == 422
|
||||
assert (
|
||||
client.post(
|
||||
"/v1/assessment/grade", json={"learner_id": LEARNER}
|
||||
).status_code
|
||||
== 422
|
||||
)
|
||||
@@ -0,0 +1 @@
|
||||
"""Process-trace grading tests (REQ-3-004)."""
|
||||
@@ -0,0 +1,361 @@
|
||||
"""Grading calibration CONTRACT test (Task 3-2-02, REQ-3-004, D-021).
|
||||
|
||||
HONEST SCOPE — what this test is and is NOT:
|
||||
|
||||
This is a CALIBRATION CONTRACT test, not an LLM quality test. The mock
|
||||
provider scripts archetype-mapped rubric scores for each fixture; the
|
||||
assertions verify that:
|
||||
|
||||
1. the ENGINE pipeline (gate → digest → prompt → D-020 → validate →
|
||||
persist) runs each archetype end-to-end and preserves the score
|
||||
ordering the scripts impose;
|
||||
2. the fixtures' digest FEATURES actually separate the archetypes
|
||||
deterministically (the non-LLM half of the calibration: paste-and-run
|
||||
vs iterative debugging vs never-passing MUST produce observably
|
||||
different digests — otherwise no rubric could ever separate them);
|
||||
3. fixture IDs are valid corpus-aligned strings (D-021).
|
||||
|
||||
It does NOT prove the real LLM scores archetypes in this order — that
|
||||
requires live-model evaluation, which is out of scope for v0.3 tests
|
||||
(tests NEVER call the cloud; conftest enforces). What the contract pins
|
||||
is the ORDERING the rubric + digest must eventually enforce:
|
||||
|
||||
strong.process_quality >= lazy.process_quality
|
||||
strong.correctness > struggling.correctness
|
||||
|
||||
If a future prompt/digest change makes the scripted mapping unachievable
|
||||
by ANY model (e.g. the digest stops separating the archetypes), test 2
|
||||
fails — that is the calibration signal.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from ai_service.corpus.telemetry import LAB_SCENARIOS
|
||||
from ai_service.corpus.trace_fixtures import (
|
||||
CALIBRATION_LEARNER_ID,
|
||||
LAZY_BUILDER,
|
||||
STRONG_BUILDER,
|
||||
STRUGGLING_BUILDER,
|
||||
TRACE_FIXTURES,
|
||||
digest_of,
|
||||
)
|
||||
from ai_service.grading.engine import GradingEngine
|
||||
from ai_service.grading.store import SQLiteGradeStore
|
||||
from ai_service.telemetry.ingest import TraceIntegrityMap
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
|
||||
#: Archetype-mapped rubric scripts (the ORDERING CONTRACT in numbers).
|
||||
#: strong: iterative verified building — highest process quality.
|
||||
#: lazy: ends pass with zero iteration — the a-4 negative (churn with no
|
||||
#: test progress caps process_quality at 1; here it's paste-churn +
|
||||
#: one late test, so 1). correctness lands high (final pass, fast).
|
||||
#: struggling: never passes — lowest correctness.
|
||||
ARCHETYPE_SCORES: dict[str, dict] = {
|
||||
"strong": {
|
||||
"criteria": {
|
||||
"process_quality": 4,
|
||||
"correctness": 4,
|
||||
"debugging_discipline": 4,
|
||||
"test_usage": 4,
|
||||
},
|
||||
"strengths": ["tight edit-test loops; every failure cycle closed"],
|
||||
"gaps": ["session ended without a final full-suite run"],
|
||||
"verdict": "mastered",
|
||||
},
|
||||
"lazy": {
|
||||
"criteria": {
|
||||
"process_quality": 1, # a-4: bulk paste + churn, one late test
|
||||
"correctness": 3,
|
||||
"debugging_discipline": 1,
|
||||
"test_usage": 1,
|
||||
},
|
||||
"strengths": ["final tests pass on the first run"],
|
||||
"gaps": ["one bulk paste with zero verified increments"],
|
||||
"verdict": "developing",
|
||||
},
|
||||
"struggling": {
|
||||
"criteria": {
|
||||
"process_quality": 2,
|
||||
"correctness": 0, # never passes
|
||||
"debugging_discipline": 1, # cycles never close
|
||||
"test_usage": 3,
|
||||
},
|
||||
"strengths": ["persisted through repeated failures"],
|
||||
"gaps": ["the same failure recurred three times"],
|
||||
"verdict": "not_yet",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class ArchetypeProvider:
|
||||
"""Mock provider scripting archetype-mapped scores keyed by fixture.
|
||||
|
||||
Keying on the digest embedded in the user turn (not on call order)
|
||||
keeps the mapping robust to gate-path noise and asserts the prompt
|
||||
really carries the fixture's digest. Records the last user turn so
|
||||
tests can re-assert D-028 (digest in prompt; raw events out).
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.digest_to_archetype: dict[str, str] = {}
|
||||
self.calls = 0
|
||||
self.last_user_prompt: str = ""
|
||||
|
||||
def register(self, fixture) -> None:
|
||||
self.digest_to_archetype[digest_of(fixture).model_dump_json()] = (
|
||||
fixture.archetype
|
||||
)
|
||||
|
||||
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
self.calls += 1
|
||||
user = next((m for m in reversed(messages) if m.role == "user"), None)
|
||||
assert user is not None
|
||||
self.last_user_prompt = user.content
|
||||
archetype = None
|
||||
for digest_json, arch in self.digest_to_archetype.items():
|
||||
if digest_json in user.content:
|
||||
archetype = arch
|
||||
break
|
||||
assert archetype is not None, "fixture digest not found in prompt"
|
||||
return json.dumps(ARCHETYPE_SCORES[archetype])
|
||||
|
||||
async def stream_chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
yield await self.chat(
|
||||
messages, model=model, temperature=temperature, response_format=response_format
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def trace_store(tmp_path: Path) -> SQLiteTraceStore:
|
||||
store = SQLiteTraceStore(db_path=tmp_path / "traces.db")
|
||||
yield store
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def grade_store(tmp_path: Path) -> SQLiteGradeStore:
|
||||
store = SQLiteGradeStore(db_path=tmp_path / "grades.db")
|
||||
yield store
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def provider() -> ArchetypeProvider:
|
||||
p = ArchetypeProvider()
|
||||
for fixture in (STRONG_BUILDER, LAZY_BUILDER, STRUGGLING_BUILDER):
|
||||
p.register(fixture)
|
||||
return p
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def engine(trace_store, grade_store, provider) -> GradingEngine:
|
||||
return GradingEngine(
|
||||
trace_store, grade_store, TraceIntegrityMap(), provider, model="gemma4:31b"
|
||||
)
|
||||
|
||||
|
||||
async def _grade_fixture(engine, trace_store, fixture) -> dict:
|
||||
for event in fixture.events:
|
||||
trace_store.append(event)
|
||||
record = await engine.grade(fixture.learner_id, fixture.task_id)
|
||||
assert record.verdict == "GRADED", (
|
||||
f"fixture {fixture.fixture_id} did not grade cleanly: {record.verdict}"
|
||||
)
|
||||
return record.scores
|
||||
|
||||
|
||||
class TestFixtureAlignment:
|
||||
"""D-021: fixture IDs align with the v0.2 corpus scenario IDs."""
|
||||
|
||||
def test_every_fixture_id_is_corpus_scenario_scoped(self):
|
||||
for fixture in TRACE_FIXTURES.values():
|
||||
assert fixture.fixture_id.count("::") == 1
|
||||
scenario_part, trace_part = fixture.fixture_id.split("::")
|
||||
assert scenario_part in LAB_SCENARIOS, (
|
||||
f"{scenario_part!r} is not a v0.2 lab scenario id"
|
||||
)
|
||||
assert trace_part.endswith("-trace")
|
||||
# double-brace-free single slug
|
||||
assert "-" in trace_part
|
||||
|
||||
def test_every_fixture_aligns_to_an_existing_scenario(self):
|
||||
assert set(f.aligned_scenario_id for f in TRACE_FIXTURES.values()) <= set(
|
||||
LAB_SCENARIOS
|
||||
)
|
||||
|
||||
def test_fixture_ids_are_valid_strings(self):
|
||||
for fixture in TRACE_FIXTURES.values():
|
||||
assert fixture.fixture_id and isinstance(fixture.fixture_id, str)
|
||||
assert fixture.learner_id == CALIBRATION_LEARNER_ID
|
||||
assert fixture.learner_id.startswith("learner-00")
|
||||
|
||||
def test_lazy_alignment_is_the_flagged_scenario(self):
|
||||
# v0.2's paste-and-run archetype is the flagged lab scenario (D-021).
|
||||
assert LAZY_BUILDER.aligned_scenario_id == "lab-scenario-flagged"
|
||||
assert LAZY_BUILDER.archetype == "lazy"
|
||||
|
||||
def test_all_three_archetypes_present(self):
|
||||
assert {f.archetype for f in TRACE_FIXTURES.values()} == {
|
||||
"strong",
|
||||
"lazy",
|
||||
"struggling",
|
||||
}
|
||||
|
||||
def test_events_are_valid_telemetry_events(self):
|
||||
from ai_service.telemetry.models import TelemetryEvent
|
||||
|
||||
for fixture in TRACE_FIXTURES.values():
|
||||
assert len(fixture.events) > 0
|
||||
for e in fixture.events:
|
||||
assert isinstance(e, TelemetryEvent)
|
||||
assert e.seq >= 0
|
||||
# contiguous seqs: a grading trace must clear the G-4 gate
|
||||
seqs = [e.seq for e in fixture.events]
|
||||
assert seqs == list(range(len(seqs))), (
|
||||
f"{fixture.fixture_id} has seq gaps — it would trip the gate"
|
||||
)
|
||||
|
||||
|
||||
class TestDigestSeparatesArchetypes:
|
||||
"""The non-LLM half of calibration: digests differ in the RIGHT direction.
|
||||
|
||||
If these deterministic assertions ever fail, no rubric — however good —
|
||||
could separate the archetypes: the digest would be lossy in the wrong
|
||||
places. These are the features the a-4 advisory and the rubric anchors
|
||||
speak about, computed over REAL trace fixtures.
|
||||
"""
|
||||
|
||||
def test_strong_shows_iterative_cycles_lazy_shows_none(self):
|
||||
strong = digest_of(STRONG_BUILDER)
|
||||
lazy = digest_of(LAZY_BUILDER)
|
||||
assert strong.error_fix_cycles >= 1
|
||||
assert lazy.error_fix_cycles == 0
|
||||
|
||||
def test_strong_tests_early_lazy_tests_late(self):
|
||||
strong = digest_of(STRONG_BUILDER)
|
||||
lazy = digest_of(LAZY_BUILDER)
|
||||
assert strong.first_test_pass_offset_s is not None
|
||||
assert lazy.first_test_pass_offset_s is not None
|
||||
assert strong.first_test_pass_offset_s < lazy.first_test_pass_offset_s
|
||||
|
||||
def test_struggling_never_passes(self):
|
||||
struggling = digest_of(STRUGGLING_BUILDER)
|
||||
assert struggling.final_test_status == "fail"
|
||||
assert struggling.test_pass_count == 0
|
||||
assert struggling.test_fail_count >= 3
|
||||
|
||||
def test_strong_and_struggling_both_end(self):
|
||||
strong = digest_of(STRONG_BUILDER)
|
||||
assert strong.final_test_status == "pass"
|
||||
assert strong.test_fail_count >= 1 # honest failure, then closed
|
||||
|
||||
def test_lazy_churns_without_test_progress(self):
|
||||
"""a-4 anchor: bulk edit + command churn with (almost) no test signal."""
|
||||
lazy = digest_of(LAZY_BUILDER)
|
||||
assert lazy.edit_count >= 2
|
||||
assert lazy.test_fail_count == 0
|
||||
assert lazy.command_count >= 2
|
||||
assert lazy.final_test_status == "pass" # ...and still one late test
|
||||
|
||||
|
||||
class TestOrderingContract:
|
||||
"""The scripted ORDERING CONTRACT through the full engine pipeline."""
|
||||
|
||||
async def test_strong_at_least_lazy_on_process_quality(
|
||||
self, engine, trace_store
|
||||
):
|
||||
strong = await _grade_fixture(engine, trace_store, STRONG_BUILDER)
|
||||
lazy = await _grade_fixture(engine, trace_store, LAZY_BUILDER)
|
||||
assert (
|
||||
strong["criteria"]["process_quality"]
|
||||
>= lazy["criteria"]["process_quality"]
|
||||
)
|
||||
|
||||
async def test_strong_above_struggling_on_correctness(self, engine, trace_store):
|
||||
strong = await _grade_fixture(engine, trace_store, STRONG_BUILDER)
|
||||
struggling = await _grade_fixture(engine, trace_store, STRUGGLING_BUILDER)
|
||||
assert strong["criteria"]["correctness"] > struggling["criteria"]["correctness"]
|
||||
|
||||
async def test_full_ordering_across_all_three_archetypes(
|
||||
self, engine, trace_store, grade_store
|
||||
):
|
||||
scores = {
|
||||
arch: await _grade_fixture(engine, trace_store, f)
|
||||
for arch, f in (
|
||||
("strong", STRONG_BUILDER),
|
||||
("lazy", LAZY_BUILDER),
|
||||
("struggling", STRUGGLING_BUILDER),
|
||||
)
|
||||
}
|
||||
# the required ordering contract (task 3-2-02):
|
||||
assert (
|
||||
scores["strong"]["criteria"]["process_quality"]
|
||||
>= scores["lazy"]["criteria"]["process_quality"]
|
||||
)
|
||||
assert (
|
||||
scores["strong"]["criteria"]["correctness"]
|
||||
> scores["struggling"]["criteria"]["correctness"]
|
||||
)
|
||||
# Deliberate NON-ordering, asserted to document the calibration
|
||||
# intent: struggling > lazy on process quality. Process quality is
|
||||
# not outcome — the struggling builder iterated (edits between
|
||||
# runs, three test runs), the lazy builder only pasted once and
|
||||
# verified once (a-4 caps that at 1). A rubric that scores a
|
||||
# never-passing iterator BELOW a paste-and-runner on process is
|
||||
# miscalibrated on its own anchors.
|
||||
assert (
|
||||
scores["struggling"]["criteria"]["process_quality"]
|
||||
> scores["lazy"]["criteria"]["process_quality"]
|
||||
)
|
||||
# verdicts also ordered: mastered >= developing > not_yet
|
||||
assert scores["strong"]["verdict"] == "mastered"
|
||||
assert scores["lazy"]["verdict"] == "developing"
|
||||
assert scores["struggling"]["verdict"] == "not_yet"
|
||||
|
||||
# all three persisted under the calibration learner, latest-state
|
||||
grades = grade_store.list_for_learner(CALIBRATION_LEARNER_ID)
|
||||
assert len(grades) == 3
|
||||
assert {g.task_id for g in grades} == {
|
||||
"task-calibration-strong",
|
||||
"task-calibration-lazy",
|
||||
"task-calibration-struggling",
|
||||
}
|
||||
|
||||
async def test_prompt_carries_fixture_digest_not_raw_events(
|
||||
self, engine, trace_store, provider
|
||||
):
|
||||
"""D-028 on the calibration path: digest in the prompt, raw events
|
||||
never — the lazy fixture plants distinctive payload strings
|
||||
("eval.py", the bulk 240-line paste) that must NOT appear."""
|
||||
for event in LAZY_BUILDER.events:
|
||||
trace_store.append(event)
|
||||
await engine.grade(LAZY_BUILDER.learner_id, LAZY_BUILDER.task_id)
|
||||
|
||||
assert provider.calls == 1
|
||||
prompt = provider.last_user_prompt
|
||||
# the fixture's digest reached the LLM...
|
||||
assert "PROCESS TRACE DIGEST (JSON):" in prompt
|
||||
assert '"error_fix_cycles"' in prompt
|
||||
# ...but no raw trace material or identity did (D-028)
|
||||
assert "eval.py" not in prompt
|
||||
assert LAZY_BUILDER.learner_id not in prompt
|
||||
assert LAZY_BUILDER.task_id not in prompt
|
||||
|
||||
async def test_struggling_digests_reach_engine_without_gate(
|
||||
self, engine, trace_store, provider
|
||||
):
|
||||
for event in STRUGGLING_BUILDER.events:
|
||||
trace_store.append(event)
|
||||
record = await engine.grade(
|
||||
STRUGGLING_BUILDER.learner_id, STRUGGLING_BUILDER.task_id
|
||||
)
|
||||
assert record.verdict == "GRADED" # honest struggle is gradable
|
||||
assert record.scores["criteria"]["correctness"] == 0
|
||||
# a never-passing session still yields a real verdict — the grade
|
||||
# is low, not missing
|
||||
assert record.scores["verdict"] == "not_yet"
|
||||
@@ -0,0 +1,449 @@
|
||||
"""GradingEngine tests (Task 3-2-01, REQ-3-004, G-4 binding).
|
||||
|
||||
Contract under test:
|
||||
- G-4 gate FIRST: seq gaps OR integrity-flagged trace →
|
||||
verdict=UNGRADABLE_TRACE_INCOMPLETE with the gap list surfaced in
|
||||
scores; the LLM is NEVER called on those paths (asserted). Empty
|
||||
trace → UNGRADABLE_EMPTY_TRACE. All are first-class GradeRecords
|
||||
(returned + persisted), not exceptions.
|
||||
- Complete trace: prompt carries the digest JSON but NO raw trace
|
||||
material (D-028) — a distinctive marker planted in a command payload
|
||||
must be absent from every message the provider received.
|
||||
- Malformed LLM JSON → the D-020 retry path recovers (scripted bad
|
||||
first, good second) → graded.
|
||||
- Re-grade overwrites the stored record (GradeStore upsert).
|
||||
|
||||
Zero network: everything runs against the mock provider (conftest rule).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from pydantic import BaseModel, ValidationError
|
||||
|
||||
from ai_service.grading.engine import GradingEngine, RubricScore
|
||||
from ai_service.grading.store import SQLiteGradeStore
|
||||
from ai_service.llm.types import Message
|
||||
from ai_service.telemetry.ingest import TraceIntegrityMap
|
||||
from ai_service.telemetry.models import TelemetryEvent
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
|
||||
T0 = datetime(2026, 9, 12, 1, 0, 0, tzinfo=UTC)
|
||||
RAW_MARKER = "SECRET-COMMAND-MARKER-7f3a"
|
||||
|
||||
|
||||
class RubricJSON:
|
||||
"""Canonical well-formed rubric payload (matches engine.RubricScore)."""
|
||||
|
||||
PAYLOAD: dict = {
|
||||
"criteria": {
|
||||
"process_quality": 3,
|
||||
"correctness": 4,
|
||||
"debugging_discipline": 3,
|
||||
"test_usage": 4,
|
||||
},
|
||||
"strengths": ["tight edit-test loops throughout"],
|
||||
"gaps": ["final commit discipline loose"],
|
||||
"verdict": "mastered",
|
||||
}
|
||||
|
||||
|
||||
class RecordingProvider:
|
||||
"""Mock LLMProvider: scripted replies + captured requests (no network).
|
||||
|
||||
Subclass-composes ai_service.llm.mock.MockProvider so the conftest
|
||||
cloud-free guard philosophy holds — but records every message list
|
||||
and counts calls, which the stock mocks don't do.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
from ai_service.llm.mock import MockProvider
|
||||
|
||||
self._mock = MockProvider()
|
||||
self.replies: list[str] = [] # consumed left-to-right; last repeats
|
||||
self.requests: list[list[Message]] = []
|
||||
self.calls = 0
|
||||
|
||||
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
self.calls += 1
|
||||
self.requests.append([Message(role=m.role, content=m.content) for m in messages])
|
||||
if self.replies:
|
||||
reply = self.replies.pop(0)
|
||||
else:
|
||||
reply = json.dumps(RubricJSON.PAYLOAD)
|
||||
return reply
|
||||
|
||||
async def stream_chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
yield await self.chat(
|
||||
messages, model=model, temperature=temperature, response_format=response_format
|
||||
)
|
||||
|
||||
|
||||
def _event(
|
||||
seq: int,
|
||||
kind: str,
|
||||
payload: dict | None = None,
|
||||
offset_s: float = 0.0,
|
||||
*,
|
||||
learner: str = "engine-learner",
|
||||
task: str = "engine-task",
|
||||
) -> TelemetryEvent:
|
||||
return TelemetryEvent(
|
||||
learner_id=learner,
|
||||
task_id=task,
|
||||
seq=seq,
|
||||
kind=kind,
|
||||
payload=payload or {},
|
||||
ts=T0 + timedelta(seconds=offset_s),
|
||||
sandbox_id="sbx-engine",
|
||||
)
|
||||
|
||||
|
||||
def _complete_trace(marker_command: bool = True) -> list[TelemetryEvent]:
|
||||
"""A healthy iterative trace; seq 0..N contiguous (no gaps).
|
||||
|
||||
`marker_command` plants a distinctive string inside a command payload
|
||||
— the D-028 assertion target: raw trace material must never reach the
|
||||
provider, so the marker must not appear in any captured request.
|
||||
"""
|
||||
if marker_command:
|
||||
cmd_payload = {"cmd": f"echo {RAW_MARKER} && pytest -q"}
|
||||
else:
|
||||
cmd_payload = {"cmd": "pytest -q"}
|
||||
return [
|
||||
_event(0, "activity", {"state": "starting"}, 0.0),
|
||||
_event(1, "file_diff", {"path": "a.py", "added": 12}, 10.0),
|
||||
_event(2, "command", cmd_payload, 20.0),
|
||||
_event(3, "test_result", {"passed": False, "exit_code": 1}, 25.0),
|
||||
_event(4, "file_diff", {"path": "a.py", "added": 4, "removed": 2}, 40.0),
|
||||
_event(5, "command", {"cmd": "pytest -q"}, 60.0),
|
||||
_event(6, "test_result", {"passed": True, "exit_code": 0}, 65.0),
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def trace_store(tmp_path: Path) -> SQLiteTraceStore:
|
||||
store = SQLiteTraceStore(db_path=tmp_path / "traces.db")
|
||||
yield store
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def grade_store(tmp_path: Path) -> SQLiteGradeStore:
|
||||
store = SQLiteGradeStore(db_path=tmp_path / "grades.db")
|
||||
yield store
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def integrity() -> TraceIntegrityMap:
|
||||
return TraceIntegrityMap()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def provider() -> RecordingProvider:
|
||||
return RecordingProvider()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def engine(trace_store, grade_store, integrity, provider) -> GradingEngine:
|
||||
return GradingEngine(
|
||||
trace_store, grade_store, integrity, provider, model="gemma4:31b"
|
||||
)
|
||||
|
||||
|
||||
def _ingest(store: SQLiteTraceStore, events: list[TelemetryEvent]) -> None:
|
||||
for e in events:
|
||||
store.append(e)
|
||||
|
||||
|
||||
class TestRubricScoreModel:
|
||||
"""The D-020 schema contract — engine-side validation rules."""
|
||||
|
||||
def _rubric(self, **overrides) -> dict:
|
||||
payload = json.loads(json.dumps(RubricJSON.PAYLOAD)) # deep copy
|
||||
payload.update(overrides)
|
||||
return payload
|
||||
|
||||
def test_well_formed_payload_validates(self):
|
||||
score = RubricScore.model_validate(RubricJSON.PAYLOAD)
|
||||
assert score.criteria["process_quality"] == 3
|
||||
assert score.verdict == "mastered"
|
||||
|
||||
def test_unknown_criterion_rejected(self):
|
||||
bad = self._rubric()
|
||||
bad["criteria"]["extra_criterion"] = 2
|
||||
with pytest.raises(ValidationError, match="criteria keys must be exactly"):
|
||||
RubricScore.model_validate(bad)
|
||||
|
||||
def test_missing_criterion_rejected(self):
|
||||
bad = self._rubric()
|
||||
del bad["criteria"]["test_usage"]
|
||||
with pytest.raises(ValidationError, match="criteria keys must be exactly"):
|
||||
RubricScore.model_validate(bad)
|
||||
|
||||
def test_out_of_range_score_rejected(self):
|
||||
bad = self._rubric()
|
||||
bad["criteria"]["process_quality"] = 5
|
||||
with pytest.raises(ValidationError, match="must be within 0-4"):
|
||||
RubricScore.model_validate(bad)
|
||||
|
||||
def test_unknown_verdict_rejected(self):
|
||||
bad = self._rubric()
|
||||
bad["verdict"] = "excellent"
|
||||
with pytest.raises(ValidationError, match="verdict must be one of"):
|
||||
RubricScore.model_validate(bad)
|
||||
|
||||
|
||||
class TestCompleteTraceGrading:
|
||||
async def test_graded_returns_validated_rubric_persisted_and_returned(
|
||||
self, engine, trace_store, grade_store, provider
|
||||
):
|
||||
_ingest(trace_store, _complete_trace())
|
||||
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "GRADED"
|
||||
assert record.model == "gemma4:31b"
|
||||
assert record.scores == RubricJSON.PAYLOAD
|
||||
assert record.scores["criteria"]["process_quality"] == 3
|
||||
assert record.digest.get("error_fix_cycles") == 1
|
||||
assert record.digest.get("final_test_status") == "pass"
|
||||
# digest persisted for auditability (D-028 reproducible input)
|
||||
assert record.digest.get("event_count") == 7
|
||||
|
||||
stored = grade_store.get("engine-learner", "engine-task")
|
||||
assert stored is not None
|
||||
assert stored.scores == record.scores
|
||||
assert stored.verdict == "GRADED"
|
||||
|
||||
async def test_prompt_contains_digest_but_no_raw_trace(self, engine, trace_store, provider):
|
||||
"""D-028: the provider saw the digest JSON but NEVER the raw trace.
|
||||
|
||||
The marker was planted inside a command payload (seq 2); it must be
|
||||
absent from every message of every request, while the digest marker
|
||||
line + a JSON object with the digest's field names must be present.
|
||||
"""
|
||||
_ingest(trace_store, _complete_trace(marker_command=True))
|
||||
|
||||
await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert provider.calls == 1 # happy path: exactly one LLM call
|
||||
all_messages = [m for req in provider.requests for m in req]
|
||||
prompt_text = "\n".join(m.content for m in all_messages)
|
||||
|
||||
assert RAW_MARKER not in prompt_text, (
|
||||
"raw trace material reached the LLM prompt (D-028 violation)"
|
||||
)
|
||||
# digest presence: marker line + a JSON object carrying digest fields
|
||||
assert "PROCESS TRACE DIGEST (JSON):" in prompt_text
|
||||
assert '"error_fix_cycles"' in prompt_text
|
||||
assert '"final_test_status"' in prompt_text
|
||||
# learner anonymity: identity strings never reach the LLM either
|
||||
assert "engine-learner" not in prompt_text
|
||||
assert "engine-task" not in prompt_text
|
||||
|
||||
async def test_system_prompt_has_rubric_and_a4_advisory(
|
||||
self, engine, trace_store, provider
|
||||
):
|
||||
_ingest(trace_store, _complete_trace())
|
||||
|
||||
await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
system = provider.requests[0][0]
|
||||
assert system.role == "system"
|
||||
assert "process_quality" in system.content
|
||||
assert "correctness" in system.content
|
||||
assert "debugging_discipline" in system.content
|
||||
assert "test_usage" in system.content
|
||||
# a-4: churn-without-test-progress is a process-quality NEGATIVE
|
||||
assert "churn is not" in system.content
|
||||
# 0-4 level anchors present for each criterion
|
||||
assert "ADVISORY" in system.content
|
||||
|
||||
|
||||
class TestGateFirst:
|
||||
"""G-4 binding: the gate fires BEFORE any grading work."""
|
||||
|
||||
async def test_gapped_trace_ungradable_llm_not_called(
|
||||
self, engine, trace_store, provider, grade_store
|
||||
):
|
||||
# seqs 0,1,3 stored — seq 2 missing (mirrors the task spec example)
|
||||
events = _complete_trace()
|
||||
gapped = [e for e in events if e.seq != 2]
|
||||
_ingest(trace_store, gapped)
|
||||
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
assert provider.calls == 0, "LLM was called despite a gapped trace (G-4)"
|
||||
# gap list surfaced in scores
|
||||
assert record.scores["missing_seqs"] == [2]
|
||||
assert record.scores["integrity_flag"] is None
|
||||
# first-class record: persisted, retrievable, empty digest
|
||||
stored = grade_store.get("engine-learner", "engine-task")
|
||||
assert stored is not None
|
||||
assert stored.verdict == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
assert stored.scores["missing_seqs"] == [2]
|
||||
assert stored.digest == {}
|
||||
assert stored.model == "none"
|
||||
|
||||
async def test_integrity_flagged_trace_ungradable_llm_not_called(
|
||||
self, engine, trace_store, integrity, provider
|
||||
):
|
||||
_ingest(trace_store, _complete_trace())
|
||||
integrity.mark("engine-learner", "engine-task", "INCOMPLETE_FLOODED")
|
||||
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
assert provider.calls == 0, "LLM was called despite integrity flag (G-4)"
|
||||
assert record.scores["integrity_flag"] == "INCOMPLETE_FLOODED"
|
||||
assert record.scores["missing_seqs"] == [] # complete rows, but untrusted
|
||||
|
||||
async def test_flag_wins_even_with_gaps(self, engine, trace_store, integrity, provider):
|
||||
"""Both gate signals set: the flag reason is surfaced (both listed)."""
|
||||
events = _complete_trace()
|
||||
_ingest(trace_store, [e for e in events if e.seq != 2])
|
||||
integrity.mark("engine-learner", "engine-task", "INCOMPLETE_FLOODED")
|
||||
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
assert provider.calls == 0
|
||||
assert record.scores["integrity_flag"] == "INCOMPLETE_FLOODED"
|
||||
assert record.scores["missing_seqs"] == [2]
|
||||
|
||||
async def test_empty_trace_ungradable(self, engine, provider, grade_store):
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "UNGRADABLE_EMPTY_TRACE"
|
||||
assert provider.calls == 0, "LLM was called for an empty trace"
|
||||
assert record.scores == {"integrity_flag": None, "missing_seqs": []}
|
||||
assert record.digest == {}
|
||||
stored = grade_store.get("engine-learner", "engine-task")
|
||||
assert stored is not None
|
||||
assert stored.verdict == "UNGRADABLE_EMPTY_TRACE"
|
||||
|
||||
|
||||
class TestD020RetryPath:
|
||||
async def test_malformed_first_response_recovers_via_retry(
|
||||
self, engine, trace_store, provider
|
||||
):
|
||||
"""First reply invalid (wrong shape, fenced), second valid → graded.
|
||||
|
||||
This exercises the D-020 layer-4 path INSIDE the engine through the
|
||||
real agents/structured.py — the engine does not reimplement it.
|
||||
"""
|
||||
_ingest(trace_store, _complete_trace())
|
||||
provider.replies = [
|
||||
'```json\n{"summary": "wrong shape"}\n```', # layer 3 rejects
|
||||
json.dumps(RubricJSON.PAYLOAD), # retry recovers
|
||||
]
|
||||
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert provider.calls == 2 # bounded to exactly one retry (layer 4)
|
||||
assert record.verdict == "GRADED"
|
||||
assert record.scores == RubricJSON.PAYLOAD
|
||||
# the retry feedback must carry the validation error back
|
||||
retry_request = provider.requests[1]
|
||||
retry_text = " ".join(m.content for m in retry_request)
|
||||
assert "previous response was invalid" in retry_text
|
||||
|
||||
async def test_persistently_malformed_raises_after_bounded_retry(
|
||||
self, engine, trace_store, provider
|
||||
):
|
||||
from ai_service.agents.structured import StructuredOutputError
|
||||
|
||||
_ingest(trace_store, _complete_trace())
|
||||
provider.replies = ["not json at all", "still not json"]
|
||||
|
||||
with pytest.raises(StructuredOutputError):
|
||||
await engine.grade("engine-learner", "engine-task")
|
||||
assert provider.calls == 2 # bounded: never more than one retry
|
||||
|
||||
|
||||
class TestRegrade:
|
||||
async def test_regrade_overwrites_stored_record(
|
||||
self, engine, trace_store, grade_store, provider
|
||||
):
|
||||
_ingest(trace_store, _complete_trace())
|
||||
|
||||
first = await engine.grade("engine-learner", "engine-task")
|
||||
assert first.verdict == "GRADED"
|
||||
|
||||
# second grade: the provider now scripts a DIFFERENT rubric outcome
|
||||
updated = json.loads(json.dumps(RubricJSON.PAYLOAD))
|
||||
updated["criteria"]["process_quality"] = 1
|
||||
updated["verdict"] = "not_yet"
|
||||
provider.replies = [json.dumps(updated)]
|
||||
|
||||
second = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert second.scores == updated
|
||||
assert second.created_at >= first.created_at
|
||||
# upsert, not append: exactly one row, holding the LATEST grade
|
||||
grades = grade_store.list_for_learner("engine-learner")
|
||||
assert len(grades) == 1
|
||||
assert grades[0].scores == updated
|
||||
assert grades[0].verdict == "GRADED"
|
||||
|
||||
async def test_regrade_after_gate_outcome_replaces_it(
|
||||
self, engine, trace_store, grade_store, provider, integrity
|
||||
):
|
||||
"""A later successful regrade wholesale-replaces a gate record —
|
||||
the GradeStore upsert contract applied across verdict kinds."""
|
||||
# first: gapped → gate record persisted
|
||||
events = _complete_trace()
|
||||
_ingest(trace_store, [e for e in events if e.seq != 2])
|
||||
gated = await engine.grade("engine-learner", "engine-task")
|
||||
assert gated.verdict == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
|
||||
# gap healed (late delivery): same pair regrades cleanly
|
||||
trace_store.append(next(e for e in events if e.seq == 2))
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "GRADED"
|
||||
stored = grade_store.get("engine-learner", "engine-task")
|
||||
assert stored is not None
|
||||
assert stored.verdict == "GRADED"
|
||||
assert stored.scores == RubricJSON.PAYLOAD
|
||||
assert len(grade_store.list_for_learner("engine-learner")) == 1
|
||||
|
||||
|
||||
class TestBoundary:
|
||||
def test_engine_module_imports_no_fastapi(self):
|
||||
"""D-027: the engine module must not import fastapi (transitively
|
||||
beyond the sanctioned agents/structured + llm + telemetry + prompts)."""
|
||||
import sys
|
||||
|
||||
import ai_service.grading.engine as engine_mod
|
||||
|
||||
code = open(engine_mod.__file__).read()
|
||||
assert "fastapi" not in code
|
||||
assert "from ..api" not in code and "from ..api." not in code
|
||||
# the one sanctioned agents import is the module-direct structured defense
|
||||
assert "from ..agents.structured import" in code
|
||||
assert "from ..agents import" not in code
|
||||
# sanity: the module really is loaded (no import cycle surprises)
|
||||
assert engine_mod.__name__ in sys.modules
|
||||
|
||||
|
||||
class TestModelValidationSmoke:
|
||||
"""RubricScore as a plain pydantic model (D-020 layer-3 target type)."""
|
||||
|
||||
def test_rubric_score_is_frozen_shape_for_scores_dict(self):
|
||||
score = RubricScore.model_validate(RubricJSON.PAYLOAD)
|
||||
dumped = score.model_dump()
|
||||
assert set(dumped["criteria"]) == {
|
||||
"process_quality",
|
||||
"correctness",
|
||||
"debugging_discipline",
|
||||
"test_usage",
|
||||
}
|
||||
assert isinstance(score, BaseModel)
|
||||
@@ -0,0 +1,170 @@
|
||||
"""Trace digest tests (Task 3-1-01, REQ-3-004).
|
||||
|
||||
Contract: `compute_digest` is deterministic, bounded (< 4 KB), and carries
|
||||
NO raw trace material (commands, file contents, payload strings) — the raw
|
||||
trace never reaches the LLM (D-028).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
import pytest
|
||||
|
||||
from ai_service.grading.features import TraceDigest, compute_digest
|
||||
from ai_service.telemetry.models import TelemetryEvent
|
||||
|
||||
T0 = datetime(2026, 9, 12, 1, 0, 0, tzinfo=UTC)
|
||||
|
||||
|
||||
def _event(
|
||||
seq: int, kind: str, payload: dict | None = None, offset_s: float = 0.0
|
||||
) -> TelemetryEvent:
|
||||
return TelemetryEvent(
|
||||
learner_id="features-learner",
|
||||
task_id="features-task",
|
||||
seq=seq,
|
||||
kind=kind,
|
||||
payload=payload or {},
|
||||
ts=T0 + timedelta(seconds=offset_s),
|
||||
sandbox_id="sbx-features",
|
||||
)
|
||||
|
||||
|
||||
def _paste_and_run_trace() -> list[TelemetryEvent]:
|
||||
"""One big file dump, a single passing test at the very end (no cycles)."""
|
||||
return [
|
||||
_event(0, "activity", {"state": "starting"}, 0.0),
|
||||
_event(1, "file_diff", {"path": "main.py", "added": 180, "removed": 0}, 5.0),
|
||||
_event(2, "command", {"cmd": "python -m pytest -q"}, 30.0),
|
||||
_event(3, "test_result", {"passed": True, "exit_code": 0}, 45.0),
|
||||
_event(4, "activity", {"state": "idle"}, 50.0),
|
||||
]
|
||||
|
||||
|
||||
def _iterative_trace() -> list[TelemetryEvent]:
|
||||
"""Many small edits; failed runs interleaved; eventual pass (>= 2 cycles)."""
|
||||
events: list[TelemetryEvent] = [
|
||||
_event(0, "activity", {"state": "starting"}, 0.0),
|
||||
_event(1, "file_diff", {"path": "a.py", "added": 12}, 10.0),
|
||||
_event(2, "command", {"cmd": "pytest -q"}, 20.0),
|
||||
_event(3, "test_result", {"passed": False, "exit_code": 1}, 25.0),
|
||||
_event(4, "file_diff", {"path": "a.py", "added": 4, "removed": 2}, 40.0),
|
||||
_event(5, "file_diff", {"path": "b.py", "added": 6}, 50.0),
|
||||
_event(6, "command", {"cmd": "pytest -q"}, 60.0),
|
||||
_event(7, "test_result", {"passed": False, "exit_code": 2}, 65.0),
|
||||
_event(8, "file_diff", {"path": "a.py", "added": 3, "removed": 1}, 80.0),
|
||||
_event(9, "command", {"cmd": "pytest -q tests/"}, 90.0),
|
||||
_event(10, "test_result", {"passed": True, "exit_code": 0}, 95.0),
|
||||
_event(11, "activity", {"state": "idle"}, 100.0),
|
||||
]
|
||||
return events
|
||||
|
||||
|
||||
def test_paste_and_run_vs_iterative_produce_observably_different_digests() -> None:
|
||||
paste = compute_digest(_paste_and_run_trace())
|
||||
iterative = compute_digest(_iterative_trace())
|
||||
assert paste.error_fix_cycles == 0
|
||||
assert iterative.error_fix_cycles == 2
|
||||
assert paste.test_fail_count == 0
|
||||
assert iterative.test_fail_count == 2
|
||||
assert paste.edit_count < iterative.edit_count
|
||||
assert iterative.first_test_pass_offset_s is not None
|
||||
assert paste.first_test_pass_offset_s is not None
|
||||
# Iterative debugs longer before the first pass.
|
||||
assert iterative.first_test_pass_offset_s > paste.first_test_pass_offset_s
|
||||
|
||||
|
||||
def test_digest_is_deterministic() -> None:
|
||||
trace = _iterative_trace()
|
||||
assert compute_digest(trace) == compute_digest(list(reversed(trace))) # seq sort normalizes
|
||||
|
||||
|
||||
def test_empty_trace_yields_valid_zeroed_digest() -> None:
|
||||
digest = compute_digest([])
|
||||
assert digest.event_count == 0
|
||||
assert digest.final_test_status == "none"
|
||||
assert digest.first_test_pass_offset_s is None
|
||||
assert digest.mean_fix_latency_s is None
|
||||
assert isinstance(digest, TraceDigest)
|
||||
|
||||
|
||||
def test_digest_json_is_bounded_under_4kb() -> None:
|
||||
big = [
|
||||
_event(i, "command", {"cmd": f"grep PATTERN-{i} file-{i}.py"}, i * 1.0)
|
||||
for i in range(200)
|
||||
]
|
||||
serialized = compute_digest(big).model_dump_json()
|
||||
assert len(serialized.encode()) < 4096, f"digest too large: {len(serialized)}B"
|
||||
|
||||
|
||||
def test_no_raw_command_string_leaks_into_digest() -> None:
|
||||
marker = "SECRET-COMMAND-MARKER-7f3a"
|
||||
trace = [
|
||||
_event(0, "command", {"cmd": f"echo {marker} && cat /etc/hostname"}, 0.0),
|
||||
_event(1, "file_diff", {"path": marker + ".py"}, 1.0),
|
||||
_event(2, "run_result", {"exit_code": 0, "stdout": marker}, 2.0),
|
||||
]
|
||||
digest_json = compute_digest(trace).model_dump_json()
|
||||
assert marker not in digest_json, "raw payload material leaked into digest"
|
||||
|
||||
|
||||
def test_idle_gaps_computed_over_threshold() -> None:
|
||||
trace = [
|
||||
_event(0, "activity", {"state": "starting"}, 0.0),
|
||||
_event(1, "activity", {"state": "idle"}, 400.0), # > 120s gap
|
||||
_event(2, "activity", {"state": "idle"}, 500.0), # 100s gap (below)
|
||||
_event(3, "activity", {"state": "stopped"}, 800.0), # > 120s gap
|
||||
]
|
||||
digest = compute_digest(trace)
|
||||
assert digest.idle_gap_count == 2
|
||||
assert digest.idle_gap_total_s == pytest.approx(400.0 + 300.0, rel=1e-6)
|
||||
|
||||
|
||||
def test_command_category_histogram() -> None:
|
||||
trace = [
|
||||
_event(0, "command", {"cmd": "npm run build"}, 0.0),
|
||||
_event(1, "command", {"cmd": "pytest -q"}, 1.0),
|
||||
_event(2, "command", {"cmd": "ls -la"}, 2.0),
|
||||
_event(3, "command", {"cmd": "rm -rf build/"}, 3.0),
|
||||
_event(4, "command", {"cmd": "curl localhost:8420/health"}, 4.0),
|
||||
_event(5, "command", {"cmd": "python mystery.py"}, 5.0),
|
||||
]
|
||||
digest = compute_digest(trace)
|
||||
assert digest.command_categories == {
|
||||
"build": 1,
|
||||
"debug": 1,
|
||||
"file": 1,
|
||||
"nav": 1,
|
||||
"other": 1,
|
||||
"test": 1,
|
||||
}
|
||||
|
||||
|
||||
def test_daemon_topology_mix_is_tolerated() -> None:
|
||||
"""P2-verify P1: live traces carry activity+file_diff only — no crash."""
|
||||
trace = [
|
||||
_event(0, "activity", {"state": "starting"}, 0.0),
|
||||
_event(1, "file_diff", {"path": "made-by-exec.txt", "added": 1}, 2.0),
|
||||
_event(2, "activity", {"state": "idle"}, 3.0),
|
||||
]
|
||||
digest = compute_digest(trace)
|
||||
assert digest.final_test_status == "none"
|
||||
assert digest.edit_count == 1
|
||||
assert digest.test_pass_count == 0
|
||||
assert digest.kind_histogram.get("file_diff") == 1
|
||||
|
||||
|
||||
def test_run_result_exit_codes_fall_back_for_test_status() -> None:
|
||||
"""No test_result events: run_result exit codes decide pass/fail."""
|
||||
trace = [
|
||||
_event(0, "file_diff", {"path": "x.py"}, 0.0),
|
||||
_event(1, "run_result", {"exit_code": 1}, 10.0),
|
||||
_event(2, "file_diff", {"path": "x.py"}, 20.0),
|
||||
_event(3, "run_result", {"exit_code": 0}, 30.0),
|
||||
]
|
||||
digest = compute_digest(trace)
|
||||
assert digest.test_fail_count == 1
|
||||
assert digest.test_pass_count == 1
|
||||
assert digest.final_test_status == "pass"
|
||||
assert digest.error_fix_cycles == 1
|
||||
@@ -0,0 +1,268 @@
|
||||
"""SQLiteGradeStore tests (REQ-3-004, D-027).
|
||||
|
||||
Each test gets its own tmp-path SQLite file — no shared disk state. Covers:
|
||||
- save / get / list_for_learner roundtrip (all fields survive,
|
||||
including nested JSON dicts and the tz-aware created_at contract)
|
||||
- upsert-on-regrade: a second save with the same (learner_id, task_id)
|
||||
REPLACES the row wholesale — scores, verdict, created_at, digest,
|
||||
model and variant_seed all reflect the latest save (documented
|
||||
contract; deliberately opposite of TraceStore.append's dedup)
|
||||
- unknown (learner, task) pair -> None; unknown learner -> empty list
|
||||
- scoping: grades for other learners/tasks are never returned
|
||||
- variant_seed stays None until P4 (D-029) and survives a roundtrip
|
||||
- rows are detached: usable after the store is closed
|
||||
- WAL + synchronous=NORMAL pragmas actually applied to the DB file
|
||||
- concurrent writer + reader against the same DB file (a-3 smoke test)
|
||||
"""
|
||||
|
||||
import concurrent.futures
|
||||
import threading
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
import sqlalchemy as sa
|
||||
|
||||
from ai_service.grading.store import GradeRecord, SQLiteGradeStore
|
||||
|
||||
_BASE_TS = datetime(2026, 9, 12, 12, 0, 0, tzinfo=UTC)
|
||||
|
||||
|
||||
def make_grade(
|
||||
task_id: str = "task-1",
|
||||
learner_id: str = "learner-1",
|
||||
variant_seed: str | None = None,
|
||||
digest: dict[str, Any] | None = None,
|
||||
scores: dict[str, Any] | None = None,
|
||||
verdict: str = "STRONG",
|
||||
model: str = "gemma4:31b",
|
||||
created_at: datetime | None = None,
|
||||
) -> GradeRecord:
|
||||
"""Canonical kwargs builder — tests override only what they assert on."""
|
||||
return GradeRecord(
|
||||
learner_id=learner_id,
|
||||
task_id=task_id,
|
||||
variant_seed=variant_seed,
|
||||
digest=digest
|
||||
if digest is not None
|
||||
else {"error_fix_cycles": 3, "command_categories": {"build": 2}},
|
||||
scores=scores
|
||||
if scores is not None
|
||||
else {
|
||||
"criteria": {"process_quality": 3, "correctness": 4},
|
||||
"strengths": ["iterative debugging"],
|
||||
"gaps": ["no final test pass"],
|
||||
},
|
||||
verdict=verdict,
|
||||
model=model,
|
||||
created_at=created_at if created_at is not None else _BASE_TS,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def store(tmp_path: Path) -> SQLiteGradeStore:
|
||||
s = SQLiteGradeStore(db_path=tmp_path / "grades.db")
|
||||
yield s
|
||||
s.close()
|
||||
|
||||
|
||||
def test_save_and_get_roundtrip(store: SQLiteGradeStore) -> None:
|
||||
grade = make_grade()
|
||||
store.save(grade)
|
||||
|
||||
fetched = store.get("learner-1", "task-1")
|
||||
assert fetched is not None
|
||||
assert fetched.learner_id == "learner-1"
|
||||
assert fetched.task_id == "task-1"
|
||||
assert fetched.variant_seed is None # null until P4 (D-029)
|
||||
assert fetched.digest == grade.digest
|
||||
assert fetched.scores == grade.scores
|
||||
assert fetched.verdict == "STRONG"
|
||||
assert fetched.model == "gemma4:31b"
|
||||
assert fetched.created_at == _BASE_TS
|
||||
assert fetched.created_at.tzinfo is UTC # tz-normalized on read
|
||||
|
||||
|
||||
def test_get_unknown_pair_returns_none(store: SQLiteGradeStore) -> None:
|
||||
store.save(make_grade())
|
||||
|
||||
assert store.get("learner-1", "task-missing") is None
|
||||
assert store.get("learner-missing", "task-1") is None
|
||||
assert store.get("nobody", "nothing") is None
|
||||
|
||||
|
||||
def test_list_for_learner_roundtrip(store: SQLiteGradeStore) -> None:
|
||||
# Created out of insertion order; list must come back chronological.
|
||||
store.save(make_grade(task_id="task-c", created_at=_BASE_TS + timedelta(hours=2)))
|
||||
store.save(make_grade(task_id="task-a", created_at=_BASE_TS))
|
||||
store.save(make_grade(task_id="task-b", created_at=_BASE_TS + timedelta(hours=1)))
|
||||
|
||||
grades = store.list_for_learner("learner-1")
|
||||
assert [g.task_id for g in grades] == ["task-a", "task-b", "task-c"]
|
||||
assert all(g.learner_id == "learner-1" for g in grades)
|
||||
hours = (timedelta(hours=0), timedelta(hours=1), timedelta(hours=2))
|
||||
assert all(
|
||||
g.created_at == _BASE_TS + offset
|
||||
for g, offset in zip(grades, hours, strict=True)
|
||||
)
|
||||
assert all(g.created_at.tzinfo is UTC for g in grades)
|
||||
|
||||
|
||||
def test_list_for_learner_unknown_learner_returns_empty_list(
|
||||
store: SQLiteGradeStore,
|
||||
) -> None:
|
||||
assert store.list_for_learner("nobody") == []
|
||||
|
||||
|
||||
def test_lists_are_scoped_to_the_learner(store: SQLiteGradeStore) -> None:
|
||||
store.save(make_grade(learner_id="learner-1", task_id="task-1"))
|
||||
store.save(make_grade(learner_id="learner-2", task_id="task-1"))
|
||||
|
||||
assert [g.task_id for g in store.list_for_learner("learner-1")] == ["task-1"]
|
||||
assert [g.learner_id for g in store.list_for_learner("learner-2")] == ["learner-2"]
|
||||
# (learner-2, task-1) is a distinct row: same task_id, different grade.
|
||||
grades_2 = store.list_for_learner("learner-2")
|
||||
assert len(grades_2) == 1
|
||||
assert grades_2[0].learner_id == "learner-2"
|
||||
|
||||
|
||||
def test_regrade_overwrites_the_stored_row(store: SQLiteGradeStore) -> None:
|
||||
"""THE contract of this store (upsert on the PK pair, latest wins).
|
||||
|
||||
The grading engine re-grades a task as its rubric or input evolves;
|
||||
the second save replaces scores, verdict, created_at, digest, model
|
||||
and variant_seed wholesale — exactly one row survives per pair.
|
||||
"""
|
||||
first = make_grade(
|
||||
verdict="DEVELOPING",
|
||||
model="gemma4:31b",
|
||||
scores={"criteria": {"process_quality": 1}},
|
||||
digest={"error_fix_cycles": 0},
|
||||
created_at=_BASE_TS,
|
||||
)
|
||||
store.save(first)
|
||||
|
||||
second = make_grade(
|
||||
verdict="EXEMPLARY",
|
||||
model="gemma4:31b-p2",
|
||||
variant_seed="seed-77",
|
||||
scores={"criteria": {"process_quality": 4}},
|
||||
digest={"error_fix_cycles": 6},
|
||||
created_at=_BASE_TS + timedelta(hours=1),
|
||||
)
|
||||
store.save(second)
|
||||
|
||||
fetched = store.get("learner-1", "task-1")
|
||||
assert fetched is not None
|
||||
# The regrade replaced every field of the first save.
|
||||
assert fetched.verdict == "EXEMPLARY"
|
||||
assert fetched.model == "gemma4:31b-p2"
|
||||
assert fetched.variant_seed == "seed-77"
|
||||
assert fetched.scores == {"criteria": {"process_quality": 4}}
|
||||
assert fetched.digest == {"error_fix_cycles": 6}
|
||||
assert fetched.created_at == _BASE_TS + timedelta(hours=1)
|
||||
|
||||
# Latest-wins also holds in list_for_learner — one row, not two.
|
||||
grades = store.list_for_learner("learner-1")
|
||||
assert len(grades) == 1
|
||||
assert grades[0].verdict == "EXEMPLARY"
|
||||
|
||||
|
||||
def test_regrade_preserves_other_pairs(store: SQLiteGradeStore) -> None:
|
||||
# An upsert on (learner-1, task-1) must not touch (learner-1, task-2).
|
||||
store.save(make_grade(task_id="task-1", verdict="STRONG"))
|
||||
store.save(make_grade(task_id="task-2", verdict="DEVELOPING"))
|
||||
store.save(make_grade(task_id="task-1", verdict="EXEMPLARY"))
|
||||
|
||||
other = store.get("learner-1", "task-2")
|
||||
assert other is not None
|
||||
assert other.verdict == "DEVELOPING" # untouched by the task-1 regrade
|
||||
assert len(store.list_for_learner("learner-1")) == 2
|
||||
|
||||
|
||||
def test_empty_scores_dict_roundtrips(store: SQLiteGradeStore) -> None:
|
||||
# Legal shape: an UNGRADABLE_TRACE_INCOMPLETE record carries a verdict
|
||||
# but no scores (and here, no digest either).
|
||||
store.save(make_grade(verdict="UNGRADABLE_TRACE_INCOMPLETE", scores={}, digest={}))
|
||||
|
||||
fetched = store.get("learner-1", "task-1")
|
||||
assert fetched is not None
|
||||
assert fetched.scores == {}
|
||||
assert fetched.digest == {}
|
||||
assert fetched.verdict == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
|
||||
|
||||
def test_rows_are_detached_after_save(store: SQLiteGradeStore, tmp_path: Path) -> None:
|
||||
# The engine hands GradeRecords across layers; rows must survive the
|
||||
# store that produced them being closed (no open-session ORM magic).
|
||||
store.save(make_grade())
|
||||
fetched = store.get("learner-1", "task-1")
|
||||
store.close()
|
||||
|
||||
assert fetched is not None
|
||||
assert fetched.verdict == "STRONG"
|
||||
assert fetched.scores["criteria"]["process_quality"] == 3
|
||||
|
||||
# A fresh store on the same file sees the same row (durability).
|
||||
reopened = SQLiteGradeStore(db_path=tmp_path / "grades.db")
|
||||
try:
|
||||
again = reopened.get("learner-1", "task-1")
|
||||
assert again is not None
|
||||
assert again.verdict == "STRONG"
|
||||
finally:
|
||||
reopened.close()
|
||||
|
||||
|
||||
def test_pragmas_are_applied(store: SQLiteGradeStore) -> None:
|
||||
# Pragmas are per-connection; query through the store's engine so the
|
||||
# connect hook (not a default sqlite3 connection) is what we inspect.
|
||||
with store._engine.connect() as conn:
|
||||
(journal_mode,) = conn.execute(sa.text("PRAGMA journal_mode")).one()
|
||||
(synchronous,) = conn.execute(sa.text("PRAGMA synchronous")).one()
|
||||
assert journal_mode == "wal"
|
||||
# synchronous=NORMAL is 1 in SQLite's pragma numbering.
|
||||
assert synchronous == 1
|
||||
|
||||
|
||||
def test_concurrent_writer_and_reader_no_database_is_locked(tmp_path: Path) -> None:
|
||||
"""One thread saves while another reads in a tight loop (a-3).
|
||||
|
||||
Without WAL + busy_timeout this pattern reliably produces
|
||||
`OperationalError: database is locked` on SQLite. The assertion is
|
||||
that every reader call completes and the latest write lands intact.
|
||||
"""
|
||||
db_path = tmp_path / "grades.db"
|
||||
n_pairs = 60 # distinct tasks -> distinct PK pairs, one regrade each
|
||||
stop_writing = threading.Event()
|
||||
|
||||
writer = SQLiteGradeStore(db_path=db_path)
|
||||
reader = SQLiteGradeStore(db_path=db_path)
|
||||
try:
|
||||
|
||||
def write_grades() -> None:
|
||||
for seq in range(n_pairs):
|
||||
writer.save(
|
||||
make_grade(
|
||||
task_id=f"task-{seq}",
|
||||
created_at=_BASE_TS + timedelta(seconds=seq),
|
||||
)
|
||||
)
|
||||
stop_writing.set()
|
||||
|
||||
def read_grades() -> None:
|
||||
while not stop_writing.is_set():
|
||||
reader.list_for_learner("learner-1")
|
||||
# Final read after the writer is done.
|
||||
assert len(reader.list_for_learner("learner-1")) == n_pairs
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool:
|
||||
futures = [pool.submit(write_grades), pool.submit(read_grades)]
|
||||
for future in futures:
|
||||
future.result(timeout=30)
|
||||
|
||||
# Every save was an upsert on its own pair: nothing lost, none doubled.
|
||||
assert len(writer.list_for_learner("learner-1")) == n_pairs
|
||||
finally:
|
||||
reader.close()
|
||||
writer.close()
|
||||
@@ -508,10 +508,13 @@ class TestReconnectFlush:
|
||||
assert set(buffered_seqs).isdisjoint(e["seq"] for e in fake_server.events)
|
||||
|
||||
held_link.resume() # outage ends → supervisor reconnects, flushes spool
|
||||
# 20s deadline: under full-suite load the supervisor thread can be
|
||||
# starved past its normal sub-second reconnect; 8s was flaky.
|
||||
assert _wait_until(
|
||||
lambda: all(
|
||||
seq in {e["seq"] for e in fake_server.events} for seq in buffered_seqs
|
||||
)
|
||||
),
|
||||
timeout_s=20.0,
|
||||
), "spooled events never flushed after reconnect"
|
||||
test_agent.stop()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user