# Praxis — v0.3 Execution Plan (Mastery Scoring + Competency Rubrics + VC Issuance) > **Milestone:** v0.3 (Mastery scoring + competency rubrics + verifiable credentials) > **Phases:** 1 execution phase (P1: mastery core + IRT + scenarios + paths + VC issuance) + final phase (P2: review + ship) > **Ship:** v0.1.3 (Phase 0) → v0.1.4 (P1) → v0.1.5 (P2 = v0.3 milestone release) > **Status:** plan (grill-amended — operator tier deferred to v0.4 per GRILL-v0.3.md Axis 2 + Axis 8) > **Autonomy:** full > **Parallelization:** enabled, max 5 concurrent agents > **Personas active:** lead-developer, backend-engineer, data-engineer, security-engineer (frontend-engineer + devops-engineer DEACTIVATED — no UI, no new deploy scripts in v0.3) > **Date:** 2026-08-03 --- ## Grill Amendments (binding — per GRILL-v0.3.md) The grill (GO-WITH-CONDITIONS, 4 MUST) restructured this plan: 1. **Axis 2 (MUST) — Split the milestone.** The operator tier (REQ-DASH-01, REQ-AUTH-01, REQ-MT-01/02 + associated NFRs) is **deferred to v0.4**. v0.3 is now a clean learner-facing mastery milestone. This restores the original ROADMAP intent (dashboard was v0.8) and avoids the hybrid SQLite+Postgres topology in v0.3. 2. **Axis 8 (MUST) — VC issuance moves to P1.** VC issuance is a learner-facing consequence of mastery (D-048), not an operator feature. Issuer keys are SQLite-backed in v0.3 (Postgres takes over in v0.4 when the operator tier arrives). 3. **Axis 3 (MUST) — VC interop + key-rotation tests added.** TASK-12-07 (external W3C verifier interop) + TASK-12-08 (key-rotation operational drill). 4. **Axis 4 (MUST) — Three technical-risk fixes.** (a) VC labeled `formative` in payload + verification + REQ-MAST-03. (b) R-AUTH-01 deferred to v0.4 with the operator surface (no auth in v0.3 → no cookie issue). (c) Evidence-extraction fallback changed from silent-fail-to-zero to `scoring_inconclusive` with learner-visible retry signal. FIX conditions (non-blocking, tracked in VERIFY): re-task SLICE-12/13 (now moot for v0.3 — operator tier deferred), wire VC trigger (resolved — VC now in P1), Postgres-failure semantics (deferred to v0.4), real-LLM smoke test (added to P1 SLICE-08), k-anonymity differencing-attack test (deferred to v0.4), reconciliation drift-correction test (deferred to v0.4), de-escalation weight clarification (static in v0.3 — dynamic re-weighting is a future feature). --- ## Phase Split Rationale (post-grill) v0.3 is now a **single execution phase** (P1) + final review/ship (P2): - **P1 (Mastery Core + VC Issuance):** rubric engine, IRT, scenario library (≥6 CS scenarios), path engine (6-week), mastery score + gate logic, VC issuer (W3C VC 2.0, Ed25519, SQLite-backed issuer keys, public verification endpoint). All learner-facing. Shippable as `v0.1.4`. - **P2 (Final):** review + audit + milestone ship (`v0.1.5` = v0.3 milestone release). The operator tier (cohort dashboard, auth, Postgres) is **v0.4** — a separate milestone with its own phase 0. This keeps v0.3 honest: one milestone, one shippable learner-facing deliverable, no hybrid storage, no operator auth surface. --- ## Deferred to v0.4 (operator tier — per grill Axis 2) The following REQ-IDs are **deferred to v0.4** and removed from v0.3 scope: - REQ-DASH-01 (cohort dashboard) — was v0.8 on original ROADMAP; v0.4 is still ahead of that but follows the grill's "split the milestone" verdict - REQ-AUTH-01 (operator auth) — no operator surface in v0.3 → no auth needed - REQ-MT-01, REQ-MT-02 (operator Postgres, cohort aggregation) — no operator tier in v0.3 - REQ-NFR-DASH-01, REQ-NFR-DASH-02, REQ-NFR-AUTH-01, REQ-NFR-MT-01 — associated NFRs v0.3 REQ-IDs (post-grill): **13** (REQ-MAST-01/02/03, REQ-SCEN-02/03/04, REQ-PATH-02 + 6 NFRs: REQ-NFR-MAST-01/02, REQ-NFR-VC-01/02, REQ-NFR-IRT-01). REQ-MAST-04 is a principle (accepted). --- # Phase 1 — Mastery Core (learner-facing mastery layer) **Branch:** `phase/01-mastery-core` → merged to `milestone/v0.3-mastery-scoring` **Ship:** `v0.1.4` (patch release, feature milestone type) **REQ-IDs covered:** REQ-MAST-01, REQ-MAST-02, REQ-SCEN-02, REQ-SCEN-03, REQ-SCEN-04, REQ-PATH-02, REQ-NFR-MAST-01, REQ-NFR-MAST-02, REQ-NFR-IRT-01 **Slices:** 8 vertical slices in 4 waves **Total tasks:** 38 | Wave | Slices | Parallel slots | Description | |------|--------|----------------|-------------| | 1 | SLICE-01, SLICE-02 | 2 | Rubric schema + scenario library schema (parallel — disjoint file territories) | | 2 | SLICE-03, SLICE-04, SLICE-05 | 3 | Rubric scoring engine + IRT engine + path engine (parallel — all depend on W1 schemas, disjoint modules) | | 3 | SLICE-06, SLICE-07 | 2 | Scenario library content (≥6 CS scenarios) + mastery score + gate logic (parallel — SLICE-06 authors scenarios, SLICE-07 wires scoring into session_recorder) | | 4 | SLICE-08 | 1 | Integration tests + mastery-gate audit log + real-LLM smoke test (depends on all prior) | | 5 | SLICE-09 | 1 | VC issuer + verification endpoint + interop/rotation tests (depends on SLICE-07 gate-open trigger) | ### Wave dependency graph ``` Wave 1 ──────────────────────────────────────── SLICE-01 (rubric YAML schema + loader) SLICE-02 (scenario library schema + index + loader) │ ▼ Wave 2 ──────────────────────────────────────── SLICE-03 (rubric scoring engine: evidence extractor + rule scorer) ← depends on SLICE-01 SLICE-04 (IRT engine + theta persistence) ← depends on SLICE-02 (scenario difficulty) SLICE-05 (path engine: 6-week structure + progression) ← depends on SLICE-02 (scenario library) │ ▼ Wave 3 ──────────────────────────────────────── SLICE-06 (≥6 expert CS scenarios + index.yaml + rubric mapping) ← depends on SLICE-01, SLICE-02 SLICE-07 (mastery score + gate logic + session_recorder hooks) ← depends on SLICE-03, SLICE-04, SLICE-05 │ ▼ Wave 4 ──────────────────────────────────────── SLICE-08 (integration tests + mastery-gate audit log in SQLite + real-LLM smoke) ← depends on all prior │ ▼ Wave 5 ──────────────────────────────────────── SLICE-09 (VC issuer: Ed25519 + JCS + Status List + verification + interop + rotation) ← depends on SLICE-07 (gate-open trigger) ``` ### Persona load distribution (P1) | Persona | Tasks | Primary territory | |---------|-------|-------------------| | backend-engineer | 20 | `server/mastery/**`, `server/scenarios/library.py`, `server/paths/**`, `server/session_recorder.py` extension | | security-engineer | 8 | `server/vc/**` (Ed25519 issuer, JCS, Status List, verification endpoint, interop + rotation tests) | | data-engineer | 6 | `db/migrations/0003_mastery.sql` (learner_ability, mastery_progress, issuer_keys, issued_credentials, mastery_gate_events tables), `db/store.py` v0.3 additions | | lead-developer | 6 | `pyproject.toml` deps, integration test orchestration, cross-persona coordination | | frontend-engineer | 0 | DEACTIVATED (no UI in v0.3 — dashboard is v0.4) | | devops-engineer | 0 | DEACTIVATED (no new deploy scripts) | **Total P1 tasks: 40** (was 38 + 8 VC - 6 rebalanced; +2 grill interop/rotation tests) --- ## SLICE-01: Rubric Schema + Loader (W1) - **Goal:** Define the competency rubric YAML format + Pydantic model + loader so scenarios can reference rubric criteria. - **REQ-IDs covered:** REQ-MAST-01 (partial — schema only), REQ-NFR-MAST-01 (determinism foundation) - **Wave:** 1 - **Dependencies:** none - **Persona:** data-engineer (schema), backend-engineer (loader) ### Tasks #### TASK-01-01 — Rubric YAML schema definition - **Persona:** data-engineer - **File:** `rubrics/customer_service.yaml` (new — refund/complaint archetype per RESEARCH §6.2) - **Content:** 4 criteria (empathy 0.35, resolution 0.30, de-escalation 0.20, professionalism 0.15), 5-level anchors each (level 1=fail … 5=mastery/entrustable, per RESEARCH §2), per-archetype weights (D-039 amendment). Professionalism = conjunctive floor ≥2. #### TASK-01-02 — Rubric Pydantic model - **Persona:** backend-engineer - **File:** `server/mastery/rubric_schema.py` (new) - **Content:** `Rubric`, `RubricCriterion`, `RubricLevel` models. Fields: id, skill, criteria[{id, name, weight, levels[{level, anchor, signals[]}]}]. Validate weights sum to 1.0. Validate 5 levels per criterion. #### TASK-01-03 — Rubric loader - **Persona:** backend-engineer - **File:** `server/mastery/rubric_loader.py` (new) - **Content:** `load_rubric(skill: str) -> Rubric` — loads `rubrics/.yaml`, parses via Pydantic. Caches in-memory. Validates against schema. #### TASK-01-04 — Rubric unit tests - **Persona:** backend-engineer - **File:** `tests/test_rubric_schema.py` (new) - **Content:** load valid rubric, reject invalid weights, reject missing levels, criterion lookup by id, weight sum validation. --- ## SLICE-02: Scenario Library Schema + Index + Loader (W1) - **Goal:** Extend the v0.1 scenario schema (D-018) with rubric mapping + library index manifest + loader for multi-scenario selection. - **REQ-IDs covered:** REQ-SCEN-03 (partial — schema), REQ-SCEN-04 (partial — format extension) - **Wave:** 1 - **Dependencies:** none (parallel with SLICE-01 — disjoint files) - **Persona:** backend-engineer ### Tasks #### TASK-02-01 — Extend Scenario schema with rubric mapping + IRT fields - **Persona:** backend-engineer - **File:** `server/scenarios/schema.py` (extend existing) - **Content:** Add `rubric_criteria: list[{criterion_id, weight, evidence_required}]` field to `Scenario`. Add `irt_target_p: float = 0.7` field (D-035 practice default). Add `version: str` (semver, D-036). Add `generated_from: str | None` (AI-variation backref, D-036). Add `intent_hash: str | None` (structural drift detection). Keep backward compat with v0.1 scenario YAML. #### TASK-02-02 — Scenario index manifest - **Persona:** backend-engineer - **File:** `scenarios/index.yaml` (new — slim manifest per RESEARCH §D) - **Content:** list of {id, path, title, difficulty, failure_mode, rubric_criteria, version, author, generated_from}. ~50 lines/scenario metadata. Updated when scenarios are added. #### TASK-02-03 — Scenario library loader - **Persona:** backend-engineer - **File:** `server/scenarios/library.py` (new) - **Content:** `ScenarioLibrary` class — loads `scenarios/index.yaml`, loads individual scenario YAMLs on demand, validates against schema. `list_by_path(path)`, `list_by_difficulty(range)`, `get(scenario_id)`, `select_for_theta(theta, path)` (IRT-aware selection targeting ~50% or ~70% per `irt_target_p`). Enforces `MIN_COVERAGE = 2` scenarios per rubric criterion (CI check, RESEARCH §D). #### TASK-02-04 — Library unit tests - **Persona:** backend-engineer - **File:** `tests/test_scenario_library.py` (new) - **Content:** load index, list by path, select_for_theta, MIN_COVERAGE validation, reject invalid semver, AI-variation backref validation. --- ## SLICE-03: Rubric Scoring Engine (W2) - **Goal:** Implement the deterministic rubric scoring flow: LLM-extracts-evidence, rules-score-evidence (D-038, REQ-NFR-MAST-01). - **REQ-IDs covered:** REQ-MAST-01 (scoring logic), REQ-NFR-MAST-01 (determinism) - **Wave:** 2 - **Dependencies:** SLICE-01 (rubric schema) - **Persona:** backend-engineer ### Tasks #### TASK-03-01 — Evidence extractor (LLM, off-voice-path) - **Persona:** backend-engineer - **File:** `server/mastery/evidence_extractor.py` (new) - **Content:** `async extract_evidence(turns, rubric_criteria) -> list[Evidence]`. Calls deepseek-v4-flash:cloud, temp=0, JSON-schema-validated output: `[{criterion_id, quote, signals: [...]}]`. **Critical: fuzzy-match quote against transcript (rapidfuzz or difflib) → reject + re-extract on mismatch (R-MAST-02).** Max 2 re-extraction attempts; **on final failure, mark scenario as `scoring_inconclusive` — do NOT count toward gate, do NOT penalize learner, surface 'technical issue, please retry' in the debrief (grill Axis 4 MUST #3 — silent fail-to-zero is unacceptable).** Log the failure for operator review. #### TASK-03-02 — Rule-based scorer (deterministic) - **Persona:** backend-engineer - **File:** `server/mastery/rubric_scorer.py` (new) - **Content:** `score(evidence, rubric) -> list[CriterionScore]`. Maps signals → 1-5 level per criterion via rubric YAML level anchors (each level has a `signals[]` list — match evidence signals to level signals). Deterministic — no LLM. Output: `[{criterion_id, level, weight, evidence_quote}]`. #### TASK-03-03 — Mastery Score computation (deterministic) - **Persona:** backend-engineer - **File:** `server/mastery/mastery_score.py` (new) - **Content:** `compute_scenario_score(criterion_scores, rubric) -> ScenarioScore` (weighted mean + conjunctive floor: every criterion ≥2, scenario mean ≥3.0 to pass). `compute_path_score(passing_scenario_scores) -> PathScore` (mean over passing scenarios only). `check_gate(path_score, distinct_passed_count) -> bool` (≥3 distinct passed AND ≥3.5 — D-032). #### TASK-03-04 — Scoring unit tests - **Persona:** backend-engineer - **File:** `tests/test_rubric_scoring.py` (new) - **Content:** evidence extraction with mocked LLM, quote fuzzy-match rejection, rule-based scoring determinism (same input → same output), conjunctive floor enforcement, gate logic. #### TASK-03-05 — Evidence extractor integration test (mocked LLM) - **Persona:** backend-engineer - **File:** `tests/test_evidence_extractor_integration.py` (new) - **Content:** end-to-end extraction → scoring with a mocked LLM returning canned evidence. Verify JSON schema validation, quote matching, deterministic scoring. --- ## SLICE-04: IRT Engine + Theta Persistence (W2) - **Goal:** Implement 1PL/Rasch IRT with Bayesian theta update, persisted to SQLite (D-046, REQ-NFR-IRT-01). - **REQ-IDs covered:** REQ-SCEN-02, REQ-NFR-IRT-01 - **Wave:** 2 - **Dependencies:** SLICE-02 (scenario difficulty field) - **Persona:** backend-engineer (engine), data-engineer (SQLite table) ### Tasks #### TASK-04-01 — IRT engine - **Persona:** backend-engineer - **File:** `server/mastery/irt.py` (new) - **Content:** `class IRTEngine`: `P_success(theta, b) -> float` (logistic(θ−b)). `update_theta(theta, sigma_sq, outcome, b) -> (new_theta, new_sigma_sq)` (Gaussian-approximation Bayesian: θ ← θ + (outcome − P) × σ²/(σ² + 1); σ² shrinks per observation). `select_scenario(theta, library, path, target_p) -> Scenario` (picks scenario with b closest to θ − logit(target_p)). Cold-start: θ=0, σ²=1; fall back to `scenario.difficulty` until ≥5 observations (R-IRT-01). #### TASK-04-02 — Theta persistence (SQLite) - **Persona:** data-engineer - **File:** `db/migrations/0003_mastery.sql` (new — adds learner_ability + mastery_progress tables), `db/store.py` (extend) - **Content:** `learner_ability` table (learner_id, path, theta REAL, sigma_sq REAL, observations INTEGER, updated_at). `mastery_progress` table (learner_id, path, current_week INTEGER, scenarios_passed_json TEXT, mastery_score REAL, gate_open bool, updated_at). `PraxisStore.get_ability()`, `set_ability()`, `get_progress()`, `set_progress()` async methods. #### TASK-04-03 — IRT unit tests - **Persona:** backend-engineer - **File:** `tests/test_irt.py` (new) - **Content:** P_success correctness, theta update convergence, cold-start fallback, select_scenario targeting, sigma_sq shrinkage. #### TASK-04-04 — Theta persistence integration test - **Persona:** data-engineer - **File:** `tests/test_learner_ability_db.py` (new) - **Content:** get/set ability round-trip, get/set progress round-trip, migration idempotency, concurrent writes (aiosqlite). --- ## SLICE-05: Path Engine (W2) - **Goal:** Implement the 6-week path structure with mastery gates (D-037, REQ-PATH-02). - **REQ-IDs covered:** REQ-PATH-02 - **Wave:** 2 - **Dependencies:** SLICE-02 (scenario library — paths reference scenarios) - **Persona:** backend-engineer ### Tasks #### TASK-05-01 — Path YAML schema + Pydantic model - **Persona:** backend-engineer - **File:** `server/paths/schema.py` (new) - **Content:** `Path` model: slug, name, skill, weeks[{week, title, scenario_ids[], gate: {required_scenarios: int, required_score: float}}]. Validate 6 weeks. Validate scenario_ids exist in library. #### TASK-05-02 — Customer Service path YAML - **Persona:** backend-engineer - **File:** `paths/customer_service.yaml` (new) - **Content:** 6 weeks per PRD §6.4. Week 1: basics (refund scenario). Week 2: escalation. Week 3: policy exceptions. Week 4: multi-issue. Week 5: recovery. Week 6: mastery demonstration. Each week references ≥1 scenario from the library (SLICE-06). Gate: ≥3 distinct scenarios passed, score ≥3.5 (D-032). #### TASK-05-03 — Path engine (progression logic) - **Persona:** backend-engineer - **File:** `server/paths/engine.py` (new) - **Content:** `PathEngine`: `load_path(slug) -> Path`. `current_week(progress) -> int`. `check_gate(progress, week) -> bool` (delegates to mastery_score.check_gate). `advance_week(progress) -> progress` (D-048). `is_path_complete(progress) -> bool` (week 6 gate open). #### TASK-05-04 — Path unit tests - **Persona:** backend-engineer - **File:** `tests/test_path_engine.py` (new) - **Content:** load path, validate 6 weeks, gate check, week advancement, path completion. --- ## SLICE-06: Scenario Library Content (W3) - **Goal:** Author ≥6 expert Customer Service scenarios filling the 6-week path (D-047, REQ-SCEN-03). - **REQ-IDs covered:** REQ-SCEN-03, REQ-SCEN-04 (expert-authored; AI variations in P2 or later) - **Wave:** 3 - **Dependencies:** SLICE-01 (rubric), SLICE-02 (library schema) - **Persona:** lead-developer (content authoring — domain expertise), backend-engineer (validation) ### Tasks #### TASK-06-01 — Author 6 CS scenarios - **Persona:** lead-developer - **Files:** `scenarios/customer_service/cs_refund_ca_v01.yaml` (exists — extend with rubric mapping), `scenarios/customer_service/cs_escalation_ca_v02.yaml` (new), `scenarios/customer_service/cs_policy_exception_ca_v03.yaml` (new), `scenarios/customer_service/cs_multi_issue_ca_v04.yaml` (new), `scenarios/customer_service/cs_recovery_ca_v05.yaml` (new), `scenarios/customer_service/cs_mastery_demonstration_ca_v06.yaml` (new) - **Content:** Each scenario: extends v0.1 schema with `rubric_criteria` (mapped to the 4 CS criteria), `irt_target_p` (0.7 for practice weeks, 0.5 for mastery-demonstration week 6), `version: 1.0.0`, `author: expert`. Difficulty 1-5 across weeks. Failure modes vary (escalates_unresolved, policy_rigid, multi_issue_drop, recovery_missed). #### TASK-06-02 — Update index.yaml manifest - **Persona:** lead-developer - **File:** `scenarios/index.yaml` (update) - **Content:** All 6 scenarios listed with metadata. `MIN_COVERAGE = 2` per criterion verified (each of empathy/resolution/de-escalation/professionalism exercised by ≥2 scenarios). #### TASK-06-03 — Scenario validation tests - **Persona:** backend-engineer - **File:** `tests/test_scenario_library_content.py` (new) - **Content:** all 6 scenarios load via schema, rubric_criteria reference valid criterion IDs, MIN_COVERAGE per criterion, semver valid, index.yaml in sync with files. --- ## SLICE-07: Mastery Score + Gate Logic + Session Recorder Hooks (W3) - **Goal:** Wire the rubric scoring + IRT + path progression into the session end flow (server/session_recorder.py). - **REQ-IDs covered:** REQ-MAST-02, REQ-NFR-MAST-02 (auditability — SQLite log) - **Wave:** 3 - **Dependencies:** SLICE-03 (scoring), SLICE-04 (IRT), SLICE-05 (path) - **Persona:** backend-engineer ### Tasks #### TASK-07-01 — Extend session_recorder.py with mastery hooks - **Persona:** backend-engineer - **File:** `server/session_recorder.py` (extend existing) - **Content:** After existing `end()` logic: (1) call `evidence_extractor.extract_evidence(turns, scenario.rubric_criteria)`, (2) `rubric_scorer.score(evidence, rubric)`, (3) `mastery_score.compute_scenario_score(...)`, (4) `irt.update_theta(...)`, (5) `path_engine.check_gate + advance_week`, (6) record `mastery_gate_event` in SQLite `mastery_gate_events` table (REQ-NFR-MAST-02 audit), (7) **if week-final gate open → call `vc_issuer.issue_credential(...)` (SLICE-09) — VC issuance is wired here, not in a later phase (grill Axis 8 MUST)**. All off the voice path (async, after session end). If evidence extraction returns `scoring_inconclusive`, skip steps 2-7 and surface retry in debrief. #### TASK-07-02 — Mastery gate event SQLite table - **Persona:** data-engineer - **File:** `db/migrations/0003_mastery.sql` (extend), `db/store.py` (extend) - **Content:** `mastery_gate_events` table (id, learner_id, path, week, scenarios_passed_json, rubric_scores_json, mastery_score, gate_opened_at). `PraxisStore.record_gate_event()` async method. #### TASK-07-03 — Mastery integration test (end-to-end scoring flow) - **Persona:** backend-engineer - **File:** `tests/test_mastery_integration.py` (new) - **Content:** simulate a session with turns → run mastery flow → verify scenario score, theta update, progress advancement, gate event recorded. Mocked LLM for evidence extraction. Verify determinism (same input → same scores). #### TASK-07-04 — IRT selection integration (next-scenario recommendation) - **Persona:** backend-engineer - **File:** `server/scenarios/library.py` (extend), `tests/test_irt_selection_integration.py` (new) - **Content:** `library.select_for_theta(theta, path)` picks the next scenario. Integration test: given a theta and a path, verify the selected scenario targets the right P. --- ## SLICE-08: Integration Tests + Mastery-Gate Audit Log (W4) - **Goal:** End-to-end P1 integration tests + verify the mastery-gate audit log is complete and queryable. - **REQ-IDs covered:** REQ-NFR-MAST-02 (full auditability) - **Wave:** 4 - **Dependencies:** all prior slices - **Persona:** lead-developer (orchestration), backend-engineer (tests) ### Tasks #### TASK-08-01 — End-to-end P1 smoke test - **Persona:** lead-developer - **File:** `scripts/test_mastery_e2e.py` (new) - **Content:** simulate 3 sessions across 3 distinct scenarios → verify mastery gate opens after 3 passing scenarios with score ≥3.5. Verify theta converges. Verify progress advances. Verify gate events recorded. #### TASK-08-02 — Audit log queryability test - **Persona:** backend-engineer - **File:** `tests/test_gate_audit_log.py` (new) - **Content:** query mastery_gate_events by learner, by path, by date range. Verify evidence (scenarios_passed, rubric_scores) is persisted and reconstructable. #### TASK-08-03 — P1 verification matrix - **Persona:** lead-developer - **File:** `.ciagent/VERIFY-P1.md` (new — pre-verify checklist for the verify stage) - **Content:** REQ-ID → test mapping. Confirm all P1 REQ-IDs have covering tests. #### TASK-08-04 — Real-LLM evidence extraction smoke test (grill Axis 7 FIX #1) - **Persona:** backend-engineer - **File:** `scripts/test_real_llm_evidence.py` (new — staging-gated, requires OLLAMA_API_KEY) - **Content:** run one real session transcript through the *actual* deepseek-v4-flash:cloud evidence extractor. Verify output is valid JSON with fuzzy-matching quotes. This runs only in staging (gated by `PRAXIS_RUN_REAL_LLM_TESTS=1` env). Mocked-LLM tests stay in CI. Validates that the extraction prompt works, not just the scoring logic. --- ## SLICE-09: VC Issuer + Verification Endpoint + Interop/Rotation Tests (W5) - **Goal:** Implement Ed25519-signed W3C VC 2.0 issuance + public verification + Status List revocation, SQLite-backed issuer keys (D-033, D-042, D-043, REQ-MAST-03, REQ-NFR-VC-01, REQ-NFR-VC-02). VC labeled `formative` per grill Axis 4 MUST #1. - **REQ-IDs covered:** REQ-MAST-03, REQ-NFR-VC-01, REQ-NFR-VC-02 - **Wave:** 5 - **Dependencies:** SLICE-07 (gate-open trigger — TASK-07-01 step 7 calls issue_credential) - **Persona:** security-engineer (issuer + crypto), data-engineer (SQLite issuer_keys/issued_credentials tables) ### Tasks #### TASK-09-01 — SQLite issuer keys + issued_credentials tables - **Persona:** data-engineer - **File:** `db/migrations/0003_mastery.sql` (extend), `db/store.py` (extend) - **Content:** `issuer_keys` table (id, public_key TEXT, private_key_enc BLOB, status TEXT active|superseded, created_at). `issued_credentials` table (id, learner_id, vc_payload_json, signature_b64, status active|revoked, issued_at). `PraxisStore` async methods: `init_issuer_key()`, `get_active_signing_key()`, `get_public_key(key_id)`, `insert_credential()`, `get_credential()`, `set_credential_status()`. Private key encrypted at rest with `PRAXIS_VC_ISSUER_KEY` root key from env (D-042). #### TASK-09-02 — Ed25519 issuer key management + VC payload builder + JCS + signing - **Persona:** security-engineer - **File:** `server/vc/issuer_keys.py` (new), `server/vc/issuer.py` (new) - **Content:** `init_issuer_key(store, root_key) -> KeyPair` — generate Ed25519 (pynacl), encrypt private key, store in SQLite. `build_vc_payload(learner_ref, path, scenarios_passed, rubric_score, completed_weeks, evidence) -> dict` (W3C VC 2.0: `scenariosPassed`, `rubricScore`, `completedWeeks: 6`, `evidence`, `issuedAt`, `validUntil: +3y`, **`credentialTier: "formative"`** per grill Axis 4). `canonicalize(payload) -> bytes` (JCS via canonicaljson). `sign(payload, signing_key) -> str` (eddsa-jcs-2022). `issue_credential(...) -> str` (stores in SQLite). #### TASK-09-03 — Bitstring Status List (revocation) - **Persona:** security-engineer - **File:** `server/vc/status_list.py` (new) - **Content:** `BitstringStatusList` — one bitstring per status list, indexed by credential sequence. `set_status(credential_idx, revoked)`, `get_status(credential_idx) -> bool`. Persisted in SQLite (`status_lists` table or adjacent to issuer_keys). Revocation latency = next verify call (status list fetched from SQLite on every verification — no cache, REQ-NFR-VC-02). #### TASK-09-04 — Public verification endpoint - **Persona:** security-engineer - **File:** `server/vc/verification.py` (new), `server/__main__.py` (extend — add route) - **Content:** `GET /vc/verify/` — public, unauthenticated (D-043). Fetch credential from SQLite, fetch issuer public key from `verificationMethod` URL, validate Ed25519 signature, check status list. Return `{valid, status, issuer, credential, mastery, credentialTier: "formative", verifiedAt}`. No PII beyond what the credential asserts. #### TASK-09-05 — VC unit tests - **Persona:** security-engineer - **File:** `tests/test_vc_issuer.py` (new) - **Content:** key generation, sign/verify round-trip, tamper detection (flip a byte → verify fails), JCS canonicalization determinism, status list set/get, revocation invalidates verification. #### TASK-09-06 — VC integration test (issue → verify round-trip + key rotation) - **Persona:** security-engineer - **File:** `tests/test_vc_integration.py` (new) - **Content:** issue a credential, GET /vc/verify/ → valid: true, credentialTier: formative. Revoke → GET → valid: false, status: revoked. Tamper payload → verify fails. Key rotation: old VC still verifies against archived public key. #### TASK-09-07 — VC interop test (grill Axis 3 MUST #1 — external W3C verifier) - **Persona:** security-engineer - **File:** `tests/test_vc_interop.py` (new — staging-gated, requires external verifier dependency) - **Content:** verify a Praxis-issued VC against at least one *external* W3C VC verifier (e.g., `digitalbazaar/vc-verifier` or a JS `@digitalcredentials/vc` verifier via subprocess). Round-trip self-verification is insufficient for cryptographic claims. This is the grill's binding MUST — custom crypto code without interop verification is an unmitigated liability. #### TASK-09-08 — Key-rotation operational drill (grill Axis 3 MUST #2) - **Persona:** security-engineer - **File:** `tests/test_vc_key_rotation_drill.py` (new) - **Content:** end-to-end operational drill — issue N VCs with key A, rotate to key B (archive A as superseded), issue M VCs with key B, verify all N+M VCs still verify (N against archived key A, M against active key B), revoke one of each, verify revocation. This is the *one* crypto procedure that, if broken, silently invalidates every credential ever issued. --- # Final Phase (P2) — Review + Audit + Milestone Ship **Branch:** `phase/02-final-review-ship` → merged to `milestone/v0.3-mastery-scoring` → merged to `main` **Ship:** `v0.1.5` (final patch = v0.3 milestone release) **REQ-IDs covered:** all v0.3 REQ-IDs (milestone-complete verification) ### Tasks (delegated to ciagent-review + ciagent-audit + ciagent-ship) 1. Run branch gate → create `phase/02-final-review-ship` 2. `ciagent-review` — multi-persona review across P1; auto-apply P0 fixes, flag P1+ 3. `ciagent-audit` — reconstruction test, file discipline, branch hygiene, commit discipline 4. `ciagent-ship` — merge phase/02 → milestone/v0.3 → main; tag v0.1.5; create release with full milestone summary 5. Update REQUIREMENTS.md (all v0.3 REQ → complete), ROADMAP.md (v0.3 → complete; v0.4 = operator tier) 6. Commit: `docs(milestone): complete v0.3-mastery-scoring` 7. Clear checkpoint --- # REQ-ID Coverage Matrix (post-grill) | REQ-ID | Phase | Slice(s) | Coverage | |--------|-------|----------|----------| | REQ-MAST-01 | P1 | SLICE-01, 03 | rubric schema + scoring | | REQ-MAST-02 | P1 | SLICE-07 | mastery score + gate logic | | REQ-MAST-03 | P1 | SLICE-09 | VC issuer (formative-tier, SQLite-backed) | | REQ-MAST-04 | — | — | principle (accepted) | | REQ-SCEN-02 | P1 | SLICE-04 | IRT dynamic difficulty | | REQ-SCEN-03 | P1 | SLICE-02, 06 | scenario library | | REQ-SCEN-04 | P1 | SLICE-02, 06 | expert-authored format + AI variation hooks | | REQ-PATH-02 | P1 | SLICE-05 | 6-week path structure | | REQ-NFR-MAST-01 | P1 | SLICE-03 | deterministic scoring | | REQ-NFR-MAST-02 | P1 | SLICE-07, 09 | gate auditability (SQLite) | | REQ-NFR-VC-01 | P1 | SLICE-09 | tamper-evidence + interop test (TASK-09-07) | | REQ-NFR-VC-02 | P1 | SLICE-09 | revocation latency (next verify call) | | REQ-NFR-IRT-01 | P1 | SLICE-04 | IRT <100ms | **Deferred to v0.4 (operator tier — per grill Axis 2):** REQ-DASH-01, REQ-AUTH-01, REQ-MT-01, REQ-MT-02, REQ-NFR-DASH-01, REQ-NFR-DASH-02, REQ-NFR-AUTH-01, REQ-NFR-MT-01. **v0.3 total: 13 REQ-IDs covered (7 functional + 6 NFR). 0 partial. 0 deferred within v0.3. 8 REQ-IDs deferred to v0.4.** --- # Open Questions Deferred to EXECUTE 1. **R-VC-02 (validUntil):** 3-year default, configurable per path. Confirm in SLICE-09. 2. **R-IRT-01 (cold start):** Fall back to scenario.difficulty until ≥5 observations. Confirm in SLICE-04. 3. **R-MAST-03 (per-archetype weights):** Ship refund/complaint weights only in v0.3 (static — dynamic branch-dependent re-weighting is a future feature per grill Axis 9 FIX). Confirm in SLICE-06. 4. **VC interop test dependency:** TASK-09-07 requires an external W3C verifier. Confirm which verifier is available (digitalbazaar/vc-verifier or @digitalcredentials/vc) and whether it runs in CI or staging-only. ---