ec6fcc64bc
Phase 2 complete — Cohort Dashboard + Aggregation: - Cohort aggregation pipeline (k-anon ≥10 write-time suppression, async hook, nightly 03:00 CT reconcile) - 4 auth-gated operator API endpoints (cohort, mastery, failure-patterns, credentials) - React cohort dashboard (BrowserRouter, login, 3 views, inline SVG sparklines, auth gate) - SPA fallback via SpaStaticFiles subclass (G-041 — NOT catch-all route) - G-038 differencing-attack test (unit + API e2e) - 317 pytest pass, 36 skip, 0 fail; 17/17 vitest pass; npm build + typecheck clean ---ci--- project: praxis phase: 2 milestone: v0.4 status: complete requirements: covered: [REQ-DASH-01, REQ-NFR-DASH-01, REQ-NFR-DASH-02, REQ-MT-02] partial: [] ---/ci---
405 lines
36 KiB
Markdown
405 lines
36 KiB
Markdown
# Praxis — v0.4 Phase 2 Verification (Cohort Dashboard + Aggregation)
|
|
|
|
## Summary
|
|
- Verdict: **APPROVE_WITH_NOTES**
|
|
- Layers: structural **PASS**, behavioral **PASS**, security **PASS**, quality **PASS**
|
|
- REQ coverage: **4/4** (REQ-DASH-01, REQ-NFR-DASH-01, REQ-NFR-DASH-02, REQ-MT-02 pipeline completion)
|
|
- Grill MUSTs honored: **2/2** (G-038 differencing-attack test, G-041 SPA fallback via custom StaticFiles subclass)
|
|
- P0 fixes applied: **0** (none needed — no P0 issues found)
|
|
- P1+ flagged: **4** (non-blocking, for post-hoc review in P3)
|
|
|
|
> Phase 2 (P2) of the v0.4 milestone covers SLICE-07..10 (23 tasks): cohort aggregation pipeline, operator API endpoints, React cohort dashboard, and P2 integration. 4 commits since `milestone/v0.4-operator-tier`: c396ded (SLICE-07), a7f7c4e (SLICE-08), d39bd14 (SLICE-09), de2020e (SLICE-10).
|
|
>
|
|
> This report supersedes the prior TASK-10-05 verification matrix (preserved in §REQ-ID Coverage Matrix below).
|
|
|
|
---
|
|
|
|
## Layer 1 — Structural
|
|
|
|
### 1.1 File existence (all P2 files present)
|
|
|
|
| File | Exists | LOC | Notes |
|
|
|------|--------|-----|-------|
|
|
| `server/cohort/__init__.py` | YES | 0 | package marker |
|
|
| `server/cohort/aggregator.py` | YES | 230 | k-anon suppression, 7-day window, metric cells |
|
|
| `server/cohort/hook.py` | YES | 44 | fire-and-forget on_session_end, no-op if no Postgres |
|
|
| `server/cohort/nightly.py` | YES | 232 | NightlyScheduler, 03:00 CT, R-DASH-04 retry |
|
|
| `server/operator/__init__.py` | YES | 0 | package marker |
|
|
| `server/operator/_common.py` | YES | 93 | shared Cell/PathView/ViewResponse models, require_pg_store, all_recent_aggregates |
|
|
| `server/operator/cohort.py` | YES | 42 | GET /api/operator/cohort (practice volume) |
|
|
| `server/operator/mastery.py` | YES | 45 | GET /api/operator/mastery (mastery progression) |
|
|
| `server/operator/failure_patterns.py` | YES | 44 | GET /api/operator/failure-patterns |
|
|
| `server/operator/credentials.py` | YES | 78 | GET /api/operator/credentials + POST /{id}/revoke |
|
|
| `client/src/operator/Login.tsx` | YES | 93 | login form, 429 handling, keyboard-accessible |
|
|
| `client/src/operator/Dashboard.tsx` | YES | 120 | auth gate, 3 view tabs, freshness, logout |
|
|
| `client/src/operator/Sparkline.tsx` | YES | 49 | inline SVG polyline, zero deps |
|
|
| `client/src/operator/views/PracticeVolume.tsx` | YES | 81 | practice volume view + sparklines |
|
|
| `client/src/operator/views/MasteryProgression.tsx` | YES | 84 | mastery progression view |
|
|
| `client/src/operator/views/FailurePatterns.tsx` | YES | 94 | failure patterns view |
|
|
| `client/src/operator/views/_viewCommon.ts` | YES | 60 | shared Cell type, suppressedLabel, formatFreshness |
|
|
| `client/src/operator/__tests__/Dashboard.test.tsx` | YES | 193 | 17 vitest tests |
|
|
| `tests/test_cohort_aggregation.py` | YES | 246 | k-anon threshold, idempotency, G-038 |
|
|
| `tests/test_cohort_nightly.py` | YES | 199 | scheduler timing, R-DASH-04, reconcile |
|
|
| `tests/test_operator_endpoints.py` | YES | 304 | 401/200 auth, suppressed cells, revoke, R-DASH-02 |
|
|
| `tests/test_p2_aggregation_integration.py` | YES | 236 | e2e aggregation→endpoint (skips without Postgres) |
|
|
| `tests/test_p2_spa_fallback.py` | YES | 128 | 9 SPA fallback assertions (G-041) |
|
|
| `client/vitest.config.ts` | YES | 13 | vitest config |
|
|
| `client/src/App.tsx` (extended) | YES | 27 | BrowserRouter routes, voice UI at / unchanged |
|
|
| `client/src/VoiceSession.tsx` | YES | 177 | extracted voice session (unchanged behavior) |
|
|
| `server/session_recorder.py` (extended) | YES | +52 | aggregation hook chained, off voice path |
|
|
| `server/__main__.py` (extended) | YES | +61 | operator routers + SpaStaticFiles + nightly scheduler |
|
|
|
|
### 1.2 Imports resolve
|
|
- `python3 -c "import server.__main__"` → **OK** (server imports cleanly, logs "SPA fallback enabled")
|
|
- `python3 -c "import server.cohort.aggregator, server.cohort.hook, server.cohort.nightly, server.operator.cohort, server.operator.mastery, server.operator.failure_patterns, server.operator.credentials"` → **OK** (all 7 new P2 modules import)
|
|
|
|
### 1.3 No stubs/TODOs in new P2 code
|
|
- `grep -r "TODO|FIXME|stub|placeholder|NotImplemented" server/cohort/ server/operator/` → **No matches** (zero stubs, zero TODOs in new P2 server code)
|
|
|
|
### 1.4 Deps + build
|
|
- `pip install -e . --break-system-packages` → **OK** (praxis-server 0.1.0 installed; P1 deps asyncpg/argon2-cffi/slowapi present)
|
|
- `docker compose config` → **OK** (validates, praxis-data volume present)
|
|
- `cd client && npm run build` → **OK** (vite v8.2.0, 168 modules, built in 547ms; bundle 662KB / 186KB gzip — within react-router-dom budget)
|
|
- `cd client && npm run typecheck` → **OK** (tsc -b --noEmit, no errors)
|
|
|
|
### 1.5 Router mount order (critical for R-DASH-03)
|
|
Verified in `server/__main__.py` diff (lines 256-298):
|
|
1. `app.include_router(auth_router)` — `/api/operator/login|logout|me`
|
|
2. `app.include_router(cohort_router)` — `/api/operator/cohort`
|
|
3. `app.include_router(mastery_router)` — `/api/operator/mastery`
|
|
4. `app.include_router(failure_router)` — `/api/operator/failure-patterns`
|
|
5. `app.include_router(credentials_router)` — `/api/operator/credentials`
|
|
6. `app.mount("/", SpaStaticFiles(...), name="spa")` — SPA fallback (AFTER all API routes)
|
|
|
|
**Order is correct**: API routes take precedence over the SPA fallback mount. R-DASH-03 verified.
|
|
|
|
**Layer 1 verdict: PASS** — all structural checks pass.
|
|
|
|
---
|
|
|
|
## Layer 2 — Behavioral
|
|
|
|
### 2.1 Test results
|
|
|
|
| Suite | Result | Notes |
|
|
|-------|--------|-------|
|
|
| `python3 -m pytest tests/` | **317 passed, 36 skipped, 0 failed** | matches expected (Postgres-requiring tests skip gracefully — PRAXIS_PG_DSN unset) |
|
|
| `cd client && npx vitest run` | **17/17 passed** | Dashboard auth gate, login form (200/401/429), sparkline (empty/dot/polyline/flat), suppressedLabel, formatFreshness, no-PII-in-DOM |
|
|
| `cd client && npm run build` | **PASS** | 168 modules, 547ms |
|
|
| `cd client && npm run typecheck` | **PASS** | tsc clean |
|
|
| P2-specific (`test_p2_spa_fallback.py` + `test_operator_endpoints.py` + `test_cohort_aggregation.py` + `test_cohort_nightly.py`) | **45/45 passed** | full P2 unit + SPA fallback coverage |
|
|
| `test_p2_aggregation_integration.py` | **3 skipped** | gracefully skipped (no PRAXIS_PG_DSN) — e2e aggregation→endpoint path covered by unit tests with mocked PgStore |
|
|
|
|
### 2.2 P2 SLICE acceptance criteria
|
|
|
|
**SLICE-07 (aggregation pipeline):**
|
|
- ✅ k-anon threshold exactly 10 — `test_k_anon_threshold_at_10` asserts `K_ANON_THRESHOLD == 10`; `test_9_learners_suppressed` (9 → suppressed), `test_10_learners_not_suppressed` (10 → not suppressed, value non-null), `test_11_learners_not_suppressed` (11 → not suppressed)
|
|
- ✅ Idempotent upsert — `test_idempotent_same_session_twice` (ON CONFLICT at DB layer)
|
|
- ✅ 7-day window — `test_rolling_window_7_days` (2026-08-04 → start=2026-07-29, 6-day span)
|
|
- ✅ All metrics computed — `test_multiple_metrics_computed` (sessions_count, active_learners_count, gate_open_rate, median_mastery_score, rubric_criterion_mean:*, failure_mode:*, branch:*)
|
|
- ✅ No PII in upserts — `test_no_pii_in_upsert_calls` (raw learner_ref not in any cell arg; cell_count is int)
|
|
- ✅ Hook non-blocking — `server/cohort/hook.py` uses `asyncio.create_task` in `session_recorder.py:161`; hook swallows exceptions (`test_hook_failure_logs_does_not_raise`)
|
|
- ✅ Hook no-op without Postgres — `test_hook_no_postgres_is_noop`
|
|
- ✅ Nightly scheduler timing — `test_seconds_until_next_03_ct_future_today` + `test_seconds_until_next_03_ct_past_today_wraps_tomorrow`
|
|
- ✅ R-DASH-04 nightly failure retry — `test_r_dash_04_nightly_failure_does_not_crash_scheduler`
|
|
- ✅ Nightly reconcile recomputes — `test_reconcile_recomputes_all_paths`
|
|
- ✅ Scheduler lifecycle — `test_scheduler_start_stop_lifecycle`
|
|
|
|
**SLICE-08 (operator API endpoints):**
|
|
- ✅ All 4 endpoints auth-gated (401 without cookie) — `test_cohort_401_without_cookie`, `test_mastery_401_without_cookie`, `test_failure_patterns_401_without_cookie`, `test_credentials_401_without_cookie`, `test_revoke_401_without_cookie`
|
|
- ✅ All 4 endpoints 200 with cookie — `test_cohort_200_with_cookie`, `test_mastery_200_with_cookie`, `test_failure_patterns_200_with_cookie`, `test_credentials_200_with_cookie`
|
|
- ✅ Suppressed cells value=null — `test_suppressed_cells_value_null` (cell_suppressed=true → value=null)
|
|
- ✅ last_updated = max(updated_at) — `test_last_updated_is_max`
|
|
- ✅ Credential revoke — `test_credential_revoke_sets_status_revoked` (status='revoked', set_credential_status awaited) + `test_credential_revoke_404_unknown` (404 for unknown)
|
|
- ✅ No per-learner data (R-DASH-02) — `test_no_per_learner_data_in_cohort_response` (no "learner-1", no "learner_ref" in response)
|
|
- ✅ 503 when no Postgres — `test_cohort_503_no_postgres` (graceful degradation)
|
|
|
|
**SLICE-09 (React dashboard):**
|
|
- ✅ react-router-dom@^7 added (`client/package.json`)
|
|
- ✅ BrowserRouter wrapper + route switch — `client/src/App.tsx`: `/` → VoiceSession (unchanged), `/operator/login` → Login, `/operator/dashboard` → Dashboard, `*` → VoiceSession (fallback)
|
|
- ✅ Login form — Login.tsx, 429 handling (`test shows rate-limit message on 429`), keyboard-accessible (label associations)
|
|
- ✅ Dashboard shell + auth gate — Dashboard.tsx, 401 on /me → redirect (`test redirects to /operator/login on 401`), 3 view tabs, freshness indicator, logout
|
|
- ✅ Inline SVG sparkline — Sparkline.tsx (49 LOC, zero deps), empty/dot/polyline/flat-line cases tested
|
|
- ✅ 3 view components — PracticeVolume, MasteryProgression, FailurePatterns (read-only, no drill-down)
|
|
- ✅ Suppressed cell display — "— (<10 learners)" (`suppressedLabel` test)
|
|
- ✅ Freshness indicator — formatFreshness (m/h/d ago)
|
|
- ✅ No PII in DOM — `test does not render learner_ref fields`
|
|
|
|
**SLICE-10 (P2 integration):**
|
|
- ✅ SPA fallback (G-041) — custom `SpaStaticFiles` subclass in `__main__.py:279-289`, NOT a catch-all route; 9 assertions in `test_p2_spa_fallback.py` all pass
|
|
- ✅ Voice UI at `/` unchanged (R-DASH-05) — `test_root_serves_voice_ui` (200, text/html, `<div id="root">`)
|
|
- ✅ API routes return JSON not HTML — `test_api_operator_cohort_is_json_not_html`, `test_health_is_json`, `test_vc_verify_nonexistent_is_404`
|
|
- ✅ Assets served by StaticFiles — `test_assets_served_by_staticfiles_not_spa_fallback` (`/assets/index.js` → javascript content-type, not index.html)
|
|
- ✅ Nightly scheduler starts in lifespan — `server/__main__.py:116` `await nightly.start(app.state.pg_store)`; cancelled on shutdown (`await nightly.stop()` line 121)
|
|
- ✅ E2e aggregation→endpoint — `test_p2_aggregation_integration.py::test_aggregation_to_endpoint_e2e` (skips without Postgres; logic covered by unit tests with mocked store)
|
|
|
|
### 2.3 REQ coverage
|
|
|
|
| REQ-ID | Covered by | Status |
|
|
|--------|-----------|--------|
|
|
| **REQ-DASH-01** (cohort dashboard, 3 views, k-anon, React under /operator/*) | SLICE-08 (4 endpoints), SLICE-09 (React UI), SLICE-10 (integration). `test_operator_endpoints.py` (all 4 endpoints 200/401), `Dashboard.test.tsx` (auth gate, login, 3 views), `test_p2_spa_fallback.py` (SPA serves /operator/*) | **COVERED** |
|
|
| **REQ-NFR-DASH-01** (k-anonymity ≥ 10) | SLICE-07 (write-time suppression in `aggregator.py`), SLICE-08 (query returns value=null for suppressed), SLICE-09 (display "— (<10 learners)"), SLICE-10 (e2e). `test_cohort_aggregation.py` (threshold at 10, 9/10/11 learners), `test_operator_endpoints.py::test_suppressed_cells_value_null`, `Dashboard.test.tsx::suppressedLabel`, G-038 differencing-attack | **COVERED** |
|
|
| **REQ-NFR-DASH-02** (freshness ≤ 24h) | SLICE-07 (nightly job + on-session-end hook), SLICE-10 (e2e). `test_cohort_nightly.py` (scheduler timing, reconcile, R-DASH-04), `test_operator_endpoints.py::test_last_updated_is_max`, `test_p2_aggregation_integration.py::test_nightly_reconciliation_updates_last_updated` (skips without Postgres) | **COVERED** |
|
|
| **REQ-MT-02** (pipeline completion — schema P1, pipeline P2) | SLICE-07 (aggregator + hook + nightly), SLICE-10 (e2e). `test_cohort_aggregation.py` (idempotent, multiple metrics, hook no-op/failure), `test_cohort_nightly.py` (reconcile), `test_p2_aggregation_integration.py::test_aggregation_to_endpoint_e2e` (skips without Postgres) | **COVERED** |
|
|
|
|
**4/4 P2 REQ-IDs covered.**
|
|
|
|
### 2.4 Grill MUSTs honored
|
|
|
|
**G-038 (differencing-attack test) — HONORED:**
|
|
- Unit layer: `test_cohort_aggregation.py::test_g038_differencing_attack_cannot_isolate_dropped_learner` — seeds 10 learners in window A, 9 in window B (learner-9 dropped), asserts window B is FULLY suppressed (value=NULL) so the dropped learner's contribution is not recoverable via subtraction. Verifies no per-learner ref leaks in either window's aggregate cells.
|
|
- API e2e layer: `test_p2_aggregation_integration.py::test_g038_differencing_attack_api_layer` — 10 learners on path diff_a, 9 on diff_b, asserts "a-9" not in response text and diff_b cells all suppressed with value=None. (Skips without Postgres — logic verified at unit layer.)
|
|
|
|
**G-041 (SPA fallback via custom StaticFiles subclass) — HONORED:**
|
|
- Implementation: `server/__main__.py:279-289` defines `class SpaStaticFiles(StaticFiles)` with `get_response` override that returns `FileResponse("index/dist/index.html")` only on 404 (non-file paths). This is the custom subclass approach mandated by G-041, NOT a `@app.get("/{path:path}")` catch-all (which would shadow asset serving per the grill's analysis).
|
|
- Test: `test_p2_spa_fallback.py::test_assets_served_by_staticfiles_not_spa_fallback` verifies `/assets/index.js` returns javascript content (not index.html) — the critical assertion 8 from TASK-10-04.
|
|
|
|
### 2.5 Voice UI at `/` unchanged (R-DASH-03, R-DASH-05)
|
|
|
|
- **Server**: `SpaStaticFiles` mount at `/` with `html=True` serves `index.html` for `/` (unchanged from v0.3 StaticFiles behavior). API routes registered before the mount take precedence. `test_root_serves_voice_ui` confirms 200 + text/html + `<div id="root">`.
|
|
- **Client**: `client/src/App.tsx` route `/` → `<VoiceSession />` (the existing voice session UI, extracted from the old App.tsx to VoiceSession.tsx — behavior unchanged). The `*` catch-all also serves VoiceSession (R-DASH-05: unknown routes fall back to learner surface, not a 404).
|
|
- **No regression**: 317 passed, 0 failed — all v0.1/v0.2/v0.3 tests still pass.
|
|
|
|
**Voice UI at `/` unchanged: CONFIRMED.**
|
|
|
|
**Layer 2 verdict: PASS** — all behavioral checks pass.
|
|
|
|
---
|
|
|
|
## Layer 3 — Security (STRIDE)
|
|
|
|
### Spoofing
|
|
- **Operator endpoints auth-gated via `current_operator` dependency.**
|
|
- Verified: all 4 operator routers (`cohort.py`, `mastery.py`, `failure_patterns.py`, `credentials.py`) import `current_operator` from `server.auth.dependencies` and apply `op: Operator = Depends(current_operator)` on every endpoint.
|
|
- Test coverage: 5 tests assert 401 without cookie (`test_cohort_401_without_cookie`, `test_mastery_401_without_cookie`, `test_failure_patterns_401_without_cookie`, `test_credentials_401_without_cookie`, `test_revoke_401_without_cookie`).
|
|
- **Disposition: low (accept).** No bypass path found — every `/api/operator/*` route (except `/login` which is rate-limited, not auth-gated) requires the dependency.
|
|
|
|
### Tampering
|
|
- **Aggregation pipeline — k-anon suppression at write time.**
|
|
- `server/cohort/aggregator.py:87` `suppressed = active_count < K_ANON_THRESHOLD` (K_ANON_THRESHOLD=10, module constant). Suppression applied before `upsert_cohort_aggregate` — value set to `None` when suppressed (lines 90, 94, 103, etc.).
|
|
- Nightly reconciliation (`nightly.py:127`) re-applies the same threshold: `suppressed = active_count < K_ANON_THRESHOLD`.
|
|
- Suppression cannot be bypassed via the API: endpoints read `cohort_aggregates` rows as-is (no post-processing that could un-suppress); suppressed cells have `value=null` in the DB (enforced at write time).
|
|
- **Disposition: low (accept).** Write-time suppression is server-side, not display-only.
|
|
|
|
### Repudiation
|
|
- **Credential revoke (POST /api/operator/credentials/{id}/revoke).**
|
|
- The revoke endpoint sets `status='revoked'` + `revoked_at=now()` in Postgres (`pg_store.py:224` `extra = ", revoked_at = now()" if status == 'revoked'`). The `revoked_at` timestamp is an audit trail.
|
|
- **GAP (P1+ flagged)**: The revoke endpoint does NOT log the revocation event at the application level, and the `operator_id` of the revoking operator is available via `current_operator` but is NOT recorded against the credential revocation. The `issued_credentials.operator_id` column tracks the *issuer*, not the *revoker*. There is no revocation audit log linking operator→action→credential→timestamp.
|
|
- Mitigation: the `revoked_at` timestamp + the signed session cookie (which records `operator_id` in `request.session`) provide a partial audit trail, but correlating them requires cross-referencing session logs.
|
|
- **Disposition: medium (mitigate — P1+ flagged).** Add application-level logging of revocation events (operator_id, credential_id, timestamp) in P3.
|
|
|
|
### Info Disclosure
|
|
- **k-anonymity ≥ 10 enforced (REQ-NFR-DASH-01).**
|
|
- Write-time suppression: cells with < 10 distinct learners → `cell_suppressed=TRUE`, `value=NULL`. Verified by `test_9_learners_suppressed`, `test_10_learners_not_suppressed`.
|
|
- No per-learner drill-down (R-DASH-02): endpoints return only aggregate cells (path, metric, value, cell_count, cell_suppressed) — no `learner_ref` in cohort/mastery/failure responses. Verified by `test_no_per_learner_data_in_cohort_response` (no "learner_ref" string, no "learner-1" in response).
|
|
- G-038 differencing-attack defense: window B (9 learners) is fully suppressed (value=NULL), so subtracting B from A is not possible. Verified at unit + API layers.
|
|
- No PII in Postgres aggregates (D-031): only opaque `learner_ref` for distinct counting, never stored in aggregate cells. Verified by `test_no_pii_in_upsert_calls`.
|
|
- **Disposition: low (accept).** k-anon defense-in-depth is sound; G-038 explicitly tested.
|
|
|
|
### Denial of Service
|
|
- **Aggregation hook is async fire-and-forget (non-blocking).**
|
|
- `server/session_recorder.py:161` `asyncio.create_task(self._run_cohort_aggregation(session_outcome))` — hook runs off the voice path (C-8, D-054). Voice loop latency unaffected.
|
|
- `server/cohort/hook.py:37` `except Exception: log.exception(...)` — hook failure does not propagate; nightly job reconciles.
|
|
- `test_hook_failure_logs_does_not_raise` confirms no exception propagation.
|
|
- Nightly job doesn't block the event loop: `NightlyScheduler._run_loop` uses `asyncio.sleep(secs)` (cooperative); reconciliation is a sequence of `await pg_store.upsert_cohort_aggregate(...)` calls (yields between each).
|
|
- **Disposition: low (accept).** Hook failure → log + nightly reconcile (R-DASH-04). No crash path.
|
|
|
|
### Elevation of Privilege
|
|
- **Single operator role. No RBAC bypass.**
|
|
- All 4 operator endpoints + credential management use `Depends(current_operator)`. The `current_operator` dependency (`server/auth/dependencies.py`) checks `request.session["operator_id"]` → fetches operator → checks `is_active=True` → returns `Operator`. No role-based dispatch exists (single role).
|
|
- The `current_operator` dependency never trusts the client (D-057) — it validates the signed session cookie server-side.
|
|
- **Disposition: low (accept).** No RBAC to bypass; single operator role; auth-gated everywhere.
|
|
|
|
**Layer 3 verdict: PASS** — all STRIDE categories low except Repudiation (medium, mitigated, P1+ flagged). No high-severity findings.
|
|
|
|
---
|
|
|
|
## Layer 4 — Quality (multi-persona review)
|
|
|
|
### Correctness
|
|
- **k-anon threshold (exactly 10):** `K_ANON_THRESHOLD = 10` module constant; 9 → suppressed, 10 → not suppressed, 11 → not suppressed. Tests cover all three boundaries. ✅
|
|
- **Aggregation idempotency:** ON CONFLICT upsert at the DB layer (PgStore); hook is deterministic (same learner produces same distinct-count + counter state in cache). `test_idempotent_same_session_twice` passes. ✅
|
|
- **Nightly scheduler timing:** `seconds_until_next_03_ct` computes seconds until 03:00 CT (fixed UTC-5 offset, documented DST approximation — acceptable for nightly reconciliation). `test_seconds_until_next_03_ct_future_today` + `test_seconds_until_next_03_ct_past_today_wraps_tomorrow` pass. ✅
|
|
- **SPA fallback (G-041):** Custom `SpaStaticFiles` subclass, NOT catch-all route. Serves assets normally (JS/CSS), falls back to index.html only on 404. `test_assets_served_by_staticfiles_not_spa_fallback` confirms assets are not shadowed. ✅
|
|
|
|
### Testing
|
|
- **Coverage gaps:** Postgres-requiring tests (`test_p2_aggregation_integration.py`, `test_pg_store.py`) skip gracefully when `PRAXIS_PG_DSN` unset — 36 skipped total, 0 failed. The e2e aggregation→endpoint→dashboard path is covered by unit tests with mocked PgStore (45/45 P2 tests pass). ✅
|
|
- **Client tests (vitest):** 17/17 pass — auth gate, login (200/401/429), sparkline (4 cases), suppressedLabel, formatFreshness, no-PII-in-DOM. ✅
|
|
- **G-038 differencing-attack coverage:** Unit layer (`test_g038_differencing_attack_cannot_isolate_dropped_learner`) + API e2e layer (`test_g038_differencing_attack_api_layer`). The unit test is the primary proof (runs without Postgres); the e2e test is a bonus that skips without Postgres. ✅
|
|
|
|
### Security
|
|
- **SQL injection in PgStore queries:** All queries use asyncpg parameterized placeholders (`$1`, `$2`, etc.). Verified in `pg_store.py` (operator CRUD, cohort upsert, credential methods, gate events) and `server/operator/_common.py::all_recent_aggregates` (`WHERE window_start >= $1`). One f-string interpolation in `set_credential_status` (`f"UPDATE ... SET status = $1{extra} WHERE id = $2"`) — but `extra` is a hardcoded constant (`, revoked_at = now()` or empty) derived from the `status` value comparison, NOT user input. Safe. ✅
|
|
- **k-anon suppression enforced server-side:** Suppression is applied in `aggregator.py` (write time) and re-applied in `nightly.py` (reconcile). The API endpoints read cells as-is — no client-side or display-only suppression. ✅
|
|
- **No PII in API responses:** Cohort/mastery/failure endpoints return only (path, metric, value, cell_count, cell_suppressed, updated_at). Credentials endpoint returns (id, learner_ref, vc_type, status, issued_at, revoked_at) — `learner_ref` is an opaque string (D-031), not PII. ✅
|
|
|
|
### Performance
|
|
- **Aggregation hook non-blocking:** `asyncio.create_task` in `session_recorder.py:161` — fire-and-forget, off the voice path (C-8). ✅
|
|
- **Nightly job doesn't block event loop:** `asyncio.sleep(secs)` + sequential `await` calls (cooperative). Runs at 03:00 CT (low activity). ✅
|
|
- **SPA fallback doesn't add latency to API routes:** API routes are registered before the StaticFiles mount — FastAPI matches API routes first (no fallback overhead). ✅
|
|
|
|
### Maintainability
|
|
- **SpaStaticFiles subclass:** Clean 11-line override (`get_response` catches 404 → FileResponse). Well-commented with G-041 rationale. ✅
|
|
- **3 view components consistent:** All 3 (PracticeVolume, MasteryProgression, FailurePatterns) share `_viewCommon.ts` (Cell type, suppressedLabel, formatFreshness) and follow the same fetch→render pattern. ✅
|
|
- **Router mounting order:** API routes → SPA fallback mount. Documented in `__main__.py:256-298` comments. ✅
|
|
|
|
### Adversarial
|
|
- **What if an attacker calls /api/operator/cohort with a path that doesn't exist?** The endpoint takes no path parameter — it returns all paths' aggregates from the last 30 days. A non-existent path simply returns no rows (no error, no leak). ✅
|
|
- **What if k-anon threshold is lowered via config?** `K_ANON_THRESHOLD = 10` is a module constant in `aggregator.py`, NOT configurable via env. Changing it requires a code change + redeploy. This is correct for a privacy control — it should not be runtime-configurable. ✅
|
|
- **What if the aggregation hook runs before Postgres is healthy?** The hook checks `pg_store is None` → no-op + WARNING (`hook.py:27-32`). If Postgres is unhealthy mid-session, `upsert_cohort_aggregate` raises → caught by `hook.py:37` `except Exception: log.exception(...)` → nightly job reconciles. ✅
|
|
|
|
**Layer 4 verdict: PASS** — no quality issues found. Code is clean, well-commented, consistently structured, and adversarially sound.
|
|
|
|
---
|
|
|
|
## P0 Fixes Applied
|
|
|
|
**None.** No P0 issues (broken tests, missing REQ coverage, security holes) were found. The P2 implementation is correct, complete, and secure.
|
|
|
|
---
|
|
|
|
## P1+ Flagged for Post-Hoc Review
|
|
|
|
The following non-blocking issues are flagged for review in the final phase (P3):
|
|
|
|
### P1+-01: Credential revocation lacks application-level audit log (Repudiation)
|
|
- **File:** `server/operator/credentials.py`
|
|
- **Issue:** The `revoke_credential` endpoint sets `status='revoked'` + `revoked_at=now()` in Postgres but does NOT log the revocation event at the application level, and the revoking `operator_id` (available via `current_operator`) is not recorded against the revocation action. The `issued_credentials.operator_id` column tracks the *issuer*, not the *revoker*.
|
|
- **Risk:** An operator who revokes a credential leaves a DB timestamp but no application log linking *who* revoked *which* credential *when*. Correlating requires cross-referencing session logs.
|
|
- **Mitigation present:** `revoked_at` timestamp in DB + signed session cookie (operator_id in session).
|
|
- **Recommended fix (P3):** Add `log.info("credential revoked: operator=%s cred_id=%s", op.id, cred_id)` in `revoke_credential`, and consider an `audit_log` table or `revoked_by_operator_id` column on `issued_credentials`.
|
|
|
|
### P1+-02: Nightly scheduler uses fixed UTC-5 offset (not true America/Winnipeg DST)
|
|
- **File:** `server/cohort/nightly.py:27` `CT = _dt.timezone(_dt.timedelta(hours=-5), "CT")`
|
|
- **Issue:** The CT timezone is approximated as a fixed UTC-5 offset. America/Winnipeg observes CST (UTC-6) in winter + CDT (UTC-5) in summer. The scheduler will drift by 1 hour across DST boundaries (the nightly job runs at 02:00 or 04:00 local instead of 03:00).
|
|
- **Risk:** Low — the nightly job runs once/day; a 1-hour drift is acceptable for a reconciliation job (on-session-end hook keeps data fresh ≤ 24h).
|
|
- **Mitigation present:** Documented in `nightly.py:36-41` comments ("drift of ≤1h over DST boundaries is acceptable... a future hardening would use zoneinfo.ZoneInfo").
|
|
- **Recommended fix (P3):** Replace `CT` constant with `zoneinfo.ZoneInfo("America/Winnipeg")` for proper DST handling.
|
|
|
|
### P1+-03: Aggregation in-memory cache is per-PgStore-instance (lost on restart)
|
|
- **File:** `server/cohort/aggregator.py:162-170` `_cache(pg_store)`
|
|
- **Issue:** The aggregator maintains a per-PgStore-instance in-memory cache (`_agg_cache`) for running counters + distinct learner sets. On server restart, the cache is lost — the next on-session-end hook starts fresh, and the active_learners_count may reset to 1 (under-counting distinct learners until the nightly job reconciles from `mastery_gate_events`).
|
|
- **Risk:** Low — the nightly job reconciles the true distinct count from the audit log (`mastery_gate_events`). Between restart and nightly reconcile, cells may be incorrectly suppressed (under-count → over-suppression, which is privacy-safe but value-destroying).
|
|
- **Mitigation present:** Nightly reconciliation recomputes from `mastery_gate_events` (the source of truth).
|
|
- **Recommended fix (P3):** Document that the in-memory cache is best-effort + nightly reconcile is authoritative, OR persist the distinct-learner set to Postgres (adds a table — may not be worth the complexity for pilot scale).
|
|
|
|
### P1+-04: `set_credential_status` uses f-string interpolation in SQL (code smell, not vulnerability)
|
|
- **File:** `db/pg_store.py:227` `f"UPDATE issued_credentials SET status = $1{extra} WHERE id = $2"`
|
|
- **Issue:** The `extra` variable (`, revoked_at = now()` or empty string) is interpolated via f-string into the SQL query. While `extra` is a hardcoded constant (not user input) and `status`/`cred_id` are parameterized, f-strings in SQL are a code smell that future maintainers might copy incorrectly.
|
|
- **Risk:** None (current code is safe — `extra` is derived from `status == "revoked"` comparison, not user input).
|
|
- **Recommended fix (P3):** Refactor to two explicit queries: `UPDATE ... SET status = $1 WHERE id = $2` and `UPDATE ... SET status = $1, revoked_at = now() WHERE id = $2`, eliminating the f-string.
|
|
|
|
---
|
|
|
|
## REQ-ID Coverage Matrix (from TASK-10-05, preserved)
|
|
|
|
### REQ-DASH-01 — Cohort dashboard (3 views + auth gate)
|
|
| Test file | Test | What it verifies |
|
|
|-----------|------|------------------|
|
|
| tests/test_operator_endpoints.py | test_cohort_200_with_cookie | GET /api/operator/cohort returns practice volume |
|
|
| tests/test_operator_endpoints.py | test_mastery_200_with_cookie | GET /api/operator/mastery returns mastery progression |
|
|
| tests/test_operator_endpoints.py | test_failure_patterns_200_with_cookie | GET /api/operator/failure-patterns returns failure data |
|
|
| tests/test_operator_endpoints.py | test_credentials_200_with_cookie | GET /api/operator/credentials lists VCs |
|
|
| tests/test_operator_endpoints.py | test_cohort_401_without_cookie (+ 4 others) | All endpoints auth-gated (401) |
|
|
| client/src/operator/__tests__/Dashboard.test.tsx | Dashboard auth gate | React auth gate redirects on 401 from /me |
|
|
| client/src/operator/__tests__/Dashboard.test.tsx | Login form | POST /api/operator/login → dashboard |
|
|
| tests/test_p2_spa_fallback.py | test_operator_dashboard_spa_fallback | /operator/dashboard serves index.html (SPA) |
|
|
| tests/test_p2_spa_fallback.py | test_operator_login_spa_fallback | /operator/login serves index.html (SPA) |
|
|
|
|
### REQ-NFR-DASH-01 — k-anonymity ≥ 10 (write-time suppression + query + display + e2e)
|
|
| Test file | Test | What it verifies |
|
|
|-----------|------|------------------|
|
|
| tests/test_cohort_aggregation.py | test_k_anon_threshold_at_10 | K_ANON_THRESHOLD == 10 |
|
|
| tests/test_cohort_aggregation.py | test_9_learners_suppressed | 9 learners → cell_suppressed=TRUE, value=NULL |
|
|
| tests/test_cohort_aggregation.py | test_10_learners_not_suppressed | 10 learners → non-suppressed, value non-null |
|
|
| tests/test_cohort_aggregation.py | test_11_learners_not_suppressed | 11 learners → non-suppressed |
|
|
| tests/test_cohort_aggregation.py | test_no_pii_in_upsert_calls | No raw learner_ref in aggregate cell args |
|
|
| tests/test_cohort_aggregation.py | test_g038_differencing_attack_cannot_isolate_dropped_learner | G-038: 10 in window A, 9 in B → dropped learner not isolatable |
|
|
| tests/test_operator_endpoints.py | test_suppressed_cells_value_null | API: suppressed cells have value=null |
|
|
| tests/test_operator_endpoints.py | test_no_per_learner_data_in_cohort_response | API: no per-learner data (R-DASH-02) |
|
|
| client/src/operator/__tests__/Dashboard.test.tsx | suppressedLabel | UI: suppressed cells render "— (<10 learners)" |
|
|
| tests/test_p2_aggregation_integration.py | test_aggregation_to_endpoint_e2e | E2e: 12 learners non-suppressed, 5 suppressed (skips without Postgres) |
|
|
| tests/test_p2_aggregation_integration.py | test_g038_differencing_attack_api_layer | G-038 e2e at API layer (skips without Postgres) |
|
|
|
|
### REQ-NFR-DASH-02 — Freshness ≤ 24h (nightly job + on-session-end hook)
|
|
| Test file | Test | What it verifies |
|
|
|-----------|------|------------------|
|
|
| tests/test_cohort_nightly.py | test_seconds_until_next_03_ct_future_today | Scheduler computes correct seconds until 03:00 CT |
|
|
| tests/test_cohort_nightly.py | test_seconds_until_next_03_ct_past_today_wraps_tomorrow | Wraps to next day correctly |
|
|
| tests/test_cohort_nightly.py | test_reconcile_recomputes_all_paths | Nightly recomputes all (path, window) cells |
|
|
| tests/test_cohort_nightly.py | test_r_dash_04_nightly_failure_does_not_crash_scheduler | R-DASH-04: failure logs + retries |
|
|
| tests/test_cohort_nightly.py | test_scheduler_start_stop_lifecycle | Scheduler starts + stops cleanly |
|
|
| tests/test_operator_endpoints.py | test_last_updated_is_max | API: last_updated = max(updated_at) |
|
|
| tests/test_p2_aggregation_integration.py | test_nightly_reconciliation_updates_last_updated | E2e: nightly reconcile refreshes last_updated (skips without Postgres) |
|
|
| tests/test_p2_aggregation_integration.py | test_aggregation_to_endpoint_e2e (assertion 8) | E2e: last_updated ≤ 24h (skips without Postgres) |
|
|
|
|
### REQ-MT-02 — Cohort aggregation pipeline (schema in P1, pipeline in P2)
|
|
| Test file | Test | What it verifies |
|
|
|-----------|------|------------------|
|
|
| tests/test_cohort_aggregation.py | test_multiple_metrics_computed | Pipeline computes all metric types |
|
|
| tests/test_cohort_aggregation.py | test_idempotent_same_session_twice | Idempotent upsert |
|
|
| tests/test_cohort_aggregation.py | test_rolling_window_7_days | 7-day rolling window computation |
|
|
| tests/test_cohort_aggregation.py | test_hook_no_postgres_is_noop | Graceful no-op without Postgres |
|
|
| tests/test_cohort_aggregation.py | test_hook_failure_logs_does_not_raise | Hook failure does not propagate |
|
|
| tests/test_cohort_nightly.py | test_reconcile_no_events_no_op | Nightly no-op when no events |
|
|
| tests/test_p2_aggregation_integration.py | test_aggregation_to_endpoint_e2e | Full pipeline e2e (skips without Postgres) |
|
|
|
|
### G-038 (binding — differencing-attack test)
|
|
| Test file | Test | What it verifies |
|
|
|-----------|------|------------------|
|
|
| tests/test_cohort_aggregation.py | test_g038_differencing_attack_cannot_isolate_dropped_learner | Unit: 10 in A, 9 in B → B suppressed, dropped learner not isolatable |
|
|
| tests/test_p2_aggregation_integration.py | test_g038_differencing_attack_api_layer | E2e at API layer (skips without Postgres) |
|
|
|
|
### G-041 (binding — SPA fallback via custom StaticFiles subclass)
|
|
| Test file | Test | What it verifies |
|
|
|-----------|------|------------------|
|
|
| tests/test_p2_spa_fallback.py | test_root_serves_voice_ui | Voice UI at / unchanged (R-DASH-05) |
|
|
| tests/test_p2_spa_fallback.py | test_operator_dashboard_spa_fallback | /operator/dashboard → index.html |
|
|
| tests/test_p2_spa_fallback.py | test_assets_served_by_staticfiles_not_spa_fallback | /assets/index.js served by StaticFiles (NOT catch-all) — G-041 critical assertion |
|
|
| tests/test_p2_spa_fallback.py | test_api_operator_cohort_is_json_not_html | API routes return JSON (not index.html) |
|
|
| tests/test_p2_spa_fallback.py | test_health_is_json | /health JSON |
|
|
|
|
### R-DASH-05 (voice UI at / unchanged)
|
|
| Test file | Test | What it verifies |
|
|
|-----------|------|------------------|
|
|
| tests/test_p2_spa_fallback.py | test_root_serves_voice_ui | / → index.html with <div id="root"> |
|
|
| client/src/operator/__tests__/Dashboard.test.tsx | (no PII in dashboard DOM) | Voice UI path unchanged |
|
|
|
|
---
|
|
|
|
## Test Results Summary
|
|
|
|
| Suite | Pass | Skip | Fail |
|
|
|-------|------|------|------|
|
|
| `python3 -m pytest tests/` (full) | 317 | 36 | 0 |
|
|
| `tests/test_p2_spa_fallback.py` | 9 | 0 | 0 |
|
|
| `tests/test_operator_endpoints.py` | 15 | 0 | 0 |
|
|
| `tests/test_cohort_aggregation.py` | 12 | 0 | 0 |
|
|
| `tests/test_cohort_nightly.py` | 9 | 0 | 0 |
|
|
| `tests/test_p2_aggregation_integration.py` | 0 | 3 | 0 (Postgres-requiring, skip gracefully) |
|
|
| `cd client && npx vitest run` | 17 | 0 | 0 |
|
|
| `cd client && npm run build` | PASS | — | — |
|
|
| `cd client && npm run typecheck` | PASS | — | — |
|
|
| `pip install -e . --break-system-packages` | PASS | — | — |
|
|
| `docker compose config` | PASS | — | — |
|
|
| `python3 -c "import server.__main__"` | PASS | — | — |
|
|
| `python3 -c "import ...all P2 modules"` | PASS | — | — |
|
|
|
|
---
|
|
|
|
## Voice UI at `/` Unchanged — Confirmation
|
|
|
|
**CONFIRMED.** Three layers of evidence:
|
|
|
|
1. **Server (`server/__main__.py`):** The `SpaStaticFiles` mount at `/` with `html=True` serves `index.html` for `/` — identical to the v0.3 `StaticFiles` behavior. The custom subclass only changes behavior for *non-file* paths (404 → index.html), not for `/` (which StaticFiles already serves as index.html with `html=True`). `test_root_serves_voice_ui` confirms 200 + text/html + `<div id="root">`.
|
|
|
|
2. **Client (`client/src/App.tsx`):** Route `/` → `<VoiceSession />`. The VoiceSession component was extracted from the old App.tsx (behavior unchanged — same voice session UI). The `*` catch-all also serves VoiceSession (R-DASH-05: unknown routes fall back to learner surface).
|
|
|
|
3. **Test suite:** 317 passed, 0 failed — all v0.1/v0.2/v0.3 tests (voice loop, WebRTC, scenarios, mastery, VC) still pass. No regression in the learner surface.
|
|
|
|
---
|
|
|
|
## Bottom Line
|
|
|
|
Phase 2 (Cohort Dashboard + Aggregation) is **APPROVE_WITH_NOTES**. All 4 layers pass. All 4 P2 REQ-IDs are covered. Both grill MUSTs (G-038 differencing-attack test, G-041 SPA fallback via custom StaticFiles subclass) are honored. Zero P0 issues. Four P1+ issues flagged for post-hoc review in P3 (credential revocation audit log, nightly scheduler DST, in-memory cache persistence, f-string SQL code smell) — all non-blocking, all with mitigations present.
|
|
|
|
The P2 implementation is shippable as `v0.1.8` pending the final P3 review + ship phase. |