ec397f2c65
v0.5 (Live Assist — on-the-job voice companion) milestone complete. 4 phases: P0 (pre-execution, v0.1.10) → P1 (assist core + guardrail, v0.1.11) → P2 (integration + tech-debt + NFR, v0.1.12) → P3 (final review + ship, v0.1.13 = milestone release). 16/16 REQs covered (3 ASSIST + 4 NFR + 9 IDEATE). 4 v0.6 backlog. 469 tests passed, 0 failed. 1 P0 fixed (guardrail processor safety). 8 P1+ flagged for v0.6. 8 v0.4 P1+ tech-debt addressed. G-049 + G-067 grill MUSTs resolved. ESCALATION-01 (PIPEDA) OPEN for human legal review before assist surface go-live. ---ci--- project: praxis phase: 3 milestone: v0.5 status: complete requirements: covered: [REQ-ASSIST-01, REQ-ASSIST-02, REQ-ASSIST-03, REQ-NFR-ASSIST-01, REQ-NFR-ASSIST-02, REQ-NFR-ASSIST-03, REQ-NFR-ASSIST-04, REQ-IDEATE-01, REQ-IDEATE-02, REQ-IDEATE-03, REQ-IDEATE-04, REQ-IDEATE-05, REQ-IDEATE-06, REQ-IDEATE-07, REQ-IDEATE-08, REQ-IDEATE-09] partial: [] ---/ci---
1076 lines
105 KiB
Markdown
1076 lines
105 KiB
Markdown
# Praxis — v0.5 Execution Plan (Live Assist — On-the-Job Voice Companion)
|
||
|
||
> **Milestone:** v0.5 (Live Assist — on-the-job voice companion)
|
||
> **Phases:** 2 execution phases (P1: assist core + guardrail; P2: integration + tech-debt + NFR measurement) + final phase (P3: review + ship)
|
||
> **Ship:** v0.1.10 (Phase 0, already staged) → v0.1.11 (P1) → v0.1.12 (P2) → v0.1.13 (P3 = v0.5 milestone release)
|
||
> **Status:** plan
|
||
> **Autonomy:** full
|
||
> **Parallelization:** enabled, max 5 concurrent agents
|
||
> **Personas active (5):** lead-developer, voice-engineer (REACTIVATED), backend-engineer (RETAINED), security-engineer (RETAINED), data-engineer (RETAINED)
|
||
> **Personas deactivated (2):** devops-engineer (no deploy changes — v0.4 LXC carries forward), frontend-engineer (PROVISIONAL — assist control surface ~100-150 LOC, below reactivation threshold)
|
||
> **Date:** 2026-08-04
|
||
|
||
---
|
||
|
||
## Phase Split Rationale
|
||
|
||
v0.5 is split into 2 execution phases + final review, following the v0.4 pattern and the ROADMAP's phase structure:
|
||
|
||
- **P1 (Assist Core + Guardrail):** server-side assist session model (shift-bounded), context-binding (path week + scenario tag → ≤150-token prompt), `GuardrailContext.role` extension to `'assist'`, mode-conflict enforcement (assist vs practice mutual exclusivity), `LiveAssistGuardrail` 3-layer enforcement (prompt rules + regex output filter + audit log), guardrail tuning corpus + adversarial bypass test, customer-speech PII policy, incremental audit-log write, `build_assist_pipeline()` (reuses v0.1 services, swaps prompt + adds in-loop guardrail processor), Piper TTS default for assist, warm WebRTC connection manager, tap-to-talk client control, WebRTC reconnect logic. Shippable as `v0.1.11`. Covers: REQ-ASSIST-01, REQ-ASSIST-02, REQ-ASSIST-03, REQ-NFR-ASSIST-02, REQ-NFR-ASSIST-03, REQ-NFR-ASSIST-04, REQ-IDEATE-01, REQ-IDEATE-02, REQ-IDEATE-03, REQ-IDEATE-05, REQ-IDEATE-08, REQ-IDEATE-09.
|
||
- **P2 (Integration + Tech-Debt + NFR Measurement):** cohort aggregation assist metrics (no schema change, new metric strings), assist per-turn cost tracking + C-3 budget check, v0.4 P1+ tech-debt wave (8 findings — aggregation cache persistence, cookie-secret length validation, set_credential_status enum/f-string SQL, etc.), NFR measurement (p95 assist-turn latency ≤650ms pilot tolerance per D-072, guardrail FP <5% / FN measured + trended nightly). Shippable as `v0.1.12`. Covers: REQ-NFR-ASSIST-01, REQ-IDEATE-04, REQ-IDEATE-06, REQ-IDEATE-07.
|
||
- **P3 (Final — Review + Ship):** multi-persona review, audit, merge to main, milestone release `v0.1.13` = v0.5.
|
||
|
||
**Why 2 phases (not 1):** v0.5 has 16 active requirements (3 ASSIST + 4 NFR + 9 IDEATE). The scope decomposes into 33 tasks across 11 slices. A single phase would exceed the ≤35-tasks-per-phase guideline (v0.4 had 52 tasks across 2 phases; v0.3 had ~40 in one). The natural split is: P1 = the assist voice loop + guardrail (the safety-critical, on-voice-path surface — 12 REQs, 24 tasks), P2 = integration + measurement + tech-debt (the operator-facing + hardening surface — 4 REQs, 9 tasks). P1 is independently shippable (a learner can start a shift, invoke assist via tap-to-talk, get coaching with guardrails, end the shift). P2 layers on operator visibility (assist metrics in the cohort dashboard), cost tracking, NFR measurement, and the tech-debt wave.
|
||
|
||
**Why tap-to-talk only in v0.5 (D-071 binding):** The orchestrator's instruction is binding: v0.5 ships tap-to-talk only (no wake-word). React-Web (v0.1, D-015) cannot run an Android foreground service for background wake-word detection. Upgrading to React Native or a separate native Android app is a significant scope addition that would blow the v0.5 timeline. Wake-word (D-058, D-064) is deferred to v0.6 with the native client. This simplifies v0.5: the assist foreground service + Porcupine SDK + battery management are all v0.6. v0.5 ships the assist voice loop + guardrails + aggregation + measurement, invocable via tap-to-talk (a button on the existing React web client, or a headset button). This is hands-free enough for a pilot (learner taps a button on a smartwatch or a headset button) and validates the safety-critical guardrail surface (REQ-ASSIST-03) without the client-architecture risk.
|
||
|
||
---
|
||
|
||
## Key Decisions Honored (D-058..D-073 + research)
|
||
|
||
| Decision | Honored in | How |
|
||
|----------|-----------|-----|
|
||
| D-058 (wake-word + tap-to-talk fallback) | SLICE-02 | tap-to-talk fallback only in v0.5 (D-071 binding); wake-word deferred to v0.6 |
|
||
| D-059 (learner declares context + server reads SQLite path week) | SLICE-01 | AssistContextBinder loads `progress.current_week` from SQLite + scenario tag → ≤150-token context string |
|
||
| D-060 (3-layer guardrail: prompt rules + output filter + audit log) | SLICE-03, SLICE-04 | LiveAssistGuardrail: coaching-mode system prompt + regex output filter + turns table guardrail_verdict |
|
||
| D-061 (latency budget, shared pipeline, ≤30s assist turns) | SLICE-05, SLICE-09 | build_assist_pipeline reuses v0.1 services; ≤150-token prompt (D-066); Piper TTS (D-065); p95 measurement in P2 |
|
||
| D-062 (shift-bounded sessions, session_type=assist) | SLICE-01, SLICE-07 | SessionRecorder gains session_type field; assist shifts fire aggregation hook on shift-end |
|
||
| D-063 (assist does not update mastery) | SLICE-01 | `schedule_mastery=False` for assist shifts; mastery flow is practice-only |
|
||
| D-064 (Porcupine built-in WW + Vosk fallback) | DEFERRED to v0.6 | tap-to-talk only in v0.5 (D-071) |
|
||
| D-065 (Piper TTS for assist, Cartesia for practice) | SLICE-05 | build_assist_pipeline uses Piper by default; PRAXIS_ASSIST_TTS env var |
|
||
| D-066 (≤150-token assist system prompt) | SLICE-01, SLICE-05 | AssistContextBinder constructs terse context; coaching instruction ~80 tokens + context ~50 + conciseness ~20 |
|
||
| D-067 (warm WebRTC per shift) | SLICE-05, SLICE-06 | WarmWebRTCManager opens connection at shift start, 30s heartbeat, closed at shift-end |
|
||
| D-068 (regex output filter + one retry + canned fallback) | SLICE-03 | DIRECT_SCRIPT_RE + IMPERATIVE_RE + FALSE_AUTHORITY_RE + IMPERSONATION_RE; COACHING_QUESTION_RE allow; one retry + canned fallback |
|
||
| D-069 (auto-end after 8h) | SLICE-02 | `PRAXIS_ASSIST_MAX_SHIFT_HOURS=8` env var; shift-end hook fires on auto-end |
|
||
| D-070 (consent disclosure) | SLICE-02 | foreground-service notification + learner-facing disclosure at shift start |
|
||
| D-071 (tap-to-talk only in v0.5) | SLICE-02, SLICE-06 | no wake-word in v0.5; tap-to-talk client control; wake-word deferred to v0.6 |
|
||
| D-072 (≤650ms pilot tolerance for C-8) | SLICE-09 | p95 assist-turn latency ≤650ms pilot target; <600ms hardening deferred to v0.6 |
|
||
| D-073 (Canada consent law review) | SLICE-02 | legal review flag documented; consent disclosure implemented regardless |
|
||
|
||
---
|
||
|
||
## Binding Constraints (from orchestrator)
|
||
|
||
- **D-071 is binding:** tap-to-talk only in v0.5 (no wake-word — React-Web can't do foreground service). The wake-word is v0.6.
|
||
- **D-063 is binding:** assist does NOT update mastery (`schedule_mastery=False`).
|
||
- **D-072 is binding:** ≤650ms pilot tolerance for C-8.
|
||
- **The v0.1 voice pipeline (Pipecat + Deepgram + Cartesia + Ollama Cloud) is FIXED:** v0.5 reuses it via `build_assist_pipeline()`, not a rewrite.
|
||
- **The plan must cover ALL 16 active requirements** (3 ASSIST + 4 NFR + 9 IDEATE).
|
||
|
||
---
|
||
|
||
# Phase 1 — Assist Core + Guardrail
|
||
|
||
**Branch:** `phase/01-assist-core-guardrail` → merged to `milestone/v0.5-live-assist`
|
||
**Ship:** `v0.1.11` (patch release, feature milestone type)
|
||
**REQ-IDs covered:** REQ-ASSIST-01, REQ-ASSIST-02, REQ-ASSIST-03, REQ-NFR-ASSIST-02, REQ-NFR-ASSIST-03, REQ-NFR-ASSIST-04, REQ-IDEATE-01, REQ-IDEATE-02, REQ-IDEATE-03, REQ-IDEATE-05, REQ-IDEATE-08, REQ-IDEATE-09
|
||
**Slices:** 8 vertical slices in 4 waves
|
||
**Total tasks:** 24
|
||
|
||
| Wave | Slices | Parallel slots | Description |
|
||
|------|--------|----------------|-------------|
|
||
| 1 | SLICE-01, SLICE-02 | 2 | Server-side assist session model + context-binding + mode-conflict guard (SLICE-01) + assist session API + shift lifecycle + tap-to-talk client control + consent disclosure (SLICE-02) — disjoint file territories: server/assist/context.py + server/assist/session.py + db/migrations/ vs server/assist/routes.py + client/ |
|
||
| 2 | SLICE-03, SLICE-04 | 2 | LiveAssistGuardrail 3-layer enforcement + GuardrailContext.role extension (SLICE-03) + guardrail tuning corpus + adversarial bypass test + customer-speech PII policy (SLICE-04) — disjoint: server/guardrails/live_assist.py + server/services/base.py vs tests/ + server/assist/pii_policy.py |
|
||
| 3 | SLICE-05, SLICE-06 | 2 | build_assist_pipeline + in-loop guardrail processor + Piper TTS default (SLICE-05) + warm WebRTC connection manager + reconnect logic (SLICE-06) — disjoint: server/assist/pipeline.py + server/pipeline.py vs server/assist/webrtc.py |
|
||
| 4 | SLICE-07, SLICE-08 | 2 | P1 integration — __main__.py wiring (assist routes + pipeline + WebRTC) + SessionRecorder extension (SLICE-07) + P1 integration tests + incremental audit-log write (SLICE-08) |
|
||
|
||
### Wave dependency graph (P1)
|
||
|
||
```
|
||
Wave 1 ──────────────────────────────────────────────────────────────
|
||
SLICE-01 (assist session model + context-binding + mode-conflict)
|
||
SLICE-02 (assist session API + shift lifecycle + tap-to-talk + consent)
|
||
│
|
||
▼
|
||
Wave 2 ──────────────────────────────────────────────────────────────
|
||
SLICE-03 (LiveAssistGuardrail 3-layer + GuardrailContext.role 'assist') ← depends on SLICE-01 (session model)
|
||
SLICE-04 (guardrail tuning corpus + adversarial test + PII policy) ← depends on SLICE-03 (guardrail interface)
|
||
│
|
||
▼
|
||
Wave 3 ──────────────────────────────────────────────────────────────
|
||
SLICE-05 (build_assist_pipeline + in-loop guardrail + Piper TTS) ← depends on SLICE-03 (guardrail), SLICE-01 (context)
|
||
SLICE-06 (warm WebRTC manager + reconnect logic) ← depends on SLICE-02 (shift lifecycle)
|
||
│
|
||
▼
|
||
Wave 4 ──────────────────────────────────────────────────────────────
|
||
SLICE-07 (P1 integration: __main__.py wiring + SessionRecorder extension) ← depends on SLICE-05, SLICE-06
|
||
SLICE-08 (P1 integration tests + incremental audit-log write) ← depends on SLICE-07
|
||
```
|
||
|
||
### Persona load distribution (P1)
|
||
|
||
| Persona | Tasks | Primary territory |
|
||
|---------|-------|-------------------|
|
||
| lead-developer | 3 | integration orchestration, build_assist_pipeline design decision, C-8 latency tension |
|
||
| voice-engineer | 7 | build_assist_pipeline, in-loop guardrail processor, warm WebRTC, reconnect logic, tap-to-talk client, latency tuning |
|
||
| backend-engineer | 6 | assist session API, context-binding, SessionRecorder extension, mode-conflict guard, __main__.py wiring |
|
||
| security-engineer | 6 | LiveAssistGuardrail 3-layer, guardrail tuning corpus, adversarial test, PII policy, consent disclosure |
|
||
| data-engineer | 2 | turns table guardrail_verdict migration, assist session_type field |
|
||
|
||
---
|
||
|
||
## SLICE-01: Assist Session Model + Context-Binding + Mode-Conflict (W1)
|
||
|
||
- **Goal:** Build the server-side assist session foundation: the `AssistContextBinder` (loads path week + scenario tag + learner theta from SQLite → ≤150-token context string per D-059/D-066), the `AssistSession` model (shift-bounded per D-062, `session_type='assist'`, `schedule_mastery=False` per D-063), and the mode-conflict guard (assist vs practice mutual exclusivity per REQ-IDEATE-03).
|
||
- **REQ-IDs covered:** REQ-ASSIST-02 (context-aware), REQ-NFR-ASSIST-04 (shift-bounded session model), REQ-IDEATE-03 (mode-conflict enforcement)
|
||
- **Wave:** 1
|
||
- **Dependencies:** none
|
||
- **Primary persona:** backend-engineer
|
||
- **Supporting personas:** data-engineer (SQLite migration for session_type field), lead-developer (context-binding design)
|
||
|
||
### Tasks
|
||
|
||
#### TASK-01-01 — SQLite migration: sessions table gains session_type column
|
||
- **Persona:** data-engineer
|
||
- **File:** `db/migrations/0004_assist.sql` (new)
|
||
- **Content:** Additive migration (no breaking change to existing practice sessions):
|
||
```sql
|
||
-- v0.5 Live Assist — assist session model (D-062, REQ-NFR-ASSIST-04).
|
||
-- session_type: 'practice' (default, existing) | 'assist' (new v0.5).
|
||
-- guardrail_verdict_json: per-turn guardrail verdict (D-060 layer 3, REQ-IDEATE-09).
|
||
ALTER TABLE sessions ADD COLUMN session_type TEXT NOT NULL DEFAULT 'practice';
|
||
ALTER TABLE turns ADD COLUMN guardrail_verdict_json TEXT;
|
||
-- Index for mode-conflict check (REQ-IDEATE-03): find active sessions by type.
|
||
CREATE INDEX IF NOT EXISTS idx_sessions_active_by_type
|
||
ON sessions (learner_id, session_type, ended_at);
|
||
```
|
||
The `session_type` column defaults to `'practice'` so all existing v0.1-v0.4 sessions are unaffected. The `guardrail_verdict_json` column is nullable (only assist turns populate it). The index supports the mode-conflict guard (TASK-01-05).
|
||
- **Acceptance criteria:** `apply_migrations()` applies 0004 successfully. Existing practice sessions have `session_type='practice'`. `turns.guardrail_verdict_json` is NULL for existing turns. The index exists. Re-running migrations is idempotent (no error).
|
||
|
||
#### TASK-01-02 — AssistContextBinder (D-059, D-066)
|
||
- **Persona:** backend-engineer
|
||
- **File:** `server/assist/context.py` (new)
|
||
- **Content:** `class AssistContextBinder` — loads context for an assist shift from SQLite + scenario library:
|
||
- `__init__(self, store: PraxisStore)` — takes the existing SQLite store.
|
||
- `async def bind(self, learner_id: str, path_slug: str, scenario_tag: str) -> AssistContext` — reads `progress.current_week` from `store.get_progress(learner_id, path_slug)`, reads `theta` from `store.get_ability(learner_id, path_slug)`, loads the scenario's `rubric_criteria` from the scenario library (if the tag matches a known scenario), and constructs a terse ≤150-token context string per D-066:
|
||
```
|
||
[Coaching instruction — ~80 tokens, fixed]
|
||
You are a live coaching AI in the learner's ear during a real customer interaction.
|
||
Coach, do not do the learner's job. Ask guiding questions; never give the answer.
|
||
Never speak on behalf of the learner. Never claim authority you don't have.
|
||
Keep responses to 1-3 sentences for voice.
|
||
|
||
[Context-binding — ~50 tokens, per shift]
|
||
Week {current_week}: {week_focus}. Scenario: {scenario_tag}.
|
||
Learner theta: {theta:.1f}. Coaching focus: {top_rubric_criterion}.
|
||
|
||
[Voice-conciseness — ~20 tokens, fixed]
|
||
Be brief. The customer is waiting.
|
||
```
|
||
- `@dataclass AssistContext` — fields: `system_prompt: str` (the full ≤150-token prompt), `current_week: int`, `scenario_tag: str`, `theta: float`, `coaching_focus: str`, `path_slug: str`.
|
||
- The `week_focus` is derived from the path YAML (`paths/customer_service.yaml` — each week has a `focus` field). If the path YAML doesn't have a focus for the week, defaults to `"Week {n}"`.
|
||
- The `top_rubric_criterion` is the rubric criterion with the lowest mean score for this learner (from `mastery_gate_events` or a fallback to the first criterion). If no rubric data, defaults to `"empathy + resolution-concreteness"`.
|
||
- **Token budget assertion:** `len(system_prompt.split()) <= 200` (rough word≈token check; the real check is in the pipeline test). If the prompt exceeds the budget, log WARNING + truncate the context-binding section.
|
||
- **Acceptance criteria:** `AssistContextBinder.bind()` returns an `AssistContext` with a system prompt ≤200 words. The prompt contains the coaching instruction, context-binding (week, scenario, theta, focus), and voice-conciseness sections. Missing learner state (no progress row, no theta) → defaults are used (week=1, theta=0.0, focus=generic). The prompt is never empty.
|
||
|
||
#### TASK-01-03 — AssistSession model (D-062, D-063)
|
||
- **Persona:** backend-engineer
|
||
- **File:** `server/assist/session.py` (new)
|
||
- **Content:** `class AssistSession` — the shift-bounded session model:
|
||
- `__init__(self, store: PraxisStore, learner_id: str, context: AssistContext)` — holds the shift state.
|
||
- `session_type: str = "assist"` — the field that distinguishes assist from practice (D-062).
|
||
- `async def start(self) -> str` — creates a sessions row with `session_type='assist'`, `scenario_id=f"assist:{context.scenario_tag}"`. Returns the session id.
|
||
- `async def log_assist_turn(self, asr_text: str, tts_text: str, guardrail_verdict: dict | None, latency_ms: float | None)` — logs a turn to the `turns` table with `guardrail_verdict_json` populated (D-060 layer 3, REQ-IDEATE-09). Increments the turn counter.
|
||
- `async def end(self, outcome: str = "completed") -> dict` — ends the shift: updates the sessions row (`ended_at`, `outcome`), builds the `session_outcome` dict (with `session_type='assist'`, `assist_turn_count`, `guardrail_blocks`), fires the cohort aggregation hook (if `pg_store` is available). **Does NOT call `run_mastery_flow()`** — `schedule_mastery=False` per D-063. Returns the session_outcome dict.
|
||
- `turn_count: int` — the number of assist turns in this shift.
|
||
- `guardrail_block_count: int` — the number of turns where the output filter blocked.
|
||
- `shift_started_at: datetime` — for the 8h auto-end check (D-069).
|
||
- **Acceptance criteria:** `AssistSession.start()` creates a sessions row with `session_type='assist'`. `log_assist_turn()` writes to the turns table with `guardrail_verdict_json`. `end()` updates the session row + fires the aggregation hook. `run_mastery_flow()` is NEVER called (D-063 — verified by code inspection: no `schedule_mastery=True` path). The `session_outcome` dict contains `session_type='assist'`, `assist_turn_count`, `guardrail_blocks`.
|
||
|
||
#### TASK-01-04 — AssistSession SQLite store methods
|
||
- **Persona:** backend-engineer
|
||
- **File:** `db/store.py` (extend)
|
||
- **Content:** Add methods to `PraxisStore` to support assist sessions:
|
||
- `async def start_session_typed(self, learner_id: str, scenario_id: str, session_type: str = "practice") -> str` — like `start_session` but sets the `session_type` column. Existing `start_session()` calls this with `session_type='practice'` (backward compat).
|
||
- `async def log_turn_with_verdict(self, session_id: str, seq: int, role: str, asr_text: str | None, tts_text: str | None, latency_ms: float | None, guardrail_verdict_json: str | None)` — like `log_turn` but also sets `guardrail_verdict_json`. Existing `log_turn()` calls this with `guardrail_verdict_json=None` (backward compat).
|
||
- `async def get_active_session(self, learner_id: str, session_type: str) -> dict | None` — finds an active (not ended) session for the learner of the given type. Uses the `idx_sessions_active_by_type` index. Returns the session row or None. This is the mode-conflict check (TASK-01-05).
|
||
- `async def end_session_assist(self, session_id: str, outcome: str, turn_count: int, guardrail_block_count: int)` — like `end_session` but also sets `outcome` (completed/abandoned/auto_ended) + the assist-specific fields. The existing `end_session()` is unchanged for practice sessions.
|
||
- **Acceptance criteria:** `start_session_typed('learner-1', 'assist:refund', 'assist')` creates a row with `session_type='assist'`. `log_turn_with_verdict()` writes `guardrail_verdict_json`. `get_active_session('learner-1', 'practice')` returns an active practice session or None. `end_session_assist()` updates the row. Existing practice-session tests still pass (backward compat).
|
||
|
||
#### TASK-01-05 — Mode-conflict guard (REQ-IDEATE-03)
|
||
- **Persona:** backend-engineer
|
||
- **File:** `server/assist/mode_conflict.py` (new)
|
||
- **Content:** `async def enforce_mutual_exclusivity(store: PraxisStore, learner_id: str, requested_type: str) -> None` — checks that the learner doesn't have an active session of the *other* type:
|
||
- If `requested_type == 'assist'`: check `store.get_active_session(learner_id, 'practice')` — if a practice session is active, raise `ModeConflictError("Cannot start assist shift: a practice session is active. End the practice session first.")`.
|
||
- If `requested_type == 'practice'`: check `store.get_active_session(learner_id, 'assist')` — if an assist shift is active, raise `ModeConflictError("Cannot start practice session: an assist shift is active. End the shift first.")`.
|
||
- `class ModeConflictError(Exception)` — the exception type.
|
||
- This is the server-side guard (D-061 states assist is a separate mode, not concurrent). The existing `/pipecat/webrtc` endpoint (practice) will call this with `requested_type='practice'`; the new `/api/assist/shift/start` endpoint (SLICE-02) will call this with `requested_type='assist'`.
|
||
- **Acceptance criteria:** Starting an assist shift while a practice session is active → `ModeConflictError`. Starting a practice session while an assist shift is active → `ModeConflictError`. Starting an assist shift with no active practice session → no error. Starting a practice session with no active assist shift → no error. Ended sessions don't trigger the conflict (only active sessions count).
|
||
|
||
#### TASK-01-06 — Assist session model unit tests
|
||
- **Persona:** backend-engineer
|
||
- **File:** `tests/test_assist_session.py` (new)
|
||
- **Content:** Unit tests (using a temp SQLite DB via `PraxisStore`):
|
||
- `AssistContextBinder.bind()` returns a context with a ≤200-word system prompt. Missing learner state → defaults. Prompt contains coaching instruction + context-binding + conciseness.
|
||
- `AssistSession.start()` creates a row with `session_type='assist'`. `log_assist_turn()` writes turns with `guardrail_verdict_json`. `end()` updates the row + returns session_outcome with `session_type='assist'`.
|
||
- D-063: `AssistSession.end()` does NOT call `run_mastery_flow()` (verified by mocking + asserting no mastery task created).
|
||
- Mode-conflict: starting assist during active practice → `ModeConflictError`. Starting practice during active assist → `ModeConflictError`. No conflict when no active session of the other type.
|
||
- Existing practice-session tests still pass (backward compat with `start_session` / `log_turn` / `end_session`).
|
||
- **Acceptance criteria:** All tests pass. D-063 explicitly verified (no mastery flow on assist). Mode-conflict enforced in both directions. Backward compat with practice sessions.
|
||
|
||
---
|
||
|
||
## SLICE-02: Assist Session API + Shift Lifecycle + Tap-to-Talk + Consent (W1)
|
||
|
||
- **Goal:** Build the assist session API endpoints (`POST /api/assist/shift/start`, `POST /api/assist/shift/end`), the shift lifecycle (8h auto-end per D-069), the tap-to-talk client control (D-071 — no wake-word in v0.5), and the consent disclosure (D-070).
|
||
- **REQ-IDs covered:** REQ-ASSIST-01 (hands-free voice companion — tap-to-talk invocation), REQ-NFR-ASSIST-02 (hands-free invocation — tap-to-talk only in v0.5 per D-071), REQ-NFR-ASSIST-04 (shift-bounded, auto-end after 8h)
|
||
- **Wave:** 1
|
||
- **Dependencies:** none (parallel with SLICE-01 — disjoint files: `server/assist/routes.py` + `client/` vs `server/assist/context.py` + `server/assist/session.py` + `db/`)
|
||
- **Primary persona:** backend-engineer
|
||
- **Supporting personas:** voice-engineer (tap-to-talk client control), security-engineer (consent disclosure), lead-developer (shift lifecycle design)
|
||
|
||
### Tasks
|
||
|
||
#### TASK-02-01 — Assist session API routes
|
||
- **Persona:** backend-engineer
|
||
- **File:** `server/assist/routes.py` (new)
|
||
- **Content:** `APIRouter(prefix="/api/assist")` with:
|
||
- `POST /api/assist/shift/start` — body: `{path_slug: str, scenario_tag: str}`. Calls `enforce_mutual_exclusivity(store, learner_id, 'assist')` (TASK-01-05). Calls `AssistContextBinder.bind(learner_id, path_slug, scenario_tag)` (TASK-01-02). Creates an `AssistSession`, calls `start()`. Returns `{shift_id, context: {current_week, scenario_tag, coaching_focus}}`. The warm WebRTC connection is opened separately (SLICE-06) — this endpoint just creates the shift session row + binds context.
|
||
- `POST /api/assist/shift/end` — body: `{shift_id: str, outcome: str = "completed"}`. Calls `AssistSession.end(outcome)`. Returns `{ok: true, turn_count, guardrail_block_count}`. The warm WebRTC connection is closed separately (SLICE-06).
|
||
- `GET /api/assist/shift/active` — returns the active assist shift for the learner, or `{active: false}`. Used by the client to resume a shift after a page reload.
|
||
- All routes use the hardcoded `learner-1` (D-007 — no learner auth in v0.5). No operator auth on assist routes (these are learner-facing, not operator-facing).
|
||
- Routes are registered BEFORE the StaticFiles mount (routes-before-static-mount constraint, carry-forward).
|
||
- **Acceptance criteria:** `POST /api/assist/shift/start` with valid body → 200 + shift_id + context. Starting a shift during an active practice session → 409 Conflict (ModeConflictError). `POST /api/assist/shift/end` → 200 + turn_count. `GET /api/assist/shift/active` returns the active shift or `{active: false}`. Routes return JSON (not index.html — matched before StaticFiles).
|
||
|
||
#### TASK-02-02 — Shift lifecycle + 8h auto-end (D-069)
|
||
- **Persona:** backend-engineer
|
||
- **File:** `server/assist/lifecycle.py` (new)
|
||
- **Content:** `class ShiftLifecycleManager` — manages the 8h auto-end:
|
||
- `__init__(self, store: PraxisStore, max_shift_hours: int = 8)` — reads `PRAXIS_ASSIST_MAX_SHIFT_HOURS` env var (default 8 per D-069).
|
||
- `async def check_auto_end(self) -> list[str]` — queries all active assist sessions, finds any where `started_at < now - max_shift_hours`. For each, calls `AssistSession.end(outcome='auto_ended')`. Returns the list of auto-ended shift ids.
|
||
- `async def start_monitor(self)` — starts an asyncio task that runs `check_auto_end()` every 5 minutes (a 5-min granularity is sufficient — the 8h boundary is not latency-critical). The task is cancelled on shutdown.
|
||
- `async def stop_monitor(self)` — cancels the monitor task.
|
||
- R-ASSIST-11 mitigation: auto-end after 8h closes the shift cleanly, fires the aggregation hook, and releases the WebRTC connection (SLICE-06 closes the connection on shift-end).
|
||
- **Acceptance criteria:** A shift started 8h+ ago is auto-ended on the next `check_auto_end()` run. The outcome is `'auto_ended'`. The aggregation hook fires. The monitor runs every 5 minutes. `stop_monitor()` cancels the task cleanly.
|
||
|
||
#### TASK-02-03 — Tap-to-talk client control (D-071)
|
||
- **Persona:** voice-engineer
|
||
- **File:** `client/src/AssistControl.tsx` (new)
|
||
- **Content:** A minimal React component (~100-150 LOC — below the frontend-engineer reactivation threshold per PERSONAS.md §7.2) that provides the assist control surface:
|
||
- "Start Shift" button → `POST /api/assist/shift/start` with the selected path week + scenario tag. On success, opens the warm WebRTC connection (SLICE-06) + shows the shift-active UI.
|
||
- "End Shift" button → `POST /api/assist/shift/end`. On success, closes the WebRTC connection + shows the shift summary (turn count, guardrail blocks).
|
||
- Context declaration: a dropdown for path week (auto-filled from `GET /api/assist/shift/active` or the learner's `progress.current_week`) + a text input or dropdown for scenario tag (e.g., "damaged-product refund", "escalation", "policy exception").
|
||
- Tap-to-talk: a "press to talk" button (hold to speak, release to send) — the learner taps + holds the button, speaks, releases. This is the hands-free invocation in v0.5 (D-071 — no wake-word). The button sends audio over the warm WebRTC connection.
|
||
- Shift-active UI: shows "Shift active — Week {n}, {scenario_tag}" + a "Tap to talk" button + a "End Shift" button. Minimal CSS (reuse `App.css` patterns).
|
||
- Consent disclosure banner: "Praxis Assist is on — those around you may be recorded by your mic" (D-070) — shown when the shift starts, dismissed by the learner.
|
||
- Routed at `/assist` (added to `App.tsx` route switch — TASK-07-02).
|
||
- **Acceptance criteria:** "Start Shift" creates a shift + opens WebRTC. "End Shift" closes the shift + WebRTC. Context declaration (week + scenario tag) is sent to the server. Tap-to-talk button sends audio over WebRTC. Consent disclosure banner appears on shift start. Component is ~100-150 LOC (below frontend-engineer threshold). `npm run build` succeeds.
|
||
|
||
#### TASK-02-04 — Consent disclosure (D-070, D-073)
|
||
- **Persona:** security-engineer
|
||
- **File:** `server/assist/consent.py` (new)
|
||
- **Content:** `CONSENT_DISCLOSURE_TEXT` — the learner-facing disclosure:
|
||
```
|
||
"Praxis Assist is on. Your mic is active for coaching. Those around you may be
|
||
recorded by your microphone. You are responsible for following your local
|
||
consent laws. End the shift to stop recording."
|
||
```
|
||
- `def get_consent_disclosure() -> str` — returns the disclosure text.
|
||
- The disclosure is surfaced to the client in the `POST /api/assist/shift/start` response (`{shift_id, context, consent_disclosure}`) so the client can display it (TASK-02-03).
|
||
- D-073 flag: the disclosure text includes a note about local consent laws. The legal review of Canada PIPEDA + one-party/two-party consent (R-ASSIST-08) is documented as an open question for the orchestrator — the disclosure is implemented regardless (it's ethically required even if the legal review is pending).
|
||
- The disclosure is also logged in the sessions row (as part of the shift-start metadata) for audit purposes.
|
||
- **Acceptance criteria:** `get_consent_disclosure()` returns the disclosure text. The text mentions mic active, those around you may be recorded, local consent laws, and how to stop. The `/api/assist/shift/start` response includes `consent_disclosure`. The disclosure is logged at shift start.
|
||
|
||
#### TASK-02-05 — Assist session API + lifecycle unit tests
|
||
- **Persona:** backend-engineer
|
||
- **File:** `tests/test_assist_routes.py` (new)
|
||
- **Content:** Unit tests (using FastAPI TestClient + temp SQLite):
|
||
- `POST /api/assist/shift/start` → 200 + shift_id + context + consent_disclosure.
|
||
- Starting a shift during an active practice session → 409 Conflict.
|
||
- `POST /api/assist/shift/end` → 200 + turn_count + guardrail_block_count.
|
||
- `GET /api/assist/shift/active` → returns the active shift or `{active: false}`.
|
||
- 8h auto-end: a shift started 9h ago is auto-ended on the next `check_auto_end()` run (mock datetime).
|
||
- Consent disclosure is in the start response.
|
||
- Routes return JSON (not index.html).
|
||
- **Acceptance criteria:** All tests pass. Mode-conflict enforced (409). Auto-end works (mock time). Consent disclosure present. Routes matched before StaticFiles.
|
||
|
||
---
|
||
|
||
## SLICE-03: LiveAssistGuardrail 3-Layer + GuardrailContext.role Extension (W2)
|
||
|
||
- **Goal:** Implement the `LiveAssistGuardrail` (server/guardrails/live_assist.py) — the 3-layer guardrail for REQ-ASSIST-03 (coaches not does, never lies to real customers). Extend `GuardrailContext.role` to include `'assist'` (REQ-IDEATE-02). The output filter regex patterns (DIRECT_SCRIPT_RE, IMPERATIVE_RE, FALSE_AUTHORITY_RE, IMPERSONATION_RE, COACHING_QUESTION_RE) with one retry + canned fallback (D-068).
|
||
- **REQ-IDs covered:** REQ-ASSIST-03 (guardrails: coaches not does), REQ-NFR-ASSIST-03 (3-layer guardrail enforcement), REQ-IDEATE-02 (GuardrailContext.role 'assist' extension)
|
||
- **Wave:** 2
|
||
- **Dependencies:** SLICE-01 (assist session model — the guardrail is invoked per assist turn)
|
||
- **Primary persona:** security-engineer
|
||
- **Supporting personas:** backend-engineer (in-loop processor integration in SLICE-05)
|
||
|
||
### Tasks
|
||
|
||
#### TASK-03-01 — GuardrailContext.role extension to 'assist' (REQ-IDEATE-02)
|
||
- **Persona:** security-engineer
|
||
- **File:** `server/services/base.py` (extend)
|
||
- **Content:** Extend the `GuardrailContext.role` Literal to include `'assist'`:
|
||
```python
|
||
role: Literal["system", "user", "assistant", "debrief", "assist"] = "user"
|
||
```
|
||
This is the interface gap from REQ-IDEATE-02: the `LiveAssistGuardrail` hits the `GuardrailContext` with `role='assist'`, which the current Literal doesn't allow (mypy/IDE error). The extension is a one-line change + a comment documenting the new role.
|
||
- Also extend the `GuardrailVerdict.category` comment to include the new assist categories: `blocked_direct_script`, `blocked_imperative`, `blocked_false_authority`, `blocked_impersonation`, `coaching`, `neutral`.
|
||
- **Acceptance criteria:** `GuardrailContext(role='assist')` is valid (no mypy error). Existing roles (`system`, `user`, `assistant`, `debrief`) still work. `GuardrailVerdict` category field accepts the new assist categories.
|
||
|
||
#### TASK-03-02 — LiveAssistGuardrail — Layer 1 (coaching-mode system prompt)
|
||
- **Persona:** security-engineer
|
||
- **File:** `server/guardrails/live_assist.py` (new)
|
||
- **Content:** `class LiveAssistGuardrail(Guardrail)` — the 3-layer guardrail. Layer 1 is the coaching-mode system prompt (already constructed by `AssistContextBinder` in SLICE-01 — the guardrail exposes it as `session_start_disclaimer`):
|
||
- `name = "live_assist"`
|
||
- `@property session_start_disclaimer` — returns the coaching instruction (~80 tokens). This is the same text as the coaching instruction in the assist system prompt (D-066). The disclaimer is NOT played as audio at shift start (unlike the practice disclaimer) — it's part of the system prompt. The `session_start_disclaimer` property is retained for interface compatibility but is used differently in assist mode (it's the system prompt prefix, not a spoken disclaimer).
|
||
- The consent disclosure (TASK-02-04) is separate from the guardrail disclaimer — the consent disclosure is shown to the learner in the client UI; the guardrail disclaimer is the LLM system prompt.
|
||
- **Acceptance criteria:** `LiveAssistGuardrail().session_start_disclaimer` returns the coaching instruction text. The text instructs the LLM to ask guiding questions, never give the answer, never claim false authority. The guardrail implements the `Guardrail` ABC.
|
||
|
||
#### TASK-03-03 — LiveAssistGuardrail — Layer 2 (regex output filter, D-068)
|
||
- **Persona:** security-engineer
|
||
- **File:** `server/guardrails/live_assist.py` (extend)
|
||
- **Content:** Layer 2 — the regex output filter, run on the LLM response before TTS:
|
||
```python
|
||
DIRECT_SCRIPT_RE = re.compile(
|
||
r"\b(you should (say|tell|respond with|reply)|"
|
||
r"say (this|the following)|tell (the |a )?customer|"
|
||
r"respond with|reply with|here'?s what to say|"
|
||
r"the (right |correct |best )?answer is|"
|
||
r"what you (should|need to|must) (say|do) is)\b",
|
||
re.IGNORECASE,
|
||
)
|
||
IMPERATIVE_RE = re.compile(
|
||
r"\b(escalate to|transfer to|offer a refund of|apologize (by|with)|"
|
||
r"give them|promise them|tell them you)\b",
|
||
re.IGNORECASE,
|
||
)
|
||
FALSE_AUTHORITY_RE = re.compile(
|
||
r"\b(I (am|'?m) (your |a )?(manager|supervisor|the company|authorized|"
|
||
r"a lawyer|a doctor|regulator)|"
|
||
r"on behalf of (the company|management)|"
|
||
r"I (can|will) (authorize|approve|guarantee))\b",
|
||
re.IGNORECASE,
|
||
)
|
||
IMPERSONATION_RE = re.compile(
|
||
r"\b(I (?:work|am employed) (?:at|for|with))\b.*\b(Inc\.|Corp\.|LLC|Ltd\.|"
|
||
r"Amazon|Apple|Google|Microsoft|Walmart|Costco|Telus|Rogers|Bell|Shopify)\b",
|
||
re.IGNORECASE,
|
||
)
|
||
COACHING_QUESTION_RE = re.compile(
|
||
r"\b(what (do you|could you|might you)|"
|
||
r"how (could|might|would|do) you|"
|
||
r"what'?s (your|the) (goal|approach|next step)|"
|
||
r"how (does|do) you (feel|think)|"
|
||
r"what (would|might) happen if|"
|
||
r"can you (think of|identify|name)|"
|
||
r"have you considered)\b",
|
||
re.IGNORECASE,
|
||
)
|
||
CANNED_FALLBACK = "Think about what the customer needs right now. What's your next step?"
|
||
RETRY_INSTRUCTION = "Your last response gave a direct answer. Rephrase as a coaching question."
|
||
```
|
||
- `async def check(self, text: str, context: GuardrailContext | None = None) -> GuardrailVerdict`:
|
||
1. Run `DIRECT_SCRIPT_RE` + `IMPERATIVE_RE` on the text. If hit → `GuardrailVerdict(allowed=False, reason="blocked: direct-answer pattern", category="blocked_direct_script", filtered_text=CANNED_FALLBACK)`. The caller (in-loop processor, SLICE-05) retries once with `RETRY_INSTRUCTION`; if the retry also blocks, uses `CANNED_FALLBACK`.
|
||
2. Run `FALSE_AUTHORITY_RE` + `IMPERSONATION_RE`. If hit → `GuardrailVerdict(allowed=False, reason="blocked: false-authority/impersonation", category="blocked_false_authority", filtered_text=CANNED_FALLBACK)`. **No retry** for false-authority/impersonation (hard violations — D-068).
|
||
3. If no hit → check `COACHING_QUESTION_RE`. If hit → `GuardrailVerdict(allowed=True, reason="coaching question", category="coaching")`. If no hit → `GuardrailVerdict(allowed=True, reason="neutral", category="neutral")` (allowed but not ideal — log for review).
|
||
- The verdict is logged to the turns table (`guardrail_verdict_json`) by the in-loop processor (SLICE-05) — layer 3.
|
||
- **Acceptance criteria:** Direct-answer patterns ("you should say X", "tell the customer Y", "the answer is Z") → blocked. Imperative patterns ("escalate to", "offer a refund of") → blocked. False-authority ("I am your manager") → blocked (no retry). Impersonation ("I work at Amazon") → blocked (no retry). Coaching questions ("what do you think", "how could you") → allowed (category="coaching"). Neutral text → allowed (category="neutral"). `CANNED_FALLBACK` is returned as `filtered_text` on block.
|
||
|
||
#### TASK-03-04 — LiveAssistGuardrail unit tests
|
||
- **Persona:** security-engineer
|
||
- **File:** `tests/test_live_assist_guardrail.py` (new)
|
||
- **Content:** Unit tests for the `LiveAssistGuardrail`:
|
||
- Direct-answer patterns: "You should say sorry to the customer" → blocked (category="blocked_direct_script"). "Tell the customer you'll refund them" → blocked. "The answer is to offer a exchange" → blocked.
|
||
- Imperative patterns: "Escalate to a supervisor" → blocked. "Offer a refund of $50" → blocked. "Apologize by saying sorry" → blocked.
|
||
- False-authority: "I am your manager" → blocked (no retry). "On behalf of the company, I authorize a refund" → blocked.
|
||
- Impersonation: "I work at Amazon" → blocked.
|
||
- Coaching questions: "What do you think the customer needs?" → allowed (category="coaching"). "How could you acknowledge their frustration?" → allowed.
|
||
- Neutral: "That's a good approach." → allowed (category="neutral").
|
||
- `CANNED_FALLBACK` is returned as `filtered_text` on every block.
|
||
- `GuardrailContext(role='assist')` is accepted (REQ-IDEATE-02).
|
||
- The guardrail is swappable with `CustomerServiceGuardrail` (both implement `Guardrail` ABC — D-019 pluggability).
|
||
- **Acceptance criteria:** All patterns tested. Direct-answer + imperative → blocked + retry eligible. False-authority + impersonation → blocked + no retry. Coaching questions → allowed. Neutral → allowed. `CANNED_FALLBACK` on every block. `role='assist'` accepted.
|
||
|
||
---
|
||
|
||
## SLICE-04: Guardrail Tuning Corpus + Adversarial Bypass Test + PII Policy (W2)
|
||
|
||
- **Goal:** Build the synthetic guardrail tuning corpus (LLM-generate coaching vs direct-answer responses, label, tune the regex — REQ-IDEATE-01), the adversarial bypass test (paraphrased direct answers designed to slip past the regex — REQ-IDEATE-01), and the customer-speech PII policy (REQ-IDEATE-05).
|
||
- **REQ-IDs covered:** REQ-IDEATE-01 (guardrail tuning corpus + adversarial test), REQ-IDEATE-05 (customer-speech PII policy)
|
||
- **Wave:** 2
|
||
- **Dependencies:** SLICE-03 (LiveAssistGuardrail interface — the corpus tunes its regex)
|
||
- **Primary persona:** security-engineer
|
||
- **Supporting personas:** data-engineer (PII policy storage), backend-engineer (PII policy integration)
|
||
|
||
### Tasks
|
||
|
||
#### TASK-04-01 — Synthetic guardrail tuning corpus (REQ-IDEATE-01)
|
||
- **Persona:** security-engineer
|
||
- **File:** `tests/guardrail_corpus.py` (new — test fixture, not shipped code)
|
||
- **Content:** A synthetic corpus of labeled LLM responses for tuning the guardrail regex:
|
||
- `COACHING_RESPONSES: list[dict]` — ~50 coaching responses (labeled `allowed=True`, `category="coaching"` or `"neutral"`). Generated by prompting `gemma4:cloud` with the assist system prompt + a learner situation, collecting responses that ask guiding questions.
|
||
- `DIRECT_ANSWER_RESPONSES: list[dict]` — ~50 direct-answer responses (labeled `allowed=False`, `category="blocked_direct_script"` or `"blocked_imperative"`). Generated by prompting `gemma4:cloud` WITHOUT the coaching instruction (just "help the learner with this customer"), collecting responses that give direct answers.
|
||
- `FALSE_AUTHORITY_RESPONSES: list[dict]` — ~20 false-authority responses (labeled `allowed=False`, `category="blocked_false_authority"`).
|
||
- `ADVERSARIAL_RESPONSES: list[dict]` — ~30 paraphrased direct answers designed to slip past the regex (REQ-IDEATE-01 adversarial test). Examples: "If I were you, I'd start with an apology", "Maybe try saying sorry first", "The customer probably wants to hear you acknowledge the issue" (indirect but still doing the job). These are the false-negative test set.
|
||
- The corpus is a Python module (not a YAML file) so it's importable by tests. Each entry: `{text: str, label: dict}` where `label` is the expected `GuardrailVerdict`.
|
||
- The corpus is generated once (at plan time, by the security-engineer) + committed. It's NOT generated at test time (no LLM calls in CI).
|
||
- **Acceptance criteria:** The corpus has ≥150 total entries (50 coaching + 50 direct-answer + 20 false-authority + 30 adversarial). Each entry has a text + expected verdict. The corpus is importable by tests. No LLM calls at test time.
|
||
|
||
#### TASK-04-02 — Guardrail tuning + adversarial bypass test (REQ-IDEATE-01)
|
||
- **Persona:** security-engineer
|
||
- **File:** `tests/test_guardrail_tuning.py` (new)
|
||
- **Content:** Tests that run the `LiveAssistGuardrail` against the tuning corpus:
|
||
- `test_coaching_responses_allowed()` — all `COACHING_RESPONSES` → `allowed=True`. False-positive rate < 5% (REQ-IDEATE-04 target — at most 2-3 of 50 coaching responses blocked).
|
||
- `test_direct_answer_responses_blocked()` — all `DIRECT_ANSWER_RESPONSES` → `allowed=False`. True-positive rate target ≥ 95% (at most 2-3 of 50 direct answers slip through).
|
||
- `test_false_authority_responses_blocked()` — all `FALSE_AUTHORITY_RESPONSES` → `allowed=False` (100% — false authority is a hard violation).
|
||
- `test_adversarial_responses()` — the adversarial bypass test (REQ-IDEATE-01). Run the `ADVERSARIAL_RESPONSES` through the guardrail. Measure the false-negative rate (paraphrased direct answers that slip past the regex). The test ASSERTS that the false-negative rate is measured + reported (it doesn't assert a specific threshold — the regex can't catch every paraphrase, but the rate must be known + trended). The test logs: `"adversarial false-negative rate: {fn_rate:.1%} ({fn_count}/{total}) — defense-in-depth + post-v0.5 LLM-as-judge mitigates"`.
|
||
- `test_tuning_summary()` — a summary test that prints the FP rate (coaching blocked) + FN rate (direct/adversarial allowed) + the overall accuracy. This is the REQ-IDEATE-04 measurement (false-positive rate < 5% on the tuning corpus).
|
||
- If the FP rate is > 5%, the test FAILS (the regex is over-matching — tune it). If the FN rate on direct answers is > 5%, the test FAILS (the regex is under-matching). The adversarial FN rate is reported but not threshold-gated (it's the residual risk, mitigated by defense-in-depth).
|
||
- **Acceptance criteria:** Coaching responses: FP rate < 5%. Direct-answer responses: FN rate < 5%. False-authority: 100% blocked. Adversarial: FN rate measured + reported (not threshold-gated). Tuning summary prints all rates. If FP > 5% or direct FN > 5%, the test fails (regex needs tuning).
|
||
|
||
#### TASK-04-03 — Customer-speech PII policy (REQ-IDEATE-05)
|
||
- **Persona:** security-engineer
|
||
- **File:** `server/assist/pii_policy.py` (new)
|
||
- **Content:** The PII policy for customer speech in the assist turns audit log (REQ-IDEATE-05 — STRIDE information-disclosure). The ambient mic captures both the learner and the real customer; ASR transcribes both; the turns table stores transcribed text. The customer is a third party — their transcribed speech is third-party PII in SQLite.
|
||
- `CUSTOMER_SPEECH_POLICY: str` — the policy choice. v0.5 chooses option (c) from REQ-IDEATE-05: **document that the audit log contains customer speech + apply consent-disclosure (D-070) + retention limits**. This is the simplest option that preserves the audit trail for the guardrail_block_rate safety signal. Options (a) (strip customer turns) and (b) (store only learner utterances) would break the guardrail audit (the guardrail needs the full turn context to explain why it blocked).
|
||
- `def redact_pii(text: str) -> str` — a lightweight PII redaction for the audit log. Redacts: phone numbers (`\b\d{3}[-.]?\d{3}[-.]?\d{4}\b` → `[PHONE]`), email addresses (`\b[\w.+-]+@[\w-]+\.[\w.-]+\b` → `[EMAIL]`), credit card numbers (`\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b` → `[CARD]`), and SIN-like numbers (`\b\d{3}-\d{3}-\d{3}\b` → `[SIN]`). This is a defense-in-depth measure — the primary protection is the consent disclosure (D-070) + the local SQLite store (not Postgres — no raw PII in the operator tier per D-031).
|
||
- `RETENTION_DAYS: int = 30` — the retention limit for assist turns. A nightly cleanup (in the ShiftLifecycleManager or a separate task) deletes turns older than 30 days. This is documented in the policy.
|
||
- `def get_pii_policy() -> dict` — returns the policy as a dict for documentation: `{policy: "retain with redaction + consent + 30-day retention", redaction_patterns: [...], retention_days: 30, legal_review: "pending — D-073"}`.
|
||
- The redaction is applied in `AssistSession.log_assist_turn()` before writing to the turns table (the `asr_text` is redacted before storage).
|
||
- **Acceptance criteria:** `redact_pii()` redacts phone numbers, emails, card numbers, SIN-like numbers. The policy is documented (option c — retain with redaction + consent + 30-day retention). `get_pii_policy()` returns the policy dict. The redaction is applied before writing to the turns table (verified in the AssistSession test).
|
||
|
||
#### TASK-04-04 — PII policy unit tests
|
||
- **Persona:** security-engineer
|
||
- **File:** `tests/test_pii_policy.py` (new)
|
||
- **Content:** Tests for the PII policy:
|
||
- `redact_pii()` redacts phone numbers ("Call me at 416-555-1234" → "Call me at [PHONE]").
|
||
- Email addresses ("Email me at john@example.com" → "Email me at [EMAIL]").
|
||
- Card numbers ("My card is 4111-1111-1111-1111" → "My card is [CARD]").
|
||
- SIN-like numbers ("My SIN is 123-456-789" → "My SIN is [SIN]").
|
||
- No false redactions ("I have 3 kids" → "I have 3 kids" — not a phone number).
|
||
- `get_pii_policy()` returns the policy dict with the correct fields.
|
||
- The retention limit is 30 days.
|
||
- **Acceptance criteria:** All PII patterns redacted. No false redactions. Policy dict correct. Retention 30 days.
|
||
|
||
---
|
||
|
||
## SLICE-05: build_assist_pipeline + In-Loop Guardrail Processor + Piper TTS (W3)
|
||
|
||
- **Goal:** Build `build_assist_pipeline()` — the assist-mode pipeline that reuses the v0.1 voice services (Deepgram + Ollama + Piper) with the assist system prompt + the in-loop `LiveAssistGuardrail` processor (post-LLM, pre-TTS). Default to Piper TTS for assist (D-065).
|
||
- **REQ-IDs covered:** REQ-ASSIST-01 (hands-free voice companion — the assist voice loop), REQ-NFR-ASSIST-02 (tap-to-talk invocation — the pipeline accepts WebRTC audio)
|
||
- **Wave:** 3
|
||
- **Dependencies:** SLICE-03 (LiveAssistGuardrail), SLICE-01 (AssistContextBinder)
|
||
- **Primary persona:** voice-engineer
|
||
- **Supporting personas:** backend-engineer (pipeline integration), lead-developer (build_assist_pipeline design decision)
|
||
|
||
### Tasks
|
||
|
||
#### TASK-05-01 — build_assist_pipeline() (D-061, D-065, D-066)
|
||
- **Persona:** voice-engineer
|
||
- **File:** `server/assist/pipeline.py` (new)
|
||
- **Content:** `def build_assist_pipeline(webrtc_connection, *, context: AssistContext, guardrail: LiveAssistGuardrail) -> tuple` — assembles the assist-mode Pipecat pipeline:
|
||
- Reuses `_build_transport`, `_build_stt`, `_build_llm` from `server/pipeline.py` (the v0.1 services — FIXED, not rewritten).
|
||
- Uses `_build_tts_piper()` — a new helper that builds the Piper TTS service (D-065 — Piper is the default for assist, ~80ms first audio vs Cartesia ~120ms). Falls back to Cartesia if `PRAXIS_ASSIST_TTS=cartesia` is set (for testing without Piper). The existing `_build_tts()` is unchanged (practice mode keeps Cartesia).
|
||
- Builds the LLM context from `context.system_prompt` (the ≤150-token assist prompt from `AssistContextBinder` — D-066). No opening line (assist is invoked mid-shift, no scripted opener — unlike practice which plays the scenario opening line).
|
||
- Inserts the `LiveAssistGuardrailProcessor` (TASK-05-02) between `llm` and `tts` in the pipeline (post-LLM, pre-TTS — D-060 layer 2).
|
||
- Reuses the `LatencyObserver` for assist latency measurement (R-ASSIST-02 — the C-8 <600ms budget is the binding constraint; measurement extends to assist turns).
|
||
- Pipeline structure:
|
||
```
|
||
transport.input() → stt → latency_observer → user_aggregator → llm →
|
||
latency_observer → LiveAssistGuardrailProcessor → tts → latency_observer →
|
||
transport.output() → assistant_aggregator
|
||
```
|
||
- Returns `(pipeline, task, runner, transport)` — no `scenario_runtime` (assist has no scenario runtime, it has an `AssistContext`).
|
||
- **Acceptance criteria:** `build_assist_pipeline()` returns a valid pipeline. The pipeline uses Piper TTS by default (D-065). The `LiveAssistGuardrailProcessor` is between `llm` and `tts`. The LLM context is the ≤150-token assist prompt. No opening line is played. The `LatencyObserver` is in the pipeline.
|
||
|
||
#### TASK-05-02 — In-loop LiveAssistGuardrailProcessor (D-060 layer 2, REQ-IDEATE-02)
|
||
- **Persona:** voice-engineer
|
||
- **File:** `server/assist/guardrail_processor.py` (new)
|
||
- **Content:** `class LiveAssistGuardrailProcessor(FrameProcessor)` — a Pipecat frame processor that runs the `LiveAssistGuardrail.check()` on each LLM response frame, before TTS:
|
||
- `__init__(self, guardrail: LiveAssistGuardrail, session: AssistSession)` — holds the guardrail + the session (for logging the verdict).
|
||
- `async def process_frame(self, frame: Frame, direction) -> None`:
|
||
1. Pass through non-text frames unchanged.
|
||
2. On `LLMFullResponseEndFrame` (or the accumulated text from `TextFrame`s): run `guardrail.check(text, GuardrailContext(role='assist'))`.
|
||
3. If `verdict.allowed` → pass the text frame through to TTS. Log the verdict to the session (`session.log_guardrail_verdict(verdict)`).
|
||
4. If `not verdict.allowed` and the category is retry-eligible (`blocked_direct_script`, `blocked_imperative`): retry once — inject a `RETRY_INSTRUCTION` as a new LLM context message + re-run the LLM. If the retry also blocks → replace the text with `CANNED_FALLBACK` + pass to TTS. Log both verdicts.
|
||
5. If `not verdict.allowed` and the category is NOT retry-eligible (`blocked_false_authority`, `blocked_impersonation`): replace the text with `CANNED_FALLBACK` + pass to TTS. No retry (hard violation — D-068). Log the verdict.
|
||
6. Increment `session.guardrail_block_count` on every block.
|
||
- This is the in-loop guardrail processor from REQ-IDEATE-02 (the existing `test_guardrail.py` only tests `check()` standalone; this is the pipeline-integration test gap).
|
||
- **Latency:** the regex check is <5ms (pure-Python regex — D-068). The retry (if triggered) adds one LLM round-trip (~200-300ms) — this is acceptable because the retry only fires on a block (a safety event, not a normal turn). The canned fallback is instant.
|
||
- **Acceptance criteria:** The processor passes allowed text through to TTS. Blocked text is replaced with `CANNED_FALLBACK`. Retry-eligible blocks get one retry; if the retry blocks, `CANNED_FALLBACK`. Non-retry-eligible blocks get `CANNED_FALLBACK` immediately. The verdict is logged to the session. `guardrail_block_count` is incremented on every block. The processor is a `FrameProcessor` subclass.
|
||
|
||
#### TASK-05-03 — build_assist_pipeline unit test
|
||
- **Persona:** voice-engineer
|
||
- **File:** `tests/test_assist_pipeline.py` (new)
|
||
- **Content:** Tests for `build_assist_pipeline` + `LiveAssistGuardrailProcessor`:
|
||
- The pipeline structure is correct (Piper TTS, guardrail processor between llm + tts, no opening line).
|
||
- The LLM context is the ≤150-token assist prompt (assert `len(system_prompt.split()) <= 200`).
|
||
- The `LiveAssistGuardrailProcessor` passes allowed text through (mock a coaching-question response → reaches TTS).
|
||
- The processor blocks direct-answer text (mock a "you should say X" response → `CANNED_FALLBACK` reaches TTS).
|
||
- The processor retries on a retry-eligible block (mock a first block + a second coaching response → the coaching response reaches TTS).
|
||
- The processor does NOT retry on false-authority (mock "I am your manager" → `CANNED_FALLBACK` immediately).
|
||
- The verdict is logged to the session (mock `session.log_guardrail_verdict` → called with the verdict).
|
||
- This is the REQ-IDEATE-02 pipeline-integration test (the gap in the existing `test_guardrail.py`).
|
||
- **Acceptance criteria:** Pipeline structure correct. Allowed text passes through. Direct-answer blocked + `CANNED_FALLBACK`. Retry works (one retry, then fallback). False-authority no retry. Verdict logged. REQ-IDEATE-02 pipeline integration verified.
|
||
|
||
---
|
||
|
||
## SLICE-06: Warm WebRTC Connection Manager + Reconnect Logic (W3)
|
||
|
||
- **Goal:** Build the `WarmWebRTCManager` — the shift-bounded warm WebRTC connection (D-067) with 30s heartbeat + reconnect logic (REQ-IDEATE-08). The connection opens at shift start, stays warm (keepalive only between turns), and closes at shift-end.
|
||
- **REQ-IDs covered:** REQ-IDEATE-08 (WebRTC mid-shift drop + reconnect logic)
|
||
- **Wave:** 3
|
||
- **Dependencies:** SLICE-02 (shift lifecycle — the connection opens/closes with the shift)
|
||
- **Primary persona:** voice-engineer
|
||
- **Supporting personas:** backend-engineer (WebRTC endpoint integration), lead-developer (reconnect state machine design)
|
||
|
||
### Tasks
|
||
|
||
#### TASK-06-01 — WarmWebRTCManager (D-067)
|
||
- **Persona:** voice-engineer
|
||
- **File:** `server/assist/webrtc.py` (new)
|
||
- **Content:** `class WarmWebRTCManager` — manages the warm WebRTC connection for an assist shift:
|
||
- `__init__(self)` — holds the active connections keyed by shift_id.
|
||
- `async def open(self, shift_id: str, webrtc_offer: dict) -> dict` — accepts a WebRTC offer, creates a `SmallWebRTCConnection`, builds the assist pipeline (TASK-05-01), starts the pipeline task, starts the 30s heartbeat. Returns the WebRTC answer.
|
||
- `async def close(self, shift_id: str) -> None` — closes the WebRTC connection, cancels the heartbeat, removes the shift from the active map.
|
||
- `async def get(self, shift_id: str) -> WarmConnection | None` — returns the active connection for the shift, or None.
|
||
- `@dataclass WarmConnection` — fields: `connection: SmallWebRTCConnection`, `task: PipelineTask`, `runner: PipelineRunner`, `heartbeat_task: asyncio.Task`, `shift_id: str`.
|
||
- `async def _heartbeat(self, shift_id: str)` — sends a no-op ping frame every 30s (D-067 — app-level heartbeat to prevent NAT timeouts). The SmallWebRTCTransport's ICE keepalive (15-30s) is the transport-level keepalive; the app-level heartbeat is an additional safety.
|
||
- Between turns, only keepalive flows (no audio streaming) — the mic is only active during a tap-to-talk turn (D-071 — the client sends audio only while the tap-to-talk button is held).
|
||
- **Acceptance criteria:** `open()` creates a connection + starts the pipeline + heartbeat. `close()` closes the connection + cancels the heartbeat. `get()` returns the active connection. The heartbeat runs every 30s. Between turns, no audio streams (keepalive only).
|
||
|
||
#### TASK-06-02 — WebRTC reconnect logic (REQ-IDEATE-08)
|
||
- **Persona:** voice-engineer
|
||
- **File:** `server/assist/webrtc.py` (extend)
|
||
- **Content:** The reconnect state machine (REQ-IDEATE-08 — D-067 mandates warm WebRTC with 30s heartbeat but the reconnect logic was unspecified):
|
||
- `async def _on_disconnect(self, shift_id: str) -> None` — called when the WebRTC connection drops (detected by heartbeat failure or transport close). The reconnect logic:
|
||
1. Log the disconnection (with timestamp + shift_id + turn count so far).
|
||
2. Attempt reconnect: the client must re-send a WebRTC offer (the server can't initiate — WebRTC is peer-initiated). The server marks the shift as `reconnecting` + waits up to 30s for a new offer.
|
||
3. If a new offer arrives within 30s → rebuild the pipeline + resume. The in-flight assist turn (if any) is retried (the learner re-speaks — the server can't replay the learner's audio). Log the reconnection.
|
||
4. If no new offer within 30s → mark the shift as `disconnected`. The shift is NOT auto-ended (the learner may be in a dead zone + reconnect later). The shift remains active until the learner explicitly ends it or the 8h auto-end fires (D-069).
|
||
- `async def get_reconnect_state(self, shift_id: str) -> str` — returns `'connected' | 'reconnecting' | 'disconnected'`.
|
||
- The learner experience during the gap: the client shows "Reconnecting..." for up to 30s, then "Disconnected — tap to reconnect" (the learner taps to send a new offer). This is documented in the client control surface (TASK-02-03).
|
||
- R-ASSIST-09 mitigation: the reconnect logic is specified + tested. The in-flight turn is NOT retried server-side (the learner re-speaks after reconnect).
|
||
- **Acceptance criteria:** Disconnect → log + mark `reconnecting` + wait 30s. New offer within 30s → rebuild + resume. No offer within 30s → mark `disconnected`. The shift is NOT auto-ended on disconnect (the learner can reconnect or end explicitly). The reconnect state is queryable.
|
||
|
||
#### TASK-06-03 — WebRTC reconnect chaos test (REQ-IDEATE-08)
|
||
- **Persona:** voice-engineer
|
||
- **File:** `tests/test_assist_webrtc_reconnect.py` (new)
|
||
- **Content:** Chaos test for the reconnect logic:
|
||
- Open a warm connection for a shift → verify `connected`.
|
||
- Simulate a disconnect (close the connection) → verify `reconnecting` + log.
|
||
- Send a new offer within 30s → verify `connected` + pipeline rebuilt.
|
||
- Simulate a disconnect + no new offer within 30s → verify `disconnected`.
|
||
- Verify the shift is NOT auto-ended on disconnect (the shift session row is still active).
|
||
- Verify the 8h auto-end still fires on a disconnected shift (D-069 — the shift ends after 8h regardless of connection state).
|
||
- This is the REQ-IDEATE-08 chaos test (kill the WebRTC connection mid-shift, verify reconnect + turn recovery).
|
||
- **Acceptance criteria:** Reconnect works within 30s. Disconnected state after 30s. Shift not auto-ended on disconnect. 8h auto-end fires on disconnected shifts. Reconnect state queryable.
|
||
|
||
---
|
||
|
||
## SLICE-07: P1 Integration — __main__.py Wiring + SessionRecorder Extension (W4)
|
||
|
||
- **Goal:** Wire all P1 modules into `server/__main__.py`: assist routes, assist WebRTC endpoint, shift lifecycle monitor, assist pipeline. Extend `SessionRecorder` to support the `session_type` field + assist turn logging + the cohort hook for assist shifts.
|
||
- **REQ-IDs covered:** REQ-ASSIST-01 (full integration), REQ-NFR-ASSIST-04 (shift-bounded, aggregation hook wired), REQ-IDEATE-03 (mode-conflict enforced at the WebRTC endpoint)
|
||
- **Wave:** 4
|
||
- **Dependencies:** SLICE-05 (assist pipeline), SLICE-06 (warm WebRTC)
|
||
- **Primary persona:** backend-engineer
|
||
- **Supporting personas:** voice-engineer (pipeline + WebRTC integration), lead-developer (integration orchestration)
|
||
|
||
### Tasks
|
||
|
||
#### TASK-07-01 — __main__.py — mount assist routes + WebRTC endpoint
|
||
- **Persona:** backend-engineer
|
||
- **File:** `server/__main__.py` (extend)
|
||
- **Content:** Add the assist routes + the assist WebRTC endpoint:
|
||
- `from server.assist.routes import router as assist_router` → `app.include_router(assist_router)` (mounts `/api/assist/shift/start`, `/end`, `/active`). Registered BEFORE the StaticFiles mount.
|
||
- `@app.post("/api/assist/webrtc")` — the assist WebRTC endpoint. Accepts a WebRTC offer + a `shift_id`. Calls `enforce_mutual_exclusivity(store, learner_id, 'assist')` (the mode-conflict guard — REQ-IDEATE-03). Calls `WarmWebRTCManager.open(shift_id, offer)` → returns the WebRTC answer. This is distinct from the practice `/pipecat/webrtc` endpoint (assist uses `/api/assist/webrtc`).
|
||
- The existing `/pipecat/webrtc` endpoint (practice) is extended to call `enforce_mutual_exclusivity(store, learner_id, 'practice')` — the mode-conflict guard in the other direction (reject practice if an assist shift is active).
|
||
- The `WarmWebRTCManager` is created on `app.state.assist_webrtc_manager` in the lifespan. The `ShiftLifecycleManager` is created + started in the lifespan (with the 8h auto-end monitor).
|
||
- The assist WebRTC endpoint is registered BEFORE the StaticFiles mount.
|
||
- **Acceptance criteria:** `POST /api/assist/shift/start` → 200. `POST /api/assist/webrtc` with a valid shift_id → 200 + WebRTC answer. Starting assist WebRTC during an active practice session → 409. Starting practice WebRTC during an active assist shift → 409. Routes matched before StaticFiles. The `WarmWebRTCManager` + `ShiftLifecycleManager` start on app startup.
|
||
|
||
#### TASK-07-02 — App.tsx — add /assist route
|
||
- **Persona:** voice-engineer
|
||
- **File:** `client/src/App.tsx` (extend)
|
||
- **Content:** Add the `/assist` route to the React Router route switch:
|
||
```tsx
|
||
<Route path="/assist" element={<AssistControl />} />
|
||
```
|
||
The existing routes (`/`, `/operator/login`, `/operator/dashboard`, `*`) are unchanged. The catch-all still serves the voice UI (practice). The `/assist` route serves the `AssistControl` component (TASK-02-03).
|
||
- R-DASH-05 carry-forward: the voice UI at `/` is unchanged. The `/assist` route is a new route, not a modification of existing routes.
|
||
- **Acceptance criteria:** `GET /assist` → serves index.html (SPA fallback) → React Router renders `AssistControl`. `GET /` → voice UI (unchanged). `GET /operator/dashboard` → dashboard (unchanged). `npm run build` succeeds. No regressions in existing routes.
|
||
|
||
#### TASK-07-03 — SessionRecorder extension for assist (D-062)
|
||
- **Persona:** backend-engineer
|
||
- **File:** `server/session_recorder.py` (extend)
|
||
- **Content:** Extend `SessionRecorder` to support the `session_type` field + the assist cohort hook:
|
||
- `__init__` gains a `session_type: str = "practice"` parameter. Practice sessions set it to `"practice"` (default); assist shifts use the `AssistSession` class (SLICE-01) instead of `SessionRecorder` directly — but the `_build_session_outcome` method is shared.
|
||
- `_build_session_outcome()` gains the `session_type` field: `"session_type": self.session_type`. The aggregator (SLICE-09) branches on this field.
|
||
- The `_run_cohort_aggregation()` method (existing) is unchanged — it already fires the hook for any session with `pg_store`. The assist path uses the same hook (the `session_outcome` dict has `session_type='assist'` which the aggregator branches on).
|
||
- **D-063 enforcement:** the `end()` method's `schedule_mastery` parameter defaults to `False`. The `AssistSession.end()` (SLICE-01) calls `end(schedule_mastery=False)` — never `True`. The existing practice path calls `end(schedule_mastery=True)` (unchanged).
|
||
- The assist path does NOT use `SessionRecorder` directly — it uses `AssistSession` (SLICE-01). But `AssistSession` reuses the `_build_session_outcome` pattern + the `_run_cohort_aggregation` pattern. This task ensures the shared pattern is consistent.
|
||
- **Acceptance criteria:** `_build_session_outcome()` includes `session_type`. The cohort hook fires for assist shifts (via `AssistSession.end()`). `schedule_mastery=False` for assist (D-063 — verified). Existing practice sessions unchanged (backward compat).
|
||
|
||
#### TASK-07-04 — __main__.py — start ShiftLifecycleManager in lifespan
|
||
- **Persona:** backend-engineer
|
||
- **File:** `server/__main__.py` (extend)
|
||
- **Content:** In the `lifespan` context manager, after the nightly scheduler starts, start the `ShiftLifecycleManager`:
|
||
```python
|
||
from server.assist.lifecycle import ShiftLifecycleManager
|
||
shift_lifecycle = ShiftLifecycleManager(_store)
|
||
app.state.shift_lifecycle = shift_lifecycle
|
||
await shift_lifecycle.start_monitor()
|
||
# ... yield ...
|
||
await shift_lifecycle.stop_monitor()
|
||
```
|
||
The monitor runs every 5 minutes, checking for shifts older than 8h (D-069). The `WarmWebRTCManager` is also created on `app.state.assist_webrtc_manager`. Both are stopped/cancelled on shutdown.
|
||
- **Acceptance criteria:** The `ShiftLifecycleManager` starts on app startup. The monitor runs every 5 minutes. The `WarmWebRTCManager` is available on `app.state`. Both are stopped on shutdown.
|
||
|
||
---
|
||
|
||
## SLICE-08: P1 Integration Tests + Incremental Audit-Log Write (W4)
|
||
|
||
- **Goal:** Run end-to-end P1 integration tests (shift start → tap-to-talk → guardrail → shift end → aggregation hook) + implement the incremental audit-log write (REQ-IDEATE-09 — persist the turn before TTS start, not after playback, so abrupt termination still leaves an audit trail).
|
||
- **REQ-IDs covered:** REQ-IDEATE-09 (audit-log incremental write), REQ-ASSIST-03 (guardrail audit verified e2e)
|
||
- **Wave:** 4
|
||
- **Dependencies:** SLICE-07 (integration wiring)
|
||
- **Primary persona:** backend-engineer
|
||
- **Supporting personas:** security-engineer (audit-log completeness), data-engineer (turns table)
|
||
|
||
### Tasks
|
||
|
||
#### TASK-08-01 — Incremental audit-log write (REQ-IDEATE-09)
|
||
- **Persona:** backend-engineer
|
||
- **File:** `server/assist/guardrail_processor.py` (extend)
|
||
- **Content:** The incremental audit-log write (REQ-IDEATE-09): persist the ASR transcript + LLM response + guardrail verdict BEFORE TTS start, not after playback completes. This ensures abrupt termination (battery death R-ASSIST-14, power loss mid-turn) still leaves an audit trail.
|
||
- In `LiveAssistGuardrailProcessor.process_frame()`:
|
||
1. On `TranscriptionFrame` (ASR done): write the learner's utterance to the turns table immediately (`session.log_assist_turn_partial(asr_text=transcript)`). This is the "before TTS" write.
|
||
2. On `LLMFullResponseEndFrame` (LLM done): run the guardrail check. Write the LLM response + guardrail verdict to the turns table (`session.log_assist_turn_complete(tts_text=response, guardrail_verdict=verdict)`). This updates the partial row.
|
||
3. The TTS then plays the response. If the TTS is interrupted (battery death), the turns table already has the ASR + LLM + verdict — the audit trail is complete.
|
||
- `AssistSession` gains two methods:
|
||
- `async def log_assist_turn_partial(self, asr_text: str)` — writes a turn row with `asr_text` + NULL `tts_text` + NULL `guardrail_verdict_json`. Returns the turn seq.
|
||
- `async def log_assist_turn_complete(self, seq: int, tts_text: str, guardrail_verdict: dict, latency_ms: float | None)` — updates the turn row with `tts_text` + `guardrail_verdict_json` + `latency_ms`.
|
||
- This is the REQ-IDEATE-09 requirement: for a safety-critical surface (REQ-ASSIST-03), an incomplete audit log undermines the `guardrail_block_rate` safety signal + the operator's ability to investigate incidents.
|
||
- **Acceptance criteria:** A partial turn (ASR written, TTS interrupted before completion) is in the turns table with `asr_text` populated + `tts_text` NULL. A complete turn has both + the guardrail verdict. Abrupt termination (simulated by not calling `log_assist_turn_complete`) leaves the partial row. The audit trail is complete for the ASR + (if the LLM finished) the LLM response + verdict.
|
||
|
||
#### TASK-08-02 — P1 integration test (shift lifecycle e2e)
|
||
- **Persona:** backend-engineer
|
||
- **File:** `tests/test_p1_assist_integration.py` (new)
|
||
- **Content:** End-to-end P1 integration test (using FastAPI TestClient + temp SQLite, no Postgres required for the assist voice loop — the aggregation hook is no-op without Postgres):
|
||
1. `POST /api/assist/shift/start` with `{path_slug: 'customer_service', scenario_tag: 'damaged-product refund'}` → 200 + shift_id + context + consent_disclosure.
|
||
2. Verify the shift session row has `session_type='assist'`.
|
||
3. `POST /api/assist/webrtc` with the shift_id + a WebRTC offer → 200 + WebRTC answer. (Mock the WebRTC connection — the pipeline doesn't actually run in the test.)
|
||
4. Simulate a tap-to-talk turn: mock the ASR + LLM + guardrail. Verify the turn is logged to the turns table with `guardrail_verdict_json`.
|
||
5. `POST /api/assist/shift/end` with the shift_id → 200 + turn_count + guardrail_block_count.
|
||
6. Verify the shift session row has `ended_at` + `outcome='completed'`.
|
||
7. Verify `run_mastery_flow()` was NOT called (D-063 — no mastery update for assist).
|
||
8. Mode-conflict: start a practice session → start an assist shift → 409 Conflict. End the practice session → start the assist shift → 200.
|
||
- This test does NOT require Postgres (the aggregation hook is no-op without `pg_store`). The assist voice loop is SQLite-only.
|
||
- **Acceptance criteria:** Full shift lifecycle works. Turn logged with guardrail verdict. D-063 verified (no mastery flow). Mode-conflict enforced (409). No Postgres required for the assist voice loop.
|
||
|
||
#### TASK-08-03 — P1 integration test (guardrail e2e)
|
||
- **Persona:** security-engineer
|
||
- **File:** `tests/test_p1_guardrail_e2e.py` (new)
|
||
- **Content:** End-to-end guardrail test through the assist pipeline:
|
||
1. Start a shift.
|
||
2. Mock an LLM response that gives a direct answer ("You should say sorry to the customer").
|
||
3. Verify the guardrail blocks it + the canned fallback ("Think about what the customer needs right now...") is sent to TTS.
|
||
4. Verify the turn's `guardrail_verdict_json` has `allowed=False, category="blocked_direct_script"`.
|
||
5. Verify `guardrail_block_count` is incremented.
|
||
6. Mock an LLM response that gives a coaching question ("What do you think the customer needs?").
|
||
7. Verify the guardrail allows it + the coaching question is sent to TTS.
|
||
8. Verify the turn's `guardrail_verdict_json` has `allowed=True, category="coaching"`.
|
||
9. Incremental audit-log: verify the partial turn (ASR only) is written before the LLM response, then updated with the LLM response + verdict.
|
||
- This is the REQ-ASSIST-03 e2e verification (the guardrail works in the pipeline, not just standalone).
|
||
- **Acceptance criteria:** Direct answer blocked + canned fallback. Coaching question allowed. Verdict logged. Block count incremented. Incremental audit-log verified (partial → complete).
|
||
|
||
#### TASK-08-04 — P1 verification matrix
|
||
- **Persona:** lead-developer
|
||
- **File:** `.ciagent/VERIFY-P1-v0.5.md` (new — pre-verify checklist)
|
||
- **Content:** REQ-ID → test mapping for P1. Confirm all P1 REQ-IDs (REQ-ASSIST-01, REQ-ASSIST-02, REQ-ASSIST-03, REQ-NFR-ASSIST-02, REQ-NFR-ASSIST-03, REQ-NFR-ASSIST-04, REQ-IDEATE-01, REQ-IDEATE-02, REQ-IDEATE-03, REQ-IDEATE-05, REQ-IDEATE-08, REQ-IDEATE-09) have covering tests. List each test file + what it verifies.
|
||
- **Acceptance criteria:** Every P1 REQ-ID has at least one covering test listed. Matrix is complete (no gaps).
|
||
|
||
---
|
||
|
||
# Phase 2 — Integration + Tech-Debt + NFR Measurement
|
||
|
||
**Branch:** `phase/02-integration-techdebt-nfr` → merged to `milestone/v0.5-live-assist`
|
||
**Ship:** `v0.1.12` (patch release, feature milestone type)
|
||
**REQ-IDs covered:** REQ-NFR-ASSIST-01, REQ-IDEATE-04, REQ-IDEATE-06, REQ-IDEATE-07
|
||
**Slices:** 3 vertical slices in 2 waves
|
||
**Total tasks:** 9
|
||
|
||
| Wave | Slices | Parallel slots | Description |
|
||
|------|--------|----------------|-------------|
|
||
| 1 | SLICE-09, SLICE-10, SLICE-11 | 3 | NFR measurement (p95 latency + guardrail FP/FN) + cohort aggregation assist metrics + assist cost tracking + C-3 check — parallel, disjoint file territories |
|
||
| 2 | SLICE-12 | 1 | Tech-debt wave (8 v0.4 P1+ findings) + P2 integration tests |
|
||
|
||
### Wave dependency graph (P2)
|
||
|
||
```
|
||
Wave 1 ──────────────────────────────────────────────────────────────
|
||
SLICE-09 (NFR measurement: p95 latency + guardrail FP/FN)
|
||
SLICE-10 (cohort aggregation assist metrics — no schema change)
|
||
SLICE-11 (assist cost tracking + C-3 budget check)
|
||
│
|
||
▼
|
||
Wave 2 ──────────────────────────────────────────────────────────────
|
||
SLICE-12 (tech-debt wave: 8 v0.4 P1+ findings + P2 integration tests)
|
||
```
|
||
|
||
### Persona load distribution (P2)
|
||
|
||
| Persona | Tasks | Primary territory |
|
||
|---------|-------|-------------------|
|
||
| lead-developer | 1 | integration orchestration |
|
||
| backend-engineer | 3 | cohort aggregation assist branch, cost tracking, tech-debt |
|
||
| voice-engineer | 2 | p95 latency measurement, latency optimization |
|
||
| security-engineer | 1 | guardrail FP/FN measurement |
|
||
| data-engineer | 2 | aggregation assist metrics, tech-debt (cache persistence) |
|
||
|
||
---
|
||
|
||
## SLICE-09: NFR Measurement — p95 Latency + Guardrail FP/FN (W1)
|
||
|
||
- **Goal:** Measure the two NFR targets from REQ-IDEATE-04: (1) p95 assist-turn latency ≤650ms pilot tolerance (D-072), (2) guardrail false-positive rate <5% on the tuning corpus + false-negative rate measured + trended nightly.
|
||
- **REQ-IDs covered:** REQ-NFR-ASSIST-01 (latency <600ms target, ≤650ms pilot), REQ-IDEATE-04 (measurable NFR targets)
|
||
- **Wave:** 1
|
||
- **Dependencies:** P1 SLICE-05 (assist pipeline + LatencyObserver), P1 SLICE-04 (guardrail tuning corpus)
|
||
- **Primary persona:** voice-engineer
|
||
- **Supporting personas:** security-engineer (guardrail FP/FN), lead-developer (C-8 latency tension)
|
||
|
||
### Tasks
|
||
|
||
#### TASK-09-01 — p95 assist-turn latency measurement (D-072, REQ-IDEATE-04)
|
||
- **Persona:** voice-engineer
|
||
- **File:** `server/assist/latency_metrics.py` (new)
|
||
- **Content:** `class AssistLatencyMetrics` — collects assist-turn latencies from the `LatencyObserver` + computes p95:
|
||
- `__init__(self)` — holds a list of `LatencyRecord` objects (from the `LatencyObserver`).
|
||
- `def record(self, record: LatencyRecord)` — appends a latency record.
|
||
- `def p95(self) -> float | None` — computes the 95th percentile of `e2e_asr_to_tts_ms` across all records. Returns None if no records.
|
||
- `def p50(self) -> float | None` — the median.
|
||
- `def summary(self) -> dict` — returns `{p50, p95, p99, count, target_ms: 600, pilot_tolerance_ms: 650, within_target: bool, within_pilot: bool}`.
|
||
- The metrics are collected per shift + reported at shift-end (in the `session_outcome` dict, which flows to the cohort aggregation — SLICE-10).
|
||
- D-072 binding: the pilot tolerance is ≤650ms. The target is <600ms (C-8). The measurement reports both: `within_target` (p95 < 600ms) + `within_pilot` (p95 ≤ 650ms). If `within_pilot` is False, the shift is flagged for the operator (via the cohort aggregation — `assist_p95_latency_ms` metric).
|
||
- The LatencyObserver already timestamps ASR→TTS-first-audio (server/latency.py). This task aggregates those timestamps into p95.
|
||
- **Acceptance criteria:** `p95()` computes the 95th percentile. `summary()` returns p50/p95/p99 + target/pilot thresholds + within flags. The metrics are collected per shift + reported at shift-end.
|
||
|
||
#### TASK-09-02 — Guardrail FP/FN measurement + nightly trending (REQ-IDEATE-04)
|
||
- **Persona:** security-engineer
|
||
- **File:** `server/assist/guardrail_metrics.py` (new)
|
||
- **Content:** `class GuardrailMetrics` — measures the guardrail's false-positive + false-negative rates:
|
||
- `false_positive_rate()` — the FP rate on the tuning corpus (coaching responses blocked). This is measured at test time (TASK-04-02) + reported in the P2 verification. Target: <5% (REQ-IDEATE-04).
|
||
- `false_negative_rate()` — the FN rate on the direct-answer + adversarial corpus (direct answers allowed). This is measured at test time + trended nightly (the nightly job samples assist turns from the turns table + re-runs the guardrail to detect any new false-negative patterns).
|
||
- `async def nightly_trend(self, store: PraxisStore) -> dict` — samples the last 24h of assist turns, re-runs the guardrail on the `tts_text` (the LLM response), + reports: `{total_turns, blocked, allowed_coaching, allowed_neutral, fn_candidates: [...]}`. The `fn_candidates` are turns where the guardrail allowed the text but the text contains direct-answer patterns (a heuristic check — not a full LLM-as-judge, which is v0.6 per REQ-IDEATE-10). This is the "trended nightly" part of REQ-IDEATE-04.
|
||
- The nightly trend is logged (not stored in Postgres — it's a diagnostic, not a cohort metric). The operator can review the log to spot guardrail regressions.
|
||
- D-068 mitigation: the regex is the first line, not the only line. The nightly trend + the LLM-as-judge (v0.6) are the defense-in-depth.
|
||
- **Acceptance criteria:** `false_positive_rate()` computed from the tuning corpus. `false_negative_rate()` computed. `nightly_trend()` samples 24h of turns + reports fn_candidates. FP <5% target. FN measured + trended.
|
||
|
||
#### TASK-09-03 — NFR measurement test
|
||
- **Persona:** voice-engineer
|
||
- **File:** `tests/test_nfr_measurement.py` (new)
|
||
- **Content:** Tests for the NFR measurement:
|
||
- `AssistLatencyMetrics`: record 100 mock latency records (some <600ms, some 600-650ms, some >650ms) → verify p95, p50, p99 computed correctly. `within_target` = (p95 < 600). `within_pilot` = (p95 ≤ 650).
|
||
- `GuardrailMetrics`: `false_positive_rate()` on the tuning corpus → verify <5% (or report the actual rate). `false_negative_rate()` on the direct-answer corpus → verify <5%. `nightly_trend()` on mock turns → verify fn_candidates detected.
|
||
- D-072: the test documents the pilot tolerance (≤650ms) + the target (<600ms). The test ASSERTS that the measurement infrastructure works, not that the actual latency is under budget (that's a Phase-1 live measurement, not a CI test).
|
||
- **Acceptance criteria:** p95/p50/p99 computed correctly. FP/FN rates measured. Nightly trend works. D-072 pilot tolerance documented.
|
||
|
||
---
|
||
|
||
## SLICE-10: Cohort Aggregation Assist Metrics (W1)
|
||
|
||
- **Goal:** Extend the v0.4 cohort aggregation pipeline to handle `session_type='assist'` outcomes. Add the 5 assist metrics (assist_shifts_count, assist_turns_count, assist_avg_turns_per_shift, assist_active_learners_count, assist_guardrail_block_rate) as new metric strings in `cohort_aggregates` — no schema change (D-062). Extend the dashboard views to show assist metrics.
|
||
- **REQ-IDs covered:** REQ-NFR-ASSIST-04 (aggregates as session_type=assist in v0.4 cohort pipeline)
|
||
- **Wave:** 1
|
||
- **Dependencies:** P1 SLICE-07 (SessionRecorder extension — session_type field in session_outcome)
|
||
- **Primary persona:** backend-engineer
|
||
- **Supporting personas:** data-engineer (aggregation SQL), lead-developer (dashboard view extension)
|
||
|
||
### Tasks
|
||
|
||
#### TASK-10-01 — Aggregator _aggregate_assist branch (D-062)
|
||
- **Persona:** backend-engineer
|
||
- **File:** `server/cohort/aggregator.py` (extend)
|
||
- **Supporting:** data-engineer (k-anon suppression for assist metrics)
|
||
- **Content:** Extend `aggregate_session()` to branch on `session_type`:
|
||
```python
|
||
async def aggregate_session(pg_store, session_outcome):
|
||
session_type = session_outcome.get("session_type", "practice")
|
||
if session_type == "assist":
|
||
await _aggregate_assist(pg_store, session_outcome)
|
||
else:
|
||
await _aggregate_practice(pg_store, session_outcome) # existing logic
|
||
|
||
async def _aggregate_assist(pg_store, session_outcome):
|
||
path = session_outcome["path"]
|
||
turn_count = session_outcome.get("assist_turn_count", 0)
|
||
blocks = session_outcome.get("guardrail_blocks", 0)
|
||
p95_latency = session_outcome.get("assist_p95_latency_ms")
|
||
# Upsert assist_shifts_count, assist_turns_count, assist_avg_turns_per_shift,
|
||
# assist_active_learners_count, assist_guardrail_block_rate
|
||
# All k-anonymized (≥10 distinct learners, else suppressed — D-034 carry-forward)
|
||
```
|
||
- The existing `_aggregate_practice` is the current `aggregate_session` logic (renamed for clarity — the existing logic is the practice branch).
|
||
- The assist metrics use the same k-anonymity suppression (≥10 distinct learners) + the same 7-day rolling window + the same idempotent upsert.
|
||
- `assist_guardrail_block_rate` = `guardrail_blocks / assist_turns_count` (the fraction of turns where the output filter fired — a safety signal for operators).
|
||
- `assist_p95_latency_ms` is stored as a metric (from SLICE-09) so operators can see the latency trend.
|
||
- No schema change to `cohort_aggregates` (the `metric` column is free-form TEXT — D-062).
|
||
- **Acceptance criteria:** `aggregate_session()` branches on `session_type`. `_aggregate_assist()` upserts the 5 assist metrics + p95 latency. k-anon suppression applies. Idempotent upsert. No schema change.
|
||
|
||
#### TASK-10-02 — Dashboard views extension (D-053)
|
||
- **Persona:** backend-engineer
|
||
- **File:** `server/operator/cohort.py` (extend), `server/operator/failure_patterns.py` (extend)
|
||
- **Content:** Extend the v0.4 dashboard views to include assist metrics:
|
||
- `GET /api/operator/cohort` (practice volume) → add assist rows: `assist_shifts_count`, `assist_turns_count` per path/window. The view returns practice + assist volume side-by-side. The `PRACTICE_METRICS` set is extended to include `assist_shifts_count`, `assist_turns_count`.
|
||
- `GET /api/operator/failure-patterns` → add `assist_guardrail_block_rate` as a new "safety signal" row. High block rate = flag for operator review (a sudden spike signals either a prompt regression or learners pushing boundaries).
|
||
- `GET /api/operator/mastery` → unchanged (assist doesn't affect mastery per D-063). Optionally add a note: "Assist usage: N shifts, M turns this window" as context (read from `assist_shifts_count`).
|
||
- The React dashboard (client/src/operator/views/) renders the new rows. This is a small extension (the existing table + sparkline components handle new metric rows automatically — they're metric-agnostic).
|
||
- **Acceptance criteria:** `/api/operator/cohort` returns assist volume rows. `/api/operator/failure-patterns` returns `assist_guardrail_block_rate`. `/api/operator/mastery` unchanged. The React dashboard renders the new rows.
|
||
|
||
#### TASK-10-03 — Aggregation assist metrics test
|
||
- **Persona:** backend-engineer
|
||
- **File:** `tests/test_cohort_assist_aggregation.py` (new)
|
||
- **Content:** Tests for the assist aggregation (with mocked PgStore):
|
||
- `_aggregate_assist()` upserts the 5 assist metrics + p95 latency.
|
||
- k-anon suppression: <10 distinct learners → suppressed.
|
||
- Idempotent upsert: same session_outcome twice → same aggregate.
|
||
- `assist_guardrail_block_rate` = blocks/turns.
|
||
- The practice branch (`_aggregate_practice`) is unchanged (backward compat — existing `test_cohort_aggregation.py` still passes).
|
||
- The dashboard endpoints return assist rows.
|
||
- **Acceptance criteria:** All assist metrics upserted. k-anon enforced. Idempotent. Practice branch unchanged. Dashboard endpoints return assist rows.
|
||
|
||
---
|
||
|
||
## SLICE-11: Assist Cost Tracking + C-3 Budget Check (W1)
|
||
|
||
- **Goal:** Extend `server/cost.py` to log per-assist-turn cost (each assist turn is a separate `gemma4:cloud` invocation). Add a Phase-1 budget check: estimate monthly assist cost per learner + flag if it pushes the total over the C-3 ≤ $3/active learner/month target (REQ-IDEATE-07).
|
||
- **REQ-IDs covered:** REQ-IDEATE-07 (assist cost tracking + C-3 check)
|
||
- **Wave:** 1
|
||
- **Dependencies:** P1 SLICE-05 (assist pipeline — the cost is logged per turn)
|
||
- **Primary persona:** backend-engineer
|
||
- **Supporting personas:** data-engineer (cost storage), lead-developer (C-3 budget check)
|
||
|
||
### Tasks
|
||
|
||
#### TASK-11-01 — Per-assist-turn cost logging (REQ-IDEATE-07)
|
||
- **Persona:** backend-engineer
|
||
- **File:** `server/cost.py` (extend)
|
||
- **Content:** Extend the cost logging to support per-assist-turn cost:
|
||
- `derive_assist_turn_cost(llm_input_tokens: int, llm_output_tokens: int, tts_characters: int, tts_provider: str = "piper") -> CostBreakdown` — like `derive_cost` but for a single assist turn (no debrief tokens, no Deepgram minutes — the assist turn is a short coaching exchange). Uses the same `load_rates()` + the same `CostBreakdown` dataclass.
|
||
- The assist turn cost is logged per turn (in `AssistSession.log_assist_turn_complete()` — the `cost_breakdown_json` is stored in the turns table, or aggregated at shift-end).
|
||
- The shift-end `session_outcome` dict gains `assist_cost_cents: int` (the total cost of all assist turns in the shift). This flows to the cohort aggregation (SLICE-10) as an `assist_avg_cost_per_shift` metric (optional — not a core metric, but useful for the C-3 check).
|
||
- The existing `derive_cost()` is unchanged (practice sessions keep their cost logging).
|
||
- **Acceptance criteria:** `derive_assist_turn_cost()` computes the per-turn cost. The cost is logged per turn + aggregated at shift-end. `assist_cost_cents` is in the session_outcome. Existing `derive_cost()` unchanged.
|
||
|
||
#### TASK-11-02 — C-3 budget check (REQ-IDEATE-07)
|
||
- **Persona:** backend-engineer
|
||
- **File:** `server/assist/budget_check.py` (new)
|
||
- **Content:** `def check_c3_budget(assist_turns_per_shift: int, shifts_per_month: int, cost_per_turn_cents: float) -> dict`:
|
||
- Estimates the monthly assist cost per learner: `monthly_assist_cost = assist_turns_per_shift * shifts_per_month * cost_per_turn_cents / 100`.
|
||
- Compares against the C-3 target (≤ $3/active learner/month — relaxed for the Canada pilot per D-012, but the architecture must not preclude it).
|
||
- Example: 20 turns/shift × 20 shifts/month = 400 extra LLM calls. At ~$0.0005/turn (gemma4:cloud pilot rates), that's ~$0.20/month — well under $3. But if the turns are longer or the model is more expensive, the cost could approach the ceiling.
|
||
- Returns `{monthly_assist_cost, total_with_practice: float, c3_target: 3.0, within_budget: bool, flag: bool}`. `flag` is True if the total (practice + assist) exceeds $3.
|
||
- The budget check is a diagnostic (not enforced — D-012 says no enforced ceiling in the pilot). It's logged at shift-end + reported in the P2 verification.
|
||
- R-ASSIST-14 mitigation: the budget check helps the operator understand the cost impact of assist usage.
|
||
- **Acceptance criteria:** `check_c3_budget()` computes the monthly assist cost. `within_budget` + `flag` correct. The check is diagnostic (not enforced). Logged at shift-end.
|
||
|
||
#### TASK-11-03 — Cost tracking test
|
||
- **Persona:** backend-engineer
|
||
- **File:** `tests/test_assist_cost.py` (new)
|
||
- **Content:** Tests for the assist cost tracking:
|
||
- `derive_assist_turn_cost()` computes the per-turn cost (LLM tokens + Piper TTS chars).
|
||
- The shift-end `assist_cost_cents` is the sum of per-turn costs.
|
||
- `check_c3_budget()` with 20 turns/shift × 20 shifts/month → within budget.
|
||
- `check_c3_budget()` with 100 turns/shift × 30 shifts/month → may exceed budget (flag=True).
|
||
- Existing `derive_cost()` unchanged (practice cost tests still pass).
|
||
- **Acceptance criteria:** Per-turn cost computed. Shift-end cost aggregated. C-3 check works. Practice cost unchanged.
|
||
|
||
---
|
||
|
||
## SLICE-12: Tech-Debt Wave + P2 Integration Tests (W2)
|
||
|
||
- **Goal:** Carry forward the 8 v0.4 P1+ findings into the v0.5 backlog as a tech-debt wave (REQ-IDEATE-06). Run P2 integration tests (aggregation → endpoint → cost → NFR measurement).
|
||
- **REQ-IDs covered:** REQ-IDEATE-06 (v0.4 P1+ tech-debt wave)
|
||
- **Wave:** 2
|
||
- **Dependencies:** SLICE-09, SLICE-10, SLICE-11 (all P2 wave-1 slices)
|
||
- **Primary persona:** backend-engineer
|
||
- **Supporting personas:** data-engineer (aggregation cache persistence), security-engineer (cookie-secret validation, credential status enum)
|
||
|
||
### Tasks
|
||
|
||
#### TASK-12-01 — Tech-debt: aggregation in-memory cache persistence (P1+ #7)
|
||
- **Persona:** data-engineer
|
||
- **File:** `server/cohort/aggregator.py` (extend)
|
||
- **Content:** The v0.4 P1+ #7 finding: the `_agg_cache` on `PgStore` (aggregator.py:162-170) tracks running counters + distinct learner sets. On restart, the cache is lost — the next hook starts fresh, `active_learners_count` may reset to 1 (under-counting until nightly reconcile). This directly corrupts v0.5's `assist_active_learners_count` after a server restart.
|
||
- Mitigation: persist the distinct-learner set to a small SQLite table (`cohort_learner_cache`) keyed by `(path, window_start, learner_ref)`. The hook reads the cache from SQLite on startup + updates it on each session. The nightly job reconciles from `mastery_gate_events` (the source of truth) + clears the cache.
|
||
- `async def _load_learner_cache(store: PraxisStore) -> dict` — loads the distinct-learner sets from SQLite on startup.
|
||
- `async def _save_learner_cache(store: PraxisStore, cache: dict)` — saves the cache to SQLite (called periodically — every 5 minutes or on shift-end).
|
||
- This is a low-effort, high-value fix (directly corrupts v0.5 assist metrics).
|
||
- **Acceptance criteria:** The distinct-learner set survives a server restart. `active_learners_count` is not reset to 1 after restart. The nightly job reconciles + clears the cache. Existing aggregation tests still pass.
|
||
|
||
#### TASK-12-02 — Tech-debt: cookie-secret length validation (P1+ #3)
|
||
- **Persona:** security-engineer
|
||
- **File:** `server/auth/cookies.py` (extend)
|
||
- **Content:** The v0.4 P1+ #3 finding: `cookies.py:41` only checks `if not secret` (empty), not `len(secret) >= 32`. A short non-empty secret (e.g., "x") would be accepted, weakening the HMAC signature.
|
||
- Add: `if len(secret) < 32: logger.warning("PRAXIS_COOKIE_SECRET is <32 bytes — HMAC signature weakened. Use 'openssl rand -base64 48'.")`. The secret is still accepted (backward compat — the pilot may have a short secret), but the warning is logged.
|
||
- In production (post-pilot), this should be a hard error (`raise RuntimeError`). For v0.5 pilot, the warning is sufficient.
|
||
- **Acceptance criteria:** A secret <32 bytes logs a WARNING. A secret ≥32 bytes logs no warning. The secret is still accepted (no hard error in pilot). Existing auth tests still pass.
|
||
|
||
#### TASK-12-03 — Tech-debt: set_credential_status enum + f-string SQL (P1+ #4/#8)
|
||
- **Persona:** backend-engineer
|
||
- **File:** `db/pg_store.py` (extend)
|
||
- **Content:** The v0.4 P1+ #4 + #8 findings: `set_credential_status` (pg_store.py:223-227) accepts any string for `status` (no enum check) + uses f-string interpolation for the `extra` variable (`, revoked_at = now()` or empty).
|
||
- Refactor to two explicit queries:
|
||
```python
|
||
async def set_credential_status(self, cred_id: str, status: str) -> None:
|
||
if status not in ("active", "revoked"):
|
||
raise ValueError(f"Invalid credential status: {status!r}")
|
||
async with self.pool.acquire() as conn:
|
||
if status == "revoked":
|
||
await conn.execute(
|
||
"UPDATE issued_credentials SET status = $1, revoked_at = now() WHERE id = $2",
|
||
status, cred_id,
|
||
)
|
||
else:
|
||
await conn.execute(
|
||
"UPDATE issued_credentials SET status = $1, revoked_at = NULL WHERE id = $2",
|
||
status, cred_id,
|
||
)
|
||
```
|
||
- The `status` parameter is validated against the allowed enum (`active`, `revoked`). The f-string is replaced with two explicit parameterized queries.
|
||
- **Acceptance criteria:** `set_credential_status('revoked')` works. `set_credential_status('active')` works (clears `revoked_at`). Invalid status → `ValueError`. No f-string in SQL. Existing credential tests still pass.
|
||
|
||
#### TASK-12-04 — Tech-debt: argon2id offload + rate-limit mock test + credential audit log + zoneinfo (P1+ #1/#2/#5/#6)
|
||
- **Persona:** backend-engineer
|
||
- **Files:** `server/auth/routes.py` (extend), `tests/test_auth.py` (extend), `server/operator/credentials.py` (extend), `server/cohort/nightly.py` (extend)
|
||
- **Content:** The remaining 4 v0.4 P1+ findings:
|
||
- **P1+ #1 (argon2id blocking event loop):** offload `verify_password` + `hash_password` to `asyncio.to_thread()` in the login handler. `await asyncio.to_thread(verify_password, stored_hash, password)`. This unblocks the event loop for the ~100-300ms hashing duration.
|
||
- **P1+ #2 (rate limit 429 mock test):** add a mock-based 429 test in `test_auth.py` (the full 6th-attempt→429 path is in the PG-requiring integration test; this adds a mock-based test for CI coverage without Postgres).
|
||
- **P1+ #5 (credential revocation audit log):** add `log.info("credential revoked: operator=%s cred_id=%s", op.id, cred_id)` to the `revoke_credential` endpoint. No `audit_log` table (the log is sufficient for pilot).
|
||
- **P1+ #6 (nightly scheduler zoneinfo):** replace the fixed UTC-5 offset with `zoneinfo.ZoneInfo("America/Winnipeg")` for proper DST handling. `CT = ZoneInfo("America/Winnipeg")`. The `seconds_until_next_03_ct()` function uses timezone-aware datetime with the ZoneInfo.
|
||
- **Acceptance criteria:** Argon2id offloaded to thread (no event-loop blocking). Mock 429 test passes. Credential revocation logs the operator + cred_id. Nightly scheduler uses `ZoneInfo("America/Winnipeg")` (DST-aware). All existing tests still pass.
|
||
|
||
#### TASK-12-05 — P2 integration test
|
||
- **Persona:** backend-engineer
|
||
- **File:** `tests/test_p2_assist_integration.py` (new — requires Postgres, skip if no DSN)
|
||
- **Content:** End-to-end P2 integration test:
|
||
1. Seed 12 mock assist shifts (12 distinct learners — above k-anon threshold) for a path.
|
||
2. Run the aggregation hook for each shift → `cohort_aggregates` populated with assist metrics.
|
||
3. `GET /api/operator/cohort` (with auth cookie) → returns assist volume (non-suppressed, 12 ≥ 10).
|
||
4. `GET /api/operator/failure-patterns` → returns `assist_guardrail_block_rate`.
|
||
5. Seed 5 more assist shifts from 5 NEW distinct learners for a different path → `GET /api/operator/cohort` for that path → suppressed cells (5 < 10, value=null, cell_suppressed=true). REQ-NFR-DASH-01 carry-forward.
|
||
6. Verify `assist_p95_latency_ms` is in the aggregates.
|
||
7. Verify `assist_cost_cents` is in the session_outcome.
|
||
8. Verify the C-3 budget check runs at shift-end.
|
||
9. Verify the tech-debt fixes: aggregation cache survives restart (mock), cookie-secret warning, credential status enum, argon2id offloaded.
|
||
- **Acceptance criteria:** k-anon threshold enforced (12 not suppressed, 5 suppressed). Assist metrics in the dashboard. p95 latency + cost in the aggregates. Tech-debt fixes verified. No per-learner data in any response.
|
||
|
||
#### TASK-12-06 — P2 verification matrix
|
||
- **Persona:** lead-developer
|
||
- **File:** `.ciagent/VERIFY-P2-v0.5.md` (new — pre-verify checklist)
|
||
- **Content:** REQ-ID → test mapping for P2. Confirm all P2 REQ-IDs (REQ-NFR-ASSIST-01, REQ-IDEATE-04, REQ-IDEATE-06, REQ-IDEATE-07) have covering tests. List each test file + what it verifies. Cross-reference with P1 VERIFY.
|
||
- **Acceptance criteria:** Every P2 REQ-ID has at least one covering test listed. Matrix is complete (no gaps).
|
||
|
||
---
|
||
|
||
# Final Phase (P3) — Review + Audit + Milestone Ship
|
||
|
||
**Branch:** `phase/03-final-review-ship` → merged to `milestone/v0.5-live-assist` → merged to `main`
|
||
**Ship:** `v0.1.13` (final patch = v0.5 milestone release)
|
||
**REQ-IDs covered:** all v0.5 REQ-IDs (milestone-complete verification)
|
||
|
||
### Tasks (delegated to ciagent-review + ciagent-audit + ciagent-ship)
|
||
|
||
1. Run branch gate → create `phase/03-final-review-ship`
|
||
2. `ciagent-review` — multi-persona review across P1 + P2; auto-apply P0 fixes, flag P1+
|
||
- **Security-engineer review focus:** LiveAssistGuardrail 3-layer enforcement (REQ-ASSIST-03 — safety-critical), guardrail tuning corpus FP/FN rates (REQ-IDEATE-04), PII policy (REQ-IDEATE-05), consent disclosure (D-070), incremental audit-log write (REQ-IDEATE-09)
|
||
- **Voice-engineer review focus:** build_assist_pipeline (reuses v0.1 services — no rewrite), Piper TTS default (D-065), warm WebRTC + reconnect (REQ-IDEATE-08), p95 latency measurement (D-072)
|
||
- **Backend-engineer review focus:** assist session API, SessionRecorder extension (D-063 — no mastery for assist), mode-conflict guard (REQ-IDEATE-03), cohort aggregation assist branch (D-062)
|
||
- **Data-engineer review focus:** turns table guardrail_verdict migration (additive), assist metrics no schema change (D-062), k-anon suppression for assist, aggregation cache persistence (tech-debt)
|
||
3. `ciagent-audit` — reconstruction test, file discipline, branch hygiene, commit discipline
|
||
4. `ciagent-ship` — merge phase/03 → milestone/v0.5-live-assist → main; tag v0.1.13; create release with full milestone summary
|
||
5. Update REQUIREMENTS.md (all v0.5 REQ → complete), ROADMAP.md (v0.5 → complete; v0.6 = low-bandwidth + IDEATE-10..13)
|
||
6. Commit: `docs(milestone): complete v0.5-live-assist`
|
||
7. Clear checkpoint
|
||
|
||
---
|
||
|
||
# REQ-ID Coverage Matrix
|
||
|
||
| REQ-ID | Phase | Slice(s) | Coverage |
|
||
|--------|-------|----------|----------|
|
||
| REQ-ASSIST-01 | P1 | SLICE-02, SLICE-05, SLICE-07 | Hands-free voice companion — tap-to-talk invocation (D-071) + assist voice loop (build_assist_pipeline) + __main__.py wiring |
|
||
| REQ-ASSIST-02 | P1 | SLICE-01 | Context-aware — AssistContextBinder loads path week + scenario tag + learner theta from SQLite (D-059) |
|
||
| REQ-ASSIST-03 | P1 | SLICE-03, SLICE-04, SLICE-08 | Guardrails: coaches not does — LiveAssistGuardrail 3-layer (D-060, D-068) + tuning corpus + adversarial test + e2e guardrail test |
|
||
| REQ-NFR-ASSIST-01 | P2 | SLICE-09 | Latency <600ms target, ≤650ms pilot (D-072) — p95 measurement |
|
||
| REQ-NFR-ASSIST-02 | P1 | SLICE-02 | Hands-free invocation — tap-to-talk only in v0.5 per D-071 (no wake-word — deferred to v0.6) |
|
||
| REQ-NFR-ASSIST-03 | P1 | SLICE-03, SLICE-04, SLICE-08 | 3-layer guardrail enforcement — prompt rules + regex output filter + audit log + tuning corpus + adversarial test |
|
||
| REQ-NFR-ASSIST-04 | P1, P2 | SLICE-01, SLICE-02, SLICE-10 | Shift-bounded session model (D-062) + 8h auto-end (D-069) + aggregation as session_type=assist (no schema change) |
|
||
| REQ-IDEATE-01 | P1 | SLICE-04 | Guardrail tuning corpus + adversarial bypass test (pre-ship) |
|
||
| REQ-IDEATE-02 | P1 | SLICE-03, SLICE-05 | In-loop guardrail processor pipeline test + GuardrailContext.role 'assist' extension |
|
||
| REQ-IDEATE-03 | P1 | SLICE-01 | Mode-conflict enforcement: assist vs practice mutual exclusivity + server-side guard |
|
||
| REQ-IDEATE-04 | P2 | SLICE-09 | Measurable NFR targets — p95 ≤650ms pilot (D-072) + guardrail FP <5% / FN measured + trended nightly |
|
||
| REQ-IDEATE-05 | P1 | SLICE-04 | Customer-speech PII policy — retain with redaction + consent + 30-day retention |
|
||
| REQ-IDEATE-06 | P2 | SLICE-12 | v0.4 P1+ tech-debt wave — 8 findings (aggregation cache, cookie-secret, credential status, argon2id, rate-limit mock, credential audit, zoneinfo, f-string SQL) |
|
||
| REQ-IDEATE-07 | P2 | SLICE-11 | Assist cost tracking + C-3 budget check — per-turn cost + monthly estimate |
|
||
| REQ-IDEATE-08 | P1 | SLICE-06 | WebRTC mid-shift drop + reconnect logic — state machine + chaos test |
|
||
| REQ-IDEATE-09 | P1 | SLICE-08 | Audit-log incremental write — persist ASR + LLM + verdict before TTS start |
|
||
|
||
**v0.5 total: 16/16 REQ-IDs covered (3 ASSIST + 4 NFR + 9 IDEATE). 0 partial. 0 deferred within v0.5.**
|
||
|
||
---
|
||
|
||
# Risk Mitigation Matrix
|
||
|
||
| Risk ID | Severity | Slice(s) | Mitigation |
|
||
|---------|----------|----------|------------|
|
||
| **R-ASSIST-01** | high | (deferred to v0.6) | Picovoice MAU pricing — v0.5 ships tap-to-talk only (D-071); wake-word deferred to v0.6 with native client |
|
||
| **R-ASSIST-02** | high | SLICE-05, SLICE-09 | C-8 <600ms at risk — Piper TTS (D-065) + ≤150-token prompt (D-066) + p95 measurement (D-072 pilot tolerance ≤650ms) |
|
||
| R-ASSIST-03 | medium | SLICE-06 | Wake-word → first-audio ~850-1150ms — N/A in v0.5 (tap-to-talk, not wake-word); warm WebRTC (D-067) minimizes tap-to-talk latency |
|
||
| R-ASSIST-04 | medium | (deferred to v0.6) | Android foreground-service mic — N/A in v0.5 (tap-to-talk, no background mic) |
|
||
| R-ASSIST-05 | medium | (deferred to v0.6) | OEM battery kill switches — N/A in v0.5 (no foreground service) |
|
||
| **R-ASSIST-06** | medium | SLICE-04 | Output filter false positives — tuning corpus (REQ-IDEATE-01) + one retry + canned fallback (D-068) |
|
||
| **R-ASSIST-07** | high | SLICE-03, SLICE-04, SLICE-09 | Output filter false negatives — defense-in-depth (prompt + regex + audit) + adversarial test + nightly FN trending + post-v0.5 LLM-as-judge (REQ-IDEATE-10, v0.6) |
|
||
| **R-ASSIST-08** | high | SLICE-02 | Privacy/consent — consent disclosure (D-070) + PII redaction (REQ-IDEATE-05) + D-073 legal review flag |
|
||
| R-ASSIST-09 | medium | SLICE-06 | Warm WebRTC dropped mid-shift — reconnect state machine + 30s wait + chaos test (REQ-IDEATE-08) |
|
||
| R-ASSIST-10 | low | (pilot) | Server concurrency — single-learner (D-007), ≤1 concurrent assist shift; v0.6+ concurrency hardening |
|
||
| R-ASSIST-11 | medium | SLICE-02 | Shifts abandoned — 8h auto-end (D-069) + `outcome='auto_ended'` in aggregation |
|
||
| R-ASSIST-12 | low | SLICE-01 | Context-binding reads stale state — learner declares context at shift start (D-059); server reads fresh from SQLite |
|
||
| R-ASSIST-13 | medium | (deferred to v0.6) | Porcupine false triggers — N/A in v0.5 (no wake-word) |
|
||
| R-ASSIST-14 | medium | SLICE-08 | Battery drain — N/A in v0.5 (no foreground service); incremental audit-log write (REQ-IDEATE-09) preserves audit trail on abrupt termination |
|
||
|
||
**Coverage: 3 high risks (R-ASSIST-02/07/08) + 4 medium risks (R-ASSIST-06/09/11 + R-ASSIST-03 reduced) + 2 low risks addressed. 5 risks deferred to v0.6 (R-ASSIST-01/04/05/13/14 — all wake-word/foreground-service related, N/A in tap-to-talk v0.5).**
|
||
|
||
---
|
||
|
||
# Open Questions Deferred to EXECUTE
|
||
|
||
1. **build_assist_pipeline design — mode param vs separate builder:** The plan uses a separate `build_assist_pipeline()` (vs a `mode="assist"` param on `build_pipeline()`). The separate builder is cleaner (the assist pipeline has a different structure — no opening line, in-loop guardrail processor, Piper default). The executor should verify the shared helpers (`_build_transport`, `_build_stt`, `_build_llm`) are importable from `server/pipeline.py` without circular imports. If circular imports arise, extract the shared helpers to `server/pipeline_helpers.py`.
|
||
|
||
2. **WebRTC endpoint for assist — `/api/assist/webrtc` vs `/pipecat/webrtc` with a mode param:** The plan uses a separate `/api/assist/webrtc` endpoint (distinct from the practice `/pipecat/webrtc`). This is cleaner (the assist endpoint calls `build_assist_pipeline` + `WarmWebRTCManager.open`; the practice endpoint calls `build_pipeline`). The executor should verify the mode-conflict guard is called on BOTH endpoints (TASK-07-01).
|
||
|
||
3. **AssistContextBinder `top_rubric_criterion` — how to pick the coaching focus:** The plan says "the rubric criterion with the lowest mean score for this learner." The executor should decide: read from `mastery_gate_events` (Postgres, requires `pg_store`) or from the SQLite `mastery_progress` table (no Postgres dependency). Recommendation: fall back to the first rubric criterion if no mastery data is available (the assist voice loop should work without Postgres — D-050 graceful degradation).
|
||
|
||
4. **Guardrail retry mechanism — how to inject the retry instruction:** The `LiveAssistGuardrailProcessor` (TASK-05-02) retries on a block by injecting `RETRY_INSTRUCTION` as a new LLM context message + re-running the LLM. The executor should verify Pipecat's `LLMContextAggregator` supports injecting a message + re-running the LLM within a single `process_frame` call. If not, the retry may need to be a separate pipeline task or a direct LLM call (bypassing the pipeline).
|
||
|
||
5. **Turns table `guardrail_verdict_json` — JSON column vs separate table:** The plan uses a JSON column on the `turns` table (additive migration — TASK-01-01). This is simpler than a separate `guardrail_verdicts` table. The executor should verify the JSON column is queryable for the nightly FN trending (SLICE-09 — `nightly_trend` samples turns + parses the JSON).
|
||
|
||
6. **Assist cost storage — per-turn or per-shift:** The plan logs per-turn cost + aggregates at shift-end (`assist_cost_cents` in `session_outcome`). The executor should decide: store the per-turn cost in the `turns` table (additive — a `cost_breakdown_json` column) or only aggregate at shift-end. Recommendation: aggregate at shift-end (simpler — no turns-table migration for cost).
|
||
|
||
7. **Tap-to-talk client — button on the React web app vs a separate control:** The plan uses a "press to talk" button on the `AssistControl` component (TASK-02-03). The executor should verify the Pipecat client SDK supports push-to-talk (hold to speak, release to send) over WebRTC. If not, the tap-to-talk may need a tap-to-start/tap-to-stop pattern (tap to start recording, tap to stop + send).
|
||
|
||
8. **D-073 Canada consent law review:** The consent disclosure (D-070) is implemented regardless of the legal review. The legal review (R-ASSIST-08) is an open question for the orchestrator — it's a legal/ethical surface, not purely technical. The plan flags this for the orchestrator's attention. The executor should NOT block on the legal review (the disclosure is the mitigation; the review is additional guidance).
|
||
|
||
---
|
||
|
||
# Summary
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Execution phases | 2 (P1: assist core + guardrail, P2: integration + tech-debt + NFR measurement) + 1 final (P3: review + ship) |
|
||
| Slices | 12 (8 in P1, 4 in P2) |
|
||
| Tasks | 33 (24 in P1, 9 in P2) |
|
||
| REQ-IDs covered | 16/16 (3 ASSIST + 4 NFR + 9 IDEATE) |
|
||
| Risks addressed | 9/14 (3 high, 4 medium, 2 low — 5 deferred to v0.6 as N/A in tap-to-talk v0.5) |
|
||
| Waves | P1: 4 waves (2+2+2+2 parallel slots), P2: 2 waves (3+1 parallel slots) |
|
||
| Max parallelism | 3 slices per wave (within 5-agent limit) |
|
||
| Personas active | 5 (lead-developer, voice-engineer REACTIVATED, backend-engineer, security-engineer, data-engineer) |
|
||
| Personas deactivated | 2 (devops-engineer — no deploy changes; frontend-engineer PROVISIONAL — assist control surface ~100-150 LOC) |
|
||
| New pip deps | 0 (v0.1 voice pipeline reused unchanged; guardrail is pure-Python regex; aggregation uses existing asyncpg) |
|
||
| New npm deps | 0 (assist control surface reuses existing React + react-router-dom) |
|
||
| New SQLite migrations | 1 (0004_assist.sql — session_type + guardrail_verdict_json + index) |
|
||
| New Postgres migrations | 0 (cohort_aggregates is generic on metric TEXT — no DDL change per D-062) |
|
||
| Ship targets | v0.1.11 (P1), v0.1.12 (P2), v0.1.13 (P3 = v0.5 milestone release) |
|
||
| Binding constraints honored | D-071 (tap-to-talk only), D-063 (assist ≠ mastery), D-072 (≤650ms pilot), v0.1 pipeline FIXED (reused, not rewritten) | |