Compare commits
103 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 5838bdeb51 | |||
| 0a091c2e50 | |||
| 73b493915f | |||
| baa9ef5d2d | |||
| d72cea1786 | |||
| 192fa6a655 | |||
| 4193552460 | |||
| 0324067fa7 | |||
| 75bde88e0d | |||
| e14e1fa3d2 | |||
| 843f16d737 | |||
| 3291914eaa | |||
| 4465e6325d | |||
| c9709e0f38 | |||
| 62bff575af | |||
| 8a43e95a4d | |||
| ee5be13c94 | |||
| 1da814e790 | |||
| cc7ec5d03f | |||
| 5c2829d9df | |||
| ec4648b37a | |||
| c5369b407b | |||
| 282f5ef150 | |||
| 3010bc4b96 | |||
| 35c4c386b5 | |||
| d8d2cebfc5 | |||
| b2a2a4023b | |||
| de431852c4 | |||
| 64e4842976 | |||
| a64733a262 | |||
| 0072689e4d | |||
| 3110be2f15 | |||
| bd5b0fee95 | |||
| 2e6d92dfa1 | |||
| e3c8cc7145 | |||
| 1e5ba6568a | |||
| f3f3746da7 | |||
| dd9bda27c9 | |||
| c80381c4df | |||
| 4238a06bca | |||
| 873d22069f | |||
| 5441e10a0e | |||
| 9fedeea767 | |||
| 6873ce6777 | |||
| c90bc7e618 | |||
| ff183b8fca | |||
| 08badd5ed6 | |||
| 9d8be6f466 | |||
| 6d6639b268 | |||
| 83246ed65b | |||
| 33f4c2d61d | |||
| 2c2f68e00e | |||
| 135ea21a61 | |||
| 1648467828 | |||
| 1a68d808fb | |||
| 2c68b44c1a | |||
| fb2db4ee97 | |||
| 485d86e117 | |||
| e798e1a6da | |||
| cd45097e52 | |||
| 1d03f0c8f5 | |||
| 12b2300f6f | |||
| e1460aed3c | |||
| b7a56d35bc | |||
| 16fb52d8f7 | |||
| a905eb8c67 | |||
| ed243594d2 | |||
| c760f9af2b | |||
| b4ae388f22 | |||
| 925ab096fb | |||
| 82ae839cd4 | |||
| b6c1bc9d54 | |||
| f281eeaf62 | |||
| 22d4fa212c | |||
| 007865a5a1 | |||
| 04bdccf189 | |||
| f3071e4b79 | |||
| 3d72fd28ec | |||
| 97893f2386 | |||
| e63b996361 | |||
| 0b34255855 | |||
| 6ab0ae2c0a | |||
| 9ff86f9cd0 | |||
| 4acffac71e | |||
| 82b9de382a | |||
| 430b4a727d | |||
| b52bef93e5 | |||
| e798c52edb | |||
| b303a41a45 | |||
| 3322c1dab9 | |||
| 0bde9cbf2e | |||
| 85028678c7 | |||
| 1a46606827 | |||
| 0fccb8d250 | |||
| 2474678da6 | |||
| 3f58fc3454 | |||
| edf586b03a | |||
| 4439486858 | |||
| b49b9189fe | |||
| f75352d0f0 | |||
| 26b4a5be60 | |||
| b6850fc036 | |||
| cfdceac17a |
+119
-25
@@ -6,6 +6,33 @@ Nextcraft is a TypeScript monorepo (pnpm workspaces + turborepo) with a Next.js
|
||||
|
||||
**v0.3 additions (Credential Engines):** real credential engines replace v0.2 mock inputs — a sandbox fabric (isolated per-learner coding environments via Linux user/mount/pid/net namespaces), a live build-telemetry pipeline (WebSocket ingest + SQLite-ordered event log), a process-trace grading engine, seeded per-learner variant task generation, and a voice-based oral defense (STT/TTS via a new provider-agnostic voice layer). **First real persistence introduced: SQLite** (`ai_service/telemetry/`, grading, variant, defense stores). Lab/Assessor/Proctor agents are re-grounded onto real telemetry/traces. **Identity/age-gating (KYC) deferred per founder directive** — no security engineer persona; secrets-hygiene checklist only.
|
||||
|
||||
**v0.4 additions (Distribution & Bootstrap CLI, founder directive D-016):** a new `apps/cli` package — the `nextcraft` bootstrap CLI (`doctor`/`bootstrap`/`verify`/`dev`) compiled to a self-contained linux x64 binary via **Node SEA** (probe-verified: Go/Rust absent, node v24.15.0 SEA-capable), installed by a repo-served one-liner script that resolves the latest Gitea release, downloads binary + sha256 sidecar, verifies, and installs to `~/.local/bin`. Every release from v0.4 onward attaches the binary + checksum as release assets (the "ongoing binaries" requirement). The CLI is a thin wrapper: all orchestration logic stays in `apps/ai-service/scripts/` (bootstrap.sh/dev.sh) — the CLI composes them via subprocess (A-202), duplicating nothing. Previously-planned v0.4 seams (real STT/TTS, KYC, design/sim envs, seq-lease) move to v0.5.
|
||||
|
||||
### v0.6 Research Conclusions (Phase 0 RESEARCH)
|
||||
|
||||
32. **D-048 Marketplace module layout mirrors D-031** — one new engine module `ai_service/marketplace/` inside apps/ai-service (no new apps, no new ports): `base.py` (pydantic contracts: `Posting`, `Employer`, `PostingTags`, `IngestPosting`), `store.py` (6th D-027 store family, D-049), `ingest.py` (PostingSource adapters, D-050), `tagging.py` (D-052), `embedding/` (protocol + mock + openai_compat + factory, D-051), `search.py` (hybrid facet+cosine ranking, A-405), `enrichment.py` (D-053), `resume.py` (D-054), with the HTTP surface in `ai_service/api/marketplace.py` (`/v1/marketplace/*` router) + `api/marketplace_admin.py` (`/v1/marketplace/admin/*`, D-057) composed via `deps.py` app.state DI (`get_marketplace_store`, `get_embedding_provider`). Boundary rules identical to identity/voice: `marketplace/` never imports `agents/` or `api/`; settings in `config.py` (backend-engineer territory); the G-18 stub MOVES from `api/identity.py` into the new router and becomes real (D-058) — `identity/` keeps only the `require_verified_adult` gate dependency it exports today.
|
||||
33. **D-049 Sixth store = one SQLite DB, five tables, DefenseStore conventions** — the marketplace store joins the shared `~/.nextcraft/data/nextcraft.db` (all 5 prior stores already share it; `Settings().db_path`). Tables: `posting` (id PK, source, external_id, employer_id, title, description, skills/seniority/category/location/remote/salary JSON+portable columns, status `draft|published|closed`, posted_at, unique constraint on (source, external_id) for idempotent upsert per A-402); `employer` (id PK, ingested raw fields, unique per source); `embedding` (posting_id PK, `vector` BLOB of struct-packed little-endian float32, `model`, `dim` — A-403/A-404 per-row provenance so lazy re-embed on provider change works); `application` (id PK, posting_id, learner_id, status `submitted`, created_at, unique (posting_id, learner_id) for A-408 idempotent apply); `employer_enrichment` (append-only: id PK, employer_id, field, value, source `llm|manual`, model, created_at — A-409). All tables mirror the probe-verified identity/store.py mechanics: WAL/synchronous=NORMAL/busy_timeout/foreign_keys pragmas, `sa_type=String/JSON` portable columns, `_validate()` hooks, Protocol + SQLite impl pair, surfaced IntegrityError, tz normalization. `VectorStore` is a protocol method-group on the marketplace store (pgvector-swap-ready, D-019/D-027 pattern).
|
||||
34. **D-050 PostingSource = protocol + two adapters, no scheduler** — `PostingSource` protocol: `fetch() -> Iterable[IngestPosting]` (pydantic, A-402 shape: source, external_id, employer + posting fields). Two v0.6 implementations: `seeded_corpus` (in-repo JSON seed ported ONCE from `packages/mock-data/jobs.ts` (20 jobs) + `employers.ts` (10 employers) per D-021 convention — shared IDs, mock `matchScore` DROPPED since D-051 computes real ones) and `admin_api` (the founder-authored path; the admin router calls ingest directly rather than an HTTP hop). Ingest upserts idempotent on (source, external_id); a `POST /v1/marketplace/admin/ingest` route triggers a source run explicitly (founder-invoked — no cron/crawler, A-401/A-402). The 3M+ external corpus lands later as a third adapter with zero store/route changes.
|
||||
35. **D-051 EmbeddingProvider = D-030 mirror with a corrected real path** — protocol `embed(texts: list[str]) -> list[EmbeddingVector]` (frozen pydantic: `values: list[float]`, `model`, `dim`) in `marketplace/embedding/base.py`. Deterministic mock: 256-dim feature-hash of normalized tokens (pure stdlib, similarity deterministic — tests never call the cloud). Real: `OpenAIEmbedProvider` on the shared httpx pool posting `{"model", "input": [texts]}` to `{AI_EMBED_BASE_URL}/embeddings` (OpenAI-compatible; `_sanitize` key redaction mirroring `openai_audio.py`). **Probe correction to A-404:** ollama-cloud does NOT expose `/v1/embeddings` (404 path-not-found; chat/completions auth-gates 401) — so `AI_EMBED_BASE_URL` must point at a different OpenAI-compatible endpoint (local ollama `:11434/v1` supports it, or any embeddings cloud); provider selection `AI_EMBED_PROVIDER=mock|openai-embed` (default mock, G-11 boot-safe fallback with loud log — exact `voice/factory.py` pattern, `AI_EMBED_MODEL`/`AI_EMBED_API_KEY` env-only). Storage per A-403: struct-packed `<{dim}f` float32 BLOB + model + dim per row; cosine in pure Python (sum of products / norm product, ≤10⁴ postings); candidate side embeds competency-profile text on demand, never stored. Confidence in A-404's parenthetical drops it to ~0.5 but the protocol design is unaffected (endpoint-agnostic by config is the D-014 pattern anyway).
|
||||
36. **D-052 Tagging = D-020 structured_completion, no new agent** — `marketplace/tagging.py` composes the EXISTING `structured_completion(provider, messages, model, schema, schema_hint)` (agents/structured.py, all 4 layers) with a `PostingTags` pydantic schema (skills: list[str] ≤10, seniority enum, category enum) + a versioned prompt constant in `prompts/` (or module-local, matching variants' pattern); no BaseAgent needed — tagging is a one-shot extraction like grading's rubric call. Mock-first for free: `settings.provider` defaults to `mock` and `structured_completion` runs against `MockLLMProvider.chat`, so tests stay cloud-free. Tags persisted as a JSON column on the posting row (D-049); tag facets feed the search filter sidebar (A-405 hard filters); re-tagging on posting update overwrites the tags column (single source of truth, no append-only need — unlike enrichment D-053, tags are derived data).
|
||||
37. **D-053 Enrichment = append-only provenance table + merge-on-read** — per A-409: `employer_enrichment` rows are append-only `(employer_id, field, value, source: llm|manual, model, created_at)`; read precedence **manual > llm > ingested-raw** computed at API-response time (a small `merge_employer()` over rows, raw fields last); every enriched field carries `{value, source}` provenance markers in the API response (A-304 honesty — LLM copy visibly marked). Trigger: a new employer's first published posting fires a one-shot LLM structured extraction (D-020, same layering as D-052, schema: description, social_links, logo_ref) — founder override = upsert a `manual`-source row, never a mutation of `llm` rows. No employer self-serve anywhere (A-407).
|
||||
38. **D-054 Resume parse = pypdf (probe-verified) → D-020 draft extraction** — pypdf 6.19.0 pip-reachable through the existing venv channel (probe 1; the ONLY new dependency, pinned `pypdf>=6.19,<7` in pyproject.toml). Pipeline: upload guard (PDF/TXT/MD only, ≤2 MB, A-410) → text extraction (`pypdf.PdfReader` page-text join for PDFs; raw decode for TXT/MD) → empty-extract on a PDF = honest 422 "text layer required" (no OCR) → cap ~100K chars → D-020 structured extraction into a `ParsedProfile` pydantic schema mapped to the competency-profile shape (competencies, skills, experience bands) → response returns as **DRAFT** (`status: "draft"`) — nothing persists to the learner profile until an explicit confirm call (REQ-6-005 review gate). Raw resume bytes never stored, never logged (A-305 pattern; the existing `_scrub_pii_validation` handler precedent applies). Tests use small generated PDFs or TXT fixtures — cloud-free.
|
||||
39. **D-055 Web migration = client shells + engine-client extension; remove 2 generateStaticParams** — board/detail/employer-profile become one static client shell each fetching at runtime per A-406 (the proven build/defend pattern): `marketplace/page.tsx` keeps its server shell but `JobFilters` receives fetch-backed data (it already owns all filter logic client-side — the facet contract survives unchanged); `jobs/[jobId]` and `employers/[employerId]` drop `generateStaticParams` (the ONLY two removals — learner catalog/learn/build/defend keep theirs, their data stays mock) and render a client shell that fetches then renders (or `notFound()` on 404). `engine-client.ts` gains a marketplace section: `listPostings(filters)`, `getPosting(id)`, `getEmployer(id)`, `searchPostings(query)` → typed against new `packages/types` marketplace wire shapes (data-engineer dual-schema sync); 403 discrimination (verify-CTA vs allowlist) already works. `packages/mock-data` jobs.ts/employers.ts go dormant for the marketplace path (still imported by employer/admin mockups — NOT deleted, A-406/A-407); the seed port (D-050) copies values once. No build-time DB reads — D-039 export build stays repo-only.
|
||||
40. **D-056 SEO = build-time metadata on static set + client JSON-LD on runtime set + ai-service sitemap/robots** — per A-411, split: (a) build-time-static pages (landing, catalog/stack detail, marketplace board, pricing) get `export const metadata` (probe 7: the static-export-compatible API the app already uses in layout.tsx/enroll) + OG tags + server-rendered JSON-LD (`Organization` on employer-adjacent static, `ItemList` on the board) rendered into the export; (b) runtime-data pages (job detail, employer profile) inject `JobPosting`/`Organization` JSON-LD + title client-side after fetch — accepted JS-crawler limitation, recorded; (c) `/sitemap.xml` + `/robots.txt` as plain FastAPI GET routes on the api app enumerating the static page list + all `published` postings — **probe 6 verified mount order: API routes beat the `StaticFiles("/")` mount**, so they win over the export even in single-port production; do NOT use Next `app/sitemap.ts` (it enumerates at build time and cannot see runtime postings under `output: 'export'`). robots.txt points at `/sitemap.xml`; sitemap URLs use the deployed origin (env-configured `AI_PUBLIC_BASE_URL`-style setting, backend-engineer territory).
|
||||
41. **D-057 Admin guard = AI_ADMIN_TOKEN header check, env-only, fail-closed** — all `/v1/marketplace/admin/*` routes (postings create + lifecycle transitions draft→published→closed, ingest trigger, enrichment manual-override, applications review) take a FastAPI dependency `require_admin` comparing `X-Admin-Token` (or `Authorization: Bearer`) against `Settings().admin_token` (`AI_ADMIN_TOKEN`, env-only per D-014, probe: no admin token exists in config today — greenfield). Unset token = admin routes 403-fail-closed (never open; a loud boot warning tells the founder to set it, G-11 doctrine). This is the A-401 operator surface; D-043 learner gate composition is untouched — admin routes do NOT pass the learner allowlist/identity gates (different principal class).
|
||||
42. **D-058 Apply route goes real, composition unchanged** — `POST /v1/marketplace/apply` (moved from api/identity.py into api/marketplace.py, D-048) keeps the exact D-043 order (G-5 allowlist → `require_verified_adult` → rate caps), then: posting must exist and be `published` (404/409 otherwise), insert an `application` row (status `submitted`, unique (posting_id, learner_id) → duplicate apply is idempotent 200-with-existing, A-408), return an honest confirmation payload `{application_id, posting_id, status: "submitted"}` (the `stub/mock` markers die). Founder reviews via `GET /v1/marketplace/admin/applications` (D-057-guarded); no notification infra. Body becomes `{posting_id}` + the learner identity from the verified record rather than a client-asserted `job_id`.
|
||||
43. **D-059 v0.6 wave order = ROADMAP order confirmed with two couplings** — P1 postings backend must ship WITH the web client shells in the same phase (a live store behind mock-reading pages proves nothing — REQ-6-001's success criterion is surfaces reading real data); P2 tagging before P3 search because facets drive the sidebar filters AND the posting text embedded in P3 should include the normalized tags; P4's two workstreams (enrichment REQ-6-004, resume REQ-6-005) are independent of each other — parallelizable within the phase; P5 SEO depends only on P1 (routes exist to enumerate) but stays last-before-final per ROADMAP since sitemap needs `published` postings and the board metadata should reflect final facets; P6 final review + ship. Tagging/enrichment/resume all route through the same D-020 call site — one prompt-review pass covers all three.
|
||||
|
||||
### v0.5 Research Conclusions (Phase 0 RESEARCH)
|
||||
|
||||
25. **D-040 Voice real path = `OpenAIAudioProvider` on the shared httpx pool (D-R01)** — implements the D-030 protocol: `transcribe()` = multipart `POST {AI_VOICE_BASE_URL}/audio/transcriptions` (`file` + `model`, response_format=json → `TranscriptSegment`), `synthesize()` = streaming `POST /audio/speech` (JSON body, raw byte stream, `input` ≤4096 chars). Carries `descriptor = VoiceDescriptor(mode="server", ...)` which defense.py:143 already prefers — mode selection drops in with zero API changes. Constructor takes the lifespan `httpx.AsyncClient` (D-017; read=300s already tolerates multi-minute clips). Factory signature becomes `voice_provider_from_settings(settings, http_client)`. Errors sanitized with key redaction (mirror `llm/openai_compat.py:_sanitize`).
|
||||
26. **D-041 Voice route fixes (D-R02)** — defense.py:189 fmt derivation must strip codec params (`"webm;codecs=opus"` → `webm`, else real STT 400s); ~10 MB audio guard (422/413) before provider call; TTS route media_type becomes format-aware (or force `response_format=wav`). Web: defense-session POSTs the recorded blob (FormData), engine-client gains the audio-answer variant.
|
||||
27. **D-042 Identity = 5th D-027 store + provider protocol (D-R03)** — `ai_service/identity/` (D-031): `IdentityProvider` protocol (submit/poll/verify), deterministic mock, SQLite store modeled on DefenseStore (WAL, FK on, portable columns). Stores **derived age_band** (`16-17`/`18+`, never raw DOB) + document **refs** (never raw docs); mock verdicts carry a `mock` marker so downstream never displays them as production-verified (A-304). PII hygiene pinned by caplog sentinel test. **Negative finding:** no age-gate UI or enrollment API exists today — the flow is built, not swapped (D-010's "visual flow" is vestigial: one mock field).
|
||||
28. **D-043 Gate composition (D-R04)** — api/-layer dependencies in order: G-5 allowlist (403 pilot guard, retained) → identity verdict (403 + verify-CTA payload) → rate caps (429). School 16+ gates variant generation + sandbox create + defense start; marketplace 18+ via reusable `require_verified_adult` demonstrated on one minimal gated route (marketplace has no backend today — the thin route proves the contract end-to-end). All middleware-layer, never in the manager.
|
||||
29. **D-044 Environment registry at the template/variant layer (D-R05)** — `TaskTemplate.environment: Literal["build","design","simulation"]` → carried through VariantRecord → API → TS `TaskVariant`; surfaces the existing-but-dead `test_command` field (Run/Test buttons stop hardcoding pytest). Manager/backend/workdir unchanged (D-024 untouched — A-307: an environment is a typing over starter files + command policy). Per-kind exec command policy at the api/ exec route (template-declared harness + generic file/nav commands, 422 on violation). Grading digest stays kind-agnostic by construction (features derive from event kinds — pinned with a digest test over a synthetic design-kind trace). Starter contents: design = SVG/HTML/schematic artifacts + validate/render harness (stack-designer c001/c002 already sanctioned); simulation = parameterized benchmark script + dataset files (stack-science, stack-operator).
|
||||
30. **D-045 Seq-ack protocol (D-R06/D-R07)** — new WS frame `{"type":"seq_ack","seq":N}` emitted per successful append from `IngestSession._append` (durable `latest_seq`; O(1); advisory — gap detection stays authoritative, G-3/G-4 unchanged). The capture agent's supervisor loop (which today discards all non-close frames) parses text frames and trims spool+pending to `seq > ack` under `_emit_lock` via atomic `Spool.rewrite` — closing the one-line replay-margin gap (`_flush_locked` pops N in-flight frames but `replay_margin()` requeues only `_last_sent`). An explicit spool bound is added (A-309's "spool cap exists" premise was factually wrong on disk). Regression test: mid-burst kill in the test_durability.py real-server harness (uvicorn + KillableProxy) — the exact scenario the P07 de-flake documented as uncovered.
|
||||
31. **D-046 v0.5 wave order (D-R08)** — P1 seq-lease (smallest, protocol-only, fixes transport before env phases add reconnecting producers), P2 voice, P3 identity, P4 environments, P5 final. Phase numbers renumbered accordingly (was 1=voice..4=seq-lease in the initial ROADMAP draft).
|
||||
|
||||
### Confirmed Technology Stack (v0.2)
|
||||
|
||||
| Technology | Version | Purpose |
|
||||
@@ -26,6 +53,8 @@ Nextcraft is a TypeScript monorepo (pnpm workspaces + turborepo) with a Next.js
|
||||
| pydantic | 2.13.x | Request/response models, structured outputs |
|
||||
| pydantic-settings | 2.15.x | Settings + env-file loading (replaces python-dotenv) |
|
||||
| httpx | 0.28.x | Async LLM HTTP client (ollama-cloud + local providers) |
|
||||
| sqlmodel / sqlalchemy | 0.0.24 / 2.x | Typed SQLite persistence for the v0.3 engine stores (D-027) |
|
||||
| python-multipart | 0.0.x | Multipart audio upload for the defense answer route (REQ-3-006) |
|
||||
| sse-starlette | 3.4.x | SSE framing, ping keep-alive |
|
||||
| pytest | 9.x | Test runner |
|
||||
| pytest-asyncio | 1.4.x | Async tests (auto mode) |
|
||||
@@ -45,6 +74,16 @@ Deliberately **not** used: openai-python SDK (the `LLMProvider` protocol is the
|
||||
7. **D-022 Monorepo integration** — zero-dependency shim `package.json` in apps/ai-service + `ai#*` turbo passthrough tasks (`cache:false, outputs:[]`) + root `ai:dev`/`ai:test` scripts + idempotent venv bootstrap.
|
||||
8. **D-023 Testing** — pytest-asyncio auto mode; TestClient `client.stream()` for SSE; httpx MockTransport for byte-exact provider parser tests; scripted mock provider incl. failure modes. Tests never call the cloud.
|
||||
|
||||
### v0.4 Architecture Decisions (from Research — Distribution & Bootstrap CLI)
|
||||
|
||||
18. **D-033 Binary toolchain = Node SEA (probe-verified)** — Go and Rust are absent from this box; node v24.15.0 ships SEA support (`--experimental-sea-config`, postject-free on linux via `cp node nextcraft && node sea-config` … blob injection with the system `dd`/`npx postject` if needed). CLI source lives in `apps/cli` (TypeScript, compiled to a single CJS bundle by esbuild, then SEA-injected into a copy of the node binary → `nextcraft-linux-x64`). Fallback if SEA breaks: python3 `zipapp` (3.11.2 available). No new toolchain deps beyond dev-scoped esbuild.
|
||||
19. **D-034 CLI = thin wrapper, orchestration stays in scripts/** — `nextcraft` composes `apps/ai-service/scripts/bootstrap.sh` and `scripts/dev.sh` equivalents via `spawn` with inherited stdio and timeout guards (A-202/A-209). doctor/bootstrap/verify implement only *checking* logic (prereqs, env template, health) — never re-implement installs. This keeps one source of truth for bootstrap semantics.
|
||||
20. **D-035 Install path = repo raw `install.sh` + Gitea latest-release API** — the one-liner `curl -fsSL <forge>/coreci/nextcraft/raw/main/scripts/install.sh | bash` resolves `GET /api/v1/repos/coreci/nextcraft/releases/latest`, downloads the `nextcraft-linux-x64` + `nextcraft-linux-x64.sha256` assets, verifies sha256 (`shasum -a 256`), installs to `~/.local/bin` (PATH hint), and degrades to printed source-bootstrap instructions when no binary asset exists or the platform mismatches (A-203/A-204/A-206).
|
||||
21. **D-036 Ongoing binaries = ship-workflow asset step** — the release pipeline (v0.3's `ShipWorkflow.createRelease` equivalent, executed as the ship step's asset stage) builds the binary + checksum and attaches both to every Gitea release from v0.4 onward (A-205). Token resolution stays `.env*`-only (D-006/D-014); binaries are linux x64 only for v0.4 (macOS arm64 deferred — unverifiable on this box).
|
||||
22. **D-037 CLI package layout** — `apps/cli` is a pnpm workspace package (`@nextcraft/cli`): `src/` (entry, commands/, checks/, lib/), `scripts/build-binary.mjs` (esbuild bundle → SEA inject), unit tests runnable via `pnpm --filter @nextcraft/cli test` (node:test, no new test framework). Root `package.json` gains `cli:*` passthrough scripts mirroring the `ai:*` pattern (D-022).
|
||||
23. **D-038 Network mode (v0.3.5)** — dev binds 0.0.0.0 (`AI_HOST`, default 0.0.0.0, revert via 127.0.0.1); CORS + WS-origin gates read `AI_CORS_ORIGINS` (default `*` — any origin, safe only because credentials are never enabled; explicit comma list restricts); the web client derives the API base URL from the browser hostname at runtime (`engine-base-url.ts`: `NEXT_PUBLIC_AI_SERVICE_URL` override → `http://${window.location.hostname}:8420` → `localhost` server-side). Hotfix also fixes: SEA direct-run detection (`require("node:sea").isSea()` — argv shape differs by invocation style), installer honesty gate (silent `--version` = hard fail), bootstrap venv recovery (poisoned partial `.venv` removal + distro-specific `apt install python3.XX-venv` hint), and doctor venv-capability probe with bootstrap preflight.
|
||||
24. **D-039 Single-port deploy + unattended dev (v0.3.6)** — only :8420 is reachable behind HAProxy, so the web app ships as a **static export** (`output: 'export'`, `NEXT_PUBLIC_AI_SERVICE_URL=self` → relative same-origin fetches) served by the ai-service itself (`AI_WEB_STATIC_DIR` StaticFiles mount, default off; `nextcraft dev` auto-wires it when `apps/web/out` exists). Daemon surface: `dev -d` (detached, `~/.nextcraft/run/<clone-hash>/dev.{pid,log}`), `stop`, `log [-n N|-f]`. Durable state (DB `~/.nextcraft/data/`, sandbox workdirs `~/.nextcraft/sandboxes/`) moves **out of the repo** (founder directive; `expanduser` validator makes `AI_DB_PATH=~/...` env overrides work). API routes beat the static mount; unknown paths serve the export's 404.html.
|
||||
|
||||
### v0.3 Architecture Decisions (from Research — Credential Engines)
|
||||
|
||||
9. **D-024 Sandbox isolation = Linux namespaces via `unshare`** — per-learner sandbox runs as a subprocess entered into fresh user+mount+pid+network namespaces (`unshare --user --map-root-user --mount --pid --fork --net`). Probe-verified on this box: in-namespace uid=0, **network fully isolated** (0 interfaces), learner writes land in a per-sandbox directory; proc-remount not permitted here but not required. Chosen because no container runtime (docker/podman/bwrap/firejail) exists on the box and there is no sudo. A `SandboxBackend` protocol abstracts the spawner so a future containerd/runc backend can replace namespace-spawning without touching callers.
|
||||
@@ -65,14 +104,14 @@ Deliberately **not** used: openai-python SDK (the `LLMProvider` protocol is the
|
||||
|
||||
| Component | Description | Boundaries | Depends On |
|
||||
|-----------|-------------|------------|------------|
|
||||
| `ai_service/main.py` | FastAPI app factory, lifespan (httpx client pool, provider factory), CORS (localhost only), /health | App entry | config, llm, agents, api |
|
||||
| `ai_service/main.py` | FastAPI app factory, lifespan (httpx client pool, provider factory, SandboxManager + reaper loop, SQLite engine stores on app.state), CORS (localhost only, incl. PUT for file writes), /health; v0.3.6: optional StaticFiles mount of the exported web app when `AI_WEB_STATIC_DIR` is set (single-port deploy, D-039) | App entry | config, llm, agents, api, engines |
|
||||
| `ai_service/config.py` | pydantic-settings Settings (env_prefix="AI_", env_file, SecretStr key) | Configuration only | None |
|
||||
| `ai_service/api/` | Endpoints: chat.py (POST /v1/chat/stream, SSE), lab.py, assessment.py, proctor.py, mentor.py; deps.py (DI) | Composes agents + sessions; never imported by llm/ or agents/ | agents, llm |
|
||||
| `ai_service/api/` | Endpoints: chat.py (POST /v1/chat/stream, SSE), lab.py, assessment.py (POST /v1/assessment/evaluate v0.2 + POST /v1/assessment/grade v0.3), proctor.py, mentor.py, sandboxes.py (lifecycle + files/exec routes, G-5 abuse gates), telemetry.py (WS ingest + trace/gaps reads), variants.py (seeded per-learner variants), defense.py (defense loop, REQ-3-006); deps.py (DI) | Composes agents + sessions + engines; never imported by llm/ or agents/ | agents, llm, sandbox, telemetry, grading, variants, voice |
|
||||
| `ai_service/llm/` | types.py (Message; ChatDelta/ChoiceDelta removed in P3 — no consumers), base.py (LLMProvider protocol), openai_compat.py (ollama-cloud + local), mock.py (deterministic), factory.py | Never imports agents/ or api/ | config |
|
||||
| `ai_service/agents/` | base.py (BaseAgent ABC), registry.py, session.py (SessionStore), structured.py (JSON defense), coach/tutor/lab/assessor/proctor/mentor.py | Never imports api/ | llm, prompts, corpus |
|
||||
| `ai_service/agents/` | base.py (BaseAgent ABC), registry.py, session.py (SessionStore), structured.py (JSON defense), coach/tutor/lab/assessor/proctor/mentor.py + examiner.py (seventh agent, v0.3) | Never imports api/ | llm, prompts, corpus, telemetry |
|
||||
| `ai_service/prompts/` | Per-agent system prompt constants + render_context functions (str.format_map) | Data only | None |
|
||||
| `ai_service/corpus/` | Mock engine inputs: learner_context.py, telemetry.py (Lab/Proctor scenarios), artifacts.py (pre-baked artifacts, rubrics, transcripts) | Pydantic-typed; aligned with TS packages/mock-data by convention | None |
|
||||
| `scripts/` | bootstrap.sh (venv + pip install idempotent), dev.sh (exports keys from .ciagent/.env.secrets → uvicorn), test.sh (pytest), lint.sh (ruff check, G-3) | Dev entry points | pyproject.toml |
|
||||
| `ai_service/corpus/` | Mock engine inputs: learner_context.py, telemetry.py (Lab/Proctor scenarios), artifacts.py (pre-baked artifacts, rubrics, transcripts). Since v0.3 P6 these are DORMANT, test-only fixtures (dormant-header noted) — the live learner path uses real engine inputs | Pydantic-typed; aligned with TS packages/mock-data by convention | None |
|
||||
| `scripts/` | bootstrap.sh (venv + pip install idempotent), dev.sh (exports keys from .ciagent/.env.secrets → uvicorn), test.sh (pytest), lint.sh (ruff check, v0.2 G-3) | Dev entry points | pyproject.toml |
|
||||
| `tests/` | conftest.py (mock provider, settings override, TestClient), health, llm (MockTransport parser), agents (framework + per-agent), api (SSE stream tests) | Mock provider only — no cloud | all |
|
||||
|
||||
**Module boundary rules:** `llm/` never imports `agents/` or `api/`; `agents/` never imports `api/`; `api/` composes both via DI. `corpus/` is the only home of mock engine data. Prompts are code — versioned and reviewed in git.
|
||||
@@ -82,35 +121,50 @@ Deliberately **not** used: openai-python SDK (the `LLMProvider` protocol is the
|
||||
| Component | Description | Boundaries | Depends On |
|
||||
|-----------|-------------|------------|------------|
|
||||
| `ai_service/sandbox/` | `backend.py` (SandboxBackend protocol), `unshare_backend.py` (userns/mount/pid/net spawner, D-024), `manager.py` (lifecycle: create/list/snapshot/destroy + concurrency guard D-032), `workdir.py` (per-sandbox fs layout) | Never imports api/ or agents/; spawns subprocesses only | config |
|
||||
| `ai_service/telemetry/` | `models.py` (TelemetryEvent, TraceSpan), `store.py` (TraceStore protocol + SQLite impl D-027), `ingest.py` (WebSocket /v1/telemetry/ingest, seq gap detection D-026) | Persistence; never imports agents/ | config |
|
||||
| `ai_service/grading/` | `features.py` (deterministic trace digest D-028), `engine.py` (rubric scoring orchestration), `store.py` (GradeStore) | LLM only via digest; never sees raw trace | llm, telemetry, prompts |
|
||||
| `ai_service/variants/` | `templates.py` (task template library), `generator.py` (seeded LLM instantiation D-029), `store.py` (VariantStore) | LLM via structured output | llm, grading |
|
||||
| `ai_service/voice/` | `base.py` (VoiceProvider protocol D-030), `openai_audio.py` (STT/TTS vs compatible endpoint), `browser.py` (native SR/TTS fallback descriptor), `mock.py` (deterministic) | Never imports agents/ or api/ | config |
|
||||
| `ai_service/telemetry/` | `models.py` (TelemetryEvent, TraceSpan), `store.py` (TraceStore protocol + SQLite impl D-027), `ingest.py` (WebSocket /v1/telemetry/ingest, seq gap detection D-026; v0.5 REQ-5-007/D-045: emits advisory `seq_ack` frames — highest-contiguous received seq per successful append; capture agent trims its spool to the ack, closing the replay-margin gap) | Persistence; never imports agents/ | config |
|
||||
| `ai_service/grading/` | `features.py` (deterministic trace digest D-028 — kind-agnostic by construction, pinned over design/sim traces in v0.5), `engine.py` (rubric scoring orchestration), `store.py` (GradeStore) | LLM only via digest; never sees raw trace | llm, telemetry, prompts |
|
||||
| `ai_service/variants/` | `templates.py` (task template library; v0.5 REQ-5-005/D-044: `environment: Literal[build,design,simulation]` registry + per-kind starter files + harness/test commands, G-15 shlex-roundtrip validation), `generator.py` (seeded LLM instantiation D-029), `store.py` (VariantStore; idempotent `_ensure_v05_columns` backfill for pre-v0.5 DBs) | LLM via structured output | llm, grading |
|
||||
| `ai_service/voice/` | `base.py` (VoiceProvider protocol D-030), `browser.py` (native SR/TTS fallback descriptor), `mock.py` (deterministic), `openai_audio.py` (v0.5 REQ-5-001, D-040: real server STT/TTS against OpenAI-compatible `/audio/transcriptions` + `/audio/speech` on the shared httpx pool — CUT-1/G-7 seam CLOSED), `factory.py` (provider selection by `AI_VOICE_PROVIDER`, G-11 boot-safe fallback to mock), `defense_store.py` (DefenseStore: transcripts + integrity signals, D-027) | Never imports agents/ or api/ | config |
|
||||
| `ai_service/identity/` | **v0.5 NEW (REQ-5-003/004, D-042/43):** `base.py` (IdentityProvider protocol: submit/poll/verify), `mock.py` (deterministic approve-on-policy mock; verdicts carry a `mock` marker, A-304), `store.py` (5th D-027 store: identity_record table — derived `age_band`, document **refs**, PII never stored raw); age-gate dependencies `require_verified_age`/`require_verified_adult` (gate composition D-043: G-5 allowlist → identity verdict → rate caps; mounted on variants/sandbox-create/defense-start + the G-18 marketplace stub), exposed via `api/identity.py` (`/v1/identity/*`) | Never imports agents/; api/ composes it via DI | config |
|
||||
| `ai_service/agents/examiner.py` | Seventh agent: oral defense examiner; streams over existing SSE, consumes process traces + emits integrity signals | reuses BaseAgent (D-018) | llm, prompts, telemetry |
|
||||
| `ai_service/data/*.db` | SQLite databases (telemetry/grades/variants/defenses) | gitignored | — |
|
||||
| `ai_service/data/*.db` | SQLite databases (telemetry/grades/variants/defenses) — **v0.3.6: default moved to `~/.nextcraft/data/nextcraft.db` (state out of the repo; AI_DB_PATH overrides, ~ expanded)** | outside repo (home) | — |
|
||||
| `scripts/sandbox-agent.py` | Tiny in-namespace capture process shipped into the sandbox; streams telemetry to ingest | standalone | stdlib only |
|
||||
|
||||
**Boundary additions:** `sandbox/`, `telemetry/`, `grading/`, `variants/`, `voice/` are engine modules — they never import `api/` (which composes them via DI) and never import `agents/` (agents call engines through narrow interfaces, not vice versa).
|
||||
|
||||
### apps/cli — Nextcraft Bootstrap CLI (v0.4 NEW)
|
||||
|
||||
| Component | Description | Boundaries | Depends On |
|
||||
|-----------|-------------|------------|------------|
|
||||
| `src/index.ts` | Entry: arg parsing (no deps beyond node stdlib at runtime), command dispatch, `--help`/`--version`, exit-code contract (0 ok / 1 failure / 2 usage) | CLI surface only | commands/ |
|
||||
| `src/commands/` | `doctor.ts` (prereq checks + actionable errors), `bootstrap.ts` (pnpm install + scripts/bootstrap.sh wrapper + env template copy + key validation), `verify.ts` (health: venv imports, ports, env, build readiness), `dev.ts` (thin passthrough to scripts/dev.sh) | Compose checks/ + lib/; spawn scripts — never re-implement them | checks/, lib/ |
|
||||
| `src/checks/` | Pure check functions: `check-command.ts` (binary-on-PATH + version compare), `check-env.ts` (template diff, required/optional key classification) | Pure logic, unit-testable, no fs side effects at import | None |
|
||||
| `src/lib/` | `spawn.ts` (subprocess with timeout + inherited stdio), `log.ts` (✓/✗/warn output formatter) | Shared utilities | None |
|
||||
| `scripts/build-binary.mjs` | esbuild → CJS bundle → Node SEA injection → `dist/nextcraft-linux-x64` + sha256 sidecar | Build-time only | esbuild (dev dep) |
|
||||
| `scripts/install.sh` | The one-liner install script served from repo raw: Gitea latest-release resolve → download + checksum verify → ~/.local/bin; source-bootstrap fallback | Standalone POSIX sh | forge API |
|
||||
| `tests/` | node:test unit tests: command dispatch, check logic, env template diff, install-script shellcheck-style assertions | Fixtures only — never mutate repo state | src/ |
|
||||
|
||||
**Boundary rules:** the CLI never imports from `apps/web`, `packages/*`, or `ai_service` Python modules — it orchestrates them exclusively via subprocess/filesystem. Runtime deps: node stdlib only (no runtime npm deps; esbuild is dev-only). The binary embeds the bundle; `scripts/bootstrap.sh` remains the single source of bootstrap truth (D-034).
|
||||
|
||||
### apps/web — Next.js Application
|
||||
|
||||
| Component | Description | Boundaries | Depends On |
|
||||
|-----------|-------------|------------|------------|
|
||||
| `app/(learner)/` | Learner surface route group: landing, catalog, competency stack, dashboard, byte viewer, sandbox mockup, assessment mockup | Learner-only routes and layouts | packages/ui, packages/mock-data, packages/types |
|
||||
| `app/(learner)/` | Learner surface route group: landing, catalog, competency stack, dashboard, byte viewer, build surface (`/build/[competencyId]` — real in-browser build), defense surface (`/defend/[competencyId]` — live oral defense + grading) | Learner-only routes and layouts | packages/ui, packages/mock-data, packages/types |
|
||||
| `app/(marketplace)/` | Marketplace surface route group: job board, job detail, employer profile, search/filter, pricing | Marketplace-only routes and layouts | packages/ui, packages/mock-data, packages/types |
|
||||
| `app/(employer)/` | Employer dashboard route group: overview, talent search, candidate profile, posting management | Employer-only routes and layouts | packages/ui, packages/mock-data, packages/types |
|
||||
| `app/(admin)/` | Admin surface route group: overview, learner management, competency graph viewer, moderation | Admin-only routes and layouts | packages/ui, packages/mock-data, packages/types |
|
||||
| `app/layout.tsx` | Root layout: theme provider, navigation shell, responsive container | All routes | packages/ui |
|
||||
| `components/` | Surface-specific components (learner/, marketplace/, employer/, admin/) plus shared chrome (navigation-shell, header/footer, role-switcher, theme-provider, breadcrumbs, dark-mode-toggle) | App-level components | packages/ui |
|
||||
| `hooks/` | use-chat-stream.ts — SSE client hook: fetch + ReadableStream, byte buffering + frame reassembly, idempotent AbortController cleanup | Client components only | ai-service SSE |
|
||||
| `lib/` | sse.ts (shared SSE frame parser — CRLF normalization + `: ping` immunity, G-1), breadcrumbs.ts, format.ts | Pure utilities | None |
|
||||
| `components/` | Surface-specific components (learner/, marketplace/, employer/, admin/) plus shared chrome (navigation-shell, header/footer, role-switcher, theme-provider, breadcrumbs, dark-mode-toggle); v0.3 learner: build-surface, sandbox-terminal (read-only exec output), defense-session | App-level components | packages/ui |
|
||||
| `hooks/` | use-chat-stream.ts — SSE client hook: fetch + ReadableStream, byte buffering + frame reassembly, idempotent AbortController cleanup; use-sandbox-session.ts (v0.3) — sandbox lifecycle for the build session: create on task open, destroy on unmount, mid-start failure cleanup, 503/403/429 honest surfaces | Client components only | ai-service SSE / engine API |
|
||||
| `lib/` | sse.ts (shared SSE frame parser — CRLF normalization + `: ping` immunity, v0.2 G-1), breadcrumbs.ts, format.ts, engine-base-url.ts (v0.3.5: runtime API base — env override → browser hostname → localhost), engine-client.ts (v0.3: typed fetch client for /v1/sandboxes, files/exec, variants, grade, defense, traces) | Pure utilities | None |
|
||||
|
||||
### packages/ui — Shared Component Library
|
||||
|
||||
| Component | Description | Boundaries | Depends On |
|
||||
|-----------|-------------|------------|------------|
|
||||
| `tokens/` | Design tokens as TS constants: colors, spacing, radii, shadows, breakpoints (mirrored as Tailwind v4 `@theme` tokens in apps/web globals.css) | Foundation layer — no dependencies | None |
|
||||
| `primitives/` | Button, Input, Card, Badge, Avatar — each with a Storybook story | Atomic UI components | tokens, packages/types |
|
||||
| `primitives/` | Button, Input, Card, Badge, Avatar (v0.1) + TerminalFrame, TelemetryStatus, MicControl, GradeBadge, TranscriptViewer (v0.3 build/defense surfaces) — each with a Storybook story | Atomic UI components | tokens, packages/types |
|
||||
|
||||
Composite/layout/theme components (navigation shell, tables, chat panels, graph viewer, theme provider) live in `apps/web/components/` as app-level components, not in packages/ui.
|
||||
|
||||
@@ -134,11 +188,42 @@ Composite/layout/theme components (navigation shell, tables, chat panels, graph
|
||||
| `marketplace.ts` | Job, Employer, Candidate, JobPosting, TalentMatch, SearchFilter | Marketplace types | None |
|
||||
| `user.ts` | Learner, Admin, EmployerUser, AgeGroup, Role | User types | None |
|
||||
| `ui.ts` | Component props, theme config, breakpoint definitions | UI types | None |
|
||||
| `telemetry.ts` | TelemetryEvent/ExecResult wire shapes for the live build surface (v0.3) | Engine types | None |
|
||||
| `variants.ts` | Variant/TaskTemplate shapes for per-learner task statements (v0.3) | Engine types | None |
|
||||
| `grading.ts` | GradeRecord/RubricScore shapes for live grading display (v0.3) | Engine types | None |
|
||||
| `defense.ts` | DefenseSession/transcript/integrity-signal shapes for the defense surface (v0.3) | Engine types | None |
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
### v0.3 credential flow (current)
|
||||
|
||||
```
|
||||
[learner build surface /build/*] [learner defense surface /defend/*]
|
||||
file CRUD + Run/Test (HTTP) mic MediaRecorder / typed + TTS playback
|
||||
│ │
|
||||
▼ ▼
|
||||
[api/sandboxes files/exec] ──exec──▶ [namespace sandbox] [api/defense start/answer/finish]
|
||||
│ │ capture agent │
|
||||
│ ▼ (WS telemetry) ▼
|
||||
│ [api/telemetry ingest] [DefenseStore (SQLite)]
|
||||
│ │ SQLite │ transcript + integrity signals
|
||||
│ ▼ │
|
||||
│ [TraceStore] ────▶ [GradingEngine: digest (grading/features)
|
||||
│ │ + rubric LLM (D-028)] ──▶ [GradeStore]
|
||||
│ ▼ ▼
|
||||
└──▶ Lab agent (live digest) Assessor (grade output) / Proctor (integrity)
|
||||
Examiner agent (SSE) ◀── defense sessions
|
||||
variants: [api/variants] ◀── [VariantStore (seeded, D-029)] ── per-learner task statements
|
||||
```
|
||||
|
||||
- Lab consumes the live trace digest; Assessor consumes grading-engine output; Proctor consumes telemetry + defense integrity signals (REQ-3-007) — no mock fallback in the learner path (v0.2 corpus scenarios are dormant test-only fixtures).
|
||||
- The learner's path is: variant task → in-sandbox build (telemetry streams to SQLite) → grade My Work (rubric scores from the real trace) → oral defense → verdict.
|
||||
- Flooded/gapped traces are terminal: ingest closes 1008 and marks INCOMPLETE_FLOODED (G-3); the grader returns UNGRADABLE_TRACE_INCOMPLETE (G-4) — no credential from an incomplete trace.
|
||||
|
||||
### v0.2 chat flow (complete, still live)
|
||||
|
||||
```
|
||||
[packages/mock-data + packages/types] [ai_service/corpus]
|
||||
│ (TS, web surfaces) │ (Python, agent inputs)
|
||||
@@ -153,21 +238,27 @@ Composite/layout/theme components (navigation shell, tables, chat panels, graph
|
||||
(https://ollama.com/v1)
|
||||
```
|
||||
|
||||
- Web surfaces remain server-component-first; client components (chat, filters, graph viewer, dark mode toggle) fetch directly from ai-service over SSE (A-002: no Next.js API-route proxy in v0.2).
|
||||
- The LLM provider layer is a dumb pipe — OpenAI-compatible chunks pass through byte-identical; envelope logic (meta/done/error) lives only in the API layer (D-016).
|
||||
- Lab/Assessor/Proctor read mock scenarios from `ai_service/corpus/` — real engines are v0.3+.
|
||||
- Web surfaces remain server-component-first; client components (chat, filters, graph viewer, dark mode toggle) fetch directly from ai-service over SSE (A-002: no Next.js API-route proxy).
|
||||
- The LLM provider layer is a dumb pipe — OpenAI-compatible chunks pass through byte-identically; envelope logic (meta/done/error) lives only in the API layer (D-016).
|
||||
- The v0.2 corpus scenarios (`ai_service/corpus/`) are retained as dormant, test-only fixtures (dormant-header noted); they are no longer inputs to the live learner path.
|
||||
- All automated tests use the deterministic mock provider; the cloud is for manual probes only.
|
||||
|
||||
---
|
||||
|
||||
## Build Order (v0.3)
|
||||
## Build Order (v0.4)
|
||||
|
||||
1. **Bootstrap CLI core** — apps/cli package: doctor checks (node/pnpm/python3/git/unshare), bootstrap wrapper (pnpm install + scripts/bootstrap.sh + .env template + key validation), verify health check, dev passthrough; unit tests
|
||||
2. **Binary build + release pipeline** — esbuild bundle → Node SEA binary (`nextcraft-linux-x64`) + sha256 sidecar; install.sh one-liner (Gitea latest-release resolve + checksum verify + PATH install); release-asset upload wired into the ship flow (ongoing binaries from v0.4 onward)
|
||||
3. **Install docs + fresh-clone E2E** — README quickstart (one-liner → doctor → bootstrap → dev), CLI reference, fresh-clone end-to-end test proving a clean clone reaches a running stack
|
||||
|
||||
## Build Order (v0.3 — complete)
|
||||
|
||||
1. **Sandbox fabric** — SandboxBackend protocol + unshare namespace spawner + lifecycle manager (create/list/snapshot/destroy) + concurrency guard + per-sandbox workdir; isolation + resource-limit probes
|
||||
2. **Live build telemetry** — TelemetryEvent models + SQLite TraceStore + WebSocket ingest endpoint + seq gap detection + in-sandbox capture agent
|
||||
3. **Process-trace grading engine** — deterministic feature/digest computation + rubric scoring via LLM structured output + GradeStore; calibrated against v0.2 mock corpora
|
||||
4. **Variant task generation** — template library + seeded LLM instantiation + VariantStore + difficulty normalization anchors
|
||||
5. **Oral / voice defense** — VoiceProvider protocol + STT/TTS + mock + browser fallback + Examiner agent + transcript/integrity-signal capture
|
||||
6. **Agent re-grounding + learner surface integration** — Lab/Assessor/Proctor consume real telemetry/grades/defense signals; learner sandbox mockup → real in-browser xterm.js build/run; assessment mockup → live defense + live grading
|
||||
6. **Agent re-grounding + learner surface integration** — Lab/Assessor/Proctor consume real telemetry/grades/defense signals; learner sandbox mockup → real in-browser build/run (Run/Test buttons executing in a namespace sandbox, read-only exec-output panel — no interactive shell, CUT-2/G-8); assessment mockup → live defense + live grading
|
||||
|
||||
---
|
||||
|
||||
@@ -184,16 +275,19 @@ The v0.1 build order (monorepo → types → mock data → tokens → primitives
|
||||
|
||||
---
|
||||
|
||||
## Future Architecture (Post-v0.3, for reference)
|
||||
## Future Architecture (Post-v0.4, for reference)
|
||||
|
||||
v0.3 delivers the real credential engines; later milestones fill in the remaining platform:
|
||||
v0.4 delivers distribution (CLI + binary releases); later milestones fill in the remaining platform:
|
||||
|
||||
- **In-memory sessions → PostgreSQL + Drizzle/SQLModel** — SessionStore + v0.3 TraceStore/GradeStore/VariantStore/DefenseStore protocols swap SQLite→Postgres with no API changes
|
||||
- **userns subprocess sandboxes → containerd/runc backend** — D-024 `SandboxBackend` protocol swap; same lifecycle API
|
||||
- **Coding-IDE sandbox → design tool + simulation environments** — REQ-F-021 full scope (v0.4)
|
||||
- **Coding-IDE sandbox → design tool + simulation environments** — REQ-F-021 full scope (v0.5)
|
||||
- **Mock provider → per-agent model routing** — provider factory already selects by config; per-agent `AI_<AGENT>_MODEL` overrides
|
||||
- **No auth → real KYC + sessions** — **deferred per founder directive**; REQ-F-017 identity/age-gating lands post-v0.3 (v0.4+). Age-gating remains the v0.1 visual flow mockup
|
||||
- **No auth → real KYC + sessions** — **deferred per founder directive; moved to v0.5 with D-016**; REQ-F-017 identity/age-gating lands post-v0.4. Age-gating remains the v0.1 visual flow mockup
|
||||
- **Mock voice → real server STT/TTS (openai-audio provider)** — CUT-1/G-7 seam moved to v0.5 per D-016; VoiceProvider protocol is the drop-in point
|
||||
- **linux x64 binary → macOS arm64 + auto-update** — D-036 defers non-linux targets (unverifiable on this box); `nextcraft upgrade` (self-replace from latest release) is the natural v0.5+ follow-up
|
||||
- **Exec-telemetry seq-lease / replay-margin fix** — the P6-lesson one-line ACK gap moves to v0.5 per D-016
|
||||
- **No search → Semantic vector search (pgvector)** — Filter UI replaced with vector similarity search
|
||||
- **No payments → Payment processing** — Pricing page replaced with real subscription/payment flows
|
||||
|
||||
The monorepo structure (apps/web + apps/ai-service + packages/*) accommodates further apps without restructuring.
|
||||
The monorepo structure (apps/web + apps/ai-service + apps/cli + packages/*) accommodates further apps without restructuring.
|
||||
@@ -1,8 +1,9 @@
|
||||
{
|
||||
"phase": 1,
|
||||
"stage": "complete",
|
||||
"milestone": "v0.3",
|
||||
"phase": 2,
|
||||
"stage": "verify",
|
||||
"milestone": "v0.6",
|
||||
"milestone_name": "marketplace-engine",
|
||||
"phase_role": "execution",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-09-11T18:43:22Z"
|
||||
}
|
||||
"updated_at": "2026-09-22T00:30:00Z"
|
||||
}
|
||||
@@ -1,3 +1,51 @@
|
||||
# Nextcraft v0.6 — GRILL.md (Adversarial Review Verdict)
|
||||
|
||||
**Stage:** GRILL, Phase 0 pre-execution · **Verdict:** GO-WITH-BINDINGS · **Confidence:** 0.72 · **Date:** 2026-09-21
|
||||
|
||||
## Summary
|
||||
|
||||
The v0.6 Marketplace Engine plan (46 tasks, 6 phases) is well-researched and tightly scoped, but its central P1 web-migration mechanism — removing `generateStaticParams` from the two marketplace dynamic routes — is **provably build-breaking**: Next 15.5.25 (installed `node_modules/next/dist/build/index.js:1244`) hard-throws error E87 for any dynamic route without `generateStaticParams` under `output: 'export'`, and even a function returning `[]` throws (`length > 0` check). The claimed "proven build/defend pattern" precedent does not exist as described — build/defend routes KEPT their `generateStaticParams`. Four quieter gaps (ingest status semantics, pagination-vs-client-filters, learner-profile sidecar persistence, sitemap origin) each ship a broken happy path or an audit flag. All fixed via binding decisions G-101..G-110 + 8 plan edits applied by the orchestrator before execute.
|
||||
|
||||
## Per-Axis Findings
|
||||
|
||||
| Axis | Verdict | Rationale |
|
||||
|------|---------|-----------|
|
||||
| Feasibility | PASS | Personas/territories match proven patterns (voice factory, identity store, D-020 layer); 46 tasks comparable to shipped v0.5. |
|
||||
| Scope discipline | PASS | Pricing/employer-auth/scheduler/crawler all absent (MH-M6, A-407/A-412, D-050); only borderline add is admin posting-edit, defensible as the re-tag trigger within A-401. |
|
||||
| Vertical-slice integrity | FAIL → fixed | P1 backend+shells coupling is right, but the shell mechanism threw E87 at build and runtime-created postings would get no exported page — fixed by G-101 (placeholder param + FastAPI catch-all shell routes). |
|
||||
| Testability | AT-RISK → fixed | MH-1e pinned a build that could not emit; MH-3d's "matchScore exists nowhere" overclaimed (mock-data retains it for employer/admin mockups) — scoped by G-107/G-110. |
|
||||
| Security posture | PASS | Fail-closed admin, D-043 gate order, upload bounds, PII sentinel tests verified against existing code; hardening added by G-105/G-108. |
|
||||
| Performance/capacity | PASS | Brute-force cosine ≤10⁴ honest; SQLite WAL/busy_timeout verified in the identity store; sitemap fine to 50k URLs. |
|
||||
| Operational readiness | AT-RISK → fixed | D-039 deploy fabric intact (mount order verified), pypdf via bootstrap channel sound; founder workflow breaks at ingest-status (G-102) and sitemap-origin (G-105) — now defined. |
|
||||
| Dependency risk | PASS | pypdf verified as the only new dep; JSON-LD/sitemap hand-rolled; embedding mock stdlib-only. A-404's stale "ollama-cloud exposes /embeddings" parenthetical annotated as superseded by D-051. |
|
||||
| Adoption/UX | AT-RISK → fixed | Happy-path steps 2-4 contradicted until ingest status defined (G-102); verify-CTA/404/409 states well-specified; sitemap URL-resolvability pinned (G-109). |
|
||||
|
||||
## Binding Decisions (applied to PLAN.md before execute)
|
||||
|
||||
- **G-101** — Do NOT remove `generateStaticParams` (E87 build break under `output:'export'`). Keep it returning one placeholder param per route; export one client shell; add FastAPI catch-all `GET /marketplace/jobs/{id}` + `/employers/{id}` (trailing-slash variants) serving the shell — same routes-beat-the-mount mechanism as sitemap. [Vertical-slice integrity]
|
||||
- **G-102** — Ingest-status contract: seed-port postings enter `published` with original `postedAt`; admin-authored start `draft`. [Adoption/UX]
|
||||
- **G-103** — Board fetch = paged fetch-all (≤10⁴ documented); API returns `total_published`; "Showing X of Y" over the fetched set with honest truncation notice. [Performance]
|
||||
- **G-104** — Confirmed learner profiles persist in the marketplace store as a 6th table `learner_profile` (learner_id PK, profile JSON, confirmed_at, model) — NOT a filesystem sidecar. D-049 amended five→six tables. [Scope discipline]
|
||||
- **G-105** — Unset `AI_PUBLIC_BASE_URL`: loud G-11 boot warning; sitemap serves empty static set + warning (never malformed/relative URLs); robots omits the Sitemap directive; boot never crashes. [Security]
|
||||
- **G-106** — Match scores appear only in search context; cards without an active query show no score; per-learner profile-driven card scores post-v0.6; REQ-6-003 "scores on job cards" satisfied conditionally — recorded. [Adoption/UX]
|
||||
- **G-107** — "Mock matchScore absent" scoped to marketplace-path API payloads + marketplace surfaces; grep excludes packages/mock-data and employer/admin route groups. [Testability]
|
||||
- **G-108** — `admin_token` as `SecretStr`; compare via `secrets.compare_digest`. [Security]
|
||||
- **G-109** — Every sitemap job/employer URL must serve 200 on the single-port app (G-101 shells); board ItemList JSON-LD enumerates static structure only — never runtime postings. [Operability]
|
||||
- **G-110** — P1 web tests pin the E87 regression class itself (shells emitted; catch-all serves arbitrary ids; `notFound()` at non-placeholder id via full page load); the honest-501 stub pin at tests/identity/test_identity.py:444 rewritten onto the marketplace router in the same phase. [Testability]
|
||||
|
||||
## Plan Edits Applied
|
||||
|
||||
1. Task 1-3-04 + MH-1e rewritten per G-101 (placeholder shells + catch-all routes; E87 test class; stub-pin rewrite)
|
||||
2. Task 1-2-01 + MH-1f ingest-status contract (G-102)
|
||||
3. Task 1-3-02 paged fetch-all + total_published (G-103)
|
||||
4. Task 4-B-03 learner_profile table replaces the JSON sidecar (G-104; D-049 → six tables)
|
||||
5. Task 5-2-01 + MH-5a/5b unset-origin behavior + URL-resolvability (G-105/G-109)
|
||||
6. Task 3-3-01 + MH-3d matchScore scope + no-query-no-score (G-106/G-107)
|
||||
7. Task 1-2-03 SecretStr + compare_digest (G-108)
|
||||
8. PROJECT.md A-404 stale parenthetical annotated as superseded by D-051 probe
|
||||
|
||||
---
|
||||
|
||||
# Nextcraft v0.3 — GRILL.md (Adversarial Review Verdict)
|
||||
|
||||
**Stage:** GRILL, Phase 0 pre-execution · **Verdict:** GO-WITH-CHANGES · **Confidence:** 0.72
|
||||
@@ -46,3 +94,71 @@ The credential-pipeline architecture (telemetry → trace → grade → defense)
|
||||
## Outcome
|
||||
|
||||
**GO** — all six binding decisions and both scope cuts applied to PLAN.md / REQUIREMENTS.md / ROADMAP.md / PROJECT.md before Phase 1 execution. No axis requires escalation (all resolvable at confidence ≥ 0.85). The milestone no longer claims resource enforcement it cannot deliver, and the no-auth abuse vector is closed at MVP scale.
|
||||
|
||||
---
|
||||
# Nextcraft v0.4 — GRILL.md (Adversarial Review Verdict)
|
||||
|
||||
**Stage:** GRILL, Phase 0 pre-execution · **Verdict:** GO-WITH-CHANGES · **Confidence:** 0.83
|
||||
|
||||
## Summary
|
||||
|
||||
The distribution milestone is small, founder-directed (D-016, confidence 0.99), and additive (zero changes to the running credential pipeline). The plan's central risk: **Node SEA was probe-verified as a flag, not as a working build** — the v0.3 lesson (A-101: probe the mechanism, not the existence) applies. Second gap: a binary whose `--version` lies (stale package.json) would poison the "ongoing binaries" contract. Third: sed-based JSON parsing in install.sh is a fragility + integrity risk. Fourth: "ongoing binaries" has no enforcement mechanism beyond prose. All four closed by binding decisions G-101..G-104 below. No scope cuts required — the milestone is already minimal.
|
||||
|
||||
## Per-Axis Findings
|
||||
|
||||
| Axis | Verdict | Rationale |
|
||||
|------|---------|-----------|
|
||||
| Business case | PASS | Founder directive explicit + recorded (D-016). Evidence of need: live Gitea probe shows latest release v0.2.8 with ZERO assets; bootstrap requires repo archaeology (scripts found only via package.json spelunking). |
|
||||
| Scope | PASS | 5 REQs, 3 execution phases, one focused surface (apps/cli + scripts). Smallest milestone yet. macOS arm64 already cut (D-036, unverifiable here). |
|
||||
| Feasibility | CONCERN (fixed) | SEA flag exists on node v24.15.0, but no end-to-end SEA binary was built during RESEARCH. postject availability assumed (`npx postject` — needs npm registry reachability, unproven). Zipapp fallback requires python3 on target — an honest-degradation ladder, not a silent downgrade. → G-101. |
|
||||
| Honest versioning | CONCERN (fixed) | `--version` from package.json would print a stale hardcoded version inside a per-release binary — breaks upgrade detection + the one-liner's re-run-to-upgrade promise. → G-102. |
|
||||
| Install integrity | CONCERN (fixed) | sed/grep JSON parsing is brittle; a parse failure must never fall through to installing an unverified artifact. Exact asset-name matching + hard-degrade to source instructions. → G-103. |
|
||||
| Sequencing | PASS | P1 CLI (source-runnable) → P2 binary+pipeline → P3 docs+E2E matches dependency order; each phase ships independently. |
|
||||
| Cost/quota | PASS | Zero new paid infra; binaries built on-box; Gitea releases free. Dev-only esbuild dep. |
|
||||
| Risks | CONCERN (fixed) | Top 3: SEA end-to-end (→ G-101 live probe FIRST in P2), npm registry reachability for esbuild (→ proven by P1's pnpm install must-have), Gitea asset-upload token scope (→ live-proven at the v0.3.2 ship itself). |
|
||||
| Adoption/operability | PASS | Consumer = founder + future pilots; one command replaces README archaeology. Rollback trivial (rm ~/.local/bin/nextcraft). No server changes. |
|
||||
|
||||
## Binding Decisions (applied to PLAN.md)
|
||||
|
||||
- **G-101 (BINDING) — SEA live-build probe is the FIRST P2 action.** Task 2-1-01 builds a real binary before anything depends on it; the build script encodes the fallback ladder explicitly (SEA → zipapp with "requires python3" honesty). If SEA fails on this box, zipapp becomes primary with the docs stating the requirement — no silent claim of node-less operation.
|
||||
- **G-102 (BINDING) — Version stamping at build time.** `build-binary` accepts the shipping tag and stamps it into the bundle (`NEXTCRAFT_VERSION` replace); `--version` prints it; install E2E asserts the installed binary reports the tag it was downloaded from. A binary may never report a version it was not built as.
|
||||
- **G-103 (BINDING) — Install-script integrity hard-degrade.** install.sh matches assets by EXACT name (`nextcraft-linux-x64`, `nextcraft-linux-x64.sha256`); any parse/lookup/download failure degrades to source-bootstrap instructions (exit 0) — never installs unverified or name-approximate artifacts. Checksum mismatch = hard stop, exit 1, explicit do-not-run message. dash-safe POSIX sh, no jq.
|
||||
- **G-104 (BINDING) — Ongoing-binaries enforcement.** Every ship from v0.3.2 onward MUST run `scripts/release-assets.sh <tag>` after tag+merge (best-effort, non-blocking, `release_pending` escalation on failure — but attempted + logged every release). The final-phase audit gate includes "milestone release carries both assets" as a check. This makes the founder's "ongoing binaries" directive a pipeline property, not prose.
|
||||
|
||||
## Escalations
|
||||
|
||||
None. All four concerns resolved at confidence ≥ 0.85. No axis requires founder escalation (directive already explicit).
|
||||
|
||||
## Outcome
|
||||
|
||||
**GO** — G-101..G-104 applied to PLAN.md before Phase 1 execution. The milestone claims only what its probes prove, and the ongoing-binaries contract has an enforcement mechanism.
|
||||
|
||||
---
|
||||
|
||||
# v0.5 GRILL (Phase 0, 2026-09-13) — Real Server Voice, KYC/Identity, Sandbox Environments, Seq-Lease
|
||||
|
||||
**Verdict: GO-WITH-CHANGES · Confidence: 0.78** — CUT-3 + G-9..G-18 binding; applied to PLAN.md before P1 execution.
|
||||
|
||||
**Evidence:** every load-bearing plan claim verified against ground-truth code and held (fmt bug defense.py:189, descriptor flow defense.py:143, supervisor frame discard sandbox-agent.py:487-496, no spool cap, dead test_command field, one-line replay margin sandbox-agent.py:416-431, factory rejection). The research layer (D-040..D-046) sold no vapor; the misses were all *interaction* surface.
|
||||
|
||||
## Binding items (all applied to PLAN.md)
|
||||
|
||||
- **CUT-3**: MH-3e identity E2E downgraded from real-app uvicorn harness to `TestClient(create_app(...))` — identity is plain JSON HTTP; the uvicorn harness stays where transport matters (P1 durability, P4 in-ns exec).
|
||||
- **G-9**: `verified_pilot` conftest fixture MUST land with the identity gates (Wave 3-2) — variants/defense are ungated today; without the fixture the existing 413-test suite 403s en masse. MH-M6 reworded: regressions zero *for verified allowlisted pilot learners*.
|
||||
- **G-10**: engine-client must discriminate 403 reasons (`verify_cta` → new `VerifyRequiredError`; allowlist detail → existing `NotAllowlistedError`) — today every 403 collapses into NotAllowlistedError, which would render a verify-CTA as an allowlist lie.
|
||||
- **G-11**: misconfigured `openai-audio` must never crash the boot — lifespan catches the factory error, logs loudly, falls back to mock (descriptor honestly reads mock). Unattended deploy (D-039) survival rule.
|
||||
- **G-12**: client recording bound — auto-stop at 180s default + visible timer + timeslice; 413 renders "re-record" honestly (never silent answer loss).
|
||||
- **G-13**: identity submit caps — one active pending per learner (409 on resubmit) + per-learner rate cap (429); each submission becomes vendor money later.
|
||||
- **G-14**: spool-bound overflow is by-design gap creation — pinned: dropped-counter > 0 → replayed trace gapped → ungradable (never silently-truncated-but-gradable); worst-case spool ≈256MB documented against the 512MB G-2 sweep.
|
||||
- **G-15**: argv contract — templates validate shlex-roundtrip at definition (no quotes/globs); TS splits whitespace-only (no shlex in browsers); exec policy matches EXACT argv tokens (never prefix); `sh -c` passthrough disallowed for design/sim kinds (the digest-gaming vector).
|
||||
- **G-16**: `voice_tts_format` is a `Literal["mp3","wav","opus"]` enum, not a free string — it feeds a Content-Type.
|
||||
- **G-17**: MH-4e "seq-ack intact" was hope-shaped; merged into MH-4d as concrete assertions (stored seqs contiguous 0..N exactly once; spool ≤ ack margin; or cite the P1 suite where a fake agent runs).
|
||||
- **G-18**: the marketplace gated stub proves the gate then returns 501 + `stub: true` + mock markers — never a fabricated "applied" outcome (A-304 honesty house rule).
|
||||
|
||||
## Advisories (recorded)
|
||||
|
||||
a-6 ack on dedup'd appends too (tight margin); a-7 MH-1d 3× runs = one-time ship validation, not per-CI; a-8 manual voice probe must be executable-by-anyone-with-keys (fixture wav + fixed phrase, falsifiable asserts); a-9 413 Content-Length fast path before buffering; a-10 ROADMAP P4 "Depends On: 4" self-typo → fix to 0; a-11 TS `TaskVariant` fields required on the wire; a-12 identity insert-only growth fine at pilot scale; a-13 `recorder.start(1000)`; a-14 do NOT build ack batching unless a flood test shows drainer stall; a-15 provider must always carry `descriptor` (missing it would badge server as mock).
|
||||
|
||||
## Escalations
|
||||
|
||||
None. All four seams remain founder-locked via D-016 / REQ-5-001..007; every finding resolved at confidence ≥ 0.65.
|
||||
|
||||
+188
-156
@@ -2,13 +2,15 @@
|
||||
|
||||
## Persona Roster
|
||||
|
||||
> **v0.3 update (RESEARCH, lead-developer assessment):** backend-engineer territory extended to the new engine modules (telemetry/grading/variants persistence + APIs). New phase-relevant custom personas added: **sandbox-engineer** (Linux-namespace isolation infra) and **voice-engineer** (STT/TTS + Examiner agent audio pipeline). ai-engineer re-scoped to LLM/agents/prompts + grading/variant/voice *model-facing* logic. **security-auditor stays inactive** (KYC deferred per founder directive). frontend-engineer gains real-sandbox (xterm.js), live-telemetry, and live-defense surfaces.
|
||||
> **v0.6 update (RESEARCH, lead-developer assessment):** milestone = Marketplace Engine (D-047). **Activated:** data-engineer (6th D-027 store family — five tables, seed port, TS wire types dual-schema sync, its core constraint territory), ai-engineer full activation (tagging/enrichment/resume D-020 prompts + EmbeddingProvider protocol — LLM-facing work returns), frontend-engineer (client shells, engine-client marketplace extension, generateStaticParams removal ×2, SEO metadata + client JSON-LD), backend-engineer (module wiring, config surface: AI_EMBED_*/AI_ADMIN_TOKEN/AI_PUBLIC_BASE_URL, sitemap/robots routes). **security-auditor re-activated (phase-specific)** for admin-token fail-closed (P1), resume upload bounds + PII-logs (P4), sitemap origin (P5), final review (P6). **Deactivated:** identity-engineer (v0.5 custom — module shipped; only the `require_verified_adult` gate import is reused), voice-engineer, sandbox-engineer (no voice/sandbox/telemetry work), cli-engineer, design-system-engineer (existing primitives cover the marketplace shells).
|
||||
|
||||
> **v0.5 update (RESEARCH, lead-developer assessment):** milestone = Real Server Voice, KYC/Identity, Sandbox Environments, Seq-Lease. **Reactivated:** voice-engineer (real server STT/TTS, its v0.3 territory), sandbox-engineer (design/sim environment kinds + the seq-ack protocol on the fabric/agent it owns), frontend-engineer (defense audio upload, identity enrollment flow, kind-aware build surface). **New custom persona: identity-engineer** (KYC/identity domain — 5th store, provider protocol, age-gate dependencies, PII hygiene). **security-auditor re-activated (phase-specific)** for identity PII + age-gate bypass + audio upload attack surface (phases 1-4 review, final phase). ai-engineer light-touch (no LLM-facing work this milestone). backend-engineer retains settings/factory wiring + turbo/root scripts. cli-engineer inactive (v0.3.6 hotfix shipped; no CLI work planned). design-system-engineer/data-engineer inactive (one TS types extension only — data-engineer light-touch for variants/identity types).
|
||||
|
||||
### lead-developer
|
||||
```yaml
|
||||
active: true
|
||||
phase_specific: false
|
||||
reason: Coordinates task decomposition across web, AI service, engine, and sandbox territories; resolves conflicts between frontend, backend, AI, sandbox, and voice personas
|
||||
reason: Coordinates the v0.6 marketplace territory split (store/ingest/embedding/tagging/search/enrichment/resume across data/ai/backend/frontend personas); owns wave-order coupling decisions (D-059: P1 ships backend WITH client shells)
|
||||
domain: coordination
|
||||
frameworks:
|
||||
- next.js
|
||||
@@ -27,99 +29,34 @@ territory:
|
||||
- "apps/ai-service/pyproject.toml"
|
||||
```
|
||||
|
||||
### frontend-engineer
|
||||
```yaml
|
||||
active: true
|
||||
phase_specific: false
|
||||
reason: Phase 6 real-engine learner-surface integration — xterm.js in-browser terminal, file-tree/run/test controls, live telemetry panels, live voice defense UI, live grading display. Owns all page components, layouts, surface-specific UI.
|
||||
domain: frontend
|
||||
frameworks:
|
||||
- react
|
||||
- next.js
|
||||
- tailwindcss
|
||||
- lucide-react
|
||||
- recharts
|
||||
- react-flow
|
||||
- "@xterm/xterm"
|
||||
- "@xterm/addon-fit"
|
||||
constraints:
|
||||
- component-first
|
||||
- server-components-default
|
||||
- minimal-client-js
|
||||
- sse-client-buffering (buffer bytes, split frames on \n\n, join data: lines)
|
||||
- abortcontroller-cleanup (idempotent abort in effect cleanup)
|
||||
- websocket-lifecycle (typed messages, reconnect backoff, cleanup)
|
||||
- mediarecorder-permission-ux (mic consent, graceful no-mic fallback)
|
||||
- responsive-all-breakpoints
|
||||
- dark-mode-support
|
||||
territory:
|
||||
- "apps/web/**"
|
||||
- "packages/ui/**"
|
||||
- "packages/mock-data/**"
|
||||
- "packages/types/**"
|
||||
```
|
||||
|
||||
### data-engineer
|
||||
```yaml
|
||||
active: true
|
||||
phase_specific: false
|
||||
reason: Owns TS mock data layer schema and typed definitions + TS types for telemetry/trace/grade/variant/defense shapes the web surfaces consume. Does NOT own the Python corpus or engine stores — aligned by convention (D-021).
|
||||
reason: v0.6 core activation — owns the 6th D-027 store family (D-049: posting/employer/embedding/application/employer_enrichment tables, WAL/pragmas/portable columns/validate hooks), the seed port from packages/mock-data jobs.ts+employers.ts (D-050, one-time, matchScore dropped), and the TS marketplace wire types (dual-schema sync with pydantic contracts)
|
||||
domain: data
|
||||
frameworks:
|
||||
- sqlmodel
|
||||
- pydantic
|
||||
- typescript
|
||||
constraints:
|
||||
- schema-first
|
||||
- type-safe
|
||||
- migration-ready
|
||||
- mock-data-only
|
||||
- dual-schema-sync (TS/Python changes made in both places)
|
||||
- idempotent-upserts ((source, external_id) + (posting_id, learner_id) unique constraints)
|
||||
territory:
|
||||
- "apps/ai-service/ai_service/marketplace/base.py"
|
||||
- "apps/ai-service/ai_service/marketplace/store.py"
|
||||
- "apps/ai-service/ai_service/marketplace/ingest.py"
|
||||
- "packages/types/**"
|
||||
- "packages/mock-data/**"
|
||||
```
|
||||
|
||||
### backend-engineer
|
||||
```yaml
|
||||
active: true
|
||||
phase_specific: false
|
||||
reason: Owns apps/ai-service app shell, config, API endpoints (incl. WebSocket telemetry ingest), engine persistence (SQLite stores), scripts, and test harness. Extended for v0.3 engine modules.
|
||||
domain: backend
|
||||
frameworks:
|
||||
- fastapi
|
||||
- uvicorn
|
||||
- pydantic
|
||||
- pydantic-settings
|
||||
- httpx
|
||||
- pytest
|
||||
- sqlmodel
|
||||
- sqlalchemy
|
||||
- websockets
|
||||
- aiofiles
|
||||
constraints:
|
||||
- provider-agnostic-boundaries (engine modules import nothing from agents/ or api/)
|
||||
- streaming-first
|
||||
- sqlite-first-persistence (protocol-wrapped stores, Postgres-ready, D-027)
|
||||
- secrets-via-env-only
|
||||
- mock-provider-in-tests
|
||||
- websocket-contract (typed envelopes, seq gap detection, D-026)
|
||||
territory:
|
||||
- "apps/ai-service/ai_service/main.py"
|
||||
- "apps/ai-service/ai_service/config.py"
|
||||
- "apps/ai-service/ai_service/api/**"
|
||||
- "apps/ai-service/ai_service/telemetry/store.py"
|
||||
- "apps/ai-service/ai_service/telemetry/ingest.py"
|
||||
- "apps/ai-service/ai_service/grading/store.py"
|
||||
- "apps/ai-service/ai_service/variants/store.py"
|
||||
- "apps/ai-service/scripts/**"
|
||||
- "apps/ai-service/package.json"
|
||||
- "apps/ai-service/tests/api/**"
|
||||
- "turbo.json"
|
||||
```
|
||||
|
||||
### ai-engineer
|
||||
```yaml
|
||||
active: true
|
||||
phase_specific: false
|
||||
reason: Owns the LLM provider layer, agent framework, prompt library, structured outputs, and the model-facing logic of v0.3 engines — trace-digest→rubric grading prompts (grading/features.py+engine.py), variant instantiation (variants/templates.py+generator.py), and the Examiner agent. Owns the deterministic-mock corpora.
|
||||
reason: v0.6 full activation — LLM-facing work returns: tagging prompt+schema (D-052), enrichment extraction (D-053), resume parse extraction (D-054) — all through the existing D-020 structured_completion layer; plus the EmbeddingProvider protocol + deterministic feature-hash mock (D-051). Guards the agent/engine boundaries the marketplace surfaces touch
|
||||
domain: ai
|
||||
frameworks:
|
||||
- pydantic
|
||||
@@ -127,116 +64,211 @@ frameworks:
|
||||
- pytest
|
||||
constraints:
|
||||
- provider-agnostic-protocol
|
||||
- prompts-are-code
|
||||
- json-defensive-parsing
|
||||
- never-call-cloud-in-tests
|
||||
- delta-passthrough
|
||||
- llm-sees-digest-not-raw-trace (D-028)
|
||||
- seeded-variant-reproducibility (D-029)
|
||||
- mock-first (structured_completion runs against MockLLMProvider)
|
||||
territory:
|
||||
- "apps/ai-service/ai_service/marketplace/tagging.py"
|
||||
- "apps/ai-service/ai_service/marketplace/enrichment.py"
|
||||
- "apps/ai-service/ai_service/marketplace/resume.py"
|
||||
- "apps/ai-service/ai_service/marketplace/embedding/**"
|
||||
- "apps/ai-service/ai_service/marketplace/search.py"
|
||||
- "apps/ai-service/ai_service/llm/**"
|
||||
- "apps/ai-service/ai_service/agents/**"
|
||||
- "apps/ai-service/ai_service/agents/structured.py"
|
||||
- "apps/ai-service/ai_service/prompts/**"
|
||||
- "apps/ai-service/ai_service/corpus/**"
|
||||
- "apps/ai-service/ai_service/grading/features.py"
|
||||
- "apps/ai-service/ai_service/grading/engine.py"
|
||||
- "apps/ai-service/ai_service/variants/templates.py"
|
||||
- "apps/ai-service/ai_service/variants/generator.py"
|
||||
- "apps/ai-service/tests/llm/**"
|
||||
- "apps/ai-service/tests/agents/**"
|
||||
```
|
||||
|
||||
### sandbox-engineer
|
||||
```yaml
|
||||
active: true
|
||||
phase_specific: true
|
||||
reason: v0.3 custom persona (RESEARCH) — owns the sandbox fabric: SandboxBackend protocol, unshare-based Linux user/mount/pid/net namespace spawner, per-sandbox workdir, resource limits, lifecycle manager, concurrency guard, and the in-sandbox capture agent. Probe-verified isolation on this box (D-024).
|
||||
domain: infra
|
||||
frameworks:
|
||||
- python
|
||||
- linux-namespaces
|
||||
- asyncio
|
||||
- pytest
|
||||
constraints:
|
||||
- isolation-verified (probe must show in-ns uid=0, network isolated, writes to workdir only)
|
||||
- backend-protocol-swap (no containerd assumption; D-024)
|
||||
- resource-limits-enforced (cpu/mem/time quotas observable)
|
||||
- no-daemon (subprocess-only; no docker/containerd service)
|
||||
- capacity-guard (1-5 concurrent; 503 when full, D-032)
|
||||
territory:
|
||||
- "apps/ai-service/ai_service/sandbox/**"
|
||||
- "apps/ai-service/scripts/sandbox-agent.py"
|
||||
- "apps/ai-service/tests/sandbox/**"
|
||||
```
|
||||
|
||||
### voice-engineer
|
||||
```yaml
|
||||
active: true
|
||||
phase_specific: true
|
||||
reason: v0.3 custom persona (RESEARCH) — owns the voice layer: VoiceProvider protocol, STT/TTS against a compatible endpoint, deterministic mock (tests never call a voice API), browser-native fallback, and the media-path wiring consumed by the Examiner agent and assessment UI.
|
||||
domain: ai-media
|
||||
frameworks:
|
||||
- pydantic
|
||||
- httpx
|
||||
- pytest
|
||||
- web-mediarecorder
|
||||
constraints:
|
||||
- provider-agnostic-protocol (D-030)
|
||||
- never-call-voice-api-in-tests
|
||||
- browser-native-fallback (no-key path still functions)
|
||||
- bounded-turn-latency (conversational feel budget)
|
||||
territory:
|
||||
- "apps/ai-service/ai_service/voice/**"
|
||||
- "apps/ai-service/tests/voice/**"
|
||||
```
|
||||
|
||||
### design-system-engineer
|
||||
### backend-engineer
|
||||
```yaml
|
||||
active: true
|
||||
phase_specific: false
|
||||
reason: Owns the shared component library, design tokens, and visual consistency. v0.3 duty: new primitives for the real build/assessment surfaces (terminal frame, telemetry status indicator, mic/record control, grade badge, defense transcript viewer).
|
||||
reason: Owns the v0.6 config surface (AI_EMBED_PROVIDER/BASE_URL/MODEL/API_KEY, AI_ADMIN_TOKEN, AI_PUBLIC_BASE_URL — all env-only), main.py lifespan wiring (marketplace store + embedding provider factory on app.state), the /v1/marketplace/* + admin routers' DI, sitemap/robots routes (D-056), .env.example documentation
|
||||
domain: backend
|
||||
frameworks:
|
||||
- fastapi
|
||||
- pydantic-settings
|
||||
- bash
|
||||
- turborepo
|
||||
- pnpm
|
||||
constraints:
|
||||
- secrets-via-env-only (D-014; keys never in code or commits)
|
||||
- idempotent-scripts
|
||||
- fail-closed-admin (unset AI_ADMIN_TOKEN = admin routes 403 + loud boot warning)
|
||||
territory:
|
||||
- "apps/ai-service/ai_service/config.py"
|
||||
- "apps/ai-service/ai_service/main.py"
|
||||
- "apps/ai-service/ai_service/api/marketplace.py"
|
||||
- "apps/ai-service/ai_service/api/marketplace_admin.py"
|
||||
- "apps/ai-service/ai_service/api/deps.py"
|
||||
- "apps/ai-service/scripts/**"
|
||||
- "apps/ai-service/.env.example"
|
||||
- "apps/ai-service/pyproject.toml"
|
||||
- "package.json"
|
||||
- "turbo.json"
|
||||
```
|
||||
|
||||
### frontend-engineer
|
||||
```yaml
|
||||
active: true
|
||||
phase_specific: false
|
||||
reason: v0.6 marketplace surfaces go live: board/detail/employer-profile client shells (D-055, build/defend pattern), engine-client marketplace functions (listPostings/getPosting/getEmployer/searchPostings/apply), generateStaticParams removal on jobs/[jobId]+employers/[employerId] ONLY, apply-flow wiring on the real route, SEO metadata (export const metadata on static set) + client-side JobPosting/Organization JSON-LD on runtime pages
|
||||
domain: frontend
|
||||
frameworks:
|
||||
- react
|
||||
- next.js
|
||||
- tailwindcss
|
||||
- storybook
|
||||
- lucide-react
|
||||
constraints:
|
||||
- design-token-driven
|
||||
- wcag-aa-contrast
|
||||
- dark-mode-required
|
||||
- consistent-across-surfaces
|
||||
- component-first
|
||||
- server-components-default
|
||||
- honest-state-surfaces (404 on missing posting, verify-CTA on 403, empty boards)
|
||||
- no-build-time-db-reads (D-039 export build stays repo-only)
|
||||
territory:
|
||||
- "apps/web/**"
|
||||
- "packages/ui/**"
|
||||
```
|
||||
|
||||
### security-auditor
|
||||
```yaml
|
||||
active: true
|
||||
phase_specific: true
|
||||
reason: v0.6 re-activated — admin token fail-closed + route coverage (P1), apply gate composition preserved D-043 (P1), embedding provider key redaction (P3), resume upload bounds + raw-bytes-never-logged (P4), enrichment provenance honesty (P4), sitemap origin config (P5), final review (P6)
|
||||
domain: security
|
||||
frameworks:
|
||||
- pytest
|
||||
- httpx
|
||||
constraints:
|
||||
- STRIDE-classified
|
||||
- fail-closed (unset admin token never opens routes)
|
||||
- bounded-uploads (resume 2MB/format guard)
|
||||
- secrets-never-logged (_sanitize key redaction)
|
||||
territory:
|
||||
- "apps/ai-service/ai_service/api/marketplace.py"
|
||||
- "apps/ai-service/ai_service/api/marketplace_admin.py"
|
||||
- "apps/ai-service/ai_service/marketplace/resume.py"
|
||||
- "apps/ai-service/ai_service/marketplace/embedding/openai_compat.py"
|
||||
- "apps/ai-service/ai_service/api/identity.py"
|
||||
```
|
||||
|
||||
### identity-engineer
|
||||
```yaml
|
||||
active: false
|
||||
phase_specific: false
|
||||
reason: Identity/age-gating (KYC) deferred beyond v0.3 per founder directive (A-110) — no real auth or PII backend lands this milestone. Security coverage remains: verifier's STRIDE layer + Phase 7 secrets-hygiene checklist (keys absent from code/logs/commits/errors, localhost-only CORS, no PII in prompts). Sandbox isolation safety is owned by sandbox-engineer's probe-verified constraint.
|
||||
domain: security
|
||||
frameworks: []
|
||||
constraints: []
|
||||
territory: []
|
||||
reason: v0.5 custom persona — module shipped complete (identity/, gates, /v1/identity/*); v0.6 only REUSES the require_verified_adult import on the marketplace apply route (D-058). No identity-domain work.
|
||||
domain: identity
|
||||
frameworks:
|
||||
- fastapi
|
||||
- sqlmodel
|
||||
- pytest
|
||||
constraints:
|
||||
- pii-never-stored-raw
|
||||
- pii-never-logged
|
||||
- mock-verdict-honesty
|
||||
- gate-composition-order
|
||||
territory:
|
||||
- "apps/ai-service/ai_service/identity/**"
|
||||
- "apps/ai-service/tests/identity/**"
|
||||
```
|
||||
|
||||
### voice-engineer
|
||||
```yaml
|
||||
active: false
|
||||
phase_specific: false
|
||||
reason: v0.5 territory — voice shipped complete (openai-audio provider, defense routes, browser SR/TTS client); no v0.6 voice work. The EmbeddingProvider protocol MIRRORS the voice pattern (D-051) but the voice module itself is untouched.
|
||||
domain: ai-media
|
||||
frameworks:
|
||||
- httpx
|
||||
- fastapi
|
||||
- pytest
|
||||
- react
|
||||
constraints:
|
||||
- provider-agnostic-protocol
|
||||
- never-call-cloud-in-tests
|
||||
- key-redaction
|
||||
- bounded-audio-in-memory
|
||||
territory:
|
||||
- "apps/ai-service/ai_service/voice/**"
|
||||
- "apps/ai-service/tests/voice/**"
|
||||
- "apps/web/components/learner/defense-session.tsx"
|
||||
```
|
||||
|
||||
### sandbox-engineer
|
||||
```yaml
|
||||
active: false
|
||||
phase_specific: false
|
||||
reason: v0.3/v0.5 territory — fabric + env kinds + seq-ack shipped complete; no v0.6 sandbox/telemetry work (marketplace data plane never enters the namespace fabric).
|
||||
domain: infra
|
||||
frameworks:
|
||||
- python
|
||||
- linux-namespaces
|
||||
- pytest
|
||||
constraints:
|
||||
- stdlib-only-agent
|
||||
- acks-are-advisory
|
||||
- thread-safe-spool-trim
|
||||
- digest-kind-agnostic
|
||||
territory:
|
||||
- "apps/ai-service/ai_service/sandbox/**"
|
||||
- "apps/ai-service/ai_service/telemetry/**"
|
||||
- "apps/ai-service/scripts/sandbox-agent.py"
|
||||
- "apps/ai-service/ai_service/variants/templates.py"
|
||||
- "apps/ai-service/tests/sandbox/**"
|
||||
- "apps/ai-service/tests/telemetry/**"
|
||||
```
|
||||
|
||||
### cli-engineer
|
||||
```yaml
|
||||
active: false
|
||||
phase_specific: false
|
||||
reason: No CLI work in v0.6 (the binary attaches to every release per the ongoing-binaries requirement; the ship workflow handles it — no apps/cli changes).
|
||||
domain: cli
|
||||
frameworks:
|
||||
- node
|
||||
- typescript
|
||||
- node:test
|
||||
- esbuild
|
||||
- node-sea
|
||||
- posix-sh
|
||||
constraints:
|
||||
- stdlib-only-runtime
|
||||
- thin-wrapper
|
||||
- timeout-every-spawn
|
||||
- fail-loud-exit-codes
|
||||
territory:
|
||||
- "apps/cli/**"
|
||||
- "scripts/install.sh"
|
||||
- "scripts/release-assets.sh"
|
||||
```
|
||||
|
||||
### design-system-engineer
|
||||
```yaml
|
||||
active: false
|
||||
phase_specific: false
|
||||
reason: No design-token or primitive work in v0.6 (existing primitives + JobFilters cover the marketplace shells); roster retained.
|
||||
domain: frontend
|
||||
frameworks:
|
||||
- tailwindcss
|
||||
- storybook
|
||||
constraints:
|
||||
- design-token-driven
|
||||
- wcag-aa-contrast
|
||||
territory:
|
||||
- "packages/ui/**"
|
||||
```
|
||||
|
||||
## Phase-Specific Personas
|
||||
|
||||
| Persona | Phases | Removed After |
|
||||
|---------|--------|---------------|
|
||||
| sandbox-engineer | 1 (primary), 2, 6 | persists while sandbox fabric exists |
|
||||
| voice-engineer | 5 (primary), 6 | persists while voice defense exists |
|
||||
| security-auditor | 1 (admin token + apply gates), 3 (embedding key redaction), 4 (resume bounds + PII), 5 (sitemap origin), 6 (final review) | milestone complete |
|
||||
|
||||
All other active personas span the entire milestone. data-engineer and design-system-engineer are light-touch outside their phases.
|
||||
All other personas span the milestone. Deactivated personas receive no tasks.
|
||||
|
||||
## Territory Conflict Resolution
|
||||
|
||||
| Conflict | Resolution |
|
||||
|----------|------------|
|
||||
| frontend-engineer vs data-engineer (packages/types, packages/mock-data) | data-engineer owns type definitions and mock data schema (incl. new telemetry/grade/variant/defense TS types); frontend-engineer consumes them. |
|
||||
| frontend-engineer vs design-system-engineer (packages/ui) | design-system-engineer owns design tokens and primitive components (terminal frame, mic control, grade badge); frontend-engineer owns composite components and page-level UI. |
|
||||
| ai-engineer vs backend-engineer (grading/variants) | ai-engineer owns the model-facing files (features/engine/templates/generator = LLM logic + prompts); backend-engineer owns the persistence stores + API endpoints. Boundary: stores are pure SQLite; engine logic is pure compute. |
|
||||
| sandbox-engineer vs backend-engineer (sandbox/) | sandbox-engineer owns `ai_service/sandbox/**` + capture agent; backend-engineer owns the API route that composes `sandbox/manager.py` via DI. manager.py has a narrow typed interface consumed by api/. |
|
||||
| voice-engineer vs ai-engineer (Examiner agent) | ai-engineer owns `agents/examiner.py` + its prompt; voice-engineer owns `voice/**` (audio in/out). Examiner calls `voice/` through the `VoiceProvider` protocol — never imports concrete providers. |
|
||||
| ai-engineer vs data-engineer (mock duplication) | ai-engineer owns `ai_service/corpus/` (Python); data-engineer owns `packages/mock-data` (TS). Shared IDs/shapes aligned by convention (D-021). |
|
||||
| lead-developer vs any | lead-developer coordinates only, does not directly modify code files. |
|
||||
| data-engineer vs backend-engineer (store.py / api routes) | data-engineer owns base.py/store.py/ingest.py schema + persistence; backend-engineer owns config.py/main.py/api routers/DI wiring — the marketplace store is consumed via deps.py, never imported directly by routes |
|
||||
| data-engineer vs ai-engineer (embedding storage) | data-engineer owns the embedding TABLE + BLOB codec; ai-engineer owns the EmbeddingProvider protocol + mock hash + search cosine math; the store's put/get_vector is the seam |
|
||||
| ai-engineer vs backend-engineer (factory/config) | backend-engineer owns config surface + factory wiring in main.py; ai-engineer owns provider implementations (openai_compat.py + mock.py) |
|
||||
| frontend-engineer vs data-engineer (wire types) | data-engineer owns packages/types marketplace shapes (dual-schema sync); frontend-engineer consumes them in engine-client + shells; wire contract = pydantic contracts in marketplace/base.py |
|
||||
| security-auditor vs backend-engineer (admin routes) | backend-engineer implements require_admin; security-auditor reviews fail-closed behavior + may patch security defects directly in api/marketplace_admin.py + resume.py (its territory) |
|
||||
| lead-developer vs any | lead-developer coordinates only, does not directly modify code files |
|
||||
+166
-363
@@ -1,438 +1,241 @@
|
||||
# Nextcraft v0.3 — PLAN.md
|
||||
# Nextcraft — PLAN.md (v0.6 Marketplace Engine)
|
||||
|
||||
## Overview
|
||||
|
||||
This plan covers execution phases 1-6 of milestone v0.3 (Credential Engines): the real engines that replace v0.2's mock inputs — a namespace-isolated sandbox fabric, live build telemetry over WebSocket + SQLite, a process-trace grading engine, seeded per-learner variant task generation, and an oral/voice defense with a seventh Examiner agent — plus re-grounding the Lab/Assessor/Proctor agents onto real engine inputs and wiring the v0.1 learner surfaces to the real build/defense/grading paths. Phases are strictly sequential (P1→P6); within each phase, Wave 1 tasks are parallelizable (no cross-file dependencies) and later waves depend on earlier ones.
|
||||
**Milestone:** v0.6 — Marketplace Engine (D-047): the real marketplace backend behind the existing surfaces.
|
||||
**Tag line:** v0.5.x patches (P0 → v0.5.0 … P6 → v0.5.6 = milestone release) on the v0.5.x line
|
||||
**Branch:** milestone/v0.6-marketplace-engine
|
||||
|
||||
**Environment facts (apply throughout):** Python 3.11.2 via `python3 -m venv` (no uv, no system pip); pnpm 12.3.4 via corepack; turborepo; ai-service port **8420**; default model `gemma4:31b` (config via `AI_TUTOR_MODEL`); ollama-cloud base `https://ollama.com/v1` (OpenAI-compatible, Bearer auth); keys live only in gitignored `.ciagent/.env.secrets` (exported by `scripts/dev.sh`) — never in code, commits, or logs; all automated tests use the deterministic mock LLM and mock voice providers and **never call cloud or voice APIs**. New ai-service deps this milestone: `sqlmodel`, `sqlalchemy`, `websockets`, `aiofiles` (all PyPI-verified; added in Task 1-1-02). SQLite data (`apps/ai-service/ai_service/data/`) and sandbox dirs (`apps/ai-service/sandboxes/`) are already gitignored. **KYC/age-gating is deferred per founder directive (A-110)** — no identity work; age-gating stays the v0.1 visual flow mockup. **GRILL scope decisions (this revision):** real server STT/TTS (`OpenAIAudioProvider`) deferred to v0.4 — voice defense is mock+browser-first (CUT-1/G-7); the interactive xterm.js shell relay is deferred to v0.4 — the build panel is run/test-buttons + read-only exec output (CUT-2/G-8), so `@xterm/*` is NOT a v0.3 dependency; sandbox abuse control (per-learner caps + allowlist, G-5) ships even though KYC is deferred; sandbox resource limits are partially enforced (memory/CPU/wall-clock + workdir-size sweep; per-sandbox pids and hard disk quota are accepted gaps, G-1/G-2).
|
||||
**Phase list:** P1 Postings backend + aggregation adapter (REQ-6-001) → P2 AI tagging (REQ-6-002) → P3 Semantic vector search (REQ-6-003) → P4 Enrichment + resume parsing (REQ-6-004/005, two parallel workstreams) → P5 SEO programmatic pages (REQ-6-006) → P6 Final review + ship.
|
||||
|
||||
| Phase | Name | Requirements | Waves | Personas |
|
||||
|-------|------|-------------|-------|----------|
|
||||
| 1 | Sandbox fabric | REQ-3-001, 002 | 3 | sandbox-engineer, backend-engineer, ai-engineer (W1 lint only) |
|
||||
| 2 | Live build telemetry | REQ-3-003 | 4 | sandbox-engineer, backend-engineer, data-engineer |
|
||||
| 3 | Process-trace grading engine | REQ-3-004 | 3 | ai-engineer, backend-engineer |
|
||||
| 4 | Variant task generation | REQ-3-005 | 3 | ai-engineer, backend-engineer, data-engineer |
|
||||
| 5 | Oral / voice defense | REQ-3-006 | 4 | voice-engineer, ai-engineer, backend-engineer |
|
||||
| 6 | Agent re-grounding + learner surface integration | REQ-3-007, 008 | 5 | ai-engineer, frontend-engineer, design-system-engineer, data-engineer, backend-engineer, lead-developer |
|
||||
**Wave order (D-059, binding):** P1 ships the backend **WITH the web client shells** in the same phase — a live store behind mock-reading pages proves nothing; REQ-6-001's success criterion is surfaces reading real data. P2 before P3 because tag facets drive the sidebar filters AND the P3 posting embedding text includes the normalized tags. P4's two workstreams (enrichment / resume) are independent — parallelizable within the phase. P5 SEO depends only on P1 (routes exist to enumerate) but stays last-before-final: the sitemap needs `published` postings and the board metadata should reflect final facets. P6 final review + ship.
|
||||
|
||||
**Research decisions D-048..D-059 are binding design contracts.** This plan operationalizes them; it does not re-litigate them. Assumptions A-401..A-412 are resolved fact.
|
||||
|
||||
**Environment facts (unchanged):** python 3.11.2 (venv at apps/ai-service/.venv, pip-only — A-009), node v24.15.0, pnpm 12.3.4, SQLite shared `Settings().db_path` (`~/.nextcraft/data/nextcraft.db`), tests NEVER call the cloud (mock providers + httpx MockTransport), secrets env-only (D-014), runtime state under `~/.nextcraft/` (D-039).
|
||||
|
||||
**Verification hooks (every phase, release-blocking):** `pnpm ai:test` (488 backend tests), web `tsx --test` (22), CLI `node:test` (61), `pnpm typecheck` (8 projects), `pnpm ai:lint` (ruff check), `pnpm build` (static export emits). **Hermeticity rule (v0.5 P04 lesson): every new test pins `Settings(db_path=tmp_path/...)` — never `~/.nextcraft`, never the cloud.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Sandbox Fabric
|
||||
## Phase 1: Postings Backend + Aggregation Adapter (REQ-6-001)
|
||||
|
||||
**Requirements:** REQ-3-001, REQ-3-002
|
||||
**Goal:** `SandboxBackend` protocol + `unshare`-based namespace spawner (D-024) + lifecycle manager with concurrency guard (D-032) + per-sandbox workdir; isolation and resources probe-verified on this box; `/v1/sandboxes` API live; deps + gitignore landed
|
||||
**Goal:** The 6th D-027 store family, the PostingSource adapter contract with the seeded corpus, the `/v1/marketplace/*` + admin API, the real apply route — and the web surfaces reading all of it at runtime.
|
||||
|
||||
### Wave 1: Foundations (parallel — no shared files)
|
||||
### Wave 1-1: backend core
|
||||
|
||||
#### Task 1-1-01: SandboxBackend protocol + unshare spawner + probe test
|
||||
- **Persona:** sandbox-engineer — **REQ:** REQ-3-001, REQ-3-002
|
||||
- **Files:** `apps/ai-service/ai_service/sandbox/__init__.py`, `apps/ai-service/ai_service/sandbox/backend.py`, `apps/ai-service/ai_service/sandbox/workdir.py`, `apps/ai-service/ai_service/sandbox/unshare_backend.py`, `apps/ai-service/tests/sandbox/__init__.py`, `apps/ai-service/tests/sandbox/test_isolation.py`
|
||||
- **Action:** `backend.py`: `SandboxBackend` protocol + `SandboxSpec` (sandbox_id, learner_id, workdir, resource limits) + `SandboxHandle` (id, pid, workdir, created_at); `spawn(spec)`, `exec(handle, cmd)`, `snapshot(handle) -> Path`, `destroy(handle)`. `workdir.py`: per-sandbox layout under `apps/ai-service/sandboxes/<id>/` (workspace/ writable, snapshot() = recursive copy to `snapshots/<ts>/`) — no symlinks as the snapshot mechanism. `unshare_backend.py`: subprocess spawner — `unshare --user --map-root-user --mount --pid --fork --net` with the per-sandbox dir bind-mounted (`--bind <dir> /work`) and `chdir /work` (D-024); pipes for stdout/stderr; async wrappers. `test_isolation.py` — **re-verify box isolation properties (runs on this box, guarded by probe skip):** (a) `id -u` inside namespace prints `0`; (b) `ip link` inside namespace shows 0 usable interfaces (loopback-only/no carrier) — network isolated; (c) file written to `/work/inside.txt` lands at `sandboxes/<id>/workspace/inside.txt` on the host; (d) attempt to write outside the mount (e.g. host tmp path via bind) does not escape the per-sandbox dir; (e) `/proc` visibility degraded (proc-remount not permitted per A-101 — assert the probe documents this, not that it fails).
|
||||
- **Verify:** `pnpm ai:test` — `tests/sandbox/test_isolation.py` green on this box (probe-gated: skips with an explicit reason if userns unavailable); `lint` clean
|
||||
- **Task 1-1-01** (data-engineer): `ai_service/marketplace/base.py` — pydantic contracts per D-048: `Employer` (id, source, external_id, name + raw ingested fields), `IngestPosting` (A-402 shape: source, external_id, employer + posting fields), `Posting` (full row shape: status `draft|published|closed`, tags JSON column, portable scalar columns, posted_at), `PostingTags` (skills ≤10, seniority, category). Module mirrors D-031 boundary: never imports `agents/` or `api/`.
|
||||
- **Task 1-1-02** (data-engineer): `ai_service/marketplace/store.py` — `MarketplaceStore` protocol + SQLite impl per D-049 **(amended by G-104: six tables)** (`posting`, `employer`, `embedding`, `application`, `employer_enrichment`, `learner_profile`) on the shared `Settings().db_path`; identity/store.py mechanics exactly: WAL/synchronous=NORMAL/busy_timeout/foreign_keys pragmas, `sa_type` portable columns, `_validate()` hooks, surfaced IntegrityError, tz normalization. Unique constraints: `(source, external_id)` on posting + `(posting_id, learner_id)` on application (idempotent upserts A-402/A-408); employer unique per `(source, external_id)`; `embedding` table created now (posting_id PK, `vector` BLOB, `model`, `dim`), its codec/methods land in P3 (the data↔ai seam, per territory table); `learner_profile` (learner_id PK, profile JSON, confirmed_at, model — G-104, consumed in P4).
|
||||
- **Task 1-1-03** (backend-engineer): `config.py` — `admin_token: str = ""` (`AI_ADMIN_TOKEN`), `public_base_url: str = ""` (`AI_PUBLIC_BASE_URL`, placeholder consumed by P5 sitemap); `.env.example` documents both (keys never in code/commits). `api/deps.py` — `get_marketplace_store` (app.state DI; routes NEVER import the store directly — territory rule). `main.py` lifespan — `app.state.marketplace_store`; **loud boot warning when `admin_token` is unset** (G-11; admin routes stay 403-fail-closed regardless).
|
||||
|
||||
#### Task 1-1-02: v0.3 dependencies + gitignore + config additions
|
||||
- **Persona:** backend-engineer — **REQ:** REQ-3-001
|
||||
- **Files:** `apps/ai-service/pyproject.toml` (update), `apps/ai-service/ai_service/config.py` (update), `apps/ai-service/.env.example` (update), `apps/ai-service/scripts/bootstrap.sh` (update if needed), root `package.json` (no change), `turbo.json` (no change)
|
||||
- **Action:** Add pinned deps: `sqlmodel`, `sqlalchemy`, `websockets`, `aiofiles` to pyproject. `config.py` additions (env_prefix `AI_`): `AI_SANDBOX_DIR` (default `apps/ai-service/sandboxes`), `AI_SANDBOX_MAX_CONCURRENT` (default 5, D-032), `AI_SANDBOX_CPU_LIMIT` (default 1 core / cpu.max), `AI_SANDBOX_MEM_LIMIT_MB` (default 512), `AI_SANDBOX_PIDS_LIMIT` (default 256), `AI_SANDBOX_TIMEOUT_S` (default 1800), `AI_DB_PATH` (default `ai_service/data/nextcraft.db`), `AI_VOICE_BASE_URL` / `AI_VOICE_API_KEY` / `AI_VOICE_STT_MODEL` / `AI_VOICE_TTS_MODEL` (all **optional**, default empty — mock-first, D-030; documented in `.env.example` and README). Confirm `.gitignore` already covers `ai_service/data/` + `sandboxes/` (it does — v0.3 block present). Re-run bootstrap idempotently to install new deps.
|
||||
- **Verify:** `pnpm ai:bootstrap` re-installs cleanly (no-op venv, new wheels land); `python -c "import sqlmodel, sqlalchemy, websockets, aiofiles"` succeeds in the venv; settings parse with new keys unset
|
||||
### Wave 1-2: ingest + routes
|
||||
|
||||
#### Task 1-1-03: Resource-limit probe documentation + harness ruff pass
|
||||
- **Persona:** ai-engineer — **REQ:** REQ-3-001
|
||||
- **Files:** `apps/ai-service/README.md` (update: sandbox section + probe transcript), `apps/ai-service/pyproject.toml` (no change), `apps/ai-service/tests/sandbox/test_isolation.py` (no change)
|
||||
- **Action:** Record the A-101 probe transcript in README (verbatim commands + observed output from this box: `unshare --user --map-root-user --mount --pid --fork --net id -u` → `0`; `ip link` → loopback only; write containment). Document the v0.3 resource-limit mechanism choice: cgroup-v2 delegation via per-sandbox scope files is **not available** on this box without sudo → enforcement = subprocess-level (`ulimit`-equivalent via `preexec_fn`: RLIMIT_AS for memory, RLIMIT_CPU for CPU-seconds, RLIMIT_NPROC for pids) + hard wall-clock timeout kill in the manager. This is the locked v0.3 mechanism (D-024 + no-sudo constraint). Run ruff over the new tree; fix all findings.
|
||||
- **Verify:** `pnpm ai:lint` exits 0; README shows the probe transcript and the rlimit mechanism note
|
||||
- **Task 1-2-01** (data-engineer): `ai_service/marketplace/ingest.py` — `PostingSource` protocol `fetch() -> Iterable[IngestPosting]`; `seeded_corpus` adapter over `marketplace/seed_corpus.json` — **a Python fixture file ported ONCE from `packages/mock-data/jobs.ts` (20 jobs) + `employers.ts` (10 employers)**, shared IDs per D-021 convention, mock `matchScore` DROPPED (D-051 computes real ones); `run(source)` upserts idempotently on `(source, external_id)` (employer rows upserted the same way). **Ingest-status contract (G-102): seed-port postings enter as `published` with their original `postedAt`; admin-authored postings start `draft`** — otherwise the board/facets/P1's success criterion are empty after ingest. No scheduler/crawler — the 3M+ corpus lands later as a third adapter with zero store/route changes.
|
||||
- **Task 1-2-02** (backend-engineer): `ai_service/api/marketplace.py` — public router: `GET /v1/marketplace/postings` (list, limit/offset, published only), `GET /v1/marketplace/postings/{id}` (404 honest), `GET /v1/marketplace/employers/{id}` (404 honest; raw fields now, merged+provenance in P4). Mounted in main.py. **The G-18 stub route MOVES here from `api/identity.py` and is deleted there** (identity keeps only its gate exports, D-048).
|
||||
- **Task 1-2-03** (backend-engineer + security-auditor): `ai_service/api/marketplace_admin.py` — `require_admin` FastAPI dependency (X-Admin-Token or Bearer vs `Settings().admin_token` as **`SecretStr`, compared via `secrets.compare_digest` (G-108)**): **unset token → every admin route 403, never open (D-057 fail-closed)**; routes: `POST /admin/postings` (create draft), `POST /admin/postings/{id}/publish` + `/close` (lifecycle `draft→published→closed`, invalid transition → 409), `POST /admin/ingest` (runs the seeded source), `GET /admin/applications` (review list, filterable by posting). Admin routes do NOT pass learner gates (different principal class, D-057).
|
||||
- **Task 1-2-04** (backend-engineer): apply route per D-058 — `POST /v1/marketplace/apply`, body `{posting_id}`: **gate composition binding (D-043, unchanged order): G-5 allowlist → `require_verified_adult` (imported from identity) → rate caps**; then posting must exist and be `published` (404 / 409); insert `application` row (status `submitted`); duplicate `(posting_id, learner_id)` → idempotent 200-with-existing; honest confirmation `{application_id, posting_id, status: "submitted"}`. All `stub`/`mock` markers die.
|
||||
|
||||
### Wave 2: Lifecycle manager (depends on Wave 1)
|
||||
### Wave 1-3: web client shells (D-055)
|
||||
|
||||
#### Task 1-2-01: Sandbox manager + concurrency guard + snapshots
|
||||
- **Persona:** sandbox-engineer — **REQ:** REQ-3-001, REQ-3-002
|
||||
- **Files:** `apps/ai-service/ai_service/sandbox/manager.py`, `apps/ai-service/tests/sandbox/test_manager.py`
|
||||
- **Action:** `SandboxManager`: `create(learner_id) -> SandboxHandle` (guard: active count ≥ `AI_SANDBOX_MAX_CONCURRENT` → raise `PoolFullError` → API maps to **503**, D-032; no queue); `list() -> list[SandboxHandle]`; `get(id)`; `snapshot(id) -> Path` (delegates to workdir); `destroy(id)` (kill process tree, keep or purge workdir per flag); `reap_expired()` background hook for `AI_SANDBOX_TIMEOUT_S` which also performs a **workdir-size sweep**: any sandbox whose `workdir` exceeds `AI_SANDBOX_MAX_WORKDIR_MB` (new config, default 512) is snapshotted-then-destroyed and the event logged as an integrity signal (G-2 — soft disk cap, best-effort, not kernel-enforced); the sweep runs on the same timer as the timeout reaper. Enforce rlimits per spawner (Task 1-1-03: RLIMIT_AS + RLIMIT_CPU + RLIMIT_FSIZE=50MB as a cheap single-file disk guard (a-2); RLIMIT_NPROC noted as shared-per-host-uid, not relied on (G-1)) at exec time. Handle registry persisted **in-memory** (v0.3, single process; not a store — see D-019 precedent) with a clear note that handles are process-local. Startup reaper (a-1): on lifespan boot, scan `AI_SANDBOX_DIR` for workdirs whose recorded pid is dead and reap them, logging a warning. Narrow typed interface only — manager never imports api/ (boundary rule).
|
||||
- **Verify:** `pnpm ai:test` — test_manager covers create/list/destroy/snapshot, 6th create raises PoolFullError (503 path), destroy kills the namespace process (pid gone), snapshot dir exists with workspace contents, timeout reaper removes a stale handle
|
||||
- **Task 1-3-01** (data-engineer): `packages/types/marketplace.ts` — marketplace wire shapes (`PostingSummary`, `PostingDetail`, `EmployerProfile` with `{value, source}` provenance fields ready for P4, `ApplicationConfirmation`, `PostingFacets`) — **dual-schema sync with `marketplace/base.py`**: field-for-field parity, same change in both places.
|
||||
- **Task 1-3-02** (frontend-engineer): `apps/web/lib/engine-client.ts` marketplace section — `listPostings(filters)` (**paged fetch-all: walks limit/offset pages until exhausted, documented ≤10⁴ bound; response carries `total_published`; "Showing X of Y" computed over the fetched set with an honest truncation notice — G-103**), `getPosting(id)`, `getEmployer(id)`, `applyPosting(posting_id)` typed against the new `packages/types` shapes (`searchPostings` lands P3); 403 discrimination (verify-CTA vs allowlist) already works and is reused.
|
||||
- **Task 1-3-03** (frontend-engineer): board — `(marketplace)/marketplace/page.tsx` keeps its server shell; the grid + `JobFilters` become fetch-backed (client shell over `listPostings`); honest empty state (zero postings → message, never fake cards); filter option lists stay UI constants this phase (P2 derives them from real tags).
|
||||
- **Task 1-3-04** (frontend-engineer + backend-engineer): detail + employer shells **(G-101 — do NOT remove `generateStaticParams`; Next 15 hard-throws E87 under `output:'export'` for any dynamic route without it, even returning `[]`)**: keep `generateStaticParams` on `jobs/[jobId]` + `employers/[employerId]` returning ONE placeholder param (`{jobId: '_'}`) so exactly one client-shell page exports per route; add FastAPI catch-all routes `GET /marketplace/jobs/{id}` + `GET /marketplace/employers/{id}` (trailing-slash variants included — `trailingSlash: true`) serving that exported shell file — the same routes-beat-the-mount mechanism P5 relies on for sitemap; runtime client fetch → render, `notFound()` on 404; apply button wired: 403 verify-CTA renders the enrollment prompt, 409 closed renders honestly, success shows the confirmation payload. `packages/mock-data` jobs.ts/employers.ts go dormant for the marketplace path (still imported by employer/admin mockups — NOT deleted). Rewrite `tests/identity/test_identity.py:444`'s honest-501 pin onto the marketplace router in this phase (G-110).
|
||||
|
||||
#### Task 1-2-02: Resource-limit enforcement test
|
||||
- **Persona:** sandbox-engineer — **REQ:** REQ-3-002
|
||||
- **Files:** `apps/ai-service/tests/sandbox/test_resource_limits.py`
|
||||
- **Action:** Concrete enforcement probes (guarded like isolation tests): (a) spawn a process that allocates > `RLIMIT_AS` → assert it dies with MemoryError/killed within a bound; (b) spawn a CPU spinner past `RLIMIT_CPU` → assert SIGXCPU/kill; (c) single huge file > `RLIMIT_FSIZE` → assert write failure (a-2 partial disk guard); (d) wall-clock: spawn `sleep 9999` with a small manager timeout → reaper destroys it; (e) **disk sweep (G-2)**: write > `AI_SANDBOX_MAX_WORKDIR_MB` across many files → assert the manager sweep destroys the sandbox and logs the integrity signal. Assert limits are observable (handle reports its limit set). NOTE (G-1): per-sandbox `RLIMIT_NPROC` is shared at the host uid — the fork-bomb probe is documented as shared-budget behavior, NOT asserted as per-sandbox isolation.
|
||||
- **Verify:** `pnpm ai:test` — test_resource_limits green; limits proven enforced and observable
|
||||
### Wave 1-4: tests
|
||||
|
||||
### Wave 3: API exposure (depends on Wave 2)
|
||||
|
||||
#### Task 1-3-01: Sandboxes API module
|
||||
- **Persona:** backend-engineer — **REQ:** REQ-3-001, REQ-3-002
|
||||
- **Files:** `apps/ai-service/ai_service/api/sandboxes.py`, `apps/ai-service/ai_service/main.py` (update: include router + lifespan manager), `apps/ai-service/ai_service/api/deps.py` (update), `apps/ai-service/tests/api/test_sandboxes.py`
|
||||
- **Action:** DI exposes a singleton `SandboxManager`. Endpoints: `POST /v1/sandboxes {learner_id}` → 201 handle (503 when pool full); `GET /v1/sandboxes` → list; `GET /v1/sandboxes/{id}` → handle; `POST /v1/sandboxes/{id}/snapshot` → snapshot path; `DELETE /v1/sandboxes/{id}` → 204. All behind localhost CORS (A-008). Lifespan creates/destroys the manager; on shutdown destroys any live sandboxes (no orphans). **Abuse control (G-5, NOT KYC):** even in no-auth v0.3 the sandbox API enforces per-`learner_id` rate limiting (`AI_SANDBOX_MAX_PER_LEARNER`, default 1 active → 429) and a global create-rate cap (`AI_SANDBOX_CREATES_PER_MIN`, default 10 → 429); `learner_id` is validated against a server-side allowlist from config (`AI_LEARNER_ALLOWLIST`, default the single mock pilot id → unknown ids rejected 403). This ships in the no-auth milestone so a rogue local process can't exhaust shared NPROC/disk.
|
||||
- **Verify:** `pnpm ai:test` — test_sandboxes green (create→list→snapshot→delete roundtrip via TestClient; 6th create → 503; delete of unknown id → 404; abuse control (G-5): non-allowlisted learner_id → 403; >1 active sandbox for one learner → 429; burst of >10 creates/min → 429); manual probe: `curl -X POST localhost:8420/v1/sandboxes -d '{"learner_id":"l1"}'` returns a handle id
|
||||
|
||||
### Must-Haves (Phase 1)
|
||||
- [ ] Isolation probe test green on this box: in-namespace uid=0, network isolated (0 usable interfaces), host writes confined to the per-sandbox bind dir (A-101 re-verified as an automated test, not just research notes)
|
||||
- [ ] Resource limits enforced + observable: memory (RLIMIT_AS) + CPU (RLIMIT_CPU) rlimits kill violating processes; wall-clock reaper destroys stale sandboxes; disk usage capped by a periodic workdir-size sweep in the manager (soft cap, configurable `AI_SANDBOX_MAX_WORKDIR_MB`, default 512MB — NOT kernel-enforced); per-sandbox NPROC is shared across sandboxes at the host uid — documented, not relied on for isolation (G-1, G-2 — test_resource_limits green)
|
||||
- [ ] No cross-tenant access: sandbox A cannot read sandbox B's workdir (isolation test asserts containment)
|
||||
- [ ] Lifecycle API works end-to-end: create/list/snapshot/destroy via TestClient; pool full → **503** (D-032, no queue)
|
||||
- [ ] Snapshot produces a restorable directory copy under the sandbox's own snapshots/ dir
|
||||
- [ ] `pnpm ai:test` and `pnpm ai:lint` green; new deps (`sqlmodel`, `sqlalchemy`, `websockets`, `aiofiles`) installed via idempotent bootstrap
|
||||
- [ ] Boundary rules hold: `sandbox/` imports nothing from `api/` or `agents/`; only `api/sandboxes.py` composes the manager via DI
|
||||
- [ ] No docker/podman/sudo anywhere in the spawner path (D-024); `SandboxBackend` protocol is the only coupling to the spawner (containerd swap possible later)
|
||||
- **Task 1-4-01** (data-engineer): store contract tests (tmp_path db — hermeticity rule): five tables created on the shared file, unique constraints fire on duplicates, pragmas + `_validate` hooks pinned; ingest idempotency (double-run → identical rows, no dupes); **seed-port fidelity check** (20 postings + 10 employers, IDs match the TS fixtures, `matchScore` absent from every API payload).
|
||||
- **Task 1-4-02** (backend-engineer + security-auditor): route tests — admin token fail-closed matrix (unset → 403 + boot warning in caplog; wrong/absent header → 403; correct → 200); lifecycle 409 on invalid transitions; apply gate composition (allowlist 403 fires FIRST for non-pilot ids, verify-CTA 403 for unverified, 429 at cap, 404/409 states, duplicate → 200 idempotent); 404s on missing postings/employers.
|
||||
- **Task 1-4-03** (frontend-engineer): web tests (`tsx --test` pattern) — engine-client marketplace functions; shells render fetched data; `notFound()` on 404; empty-board state; apply-flow states.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Live Build Telemetry
|
||||
## Phase 2: AI Tagging (REQ-6-002)
|
||||
|
||||
**Requirements:** REQ-3-003
|
||||
**Goal:** First real persistence (D-027: SQLite via SQLModel) with a `TraceStore` protocol; `TelemetryEvent` model with per-(learner,task) monotonic `seq`; WebSocket ingest endpoint (D-026) with gap detection; stdlib-only in-sandbox capture agent streams real sandbox activity into ai-service; trace retrievable by learner+task
|
||||
**Goal:** Posting text → skills/seniority/category via D-020 structured output (mock-first, cloud-free); tags persisted; facets drive the sidebar.
|
||||
|
||||
### Wave 1: Models + stores + TS types (parallel — no shared files)
|
||||
### Wave 2-1: tagging + facets
|
||||
|
||||
#### Task 2-1-01: Telemetry event models
|
||||
- **Persona:** backend-engineer — **REQ:** REQ-3-003
|
||||
- **Files:** `apps/ai-service/ai_service/telemetry/__init__.py`, `apps/ai-service/ai_service/telemetry/models.py`, `apps/ai-service/tests/telemetry/__init__.py`, `apps/ai-service/tests/telemetry/test_models.py`
|
||||
- **Action:** Pydantic/SQLModel `TelemetryEvent`: `learner_id`, `task_id`, `seq` (int, monotonic per (learner,task)), `kind` (`command` | `file_diff` | `run_result` | `test_result` | `activity` | `stdin` | `stdout`), `payload` (JSON), `ts` (datetime, monotonic-envelope), `sandbox_id`. `TraceSpan` derived view (ordered events for one (learner,task)). Validation: seq ≥ 0, kind enum, non-empty ids.
|
||||
- **Verify:** `pnpm ai:test` — test_models green (validation rules enforced, JSON payload roundtrip)
|
||||
- **Task 2-1-01** (ai-engineer): `marketplace/tagging.py` — `PostingTags` schema (skills `list[str]` ≤10, seniority enum, category enum) + versioned prompt constant; `tag_posting(text)` composes the EXISTING `structured_completion(provider, messages, model, schema, schema_hint)` (D-052: no new agent — one-shot extraction like grading's rubric call). Mock-first for free: mock provider default keeps tests cloud-free.
|
||||
- **Task 2-1-02** (data-engineer): persistence hooks — ingest pipeline tags every posting after upsert; re-tag on posting update **overwrites** the tags column (derived data, single source of truth — D-052; unlike enrichment, no append-only).
|
||||
- **Task 2-1-03** (backend-engineer): `PUT /v1/marketplace/admin/postings/{id}` (edit title/description/skills → triggers re-tag) + facets in API responses: `GET /v1/marketplace/postings` carries a `facets` payload (skills/seniority/category value counts) computed from stored tags of published postings.
|
||||
- **Task 2-1-04** (frontend-engineer): `JobFilters` facet lists derive from the API facets payload (static option lists die); counts render honestly.
|
||||
|
||||
#### Task 2-1-02: TraceStore protocol + SQLite implementation
|
||||
- **Persona:** backend-engineer — **REQ:** REQ-3-003
|
||||
- **Files:** `apps/ai-service/ai_service/telemetry/store.py`, `apps/ai-service/tests/telemetry/test_store.py`, `apps/ai-service/ai_service/data/.gitkeep`
|
||||
- **Action:** `TraceStore` protocol (D-027, Postgres-migration-ready): `append(event) -> None` (idempotent on (learner,task,seq) — at-least-once dedup), `get_trace(learner_id, task_id) -> list[TelemetryEvent]` (ordered by seq), `gaps(learner_id, task_id) -> list[int]` (missing seqs), `latest_seq(learner_id, task_id) -> int`, `list_tasks(learner_id) -> list[str]`, `close()`. `SQLiteTraceStore(SQLModel)`: single `telemetry_event` table, composite PK ((learner_id, task_id, seq)), indexes on (learner_id, task_id). Engine creation from `AI_DB_PATH`; `SQLModel.metadata.create_all` at app lifespan. Enable `PRAGMA journal_mode=WAL` + `synchronous=NORMAL` at engine creation (a-3) so concurrent ingest (writer) and trace reads (grader) don't hit `database is locked` under concurrent sandboxes.
|
||||
- **Verify:** `pnpm ai:test` — test_store green (append/ordered-get/dedup-on-retry/gap detection/latest_seq; tmp-path SQLite per test)
|
||||
### Wave 2-2: tests
|
||||
|
||||
#### Task 2-1-03: TS types for telemetry/traces
|
||||
- **Persona:** data-engineer — **REQ:** REQ-3-003
|
||||
- **Files:** `packages/types/telemetry.ts` (new), `packages/types/index.ts` (update)
|
||||
- **Action:** TS `TelemetryEvent`, `TraceSpan`, `TelemetryKind` mirroring the Python model field-for-field (cross-referencing header, same string enums). Consumed by Phase 6 web surfaces; no runtime code.
|
||||
- **Verify:** `pnpm typecheck` passes; TS type keys match Python model keys exactly
|
||||
- **Task 2-2-01** (ai-engineer): tests — tagging via `MockLLMProvider` (cloud-free pin); D-020 schema-violation → single bounded retry pinned; skills ≤10 enforced on an 11-skill mock response; facet derivation test (API facets exactly match stored tags of published-only postings); re-tag overwrite pin (old tags gone after update).
|
||||
|
||||
### Wave 2: Capture agent + ingest (depends on Wave 1)
|
||||
|
||||
#### Task 2-2-01: In-sandbox capture agent (stdlib-only)
|
||||
- **Persona:** sandbox-engineer — **REQ:** REQ-3-003
|
||||
- **Files:** `apps/ai-service/scripts/sandbox-agent.py`, `apps/ai-service/tests/sandbox/test_sandbox_agent.py`
|
||||
- **Action:** Tiny standalone process (D-031, **stdlib only** — no deps shipped into the namespace): wraps a shell inside the sandbox; captures commands, file diffs (mtime/content polling of `workspace/` at 250ms), run/test results, activity; assigns per-(learner,task) `seq`; buffers to a local spool file on disconnect (at-least-once, D-026); reconnects with **exponential backoff** and flushes spool in order; small WebSocket client implemented over raw `socket` (RFC6455 client handshake + frames — stdlib only, no `websockets` in-namespace). Configured via env baked at spawn (`NC_LEARNER_ID`, `NC_TASK_ID`, `NC_INGEST_URL`).
|
||||
- **Verify:** `pnpm ai:test` — unit tests with a loopback fake WS server: ordered seq emission, spool-on-disconnect, reconnect flush preserves order (no loss, dupes deduped server-side), no third-party imports in the file (asserted by AST scan)
|
||||
|
||||
#### Task 2-2-02: WebSocket ingest endpoint
|
||||
- **Persona:** backend-engineer — **REQ:** REQ-3-003
|
||||
- **Files:** `apps/ai-service/ai_service/telemetry/ingest.py`, `apps/ai-service/ai_service/api/sandboxes.py` (update: register WS route), `apps/ai-service/ai_service/main.py` (update), `apps/ai-service/tests/api/test_telemetry_ingest.py`
|
||||
- **Action:** `WS /v1/telemetry/ingest` (D-026): accepts connections carrying learner_id/task_id/sandbox_id; validates + appends events to `TraceStore` (idempotent — server-side dedup on (learner,task,seq)); emits **gap warnings** when seq skips (logged + surfaced in a per-connection status); ping/pong keepalive. **Backpressure / flood control (G-3 — replaces silent drop):** bounded inbound queue; on overflow OR when a per-connection cap `AI_TELEMETRY_MAX_EVENTS_PER_TASK` (default 50000) is exceeded → **reject with a 1008 policy-violation close and mark the (learner,task) trace `INCOMPLETE_FLOODED`** (an integrity signal consumed by Proctor). Silent drop-oldest is FORBIDDEN because it corrupts grading input and is indistinguishable from trace-gaming. `GET /v1/telemetry/traces/{learner_id}/{task_id}` returns the ordered trace; `GET /v1/telemetry/gaps/{learner_id}/{task_id}` returns missing seqs.
|
||||
- **Verify:** `pnpm ai:test` — test_telemetry_ingest green (TestClient websocket: connect → send 3 events → trace retrievable ordered; resend event 2 → deduped; skip seq 5 → gap reported; unknown sandbox tolerated in v0.3 no-auth mode)
|
||||
|
||||
### Wave 3: Sandbox telemetry wiring (depends on Wave 2)
|
||||
|
||||
#### Task 2-3-01: Spawn sandboxes with the capture agent
|
||||
- **Persona:** sandbox-engineer — **REQ:** REQ-3-003
|
||||
- **Files:** `apps/ai-service/ai_service/sandbox/manager.py` (update), `apps/ai-service/ai_service/sandbox/unshare_backend.py` (update), `apps/ai-service/tests/sandbox/test_telemetry_wiring.py`
|
||||
- **Action:** `create()` gains optional `task_id`; when set the spawner copies `scripts/sandbox-agent.py` into the sandbox workdir, injects `NC_*` env, and launches the agent as a child of the namespace process (agent lifecycle tied to sandbox lifecycle; destroy kills the agent). No capture when task_id absent (pure shell sandbox).
|
||||
- **Verify:** `pnpm ai:test` — end-to-end on this box: create sandbox with task_id → run 2 commands via exec → events arrive at the ingest endpoint and land in SQLite in order
|
||||
|
||||
### Wave 4: Reliability probe (depends on Wave 3)
|
||||
|
||||
#### Task 2-4-01: Dropped-connection durability probe
|
||||
- **Persona:** backend-engineer — **REQ:** REQ-3-003
|
||||
- **Files:** `apps/ai-service/tests/telemetry/test_durability.py`
|
||||
- **Action:** Integration probe: run a capture agent against ingest, kill the WS connection mid-stream (simulate network failure), keep generating events, reconnect, assert the SQLite trace contains **every** event exactly once in order (spool + dedup). Document at-least-once semantics + replay path in README.
|
||||
- **Verify:** `pnpm ai:test` — test_durability green; README documents semantics
|
||||
|
||||
### Must-Haves (Phase 2)
|
||||
- [ ] Real telemetry from a live sandbox arrives at ai-service: shell commands, file diffs, run/test results appear as ordered events in SQLite (end-to-end, no mocks)
|
||||
- [ ] Per-(learner,task) monotonic `seq`; gap detection reports missing seqs; replay yields the complete ordered trace
|
||||
- [ ] At-least-once proven: transient disconnect + reconnect loses no events; duplicates deduped server-side (durability probe green)
|
||||
- [ ] Capture agent is stdlib-only (AST-verified) and its lifecycle is tied to the sandbox (destroy kills it)
|
||||
- [ ] Trace retrievable by learner+task via `GET /v1/telemetry/traces/...`; unknown trace → 404
|
||||
- [ ] `TraceStore` protocol respected: no api/ code touches SQLite directly; `telemetry/` never imports `agents/` (D-027)
|
||||
- [ ] Flood control (G-3): burst past `AI_TELEMETRY_MAX_EVENTS_PER_TASK` → connection closed 1008 + trace marked `INCOMPLETE_FLOODED`; no silent event drop on overflow
|
||||
- [ ] `pnpm ai:test` green; `packages/types` telemetry TS types compile (`pnpm typecheck`)
|
||||
**Verification hooks P2:** all suites green (488+ / web 22+ / CLI 61 / typecheck / ruff / export build). Tagging tests stay mock-provider-only.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Process-Trace Grading Engine
|
||||
## Phase 3: Semantic Vector Search (REQ-6-003)
|
||||
|
||||
**Requirements:** REQ-3-004
|
||||
**Goal:** Deterministic feature computation over traces (D-028) → compact digest → LLM rubric scoring via existing D-020 JSON defense → validated structured scores stored in `GradeStore`; LLM never sees the raw trace; engine calibrated against v0.2 mock corpora so process quality separates paste-and-run from iterative debugging
|
||||
**Goal:** EmbeddingProvider protocol (mock-first, deterministic), SQLite-local vectors, hybrid facet+cosine ranking, computed match scores replacing the mock `matchScore`.
|
||||
|
||||
### Wave 1: Features + grades store (parallel — no shared files)
|
||||
### Wave 3-1: provider + storage
|
||||
|
||||
#### Task 3-1-01: Deterministic trace digest (features)
|
||||
- **Persona:** ai-engineer — **REQ:** REQ-3-004
|
||||
- **Files:** `apps/ai-service/ai_service/grading/__init__.py`, `apps/ai-service/ai_service/grading/features.py`, `apps/ai-service/tests/grading/__init__.py`, `apps/ai-service/tests/grading/test_features.py`
|
||||
- **Action:** Pure compute module (D-028): `compute_digest(trace: list[TelemetryEvent]) -> TraceDigest`. Deterministic features: test pass/fail counts + final status; edit count; error/fix cycle count + mean fix latency; idle gaps (>Ns, count + total); command category histogram (build/test/file/nav/debug/other); session duration; first-test-pass offset. `TraceDigest` pydantic model — compact (bounded size, no raw commands), LLM-safe.
|
||||
- **Verify:** `pnpm ai:test` — test_features green over synthetic traces: paste-and-run trace (0 error/fix cycles, single test pass at end) vs iterative trace (many cycles) produce observably different digests
|
||||
- **Task 3-1-01** (ai-engineer): `marketplace/embedding/base.py` — `EmbeddingProvider` protocol `embed(texts: list[str]) -> list[EmbeddingVector]` (frozen pydantic: `values: list[float]`, `model`, `dim` — D-051, D-030 mirror); `embedding/mock.py` — deterministic 256-dim feature-hash of normalized tokens (pure stdlib; similarity deterministic).
|
||||
- **Task 3-1-02** (ai-engineer): `embedding/openai_compat.py` — `POST {AI_EMBED_BASE_URL}/embeddings` (`{"model", "input": [texts]}`) on the shared httpx pool (D-017); `_sanitize` key redaction mirroring `openai_audio.py`. **Probe-corrected (D-051): ollama-cloud does NOT expose /embeddings — `AI_EMBED_BASE_URL` must point at another OpenAI-compatible endpoint (e.g. local ollama `:11434/v1`); default stays mock.**
|
||||
- **Task 3-1-03** (backend-engineer): `config.py` `embed_provider`/`embed_base_url`/`embed_api_key` (SecretStr)/`embed_model` + `embedding/factory.py` wiring in main.py lifespan: `AI_EMBED_PROVIDER=mock|openai-embed`, **G-11 boot-safe — `openai-embed` selected but unconfigured → loud log naming the fix, fallback to mock, boot never crashes**; `app.state.embedding_provider` + `deps.get_embedding_provider`.
|
||||
- **Task 3-1-04** (data-engineer): store `put_vector`/`get_vector` — struct-packed little-endian float32 BLOB codec (`<{dim}f`), `model`+`dim` persisted per row (A-403/A-404: lazy re-embed on provider change works); roundtrip test (the data↔ai seam per territory table).
|
||||
|
||||
#### Task 3-1-02: GradeStore protocol + SQLite implementation
|
||||
- **Persona:** backend-engineer — **REQ:** REQ-3-004
|
||||
- **Files:** `apps/ai-service/ai_service/grading/store.py`, `apps/ai-service/tests/grading/test_store.py`
|
||||
- **Action:** `GradeStore` protocol (D-027): `save(grade)`, `get(learner_id, task_id)`, `list_for_learner(learner_id)`, `close()`. SQLModel `GradeRecord`: learner_id, task_id, variant_seed (null until P4), digest (JSON), scores (JSON), verdict, model, created_at. PK (learner_id, task_id). Postgres-migration-ready.
|
||||
- **Verify:** `pnpm ai:test` — test_store green (save/get/list roundtrip, overwrite-on-regrade documented)
|
||||
### Wave 3-2: pipeline + ranking
|
||||
|
||||
### Wave 2: Grading engine + calibration (depends on Wave 1)
|
||||
- **Task 3-2-01** (ai-engineer): embed-on-publish — the publish transition embeds posting text + tags and stores the vector; **lazy re-embed**: a row whose `model`/`dim` ≠ the provider's is re-embedded on next access; candidate-side competency-profile text is embedded **on demand, never stored**.
|
||||
- **Task 3-2-02** (ai-engineer): `marketplace/search.py` — pure-Python cosine (sum of products / norm product; ≤10⁴ postings, no numpy — A-403); **hybrid per A-405/D-059: facet hard-filters (SQL on portable columns) first, then cosine ranking over the filtered set; no facets → pure semantic ranking**; `match_score` = cosine(candidate, posting); skill-breakdown = posting-tag overlap with the query + per-skill vector contribution (deterministic feature-hash dot-product shares — testable).
|
||||
- **Task 3-2-03** (backend-engineer): `GET /v1/marketplace/search` (`q` + facet params → ranked results with `match_score` + `skill_breakdown`); DI-composed, no store import in routes.
|
||||
|
||||
#### Task 3-2-01: Rubric scoring engine
|
||||
- **Persona:** ai-engineer — **REQ:** REQ-3-004
|
||||
- **Files:** `apps/ai-service/ai_service/grading/engine.py`, `apps/ai-service/ai_service/prompts/grading.py` (new), `apps/ai-service/tests/grading/test_engine.py`
|
||||
- **Action:** `GradingEngine.grade(learner_id, task_id) -> GradeRecord`: **trace-completeness gate (G-4)** — first call `TraceStore.gaps()` + check the trace `INCOMPLETE_FLOODED` flag; if gaps are non-empty OR the trace is flagged incomplete → return `verdict=UNGRADABLE_TRACE_INCOMPLETE` (a first-class verdict, not an exception) surfacing the gap list; a credential is NEVER issued from a gapped/incomplete trace. Otherwise: load trace via `TraceStore` → `compute_digest` → render rubric prompt (`prompts/grading.py`: criteria + level anchors for process quality, correctness, debugging discipline, test usage; a-4: treat high edit/command churn with no test-progress as a process-quality negative) → LLM structured output through the **existing D-020 4-layer defense** (`agents/structured.py` reused — engine composes it, never duplicates it) → validate `RubricScore` model (per-criterion 0-4 + strengths + gaps + verdict) → persist via `GradeStore`. Grading depends on `llm/` + `telemetry/` + `prompts/` only (boundary). Mock provider scripts deterministic rubric JSON for tests, including the INCOMPLETE path.
|
||||
- **Verify:** `pnpm ai:test` — test_engine green (mock provider: digest-only prompt asserted — **raw trace string absent from prompt**; validated scores returned; malformed JSON exercises D-020 retry; unknown trace → error)
|
||||
### Wave 3-3: web + tests
|
||||
|
||||
#### Task 3-2-02: Calibration against v0.2 mock corpora
|
||||
- **Persona:** ai-engineer — **REQ:** REQ-3-004
|
||||
- **Files:** `apps/ai-service/ai_service/corpus/trace_fixtures.py` (new), `apps/ai-service/tests/grading/test_calibration.py`
|
||||
- **Action:** Synthetic trace fixtures aligned with v0.2 `corpus/telemetry.py` + `corpus/artifacts.py` scenario IDs (strong/lazy/struggling builder archetypes). Assert grading separates them: strong archetype scores ≥ lazy archetype on process-quality criterion (mock provider maps digest shape → scripted scores; test asserts the ordering contract + that fixture IDs align with existing corpus IDs, D-021).
|
||||
- **Verify:** `pnpm ai:test` — test_calibration enforces the ordering contract
|
||||
- **Task 3-3-01** (frontend-engineer): `engine-client.searchPostings(q, filters)`; board search bar wired to the search API; **match scores appear only in search context (G-106): cards without an active query show no score (honest absence, never a fabricated number); per-learner profile-driven card scores are explicitly post-v0.6 — REQ-6-003's "scores on job cards" satisfied conditionally in search context, recorded**; skill-breakdown panel on detail consumes the computed payload.
|
||||
- **Task 3-3-02** (ai-engineer + security-auditor): tests — deterministic similarity pins (same text → identical vector; related texts > unrelated texts cosine); codec roundtrip; factory fallback + boot-safe; `_sanitize` key-redaction pin (no key in any error); cosine math corner cases (zero vector); hybrid ordering (facet filters before ranking; pure-rank with no facets); embed-on-publish + lazy re-embed on model change.
|
||||
|
||||
### Wave 3: Grading endpoint (depends on Wave 2)
|
||||
|
||||
#### Task 3-3-01: Assessment grade endpoint (real traces)
|
||||
- **Persona:** backend-engineer — **REQ:** REQ-3-004
|
||||
- **Files:** `apps/ai-service/ai_service/api/assessment.py` (update), `apps/ai-service/tests/api/test_grading.py`
|
||||
- **Action:** `POST /v1/assessment/grade {learner_id, task_id}` → `GradingEngine` → validated `RubricScore` JSON; `GET /v1/assessment/grade/{learner_id}/{task_id}` → stored grade; unknown trace → 404. Composed via DI (api/ owns wiring; engine knows nothing of FastAPI).
|
||||
- **Verify:** `pnpm ai:test` — test_grading green (grade roundtrip via TestClient with mock provider; 404 on unknown; GET after POST returns same scores)
|
||||
|
||||
### Must-Haves (Phase 3)
|
||||
- [ ] Engine emits structured rubric-aligned scores from a **real process trace** (not pre-baked input) — TestClient roundtrip green
|
||||
- [ ] Deterministic features computed in code (test pass/fail, edit count, error/fix cycles, idle gaps, command categories); LLM receives the **digest only** — test asserts the raw trace never reaches the prompt (D-028)
|
||||
- [ ] Scores distinguish process quality: iterative-debugging archetype out-scores paste-and-run on the process criterion (calibration contract test)
|
||||
- [ ] Grades persisted + retrievable by learner+task via GradeStore (SQLite, protocol-wrapped, D-027)
|
||||
- [ ] Boundary rules hold: `grading/` imports no `api/`/`agents/` internals except the shared D-020 structured defense; `pnpm ai:test` + `pnpm ai:lint` green
|
||||
- [ ] Incomplete-trace gate (G-4): gapped or `INCOMPLETE_FLOODED` trace → `verdict=UNGRADABLE_TRACE_INCOMPLETE` with the gap list; no credential issued from an incomplete trace (test green)
|
||||
**Verification hooks P3:** all suites green; embedding tests never leave the process (mock provider + MockTransport for openai_compat byte-contract tests).
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Variant Task Generation
|
||||
## Phase 4: Enrichment + Resume Parsing (REQ-6-004, REQ-6-005)
|
||||
|
||||
**Requirements:** REQ-3-005
|
||||
**Goal:** Template library with typed parameter slots (D-029) + seeded LLM instantiation + per-learner variant registry (SQLite) with difficulty-normalization anchors; two learners on the same competency get provably distinct, reproducible, auditable tasks
|
||||
**Goal:** Two independent workstreams, parallelizable (D-059). (A) employer enrichment with append-only provenance; (B) bounded resume upload → draft extraction → learner confirm gate.
|
||||
|
||||
### Wave 1: Templates + store (parallel — no shared files)
|
||||
### Workstream A: enrichment (REQ-6-004)
|
||||
|
||||
#### Task 4-1-01: Task template library
|
||||
- **Persona:** ai-engineer — **REQ:** REQ-3-005
|
||||
- **Files:** `apps/ai-service/ai_service/variants/__init__.py`, `apps/ai-service/ai_service/variants/templates.py`, `apps/ai-service/tests/variants/__init__.py`, `apps/ai-service/tests/variants/test_templates.py`
|
||||
- **Action:** ≥3 initial task templates bound to existing competency IDs (D-021 alignment). `TaskTemplate`: id, competency_id, statement skeleton with `{slot}` placeholders, `ParameterSlot[]` (name, type: enum/int-range/string-set, allowed values), `rubric anchors` (difficulty normalization: expected feature envelope — e.g. expected edit-count band — used by grading context), starter-file scaffolds served to the sandbox. Seeded slot sampler is pure code (`random.Random(seed)`), fully reproducible.
|
||||
- **Verify:** `pnpm ai:test` — test_templates green (slot validation: bad value rejected; seeded sampling reproducible across runs; all templates bind to real competency IDs)
|
||||
- **Task 4-A-01** (ai-engineer): `marketplace/enrichment.py` — `EmployerEnrichment` schema (description, social_links, logo_ref) + prompt; one-shot `enrich_employer()` via structured_completion (D-053, same D-020 layering as tagging; mock-first).
|
||||
- **Task 4-A-02** (data-engineer): store — `employer_enrichment` rows are **append-only** `(employer_id, field, value, source: llm|manual, model, created_at)`; `merge_employer(employer_id)` read precedence **manual > llm > ingested-raw** (A-409); manual override = new `manual` row, NEVER a mutation of `llm` rows.
|
||||
- **Task 4-A-03** (backend-engineer): **enrich-on-first-publish** — a publish whose employer has no prior published posting fires the one-shot extraction (re-publishing the same employer does not re-run); employer GET returns merged fields with per-field `{value, source}` provenance markers (A-304 honesty — LLM copy visibly marked); `POST /v1/marketplace/admin/enrichment` (manual-override upsert, D-057-guarded).
|
||||
- **Task 4-A-04** (frontend-engineer): employer profile shell consumes merged+marked fields; provenance badges visible per field.
|
||||
- **Task 4-A-05** (security-auditor + ai-engineer): tests — precedence pin (manual beats llm beats raw, each combination); append-only pin (no UPDATE/DELETE path exists on enrichment rows); marker honesty (mock-llm copy carries the llm marker end-to-end); trigger-once pin.
|
||||
|
||||
#### Task 4-1-02: VariantStore protocol + SQLite implementation
|
||||
- **Persona:** backend-engineer — **REQ:** REQ-3-005
|
||||
- **Files:** `apps/ai-service/ai_service/variants/store.py`, `apps/ai-service/tests/variants/test_store.py`
|
||||
- **Action:** `VariantStore` protocol (D-027): `save(variant)`, `get(learner_id, template_id_or_task_id)`, `list_for_learner(learner_id)`, `list_by_template(template_id)`, `close()`. SQLModel `VariantRecord`: learner_id, task_id (the grading/telemetry task key), template_id, seed, params (JSON), statement (rendered), created_at. Unique (learner_id, template_id).
|
||||
- **Verify:** `pnpm ai:test` — test_store green (roundtrip, unique constraint, audit listing)
|
||||
### Workstream B: resume parsing (REQ-6-005)
|
||||
|
||||
### Wave 2: Generator (depends on Wave 1)
|
||||
- **Task 4-B-01** (backend-engineer): pypdf pin — `pypdf>=6.19,<7` in `apps/ai-service/pyproject.toml` + idempotent install via the existing bootstrap venv channel (probe-verified 6.19.0, A-009; the ONLY new dep).
|
||||
- **Task 4-B-02** (ai-engineer): `marketplace/resume.py` — `ParsedProfile` schema mapped to the competency-profile shape (competencies, skills, experience bands) + prompt; parse via D-020 structured extraction; result carries `status: "draft"`.
|
||||
- **Task 4-B-03** (backend-engineer): `POST /v1/marketplace/resume` (multipart; **PDF/TXT/MD only, ≤2 MB, text-layer required** — empty pypdf extract → honest 422 "text layer required", NO OCR in v0.6; extracted text capped ~100K chars before the LLM call; raw bytes never stored, never logged — A-305, `_scrub_pii_validation` precedent) and `POST /v1/marketplace/resume/confirm` (learner-gated: G-5 → verified → caps; **persists the confirmed profile JSON to a 6th store table `learner_profile` (learner_id PK, profile JSON, confirmed_at, model — G-104; NOT a filesystem sidecar — the D-027 store family is the only persistence mechanism)**; the draft NEVER persists without confirm).
|
||||
- **Task 4-B-04** (frontend-engineer): learner dashboard profile section — upload → draft fields review → confirm; bounds errors render honestly (413 oversize / 422 format+text-layer); nothing saves before the confirm click.
|
||||
- **Task 4-B-05** (security-auditor + ai-engineer): tests — small **generated** PDF + TXT fixtures parse (cloud-free); bounds matrix (format rejection, >2 MB, empty-extract 422, char cap); draft-then-confirm flow; PII sentinel caplog test (resume bytes/sentinel strings in NO log record, NO stored raw form).
|
||||
|
||||
#### Task 4-2-01: Seeded LLM variant generator
|
||||
- **Persona:** ai-engineer — **REQ:** REQ-3-005
|
||||
- **Files:** `apps/ai-service/ai_service/variants/generator.py`, `apps/ai-service/ai_service/prompts/variant.py` (new), `apps/ai-service/tests/variants/test_generator.py`
|
||||
- **Action:** `VariantGenerator.generate(learner_id, template_id) -> VariantRecord`: derive seed (`sha256(template_id|learner_id|milestone)` — reproducible, D-029); sample typed slots in code; render a fill prompt (statement skeleton + concrete slot values) → LLM via D-020 structured defense → unique task statement + starter files → validate → persist (seed + params + statement) via `VariantStore`. Cache: existing (learner,template) returns the stored variant (no duplicate work). Mock provider scripts deterministic statements per seed for tests.
|
||||
- **Verify:** `pnpm ai:test` — test_generator green: two different learner_ids → distinct statements for the same template; same learner twice → identical stored variant (reproducible); params JSON contains only schema-valid slot values; **fairness envelope (a-5):** two variants of one template compute digests within the template's expected feature envelope (comparable slot complexity/difficulty features) — "same bar" is testable, not asserted
|
||||
|
||||
### Wave 3: Variant endpoint + TS types (depends on Wave 2)
|
||||
|
||||
#### Task 4-3-01: Variant task endpoint
|
||||
- **Persona:** backend-engineer — **REQ:** REQ-3-005
|
||||
- **Files:** `apps/ai-service/ai_service/api/variants.py` (new), `apps/ai-service/ai_service/main.py` (update), `apps/ai-service/tests/api/test_variants.py`
|
||||
- **Action:** `POST /v1/variants {learner_id, template_id or competency_id}` → generated (or cached) variant: statement, starter files, task_id, seed; `GET /v1/variants/{task_id}` → stored variant; `GET /v1/variants?learner_id=` → learner's variants. DI wiring in api/ only.
|
||||
- **Verify:** `pnpm ai:test` — test_variants green (generate→get roundtrip; cache hit on regenerate; distinct learners → distinct statements asserted at the API layer)
|
||||
|
||||
#### Task 4-3-02: TS types for variants (+ grades)
|
||||
- **Persona:** data-engineer — **REQ:** REQ-3-005
|
||||
- **Files:** `packages/types/variants.ts` (new), `packages/types/grading.ts` (new), `packages/types/index.ts` (update)
|
||||
- **Action:** TS `TaskVariant`, `VariantParams`, `RubricScore`, `GradeRecord` matching Python models (cross-referencing headers; same field names). Consumed by Phase 6 surfaces.
|
||||
- **Verify:** `pnpm typecheck` passes
|
||||
|
||||
### Must-Haves (Phase 4)
|
||||
- [ ] Two learners requesting the same competency receive **provably distinct** task variants (API-level test)
|
||||
- [ ] Seed derivation reproducible: same (template, learner) → same variant, served from cache without a second LLM call (D-029)
|
||||
- [ ] Variant seed + typed params persisted + auditable (VariantStore listing; proctoring cross-check path exists)
|
||||
- [ ] Difficulty normalization anchors present per template and shipped to the grader prompt context
|
||||
- [ ] Starter-file scaffolds defined per template (P6 wires them into the sandbox workdir)
|
||||
- [ ] `pnpm ai:test` + `pnpm typecheck` green; `variants/` imports no api/ (boundary)
|
||||
**Verification hooks P4:** all suites green; resume + enrichment tests stay mock-provider; resume fixtures generated in tmp (no binaries committed beyond minimal fixtures).
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Oral / Voice Defense
|
||||
## Phase 5: SEO Programmatic Pages (REQ-6-006)
|
||||
|
||||
**Requirements:** REQ-3-006
|
||||
**Goal:** `VoiceProvider` protocol with mock + browser fallback (D-030) + seventh `Examiner` agent streaming over existing SSE + `DefenseStore` persisting transcript + integrity signals. **Real server STT/TTS (`OpenAIAudioProvider`) is DEFERRED to v0.4 (with KYC, when there's a real key + real users)** — voice is mock-first (D-030) and the `/audio/*` real path could never be exercised in CI, so v0.3 proves the full defense *dialogue* + integrity-signal pipeline over mock + browser-native fallback only; the protocol seam keeps the real provider a drop-in later.
|
||||
**Goal:** Metadata + JSON-LD + sitemap/robots on the static-export fabric, D-039 unchanged.
|
||||
|
||||
### Wave 1: Voice provider layer (parallel — no shared files)
|
||||
### Wave 5-1: web metadata
|
||||
|
||||
#### Task 5-1-01: VoiceProvider protocol + mock provider + browser fallback
|
||||
- **Persona:** voice-engineer — **REQ:** REQ-3-006
|
||||
- **Files:** `apps/ai-service/ai_service/voice/__init__.py`, `apps/ai-service/ai_service/voice/base.py`, `apps/ai-service/ai_service/voice/mock.py`, `apps/ai-service/ai_service/voice/browser.py`, `apps/ai-service/ai_service/voice/factory.py`, `apps/ai-service/tests/voice/__init__.py`, `apps/ai-service/tests/voice/test_mock.py`, `apps/ai-service/tests/voice/test_factory.py`
|
||||
- **Action:** `VoiceProvider` protocol mirroring `LLMProvider` (D-030): `transcribe(audio: bytes, fmt) -> TranscriptSegment` + `synthesize(text, voice) -> AsyncIterator[bytes]`. `MockVoiceProvider`: deterministic canned transcript (scripted per test), canned 1kHz-tone WAV bytes, scripted failure modes. `browser.py`: fallback **descriptor** (`sr_available: true`, endpoint hints) the web client uses to select browser-native `SpeechRecognition`/`speechSynthesis` when no server provider. `factory.py`: `AI_VOICE_PROVIDER=browser | mock` (default mock when no key). **`OpenAIAudioProvider` (real server STT/TTS) intentionally NOT built in v0.3 — deferred to v0.4**; the protocol is its future seam. `voice/` never imports `agents/` or `api/`.
|
||||
- **Verify:** `pnpm ai:test` — test_mock + test_factory green (deterministic transcribe/synthesize; failure modes; factory selects mock with empty key, browser when provider=browser; zero network calls)
|
||||
- **Task 5-1-01** (frontend-engineer): build-time metadata — `export const metadata` (title/description/OG) on the **static set: landing, catalog/stack detail, marketplace board, pricing** (the existing layout.tsx/enroll pattern; static-export-compatible, probe 7).
|
||||
- **Task 5-1-02** (frontend-engineer): server-rendered JSON-LD on the board — `ItemList` + `Organization` — rendered into the export.
|
||||
- **Task 5-1-03** (frontend-engineer): client-side JSON-LD on the **runtime set** — `JobPosting` on job detail, `Organization` on employer profile, injected after fetch (A-411 accepted limitation: JS-less crawlers see the shell — recorded).
|
||||
|
||||
#### Task 5-1-03: DefenseStore protocol + SQLite implementation
|
||||
- **Persona:** backend-engineer — **REQ:** REQ-3-006
|
||||
- **Files:** `apps/ai-service/ai_service/voice/defense_store.py`, `apps/ai-service/tests/voice/test_defense_store.py`
|
||||
- **Action:** `DefenseStore` protocol (D-027): `start(defense)`, `append_turn(defense_id, turn)`, `finalize(defense_id, integrity_signals)`, `get(defense_id)`, `list_for_learner(learner_id)`, `close()`. SQLModel `DefenseRecord` (id, learner_id, task_id, status, created/finished_at) + `DefenseTurn` (defense_id FK, turn seq, role examiner|learner, text, ts, latency_ms) + integrity signals JSON on the record (long pauses, off-scope cadence markers — A-109).
|
||||
- **Verify:** `pnpm ai:test` — test_defense_store green (start→append turns→finalize→get roundtrip; ordered turns by seq)
|
||||
### Wave 5-2: sitemap/robots + tests
|
||||
|
||||
### Wave 2: Examiner agent (depends on Wave 1)
|
||||
- **Task 5-2-01** (backend-engineer + security-auditor): `GET /sitemap.xml` + `GET /robots.txt` as plain FastAPI routes on the api app — **probe 6 verified: API routes beat the `StaticFiles("/")` mount**, so they win over the export in single-port production; enumerate the static page list (code constant) + all `published` postings; URLs from the `AI_PUBLIC_BASE_URL` origin; robots points at `/sitemap.xml`. **Unset `AI_PUBLIC_BASE_URL` (G-105): loud G-11 boot warning; sitemap serves the empty static-page set (never malformed/relative URLs) and robots omits the Sitemap directive (absolute-URL spec requirement); fail honest at request time; boot never crashes.** **Do NOT use Next `app/sitemap.ts`** (build-time enumeration cannot see runtime postings under `output: 'export'`).
|
||||
- **Task 5-2-02** (frontend-engineer + backend-engineer): tests — sitemap contents reflect published postings (draft/closed excluded); robots points at the sitemap with the configured origin; metadata/OG present in the exported HTML (`out/` probe); mount-order test (routes beat the static mount); client JSON-LD injection web test.
|
||||
|
||||
#### Task 5-2-01: Examiner agent (seventh agent)
|
||||
- **Persona:** ai-engineer — **REQ:** REQ-3-006
|
||||
- **Files:** `apps/ai-service/ai_service/agents/examiner.py`, `apps/ai-service/ai_service/prompts/examiner.py` (new), `apps/ai-service/ai_service/agents/registry.py` (update: central registration, G-4 pattern), `apps/ai-service/tests/agents/test_examiner.py`
|
||||
- **Action:** `ExaminerAgent(BaseAgent)` (D-030/A-109): builds questions from learner transcript + trace digest + (P4) variant statement; probes understanding + challenges process choices ("why did you choose X at step N?"); streams questions over the existing SSE pipeline; `structured` verdict mode returns verdict + per-answer integrity signal list (long pause flags, off-scope answers) computed from turn metadata; calls voice **only through the `VoiceProvider` protocol** (PERSONAS conflict rule — never concrete providers). Session-scoped history reused from v0.2.
|
||||
- **Verify:** `pnpm ai:test` — test_examiner green (question stream references trace-digest facts; verdict structured output validates via D-020 defense; registry resolves all seven agents; mock-VoiceProvider wiring through protocol only — asserted by import scan in test)
|
||||
|
||||
### Wave 3: Defense endpoints (depends on Wave 2)
|
||||
|
||||
#### Task 5-3-01: Defense session + audio endpoints
|
||||
- **Persona:** backend-engineer — **REQ:** REQ-3-006
|
||||
- **Files:** `apps/ai-service/ai_service/api/defense.py` (new), `apps/ai-service/ai_service/main.py` (update), `apps/ai-service/tests/api/test_defense.py`
|
||||
- **Action:** `POST /v1/defense/start {learner_id, task_id}` → creates DefenseRecord + streams the first examiner question (SSE, agent=examiner); `POST /v1/defense/{id}/answer` (multipart audio from browser MediaRecorder, or `{text}` for typed fallback) → STT via VoiceProvider → append learner turn → stream examiner follow-up (SSE) → TTS audio chunks over `GET /v1/defense/{id}/audio/{turn_id}`; `POST /v1/defense/{id}/finish` → verdict + integrity signals persisted; `GET /v1/defense/{id}` → full transcript + signals. Browser-fallback mode: when provider=browser, start returns the fallback descriptor instead of server audio.
|
||||
- **Verify:** `pnpm ai:test` — test_defense green (full loop with mock voice + mock LLM: start → answer(text) → answer(audio bytes) → finish → transcript retrievable with per-turn latency; unknown id → 404; `GET` signals present after finish)
|
||||
|
||||
### Wave 4: Examiner latency instrumentation (depends on Wave 3)
|
||||
|
||||
#### Task 5-4-01: Per-turn latency instrumentation (mock-based)
|
||||
- **Persona:** voice-engineer — **REQ:** REQ-3-006
|
||||
- **Files:** `apps/ai-service/tests/voice/test_latency.py`, `apps/ai-service/README.md` (update: conversational-budget doc + v0.4 voice note)
|
||||
- **Action:** Instrument per-turn latency (STT ms + LLM TTFT ms + TTS ms) recorded on each DefenseTurn; deterministic test over mock providers asserts instrumentation presence + `latency_ms` populated + budget constant defined (mock runs are near-instant — wall-clock asserted in v0.4 against a real endpoint). README documents the conversational-latency target as a v0.4 acceptance criterion (real STT/TTS deferred per CUT-1/G-7).
|
||||
- **Verify:** `pnpm ai:test` — test_latency green (latency_ms fields populated on every turn; budget constant defined); README documents the deferred real-voice acceptance probe
|
||||
|
||||
### Must-Haves (Phase 5)
|
||||
- [ ] Spoken defense runs end-to-end over HTTP with mock providers: start → answer (audio + typed fallback) → examiner follow-up streams → verdict + transcript persisted (automated)
|
||||
- [ ] Examiner is the seventh registered agent; streams over the existing SSE envelope (meta agent=examiner)
|
||||
- [ ] Instrumented per-turn latency fields populated on every DefenseTurn (STT ms + LLM TTFT ms + TTS ms); conversational budget named (A-109)
|
||||
- [ ] `VoiceProvider` protocol respected: examiner + api touch voice only via the protocol; mock-first — **no task requires a real voice key to pass**
|
||||
- [ ] Browser-native SR/TTS fallback descriptor returned when no server voice provider configured (mock/browser are first-class, D-030)
|
||||
- [ ] Real server STT/TTS (`OpenAIAudioProvider`) explicitly deferred to v0.4 (with real keys/users); the defense pipeline is fully proven over mock+browser — documented in README + release note
|
||||
- [ ] Optional future voice config noted for v0.4 (`AI_VOICE_BASE_URL` / `AI_VOICE_API_KEY`) in `.env.example` + README; keys only in gitignored `.ciagent/.env.secrets`; tests never call a voice API
|
||||
- [ ] Boundary rules hold: `voice/` imports no `agents/`/`api/`; `pnpm ai:test` + `pnpm ai:lint` green
|
||||
**Verification hooks P5:** all suites green + export build emits with metadata; sitemap tests pin `db_path=tmp_path` + `public_base_url` explicitly (hermetic, no `~/.nextcraft`).
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Agent Re-grounding + Learner Surface Integration
|
||||
## Phase 6: Final Review + Ship (milestone release v0.5.6)
|
||||
|
||||
**Requirements:** REQ-3-007, REQ-3-008
|
||||
**Goal:** Lab/Assessor/Proctor consume real engine inputs (live telemetry, grading output, defense signals) with **no mock fallback in the learner path**; the v0.1 sandbox + assessment mockups become real — in-browser build/run (Run/Test buttons + read-only exec output, CUT-2 — no interactive shell), live telemetry panel, browser/typed voice defense, live grading; `pnpm build` + `pnpm typecheck` + full `pnpm ai:test` green
|
||||
**Note:** E2E verification runs against the real engines over HTTP with `AI_PROVIDER=mock` + `AI_VOICE_PROVIDER=mock` permitted (G-2 precedent) — the requirement is real engine plumbing (sandbox/telemetry/grading/defense over real endpoints, no corpus mocks in the learner path); a cloud outage must not block P6.
|
||||
|
||||
### Wave 1: Agent re-grounding (parallel — no shared files)
|
||||
|
||||
#### Task 6-1-01: Lab agent on live telemetry
|
||||
- **Persona:** ai-engineer — **REQ:** REQ-3-007
|
||||
- **Files:** `apps/ai-service/ai_service/agents/lab.py` (update), `apps/ai-service/ai_service/prompts/lab.py` (update: render real trace digest), `apps/ai-service/tests/agents/test_lab_live.py` (new)
|
||||
- **Action:** Lab consumes a **live trace digest** (grading/features `compute_digest` over `TraceStore` events) instead of `corpus/telemetry.py`. build_messages renders digest facts (recent commands, failing tests, idle). Mock-provider scripts assert digest-derived content. v0.2 corpus path removed from the agent (dormant corpus retained until Task 6-1-04 check).
|
||||
- **Verify:** `pnpm ai:test` — test_lab_live green: feedback references events actually present in a seeded SQLite trace (not corpus fixtures); no `corpus.telemetry` import in `agents/lab.py` (AST-asserted)
|
||||
|
||||
#### Task 6-1-02: Assessor agent on grading output
|
||||
- **Persona:** ai-engineer — **REQ:** REQ-3-007
|
||||
- **Files:** `apps/ai-service/ai_service/agents/assessor.py` (update), `apps/ai-service/ai_service/prompts/assessor.py` (update), `apps/ai-service/tests/agents/test_assessor_live.py` (new)
|
||||
- **Action:** Assessor consumes `GradeStore` output (validated `RubricScore` + digest) for learner+task instead of pre-baked artifacts/transcripts; renders strengths/gaps/verdict with rubric-anchored coaching framing. Structured output unchanged (D-020).
|
||||
- **Verify:** `pnpm ai:test` — test_assessor_live green: given a real stored grade, Assessor output reflects its scores; corpus artifact path gone from the agent (AST-asserted)
|
||||
|
||||
#### Task 6-1-03: Proctor agent on telemetry + defense signals
|
||||
- **Persona:** ai-engineer — **REQ:** REQ-3-007
|
||||
- **Files:** `apps/ai-service/ai_service/agents/proctor.py` (update), `apps/ai-service/ai_service/prompts/proctor.py` (update), `apps/ai-service/tests/agents/test_proctor_live.py` (new)
|
||||
- **Action:** Proctor consumes real integrity inputs: idle gaps + command cadence from the trace digest + defense integrity signals from `DefenseStore` → classified signals + coaching interventions (supportive tone retained). Cross-checks variant seed params (P4) for off-template work.
|
||||
- **Verify:** `pnpm ai:test` — test_proctor_live green: signals derived from seeded real trace + defense records; corpus proctor scenarios no longer imported (AST-asserted)
|
||||
|
||||
#### Task 6-1-04: Corpus dormancy + mockup removal verification
|
||||
- **Persona:** lead-developer — **REQ:** REQ-3-007
|
||||
- **Files:** `apps/ai-service/ai_service/corpus/telemetry.py`, `apps/ai-service/ai_service/corpus/artifacts.py`, `apps/ai-service/ai_service/corpus/` (README note), `apps/ai-service/tests/test_corpus_dormancy.py` (new)
|
||||
- **Action:** Verify no production code path imports `corpus/telemetry.py` or `corpus/artifacts.py` anymore (test scans imports across `agents/`, `api/`, engines). Retain files as Phase-3 calibration history with a header note marking them **dormant — v0.2 mocks, not used at runtime**; learner-context corpus stays (agents still need learner context). Disposes v0.2's G-5-class dead-code risk deliberately.
|
||||
- **Verify:** `pnpm ai:test` — test_corpus_dormancy green (zero runtime importers); suite otherwise unchanged
|
||||
|
||||
### Wave 2: Client plumbing + design primitives (parallel — no shared files)
|
||||
|
||||
#### Task 6-2-01: Sandbox build-panel engine client (run/test-only — no raw shell relay)
|
||||
- **Persona:** frontend-engineer — **REQ:** REQ-3-008
|
||||
- **Files:** `apps/web/hooks/use-sandbox-session.ts` (new), `apps/web/lib/engine-client.ts` (new), `apps/web/.env.example` (update)
|
||||
- **Action:** `engine-client.ts`: typed fetch client for `/v1/sandboxes` (create/destroy — learner_id from the v0.3 mock session constant, allowlisted server-side per G-5), `/v1/sandboxes/{id}/files` (workspace CRUD), `/v1/sandboxes/{id}/exec` (run/test), `/v1/variants`, `/v1/assessment/grade`, `/v1/telemetry/traces`, `/v1/defense/*`; base `NEXT_PUBLIC_AI_SERVICE_URL`. `use-sandbox-session.ts`: create sandbox+variant on task open → destroy on unmount (idempotent cleanup, AbortController pattern); 503 pool-full → user-facing "environment busy, retry" (D-032); 403/429 abuse-control surfaced honestly (G-5). **CUT-2 (G-8): NO raw interactive WS terminal relay (keystroke-level stdin/stdout) in v0.3** — the credential pipeline needs *process events* (from Run/Test + file edits), not a live shell; the interactive xterm relay is the most fragile real-time piece and is deferred to v0.4. The build panel is a **Run/Test output viewer** (exec results + telemetry pulse render), not an interactive shell. `@xterm/xterm` is therefore NOT a dependency in v0.3.
|
||||
- **Verify:** `pnpm install && pnpm typecheck` pass; hook unmount destroys the sandbox (manual probe: `curl localhost:8420/v1/sandboxes` shows count drop after navigation); RUN/TEST buttons produce streamed output + telemetry events in the trace
|
||||
|
||||
#### Task 6-2-02: New design primitives
|
||||
- **Persona:** design-system-engineer — **REQ:** REQ-3-008
|
||||
- **Files:** `packages/ui/src/primitives/terminal-frame.tsx` (new), `packages/ui/src/primitives/mic-control.tsx` (new), `packages/ui/src/primitives/grade-badge.tsx` (new), `packages/ui/src/primitives/telemetry-status.tsx` (new), `packages/ui/src/primitives/transcript-viewer.tsx` (new), `packages/ui/src/primitives/index.ts` (update), `packages/ui/src/index.ts` (update)
|
||||
- **Action:** Token-driven primitives: TerminalFrame (CUT-2: a read-only exec-output viewer chrome — streams Run/Test results, NOT an interactive shell), MicControl (record/stop with consent state + no-mic fallback state, MediaRecorder permission UX), GradeBadge (verdict rendering), TelemetryStatus (live event pulse / disconnected indicator), TranscriptViewer (examiner/learner turn list). Dark mode + WCAG AA; stories for each.
|
||||
- **Verify:** primitives import from `@nextcraft/ui`; Storybook stories render dark + light; `pnpm build` (ui package) passes
|
||||
|
||||
#### Task 6-2-03: Defense TS types
|
||||
- **Persona:** data-engineer — **REQ:** REQ-3-008
|
||||
- **Files:** `packages/types/defense.ts` (new), `packages/types/index.ts` (update)
|
||||
- **Action:** TS `DefenseSession`, `DefenseTurn`, `IntegritySignal`, `Verdict` mirroring P5 Python models (cross-referencing header).
|
||||
- **Verify:** `pnpm typecheck` passes
|
||||
|
||||
### Wave 3: Real build surface (depends on Wave 2)
|
||||
|
||||
#### Task 6-3-01: Sandbox mockup → real in-browser IDE
|
||||
- **Persona:** frontend-engineer — **REQ:** REQ-3-008
|
||||
- **Files:** `apps/web/app/(learner)/build/[competencyId]/page.tsx` (rewrite), `apps/web/components/learner/sandbox-terminal.tsx` (new), `apps/web/components/learner/file-tree.tsx` (new), `apps/web/components/learner/run-controls.tsx` (new), `apps/web/components/learner/lab-feedback-panel.tsx` (update: live trace)
|
||||
- **Action:** Replace the mockup with the real build environment (A-103, CUT-2): file tree (HTTP CRUD into the sandbox workdir via `/v1/sandboxes/{id}/files` routes added to api/sandboxes — read/write/list workspace files), syntax-highlight editor (existing), **Run**/**Test** buttons (exec in sandbox; results stream to a read-only TerminalFrame output panel — no interactive shell), starter files from the P4 variant scaffold. Lab panel posts `learner_id+task_id` → streams Lab feedback over the **live** trace (no scenario IDs). Telemetry sidebar shows live TelemetryStatus. Pool-full 503 → busy state with retry; 403/429 surfaced.
|
||||
- **Verify:** with ai-service up: open `/build/comp-01` → variant statement + starter files load → edit a file → **Run** executes the command in-sandbox and output renders in the panel → **Test** runs the test suite in-sandbox → Lab panel streams digest-derived feedback → telemetry status shows live events. Manual probe documented; `pnpm typecheck` green
|
||||
|
||||
### Wave 4: Live defense + grading surfaces (depends on Waves 2-3)
|
||||
|
||||
#### Task 6-4-01: Assessment mockup → live defense + live grading
|
||||
- **Persona:** frontend-engineer — **REQ:** REQ-3-008
|
||||
- **Files:** `apps/web/app/(learner)/defend/[competencyId]/page.tsx` (rewrite), `apps/web/components/learner/defense-session.tsx` (new), `apps/web/components/learner/assessor-results-panel.tsx` (update), `apps/web/components/learner/proctor-banner.tsx` (update), `apps/web/components/learner/oral-defense-interface.tsx` (rewrite or remove)
|
||||
- **Action:** Real assessment flow: **Start Defense** → POST `/v1/defense/start` → examiner question streams → learner answers via MicControl (MediaRecorder webm/opus → multipart POST) with typed fallback when mic denied or `provider=browser` (native `SpeechRecognition`/`speechSynthesis` path per fallback descriptor) → follow-ups stream → **Finish** → verdict + integrity signals panel (TranscriptViewer, GradeBadge) + **Grade My Work** → POST `/v1/assessment/grade` → structured rubric bars render. Proctor banner reads signals for task_id. Loading + error states throughout; no mock defense data remains in the learner path.
|
||||
- **Verify:** with ai-service up: full defense loop runs in-browser (typed fallback acceptable in CI-less manual probe; mic path exercised manually with permission granted); grade panel renders real rubric scores; `pnpm typecheck` green
|
||||
|
||||
### Wave 5: End-to-end verification (depends on Waves 3-4)
|
||||
|
||||
#### Task 6-5-01: Full learner-path E2E probe + green builds
|
||||
- **Persona:** backend-engineer — **REQ:** REQ-3-007, REQ-3-008
|
||||
- **Files:** `apps/ai-service/tests/api/test_e2e_credential_flow.py` (new), `apps/ai-service/README.md` (update: E2E probe doc)
|
||||
- **Action:** Endpoint-level E2E (mock LLM/voice providers, real engines): create variant → create sandbox with task_id → ingest trace events via real sandbox exec → POST grade → start defense (typed answers) → finish → assert: trace persisted, grade stored + digest-linked, defense transcript + signals stored, Proctor/Assessor endpoints serve them. No corpus fixture anywhere in the flow (AST-asserted). Run `pnpm build` + `pnpm typecheck` + full `pnpm ai:test` at repo root; fix all failures before phase ship.
|
||||
- **Verify:** `pnpm ai:test` green incl. test_e2e_credential_flow; `pnpm build` + `pnpm typecheck` green; README E2E probe section documents the manual browser pass
|
||||
|
||||
### Must-Haves (Phase 6)
|
||||
- [ ] Lab/Assessor/Proctor operate on real inputs with **no mock fallback in the learner path** (AST-verified: no corpus telemetry/artifact/proctor imports in production paths)
|
||||
- [ ] Learner builds in-browser for real (CUT-2): Run/Test buttons execute in a namespace sandbox and stream output to a read-only panel; file tree CRUD works; starter files come from the learner's variant scaffold
|
||||
- [ ] Live telemetry: build activity streams to ai-service and the sidebar shows live status (TelemetryStatus); trace persisted in SQLite
|
||||
- [ ] Live defense: start → answer (mic or typed fallback) → examiner follow-ups → finish → transcript + integrity signals + verdict rendered; browser-native path works with no server voice key
|
||||
- [ ] Live grading: grade request returns structured rubric scores computed from the real trace digest; grade panel renders them
|
||||
- [ ] 503 pool-full surfaced honestly in UI; navigating away destroys the sandbox (no leaked sandboxes — `GET /v1/sandboxes` manual probe)
|
||||
- [ ] Examiner remains protocol-clean (voice only via `VoiceProvider`); module boundary rules hold across all new code
|
||||
- [ ] `pnpm build` and `pnpm typecheck` pass; full `pnpm ai:test` green; no cloud/voice calls in any automated test
|
||||
- [ ] Release-note input (for P7): v0.3 ships real engines; **identity/age-gating (KYC) remains deferred — age-gating is still a visual mockup** (A-110); sandbox scope is coding-IDE only (design tool/simulation deferred to v0.4, D-025)
|
||||
- **Task 6-1-01** (lead-developer → ci-review personas): multi-persona review of the full milestone branch diff (all phases — correctness, testing, security, performance, maintainability); **known follow-ups checked for scope creep: seed dormancy verified (mock-data untouched except marketplace path), employer dashboard stays mock (A-407), pricing stays mock (A-412) — zero payments/employer-auth/scheduler code**. P0s fixed in-phase.
|
||||
- **Task 6-1-02** (ci-audit): reconstruction test (git log ↔ .ciagent/ files), file/branch/commit discipline, tag hygiene; critical fixes in-phase.
|
||||
- **Task 6-1-03** (lead-developer → ci-ship): merge milestone → main; tag **v0.5.6** (milestone release); Gitea release with full summary + `nextcraft-linux-x64` + `.sha256` assets (D-036); delete milestone branches; mark REQ-6-001..006 complete; clear checkpoint.
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: Final Review + Ship (no planned tasks)
|
||||
## Must-Haves (per phase — concrete, falsifiable, test-linked)
|
||||
|
||||
Orchestrated by the SHIP stage, not this plan: multi-persona code review (correctness, testing, module boundaries, secrets hygiene — keys absent from code/logs/commits/errors, localhost-only CORS, no PII in prompts), project health audit (reconstruction, .ciagent/ discipline, branch/commit hygiene), then merge milestone → main, tag the final v0.2.x patch, create the Gitea release, mark all 8 v0.3 requirements complete.
|
||||
### Phase 1
|
||||
|
||||
**Release-note honesty:** the release note must state (a) Lab/Assessor/Proctor now run on real engine inputs (v0.2 mock-input caveat retired), (b) sandbox scope = coding IDE only — design tool + simulation environments deferred to v0.4 (D-025), (c) identity/age-gating (KYC) deferred per founder directive — age-gating remains the v0.1 visual flow mockup; abuse control (per-learner sandbox caps + server-side learner allowlist, G-5) ships in place of auth (A-110), (d) voice runs mock-first with browser-native fallback — real server STT/TTS deferred to v0.4 (CUT-1), (e) sandbox resource limits are partially enforced (memory/CPU/wall-clock kernel-enforced via rlimits; per-sandbox pids + hard disk quota are NOT — mitigated by a workdir-size sweep and per-learner caps; full enforcement requires cgroup delegation, deferred to the post-MVP containerd backend, D-024/G-1/G-2).
|
||||
- **MH-1a**: Store contract tests (tmp_path db): five tables created on the shared `db_path`; `(source, external_id)` and `(posting_id, learner_id)` unique constraints fire on duplicates; WAL/pragmas/`_validate` hooks pinned.
|
||||
- **MH-1b**: Ingest idempotency — running the seeded source twice yields identical row counts (no dupes); upsert overwrites in place.
|
||||
- **MH-1c**: Admin fail-closed matrix — unset `AI_ADMIN_TOKEN` → every admin route 403 + boot warning in caplog; wrong/absent token → 403; correct token → 200. No admin route passes unguarded.
|
||||
- **MH-1d**: Apply gate composition **provable by test** — G-5 403 fires first, verify-CTA 403 second, 429 at cap; published-only (404 missing / 409 draft+closed); duplicate → 200-with-existing; confirmation payload exact; **G-18 stub markers gone** (grep + response-schema pin).
|
||||
- **MH-1e**: Web shells read real data — board/detail/employer consume `/v1/marketplace/*` (web tests pin the engine-client functions); `notFound()` on 404; honest empty board; **E87 regression class pinned (G-101/G-110): placeholder shells emitted for both routes (build does not throw), catch-all serves arbitrary ids, hydration + `notFound()` works at a non-placeholder id via full page load**; the honest-501 stub pin in tests/identity/test_identity.py:444 rewritten onto the marketplace router.
|
||||
- **MH-1f**: Seed fidelity — 20 postings + 10 employers, IDs match the TS fixtures, `matchScore` absent from all marketplace-path payloads; **seed postings enter `published` with original `postedAt` (G-102)**.
|
||||
|
||||
**Secrets-hygiene checklist (P7):** `.ciagent/.env.secrets` gitignored and never committed; `AI_VOICE_API_KEY` / `AI_TUTOR_API_KEY` referenced only via env; SQLite DBs + sandbox dirs gitignored; no keys in logs, error messages, or test fixtures.
|
||||
### Phase 2
|
||||
|
||||
**Disposal checks (G-5 class):** v0.2 dormant corpus files carry the dormant-header note (Task 6-1-04); any now-unused mock-data exports for the old sandbox/defense mockups (e.g. `aiTutorResponses`-class leftovers) must be removed or deprecated by review.
|
||||
- **MH-2a**: Tagging cloud-free via `MockLLMProvider`; `PostingTags` schema enforced (skills ≤10 pin on an 11-skill response); D-020 single bounded retry pinned on schema violation.
|
||||
- **MH-2b**: Tag-on-ingest + re-tag-on-update **overwrites** (old tags absent after update — single source of truth).
|
||||
- **MH-2c**: Facets derive from real stored tags — API facets payload matches published-only tags exactly; `JobFilters` option lists come from the payload (web test).
|
||||
|
||||
---
|
||||
### Phase 3
|
||||
|
||||
- **MH-3a**: Codec roundtrip — struct-packed float32 BLOB via `put_vector`/`get_vector` with `model`+`dim` per row.
|
||||
- **MH-3b**: Deterministic similarity — same text → identical vector; related > unrelated cosine (pinned numbers).
|
||||
- **MH-3c**: Factory G-11 boot-safe — `openai-embed` unconfigured → boot survives with loud log + mock fallback; `_sanitize` redaction: no key string in any error (pinned).
|
||||
- **MH-3d**: Hybrid ordering — facets hard-filter before cosine ranking; no facets → pure semantic rank; `match_score` = computed cosine, **mock `matchScore` absent from marketplace-path API payloads and marketplace surfaces (G-107: the pin's grep excludes `packages/mock-data` and the employer/admin route groups, which stay mock)**; absent score = honest absence (no active query → no score, G-106).
|
||||
- **MH-3e**: Embed-on-publish + lazy re-embed on model change (pinned); candidate-side embeddings never stored.
|
||||
- **MH-3f**: Skill-breakdown = tag overlap + vector contribution, deterministic pin.
|
||||
|
||||
### Phase 4
|
||||
|
||||
- **MH-4a**: Enrichment append-only (no update/delete path) + precedence manual > llm > raw, every pairing pinned.
|
||||
- **MH-4b**: Provenance markers — every enriched field carries `{value, source}` in the API; mock-llm copy visibly marked end-to-end.
|
||||
- **MH-4c**: Enrich-on-first-publish fires exactly once per employer (re-publish no-op pin).
|
||||
- **MH-4d**: Resume bounds matrix — PDF/TXT/MD only (other → 422), >2 MB → 413, empty PDF extract → 422 "text layer required", ~100K char cap.
|
||||
- **MH-4e**: Draft-then-confirm — parse returns `status: "draft"`, nothing persisted; confirm persists; **PII sentinel caplog test green** (raw bytes never logged/stored).
|
||||
- **MH-4f**: pypdf pinned `>=6.19,<7` in pyproject + bootstrap installs it; generated PDF fixture parses.
|
||||
|
||||
### Phase 5
|
||||
|
||||
- **MH-5a**: Sitemap reflects published postings (draft/closed excluded) + static page list; origin from `AI_PUBLIC_BASE_URL`; **every sitemap job/employer URL must serve 200 on the single-port app via the G-101 catch-all shells (URL-resolvability pinned — G-109, never advertise URLs the deploy 404s)**; unset origin → empty set + warning, robots omits the Sitemap directive (G-105).
|
||||
- **MH-5b**: robots.txt points at `/sitemap.xml`.
|
||||
- **MH-5c**: Metadata + OG present in the exported HTML (`out/` probe); board ItemList+Organization JSON-LD server-rendered.
|
||||
- **MH-5d**: Runtime JSON-LD injected client-side after fetch (web test); JS-crawler limitation recorded.
|
||||
- **MH-5e**: Mount-order — sitemap/robots routes beat the static export in single-port mode (route test).
|
||||
|
||||
## Milestone Must-Haves
|
||||
|
||||
- **MH-M1**: All six requirements complete with their REQUIREMENTS.md success criteria: REQ-6-001 (store+lifecycle+routes+adapter, surfaces read real data, D-043 gates composed), REQ-6-002 (tags via D-020, facets drive filters), REQ-6-003 (protocol+SQLite vectors, similarity ranking, computed match scores), REQ-6-004 (enrichment with provenance), REQ-6-005 (resume auto-fill with review gate, bounded), REQ-6-006 (metadata+JSON-LD+sitemap/robots, D-039 unchanged).
|
||||
- **MH-M2**: Zero regressions on existing suites at every phase ship: 488 backend pytest, web tsx --test 22, CLI 61, typecheck 8, ruff, export build.
|
||||
- **MH-M3**: G-18 stub markers gone (MH-1d) — the one honest-501 marketplace route is now real.
|
||||
- **MH-M4**: The D-043 gate order provable by test (MH-1d) — allowlist → verified-18+ → caps, unchanged.
|
||||
- **MH-M5**: Hermeticity — every new test pins `db_path` (and any state dir) to `tmp_path`; no test touches `~/.nextcraft` or the cloud (v0.5 P04 lesson).
|
||||
- **MH-M6**: Scope discipline — pricing stays mock (A-412), employer dashboard stays mock (A-407), no payments/employer-auth/scheduler/crawler code; seed corpus is the only ingestion source.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Dual-schema drift (TS wire types ↔ pydantic contracts)** — mitigated by the dual-schema-sync constraint (data-engineer owns both sides of every change) + parity checks in review.
|
||||
- **Match-score removal breaking non-marketplace mockups** — jobs.ts/employers.ts stay intact for employer/admin mockups; only the marketplace path switches (A-406); pinned by build + web tests.
|
||||
- **`AI_EMBED_BASE_URL` misconfiguration** (ollama-cloud has no /embeddings — D-051 probe) — default mock; G-11 boot-safe fallback with loud log; endpoint is config, never code.
|
||||
- **Admin token operational risk** — unset token locks the founder out of admin routes (fail-closed is the intent); loud boot warning names the fix; documented in README quickstart.
|
||||
- **Brute-force cosine at scale** — pilot-only (≤10⁴ postings, A-403); documented; `VectorStore` protocol keeps the pgvector swap API-compatible.
|
||||
- **pypdf text-layer variance across real resumes** — honest 422 on empty extract (no OCR); TXT/MD path always works; generated-fixture tests pin both.
|
||||
- **Client-shipped runtime fetch + JS-crawlers see shells on runtime pages** — accepted limitation (A-411), recorded; static set carries the SEO weight.
|
||||
- **E87 static-export build break (grill-verified, G-101)** — Next 15 `output:'export'` hard-throws for dynamic routes lacking `generateStaticParams` (even `[]`); mitigation: one placeholder param per route + FastAPI catch-all shell serving (routes beat the mount, probe 6); pinned by release-blocking tests (G-110).
|
||||
|
||||
## User-Facing Surface
|
||||
|
||||
The primary user-facing surface is the **learner build + defend flow** at `http://localhost:3000`, backed by the real engines in ai-service at `http://localhost:8420`:
|
||||
1. **Marketplace job board** (`/marketplace`): reads real published postings from `/v1/marketplace/postings`; facet sidebar derives from real AI tags; match scores are computed, not mocked.
|
||||
2. **Job detail** (`/marketplace/jobs/[jobId]`): real posting fetch; full description, tags, employer info, computed skill-breakdown; honest 404 on missing postings; apply flow live with verify-CTA on 403 and 409 on closed.
|
||||
3. **Employer profile** (`/marketplace/employers/[employerId]`): merged raw+enriched fields with visible per-field provenance markers.
|
||||
4. **Learner profile auto-fill**: resume upload → draft review → confirm (nothing saved silently).
|
||||
5. **Founder/admin surface via curl + engineering docs**: `AI_ADMIN_TOKEN`-guarded posting authoring, lifecycle, ingest trigger, enrichment override, applications review — documented in a README marketplace quickstart addition (curl examples included).
|
||||
6. **SEO**: sitemap/robots served by the API; metadata + JSON-LD on the page set.
|
||||
|
||||
- `/dashboard` — AI tutor chat (Coach/Tutor, streaming) + Mentor panel (unchanged from v0.2)
|
||||
- `/learn/[competencyId]` — byte viewer with streaming Tutor explanations (unchanged)
|
||||
- `/build/[competencyId]` — **real in-browser build environment**: file tree, syntax editor, **Run/Test buttons that execute in a namespace sandbox and stream output to a read-only panel** (CUT-2 — no interactive shell), live telemetry status, live Lab feedback, per-learner variant task statement
|
||||
- `/defend/[competencyId]` — **live oral defense + live grading**: Examiner voice/typed dialogue, transcript + integrity signals, real rubric scores from the process trace
|
||||
## Happy Path (end-to-end, written BEFORE execute)
|
||||
|
||||
The marketplace, employer, and admin surfaces are unchanged from v0.1/v0.2.
|
||||
|
||||
## Happy Path
|
||||
|
||||
1. Learner opens `/build/comp-01` → a per-learner **variant statement** and starter files load; a namespace sandbox is created for the session
|
||||
2. Learner edits files in the tree and clicks **Run**/**Test** → commands execute in the sandbox and real output renders in the build panel; the telemetry sidebar pulses as events stream to ai-service and persist in SQLite
|
||||
3. The Lab panel streams feedback derived from the **live trace digest** (real commands, real failures)
|
||||
4. Learner opens `/defend/comp-01` → **Start Defense**: the Examiner streams an opening question ("Walk me through your build — why did you structure it this way?")
|
||||
5. Learner answers by voice (mic consent → MediaRecorder → STT) or typed fallback → examiner follow-ups probe the trace ("You hit three test failures before passing — what changed?"); TTS plays examiner audio (or browser speech in fallback)
|
||||
6. Learner finishes the defense → transcript + integrity signals appear; verdict renders in a GradeBadge
|
||||
7. Learner clicks **Grade My Work** → the grading engine computes the digest from the real trace, rubric-scores it, and the panel renders per-criterion bars + strengths/gaps/verdict
|
||||
8. Proctor banner shows integrity signals from the live trace + defense in coaching tone; Mentor panel on `/dashboard` can narrate the real outcome
|
||||
9. Navigating away destroys the sandbox (pool slot freed); killing ai-service shows inline error + retry states on every panel, with no crashes
|
||||
1. Founder sets `AI_ADMIN_TOKEN=ops-secret` in `.env`; boots the stack; log shows the marketplace store on the shared DB (no unset-token warning).
|
||||
2. Founder runs `POST /v1/marketplace/admin/ingest` (X-Admin-Token header) → seeded corpus populates: 20 postings + 10 employers, upsert-idempotent.
|
||||
3. Founder authors a posting via `POST /v1/marketplace/admin/postings` (draft) → `POST .../publish` → posting is published (auto-tagged, embedded).
|
||||
4. Learner opens `/marketplace` → board client shell fetches real postings → filters by a facet derived from real tags → opens `/marketplace/jobs/{id}` detail.
|
||||
5. An 18+ verified, allowlisted learner clicks Apply → passes G-5 → verified-18+ → rate caps → posting published → `application` row (status `submitted`) → honest confirmation; a second click returns the same application (idempotent).
|
||||
6. Founder reviews via `GET /v1/marketplace/admin/applications` (token-guarded) and contacts the employer out-of-band (no notification infra, A-408).
|
||||
7. `GET /sitemap.xml` lists the static pages + every published posting.
|
||||
|
||||
## UX Acceptance Criteria
|
||||
|
||||
1. Run/Test output visibly reflects the real sandbox execution (command round-trip to the sandbox, real stdout/stderr), not a replay animation
|
||||
2. The learner path contains **no mock engine data** — scenarios, canned artifacts, and scripted defense transcripts from v0.2 are gone from runtime
|
||||
3. Variant statements visibly differ between two learner sessions on the same competency
|
||||
4. Mic permission flow is graceful: consent prompt, recording indicator, no-mic/typed fallback, and browser-native speech path when no server voice key is configured
|
||||
5. Defense transcript renders turn-by-turn with latency shown; integrity signals render in coaching (supportive) tone
|
||||
6. Grade results render as structured per-criterion bars with verdict, from the real trace — not from final-output-only heuristics
|
||||
7. Pool-full (503) shows an honest "environment busy — retry" state; navigation/unmount destroys sandboxes with no leaks
|
||||
8. When ai-service is unreachable: inline error + retry on every panel — no crashes, no console errors, no blank UI
|
||||
9. All new UI uses design tokens, supports dark mode, meets WCAG AA contrast, responsive at 375px / 768px / 1280px
|
||||
10. `pnpm build` and `pnpm typecheck` pass with zero errors; `pnpm ai:test` green cloud-free and voice-key-free
|
||||
1. Board shows real postings with an honest empty state (no fake cards when the store is empty).
|
||||
2. Missing posting id → honest 404 page (no fabricated job).
|
||||
3. Unverified learner clicking Apply → verify-CTA prompt linking to enrollment (never a dead JSON error or a fake "applied").
|
||||
4. Match scores are computed cosine values only; no mock `matchScore` anywhere; absent score = absent query context (honest).
|
||||
5. Tag facets derive from real stored tags (API payload → sidebar), never hardcoded lists.
|
||||
6. Per-field provenance markers visible on the employer profile (raw/llm/manual).
|
||||
7. Sitemap lists published postings; robots points at the sitemap; metadata present in the exported HTML.
|
||||
8. Resume auto-fill never persists without an explicit confirm; oversize/wrong-format/text-less PDFs get honest, actionable errors.
|
||||
9. All existing accessibility baselines hold (WCAG AA on new states).
|
||||
+105
-17
@@ -8,45 +8,130 @@ Nextcraft is an AI-native outcome school where graduates prove what they can bui
|
||||
|
||||
---
|
||||
|
||||
## Current Milestone: v0.3 — Credential Engines
|
||||
## Hotfix Round (2026-09-21, founder directive)
|
||||
|
||||
**Scope:** Replace v0.2's mock engine inputs with real credential engines. Build the sandbox fabric (sandboxed IDE / design tool / simulation), the live in-environment build-telemetry pipeline, the process-trace grading engine, per-learner variant task generation, and the oral/voice defense with AI examiner. Lab/Assessor/Proctor agents move from mock inputs to real engine inputs; the six tutor agents operate on authentic telemetry and artifacts.
|
||||
**v0.4.6 — deploy-static-404 + dev experience.** Third attempt at the site-down issue (domain root → API JSON 404). Root cause found by host probe (10.0.0.105): the deployed uvicorn ran mount-less — `apps/web/out` was never built on the host, `dev -d` auto-wiring silently skipped `AI_WEB_STATIC_DIR`, node/pnpm were invisible to non-interactive shells, and NOTHING detected the mount-less boot. Founder directive: fix the deploy experience so the one-liner install handles all of it — smooth first run, upgrades, and reboot persistence. Delivered: `nextcraft deploy` (clone/update → install → venv → export build → restart → verify, idempotent), `deploy --service` (generated systemd unit, enabled), toolchain discovery (PATH → nvm → PNPM_HOME → corepack; prefix carries node+pnpm dirs for split installs), loud mount-less boot warning, `verify` live mount probe (GET / must be HTML), install.sh next-steps rewritten. Deployed live on the host via the new command; site 200, service active+enabled.
|
||||
|
||||
**v0.4.7 — pre-public hardening (the 4 P1 flags).** WS ingest HMAC tokens (grading input no longer LAN-poisonable; manager mints into NC_INGEST_URL, endpoint verifies, per-boot secret default), nonce+counter defense-id mint (same-ms collision → permanent 500 closed), browser-mode SR/TTS client implementation (the descriptor's v0.5 contract finally honored — recognizeOnce/speakText pure module + defense-session wiring + honest unsupported state), 8 module pins. All P1 flags from the v0.5 final review are now closed; the recorded pre-public hardening backlog is empty.
|
||||
|
||||
---
|
||||
|
||||
## Milestone v0.6 — Marketplace Engine (ACTIVE)
|
||||
|
||||
**Scope (auto-decided at SPECIFY per full autonomy — founder scope question left unanswered 2026-09-21):**
|
||||
|
||||
The marketplace has no backend today: every job board, job detail, employer profile, search, and pricing surface reads `packages/mock-data` directly; the only live route is the G-18 `POST /v1/marketplace/apply` gated stub (18+ verified, honest 501). The guiding outcome — ≥1,000 learners placed in AI-orchestration roles via the Nextcraft marketplace — runs *through* this surface, so the marketplace engine is the highest-leverage next build.
|
||||
|
||||
**Scope decision (D-047):** Activate **REQ-F-011..016** as REQ-6-001..006 — the real marketplace backend behind the existing UI:
|
||||
|
||||
1. **Job aggregation pipeline** (REQ-6-001, from REQ-F-011): job ingestion from employer postings + a seeded/synthetic corpus (3M+ external jobs is post-MVP — the pipeline ingests at pilot scale with the same adapter contract); postings store (6th D-027 store family); posting lifecycle (draft → published → closed).
|
||||
2. **AI-powered tagging** (REQ-6-002, from REQ-F-012): skills/seniority/category extraction from posting text via the existing LLM structured-output layer (D-020), provider-agnostic, mock-first.
|
||||
3. **Semantic vector search** (REQ-6-003, from REQ-F-013): embedding-backed role matching — candidate-competency vs posting-requirement similarity; SQLite-local at pilot scale (no pgvector before Postgres migration), deterministic embedding provider protocol (mock-first, real provider env-configured).
|
||||
4. **Company enrichment** (REQ-6-004, from REQ-F-014): employer profile enrichment from posting/company data (descriptions, social links, logo refs) — LLM-assisted, human-in-the-loop founder review deferred to post-pilot.
|
||||
5. **Resume/profile parsing** (REQ-6-005, from REQ-F-015): learner profile auto-fill from an uploaded resume — LLM structured extraction mapping to the competency profile shape.
|
||||
6. **SEO-optimized programmatic pages** (REQ-6-006, from REQ-F-016): server-rendered job/stack/landing pages with metadata, sitemap, structured data — built on the existing Next.js static-export single-port deploy (D-039-compatible).
|
||||
|
||||
**Deferred beyond v0.6 (unchanged):** REQ-F-018 payments/subscriptions, REQ-F-019 human tutor marketplace, REQ-F-020 CIRR placement tracking.
|
||||
|
||||
**Success (milestone-level):** the marketplace surfaces read from a real backend (postings, search, matching, employer profiles) instead of mock data; job posting + tagging + search work end-to-end for a pilot employer; the deploy stays one-command on the v0.4.x/v0.5.x fabric.
|
||||
|
||||
## Prior Milestone: v0.5 — Real Server Voice, KYC/Identity, Sandbox Environments, Seq-Lease (COMPLETE, shipped as v0.4.5; hotfixes v0.4.6 deploy-static-404, v0.4.7 pre-public hardening)
|
||||
|
||||
**Scope (the four seams D-016 deferred out of v0.4, locked at v0.5 Phase 0 SPECIFY):**
|
||||
|
||||
1. **Real server STT/TTS** (CUT-1/G-7 seam): implement the `openai-audio` VoiceProvider against OpenAI-compatible `/audio/transcriptions` (STT) + `/audio/speech` (TTS) — the D-030 protocol drop-in; mock stays first-class for tests; browser fallback stays for the no-key path. Voice defense (Examiner flows) gains the real server path end-to-end.
|
||||
2. **KYC/identity + age-gating backend** (REQ-F-017): real identity verification behind a provider protocol (mock-first, D-014 pattern), age-gate enforcement (school 16+, marketplace 18+ with verified identity) replacing the v0.1 visual-only flow, session/learner identity wired into the API surface (the G-5 allowlist evolves toward real identity).
|
||||
3. **Design/simulation sandbox environments** (REQ-F-021 remainder): extend the D-024/D-025 namespace fabric beyond the coding IDE — design-tool and simulation environment types alongside the existing build environment, one lifecycle, per-type telemetry.
|
||||
4. **Exec-telemetry seq-lease / replay-margin fix**: the one-line ACK gap documented in the P6 lesson (P07 review): reconnect replay margin so at-least-once ingest acknowledges received seqs and the capture agent resumes from the ack — closing the documented gap.
|
||||
|
||||
**Success (milestone-level):** voice defense runs on real server STT/TTS when keys exist; age-gating is enforced by the backend (not a mockup page); the sandbox fabric provisions design/sim environments; the reconnect-replay gap is closed with a regression test.
|
||||
|
||||
## Prior Milestone: v0.4 — Distribution & Bootstrap CLI (COMPLETE, shipped as v0.3.4; hotfixes v0.3.5 fresh-box experience, v0.3.6 single-port unattended deploy)
|
||||
|
||||
**Scope (founder directive, 2026-09-12):** Streamline installing Nextcraft. Ship a bootstrap CLI with a single-liner install script, and publish release binaries on an ongoing basis for every release going forward.
|
||||
|
||||
**Delivered:** `nextcraft` CLI — `doctor` (prerequisite checks), `bootstrap` (deps + venv + env from templates + key validation), `verify` (health check), `dev` (thin passthrough to scripts/dev.sh); one-liner install script downloading the linux x64 binary from the latest Gitea release with sha256 + version integrity gates; binary release pipeline attached to every ship from v0.3.2 onward; install/quickstart documentation backed by a fresh-clone E2E test. All 5 requirements (REQ-4-001..005) complete.
|
||||
|
||||
**Status of v0.3:** Complete and shipped (v0.2.8). Credential engines live: namespace-isolated sandbox fabric, live build telemetry, process-trace grading, seeded variants, oral defense; real learner build/defense/grading surfaces.
|
||||
|
||||
**Status of v0.2:** Complete and shipped (v0.2.0). Six AI tutor agents live over mockengine inputs (D-015).
|
||||
|
||||
**Deferred from earlier plan:** REQ-F-017 (identity verification + 16+/18+ age-gating) is explicitly deferred to a later milestone per founder directive. Age-gating remains the v0.1-style visual flow mockup; no real KYC backend is built in v0.3.
|
||||
**Deferred from earlier plan:** REQ-F-017 (identity verification + age-gating KYC), real server STT/TTS, design/simulation sandbox environments, and the exec-telemetry seq-lease are all deferred to v0.5. Age-gating remains the v0.1-style visual flow mockup.
|
||||
|
||||
**Tech stack:** v0.1 TS monorepo (pnpm/turborepo, Next.js) + v0.2 Python FastAPI ai-service + new credential-engine services (sandbox fabric orchestrator, telemetry ingest, grading engine) in Python/TypeScript as determined at RESEARCH.
|
||||
|
||||
---
|
||||
|
||||
## Requirements (Validated)
|
||||
## v0.4 Requirements (Complete)
|
||||
|
||||
The following requirements have been validated during specification and are locked for milestone v0.3 (REQ-F-007..010 and REQ-F-021 activated from the deferred pool; REQ-F-017 deferred per founder directive):
|
||||
All 5 v0.4 requirements (REQ-4-001..005) are complete and shipped as v0.3.4:
|
||||
1. Bootstrap CLI — `nextcraft` executable with `doctor` / `bootstrap` / `verify` / `dev` (REQ-4-001, REQ-4-002)
|
||||
2. One-liner install — `curl | sh` fetching the linux x64 binary from the latest Gitea release with sha256 + version integrity verification (REQ-4-003)
|
||||
3. Ongoing release binaries — every release from v0.3.2 onward ships the CLI binary + checksum as release assets (REQ-4-004)
|
||||
4. Install documentation — README quickstart + CLI reference verified by a fresh-clone E2E test (REQ-4-005)
|
||||
|
||||
1. Sandbox fabric — sandboxed IDE, design tool, and simulation environments with isolated execution and lifecycle management (REQ-F-021)
|
||||
2. Live build telemetry — in-environment capture of process events (keystrokes, commands, file diffs, run/test results) streamed to ai-service (REQ-F-010)
|
||||
3. Process-trace grading engine — grade artifacts from their process traces, not just final output (REQ-F-007); feeds the Assessor agent real inputs
|
||||
4. Variant task generation — per-learner task variants so no two learners receive identical prompts (REQ-F-008)
|
||||
5. Oral/voice defense — AI examiner conducts spoken defense of submitted work (REQ-F-009); feeds the Proctor/Mentor agents
|
||||
6. Agent re-grounding — Lab/Assessor/Proctor consume real engine inputs (telemetry, traces, defenses) instead of v0.2 mocks
|
||||
7. Learner surface integration — wire the v0.1 sandbox + assessment mockups to the real engines (build/run in-browser, live telemetry, live defense)
|
||||
## v0.3 Requirements (Complete)
|
||||
|
||||
## v0.2 Requirements (Complete)
|
||||
|
||||
All 12 v0.2 requirements (REQ-2-001..012) are complete and shipped as v0.2.0. See REQUIREMENTS.md traceability matrix.
|
||||
All 8 v0.3 requirements (REQ-3-001..008) are complete and shipped as v0.2.8. See REQUIREMENTS.md traceability matrix.
|
||||
|
||||
## v0.1 Requirements (Complete)
|
||||
|
||||
All 28 v0.1 requirements (REQ-001..028) are complete and shipped as v0.1.0. See REQUIREMENTS.md traceability matrix.
|
||||
|
||||
## Clarified Assumptions (v0.6 CLARIFY stage, full autonomy — auto-resolved)
|
||||
|
||||
| # | Ambiguity | Resolution | Confidence |
|
||||
|---|-----------|------------|-------------|
|
||||
| A-401 | Who authors postings at pilot (no employer auth exists)? | **Founder-authored; employer self-serve deferred.** Postings enter via founder/admin API routes (`/v1/marketplace/admin/postings` create + lifecycle transitions draft→published→closed), guarded by an `AI_ADMIN_TOKEN` env-header check (keys env-only, D-014; operator route — D-043 learner gate composition untouched). The employer dashboard's posting-management mockup stays mock; real employer self-serve lands with the deferred B2B employer network milestone. | 0.85 |
|
||||
| A-402 | Ingestion adapter contract shape? | **`PostingSource` protocol in `ai_service/marketplace/`** (D-031 module, D-014 mock-first): `fetch() -> Iterable[IngestPosting]` — pydantic-typed with `source`, `external_id`, employer + posting fields. Two v0.6 implementations: `seeded_corpus` (in-repo JSON seed, ported once from packages/mock-data jobs.ts/employers.ts per D-021 convention) and `admin_api` (A-401 founder path). Upserts idempotent on `(source, external_id)`. No scheduler/crawler — the 3M+ external corpus drops in later as a new adapter with zero store/route changes. | 0.80 |
|
||||
| A-403 | Where do embeddings live at SQLite scale (pip-only venv, no pgvector)? | **`embedding` table inside the marketplace SQLite store (6th D-027 family)**: posting_id → BLOB of struct-packed float32 + model + dim columns; the candidate side embeds the learner's competency profile text on demand (not stored). **Brute-force cosine in pure Python** at query time (pilot scale ≤ ~10⁴ postings — no numpy, no FAISS/chromadb, no new deps). `VectorStore` protocol-wrapped so the Postgres migration swaps to pgvector with no API changes (D-027/D-019 pattern). | 0.85 |
|
||||
| A-404 | Embedding provider + dimensionality? | **`EmbeddingProvider` protocol (mirror of D-030 VoiceProvider)**: deterministic mock (256-dim feature-hash of posting/profile tokens — similarity is deterministic; tests never call the cloud) + real OpenAI-compatible `POST {AI_EMBED_BASE_URL}/embeddings` via `AI_EMBED_MODEL`/`AI_EMBED_API_KEY` (env-only; ollama-cloud exposes /embeddings). Dimension is **not hardcoded** — persisted per row with the model name; on provider/model change, postings re-embed lazily. **[SUPERSEDED by D-051 probe: ollama-cloud does NOT expose /embeddings (404 path-not-found) — `AI_EMBED_BASE_URL` must target another OpenAI-compatible endpoint (e.g. local ollama `:11434/v1`); protocol design unaffected, default stays mock]** | 0.75 |
|
||||
| A-405 | Does semantic search replace the filter sidebar? | **Hybrid — augments, not replaces.** REQ-6-002 tag facets (skills/seniority/category + location/remote/salary) remain hard filters driving the existing sidebar; the semantic query + similarity compose as a **ranking layer over the facet-filtered set** (no facets selected → pure semantic ranking). Match score on job cards = cosine(candidate-competency vector, posting vector), replacing the mock `matchScore`; skill-breakdown panel derives from tag overlap + per-skill vector contribution. ARCHITECTURE.md's old "filter UI replaced" future-note is superseded by this hybrid. | 0.80 |
|
||||
| A-406 | What happens to packages/mock-data jobs.ts/employers.ts (runtime data vs static export)? | **Marketplace web surfaces switch from build-time mock imports to runtime client fetch** (A-002 direct-fetch + engine-client pattern, proven by build/defend surfaces): board/detail/employer-profile become client shells fetching `/v1/marketplace/*`; `generateStaticParams` on jobs/[jobId] and employers/[employerId] is removed (one static shell per route, data at runtime). jobs.ts/employers.ts are ported **once** into the Python seed fixture (A-402) and go dormant for the marketplace path (mock-data remains for admin/employer-dashboard mockups + shared types). No build-time DB reads — the D-039 deploy export build stays repo-only. | 0.80 |
|
||||
| A-407 | Employer auth model at pilot? | **None — employers are data, not principals, in v0.6.** Employer records are ingested/enriched entities keyed by `employer_id` on postings; no employer login, sessions, or accounts. The employer dashboard surface stays the v0.1 mockup; enrichment is founder-triggered/admin-reviewed (A-409), never employer-edited. Real employer auth lands post-payments with the B2B employer network. | 0.85 |
|
||||
| A-408 | Apply-flow shape beyond the 501 stub? | **The existing `POST /v1/marketplace/apply` route becomes real, D-043 composition unchanged** (G-5 allowlist → verified-18+ verdict → rate caps): body `{posting_id}` → posting must be `published` → insert an `application` row (status `submitted`, identity ref, timestamp) in the marketplace store → honest confirmation payload. Duplicate apply (same learner + posting) is idempotent. **No notification infra**: founder reviews applications via `/v1/marketplace/admin/applications` and contacts employers out-of-band. | 0.75 |
|
||||
| A-409 | Enrichment provenance rules? | **Separate `employer_enrichment` table — raw fields never overwritten** (REQ-6-004): append-only rows `(employer_id, field, value, source: llm\|manual, model, created_at)`. Read precedence **manual > llm > ingested-raw**; API responses merge and carry per-field provenance markers (A-304 honesty pattern — LLM-authored copy visibly marked). Enrich-on-post: new employer's first published posting triggers LLM structured extraction (D-020, mock-first) of description/social links/logo refs. Founder override = upsert a `manual`-source row, never a mutation of `llm` rows. | 0.85 |
|
||||
| A-410 | Resume upload format bounds? | **PDF/TXT/MD only, ≤2 MB, text-layer required.** PDF text via `pypdf` (pure-Python, pip-reachable through the A-009 venv-pip channel — the only new dep); extracted text capped ~100K chars before the D-020 structured-extraction call mapping to the competency-profile shape. Image-only/scanned PDFs → honest 422 ("text layer required") — **no OCR in v0.6**. Parsed result returns as a DRAFT: nothing persists to the learner profile until explicit learner confirm (REQ-6-005 review gate); raw resume bytes never stored, never logged (A-305 pattern). | 0.80 |
|
||||
| A-411 | Which pages get programmatic SEO treatment under static export — and how? | **Build-time-static set: landing, catalog/stack detail, marketplace board, pricing** — full server-component metadata (title/desc/OG) + JSON-LD (Organization, stack/Course, ItemList) rendered at export (D-039 build unchanged). **Runtime-data set: job detail + employer profile** — client shells (A-406) inject `JobPosting`/`Organization` JSON-LD + title client-side after fetch (best-effort per-job SEO at pilot; recorded as an accepted limitation — JS-less crawlers see the shell). **Sitemap/robots: `/sitemap.xml` + `/robots.txt` served dynamically by ai-service** (API routes beat the static mount, D-039), enumerating static pages + published postings; robots points at `/sitemap.xml`. | 0.70 |
|
||||
| A-412 | Pricing page — real or mock? | **Stays mock — zero backend.** Payments/subscriptions are REQ-F-018, explicitly deferred beyond v0.6 (D-047). The pricing page gains only build-time metadata/OG (A-411 static set); no pricing API routes, no package enforcement — posting is free at pilot (founder-mediated). | 0.95 |
|
||||
|
||||
## Clarified Assumptions (v0.5 CLARIFY stage, full autonomy — auto-resolved)
|
||||
|
||||
| # | Ambiguity | Resolution | Confidence |
|
||||
|---|-----------|------------|------------|
|
||||
| A-301 | Which STT/TTS endpoint for `openai-audio`? | **OpenAI-compatible `/audio/transcriptions` + `/audio/speech`** via `AI_VOICE_BASE_URL` (ollama-cloud does not expose audio endpoints; any OpenAI-compatible audio API works — the provider is endpoint-agnostic by config, D-014 pattern). Model via `AI_VOICE_MODEL` (default `whisper-1` STT / `tts-1` TTS class names, configurable). Keys env-only, never committed. | 0.75 |
|
||||
| A-302 | Does real voice replace browser fallback? | **No — layered.** `openai-audio` when configured, browser-native second-class fallback (CUT-1 keeps it first-class for the no-key path), deterministic mock for tests. The defense surface probes provider capability and degrades with a visible badge (mic mock vs browser vs server). | 0.85 |
|
||||
| A-303 | KYC vendor in v0.5? | **No vendor — protocol + mock-first backend.** `IdentityProvider` protocol (submit/poll/verify), deterministic mock (approve-on-policy), SQLite store. A real vendor (Stripe Identity/Persona/Onfido class) drops in later without API changes. Solo-founder economics: no vendor spend before pilot. | 0.85 |
|
||||
| A-304 | What counts as "verified 18+" for marketplace? | **Provider verdict = DOB-verified 18+; until a real vendor exists the mock verdict is explicit and marked `mock` in API responses** so downstream surfaces can label unverified state honestly (never display mock-verified as production-verified). | 0.80 |
|
||||
| A-305 | PII storage? | **Server-side only, minimal**: document refs + provider verdicts in SQLite; raw documents NEVER stored, NEVER logged (payload scrubbing pinned by test). Enrollment DOB stored as derived age-band, not raw DOB where possible. | 0.85 |
|
||||
| A-306 | Age-gate enforcement points? | **School enrollment (16+)**: identity submit → verify → age check at enrollment API. **Marketplace (18+ verified)**: gated marketplace routes check verified-identity verdict; unverified → 403 with verify-CTA payload. G-5 learner allowlist REMAINS as a pilot guard (identity doesn't replace sandbox rate caps). | 0.82 |
|
||||
| A-307 | What are "design" and "simulation" environments concretely? | **Same namespace fabric, typed starter contents + allowed commands.** `design`: canvas-style artifacts (files the learner edits: SVG/HTML/schematic text), Run = validator/renderer harness command; `simulation`: parameterized run harness (benchmark scripts + dataset files), Run = bounded harness execution. No new isolation tech — D-024 unchanged; the ENVIRONMENT is a typing over starter files + command policy. | 0.78 |
|
||||
| A-308 | Does the learner UI need new surfaces for design/sim? | **Reuse the build surface.** The existing /build flow accepts `environment` kind; the file tree, editor, Run/Test buttons, and read-only output panel (CUT-2) all carry over; only starter contents and command policy differ per kind. No new route groups; variants carry the kind. | 0.80 |
|
||||
| A-309 | Seq-lease protocol shape? | **Ingest acks highest-contiguous received seq per (learner, task) on the existing WS (JSON ack frame); the capture agent trims its spool to the ack and replays from there on reconnect.** Replay margin bounded (spool cap already exists). No new endpoint; G-3 flood semantics unchanged; acks are advisory hints, gap detection stays authoritative. | 0.80 |
|
||||
| A-310 | Where does identity live in the app? | **New `ai_service/identity/` module (D-031 pattern — extend ai-service, no new apps)**: provider protocol + mock, SQLite store (D-027 family), `/v1/identity/*` router. Web enrollment + marketplace surfaces call it via engine-client. | 0.85 |
|
||||
|
||||
## Clarified Assumptions (v0.4 CLARIFY stage, full autonomy — auto-resolved)
|
||||
|
||||
| # | Ambiguity | Resolution | Confidence |
|
||||
|---|-----------|------------|-------------|
|
||||
| A-201 | CLI language/toolchain for the binary? | **Probe-driven at RESEARCH** — Go → Rust → Node SEA → Python zipapp fallback chain; spec stays toolchain-agnostic so PLAN locks the probe-verified toolchain | 0.70 |
|
||||
| A-202 | Does bootstrap replace scripts/bootstrap.sh? | **No — reuse it.** CLI wraps existing `scripts/bootstrap.sh` + `scripts/dev.sh` via subprocess; zero orchestration logic duplicated in the CLI (thin passthrough pattern) | 0.85 |
|
||||
| A-203 | Where does the one-liner fetch the binary? | **Gitea latest-release API** (`/repos/{owner}/{repo}/releases/latest`) → download `nextcraft-linux-x64` + `.sha256` asset; repo raw serves `install.sh` as the stable URL | 0.80 |
|
||||
| A-204 | Install target + PATH? | **~/.local/bin** (XDG-style, no sudo), PATH hint printed when missing; `--dest` override flag | 0.85 |
|
||||
| A-205 | Binary "ongoing releases" scope? | **Every ship from v0.4 onward** attaches `nextcraft-linux-x64` + sha256 sidecar to the Gitea release — the ship workflow gains an asset step; retroactive binaries for old releases NOT required | 0.90 |
|
||||
| A-206 | No binary available yet / non-linux? | **Graceful degradation**: install script prints source-bootstrap instructions (git clone + scripts/bootstrap.sh) — never a hard fail | 0.88 |
|
||||
| A-207 | Checksum trust root? | **sha256 sidecar shipped as a release asset next to the binary** (same release, same channel); script verifies download against it. Signature/PKI out of scope for v0.4 (single forge, TLS transport) | 0.75 |
|
||||
| A-208 | Which prerequisites does doctor check? | node ≥18, pnpm ≥8, python3 ≥3.11, git, `unshare` availability (sandbox fabric needs it) — versions from the existing bootstrap tooling, not invented | 0.85 |
|
||||
| A-209 | Does `dev` manage multiple processes? | **No.** Thin passthrough to scripts/dev.sh only — the CLI stays bootstrap-scoped (D-016); orchestration remains in dev.sh | 0.82 |
|
||||
| A-210 | `.env.secrets` handling by bootstrap? | **Template copy only for `.env.example` → `.env`; secrets NEVER generated, NEVER committed; bootstrap validates presence of optional keys and warns (not blocks) when missing — mock-first providers keep the stack runnable** | 0.90 |
|
||||
|
||||
## Clarified Assumptions (v0.3 CLARIFY stage, full autonomy — auto-resolved)
|
||||
|
||||
| # | Ambiguity | Resolution | Confidence |
|
||||
|---|-----------|------------|-------------|
|
||||
| A-101 | Sandbox isolation technology? | **`unshare` user+mount+pid+net namespace subprocess isolation** per sandbox (probe-verified: in-ns uid=0, network fully isolated with 0 interfaces, writes land in an isolated bind-mounted workdir; proc-remount is not permitted in this context but is not required). No Docker/Podman/VMs — none present on the box; no sudo. A `SandboxBackend` protocol keeps a future containerd swap possible. Falls back further to a plain chroot-free subprocess with a cwd-jail if userns ever unavailable (tested path is userns). | 0.8 |
|
||||
| A-102 | Sandbox scope in v0.3? | **Coding IDE only** (web terminal + file tree + run/test). The "design tool" and "simulation" environments specified in REQ-F-021 are deferred to v0.4 — a single real build environment is enough to prove the credential pipeline end-to-end (telemetry → trace → grade → defense). | 0.75 |
|
||||
| A-103 | Live in-browser build UX? | **WebSocket xterm.js terminal** attached to the bwrap sandbox shell + HTTP file-tree/CRUD + run/test buttons. No full Monaco LSP in v0.3 — a code editor with syntax highlight (existing) + real shell is sufficient and far cheaper. | 0.72 |
|
||||
| A-103 | Live in-browser build UX? | **Run/Test buttons executing in the namespace sandbox + HTTP file-tree/CRUD + read-only exec-output panel** (CUT-2/G-8 — the interactive xterm.js shell relay is deferred to v0.4; `@xterm/*` is not a v0.3 dependency). No full Monaco LSP in v0.3 — a code editor with syntax highlight (existing) is sufficient and far cheaper. | 0.72 |
|
||||
| A-104 | Telemetry transport? | **WebSocket** from sandbox to a new ingestion endpoint on ai-service for live events; **SQLite-backed** ordered event log (`ai_service/telemetry/`) gives durability + at-least-once delivery + replay. Events carry monotonic `seq` per (learner,task) so gaps are detectable. | 0.8 |
|
||||
| A-105 | Where do traces live? | **SQLite** (`ai_service` data dir), introducing the first real persistence. SQLModel/SQLAlchemy for typed access. Chosen over Postgres because solo-founder + single box + low write volume; the `TraceStore` protocol is Postgres-migration-ready like SessionStore was. | 0.75 |
|
||||
| A-106 | Process-trace grading model? | **LLM-based grader**: structure the trace into a compact timeline digest (command categories, error/fix cycles, idle gaps, test passes) → Assessor-style rubric prompt → structured score via existing D-020 JSON defense. Deterministic features (test pass/fail, edit count) computed in code, not left to the LLM. | 0.7 |
|
||||
@@ -54,7 +139,7 @@ All 28 v0.1 requirements (REQ-001..028) are complete and shipped as v0.1.0. See
|
||||
| A-108 | Voice defense — STT/TTS providers? | **Provider-agnostic, mock-first like the LLM layer (D-014).** Real path: browser `MediaRecorder` → audio to ai-service → **OpenAI-compatible `/audio/transcriptions`** (Whisper STT) and **`/audio/speech`** (TTS) against ollama-cloud or a compatible endpoint; fallbacks: browser `SpeechRecognition`/`speechSynthesis` when no server keys. `VoiceProvider` protocol + deterministic mock (returns canned transcript) so tests never call a voice API. | 0.62 |
|
||||
| A-109 | Defense dialogue shape? | Reuse BaseAgent: an `Examiner` agent (seventh agent) streams examiner questions over the existing SSE pipeline; integrity signals (long pauses, off-scope answers, reading-from-notes cadence) emitted alongside the transcript to Proctor. | 0.8 |
|
||||
| A-110 | KYC / age-gating in v0.3? | **Deferred per founder directive.** No real identity backend. Age-gating stays the v0.1 visual flow mockup. Personas omit a security-engineer; security review via verifier + Phase 7 secrets-hygiene checklist. **Abuse control is NOT deferred with KYC (G-5):** v0.3 ships per-learner sandbox caps (`AI_SANDBOX_MAX_PER_LEARNER`), a global create-rate cap, and a server-side `learner_id` allowlist (`AI_LEARNER_ALLOWLIST`) so the unauthenticated surface cannot exhaust shared NPROC/disk. Documented in the release note. | 0.98 |
|
||||
| A-111 | New services vs extend ai-service? | **Extend ai-service**, don't fork new Python apps. Telemetry ingestion, trace grading, variant generation, voice, and sandbox orchestration all live as new modules in `apps/ai-service` (they share the LLM provider pool + config + session infra). Only the in-sandbox capture agent is a separate tiny Python process shipped into the bwrap environment. | 0.82 |
|
||||
| A-111 | New services vs extend ai-service? | **Extend ai-service**, don't fork new Python apps. Telemetry ingestion, trace grading, variant generation, voice, and sandbox orchestration all live as new modules in `apps/ai-service` (they share the LLM provider pool + config + session infra). Only the in-sandbox capture agent is a separate tiny Python process shipped into the namespace sandbox. | 0.82 |
|
||||
| A-112 | Sandbox on a single dev/school box — capacity? | v0.3 targets **1–5 concurrent sandboxes** (founder + pilot learners). No horizontal scaling, no queue. Concurrency guard returns 503 when full. Scaling is post-MVP. | 0.8 |
|
||||
|
||||
## Clarified Assumptions (v0.2 CLARIFY stage, full autonomy — auto-resolved)
|
||||
@@ -133,6 +218,9 @@ The following remain deferred beyond v0.3 and will be activated in subsequent mi
|
||||
| D-013 | v0.1 prototype founder-agreed; D-001 business-logic gate unlocked | Founder approved starting v0.2 with AI Tutor Architecture, which constitutes agreement of the v0.1 prototype per D-001. Recorded at v0.2 SPECIFY. | Business logic authorized from v0.2 onward |
|
||||
| D-014 | Provider-agnostic LLM layer; ollama-cloud as initial provider | OpenAI-compatible client abstraction with pluggable providers: ollama-cloud (https://ollama.com/v1, default), local OpenAI-compatible endpoint, deterministic mock (tests/CI). Keys in gitignored .ciagent/.env.secrets, never in code or commits. | apps/ai-service llm package with 3 providers; default=ollama-cloud |
|
||||
| D-015 | All six agents implemented as real LLM services; engines mocked | Coach/Tutor/Mentor fully real. Lab/Assessor/Proctor are real LLM logic over mock inputs (simulated telemetry, pre-baked artifacts) since sandbox fabric, assessment engine, and identity verification are v0.3+. Consistent with v0.1's mock-data approach. | REQ-F-001..006 complete in v0.2; real engines deferred to v0.3+ |
|
||||
| D-016 | v0.4 = Distribution & Bootstrap CLI (founder directive supersedes previously-named v0.4 seams) | Founder directive 2026-09-12: focus this milestone on streamlining install, a bootstrap CLI with a one-liner install script, ongoing release binaries. Real server STT/TTS, KYC, design/simulation envs, seq-lease move to v0.5. | Milestone scope locked at SPECIFY; binary = CLI-only, linux x64 |
|
||||
| D-047 | v0.6 = Marketplace Engine (auto-decided at SPECIFY, full autonomy; founder scope question left unanswered 2026-09-21) | Activates REQ-F-011..016 as REQ-6-001..006: real marketplace backend (aggregation, tagging, vector search, enrichment, resume parsing, SEO pages). Rationale: the guiding outcome (≥1,000 placements via the marketplace) runs through the only surface with zero backend; every prior milestone fed the credential side, placement side is the bottleneck. Confidence 0.80 ≥ threshold 0.60 (D-007). External 3M+ job corpus scaled to pilot (adapter contract preserves the full-scale path). | Milestone scope locked at SPECIFY; tags on v0.5.x line; feature milestone |
|
||||
| D-039 | Single-port deploy + unattended dev (v0.3.6 hotfix, founder directive) | Only :8420 reachable behind HAProxy; site was down (domain root hit the API 404). Web app = static export served same-origin by the ai-service (`AI_WEB_STATIC_DIR`); `dev -d`/`stop`/`log` daemonize ops; durable state (DB, sandboxes, pid/log) moves to `~/.nextcraft/` out of the repo. | One port serves UI+API; unattended deploy recipe in deploy/README.md |
|
||||
|
||||
---
|
||||
|
||||
|
||||
+154
-47
@@ -1,5 +1,79 @@
|
||||
# Nextcraft — REQUIREMENTS.md
|
||||
|
||||
## v0.6 Requirements (ACTIVE — Marketplace Engine, milestone v0.6)
|
||||
|
||||
### Job Aggregation & Postings
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-6-001 | Job posting backend (from REQ-F-011, pilot-scaled): postings store (6th D-027 store family — SQLite, insert-safe, protocol-wrapped), employer posting lifecycle (draft → published → closed), ingestion adapter contract (employer-authored postings + seeded corpus at pilot scale; 3M+ external aggregation preserved by the adapter interface), marketplace API routes replacing the mock-data reads on job board/detail/employer surfaces, 18+ verified identity gate on apply/post flows (D-043 composition retained) | critical | 1 | pending |
|
||||
| REQ-6-002 | AI-powered tagging (from REQ-F-012): skills/seniority/category extraction from posting text via the LLM structured-output layer (D-020) — deterministic mock-first, provider-agnostic; tags persisted with the posting; tag facets drive the search filter sidebar | high | 2 | pending |
|
||||
|
||||
### Search & Matching
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-6-003 | Semantic vector search (from REQ-F-013): embedding provider protocol (mock-first, deterministic; real provider env-configured per D-014), vector store at pilot scale (SQLite-local — no pgvector before the Postgres migration), candidate-competency ↔ posting-requirement similarity ranking, match-score surfacing on job cards + candidate matches (replaces mock match scores) | critical | 3 | pending |
|
||||
|
||||
### Enrichment & Parsing
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-6-004 | Company enrichment (from REQ-F-014): employer profile enrichment from posting/company data (descriptions, social links, logo refs) via LLM structured output; enrich-on-post with manual founder override path; enriched data stored separately from raw (provenance marked) | medium | 4 | pending |
|
||||
| REQ-6-005 | Resume/profile parsing (from REQ-F-015): learner profile auto-fill from an uploaded resume — LLM structured extraction mapping to the competency-profile shape, learner-review-before-save (never silently overwrite), upload bounded (size/format guard, house pattern) | medium | 4 | pending |
|
||||
|
||||
### SEO & Programmatic Pages
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-6-006 | SEO-optimized programmatic pages (from REQ-F-016): server-rendered job/stack/landing pages with metadata (title/description/OG), sitemap.xml + robots.txt, JSON-LD structured data (JobPosting schema), static-export compatible (D-039 single-port deploy unchanged) | high | 5 | pending |
|
||||
|
||||
## v0.5 Requirements (Complete — Real Server Voice, KYC/Identity, Sandbox Environments, Seq-Lease; shipped as v0.4.5)
|
||||
|
||||
### Real Server Voice
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-5-001 | `openai-audio` VoiceProvider: server STT (`/audio/transcriptions`) + TTS (`/audio/speech`) against an OpenAI-compatible endpoint via the existing D-030 protocol; provider selection by `AI_VOICE_PROVIDER` (+ base URL/key from env, never committed); deterministic mock stays first-class; browser fallback unchanged | critical | 2 | complete |
|
||||
| REQ-5-002 | Voice defense real path end-to-end: examiner dialogue answers transcribed server-side (audio upload → transcript), examiner questions spoken via server TTS (audio returned to the client); transcripts + integrity signals unchanged | critical | 2 | complete |
|
||||
|
||||
### Identity & Age-Gating
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-5-003 | Identity provider protocol + mock-first backend (REQ-F-017): verify-identity flow (submit → pending → verified/rejected with document refs), provider-agnostic (mock default; a real KYC vendor drops in later), PII stored server-side only, never logged | critical | 3 | complete |
|
||||
| REQ-5-004 | Age-gating enforced by the backend: school floor 16+ verified at enrollment, marketplace 18+ with verified identity — API surfaces reject under-age/unverified callers on gated routes (replaces the v0.1 visual-only flow; G-5 allowlist evolves toward real identity, allowlist remains as pilot guard) | critical | 3 | complete |
|
||||
|
||||
### Sandbox Environments
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-5-005 | Design + simulation sandbox environment types (REQ-F-021 remainder): extend the namespace fabric with environment kinds beyond the coding IDE (design-tool: canvas/editor surfaces with file artifacts; simulation: run/benchmark harnesses) — one lifecycle, one telemetry path, per-type starter contents + allowed commands | high | 4 | complete |
|
||||
| REQ-5-006 | Environment-typed learner surface: the build/defend flow accepts environment kind, telemetry captures per-kind events, grading digest stays kind-agnostic | high | 4 | complete |
|
||||
|
||||
### Telemetry Durability
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-5-007 | Exec-telemetry seq-lease/replay-margin fix: WS ingest acknowledges received seqs; capture agent resumes from the ack on reconnect (bounded replay margin) — closes the P6-lesson one-line ACK gap with a real-server regression test | high | 1 | complete |
|
||||
|
||||
## v0.4 Requirements (Complete — Distribution & Bootstrap CLI, shipped as v0.3.4)
|
||||
|
||||
### Bootstrap CLI
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-4-001 | `nextcraft` CLI (linux x64 binary): `doctor` command checking prerequisites (node, pnpm, python3, git, unshare) with actionable error messages | critical | 1 | complete |
|
||||
| REQ-4-002 | `bootstrap` command: pnpm install, ai-service venv + pinned deps, .env from templates, key validation, .env.secrets handling; `verify` health check (ports, imports, builds); `dev` thin passthrough to scripts/dev.sh | critical | 1 | complete |
|
||||
|
||||
### Distribution
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-4-003 | One-liner install script (`curl -fsSL <url> \| bash`): detects linux x64, resolves latest release from Gitea API, downloads binary + checksum, verifies sha256, installs to ~/.local/bin (PATH hint), degrades to source-bootstrap instructions when no binary | critical | 2 | complete |
|
||||
| REQ-4-004 | Binary release pipeline: reproducible linux x64 build script, sha256 checksum sidecar, upload as release assets on every ship from v0.4 onward (ongoing binaries requirement) | critical | 2 | complete |
|
||||
| REQ-4-005 | Install + quickstart documentation: README one-liner quickstart, CLI command reference, fresh-clone-to-running-stack end-to-end verification | high | 3 | complete |
|
||||
|
||||
## v0.3 Requirements (Credential Engines)
|
||||
|
||||
### Sandbox & Telemetry
|
||||
@@ -8,22 +82,22 @@
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-3-001 | Sandbox fabric: isolated per-learner execution environments (sandboxed IDE, design tool, simulation) with lifecycle management | critical | 1 | complete |
|
||||
| REQ-3-002 | Sandbox isolation + resource limits: per-learner isolation boundary, CPU/memory quotas (rlimits), wall-clock time quota, disk-quota via per-sandbox workdir usage sweep (best-effort, not kernel-enforced), no cross-tenant access, snapshot support. **Known gap (v0.3): per-sandbox pids and hard disk caps are NOT kernel-enforceable without cgroup delegation/sudo — documented as accepted risk** | critical | 1 | complete |
|
||||
| REQ-3-003 | Live build telemetry: in-environment capture of process events (commands, file diffs, run/test results, activity) streamed reliably to ai-service with per-learner trace persistence | critical | 2 | pending |
|
||||
| REQ-3-003 | Live build telemetry: in-environment capture of process events (commands, file diffs, run/test results, activity) streamed reliably to ai-service with per-learner trace persistence | critical | 2 | complete |
|
||||
|
||||
### Credential Engines
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-3-004 | Process-trace grading engine: grade artifacts from their full process traces; rubric-aligned structured scores; feeds Assessor real inputs | critical | 3 | pending |
|
||||
| REQ-3-005 | Variant task generation: per-learner task variants (no two learners get identical prompts); variant seed registry; difficulty normalization | high | 4 | pending |
|
||||
| REQ-3-006 | Oral/voice defense: AI examiner conducts spoken defense (STT → dialogue → TTS); transcript + integrity signals captured; feeds Proctor/Mentor | high | 5 | pending |
|
||||
| REQ-3-004 | Process-trace grading engine: grade artifacts from their full process traces; rubric-aligned structured scores; feeds Assessor real inputs | critical | 3 | complete |
|
||||
| REQ-3-005 | Variant task generation: per-learner task variants (no two learners get identical prompts); variant seed registry; difficulty normalization | high | 4 | complete |
|
||||
| REQ-3-006 | Oral/voice defense: AI examiner conducts spoken defense (STT → dialogue → TTS); transcript + integrity signals captured; feeds Proctor/Mentor | high | 5 | complete |
|
||||
|
||||
### Agent Re-grounding & Integration
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-3-007 | Agent re-grounding: Lab consumes live telemetry; Assessor consumes grading-engine output; Proctor consumes telemetry + defense integrity signals (replace v0.2 mocks) | critical | 6 | pending |
|
||||
| REQ-3-008 | Learner surface integration: sandbox mockup → real in-browser build/run with live telemetry; assessment mockup → live defense + live grading | critical | 6 | pending |
|
||||
| REQ-3-007 | Agent re-grounding: Lab consumes live telemetry; Assessor consumes grading-engine output; Proctor consumes telemetry + defense integrity signals (replace v0.2 mocks) | critical | 6 | complete |
|
||||
| REQ-3-008 | Learner surface integration: sandbox mockup → real in-browser build/run with live telemetry; assessment mockup → live defense + live grading | critical | 6 | complete |
|
||||
|
||||
---
|
||||
|
||||
@@ -65,58 +139,58 @@
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-001 | Monorepo scaffolding: pnpm workspaces, turborepo, Next.js app, TypeScript config, ESLint, Prettier | critical | 1 | complete |
|
||||
| REQ-002 | Shared component library: design tokens, Button, Input, Card, Badge, Avatar, Dialog, Navigation, Table, Tabs, Progress, Tooltip, Skeleton, Toast | critical | 1 | pending |
|
||||
| REQ-003 | Mock data layer: typed mock data for competency stacks, job listings, candidate profiles, employers, learner progress | critical | 1 | pending |
|
||||
| REQ-004 | Routing and navigation: App Router route groups for (learner), (marketplace), (employer), (admin); shared layout components; cross-surface navigation | critical | 1 | pending |
|
||||
| REQ-005 | Responsive layout system: mobile, tablet, desktop breakpoints; container components; grid system | high | 1 | pending |
|
||||
| REQ-002 | Shared component library: design tokens, Button, Input, Card, Badge, Avatar, Dialog, Navigation, Table, Tabs, Progress, Tooltip, Skeleton, Toast | critical | 1 | complete |
|
||||
| REQ-003 | Mock data layer: typed mock data for competency stacks, job listings, candidate profiles, employers, learner progress | critical | 1 | complete |
|
||||
| REQ-004 | Routing and navigation: App Router route groups for (learner), (marketplace), (employer), (admin); shared layout components; cross-surface navigation | critical | 1 | complete |
|
||||
| REQ-005 | Responsive layout system: mobile, tablet, desktop breakpoints; container components; grid system | high | 1 | complete |
|
||||
|
||||
### Learner Surface
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-006 | Landing page: hero, value proposition, program highlights, how-it-works (Byte→Build→Demonstrate→Defend), testimonials mockup, CTA to program catalog | critical | 2 | pending |
|
||||
| REQ-007 | Program catalog: grid of competency stacks (AI Orchestration Engineer, AI Safety & Governance Lead, Human-AI Product Designer, AI-Augmented Field Operator, Computational Sciences Practitioner); stack cards with role descriptions | critical | 2 | pending |
|
||||
| REQ-008 | Competency stack view: selected stack detail with 12-18 competencies listed, progress indicators, microcredential badges, mastery status | critical | 2 | pending |
|
||||
| REQ-009 | Learner dashboard: active competencies, progress graph, recent artifacts, upcoming defenses, AI tutor chat mockup, milestone tracker | critical | 2 | pending |
|
||||
| REQ-010 | Byte tutorial viewer: 3-7 minute micro-tutorial layout with concept panel, worked example panel, code/design/simulation viewer mockup | high | 2 | pending |
|
||||
| REQ-011 | Build sandbox mockup: sandboxed IDE/design tool/simulation UI mockup with toolbar, file explorer, editor area, telemetry sidebar (process capture indicators) | high | 2 | pending |
|
||||
| REQ-012 | Assessment/defense mockup: rubric display, AI reviewer panel, oral defense interface (voice/mic mockup), process trace timeline, artifact viewer | high | 2 | pending |
|
||||
| REQ-006 | Landing page: hero, value proposition, program highlights, how-it-works (Byte→Build→Demonstrate→Defend), testimonials mockup, CTA to program catalog | critical | 2 | complete |
|
||||
| REQ-007 | Program catalog: grid of competency stacks (AI Orchestration Engineer, AI Safety & Governance Lead, Human-AI Product Designer, AI-Augmented Field Operator, Computational Sciences Practitioner); stack cards with role descriptions | critical | 2 | complete |
|
||||
| REQ-008 | Competency stack view: selected stack detail with 12-18 competencies listed, progress indicators, microcredential badges, mastery status | critical | 2 | complete |
|
||||
| REQ-009 | Learner dashboard: active competencies, progress graph, recent artifacts, upcoming defenses, AI tutor chat mockup, milestone tracker | critical | 2 | complete |
|
||||
| REQ-010 | Byte tutorial viewer: 3-7 minute micro-tutorial layout with concept panel, worked example panel, code/design/simulation viewer mockup | high | 2 | complete |
|
||||
| REQ-011 | Build sandbox mockup: sandboxed IDE/design tool/simulation UI mockup with toolbar, file explorer, editor area, telemetry sidebar (process capture indicators) | high | 2 | complete |
|
||||
| REQ-012 | Assessment/defense mockup: rubric display, AI reviewer panel, oral defense interface (voice/mic mockup), process trace timeline, artifact viewer | high | 2 | complete |
|
||||
|
||||
### Marketplace Surface
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-013 | Job board listing: searchable grid of AI-era job listings, filter sidebar (skills, seniority, location, salary), result cards with match score | critical | 3 | pending |
|
||||
| REQ-014 | Job detail page: full job description, required competencies, employer info, application CTA, related jobs, AI-matched skills breakdown | critical | 3 | pending |
|
||||
| REQ-015 | Employer profile: company overview, logo, description, social links, open positions, company culture mockup | high | 3 | pending |
|
||||
| REQ-016 | Search/filter UI: semantic search bar, skill tags, category filters, seniority filter, remote/on-site toggle, saved searches mockup | critical | 3 | pending |
|
||||
| REQ-017 | Pricing page: job posting packages (single, bundle, enterprise), talent access plans, subscription tiers, feature comparison table | high | 3 | pending |
|
||||
| REQ-013 | Job board listing: searchable grid of AI-era job listings, filter sidebar (skills, seniority, location, salary), result cards with match score | critical | 3 | complete |
|
||||
| REQ-014 | Job detail page: full job description, required competencies, employer info, application CTA, related jobs, AI-matched skills breakdown | critical | 3 | complete |
|
||||
| REQ-015 | Employer profile: company overview, logo, description, social links, open positions, company culture mockup | high | 3 | complete |
|
||||
| REQ-016 | Search/filter UI: semantic search bar, skill tags, category filters, seniority filter, remote/on-site toggle, saved searches mockup | critical | 3 | complete |
|
||||
| REQ-017 | Pricing page: job posting packages (single, bundle, enterprise), talent access plans, subscription tiers, feature comparison table | high | 3 | complete |
|
||||
|
||||
### Employer Dashboard
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-018 | Employer dashboard overview: active postings, applicant pipeline, talent matches, analytics mockup (charts, placement stats) | critical | 4 | pending |
|
||||
| REQ-019 | Talent search: searchable candidate database with AI-matched filters, candidate cards showing competency stack, microcredentials, artifact count, defense score | critical | 4 | pending |
|
||||
| REQ-020 | Candidate profile view: full candidate profile with artifact gallery, process trace summary, oral defense transcripts, competency graph, microcredential verification | high | 4 | pending |
|
||||
| REQ-021 | Posting management: create/edit/delete job postings, posting status tracking, applicant list per posting, interview pipeline mockup | high | 4 | pending |
|
||||
| REQ-018 | Employer dashboard overview: active postings, applicant pipeline, talent matches, analytics mockup (charts, placement stats) | critical | 4 | complete |
|
||||
| REQ-019 | Talent search: searchable candidate database with AI-matched filters, candidate cards showing competency stack, microcredentials, artifact count, defense score | critical | 4 | complete |
|
||||
| REQ-020 | Candidate profile view: full candidate profile with artifact gallery, process trace summary, oral defense transcripts, competency graph, microcredential verification | high | 4 | complete |
|
||||
| REQ-021 | Posting management: create/edit/delete job postings, posting status tracking, applicant list per posting, interview pipeline mockup | high | 4 | complete |
|
||||
|
||||
### Admin Surface
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-022 | Admin overview: platform metrics (learners, employers, placements, completion rate, NPS), recent activity feed, system health mockup | critical | 5 | pending |
|
||||
| REQ-023 | Learner management: searchable learner table, learner detail view, progress tracking, competency completion status, credential issuance log | high | 5 | pending |
|
||||
| REQ-024 | Competency graph viewer: interactive visualization of competency stacks and their relationships, node/edge graph using react-flow, stack details on node click | high | 5 | pending |
|
||||
| REQ-025 | Marketplace moderation: job posting review queue, employer verification queue, flagged content, content moderation tools mockup | high | 5 | pending |
|
||||
| REQ-022 | Admin overview: platform metrics (learners, employers, placements, completion rate, NPS), recent activity feed, system health mockup | critical | 5 | complete |
|
||||
| REQ-023 | Learner management: searchable learner table, learner detail view, progress tracking, competency completion status, credential issuance log | high | 5 | complete |
|
||||
| REQ-024 | Competency graph viewer: interactive visualization of competency stacks and their relationships, node/edge graph using react-flow, stack details on node click | high | 5 | complete |
|
||||
| REQ-025 | Marketplace moderation: job posting review queue, employer verification queue, flagged content, content moderation tools mockup | high | 5 | complete |
|
||||
|
||||
### Polish & Integration
|
||||
|
||||
| ID | Description | Priority | Phase | Status |
|
||||
|----|-------------|----------|-------|--------|
|
||||
| REQ-026 | Cross-surface navigation: role switcher (learner/employer/admin), breadcrumbs, consistent header/footer across all surfaces | critical | 6 | pending |
|
||||
| REQ-027 | Visual consistency audit: typography scale, color palette, spacing system, dark mode toggle, accessibility baseline (WCAG AA contrast) | high | 6 | pending |
|
||||
| REQ-028 | Component library finalization: Storybook setup, component documentation, prop tables, usage examples | medium | 6 | pending |
|
||||
| REQ-026 | Cross-surface navigation: role switcher (learner/employer/admin), breadcrumbs, consistent header/footer across all surfaces | critical | 6 | complete |
|
||||
| REQ-027 | Visual consistency audit: typography scale, color palette, spacing system, dark mode toggle, accessibility baseline (WCAG AA contrast) | high | 6 | complete |
|
||||
| REQ-028 | Component library finalization: Storybook setup, component documentation, prop tables, usage examples | medium | 6 | complete |
|
||||
|
||||
---
|
||||
|
||||
@@ -136,18 +210,18 @@
|
||||
|
||||
| ID | Description | Priority | Milestone | Status |
|
||||
|----|-------------|----------|-----------|--------|
|
||||
| REQ-F-011 | AI job aggregation pipeline (3M+ jobs from 120K companies) | high | v0.4+ | deferred |
|
||||
| REQ-F-012 | AI-powered tagging (skills, categories, seniority) | high | v0.4+ | deferred |
|
||||
| REQ-F-013 | Semantic vector search for role matching | high | v0.4+ | deferred |
|
||||
| REQ-F-014 | AI company enrichment (logos, descriptions, social links) | medium | v0.4+ | deferred |
|
||||
| REQ-F-015 | AI resume parsing (profile auto-fill) | medium | v0.4+ | deferred |
|
||||
| REQ-F-016 | SEO-optimized programmatic pages | medium | v0.4+ | deferred |
|
||||
| REQ-F-011 | AI job aggregation pipeline (3M+ jobs from 120K companies) | high | v0.6 | activated → REQ-6-001 (pilot-scaled; adapter contract preserves full scale) |
|
||||
| REQ-F-012 | AI-powered tagging (skills, categories, seniority) | high | v0.6 | activated → REQ-6-002 |
|
||||
| REQ-F-013 | Semantic vector search for role matching | high | v0.6 | activated → REQ-6-003 |
|
||||
| REQ-F-014 | AI company enrichment (logos, descriptions, social links) | medium | v0.6 | activated → REQ-6-004 |
|
||||
| REQ-F-015 | AI resume parsing (profile auto-fill) | medium | v0.6 | activated → REQ-6-005 |
|
||||
| REQ-F-016 | SEO-optimized programmatic pages | medium | v0.6 | activated → REQ-6-006 |
|
||||
|
||||
### Platform Infrastructure
|
||||
|
||||
| ID | Description | Priority | Milestone | Status |
|
||||
|----|-------------|----------|-----------|--------|
|
||||
| REQ-F-017 | Identity verification and age-gating (16+/18+) — real KYC backend | high | v0.4+ | deferred (deferred from v0.3 per founder directive) |
|
||||
| REQ-F-017 | Identity verification and age-gating (16+/18+) — real KYC backend | high | v0.5 | activated → complete (REQ-5-003/004) |
|
||||
| REQ-F-018 | Payment processing and subscription management | high | v0.3+ | deferred |
|
||||
| REQ-F-019 | Human tutor marketplace (third-party courses) | medium | v0.4+ | deferred |
|
||||
| REQ-F-020 | CIRR-style placement tracking and audit | medium | v0.5+ | deferred |
|
||||
@@ -170,18 +244,51 @@
|
||||
|
||||
## Traceability Matrix
|
||||
|
||||
### v0.3 (current milestone)
|
||||
### v0.6 (active)
|
||||
|
||||
| Requirement | Phase | Status |
|
||||
|-------------|-------|--------|
|
||||
| REQ-6-001 | 1 | pending |
|
||||
| REQ-6-002 | 2 | pending |
|
||||
| REQ-6-003 | 3 | pending |
|
||||
| REQ-6-004 | 4 | pending |
|
||||
| REQ-6-005 | 4 | pending |
|
||||
| REQ-6-006 | 5 | pending |
|
||||
|
||||
### v0.5 (complete)
|
||||
|
||||
| Requirement | Phase | Status |
|
||||
|-------------|-------|--------|
|
||||
| REQ-5-007 | 1 | complete |
|
||||
| REQ-5-001 | 2 | complete |
|
||||
| REQ-5-002 | 2 | complete |
|
||||
| REQ-5-003 | 3 | complete |
|
||||
| REQ-5-004 | 3 | complete |
|
||||
| REQ-5-005 | 4 | complete |
|
||||
| REQ-5-006 | 4 | complete |
|
||||
|
||||
### v0.4 (complete)
|
||||
|
||||
| Requirement | Phase | Status |
|
||||
|-------------|-------|--------|
|
||||
| REQ-4-001 | 1 | complete |
|
||||
| REQ-4-002 | 1 | complete |
|
||||
| REQ-4-003 | 2 | complete |
|
||||
| REQ-4-004 | 2 | complete |
|
||||
| REQ-4-005 | 3 | complete |
|
||||
|
||||
### v0.3 (complete)
|
||||
|
||||
| Requirement | Phase | Status |
|
||||
|-------------|-------|--------|
|
||||
| REQ-3-001 | 1 | complete |
|
||||
| REQ-3-002 | 1 | complete |
|
||||
| REQ-3-003 | 2 | pending |
|
||||
| REQ-3-004 | 3 | pending |
|
||||
| REQ-3-005 | 4 | pending |
|
||||
| REQ-3-006 | 5 | pending |
|
||||
| REQ-3-007 | 6 | pending |
|
||||
| REQ-3-008 | 6 | pending |
|
||||
| REQ-3-003 | 2 | complete |
|
||||
| REQ-3-004 | 3 | complete |
|
||||
| REQ-3-005 | 4 | complete |
|
||||
| REQ-3-006 | 5 | complete |
|
||||
| REQ-3-007 | 6 | complete |
|
||||
| REQ-3-008 | 6 | complete |
|
||||
|
||||
### v0.2 (complete)
|
||||
|
||||
|
||||
+88
-116
@@ -2,30 +2,56 @@
|
||||
|
||||
## Overview
|
||||
|
||||
**Milestone v0.3** — Credential Engines: Replace v0.2's mock engine inputs with real credential engines. Build the sandbox fabric (sandboxed IDE / design tool / simulation), the live in-environment build-telemetry pipeline, the process-trace grading engine, per-learner variant task generation, and the oral/voice defense with AI examiner. Lab/Assessor/Proctor agents move from mock inputs to real engine inputs.
|
||||
**Milestone v0.6 — Marketplace Engine (ACTIVE, started 2026-09-21).** The real marketplace backend behind the existing surfaces: job posting backend + aggregation adapter (REQ-6-001), AI tagging (REQ-6-002), semantic vector search + match scoring (REQ-6-003), company enrichment + resume parsing (REQ-6-004/005), SEO programmatic pages (REQ-6-006). Scope decision D-047 (auto-decided at SPECIFY, full autonomy). Tags on the v0.5.x line (v0.5.0 phase 0 → v0.5.N final = milestone release); branch `milestone/v0.6-marketplace-engine`.
|
||||
|
||||
**Deferred per founder directive:** REQ-F-017 identity verification + age-gating (real KYC backend) is deferred beyond v0.3. Age-gating remains the v0.1 visual flow mockup.
|
||||
**Hotfix v0.4.7 (2026-09-21) — pre-public hardening.** The 4 P1 flags from the v0.5 final review closed: HMAC ingest tokens on the WS grading input (LAN poisoning closed), nonce+counter defense-id mint (same-ms collision closed), browser-mode client SR/TTS per the descriptor contract (gap closed), 8 browser-voice module pins. Deployed via `nextcraft deploy --service`; site 200, service enabled.
|
||||
|
||||
**Hotfix v0.4.6 (2026-09-21) — site-down + deploy experience.** The domain root (nextcraft.coreci.dev) served the API's JSON 404 for three releases: the deployed service ran mount-less (web export never built on the host; dev -d auto-wiring silently skipped; nothing detected it). Fixed the class: `nextcraft deploy` (one-command clone/update → install → venv → export build → restart → verify), `deploy --service` (generated systemd unit, reboot-persistent), toolchain discovery (nvm/PNPM_HOME/corepack, incl. split installs — the host's exact layout), loud mount-less boot warning, verify live mount probe. Deployed live on the host via the new command; site 200, service enabled.
|
||||
|
||||
**Hotfix v0.3.5/v0.3.6 (2026-09-13) — fresh-box + single-port unattended deploy.**
|
||||
|
||||
**Milestone v0.5 — COMPLETE (shipped as v0.4.5, 2026-09-13).** Real Server Voice, KYC/Identity, Sandbox Environments, Seq-Lease: the four seams D-016 deferred out of v0.4. Server STT/TTS for voice defense (`openai-audio` VoiceProvider), identity verification + backend-enforced age-gating (REQ-F-017), design/simulation sandbox environments (REQ-F-021 remainder), and the exec-telemetry seq-lease/replay-margin fix.
|
||||
|
||||
**Milestone v0.4 — COMPLETE (shipped as v0.3.4, 2026-09-13; hotfixes v0.3.5 fresh-box, v0.3.6 single-port unattended deploy).** Distribution & Bootstrap CLI: `nextcraft` CLI (doctor/bootstrap/verify/dev) shipped as a linux x64 SEA binary, one-liner install script with checksum + version integrity gates, and binaries published on **every ongoing release** (v0.3.2 onward). v0.3.6 adds the single-port same-origin deploy (static export served by the ai-service on :8420) and unattended ops (`dev -d`/`stop`/`log`), with runtime state moved to `~/.nextcraft/`.
|
||||
|
||||
**Milestone v0.3** — Credential Engines: complete, shipped as v0.2.8 (2026-09-12). Real sandbox fabric, live build telemetry, process-trace grading, per-learner variants, oral defense, real learner surfaces.
|
||||
|
||||
**Deferred per founder directive (D-016):** REQ-F-017 identity verification + age-gating (real KYC backend), real server STT/TTS, design/simulation sandbox environments, and the exec-telemetry seq-lease are deferred to v0.5. Age-gating remains the v0.1 visual flow mockup.
|
||||
|
||||
**Prior milestone:** v0.2 (ai-tutor-architecture) — complete, shipped as v0.2.0, six tutor agents live over mock engine inputs (D-015).
|
||||
|
||||
**Milestone type:** Feature (new credential-engine services + real agent inputs)
|
||||
**Tag line:** v0.2.x (patches on the v0.2 line; milestone release as the final v0.2.x patch)
|
||||
**Branch:** milestone/v0.3-credential-engines
|
||||
**Milestone type:** Feature (voice provider + identity backend + new env types + ingest protocol fix)
|
||||
**Tag line:** v0.4.x (patches on the v0.4 line; milestone release as the final v0.4.x patch)
|
||||
**Branch:** milestone/v0.5-real-voice-identity-envs
|
||||
|
||||
---
|
||||
|
||||
## Phase List
|
||||
|
||||
### v0.6 (active)
|
||||
|
||||
| # | Name | Status | Depends On | Requirements | Success Criteria |
|
||||
|---|------|--------|------------|--------------|------------------|
|
||||
| 0 | Pre-execution | in-progress | — | — | Specification, clarify, research, plan complete; .ciagent/ files updated for v0.3 |
|
||||
| 1 | Sandbox fabric | complete | 0 | REQ-3-001, REQ-3-002 | Isolated per-learner sandbox environments provisioned (IDE / design / simulation); lifecycle API (create/destroy/snapshot); resource limits enforced; no cross-tenant access |
|
||||
| 2 | Live build telemetry | pending | 1 | REQ-3-003 | In-environment capture of process events (commands, file diffs, run/test results, keystroke-level activity) streamed to ai-service; reliable transport; per-learner trace persistence |
|
||||
| 3 | Process-trace grading engine | pending | 2 | REQ-3-004 | Grades artifacts from their full process traces (not just final output); emits structured rubric-aligned scores; feeds Assessor real inputs |
|
||||
| 4 | Variant task generation | pending | 1 | REQ-3-005 | Per-learner task variants generated so no two learners receive identical prompts; variant seed recorded for grading fairness |
|
||||
| 5 | Oral / voice defense | pending | 3 | REQ-3-006 | AI examiner conducts spoken defense of submitted work; STT → dialogue → TTS; transcript + integrity signals captured; feeds Proctor/Mentor |
|
||||
| 6 | Agent re-grounding + learner surface integration | pending | 2,3,4,5 | REQ-3-007, REQ-3-008 | Lab/Assessor/Proctor consume real engine inputs; v0.1 sandbox + assessment mockups wired to real engines (in-browser build/run, live telemetry, live defense) |
|
||||
| 7 | Final review + ship | pending | 6 | — | Code review clean; audit passes; milestone tagged (v0.2.x final patch); release created on Gitea |
|
||||
| 0 | Pre-execution | complete | — | — | Specification, clarify, research, plan, grill, MVP/UX check complete; .ciagent/ files updated for v0.6 (shipped as v0.5.0) |
|
||||
| 1 | Postings backend + aggregation adapter | complete (v0.5.1) | 0 | REQ-6-001 | Postings store + lifecycle + `/v1/marketplace/*` routes live; job board/detail/employer surfaces read real data; 18+ gates composed per D-043; seeded ingestion adapter proven |
|
||||
| 2 | AI tagging | in-progress | 1 | REQ-6-002 | Posting text → skills/seniority/category tags via LLM structured output (mock-first); tag facets drive search filters; tags persisted |
|
||||
| 3 | Semantic vector search | pending | 2 | REQ-6-003 | Embedding provider protocol (mock-first) + SQLite-local vectors; competency↔posting similarity ranking; match scores surface on job cards + candidate matches |
|
||||
| 4 | Enrichment + resume parsing | pending | 3 | REQ-6-004, REQ-6-005 | Employer enrichment from posting data with provenance; resume upload → profile auto-fill with learner review gate; bounded uploads |
|
||||
| 5 | SEO programmatic pages | pending | 4 | REQ-6-006 | Metadata + JSON-LD + sitemap/robots on job/stack/landing pages; static-export compatible (single-port deploy unchanged) |
|
||||
| 6 | Final review + ship | pending | 1-5 | — | Review clean; audit passes; milestone tagged (final v0.5.x patch); release with binary assets on Gitea; milestone merged to main |
|
||||
|
||||
---
|
||||
|
||||
## Phase List (v0.5 — complete)
|
||||
|
||||
| # | Name | Status | Depends On | Requirements | Success Criteria |
|
||||
|---|------|--------|------------|--------------|------------------|
|
||||
| 0 | Pre-execution | complete | — | — | Specification, clarify, research, plan, grill complete; .ciagent/ files updated for v0.5 |
|
||||
| 1 | Seq-lease + replay margin | complete | 0 | REQ-5-007 | Ingest acks received seqs (`seq_ack` frame); capture agent trims spool to ack on reconnect (bounded margin); real-server mid-burst regression test closes the P07-documented gap |
|
||||
| 2 | Real server voice | complete | 0 | REQ-5-001, REQ-5-002 | `openai-audio` provider passes STT/TTS contract tests (MockTransport); voice defense runs server-side end-to-end when keys exist; mock/browser paths unchanged; suite green |
|
||||
| 3 | Identity + age-gating | complete | 0 | REQ-5-003, REQ-5-004 | Identity protocol + mock backend + verification flow API; gated routes enforce 16+/18+ (allowlist → identity → rate caps); PII hygiene pinned by caplog test |
|
||||
| 4 | Design/sim environments | complete | 0 | REQ-5-005, REQ-5-006 | Template-layer env registry (build/design/simulation); per-kind starter contents + exec policy; test_command surfaced; grading digest kind-agnostic (pinned) |
|
||||
| 5 | Final review + ship | complete | 1-4 | — | Code review clean; audit passes; milestone tagged (final v0.4.x patch); release with binary assets on Gitea |
|
||||
|
||||
---
|
||||
|
||||
@@ -33,147 +59,93 @@
|
||||
|
||||
### Phase 0: Pre-execution
|
||||
|
||||
**Goal:** Establish v0.3 specification, clarify ambiguities, research credential-engine architecture (sandbox isolation, telemetry transport, trace grading, variant generation, voice IO), create detailed plans.
|
||||
**Goal:** Lock the v0.5 specification (four deferred seams), clarify ambiguities, research the audio endpoint contract + KYC provider landscape + env-type design + the seq-lease protocol, plan waves, grill adversarially.
|
||||
|
||||
**Stages:** SPECIFY → CLARIFY → RESEARCH → PLAN → GRILL → MVP/UX CHECK → SHIP
|
||||
|
||||
**Deliverables:**
|
||||
- Updated .ciagent/config.json, PROJECT.md, REQUIREMENTS.md, ROADMAP.md, ARCHITECTURE.md, PERSONAS.md, PLAN.md
|
||||
|
||||
**Success criteria:** All .ciagent/ files updated for v0.3; phase 0 shipped as first v0.2.x patch.
|
||||
**Success criteria:** All .ciagent/ files updated for v0.5; phase 0 shipped as v0.4.0.
|
||||
|
||||
---
|
||||
### Phase 1: Seq-Lease + Replay Margin (REQ-5-007)
|
||||
|
||||
### Phase 1: Sandbox Fabric
|
||||
|
||||
**Goal:** Provision and manage isolated per-learner execution environments.
|
||||
|
||||
**Requirements:** REQ-3-001, REQ-3-002
|
||||
**Goal:** Close the reconnect-replay ACK gap from the v0.3 P6 lesson.
|
||||
|
||||
**Key deliverables:**
|
||||
- Sandbox orchestrator service: create/list/destroy/snapshot sandbox instances (IDE, design tool, simulation)
|
||||
- Isolation boundary: per-learner containerization or VM-grade isolation; no cross-tenant filesystem/network access
|
||||
- Resource limits: CPU/memory/disk/time quotas per sandbox
|
||||
- Sandbox lifecycle API consumed by ai-service and the web learner surface
|
||||
- Ingest WS: ack frames carrying highest-contiguous received seq per (learner, task); capture agent tracks ack and resumes spool replay from acked position (bounded replay margin)
|
||||
- Real-server regression test: mid-burst kill → reconnect → no loss/dup exactly-once within margin (deterministic — waits for server-side observation like the P07 fix)
|
||||
|
||||
**Success criteria:**
|
||||
- A sandbox can be created, written to, snapshotted, and destroyed via API
|
||||
- Isolation verified: a sandbox cannot read another learner's data
|
||||
- Resource limits enforced and observable — enforcement mechanism: rlimits (memory/CPU) + wall-clock reaper + workdir-size sweep; per-sandbox pids and hard-disk-quota are accepted v0.3 gaps (no cgroup delegation/sudo on this box, G-1)
|
||||
**Success criteria:** regression test green on repeated runs; margin documented; flood/gap semantics (G-3/G-4) unchanged.
|
||||
|
||||
---
|
||||
### Phase 2: Real Server Voice (REQ-5-001, REQ-5-002)
|
||||
|
||||
### Phase 2: Live Build Telemetry
|
||||
|
||||
**Goal:** Capture in-environment process events and stream them to ai-service reliably.
|
||||
|
||||
**Requirements:** REQ-3-003
|
||||
**Goal:** The `openai-audio` VoiceProvider against OpenAI-compatible STT/TTS endpoints; voice defense real path end-to-end.
|
||||
|
||||
**Key deliverables:**
|
||||
- Telemetry capture agent (in-sandbox): commands, file diffs, run/test results, keystroke-level/activity events
|
||||
- Telemetry transport: durable, ordered, resumable stream to ai-service ingestion endpoint
|
||||
- Trace persistence: per-learner, per-task process traces stored for grading and proctoring
|
||||
- Transport hardening: retries, backpressure, exactly-once-or-at-least-once semantics documented
|
||||
- `ai_service/voice/openai_audio.py` — provider per D-030 protocol (STT: multipart upload → transcript; TTS: text → audio bytes); httpx via the app pool; timeouts; error mapping
|
||||
- `AI_VOICE_BASE_URL`/`AI_VOICE_API_KEY`/`AI_VOICE_MODEL` settings resolution (env-only, never committed); `AI_VOICE_PROVIDER=openai-audio|browser|mock` selection in factory
|
||||
- Defense answer route: real STT path (audio upload transcribed server-side when provider=openai-audio; browser/mock paths unchanged); examiner TTS question audio
|
||||
- Tests: MockTransport byte-contract tests (multipart shape, response parse, error modes); defense-flow tests stay mock-only (never call the cloud)
|
||||
|
||||
**Success criteria:**
|
||||
- Sandbox activity produces a complete ordered process trace in ai-service
|
||||
- Stream survives transient network failure without trace loss
|
||||
- Trace retrievable by learner+task ID for grading
|
||||
**Success criteria:** provider contract pinned by tests; voice defense works against a scripted audio endpoint; `pnpm ai:test` green; manual cloud probe documented.
|
||||
|
||||
---
|
||||
### Phase 3: Identity + Age-Gating (REQ-5-003, REQ-5-004)
|
||||
|
||||
### Phase 3: Process-Trace Grading Engine
|
||||
|
||||
**Goal:** Grade learner artifacts from their full process traces.
|
||||
|
||||
**Requirements:** REQ-3-004
|
||||
**Goal:** Real identity verification backend behind a provider protocol; backend-enforced age gates.
|
||||
|
||||
**Key deliverables:**
|
||||
- Trace analyzer: reconstructs build/decision timeline from a process trace
|
||||
- Grading engine: rubric-aligned scoring over the trace (process quality, not just final artifact)
|
||||
- Structured score output consumable by the Assessor agent
|
||||
- Calibration against v0.2 mock corpora to validate grading dimensions
|
||||
- `ai_service/identity/` — IdentityProvider protocol (submit/poll/verify), mock provider (deterministic), store (SQLite, D-027 family), API router (`/v1/identity/*`)
|
||||
- Age-gate enforcement: school 16+ (enrollment), marketplace 18+ verified (gated routes reject under-age/unverified with 403) — dependency-injected gate, allowlist (G-5) retained as pilot guard
|
||||
- PII hygiene: document refs stored, never raw docs in logs; secrets-hygiene checklist extended
|
||||
|
||||
**Success criteria:**
|
||||
- Engine emits structured rubric-aligned scores from a real process trace
|
||||
- Scores distinguish process quality (e.g., iterative debugging vs. paste-and-run)
|
||||
- Output feeds Assessor; replaces pre-baked artifact corpus inputs
|
||||
**Success criteria:** verification flow API green under mock; gated routes enforce ages in tests; zero PII in captured logs (pinned by test).
|
||||
|
||||
---
|
||||
### Phase 4: Design/Sim Environments (REQ-5-005, REQ-5-006)
|
||||
|
||||
### Phase 4: Variant Task Generation
|
||||
|
||||
**Goal:** Generate per-learner task variants so no two learners receive identical prompts.
|
||||
|
||||
**Requirements:** REQ-3-005
|
||||
**Goal:** Environment kinds beyond the coding IDE on the existing namespace fabric.
|
||||
|
||||
**Key deliverables:**
|
||||
- Variant generator: parameterized task templates → unique per-learner instances
|
||||
- Variant seed registry: record variant parameters for grading fairness and proctoring
|
||||
- Difficulty normalization: variants calibrated to equivalent difficulty
|
||||
- Environment-type registry in the sandbox fabric: `build` (existing), `design` (canvas/file artifacts), `simulation` (run/benchmark harness) — per-type starter contents + allowed exec commands, one lifecycle, one telemetry path
|
||||
- Learner surface + engine-client accept environment kind; telemetry events carry kind; grading digest stays kind-agnostic (D-028 unchanged)
|
||||
|
||||
**Success criteria:**
|
||||
- Two learners requesting the same competency receive distinct task variants
|
||||
- Variant parameters persisted and auditable
|
||||
- Grading engine scores variants equitably
|
||||
**Success criteria:** all three kinds provisionable + provable isolation; per-kind starter files land; kind flows through telemetry to the digest.
|
||||
|
||||
### Phase 5: Final Review + Ship
|
||||
|
||||
**Goal:** Code review, audit, milestone release with binary assets.
|
||||
|
||||
**Success criteria:** review P0s fixed in-phase; audit reconstruction clean; final v0.4.x patch tagged; Gitea release with `nextcraft-linux-x64` + `.sha256`; milestone merged to main; branches deleted.
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Oral / Voice Defense
|
||||
## Hotfixes (v0.4.x line, post-v0.5)
|
||||
|
||||
**Goal:** AI examiner conducts a spoken defense of the learner's submitted work.
|
||||
| Tag | Date | Summary |
|
||||
|-----|------|---------|
|
||||
| v0.4.7 | 2026-09-21 | Pre-public hardening (the 4 P1 flags from the v0.5 final review): HMAC ingest tokens on the WS grading input (LAN trace-poisoning closed, per-boot secret, manager mints / endpoint verifies), nonce+counter defense-id mint (same-ms PK collision closed), browser-mode client SR/TTS implemented per the descriptor contract (the v0.5 gap), 8 browser-voice module pins. 488 backend (+10), web 22/22 (+8), CLI 61/61. Deployed via `nextcraft deploy --service`; site 200, service enabled. |
|
||||
| v0.4.6 | 2026-09-21 | Site-down hotfix + one-command deploy: `nextcraft deploy [--service]` (clone/update → install → venv → export build → restart → verify), systemd unit generation (reboot persistence), toolchain discovery (nvm/PNPM_HOME/corepack + split installs), loud mount-less boot, verify live mount probe, subcommand flag dispatch. Deployed live via the command itself; nextcraft.coreci.dev → 200 HTML. |
|
||||
|
||||
**Requirements:** REQ-3-006
|
||||
## v0.6 (Active — Marketplace Engine)
|
||||
|
||||
**Key deliverables:**
|
||||
- Voice pipeline: STT → defense dialogue (LLM examiner) → TTS
|
||||
- Examiner agent: probes understanding, challenges process choices from the trace
|
||||
- Transcript + integrity signals captured for Proctor/Mentor
|
||||
- Latency budget: defense feels conversational (bounded turn latency)
|
||||
Real backend behind the marketplace surfaces: postings store + aggregation adapter + marketplace API (REQ-6-001), AI tagging (REQ-6-002), embedding-backed semantic search + match scores (REQ-6-003), company enrichment + resume parsing (REQ-6-004/005), SEO programmatic pages (REQ-6-006). 7 phases (P0–P6). Scope D-047 (auto-decided at SPECIFY). Tags v0.5.0+ on the v0.5.x line; final patch = milestone release.
|
||||
|
||||
**Success criteria:**
|
||||
- A spoken defense runs end-to-end (speak → examiner question → learner response → verdict)
|
||||
- Transcript + integrity signals persisted and consumable by Proctor
|
||||
- Turn latency within the documented budget
|
||||
## v0.5 (Complete — Shipped as v0.4.5)
|
||||
|
||||
---
|
||||
Real Server Voice, KYC/Identity, Sandbox Environments, Seq-Lease: real server STT/TTS (`openai-audio` VoiceProvider with boot-safe fallback), the identity module (5th D-027 store, provider protocol + mock, D-043 age-gate composition on variants/sandboxes/defense/marketplace), design/simulation environment kinds at the template layer with per-kind exec policy, and the seq-ack protocol closing the P07 replay-margin gap. 6 phases (P0–P5). All 7 requirements (REQ-5-001..007) complete. Tags v0.4.0–v0.4.4 per phase, milestone release v0.4.5.
|
||||
|
||||
### Phase 6: Agent Re-grounding + Learner Surface Integration
|
||||
## v0.4 (Complete — Shipped as v0.3.4)
|
||||
|
||||
**Goal:** Move Lab/Assessor/Proctor to real engine inputs; wire learner surface to the real engines.
|
||||
Distribution & Bootstrap CLI: `nextcraft` CLI (doctor/bootstrap/verify/dev) as a self-contained linux x64 SEA binary, one-liner install with sha256 + version integrity gates, release-asset pipeline attaching binaries to every ongoing release (v0.3.2 onward), install/quickstart docs backed by a fresh-clone E2E test. 5 phases (P0–P4). All 5 requirements (REQ-4-001..005) complete. Tags v0.3.0–v0.3.3 per phase, milestone release v0.3.4.
|
||||
|
||||
**Requirements:** REQ-3-007, REQ-3-008
|
||||
## v0.3 (Complete — Shipped as v0.2.8)
|
||||
|
||||
**Key deliverables:**
|
||||
- Lab agent consumes live sandbox telemetry (replaces v0.2 mock telemetry)
|
||||
- Assessor agent consumes grading-engine output (replaces pre-baked artifacts)
|
||||
- Proctor consumes telemetry + defense integrity signals (replaces mock telemetry)
|
||||
- Learner sandbox mockup → real in-browser build/run; assessment mockup → live defense + live grading
|
||||
|
||||
**Success criteria:**
|
||||
- Lab/Assessor/Proctor operate on real inputs with no mock fallback in the learner path
|
||||
- Learner can build in-browser and see live telemetry + live feedback
|
||||
- Assessment surface runs a live defense and shows live grading
|
||||
- `pnpm build` and `pnpm typecheck` pass
|
||||
|
||||
---
|
||||
|
||||
### Phase 7: Final Review + Ship
|
||||
|
||||
**Goal:** Code review, audit, milestone release.
|
||||
|
||||
**Key deliverables:**
|
||||
- Multi-persona code review (correctness, testing, security, performance, maintainability)
|
||||
- Project health audit (reconstruction test, .ciagent/ file discipline, branch hygiene, commit discipline)
|
||||
- Milestone ship: merge milestone → main, tag final v0.2.x patch, create Gitea release
|
||||
|
||||
**Success criteria:**
|
||||
- Code review: P0 fixes applied, P1+ documented
|
||||
- Audit: all checks pass, project state reconstructable from git log
|
||||
- Ship: milestone tagged, branch merged to main, Gitea release created — release note states identity/age-gating (KYC) is deferred and age-gating remains a visual mockup
|
||||
- All v0.3 requirements marked complete
|
||||
|
||||
---
|
||||
Credential Engines: real sandbox fabric (Linux namespaces), live build telemetry
|
||||
(at-least-once/exactly-once), process-trace grading (G-4 gated), seeded per-learner
|
||||
variants (fairness anchors wired to grading), oral defense with integrity signals
|
||||
(mock-first voice, browser fallback), and real learner build/defense/grading surfaces.
|
||||
8 phases. All 8 requirements (REQ-3-001..008) complete. Tags v0.2.1–v0.2.7 per phase,
|
||||
milestone release v0.2.8.
|
||||
|
||||
## v0.2 (Complete — Shipped as v0.2.0)
|
||||
|
||||
|
||||
@@ -46,9 +46,9 @@
|
||||
"projects": [],
|
||||
"active_project": null,
|
||||
"milestone": {
|
||||
"version": "v0.3",
|
||||
"name": "credential-engines",
|
||||
"version": "v0.6",
|
||||
"name": "marketplace-engine",
|
||||
"type": "feature",
|
||||
"branch": "milestone/v0.3-credential-engines"
|
||||
"branch": "milestone/v0.6-marketplace-engine"
|
||||
}
|
||||
}
|
||||
@@ -20,6 +20,8 @@ dist/
|
||||
.next/
|
||||
.turbo/
|
||||
*.tsbuildinfo
|
||||
# Next.js static export (v0.3.6 build output, served by the ai-service)
|
||||
apps/web/out/
|
||||
|
||||
# Storybook
|
||||
storybook-static/
|
||||
@@ -55,3 +57,6 @@ apps/ai-service/ai_service/data/
|
||||
apps/ai-service/**/sandboxes/
|
||||
*.db
|
||||
*.db-journal
|
||||
|
||||
# in-sandbox capture agent runtime spool
|
||||
.nc-agent/
|
||||
|
||||
@@ -2,8 +2,92 @@
|
||||
|
||||
AI-native outcome school + marketplace — graduates prove what they can build, not what they can write.
|
||||
|
||||
## Quickstart
|
||||
|
||||
One-liner install (linux x64):
|
||||
|
||||
```sh
|
||||
curl -fsSL https://git.coreci.dev/coreci/nextcraft/raw/main/scripts/install.sh | sh
|
||||
```
|
||||
|
||||
That downloads the latest release's `nextcraft` CLI binary, verifies its sha256 checksum, and installs it to `~/.local/bin` (PATH hint printed if needed). Every release ships fresh binaries — re-run the one-liner to upgrade.
|
||||
|
||||
### Server (deploy the whole stack, one command)
|
||||
|
||||
```sh
|
||||
nextcraft deploy --service
|
||||
```
|
||||
|
||||
That clones (or updates) the repo, installs deps, builds the web UI, restarts the service, verifies the site is served, and installs a systemd unit so the stack survives reboots. Idempotent — same command for first install and upgrades. Toolchain discovery included: node/pnpm behind nvm/PNPM_HOME/corepack are found even when non-interactive shells can't see them.
|
||||
|
||||
### Development (from a clone of this repo)
|
||||
|
||||
```sh
|
||||
nextcraft doctor # check prerequisites: node >= 18, pnpm >= 8, python3 >= 3.11, git, unshare
|
||||
nextcraft bootstrap # pnpm install + ai-service venv + .env from template (idempotent)
|
||||
nextcraft verify # health check: venv imports, uvicorn, ports, env, web export, live mount probe
|
||||
nextcraft dev # run the ai-service dev server on :8420 (web dev server: pnpm dev)
|
||||
```
|
||||
|
||||
The E2E test (`apps/cli/tests/fresh-clone-e2e.test.ts`) proves this exact sequence on a fresh clone.
|
||||
|
||||
### No binary / non-linux?
|
||||
|
||||
The installer degrades to printed source instructions. Manual equivalent:
|
||||
|
||||
```sh
|
||||
git clone https://git.coreci.dev/coreci/nextcraft.git && cd nextcraft
|
||||
pnpm install
|
||||
bash apps/ai-service/scripts/bootstrap.sh
|
||||
cp apps/ai-service/.env.example apps/ai-service/.env
|
||||
pnpm ai:dev
|
||||
```
|
||||
|
||||
## CLI reference (`nextcraft`)
|
||||
|
||||
| Command | What it does | Exit codes |
|
||||
|---------|--------------|------------|
|
||||
| `doctor` | Checks prerequisites: node >= 18, pnpm >= 8 (discovered via PATH/nvm/PNPM_HOME/corepack), python3 >= 3.11, git, unshare (sandbox fabric). Every ✗ prints a fix hint. | 0 all pass, 1 any fail |
|
||||
| `bootstrap` | Sets up the monorepo from a fresh clone: (1) locates the repo root, (2) `pnpm install` (discovered toolchain), (3) ai-service venv via `apps/ai-service/scripts/bootstrap.sh`, (4) copies `.env.example` → `.env` if absent, (5) warns on missing optional keys. Idempotent — safe to re-run. | 0 ok, 1 step failed |
|
||||
| `verify` | Health check: ai-service venv + `import ai_service`, uvicorn importable, `.env` present (warn-only), `AI_PORT` state — free, or **busy + serving the web UI** (live mount probe: `/` must return HTML, not the API's JSON 404), web export present (`apps/web/out`), workspace `node_modules`. | 0 ok, 1 failures |
|
||||
| `deploy` | One-command deploy/upgrade: clone-or-update repo (`--dir`, `--ref`), `pnpm install`, ai-service venv, `NEXT_PUBLIC_AI_SERVICE_URL=self pnpm build`, restart, verify. Fails loudly if the site isn't served. | 0 ok, 1 step failed |
|
||||
| `deploy --service` | Same, plus generates + installs a systemd unit (real paths, `Restart=always`, enabled) so the stack survives reboots; systemd becomes the supervisor. | 0 ok, 1 step failed |
|
||||
| `dev` | Thin passthrough to `apps/ai-service/scripts/dev.sh` (exports secrets from `.ciagent/.env.secrets` if present, runs uvicorn on :8420). Ctrl+C stops it. The web dev server is separate: `pnpm dev`. | child's exit code |
|
||||
| `dev -d` / `dev --detach` | Same, but as an **unattended daemon**: detached, output appended to `~/.nextcraft/run/<clone>/dev.log`, pidfile beside it. Refuses to double-start (and refuses when the systemd service is active). Auto-serves the built web UI same-origin when `apps/web/out` exists. | 0 started, 1 already running |
|
||||
| `stop` | Stops the systemd service when it supervises the stack; otherwise stops the daemon started by `dev -d` (SIGTERM, SIGKILL after 5s) and cleans the pidfile. | 0 stopped/clean, 1 kill failed |
|
||||
| `log` | Tails the daemon log: last 50 lines by default, `-n N` for more, `-f`/`--follow` to stream. | 0 ok, 1 no log yet |
|
||||
| `--help` / `-h` | Usage for the CLI or any command. | 0 |
|
||||
| `--version` | Prints the version this binary was built as (matches the release tag). | 0 |
|
||||
|
||||
Exit-code contract: `0` success, `1` check/step failure (hint printed), `2` usage error.
|
||||
|
||||
### Remote server / single-port deployment
|
||||
|
||||
`nextcraft dev` binds the API on **0.0.0.0:8420** (and `pnpm dev` serves the web app on all interfaces), so the stack works from other machines out of the box:
|
||||
|
||||
- Browse `http://<your-host>:3000` — the web app targets `http://<your-host>:8420` automatically (derived from the browser's hostname).
|
||||
- CORS admits any origin (`AI_CORS_ORIGINS=*` in `apps/ai-service/.env`). This is safe **only** because credentials are never enabled; to restrict, set an explicit list: `AI_CORS_ORIGINS=http://<your-host>:3000`.
|
||||
- To revert to loopback-only: `AI_HOST=127.0.0.1` in `apps/ai-service/.env`.
|
||||
- Security note: this is an unauthenticated dev API reachable from any network the box exposes. Mitigations that still apply: per-learner sandbox caps + global rate caps + learner allowlist (G-5), telemetry flood control (traces marked `INCOMPLETE_FLOODED` are refused by the grader). Expose only on trusted networks until identity/KYC lands (v0.5).
|
||||
|
||||
**Production behind one port (v0.3.6):** when only one port is reachable (e.g. behind HAProxy), the one-command deploy builds the web app as a static export and lets the ai-service serve it same-origin on :8420 — UI and API on one port, no CORS, no mixed content:
|
||||
|
||||
```sh
|
||||
nextcraft deploy --service # install/upgrade + web export + restart + verify + systemd unit
|
||||
```
|
||||
|
||||
Durable state (SQLite DB, sandbox workdirs, daemon pid/log) lives in `~/.nextcraft/` — never inside the repo. Full recipe incl. HAProxy timeouts and a systemd unit: [deploy/README.md](deploy/README.md).
|
||||
|
||||
## Docs
|
||||
|
||||
- [apps/cli/README.md](apps/cli/README.md) — CLI internals: build, binary pipeline, troubleshooting
|
||||
- [.ciagent/PROJECT.md](.ciagent/PROJECT.md) — product spec and milestone history
|
||||
- [.ciagent/ARCHITECTURE.md](.ciagent/ARCHITECTURE.md) — system architecture
|
||||
|
||||
## Status
|
||||
|
||||
**Milestone v0.1** — UI/UX Prototype (high-fidelity interactive, all mock data)
|
||||
**Milestone v0.5** — Real Server Voice, KYC/Identity, Sandbox Environments, Seq-Lease (shipped as v0.4.5)
|
||||
|
||||
Prior: v0.3 Credential Engines (shipped v0.2.8) · v0.2 AI Tutor Architecture (v0.2.0) · v0.1 UI/UX Prototype (v0.1.0)
|
||||
|
||||
Initialized via CIAgent v0.7.0
|
||||
@@ -2,6 +2,13 @@
|
||||
# Real keys live in .ciagent/.env.secrets (gitignored) and are exported by scripts/dev.sh
|
||||
|
||||
AI_PORT=8420
|
||||
# Network mode (v0.3.5, D-038): dev server binds 0.0.0.0 so remote machines can
|
||||
# reach the stack. Set to 127.0.0.1 to revert to loopback-only.
|
||||
AI_HOST=0.0.0.0
|
||||
# CORS + WS-origin policy: '*' (default) admits any origin — safe because
|
||||
# credentials are never enabled. Restrict with a comma list, e.g.:
|
||||
# AI_CORS_ORIGINS=http://nextcraft-1:3000
|
||||
AI_CORS_ORIGINS=*
|
||||
AI_PROVIDER=ollama-cloud
|
||||
AI_MODEL=gemma4:31b
|
||||
AI_OLLAMA_CLOUD_BASE_URL=https://ollama.com/v1
|
||||
@@ -9,8 +16,10 @@ AI_OLLAMA_CLOUD_API_KEY=
|
||||
AI_LOCAL_BASE_URL=http://localhost:11434/v1
|
||||
AI_JSON_MODE=auto
|
||||
|
||||
# Sandbox fabric (v0.3)
|
||||
AI_SANDBOX_DIR=sandboxes
|
||||
# Sandbox fabric (v0.3). v0.3.6: durable state defaults to ~/.nextcraft/ —
|
||||
# OUTSIDE the repo (the old repo-relative 'sandboxes' default polluted the
|
||||
# git tree). Set an absolute path (~/ works) or a repo-relative one only for
|
||||
# throwaway dev clones.
|
||||
AI_SANDBOX_MAX_CONCURRENT=5
|
||||
AI_SANDBOX_TIMEOUT_S=900
|
||||
AI_SANDBOX_MAX_WORKDIR_MB=512
|
||||
@@ -23,5 +32,56 @@ AI_SANDBOX_MAX_PER_LEARNER=1
|
||||
# global creates per rolling 60s window (in-memory) → 429 when exceeded
|
||||
AI_SANDBOX_CREATES_PER_MIN=10
|
||||
|
||||
# Persistence (SQLite)
|
||||
AI_DB_PATH=ai_service/data/nextcraft.db
|
||||
# Persistence (SQLite). v0.3.6: default moved out of the repo to
|
||||
# ~/.nextcraft/data/nextcraft.db (~/ paths in env overrides are expanded).
|
||||
# Uncomment + edit ONLY to relocate:
|
||||
# AI_DB_PATH=~/.nextcraft/data/nextcraft.db
|
||||
|
||||
# Single-port deploy (v0.3.6): directory of the exported web app
|
||||
# (apps/web/out). When set, the UI is served from this same service on :8420
|
||||
# — build it with `NEXT_PUBLIC_AI_SERVICE_URL=self pnpm build`. The
|
||||
# `nextcraft dev` command auto-sets this when apps/web/out exists.
|
||||
# AI_WEB_STATIC_DIR=../../web/out
|
||||
# --- Identity (REQ-5-003, D-042) ---
|
||||
# 'mock' (default — deterministic, no vendor spend pre-pilot; verdicts carry
|
||||
# mock=True forever per A-304). A real KYC vendor drops in via the
|
||||
# IdentityProvider protocol without API changes.
|
||||
AI_IDENTITY_PROVIDER=mock
|
||||
# G-13: identity submit caps — one active pending per learner (409), and a
|
||||
# per-learner submit rate ceiling (429 over a rolling 60s window).
|
||||
AI_IDENTITY_SUBMITS_PER_MIN=3
|
||||
|
||||
# --- Voice (REQ-3-006 D-030; real server path REQ-5-001, D-040) ---
|
||||
# 'mock' (default; no key needed — tests/dev), 'browser' (client-native
|
||||
# SR/TTS), or 'openai-audio' (real server STT/TTS, live since v0.5).
|
||||
AI_VOICE_PROVIDER=mock
|
||||
# openai-audio requires BOTH (unconfigured → app boots, voice falls back to
|
||||
# mock with a loud log — G-11; the badge then honestly reports mock):
|
||||
# AI_VOICE_BASE_URL=https://your-audio-endpoint/v1
|
||||
# AI_VOICE_API_KEY=
|
||||
# Optional model/voice/format knobs (defaults shown):
|
||||
# AI_VOICE_STT_MODEL=whisper-1
|
||||
# AI_VOICE_TTS_MODEL=tts-1
|
||||
# AI_VOICE_TTS_VOICE=alloy
|
||||
# AI_VOICE_TTS_FORMAT=mp3 (enum: mp3 | wav | opus)
|
||||
# AI_VOICE_MAX_AUDIO_MB=10
|
||||
# Manual probe recipe (executable by anyone with the keys; never CI):
|
||||
# STT: curl -sS $AI_VOICE_BASE_URL/audio/transcriptions \
|
||||
# -H "Authorization: Bearer $AI_VOICE_API_KEY" \
|
||||
# -F file=@test/fixtures/answer.wav -F model=whisper-1 | jq -e '.text'
|
||||
# TTS: curl -sS $AI_VOICE_BASE_URL/audio/speech \
|
||||
# -H "Authorization: Bearer $AI_VOICE_API_KEY" \
|
||||
# -H 'Content-Type: application/json' \
|
||||
# -d '{"model":"tts-1","input":"Nextcraft","voice":"alloy"}' \
|
||||
# -o /tmp/probe.mp3 && file /tmp/probe.mp3 | grep -i audio
|
||||
|
||||
# --- Marketplace (REQ-6-001, D-057) ---
|
||||
# Founder/admin surface guard (/v1/marketplace/admin/*): send as the
|
||||
# 'X-Admin-Token' header OR 'Authorization: Bearer <token>'; compared
|
||||
# constant-time (G-108). UNSET = every admin route 403s (fail-closed,
|
||||
# never open) and the boot log warns loudly. Never commit real tokens.
|
||||
# AI_ADMIN_TOKEN=
|
||||
# P5 (D-056) placeholder: the deployed origin for sitemap.xml / robots.txt
|
||||
# absolute URLs (e.g. https://nextcraft.example). Unset → SEO phase
|
||||
# serves the empty static-page set with a boot warning (G-105).
|
||||
# AI_PUBLIC_BASE_URL=
|
||||
|
||||
@@ -194,6 +194,65 @@ Unprivileged user namespaces are on the box's kernel and need neither a
|
||||
daemon, nor suid helpers, nor network access — they are the only isolation
|
||||
primitive that works here, so that's what v0.3 uses.
|
||||
|
||||
## Telemetry delivery semantics (v0.3, REQ-3-003)
|
||||
|
||||
Delivery is **at-least-once**; storage is **exactly-once** — the two compose:
|
||||
|
||||
- The in-sandbox capture agent (stdlib-only, `scripts/sandbox-agent.py`)
|
||||
spools every event to a durable JSONL file (fsync per append) BEFORE any
|
||||
send attempt, so no event can be lost to a dead socket or a SIGKILL.
|
||||
- The WS ingest endpoint (`WS /v1/telemetry/ingest?learner_id&task_id`,
|
||||
D-026) dedups server-side on the `(learner_id, task_id, seq)` primary key:
|
||||
re-sends (reconnect flushes, replay margin) are collapsed, never upserted.
|
||||
- On disconnect the agent reconnects with exponential backoff and flushes
|
||||
the spool in `seq` order; a transient outage therefore loses nothing and
|
||||
stores each event exactly once (`tests/telemetry/test_durability.py`
|
||||
proves this end-to-end against a real namespace sandbox + live server).
|
||||
- Replay/read path: `GET /v1/telemetry/traces/{learner}/{task}` returns the
|
||||
complete ordered trace; `GET /v1/telemetry/gaps/{learner}/{task}` returns
|
||||
missing seqs for gap detection.
|
||||
- Flood boundary (G-3): a connection exceeding `AI_TELEMETRY_MAX_EVENTS_PER_TASK`
|
||||
(default 50,000) is closed with WS code 1008 and its trace is marked
|
||||
`INCOMPLETE_FLOODED` — a terminal integrity flag the grader refuses to
|
||||
grade. Silent event dropping is forbidden: it would corrupt grading input.
|
||||
|
||||
## Voice defense (v0.3, REQ-3-006)
|
||||
|
||||
Voice is **mock-first** (D-030): the defense pipeline is fully proven over
|
||||
the deterministic `MockVoiceProvider` + browser-native fallback — no task
|
||||
requires a real voice key. Real server STT/TTS (`OpenAIAudioProvider` over
|
||||
OpenAI-compatible `/audio/transcriptions` + `/audio/speech`) is **deferred
|
||||
to v0.4** together with KYC (GRILL CUT-1 / G-7): it could never be exercised
|
||||
in CI, so v0.3 ships the protocol seam instead of an unverifiable claim.
|
||||
|
||||
- `AI_VOICE_PROVIDER=mock` (default) — deterministic canned STT/TTS
|
||||
- `AI_VOICE_PROVIDER=browser` — the web client uses SpeechRecognition +
|
||||
speechSynthesis; the server keeps text-turn persistence
|
||||
- Conversational budget: a defense turn should complete in **< 4s**
|
||||
(`DEFENSE_TURN_BUDGET_MS` in `tests/voice/test_latency.py`). v0.3
|
||||
asserts instrumentation (stt_ms/llm_ms/tts_ms populated per turn); the
|
||||
wall-clock acceptance probe against a real voice endpoint is a v0.4
|
||||
criterion, run manually with `AI_VOICE_PROVIDER` set to the real
|
||||
provider and keys in `.ciagent/.env.secrets` (never in code/commits).
|
||||
|
||||
## End-to-end credential flow (v0.3, REQ-3-007/008)
|
||||
|
||||
`tests/api/test_e2e_credential_flow.py` runs the full pipeline against a REAL
|
||||
uvicorn server with REAL namespace sandboxes (mock LLM/voice per G-2
|
||||
precedent): variant -> telemetry-wired sandbox -> in-sandbox exec -> trace
|
||||
persistence -> process-trace grade (variant seed stamped) -> assessor
|
||||
coaching -> oral defense -> verdict + integrity signals -> proctor. It
|
||||
asserts no corpus fixture appears anywhere in the learner path.
|
||||
|
||||
Manual browser pass (documented, not automated): `pnpm ai:dev` + `pnpm dev`,
|
||||
then open `/build/stack-orchestration-c007` — variant statement + starter
|
||||
files load, edit a file, Run/Test execute in the sandbox with output in the
|
||||
read-only panel, the telemetry status pulses, Lab streams feedback from the
|
||||
live digest; then `/defend/stack-orchestration-c007` — Start Defense, typed
|
||||
answers (mic path needs permission), Finish, Grade My Work renders the real
|
||||
rubric bars. Navigating away destroys the sandbox
|
||||
(`curl localhost:8420/v1/sandboxes` shows the count drop).
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
|
||||
@@ -1,53 +1,34 @@
|
||||
"""AssessorAgent — rubric application to pre-baked artifacts (REQ-2-008).
|
||||
"""AssessorAgent — rubric coaching over REAL grading output (REQ-3-007).
|
||||
|
||||
Structured-output showcase: applies the 4-layer defense (D-020) to return
|
||||
a pydantic-validated rubric score. Mock engine inputs (corpus artifacts +
|
||||
transcripts); real process-trace grading is v0.3+.
|
||||
v0.3 re-grounding: the Assessor no longer invents scores from corpus
|
||||
artifacts — the process-trace grading engine (Phase 3) computes and
|
||||
persists the validated RubricScore. This agent now renders the STORED
|
||||
grade as rubric-anchored coaching: explains the criteria, cites strengths
|
||||
and gaps, and frames next steps. Corpus artifacts are retired from this
|
||||
path (corpus dormancy, Task 6-1-04).
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..corpus.artifacts import (
|
||||
ArtifactSubmission,
|
||||
AssessmentRubric,
|
||||
DefenseTranscript,
|
||||
render_rubric,
|
||||
render_transcript,
|
||||
)
|
||||
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||
from ..grading.store import GradeRecord
|
||||
from ..prompts.assessor import SYSTEM_PROMPT, render_context
|
||||
from .base import BaseAgent
|
||||
|
||||
|
||||
class CriterionScore(BaseModel):
|
||||
criterion_id: str
|
||||
name: str
|
||||
score: int = Field(ge=0, le=100)
|
||||
evidence: str
|
||||
class GradeCoaching(BaseModel):
|
||||
"""Rubric-anchored coaching rendered FROM the stored grade (not invented)."""
|
||||
|
||||
summary: str = Field(min_length=1)
|
||||
strengths: list[str] = Field(min_length=1, max_length=3)
|
||||
gaps: list[str] = Field(min_length=1, max_length=3)
|
||||
next_steps: list[str] = Field(min_length=1, max_length=3)
|
||||
|
||||
|
||||
class RubricScore(BaseModel):
|
||||
rubric_id: str
|
||||
artifact_id: str
|
||||
competency_id: str
|
||||
scores: list[CriterionScore]
|
||||
strengths: list[str] = Field(min_length=1, max_length=2)
|
||||
gaps: list[str] = Field(min_length=1, max_length=2)
|
||||
verdict: str # "mastered" | "developing" | "not_yet"
|
||||
|
||||
def weighted_total(self, rubric: AssessmentRubric) -> float:
|
||||
by_id = {c.criterion_id: c for c in rubric.criteria}
|
||||
total = 0.0
|
||||
for s in self.scores:
|
||||
total += s.score * by_id[s.criterion_id].weight
|
||||
return total
|
||||
|
||||
|
||||
RUBRIC_SCORE_SCHEMA_HINT = (
|
||||
'{"rubric_id": "<id>", "artifact_id": "<id>", "competency_id": "<id>", '
|
||||
'"scores": [{"criterion_id": "<id>", "name": "<name>", "score": <0-100>, '
|
||||
'"evidence": "<one sentence>"}], "strengths": ["<one sentence>"], '
|
||||
'"gaps": ["<one sentence>"], "verdict": "mastered"|"developing"|"not_yet"}'
|
||||
GRADE_COACHING_SCHEMA_HINT = (
|
||||
'{"summary": "<two sentences on the grade>", '
|
||||
'"strengths": ["<one sentence>"], "gaps": ["<one sentence>"], '
|
||||
'"next_steps": ["<one sentence>"]}'
|
||||
)
|
||||
|
||||
|
||||
@@ -58,35 +39,25 @@ class AssessorAgent(BaseAgent):
|
||||
ctx = learner_context or get_learner_context()
|
||||
return SYSTEM_PROMPT.format_map(render_context(ctx))
|
||||
|
||||
def build_evaluation_input(
|
||||
async def coach_grade(
|
||||
self,
|
||||
artifact: ArtifactSubmission,
|
||||
rubric: AssessmentRubric,
|
||||
transcript: DefenseTranscript | None,
|
||||
) -> str:
|
||||
parts = [
|
||||
f"ARTIFACT: {artifact.name} ({artifact.artifact_id})",
|
||||
f"Evidence excerpt: {artifact.evidence_excerpt}",
|
||||
"",
|
||||
render_rubric(rubric),
|
||||
]
|
||||
if transcript is not None:
|
||||
parts += ["", render_transcript(transcript)]
|
||||
return "\n".join(parts)
|
||||
|
||||
async def evaluate(
|
||||
self,
|
||||
artifact: ArtifactSubmission,
|
||||
rubric: AssessmentRubric,
|
||||
transcript: DefenseTranscript | None,
|
||||
grade: GradeRecord,
|
||||
learner_context: LearnerContext | None = None,
|
||||
) -> RubricScore:
|
||||
evaluation_input = self.build_evaluation_input(artifact, rubric, transcript)
|
||||
score: RubricScore = await self.structured_reply(
|
||||
) -> GradeCoaching:
|
||||
"""Render the STORED grade as coaching via the D-020 defense."""
|
||||
grade_json = {
|
||||
"verdict": grade.verdict,
|
||||
"scores": grade.scores,
|
||||
"digest": grade.digest,
|
||||
}
|
||||
coaching: GradeCoaching = await self.structured_reply(
|
||||
history=None,
|
||||
user_input=evaluation_input,
|
||||
user_input=(
|
||||
"The learner's process-trace grade (computed by the grading "
|
||||
f"engine) is:\n{grade_json!r}\nExplain it as coaching."
|
||||
),
|
||||
learner_context=learner_context,
|
||||
schema=RubricScore,
|
||||
schema_hint=RUBRIC_SCORE_SCHEMA_HINT,
|
||||
schema=GradeCoaching,
|
||||
schema_hint=GRADE_COACHING_SCHEMA_HINT,
|
||||
)
|
||||
return score
|
||||
return coaching
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
"""ExaminerAgent — the seventh agent: oral-defense examiner (REQ-3-006, A-109).
|
||||
|
||||
BOUNDARY DECISION (PERSONAS conflict rule, honored by construction): the
|
||||
examiner is a TEXT agent. It composes the LLM provider through BaseAgent and
|
||||
consumes defense transcript turns; it NEVER imports voice/ — STT/TTS belong
|
||||
to the API endpoints (they move audio bytes; the agent moves question text).
|
||||
Integrity signals (long pauses, off-scope cadence) are computed by the
|
||||
endpoint layer from turn metadata (latency_ms etc.), not by the agent.
|
||||
|
||||
Digest discipline (D-028 mirror): questions are grounded in the compact
|
||||
TraceDigest + variant statement — never the raw trace, never learner ids.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from ..grading.features import TraceDigest
|
||||
from ..llm.types import Message
|
||||
from ..prompts.examiner import SYSTEM_PROMPT, VERDICT_SCHEMA_HINT, render_digest_context
|
||||
from .base import BaseAgent
|
||||
|
||||
|
||||
class DefenseVerdict(BaseModel):
|
||||
"""D-20-validated final defense verdict (structured mode)."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
verdict: str = Field(pattern="^(mastered|developing|not_yet)$")
|
||||
understanding: str = Field(min_length=1)
|
||||
process_justification: str = Field(min_length=1)
|
||||
communication: str = Field(min_length=1)
|
||||
strengths: list[str] = Field(min_length=1, max_length=2)
|
||||
gaps: list[str] = Field(min_length=1, max_length=2)
|
||||
|
||||
|
||||
class ExaminerAgent(BaseAgent):
|
||||
"""Conducts the oral defense: next_question + final_verdict."""
|
||||
|
||||
name = "examiner"
|
||||
|
||||
def system_prompt(self, learner_context=None) -> str: # noqa: ANN001
|
||||
"""Examiner is context-free (digest-anonymous, D-028 mirror)."""
|
||||
return SYSTEM_PROMPT
|
||||
|
||||
def build_defense_messages(
|
||||
self,
|
||||
trace_digest: TraceDigest | None = None,
|
||||
variant_statement: str | None = None,
|
||||
history: list[Message] | None = None,
|
||||
) -> list[Message]:
|
||||
"""System + grounding + defense transcript (no learner id — D-028)."""
|
||||
digest_json = (
|
||||
trace_digest.model_dump_json() if trace_digest is not None else "{}"
|
||||
)
|
||||
messages: list[Message] = [
|
||||
Message(role="system", content=SYSTEM_PROMPT),
|
||||
Message(role="user", content=render_digest_context(digest_json, variant_statement)),
|
||||
Message(
|
||||
role="assistant",
|
||||
content="Understood. I will question the learner about this build session.",
|
||||
),
|
||||
]
|
||||
for m in history or []:
|
||||
messages.append(m)
|
||||
return messages
|
||||
|
||||
async def next_question(
|
||||
self,
|
||||
history: list[Message],
|
||||
trace_digest: TraceDigest | None = None,
|
||||
variant_statement: str | None = None,
|
||||
) -> str:
|
||||
"""One examiner question (streamed over SSE by the endpoints)."""
|
||||
messages = self.build_defense_messages(trace_digest, variant_statement, history)
|
||||
messages.append(
|
||||
Message(role="user", content="Ask the learner your next question now.")
|
||||
)
|
||||
reply = await self.provider.chat(messages, model=self.settings.model)
|
||||
return reply
|
||||
|
||||
async def final_verdict(
|
||||
self,
|
||||
history: list[Message],
|
||||
trace_digest: TraceDigest | None = None,
|
||||
variant_statement: str | None = None,
|
||||
) -> DefenseVerdict:
|
||||
"""Structured verdict via the D-020 4-layer defense."""
|
||||
from .structured import structured_completion # module-direct (G-4)
|
||||
|
||||
messages = self.build_defense_messages(trace_digest, variant_statement, history)
|
||||
messages.append(
|
||||
Message(
|
||||
role="user",
|
||||
content="The defense is finished. Return the final verdict JSON now.",
|
||||
)
|
||||
)
|
||||
return await structured_completion(
|
||||
self.provider,
|
||||
messages,
|
||||
model=self.settings.model,
|
||||
schema=DefenseVerdict,
|
||||
schema_hint=VERDICT_SCHEMA_HINT,
|
||||
)
|
||||
@@ -1,17 +1,18 @@
|
||||
"""LabAgent — in-flow feedback over simulated sandbox telemetry (REQ-2-007).
|
||||
"""LabAgent — in-flow feedback over LIVE sandbox telemetry (REQ-3-007).
|
||||
|
||||
Scenario-driven: consumes a LabTelemetryScenario from the corpus, renders
|
||||
the event timeline into the conversation, streams concrete feedback.
|
||||
No session chat — each request is one scenario read.
|
||||
v0.3 re-grounding: consumes a TraceDigest computed from the learner's real
|
||||
trace (grading/features.compute_digest over TraceStore events) — the v0.2
|
||||
corpus scenarios are retired from this path (corpus dormancy, Task 6-1-04).
|
||||
No session chat — each request is one live-trace read.
|
||||
"""
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from ..config import Settings
|
||||
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||
from ..corpus.telemetry import LabTelemetryScenario, summarize_scenario
|
||||
from ..grading.features import TraceDigest
|
||||
from ..llm.base import LLMProvider
|
||||
from ..prompts.lab import SYSTEM_PROMPT, render_context
|
||||
from ..prompts.lab import SYSTEM_PROMPT, render_context, render_digest_timeline
|
||||
from .base import BaseAgent
|
||||
|
||||
|
||||
@@ -27,10 +28,11 @@ class LabAgent(BaseAgent):
|
||||
|
||||
async def stream_feedback(
|
||||
self,
|
||||
scenario: LabTelemetryScenario,
|
||||
digest: TraceDigest | None,
|
||||
learner_context: LearnerContext | None = None,
|
||||
) -> AsyncIterator[str]:
|
||||
timeline = summarize_scenario(scenario)
|
||||
"""Feedback grounded in the learner's live trace digest."""
|
||||
timeline = render_digest_timeline(digest)
|
||||
async for token in self.stream_reply(
|
||||
history=None, user_input=timeline, learner_context=learner_context
|
||||
):
|
||||
|
||||
@@ -1,33 +1,41 @@
|
||||
"""ProctorAgent — integrity signals + coaching interventions (REQ-2-009).
|
||||
"""ProctorAgent — integrity signals + coaching over REAL inputs (REQ-3-007).
|
||||
|
||||
Consumes a ProctorScenario from the corpus, returns pydantic-validated
|
||||
signal classifications via structured_reply (4-layer defense).
|
||||
Mock engine inputs; real identity/attention signals are v0.3+.
|
||||
v0.3 re-grounding: consumes the learner's live trace digest (idle gaps,
|
||||
command cadence), the DefenseStore integrity signals (long pauses from the
|
||||
oral defense), and the variant seed cross-check — NOT v0.2 corpus
|
||||
scenarios. The proctor COACHES: it classifies signals supportively and
|
||||
recommends one intervention; it never punishes and never accuses.
|
||||
|
||||
Integrity inputs (computed server-side, passed in by the API layer):
|
||||
- trace digest: idle_gap_count/total, command_categories histogram,
|
||||
error/fix cycles, huge-burst indicators (edit_count vs test runs)
|
||||
- defense signals: long_pauses list from the finished defense (A-109)
|
||||
- variant: seed + params when the task is variant-derived (off-template
|
||||
work is a cross-check input, not an accusation)
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||
from ..corpus.telemetry import ProctorScenario, summarize_proctor_scenario
|
||||
from ..grading.features import TraceDigest
|
||||
from ..prompts.proctor import SYSTEM_PROMPT, render_context
|
||||
from .base import BaseAgent
|
||||
|
||||
|
||||
class IntegritySignal(BaseModel):
|
||||
signal_type: str # e.g. "context_switch" | "idle_gap" | "large_paste"
|
||||
signal_type: str # "idle_gap" | "long_pause" | "burst_edit" | "off_template"
|
||||
severity: str # "low" | "medium" | "high"
|
||||
note: str
|
||||
|
||||
|
||||
class ProctorAssessment(BaseModel):
|
||||
scenario_id: str
|
||||
signals: list[IntegritySignal] = Field(min_length=0)
|
||||
intervention: str # ONE supportive coaching recommendation
|
||||
summary: str
|
||||
|
||||
|
||||
PROCTOR_ASSESSMENT_SCHEMA_HINT = (
|
||||
'{"scenario_id": "<id>", "signals": [{"signal_type": "<type>", '
|
||||
'{"signals": [{"signal_type": "<type>", '
|
||||
'"severity": "low"|"medium"|"high", "note": "<one sentence>"}], '
|
||||
'"intervention": "<one supportive recommendation>", '
|
||||
'"summary": "<one sentence>"}'
|
||||
@@ -43,13 +51,27 @@ class ProctorAgent(BaseAgent):
|
||||
|
||||
async def assess(
|
||||
self,
|
||||
scenario: ProctorScenario,
|
||||
digest: TraceDigest | None,
|
||||
defense_signals: dict | None = None,
|
||||
variant_context: dict | None = None,
|
||||
learner_context: LearnerContext | None = None,
|
||||
) -> ProctorAssessment:
|
||||
timeline = summarize_proctor_scenario(scenario)
|
||||
"""Classify REAL integrity inputs into supportive signals + coaching."""
|
||||
parts: list[str] = []
|
||||
if digest is not None:
|
||||
parts.append(f"Build-session digest:\n{digest.model_dump_json()}")
|
||||
else:
|
||||
parts.append("No build telemetry recorded for this task yet.")
|
||||
if defense_signals:
|
||||
parts.append(f"Oral-defense integrity signals:\n{defense_signals}")
|
||||
if variant_context:
|
||||
parts.append(f"Variant audit context (seed + params):\n{variant_context}")
|
||||
assessment: ProctorAssessment = await self.structured_reply(
|
||||
history=None,
|
||||
user_input=timeline,
|
||||
user_input=(
|
||||
"Assess this learner's integrity signals supportively.\n\n"
|
||||
+ "\n\n".join(parts)
|
||||
),
|
||||
learner_context=learner_context,
|
||||
schema=ProctorAssessment,
|
||||
schema_hint=PROCTOR_ASSESSMENT_SCHEMA_HINT,
|
||||
|
||||
@@ -14,13 +14,14 @@ AgentFactory = Callable[[LLMProvider, Settings], BaseAgent]
|
||||
|
||||
|
||||
def register_builtin_agents(registry: "AgentRegistry") -> None:
|
||||
"""Central registration of all six shipped tutor agents (G-4: one pattern).
|
||||
"""Central registration of all seven shipped agents (G-4: one pattern).
|
||||
|
||||
coach, tutor, lab, assessor, proctor, mentor. New agents register here
|
||||
in their landing phase.
|
||||
coach, tutor, lab, assessor, proctor, mentor, examiner (Phase 5).
|
||||
New agents register here in their landing phase.
|
||||
"""
|
||||
from .assessor import AssessorAgent
|
||||
from .coach import CoachAgent
|
||||
from .examiner import ExaminerAgent
|
||||
from .lab import LabAgent
|
||||
from .mentor import MentorAgent
|
||||
from .proctor import ProctorAgent
|
||||
@@ -38,6 +39,9 @@ def register_builtin_agents(registry: "AgentRegistry") -> None:
|
||||
registry.register(
|
||||
"mentor", lambda provider, settings: MentorAgent(provider, settings)
|
||||
)
|
||||
registry.register(
|
||||
"examiner", lambda provider, settings: ExaminerAgent(provider, settings)
|
||||
)
|
||||
|
||||
|
||||
class UnknownAgentError(KeyError):
|
||||
|
||||
@@ -5,10 +5,13 @@ Boundary rule: api/ composes agents/ and llm/; they never import api/.
|
||||
|
||||
from .assessment import router as assessment_router
|
||||
from .chat import router as chat_router
|
||||
from .defense import router as defense_router
|
||||
from .lab import router as lab_router
|
||||
from .mentor import router as mentor_router
|
||||
from .proctor import router as proctor_router
|
||||
from .sandboxes import router as sandboxes_router
|
||||
from .telemetry import router as telemetry_router
|
||||
from .variants import router as variants_router
|
||||
|
||||
__all__ = [
|
||||
"assessment_router",
|
||||
@@ -17,4 +20,7 @@ __all__ = [
|
||||
"mentor_router",
|
||||
"proctor_router",
|
||||
"sandboxes_router",
|
||||
"telemetry_router",
|
||||
"variants_router",
|
||||
"defense_router",
|
||||
]
|
||||
|
||||
@@ -1,47 +1,190 @@
|
||||
"""POST /v1/assessment/evaluate — structured rubric scores (REQ-2-008).
|
||||
"""/v1/assessment — rubric evaluation + trace grading endpoints (REQ-2-008, REQ-3-004).
|
||||
|
||||
JSON response (not SSE): a pydantic-validated RubricScore. Unknown
|
||||
artifact → 404. The Assessor's structured output IS the payload.
|
||||
Two endpoint families share this router:
|
||||
|
||||
POST /v1/assessment/evaluate (v0.2, REQ-2-008) — corpus
|
||||
artifact evaluation through
|
||||
the Assessor agent.
|
||||
POST /v1/assessment/grade (v0.3, REQ-3-004) — grade a
|
||||
REAL process trace through
|
||||
the GradingEngine.
|
||||
GET /v1/assessment/grade/{learner_id}/{task_id} — stored latest grade.
|
||||
|
||||
Grading status-code mapping (the engine's outcomes are CONTRACT, not errors):
|
||||
|
||||
GradeRecord(verdict=GRADED) → 200 — rubric scores +
|
||||
verdict (in scores.verdict)
|
||||
+ digest summary.
|
||||
GradeRecord(UNGRADABLE_TRACE_INCOMPLETE) → 200 — the ungradable
|
||||
record IS a valid result:
|
||||
the trace cannot be graded,
|
||||
and the gate surfaces WHY
|
||||
(scores.missing_seqs +
|
||||
scores.integrity_flag).
|
||||
Persisted like any grade.
|
||||
GradeRecord(UNGRADABLE_EMPTY_TRACE) → 200 — no events stored for
|
||||
the pair. This covers BOTH
|
||||
a known pair whose trace
|
||||
ended up empty AND a task
|
||||
that never had a trace at
|
||||
all: the engine cannot
|
||||
distinguish them (zero
|
||||
stored events is zero
|
||||
events), and grading an
|
||||
absent trace genuinely has
|
||||
the empty-trace outcome —
|
||||
a 404 here would erase the
|
||||
durable gate record the
|
||||
engine persists for the
|
||||
pair. PLAN's 404 applies to
|
||||
GET of a never-graded pair.
|
||||
StructuredOutputError → 502 — the trace was
|
||||
gradable but the provider
|
||||
failed the D-020 budget;
|
||||
provider failure (bad
|
||||
gateway to the model), same
|
||||
mapping as evaluate.
|
||||
GET of an unknown (never-graded) pair → 404.
|
||||
|
||||
DI (D-027/D-032 house pattern): engine + store arrive via deps.get_grading_engine
|
||||
/ get_grade_store from app.state; this module owns all FastAPI wiring — the
|
||||
engine knows nothing of HTTP. UNGRADABLE_* bodies are rendered by the same
|
||||
GradeResponse model as GRADED ones (a gate record's `scores` holds the gate
|
||||
detail instead of rubric scores), so consumers read ONE shape.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..agents.assessor import RubricScore
|
||||
from ..agents.registry import AgentRegistry
|
||||
from ..agents.structured import StructuredOutputError
|
||||
from ..config import Settings
|
||||
from ..corpus.artifacts import get_artifact_bundle, get_transcript_for_artifact
|
||||
from ..corpus.learner_context import get_learner_context
|
||||
from .deps import get_agent_registry, get_provider, get_settings
|
||||
from ..grading.engine import GradingEngine
|
||||
from ..grading.store import GradeRecord, GradeStore
|
||||
from .deps import (
|
||||
get_agent_registry,
|
||||
get_grade_store,
|
||||
get_grading_engine,
|
||||
get_provider,
|
||||
get_settings,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/v1")
|
||||
|
||||
|
||||
class AssessmentRequest(BaseModel):
|
||||
artifact_id: str = Field(min_length=1)
|
||||
learner_id: str | None = None
|
||||
# --- v0.2 artifact evaluation (REQ-2-008) --------------------------------------
|
||||
|
||||
|
||||
@router.post("/assessment/evaluate", response_model=RubricScore)
|
||||
class EvaluateRequest(BaseModel):
|
||||
learner_id: str = Field(min_length=1)
|
||||
task_id: str = Field(min_length=1)
|
||||
|
||||
|
||||
@router.post("/assessment/evaluate")
|
||||
async def assessment_evaluate(
|
||||
body: AssessmentRequest,
|
||||
body: EvaluateRequest,
|
||||
registry: AgentRegistry = Depends(get_agent_registry),
|
||||
settings: Settings = Depends(get_settings),
|
||||
provider=Depends(get_provider),
|
||||
) -> RubricScore:
|
||||
bundle = get_artifact_bundle(body.artifact_id)
|
||||
if bundle is None:
|
||||
grade_store=Depends(get_grade_store),
|
||||
) -> dict:
|
||||
"""Assessor coaching rendered FROM the learner's stored grade (REQ-3-007).
|
||||
|
||||
The grading engine computes the scores (POST /assessment/grade); this
|
||||
endpoint explains them. No stored grade yet -> 404 (grade first).
|
||||
"""
|
||||
grade = grade_store.get(body.learner_id, body.task_id)
|
||||
if grade is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"unknown artifact {body.artifact_id!r}"
|
||||
status_code=404,
|
||||
detail=f"no stored grade for {body.learner_id}/{body.task_id} - grade first",
|
||||
)
|
||||
artifact, rubric = bundle
|
||||
transcript = get_transcript_for_artifact(artifact.artifact_id)
|
||||
agent = registry.get(provider, settings, "assessor")
|
||||
learner_context = get_learner_context(body.learner_id)
|
||||
try:
|
||||
return await agent.evaluate(artifact, rubric, transcript, learner_context)
|
||||
coaching = await agent.coach_grade(grade, learner_context)
|
||||
except Exception as exc:
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"assessment evaluation failed: {exc}",
|
||||
) from exc
|
||||
return {
|
||||
"learner_id": grade.learner_id,
|
||||
"task_id": grade.task_id,
|
||||
"grade_verdict": grade.verdict,
|
||||
"grade_scores": grade.scores,
|
||||
"coaching": coaching.model_dump(),
|
||||
}
|
||||
|
||||
|
||||
# --- # --- v0.3 trace grading (REQ-3-004) ---------------------------------------------
|
||||
|
||||
|
||||
class GradeRequest(BaseModel):
|
||||
learner_id: str = Field(min_length=1)
|
||||
task_id: str = Field(min_length=1)
|
||||
|
||||
|
||||
class GradeResponse(BaseModel):
|
||||
"""GradeRecord over HTTP — one shape for GRADED and UNGRADABLE_* alike.
|
||||
|
||||
`scores` holds the validated rubric (criteria 0-4, strengths, gaps,
|
||||
rubric verdict) for a GRADED record, or the gate detail
|
||||
({integrity_flag, missing_seqs}) for an UNGRADABLE_* record — never both.
|
||||
`digest` is the compact trace summary that fed the rubric prompt (empty
|
||||
for gate records: nothing was graded).
|
||||
"""
|
||||
|
||||
learner_id: str
|
||||
task_id: str
|
||||
variant_seed: str | None
|
||||
digest: dict[str, Any]
|
||||
scores: dict[str, Any]
|
||||
verdict: str
|
||||
model: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
def _grade_response(record: GradeRecord) -> GradeResponse:
|
||||
return GradeResponse.model_validate(record, from_attributes=True)
|
||||
|
||||
|
||||
@router.post("/assessment/grade", response_model=GradeResponse)
|
||||
async def assessment_grade(
|
||||
body: GradeRequest,
|
||||
engine: GradingEngine = Depends(get_grading_engine),
|
||||
) -> GradeResponse:
|
||||
"""Run the grading engine for one (learner_id, task_id) trace.
|
||||
|
||||
Gate outcomes (UNGRADABLE_*) are 200s — they are first-class results the
|
||||
engine persists, not failures. Only a provider that exhausts the D-020
|
||||
budget turns into a 502; nothing is persisted on that path.
|
||||
"""
|
||||
try:
|
||||
record = await engine.grade(body.learner_id, body.task_id)
|
||||
except StructuredOutputError as exc:
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"grading failed: {exc}",
|
||||
) from exc
|
||||
return _grade_response(record)
|
||||
|
||||
|
||||
@router.get("/assessment/grade/{learner_id}/{task_id}", response_model=GradeResponse)
|
||||
async def assessment_get_grade(
|
||||
learner_id: str,
|
||||
task_id: str,
|
||||
store: GradeStore = Depends(get_grade_store),
|
||||
) -> GradeResponse:
|
||||
"""Latest stored grade for the pair; 404 when none was ever stored."""
|
||||
record = store.get(learner_id, task_id)
|
||||
if record is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"no stored grade for {learner_id!r}/{task_id!r}",
|
||||
)
|
||||
return _grade_response(record)
|
||||
|
||||
@@ -0,0 +1,393 @@
|
||||
"""Oral-defense endpoints (Task 5-3-01, REQ-3-006, A-109).
|
||||
|
||||
Full defense loop over HTTP with mock-first voice (D-030) and the seventh
|
||||
Examiner agent (SSE question streaming happens through the chat pipeline;
|
||||
these endpoints are the session orchestration + transcript persistence):
|
||||
|
||||
POST /v1/defense/start {learner_id, task_id}
|
||||
POST /v1/defense/{id}/answer {text} | multipart audio (STT)
|
||||
GET /v1/defense/{id}/audio/{turn_id} TTS bytes (streaming)
|
||||
POST /v1/defense/{id}/finish verdict + integrity signals
|
||||
GET /v1/defense/{id} transcript + signals
|
||||
|
||||
Integrity signals (A-109) are computed server-side from turn metadata:
|
||||
long pauses = learner turns whose latency_ms exceeds PAUSE_THRESHOLD_MS.
|
||||
The defense does NOT gate on trace completeness (the grader does, G-4);
|
||||
an incomplete trace is surfaced as `trace_complete: false` so the UI can
|
||||
disclose it before the learner defends.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import itertools
|
||||
import secrets
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from fastapi import APIRouter, Depends, File, Form, HTTPException, Request, UploadFile
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..agents.examiner import ExaminerAgent
|
||||
from ..grading.features import TraceDigest, compute_digest
|
||||
from ..llm.types import Message
|
||||
from ..voice.base import VoiceDescriptor
|
||||
from ..voice.browser import BROWSER_FALLBACK_DESCRIPTOR
|
||||
from ..voice.defense_store import DefenseRecord, DefenseStore, DefenseTurn
|
||||
from .deps import (
|
||||
get_examiner,
|
||||
get_settings,
|
||||
get_trace_store,
|
||||
get_variant_store,
|
||||
get_voice_provider,
|
||||
get_voice_store,
|
||||
)
|
||||
from .identity import require_verified_age # noqa: E402 (gate dep, D-043)
|
||||
|
||||
router = APIRouter(prefix="/v1/defense", tags=["defense"])
|
||||
|
||||
#: A-109: learner turns slower than this are flagged as long pauses (ms).
|
||||
PAUSE_THRESHOLD_MS = 15_000
|
||||
|
||||
_ROLE_EXAMINER = "examiner"
|
||||
_ROLE_LEARNER = "learner"
|
||||
|
||||
# v0.4.7 (P1, final-review flag): the old id mint `dfn-{ms:x}` collided on
|
||||
# same-millisecond starts — the insert-only store's PK then 500'd the second
|
||||
# start permanently (the identity-resubmit P0-C1 class). Nonce + counter
|
||||
# keeps ids unique within and across process restarts (the identity/mock.py
|
||||
# pattern); ids stay opaque handles, never derived from PII.
|
||||
_DEFENSE_NONCE = secrets.randbits(32)
|
||||
_DEFENSE_COUNTER = itertools.count(1)
|
||||
|
||||
|
||||
def mint_defense_id(learner_id: str) -> str:
|
||||
seq = next(_DEFENSE_COUNTER)
|
||||
return f"dfn-{_DEFENSE_NONCE:08x}{seq:08x}-{learner_id[:8]}"
|
||||
|
||||
|
||||
class StartRequest(BaseModel):
|
||||
learner_id: str = Field(min_length=1)
|
||||
task_id: str = Field(min_length=1)
|
||||
|
||||
|
||||
class StartResponse(BaseModel):
|
||||
defense_id: str
|
||||
voice_descriptor: dict
|
||||
trace_complete: bool
|
||||
first_question: str
|
||||
|
||||
|
||||
class AnswerResponse(BaseModel):
|
||||
question: str
|
||||
turn_latency: dict[str, int | None]
|
||||
|
||||
|
||||
class FinishResponse(BaseModel):
|
||||
verdict: dict
|
||||
integrity_signals: dict
|
||||
|
||||
|
||||
async def _digest_for_task(
|
||||
trace_store, learner_id: str, task_id: str
|
||||
) -> tuple[TraceDigest | None, bool]:
|
||||
"""Digest of the learner's trace for this task + completeness flag."""
|
||||
if not trace_store.list_tasks(learner_id) or task_id not in trace_store.list_tasks(
|
||||
learner_id
|
||||
):
|
||||
return None, True # no trace at all is "complete" for defense purposes
|
||||
trace = trace_store.get_trace(learner_id, task_id)
|
||||
gaps = trace_store.gaps(learner_id, task_id)
|
||||
return (compute_digest(trace) if trace else None), (len(gaps) == 0)
|
||||
|
||||
|
||||
def _voice_descriptor(settings) -> VoiceDescriptor:
|
||||
"""The capability descriptor for the configured voice mode (D-030).
|
||||
|
||||
Must-Have #6: browser mode returns BROWSER_FALLBACK_DESCRIPTOR so the
|
||||
web client selects native SpeechRecognition/speechSynthesis; mock mode
|
||||
returns the mock descriptor. (A real server provider returns
|
||||
mode="server" — the protocol seam.)
|
||||
"""
|
||||
if (settings.voice_provider or "mock").strip().lower() == "browser":
|
||||
return BROWSER_FALLBACK_DESCRIPTOR
|
||||
return VoiceDescriptor(
|
||||
mode="mock", sr_available=True, tts_available=True, hint=""
|
||||
)
|
||||
|
||||
|
||||
@router.post("/start", response_model=StartResponse)
|
||||
async def start_defense(
|
||||
body: StartRequest,
|
||||
request: Request,
|
||||
examiner: ExaminerAgent = Depends(get_examiner),
|
||||
voice_store: DefenseStore = Depends(get_voice_store),
|
||||
voice_provider=Depends(get_voice_provider),
|
||||
trace_store=Depends(get_trace_store),
|
||||
variant_store=Depends(get_variant_store),
|
||||
settings=Depends(get_settings),
|
||||
) -> StartResponse:
|
||||
# v0.5 identity gate (D-043): allowlist first (G-5), then the school
|
||||
# 16+ verified verdict, before any defense machinery runs.
|
||||
if body.learner_id not in settings.learner_allowlist:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail=(
|
||||
f"learner_id {body.learner_id!r} is not on the sandbox "
|
||||
"allowlist (G-5)"
|
||||
),
|
||||
)
|
||||
await require_verified_age(
|
||||
16, body.learner_id, request.app.state.identity_store
|
||||
)
|
||||
record = voice_store.start(
|
||||
DefenseRecord(
|
||||
id=mint_defense_id(body.learner_id),
|
||||
learner_id=body.learner_id,
|
||||
task_id=body.task_id,
|
||||
status="in_progress",
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
digest, trace_complete = await _digest_for_task(trace_store, body.learner_id, body.task_id)
|
||||
variant = variant_store.get_by_task(body.task_id)
|
||||
statement = variant.statement if variant is not None else None
|
||||
|
||||
started = time.perf_counter()
|
||||
question = await examiner.next_question(
|
||||
history=[], trace_digest=digest, variant_statement=statement
|
||||
)
|
||||
llm_ms = int((time.perf_counter() - started) * 1000)
|
||||
voice_store.append_turn(
|
||||
record.id,
|
||||
DefenseTurn(
|
||||
defense_id=record.id,
|
||||
seq=0,
|
||||
role=_ROLE_EXAMINER,
|
||||
text=question,
|
||||
ts=datetime.now(UTC),
|
||||
latency_ms=llm_ms,
|
||||
created_at=datetime.now(UTC),
|
||||
),
|
||||
)
|
||||
descriptor = getattr(voice_provider, "descriptor", None) or _voice_descriptor(
|
||||
settings
|
||||
)
|
||||
return StartResponse(
|
||||
defense_id=record.id,
|
||||
voice_descriptor=descriptor.model_dump(),
|
||||
trace_complete=trace_complete,
|
||||
first_question=question,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/{defense_id}/answer", response_model=AnswerResponse)
|
||||
async def answer_defense(
|
||||
defense_id: str,
|
||||
text: str | None = Form(default=None),
|
||||
audio: UploadFile | None = File(default=None),
|
||||
voice_store: DefenseStore = Depends(get_voice_store),
|
||||
voice_provider=Depends(get_voice_provider),
|
||||
examiner: ExaminerAgent = Depends(get_examiner),
|
||||
trace_store=Depends(get_trace_store),
|
||||
variant_store=Depends(get_variant_store),
|
||||
settings=Depends(get_settings),
|
||||
) -> AnswerResponse:
|
||||
record = voice_store.get(defense_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail=f"no defense {defense_id!r}")
|
||||
if record.status == "finished":
|
||||
# The store owns the finished transition but does NOT police turn
|
||||
# sequencing (defense_store.py: "turns after finalize are a sequencing
|
||||
# bug for the endpoints to prevent") — this is the endpoint half of
|
||||
# that contract: a sealed transcript is append-only-no-more.
|
||||
raise HTTPException(
|
||||
status_code=409, detail="defense is finished; start a new defense"
|
||||
)
|
||||
if text is None and audio is None:
|
||||
raise HTTPException(status_code=422, detail="provide {text} or audio")
|
||||
|
||||
# STT (typed fallback bypasses the voice provider entirely).
|
||||
stt_ms: int | None = None
|
||||
if audio is not None:
|
||||
stt_started = time.perf_counter()
|
||||
raw = await audio.read()
|
||||
if not raw:
|
||||
# Empty upload is a client error (422), not a provider crash
|
||||
# (500): validate before the provider call so every provider
|
||||
# sees the same contract.
|
||||
raise HTTPException(status_code=422, detail="audio upload is empty")
|
||||
max_bytes = settings.voice_max_audio_mb * 1024 * 1024
|
||||
if len(raw) > max_bytes:
|
||||
# D-041/G-12: bounded audio BEFORE the provider call — the
|
||||
# client renders this as an honest re-record prompt.
|
||||
raise HTTPException(
|
||||
status_code=413,
|
||||
detail=(
|
||||
f"audio exceeds {settings.voice_max_audio_mb}MB "
|
||||
"— re-record a shorter answer"
|
||||
),
|
||||
)
|
||||
# D-041: strip codec params — MediaRecorder sends
|
||||
# 'audio/webm;codecs=opus'; the bare extension is the provider
|
||||
# contract ('webm'), else real STT endpoints reject the multipart.
|
||||
fmt = (audio.content_type or "audio/wav").split("/")[-1].split(";")[0].strip()
|
||||
try:
|
||||
segment = await voice_provider.transcribe(raw, fmt)
|
||||
except RuntimeError as exc:
|
||||
# Provider failure is the 502 house pattern (assessment.py /
|
||||
# proctor.py), not a 500: a real endpoint outage (or the
|
||||
# default mock's unscripted queue — final-review cross-phase
|
||||
# P0) must surface as an honest upstream error. Both providers
|
||||
# raise RuntimeError with sanitized text (mock: MockVoiceFailure;
|
||||
# openai-audio: key-redacted _sanitize).
|
||||
raise HTTPException(
|
||||
status_code=502, detail=f"voice transcription failed: {exc}"
|
||||
) from exc
|
||||
stt_ms = int((time.perf_counter() - stt_started) * 1000)
|
||||
text = segment.text
|
||||
|
||||
turns = record.turns if hasattr(record, "turns") else []
|
||||
history = [
|
||||
Message(role="assistant" if t.role == _ROLE_EXAMINER else "user", content=t.text)
|
||||
for t in turns
|
||||
]
|
||||
next_seq = len(turns)
|
||||
|
||||
voice_store.append_turn(
|
||||
defense_id,
|
||||
DefenseTurn(
|
||||
defense_id=defense_id,
|
||||
seq=next_seq,
|
||||
role=_ROLE_LEARNER,
|
||||
text=text or "",
|
||||
ts=datetime.now(UTC),
|
||||
latency_ms=stt_ms,
|
||||
created_at=datetime.now(UTC),
|
||||
),
|
||||
)
|
||||
|
||||
digest, _ = await _digest_for_task(trace_store, record.learner_id, record.task_id)
|
||||
variant = variant_store.get_by_task(record.task_id)
|
||||
|
||||
llm_started = time.perf_counter()
|
||||
question = await examiner.next_question(
|
||||
history=history + [Message(role="user", content=text or "")],
|
||||
trace_digest=digest,
|
||||
variant_statement=variant.statement if variant is not None else None,
|
||||
)
|
||||
llm_ms = int((time.perf_counter() - llm_started) * 1000)
|
||||
|
||||
voice_store.append_turn(
|
||||
defense_id,
|
||||
DefenseTurn(
|
||||
defense_id=defense_id,
|
||||
seq=next_seq + 1,
|
||||
role=_ROLE_EXAMINER,
|
||||
text=question,
|
||||
ts=datetime.now(UTC),
|
||||
latency_ms=llm_ms,
|
||||
created_at=datetime.now(UTC),
|
||||
),
|
||||
)
|
||||
return AnswerResponse(
|
||||
question=question,
|
||||
turn_latency={"stt_ms": stt_ms, "llm_ms": llm_ms, "tts_ms": None},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{defense_id}/audio/{turn_id}")
|
||||
async def defense_audio(
|
||||
defense_id: str,
|
||||
turn_id: int,
|
||||
voice_store: DefenseStore = Depends(get_voice_store),
|
||||
voice_provider=Depends(get_voice_provider),
|
||||
settings=Depends(get_settings),
|
||||
):
|
||||
record = voice_store.get(defense_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail=f"no defense {defense_id!r}")
|
||||
turn = next((t for t in record.turns if t.seq == turn_id), None)
|
||||
if turn is None or turn.role != _ROLE_EXAMINER:
|
||||
raise HTTPException(status_code=404, detail=f"no examiner turn {turn_id!r}")
|
||||
|
||||
async def stream():
|
||||
async for chunk in voice_provider.synthesize(turn.text):
|
||||
yield chunk
|
||||
|
||||
# G-16/D-041: the TTS format is a settings enum; the media_type maps
|
||||
# from it (was hardcoded audio/wav — wrong for every real format).
|
||||
return StreamingResponse(
|
||||
stream(), media_type=f"audio/{settings.voice_tts_format}"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/{defense_id}/finish", response_model=FinishResponse)
|
||||
async def finish_defense(
|
||||
defense_id: str,
|
||||
voice_store: DefenseStore = Depends(get_voice_store),
|
||||
examiner: ExaminerAgent = Depends(get_examiner),
|
||||
trace_store=Depends(get_trace_store),
|
||||
variant_store=Depends(get_variant_store),
|
||||
) -> FinishResponse:
|
||||
record = voice_store.get(defense_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail=f"no defense {defense_id!r}")
|
||||
|
||||
turns = record.turns if hasattr(record, "turns") else []
|
||||
history = [
|
||||
Message(role="assistant" if t.role == _ROLE_EXAMINER else "user", content=t.text)
|
||||
for t in turns
|
||||
]
|
||||
digest, _ = await _digest_for_task(trace_store, record.learner_id, record.task_id)
|
||||
variant = variant_store.get_by_task(record.task_id)
|
||||
verdict = await examiner.final_verdict(
|
||||
history=history,
|
||||
trace_digest=digest,
|
||||
variant_statement=variant.statement if variant is not None else None,
|
||||
)
|
||||
|
||||
signals: dict = {
|
||||
"long_pauses": [
|
||||
{"turn": t.seq, "latency_ms": t.latency_ms}
|
||||
for t in turns
|
||||
if t.role == _ROLE_LEARNER and (t.latency_ms or 0) > PAUSE_THRESHOLD_MS
|
||||
],
|
||||
"pause_threshold_ms": PAUSE_THRESHOLD_MS,
|
||||
# Must-Have #1: "verdict + transcript persisted" — the verdict is
|
||||
# stored INSIDE integrity_signals so GET /{id} after finish can
|
||||
# re-serve it (the finish response alone would lose it). Signals
|
||||
# are a JSON object dict (DefenseStore.finalize contract), so the
|
||||
# verdict nests under the "verdict" key alongside the A-109
|
||||
# markers the Proctor/Mentor feeds read.
|
||||
"verdict": verdict.model_dump(),
|
||||
}
|
||||
voice_store.finalize(defense_id, signals)
|
||||
return FinishResponse(verdict=verdict.model_dump(), integrity_signals=signals)
|
||||
|
||||
|
||||
@router.get("/{defense_id}")
|
||||
async def get_defense(
|
||||
defense_id: str,
|
||||
voice_store: DefenseStore = Depends(get_voice_store),
|
||||
):
|
||||
record = voice_store.get(defense_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail=f"no defense {defense_id!r}")
|
||||
return {
|
||||
"defense_id": record.id,
|
||||
"learner_id": record.learner_id,
|
||||
"task_id": record.task_id,
|
||||
"status": record.status,
|
||||
"turns": [
|
||||
{
|
||||
"seq": t.seq,
|
||||
"role": t.role,
|
||||
"text": t.text,
|
||||
"ts": t.ts,
|
||||
"latency_ms": t.latency_ms,
|
||||
}
|
||||
for t in record.turns
|
||||
],
|
||||
"integrity_signals": record.integrity_signals or {},
|
||||
}
|
||||
@@ -2,12 +2,22 @@
|
||||
|
||||
from fastapi import Request
|
||||
|
||||
from ..agents.examiner import ExaminerAgent
|
||||
from ..agents.registry import AgentRegistry
|
||||
from ..agents.session import SessionStore
|
||||
from ..config import Settings
|
||||
from ..grading.engine import GradingEngine
|
||||
from ..grading.store import GradeStore
|
||||
from ..llm.base import LLMProvider
|
||||
from ..marketplace.store import MarketplaceStore
|
||||
from ..sandbox.manager import SandboxManager
|
||||
from ..sandbox.workdir import SandboxDir
|
||||
from ..telemetry.ingest import TraceIntegrityMap
|
||||
from ..telemetry.store import TraceStore
|
||||
from ..variants.generator import VariantGenerator
|
||||
from ..variants.store import VariantStore
|
||||
from ..voice.base import VoiceProvider
|
||||
from ..voice.defense_store import DefenseStore
|
||||
|
||||
|
||||
def get_settings(request: Request) -> Settings:
|
||||
@@ -33,3 +43,44 @@ def get_sandbox_manager(request: Request) -> SandboxManager:
|
||||
def get_sandbox_test_layout(request: Request) -> SandboxDir | None:
|
||||
"""Optional test seam (app.state.sandbox_test_layout); always None in prod."""
|
||||
return getattr(request.app.state, "sandbox_test_layout", None)
|
||||
|
||||
|
||||
def get_trace_store(request: Request) -> TraceStore:
|
||||
return request.app.state.trace_store
|
||||
|
||||
|
||||
def get_trace_integrity(request: Request) -> TraceIntegrityMap:
|
||||
return request.app.state.trace_integrity
|
||||
|
||||
|
||||
def get_grade_store(request: Request) -> GradeStore:
|
||||
return request.app.state.grade_store
|
||||
|
||||
|
||||
def get_grading_engine(request: Request) -> GradingEngine:
|
||||
return request.app.state.grading_engine
|
||||
|
||||
|
||||
def get_variant_generator(request: Request) -> VariantGenerator:
|
||||
return request.app.state.variant_generator
|
||||
|
||||
|
||||
def get_variant_store(request: Request) -> VariantStore:
|
||||
return request.app.state.variant_store
|
||||
|
||||
def get_voice_store(request: Request) -> DefenseStore:
|
||||
return request.app.state.defense_store
|
||||
|
||||
|
||||
def get_voice_provider(request: Request) -> VoiceProvider:
|
||||
return request.app.state.voice_provider
|
||||
|
||||
|
||||
def get_examiner(request: Request) -> ExaminerAgent:
|
||||
return request.app.state.examiner_agent
|
||||
|
||||
|
||||
def get_marketplace_store(request: Request) -> MarketplaceStore:
|
||||
"""v0.6 marketplace (D-049): routes NEVER import the store directly —
|
||||
the api/ layer composes it via app.state DI (territory rule)."""
|
||||
return request.app.state.marketplace_store
|
||||
|
||||
@@ -0,0 +1,265 @@
|
||||
"""Identity API + age-gate dependencies (REQ-5-003/004, D-042/43).
|
||||
|
||||
Flow (all under mock provider by default; A-304 mock markers ride every
|
||||
response so downstream surfaces never treat mock-verified as real):
|
||||
POST /v1/identity/submit submission → pending (G-13 caps first)
|
||||
GET /v1/identity/status/{lid} latest record + mock marker
|
||||
POST /v1/identity/verify/{sid} poll provider → terminal transition
|
||||
|
||||
Gate dependencies (D-043 binding composition order, mounted by the gated
|
||||
routes — variants/sandbox-create/defense-start for school 16+; the
|
||||
marketplace 18+ apply route in api/marketplace.py):
|
||||
allowlist (403, G-5 pilot guard) → identity verdict (403 + verify-CTA)
|
||||
→ rate caps (429, owned by the calling routes)
|
||||
|
||||
PII (A-305): raw DOB enters via the submission, is used to derive the
|
||||
band, and is NEVER stored or logged (caplog sentinel test pins it).
|
||||
|
||||
v0.6 (D-048/D-058): the G-18 marketplace stub MOVED to api/marketplace.py
|
||||
and became the real apply route; identity keeps ONLY the gate exports.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
from ..config import Settings
|
||||
from ..identity.base import IdentityProvider, IdentitySubmission
|
||||
from ..identity.store import IdentityRecord, IdentityStore
|
||||
from .deps import get_settings
|
||||
|
||||
router = APIRouter(prefix="/v1/identity", tags=["identity"])
|
||||
|
||||
|
||||
# -- DI ------------------------------------------------------------------------
|
||||
|
||||
|
||||
def get_identity_store(request: Request) -> IdentityStore:
|
||||
return request.app.state.identity_store
|
||||
|
||||
|
||||
def get_identity_provider(request: Request) -> IdentityProvider:
|
||||
return request.app.state.identity_provider
|
||||
|
||||
|
||||
# -- models ----------------------------------------------------------------------
|
||||
|
||||
|
||||
class SubmitBody(BaseModel):
|
||||
learner_id: str = Field(min_length=1)
|
||||
#: ISO date. Validated AT THE BOUNDARY (D1 verifier fix): a malformed
|
||||
#: value would otherwise blow up as a 500 inside derive_age_band on the
|
||||
#: verify path — echoing the raw DOB into the traceback (A-305) and
|
||||
#: leaving a poisoned pending record that G-13 turns into a permanent
|
||||
#: learner lockout.
|
||||
date_of_birth: str = Field(description="ISO date; never stored or logged")
|
||||
document_refs: list[str] = Field(default_factory=list)
|
||||
|
||||
@field_validator("date_of_birth")
|
||||
@classmethod
|
||||
def _validate_dob(cls, value: str) -> str:
|
||||
try:
|
||||
datetime.fromisoformat(value)
|
||||
except ValueError as exc:
|
||||
# 422 with the input scrubbed (A-305/D1 — the app-level
|
||||
# RequestValidationError handler redacts PII field inputs).
|
||||
raise ValueError("date_of_birth must be an ISO date (YYYY-MM-DD)") from exc
|
||||
return value
|
||||
|
||||
|
||||
class SubmitResponse(BaseModel):
|
||||
submission_id: str
|
||||
status: str
|
||||
#: A-304: honesty marker — a mock verdict is NEVER production-verified.
|
||||
mock: bool = True
|
||||
detail: str = ""
|
||||
|
||||
|
||||
class StatusResponse(BaseModel):
|
||||
learner_id: str
|
||||
status: str
|
||||
age_band: str | None
|
||||
mock: bool
|
||||
verified_at: str | None
|
||||
|
||||
|
||||
class VerifyResponse(SubmitResponse):
|
||||
age_band: str | None
|
||||
|
||||
|
||||
# -- G-13 submit caps ---------------------------------------------------------------
|
||||
|
||||
|
||||
class _SubmitRateLimiter:
|
||||
"""Per-learner submit rate cap (in-memory, process-local — the G-5
|
||||
creates-per-min pattern from the sandboxes route)."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._window: dict[str, list[float]] = {}
|
||||
|
||||
def check(self, learner_id: str, per_min: int) -> None:
|
||||
now = time.monotonic()
|
||||
window = self._window.setdefault(learner_id, [])
|
||||
window[:] = [t for t in window if now - t < 60.0]
|
||||
if len(window) >= per_min:
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail="identity submit rate exceeded — wait a minute",
|
||||
)
|
||||
window.append(now)
|
||||
|
||||
|
||||
_rate_limiter = _SubmitRateLimiter()
|
||||
|
||||
|
||||
# -- verification flow ------------------------------------------------------------
|
||||
|
||||
|
||||
@router.post("/submit", response_model=SubmitResponse)
|
||||
async def submit_identity(
|
||||
body: SubmitBody,
|
||||
store: IdentityStore = Depends(get_identity_store),
|
||||
provider: IdentityProvider = Depends(get_identity_provider),
|
||||
settings: Settings = Depends(get_settings),
|
||||
) -> SubmitResponse:
|
||||
# G-13: one active pending submission per learner — resubmit while
|
||||
# pending echoes the pending state (409), not a second submission.
|
||||
if store.count_pending_for_learner(body.learner_id) > 0:
|
||||
latest = store.latest_for_learner(body.learner_id)
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"reason": "submission_pending",
|
||||
"submission_id": latest.id if latest else None,
|
||||
"status": "pending",
|
||||
},
|
||||
)
|
||||
_rate_limiter.check(body.learner_id, settings.identity_submits_per_min)
|
||||
|
||||
submission = IdentitySubmission(
|
||||
learner_id=body.learner_id,
|
||||
date_of_birth=body.date_of_birth,
|
||||
document_refs=body.document_refs,
|
||||
)
|
||||
submission_id = await provider.submit(submission)
|
||||
|
||||
record = store.insert(
|
||||
IdentityRecord(
|
||||
id=submission_id,
|
||||
learner_id=body.learner_id,
|
||||
status="pending",
|
||||
provider="mock" if settings.identity_provider == "mock" else settings.identity_provider,
|
||||
document_refs=body.document_refs, # A-305: opaque handles, never contents
|
||||
submitted_at=datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
return SubmitResponse(
|
||||
submission_id=record.id, status=record.status, mock=record.mock
|
||||
)
|
||||
|
||||
|
||||
@router.get("/status/{learner_id}", response_model=StatusResponse)
|
||||
async def identity_status(
|
||||
learner_id: str,
|
||||
store: IdentityStore = Depends(get_identity_store),
|
||||
) -> StatusResponse:
|
||||
record = store.latest_for_learner(learner_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail="no identity record")
|
||||
return StatusResponse(
|
||||
learner_id=learner_id,
|
||||
status=record.status,
|
||||
age_band=record.age_band,
|
||||
mock=record.mock,
|
||||
verified_at=record.verified_at.isoformat() if record.verified_at else None,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/verify/{submission_id}", response_model=VerifyResponse)
|
||||
async def verify_identity(
|
||||
submission_id: str,
|
||||
store: IdentityStore = Depends(get_identity_store),
|
||||
provider: IdentityProvider = Depends(get_identity_provider),
|
||||
) -> VerifyResponse:
|
||||
record = store.get(submission_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail="no such submission")
|
||||
if record.status != "pending":
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=f"submission already {record.status}",
|
||||
)
|
||||
|
||||
verdict = await provider.poll(submission_id)
|
||||
# A-305: derive the band from the provider's verdict; the raw DOB never
|
||||
# entered the store and is not logged here.
|
||||
updated = store.mark_verified(
|
||||
submission_id, verdict.model_dump(), verdict.age_band
|
||||
)
|
||||
assert updated is not None # record existed a moment ago
|
||||
return VerifyResponse(
|
||||
submission_id=updated.id,
|
||||
status=updated.status,
|
||||
age_band=updated.age_band,
|
||||
mock=updated.mock,
|
||||
detail=verdict.detail,
|
||||
)
|
||||
|
||||
|
||||
# -- age-gate dependencies (D-043 composition) -------------------------------------
|
||||
|
||||
|
||||
def _verify_cta_payload(
|
||||
reason: str, min_age: int, record: IdentityRecord | None
|
||||
) -> dict:
|
||||
"""A-306/UX acceptance #2: an actionable 403 — never a bare error."""
|
||||
return {
|
||||
"reason": reason,
|
||||
"min_age": min_age,
|
||||
"current_status": record.status if record else "none",
|
||||
"verify_cta": "/enroll",
|
||||
}
|
||||
|
||||
|
||||
async def require_verified_age(
|
||||
min_age: int,
|
||||
learner_id: str,
|
||||
store: IdentityStore,
|
||||
) -> IdentityRecord:
|
||||
"""The identity half of the D-043 composition (allowlist runs FIRST in
|
||||
the calling routes; this is the second gate; caps come after).
|
||||
|
||||
School 16+ → min_age=16; marketplace 18+ verified → min_age=18.
|
||||
"""
|
||||
record = store.latest_for_learner(learner_id)
|
||||
if record is None or record.status != "verified":
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail=_verify_cta_payload("identity_verification_required", min_age, record),
|
||||
)
|
||||
# D2 (verifier): FAIL CLOSED. Only canonical bands can pass — None,
|
||||
# unknown, or under-16 bands reject (the gate is the security boundary
|
||||
# for the future vendor and direct store writes; it never trusts a
|
||||
# band it does not recognize).
|
||||
if record.age_band not in ("16-17", "18+"):
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail=_verify_cta_payload("identity_verification_required", min_age, record),
|
||||
)
|
||||
if min_age > 16 and record.age_band != "18+":
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail=_verify_cta_payload("age_gate_18_plus", 18, record),
|
||||
)
|
||||
return record
|
||||
|
||||
|
||||
#: Convenience alias for the marketplace 18+ composition (D-043's
|
||||
#: `require_verified_adult` — direct calls use require_verified_age(18, ...)).
|
||||
#: v0.6 (D-058): the marketplace surface itself moved to api/marketplace.py;
|
||||
#: identity keeps ONLY this gate export (D-048).
|
||||
require_verified_adult = require_verified_age
|
||||
@@ -1,27 +1,36 @@
|
||||
"""POST /v1/lab/feedback — SSE stream of Lab in-flow feedback (REQ-2-007).
|
||||
"""POST /v1/lab/feedback — SSE stream of Lab in-flow feedback (REQ-3-007).
|
||||
|
||||
D-016 envelope with agent=lab. Unknown scenario → 404 before streaming.
|
||||
v0.3 re-grounding: LIVE trace digest. Request carries {learner_id, task_id};
|
||||
the digest is computed from the learner's real TraceStore events (D-028)
|
||||
and handed to the Lab agent. No corpus scenarios. Empty/unknown trace is NOT
|
||||
an error — Lab gets a "no telemetry yet" timeline and coaches the baseline.
|
||||
D-016 envelope with agent=lab.
|
||||
"""
|
||||
|
||||
import json
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from fastapi import APIRouter, Depends
|
||||
from pydantic import BaseModel, Field
|
||||
from sse_starlette.sse import EventSourceResponse
|
||||
|
||||
from ..agents.registry import AgentRegistry
|
||||
from ..config import Settings
|
||||
from ..corpus.learner_context import get_learner_context
|
||||
from ..corpus.telemetry import get_lab_scenario
|
||||
from .deps import get_agent_registry, get_provider, get_settings
|
||||
from ..grading.features import compute_digest
|
||||
from .deps import (
|
||||
get_agent_registry,
|
||||
get_provider,
|
||||
get_settings,
|
||||
get_trace_store,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/v1")
|
||||
|
||||
|
||||
class LabFeedbackRequest(BaseModel):
|
||||
scenario_id: str = Field(min_length=1)
|
||||
learner_id: str | None = None
|
||||
learner_id: str = Field(min_length=1)
|
||||
task_id: str = Field(min_length=1)
|
||||
|
||||
|
||||
@router.post("/lab/feedback")
|
||||
@@ -30,25 +39,27 @@ async def lab_feedback(
|
||||
registry: AgentRegistry = Depends(get_agent_registry),
|
||||
settings: Settings = Depends(get_settings),
|
||||
provider=Depends(get_provider),
|
||||
trace_store=Depends(get_trace_store),
|
||||
) -> EventSourceResponse:
|
||||
scenario = get_lab_scenario(body.scenario_id)
|
||||
if scenario is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"unknown scenario {body.scenario_id!r}"
|
||||
)
|
||||
agent = registry.get(provider, settings, "lab")
|
||||
learner_context = get_learner_context(body.learner_id)
|
||||
trace = (
|
||||
trace_store.get_trace(body.learner_id, body.task_id)
|
||||
if body.task_id in trace_store.list_tasks(body.learner_id)
|
||||
else []
|
||||
)
|
||||
digest = compute_digest(trace) if trace else None
|
||||
|
||||
async def event_stream() -> AsyncIterator[dict]:
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "meta",
|
||||
"agent": "lab",
|
||||
"scenario_id": body.scenario_id,
|
||||
"task_id": body.task_id,
|
||||
"model": settings.model,
|
||||
})}
|
||||
first_byte = True
|
||||
try:
|
||||
async for token in agent.stream_feedback(scenario, learner_context):
|
||||
async for token in agent.stream_feedback(digest, learner_context):
|
||||
first_byte = False
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "delta", "content": token
|
||||
|
||||
@@ -0,0 +1,272 @@
|
||||
"""/v1/marketplace — public postings API (REQ-6-001, D-048, D-058).
|
||||
|
||||
The board's runtime data source (published-only, honest 404s, no
|
||||
matchScore anywhere — G-106/G-107) + the REAL apply route: the v0.5
|
||||
placeholder that lived in api/identity.py moved here and became this
|
||||
module's /apply (D-058) — gate composition UNCHANGED (D-043 binding
|
||||
order): G-5 allowlist (403) → require_verified_adult (403 + verify-CTA)
|
||||
→ rate caps (429), then posting exists (404) and is published (409),
|
||||
then an idempotent insert (A-408: duplicate apply → 200 with the
|
||||
EXISTING application).
|
||||
|
||||
Routes never import the store directly — deps.py app.state DI only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from collections import deque
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..config import Settings
|
||||
from ..identity.store import IdentityStore
|
||||
from ..marketplace.base import Employer, Posting
|
||||
from .deps import get_marketplace_store, get_settings
|
||||
from .identity import require_verified_age
|
||||
|
||||
router = APIRouter(prefix="/v1/marketplace", tags=["marketplace"])
|
||||
|
||||
|
||||
def _identity_store(request: Request) -> IdentityStore:
|
||||
return request.app.state.identity_store
|
||||
|
||||
|
||||
def _enforce_allowlist(learner_id: str, settings: Settings) -> None:
|
||||
"""G-5 pilot guard — allowlist runs FIRST in the composition (D-043)."""
|
||||
if learner_id not in settings.learner_allowlist:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail=f"learner_id {learner_id!r} is not on the sandbox allowlist (G-5)",
|
||||
)
|
||||
|
||||
|
||||
# In-memory apply-rate window (monotonic timestamps, process-local) — the
|
||||
# G-5 creates-per-min pattern from the sandboxes route (module state is
|
||||
# acceptable: one process, one box, D-019/D-027 precedent).
|
||||
_APPLY_TIMES: dict[str, deque[float]] = {}
|
||||
|
||||
#: Applies per rolling 60s window per learner (A-408 anti-spam; the
|
||||
#: idempotent duplicate makes honest retries free — only DISTINCT
|
||||
#: posting applies consume budget).
|
||||
_APPLY_PER_LEARNER_PER_MIN = 5
|
||||
|
||||
|
||||
def _check_apply_rate(learner_id: str) -> None:
|
||||
now = time.monotonic()
|
||||
window = _APPLY_TIMES.setdefault(learner_id, deque())
|
||||
while window and now - window[0] > 60.0:
|
||||
window.popleft()
|
||||
if len(window) >= _APPLY_PER_LEARNER_PER_MIN:
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail="marketplace apply rate exceeded — wait a minute",
|
||||
)
|
||||
window.append(now)
|
||||
|
||||
|
||||
# -- response contracts -------------------------------------------------------
|
||||
|
||||
|
||||
class PostingResponse(BaseModel):
|
||||
"""One posting over HTTP. G-107: `matchScore` is deliberately absent
|
||||
from the schema — scores are computed search-time context (P3),
|
||||
never list/detail payload fields."""
|
||||
|
||||
id: str
|
||||
source: str
|
||||
external_id: str
|
||||
employer_id: str
|
||||
title: str
|
||||
description: str
|
||||
required_competencies: list[str]
|
||||
skills: list[str]
|
||||
seniority: str | None
|
||||
category: str | None
|
||||
location: str
|
||||
remote: bool
|
||||
salary_min: int | None
|
||||
salary_max: int | None
|
||||
status: str
|
||||
tags: dict | None
|
||||
posted_at: str | None
|
||||
|
||||
|
||||
class PostingListResponse(BaseModel):
|
||||
"""G-103: the page + the honest total_published so the board's paged
|
||||
fetch-all can render "Showing X of Y" with an honest truncation. P2
|
||||
(MH-2c): facets — value counts over stored tags of published-only
|
||||
postings, data-driving the filter sidebar."""
|
||||
|
||||
postings: list[PostingResponse]
|
||||
total_published: int
|
||||
facets: dict
|
||||
|
||||
|
||||
class EmployerResponse(BaseModel):
|
||||
"""Raw ingested fields (P1). P4 (D-053) merges enrichment on read and
|
||||
adds per-field {value, source} provenance markers."""
|
||||
|
||||
id: str
|
||||
source: str
|
||||
external_id: str
|
||||
name: str
|
||||
logo: str
|
||||
description: str
|
||||
industry: str
|
||||
size: str
|
||||
location: str
|
||||
website: str
|
||||
social_links: dict[str, str]
|
||||
culture: str
|
||||
|
||||
|
||||
class ApplyBody(BaseModel):
|
||||
"""D-058: the learner identity comes from the VERIFIED record, not a
|
||||
client assertion — but the pilot surface passes learner_id in the
|
||||
body like every other gated route (G-5 composition unchanged)."""
|
||||
|
||||
learner_id: str = Field(min_length=1)
|
||||
posting_id: str = Field(min_length=1)
|
||||
|
||||
|
||||
class ApplyResponse(BaseModel):
|
||||
"""Honest confirmation (D-058): exactly the application state, no
|
||||
stub/mock markers, no fabricated outcome."""
|
||||
|
||||
application_id: str
|
||||
posting_id: str
|
||||
status: str
|
||||
|
||||
|
||||
def _posting_response(posting: Posting) -> PostingResponse:
|
||||
return PostingResponse(
|
||||
id=posting.id,
|
||||
source=posting.source,
|
||||
external_id=posting.external_id,
|
||||
employer_id=posting.employer_id,
|
||||
title=posting.title,
|
||||
description=posting.description,
|
||||
required_competencies=posting.required_competencies,
|
||||
skills=posting.skills,
|
||||
seniority=posting.seniority,
|
||||
category=posting.category,
|
||||
location=posting.location,
|
||||
remote=posting.remote,
|
||||
salary_min=posting.salary_min,
|
||||
salary_max=posting.salary_max,
|
||||
status=posting.status,
|
||||
tags=dict(posting.tags) if posting.tags else None,
|
||||
posted_at=posting.posted_at.isoformat() if posting.posted_at else None,
|
||||
)
|
||||
|
||||
|
||||
def _employer_response(employer: Employer) -> EmployerResponse:
|
||||
return EmployerResponse(
|
||||
id=employer.id,
|
||||
source=employer.source,
|
||||
external_id=employer.external_id,
|
||||
name=employer.name,
|
||||
logo=employer.logo,
|
||||
description=employer.description,
|
||||
industry=employer.industry,
|
||||
size=employer.size,
|
||||
location=employer.location,
|
||||
website=employer.website,
|
||||
social_links=dict(employer.social_links),
|
||||
culture=employer.culture,
|
||||
)
|
||||
|
||||
|
||||
# -- public endpoints ---------------------------------------------------------
|
||||
|
||||
|
||||
@router.get("/postings", response_model=PostingListResponse)
|
||||
async def list_postings(
|
||||
store=Depends(get_marketplace_store),
|
||||
limit: int = Query(default=20, ge=1, le=100),
|
||||
offset: int = Query(default=0, ge=0),
|
||||
) -> PostingListResponse:
|
||||
"""Published postings only (draft/closed never board), posted_at DESC,
|
||||
with the honest total (G-103) + facets from published-only stored
|
||||
tags (MH-2c — facets computed once per page request, identical on
|
||||
every page of the fetch-all)."""
|
||||
postings, total = store.list_postings(limit=limit, offset=offset)
|
||||
return PostingListResponse(
|
||||
postings=[_posting_response(p) for p in postings],
|
||||
total_published=total,
|
||||
facets=store.facets(),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/postings/{posting_id}", response_model=PostingResponse)
|
||||
async def get_posting(
|
||||
posting_id: str,
|
||||
store=Depends(get_marketplace_store),
|
||||
) -> PostingResponse:
|
||||
"""Honest 404 on any unknown id — never a fabricated job."""
|
||||
posting = store.get_posting(posting_id)
|
||||
if posting is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"no posting with id {posting_id!r}"
|
||||
)
|
||||
return _posting_response(posting)
|
||||
|
||||
|
||||
@router.get("/employers/{employer_id}", response_model=EmployerResponse)
|
||||
async def get_employer(
|
||||
employer_id: str,
|
||||
store=Depends(get_marketplace_store),
|
||||
) -> EmployerResponse:
|
||||
"""Honest 404 on unknown employers; raw fields in P1 (merged +
|
||||
provenance in P4, D-053)."""
|
||||
employer = store.get_employer(employer_id)
|
||||
if employer is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"no employer with id {employer_id!r}"
|
||||
)
|
||||
return _employer_response(employer)
|
||||
|
||||
|
||||
# -- the real apply route (D-058; the G-18 stub's replacement) ------------------
|
||||
|
||||
|
||||
@router.post("/apply", response_model=ApplyResponse)
|
||||
async def apply_posting(
|
||||
body: ApplyBody,
|
||||
request: Request,
|
||||
store=Depends(get_marketplace_store),
|
||||
settings: Settings = Depends(get_settings),
|
||||
) -> ApplyResponse:
|
||||
"""Apply to a posting. Gate composition (D-043, unchanged order):
|
||||
G-5 allowlist → require_verified_adult (18+ verified, verify-CTA
|
||||
403) → rate caps → posting exists (404) + published (409) →
|
||||
idempotent insert (A-408: a duplicate returns the EXISTING
|
||||
application with 200)."""
|
||||
_enforce_allowlist(body.learner_id, settings)
|
||||
await require_verified_age(18, body.learner_id, _identity_store(request))
|
||||
_check_apply_rate(body.learner_id)
|
||||
|
||||
posting = store.get_posting(body.posting_id)
|
||||
if posting is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"no posting with id {body.posting_id!r}"
|
||||
)
|
||||
if posting.status != "published":
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"reason": "posting_not_published",
|
||||
"status": posting.status,
|
||||
},
|
||||
)
|
||||
|
||||
application, _created = store.insert_application(
|
||||
posting_id=body.posting_id, learner_id=body.learner_id
|
||||
)
|
||||
return ApplyResponse(
|
||||
application_id=application.id,
|
||||
posting_id=application.posting_id,
|
||||
status=application.status,
|
||||
)
|
||||
@@ -0,0 +1,339 @@
|
||||
"""/v1/marketplace/admin — founder/operator surface (REQ-6-001, D-057).
|
||||
|
||||
Guard: `require_admin` compares X-Admin-Token OR Authorization: Bearer
|
||||
against Settings().admin_token via secrets.compare_digest (G-108).
|
||||
UNSET token → every admin route 403s — fail-closed, NEVER open (a loud
|
||||
boot warning in main.py names the fix, G-11). Admin routes do NOT pass
|
||||
the learner gates (different principal class, D-057/D-043 untouched).
|
||||
|
||||
Routes: create draft posting, publish/close lifecycle (invalid
|
||||
transition → 409), seeded-corpus ingest trigger, applications review
|
||||
(filterable by posting). No cron/crawler — ingest is founder-invoked
|
||||
(A-401/A-402).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import secrets
|
||||
import uuid
|
||||
from typing import Annotated, Any
|
||||
|
||||
from fastapi import APIRouter, Depends, Header, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..config import Settings
|
||||
from ..llm.base import LLMProvider
|
||||
from ..marketplace.base import Posting
|
||||
from ..marketplace.ingest import SeededCorpusSource, run_ingest
|
||||
from ..marketplace.store import InvalidTransition, MarketplaceStore
|
||||
from ..marketplace.tagging import TaggingUnavailable, tag_posting
|
||||
from .deps import get_marketplace_store, get_provider, get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/v1/marketplace/admin", tags=["marketplace-admin"])
|
||||
|
||||
|
||||
def _extract_admin_token(
|
||||
x_admin_token: Annotated[str | None, Header(include_in_schema=False)] = None,
|
||||
authorization: Annotated[str | None, Header(include_in_schema=False)] = None,
|
||||
) -> str | None:
|
||||
"""Bearer of the request: X-Admin-Token wins; else Authorization:
|
||||
Bearer <token>. None when neither header carries a candidate."""
|
||||
if x_admin_token:
|
||||
return x_admin_token
|
||||
if authorization:
|
||||
scheme, _, token = authorization.partition(" ")
|
||||
if scheme.strip().lower() == "bearer" and token.strip():
|
||||
return token.strip()
|
||||
return None
|
||||
|
||||
|
||||
def require_admin(
|
||||
request_token: Annotated[str | None, Depends(_extract_admin_token)],
|
||||
settings: Annotated[Settings, Depends(get_settings)],
|
||||
) -> Settings:
|
||||
"""D-057 fail-closed guard: constant-time compare (G-108) against the
|
||||
env-only AI_ADMIN_TOKEN (SecretStr — never repr'd/logged); UNSET → 403
|
||||
for EVERY request (never open). Returns the settings for the route
|
||||
handlers."""
|
||||
configured = settings.admin_token.get_secret_value()
|
||||
if not configured or request_token is None:
|
||||
# Fail-closed: no configured token, or no candidate at all — the
|
||||
# surface is locked, not open (D-057).
|
||||
raise HTTPException(status_code=403, detail="admin token required")
|
||||
if not secrets.compare_digest(request_token, configured):
|
||||
raise HTTPException(status_code=403, detail="invalid admin token")
|
||||
return settings
|
||||
|
||||
|
||||
# -- contracts ----------------------------------------------------------------
|
||||
|
||||
|
||||
class AdminPostingCreate(BaseModel):
|
||||
"""Admin-authored posting body. Starts as a DRAFT (G-102: only
|
||||
seed-port postings enter published); publish is an explicit call."""
|
||||
|
||||
employer_id: str = Field(min_length=1)
|
||||
title: str = Field(min_length=1)
|
||||
description: str = ""
|
||||
required_competencies: list[str] = Field(default_factory=list)
|
||||
skills: list[str] = Field(default_factory=list)
|
||||
seniority: str | None = None
|
||||
category: str | None = None
|
||||
location: str = ""
|
||||
remote: bool = False
|
||||
salary_min: int | None = None
|
||||
salary_max: int | None = None
|
||||
|
||||
|
||||
class AdminPostingResponse(BaseModel):
|
||||
id: str
|
||||
source: str
|
||||
external_id: str
|
||||
employer_id: str
|
||||
title: str
|
||||
status: str
|
||||
|
||||
|
||||
class IngestResponse(BaseModel):
|
||||
postings_upserted: int
|
||||
employers_upserted: int
|
||||
|
||||
|
||||
class AdminPostingEdit(BaseModel):
|
||||
"""P2 (Task 2-1-03): partial content edit; fields omitted stay put.
|
||||
A successful edit RE-TAGS the posting (tags are derived data,
|
||||
D-052 — the tagger overwrites)."""
|
||||
|
||||
title: str | None = Field(default=None, min_length=1)
|
||||
description: str | None = None
|
||||
skills: list[str] | None = None
|
||||
|
||||
|
||||
class AdminApplication(BaseModel):
|
||||
id: str
|
||||
posting_id: str
|
||||
learner_id: str
|
||||
status: str
|
||||
created_at: str
|
||||
|
||||
|
||||
class AdminApplicationsResponse(BaseModel):
|
||||
applications: list[AdminApplication]
|
||||
|
||||
|
||||
# -- routes --------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.post("/postings", status_code=201, response_model=AdminPostingResponse)
|
||||
async def create_posting(
|
||||
body: AdminPostingCreate,
|
||||
store: MarketplaceStore = Depends(get_marketplace_store),
|
||||
settings: Settings = Depends(require_admin),
|
||||
) -> AdminPostingResponse:
|
||||
"""Create a DRAFT posting (founder authoring path, D-050 admin_api
|
||||
adapter — source=admin). Publish is a separate lifecycle call."""
|
||||
posting = Posting(
|
||||
id=f"adm-{uuid.uuid4().hex[:16]}",
|
||||
source="admin",
|
||||
external_id=f"adm-{uuid.uuid4().hex[:16]}",
|
||||
employer_id=body.employer_id,
|
||||
title=body.title,
|
||||
description=body.description,
|
||||
required_competencies=body.required_competencies,
|
||||
skills=body.skills,
|
||||
seniority=body.seniority, # type: ignore[arg-type]
|
||||
category=body.category,
|
||||
location=body.location,
|
||||
remote=body.remote,
|
||||
salary_min=body.salary_min,
|
||||
salary_max=body.salary_max,
|
||||
status="draft",
|
||||
)
|
||||
try:
|
||||
stored = store.insert_posting(posting)
|
||||
except Exception as exc: # surfaced IntegrityError from the store
|
||||
raise HTTPException(status_code=409, detail=str(exc)) from exc
|
||||
return AdminPostingResponse(
|
||||
id=stored.id,
|
||||
source=stored.source,
|
||||
external_id=stored.external_id,
|
||||
employer_id=stored.employer_id,
|
||||
title=stored.title,
|
||||
status=stored.status,
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/postings/{posting_id}/publish", response_model=AdminPostingResponse
|
||||
)
|
||||
async def publish_posting(
|
||||
posting_id: str,
|
||||
store: MarketplaceStore = Depends(get_marketplace_store),
|
||||
settings: Settings = Depends(require_admin),
|
||||
) -> AdminPostingResponse:
|
||||
"""draft → published (the only valid move from draft; anything else
|
||||
409s). Stamps posted_at NOW when the row has none (G-102: seed-port
|
||||
postings keep their original postedAt)."""
|
||||
try:
|
||||
stored = store.set_posting_status(posting_id, "published")
|
||||
except InvalidTransition as exc:
|
||||
raise HTTPException(status_code=409, detail=str(exc)) from exc
|
||||
if stored is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"no posting with id {posting_id!r}"
|
||||
)
|
||||
return AdminPostingResponse(
|
||||
id=stored.id,
|
||||
source=stored.source,
|
||||
external_id=stored.external_id,
|
||||
employer_id=stored.employer_id,
|
||||
title=stored.title,
|
||||
status=stored.status,
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/postings/{posting_id}/close", response_model=AdminPostingResponse
|
||||
)
|
||||
async def close_posting(
|
||||
posting_id: str,
|
||||
store: MarketplaceStore = Depends(get_marketplace_store),
|
||||
settings: Settings = Depends(require_admin),
|
||||
) -> AdminPostingResponse:
|
||||
"""published → closed (the only valid move from published; 409 on
|
||||
closing a draft or re-closing)."""
|
||||
try:
|
||||
stored = store.set_posting_status(posting_id, "closed")
|
||||
except InvalidTransition as exc:
|
||||
raise HTTPException(status_code=409, detail=str(exc)) from exc
|
||||
if stored is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"no posting with id {posting_id!r}"
|
||||
)
|
||||
return AdminPostingResponse(
|
||||
id=stored.id,
|
||||
source=stored.source,
|
||||
external_id=stored.external_id,
|
||||
employer_id=stored.employer_id,
|
||||
title=stored.title,
|
||||
status=stored.status,
|
||||
)
|
||||
|
||||
|
||||
@router.put("/postings/{posting_id}", response_model=AdminPostingResponse)
|
||||
async def edit_posting(
|
||||
posting_id: str,
|
||||
body: AdminPostingEdit,
|
||||
store: MarketplaceStore = Depends(get_marketplace_store),
|
||||
settings: Settings = Depends(require_admin),
|
||||
provider: LLMProvider = Depends(get_provider),
|
||||
) -> AdminPostingResponse:
|
||||
"""P2 (Task 2-1-03): edit title/description/skills → RE-TAG the
|
||||
posting (D-052: tags are derived data — the new extraction
|
||||
OVERWRITES the old tags column; old tags gone). Tag failure is
|
||||
honest: the edit persists, tags stay as they were, a warning logs."""
|
||||
updated = store.update_posting_fields(
|
||||
posting_id,
|
||||
title=body.title,
|
||||
description=body.description,
|
||||
skills=body.skills,
|
||||
)
|
||||
if updated is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"no posting with id {posting_id!r}"
|
||||
)
|
||||
try:
|
||||
tags = await tag_posting(
|
||||
provider,
|
||||
model=settings.model,
|
||||
title=updated.title,
|
||||
description=updated.description,
|
||||
skills_hint=updated.skills,
|
||||
)
|
||||
updated = store.set_posting_tags(posting_id, tags) or updated
|
||||
except TaggingUnavailable:
|
||||
# Honest: the EDIT landed; tags keep their previous value (the
|
||||
# warning already logged in tagging.py). Never 500 on tag failure.
|
||||
pass
|
||||
return AdminPostingResponse(
|
||||
id=updated.id,
|
||||
source=updated.source,
|
||||
external_id=updated.external_id,
|
||||
employer_id=updated.employer_id,
|
||||
title=updated.title,
|
||||
status=updated.status,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/ingest", response_model=IngestResponse)
|
||||
async def ingest(
|
||||
store: MarketplaceStore = Depends(get_marketplace_store),
|
||||
settings: Settings = Depends(require_admin),
|
||||
provider: LLMProvider = Depends(get_provider),
|
||||
) -> IngestResponse:
|
||||
"""Run the seeded corpus source now (founder-invoked — no scheduler,
|
||||
A-401/A-402). Idempotent on (source, external_id): a double-run
|
||||
yields identical rows (MH-1b). Every posting is TAGGED after
|
||||
upsert (P2, D-052) — a tag failure leaves the posting UNTAGGED with
|
||||
a warning, never a crashed ingest."""
|
||||
source = SeededCorpusSource()
|
||||
result = run_ingest(source, store)
|
||||
tagged = 0
|
||||
untagged = 0
|
||||
for posting in source.fetch():
|
||||
stored = store.get_posting(posting.id)
|
||||
if stored is None: # pragma: no cover - upsert just ran
|
||||
continue
|
||||
try:
|
||||
tags = await tag_posting(
|
||||
provider,
|
||||
model=settings.model,
|
||||
title=stored.title,
|
||||
description=stored.description,
|
||||
skills_hint=stored.skills,
|
||||
)
|
||||
store.set_posting_tags(stored.id, tags)
|
||||
tagged += 1
|
||||
except TaggingUnavailable:
|
||||
untagged += 1
|
||||
if untagged:
|
||||
logger.warning(
|
||||
"marketplace ingest tagged %s/%s postings — %s left untagged "
|
||||
"(provider failures are honest None, never fabricated)",
|
||||
tagged,
|
||||
tagged + untagged,
|
||||
untagged,
|
||||
)
|
||||
return IngestResponse(
|
||||
postings_upserted=result.postings_upserted,
|
||||
employers_upserted=result.employers_upserted,
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/applications", response_model=AdminApplicationsResponse
|
||||
)
|
||||
async def list_applications(
|
||||
posting_id: str | None = None,
|
||||
store: MarketplaceStore = Depends(get_marketplace_store),
|
||||
settings: Settings = Depends(require_admin),
|
||||
) -> AdminApplicationsResponse:
|
||||
"""Applications review list (newest first), filterable by posting.
|
||||
The founder contacts employers out-of-band (no notification infra,
|
||||
A-408)."""
|
||||
rows: list[Any] = store.list_applications(posting_id=posting_id)
|
||||
return AdminApplicationsResponse(
|
||||
applications=[
|
||||
AdminApplication(
|
||||
id=a.id,
|
||||
posting_id=a.posting_id,
|
||||
learner_id=a.learner_id,
|
||||
status=a.status,
|
||||
created_at=a.created_at.isoformat() if a.created_at else "",
|
||||
)
|
||||
for a in rows
|
||||
]
|
||||
)
|
||||
@@ -0,0 +1,54 @@
|
||||
"""G-101 placeholder-shell serving (v0.6).
|
||||
|
||||
Next 15's static export emits exactly one placeholder page per dynamic
|
||||
marketplace route (`out/marketplace/jobs/_/index.html`). Under the
|
||||
single-port deploy (D-039) these routes serve that shell for ANY id so
|
||||
runtime-created postings stay reachable and every sitemap URL resolves
|
||||
(G-109). The shell client-fetches the real data by path id and
|
||||
notFound()s on 404.
|
||||
|
||||
Registered before the StaticFiles mount so the routes win (the same
|
||||
routes-beat-the-mount mechanism the sitemap relies on). In dev mode
|
||||
(web_static_dir unset) the Next dev server handles routing itself —
|
||||
no-op.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, FastAPI
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
from ..config import Settings
|
||||
|
||||
_SHELL_ROUTES = (
|
||||
("/marketplace/jobs/{job_id}", "marketplace/jobs/_/index.html"),
|
||||
("/marketplace/jobs/{job_id}/", "marketplace/jobs/_/index.html"),
|
||||
("/marketplace/employers/{employer_id}", "marketplace/employers/_/index.html"),
|
||||
("/marketplace/employers/{employer_id}/", "marketplace/employers/_/index.html"),
|
||||
)
|
||||
|
||||
|
||||
def register_marketplace_shell_routes(app: FastAPI, settings: Settings) -> None:
|
||||
web_dir = Path(str(settings.web_static_dir or ""))
|
||||
if str(settings.web_static_dir) in ("", ".") or not web_dir.is_dir():
|
||||
# Dev/no-mount boots: the Next dev server or a separate web origin
|
||||
# handles these paths. Nothing to serve here.
|
||||
return
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
for path, shell in _SHELL_ROUTES:
|
||||
|
||||
def _serve(shell_file: str = shell) -> FileResponse:
|
||||
page = web_dir / shell_file
|
||||
if page.is_file():
|
||||
return FileResponse(page, media_type="text/html")
|
||||
raise RuntimeError( # pragma: no cover - deploy misconfig
|
||||
f"marketplace shell missing: {page} — rebuild the web export"
|
||||
)
|
||||
|
||||
router.get(path, include_in_schema=False)(_serve)
|
||||
|
||||
app.include_router(router)
|
||||
@@ -1,7 +1,8 @@
|
||||
"""POST /v1/proctor/signals — structured integrity signals (REQ-2-009).
|
||||
"""POST /v1/proctor/signals — integrity signals over REAL inputs (REQ-3-007).
|
||||
|
||||
JSON response (not SSE): a pydantic-validated ProctorAssessment.
|
||||
Unknown scenario → 404. Coaching-shaped interventions only.
|
||||
v0.3 re-grounding: live trace digest + DefenseStore long-pause signals +
|
||||
variant seed cross-check, no corpus scenarios. The proctor coaches:
|
||||
a pydantic-validated ProctorAssessment (JSON response, not SSE).
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
@@ -11,15 +12,22 @@ from ..agents.proctor import ProctorAssessment
|
||||
from ..agents.registry import AgentRegistry
|
||||
from ..config import Settings
|
||||
from ..corpus.learner_context import get_learner_context
|
||||
from ..corpus.telemetry import get_proctor_scenario
|
||||
from .deps import get_agent_registry, get_provider, get_settings
|
||||
from ..grading.features import compute_digest
|
||||
from .deps import (
|
||||
get_agent_registry,
|
||||
get_provider,
|
||||
get_settings,
|
||||
get_trace_store,
|
||||
get_variant_store,
|
||||
get_voice_store,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/v1")
|
||||
|
||||
|
||||
class ProctorRequest(BaseModel):
|
||||
scenario_id: str = Field(min_length=1)
|
||||
learner_id: str | None = None
|
||||
learner_id: str = Field(min_length=1)
|
||||
task_id: str = Field(min_length=1)
|
||||
|
||||
|
||||
@router.post("/proctor/signals", response_model=ProctorAssessment)
|
||||
@@ -28,16 +36,40 @@ async def proctor_signals(
|
||||
registry: AgentRegistry = Depends(get_agent_registry),
|
||||
settings: Settings = Depends(get_settings),
|
||||
provider=Depends(get_provider),
|
||||
trace_store=Depends(get_trace_store),
|
||||
variant_store=Depends(get_variant_store),
|
||||
voice_store=Depends(get_voice_store),
|
||||
) -> ProctorAssessment:
|
||||
scenario = get_proctor_scenario(body.scenario_id)
|
||||
if scenario is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"unknown scenario {body.scenario_id!r}"
|
||||
)
|
||||
"""Real integrity inputs: live digest + defense signals + variant context."""
|
||||
agent = registry.get(provider, settings, "proctor")
|
||||
learner_context = get_learner_context(body.learner_id)
|
||||
|
||||
trace = (
|
||||
trace_store.get_trace(body.learner_id, body.task_id)
|
||||
if body.task_id in trace_store.list_tasks(body.learner_id)
|
||||
else []
|
||||
)
|
||||
digest = compute_digest(trace) if trace else None
|
||||
|
||||
defense_signals = None
|
||||
for record in voice_store.list_for_learner(body.learner_id):
|
||||
if record.task_id == body.task_id and record.status == "finished":
|
||||
defense_signals = record.integrity_signals or None
|
||||
break
|
||||
|
||||
variant = variant_store.get_by_task(body.task_id)
|
||||
variant_context = (
|
||||
{"template_id": variant.template_id, "seed": variant.seed, "params": variant.params}
|
||||
if variant is not None
|
||||
else None
|
||||
)
|
||||
try:
|
||||
return await agent.assess(scenario, learner_context)
|
||||
return await agent.assess(
|
||||
digest,
|
||||
defense_signals=defense_signals,
|
||||
variant_context=variant_context,
|
||||
learner_context=learner_context,
|
||||
)
|
||||
except Exception as exc:
|
||||
raise HTTPException(
|
||||
status_code=502, detail=f"proctor assessment failed: {exc}"
|
||||
|
||||
@@ -23,8 +23,9 @@ never part of the API contract.
|
||||
|
||||
import time
|
||||
from collections import deque
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Response
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from ..config import Settings
|
||||
@@ -50,6 +51,9 @@ _CREATE_TIMES: deque[float] = deque()
|
||||
|
||||
class SandboxCreateRequest(BaseModel):
|
||||
learner_id: str = Field(min_length=1)
|
||||
# Optional task key: when set, the sandbox is telemetry-wired (REQ-3-003)
|
||||
# — the in-sandbox capture agent streams workspace events to the ingest.
|
||||
task_id: str | None = None
|
||||
|
||||
|
||||
class SandboxResponse(BaseModel):
|
||||
@@ -90,6 +94,10 @@ class SnapshotResponse(BaseModel):
|
||||
# -- abuse control (G-5; middleware layer, not auth) ---------------------------
|
||||
|
||||
|
||||
from ..variants.templates import get_template # noqa: E402
|
||||
from .identity import require_verified_age # noqa: E402 (gate dep, D-043)
|
||||
|
||||
|
||||
def _enforce_allowlist(learner_id: str, settings: Settings) -> None:
|
||||
if learner_id not in settings.learner_allowlist:
|
||||
raise HTTPException(
|
||||
@@ -135,14 +143,20 @@ def _check_global_create_rate(settings: Settings) -> None:
|
||||
@router.post("", status_code=201, response_model=SandboxResponse)
|
||||
async def create_sandbox(
|
||||
body: SandboxCreateRequest,
|
||||
request: Request,
|
||||
manager: SandboxManager = Depends(get_sandbox_manager),
|
||||
settings: Settings = Depends(get_settings),
|
||||
) -> SandboxResponse:
|
||||
# v0.5 identity gate (D-043, REQ-5-004): allowlist (G-5) → identity
|
||||
# verdict (403 + verify-CTA) → caps (429) — the binding composition.
|
||||
_enforce_allowlist(body.learner_id, settings)
|
||||
await require_verified_age(
|
||||
16, body.learner_id, request.app.state.identity_store
|
||||
)
|
||||
_check_per_learner_cap(await manager.list(), body.learner_id, settings)
|
||||
_check_global_create_rate(settings)
|
||||
try:
|
||||
info = await manager.create(body.learner_id)
|
||||
info = await manager.create(body.learner_id, task_id=body.task_id)
|
||||
except PoolFullError as exc:
|
||||
raise HTTPException(status_code=503, detail=str(exc)) from exc
|
||||
return _to_response(info)
|
||||
@@ -217,3 +231,191 @@ async def delete_sandbox(
|
||||
if test_layout is not None:
|
||||
result.headers["X-Workspace-Copy"] = str(test_layout.workspace)
|
||||
return result
|
||||
|
||||
|
||||
# -- workspace files + exec (Phase 6, REQ-3-008; CUT-2) -------------------------
|
||||
#
|
||||
# The build surface reads/writes/list workspace files and runs Run/Test
|
||||
# commands through the manager's backend. NO interactive shell relay (CUT-2:
|
||||
# keystroke-level stdin/stdout is v0.4) — each exec is a bounded command with
|
||||
# captured output. Paths are WORKSPACE-RELATIVE; traversal outside the
|
||||
# workspace is rejected (the workdir bind is the boundary, but the API adds
|
||||
# its own containment check — defense in depth).
|
||||
|
||||
|
||||
class FileWriteRequest(BaseModel):
|
||||
path: str = Field(min_length=1)
|
||||
content: str
|
||||
|
||||
|
||||
class ExecRequest(BaseModel):
|
||||
cmd: list[str] = Field(min_length=1)
|
||||
|
||||
|
||||
class ExecResponse(BaseModel):
|
||||
cmd: list[str]
|
||||
returncode: int
|
||||
stdout: str
|
||||
stderr: str
|
||||
duration_s: float
|
||||
|
||||
|
||||
async def _workspace_dir(manager: SandboxManager, sandbox_id: str):
|
||||
"""Resolve the sandbox workspace (tracked layout or shell layout)."""
|
||||
info = await manager.get(sandbox_id) # raises SandboxNotFoundError -> 404
|
||||
backend = manager._backend # noqa: SLF001 - API owns the composition seam
|
||||
tracked = getattr(backend, "_tracked", {}).get(sandbox_id)
|
||||
if tracked is not None:
|
||||
return tracked.workspace, info
|
||||
return info.workdir / "workspace", info
|
||||
|
||||
|
||||
def _safe_rel_path(raw: str) -> Path:
|
||||
"""Workspace-relative path; reject absolute/traversal paths."""
|
||||
candidate = Path(raw)
|
||||
if candidate.is_absolute() or ".." in candidate.parts:
|
||||
raise HTTPException(status_code=422, detail=f"invalid workspace path {raw!r}")
|
||||
return candidate
|
||||
|
||||
|
||||
def _resolve_in_workspace(workspace: Path, rel: Path) -> Path:
|
||||
"""Resolve `rel` under `workspace`, refusing symlink escapes (P7).
|
||||
|
||||
The lexical check in `_safe_rel_path` cannot see symlinks: an exec can
|
||||
plant `ln -s /etc target` in the workspace and a follow-up read/write
|
||||
would follow it OUT of the bind. Resolve with the workspace as the
|
||||
anchor (strict: a symlink chain escaping raises) and confirm the
|
||||
normalized target still sits inside the workspace — defense in depth
|
||||
for both read_file and write_file.
|
||||
"""
|
||||
try:
|
||||
target = (workspace / rel).resolve(strict=False)
|
||||
target.relative_to(workspace.resolve(strict=False))
|
||||
except ValueError:
|
||||
raise HTTPException(
|
||||
status_code=422, detail=f"path escapes the workspace: {rel.as_posix()!r}"
|
||||
) from None
|
||||
return target
|
||||
|
||||
|
||||
@router.get("/{sandbox_id}/files")
|
||||
async def list_files(
|
||||
sandbox_id: str,
|
||||
manager: SandboxManager = Depends(get_sandbox_manager),
|
||||
) -> dict:
|
||||
try:
|
||||
workspace, _ = await _workspace_dir(manager, sandbox_id)
|
||||
except SandboxNotFoundError:
|
||||
raise HTTPException(status_code=404, detail=f"no sandbox {sandbox_id!r}") from None
|
||||
return {"files": sorted(p.name for p in workspace.iterdir()) if workspace.is_dir() else []}
|
||||
|
||||
|
||||
@router.get("/{sandbox_id}/files/{path:path}")
|
||||
async def read_file(
|
||||
sandbox_id: str,
|
||||
path: str,
|
||||
manager: SandboxManager = Depends(get_sandbox_manager),
|
||||
) -> dict:
|
||||
try:
|
||||
workspace, _ = await _workspace_dir(manager, sandbox_id)
|
||||
except SandboxNotFoundError:
|
||||
raise HTTPException(status_code=404, detail=f"no sandbox {sandbox_id!r}") from None
|
||||
rel = _safe_rel_path(path)
|
||||
target = _resolve_in_workspace(workspace, rel)
|
||||
if not target.is_file():
|
||||
raise HTTPException(status_code=404, detail=f"no file {path!r}")
|
||||
return {"path": path, "content": target.read_text(errors="replace")}
|
||||
|
||||
|
||||
@router.put("/{sandbox_id}/files/{path:path}")
|
||||
async def write_file(
|
||||
sandbox_id: str,
|
||||
path: str,
|
||||
body: FileWriteRequest,
|
||||
manager: SandboxManager = Depends(get_sandbox_manager),
|
||||
) -> dict:
|
||||
try:
|
||||
workspace, _ = await _workspace_dir(manager, sandbox_id)
|
||||
except SandboxNotFoundError:
|
||||
raise HTTPException(status_code=404, detail=f"no sandbox {sandbox_id!r}") from None
|
||||
rel = _safe_rel_path(body.path)
|
||||
target = _resolve_in_workspace(workspace, rel)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(body.content)
|
||||
return {"path": body.path, "written": True}
|
||||
|
||||
|
||||
#: G-15 (REQ-5-005): per-kind exec command policy — EXACT argv[0] token
|
||||
#: matching, never prefix/substring (trivially bypassed via flags/-c
|
||||
#: passthrough). 'sh -c' passthrough is DISALLOWED for design/simulation
|
||||
#: kinds: the gaming vector would be faking build-style test cycles into a
|
||||
#: kind-agnostic digest. Build kinds keep v0.3 behavior (any command —
|
||||
#: the CUT-2 surface is Run/Test buttons, not a shell relay).
|
||||
#: python (bare) is deliberately absent — in-ns PATH resolves only python3
|
||||
#: (verifier P1); pip is absent (no network in the namespace).
|
||||
_GENERIC_FIRST_TOKENS = frozenset(
|
||||
{"ls", "cat", "pwd", "echo", "python3", "pytest"}
|
||||
)
|
||||
|
||||
|
||||
def _enforce_exec_policy(
|
||||
cmd: list[str], environment: str | None, allowed: set[str] | None = None
|
||||
) -> None:
|
||||
"""422 with the allowed set when a design/sim command is out of policy.
|
||||
|
||||
`allowed` defaults to the generic set; the exec route unions in the
|
||||
template's DECLARED harness argv[0] (a future non-python harness
|
||||
template must not reject its own Run command)."""
|
||||
if environment not in ("design", "simulation"):
|
||||
return # build kind: unchanged v0.3 semantics
|
||||
allowed = set(allowed) if allowed is not None else set(_GENERIC_FIRST_TOKENS)
|
||||
first = cmd[0] if cmd else ""
|
||||
if first in ("sh", "bash"):
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail=(
|
||||
f"shell passthrough is not allowed in a {environment} "
|
||||
f"environment; allowed commands: {sorted(allowed)}"
|
||||
),
|
||||
)
|
||||
if first not in allowed:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail=(
|
||||
f"command {first!r} is not allowed in a {environment} "
|
||||
f"environment; allowed commands: {sorted(allowed)}"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/{sandbox_id}/exec", response_model=ExecResponse)
|
||||
async def exec_command(
|
||||
sandbox_id: str,
|
||||
body: ExecRequest,
|
||||
request: Request,
|
||||
manager: SandboxManager = Depends(get_sandbox_manager),
|
||||
) -> ExecResponse:
|
||||
try:
|
||||
await manager.get(sandbox_id)
|
||||
except SandboxNotFoundError:
|
||||
raise HTTPException(status_code=404, detail=f"no sandbox {sandbox_id!r}") from None
|
||||
backend = manager._backend # noqa: SLF001 - API owns the composition seam
|
||||
handle = manager._handles.get(sandbox_id) # noqa: SLF001
|
||||
if handle is None:
|
||||
raise HTTPException(status_code=404, detail=f"no live handle {sandbox_id!r}")
|
||||
# REQ-5-005 (G-15): resolve the sandbox's variant environment by its
|
||||
# task_id (manager side-table) and enforce the per-kind command policy
|
||||
# BEFORE execution.
|
||||
task_id = manager._task_ids.get(sandbox_id) # noqa: SLF001 - composition seam
|
||||
if task_id:
|
||||
variant = request.app.state.variant_store.get_by_task(task_id)
|
||||
if variant is not None:
|
||||
template = get_template(variant.template_id)
|
||||
declared = (
|
||||
{template.run_command.split()[0]} if template is not None else set()
|
||||
)
|
||||
_enforce_exec_policy(
|
||||
body.cmd, variant.environment, _GENERIC_FIRST_TOKENS | declared
|
||||
)
|
||||
result = await backend.exec(handle, body.cmd)
|
||||
return ExecResponse(**result.model_dump())
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
"""/v1/telemetry — WS ingest + trace read endpoints (REQ-3-003, D-026, G-3).
|
||||
|
||||
The router composes the telemetry engine via DI: `telemetry/ingest.py` owns
|
||||
the WS protocol (frame contract + flood control + keepalive) and this module
|
||||
only wires `app.state.trace_store` / `app.state.trace_integrity` /
|
||||
`app.state.settings` into it, plus the two HTTP read faces:
|
||||
|
||||
WS /v1/telemetry/ingest?learner_id&task_id[&sandbox_id] (D-026)
|
||||
GET /v1/telemetry/traces/{learner_id}/{task_id} ordered trace; 404 unknown
|
||||
GET /v1/telemetry/gaps/{learner_id}/{task_id} missing seqs ; 404 unknown
|
||||
|
||||
The WS route is a thin DI shell: it validates the query-param identity and
|
||||
the Origin (browser pages are gated to the localhost dev origins — CORS
|
||||
middleware does not cover WS upgrades; the stdlib capture agent sends no
|
||||
Origin and is unaffected), pulls store/integrity/settings from `app.state`,
|
||||
and calls `telemetry_ingest_endpoint(...)` — the engine stays
|
||||
FastAPI-DI-free so it's testable without a router and the api/ layer owns
|
||||
all composition.
|
||||
|
||||
Unknown-trace contract: a trace is KNOWN when it has >=1 stored event OR
|
||||
carries an integrity flag — a flooded trace with zero stored rows still 200s
|
||||
so Proctor/grader can read WHY it's unusable (G-4 consumes
|
||||
`integrity_reason`). `TraceResponse.incomplete` / `.integrity_reason` mirror
|
||||
the map so HTTP consumers never touch process internals.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, WebSocket
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ..telemetry.ingest import (
|
||||
TraceIntegrityMap,
|
||||
telemetry_ingest_endpoint,
|
||||
)
|
||||
from ..telemetry.models import TelemetryEvent
|
||||
from ..telemetry.store import TraceStore
|
||||
from ..telemetry.tokens import IngestToken
|
||||
from .deps import get_trace_integrity, get_trace_store
|
||||
|
||||
router = APIRouter(prefix="/v1/telemetry", tags=["telemetry"])
|
||||
|
||||
#: Browser Origins allowed to open the ingest socket (A-008 mirror, D-038).
|
||||
#: The stdlib capture agent sends NO Origin header (it is not a browser) and
|
||||
#: stays allowed; a malicious page loaded in the learner's browser would
|
||||
#: carry an Origin and must not be able to poison/flood the trace. CORS
|
||||
#: middleware does NOT cover WebSocket upgrades, so this gate is explicit.
|
||||
#: In network mode the configured CORS list governs (default '*' — any
|
||||
#: origin, since credentials are never used); an explicit list still rejects
|
||||
#: unlisted origins with 1008.
|
||||
_LOCAL_WS_ORIGINS = frozenset(
|
||||
{"http://localhost:3000", "http://127.0.0.1:3000", "http://localhost:8420"}
|
||||
)
|
||||
|
||||
|
||||
def _allowed_ws_origins(settings: object) -> frozenset[str]:
|
||||
configured = getattr(settings, "cors_origin_list", None)
|
||||
if configured is None:
|
||||
return _LOCAL_WS_ORIGINS
|
||||
if configured == ["*"]:
|
||||
return frozenset() # empty = wildcard = every Origin passes
|
||||
return frozenset(configured) | _LOCAL_WS_ORIGINS
|
||||
|
||||
|
||||
# --- WS ingest (D-026) ---------------------------------------------------------
|
||||
|
||||
|
||||
@router.websocket("/ingest")
|
||||
async def telemetry_ingest_ws(websocket: WebSocket) -> None:
|
||||
"""DI shell: resolve app.state services, then hand the socket to the engine.
|
||||
|
||||
The engine's session + flood logic is fully typed and testable without
|
||||
FastAPI; this shim is the only place the two layers meet.
|
||||
"""
|
||||
origin = (websocket.headers.get("origin") or "").strip()
|
||||
allowed = _allowed_ws_origins(getattr(websocket.app.state, "settings", None))
|
||||
if origin and allowed and origin not in allowed:
|
||||
# Same-origin dev pages (Next.js on :3000, the service itself on
|
||||
# :8420) pass; anything else is refused pre-accept. Non-browser
|
||||
# producers (the capture agent, tests) send no Origin and pass.
|
||||
# Wildcard (empty frozenset) passes every Origin in network mode.
|
||||
await websocket.close(
|
||||
code=1008, reason=f"origin {origin!r} not allowed for telemetry ingest"
|
||||
)
|
||||
return
|
||||
query = websocket.query_params
|
||||
learner_id = query.get("learner_id", "")
|
||||
task_id = query.get("task_id", "")
|
||||
if not learner_id or not task_id:
|
||||
# Reject BEFORE accept: closing the pre-accept handshake is the
|
||||
# cheapest denial and unambiguous for the stdlib capture agent.
|
||||
await websocket.close(code=1008, reason="learner_id/task_id query params required")
|
||||
return
|
||||
# v0.4.7 (P1): HMAC ingest token. The socket is a grading input; in
|
||||
# network mode it is LAN-reachable and identity params were self-
|
||||
# asserted — anyone could poison/flood a learner's trace (grading DoS).
|
||||
# The manager bakes mint tokens into NC_INGEST_URL; verify here. The
|
||||
# token binds (learner_id, task_id, sandbox_id) to the process secret.
|
||||
token = query.get("token", "")
|
||||
verifier: IngestToken | None = getattr(websocket.app.state, "ingest_token", None)
|
||||
if verifier is None or not verifier.verify(
|
||||
token, learner_id, task_id, query.get("sandbox_id", "")
|
||||
):
|
||||
await websocket.close(
|
||||
code=1008, reason="missing or invalid ingest token"
|
||||
)
|
||||
return
|
||||
await telemetry_ingest_endpoint(
|
||||
websocket=websocket,
|
||||
learner_id=learner_id,
|
||||
task_id=task_id,
|
||||
sandbox_id=query.get("sandbox_id", ""),
|
||||
store=websocket.app.state.trace_store,
|
||||
integrity=websocket.app.state.trace_integrity,
|
||||
settings=websocket.app.state.settings,
|
||||
)
|
||||
|
||||
|
||||
# --- HTTP reads -----------------------------------------------------------------
|
||||
|
||||
|
||||
class TraceResponse(BaseModel):
|
||||
"""Ordered trace + integrity signal (G-4 reads incomplete/reason)."""
|
||||
|
||||
learner_id: str
|
||||
task_id: str
|
||||
events: list[TelemetryEvent]
|
||||
incomplete: bool
|
||||
integrity_reason: str | None
|
||||
|
||||
|
||||
class GapsResponse(BaseModel):
|
||||
"""Missing seqs + integrity signal."""
|
||||
|
||||
learner_id: str
|
||||
task_id: str
|
||||
gaps: list[int]
|
||||
incomplete: bool
|
||||
integrity_reason: str | None
|
||||
|
||||
|
||||
def _is_known_trace(
|
||||
store: TraceStore, integrity: TraceIntegrityMap, learner_id: str, task_id: str
|
||||
) -> bool:
|
||||
"""Known = has stored events OR carries an integrity flag (a flooded trace
|
||||
with zero rows must still be readable — Proctor needs the reason)."""
|
||||
return (
|
||||
store.latest_seq(learner_id, task_id) >= 0
|
||||
or integrity.is_incomplete(learner_id, task_id)
|
||||
)
|
||||
|
||||
|
||||
@router.get("/traces/{learner_id}/{task_id}", response_model=TraceResponse)
|
||||
async def get_trace(
|
||||
learner_id: str,
|
||||
task_id: str,
|
||||
store: TraceStore = Depends(get_trace_store),
|
||||
integrity: TraceIntegrityMap = Depends(get_trace_integrity),
|
||||
) -> TraceResponse:
|
||||
if not _is_known_trace(store, integrity, learner_id, task_id):
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"unknown trace {learner_id!r}/{task_id!r}"
|
||||
)
|
||||
return TraceResponse(
|
||||
learner_id=learner_id,
|
||||
task_id=task_id,
|
||||
events=store.get_trace(learner_id, task_id),
|
||||
incomplete=integrity.is_incomplete(learner_id, task_id),
|
||||
integrity_reason=integrity.reason(learner_id, task_id),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/gaps/{learner_id}/{task_id}", response_model=GapsResponse)
|
||||
async def get_gaps(
|
||||
learner_id: str,
|
||||
task_id: str,
|
||||
store: TraceStore = Depends(get_trace_store),
|
||||
integrity: TraceIntegrityMap = Depends(get_trace_integrity),
|
||||
) -> GapsResponse:
|
||||
if not _is_known_trace(store, integrity, learner_id, task_id):
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"unknown trace {learner_id!r}/{task_id!r}"
|
||||
)
|
||||
return GapsResponse(
|
||||
learner_id=learner_id,
|
||||
task_id=task_id,
|
||||
gaps=store.gaps(learner_id, task_id),
|
||||
incomplete=integrity.is_incomplete(learner_id, task_id),
|
||||
integrity_reason=integrity.reason(learner_id, task_id),
|
||||
)
|
||||
@@ -0,0 +1,216 @@
|
||||
"""/v1/variants — seeded per-learner task variant endpoints (REQ-3-005, D-029).
|
||||
|
||||
Three faces over the variant engine (the generator + store stay FastAPI-free;
|
||||
this module owns all HTTP wiring — the D-027/D-032 house pattern):
|
||||
|
||||
POST /v1/variants {learner_id, template_id | competency_id}
|
||||
→ 200 the learner's variant — GENERATED on the first request,
|
||||
CACHED (store read, zero LLM calls) on every repeat: D-029
|
||||
reproducibility means one (learner_id, template_id) is ONE
|
||||
variant forever, so a regenerate is always a 200 of the SAME
|
||||
variant, never a second render.
|
||||
→ 404 unknown template_id, or competency_id with no bound template.
|
||||
→ 422 neither template_id nor competency_id given.
|
||||
GET /v1/variants/{task_id}
|
||||
→ 200 the stored variant owning the task key (the grading and
|
||||
telemetry join path); 404 when no variant was ever generated
|
||||
for the task.
|
||||
GET /v1/variants?learner_id=...
|
||||
→ 200 the learner's variants, chronological; [] when none.
|
||||
|
||||
Template resolution: an explicit `template_id` wins; without it the FIRST
|
||||
template bound to `competency_id` is used (`template_for_competency`,
|
||||
D-021 corpus alignment). The response carries `competency_id` resolved
|
||||
from the template library at read time — an enrichment, not a persisted
|
||||
column (the seed re-derives the whole variant, D-029) — so the learner
|
||||
surface can bind a variant to its competency without a library round-trip.
|
||||
|
||||
Distinctness (REQ-3-005): different learners on the same template draw
|
||||
different seeded params and receive distinct statements and task_ids;
|
||||
tests/api/test_variants.py asserts this end-to-end through the API.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Self
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from ..config import Settings
|
||||
from ..identity.store import IdentityStore
|
||||
from ..variants.generator import VariantGenerator
|
||||
from ..variants.store import VariantRecord, VariantStore
|
||||
from ..variants.templates import TaskTemplate, get_template, template_for_competency
|
||||
from .deps import get_settings, get_variant_generator, get_variant_store
|
||||
from .identity import require_verified_age
|
||||
|
||||
|
||||
def _identity_store(request: Request) -> IdentityStore:
|
||||
return request.app.state.identity_store
|
||||
|
||||
|
||||
def _enforce_allowlist(learner_id: str, settings: Settings) -> None:
|
||||
"""G-5 pilot guard — allowlist runs FIRST in the composition (D-043)."""
|
||||
if learner_id not in settings.learner_allowlist:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail=f"learner_id {learner_id!r} is not on the sandbox allowlist (G-5)",
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/v1/variants", tags=["variants"])
|
||||
|
||||
|
||||
# -- contracts ------------------------------------------------------------------
|
||||
|
||||
|
||||
class VariantGenerateRequest(BaseModel):
|
||||
"""One variant identity: an explicit `template_id`, or the first
|
||||
template bound to a `competency_id` (D-021). `template_id` wins when
|
||||
both are given (explicit identity beats derived); at least one is
|
||||
required — 422 otherwise.
|
||||
"""
|
||||
|
||||
learner_id: str = Field(min_length=1)
|
||||
template_id: str | None = None
|
||||
competency_id: str | None = None
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _require_template_or_competency(self) -> Self:
|
||||
if self.template_id is None and self.competency_id is None:
|
||||
raise ValueError("template_id or competency_id is required")
|
||||
return self
|
||||
|
||||
|
||||
class VariantResponse(BaseModel):
|
||||
"""VariantRecord over HTTP, plus the `competency_id` enrichment.
|
||||
|
||||
Every field except `competency_id` mirrors `VariantRecord` exactly
|
||||
(snake_case; `created_at` is an ISO 8601 UTC datetime) — the wire shape
|
||||
typed as `TaskVariant` in packages/types/variants.ts.
|
||||
"""
|
||||
|
||||
learner_id: str
|
||||
task_id: str
|
||||
template_id: str
|
||||
competency_id: str
|
||||
seed: str
|
||||
params: dict[str, str | int]
|
||||
statement: str
|
||||
starter_files: dict[str, str]
|
||||
#: REQ-5-005 (a-11): REQUIRED on the wire — always emitted.
|
||||
environment: str
|
||||
test_command: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class VariantListResponse(BaseModel):
|
||||
variants: list[VariantResponse]
|
||||
|
||||
|
||||
# -- resolution + rendering -----------------------------------------------------
|
||||
|
||||
|
||||
def _resolve_template(body: VariantGenerateRequest) -> TaskTemplate:
|
||||
"""Template for the request: the explicit id, else the first template
|
||||
bound to the competency; 404 when neither resolves."""
|
||||
if body.template_id is not None:
|
||||
template = get_template(body.template_id)
|
||||
if template is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"no task template with id {body.template_id!r}",
|
||||
)
|
||||
return template
|
||||
# The request validator guarantees the disjunction, so reaching here
|
||||
# means a competency_id was given (never None).
|
||||
assert body.competency_id is not None
|
||||
templates = template_for_competency(body.competency_id)
|
||||
if not templates:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"no task template for competency {body.competency_id!r}",
|
||||
)
|
||||
return templates[0]
|
||||
|
||||
|
||||
def _competency_for(template_id: str) -> str:
|
||||
"""competency_id enrichment for stored records (read paths)."""
|
||||
template = get_template(template_id)
|
||||
if template is None:
|
||||
# Integrity guard: a stored variant referencing a template that is
|
||||
# no longer in the library cannot be enriched; fail loudly rather
|
||||
# than fabricate a competency binding.
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"stored variant references unknown template {template_id!r}",
|
||||
)
|
||||
return template.competency_id
|
||||
|
||||
|
||||
def _to_response(record: VariantRecord, competency_id: str) -> VariantResponse:
|
||||
return VariantResponse(
|
||||
learner_id=record.learner_id,
|
||||
task_id=record.task_id,
|
||||
template_id=record.template_id,
|
||||
competency_id=competency_id,
|
||||
seed=record.seed,
|
||||
params=dict(record.params),
|
||||
statement=record.statement,
|
||||
starter_files=dict(record.starter_files),
|
||||
# a-11: REQUIRED on the wire — the server always emits both (v0.5).
|
||||
environment=record.environment or "build",
|
||||
test_command=record.test_command or "pytest -q",
|
||||
created_at=record.created_at,
|
||||
)
|
||||
|
||||
|
||||
# -- endpoints ------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.post("", response_model=VariantResponse)
|
||||
async def generate_variant(
|
||||
body: VariantGenerateRequest,
|
||||
request: Request,
|
||||
generator: VariantGenerator = Depends(get_variant_generator),
|
||||
settings: Settings = Depends(get_settings),
|
||||
) -> VariantResponse:
|
||||
"""The learner's variant for the resolved template — generated on the
|
||||
first request, cached (no LLM call) on every repeat: D-029 makes a
|
||||
regenerate a 200 of the SAME stored variant.
|
||||
|
||||
v0.5 identity gate (D-043, REQ-5-004): the school floor is 16+ verified.
|
||||
Composition order: G-5 allowlist (403) → identity verdict (403 + CTA).
|
||||
"""
|
||||
_enforce_allowlist(body.learner_id, settings)
|
||||
await require_verified_age(16, body.learner_id, _identity_store(request))
|
||||
template = _resolve_template(body)
|
||||
record = await generator.generate(body.learner_id, template.id)
|
||||
return _to_response(record, competency_id=template.competency_id)
|
||||
|
||||
|
||||
@router.get("", response_model=VariantListResponse)
|
||||
async def list_variants(
|
||||
learner_id: str,
|
||||
store: VariantStore = Depends(get_variant_store),
|
||||
) -> VariantListResponse:
|
||||
"""All stored variants for the learner, chronological; [] when none."""
|
||||
variants = [
|
||||
_to_response(record, competency_id=_competency_for(record.template_id))
|
||||
for record in store.list_for_learner(learner_id)
|
||||
]
|
||||
return VariantListResponse(variants=variants)
|
||||
|
||||
|
||||
@router.get("/{task_id}", response_model=VariantResponse)
|
||||
async def get_variant(
|
||||
task_id: str,
|
||||
store: VariantStore = Depends(get_variant_store),
|
||||
) -> VariantResponse:
|
||||
"""The stored variant owning the task key — the grading and telemetry
|
||||
join path; 404 when no variant was ever generated for the task."""
|
||||
record = store.get_by_task(task_id)
|
||||
if record is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"no stored variant for task {task_id!r}"
|
||||
)
|
||||
return _to_response(record, competency_id=_competency_for(record.template_id))
|
||||
@@ -1,14 +1,24 @@
|
||||
"""Service settings — pydantic-settings, env prefix AI_, .env support."""
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Annotated
|
||||
|
||||
from pydantic import field_validator
|
||||
from pydantic import SecretStr, field_validator
|
||||
from pydantic_settings import BaseSettings, NoDecode, SettingsConfigDict
|
||||
|
||||
# apps/ai-service/ (sandbox dir default is relative to the app, not the CWD)
|
||||
_SERVICE_ROOT = Path(__file__).resolve().parent.parent
|
||||
|
||||
# v0.3.6: durable runtime state lives OUTSIDE the repo (founder directive —
|
||||
# the repo is a git tree, not a data dir; old defaults under apps/ai-service/
|
||||
# polluted the working copy). ~/.nextcraft is the state root for DB + sandbox
|
||||
# workdirs; env overrides may use ~/ paths (expanded by the validator below).
|
||||
_STATE_ROOT = Path.home() / ".nextcraft"
|
||||
|
||||
# G-16: TTS response-format whitelist (feeds the TTS route's Content-Type).
|
||||
_TTS_FORMATS = ("mp3", "wav", "opus")
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(env_prefix="AI_", env_file=".env", extra="ignore")
|
||||
@@ -26,7 +36,7 @@ class Settings(BaseSettings):
|
||||
|
||||
# v0.3 sandbox fabric (REQ-3-001): root holding per-sandbox workdirs.
|
||||
# Relative paths resolve against the app dir (apps/ai-service/), not the CWD.
|
||||
sandbox_dir: Path = _SERVICE_ROOT / "sandboxes"
|
||||
sandbox_dir: Path = _STATE_ROOT / "sandboxes"
|
||||
|
||||
# D-032: single-box capacity, no queue — pool full → API maps to 503.
|
||||
sandbox_max_concurrent: int = 5
|
||||
@@ -63,4 +73,125 @@ class Settings(BaseSettings):
|
||||
return value
|
||||
|
||||
# D-027: SQLite path for telemetry/grades/variants/defenses stores.
|
||||
db_path: Path = _SERVICE_ROOT / "ai_service" / "data" / "nextcraft.db"
|
||||
# v0.3.6: default moved out of the repo to the home state root.
|
||||
db_path: Path = _STATE_ROOT / "data" / "nextcraft.db"
|
||||
|
||||
# v0.3.6 single-port deploy: directory of the exported Next.js app
|
||||
# (apps/web/out). When set, the app serves it at / (StaticFiles) so the
|
||||
# whole site — UI + API — answers on one port behind HAProxy; the web
|
||||
# build bakes NEXT_PUBLIC_AI_SERVICE_URL=self (relative same-origin
|
||||
# fetches). Empty default → no mount; dev/tests unchanged.
|
||||
web_static_dir: Path = Path("")
|
||||
|
||||
@field_validator("db_path", "sandbox_dir", "web_static_dir", mode="before")
|
||||
@classmethod
|
||||
def _expanduser_paths(cls, value: object) -> object:
|
||||
# Env-set paths may use ~ (e.g. AI_DB_PATH=~/.nextcraft/x.db);
|
||||
# pydantic Path does not expand it natively.
|
||||
if isinstance(value, str):
|
||||
return Path(value).expanduser()
|
||||
return value
|
||||
|
||||
@field_validator("voice_tts_format")
|
||||
@classmethod
|
||||
def _validate_tts_format(cls, value: str) -> str:
|
||||
# G-16 + G-11 consistency: unknown values NEVER crash the boot —
|
||||
# fall back to the default with a loud warning (the boot-survival
|
||||
# log lives in main.py's voice fallback; this validator normalizes).
|
||||
v = value.strip().lower()
|
||||
if v not in _TTS_FORMATS:
|
||||
logging.getLogger(__name__).warning(
|
||||
"AI_VOICE_TTS_FORMAT=%r is not one of %s — falling back to 'mp3'",
|
||||
value,
|
||||
_TTS_FORMATS,
|
||||
)
|
||||
return "mp3"
|
||||
return v
|
||||
|
||||
# G-3 flood control (NOT backpressure-by-silence): max events ingested per
|
||||
# (learner_id, task_id) trace before the WS endpoint closes the connection
|
||||
# with 1008 and marks the trace INCOMPLETE_FLOODED. Drop-oldest is
|
||||
# FORBIDDEN — it corrupts grading input (GRILL G-3).
|
||||
telemetry_max_events_per_task: int = 50000
|
||||
|
||||
# Sandbox telemetry wiring (REQ-3-003): loopback host the in-sandbox capture
|
||||
# agent dials to reach this service's WS ingest (the agent joins the sandbox
|
||||
# mount ns but NOT the net ns — exec namespaces are offline, so the agent
|
||||
# shares the host network and reaches the app over loopback). Port reuses
|
||||
# `port` (A-004); only the host is configurable — never a second port.
|
||||
telemetry_ingest_host: str = "127.0.0.1"
|
||||
|
||||
# v0.4.7 (P1, final-review flag): HMAC secret for WS ingest tokens. The
|
||||
# ingest socket is reachable on the LAN in network mode; without a token
|
||||
# check anyone could dial with arbitrary learner_id/task_id and poison or
|
||||
# flood traces (grading DoS). The manager bakes an HMAC of
|
||||
# (learner_id, task_id, sandbox_id) into NC_INGEST_URL; the WS endpoint
|
||||
# verifies it. Unset → per-boot random secret (in-process mint/verify
|
||||
# agree; restart invalidates stale URLs — sandboxes are not reboot-
|
||||
# persistent, so that is harmless). Env-set for multi-process deploys.
|
||||
telemetry_ingest_secret: str = ""
|
||||
|
||||
# v0.3.5 network mode (D-038): dev.sh binds 0.0.0.0 so remote browsers can
|
||||
# reach the stack; '*' (default) lets any origin call the API (safe ONLY
|
||||
# because credentials are never enabled — A-008). Set a comma-separated
|
||||
# origin list (e.g. 'http://nextcraft-1:3000') to restrict instead.
|
||||
# NOTE (v0.3.6): with the UI served same-origin from 8420 this is moot in
|
||||
# production (same-origin requests skip CORS); it stays for the two-port
|
||||
# dev topology.
|
||||
cors_origins: str = "*"
|
||||
|
||||
@property
|
||||
def cors_origin_list(self) -> list[str]:
|
||||
value = self.cors_origins.strip()
|
||||
if value == "*":
|
||||
return ["*"]
|
||||
return [o.strip() for o in value.split(",") if o.strip()]
|
||||
|
||||
# Identity provider selection (REQ-5-003, A-303): 'mock' (default —
|
||||
# deterministic, no vendor spend pre-pilot; verdicts carry mock=True
|
||||
# forever per A-304). A real KYC vendor drops in via the
|
||||
# IdentityProvider protocol without API changes.
|
||||
identity_provider: str = "mock"
|
||||
# G-13: identity submit caps — one active pending per learner (409 on
|
||||
# resubmit) and a per-learner submit rate ceiling.
|
||||
identity_submits_per_min: int = 3
|
||||
|
||||
# v0.6 marketplace (REQ-6-001, D-057): the founder/admin surface is
|
||||
# guarded by a shared secret — X-Admin-Token or Authorization: Bearer
|
||||
# compared via secrets.compare_digest (G-108). Env-only (D-014), never
|
||||
# logged; SecretStr keeps it out of accidental repr/logs. UNSET =
|
||||
# every /v1/marketplace/admin/* route 403s — fail closed, NEVER open
|
||||
# (a loud boot warning names the fix, G-11).
|
||||
admin_token: SecretStr = SecretStr("")
|
||||
|
||||
# v0.6 P5 placeholder (D-056): the deployed origin for sitemap.xml /
|
||||
# robots.txt absolute URLs (consumed by the SEO phase; documented
|
||||
# now so operators see it beside the admin token).
|
||||
public_base_url: str = ""
|
||||
|
||||
# Voice provider selection (REQ-5-001, D-040): 'mock' (default — the
|
||||
# no-key path is first-class; tests never call a real voice API),
|
||||
# 'browser' (client-native SR/TTS; the descriptor tells the web client),
|
||||
# or 'openai-audio' (real server STT/TTS against an OpenAI-compatible
|
||||
# audio endpoint). openai-audio requires voice_base_url + voice_api_key;
|
||||
# when unconfigured the lifespan falls back to mock with a loud log
|
||||
# (G-11 — a typo'd env must never crash the unattended boot).
|
||||
voice_provider: str = "mock"
|
||||
|
||||
# Real server voice (D-040, A-301): endpoint-agnostic by config (D-014
|
||||
# pattern) — any OpenAI-compatible audio API works. Keys env-only,
|
||||
# never committed, never logged (mirrors ollama_cloud_api_key).
|
||||
voice_base_url: str = ""
|
||||
voice_api_key: str = ""
|
||||
voice_stt_model: str = "whisper-1"
|
||||
voice_tts_model: str = "tts-1"
|
||||
voice_tts_voice: str = "alloy"
|
||||
# G-16: whitelist, not free string — this feeds the TTS route's
|
||||
# Content-Type. A str + mode-after validator (NOT a pydantic Literal):
|
||||
# a Literal would raise ValidationError at Settings construction, before
|
||||
# main.py's G-11 fallback could catch it — crashing the unattended boot
|
||||
# on a typo'd env. Invalid values fall back to the default LOUDLY.
|
||||
voice_tts_format: str = "mp3"
|
||||
# A-302/D-041: upload guard before the provider call (webm/opus is
|
||||
# ~0.5-1MB/min, so 10MB tolerates very long answers).
|
||||
voice_max_audio_mb: int = 10
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
"""Pre-baked artifacts + rubrics + defense transcripts — Assessor mock inputs (REQ-2-008).
|
||||
|
||||
v0.2 mock engine inputs (pre-baked artifacts/rubrics/transcripts) — DORMANT as of v0.3 re-
|
||||
grounding (Task 6-1-04): no production code path imports this module. Retained as Phase-3
|
||||
calibration history.
|
||||
|
||||
|
||||
Counterpart: packages/mock-data/ai-scenarios.ts (artifact IDs string-identical,
|
||||
D-021). Real process-trace grading is a v0.3+ engine (assessment engine);
|
||||
these pre-baked submissions stand in for artifact + defense evaluation.
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
"""Simulated sandbox telemetry corpus — Lab agent mock engine inputs (REQ-2-007).
|
||||
|
||||
v0.2 mock engine inputs (Lab/Proctor scenarios) — DORMANT as of v0.3 re-grounding (Task 6-1-04):
|
||||
no production code path imports this module. Retained as Phase-3 calibration history
|
||||
(corpus/trace_fixtures.py references it from TESTS only).
|
||||
|
||||
|
||||
Counterpart: packages/mock-data/ai-scenarios.ts (scenario IDs string-identical,
|
||||
D-021). Real sandbox telemetry is a v0.3+ engine (sandbox fabric); these
|
||||
scripted event streams stand in for the build-session process trace.
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
"""Synthetic trace fixtures for grading calibration (Task 3-2-02, REQ-3-004).
|
||||
|
||||
Three builder archetypes as REAL `TelemetryEvent` traces (the grading
|
||||
engine's native input — these are NOT the v0.2 corpus's simplified
|
||||
`{timestamp, kind, detail}` event shapes):
|
||||
|
||||
strong builder — iterative debugging: small edits, tests early,
|
||||
failed runs closed by targeted fixes, eventual pass.
|
||||
lazy builder — one large paste, a single late test run, pass.
|
||||
(v0.2 alignment: the `lab-scenario-flagged` paste-
|
||||
and-run archetype; D-021.)
|
||||
struggling builder — many edit/test cycles, failures never close,
|
||||
never reaches a pass.
|
||||
|
||||
ID convention (D-021 alignment, documented in each fixture):
|
||||
v0.2 corpus scenario IDs are `<domain>-scenario-<slug>` (`lab-scenario-strong`,
|
||||
`lab-scenario-struggling`, `lab-scenario-flagged`, `proctor-scenario-*` — see
|
||||
corpus/telemetry.py). Grading fixtures carry ids string-aligned to that
|
||||
convention:
|
||||
|
||||
fixture id = "<scenario id>::<archetype>-trace"
|
||||
learner id = "learner-003" (a member of the corpus learner-00N id space;
|
||||
learner-001/002 exist in learner_context.py)
|
||||
|
||||
so a fixture is greppable against its v0.2 scenario counterpart while staying
|
||||
a distinct id space (a grading trace is a real event stream, not the v0.2
|
||||
mock scenario timeline — same convention, richer event kind set).
|
||||
|
||||
Instance hygiene: fixtures store event SPECS (plain tuples) and materialize
|
||||
FRESH `TelemetryEvent` instances on every `fixture.events` access. SQLModel
|
||||
rows carry SQLAlchemy instance state once a session has flushed them —
|
||||
re-adding the SAME instance to another store is a silent no-op, which would
|
||||
poison sequential test runs (a fixture ingested by test N would vanish for
|
||||
test N+1). Materializing per access keeps every consumer independent.
|
||||
|
||||
These fixtures exist for the CALIBRATION CONTRACT (tests/grading/
|
||||
test_calibration.py): the mock provider scripts archetype-mapped scores and
|
||||
the test asserts the ORDERING the rubric must eventually enforce. They are
|
||||
NOT an LLM quality benchmark — see the test module docstring for the honest
|
||||
scope statement.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Final
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..telemetry.models import TelemetryEvent
|
||||
|
||||
_T0: Final = datetime(2026, 9, 12, 0, 0, 0, tzinfo=UTC)
|
||||
|
||||
#: Event spec: (seq, kind, payload, seconds-since-session-start).
|
||||
EventSpec = tuple[int, str, dict, float]
|
||||
|
||||
|
||||
class TraceFixture(BaseModel):
|
||||
"""One named synthetic trace + its v0.2 scenario alignment (D-021).
|
||||
|
||||
`event_specs` is the durable, session-state-free description; `events`
|
||||
materializes fresh TelemetryEvent rows from it on every access.
|
||||
"""
|
||||
|
||||
model_config = {"frozen": True}
|
||||
|
||||
fixture_id: str # "<v0.2 scenario id>::<archetype>-trace"
|
||||
archetype: str # "strong" | "lazy" | "struggling"
|
||||
aligned_scenario_id: str # the v0.2 corpus scenario this fixture mirrors
|
||||
competency_id: str # corpus competency id space (stack-orchestration-c00N)
|
||||
task_id: str # trace task id (grading operates on (learner, task))
|
||||
learner_id: str
|
||||
event_specs: tuple[EventSpec, ...] = Field(default=())
|
||||
|
||||
@property
|
||||
def events(self) -> list[TelemetryEvent]:
|
||||
"""FRESH TelemetryEvent instances — safe to ingest into any store.
|
||||
|
||||
Never cache these: an instance flushed by one SQLite session
|
||||
carries persistent identity, and re-appending it elsewhere no-ops.
|
||||
"""
|
||||
return [
|
||||
TelemetryEvent(
|
||||
learner_id=self.learner_id,
|
||||
task_id=self.task_id,
|
||||
seq=seq,
|
||||
kind=kind,
|
||||
payload=payload,
|
||||
ts=_T0 + timedelta(seconds=offset_s),
|
||||
sandbox_id="sbx-calibration",
|
||||
)
|
||||
for seq, kind, payload, offset_s in self.event_specs
|
||||
]
|
||||
|
||||
def description(self) -> str:
|
||||
return (
|
||||
f"{self.fixture_id} (archetype={self.archetype}, aligned="
|
||||
f"{self.aligned_scenario_id}, competency={self.competency_id})"
|
||||
)
|
||||
|
||||
|
||||
class _Builder:
|
||||
"""Seq-accurate event-spec builder for one (learner, task) pair."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.specs: list[EventSpec] = []
|
||||
self._seq = 0
|
||||
|
||||
def add(self, kind: str, payload: dict | None, at: float) -> None:
|
||||
self.specs.append((self._seq, kind, payload or {}, at))
|
||||
self._seq += 1
|
||||
|
||||
|
||||
def _strong_builder_specs() -> list[EventSpec]:
|
||||
"""Iterative debugging: tests early, tight edit→test loops, eventual pass.
|
||||
|
||||
Mirrors `lab-scenario-strong` (v0.2: keystrokes → file_save → run_tests →
|
||||
test_pass, c002) at full telemetry fidelity — every failed cycle is
|
||||
closed by a targeted edit followed by a re-run that passes.
|
||||
"""
|
||||
b = _Builder()
|
||||
b.add("activity", {"state": "starting"}, 0)
|
||||
b.add("file_diff", {"path": "planner.py", "added": 14}, 95) # small edit
|
||||
b.add("command", {"cmd": "pytest -q tests/test_planner.py"}, 120) # tests EARLY
|
||||
b.add("test_result", {"passed": True, "exit_code": 0}, 126)
|
||||
b.add("file_diff", {"path": "tool_node.py", "added": 22}, 180)
|
||||
b.add("command", {"cmd": "pytest -q"}, 275)
|
||||
b.add("test_result", {"passed": False, "exit_code": 1}, 281) # honest failure
|
||||
b.add("file_diff", {"path": "tool_node.py", "added": 6, "removed": 2}, 340) # targeted fix
|
||||
b.add("command", {"cmd": "pytest -q"}, 430)
|
||||
b.add("test_result", {"passed": True, "exit_code": 0}, 436) # cycle CLOSED
|
||||
b.add("command", {"cmd": "git commit -m 'tool node with retries'"}, 500)
|
||||
return b.specs
|
||||
|
||||
|
||||
def _lazy_builder_specs() -> list[EventSpec]:
|
||||
"""Paste-and-run: one large paste, a single LATE test run, instant pass.
|
||||
|
||||
Mirrors `lab-scenario-flagged` (v0.2: paste of 2,400 chars → run_tests →
|
||||
instant 6/6 pass, c003) at full telemetry fidelity — zero iteration, zero
|
||||
verification during construction, one terminal test run only.
|
||||
"""
|
||||
b = _Builder()
|
||||
b.add("activity", {"state": "starting"}, 0)
|
||||
b.add("file_diff", {"path": "eval.py", "added": 240, "removed": 0}, 30) # one bulk paste
|
||||
b.add("file_diff", {"path": "README.md", "added": 12}, 40)
|
||||
b.add("command", {"cmd": "npm run build"}, 45)
|
||||
b.add("run_result", {"exit_code": 0, "ok": True}, 60)
|
||||
b.add("command", {"cmd": "pytest -q"}, 520) # single LATE test run
|
||||
b.add("test_result", {"passed": True, "exit_code": 0}, 540) # instant pass
|
||||
return b.specs
|
||||
|
||||
|
||||
def _struggling_builder_specs() -> list[EventSpec]:
|
||||
"""Many cycles, none close: repeated failures, no eventual pass.
|
||||
|
||||
Mirrors `lab-scenario-struggling` (v0.2: repeated identical ImportErrors,
|
||||
idle gaps, no checkpoint, c002) at full telemetry fidelity — edits happen
|
||||
between failures, but the same failure recurs; no pass is ever reached.
|
||||
"""
|
||||
b = _Builder()
|
||||
b.add("activity", {"state": "starting"}, 0)
|
||||
b.add("file_diff", {"path": "main.py", "added": 40}, 20)
|
||||
b.add("command", {"cmd": "pytest -q"}, 210)
|
||||
b.add("test_result", {"passed": False, "exit_code": 1}, 215) # ImportError
|
||||
b.add("file_diff", {"path": "main.py", "added": 8, "removed": 3}, 300)
|
||||
b.add("command", {"cmd": "pytest -q"}, 520)
|
||||
b.add("test_result", {"passed": False, "exit_code": 1}, 525) # SAME error
|
||||
b.add("file_diff", {"path": "main.py", "added": 5}, 610)
|
||||
b.add("command", {"cmd": "pytest -q"}, 960)
|
||||
b.add("test_result", {"passed": False, "exit_code": 1}, 965) # STILL failing
|
||||
b.add("activity", {"state": "idle"}, 1500) # long idle
|
||||
b.add("activity", {"state": "idle"}, 2200)
|
||||
return b.specs
|
||||
|
||||
|
||||
#: Calibration learner — a member of the corpus learner-00N id space (D-021;
|
||||
#: learner-001/002 live in corpus/learner_context.py; grading fixtures use
|
||||
#: a third id so calibration traces never collide with mock-context reads).
|
||||
CALIBRATION_LEARNER_ID: Final = "learner-003"
|
||||
|
||||
|
||||
STRONG_BUILDER: Final = TraceFixture(
|
||||
fixture_id="lab-scenario-strong::strong-trace",
|
||||
archetype="strong",
|
||||
aligned_scenario_id="lab-scenario-strong",
|
||||
competency_id="stack-orchestration-c002",
|
||||
task_id="task-calibration-strong",
|
||||
learner_id=CALIBRATION_LEARNER_ID,
|
||||
event_specs=tuple(_strong_builder_specs()),
|
||||
)
|
||||
|
||||
LAZY_BUILDER: Final = TraceFixture(
|
||||
# v0.2's paste-and-run archetype is the "flagged" lab scenario (D-021):
|
||||
# large paste → instant test pass. "lazy builder" is that behavior
|
||||
# without the proctor flag; the alignment is behavioral, documented here.
|
||||
fixture_id="lab-scenario-flagged::lazy-trace",
|
||||
archetype="lazy",
|
||||
aligned_scenario_id="lab-scenario-flagged",
|
||||
competency_id="stack-orchestration-c003",
|
||||
task_id="task-calibration-lazy",
|
||||
learner_id=CALIBRATION_LEARNER_ID,
|
||||
event_specs=tuple(_lazy_builder_specs()),
|
||||
)
|
||||
|
||||
STRUGGLING_BUILDER: Final = TraceFixture(
|
||||
fixture_id="lab-scenario-struggling::struggling-trace",
|
||||
archetype="struggling",
|
||||
aligned_scenario_id="lab-scenario-struggling",
|
||||
competency_id="stack-orchestration-c002",
|
||||
task_id="task-calibration-struggling",
|
||||
learner_id=CALIBRATION_LEARNER_ID,
|
||||
event_specs=tuple(_struggling_builder_specs()),
|
||||
)
|
||||
|
||||
TRACE_FIXTURES: Final[dict[str, TraceFixture]] = {
|
||||
f.fixture_id: f
|
||||
for f in (STRONG_BUILDER, LAZY_BUILDER, STRUGGLING_BUILDER)
|
||||
}
|
||||
|
||||
|
||||
def get_trace_fixture(fixture_id: str) -> TraceFixture | None:
|
||||
return TRACE_FIXTURES.get(fixture_id)
|
||||
|
||||
|
||||
def digest_of(fixture: TraceFixture):
|
||||
"""Compute the digest for a fixture (pure compute; test-side helper)."""
|
||||
from ..grading.features import compute_digest
|
||||
|
||||
return compute_digest(fixture.events)
|
||||
@@ -0,0 +1,27 @@
|
||||
"""Process-trace grading — trace digest, rubric engine, GradeStore (REQ-3-004).
|
||||
|
||||
Boundary rule (D-027): grading/ is an engine module — it never imports
|
||||
api/; the single sanctioned agents/ dependency is the shared D-020
|
||||
structured defense (agents/structured.py), imported module-direct in
|
||||
engine.py (see its docstring for why). features.py imports telemetry/;
|
||||
store.py imports config only; engine.py composes llm/ + telemetry/ +
|
||||
prompts/ + agents.structured.
|
||||
|
||||
Wave status: features.py (TraceDigest, compute_digest) + store.py
|
||||
(GradeRecord, GradeStore, SQLiteGradeStore) landed in Wave 1 (3-1-01 +
|
||||
3-1-02); engine.py (GradingEngine, RubricScore) is Wave 2 (3-2-01).
|
||||
"""
|
||||
|
||||
from .engine import GradingEngine, RubricScore
|
||||
from .features import TraceDigest, compute_digest
|
||||
from .store import GradeRecord, GradeStore, SQLiteGradeStore
|
||||
|
||||
__all__ = [
|
||||
"GradeRecord",
|
||||
"GradeStore",
|
||||
"GradingEngine",
|
||||
"RubricScore",
|
||||
"SQLiteGradeStore",
|
||||
"TraceDigest",
|
||||
"compute_digest",
|
||||
]
|
||||
@@ -0,0 +1,313 @@
|
||||
"""GradingEngine — rubric scoring over real process traces (REQ-3-004).
|
||||
|
||||
The Wave-2 composition of the grading stack:
|
||||
|
||||
trace completeness gate (G-4, FIRST — nothing is sent to any LLM
|
||||
when the gate trips) → TraceStore.get_trace → compute_digest (D-028)
|
||||
→ prompts.grading.render_trace_digest → D-020 4-layer structured
|
||||
defense (agents/structured.py — REUSED, composed, never duplicated)
|
||||
→ RubricScore validation → GradeStore persistence → GradeRecord.
|
||||
|
||||
Gate verdicts are FIRST-CLASS RESULTS, not exceptions (G-4 is binding):
|
||||
UNGRADABLE_TRACE_INCOMPLETE — seq gaps in the store OR the trace is
|
||||
flagged by TraceIntegrityMap (INCOMPLETE_FLOODED). The gap list /
|
||||
flag reason is surfaced in `scores` for API rendering, and the
|
||||
record is PERSISTED like any grade so a learner sees why no
|
||||
credential can be issued for this trace — the gate outcome is
|
||||
durable and auditable, not a transient error string.
|
||||
UNGRADABLE_EMPTY_TRACE — no events stored for the pair.
|
||||
On either verdict `scores.criteria` is empty and the LLM is never called.
|
||||
|
||||
DI (D-027/D-032 house pattern): the engine receives trace_store,
|
||||
grade_store, integrity and provider through the constructor and knows
|
||||
NOTHING of FastAPI — api/ composes it (Task 3-3-01). `model` is injected
|
||||
alongside the provider so tests script the mock against the production
|
||||
wiring without touching Settings.
|
||||
|
||||
Boundary (D-027): grading/ imports llm/ (provider protocol + Message
|
||||
type), telemetry/ (store + integrity map), prompts/ (rubric text) and
|
||||
ONLY agents.structured — the sanctioned shared D-020 defense. We import
|
||||
the MODULE directly (`from ..agents.structured import structured_completion`)
|
||||
rather than the `agents` package, mirroring how agents/base.py consumes
|
||||
it (same direct-module import): that keeps the dependency surface to
|
||||
exactly the two names the engine needs (structured_completion,
|
||||
StructuredOutputError) and avoids executing agents/__init__ re-exports
|
||||
(BaseAgent, registry, session store) that grading has no business
|
||||
loading — a side-effect-hygiene choice that keeps this import line
|
||||
grep-auditable as "the one agents dependency". grading/ never imports api/.
|
||||
|
||||
RubricScore placement (documented decision): the validated output model
|
||||
lives HERE, not in prompts/. The pydantic model is the engine's return
|
||||
CONTRACT (the shape GradeStore.scores must hold), while prompts/grading.py
|
||||
is pure prompt text + its mirror schema HINT string — the same split as
|
||||
agents/assessor.py (model + hint) but with the model owned by the engine
|
||||
module that validates it. Prompt files hold text, per the prompts/ house
|
||||
style; engine files hold typed contracts.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
from typing import TYPE_CHECKING, Final
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from ..agents.structured import StructuredOutputError, structured_completion
|
||||
from ..llm.base import LLMProvider
|
||||
from ..llm.types import Message
|
||||
from ..prompts.grading import (
|
||||
RUBRIC_CRITERIA,
|
||||
RUBRIC_SCORE_SCHEMA_HINT,
|
||||
SYSTEM_PROMPT,
|
||||
render_trace_digest,
|
||||
)
|
||||
from ..telemetry.ingest import TraceIntegrityMap
|
||||
from ..telemetry.store import TraceStore
|
||||
from .features import compute_digest
|
||||
from .store import GradeRecord, GradeStore
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover - protocol-only import for the optional
|
||||
from ..variants.store import VariantStore # noqa: TC001 (variant-blind without it)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: First-class gate verdicts (G-4). GradeRecord.verdict values; non-empty
|
||||
#: by store contract. Rubric verdicts (mastered/developing/not_yet) ride in
|
||||
#: `scores.verdict` — `record.verdict` stays the machine-readable outcome.
|
||||
VERDICT_UNGRADABLE_INCOMPLETE: Final = "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
VERDICT_UNGRADABLE_EMPTY: Final = "UNGRADABLE_EMPTY_TRACE"
|
||||
#: record.verdict for a successfully LLM-graded trace (the rubric verdict
|
||||
#: travels inside scores); keeps verdict non-empty for every persisted row.
|
||||
VERDICT_GRADED: Final = "GRADED"
|
||||
|
||||
#: Gate-detail keys surfaced in GradeRecord.scores (tests assert on these).
|
||||
_INTEGRITY_FLAG_KEY: Final = "integrity_flag"
|
||||
|
||||
|
||||
def _anchors_context(variant) -> str: # noqa: ANN001 - VariantRecord (duck-typed)
|
||||
"""Render the variant template's difficulty anchors for the grader prompt.
|
||||
|
||||
Contains only the template id + anchor numbers — no learner-identifying
|
||||
material (D-028 anonymity preserved; the digest-leak tests keep holding).
|
||||
Lazy template import: grading must not import variants/ at module load
|
||||
(variants/prompts import-cycle safety mirrors llm/ rules).
|
||||
"""
|
||||
from ..variants.templates import get_template
|
||||
|
||||
template = get_template(variant.template_id)
|
||||
if template is None:
|
||||
return f"template={variant.template_id} (anchors unavailable)"
|
||||
a = template.rubric_anchors
|
||||
return (
|
||||
f"template={template.id}; "
|
||||
f"expected_edit_count_band={list(a.expected_edit_count_band)}; "
|
||||
f"expected_min_test_runs={a.expected_min_test_runs}; "
|
||||
f"expected_error_fix_cycles_band={list(a.expected_error_fix_cycles_band)}"
|
||||
)
|
||||
_MISSING_SEQS_KEY: Final = "missing_seqs"
|
||||
|
||||
|
||||
class RubricScore(BaseModel):
|
||||
"""Validated LLM output: per-criterion 0-4 scores + strengths + gaps + verdict.
|
||||
|
||||
The D-020 schema for the grader: `structured_completion` parses the
|
||||
model reply into THIS shape (layer 3), retrying once with the
|
||||
validation error fed back (layer 4). Exact criteria set + 0-4 ranges
|
||||
are enforced here, so the scores dict persisted to GradeStore is
|
||||
always rubric-shaped no matter what the model produced.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
criteria: dict[str, int]
|
||||
strengths: list[str] = Field(min_length=1, max_length=2)
|
||||
gaps: list[str] = Field(min_length=1, max_length=2)
|
||||
verdict: str # "mastered" | "developing" | "not_yet"
|
||||
|
||||
@field_validator("criteria")
|
||||
@classmethod
|
||||
def _criteria_rubric_shaped(cls, value: dict[str, int]) -> dict[str, int]:
|
||||
"""Exact criteria keys (no extras, no omissions) and 0-4 scores."""
|
||||
expected = set(RUBRIC_CRITERIA)
|
||||
got = set(value)
|
||||
if got != expected:
|
||||
raise ValueError(
|
||||
f"criteria keys must be exactly {sorted(expected)}, got {sorted(got)}"
|
||||
)
|
||||
for key, score in value.items():
|
||||
if not 0 <= score <= 4:
|
||||
raise ValueError(f"criterion {key!r} must be within 0-4, got {score}")
|
||||
return value
|
||||
|
||||
@field_validator("verdict")
|
||||
@classmethod
|
||||
def _verdict_known(cls, value: str) -> str:
|
||||
allowed = {"mastered", "developing", "not_yet"}
|
||||
if value not in allowed:
|
||||
raise ValueError(f"verdict must be one of {sorted(allowed)}, got {value!r}")
|
||||
return value
|
||||
|
||||
|
||||
class GradingEngine:
|
||||
"""Scores a (learner_id, task_id) trace into a persisted GradeRecord."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
trace_store: TraceStore,
|
||||
grade_store: GradeStore,
|
||||
integrity: TraceIntegrityMap,
|
||||
provider: LLMProvider,
|
||||
*,
|
||||
model: str = "gemma4:31b",
|
||||
variant_store: "VariantStore | None" = None,
|
||||
) -> None:
|
||||
self._trace_store = trace_store
|
||||
self._grade_store = grade_store
|
||||
self._integrity = integrity
|
||||
self._provider = provider
|
||||
self._model = model
|
||||
# Phase 4 (MH#4): optional variant lookup — when the graded task
|
||||
# derives from a generated variant, its template's difficulty anchors
|
||||
# ship to the grader prompt (same bar for every variant of the
|
||||
# template, a-5) and the variant seed is stamped on the record.
|
||||
# Optional so engine tests stay decoupled; main.py lifespan wires it.
|
||||
self._variant_store = variant_store
|
||||
|
||||
async def grade(self, learner_id: str, task_id: str) -> GradeRecord:
|
||||
"""Grade one trace; persist latest-state (GradeStore upserts); return it.
|
||||
|
||||
Gate FIRST (G-4): the LLM is only ever reached from the fully
|
||||
guarded path — no gate state can be masked by an LLM error.
|
||||
"""
|
||||
record = await self._grade(learner_id, task_id)
|
||||
self._grade_store.save(record)
|
||||
return record
|
||||
|
||||
# ------------------------------------------------------------------ core
|
||||
|
||||
async def _grade(self, learner_id: str, task_id: str) -> GradeRecord:
|
||||
# -- G-4 gate FIRST: integrity flag OR seq gaps. Ordering matters:
|
||||
# gaps() returns [] for an EMPTY trace, so the empty check below is
|
||||
# reachable only when no rows exist at all; a gapped or flooded
|
||||
# trace can never fall through to the LLM path.
|
||||
if self._integrity.is_incomplete(learner_id, task_id):
|
||||
reason = self._integrity.reason(learner_id, task_id) or "unknown"
|
||||
gaps = self._trace_store.gaps(learner_id, task_id)
|
||||
logger.info(
|
||||
"grade gate (G-4): %s/%s integrity-flagged (%s) — ungradable",
|
||||
learner_id,
|
||||
task_id,
|
||||
reason,
|
||||
)
|
||||
return self._ungradable(
|
||||
learner_id,
|
||||
task_id,
|
||||
detail={_INTEGRITY_FLAG_KEY: reason, _MISSING_SEQS_KEY: gaps},
|
||||
verdict=VERDICT_UNGRADABLE_INCOMPLETE,
|
||||
)
|
||||
gaps = self._trace_store.gaps(learner_id, task_id)
|
||||
if gaps:
|
||||
logger.info(
|
||||
"grade gate (G-4): %s/%s seq gaps %s — ungradable",
|
||||
learner_id,
|
||||
task_id,
|
||||
gaps,
|
||||
)
|
||||
return self._ungradable(
|
||||
learner_id,
|
||||
task_id,
|
||||
detail={_INTEGRITY_FLAG_KEY: None, _MISSING_SEQS_KEY: gaps},
|
||||
verdict=VERDICT_UNGRADABLE_INCOMPLETE,
|
||||
)
|
||||
|
||||
trace = self._trace_store.get_trace(learner_id, task_id)
|
||||
if not trace:
|
||||
logger.info(
|
||||
"grade gate: %s/%s empty trace — ungradable", learner_id, task_id
|
||||
)
|
||||
return self._ungradable(
|
||||
learner_id,
|
||||
task_id,
|
||||
detail={_INTEGRITY_FLAG_KEY: None, _MISSING_SEQS_KEY: []},
|
||||
verdict=VERDICT_UNGRADABLE_EMPTY,
|
||||
)
|
||||
|
||||
# -- Guarded path: digest (D-028) → prompt → D-020 4-layer defense.
|
||||
digest = compute_digest(trace)
|
||||
variant = self._lookup_variant(task_id)
|
||||
anchors_context = (
|
||||
_anchors_context(variant) if variant is not None else None
|
||||
)
|
||||
messages = [
|
||||
Message(role="system", content=SYSTEM_PROMPT),
|
||||
Message(
|
||||
role="user",
|
||||
content=render_trace_digest(digest, anchors_context=anchors_context),
|
||||
),
|
||||
]
|
||||
try:
|
||||
rubric = await structured_completion(
|
||||
self._provider,
|
||||
messages,
|
||||
model=self._model,
|
||||
schema=RubricScore,
|
||||
schema_hint=RUBRIC_SCORE_SCHEMA_HINT,
|
||||
)
|
||||
except StructuredOutputError as exc:
|
||||
# The trace was gradable but the model failed to produce valid
|
||||
# JSON within the D-020 budget (two attempts). Raise — the API
|
||||
# layer maps this to a 502 (assessor precedent). Persisting a
|
||||
# fabricated or partial grade here would violate the no-silent-
|
||||
# fallback rule: no credential-worthy record without a validated
|
||||
# RubricScore.
|
||||
raise StructuredOutputError(f"grading LLM failed for {task_id}: {exc}") from exc
|
||||
|
||||
logger.debug(
|
||||
"graded %s/%s: %s (model=%s)",
|
||||
learner_id,
|
||||
task_id,
|
||||
rubric.verdict,
|
||||
self._model,
|
||||
)
|
||||
return GradeRecord(
|
||||
learner_id=learner_id,
|
||||
task_id=task_id,
|
||||
variant_seed=variant.seed if variant is not None else None, # D-029
|
||||
digest=digest.model_dump(),
|
||||
scores=rubric.model_dump(),
|
||||
verdict=VERDICT_GRADED,
|
||||
model=self._model,
|
||||
created_at=datetime.now(tz=UTC),
|
||||
)
|
||||
|
||||
def _lookup_variant(self, task_id: str): # noqa: ANN202 - VariantRecord | None
|
||||
"""MH#4: resolve the graded task's variant (None when not variant-derived)."""
|
||||
if self._variant_store is None:
|
||||
return None
|
||||
return self._variant_store.get_by_task(task_id)
|
||||
|
||||
# ------------------------------------------------------------ gate record
|
||||
|
||||
@staticmethod
|
||||
def _ungradable(
|
||||
learner_id: str,
|
||||
task_id: str,
|
||||
*,
|
||||
detail: dict,
|
||||
verdict: str,
|
||||
) -> GradeRecord:
|
||||
"""Build a gate record: no digest (nothing was graded), gate detail
|
||||
surfaced in `scores` (the store allows an empty scores dict, but
|
||||
G-4 requires the gap list / flag reason surfaced — the detail IS the
|
||||
verdict's payload), model="none" (no LLM was involved; provenance
|
||||
stays honest).
|
||||
"""
|
||||
return GradeRecord(
|
||||
learner_id=learner_id,
|
||||
task_id=task_id,
|
||||
variant_seed=None,
|
||||
digest={},
|
||||
scores=detail,
|
||||
verdict=verdict,
|
||||
model="none",
|
||||
created_at=datetime.now(tz=UTC),
|
||||
)
|
||||
@@ -0,0 +1,246 @@
|
||||
"""Deterministic process-trace digest (D-028, REQ-3-004).
|
||||
|
||||
Pure compute — no LLM, no I/O. `compute_digest` reduces an ordered
|
||||
TelemetryEvent trace to a compact, bounded `TraceDigest` that is safe to
|
||||
embed in a grading prompt:
|
||||
|
||||
- FIXED fields + small histograms only; NO raw commands, NO file contents,
|
||||
NO payloads — the raw trace NEVER reaches the LLM (D-028), which also
|
||||
bounds the prompt-injection surface.
|
||||
- Tolerant to both live trace mixes: daemon-topology traces carry
|
||||
`activity` + `file_diff` kinds (workspace watcher), while REPL-driven
|
||||
traces carry `command`/`stdin`/`stdout`/`run_result`/`test_result`
|
||||
(P2 verification P1). Features derive from whatever kinds are present and
|
||||
never crash on absent kinds.
|
||||
|
||||
Feature semantics (conservative, deterministic):
|
||||
- test pass/fail counts + final status derive from `test_result` payloads
|
||||
when present, falling back to `run_result` exit codes (0 = pass).
|
||||
- an error/fix CYCLE = a failing run/test followed by >= 1 edit and then a
|
||||
later run/test (pass or fail) — the next observed result closes the cycle.
|
||||
- idle gaps = wall-clock gaps between consecutive events exceeding
|
||||
`idle_threshold_s` (default 120s): count + total seconds.
|
||||
- command category histogram classifies `command`-kind payloads: build /
|
||||
test / file / nav / debug / other.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..telemetry.models import TelemetryEvent
|
||||
|
||||
_IDLE_DEFAULT_S: float = 120.0
|
||||
|
||||
_TEST_HINTS = ("test", "pytest", "vitest", "jest", "mocha", "unittest", "go test", "npm test")
|
||||
_BUILD_HINTS = ("make", "npm run build", "pip install", "pnpm", "cargo build", "gcc", "tsc")
|
||||
_DEBUG_HINTS = ("gdb", "pdb", "print(", "debug", "strace", "ltrace", "curl", "ping")
|
||||
_NAV_HINTS = ("ls", "cd", "pwd", "cat ", "grep ", "find", "rg ", "tree", "head", "tail", "less")
|
||||
_FILE_HINTS = ("mv ", "cp ", "rm ", "mkdir", "touch", "chmod", "nano", "vim", "sed -i", "tee ")
|
||||
|
||||
|
||||
class TraceDigest(BaseModel):
|
||||
"""Compact, bounded, LLM-safe summary of a process trace (D-028).
|
||||
|
||||
Fixed fields + small histograms. Serializes well under 4 KB; contains no
|
||||
raw commands, file contents, or event payloads.
|
||||
"""
|
||||
|
||||
model_config = {"frozen": True}
|
||||
|
||||
event_count: int = Field(ge=0)
|
||||
session_duration_s: float = Field(ge=0.0)
|
||||
edit_count: int = Field(ge=0)
|
||||
command_count: int = Field(ge=0)
|
||||
run_count: int = Field(ge=0)
|
||||
test_pass_count: int = Field(ge=0)
|
||||
test_fail_count: int = Field(ge=0)
|
||||
final_test_status: str = Field(pattern="^(pass|fail|none)$")
|
||||
first_test_pass_offset_s: float | None = None
|
||||
error_fix_cycles: int = Field(ge=0)
|
||||
mean_fix_latency_s: float | None = None
|
||||
idle_gap_count: int = Field(ge=0)
|
||||
idle_gap_total_s: float = Field(ge=0.0)
|
||||
command_categories: dict[str, int] = Field(default_factory=dict)
|
||||
kind_histogram: dict[str, int] = Field(default_factory=dict)
|
||||
|
||||
|
||||
def _event_pass_status(event: TelemetryEvent) -> bool | None:
|
||||
"""True (pass) / False (fail) / None (not a result event) for one event."""
|
||||
payload = event.payload or {}
|
||||
if event.kind == "test_result":
|
||||
if "passed" in payload:
|
||||
return bool(payload["passed"])
|
||||
if "exit_code" in payload:
|
||||
return int(payload["exit_code"]) == 0
|
||||
if "status" in payload:
|
||||
return str(payload["status"]).lower() in ("pass", "passed", "ok", "success")
|
||||
return None
|
||||
if event.kind == "run_result":
|
||||
if "exit_code" in payload:
|
||||
return int(payload["exit_code"]) == 0
|
||||
if "ok" in payload:
|
||||
return bool(payload["ok"])
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _classify_command(text: str) -> str:
|
||||
lowered = text.lower()
|
||||
if any(h in lowered for h in _TEST_HINTS):
|
||||
return "test"
|
||||
if any(h in lowered for h in _BUILD_HINTS):
|
||||
return "build"
|
||||
if any(h in lowered for h in _DEBUG_HINTS):
|
||||
return "debug"
|
||||
if any(h in lowered for h in _NAV_HINTS):
|
||||
return "nav"
|
||||
if any(h in lowered for h in _FILE_HINTS):
|
||||
return "file"
|
||||
return "other"
|
||||
|
||||
|
||||
def _command_text(event: TelemetryEvent) -> str:
|
||||
payload = event.payload or {}
|
||||
return str(payload.get("cmd") or payload.get("command") or payload.get("line") or "")
|
||||
|
||||
|
||||
def compute_digest(
|
||||
trace: list[TelemetryEvent], *, idle_threshold_s: float = _IDLE_DEFAULT_S
|
||||
) -> TraceDigest:
|
||||
"""Reduce an ordered trace to a bounded digest. Never raises on odd input."""
|
||||
events = sorted(trace, key=lambda e: (e.seq, e.ts))
|
||||
if not events:
|
||||
return TraceDigest(
|
||||
event_count=0,
|
||||
session_duration_s=0.0,
|
||||
edit_count=0,
|
||||
command_count=0,
|
||||
run_count=0,
|
||||
test_pass_count=0,
|
||||
test_fail_count=0,
|
||||
final_test_status="none",
|
||||
first_test_pass_offset_s=None,
|
||||
error_fix_cycles=0,
|
||||
mean_fix_latency_s=None,
|
||||
idle_gap_count=0,
|
||||
idle_gap_total_s=0.0,
|
||||
command_categories={},
|
||||
kind_histogram={},
|
||||
)
|
||||
|
||||
kind_histogram = Counter(e.kind for e in events)
|
||||
start_ts = events[0].ts
|
||||
end_ts = events[-1].ts
|
||||
duration = max(0.0, (end_ts - start_ts).total_seconds())
|
||||
|
||||
edit_count = kind_histogram.get("file_diff", 0)
|
||||
command_count = kind_histogram.get("command", 0)
|
||||
run_count = kind_histogram.get("run_result", 0)
|
||||
|
||||
# Tests: prefer test_result events; fall back to run_result exit codes.
|
||||
test_statuses: list[tuple[TelemetryEvent, bool]] = []
|
||||
for e in events:
|
||||
if e.kind == "test_result":
|
||||
ok = _event_pass_status(e)
|
||||
if ok is not None:
|
||||
test_statuses.append((e, ok))
|
||||
if not test_statuses:
|
||||
for e in events:
|
||||
if e.kind == "run_result":
|
||||
ok = _event_pass_status(e)
|
||||
if ok is not None:
|
||||
test_statuses.append((e, ok))
|
||||
|
||||
test_pass_count = sum(1 for _, ok in test_statuses if ok)
|
||||
test_fail_count = len(test_statuses) - test_pass_count
|
||||
if not test_statuses:
|
||||
final_test_status = "none"
|
||||
else:
|
||||
final_test_status = "pass" if test_statuses[-1][1] else "fail"
|
||||
first_pass = next((e for e, ok in test_statuses if ok), None)
|
||||
first_pass_offset = (
|
||||
max(0.0, (first_pass.ts - start_ts).total_seconds()) if first_pass is not None else None
|
||||
)
|
||||
|
||||
# Error/fix cycles: a failing result starts a pending cycle; the NEXT
|
||||
# observed result closes it (regardless of outcome) — a fix attempt that
|
||||
# fails again is itself another iteration of debugging, so it closes the
|
||||
# previous cycle and opens a new one. Edits since the fail mark the
|
||||
# close as a genuine fix attempt; latency = first edit -> closing result.
|
||||
cycles = 0
|
||||
fix_latencies: list[float] = []
|
||||
pending_fail_ts: float | None = None # seconds since start
|
||||
edits_since_fail = 0
|
||||
first_edit_ts: float | None = None
|
||||
for e in events:
|
||||
t = max(0.0, (e.ts - start_ts).total_seconds())
|
||||
if e.kind == "file_diff":
|
||||
if pending_fail_ts is not None:
|
||||
if edits_since_fail == 0:
|
||||
first_edit_ts = t
|
||||
edits_since_fail += 1
|
||||
continue
|
||||
ok = _event_pass_status(e)
|
||||
if ok is None:
|
||||
continue
|
||||
if ok is False:
|
||||
if pending_fail_ts is not None and edits_since_fail > 0 and first_edit_ts is not None:
|
||||
# failed fix attempt: closes the previous cycle, opens a new one
|
||||
cycles += 1
|
||||
fix_latencies.append(t - first_edit_ts)
|
||||
pending_fail_ts = t
|
||||
edits_since_fail = 0
|
||||
first_edit_ts = None
|
||||
continue
|
||||
if ok is True and pending_fail_ts is not None:
|
||||
if edits_since_fail > 0 and first_edit_ts is not None:
|
||||
cycles += 1
|
||||
fix_latencies.append(t - first_edit_ts)
|
||||
pending_fail_ts = None
|
||||
edits_since_fail = 0
|
||||
first_edit_ts = None
|
||||
|
||||
mean_fix_latency = (
|
||||
sum(fix_latencies) / len(fix_latencies) if fix_latencies else None
|
||||
)
|
||||
|
||||
# Idle gaps between consecutive events.
|
||||
idle_gap_count = 0
|
||||
idle_gap_total = 0.0
|
||||
prev_ts = None
|
||||
for e in events:
|
||||
if prev_ts is not None:
|
||||
gap = (e.ts - prev_ts).total_seconds()
|
||||
if gap > idle_threshold_s:
|
||||
idle_gap_count += 1
|
||||
idle_gap_total += gap
|
||||
prev_ts = e.ts
|
||||
|
||||
# Command category histogram (command-kind events only).
|
||||
categories: Counter[str] = Counter()
|
||||
for e in events:
|
||||
if e.kind == "command":
|
||||
categories[_classify_command(_command_text(e))] += 1
|
||||
|
||||
return TraceDigest(
|
||||
event_count=len(events),
|
||||
session_duration_s=round(duration, 3),
|
||||
edit_count=edit_count,
|
||||
command_count=command_count,
|
||||
run_count=run_count,
|
||||
test_pass_count=test_pass_count,
|
||||
test_fail_count=test_fail_count,
|
||||
final_test_status=final_test_status,
|
||||
first_test_pass_offset_s=(
|
||||
round(first_pass_offset, 3) if first_pass_offset is not None else None
|
||||
),
|
||||
error_fix_cycles=cycles,
|
||||
mean_fix_latency_s=(round(mean_fix_latency, 3) if mean_fix_latency is not None else None),
|
||||
idle_gap_count=idle_gap_count,
|
||||
idle_gap_total_s=round(idle_gap_total, 3),
|
||||
command_categories=dict(sorted(categories.items())),
|
||||
kind_histogram=dict(sorted(kind_histogram.items())),
|
||||
)
|
||||
@@ -0,0 +1,237 @@
|
||||
"""GradeStore — grade persistence protocol + SQLite implementation (REQ-3-004, D-027).
|
||||
|
||||
Postgres-migration-ready (D-027): the protocol is the only surface the
|
||||
grading engine and API layers touch; swapping SQLiteGradeStore for a
|
||||
Postgres-backed implementation must not change call sites. The
|
||||
`grade_record` table uses only portable column types (str / JSON /
|
||||
datetime), so the same SQLModel schema stands up unchanged on Postgres.
|
||||
|
||||
Upsert, NOT append: (learner_id, task_id) is the grade identity — one row
|
||||
per learner per task holding the LATEST grade. `save` overwrites the whole
|
||||
row when the pair already exists, so a regrade replaces scores, verdict,
|
||||
created_at, digest, model and variant_seed wholesale. That is deliberately
|
||||
the opposite of TraceStore.append's dedup-keep-first contract: a trace is an
|
||||
append-only event log, a grade is latest-state, so the engine can re-grade
|
||||
a task idempotently as its rubric or input evolves.
|
||||
|
||||
Concurrency (a-3): the engine enables WAL + synchronous=NORMAL and a busy
|
||||
timeout at connection time, so a regrade writer and API readers do not hit
|
||||
`database is locked` on the single-box pilot.
|
||||
|
||||
`created_at` contract: callers stamp UTC (datetime.now(UTC)); SQLite stores
|
||||
it naive and the read paths re-label it tz-aware UTC (same boundary
|
||||
normalization as TelemetryEvent.ts, so the contract holds on any backend).
|
||||
|
||||
Boundary (D-027): `grading/` never imports `agents/` / `api/`; this module
|
||||
imports config only.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy import JSON, Index
|
||||
from sqlalchemy.orm import validates
|
||||
from sqlmodel import Field, Session, SQLModel, create_engine, select
|
||||
|
||||
from ..config import Settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GradeRecord(SQLModel, table=True):
|
||||
"""A persisted grade; (learner_id, task_id) is the PK — latest wins.
|
||||
|
||||
Written by the grading engine (one save per grade attempt), read by the
|
||||
API layer through the GradeStore protocol. Constraint enforcement
|
||||
mirrors TelemetryEvent: sqlmodel 0.0.42's metaclass drops pydantic
|
||||
constraints on table models, so SQLAlchemy `@validates` hooks enforce
|
||||
instead and the column types stay Postgres-ready (D-027).
|
||||
|
||||
Field contract:
|
||||
learner_id — non-empty learner identifier (same id space as traces).
|
||||
task_id — non-empty task identifier; grade identity is the
|
||||
(learner_id, task_id) pair — the same pair as trace
|
||||
identity, so a grade is keyed by the exact trace it
|
||||
was computed from.
|
||||
variant_seed — task-variant seed (D-029); None when the graded task
|
||||
is not variant-derived. Since Phase 4 the engine
|
||||
stamps the graded variant's seed here (MH#4) and the
|
||||
template's difficulty anchors ship to the grader
|
||||
prompt — this column is the audit join for that.
|
||||
digest — compact deterministic trace digest (D-028) that fed
|
||||
the rubric prompt; persisted for auditability so the
|
||||
LLM's input stays reproducible.
|
||||
scores — validated rubric scores (per-criterion 0-4,
|
||||
strengths, gaps); JSON dict. An empty dict is legal
|
||||
(e.g. an UNGRADABLE_TRACE_INCOMPLETE record carries a
|
||||
verdict but no scores).
|
||||
verdict — first-class verdict string (rubric verdict or
|
||||
UNGRADABLE_TRACE_INCOMPLETE); non-empty.
|
||||
model — provider model that produced the scores (provenance).
|
||||
created_at — UTC grade timestamp; a regrade replaces it (latest
|
||||
save wins).
|
||||
"""
|
||||
|
||||
__tablename__ = "grade_record"
|
||||
# The composite PK covers (learner_id, task_id) point lookups; this
|
||||
# secondary index covers list_for_learner ordered by created_at without
|
||||
# a sort step (Postgres migration target D-027).
|
||||
__table_args__ = (
|
||||
Index("ix_grade_record_learner_created", "learner_id", "created_at"),
|
||||
)
|
||||
|
||||
learner_id: str = Field(primary_key=True)
|
||||
task_id: str = Field(primary_key=True)
|
||||
# None only for non-variant tasks (MH#4 stamps variant seeds since P4).
|
||||
variant_seed: str | None = Field(default=None)
|
||||
# JSON columns: stored as TEXT on SQLite, native JSONB on Postgres (D-027).
|
||||
digest: dict[str, Any] = Field(default_factory=dict, sa_type=JSON)
|
||||
scores: dict[str, Any] = Field(default_factory=dict, sa_type=JSON)
|
||||
verdict: str
|
||||
model: str
|
||||
created_at: datetime
|
||||
|
||||
@validates("learner_id", "task_id")
|
||||
def _ids_non_empty(self, key: str, value: str) -> str:
|
||||
if not value:
|
||||
raise ValueError(f"{key} must be a non-empty identifier")
|
||||
return value
|
||||
|
||||
@validates("verdict")
|
||||
def _verdict_non_empty(self, key: str, value: str) -> str:
|
||||
if not value:
|
||||
raise ValueError(f"{key} must be a non-empty verdict string")
|
||||
return value
|
||||
|
||||
|
||||
class GradeStore(Protocol):
|
||||
"""Persistence contract for latest-state grades per (learner_id, task_id).
|
||||
|
||||
Implemented by SQLiteGradeStore (v0.3, D-027); a Postgres implementation
|
||||
must satisfy the same surface.
|
||||
"""
|
||||
|
||||
def save(self, grade: GradeRecord) -> None:
|
||||
"""Persist a grade. UPSERT on (learner_id, task_id): a regrade with
|
||||
the same pair REPLACES the stored row wholesale — the latest grade
|
||||
wins. NOT append-only; contrast TraceStore.append, which is
|
||||
dedup-keep-first for at-least-once ingest.
|
||||
"""
|
||||
...
|
||||
|
||||
def get(self, learner_id: str, task_id: str) -> GradeRecord | None:
|
||||
"""Latest stored grade for the pair; None when none exists.
|
||||
|
||||
Detached from any DB session — safe to pass across layers.
|
||||
"""
|
||||
...
|
||||
|
||||
def list_for_learner(self, learner_id: str) -> list[GradeRecord]:
|
||||
"""All stored grades for the learner, ordered by created_at
|
||||
ascending (chronological). Empty list when the learner has none.
|
||||
"""
|
||||
...
|
||||
|
||||
def close(self) -> None:
|
||||
"""Release DB connections. Store must not be used after close."""
|
||||
...
|
||||
|
||||
|
||||
def _sqlite_connect(dbapi_connection: sqlite3.Connection, _: object) -> None:
|
||||
"""Per-connection pragma setup (a-3). Mirrors telemetry/store.py.
|
||||
|
||||
journal_mode=WAL — readers never block the single writer.
|
||||
synchronous=NORMAL — safe in WAL mode, avoids full fsync-per-commit.
|
||||
busy_timeout=5000 — retry briefly under contention instead of
|
||||
`OperationalError: database is locked`.
|
||||
"""
|
||||
cursor = dbapi_connection.cursor()
|
||||
cursor.execute("PRAGMA journal_mode=WAL")
|
||||
cursor.execute("PRAGMA synchronous=NORMAL")
|
||||
cursor.execute("PRAGMA busy_timeout=5000")
|
||||
cursor.close()
|
||||
|
||||
|
||||
def _as_utc(ts: datetime) -> datetime:
|
||||
"""Timestamps travel as naive datetime on SQLite, tz-aware elsewhere.
|
||||
|
||||
SQLite (via SQLModel) drops tzinfo; Postgres TIMESTAMP WITH TIME ZONE
|
||||
keeps it. Normalizing on the read path makes the store's contract
|
||||
tz-aware UTC regardless of the backend (D-027).
|
||||
"""
|
||||
if ts.tzinfo is None:
|
||||
return ts.replace(tzinfo=UTC) # naive UTC read-side: label as UTC
|
||||
return ts.astimezone(UTC)
|
||||
|
||||
|
||||
class SQLiteGradeStore:
|
||||
"""SQLite-backed GradeStore (SQLModel). Second protocol-wrapped store
|
||||
of the D-027 family (first: SQLiteTraceStore).
|
||||
"""
|
||||
|
||||
def __init__(self, db_path: Path | None = None) -> None:
|
||||
self._db_path: Path = db_path if db_path is not None else Settings().db_path
|
||||
self._engine = create_engine(f"sqlite:///{self._db_path}")
|
||||
sa.event.listen(self._engine, "connect", _sqlite_connect)
|
||||
SQLModel.metadata.create_all(self._engine)
|
||||
|
||||
@contextmanager
|
||||
def _session(self) -> Iterator[Session]:
|
||||
# expire_on_commit=False: identical session behavior to
|
||||
# SQLiteTraceStore. save() discards the merged instance and the read
|
||||
# paths never commit, but a uniform flag across the D-027 stores
|
||||
# keeps their detachment guarantees from diverging.
|
||||
with Session(self._engine, expire_on_commit=False) as session:
|
||||
yield session
|
||||
|
||||
def save(self, grade: GradeRecord) -> None:
|
||||
# `merge` = SELECT-by-PK then UPDATE or INSERT — exactly the upsert
|
||||
# contract. The trace store deliberately avoids merge (its append is
|
||||
# dedup-keep-first); here latest-wins IS the contract, so merge is
|
||||
# the right tool. The caller's object is never attached to the
|
||||
# session and stays usable (unexpired) after save.
|
||||
with self._session() as session:
|
||||
session.merge(grade)
|
||||
session.commit()
|
||||
logger.debug(
|
||||
"grade saved (regrade overwrites): %s/%s verdict=%s model=%s",
|
||||
grade.learner_id,
|
||||
grade.task_id,
|
||||
grade.verdict,
|
||||
grade.model,
|
||||
)
|
||||
|
||||
def get(self, learner_id: str, task_id: str) -> GradeRecord | None:
|
||||
with self._session() as session:
|
||||
record = session.get(GradeRecord, (learner_id, task_id))
|
||||
if record is None:
|
||||
return None
|
||||
record.created_at = _as_utc(record.created_at)
|
||||
# Detach from the session: callers must not depend on
|
||||
# open-session ORM magic (lazy loads fail once it closes).
|
||||
session.expunge(record)
|
||||
return record
|
||||
|
||||
def list_for_learner(self, learner_id: str) -> list[GradeRecord]:
|
||||
with self._session() as session:
|
||||
stmt = (
|
||||
select(GradeRecord)
|
||||
.where(GradeRecord.learner_id == learner_id)
|
||||
# Chronological; task_id is a deterministic tie-break for
|
||||
# grades stamped within the same instant.
|
||||
.order_by(GradeRecord.created_at, GradeRecord.task_id)
|
||||
)
|
||||
results = session.exec(stmt).all()
|
||||
for row in results:
|
||||
row.created_at = _as_utc(row.created_at)
|
||||
session.expunge(row)
|
||||
return list(results)
|
||||
|
||||
def close(self) -> None:
|
||||
self._engine.dispose()
|
||||
@@ -0,0 +1,24 @@
|
||||
"""Identity engine package — provider protocol + store (REQ-5-003, D-042)."""
|
||||
|
||||
from .base import (
|
||||
AgeBand,
|
||||
IdentityProvider,
|
||||
IdentityStatus,
|
||||
IdentitySubmission,
|
||||
IdentityVerdict,
|
||||
)
|
||||
from .mock import MockIdentityProvider, derive_age_band
|
||||
from .store import IdentityRecord, IdentityStore, SQLiteIdentityStore
|
||||
|
||||
__all__ = [
|
||||
"AgeBand",
|
||||
"IdentityStatus",
|
||||
"IdentitySubmission",
|
||||
"IdentityVerdict",
|
||||
"IdentityProvider",
|
||||
"IdentityRecord",
|
||||
"IdentityStore",
|
||||
"SQLiteIdentityStore",
|
||||
"MockIdentityProvider",
|
||||
"derive_age_band",
|
||||
]
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Identity verification protocol (REQ-5-003, D-042, A-303).
|
||||
|
||||
Provider-agnostic like LLMProvider/VoiceProvider (D-014/D-030): a narrow
|
||||
protocol the identity API composes via DI, a deterministic mock, and a
|
||||
future real KYC vendor (Stripe Identity / Persona / Onfido class) that
|
||||
drops in without API changes. PII rules (A-305): the provider sees
|
||||
document REFERENCES, never raw documents; verdicts carry a mock marker
|
||||
(A-304) so downstream surfaces never display mock-verified as
|
||||
production-verified.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal, Protocol, runtime_checkable
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
AgeBand = Literal["16-17", "18+"]
|
||||
IdentityStatus = Literal["pending", "verified", "rejected"]
|
||||
|
||||
|
||||
class IdentitySubmission(BaseModel):
|
||||
"""What a learner submits: derived data + document refs only.
|
||||
|
||||
`date_of_birth` is a REAL date (the provider derives the age band) but
|
||||
raw DOB is NEVER persisted — only the derived band (A-305). Document
|
||||
refs are opaque handles (upload ids), never contents.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
learner_id: str = Field(min_length=1)
|
||||
date_of_birth: str = Field(description="ISO date; used to derive age_band, never stored")
|
||||
document_refs: list[str] = Field(
|
||||
default_factory=list,
|
||||
description="Opaque upload handles; raw documents are never stored",
|
||||
)
|
||||
|
||||
|
||||
class IdentityVerdict(BaseModel):
|
||||
"""Provider verdict — what gets stored + surfaced."""
|
||||
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
status: IdentityStatus
|
||||
age_band: AgeBand | None = None
|
||||
provider: str
|
||||
#: A-304 honesty: mock verdicts carry mock=True forever — downstream
|
||||
#: surfaces must never treat a mock verdict as production-verified.
|
||||
mock: bool = True
|
||||
detail: str = ""
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class IdentityProvider(Protocol):
|
||||
"""The KYC port: submit → (poll) → verdict. Never imports api/."""
|
||||
|
||||
async def submit(self, submission: IdentitySubmission) -> str:
|
||||
"""Start verification; returns a submission id (minted once)."""
|
||||
...
|
||||
|
||||
async def poll(self, submission_id: str) -> IdentityVerdict:
|
||||
"""Fetch the (possibly pending) verdict for a submission."""
|
||||
...
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Deterministic mock identity provider (REQ-5-003, A-303).
|
||||
|
||||
Approve-on-policy: every submission verifies unless the caller scripts a
|
||||
rejection (by learner id) or the derived age band fails the floor
|
||||
(under-16 → rejected with an age detail). Verdicts are mock-marked (A-304)
|
||||
— the marker rides every verdict so no downstream surface can ever
|
||||
display mock-verified as production-verified. Never calls the network.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import itertools
|
||||
import secrets
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from .base import IdentitySubmission, IdentityVerdict
|
||||
|
||||
|
||||
def derive_age_band(date_of_birth: str, today: datetime | None = None) -> str:
|
||||
"""Derive the age band from an ISO date. Under-16 returns "under-16".
|
||||
|
||||
Pure + deterministic; used by the store test and the API alike.
|
||||
"""
|
||||
dob = datetime.fromisoformat(date_of_birth)
|
||||
now = today or datetime.now(UTC)
|
||||
age = now.year - dob.year - (
|
||||
(now.month, now.day) < (dob.month, dob.day)
|
||||
)
|
||||
if age < 16:
|
||||
return "under-16"
|
||||
if age < 18:
|
||||
return "16-17"
|
||||
return "18+"
|
||||
|
||||
|
||||
class MockIdentityProvider:
|
||||
"""Scriptable, deterministic; no network, no vendor calls.
|
||||
|
||||
Submission ids are minted UNIQUELY per submit() call (a monotonic
|
||||
counter + the per-process seed from `secrets`): the id is the PK of
|
||||
the insert-only IdentityStore, and a deterministic id derived from
|
||||
(learner_id, date_of_birth) collides on any resubmit-after-terminal
|
||||
(e.g. a rejected learner retrying with the same DOB) — the store
|
||||
surfaces IntegrityError and the API would 500 (cross-phase P0,
|
||||
final review). Uniqueness per call is the contract; determinism of
|
||||
VERDICTS (what tests actually pin) is preserved — poll() derives the
|
||||
band purely from the stored submission.
|
||||
"""
|
||||
|
||||
def __init__(self, reject_learners: set[str] | None = None) -> None:
|
||||
self._submissions: dict[str, IdentitySubmission] = {}
|
||||
self._reject_learners = reject_learners or set()
|
||||
# Per-process nonce: ids are opaque handles (A-305) — never
|
||||
# derived from PII. Counter + nonce keeps ids unique within and
|
||||
# across provider instances on one box.
|
||||
self._nonce = secrets.randbits(32)
|
||||
self._counter = itertools.count()
|
||||
|
||||
async def submit(self, submission: IdentitySubmission) -> str:
|
||||
submission_id = f"idc-{self._nonce:08x}{next(self._counter):08x}"
|
||||
self._submissions[submission_id] = submission
|
||||
return submission_id
|
||||
|
||||
async def poll(self, submission_id: str) -> IdentityVerdict:
|
||||
submission = self._submissions.get(submission_id)
|
||||
if submission is None:
|
||||
return IdentityVerdict(
|
||||
status="rejected",
|
||||
provider="mock",
|
||||
mock=True,
|
||||
detail="unknown submission id",
|
||||
)
|
||||
if submission.learner_id in self._reject_learners:
|
||||
return IdentityVerdict(
|
||||
status="rejected",
|
||||
provider="mock",
|
||||
mock=True,
|
||||
detail="scripted rejection (test)",
|
||||
)
|
||||
band = derive_age_band(submission.date_of_birth)
|
||||
if band == "under-16":
|
||||
return IdentityVerdict(
|
||||
status="rejected",
|
||||
provider="mock",
|
||||
mock=True,
|
||||
detail="under 16 — the AI school floor is 16+ (COPPA avoidance)",
|
||||
)
|
||||
return IdentityVerdict(
|
||||
status="verified",
|
||||
age_band=band, # type: ignore[arg-type]
|
||||
provider="mock",
|
||||
mock=True,
|
||||
detail="mock verdict — not production verification",
|
||||
)
|
||||
@@ -0,0 +1,214 @@
|
||||
"""IdentityStore — verification records (REQ-5-003, D-042, D-027 FIFTH store).
|
||||
|
||||
Insert-only + latest-per-learner lookup, modeled on the DefenseStore
|
||||
conventions: WAL + synchronous=NORMAL + busy_timeout + foreign_keys=ON
|
||||
pragmas at connect time, portable column types (str/datetime/JSON) for
|
||||
Postgres parity, @validates hooks for constraints sqlmodel's metaclass
|
||||
drops, tz-aware→naive→tz-aware boundary normalization.
|
||||
|
||||
PII contract (A-305): stores the DERIVED age_band (16-17 | 18+), NEVER a
|
||||
raw date of birth; document_refs are opaque handles, NEVER contents.
|
||||
Verdict provenance is audit data: every record carries provider + the
|
||||
mock marker (A-304) so downstream surfaces can label unverified state
|
||||
honestly.
|
||||
|
||||
Insert-only growth is fine at pilot scale (a-12): learner_id indexed,
|
||||
latest-per-learner lookup, no compaction pre-vendor.
|
||||
|
||||
Boundary (D-027): `identity/` never imports `agents/` / `api/`; this
|
||||
module imports config only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
from sqlalchemy import event, text
|
||||
from sqlalchemy.types import JSON, String
|
||||
from sqlmodel import Field, Session, SQLModel, create_engine
|
||||
|
||||
from ..config import Settings
|
||||
from .base import IdentityStatus
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class IdentityRecord(SQLModel, table=True):
|
||||
"""One verification submission's lifecycle + verdict provenance."""
|
||||
|
||||
__tablename__ = "identity_record"
|
||||
|
||||
#: PK = the submission id minted once by the provider's submit().
|
||||
id: str = Field(primary_key=True)
|
||||
learner_id: str = Field(index=True)
|
||||
# Bare Literal annotations crash sqlmodel's column inference; explicit
|
||||
# sa_type + the validates hook below give the same contract
|
||||
# (VARCHAR column, Literal-rejected values — DefenseStore pattern).
|
||||
status: IdentityStatus = Field(default="pending", sa_type=String)
|
||||
provider: str
|
||||
#: Verdict provenance: the provider's raw verdict (mock-marked).
|
||||
verdict: dict[str, Any] = Field(default_factory=dict, sa_type=JSON)
|
||||
#: DERIVED band only — raw DOB is never persisted (A-305).
|
||||
age_band: str | None = Field(default=None, sa_type=String)
|
||||
#: Opaque document handles — raw documents never stored (A-305).
|
||||
document_refs: list[str] = Field(default_factory=list, sa_type=JSON)
|
||||
submitted_at: datetime
|
||||
verified_at: datetime | None = Field(default=None)
|
||||
|
||||
@property
|
||||
def mock(self) -> bool:
|
||||
"""A-304: the mock marker rides every surface (record + API)."""
|
||||
return bool(self.verdict.get("mock", True))
|
||||
|
||||
def _validate(self) -> None:
|
||||
if not self.id or not self.learner_id:
|
||||
raise ValueError("id and learner_id must be non-empty")
|
||||
if self.status not in ("pending", "verified", "rejected"):
|
||||
raise ValueError(f"invalid identity status {self.status!r}")
|
||||
# D3 (verifier): a stored band must be canonical or None (pending).
|
||||
# The gate fails closed on anything else; the store refuses to
|
||||
# create it in the first place.
|
||||
if self.age_band is not None and self.age_band not in (
|
||||
"16-17",
|
||||
"18+",
|
||||
"under-16",
|
||||
):
|
||||
raise ValueError(f"invalid age_band {self.age_band!r}")
|
||||
|
||||
def _normalize(self) -> None:
|
||||
self.submitted_at = _as_utc(self.submitted_at)
|
||||
if self.verified_at is not None:
|
||||
self.verified_at = _as_utc(self.verified_at)
|
||||
|
||||
|
||||
def _as_utc(ts: datetime) -> datetime:
|
||||
"""SQLite stores naive; read paths re-label tz-aware UTC (D-027 pattern)."""
|
||||
if ts.tzinfo is None:
|
||||
return ts.replace(tzinfo=UTC)
|
||||
return ts
|
||||
|
||||
|
||||
def _sqlite_connect(dbapi_connection: object, _: object) -> None:
|
||||
"""Per-connection pragmas — mirrors the other D-027 stores."""
|
||||
cursor = dbapi_connection.cursor() # type: ignore[attr-defined]
|
||||
cursor.execute("PRAGMA journal_mode=WAL")
|
||||
cursor.execute("PRAGMA synchronous=NORMAL")
|
||||
cursor.execute("PRAGMA busy_timeout=5000")
|
||||
cursor.execute("PRAGMA foreign_keys=ON")
|
||||
cursor.close() # type: ignore[attr-defined]
|
||||
|
||||
|
||||
class IdentityStore(Protocol):
|
||||
"""Persistence contract for identity records."""
|
||||
|
||||
def insert(self, record: IdentityRecord) -> IdentityRecord:
|
||||
"""INSERT-ONLY: a duplicate id raises IntegrityError (surfaced, not
|
||||
swallowed — a submission id is minted once)."""
|
||||
...
|
||||
|
||||
def get(self, submission_id: str) -> IdentityRecord | None:
|
||||
"""Point lookup by submission id."""
|
||||
...
|
||||
|
||||
def latest_for_learner(self, learner_id: str) -> IdentityRecord | None:
|
||||
"""Newest record for the learner (or None)."""
|
||||
...
|
||||
|
||||
def count_pending_for_learner(self, learner_id: str) -> int:
|
||||
"""G-13: active pending submissions (cap = 1)."""
|
||||
...
|
||||
|
||||
def mark_verified(
|
||||
self, submission_id: str, verdict: dict[str, Any], age_band: str | None
|
||||
) -> IdentityRecord | None:
|
||||
"""Terminal transition (verified or rejected): stamp + store the
|
||||
provider verdict + derived band. Unknown id → None."""
|
||||
...
|
||||
|
||||
|
||||
class SQLiteIdentityStore:
|
||||
"""SQLite implementation of IdentityStore (D-027)."""
|
||||
|
||||
def __init__(self, db_path: Path | None = None) -> None:
|
||||
self._db_path: Path = db_path if db_path is not None else Settings().db_path
|
||||
self._engine = create_engine(
|
||||
f"sqlite:///{self._db_path}",
|
||||
connect_args={"check_same_thread": False},
|
||||
)
|
||||
event.listen(self._engine, "connect", _sqlite_connect)
|
||||
SQLModel.metadata.create_all(self._engine)
|
||||
|
||||
def insert(self, record: IdentityRecord) -> IdentityRecord:
|
||||
record._validate()
|
||||
record._normalize()
|
||||
with Session(self._engine) as session:
|
||||
session.add(record)
|
||||
session.commit() # IntegrityError SURFACES (insert-only, minted-once)
|
||||
session.refresh(record)
|
||||
return record
|
||||
|
||||
def get(self, submission_id: str) -> IdentityRecord | None:
|
||||
with Session(self._engine) as session:
|
||||
rec = session.get(IdentityRecord, submission_id)
|
||||
if rec is None:
|
||||
return None
|
||||
session.refresh(rec)
|
||||
rec.submitted_at = _as_utc(rec.submitted_at)
|
||||
if rec.verified_at is not None:
|
||||
rec.verified_at = _as_utc(rec.verified_at)
|
||||
return rec
|
||||
|
||||
def latest_for_learner(self, learner_id: str) -> IdentityRecord | None:
|
||||
with Session(self._engine) as session:
|
||||
# D4 (verifier): submitted_at alone can tie at microsecond
|
||||
# resolution — sqlite rowid breaks the tie deterministically
|
||||
# (last inserted wins, mirroring insert-only chronology).
|
||||
# rowid is a SQLite physical column, not a SQLModel field — it
|
||||
# rides the query as raw text.
|
||||
rec = (
|
||||
session.query(IdentityRecord)
|
||||
.filter(IdentityRecord.learner_id == learner_id)
|
||||
.order_by(
|
||||
IdentityRecord.submitted_at.desc(),
|
||||
text("rowid DESC"),
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if rec is not None:
|
||||
rec.submitted_at = _as_utc(rec.submitted_at)
|
||||
if rec.verified_at is not None:
|
||||
rec.verified_at = _as_utc(rec.verified_at)
|
||||
return rec
|
||||
|
||||
def count_pending_for_learner(self, learner_id: str) -> int:
|
||||
with Session(self._engine) as session:
|
||||
return (
|
||||
session.query(IdentityRecord)
|
||||
.filter(
|
||||
IdentityRecord.learner_id == learner_id,
|
||||
IdentityRecord.status == "pending",
|
||||
)
|
||||
.count()
|
||||
)
|
||||
|
||||
def mark_verified(
|
||||
self, submission_id: str, verdict: dict[str, Any], age_band: str | None
|
||||
) -> IdentityRecord | None:
|
||||
with Session(self._engine) as session:
|
||||
rec = session.get(IdentityRecord, submission_id)
|
||||
if rec is None:
|
||||
return None
|
||||
rec.verdict = verdict
|
||||
rec.age_band = age_band
|
||||
rec.verified_at = datetime.now(UTC)
|
||||
rec.status = "verified" if verdict.get("status") == "verified" else "rejected"
|
||||
rec._validate() # D3: transitions validate like inserts
|
||||
session.commit()
|
||||
session.refresh(rec)
|
||||
return rec
|
||||
|
||||
def close(self) -> None:
|
||||
self._engine.dispose()
|
||||
@@ -14,14 +14,29 @@ from .agents.session import InMemorySessionStore
|
||||
from .api import (
|
||||
assessment_router,
|
||||
chat_router,
|
||||
defense_router,
|
||||
lab_router,
|
||||
mentor_router,
|
||||
proctor_router,
|
||||
sandboxes_router,
|
||||
telemetry_router,
|
||||
variants_router,
|
||||
)
|
||||
from .config import Settings
|
||||
from .grading.engine import GradingEngine
|
||||
from .grading.store import SQLiteGradeStore
|
||||
from .identity.mock import MockIdentityProvider
|
||||
from .identity.store import SQLiteIdentityStore
|
||||
from .llm import create_provider
|
||||
from .sandbox import SandboxManager, UnshareBackend
|
||||
from .telemetry.ingest import TraceIntegrityMap
|
||||
from .telemetry.store import SQLiteTraceStore
|
||||
from .telemetry.tokens import IngestToken
|
||||
from .variants.generator import VariantGenerator
|
||||
from .variants.store import SQLiteVariantStore
|
||||
from .voice.defense_store import SQLiteDefenseStore
|
||||
from .voice.factory import voice_provider_from_settings
|
||||
from .voice.mock import MockVoiceProvider
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -39,7 +54,11 @@ def create_app(settings: Settings | None = None) -> FastAPI:
|
||||
timeout = httpx.Timeout(connect=10.0, read=300.0, write=30.0, pool=10.0)
|
||||
app.state.http_client = httpx.AsyncClient(timeout=timeout)
|
||||
app.state.settings = settings
|
||||
app.state.provider = create_provider(settings, app.state.http_client)
|
||||
# State-injection override (same pattern as the stores): tests may
|
||||
# pre-set app.state.provider with a scripted mock; only construct the
|
||||
# configured provider when none is present.
|
||||
if getattr(app.state, "provider", None) is None:
|
||||
app.state.provider = create_provider(settings, app.state.http_client)
|
||||
app.state.session_store = InMemorySessionStore()
|
||||
app.state.agent_registry = AgentRegistry()
|
||||
register_builtin_agents(app.state.agent_registry)
|
||||
@@ -48,12 +67,140 @@ def create_app(settings: Settings | None = None) -> FastAPI:
|
||||
# app.state. Tests may pre-set app.state.sandbox_manager (dependency
|
||||
# override by state injection) to swap the backend; the lifespan then
|
||||
# adopts it instead of constructing the real UnshareBackend one.
|
||||
# v0.4.7 (P1): the ingest token is created BEFORE the manager (the
|
||||
# manager MINTS into NC_INGEST_URL; the WS endpoint VERIFIES) and
|
||||
# shared via app.state — one instance, so the per-boot random secret
|
||||
# (unset AI_TELEMETRY_INGEST_SECRET) agrees on both sides.
|
||||
if getattr(app.state, "ingest_token", None) is None:
|
||||
app.state.ingest_token = IngestToken(settings.telemetry_ingest_secret)
|
||||
manager = getattr(app.state, "sandbox_manager", None)
|
||||
if manager is None:
|
||||
manager = SandboxManager(backend=UnshareBackend(), settings=settings)
|
||||
manager = SandboxManager(
|
||||
backend=UnshareBackend(),
|
||||
settings=settings,
|
||||
ingest_token=app.state.ingest_token,
|
||||
)
|
||||
app.state.sandbox_manager = manager
|
||||
await manager.start() # a-1: reap on-disk orphans from a previous process
|
||||
|
||||
# Telemetry persistence (REQ-3-003, D-027): TraceStore wired through
|
||||
# app.state. Tests may pre-set app.state.trace_store (state-injection
|
||||
# override, same pattern as sandbox_manager) — the lifespan adopts it.
|
||||
trace_store = getattr(app.state, "trace_store", None)
|
||||
if trace_store is None:
|
||||
settings.db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
trace_store = SQLiteTraceStore(db_path=settings.db_path)
|
||||
app.state.trace_store = trace_store
|
||||
# Trace-integrity flags (G-3 INCOMPLETE_FLOODED): process-local map is
|
||||
# intentional (D-019 registry precedent); the lifespan owns it so the
|
||||
# grader and the ingest endpoint share one instance.
|
||||
if getattr(app.state, "trace_integrity", None) is None:
|
||||
app.state.trace_integrity = TraceIntegrityMap()
|
||||
|
||||
# Variant generation (REQ-3-005): VariantStore from the same
|
||||
# SQLite file as traces/grades (D-027), one VariantGenerator singleton
|
||||
# wired through app.state — the generator receives store + provider
|
||||
# via constructor DI and knows nothing of FastAPI (api/ composes it,
|
||||
# same pattern as GradingEngine). Tests may pre-set
|
||||
# app.state.variant_store / app.state.variant_generator (the same
|
||||
# state-injection override); the lifespan adopts a pre-set store but
|
||||
# NEVER rebuilds a pre-set generator (its provider binding is part
|
||||
# of the test fixture).
|
||||
# ORDER NOTE: built BEFORE the grading engine — the engine takes the
|
||||
# variant store (Phase 4 MH#4: variant anchors ship to the grader
|
||||
# prompt; variant_seed stamped on graded records).
|
||||
variant_store = getattr(app.state, "variant_store", None)
|
||||
if variant_store is None:
|
||||
settings.db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
variant_store = SQLiteVariantStore(db_path=settings.db_path)
|
||||
app.state.variant_store = variant_store
|
||||
if getattr(app.state, "variant_generator", None) is None:
|
||||
app.state.variant_generator = VariantGenerator(
|
||||
variant_store,
|
||||
app.state.provider,
|
||||
model=settings.model,
|
||||
)
|
||||
|
||||
# Oral defense (REQ-3-006): DefenseStore (same SQLite file) + the
|
||||
# mock-first voice provider (D-030) + the seventh Examiner agent.
|
||||
# Tests may pre-set app.state.defense_store / voice_provider /
|
||||
# examiner_agent (state-injection override; never rebuilt if pre-set).
|
||||
defense_store = getattr(app.state, "defense_store", None)
|
||||
if defense_store is None:
|
||||
defense_store = SQLiteDefenseStore(db_path=settings.db_path)
|
||||
app.state.defense_store = defense_store
|
||||
if getattr(app.state, "voice_provider", None) is None:
|
||||
# G-11 (boot survival): a misconfigured real provider must never
|
||||
# crash the unattended deploy — fall back to mock loudly. The
|
||||
# mock provider's descriptor honestly reports mode='mock' so the
|
||||
# UI badge cannot lie about which path is live.
|
||||
try:
|
||||
app.state.voice_provider = voice_provider_from_settings(
|
||||
settings, app.state.http_client
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"voice provider %r unavailable (%s); falling back to mock "
|
||||
"— fix the AI_VOICE_* settings and restart",
|
||||
settings.voice_provider,
|
||||
exc,
|
||||
)
|
||||
app.state.voice_provider = MockVoiceProvider()
|
||||
if getattr(app.state, "examiner_agent", None) is None:
|
||||
from .agents.examiner import ExaminerAgent
|
||||
|
||||
app.state.examiner_agent = ExaminerAgent(app.state.provider, settings)
|
||||
|
||||
# Identity verification (REQ-5-003): 5th D-027 store (same SQLite
|
||||
# file) + mock-first provider (A-303). State-injection overrides
|
||||
# preserved — tests may pre-set either.
|
||||
identity_store = getattr(app.state, "identity_store", None)
|
||||
if identity_store is None:
|
||||
identity_store = SQLiteIdentityStore(db_path=settings.db_path)
|
||||
app.state.identity_store = identity_store
|
||||
if getattr(app.state, "identity_provider", None) is None:
|
||||
app.state.identity_provider = MockIdentityProvider()
|
||||
|
||||
# Marketplace engine (v0.6, REQ-6-001): 6th D-027 store family on
|
||||
# the same SQLite file. State-injection override preserved.
|
||||
if getattr(app.state, "marketplace_store", None) is None:
|
||||
from .marketplace.store import SQLiteMarketplaceStore
|
||||
|
||||
app.state.marketplace_store = SQLiteMarketplaceStore(
|
||||
db_path=settings.db_path
|
||||
)
|
||||
|
||||
# G-11/D-057: admin routes fail CLOSED on an unset token, and the
|
||||
# founder must be told loudly at boot (never a silent lockout).
|
||||
if not settings.admin_token.get_secret_value():
|
||||
logger.warning(
|
||||
"AI_ADMIN_TOKEN is unset — /v1/marketplace/admin/* routes "
|
||||
"are 403 (fail-closed). Set AI_ADMIN_TOKEN in .env to "
|
||||
"author postings, run ingest, and review applications."
|
||||
)
|
||||
|
||||
# Grading persistence + engine (REQ-3-004): GradeStore from the same
|
||||
# SQLite file as traces (D-027), one GradingEngine singleton wired
|
||||
# through app.state — the engine receives its stores via constructor
|
||||
# DI and knows nothing of FastAPI (api/ owns composition). Tests may
|
||||
# pre-set app.state.grade_store / app.state.grading_engine (the same
|
||||
# state-injection override as sandbox_manager/trace_store) to swap
|
||||
# either; the lifespan adopts a pre-set store but NEVER rebuilds a
|
||||
# pre-set engine (its provider binding is part of the test fixture).
|
||||
grade_store = getattr(app.state, "grade_store", None)
|
||||
if grade_store is None:
|
||||
grade_store = SQLiteGradeStore(db_path=settings.db_path)
|
||||
app.state.grade_store = grade_store
|
||||
if getattr(app.state, "grading_engine", None) is None:
|
||||
app.state.grading_engine = GradingEngine(
|
||||
trace_store,
|
||||
grade_store,
|
||||
app.state.trace_integrity,
|
||||
app.state.provider,
|
||||
model=settings.model,
|
||||
variant_store=variant_store, # MH#4: anchors + seed (D-029)
|
||||
)
|
||||
|
||||
async def _reaper_loop() -> None:
|
||||
# Wall-clock timeout + G-2 workdir-size sweep, one pass per tick.
|
||||
while True:
|
||||
@@ -73,15 +220,26 @@ def create_app(settings: Settings | None = None) -> FastAPI:
|
||||
# No orphans outlive the process (a-1, shutdown half): destroy
|
||||
# everything live; workdirs stay on disk for snapshot restore.
|
||||
await manager.destroy_all()
|
||||
trace_store.close()
|
||||
grade_store.close()
|
||||
variant_store.close()
|
||||
defense_store.close()
|
||||
identity_store.close()
|
||||
await app.state.http_client.aclose()
|
||||
|
||||
app = FastAPI(title="Nextcraft AI Service", version="0.3.0", lifespan=lifespan)
|
||||
|
||||
# A-008: localhost-only CORS, no credentials
|
||||
# A-008 + D-038: no-credentials CORS. Default '*' admits remote-browser
|
||||
# origins in network mode (safe only because allow_credentials stays
|
||||
# False — never enable credentials with a wildcard). AI_CORS_ORIGINS
|
||||
# restricts to an explicit list. PUT is CONTRACT, not trivia: the learner
|
||||
# build surface writes workspace files with PUT (engine-client writeFile)
|
||||
# — v0.3 initially shipped without it and every cross-origin Save failed
|
||||
# preflight (caught in P7 review; tests/api/test_cors.py pins the policy).
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["http://localhost:3000", "http://127.0.0.1:3000"],
|
||||
allow_methods=["GET", "POST", "DELETE", "OPTIONS"],
|
||||
allow_origins=settings.cors_origin_list,
|
||||
allow_methods=["GET", "POST", "PUT", "DELETE", "OPTIONS"],
|
||||
allow_headers=["Content-Type"],
|
||||
allow_credentials=False,
|
||||
)
|
||||
@@ -100,6 +258,106 @@ def create_app(settings: Settings | None = None) -> FastAPI:
|
||||
app.include_router(mentor_router)
|
||||
app.include_router(proctor_router)
|
||||
app.include_router(sandboxes_router)
|
||||
app.include_router(telemetry_router)
|
||||
app.include_router(variants_router)
|
||||
app.include_router(defense_router)
|
||||
|
||||
# v0.5 identity (REQ-5-003/004): verification flow + age-gate deps.
|
||||
# v0.6: the G-18 marketplace stub moved to api/marketplace.py and is
|
||||
# REAL (D-058) — identity keeps only its own router + gate exports.
|
||||
from .api.identity import router as identity_router
|
||||
from .api.marketplace import router as marketplace_public_router
|
||||
from .api.marketplace_admin import router as marketplace_admin_router
|
||||
|
||||
app.include_router(identity_router)
|
||||
app.include_router(marketplace_public_router)
|
||||
app.include_router(marketplace_admin_router)
|
||||
|
||||
# A-305/D1: PII-safe 422s — FastAPI echoes the offending `input` in
|
||||
# validation errors by default; for the identity submit body that
|
||||
# leaks the raw DOB into responses + client logs. The handler scrubs
|
||||
# PII field inputs (scoped app-wide; harmless elsewhere).
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
def _scrub_pii_validation(request, exc): # type: ignore[unused-arg]
|
||||
pii_fields = frozenset({"date_of_birth"})
|
||||
scrubbed = []
|
||||
for err in exc.errors():
|
||||
err = dict(err)
|
||||
if err.get("loc") and err["loc"][-1] in pii_fields:
|
||||
# A-305/D1: never echo the submitted value (input) or the
|
||||
# ctx payload (carries the ValueError); the msg is the
|
||||
# constraint text and is safe.
|
||||
err["input"] = "[redacted]"
|
||||
err.pop("ctx", None)
|
||||
else:
|
||||
# FastAPI's default 422s are JSON-safe EXCEPT ctx payloads
|
||||
# carrying raw ValueError objects (pydantic model_validator
|
||||
# errors); strip ctx body-wide so non-PII routes keep their
|
||||
# 422 shape (msg + loc carry the meaning).
|
||||
ctx = err.get("ctx")
|
||||
if isinstance(ctx, dict):
|
||||
err["ctx"] = {
|
||||
k: v for k, v in ctx.items() if isinstance(v, (str, int, float, bool))
|
||||
}
|
||||
scrubbed.append(err)
|
||||
return JSONResponse(status_code=422, content={"detail": scrubbed})
|
||||
|
||||
app.add_exception_handler(RequestValidationError, _scrub_pii_validation)
|
||||
|
||||
# v0.6 G-101: Next 15 output:'export' hard-throws (E87) for dynamic
|
||||
# routes without generateStaticParams, so jobs/[jobId] and
|
||||
# employers/[employerId] export exactly ONE placeholder shell
|
||||
# (out/marketplace/jobs/_/index.html). These catch-all routes serve
|
||||
# that shell for ANY id — registered BEFORE the StaticFiles mount so
|
||||
# they win (the same routes-beat-the-mount mechanism the sitemap
|
||||
# relies on, G-109). The shell client-fetches the real posting by
|
||||
# the path id and notFound()s on 404 — a resolvable URL for every
|
||||
# sitemap entry.
|
||||
from .api.marketplace_shells import register_marketplace_shell_routes
|
||||
|
||||
register_marketplace_shell_routes(app, settings)
|
||||
|
||||
# v0.3.6 single-port deploy: serve the exported web app (apps/web/out)
|
||||
# from the SAME origin as the API when AI_WEB_STATIC_DIR is set. Mounted
|
||||
# AFTER all routers, so /v1/*, /health, /docs win; StaticFiles(html=True)
|
||||
# then resolves / → index.html, /dashboard/ → dashboard/index.html. A
|
||||
# 404 handler below serves the export's 404.html for unknown paths so
|
||||
# browsers see the site's not-found page instead of FastAPI's JSON.
|
||||
# Default unset → no mount, dev/tests see the plain API app.
|
||||
if str(settings.web_static_dir) not in ("", "."):
|
||||
from fastapi.responses import FileResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
|
||||
web_dir = settings.web_static_dir
|
||||
if not web_dir.is_dir():
|
||||
raise RuntimeError(
|
||||
f"AI_WEB_STATIC_DIR is set but {web_dir} does not exist — "
|
||||
"build the web app first (pnpm build) or unset the setting"
|
||||
)
|
||||
|
||||
@app.exception_handler(404)
|
||||
async def _spa_404(request, exc): # type: ignore[unused]
|
||||
not_found_page = web_dir / "404.html"
|
||||
if not_found_page.is_file():
|
||||
return FileResponse(not_found_page, status_code=404)
|
||||
raise exc
|
||||
|
||||
app.mount("/", StaticFiles(directory=web_dir, html=True), name="web")
|
||||
else:
|
||||
# v0.4.6: the v0.3.6/v0.4-era "site down" root cause was a silently
|
||||
# mount-less boot — behind HAProxy, / answers this app's JSON 404
|
||||
# and nothing detects it. Unattended deploys MUST be loud (G-11
|
||||
# doctrine). Dev/tests keep the no-mount default; production boxes
|
||||
# see a warning naming the fix.
|
||||
logger.warning(
|
||||
"AI_WEB_STATIC_DIR is unset — the exported web UI is NOT served; "
|
||||
"/ answers the API's JSON 404. For single-port deploys, build the "
|
||||
"web app (NEXT_PUBLIC_AI_SERVICE_URL=self pnpm build) and run it "
|
||||
"via `nextcraft dev`/`dev -d` (auto-wires apps/web/out) or set "
|
||||
"AI_WEB_STATIC_DIR explicitly."
|
||||
)
|
||||
return app
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
"""Marketplace engine package (REQ-6-001, D-048/D-049).
|
||||
|
||||
The 6th engine module inside apps/ai-service (no new apps, no new ports,
|
||||
D-031 boundary): pydantic contracts (`base.py`), the D-027 store family's
|
||||
sixth SQLite store (`store.py`), and PostingSource ingest adapters
|
||||
(`ingest.py`). Tagging (P2), embedding (P3), enrichment + resume (P4)
|
||||
extend this package; the HTTP surface lives in `api/marketplace*.py`
|
||||
which composes this module via DI.
|
||||
|
||||
Boundary (D-031/D-048): `marketplace/` never imports `agents/` or
|
||||
`api/`; this package imports config + stdlib only.
|
||||
"""
|
||||
@@ -0,0 +1,174 @@
|
||||
"""Marketplace pydantic contracts (REQ-6-001, D-048, A-402).
|
||||
|
||||
The wire + store shapes for the postings backend. `IngestPosting` is the
|
||||
A-402 adapter shape (what every PostingSource yields); `Posting` is the
|
||||
full row shape (status lifecycle + tags); `Employer` carries the raw
|
||||
ingested fields (enrichment + provenance merge is P4's read-time job).
|
||||
|
||||
D-051/D-053 note: the mock `matchScore` from the TS fixtures is NOT part
|
||||
of any contract — P3 computes real scores at search time; absent score =
|
||||
absent query context (G-106).
|
||||
|
||||
Boundary (D-048): mirrors D-031 — never imports `agents/` or `api/`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
PostingStatus = Literal["draft", "published", "closed"]
|
||||
|
||||
#: Lifecycle edge set (D-049/D-057): draft → published → closed. A
|
||||
#: transition not in this map is invalid (admin route → 409).
|
||||
VALID_TRANSITIONS: dict[str, str] = {
|
||||
"draft": "published",
|
||||
"published": "closed",
|
||||
}
|
||||
|
||||
Seniority = Literal["entry", "mid", "senior", "staff", "principal"]
|
||||
|
||||
#: P2 (D-052) enumerates the category taxonomy; the seed corpus carries
|
||||
#: category=null and P2's tagging pass fills it. The store accepts any
|
||||
#: short string (validation lands with the tagger); `None` = not yet
|
||||
#: tagged.
|
||||
Category = str | None
|
||||
|
||||
|
||||
class PostingTags(BaseModel):
|
||||
"""Derived tags on a posting row (D-052 schema, persisted as the
|
||||
posting's JSON `tags` column in P2; the column exists from day one).
|
||||
|
||||
`skills` is bounded at 10 by the tagger (D-052); the schema itself
|
||||
keeps the bound so the store can validate a hand-authored payload.
|
||||
`extra="forbid"`: a non-tag JSON object (e.g. the mock provider's
|
||||
canned {"summary":...}) must NOT silently validate as all-default
|
||||
tags — the untagged signal stays honest (verifier P2-3; P3
|
||||
embeddings must not treat empty-default parses as real tags).
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
skills: list[str] = Field(default_factory=list, max_length=10)
|
||||
seniority: Seniority | None = None
|
||||
category: Category = None
|
||||
|
||||
|
||||
class Employer(BaseModel):
|
||||
"""Raw ingested employer fields (D-049 `employer` table row shape).
|
||||
|
||||
P4 (D-053) enriches on read with per-field `{value, source}`
|
||||
provenance; P1 surfaces the raw fields only.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
id: str = Field(min_length=1)
|
||||
source: str = Field(min_length=1)
|
||||
external_id: str = Field(min_length=1)
|
||||
name: str = Field(min_length=1)
|
||||
logo: str = ""
|
||||
description: str = ""
|
||||
industry: str = ""
|
||||
size: str = ""
|
||||
location: str = ""
|
||||
website: str = ""
|
||||
social_links: dict[str, str] = Field(default_factory=dict)
|
||||
culture: str = ""
|
||||
|
||||
|
||||
class IngestPosting(BaseModel):
|
||||
"""The A-402 PostingSource payload: (source, external_id) identity +
|
||||
employer binding + the raw posting fields. Idempotent upsert key is
|
||||
(source, external_id) — a re-fetch overwrites the same row in place.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
id: str = Field(min_length=1)
|
||||
source: str = Field(min_length=1)
|
||||
external_id: str = Field(min_length=1)
|
||||
employer_id: str = Field(min_length=1)
|
||||
title: str = Field(min_length=1)
|
||||
description: str = ""
|
||||
required_competencies: list[str] = Field(default_factory=list)
|
||||
skills: list[str] = Field(default_factory=list)
|
||||
seniority: Seniority | None = None
|
||||
category: Category = None
|
||||
location: str = ""
|
||||
remote: bool = False
|
||||
salary_min: int | None = None
|
||||
salary_max: int | None = None
|
||||
#: G-102: seed-port postings carry their ORIGINAL postedAt; the
|
||||
#: store applies it verbatim (status=published via the ingest
|
||||
#: contract, not the field).
|
||||
posted_at: datetime | None = None
|
||||
|
||||
|
||||
class Posting(BaseModel):
|
||||
"""Full posting row (D-049 `posting` table). Status is the lifecycle
|
||||
column (draft|published|closed, D-057); `tags` is the P2-derived
|
||||
JSON column (D-052 — empty until the tagger runs). `matchScore` is
|
||||
deliberately absent (G-106/G-107).
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
id: str = Field(min_length=1)
|
||||
source: str = Field(min_length=1)
|
||||
external_id: str = Field(min_length=1)
|
||||
employer_id: str = Field(min_length=1)
|
||||
title: str = Field(min_length=1)
|
||||
description: str = ""
|
||||
required_competencies: list[str] = Field(default_factory=list)
|
||||
skills: list[str] = Field(default_factory=list)
|
||||
seniority: Seniority | None = None
|
||||
category: Category = None
|
||||
location: str = ""
|
||||
remote: bool = False
|
||||
salary_min: int | None = None
|
||||
salary_max: int | None = None
|
||||
status: PostingStatus = "draft"
|
||||
tags: PostingTags | None = None
|
||||
posted_at: datetime | None = None
|
||||
|
||||
|
||||
class Application(BaseModel):
|
||||
"""A learner's application to a posting (D-049 `application` table).
|
||||
Unique (posting_id, learner_id) → duplicate apply is idempotent
|
||||
200-with-existing (A-408/D-058). P1 status is always `submitted`;
|
||||
richer states are post-v0.6.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
id: str = Field(min_length=1)
|
||||
posting_id: str = Field(min_length=1)
|
||||
learner_id: str = Field(min_length=1)
|
||||
status: str = "submitted"
|
||||
created_at: datetime | None = None
|
||||
|
||||
|
||||
class PostingPage(BaseModel):
|
||||
"""Published-only list response (G-103): the page + the honest
|
||||
total so the board can render "Showing X of Y" over a paged
|
||||
fetch-all without fabrication."""
|
||||
|
||||
postings: list[Posting]
|
||||
total_published: int
|
||||
|
||||
|
||||
class IngestResult(BaseModel):
|
||||
"""Honest ingest run summary (founder surface)."""
|
||||
|
||||
postings_upserted: int = 0
|
||||
employers_upserted: int = 0
|
||||
|
||||
|
||||
def posting_summary(posting: Posting) -> dict[str, Any]:
|
||||
"""Serialization helper shared by the list + detail responses — one
|
||||
shape so the G-107 pin (no `matchScore` in any marketplace-path
|
||||
payload) covers both surfaces."""
|
||||
return posting.model_dump(mode="json")
|
||||
@@ -0,0 +1,97 @@
|
||||
"""PostingSource adapters + ingest runner (REQ-6-001, D-050, A-402).
|
||||
|
||||
`PostingSource` is the adapter port: `fetch() -> Iterable[IngestPosting]`.
|
||||
v0.6 ships ONE implementation, `SeededCorpusSource` over the in-repo
|
||||
`seed_corpus.json` (ported ONCE from packages/mock-data/jobs.ts +
|
||||
employers.ts per D-021 — shared ids, mock matchScore DROPPED: D-051
|
||||
computes real scores at search time, G-106/G-107). The 3M+ external
|
||||
corpus lands later as a third adapter with zero store/route changes.
|
||||
|
||||
Ingest-status contract (G-102): SEED-PORT postings enter as `published`
|
||||
with their ORIGINAL postedAt — otherwise the board/facets/P1's success
|
||||
criterion sit empty after ingest. ADMIN-AUTHORED postings start
|
||||
`draft` (the admin route's create path, not this runner).
|
||||
|
||||
No scheduler/crawler (A-401/A-402): ingest runs when the founder POSTs
|
||||
/v1/marketplace/admin/ingest; `run_ingest` is idempotent on
|
||||
(source, external_id) — a double-run yields identical row counts.
|
||||
|
||||
Boundary (D-048): never imports `agents/` or `api/`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import Iterable
|
||||
from pathlib import Path
|
||||
from typing import Protocol, runtime_checkable
|
||||
|
||||
from .base import Employer, IngestPosting, IngestResult
|
||||
from .store import MarketplaceStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: The ported TS fixture corpus ships IN THE PACKAGE (importlib-free
|
||||
#: path resolution: this file's parent) — one JSON file, no codegen.
|
||||
_SEED_PATH = Path(__file__).resolve().parent / "seed_corpus.json"
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class PostingSource(Protocol):
|
||||
"""The ingest port (D-050): yields IngestPosting payloads. Employers
|
||||
ride the corpus alongside postings (the seeded source exposes both;
|
||||
a pure-posting adapter simply fetches postings and leaves employers
|
||||
to a separate source)."""
|
||||
|
||||
def fetch(self) -> Iterable[IngestPosting]:
|
||||
...
|
||||
|
||||
def fetch_employers(self) -> Iterable[Employer]:
|
||||
...
|
||||
|
||||
|
||||
class SeededCorpusSource:
|
||||
"""The seeded corpus adapter over seed_corpus.json (D-050).
|
||||
|
||||
Pure + deterministic: reads the JSON fixture once per fetch; the
|
||||
runner upserts idempotently. Raw fields map 1:1 from the TS shapes
|
||||
(camelCase → snake_case port, matchScore dropped — G-102/G-107)."""
|
||||
|
||||
def __init__(self, path: Path = _SEED_PATH) -> None:
|
||||
self._path = path
|
||||
|
||||
def _load(self) -> dict:
|
||||
with self._path.open(encoding="utf-8") as fh:
|
||||
return json.load(fh)
|
||||
|
||||
def fetch(self) -> Iterable[IngestPosting]:
|
||||
for raw in self._load().get("postings", []):
|
||||
yield IngestPosting.model_validate(raw)
|
||||
|
||||
def fetch_employers(self) -> Iterable[Employer]:
|
||||
for raw in self._load().get("employers", []):
|
||||
yield Employer.model_validate(raw)
|
||||
|
||||
|
||||
def run_ingest(source: PostingSource, store: MarketplaceStore) -> IngestResult:
|
||||
"""Idempotent upsert run: every (source, external_id) overwrites its
|
||||
row in place — a second run yields identical counts (MH-1b)."""
|
||||
|
||||
result = IngestResult()
|
||||
# Employers first: postings reference employer ids (no FK today, but
|
||||
# the reference ordering keeps the board honest mid-run).
|
||||
for employer in source.fetch_employers():
|
||||
store.upsert_employer(employer)
|
||||
result.employers_upserted += 1
|
||||
for posting in source.fetch():
|
||||
# G-102: seed-port postings enter PUBLISHED with their original
|
||||
# postedAt (the IngestPosting carries it verbatim).
|
||||
store.upsert_posting(posting, status="published")
|
||||
result.postings_upserted += 1
|
||||
logger.info(
|
||||
"marketplace ingest: %s postings, %s employers (idempotent upserts)",
|
||||
result.postings_upserted,
|
||||
result.employers_upserted,
|
||||
)
|
||||
return result
|
||||
@@ -0,0 +1,695 @@
|
||||
{
|
||||
"employers": [
|
||||
{
|
||||
"id": "emp-openai",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "emp-openai",
|
||||
"name": "OpenAI Labs",
|
||||
"logo": "https://placehold.co/80x80/indigo/white?text=OA",
|
||||
"description": "Mock employer building frontier AI systems. We build and deploy large language models and agentic tools used by millions of developers and consumers.",
|
||||
"industry": "Artificial Intelligence",
|
||||
"size": "2,000+",
|
||||
"location": "San Francisco, CA",
|
||||
"website": "https://example.com/openai",
|
||||
"social_links": {
|
||||
"twitter": "https://example.com/openai/x",
|
||||
"linkedin": "https://example.com/openai/li"
|
||||
},
|
||||
"culture": "Research-driven, shipping-paced, safety-conscious. We pair frontier research with product rigor."
|
||||
},
|
||||
{
|
||||
"id": "emp-anthropic",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "emp-anthropic",
|
||||
"name": "Anthropic Research",
|
||||
"logo": "https://placehold.co/80x80/amber/white?text=AN",
|
||||
"description": "Mock employer focused on AI safety and alignment. We build reliable, interpretable, and steerable AI systems.",
|
||||
"industry": "AI Safety",
|
||||
"size": "1,000+",
|
||||
"location": "San Francisco, CA",
|
||||
"website": "https://example.com/anthropic",
|
||||
"social_links": {
|
||||
"twitter": "https://example.com/anthropic/x",
|
||||
"linkedin": "https://example.com/anthropic/li"
|
||||
},
|
||||
"culture": "Safety-first, evidence-based, calm-paced. We value thoroughness over speed when stakes are high."
|
||||
},
|
||||
{
|
||||
"id": "emp-huggingface",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "emp-huggingface",
|
||||
"name": "Hugging Face",
|
||||
"logo": "https://placehold.co/80x80/yellow/black?text=HF",
|
||||
"description": "Mock employer building the open-source AI community. We host models, datasets, and demos for the global ML community.",
|
||||
"industry": "Open Source AI",
|
||||
"size": "500+",
|
||||
"location": "New York, NY",
|
||||
"website": "https://example.com/hf",
|
||||
"social_links": {
|
||||
"twitter": "https://example.com/hf/x",
|
||||
"linkedin": "https://example.com/hf/li"
|
||||
},
|
||||
"culture": "Community-first, open-by-default, remote-friendly. We ship in the open with thousands of contributors."
|
||||
},
|
||||
{
|
||||
"id": "emp-scaleai",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "emp-scaleai",
|
||||
"name": "Scale AI",
|
||||
"logo": "https://placehold.co/80x80/violet/white?text=SC",
|
||||
"description": "Mock employer providing data and evaluation infrastructure for frontier AI. We power the RLHF and eval pipelines behind many model releases.",
|
||||
"industry": "AI Infrastructure",
|
||||
"size": "1,500+",
|
||||
"location": "San Francisco, CA",
|
||||
"website": "https://example.com/scale",
|
||||
"social_links": {
|
||||
"twitter": "https://example.com/scale/x",
|
||||
"linkedin": "https://example.com/scale/li"
|
||||
},
|
||||
"culture": "Infrastructure-minded, quality-obsessed, enterprise-aware. We bridge research and production data."
|
||||
},
|
||||
{
|
||||
"id": "emp-perplexity",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "emp-perplexity",
|
||||
"name": "Perplexity",
|
||||
"logo": "https://placehold.co/80x80/teal/white?text=PX",
|
||||
"description": "Mock employer building an AI-powered answer engine. We combine retrieval, generation, and citations for trustworthy answers.",
|
||||
"industry": "AI Search",
|
||||
"size": "300+",
|
||||
"location": "San Francisco, CA",
|
||||
"website": "https://example.com/perplexity",
|
||||
"social_links": {
|
||||
"twitter": "https://example.com/perplexity/x",
|
||||
"linkedin": "https://example.com/perplexity/li"
|
||||
},
|
||||
"culture": "Answer-focused, fast-iterating, citation-proud. We treat groundedness as a product feature."
|
||||
},
|
||||
{
|
||||
"id": "emp-cohere",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "emp-cohere",
|
||||
"name": "Cohere",
|
||||
"logo": "https://placehold.co/80x80/pink/white?text=CO",
|
||||
"description": "Mock employer building enterprise-grade language models. We specialize in retrieval, multilinguality, and data privacy for regulated industries.",
|
||||
"industry": "Enterprise LLMs",
|
||||
"size": "400+",
|
||||
"location": "Toronto, Canada",
|
||||
"website": "https://example.com/cohere",
|
||||
"social_links": {
|
||||
"twitter": "https://example.com/cohere/x",
|
||||
"linkedin": "https://example.com/cohere/li"
|
||||
},
|
||||
"culture": "Enterprise-aware, research-active, multilingual. We ship models that work in 100+ languages."
|
||||
},
|
||||
{
|
||||
"id": "emp-mistral",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "emp-mistral",
|
||||
"name": "Mistral AI",
|
||||
"logo": "https://placehold.co/80x80/red/white?text=MI",
|
||||
"description": "Mock employer building open-weight frontier models in Europe. We push efficiency and openness in large language models.",
|
||||
"industry": "Open-Weight LLMs",
|
||||
"size": "200+",
|
||||
"location": "Paris, France",
|
||||
"website": "https://example.com/mistral",
|
||||
"social_links": {
|
||||
"twitter": "https://example.com/mistral/x",
|
||||
"linkedin": "https://example.com/mistral/li"
|
||||
},
|
||||
"culture": "European-rooted, efficiency-driven, open-weight-proud. We ship small models that punch above their size."
|
||||
},
|
||||
{
|
||||
"id": "emp-replicate",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "emp-replicate",
|
||||
"name": "Replicate",
|
||||
"logo": "https://placehold.co/80x80/orange/white?text=RE",
|
||||
"description": "Mock employer making ML deployment delightful. We host, serve, and scale models with a few lines of code.",
|
||||
"industry": "ML Deployment",
|
||||
"size": "150+",
|
||||
"location": "San Francisco, CA",
|
||||
"website": "https://example.com/replicate",
|
||||
"social_links": {
|
||||
"twitter": "https://example.com/replicate/x",
|
||||
"linkedin": "https://example.com/replicate/li"
|
||||
},
|
||||
"culture": "Developer-experience-obsessed, pragmatic, remote-friendly. We make shipping models feel like shipping software."
|
||||
},
|
||||
{
|
||||
"id": "emp-pinecone",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "emp-pinecone",
|
||||
"name": "Pinecone",
|
||||
"logo": "https://placehold.co/80x80/green/white?text=PC",
|
||||
"description": "Mock employer building the vector database for AI applications. We power retrieval for thousands of production RAG systems.",
|
||||
"industry": "Vector Databases",
|
||||
"size": "300+",
|
||||
"location": "New York, NY",
|
||||
"website": "https://example.com/pinecone",
|
||||
"social_links": {
|
||||
"twitter": "https://example.com/pinecone/x",
|
||||
"linkedin": "https://example.com/pinecone/li"
|
||||
},
|
||||
"culture": "Infra-focused, latency-obsessed, distributed-first. We treat retrieval as the heart of grounded AI."
|
||||
},
|
||||
{
|
||||
"id": "emp-langchain",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "emp-langchain",
|
||||
"name": "LangChain",
|
||||
"logo": "https://placehold.co/80x80/blue/white?text=LC",
|
||||
"description": "Mock employer building the orchestration layer for LLM applications. We provide frameworks for agents, retrieval, and evaluation.",
|
||||
"industry": "AI Orchestration",
|
||||
"size": "200+",
|
||||
"location": "San Francisco, CA",
|
||||
"website": "https://example.com/langchain",
|
||||
"social_links": {
|
||||
"twitter": "https://example.com/langchain/x",
|
||||
"linkedin": "https://example.com/langchain/li"
|
||||
},
|
||||
"culture": "Framework-minded, community-driven, abstractions-first. We build primitives others compose into products."
|
||||
}
|
||||
],
|
||||
"postings": [
|
||||
{
|
||||
"id": "job-001",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-001",
|
||||
"employer_id": "emp-langchain",
|
||||
"title": "AI Orchestration Engineer",
|
||||
"description": "Design and operate multi-agent systems that automate complex knowledge workflows. You will own agent topology, tool integration, and evaluation harnesses for production deployments serving millions of queries.",
|
||||
"required_competencies": [
|
||||
"stack-orchestration-c001",
|
||||
"stack-orchestration-c002",
|
||||
"stack-orchestration-c005"
|
||||
],
|
||||
"skills": [
|
||||
"LangGraph",
|
||||
"Multi-agent systems",
|
||||
"RAG",
|
||||
"Python",
|
||||
"Evaluation"
|
||||
],
|
||||
"seniority": "senior",
|
||||
"category": null,
|
||||
"location": "San Francisco, CA",
|
||||
"remote": true,
|
||||
"salary_min": 160000,
|
||||
"salary_max": 240000,
|
||||
"posted_at": "2026-08-21T10:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-002",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-002",
|
||||
"employer_id": "emp-openai",
|
||||
"title": "LLM Application Developer",
|
||||
"description": "Build delightful LLM-powered features end-to-end. You will work on retrieval, function calling, streaming UX, and robust output validation for consumer-facing products.",
|
||||
"required_competencies": [
|
||||
"stack-orchestration-c003",
|
||||
"stack-orchestration-c004",
|
||||
"stack-orchestration-c005"
|
||||
],
|
||||
"skills": [
|
||||
"Python",
|
||||
"TypeScript",
|
||||
"RAG",
|
||||
"Function calling",
|
||||
"Streaming"
|
||||
],
|
||||
"seniority": "mid",
|
||||
"category": null,
|
||||
"location": "San Francisco, CA",
|
||||
"remote": true,
|
||||
"salary_min": 140000,
|
||||
"salary_max": 210000,
|
||||
"posted_at": "2026-08-28T09:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-003",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-003",
|
||||
"employer_id": "emp-anthropic",
|
||||
"title": "AI Safety Researcher",
|
||||
"description": "Research alignment, interpretability, and robustness of frontier models. Design experiments, run evaluations, and publish findings that improve the safety of deployed AI systems.",
|
||||
"required_competencies": [
|
||||
"stack-safety-c001",
|
||||
"stack-safety-c007",
|
||||
"stack-safety-c002"
|
||||
],
|
||||
"skills": [
|
||||
"Alignment",
|
||||
"Interpretability",
|
||||
"Red teaming",
|
||||
"Python",
|
||||
"Research"
|
||||
],
|
||||
"seniority": "senior",
|
||||
"category": null,
|
||||
"location": "San Francisco, CA",
|
||||
"remote": false,
|
||||
"salary_min": 180000,
|
||||
"salary_max": 280000,
|
||||
"posted_at": "2026-08-15T12:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-004",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-004",
|
||||
"employer_id": "emp-perplexity",
|
||||
"title": "Prompt Engineer",
|
||||
"description": "Craft, test, and ship prompt strategies that power answer-engine features. Own prompt libraries, regression suites, and A/B experiments across product surfaces.",
|
||||
"required_competencies": [
|
||||
"stack-orchestration-c004",
|
||||
"stack-orchestration-c001"
|
||||
],
|
||||
"skills": [
|
||||
"Prompt engineering",
|
||||
"Evaluation",
|
||||
"A/B testing",
|
||||
"Python"
|
||||
],
|
||||
"seniority": "mid",
|
||||
"category": null,
|
||||
"location": "Remote",
|
||||
"remote": true,
|
||||
"salary_min": 110000,
|
||||
"salary_max": 175000,
|
||||
"posted_at": "2026-09-02T14:15:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-005",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-005",
|
||||
"employer_id": "emp-huggingface",
|
||||
"title": "AI Product Manager",
|
||||
"description": "Own the roadmap for AI-powered developer tools. Translate model capabilities into customer value, define success metrics, and ship agentic features with engineering.",
|
||||
"required_competencies": [
|
||||
"stack-designer-c001",
|
||||
"stack-orchestration-c001"
|
||||
],
|
||||
"skills": [
|
||||
"Product strategy",
|
||||
"AI UX",
|
||||
"Roadmapping",
|
||||
"Stakeholder mgmt"
|
||||
],
|
||||
"seniority": "senior",
|
||||
"category": null,
|
||||
"location": "New York, NY",
|
||||
"remote": true,
|
||||
"salary_min": 150000,
|
||||
"salary_max": 220000,
|
||||
"posted_at": "2026-08-30T08:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-006",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-006",
|
||||
"employer_id": "emp-pinecone",
|
||||
"title": "RAG Infrastructure Engineer",
|
||||
"description": "Build the retrieval substrate behind knowledge-grounded AI. Optimize indexing, query latency, and hybrid retrieval across billion-vector workloads.",
|
||||
"required_competencies": [
|
||||
"stack-orchestration-c005",
|
||||
"stack-orchestration-c006"
|
||||
],
|
||||
"skills": [
|
||||
"Vector databases",
|
||||
"Embeddings",
|
||||
"Go",
|
||||
"Distributed systems"
|
||||
],
|
||||
"seniority": "mid",
|
||||
"category": null,
|
||||
"location": "Remote",
|
||||
"remote": true,
|
||||
"salary_min": 130000,
|
||||
"salary_max": 200000,
|
||||
"posted_at": "2026-09-05T11:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-007",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-007",
|
||||
"employer_id": "emp-scaleai",
|
||||
"title": "AI Governance Lead",
|
||||
"description": "Stand up the AI governance program for enterprise customers. Map controls to NIST AI RMF and EU AI Act, run audits, and author model cards at scale.",
|
||||
"required_competencies": [
|
||||
"stack-safety-c005",
|
||||
"stack-safety-c003",
|
||||
"stack-safety-c013"
|
||||
],
|
||||
"skills": [
|
||||
"AI policy",
|
||||
"NIST AI RMF",
|
||||
"Audit",
|
||||
"Documentation"
|
||||
],
|
||||
"seniority": "staff",
|
||||
"category": null,
|
||||
"location": "San Francisco, CA",
|
||||
"remote": true,
|
||||
"salary_min": 170000,
|
||||
"salary_max": 230000,
|
||||
"posted_at": "2026-08-18T16:45:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-008",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-008",
|
||||
"employer_id": "emp-anthropic",
|
||||
"title": "Human-AI Interaction Designer",
|
||||
"description": "Design how users collaborate with Claude across products. Own transparency patterns, trust calibration, and agentic interaction for assistant surfaces.",
|
||||
"required_competencies": [
|
||||
"stack-designer-c001",
|
||||
"stack-designer-c003",
|
||||
"stack-designer-c008"
|
||||
],
|
||||
"skills": [
|
||||
"Figma",
|
||||
"AI UX",
|
||||
"Prototyping",
|
||||
"User research"
|
||||
],
|
||||
"seniority": "mid",
|
||||
"category": null,
|
||||
"location": "San Francisco, CA",
|
||||
"remote": true,
|
||||
"salary_min": 125000,
|
||||
"salary_max": 190000,
|
||||
"posted_at": "2026-08-25T10:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-009",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-009",
|
||||
"employer_id": "emp-cohere",
|
||||
"title": "Evaluation Engineer",
|
||||
"description": "Build and operate the eval platform for LLM-powered products. Design LLM-as-judge pipelines, regression suites, and human-eval panels for production models.",
|
||||
"required_competencies": [
|
||||
"stack-orchestration-c007",
|
||||
"stack-safety-c002"
|
||||
],
|
||||
"skills": [
|
||||
"LLM evaluation",
|
||||
"Python",
|
||||
"Statistics",
|
||||
"Data labeling"
|
||||
],
|
||||
"seniority": "mid",
|
||||
"category": null,
|
||||
"location": "Toronto, Canada",
|
||||
"remote": true,
|
||||
"salary_min": 120000,
|
||||
"salary_max": 180000,
|
||||
"posted_at": "2026-09-01T13:20:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-010",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-010",
|
||||
"employer_id": "emp-replicate",
|
||||
"title": "Robotics Operations Specialist",
|
||||
"description": "Supervise a fleet of AI-augmented robotic units in a warehouse environment. Interpret anomaly alerts, perform calibrations, and intervene on exceptions.",
|
||||
"required_competencies": [
|
||||
"stack-operator-c002",
|
||||
"stack-operator-c011",
|
||||
"stack-operator-c003"
|
||||
],
|
||||
"skills": [
|
||||
"Robotics",
|
||||
"Safety protocols",
|
||||
"IoT",
|
||||
"Predictive maintenance"
|
||||
],
|
||||
"seniority": "entry",
|
||||
"category": null,
|
||||
"location": "Austin, TX",
|
||||
"remote": false,
|
||||
"salary_min": 80000,
|
||||
"salary_max": 115000,
|
||||
"posted_at": "2026-08-22T09:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-011",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-011",
|
||||
"employer_id": "emp-recursion",
|
||||
"title": "Computational Biologist",
|
||||
"description": "Apply ML to drug discovery. Build pipelines for phenotype prediction, active learning on assay data, and molecular generation for novel targets.",
|
||||
"required_competencies": [
|
||||
"stack-science-c005",
|
||||
"stack-science-c002",
|
||||
"stack-science-c011"
|
||||
],
|
||||
"skills": [
|
||||
"Bioinformatics",
|
||||
"Python",
|
||||
"PyTorch",
|
||||
"Drug discovery"
|
||||
],
|
||||
"seniority": "senior",
|
||||
"category": null,
|
||||
"location": "Boston, MA",
|
||||
"remote": true,
|
||||
"salary_min": 145000,
|
||||
"salary_max": 215000,
|
||||
"posted_at": "2026-08-12T15:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-012",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-012",
|
||||
"employer_id": "emp-deepmind",
|
||||
"title": "Climate ML Scientist",
|
||||
"description": "Develop ML models for climate forecasting and energy grid optimization. Work with earth system scientists to downscale GCM output and quantify uncertainty.",
|
||||
"required_competencies": [
|
||||
"stack-science-c004",
|
||||
"stack-science-c014",
|
||||
"stack-science-c010"
|
||||
],
|
||||
"skills": [
|
||||
"Climate modeling",
|
||||
"PyTorch",
|
||||
"Data assimilation",
|
||||
"PINNs"
|
||||
],
|
||||
"seniority": "senior",
|
||||
"category": null,
|
||||
"location": "London, UK",
|
||||
"remote": true,
|
||||
"salary_min": 135000,
|
||||
"salary_max": 200000,
|
||||
"posted_at": "2026-08-27T11:45:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-013",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-013",
|
||||
"employer_id": "emp-langchain",
|
||||
"title": "Agent Reliability Engineer",
|
||||
"description": "Own observability and reliability for production agent workloads. Build tracing, alerting, and rollback systems for multi-step agent pipelines.",
|
||||
"required_competencies": [
|
||||
"stack-orchestration-c013",
|
||||
"stack-orchestration-c015",
|
||||
"stack-orchestration-c008"
|
||||
],
|
||||
"skills": [
|
||||
"Observability",
|
||||
"Python",
|
||||
"SRE",
|
||||
"Distributed tracing"
|
||||
],
|
||||
"seniority": "mid",
|
||||
"category": null,
|
||||
"location": "Remote",
|
||||
"remote": true,
|
||||
"salary_min": 140000,
|
||||
"salary_max": 205000,
|
||||
"posted_at": "2026-09-04T09:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-014",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-014",
|
||||
"employer_id": "emp-mistral",
|
||||
"title": "AI Red Team Lead",
|
||||
"description": "Lead adversarial testing of frontier and open-weight models. Build automated red-team suites, track vulnerabilities, and coordinate disclosure.",
|
||||
"required_competencies": [
|
||||
"stack-safety-c002",
|
||||
"stack-safety-c010",
|
||||
"stack-safety-c001"
|
||||
],
|
||||
"skills": [
|
||||
"Red teaming",
|
||||
"Prompt injection",
|
||||
"Python",
|
||||
"Leadership"
|
||||
],
|
||||
"seniority": "staff",
|
||||
"category": null,
|
||||
"location": "Paris, France",
|
||||
"remote": true,
|
||||
"salary_min": 160000,
|
||||
"salary_max": 230000,
|
||||
"posted_at": "2026-08-19T10:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-015",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-015",
|
||||
"employer_id": "emp-huggingface",
|
||||
"title": "Conversation Designer",
|
||||
"description": "Design dialogue flows, personas, and repair strategies for AI assistants across open-source products. Partner with ML engineers to align tone with model behavior.",
|
||||
"required_competencies": [
|
||||
"stack-designer-c002",
|
||||
"stack-designer-c010",
|
||||
"stack-designer-c011"
|
||||
],
|
||||
"skills": [
|
||||
"Conversation design",
|
||||
"Figma",
|
||||
"Voice UX",
|
||||
"Prototyping"
|
||||
],
|
||||
"seniority": "mid",
|
||||
"category": null,
|
||||
"location": "Remote",
|
||||
"remote": true,
|
||||
"salary_min": 105000,
|
||||
"salary_max": 160000,
|
||||
"posted_at": "2026-08-29T14:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-016",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-016",
|
||||
"employer_id": "emp-anduril",
|
||||
"title": "Field AI Technician",
|
||||
"description": "Deploy and maintain AI vision systems on defense hardware in the field. Calibrate sensors, interpret model alerts, and escalate edge cases to engineering.",
|
||||
"required_competencies": [
|
||||
"stack-operator-c004",
|
||||
"stack-operator-c009",
|
||||
"stack-operator-c001"
|
||||
],
|
||||
"skills": [
|
||||
"Computer vision",
|
||||
"Sensor calibration",
|
||||
"Field ops",
|
||||
"Python"
|
||||
],
|
||||
"seniority": "entry",
|
||||
"category": null,
|
||||
"location": "Costa Mesa, CA",
|
||||
"remote": false,
|
||||
"salary_min": 90000,
|
||||
"salary_max": 130000,
|
||||
"posted_at": "2026-08-14T08:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-017",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-017",
|
||||
"employer_id": "emp-deepmind",
|
||||
"title": "Materials ML Engineer",
|
||||
"description": "Discover novel materials with ML. Train property-prediction models, run active-learning loops over DFT calculations, and validate candidates experimentally.",
|
||||
"required_competencies": [
|
||||
"stack-science-c003",
|
||||
"stack-science-c002",
|
||||
"stack-science-c006"
|
||||
],
|
||||
"skills": [
|
||||
"Materials science",
|
||||
"PyTorch",
|
||||
"DFT",
|
||||
"Active learning"
|
||||
],
|
||||
"seniority": "senior",
|
||||
"category": null,
|
||||
"location": "London, UK",
|
||||
"remote": true,
|
||||
"salary_min": 140000,
|
||||
"salary_max": 210000,
|
||||
"posted_at": "2026-08-26T12:15:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-018",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-018",
|
||||
"employer_id": "emp-openai",
|
||||
"title": "AI Trust & Safety Analyst",
|
||||
"description": "Investigate misuse patterns, triage safety incidents, and improve policy enforcement for consumer AI products. Author postmortems and recommend mitigations.",
|
||||
"required_competencies": [
|
||||
"stack-safety-c008",
|
||||
"stack-safety-c006",
|
||||
"stack-safety-c002"
|
||||
],
|
||||
"skills": [
|
||||
"Trust & safety",
|
||||
"Incident response",
|
||||
"Policy",
|
||||
"Investigation"
|
||||
],
|
||||
"seniority": "mid",
|
||||
"category": null,
|
||||
"location": "San Francisco, CA",
|
||||
"remote": true,
|
||||
"salary_min": 115000,
|
||||
"salary_max": 170000,
|
||||
"posted_at": "2026-09-03T10:45:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-019",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-019",
|
||||
"employer_id": "emp-replicate",
|
||||
"title": "Edge AI Engineer",
|
||||
"description": "Optimize and deploy models to edge hardware. Quantize, distill, and compile models for low-latency inference on field devices with constrained budgets.",
|
||||
"required_competencies": [
|
||||
"stack-operator-c008",
|
||||
"stack-orchestration-c012",
|
||||
"stack-orchestration-c014"
|
||||
],
|
||||
"skills": [
|
||||
"Edge ML",
|
||||
"TensorRT",
|
||||
"C++",
|
||||
"Quantization"
|
||||
],
|
||||
"seniority": "mid",
|
||||
"category": null,
|
||||
"location": "Remote",
|
||||
"remote": true,
|
||||
"salary_min": 130000,
|
||||
"salary_max": 195000,
|
||||
"posted_at": "2026-08-24T11:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": "job-020",
|
||||
"source": "seed-corpus",
|
||||
"external_id": "job-020",
|
||||
"employer_id": "emp-cohere",
|
||||
"title": "AI Research Engineer",
|
||||
"description": "Push the frontier of language model capabilities. Prototype new architectures, run large-scale experiments, and contribute to publications and open-source releases.",
|
||||
"required_competencies": [
|
||||
"stack-orchestration-c001",
|
||||
"stack-science-c010",
|
||||
"stack-science-c011"
|
||||
],
|
||||
"skills": [
|
||||
"PyTorch",
|
||||
"Research",
|
||||
"Transformers",
|
||||
"Distributed training"
|
||||
],
|
||||
"seniority": "senior",
|
||||
"category": null,
|
||||
"location": "Berlin, Germany",
|
||||
"remote": true,
|
||||
"salary_min": 155000,
|
||||
"salary_max": 235000,
|
||||
"posted_at": "2026-08-20T09:00:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,695 @@
|
||||
"""MarketplaceStore — the 6th D-027 store family (D-049, amended G-104).
|
||||
|
||||
Six tables on the shared `Settings().db_path` (one SQLite file with the
|
||||
five prior stores): `posting`, `employer`, `embedding`, `application`,
|
||||
`employer_enrichment`, `learner_profile`. Mechanics mirror the
|
||||
probe-verified identity/store.py exactly: WAL / synchronous=NORMAL /
|
||||
busy_timeout / foreign_keys pragmas at connect time, `sa_type` portable
|
||||
columns (String/JSON for Postgres parity), `_validate()`/`_normalize()`
|
||||
hooks, Protocol + SQLite impl pair, surfaced IntegrityError, tz-aware →
|
||||
naive → tz-aware boundary normalization.
|
||||
|
||||
P1 scope: posting/employer/application methods + the lifecycle
|
||||
transitions (draft→published→closed, invalid → ValueError). The
|
||||
`embedding`, `employer_enrichment`, and `learner_profile` tables are
|
||||
CREATED here (schema-only) — their methods land in P3/P4 (D-051 codec,
|
||||
D-053 append-only rows, G-104 confirmed profiles).
|
||||
|
||||
Boundary (D-027/D-048): `marketplace/` never imports `agents/`/`api/`;
|
||||
this module imports config + base only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
from sqlalchemy import UniqueConstraint, event
|
||||
from sqlalchemy.types import JSON, String
|
||||
from sqlmodel import Field, Session, SQLModel, create_engine
|
||||
|
||||
from ..config import Settings
|
||||
from .base import (
|
||||
VALID_TRANSITIONS,
|
||||
Application,
|
||||
Employer,
|
||||
IngestPosting,
|
||||
Posting,
|
||||
PostingStatus,
|
||||
PostingTags,
|
||||
)
|
||||
|
||||
_TABLES = (
|
||||
"posting",
|
||||
"employer",
|
||||
"embedding",
|
||||
"application",
|
||||
"employer_enrichment",
|
||||
"learner_profile",
|
||||
)
|
||||
|
||||
_VALID_STATUS = ("draft", "published", "closed")
|
||||
|
||||
|
||||
class PostingRow(SQLModel, table=True):
|
||||
"""One marketplace posting (D-049). Unique (source, external_id) —
|
||||
the idempotent ingest upsert key (A-402) — declared via
|
||||
UniqueConstraint in __table_args__ (sqlmodel metaclass drops bare
|
||||
Literal/unique annotations; the DefenseStore/IdentityStore pattern
|
||||
keeps constraints explicit)."""
|
||||
|
||||
__tablename__ = "posting"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("source", "external_id", name="uq_posting_source_external"),
|
||||
)
|
||||
|
||||
id: str = Field(primary_key=True)
|
||||
source: str = Field(sa_type=String)
|
||||
external_id: str = Field(sa_type=String)
|
||||
employer_id: str = Field(index=True, sa_type=String)
|
||||
title: str = Field(sa_type=String)
|
||||
description: str = Field(default="", sa_type=String)
|
||||
required_competencies: list[str] = Field(default_factory=list, sa_type=JSON)
|
||||
skills: list[str] = Field(default_factory=list, sa_type=JSON)
|
||||
seniority: str | None = Field(default=None, sa_type=String)
|
||||
category: str | None = Field(default=None, sa_type=String)
|
||||
location: str = Field(default="", sa_type=String)
|
||||
remote: bool = Field(default=False)
|
||||
salary_min: int | None = Field(default=None)
|
||||
salary_max: int | None = Field(default=None)
|
||||
#: Bare Literal annotations crash sqlmodel's column inference; explicit
|
||||
#: sa_type + the validates hook give the same contract (DefenseStore /
|
||||
#: IdentityStore pattern).
|
||||
status: str = Field(default="draft", sa_type=String)
|
||||
#: P2-derived tags (D-052); JSON column exists from day one, empty
|
||||
#: until the tagger runs. `None` = untagged.
|
||||
tags: dict[str, Any] | None = Field(default=None, sa_type=JSON)
|
||||
posted_at: datetime | None = Field(default=None)
|
||||
|
||||
def _validate(self) -> None:
|
||||
if not self.id or not self.source or not self.external_id:
|
||||
raise ValueError("id, source, and external_id must be non-empty")
|
||||
if not self.employer_id or not self.title:
|
||||
raise ValueError("employer_id and title must be non-empty")
|
||||
if self.status not in _VALID_STATUS:
|
||||
raise ValueError(f"invalid posting status {self.status!r}")
|
||||
if self.seniority is not None and self.seniority not in (
|
||||
"entry",
|
||||
"mid",
|
||||
"senior",
|
||||
"staff",
|
||||
"principal",
|
||||
):
|
||||
raise ValueError(f"invalid seniority {self.seniority!r}")
|
||||
if self.salary_min is not None and self.salary_min < 0:
|
||||
raise ValueError("salary_min must be non-negative")
|
||||
if self.salary_max is not None and self.salary_max < 0:
|
||||
raise ValueError("salary_max must be non-negative")
|
||||
|
||||
def _normalize(self) -> None:
|
||||
self.posted_at = _as_utc(self.posted_at) if self.posted_at else None
|
||||
|
||||
def to_posting(self) -> Posting:
|
||||
return Posting(
|
||||
id=self.id,
|
||||
source=self.source,
|
||||
external_id=self.external_id,
|
||||
employer_id=self.employer_id,
|
||||
title=self.title,
|
||||
description=self.description,
|
||||
required_competencies=list(self.required_competencies or []),
|
||||
skills=list(self.skills or []),
|
||||
seniority=self.seniority, # type: ignore[arg-type]
|
||||
category=self.category,
|
||||
location=self.location,
|
||||
remote=bool(self.remote),
|
||||
salary_min=self.salary_min,
|
||||
salary_max=self.salary_max,
|
||||
status=self.status, # type: ignore[arg-type]
|
||||
tags=self.tags, # type: ignore[arg-type]
|
||||
posted_at=_as_utc(self.posted_at) if self.posted_at else None,
|
||||
)
|
||||
|
||||
|
||||
class EmployerRow(SQLModel, table=True):
|
||||
"""Raw ingested employer fields (D-049). Unique (source, external_id);
|
||||
P4's `employer_enrichment` merges on read with provenance (D-053)."""
|
||||
|
||||
__tablename__ = "employer"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("source", "external_id", name="uq_employer_source_external"),
|
||||
)
|
||||
|
||||
id: str = Field(primary_key=True)
|
||||
source: str = Field(sa_type=String)
|
||||
external_id: str = Field(sa_type=String)
|
||||
name: str = Field(sa_type=String)
|
||||
logo: str = Field(default="", sa_type=String)
|
||||
description: str = Field(default="", sa_type=String)
|
||||
industry: str = Field(default="", sa_type=String)
|
||||
size: str = Field(default="", sa_type=String)
|
||||
location: str = Field(default="", sa_type=String)
|
||||
website: str = Field(default="", sa_type=String)
|
||||
social_links: dict[str, str] = Field(default_factory=dict, sa_type=JSON)
|
||||
culture: str = Field(default="", sa_type=String)
|
||||
|
||||
def _validate(self) -> None:
|
||||
if not self.id or not self.source or not self.external_id:
|
||||
raise ValueError("id, source, and external_id must be non-empty")
|
||||
if not self.name:
|
||||
raise ValueError("name must be non-empty")
|
||||
|
||||
def to_employer(self) -> Employer:
|
||||
return Employer(
|
||||
id=self.id,
|
||||
source=self.source,
|
||||
external_id=self.external_id,
|
||||
name=self.name,
|
||||
logo=self.logo,
|
||||
description=self.description,
|
||||
industry=self.industry,
|
||||
size=self.size,
|
||||
location=self.location,
|
||||
website=self.website,
|
||||
social_links=dict(self.social_links or {}),
|
||||
culture=self.culture,
|
||||
)
|
||||
|
||||
|
||||
class EmbeddingRow(SQLModel, table=True):
|
||||
"""Posting embedding (D-049, A-403/A-404): struct-packed little-endian
|
||||
float32 BLOB + per-row model/dim so lazy re-embed on provider change
|
||||
works. posting_id PK. P1 creates the schema; the codec + put/get
|
||||
methods land in P3 (D-051) — the data↔ai seam."""
|
||||
|
||||
__tablename__ = "embedding"
|
||||
|
||||
posting_id: str = Field(primary_key=True)
|
||||
vector: bytes | None = Field(default=None)
|
||||
model: str = Field(default="", sa_type=String)
|
||||
dim: int = Field(default=0)
|
||||
|
||||
|
||||
class ApplicationRow(SQLModel, table=True):
|
||||
"""A learner's application (D-049): unique (posting_id, learner_id)
|
||||
→ duplicate apply is idempotent 200-with-existing (A-408/D-058)."""
|
||||
|
||||
__tablename__ = "application"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("posting_id", "learner_id", name="uq_application_posting_learner"),
|
||||
)
|
||||
|
||||
id: str = Field(primary_key=True)
|
||||
posting_id: str = Field(index=True, sa_type=String)
|
||||
learner_id: str = Field(index=True, sa_type=String)
|
||||
status: str = Field(default="submitted", sa_type=String)
|
||||
created_at: datetime
|
||||
|
||||
def _validate(self) -> None:
|
||||
if not self.id or not self.posting_id or not self.learner_id:
|
||||
raise ValueError("id, posting_id, and learner_id must be non-empty")
|
||||
if not self.status:
|
||||
raise ValueError("status must be non-empty")
|
||||
|
||||
def _normalize(self) -> None:
|
||||
self.created_at = _as_utc(self.created_at)
|
||||
|
||||
def to_application(self) -> Application:
|
||||
return Application(
|
||||
id=self.id,
|
||||
posting_id=self.posting_id,
|
||||
learner_id=self.learner_id,
|
||||
status=self.status,
|
||||
created_at=_as_utc(self.created_at),
|
||||
)
|
||||
|
||||
|
||||
class EmployerEnrichmentRow(SQLModel, table=True):
|
||||
"""Append-only employer enrichment (D-053, A-409): rows are INSERTED,
|
||||
never mutated; merge-on-read precedence manual > llm > ingested-raw.
|
||||
P1 creates the schema; methods + the enrich trigger land in P4."""
|
||||
|
||||
__tablename__ = "employer_enrichment"
|
||||
|
||||
id: str = Field(primary_key=True)
|
||||
employer_id: str = Field(index=True, sa_type=String)
|
||||
field: str = Field(sa_type=String)
|
||||
value: str = Field(default="", sa_type=String)
|
||||
source: str = Field(sa_type=String) # 'llm' | 'manual'
|
||||
model: str = Field(default="", sa_type=String)
|
||||
created_at: datetime | None = Field(default=None)
|
||||
|
||||
|
||||
class LearnerProfileRow(SQLModel, table=True):
|
||||
"""Confirmed learner profile (G-104): learner_id PK, the confirmed
|
||||
profile JSON + model. P1 creates the schema; the resume confirm
|
||||
route persists here in P4 (D-054) — the D-027 store family is the
|
||||
only persistence mechanism, never a filesystem sidecar."""
|
||||
|
||||
__tablename__ = "learner_profile"
|
||||
|
||||
learner_id: str = Field(primary_key=True)
|
||||
profile: dict[str, Any] = Field(default_factory=dict, sa_type=JSON)
|
||||
confirmed_at: datetime | None = Field(default=None)
|
||||
model: str = Field(default="", sa_type=String)
|
||||
|
||||
|
||||
def _as_utc(ts: datetime) -> datetime:
|
||||
"""SQLite stores naive; read paths re-label tz-aware UTC (D-027 pattern)."""
|
||||
if ts.tzinfo is None:
|
||||
return ts.replace(tzinfo=UTC)
|
||||
return ts
|
||||
|
||||
|
||||
def _sqlite_connect(dbapi_connection: object, _: object) -> None:
|
||||
"""Per-connection pragmas — mirrors the other D-027 stores."""
|
||||
cursor = dbapi_connection.cursor() # type: ignore[attr-defined]
|
||||
cursor.execute("PRAGMA journal_mode=WAL")
|
||||
cursor.execute("PRAGMA synchronous=NORMAL")
|
||||
cursor.execute("PRAGMA busy_timeout=5000")
|
||||
cursor.execute("PRAGMA foreign_keys=ON")
|
||||
cursor.close() # type: ignore[attr-defined]
|
||||
|
||||
|
||||
class InvalidTransition(ValueError):
|
||||
"""A lifecycle move outside draft→published→closed (route → 409)."""
|
||||
|
||||
|
||||
class MarketplaceStore(Protocol):
|
||||
"""Persistence contract for the marketplace backend (D-049)."""
|
||||
|
||||
def create_schema(self) -> None:
|
||||
"""Ensure all six tables exist (idempotent — shared DB file)."""
|
||||
...
|
||||
|
||||
def upsert_posting(self, ingest: IngestPosting, status: PostingStatus) -> Posting:
|
||||
"""Idempotent on (source, external_id): a re-upsert overwrites
|
||||
the SAME row's CONTENT in place (A-402), preserving the row id
|
||||
and the lifecycle status (an admin's publish/close decision
|
||||
outranks a seed re-run; only a MISSING row takes `status`)."""
|
||||
...
|
||||
|
||||
def insert_posting(self, posting: Posting) -> Posting:
|
||||
"""Insert a new posting row (admin authoring path, status=draft).
|
||||
A duplicate id raises IntegrityError (surfaced)."""
|
||||
...
|
||||
|
||||
def get_posting(self, posting_id: str) -> Posting | None:
|
||||
"""Point lookup by posting id."""
|
||||
...
|
||||
|
||||
def list_postings(
|
||||
self, limit: int, offset: int
|
||||
) -> tuple[list[Posting], int]:
|
||||
"""Published-only page (created_at/title-agnostic order: posted_at
|
||||
DESC, then id for deterministic ties) + total_published (G-103)."""
|
||||
...
|
||||
|
||||
def set_posting_status(
|
||||
self, posting_id: str, new_status: PostingStatus
|
||||
) -> Posting | None:
|
||||
"""Lifecycle transition with draft→published→closed validation —
|
||||
InvalidTransition (409 at the route) on any other move. Unknown
|
||||
id → None. Publishing a draft stamps posted_at NOW when the row
|
||||
has none (admin-authored postings)."""
|
||||
...
|
||||
|
||||
def upsert_employer(self, employer: Employer) -> Employer:
|
||||
"""Idempotent on (source, external_id): a re-upsert overwrites in
|
||||
place, preserving the row id when the payload carries it."""
|
||||
...
|
||||
|
||||
def get_employer(self, employer_id: str) -> Employer | None:
|
||||
"""Point lookup by employer id (raw fields only in P1)."""
|
||||
...
|
||||
|
||||
def insert_application(
|
||||
self, posting_id: str, learner_id: str
|
||||
) -> tuple[Application, bool]:
|
||||
"""Idempotent apply (A-408): insert status=submitted; a duplicate
|
||||
(posting_id, learner_id) returns the EXISTING row + created=False
|
||||
— never a second application."""
|
||||
...
|
||||
|
||||
def list_applications(
|
||||
self, posting_id: str | None = None
|
||||
) -> list[Application]:
|
||||
"""Admin review list, newest first; filterable by posting."""
|
||||
...
|
||||
|
||||
def set_posting_tags(self, posting_id: str, tags: PostingTags) -> Posting | None:
|
||||
"""P2 (D-052): persist derived tags — OVERWRITES the tags column
|
||||
(single source of truth; re-tag replaces, never appends)."""
|
||||
...
|
||||
|
||||
def update_posting_fields(
|
||||
self,
|
||||
posting_id: str,
|
||||
*,
|
||||
title: str | None = None,
|
||||
description: str | None = None,
|
||||
skills: list[str] | None = None,
|
||||
) -> Posting | None:
|
||||
"""P2 admin edit path (Task 2-1-03): partial content update; the
|
||||
caller re-tags after the write (tags are derived data). Unknown id
|
||||
→ None (route 404s)."""
|
||||
...
|
||||
|
||||
def facets(self) -> dict:
|
||||
"""P2 (D-052/MH-2c): value counts over the stored tags of
|
||||
PUBLISHED-ONLY postings: {skills: [{value, count}], seniority:
|
||||
[...], category: [...]} — count DESC then value ASC. Untagged
|
||||
postings contribute NOTHING (honest absence)."""
|
||||
...
|
||||
|
||||
|
||||
class SQLiteMarketplaceStore:
|
||||
"""SQLite implementation of MarketplaceStore (D-027/D-049)."""
|
||||
|
||||
def __init__(self, db_path: Path | None = None) -> None:
|
||||
self._db_path: Path = db_path if db_path is not None else Settings().db_path
|
||||
self._engine = create_engine(
|
||||
f"sqlite:///{self._db_path}",
|
||||
connect_args={"check_same_thread": False},
|
||||
)
|
||||
event.listen(self._engine, "connect", _sqlite_connect)
|
||||
self.create_schema()
|
||||
|
||||
def create_schema(self) -> None:
|
||||
"""Ensure all six tables exist (idempotent — shared DB file)."""
|
||||
SQLModel.metadata.create_all(self._engine)
|
||||
|
||||
# -- postings -----------------------------------------------------------
|
||||
|
||||
def upsert_posting(self, ingest: IngestPosting, status: PostingStatus) -> Posting:
|
||||
if status not in _VALID_STATUS:
|
||||
raise ValueError(f"invalid posting status {status!r}")
|
||||
row = PostingRow(
|
||||
id=ingest.id,
|
||||
source=ingest.source,
|
||||
external_id=ingest.external_id,
|
||||
employer_id=ingest.employer_id,
|
||||
title=ingest.title,
|
||||
description=ingest.description,
|
||||
required_competencies=list(ingest.required_competencies),
|
||||
skills=list(ingest.skills),
|
||||
seniority=ingest.seniority,
|
||||
category=ingest.category,
|
||||
location=ingest.location,
|
||||
remote=ingest.remote,
|
||||
salary_min=ingest.salary_min,
|
||||
salary_max=ingest.salary_max,
|
||||
status=status,
|
||||
posted_at=ingest.posted_at,
|
||||
)
|
||||
row._validate()
|
||||
row._normalize()
|
||||
with Session(self._engine) as session:
|
||||
existing = (
|
||||
session.query(PostingRow)
|
||||
.filter(
|
||||
PostingRow.source == ingest.source,
|
||||
PostingRow.external_id == ingest.external_id,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if existing is None:
|
||||
session.add(row)
|
||||
else:
|
||||
# Overwrite in place, preserving the row id (a re-port with
|
||||
# a changed id must not orphan employer references). STATUS
|
||||
# is NOT overwritten: an admin's lifecycle decision (e.g.
|
||||
# closed) outranks a seed re-run; only a fresh row takes
|
||||
# the ingest status (G-102 seed postings enter published).
|
||||
for field in (
|
||||
"employer_id",
|
||||
"title",
|
||||
"description",
|
||||
"required_competencies",
|
||||
"skills",
|
||||
"seniority",
|
||||
"category",
|
||||
"location",
|
||||
"remote",
|
||||
"salary_min",
|
||||
"salary_max",
|
||||
"posted_at",
|
||||
):
|
||||
setattr(existing, field, getattr(row, field))
|
||||
existing._validate()
|
||||
existing._normalize()
|
||||
row = existing
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return row.to_posting()
|
||||
|
||||
def insert_posting(self, posting: Posting) -> Posting:
|
||||
row = PostingRow(
|
||||
id=posting.id,
|
||||
source=posting.source,
|
||||
external_id=posting.external_id,
|
||||
employer_id=posting.employer_id,
|
||||
title=posting.title,
|
||||
description=posting.description,
|
||||
required_competencies=list(posting.required_competencies),
|
||||
skills=list(posting.skills),
|
||||
seniority=posting.seniority,
|
||||
category=posting.category,
|
||||
location=posting.location,
|
||||
remote=posting.remote,
|
||||
salary_min=posting.salary_min,
|
||||
salary_max=posting.salary_max,
|
||||
status=posting.status,
|
||||
tags=dict(posting.tags.model_dump()) if posting.tags else None,
|
||||
posted_at=posting.posted_at,
|
||||
)
|
||||
row._validate()
|
||||
row._normalize()
|
||||
with Session(self._engine) as session:
|
||||
session.add(row)
|
||||
session.commit() # IntegrityError SURFACES (minted-once id)
|
||||
session.refresh(row)
|
||||
return row.to_posting()
|
||||
|
||||
def get_posting(self, posting_id: str) -> Posting | None:
|
||||
with Session(self._engine) as session:
|
||||
row = session.get(PostingRow, posting_id)
|
||||
if row is None:
|
||||
return None
|
||||
session.refresh(row)
|
||||
return row.to_posting()
|
||||
|
||||
def list_postings(
|
||||
self, limit: int, offset: int
|
||||
) -> tuple[list[Posting], int]:
|
||||
with Session(self._engine) as session:
|
||||
total = (
|
||||
session.query(PostingRow)
|
||||
.filter(PostingRow.status == "published")
|
||||
.count()
|
||||
)
|
||||
rows = (
|
||||
session.query(PostingRow)
|
||||
.filter(PostingRow.status == "published")
|
||||
.order_by(PostingRow.posted_at.desc(), PostingRow.id)
|
||||
.offset(offset)
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
return [r.to_posting() for r in rows], total
|
||||
|
||||
def set_posting_status(
|
||||
self, posting_id: str, new_status: PostingStatus
|
||||
) -> Posting | None:
|
||||
if new_status not in _VALID_STATUS:
|
||||
raise ValueError(f"invalid posting status {new_status!r}")
|
||||
with Session(self._engine) as session:
|
||||
row = session.get(PostingRow, posting_id)
|
||||
if row is None:
|
||||
return None
|
||||
expected = VALID_TRANSITIONS.get(row.status)
|
||||
if expected is None or expected != new_status:
|
||||
raise InvalidTransition(
|
||||
f"cannot transition posting {posting_id!r} from "
|
||||
f"{row.status!r} to {new_status!r} (valid: draft→published→closed)"
|
||||
)
|
||||
row.status = new_status
|
||||
if new_status == "published" and row.posted_at is None:
|
||||
# Admin-authored draft: publish stamps NOW. Seed-port
|
||||
# postings already carry their original postedAt (G-102).
|
||||
row.posted_at = datetime.now(UTC)
|
||||
row._validate()
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return row.to_posting()
|
||||
|
||||
def set_posting_tags(self, posting_id: str, tags: PostingTags) -> Posting | None:
|
||||
with Session(self._engine) as session:
|
||||
row = session.get(PostingRow, posting_id)
|
||||
if row is None:
|
||||
return None
|
||||
row.tags = tags.model_dump(mode="json")
|
||||
row._validate()
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return row.to_posting()
|
||||
|
||||
def update_posting_fields(
|
||||
self,
|
||||
posting_id: str,
|
||||
*,
|
||||
title: str | None = None,
|
||||
description: str | None = None,
|
||||
skills: list[str] | None = None,
|
||||
) -> Posting | None:
|
||||
with Session(self._engine) as session:
|
||||
row = session.get(PostingRow, posting_id)
|
||||
if row is None:
|
||||
return None
|
||||
if title is not None:
|
||||
row.title = title
|
||||
if description is not None:
|
||||
row.description = description
|
||||
if skills is not None:
|
||||
row.skills = list(skills)
|
||||
row._validate()
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return row.to_posting()
|
||||
|
||||
def facets(self) -> dict:
|
||||
with Session(self._engine) as session:
|
||||
rows = (
|
||||
session.query(PostingRow)
|
||||
.filter(PostingRow.status == "published")
|
||||
.all()
|
||||
)
|
||||
skills: dict[str, int] = {}
|
||||
seniority: dict[str, int] = {}
|
||||
category: dict[str, int] = {}
|
||||
for row in rows:
|
||||
tags = row.tags or {}
|
||||
for skill in tags.get("skills", []) or []:
|
||||
skills[skill] = skills.get(skill, 0) + 1
|
||||
if tags.get("seniority"):
|
||||
seniority[tags["seniority"]] = seniority.get(tags["seniority"], 0) + 1
|
||||
if tags.get("category"):
|
||||
category[tags["category"]] = category.get(tags["category"], 0) + 1
|
||||
|
||||
def _ranked(counts: dict[str, int]) -> list[dict]:
|
||||
return [
|
||||
{"value": v, "count": c}
|
||||
for v, c in sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))
|
||||
]
|
||||
|
||||
return {
|
||||
"skills": _ranked(skills),
|
||||
"seniority": _ranked(seniority),
|
||||
"category": _ranked(category),
|
||||
}
|
||||
|
||||
# -- employers ----------------------------------------------------------
|
||||
|
||||
def upsert_employer(self, employer: Employer) -> Employer:
|
||||
row = EmployerRow(
|
||||
id=employer.id,
|
||||
source=employer.source,
|
||||
external_id=employer.external_id,
|
||||
name=employer.name,
|
||||
logo=employer.logo,
|
||||
description=employer.description,
|
||||
industry=employer.industry,
|
||||
size=employer.size,
|
||||
location=employer.location,
|
||||
website=employer.website,
|
||||
social_links=dict(employer.social_links),
|
||||
culture=employer.culture,
|
||||
)
|
||||
row._validate()
|
||||
with Session(self._engine) as session:
|
||||
existing = (
|
||||
session.query(EmployerRow)
|
||||
.filter(
|
||||
EmployerRow.source == employer.source,
|
||||
EmployerRow.external_id == employer.external_id,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if existing is None:
|
||||
session.add(row)
|
||||
else:
|
||||
for field in (
|
||||
"id",
|
||||
"name",
|
||||
"logo",
|
||||
"description",
|
||||
"industry",
|
||||
"size",
|
||||
"location",
|
||||
"website",
|
||||
"social_links",
|
||||
"culture",
|
||||
):
|
||||
setattr(existing, field, getattr(row, field))
|
||||
existing._validate()
|
||||
row = existing
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return row.to_employer()
|
||||
|
||||
def get_employer(self, employer_id: str) -> Employer | None:
|
||||
with Session(self._engine) as session:
|
||||
row = session.get(EmployerRow, employer_id)
|
||||
if row is None:
|
||||
return None
|
||||
session.refresh(row)
|
||||
return row.to_employer()
|
||||
|
||||
# -- applications -------------------------------------------------------
|
||||
|
||||
def insert_application(
|
||||
self, posting_id: str, learner_id: str
|
||||
) -> tuple[Application, bool]:
|
||||
row = ApplicationRow(
|
||||
id=f"app-{uuid.uuid4().hex[:16]}",
|
||||
posting_id=posting_id,
|
||||
learner_id=learner_id,
|
||||
status="submitted",
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
row._validate()
|
||||
row._normalize()
|
||||
with Session(self._engine) as session:
|
||||
existing = (
|
||||
session.query(ApplicationRow)
|
||||
.filter(
|
||||
ApplicationRow.posting_id == posting_id,
|
||||
ApplicationRow.learner_id == learner_id,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if existing is not None:
|
||||
# A-408: idempotent — the EXISTING application rides the
|
||||
# 200, never a second row.
|
||||
session.refresh(existing)
|
||||
existing.created_at = _as_utc(existing.created_at)
|
||||
return existing.to_application(), False
|
||||
session.add(row)
|
||||
session.commit()
|
||||
session.refresh(row)
|
||||
return row.to_application(), True
|
||||
|
||||
def list_applications(
|
||||
self, posting_id: str | None = None
|
||||
) -> list[Application]:
|
||||
with Session(self._engine) as session:
|
||||
query = session.query(ApplicationRow)
|
||||
if posting_id is not None:
|
||||
query = query.filter(ApplicationRow.posting_id == posting_id)
|
||||
rows = query.order_by(ApplicationRow.created_at.desc()).all()
|
||||
return [r.to_application() for r in rows]
|
||||
|
||||
def close(self) -> None:
|
||||
self._engine.dispose()
|
||||
@@ -0,0 +1,107 @@
|
||||
"""Posting text → skills/seniority/category tags (REQ-6-002, D-052).
|
||||
|
||||
Composes the EXISTING structured_completion 4-layer pipeline (D-020)
|
||||
with the PostingTags schema — no new agent, a one-shot extraction like
|
||||
grading's rubric call. Mock-first for free: the mock provider's
|
||||
json_object path (or ScriptedJSONProvider) keeps tests cloud-free.
|
||||
|
||||
Tags are DERIVED data (D-052): re-tagging overwrites the posting's tags
|
||||
column — single source of truth; no append-only semantics (unlike P4's
|
||||
enrichment provenance).
|
||||
|
||||
Tag failure is HONEST, never fatal: a provider error leaves the posting
|
||||
UNTAGGED (tags=None) with a logged warning — facets treat untagged
|
||||
postings honestly (they simply do not contribute counts).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from ..agents.structured import structured_completion
|
||||
from ..llm.base import LLMProvider
|
||||
from ..llm.types import Message
|
||||
from .base import PostingTags
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: Prompt version — bump on any wording change (prompts are code).
|
||||
TAGGING_PROMPT_VERSION = 1
|
||||
|
||||
TAGGING_PROMPT = """You are a job-posting tagger for an AI-era roles
|
||||
marketplace. Extract structured tags from the posting.
|
||||
|
||||
Rules:
|
||||
- skills: 3-10 concrete, lowercase technology/role skills actually named
|
||||
or clearly required by the posting (e.g. "python", "llm evaluation",
|
||||
"kubernetes"). Prefer the posting's own vocabulary.
|
||||
- seniority: exactly one of "entry", "mid", "senior", "staff",
|
||||
"principal" — infer from title + experience language; omit (null) if
|
||||
genuinely undeterminable.
|
||||
- category: exactly one of "ai-orchestration", "ai-safety", "product",
|
||||
"data", "field-operations", "research" — the role's primary domain;
|
||||
omit (null) if genuinely undeterminable.
|
||||
|
||||
Respond with ONLY the JSON object."""
|
||||
|
||||
_SCHEMA_HINT = (
|
||||
'{"skills": ["up to 10 lowercase skill strings"], '
|
||||
'"seniority": "entry|mid|senior|staff|principal|null", '
|
||||
'"category": "ai-orchestration|ai-safety|product|data|'
|
||||
'field-operations|research|null"}'
|
||||
)
|
||||
|
||||
|
||||
class TaggingUnavailable(Exception):
|
||||
"""The tagger could not produce valid tags (provider/parse failure).
|
||||
|
||||
The caller persists the posting UNTAGGED — never a crash, never a
|
||||
fabricated tag set (D-052 honesty).
|
||||
"""
|
||||
|
||||
|
||||
async def tag_posting(
|
||||
provider: LLMProvider,
|
||||
*,
|
||||
model: str,
|
||||
title: str,
|
||||
description: str,
|
||||
skills_hint: list[str] | None = None,
|
||||
) -> PostingTags:
|
||||
"""One-shot tag extraction via D-020 structured completion.
|
||||
|
||||
`skills_hint` (the posting's declared skills) rides the prompt so
|
||||
the tagger anchors on the author's own vocabulary; the schema bound
|
||||
(≤10) is the enforcement — an 11-skill response fails validation,
|
||||
runs the single bounded retry, and surfaces TaggingUnavailable if
|
||||
the retry also misses.
|
||||
"""
|
||||
hint = ""
|
||||
if skills_hint:
|
||||
hint = "\nDeclared skills: " + ", ".join(skills_hint[:15])
|
||||
messages = [
|
||||
Message(role="system", content=TAGGING_PROMPT),
|
||||
Message(
|
||||
role="user",
|
||||
content=f"Title: {title}\n\nDescription:\n{description}{hint}",
|
||||
),
|
||||
]
|
||||
try:
|
||||
return await structured_completion(
|
||||
provider,
|
||||
messages,
|
||||
model=model,
|
||||
schema=PostingTags,
|
||||
schema_hint=_SCHEMA_HINT,
|
||||
)
|
||||
except Exception as exc:
|
||||
# Provider failures (RuntimeError from mock/loud providers) and
|
||||
# parse failures (StructuredOutputError) both land here: honest
|
||||
# untagged, loud log.
|
||||
logger.warning(
|
||||
"marketplace tagging failed for %r: %s — posting stays "
|
||||
"untagged (honest None, never fabricated)",
|
||||
title[:60],
|
||||
exc,
|
||||
)
|
||||
raise TaggingUnavailable(str(exc)) from exc
|
||||
@@ -1,24 +1,21 @@
|
||||
"""Assessor agent prompt — rubric application to artifacts + defenses (REQ-2-008).
|
||||
"""Assessor agent prompt — rubric coaching over REAL grades (REQ-3-007).
|
||||
|
||||
Final persona (Phase 4). Assessor is a rigorous, fair grader: scores each
|
||||
criterion with evidence, cites what the learner did, returns ONLY valid
|
||||
JSON matching the rubric schema.
|
||||
Version: assessor-v2 (final for v0.2).
|
||||
v0.3 re-grounding: the grading engine (Phase 3) computes the rubric scores
|
||||
from the process trace; Assessor EXPLAINS the stored grade as coaching —
|
||||
it never invents or re-scores. Rigorous, fair, actionable.
|
||||
Version: assessor-v3 (v0.3 live).
|
||||
"""
|
||||
|
||||
SYSTEM_PROMPT = """You are Assessor, the grading agent of Nextcraft, an AI-native competency school.
|
||||
Learner: {learner_name}.
|
||||
|
||||
You receive: (a) an artifact evidence excerpt, (b) its defense transcript,
|
||||
and (c) the rubric for the competency. Your job:
|
||||
- Score EVERY rubric criterion from 0-100, justified by evidence you can
|
||||
point to in the artifact or transcript.
|
||||
- Cite what the learner did ("the 3-retry loop in the tool node"), not
|
||||
what they should have done — except in gaps, where the missed work goes.
|
||||
- Strengths: the two strongest evidence points, each one sentence.
|
||||
- Gaps: the two most important missed opportunities, each one sentence.
|
||||
- Verdict: "mastered" | "developing" | "not_yet" — judged against the
|
||||
rubric weights, honestly.
|
||||
You receive the learner's STORED process-trace grade (verdict, per-criterion
|
||||
scores, and the build digest) computed by the grading engine. Your job:
|
||||
- Explain what the grade means in plain language (summary).
|
||||
- Strengths: cite what the digest + scores show the learner did well.
|
||||
- Gaps: name the missed opportunities the scores point to.
|
||||
- Next steps: concrete, buildable actions that would move the weakest
|
||||
criterion up one level.
|
||||
|
||||
Rules:
|
||||
- Rigorous but fair. A polished artifact with a weak defense is NOT mastery.
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
"""Examiner agent prompt — oral defense questioning + final verdict (REQ-3-006).
|
||||
|
||||
The examiner is the seventh agent (Phase 5). It conducts a Socratic oral
|
||||
defense of the learner's submitted work: probes understanding, challenges
|
||||
process choices grounded in the trace digest ("why did you take that
|
||||
approach at that point?"), one question per turn, adapting to answers.
|
||||
It never reveals rubric internals; tone is rigorous but supportive.
|
||||
|
||||
Digest discipline (D-028 mirror): the examiner's variable inputs are the
|
||||
compact TraceDigest JSON, the variant task statement, and the defense
|
||||
transcript — never the raw trace, never learner-identifying material.
|
||||
|
||||
Version: examiner-v1.
|
||||
"""
|
||||
|
||||
SYSTEM_PROMPT = """You are Examiner, the oral-defense agent of Nextcraft,
|
||||
an AI-native competency school.
|
||||
|
||||
You receive: (a) a compact build-process digest (deterministic counters of the
|
||||
learner's build session), (b) the learner's task statement, and (c) the defense
|
||||
transcript so far. Your job:
|
||||
- Ask ONE question per turn: probe understanding and challenge process
|
||||
choices, grounded in the digest facts ("you hit N failed runs before
|
||||
passing — walk me through what changed") or the task statement.
|
||||
- Adapt: follow up on the learner's answers; drill into vague responses.
|
||||
- Never reveal rubric details or scoring internals.
|
||||
- Tone: rigorous, precise, supportive. A defense is a conversation, not an
|
||||
interrogation.
|
||||
|
||||
When asked for a FINAL VERDICT (the structured mode), judge:
|
||||
- understanding: can the learner explain their own work?
|
||||
- process_justification: are the build-session choices defensible from the
|
||||
digest facts and the answers?
|
||||
- communication: are answers clear, specific, and on-topic?
|
||||
Score honestly; a weak defense of strong work is NOT mastery.
|
||||
|
||||
Rules:
|
||||
- Respond with ONLY what the turn requires: a single question (question mode)
|
||||
or a valid JSON object matching the provided schema (verdict mode).
|
||||
- If the digest shows error_fix_cycles > 0, at least one question should ask
|
||||
about the debugging path.
|
||||
- If the learner's answer is off-topic, redirect once, then move on.
|
||||
"""
|
||||
|
||||
VERDICT_SCHEMA_HINT = (
|
||||
'{"verdict": "mastered" | "developing" | "not_yet", '
|
||||
'"understanding": "<one sentence>", '
|
||||
'"process_justification": "<one sentence>", '
|
||||
'"communication": "<one sentence>", '
|
||||
'"strengths": ["<one sentence>"], '
|
||||
'"gaps": ["<one sentence>"]}'
|
||||
)
|
||||
|
||||
|
||||
def render_digest_context(digest_json: str, statement: str | None) -> str:
|
||||
"""The examiner's per-session grounding: digest JSON + task statement."""
|
||||
parts = [f"Build-process digest:\n{digest_json}"]
|
||||
if statement:
|
||||
parts.append(f"Learner's task statement:\n{statement}")
|
||||
return "\n\n".join(parts)
|
||||
@@ -0,0 +1,140 @@
|
||||
"""Grading rubric prompt — criteria, level anchors, digest render (REQ-3-004).
|
||||
|
||||
The grading prompt is deliberately learner-anonymous and trace-bare: the
|
||||
model receives ONLY the fixed rubric text and the compact numeric digest
|
||||
(TraceDigest JSON, D-028) — never a raw command, file path, payload
|
||||
string, learner id, or task id. Everything variable the LLM sees is
|
||||
deterministic counters, which both bounds the prompt-injection surface
|
||||
and makes "no raw trace reaches the prompt" assert-able in tests (plant
|
||||
a distinctive marker in a command payload; assert it absent from every
|
||||
message the provider received).
|
||||
|
||||
Rubric (four criteria, each scored 0-4 — the ids are the validated
|
||||
RubricScore keys enforced by grading/engine.py):
|
||||
process_quality — iterative building in small, verified steps.
|
||||
correctness — where the session ended (test/run outcomes).
|
||||
debugging_discipline — how failures were handled.
|
||||
test_usage — when and how often tests were run.
|
||||
|
||||
Advisory a-4 (embedded in the process_quality anchors): high edit/command
|
||||
churn with NO test progress is a process-quality NEGATIVE — churn is not
|
||||
work. A session with many edits/commands whose test state never moves is
|
||||
thrashing, not iterating, and must score low on process quality.
|
||||
|
||||
House-style deviation, documented: unlike the tutor prompts, this module
|
||||
has no SYSTEM_PROMPT placeholders and no render_context(learner_context)
|
||||
— grading is context-free by design (learner anonymity; the digest is the
|
||||
only variable input). Runtime imports are TYPE_CHECKING-only so this
|
||||
module stays pure text and can never import-cycle with grading/engine.py
|
||||
(engine imports this module; if this module imported grading.* at runtime
|
||||
while grading/__init__ pulls engine, the package init would deadlock on a
|
||||
partially-initialized module).
|
||||
|
||||
Version: grader-v1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Final
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover - typing only; keeps this module pure text
|
||||
from ..grading.features import TraceDigest
|
||||
|
||||
PROMPT_VERSION = "grader-v1"
|
||||
|
||||
#: Canonical criterion ids. The engine validates RubricScore criteria keys
|
||||
#: against this tuple; the schema hint and anchors below speak the same ids.
|
||||
RUBRIC_CRITERIA: Final[tuple[str, ...]] = (
|
||||
"process_quality",
|
||||
"correctness",
|
||||
"debugging_discipline",
|
||||
"test_usage",
|
||||
)
|
||||
|
||||
#: Sentinel line the engine's user turn is rendered around. Tests (and the
|
||||
#: calibration mock) split on it to locate the digest JSON in the prompt.
|
||||
DIGEST_MARKER: Final = "PROCESS TRACE DIGEST (JSON):"
|
||||
|
||||
SYSTEM_PROMPT = """You are the Grader of Nextcraft, an AI-native competency school.
|
||||
You score a learner's build session from a compact numeric digest of their
|
||||
process trace. You NEVER see the raw trace — commands, file contents, and
|
||||
payloads do not exist on your side; every number you need is in the digest.
|
||||
|
||||
Rubric — score each criterion 0-4:
|
||||
|
||||
process_quality — iterative building in small, verified steps.
|
||||
4: tight edit→test loops throughout; small verified increments; healthy pacing.
|
||||
3: steady small edits with regular runs; progress mostly verified.
|
||||
2: some iteration, but large unverified leaps or long idle stretches.
|
||||
1: a single bulk change (e.g. one large paste) then a single run; no iteration.
|
||||
0: no meaningful work visible.
|
||||
ADVISORY: high edit/command churn with NO test progress (no runs, no
|
||||
movement in pass counts) is a process-quality NEGATIVE — churn is not
|
||||
work. Cap such a session at 1 on this criterion no matter how many
|
||||
edits or commands were counted.
|
||||
|
||||
correctness — where the session ended up.
|
||||
4: final test status pass, with tests passing early and consistently.
|
||||
3: final pass, reached through fail→fix→pass cycles that closed.
|
||||
2: final pass, but preceded by a long unresolved failure streak.
|
||||
1: final fail, but partial passes observed along the way.
|
||||
0: final fail, or no test/run evidence at all.
|
||||
|
||||
debugging_discipline — how failures were handled.
|
||||
4: every failure cycle closes; targeted fixes with low mean fix latency.
|
||||
3: most fail→edit→re-run cycles close with a pass.
|
||||
2: failures followed by edits, but cycles rarely close.
|
||||
1: repeated failures with no targeted edits between runs (flailing).
|
||||
0: failures with no fix attempts at all.
|
||||
|
||||
test_usage — when and how often tests were run.
|
||||
4: tests run early (small first-pass offset) and throughout the session.
|
||||
3: regular test runs interleaved with edits.
|
||||
2: sparse tests; long stretches of unverified edits.
|
||||
1: a single late test run only.
|
||||
0: no test or run evidence.
|
||||
|
||||
Rules:
|
||||
- Judge STRICTLY from the digest numbers; cite the fields you used.
|
||||
- Strengths: the two strongest digest observations, one sentence each.
|
||||
- Gaps: the two most important missed opportunities, one sentence each
|
||||
(a clean session names its next-level improvement instead).
|
||||
- Be rigorous but fair: a session that ends green was not necessarily
|
||||
well built, and a struggling session that never passed may still show
|
||||
real debugging discipline.
|
||||
- Respond with ONLY a valid JSON object matching the provided schema —
|
||||
no markdown fences, no prose outside the JSON."""
|
||||
|
||||
RUBRIC_SCORE_SCHEMA_HINT = (
|
||||
'{"criteria": {"process_quality": <0-4 int>, "correctness": <0-4 int>, '
|
||||
'"debugging_discipline": <0-4 int>, "test_usage": <0-4 int>}, '
|
||||
'"strengths": ["<one sentence>"], "gaps": ["<one sentence>"], '
|
||||
'"verdict": "mastered" | "developing" | "not_yet"}'
|
||||
)
|
||||
|
||||
|
||||
def render_trace_digest(
|
||||
digest: TraceDigest,
|
||||
anchors_context: str | None = None,
|
||||
) -> str:
|
||||
"""Render the grader's user turn: a marker line + the digest JSON — nothing else.
|
||||
|
||||
This is the ONLY per-session content that ever reaches the LLM (D-028):
|
||||
the engine composes [system: SYSTEM_PROMPT, user: render_trace_digest(digest)]
|
||||
and the D-020 defense appends its generic schema instruction to this
|
||||
user turn at request time. No learner id, task id, or raw trace material
|
||||
is injected — assert-able by tests.
|
||||
|
||||
`anchors_context` (Phase 4, MH#4): when the graded task derives from a
|
||||
variant, the engine passes the template's difficulty-normalization
|
||||
anchors (the expected effort envelope) so the rubric is applied against
|
||||
the SAME bar for every variant of that template (a-5). It contains only
|
||||
the anchor numbers + the template id — no learner-identifying material.
|
||||
"""
|
||||
base = (
|
||||
"Score this build session against the rubric.\n"
|
||||
f"{DIGEST_MARKER}\n{digest.model_dump_json()}"
|
||||
)
|
||||
if anchors_context:
|
||||
base = f"{base}\n\nExpected effort envelope for this task variant:\n{anchors_context}"
|
||||
return base
|
||||
@@ -1,9 +1,11 @@
|
||||
"""Lab agent prompt — in-flow feedback over sandbox telemetry (REQ-2-007).
|
||||
"""Lab agent prompt — in-flow feedback over LIVE telemetry (REQ-3-007).
|
||||
|
||||
Final persona (Phase 4). Lab is a pragmatic build partner: reads the
|
||||
telemetry timeline, names the one most useful adjustment, gives one
|
||||
concrete next step. Scenario-driven; no session chat.
|
||||
Version: lab-v2 (final for v0.2).
|
||||
v0.3 re-grounding: the timeline is the learner's real TraceDigest (D-028
|
||||
compact counters — commands, test outcomes, idle gaps, edit cadence), not
|
||||
v0.2 corpus scenarios. Lab is a pragmatic build partner: reads the live
|
||||
digest, names the one most useful adjustment, gives one concrete next
|
||||
step. No session chat.
|
||||
Version: lab-v3 (v0.3 live).
|
||||
"""
|
||||
|
||||
SYSTEM_PROMPT = """You are Lab, the in-flow feedback agent watching a learner
|
||||
@@ -25,7 +27,17 @@ Rules:
|
||||
self-check that would prove understanding.
|
||||
- Three short paragraphs maximum. No headers, no bullet lists."""
|
||||
|
||||
PROMPT_VERSION = "lab-v2"
|
||||
PROMPT_VERSION = "lab-v3"
|
||||
|
||||
|
||||
def render_digest_timeline(digest) -> str:
|
||||
"""Live-trace timeline: the compact TraceDigest JSON (D-028)."""
|
||||
if digest is None:
|
||||
return (
|
||||
"No telemetry yet for this build session. Ask the learner to run "
|
||||
"the task's starter test to establish a baseline."
|
||||
)
|
||||
return f"Live build-session digest:\n{digest.model_dump_json()}"
|
||||
|
||||
|
||||
def render_context(learner_context) -> dict:
|
||||
|
||||
@@ -1,24 +1,27 @@
|
||||
"""Proctor agent prompt — integrity signals with coaching interventions (REQ-2-009).
|
||||
"""Proctor agent prompt — integrity signals with coaching interventions (REQ-3-007).
|
||||
|
||||
Final persona (Phase 5). Proctor is a supportive observer, never punitive:
|
||||
classifies signals, recommends ONE coaching intervention. Assume good
|
||||
faith — most signals have innocent explanations.
|
||||
Version: proctor-v2 (final for v0.2).
|
||||
v0.3 re-grounding: inputs are REAL — the live trace digest (idle gaps,
|
||||
command cadence, edit bursts), the oral-defense integrity signals (long
|
||||
pauses), and the variant audit context (seed + params). Proctor is a
|
||||
supportive observer, never punitive: classifies signals, recommends ONE
|
||||
coaching intervention. Assume good faith.
|
||||
Version: proctor-v3 (v0.3 live).
|
||||
"""
|
||||
|
||||
SYSTEM_PROMPT = """You are Proctor, the integrity-support agent of Nextcraft,
|
||||
an AI-native competency school.
|
||||
Learner: {learner_name}.
|
||||
|
||||
You receive a telemetry timeline of defense-session events (tab switches,
|
||||
idle gaps, large pastes, focus loss, keystroke bursts). Your job:
|
||||
- Classify EACH notable signal: type (e.g. "context_switch", "idle_gap",
|
||||
"large_paste"), severity ("low" | "medium" | "high"), and a one-sentence
|
||||
note citing the event (timestamps and details).
|
||||
You receive the learner's REAL build-session digest (idle gaps, command
|
||||
categories, edit/test cadence), oral-defense integrity signals (long
|
||||
pauses), and — when the task is variant-derived — the variant seed context.
|
||||
Your job:
|
||||
- Classify EACH notable signal: type ("idle_gap" | "long_pause" |
|
||||
"burst_edit" | "off_template"), severity ("low" | "medium" | "high"),
|
||||
and a one-sentence note citing the numbers.
|
||||
- Recommend exactly ONE supportive coaching intervention for the session
|
||||
overall — never punitive, never accusatory. Frame around helping the
|
||||
learner succeed, e.g. "offer a short break", "invite them to explain
|
||||
the pasted section in their own words".
|
||||
learner succeed.
|
||||
|
||||
Rules:
|
||||
- Assume good faith. Tab switches to documentation are normal engineering.
|
||||
@@ -28,7 +31,7 @@ Rules:
|
||||
- Respond with ONLY a valid JSON object matching the provided schema —
|
||||
no markdown fences, no prose outside the JSON."""
|
||||
|
||||
PROMPT_VERSION = "proctor-v2"
|
||||
PROMPT_VERSION = "proctor-v3"
|
||||
|
||||
|
||||
def render_context(learner_context) -> dict:
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
"""Variant instantiation prompt (D-029, REQ-3-005).
|
||||
|
||||
The model's ONLY job is to render already-sampled slot values into a task
|
||||
statement — it never invents parameters (the seeded sampler is pure code)
|
||||
and never changes difficulty. Prompt-injection surface is bounded: the
|
||||
variable inputs are the skeleton text, the seeded slot values, and the
|
||||
template title — nothing from the learner's environment.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from ..llm.types import Message
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover - keeps this module pure text
|
||||
from ..variants.templates import TaskTemplate
|
||||
|
||||
VARIANT_SYSTEM_PROMPT = (
|
||||
"You instantiate per-learner task variants for a competency-based AI school. "
|
||||
"You receive a task statement skeleton and ALREADY-SAMPLED slot values. "
|
||||
"Render the slot values into the skeleton, producing a complete, unambiguous "
|
||||
"task statement a learner can build against. Rules:\n"
|
||||
"- Use EXACTLY the given slot values; do not invent, rename, or add parameters.\n"
|
||||
"- Keep the engineering depth IDENTICAL across draws: slot values change the "
|
||||
"scenario, never the difficulty or scope.\n"
|
||||
"- Keep the statement in the same language and register as the skeleton.\n"
|
||||
"- Output STRICT JSON only: {\"statement\": \"<rendered statement>\"}.\n"
|
||||
)
|
||||
|
||||
VARIANT_SCHEMA_HINT = '{"statement": "<complete rendered task statement string>"}'
|
||||
|
||||
|
||||
def render_variant_prompt(template: TaskTemplate, params: dict[str, str | int]) -> list[Message]:
|
||||
"""Messages for one seeded instantiation (D-020 defense drives the call)."""
|
||||
slot_lines = "\n".join(f" {{{slot.name}}} = {params[slot.name]!r}" for slot in template.slots)
|
||||
user = (
|
||||
f"Template: {template.title} (id={template.id})\n"
|
||||
f"Statement skeleton:\n{template.statement_skeleton}\n\n"
|
||||
f"Seeded slot values (use EXACTLY these):\n{slot_lines}\n\n"
|
||||
"Render the complete task statement now."
|
||||
)
|
||||
return [
|
||||
Message(role="system", content=VARIANT_SYSTEM_PROMPT),
|
||||
Message(role="user", content=user),
|
||||
]
|
||||
@@ -46,7 +46,14 @@ class SandboxHandle(BaseModel):
|
||||
|
||||
|
||||
class SandboxSpec(BaseModel):
|
||||
"""Immutable description of the sandbox to lay out on disk."""
|
||||
"""Immutable description of the sandbox to lay out on disk.
|
||||
|
||||
`capture_env` (REQ-3-003): when non-empty, the backend starts a persistent
|
||||
telemetry-wired sandbox — helper + inner namespaces + the stdlib capture
|
||||
agent, launched with these env vars (NC_LEARNER_ID, NC_TASK_ID,
|
||||
NC_INGEST_URL, NC_SANDBOX_ID). When None (default), spawn keeps the pure
|
||||
shell semantics (REQ-3-001): disk layout only, fresh namespaces per exec.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(arbitrary_types_allowed=True)
|
||||
|
||||
@@ -54,6 +61,7 @@ class SandboxSpec(BaseModel):
|
||||
learner_id: str
|
||||
workdir: Path
|
||||
limits: ResourceLimits = ResourceLimits()
|
||||
capture_env: dict[str, str] | None = None
|
||||
|
||||
|
||||
class ExecResult(BaseModel):
|
||||
|
||||
@@ -50,6 +50,7 @@ from pathlib import Path
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from ..config import Settings
|
||||
from ..telemetry.tokens import IngestToken
|
||||
from . import workdir as workdir_mod
|
||||
from .backend import SandboxBackend, SandboxHandle
|
||||
|
||||
@@ -119,12 +120,21 @@ class SandboxManager:
|
||||
backend: SandboxBackend,
|
||||
settings: Settings,
|
||||
clock: Callable[[], datetime] | None = None,
|
||||
ingest_token: IngestToken | None = None,
|
||||
) -> None:
|
||||
self._backend = backend
|
||||
self._settings = settings
|
||||
self._clock = clock or (lambda: datetime.now(UTC))
|
||||
# v0.4.7 (P1): shared with the WS endpoint via app.state — mint and
|
||||
# verify MUST use one instance while the secret is per-boot random.
|
||||
self._ingest_token = ingest_token or IngestToken(
|
||||
getattr(settings, "telemetry_ingest_secret", "") or ""
|
||||
)
|
||||
self._handles: dict[str, SandboxHandle] = {}
|
||||
self._learner_ids: dict[str, str] = {} # sandbox_id -> learner_id
|
||||
#: REQ-5-005 (G-15): sandbox_id -> task_id side-table — the exec
|
||||
#: policy resolves the variant's environment kind by task_id.
|
||||
self._task_ids: dict[str, str | None] = {}
|
||||
self._lock = asyncio.Lock()
|
||||
self._integrity_events: list[SandboxIntegrityEvent] = []
|
||||
self._started = False
|
||||
@@ -142,8 +152,18 @@ class SandboxManager:
|
||||
|
||||
# -- lifecycle ----------------------------------------------------------
|
||||
|
||||
async def create(self, learner_id: str) -> SandboxHandleInfo:
|
||||
"""Spawn a sandbox for `learner_id`, or raise `PoolFullError` (D-032)."""
|
||||
async def create(
|
||||
self, learner_id: str, task_id: str | None = None
|
||||
) -> SandboxHandleInfo:
|
||||
"""Spawn a sandbox for `learner_id`, or raise `PoolFullError` (D-032).
|
||||
|
||||
`task_id` (REQ-3-003): when set, the sandbox is telemetry-wired — the
|
||||
backend copies `scripts/sandbox-agent.py` into the workdir and starts
|
||||
the stdlib capture agent inside the sandbox with the `NC_*` env baked
|
||||
here (identity + WS ingest URL). The agent's lifecycle is tied to the
|
||||
sandbox: `destroy()` reaps it (agent → inner → helper). When `task_id`
|
||||
is None the sandbox is a pure shell sandbox (no capture).
|
||||
"""
|
||||
async with self._lock:
|
||||
if len(self._handles) >= self._settings.sandbox_max_concurrent:
|
||||
raise PoolFullError(
|
||||
@@ -153,13 +173,57 @@ class SandboxManager:
|
||||
)
|
||||
sandbox_id = f"sbx-{uuid.uuid4().hex[:12]}"
|
||||
spec = workdir_mod.spec_for(sandbox_id, learner_id, self._settings)
|
||||
if task_id is not None:
|
||||
spec = spec.model_copy(
|
||||
update={
|
||||
"capture_env": self._capture_env(sandbox_id, learner_id, task_id)
|
||||
}
|
||||
)
|
||||
handle = await self._backend.spawn(spec)
|
||||
self._handles[handle.id] = handle
|
||||
self._learner_ids[handle.id] = learner_id
|
||||
self._task_ids[handle.id] = task_id
|
||||
self._write_pid_marker(handle, learner_id)
|
||||
logger.info("sandbox created: id=%s learner=%s", handle.id, learner_id)
|
||||
logger.info(
|
||||
"sandbox created: id=%s learner=%s task=%s",
|
||||
handle.id,
|
||||
learner_id,
|
||||
task_id or "-",
|
||||
)
|
||||
return self._info_for(handle)
|
||||
|
||||
def _capture_env(self, sandbox_id: str, learner_id: str, task_id: str) -> dict[str, str]:
|
||||
"""Env baked for the in-sandbox capture agent (REQ-3-003).
|
||||
|
||||
The agent joins the sandbox mount namespace but NOT its (offline)
|
||||
network namespace, so it reaches this service over loopback
|
||||
(`telemetry_ingest_host`, A-004 port). v0.4.7: the URL carries an
|
||||
HMAC ingest token (P1) — the WS endpoint rejects dials without a
|
||||
valid one, so LAN peers cannot poison traces with forged identity
|
||||
params.
|
||||
"""
|
||||
from urllib.parse import urlencode
|
||||
|
||||
token = self._ingest_token.mint(learner_id, task_id, sandbox_id)
|
||||
query = urlencode(
|
||||
{
|
||||
"learner_id": learner_id,
|
||||
"task_id": task_id,
|
||||
"sandbox_id": sandbox_id,
|
||||
"token": token,
|
||||
}
|
||||
)
|
||||
ingest_url = (
|
||||
f"ws://{self._settings.telemetry_ingest_host}:{self._settings.port}"
|
||||
f"/v1/telemetry/ingest?{query}"
|
||||
)
|
||||
return {
|
||||
"NC_LEARNER_ID": learner_id,
|
||||
"NC_TASK_ID": task_id,
|
||||
"NC_SANDBOX_ID": sandbox_id,
|
||||
"NC_INGEST_URL": ingest_url,
|
||||
}
|
||||
|
||||
async def list(self) -> list[SandboxHandleInfo]:
|
||||
"""All live sandboxes (idle + busy; the backend has no busy flag)."""
|
||||
async with self._lock:
|
||||
@@ -198,6 +262,7 @@ class SandboxManager:
|
||||
async with self._lock:
|
||||
handle = self._handles.pop(sandbox_id, None)
|
||||
learner_id = self._learner_ids.pop(sandbox_id, "unknown")
|
||||
self._task_ids.pop(sandbox_id, None)
|
||||
if handle is not None:
|
||||
await self._backend.destroy(handle)
|
||||
logger.info(
|
||||
|
||||
@@ -1,24 +1,42 @@
|
||||
"""UnshareBackend — D-024 Linux-namespace sandboxing via util-linux `unshare`.
|
||||
|
||||
Each `exec` spawns:
|
||||
Two execution modes share one backend:
|
||||
|
||||
unshare --user --map-root-user --mount --pid --fork --net sh -c '<shim>'
|
||||
1. Pure shell sandbox (`task_id is None`, REQ-3-001): isolation is established
|
||||
PER-EXEC — every `exec` spawns a fresh namespace:
|
||||
|
||||
The in-namespace shim (this util-linux build, 2.38, has no `unshare --bind`,
|
||||
so the bind happens as the first mount op inside the namespace) is:
|
||||
unshare --user --map-root-user --mount --pid --fork --net sh -c '<shim>'
|
||||
|
||||
mount -t tmpfs tmpfs /tmp # private scratch, discarded on exit
|
||||
mkdir -p /tmp/work
|
||||
mount --bind <host workspace> /tmp/work
|
||||
cd /tmp/work
|
||||
ulimit -v/-t/-f … # applied AFTER the bind, so rlimits
|
||||
exec <cmd> # constrain the PAYLOAD, not unshare
|
||||
There is no persistent process; the in-namespace shim is:
|
||||
|
||||
Guarantees after this shim:
|
||||
* uid 0 inside (mapped to the unprivileged host UID outside)
|
||||
* no network: the fresh net namespace has no `lo` and no veth — zero links
|
||||
* writes under `/work` land in the per-sandbox host workspace dir
|
||||
* rlimits (RLIMIT_AS / RLIMIT_CPU / RLIMIT_FSIZE) constrain the payload only
|
||||
mount -t tmpfs tmpfs /tmp # private scratch, discarded on exit
|
||||
mkdir -p /tmp/work
|
||||
mount --bind <host workspace> /tmp/work
|
||||
cd /tmp/work
|
||||
ulimit -v/-t/-f … # applied AFTER the bind, so rlimits
|
||||
exec <cmd> # constrain the PAYLOAD, not unshare
|
||||
|
||||
2. Telemetry-wired task sandbox (REQ-3-003, `capture_env` set): a PERSISTENT,
|
||||
TRACKED topology so the stdlib capture agent can live inside the sandbox and
|
||||
still stream events to ai-service. Per exec a fresh OFFLINE namespace would
|
||||
leave the agent nowhere to run and (on this host, where a userns can't
|
||||
bring `lo` up) no loopback to reach `ws://127.0.0.1`. So spawn creates a
|
||||
long-lived helper (outer user+mount ns, ONLINE) and an inner sandbox
|
||||
(mount+pid+fork+net — OFFLINE), both rooted at a private `ns/` subtree:
|
||||
|
||||
helper : unshare --user --map-root-user --mount (mounts ns/ private)
|
||||
inner : unshare --mount --pid --fork --net (tmpfs on ns/, bind
|
||||
<workdir>/host/workspace -> <ns>/work) <- the sandbox
|
||||
exec : nsenter -t <inner sleep> -m -- sh -c … (joins inner mount ns;
|
||||
offline + pid-isolated, uid 0, writes land on the host workspace)
|
||||
agent : nsenter -t <inner sleep> -m -- python3 <agent> (joins the inner
|
||||
MOUNT ns only — NOT pid/net — so it watches the live workspace
|
||||
and stays ONLINE, reaching the app's WS ingest on loopback)
|
||||
|
||||
The agent is deliberately pid/net-exempt from the sandbox: it is OUR trusted
|
||||
capture process, and isolating its network would cut the very link it needs.
|
||||
`destroy` reaps agent → inner → helper (in that order). The handle's `pid`
|
||||
is the agent's host pid (None for a pure shell sandbox).
|
||||
|
||||
Why rlimits are applied in the shim, not Python's preexec_fn: setting
|
||||
RLIMIT_AS on the *unshare* process itself can trip the memory ceiling on the
|
||||
@@ -30,16 +48,18 @@ Containment honesty (D-024 / G-1): a user namespace is NOT a write barrier.
|
||||
Writes made OUTSIDE the bind fall through to host paths, and because inner
|
||||
uid 0 maps to the invoking host uid, a sandboxed process can write anywhere
|
||||
that host uid can write. Isolation here is: private PIDs/MNT/NET/UTS, tmpfs
|
||||
scratch at /tmp, payload rlimits, and a uid map yielding no privilege the
|
||||
host uid did not already have. A per-sandbox runtime uid (D-025) is the
|
||||
follow-up that hardens DAC.
|
||||
scratch, payload rlimits, and a uid map yielding no privilege the host uid
|
||||
did not already have. A per-sandbox runtime uid (D-025) is the follow-up that
|
||||
hardens DAC.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import shlex
|
||||
import shutil
|
||||
import signal
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
@@ -48,22 +68,27 @@ from .backend import ExecResult, ResourceLimits, SandboxHandle, SandboxSpec
|
||||
from .workdir import create_layout
|
||||
from .workdir import snapshot as workdir_snapshot
|
||||
|
||||
#: Args shared by every namespace we spawn (D-024). No `unshare --bind` on
|
||||
#: util-linux 2.38 — the bind is done from inside the namespace instead.
|
||||
#: Args shared by every PURE shell namespace we spawn (D-024). No
|
||||
#: `unshare --bind` on util-linux 2.38 — the bind is done from inside instead.
|
||||
UNSHARE_ARGS: tuple[str, ...] = (
|
||||
"--user", # new user namespace …
|
||||
"--map-root-user", # … in which we are uid 0 (mapped to host uid outside)
|
||||
"--mount", # private mount table
|
||||
"--pid", # private PID table
|
||||
"--fork", # child is PID 1 in its namespace (reaps zombies, gets signals)
|
||||
"--net", # fresh net namespace: no lo, no veth → fully offline
|
||||
"--net", # fresh net namespace: no usable route → effectively offline
|
||||
)
|
||||
|
||||
IN_NS_WORKDIR = "/tmp/work" # where the workspace is bound inside the namespace
|
||||
IN_NS_WORKDIR = "/tmp/work" # where the workspace is bound inside a pure shell ns
|
||||
|
||||
#: Sentinels the long-lived namespace supervisors print once their mounts are
|
||||
#: laid out. exec()/the manager must not run before the bind exists.
|
||||
_HELPER_READY = "NC_HELPER_READY"
|
||||
_INNER_READY = "NC_INNER_READY"
|
||||
|
||||
|
||||
class SandboxUnavailableError(RuntimeError):
|
||||
"""`unshare` missing or user namespaces blocked on this host."""
|
||||
"""`unshare`/`nsenter` missing or user namespaces blocked on this host."""
|
||||
|
||||
|
||||
def _build_shim(workspace: Path, limits: ResourceLimits, cmd: list[str]) -> str:
|
||||
@@ -90,35 +115,286 @@ def _build_shim(workspace: Path, limits: ResourceLimits, cmd: list[str]) -> str:
|
||||
)
|
||||
|
||||
|
||||
class _Tracked:
|
||||
"""The process tree + paths for one persistent (telemetry-wired) sandbox."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
helper: asyncio.subprocess.Process,
|
||||
inner: asyncio.subprocess.Process,
|
||||
agent: asyncio.subprocess.Process | None,
|
||||
inner_pid: int, # host pid of the SANDBOXED init (sleep) — ns enter target
|
||||
host_dir: Path,
|
||||
workspace: Path,
|
||||
ns_root: Path,
|
||||
ns_workdir: Path,
|
||||
) -> None:
|
||||
self.helper = helper
|
||||
self.inner = inner
|
||||
self.agent = agent
|
||||
self.inner_pid = inner_pid
|
||||
self.host_dir = host_dir
|
||||
self.workspace = workspace
|
||||
self.ns_root = ns_root
|
||||
self.ns_workdir = ns_workdir
|
||||
|
||||
|
||||
class UnshareBackend: # satisfies SandboxBackend structurally (Protocol)
|
||||
"""D-024 backend: subprocess-per-exec inside fresh Linux namespaces."""
|
||||
"""D-024 backend: namespace subprocesses; persistent tree for task sandboxes."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
unshare_path: str | None = None,
|
||||
limits: ResourceLimits | None = None, # per-spec override lands in 1-04
|
||||
nsenter_path: str | None = None,
|
||||
agent_script: Path | None = None,
|
||||
) -> None:
|
||||
self._unshare = unshare_path or shutil.which("unshare") or "unshare"
|
||||
self._nsenter = nsenter_path or shutil.which("nsenter") or "nsenter"
|
||||
self._limits = limits or ResourceLimits()
|
||||
# The stdlib-only capture agent script, copied into each tracked
|
||||
# workdir's host/ tree so nsenter can reach it inside the sandbox.
|
||||
# ai_service/sandbox/unshare_backend.py -> parents[2] = apps/ai-service.
|
||||
self._agent_script = agent_script or (
|
||||
Path(__file__).resolve().parents[2] / "scripts" / "sandbox-agent.py"
|
||||
)
|
||||
# Tracked (persistent) sandboxes by id; pure shell sandboxes are absent.
|
||||
self._tracked: dict[str, _Tracked] = {}
|
||||
|
||||
# -- spawn ------------------------------------------------------------------
|
||||
|
||||
async def spawn(self, spec: SandboxSpec) -> SandboxHandle:
|
||||
"""Lay out the workdir and return a handle.
|
||||
"""Lay out the workdir; if `spec.capture_env` is set, start the sandbox.
|
||||
|
||||
Isolation is established per-`exec` (each exec = fresh namespaces), so
|
||||
spawn only prepares on-disk state; there is no long-lived init process.
|
||||
A spec WITHOUT capture_env keeps REQ-3-001 semantics: spawn only
|
||||
prepares disk state and each exec forks a fresh (offline) namespace.
|
||||
A spec WITH capture_env starts the persistent helper/inner tree and the
|
||||
capture agent, and `handle.pid` carries the agent's host pid.
|
||||
"""
|
||||
create_layout(spec)
|
||||
if not spec.capture_env:
|
||||
return SandboxHandle(
|
||||
id=spec.sandbox_id,
|
||||
pid=None, # no persistent process; each exec forks short-lived PIDs
|
||||
workdir=spec.workdir,
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
tracked = await self._spawn_tracked(spec)
|
||||
self._tracked[spec.sandbox_id] = tracked
|
||||
return SandboxHandle(
|
||||
id=spec.sandbox_id,
|
||||
pid=None, # no persistent process; each exec forks short-lived PIDs
|
||||
pid=tracked.agent.pid if tracked.agent is not None else tracked.inner_pid,
|
||||
workdir=spec.workdir,
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
|
||||
async def _spawn_tracked(self, spec: SandboxSpec) -> _Tracked:
|
||||
"""Bring up helper + inner + agent for a telemetry-wired task sandbox."""
|
||||
host_dir = spec.workdir / "host"
|
||||
workspace = host_dir / "workspace"
|
||||
ns_root = host_dir / "ns"
|
||||
ns_workdir = ns_root / "work"
|
||||
for d in (workspace, ns_root):
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
# The agent script must live INSIDE the workspace: the inner ns bind
|
||||
# mounts <host_dir>/workspace -> <ns_root>/work, so only workspace
|
||||
# content is visible in-namespace at /work.
|
||||
agent_host_path = workspace / "sandbox-agent.py"
|
||||
shutil.copyfile(self._agent_script, agent_host_path)
|
||||
|
||||
helper = await self._launch_ns(
|
||||
[
|
||||
self._unshare,
|
||||
"--user",
|
||||
"--map-root-user",
|
||||
"--mount",
|
||||
"sh",
|
||||
"-c",
|
||||
(
|
||||
# Isolate ns/ so the inner tmpfs never propagates back to the
|
||||
# host mount table (make-private is best-effort on this host).
|
||||
f"mount --bind {shlex.quote(str(ns_root))} {shlex.quote(str(ns_root))}; "
|
||||
f"mount --make-private {shlex.quote(str(ns_root))} 2>/dev/null; "
|
||||
f"echo {_HELPER_READY}; exec sleep 3600"
|
||||
),
|
||||
],
|
||||
sentinel=_HELPER_READY,
|
||||
label="helper",
|
||||
)
|
||||
try:
|
||||
inner = await self._launch_ns(
|
||||
[
|
||||
*self._helper_join_argv(helper),
|
||||
self._unshare,
|
||||
"--mount",
|
||||
"--pid",
|
||||
"--fork",
|
||||
"--net",
|
||||
"sh",
|
||||
"-c",
|
||||
(
|
||||
f"mount -t tmpfs tmpfs {shlex.quote(str(ns_root))}; "
|
||||
f"mkdir -p {shlex.quote(str(ns_workdir))}; "
|
||||
f"mount --bind {shlex.quote(str(workspace))} "
|
||||
f"{shlex.quote(str(ns_workdir))}; "
|
||||
f"echo {_INNER_READY}; exec sleep 3600"
|
||||
),
|
||||
],
|
||||
sentinel=_INNER_READY,
|
||||
label="inner",
|
||||
)
|
||||
except Exception:
|
||||
await self._reap(helper)
|
||||
raise
|
||||
|
||||
await asyncio.sleep(0) # let the inner child's sleep fork settle
|
||||
inner_pid = await asyncio.to_thread(self._find_child_pid, inner.pid)
|
||||
if inner_pid is None:
|
||||
await self._reap(inner)
|
||||
await self._reap(helper)
|
||||
raise SandboxUnavailableError(
|
||||
f"could not resolve sandboxed init pid for {spec.sandbox_id}"
|
||||
)
|
||||
|
||||
tracked = _Tracked(
|
||||
helper=helper,
|
||||
inner=inner,
|
||||
agent=None,
|
||||
inner_pid=inner_pid,
|
||||
host_dir=host_dir,
|
||||
workspace=workspace,
|
||||
ns_root=ns_root,
|
||||
ns_workdir=ns_workdir,
|
||||
)
|
||||
if spec.capture_env:
|
||||
tracked.agent = await self._launch_agent(spec, tracked, agent_host_path)
|
||||
return tracked
|
||||
|
||||
# -- process launch helpers --------------------------------------------------
|
||||
|
||||
def _helper_join_argv(self, helper: asyncio.subprocess.Process) -> list[str]:
|
||||
"""nsenter argv that runs a command inside the helper's user+mount ns."""
|
||||
if helper.pid is None:
|
||||
raise SandboxUnavailableError("helper namespace process is not running")
|
||||
return [
|
||||
self._nsenter,
|
||||
"-t",
|
||||
str(helper.pid),
|
||||
"-m",
|
||||
"-U",
|
||||
"--preserve-credentials",
|
||||
"--",
|
||||
]
|
||||
|
||||
def _sandbox_join_argv(self, tracked: _Tracked) -> list[str]:
|
||||
"""nsenter argv that joins the inner sandbox MOUNT namespace (uid 0)."""
|
||||
return [self._nsenter, "-t", str(tracked.inner_pid), "-m", "--"]
|
||||
|
||||
async def _launch_agent(
|
||||
self, spec: SandboxSpec, tracked: _Tracked, agent_host_path: Path
|
||||
) -> asyncio.subprocess.Process:
|
||||
"""Launch the capture agent: joins the sandbox mount ns, NOT pid/net.
|
||||
|
||||
The nsenter chain swaps the mount table under the process, so a HOST
|
||||
cwd/relative path is invalid after the join (observed: python3
|
||||
resolved ``sandbox-agent.py`` against a stale root → ``//…`` and
|
||||
exited rc=2). The launch therefore happens through ``sh -c`` INSIDE
|
||||
the joined namespace, using only in-namespace absolute paths: the
|
||||
workspace is bind-mounted at ``<ns_root>/work``, the agent script was
|
||||
copied into the host workspace, so ``/work/sandbox-agent.py`` exists
|
||||
after the join. The agent runs ONLINE (joins mount ns only, not the
|
||||
offline net ns) so it can dial the ai-service WS ingest loopback.
|
||||
"""
|
||||
env = dict(spec.capture_env or {})
|
||||
in_ns_script = f"{tracked.ns_root / 'work' / 'sandbox-agent.py'}"
|
||||
in_ns_cwd = f"{tracked.ns_root / 'work'}"
|
||||
launch = f"cd {shlex.quote(in_ns_cwd)} && exec python3 {shlex.quote(in_ns_script)}"
|
||||
try:
|
||||
return await asyncio.create_subprocess_exec(
|
||||
*self._helper_join_argv(tracked.helper),
|
||||
*self._sandbox_join_argv(tracked),
|
||||
"sh",
|
||||
"-c",
|
||||
launch,
|
||||
stdin=asyncio.subprocess.DEVNULL,
|
||||
stdout=asyncio.subprocess.DEVNULL,
|
||||
stderr=asyncio.subprocess.DEVNULL,
|
||||
env=env,
|
||||
)
|
||||
except FileNotFoundError as exc: # pragma: no cover - env-dependent
|
||||
raise SandboxUnavailableError("python3 unavailable for capture agent") from exc
|
||||
|
||||
async def _launch_ns(
|
||||
self, argv: list[str], *, sentinel: str, label: str
|
||||
) -> asyncio.subprocess.Process:
|
||||
"""Spawn a namespace supervisor and wait for its `sentinel` line."""
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*argv,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.STDOUT,
|
||||
)
|
||||
|
||||
async def _wait_ready() -> None:
|
||||
if proc.stdout is None: # pragma: no cover (stdout is a PIPE)
|
||||
raise SandboxUnavailableError(f"{label} namespace missing stdout pipe")
|
||||
async for raw in proc.stdout:
|
||||
if raw.decode(errors="replace").strip() == sentinel:
|
||||
return
|
||||
raise SandboxUnavailableError(
|
||||
f"{label} namespace exited before signalling readiness: {argv[:3]}"
|
||||
)
|
||||
|
||||
try:
|
||||
await asyncio.wait_for(_wait_ready(), timeout=10.0)
|
||||
except TimeoutError as exc:
|
||||
await self._reap(proc)
|
||||
raise SandboxUnavailableError(
|
||||
f"{label} namespace never became ready (timeout): {argv[:3]}"
|
||||
) from exc
|
||||
except SandboxUnavailableError:
|
||||
await self._reap(proc)
|
||||
raise
|
||||
return proc
|
||||
|
||||
@staticmethod
|
||||
def _find_child_pid(parent_pid: int | None) -> int | None:
|
||||
"""First direct child of `parent_pid` (the pid-namespaced `sleep`).
|
||||
|
||||
The helper→unshare shim is inner.pid's parent chain head, but the
|
||||
SANDBOXED mount/pid namespaces belong to its forked child (the
|
||||
`sleep`). nsenter must target THAT pid to land inside the sandbox.
|
||||
Reads /proc directly — best-effort, host-local, no subprocess.
|
||||
"""
|
||||
if parent_pid is None:
|
||||
return None
|
||||
for entry in os.listdir("/proc"):
|
||||
if not entry.isdigit():
|
||||
continue
|
||||
try:
|
||||
with open(f"/proc/{entry}/stat") as fh:
|
||||
# ppid is field 4; comm (field 2) may contain spaces, so
|
||||
# parse relative to the LAST ')'.
|
||||
rest = fh.read().rsplit(") ", 1)[1].split()
|
||||
if int(rest[1]) == parent_pid: # state=rest[0], ppid=rest[1]
|
||||
return int(entry)
|
||||
except (OSError, IndexError, ValueError):
|
||||
continue
|
||||
return None
|
||||
|
||||
# -- exec --------------------------------------------------------------------
|
||||
|
||||
async def exec(self, handle: SandboxHandle, cmd: list[str]) -> ExecResult:
|
||||
"""Run `cmd` in a fresh namespace rooted at the sandbox workspace."""
|
||||
"""Run `cmd` in the sandbox workspace (cwd = the bound workspace)."""
|
||||
if not cmd:
|
||||
raise ValueError("exec requires a non-empty cmd")
|
||||
tracked = self._tracked.get(handle.id)
|
||||
if tracked is not None:
|
||||
return await self._exec_tracked(handle, tracked, cmd)
|
||||
return await self._exec_fresh(handle, cmd)
|
||||
|
||||
async def _exec_fresh(self, handle: SandboxHandle, cmd: list[str]) -> ExecResult:
|
||||
"""Pure shell sandbox: spawn one fresh offline namespace per exec."""
|
||||
workspace = handle.workdir / "workspace"
|
||||
if not workspace.is_dir():
|
||||
raise SandboxUnavailableError(f"spawn() first: no workspace at {workspace}")
|
||||
@@ -129,7 +405,6 @@ class UnshareBackend: # satisfies SandboxBackend structurally (Protocol)
|
||||
"-c",
|
||||
_build_shim(workspace, self._limits, cmd),
|
||||
]
|
||||
|
||||
started = time.monotonic()
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*argv,
|
||||
@@ -146,13 +421,128 @@ class UnshareBackend: # satisfies SandboxBackend structurally (Protocol)
|
||||
duration_s=time.monotonic() - started,
|
||||
)
|
||||
|
||||
async def _exec_tracked(
|
||||
self, handle: SandboxHandle, tracked: _Tracked, cmd: list[str]
|
||||
) -> ExecResult:
|
||||
"""Task sandbox: join the persistent inner namespace (offline, uid 0).
|
||||
|
||||
rlimits apply in the joining subshell so only the payload is limited;
|
||||
cwd is the bound workspace (`<ns>/work`).
|
||||
"""
|
||||
if tracked.inner.returncode is not None:
|
||||
raise SandboxUnavailableError(
|
||||
f"sandbox {handle.id} is not running (inner namespace exited)"
|
||||
)
|
||||
quoted_cmd = " ".join(shlex.quote(part) for part in cmd)
|
||||
rlimit_prefix = (
|
||||
f"ulimit -v {self._limits.memory_bytes // 1024}; "
|
||||
f"ulimit -t {self._limits.cpu_seconds}; "
|
||||
f"ulimit -f {self._limits.file_size_bytes // 512}; "
|
||||
)
|
||||
shell = (
|
||||
f"cd {shlex.quote(str(tracked.ns_workdir))}; "
|
||||
f"{rlimit_prefix}"
|
||||
f"exec {quoted_cmd}"
|
||||
)
|
||||
argv = [
|
||||
*self._helper_join_argv(tracked.helper),
|
||||
*self._sandbox_join_argv(tracked),
|
||||
"sh",
|
||||
"-c",
|
||||
shell,
|
||||
]
|
||||
started = time.monotonic()
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*argv,
|
||||
stdin=asyncio.subprocess.DEVNULL,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
out, err = await proc.communicate()
|
||||
return ExecResult(
|
||||
cmd=cmd,
|
||||
returncode=proc.returncode if proc.returncode is not None else -1,
|
||||
stdout=out.decode(errors="replace"),
|
||||
stderr=err.decode(errors="replace"),
|
||||
duration_s=time.monotonic() - started,
|
||||
)
|
||||
|
||||
# -- snapshot / destroy --------------------------------------------------------
|
||||
|
||||
async def snapshot(self, handle: SandboxHandle) -> Path:
|
||||
return workdir_snapshot(handle.workdir)
|
||||
workspace_root = handle.workdir
|
||||
tracked = self._tracked.get(handle.id)
|
||||
if tracked is not None:
|
||||
# Copy the tracked workspace, not the legacy <workdir>/workspace.
|
||||
dest_parent = handle.workdir / "snapshots"
|
||||
dest_parent.mkdir(parents=True, exist_ok=True)
|
||||
return workdir_snapshot_from_workspace(tracked.workspace, dest_parent)
|
||||
return workdir_snapshot(workspace_root)
|
||||
|
||||
async def destroy(self, handle: SandboxHandle) -> None:
|
||||
"""Best-effort teardown. Namespaces die with their process; nothing to kill.
|
||||
"""Best-effort teardown. Pure shell sandboxes die with their exec; for a
|
||||
tracked task sandbox reap AGENT → INNER → HELPER so no capture process
|
||||
or namespace supervisor outlives the handle (REQ-3-003 lifecycle).
|
||||
|
||||
Keeping the workdir is deliberate: snapshots must survive destroy so a
|
||||
learner's last state can be restored by the manager layer.
|
||||
"""
|
||||
tracked = self._tracked.pop(handle.id, None)
|
||||
if tracked is not None:
|
||||
# Agent first (it must not flush a "stopped" event into a dead
|
||||
# sandbox), then the namespace tree. The inner `unshare --fork`
|
||||
# shim is NOT the namespace init: killing it orphans its child
|
||||
# (the `sleep` that is PID 1 of the sandbox pid+mnt+net ns),
|
||||
# which reparents to host init and holds the tmpfs + bind for
|
||||
# a full hour (observed: ~30 leaked `sleep 3600` after a test
|
||||
# run). `--kill-child` does not reach it either (util-linux
|
||||
# 2.38 leaks the same child under this flag combo — the child
|
||||
# is reparented before unshare's signal handler runs). The
|
||||
# deterministic kill is SIGKILL on the ns-init's HOST pid,
|
||||
# which we already track as `tracked.inner_pid` (nsenter uses
|
||||
# it for exec); the kernel then tears down the namespace with
|
||||
# its init (no processes remain).
|
||||
for proc in (tracked.agent, tracked.inner, tracked.helper):
|
||||
if proc is not None:
|
||||
await self._reap(proc)
|
||||
self._kill_pid(tracked.inner_pid)
|
||||
handle.pid = None
|
||||
|
||||
@staticmethod
|
||||
def _kill_pid(pid: int | None, sig: int = signal.SIGKILL) -> None:
|
||||
"""Best-effort host-side signal; pid recycled or gone is not an error."""
|
||||
if pid is None:
|
||||
return
|
||||
try:
|
||||
os.kill(pid, sig)
|
||||
except (ProcessLookupError, PermissionError):
|
||||
pass # already dead, or not ours — nothing to do
|
||||
|
||||
@staticmethod
|
||||
async def _reap(proc: asyncio.subprocess.Process) -> None:
|
||||
"""SIGTERM then SIGKILL, tolerant of an already-dead process."""
|
||||
if proc.returncode is not None:
|
||||
return
|
||||
try:
|
||||
proc.terminate()
|
||||
except ProcessLookupError:
|
||||
return
|
||||
try:
|
||||
await asyncio.wait_for(proc.wait(), timeout=5.0)
|
||||
except TimeoutError:
|
||||
try:
|
||||
proc.kill()
|
||||
except ProcessLookupError:
|
||||
return
|
||||
try:
|
||||
await asyncio.wait_for(proc.wait(), timeout=5.0)
|
||||
except TimeoutError: # pragma: no cover - SIGKILL always wins
|
||||
pass
|
||||
|
||||
|
||||
def workdir_snapshot_from_workspace(workspace: Path, snapshots_dir: Path) -> Path:
|
||||
"""Snapshot helper for tracked sandboxes whose workspace is `<workdir>/host/workspace`
|
||||
instead of the legacy `<workdir>/workspace` layout."""
|
||||
dest = snapshots_dir / datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
|
||||
shutil.copytree(workspace, dest, symlinks=False)
|
||||
return dest
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
"""Live build telemetry — event models and the TraceStore protocol (REQ-3-003).
|
||||
|
||||
Boundary rule (D-027): telemetry/ imports from config only — never from
|
||||
agents/ or api/ (agents call engines through narrow interfaces, never
|
||||
the reverse; api/ composes stores via DI).
|
||||
"""
|
||||
|
||||
from .ingest import IngestSession, TraceIntegrityMap, telemetry_ingest_endpoint
|
||||
from .models import EventKind, TelemetryEvent, TraceSpan
|
||||
from .store import SQLiteTraceStore, TraceStore
|
||||
|
||||
__all__ = [
|
||||
"EventKind",
|
||||
"IngestSession",
|
||||
"SQLiteTraceStore",
|
||||
"TelemetryEvent",
|
||||
"TraceIntegrityMap",
|
||||
"TraceSpan",
|
||||
"TraceStore",
|
||||
"telemetry_ingest_endpoint",
|
||||
]
|
||||
@@ -0,0 +1,458 @@
|
||||
"""WS ingest protocol for learner telemetry (REQ-3-003, D-026, G-3).
|
||||
|
||||
Frame contract — trace identity travels as QUERY PARAMS on the WS upgrade
|
||||
(`WS /v1/telemetry/ingest?learner_id=...&task_id=...&sandbox_id=...`), NOT as
|
||||
a first init frame. Rationale: the in-sandbox capture agent (Task 2-2-01) is a
|
||||
stdlib-only RFC6455 client where the URL is the cheapest thing to parametrize
|
||||
(`NC_INGEST_URL` carries the query string); identity is also visible to the
|
||||
server BEFORE accept(), so a malformed handshake can be rejected without an
|
||||
accept/close round-trip. Client messages are then ONE event per JSON text
|
||||
frame — no envelope:
|
||||
|
||||
{"seq": 0, "kind": "command", "payload": {...}, "ts": "...",
|
||||
"sandbox_id": "..."} # learner_id / task_id forbidden (URL owns them)
|
||||
|
||||
Server → client frames are typed status envelopes:
|
||||
|
||||
{"type": "ack_total", "count": N} — final flush summary, then close 1000
|
||||
{"type": "seq_ack", "seq": N} — advisory: durable latest_seq after
|
||||
each successful append (D-045; the
|
||||
agent trims its spool to seq > ack)
|
||||
{"type": "gap_warning", "missing_seqs": [...]} — seq skipped ahead
|
||||
{"type": "event_rejected", "detail": "..."} — one frame failed validation
|
||||
(seq echoed when parseable)
|
||||
{"type": "event_rejected", "seq": N, "detail": "..."} — stored-field rejected (bad kind)
|
||||
{"type": "flooded", "reason": "cap_exceeded"|"queue_overflow",
|
||||
"count": N} — sent before close(1008)
|
||||
|
||||
Keepalive: the server sends an opaque ping frame every `PING_INTERVAL_S` (the
|
||||
capture agent auto-pongs at the frame layer); a peer that is silent past
|
||||
`PONG_TIMEOUT_S` is assumed wedged, but the keepalive half only LOGS — the
|
||||
receiver half owns disconnect detection (single-box pilot: TCP EOF is
|
||||
reliable; an aggressive pong-watchdog would false-positive on loaded boxes).
|
||||
|
||||
Flood control (GRILL G-3, BINDING — silent drop-oldest is FORBIDDEN):
|
||||
* per-connection inbound queue bounded at `INBOUND_QUEUE_MAX` frames; on
|
||||
overflow → close code 1008 (policy violation) + trace marked
|
||||
INCOMPLETE_FLOODED via `TraceIntegrityMap`.
|
||||
* total events for the (learner, task) exceeding
|
||||
`Settings.telemetry_max_events_per_task` → same 1008 + INCOMPLETE_FLOODED.
|
||||
`INCOMPLETE_FLOODED` is an integrity signal Proctor/Phase-3 grader read via
|
||||
`TraceIntegrityMap.is_incomplete()` (the G-4 gate): a flooded trace can
|
||||
never yield a credential.
|
||||
|
||||
Boundary (D-027): telemetry/ never imports agents/ or api/. This module
|
||||
imports only `fastapi.WebSocket` for the socket type (a protocol surface, not
|
||||
a DI framework); the session engine below depends only on the TraceStore
|
||||
protocol + Settings, and api/telemetry.py injects both through plain
|
||||
parameters.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Any, Final
|
||||
|
||||
from fastapi import WebSocket, WebSocketDisconnect
|
||||
from pydantic import BaseModel, ConfigDict, Field, ValidationError
|
||||
|
||||
from ..config import Settings
|
||||
from .models import TelemetryEvent
|
||||
from .store import TraceStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: WebSocket close code 1008 — policy violation (RFC 6455 §7.4.1).
|
||||
WS_CLOSE_POLICY_VIOLATION: Final = 1008
|
||||
|
||||
#: Bounded inbound queue depth per connection (G-3). Sized for burst-tolerance
|
||||
#: well above the capture agent's emission rate; overflow is a flood signal,
|
||||
#: not a backpressure knob.
|
||||
INBOUND_QUEUE_MAX: Final = 256
|
||||
|
||||
PING_INTERVAL_S: Final = 20.0
|
||||
|
||||
|
||||
class InboundEventFrame(BaseModel):
|
||||
"""Client → server event frame (one TelemetryEvent minus URL-owned ids).
|
||||
|
||||
`extra="forbid"`: learner_id/task_id arriving in the frame body is a
|
||||
contract violation — identity comes from the query params only, so a
|
||||
replayed frame can never lie about which trace it belongs to.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
seq: int = Field(ge=0)
|
||||
kind: str = Field(min_length=1)
|
||||
payload: dict[str, Any] = Field(default_factory=dict)
|
||||
ts: datetime
|
||||
sandbox_id: str = ""
|
||||
|
||||
|
||||
class TraceIntegrityMap:
|
||||
"""Integrity flags for traces that can never be graded (G-3/G-4).
|
||||
|
||||
Process-local and deliberately small: v0.3 runs ONE ai-service process per
|
||||
box, and the Phase-3 grader reads this flag through the same DI container
|
||||
— D-019-style in-memory registry precedent (the sandbox handle registry is
|
||||
the same shape). The flag is terminal within the process: a reconnect
|
||||
sending legal events does NOT clear it — the trace is already untrusted as
|
||||
grading input. Restarting ai-service resets flags; grading runs against a
|
||||
live service, and the SQLite trace rows themselves are durable.
|
||||
|
||||
All methods are sync: mutation is a dict write, reads are dict lookups —
|
||||
no await needed, so callers from any layer (API handlers, the grader)
|
||||
don't inherit an async surface for a nanosecond operation.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
# (learner_id, task_id) -> machine-readable reason (INCOMPLETE_FLOODED)
|
||||
self._flags: dict[tuple[str, str], str] = {}
|
||||
|
||||
def mark(self, learner_id: str, task_id: str, reason: str) -> None:
|
||||
"""Set an integrity flag. Presence of the flag is the signal; the
|
||||
reason is informational (last write wins)."""
|
||||
self._flags[(learner_id, task_id)] = reason
|
||||
|
||||
def clear(self, learner_id: str, task_id: str) -> None:
|
||||
"""Test seam: reset a flag (production ingest never clears)."""
|
||||
self._flags.pop((learner_id, task_id), None)
|
||||
|
||||
def is_incomplete(self, learner_id: str, task_id: str) -> bool:
|
||||
"""True when the trace carries ANY terminal integrity flag."""
|
||||
return (learner_id, task_id) in self._flags
|
||||
|
||||
def reason(self, learner_id: str, task_id: str) -> str | None:
|
||||
"""The flag's reason (INCOMPLETE_FLOODED), or None when unflagged."""
|
||||
return self._flags.get((learner_id, task_id))
|
||||
|
||||
|
||||
class IngestSession:
|
||||
"""One WebSocket ingest connection: receive → queue → drain → store.
|
||||
|
||||
Two tasks per connection:
|
||||
* `_receiver` — reads frames, validates shape, enqueues (bounded queue,
|
||||
G-3). Receives never block on SQLite.
|
||||
* `_drainer` — pops frames in arrival order, appends via TraceStore
|
||||
(idempotent on (learner,task,seq)), emits gap warnings, enforces the
|
||||
per-trace event cap.
|
||||
Either task detecting a flood closes the WS with 1008 and marks the trace
|
||||
INCOMPLETE_FLOODED. The events queue carries `None` as the client-
|
||||
disconnect sentinel.
|
||||
|
||||
- `telemetry_max_events_per_task` is consulted at connect and re-checked
|
||||
per append against the DURABLE row count (cap compares against stored
|
||||
events, so a skipped-ahead seq cannot burn budget that was never sent).
|
||||
Durable count via `TraceStore.count()` (COUNT(*)) — a single aggregate
|
||||
per append, never materializing trace rows (the pre-P7 code read
|
||||
`len(get_trace(...))` which was O(trace) per event / O(n²) per session).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
websocket: WebSocket,
|
||||
store: TraceStore,
|
||||
integrity: TraceIntegrityMap,
|
||||
settings: Settings,
|
||||
learner_id: str,
|
||||
task_id: str,
|
||||
sandbox_id: str,
|
||||
) -> None:
|
||||
self._ws = websocket
|
||||
self._store = store
|
||||
self._integrity = integrity
|
||||
# Snapshot of the one setting ingest consults: read once at connect so
|
||||
# a hot-reloaded Settings object mid-session can't move the cap.
|
||||
self._max_events = settings.telemetry_max_events_per_task
|
||||
self.learner_id = learner_id
|
||||
self.task_id = task_id
|
||||
self.sandbox_id = sandbox_id
|
||||
|
||||
self._queue: asyncio.Queue[InboundEventFrame | None] = asyncio.Queue(
|
||||
maxsize=INBOUND_QUEUE_MAX
|
||||
)
|
||||
self._seen: set[int] = set()
|
||||
self._next_expected: int | None = None # in-connection monotonic hint
|
||||
self._received = 0
|
||||
self._stored = 0
|
||||
self._deduped = 0
|
||||
self._rejected = 0
|
||||
self._flooded = False
|
||||
self._flood_reason = ""
|
||||
|
||||
# -- receive half ----------------------------------------------------------
|
||||
|
||||
async def run(self) -> None:
|
||||
"""Accept, run receiver+drainer, close cleanly. Owns the WS lifecycle."""
|
||||
await self._ws.accept()
|
||||
logger.info(
|
||||
"telemetry ingest connected: %s/%s sandbox=%s",
|
||||
self.learner_id,
|
||||
self.task_id,
|
||||
self.sandbox_id or "(none)",
|
||||
)
|
||||
pinger = asyncio.create_task(self._keepalive())
|
||||
receiver = asyncio.create_task(self._receiver())
|
||||
drainer = asyncio.create_task(self._drainer())
|
||||
# First terminal outcome shuts the session down: client disconnect
|
||||
# (receiver ends) → drainer flushes; drainer ended (clean close after
|
||||
# flush or a 1008 flood close) → receiver must not linger.
|
||||
pending: set[asyncio.Task[None]] = {receiver, drainer}
|
||||
try:
|
||||
done, pending = await asyncio.wait(
|
||||
pending, return_when=asyncio.FIRST_COMPLETED
|
||||
)
|
||||
if receiver in done and drainer in pending:
|
||||
try:
|
||||
await drainer # final flush → sends ack_total, close 1000
|
||||
finally:
|
||||
pending.discard(drainer)
|
||||
finally:
|
||||
for task in (pinger, *pending):
|
||||
task.cancel()
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
await task
|
||||
|
||||
async def _receiver(self) -> None:
|
||||
"""Read frames; parse+enqueue. Overflow → flood shutdown (G-3).
|
||||
|
||||
RuntimeError from receive_text is benign here: it fires when the
|
||||
socket was closed by the drainer (1008 flood close) while this task
|
||||
was parked in receive — a terminal condition, not a bug.
|
||||
"""
|
||||
try:
|
||||
while True:
|
||||
raw = await self._ws.receive_text()
|
||||
frame = self._parse(raw)
|
||||
if frame is None:
|
||||
# Rejected frame — keep the connection open; the producer
|
||||
# gets an event_rejected status frame so a malformed batch
|
||||
# is visible (and its seq is never stored). Yield so the
|
||||
# status frame flushes before we block on the next receive.
|
||||
await self._reject_frame(raw)
|
||||
await asyncio.sleep(0)
|
||||
continue
|
||||
self._received += 1
|
||||
try:
|
||||
self._queue.put_nowait(frame)
|
||||
except asyncio.QueueFull:
|
||||
# Bounded queue — overflow is a flood, never drop-oldest.
|
||||
# _trigger_flood closes the socket; fall through to the
|
||||
# tail so the disconnect sentinel is still enqueued — the
|
||||
# drainer is never left parked on an empty queue after a
|
||||
# flood (P7 review: the pre-fix code `return`ed from the
|
||||
# QueueFull branch WITHOUT the sentinel, leaking the
|
||||
# session task set — one per flooded trace).
|
||||
await self._trigger_flood("queue_overflow")
|
||||
return
|
||||
except WebSocketDisconnect:
|
||||
pass
|
||||
except RuntimeError:
|
||||
logger.debug(
|
||||
"ingest receiver: socket already closed (flood path) %s/%s",
|
||||
self.learner_id,
|
||||
self.task_id,
|
||||
)
|
||||
# Client gone (clean close, drop, or flood close): sentinel unblocks
|
||||
# the drainer for a final flush. put_nowait can only fail under flood,
|
||||
# which already terminated the session.
|
||||
with contextlib.suppress(asyncio.QueueFull):
|
||||
self._queue.put_nowait(None)
|
||||
|
||||
def _parse(self, raw: str) -> InboundEventFrame | None:
|
||||
"""Validate one frame; None means malformed (caller rejects it)."""
|
||||
try:
|
||||
return InboundEventFrame.model_validate_json(raw)
|
||||
except ValidationError:
|
||||
return None
|
||||
|
||||
async def _reject_frame(self, raw: str) -> None:
|
||||
"""Malformed envelope: log + event_rejected status frame (never stored)."""
|
||||
self._rejected += 1
|
||||
detail = "invalid event frame"
|
||||
try:
|
||||
InboundEventFrame.model_validate_json(raw)
|
||||
except ValidationError as exc:
|
||||
detail = exc.errors()[0].get("msg", "validation error")
|
||||
logger.warning(
|
||||
"telemetry frame rejected: %s/%s: %s", self.learner_id, self.task_id, detail
|
||||
)
|
||||
seq: int | None = None
|
||||
with contextlib.suppress(Exception):
|
||||
seq = int(json.loads(raw).get("seq")) # best-effort echo for the producer
|
||||
payload: dict[str, Any] = {"type": "event_rejected", "detail": detail}
|
||||
if seq is not None:
|
||||
payload["seq"] = seq
|
||||
await self._send_json(payload)
|
||||
|
||||
# -- drain half --------------------------------------------------------------
|
||||
|
||||
async def _drainer(self) -> None:
|
||||
"""Pop queued frames, append to the store, then close 1000 + summary."""
|
||||
while True:
|
||||
frame = await self._queue.get()
|
||||
if frame is None: # disconnect sentinel → flush complete
|
||||
await self._send_json(
|
||||
{
|
||||
"type": "ack_total",
|
||||
"count": self._stored,
|
||||
"deduped": self._deduped,
|
||||
"rejected": self._rejected,
|
||||
}
|
||||
)
|
||||
with contextlib.suppress(RuntimeError, WebSocketDisconnect):
|
||||
await self._ws.close(code=1000)
|
||||
return
|
||||
await self._append(frame)
|
||||
|
||||
async def _append(self, frame: InboundEventFrame) -> None:
|
||||
# Precedence: a trace already flagged INCOMPLETE_FLOODED is terminal —
|
||||
# the connection that triggered it is being torn down, and any stray
|
||||
# queued frames must not resurrect the trace's intake.
|
||||
if self._integrity.is_incomplete(self.learner_id, self.task_id):
|
||||
await self._trigger_flood("already_flagged")
|
||||
return
|
||||
|
||||
# Per-trace cap (G-3): checked against the DURABLE row count so a
|
||||
# reconnect resumes the budget instead of resetting it, and a
|
||||
# skipped-ahead seq cannot burn budget that was never sent.
|
||||
if self._flood_breached():
|
||||
await self._trigger_flood("cap_exceeded")
|
||||
return
|
||||
|
||||
# TelemetryEvent's @validates hooks fire on CONSTRUCTION (setattr), so
|
||||
# the try must wrap building the model too — an unknown kind raises
|
||||
# before `store.append` is ever reached.
|
||||
event: TelemetryEvent
|
||||
before = self._store.latest_seq(self.learner_id, self.task_id)
|
||||
try:
|
||||
event = TelemetryEvent(
|
||||
learner_id=self.learner_id,
|
||||
task_id=self.task_id,
|
||||
seq=frame.seq,
|
||||
kind=frame.kind,
|
||||
payload=frame.payload,
|
||||
ts=frame.ts,
|
||||
sandbox_id=frame.sandbox_id or self.sandbox_id,
|
||||
)
|
||||
self._store.append(event)
|
||||
except ValueError as exc: # unknown kind / invalid field
|
||||
self._rejected += 1
|
||||
await self._send_json(
|
||||
{"type": "event_rejected", "seq": frame.seq, "detail": str(exc)}
|
||||
)
|
||||
return
|
||||
after = self._store.latest_seq(self.learner_id, self.task_id)
|
||||
|
||||
if after == before and frame.seq in self._seen:
|
||||
self._deduped += 1 # at-least-once retry; stored once (idempotent)
|
||||
else:
|
||||
self._stored += 1
|
||||
self._seen.add(frame.seq)
|
||||
# Seq-ack (D-045, REQ-5-007): advisory hint carrying the durable
|
||||
# latest_seq AFTER this append — the capture agent trims its spool to
|
||||
# seq > ack on receipt, bounding the replay margin to the in-flight
|
||||
# window. Emitted on dedup'd appends too (a-6) so a replay flush
|
||||
# tightens the margin immediately. Gap detection stays authoritative
|
||||
# (_check_gap below); G-3 flood semantics untouched.
|
||||
await self._send_json({"type": "seq_ack", "seq": after})
|
||||
await self._check_gap(frame.seq)
|
||||
# SQLite appends are sync and fast; on a burst the drainer can hold
|
||||
# the loop between receives. Yield so the WS writer flushes the close
|
||||
# and the pinger/interleave stay live under the eventlet-free portal.
|
||||
await asyncio.sleep(0)
|
||||
|
||||
def _flood_breached(self) -> bool:
|
||||
"""True when this append would exceed the per-trace event budget."""
|
||||
# Durable count (NOT latest_seq+1 — a skipped-ahead seq must not burn
|
||||
# un-sent events' budget) via COUNT(*): never materialize the trace
|
||||
# per append (P7 review — the old len(get_trace(...)) built every row
|
||||
# object per event, O(trace) per append / O(n²) per session).
|
||||
durable = self._store.count(learner_id=self.learner_id, task_id=self.task_id)
|
||||
return durable >= self._max_events
|
||||
|
||||
async def _check_gap(self, incoming_seq: int) -> None:
|
||||
"""Seq skipped ahead → log + per-connection gap_warning status frame."""
|
||||
if self._next_expected is not None and incoming_seq > self._next_expected:
|
||||
missing = list(range(self._next_expected, incoming_seq))
|
||||
logger.warning(
|
||||
"telemetry gap: %s/%s missing seqs %s (arrived seq=%d)",
|
||||
self.learner_id,
|
||||
self.task_id,
|
||||
missing,
|
||||
incoming_seq,
|
||||
)
|
||||
await self._send_json({"type": "gap_warning", "missing_seqs": missing})
|
||||
if self._next_expected is None or incoming_seq >= self._next_expected:
|
||||
self._next_expected = incoming_seq + 1
|
||||
|
||||
# -- flood + keepalive ------------------------------------------------------
|
||||
|
||||
async def _trigger_flood(self, reason: str) -> None:
|
||||
"""G-3: 1008 close + INCOMPLETE_FLOODED mark. Exactly once."""
|
||||
if self._flooded:
|
||||
return
|
||||
self._flooded = True
|
||||
self._flood_reason = reason
|
||||
self._integrity.mark(self.learner_id, self.task_id, "INCOMPLETE_FLOODED")
|
||||
logger.warning(
|
||||
"telemetry flood: %s/%s reason=%s — closing 1008, trace marked "
|
||||
"INCOMPLETE_FLOODED (G-3; Proctor/grade gate will refuse it)",
|
||||
self.learner_id,
|
||||
self.task_id,
|
||||
reason,
|
||||
)
|
||||
await self._send_json(
|
||||
{"type": "flooded", "reason": reason, "count": self._received}
|
||||
)
|
||||
with contextlib.suppress(RuntimeError, WebSocketDisconnect):
|
||||
await self._ws.close(
|
||||
code=WS_CLOSE_POLICY_VIOLATION,
|
||||
reason=f"telemetry flood control (G-3): {reason}",
|
||||
)
|
||||
|
||||
async def _keepalive(self) -> None:
|
||||
"""Protocol-level ping on an interval (agent auto-pongs at frame level).
|
||||
|
||||
A send failure means the socket is already gone — the receiver half
|
||||
independently surfaces the disconnect; we just stop pinging.
|
||||
"""
|
||||
while True:
|
||||
await asyncio.sleep(PING_INTERVAL_S)
|
||||
try:
|
||||
await self._ws.send_bytes(b"\x89ping-nextcraft")
|
||||
except (RuntimeError, WebSocketDisconnect):
|
||||
return
|
||||
|
||||
async def _send_json(self, payload: dict[str, Any]) -> None:
|
||||
"""Best-effort status frame; the socket may already be gone."""
|
||||
with contextlib.suppress(RuntimeError, WebSocketDisconnect):
|
||||
await self._ws.send_json(payload)
|
||||
|
||||
|
||||
async def telemetry_ingest_endpoint(
|
||||
websocket: WebSocket,
|
||||
learner_id: str,
|
||||
task_id: str,
|
||||
store: TraceStore,
|
||||
integrity: TraceIntegrityMap,
|
||||
settings: Settings,
|
||||
sandbox_id: str = "",
|
||||
) -> None:
|
||||
"""Engine entry: build the session and run it. api/telemetry.py calls this
|
||||
with query params + app.state services already resolved — this signature
|
||||
is deliberately Depends-free (telemetry/ never knows FastAPI DI exists).
|
||||
"""
|
||||
session = IngestSession(
|
||||
websocket=websocket,
|
||||
store=store,
|
||||
integrity=integrity,
|
||||
settings=settings,
|
||||
learner_id=learner_id,
|
||||
task_id=task_id,
|
||||
sandbox_id=sandbox_id,
|
||||
)
|
||||
await session.run()
|
||||
@@ -0,0 +1,108 @@
|
||||
"""Telemetry event record — the row the trace store persists (REQ-3-003, D-027).
|
||||
|
||||
One model serves both as the JSON payload sent by producers and as the SQLite
|
||||
row schema. `payload` is stored as a JSON column (native JSONB on Postgres —
|
||||
no migration-time shape change, D-027).
|
||||
|
||||
Field contract (consumed by the trace store and the grader):
|
||||
learner_id — non-empty learner identifier.
|
||||
task_id — non-empty task/session identifier; trace identity is the
|
||||
(learner_id, task_id) pair.
|
||||
seq — sequence number per trace, >= 0. Monotonicity per
|
||||
(learner, task) is enforced by the store (Task 2-1-02);
|
||||
this model only rejects negative seqs.
|
||||
kind — event discriminator: command | file_diff | run_result |
|
||||
test_result | activity | stdin | stdout.
|
||||
payload — free-form JSON detail blob.
|
||||
ts — envelope timestamp (UTC); monotonicity enforced at ingest.
|
||||
sandbox_id — originating sandbox ("" for non-sandbox sources).
|
||||
|
||||
Boundary (D-027): telemetry/ never imports agents/ or api/.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from sqlalchemy import JSON, Index, String
|
||||
from sqlalchemy.orm import validates
|
||||
from sqlmodel import Field as SQLField
|
||||
from sqlmodel import SQLModel
|
||||
|
||||
EventKind = Literal[
|
||||
"command",
|
||||
"file_diff",
|
||||
"run_result",
|
||||
"test_result",
|
||||
"activity",
|
||||
"stdin",
|
||||
"stdout",
|
||||
]
|
||||
_EVENT_KINDS: frozenset[str] = frozenset(EventKind.__args__)
|
||||
|
||||
|
||||
class TelemetryEvent(SQLModel, table=True):
|
||||
"""A single durable telemetry event; (learner_id, task_id, seq) is PK.
|
||||
|
||||
Constraint enforcement uses SQLAlchemy `@validates` hooks: sqlmodel
|
||||
0.0.42's metaclass drops pydantic `Field(ge=...)`/`field_validator`
|
||||
constraints for table models (the decorators register but never make it
|
||||
into the core schema), while `@validates` fires on every attribute set —
|
||||
construction included — and raises ValueError on violation. seq >= 0 plus
|
||||
a VARCHAR kind column keep the DB shape Postgres-ready (D-027).
|
||||
"""
|
||||
|
||||
__tablename__ = "telemetry_event"
|
||||
# PK columns already produce a unique index; this secondary index covers
|
||||
# trace reads ordered by seq without depending on the PK column order
|
||||
# (Postgres migration target D-027).
|
||||
__table_args__ = (Index("ix_telemetry_event_trace", "learner_id", "task_id"),)
|
||||
|
||||
learner_id: str = SQLField(primary_key=True)
|
||||
task_id: str = SQLField(primary_key=True)
|
||||
seq: int = SQLField(primary_key=True)
|
||||
# Bare Literal annotations crash sqlmodel<=0.0.42's column inference
|
||||
# (issubclass(TypeAlias, Enum)); an explicit sa_type + the validates hook
|
||||
# below gives the same contract: VARCHAR column, Literal-rejected values.
|
||||
kind: EventKind = SQLField(sa_type=String)
|
||||
# JSON column: stored as TEXT on SQLite, native JSONB on Postgres (D-027).
|
||||
payload: dict[str, Any] = SQLField(default_factory=dict, sa_type=JSON)
|
||||
ts: datetime
|
||||
sandbox_id: str = SQLField(default="")
|
||||
|
||||
@validates("learner_id", "task_id")
|
||||
def _ids_non_empty(self, key: str, value: str) -> str:
|
||||
if not value:
|
||||
raise ValueError(f"{key} must be a non-empty identifier")
|
||||
return value
|
||||
|
||||
@validates("seq")
|
||||
def _seq_non_negative(self, key: str, value: int) -> int:
|
||||
if value < 0:
|
||||
raise ValueError("seq must be >= 0 (monotonicity is the store's job)")
|
||||
return value
|
||||
|
||||
@validates("kind")
|
||||
def _kind_is_known(self, key: str, value: str) -> str:
|
||||
if value not in _EVENT_KINDS:
|
||||
raise ValueError(f"unknown event kind: {value!r}")
|
||||
return value
|
||||
|
||||
|
||||
class TraceSpan(BaseModel):
|
||||
"""Derived view: the ordered event trace for one (learner_id, task_id).
|
||||
|
||||
NOT a table — materialized by the store from persisted TelemetryEvents
|
||||
(grader/Lab consume this shape; replay order is the seq column).
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
learner_id: str = Field(min_length=1)
|
||||
task_id: str = Field(min_length=1)
|
||||
events: tuple[TelemetryEvent, ...] = ()
|
||||
|
||||
@property
|
||||
def latest_seq(self) -> int:
|
||||
"""Highest seq in the span; -1 when empty (store convention)."""
|
||||
return self.events[-1].seq if self.events else -1
|
||||
@@ -0,0 +1,218 @@
|
||||
"""TraceStore — telemetry persistence protocol + SQLite implementation (REQ-3-003, D-027).
|
||||
|
||||
Postgres-migration-ready (D-027): the protocol is the only surface the API /
|
||||
grader layers touch; swapping SQLiteTraceStore for a Postgres-backed
|
||||
implementation must not change call sites. The `telemetry_event` table uses
|
||||
only portable column types (str / int / datetime / JSON), so the same SQLModel
|
||||
schema stands up unchanged on Postgres.
|
||||
|
||||
Ingest is at-least-once: duplicates carry the same (learner_id, task_id, seq)
|
||||
idempotency key, so `append` with a triplet that is already stored is a no-op.
|
||||
The pair (learner_id, task_id) identifies a trace; `seq` numbers events in it
|
||||
starting at 0.
|
||||
|
||||
Concurrency (a-3): the engine enables WAL + synchronous=NORMAL and a busy
|
||||
timeout at connection time, so the ingest writer and grader readers do not hit
|
||||
`database is locked` on the single-box pilot.
|
||||
|
||||
Boundary: `telemetry/` never imports `agents/` / `api/` and has no FastAPI
|
||||
dependency.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlmodel import Session, SQLModel, create_engine, select
|
||||
|
||||
from ..config import Settings
|
||||
from .models import TelemetryEvent
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class TraceStore(Protocol):
|
||||
"""Persistence contract for ordered per-learner task trace streams.
|
||||
|
||||
Implemented by SQLiteTraceStore (v0.3, D-027); a Postgres implementation
|
||||
must satisfy the same surface.
|
||||
"""
|
||||
|
||||
def append(self, event: TelemetryEvent) -> None:
|
||||
"""Store one event. IDEMPOTENT on (learner_id, task_id, seq):
|
||||
|
||||
at-least-once ingest retries with the same triplet are deduped
|
||||
(stored once), not rejected. Later events must not overwrite an
|
||||
existing row.
|
||||
"""
|
||||
...
|
||||
|
||||
def get_trace(self, learner_id: str, task_id: str) -> list[TelemetryEvent]:
|
||||
"""All stored events for the trace, ordered by seq ascending.
|
||||
|
||||
Detached from any DB session — safe to pass across layers. Empty list
|
||||
when the trace has no events.
|
||||
"""
|
||||
...
|
||||
|
||||
def gaps(self, learner_id: str, task_id: str) -> list[int]:
|
||||
"""Missing seqs in 0..latest for the trace ([0,2,3] stored -> [1])."""
|
||||
...
|
||||
|
||||
def latest_seq(self, learner_id: str, task_id: str) -> int:
|
||||
"""Highest stored seq for the trace; -1 when no events exist."""
|
||||
...
|
||||
|
||||
def count(self, learner_id: str, task_id: str) -> int:
|
||||
"""Number of stored events for the trace (COUNT(*), never
|
||||
materializes rows — the ingest cap consults this per append, so
|
||||
an O(trace) implementation would make ingest O(n²) per session).
|
||||
"""
|
||||
...
|
||||
|
||||
def list_tasks(self, learner_id: str) -> list[str]:
|
||||
"""Distinct task_ids with at least one event for the learner."""
|
||||
...
|
||||
|
||||
def close(self) -> None:
|
||||
"""Release DB connections. Store must not be used after close."""
|
||||
...
|
||||
|
||||
|
||||
def _sqlite_connect(dbapi_connection: sqlite3.Connection, _: object) -> None:
|
||||
"""Per-connection pragma setup (a-3).
|
||||
|
||||
journal_mode=WAL — readers never block the single writer.
|
||||
synchronous=NORMAL — safe in WAL mode, avoids full fsync-per-commit.
|
||||
busy_timeout=5000 — retry briefly under contention instead of
|
||||
`OperationalError: database is locked`.
|
||||
"""
|
||||
cursor = dbapi_connection.cursor()
|
||||
cursor.execute("PRAGMA journal_mode=WAL")
|
||||
cursor.execute("PRAGMA synchronous=NORMAL")
|
||||
cursor.execute("PRAGMA busy_timeout=5000")
|
||||
cursor.close()
|
||||
|
||||
|
||||
def _as_utc(ts: datetime) -> datetime:
|
||||
"""Timestamps travel as naive datetime on SQLite, tz-aware elsewhere.
|
||||
|
||||
SQLite (via SQLModel) drops tzinfo; Postgres TIMESTAMP WITH TIME ZONE
|
||||
keeps it. Normalizing on the read/write boundary makes the store's
|
||||
contract tz-aware UTC regardless of the backend (D-027).
|
||||
"""
|
||||
if ts.tzinfo is None:
|
||||
return ts.replace(tzinfo=UTC) # naive UTC read-side: label as UTC
|
||||
return ts.astimezone(UTC)
|
||||
|
||||
|
||||
class SQLiteTraceStore:
|
||||
"""SQLite-backed TraceStore (SQLModel). First real persistence (D-027)."""
|
||||
|
||||
def __init__(self, db_path: Path | None = None) -> None:
|
||||
self._db_path: Path = db_path if db_path is not None else Settings().db_path
|
||||
self._engine = create_engine(f"sqlite:///{self._db_path}")
|
||||
sa.event.listen(self._engine, "connect", _sqlite_connect)
|
||||
SQLModel.metadata.create_all(self._engine)
|
||||
|
||||
@contextmanager
|
||||
def _session(self) -> Iterator[Session]:
|
||||
# expire_on_commit=False: ORM objects returned from `append`'s
|
||||
# IntegrityError path stay usable without a refresh round-trip.
|
||||
with Session(self._engine, expire_on_commit=False) as session:
|
||||
yield session
|
||||
|
||||
def append(self, event: TelemetryEvent) -> None:
|
||||
# INSERT-if-absent via PK: sqlite3 raises IntegrityError on a
|
||||
# duplicate (learner_id, task_id, seq); swallow it — the row is
|
||||
# already stored, which is the dedup contract for at-least-once
|
||||
# ingest. `session.merge` would upsert instead; wrong semantics here.
|
||||
with self._session() as session:
|
||||
try:
|
||||
session.add(event)
|
||||
session.commit()
|
||||
except sa.exc.IntegrityError:
|
||||
session.rollback()
|
||||
logger.debug(
|
||||
"trace event dedup: %s/%s seq=%d already stored",
|
||||
event.learner_id,
|
||||
event.task_id,
|
||||
event.seq,
|
||||
)
|
||||
|
||||
def get_trace(self, learner_id: str, task_id: str) -> list[TelemetryEvent]:
|
||||
with self._session() as session:
|
||||
stmt = (
|
||||
select(TelemetryEvent)
|
||||
.where(TelemetryEvent.learner_id == learner_id)
|
||||
.where(TelemetryEvent.task_id == task_id)
|
||||
.order_by(TelemetryEvent.seq)
|
||||
)
|
||||
results = session.exec(stmt).all()
|
||||
# Detach from the session: callers must not depend on open-session
|
||||
# ORM magic (lazy loads fail once the session is closed).
|
||||
for row in results:
|
||||
row.ts = _as_utc(row.ts)
|
||||
session.expunge(row)
|
||||
return list(results)
|
||||
|
||||
def _stored_seqs(self, learner_id: str, task_id: str) -> list[int]:
|
||||
with self._session() as session:
|
||||
stmt = (
|
||||
select(TelemetryEvent.seq)
|
||||
.where(TelemetryEvent.learner_id == learner_id)
|
||||
.where(TelemetryEvent.task_id == task_id)
|
||||
.order_by(TelemetryEvent.seq)
|
||||
)
|
||||
# sqlmodel scalar select: rows are plain ints, not 1-tuples.
|
||||
return [int(seq) for seq in session.exec(stmt).all()]
|
||||
|
||||
def gaps(self, learner_id: str, task_id: str) -> list[int]:
|
||||
seqs = self._stored_seqs(learner_id, task_id)
|
||||
if not seqs:
|
||||
return []
|
||||
present = set(seqs)
|
||||
# seq numbering starts at 0; a gap is any seq in 0..latest not stored.
|
||||
return [seq for seq in range(seqs[-1] + 1) if seq not in present]
|
||||
|
||||
def latest_seq(self, learner_id: str, task_id: str) -> int:
|
||||
with self._session() as session:
|
||||
stmt = (
|
||||
select(sa.func.max(TelemetryEvent.seq))
|
||||
.where(TelemetryEvent.learner_id == learner_id)
|
||||
.where(TelemetryEvent.task_id == task_id)
|
||||
)
|
||||
latest: Any = session.exec(stmt).one()
|
||||
return -1 if latest is None else int(latest)
|
||||
|
||||
def count(self, learner_id: str, task_id: str) -> int:
|
||||
# COUNT(*) at the DB — no row materialization. The ingest flood cap
|
||||
# calls this per append (telemetry/ingest._flood_breached); the
|
||||
# docstring-free body keeps it obvious what the query shape is.
|
||||
with self._session() as session:
|
||||
stmt = (
|
||||
select(sa.func.count(TelemetryEvent.seq))
|
||||
.where(TelemetryEvent.learner_id == learner_id)
|
||||
.where(TelemetryEvent.task_id == task_id)
|
||||
)
|
||||
total: Any = session.exec(stmt).one()
|
||||
return int(total or 0)
|
||||
|
||||
def list_tasks(self, learner_id: str) -> list[str]:
|
||||
with self._session() as session:
|
||||
stmt = (
|
||||
select(TelemetryEvent.task_id)
|
||||
.where(TelemetryEvent.learner_id == learner_id)
|
||||
.distinct()
|
||||
.order_by(TelemetryEvent.task_id)
|
||||
)
|
||||
# sqlmodel scalar select: rows are plain strs, not 1-tuples.
|
||||
return [str(task_id) for task_id in session.exec(stmt).all()]
|
||||
|
||||
def close(self) -> None:
|
||||
self._engine.dispose()
|
||||
@@ -0,0 +1,62 @@
|
||||
"""HMAC tokens for the WS ingest socket (v0.4.7, P1 final-review flag).
|
||||
|
||||
The ingest WS is a grading input: events land in the trace store the
|
||||
Assessor/Proctor consume. In network mode the socket is reachable on the
|
||||
LAN, and the endpoint accepted any ``learner_id``/``task_id`` query params —
|
||||
anyone on the network could poison or flood a learner's trace (grading
|
||||
DoS, the G-3 flood cap only limits volume, not identity).
|
||||
|
||||
The sandbox manager bakes ``token=<hmac>`` into ``NC_INGEST_URL``; the WS
|
||||
endpoint recomputes the HMAC from the presented identity and rejects on
|
||||
mismatch. Nonce-free and stateless: the token binds (learner_id, task_id,
|
||||
sandbox_id) to the process's secret — nothing to store, nothing to leak
|
||||
beyond the URL the manager already owns. Restart rotates the secret when
|
||||
unset (per-boot random); sandboxes are not reboot-persistent, so stale
|
||||
URLs dying with the boot is the correct lifecycle.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import secrets
|
||||
|
||||
_TOKEN_TTL_BIND_FIELDS = ("learner_id", "task_id", "sandbox_id")
|
||||
|
||||
|
||||
class IngestToken:
|
||||
"""Mint/verify HMAC ingest tokens bound to the settings secret."""
|
||||
|
||||
def __init__(self, secret: str = "") -> None:
|
||||
# Empty/short configured secrets are REPLACED per boot (never warn-
|
||||
# and-accept a guessable secret on an unauthenticated socket).
|
||||
self._secret = (
|
||||
secret.encode("utf-8")
|
||||
if secret
|
||||
else secrets.token_bytes(32)
|
||||
)
|
||||
|
||||
def mint(self, learner_id: str, task_id: str, sandbox_id: str) -> str:
|
||||
msg = _bind_message(learner_id, task_id, sandbox_id)
|
||||
return hmac.new(self._secret, msg, hashlib.sha256).hexdigest()
|
||||
|
||||
def verify(
|
||||
self, token: str, learner_id: str, task_id: str, sandbox_id: str
|
||||
) -> bool:
|
||||
expected = self.mint(learner_id, task_id, sandbox_id)
|
||||
# compare_digest guards the timing channel; an empty token never
|
||||
# matches (mint always yields 64 hex chars).
|
||||
return bool(token) and hmac.compare_digest(expected, token)
|
||||
|
||||
|
||||
def _bind_message(learner_id: str, task_id: str, sandbox_id: str) -> bytes:
|
||||
# Length-prefixed concat — no delimiter injection: a (learner, task)
|
||||
# pair cannot be re-spelled into another valid binding.
|
||||
parts = [learner_id, task_id, sandbox_id]
|
||||
return b"\x00".join(
|
||||
f"{len(p)}:{p}".encode() for p in parts
|
||||
)
|
||||
|
||||
|
||||
#: Field names bound into the token (docs + tests import this).
|
||||
TOKEN_BIND_FIELDS = tuple(_TOKEN_TTL_BIND_FIELDS)
|
||||
@@ -0,0 +1,31 @@
|
||||
"""Per-learner variant task generation — templates, generator, VariantStore (REQ-3-005).
|
||||
|
||||
Boundary rule (D-027): variants/ is an engine module — it never imports
|
||||
api/; its ONLY agents/ dependency is the module-direct
|
||||
agents.structured import in generator.py (the sanctioned shared D-020
|
||||
structured defense, same exception as grading/engine.py). api/ composes
|
||||
the generator and store via DI; store.py imports config only.
|
||||
|
||||
CO-ORDINATION NOTE (ADD, don't REMOVE — same convention as grading/):
|
||||
This __init__.py is a minimal placeholder created by the VariantStore
|
||||
task (4-1-02). The templates task (4-1-01) owns this file's final shape
|
||||
— when templates.py lands, ADD its exports alongside these; do not
|
||||
remove the store exports below.
|
||||
|
||||
Wave status: store.py (VariantRecord, VariantStore, SQLiteVariantStore)
|
||||
landed in Wave 1 (task 4-1-02); templates.py is Wave 1 task 4-1-01;
|
||||
generator.py is Wave 2 (4-2-01).
|
||||
"""
|
||||
|
||||
from .store import SQLiteVariantStore, VariantRecord, VariantStore
|
||||
from .templates import TEMPLATES, TaskTemplate, get_template, template_for_competency
|
||||
|
||||
__all__ = [
|
||||
"SQLiteVariantStore",
|
||||
"TEMPLATES",
|
||||
"TaskTemplate",
|
||||
"VariantRecord",
|
||||
"VariantStore",
|
||||
"get_template",
|
||||
"template_for_competency",
|
||||
]
|
||||
@@ -0,0 +1,138 @@
|
||||
"""Seeded per-learner variant generator (D-029, REQ-3-005).
|
||||
|
||||
Contract (binding, from GRILL + PLAN Must-Haves):
|
||||
- REPRODUCIBLE: seed = sha256(template_id|learner_id|milestone); the same
|
||||
(template, learner) re-derives the same seed, params, task_id — and the
|
||||
second generate() call is a cache hit with NO LLM call.
|
||||
- DISTINCT: different learners on the same template draw different params
|
||||
(the sampler is seeded per-learner) and receive distinct statements.
|
||||
- NEVER BLOCKS ON THE LLM: the deterministic skeleton render
|
||||
(`template.render(params)`) is a complete, valid statement; if the D-020
|
||||
LLM render fails after its bounded retry, the fallback is used — and
|
||||
because the fallback is exactly `template.render(seed-params)`, it is
|
||||
auditable from the persisted seed + params without a provenance column.
|
||||
- AUDITABLE: seed + params + statement persist via VariantStore
|
||||
(insert-only first-wins) — the proctoring cross-check path.
|
||||
- FAIR (a-5): slot draws change the scenario, never the difficulty; the
|
||||
template's rubric anchors bound the expected effort envelope, so every
|
||||
variant of one template is held to the same bar.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from datetime import UTC, datetime
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from ..agents.structured import StructuredOutputError, structured_completion
|
||||
from ..llm.types import Message
|
||||
from ..prompts.variant import VARIANT_SCHEMA_HINT, render_variant_prompt
|
||||
from .store import VariantRecord
|
||||
from .templates import TaskTemplate, get_template
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover
|
||||
from ..llm.base import LLMProvider
|
||||
from .store import VariantStore
|
||||
|
||||
MILESTONE = "v0.3"
|
||||
|
||||
|
||||
class RenderedVariant(BaseModel):
|
||||
"""D-20-validated LLM render output (statement only — files come from the template)."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
statement: str = Field(min_length=20)
|
||||
|
||||
|
||||
class UnknownTemplateError(ValueError):
|
||||
"""Raised when generate() is asked for a template id not in the library."""
|
||||
|
||||
|
||||
def derive_seed(template_id: str, learner_id: str, milestone: str = MILESTONE) -> str:
|
||||
"""Reproducible per-(template, learner, milestone) seed (D-029)."""
|
||||
return hashlib.sha256(f"{template_id}|{learner_id}|{milestone}".encode()).hexdigest()
|
||||
|
||||
|
||||
def derive_task_id(seed: str) -> str:
|
||||
"""Deterministic grading/telemetry task key from the seed (16 hex chars)."""
|
||||
return f"task-{seed[:16]}"
|
||||
|
||||
|
||||
class VariantGenerator:
|
||||
"""Seeded instantiation over the template library. DI: store + provider."""
|
||||
|
||||
def __init__(self, store: VariantStore, provider: LLMProvider, model: str) -> None:
|
||||
self._store = store
|
||||
self._provider = provider
|
||||
self._model = model
|
||||
|
||||
async def generate(self, learner_id: str, template_id: str) -> VariantRecord:
|
||||
template = get_template(template_id)
|
||||
if template is None:
|
||||
raise UnknownTemplateError(f"no task template with id {template_id!r}")
|
||||
|
||||
# Cache: D-029 reproducibility — same (learner, template) is served
|
||||
# from the store with no LLM call.
|
||||
cached = self._store.get(learner_id, template_id)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
seed_hex = derive_seed(template_id, learner_id)
|
||||
task_id = derive_task_id(seed_hex)
|
||||
params = template.sample_params(_seed_int(seed_hex))
|
||||
_validate_params(template, params)
|
||||
|
||||
statement = await self._render(template, params)
|
||||
record = VariantRecord(
|
||||
learner_id=learner_id,
|
||||
task_id=task_id,
|
||||
template_id=template_id,
|
||||
seed=seed_hex,
|
||||
params=dict(params),
|
||||
statement=statement,
|
||||
starter_files=dict(template.starter_files),
|
||||
environment=template.environment,
|
||||
test_command=template.test_command,
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
self._store.save(record)
|
||||
return record
|
||||
|
||||
async def _render(self, template: TaskTemplate, params: dict[str, str | int]) -> str:
|
||||
"""LLM render via D-020; deterministic fallback never blocks task work.
|
||||
|
||||
Provenance note: unlike grades, variants carry no `model` column —
|
||||
the deterministic fallback is exactly `template.render(params)`,
|
||||
re-derivable from the persisted seed + params, so a fallback render is
|
||||
auditable without storing provenance (the seed IS the provenance).
|
||||
"""
|
||||
messages: list[Message] = render_variant_prompt(template, params)
|
||||
try:
|
||||
rendered = await structured_completion(
|
||||
self._provider,
|
||||
messages,
|
||||
model=self._model,
|
||||
schema=RenderedVariant,
|
||||
schema_hint=VARIANT_SCHEMA_HINT,
|
||||
)
|
||||
except StructuredOutputError:
|
||||
# Deterministic fallback: the skeleton + seeded slots is already a
|
||||
# complete statement, re-derivable from the persisted seed.
|
||||
return template.render(params)
|
||||
return rendered.statement
|
||||
|
||||
|
||||
def _seed_int(seed_hex: str) -> int:
|
||||
"""Stable int for random.Random from the hex seed."""
|
||||
return int(seed_hex[:16], 16)
|
||||
|
||||
|
||||
def _validate_params(template: TaskTemplate, params: dict[str, str | int]) -> None:
|
||||
"""Defense in depth: every sampled value must be schema-valid (a-5)."""
|
||||
for slot in template.slots:
|
||||
value = params.get(slot.name)
|
||||
if value is None or not slot.validate_value(value):
|
||||
raise ValueError(f"sampled params invalid for slot {slot.name!r}: {value!r}")
|
||||
@@ -0,0 +1,351 @@
|
||||
"""VariantStore — variant persistence protocol + SQLite implementation (REQ-3-005, D-027).
|
||||
|
||||
Postgres-migration-ready (D-027): the protocol is the only surface the
|
||||
variant generator and API layers touch; swapping SQLiteVariantStore for a
|
||||
Postgres-backed implementation must not change call sites. The
|
||||
`variant_record` table uses only portable column types (str / JSON /
|
||||
datetime), so the same SQLModel schema stands up unchanged on Postgres.
|
||||
|
||||
Insert-only, NOT upsert: (learner_id, template_id) is the variant identity
|
||||
and the FIRST generation is authoritative — reproducibility (D-029) means
|
||||
the seed re-derives the same variant, so the generator's cache path serves
|
||||
`get` instead of saving again. `save` is a plain INSERT; a duplicate pair
|
||||
raises sqlalchemy.exc.IntegrityError to the caller (documented behavior).
|
||||
`task_id` is unique too — it is the grading/telemetry trace key, so a
|
||||
trace or grade can never silently join to a different variant. Both
|
||||
rejections are deliberate: overwriting a stored variant would swap a
|
||||
learner's graded task underneath its trace and grade (audit corruption).
|
||||
Contrast TraceStore.append (dedup-keep-first, swallowed — at-least-once
|
||||
ingest) and GradeStore.save (upsert-latest-wins — a regrade is
|
||||
latest-state); this store is the third contract of the D-027 family.
|
||||
|
||||
Concurrency (a-3): the store enables WAL + synchronous=NORMAL and a busy
|
||||
timeout at connection time, so a generation writer and API readers do not
|
||||
hit `database is locked` on the single-box pilot.
|
||||
|
||||
`created_at` contract: callers stamp UTC (datetime.now(UTC)); SQLite
|
||||
stores it naive and the read paths re-label it tz-aware UTC (same
|
||||
boundary normalization as TelemetryEvent.ts / GradeRecord.created_at, so
|
||||
the contract holds on any backend).
|
||||
|
||||
Boundary (D-027): `variants/` never imports `agents/` / `api/`; this
|
||||
module imports config only.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy import JSON, Index, String, UniqueConstraint
|
||||
from sqlalchemy.orm import validates
|
||||
from sqlmodel import Field, Session, SQLModel, create_engine, select
|
||||
|
||||
from ..config import Settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class VariantRecord(SQLModel, table=True):
|
||||
"""A persisted task variant; (learner_id, template_id) is the PK — first wins.
|
||||
|
||||
Written once by the variant generator (Task 4-2-01), read by the API
|
||||
layer and proctoring cross-checks through the VariantStore protocol.
|
||||
Constraint enforcement mirrors TelemetryEvent / GradeRecord: sqlmodel
|
||||
0.0.42's metaclass drops pydantic constraints on table models, so
|
||||
SQLAlchemy `@validates` hooks enforce instead and the column types
|
||||
stay Postgres-ready (D-027).
|
||||
|
||||
Field contract:
|
||||
learner_id — non-empty learner identifier (same id space as
|
||||
traces and grades).
|
||||
template_id — non-empty task template identifier; variant
|
||||
identity is the (learner_id, template_id) pair —
|
||||
the pair the generator caches on (exactly one
|
||||
variant per learner per template).
|
||||
task_id — non-empty, GLOBALLY unique task identifier; the
|
||||
grading/telemetry trace key (the (learner_id,
|
||||
task_id) pair TraceStore / GradeStore key on),
|
||||
stamped at generation so a variant's trace and
|
||||
grade join back to it exactly once.
|
||||
seed — non-empty variant seed (D-029); derived from
|
||||
(template_id, learner_id, milestone) so the
|
||||
variant is reproducible and auditable.
|
||||
params — typed parameter-slot values the generator filled;
|
||||
JSON dict. An empty dict is legal (a slotless
|
||||
template).
|
||||
statement — non-empty rendered task statement shown to the
|
||||
learner (distinct per learner by construction,
|
||||
REQ-3-005).
|
||||
starter_files — workspace scaffold: filename -> file content;
|
||||
JSON dict. An empty dict is legal (no scaffold).
|
||||
created_at — UTC generation timestamp.
|
||||
"""
|
||||
|
||||
__tablename__ = "variant_record"
|
||||
# The composite PK covers (learner_id, template_id) point lookups; the
|
||||
# unique task_id covers get_by_task (the grading/telemetry join path);
|
||||
# the two secondary indexes cover list_for_learner / list_by_template
|
||||
# ordered by created_at without a sort step (Postgres target D-027).
|
||||
__table_args__ = (
|
||||
UniqueConstraint("task_id", name="uq_variant_record_task_id"),
|
||||
Index("ix_variant_record_learner_created", "learner_id", "created_at"),
|
||||
Index("ix_variant_record_template_created", "template_id", "created_at"),
|
||||
)
|
||||
|
||||
learner_id: str = Field(primary_key=True)
|
||||
template_id: str = Field(primary_key=True)
|
||||
task_id: str
|
||||
seed: str
|
||||
# JSON columns: stored as TEXT on SQLite, native JSONB on Postgres (D-027).
|
||||
params: dict[str, Any] = Field(default_factory=dict, sa_type=JSON)
|
||||
statement: str
|
||||
starter_files: dict[str, Any] = Field(default_factory=dict, sa_type=JSON)
|
||||
#: REQ-5-005 (D-044): the environment kind rides the record to the API
|
||||
#: and TS client; defaults keep pre-v0.5 rows 'build'.
|
||||
environment: str = Field(default="build", sa_type=String)
|
||||
#: The variant's real test command (was a dead template field — v0.5
|
||||
#: surfaces it so the Run/Test buttons stop hardcoding pytest).
|
||||
test_command: str = Field(default="", sa_type=String)
|
||||
created_at: datetime
|
||||
|
||||
@validates("learner_id", "template_id", "task_id")
|
||||
def _ids_non_empty(self, key: str, value: str) -> str:
|
||||
if not value:
|
||||
raise ValueError(f"{key} must be a non-empty identifier")
|
||||
return value
|
||||
|
||||
@validates("seed")
|
||||
def _seed_non_empty(self, key: str, value: str) -> str:
|
||||
if not value:
|
||||
raise ValueError(f"{key} must be a non-empty seed string")
|
||||
return value
|
||||
|
||||
@validates("statement")
|
||||
def _statement_non_empty(self, key: str, value: str) -> str:
|
||||
if not value:
|
||||
raise ValueError(f"{key} must be a non-empty statement string")
|
||||
return value
|
||||
|
||||
|
||||
class VariantStore(Protocol):
|
||||
"""Persistence contract for reproducible per-learner task variants.
|
||||
|
||||
Implemented by SQLiteVariantStore (v0.3, D-027); a Postgres
|
||||
implementation must satisfy the same surface.
|
||||
"""
|
||||
|
||||
def save(self, variant: VariantRecord) -> None:
|
||||
"""Persist a new variant. INSERT-ONLY on (learner_id, template_id):
|
||||
the FIRST generated variant is authoritative (reproducibility,
|
||||
D-029); a duplicate pair raises sqlalchemy.exc.IntegrityError to
|
||||
the caller — the generator serves cached variants via `get`
|
||||
instead of saving again. `task_id` is unique too: claiming an
|
||||
existing trace key for a different variant is equally rejected.
|
||||
NOT upsert; contrast GradeStore.save (latest-wins) and
|
||||
TraceStore.append (dedup-keep-first, swallowed).
|
||||
"""
|
||||
...
|
||||
|
||||
def get(self, learner_id: str, template_id: str) -> VariantRecord | None:
|
||||
"""The learner's stored variant for the template; None when none
|
||||
exists. Detached from any DB session — safe to pass across layers.
|
||||
"""
|
||||
...
|
||||
|
||||
def get_by_task(self, task_id: str) -> VariantRecord | None:
|
||||
"""The variant owning the task key (the grading/telemetry join
|
||||
path); None when none exists. Detached from any DB session.
|
||||
"""
|
||||
...
|
||||
|
||||
def list_for_learner(self, learner_id: str) -> list[VariantRecord]:
|
||||
"""All stored variants for the learner, ordered by created_at
|
||||
ascending (chronological; task_id breaks same-instant ties).
|
||||
Empty list when the learner has none.
|
||||
"""
|
||||
...
|
||||
|
||||
def list_by_template(self, template_id: str) -> list[VariantRecord]:
|
||||
"""All stored variants generated from the template — one row per
|
||||
learner — ordered by created_at ascending (chronological;
|
||||
learner_id breaks same-instant ties). Empty list when the
|
||||
template has none. The proctoring cross-check path (seed params
|
||||
per learner) reads through this.
|
||||
"""
|
||||
...
|
||||
|
||||
def close(self) -> None:
|
||||
"""Release DB connections. Store must not be used after close."""
|
||||
...
|
||||
|
||||
|
||||
def _sqlite_connect(dbapi_connection: sqlite3.Connection, _: object) -> None:
|
||||
"""Per-connection pragma setup (a-3). Mirrors telemetry/grading stores.
|
||||
|
||||
journal_mode=WAL — readers never block the single writer.
|
||||
synchronous=NORMAL — safe in WAL mode, avoids full fsync-per-commit.
|
||||
busy_timeout=5000 — retry briefly under contention instead of
|
||||
`OperationalError: database is locked`.
|
||||
"""
|
||||
cursor = dbapi_connection.cursor()
|
||||
cursor.execute("PRAGMA journal_mode=WAL")
|
||||
cursor.execute("PRAGMA synchronous=NORMAL")
|
||||
cursor.execute("PRAGMA busy_timeout=5000")
|
||||
cursor.close()
|
||||
|
||||
|
||||
def _as_utc(ts: datetime) -> datetime:
|
||||
"""Timestamps travel as naive datetime on SQLite, tz-aware elsewhere.
|
||||
|
||||
SQLite (via SQLModel) drops tzinfo; Postgres TIMESTAMP WITH TIME ZONE
|
||||
keeps it. Normalizing on the read path makes the store's contract
|
||||
tz-aware UTC regardless of the backend (D-027).
|
||||
"""
|
||||
if ts.tzinfo is None:
|
||||
return ts.replace(tzinfo=UTC) # naive UTC read-side: label as UTC
|
||||
return ts.astimezone(UTC)
|
||||
|
||||
|
||||
class SQLiteVariantStore:
|
||||
"""SQLite-backed VariantStore (SQLModel). Third protocol-wrapped store
|
||||
of the D-027 family (first: SQLiteTraceStore, second: SQLiteGradeStore).
|
||||
"""
|
||||
|
||||
def __init__(self, db_path: Path | None = None) -> None:
|
||||
self._db_path: Path = db_path if db_path is not None else Settings().db_path
|
||||
self._engine = create_engine(f"sqlite:///{self._db_path}")
|
||||
sa.event.listen(self._engine, "connect", _sqlite_connect)
|
||||
SQLModel.metadata.create_all(self._engine)
|
||||
# v0.5 schema (D-044) added two columns to an existing table.
|
||||
# create_all does NOT ALTER existing tables: on a box with a
|
||||
# pre-v0.5 ~/.nextcraft/data/nextcraft.db, every variant read/write
|
||||
# would raise OperationalError("no such column: variant_record.
|
||||
# environment") — a silent total breakage of the variant path
|
||||
# (final-review P0, verified empirically). Backfill the missing
|
||||
# columns with the model defaults ('build' keeps pre-v0.5 rows
|
||||
# build-kind per the field contract; '' falls back to pytest at
|
||||
# the API seam, api/variants._to_response). Idempotent: the
|
||||
# PRAGMA table_info check makes re-runs no-ops.
|
||||
self._ensure_v05_columns()
|
||||
|
||||
def _ensure_v05_columns(self) -> None:
|
||||
"""Add v0.5 columns to a pre-v0.5 variant_record table (idempotent)."""
|
||||
from sqlalchemy import text
|
||||
|
||||
with self._engine.begin() as conn:
|
||||
columns = {row[1] for row in conn.execute(text("PRAGMA table_info(variant_record)"))}
|
||||
if "environment" not in columns:
|
||||
conn.execute(
|
||||
text(
|
||||
"ALTER TABLE variant_record ADD COLUMN environment "
|
||||
"VARCHAR DEFAULT 'build' NOT NULL"
|
||||
)
|
||||
)
|
||||
logger.info("variant store: backfilled 'environment' (pre-v0.5 schema)")
|
||||
if "test_command" not in columns:
|
||||
conn.execute(
|
||||
text(
|
||||
"ALTER TABLE variant_record ADD COLUMN test_command "
|
||||
"VARCHAR DEFAULT '' NOT NULL"
|
||||
)
|
||||
)
|
||||
logger.info("variant store: backfilled 'test_command' (pre-v0.5 schema)")
|
||||
|
||||
@contextmanager
|
||||
def _session(self) -> Iterator[Session]:
|
||||
# expire_on_commit=False: identical session behavior to the other
|
||||
# D-027 stores. save() never commits on the error path and the read
|
||||
# paths never commit, but a uniform flag across the family keeps
|
||||
# their detachment guarantees from diverging.
|
||||
with Session(self._engine, expire_on_commit=False) as session:
|
||||
yield session
|
||||
|
||||
def save(self, variant: VariantRecord) -> None:
|
||||
# Plain INSERT, no merge: overwriting a stored variant would swap a
|
||||
# learner's graded task underneath its trace and grade (audit
|
||||
# corruption), so a duplicate identity is a race or bug to SURFACE,
|
||||
# not paper over. The generator's cache path (get before generate)
|
||||
# makes duplicate saves a programming error, not a normal flow.
|
||||
# The trace store swallows its IntegrityError (dedup is the
|
||||
# contract there); the grade store merges (latest-wins is the
|
||||
# contract there); this store re-raises (first-wins is the
|
||||
# contract here).
|
||||
with self._session() as session:
|
||||
try:
|
||||
session.add(variant)
|
||||
session.commit()
|
||||
except sa.exc.IntegrityError:
|
||||
session.rollback()
|
||||
logger.debug(
|
||||
"variant insert rejected (identity already stored): "
|
||||
"learner=%s template=%s task=%s",
|
||||
variant.learner_id,
|
||||
variant.template_id,
|
||||
variant.task_id,
|
||||
)
|
||||
raise
|
||||
logger.debug(
|
||||
"variant saved: %s/%s task=%s seed=%s",
|
||||
variant.learner_id,
|
||||
variant.template_id,
|
||||
variant.task_id,
|
||||
variant.seed,
|
||||
)
|
||||
|
||||
def get(self, learner_id: str, template_id: str) -> VariantRecord | None:
|
||||
with self._session() as session:
|
||||
record = session.get(VariantRecord, (learner_id, template_id))
|
||||
if record is None:
|
||||
return None
|
||||
record.created_at = _as_utc(record.created_at)
|
||||
# Detach from the session: callers must not depend on
|
||||
# open-session ORM magic (lazy loads fail once it closes).
|
||||
session.expunge(record)
|
||||
return record
|
||||
|
||||
def get_by_task(self, task_id: str) -> VariantRecord | None:
|
||||
with self._session() as session:
|
||||
stmt = select(VariantRecord).where(VariantRecord.task_id == task_id)
|
||||
record = session.exec(stmt).first()
|
||||
if record is None:
|
||||
return None
|
||||
record.created_at = _as_utc(record.created_at)
|
||||
session.expunge(record)
|
||||
return record
|
||||
|
||||
def list_for_learner(self, learner_id: str) -> list[VariantRecord]:
|
||||
with self._session() as session:
|
||||
stmt = (
|
||||
select(VariantRecord)
|
||||
.where(VariantRecord.learner_id == learner_id)
|
||||
# Chronological; task_id is a deterministic tie-break for
|
||||
# variants stamped within the same instant.
|
||||
.order_by(VariantRecord.created_at, VariantRecord.task_id)
|
||||
)
|
||||
results = session.exec(stmt).all()
|
||||
for row in results:
|
||||
row.created_at = _as_utc(row.created_at)
|
||||
session.expunge(row)
|
||||
return list(results)
|
||||
|
||||
def list_by_template(self, template_id: str) -> list[VariantRecord]:
|
||||
with self._session() as session:
|
||||
stmt = (
|
||||
select(VariantRecord)
|
||||
.where(VariantRecord.template_id == template_id)
|
||||
# Chronological; learner_id is a deterministic tie-break.
|
||||
.order_by(VariantRecord.created_at, VariantRecord.learner_id)
|
||||
)
|
||||
results = session.exec(stmt).all()
|
||||
for row in results:
|
||||
row.created_at = _as_utc(row.created_at)
|
||||
session.expunge(row)
|
||||
return list(results)
|
||||
|
||||
def close(self) -> None:
|
||||
self._engine.dispose()
|
||||
@@ -0,0 +1,474 @@
|
||||
"""Task template library for seeded variant generation (D-029, REQ-3-005).
|
||||
|
||||
A `TaskTemplate` binds a competency (D-021-aligned corpus ID), a statement
|
||||
skeleton with `{slot}` placeholders, typed `ParameterSlot`s, difficulty-
|
||||
normalization rubric anchors (the expected feature envelope that bounds
|
||||
variant fairness in the a-5 envelope test — grader-prompt shipment is the
|
||||
tracked P4 follow-up; grading is variant-blind today), and starter-file
|
||||
scaffolds served into the sandbox workdir (wired in P6).
|
||||
|
||||
Slot sampling is PURE CODE: `random.Random(seed)` over typed slots — fully
|
||||
reproducible for a given seed, independent of the LLM. The LLM only renders
|
||||
the seeded slot values into the statement skeleton (D-020 defense).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
import re
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
#: REQ-5-005 environment kinds (D-044): one namespace fabric, typed starter
|
||||
#: contents + command policy. 'build' = the v0.3 coding IDE; 'design' =
|
||||
#: artifact editing with a validator/renderer harness; 'simulation' = a
|
||||
#: parameterized run harness (benchmark scripts + datasets).
|
||||
EnvironmentKind = Literal["build", "design", "simulation"]
|
||||
|
||||
|
||||
def validate_simple_argv(value: str) -> str:
|
||||
"""G-15: command fields must roundtrip shlex.split → join → split.
|
||||
|
||||
No quotes, no shell metachars — the TS client splits on whitespace only
|
||||
(no shlex in browsers), so anything quote-aware would split differently
|
||||
on the two sides. A violation is a template-AUTHORING bug caught here,
|
||||
at definition time, in Python where shlex exists.
|
||||
"""
|
||||
import shlex
|
||||
|
||||
parts = shlex.split(value)
|
||||
if not parts:
|
||||
raise ValueError("command must not be empty")
|
||||
joined = " ".join(parts)
|
||||
if shlex.split(joined) != parts:
|
||||
raise ValueError(f"command is not whitespace-joinable: {value!r}")
|
||||
return joined
|
||||
|
||||
SlotType = Literal["enum", "int_range", "string_set"]
|
||||
|
||||
|
||||
class ParameterSlot(BaseModel):
|
||||
"""One typed fill-in for a statement skeleton."""
|
||||
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
name: str = Field(min_length=1)
|
||||
type: SlotType
|
||||
values: list[str] = Field(default_factory=list) # enum/string_set options
|
||||
lo: int | None = None # int_range bounds
|
||||
hi: int | None = None
|
||||
|
||||
@field_validator("values")
|
||||
@classmethod
|
||||
def _values_nonempty_for_enums(cls, v: list[str], info) -> list[str]:
|
||||
if info.data.get("type") in ("enum", "string_set") and not v:
|
||||
raise ValueError(f"slot {info.data.get('name')!r} needs values")
|
||||
return v
|
||||
|
||||
def sample(self, rng: random.Random) -> str | int:
|
||||
"""Deterministic sample from the seeded RNG. Validated after sampling."""
|
||||
if self.type == "enum" or self.type == "string_set":
|
||||
return rng.choice(self.values)
|
||||
if self.type == "int_range":
|
||||
lo = self.lo if self.lo is not None else 0
|
||||
hi = self.hi if self.hi is not None else lo
|
||||
if hi < lo:
|
||||
raise ValueError(f"slot {self.name!r}: hi < lo")
|
||||
return rng.randint(lo, hi)
|
||||
raise ValueError(f"unsupported slot type: {self.type!r}")
|
||||
|
||||
def validate_value(self, value: str | int) -> bool:
|
||||
"""Is `value` schema-valid for this slot? (params JSON gate, a-5.)"""
|
||||
if self.type in ("enum", "string_set"):
|
||||
return isinstance(value, str) and value in self.values
|
||||
if self.type == "int_range":
|
||||
lo = self.lo if self.lo is not None else 0
|
||||
hi = self.hi if self.hi is not None else lo
|
||||
return isinstance(value, int) and lo <= value <= hi
|
||||
return False
|
||||
|
||||
|
||||
class RubricAnchors(BaseModel):
|
||||
"""Difficulty-normalization anchors for the grader (a-5).
|
||||
|
||||
Expected FEATURE ENVELOPE (digest-space): the expected effort band
|
||||
for this template, so two variants of one template are held to the
|
||||
same bar regardless of which slot values a learner drew. The a-5
|
||||
envelope test (tests/variants/test_generator.py) binds variants to
|
||||
these bands in code, and — since Phase 4 (MH#4) — the grading engine
|
||||
ships this envelope into the grader prompt
|
||||
(grading/engine._anchors_context) and stamps the variant seed on the
|
||||
GradeRecord, so the anchors gate variant fairness in BOTH tests and
|
||||
the live rubric.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
expected_edit_count_band: tuple[int, int]
|
||||
expected_min_test_runs: int
|
||||
expected_error_fix_cycles_band: tuple[int, int]
|
||||
notes: str = ""
|
||||
|
||||
|
||||
class TaskTemplate(BaseModel):
|
||||
"""A reusable task shape; variants instantiate it per learner."""
|
||||
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
id: str = Field(min_length=1)
|
||||
competency_id: str = Field(min_length=1) # D-021 corpus alignment
|
||||
title: str
|
||||
statement_skeleton: str = Field(min_length=1) # {slot} placeholders
|
||||
slots: list[ParameterSlot] = Field(min_length=1)
|
||||
rubric_anchors: RubricAnchors
|
||||
starter_files: dict[str, str] = Field(default_factory=dict) # path -> content
|
||||
test_command: str
|
||||
#: REQ-5-005 (D-044): the environment kind rides the variant through
|
||||
#: the API to the TS client; 'build' default keeps v0.3 behavior.
|
||||
environment: EnvironmentKind = "build"
|
||||
#: The kind's Run harness (design: validator/renderer; simulation:
|
||||
#: benchmark script). Defaults to the test_command for build kinds.
|
||||
harness_command: str = ""
|
||||
|
||||
@field_validator("statement_skeleton")
|
||||
@classmethod
|
||||
def _skeleton_placeholders(cls, v: str) -> str:
|
||||
if "{" not in v or "}" not in v:
|
||||
raise ValueError("statement_skeleton needs at least one {slot}")
|
||||
return v
|
||||
|
||||
@field_validator("test_command", "harness_command")
|
||||
@classmethod
|
||||
def _simple_argv(cls, v: str) -> str:
|
||||
return validate_simple_argv(v) if v else v
|
||||
|
||||
@property
|
||||
def run_command(self) -> str:
|
||||
"""The Run button's command: kind harness when declared, else tests."""
|
||||
return self.harness_command or self.test_command
|
||||
|
||||
def render(self, params: dict[str, str | int]) -> str:
|
||||
"""Fill the skeleton with validated params."""
|
||||
for slot in self.slots:
|
||||
if slot.name not in params:
|
||||
raise ValueError(f"missing param for slot {slot.name!r}")
|
||||
if not slot.validate_value(params[slot.name]):
|
||||
raise ValueError(f"invalid value for slot {slot.name!r}: {params[slot.name]!r}")
|
||||
return self.statement_skeleton.format(**params)
|
||||
|
||||
def sample_params(self, seed: int) -> dict[str, str | int]:
|
||||
"""Seeded, reproducible, schema-valid slot values (pure code)."""
|
||||
rng = random.Random(seed)
|
||||
return {slot.name: slot.sample(rng) for slot in self.slots}
|
||||
|
||||
|
||||
# --- Template library (v0.3 initial set) --------------------------------------
|
||||
# Competency IDs are D-021-aligned with the Python corpus
|
||||
# (ai_service/corpus/learner_context.py) and the TS mock-data layer
|
||||
# (packages/mock-data/competency-stacks.ts: deterministic cid() scheme).
|
||||
|
||||
TEMPLATES: dict[str, TaskTemplate] = {
|
||||
"tpl-llm-judge": TaskTemplate(
|
||||
id="tpl-llm-judge",
|
||||
competency_id="stack-orchestration-c007",
|
||||
title="Build an LLM-as-Judge Evaluator",
|
||||
statement_skeleton=(
|
||||
"Build a small LLM-as-judge evaluator for {domain} answers. "
|
||||
"The judge must score each answer on {criterion} using a 0-4 scale, "
|
||||
"return structured JSON, and handle at least {edge_cases} edge-case "
|
||||
"answer classes (empty, off-topic, adversarial). Include a tiny "
|
||||
"repro test set of at least {test_size} examples and print a summary "
|
||||
"table of scores."
|
||||
),
|
||||
slots=[
|
||||
ParameterSlot(
|
||||
name="domain",
|
||||
type="enum",
|
||||
values=["customer-support", "code-review", "summarization", "tutoring"],
|
||||
),
|
||||
ParameterSlot(
|
||||
name="criterion",
|
||||
type="enum",
|
||||
values=["factual-accuracy", "helpfulness", "safety", "completeness"],
|
||||
),
|
||||
ParameterSlot(name="edge_cases", type="int_range", lo=2, hi=4),
|
||||
ParameterSlot(name="test_size", type="int_range", lo=3, hi=8),
|
||||
],
|
||||
rubric_anchors=RubricAnchors(
|
||||
expected_edit_count_band=(3, 25),
|
||||
expected_min_test_runs=2,
|
||||
expected_error_fix_cycles_band=(0, 4),
|
||||
notes="Slot draw changes the SCENARIO, not the engineering depth.",
|
||||
),
|
||||
starter_files={
|
||||
"README.md": (
|
||||
"# LLM-as-Judge Evaluator\n\n"
|
||||
"Implement `judge.py`:\n"
|
||||
"- `score(answer: str) -> dict` — 0-4 on the named criterion\n"
|
||||
"- structured JSON output (schema below)\n"
|
||||
"- edge-case classes handled explicitly\n"
|
||||
"- `pytest` must pass\n"
|
||||
),
|
||||
"judge.py": "def score(answer: str) -> dict:\n raise NotImplementedError\n",
|
||||
"test_judge.py": "def test_placeholder():\n assert True\n",
|
||||
},
|
||||
test_command="pytest -q",
|
||||
),
|
||||
"tpl-guardrail-schema": TaskTemplate(
|
||||
id="tpl-guardrail-schema",
|
||||
competency_id="stack-orchestration-c008",
|
||||
title="Schema Guardrail Pipeline",
|
||||
statement_skeleton=(
|
||||
"Implement an output-validation guardrail for a model returning "
|
||||
"{entity} records. Validate against a typed schema with {field_count} "
|
||||
"required fields, coerce or reject {failure_mode} failures, and emit "
|
||||
"a fallback response for invalid payloads. Cover with at least "
|
||||
"{test_size} unit tests including malformed JSON."
|
||||
),
|
||||
slots=[
|
||||
ParameterSlot(
|
||||
name="entity",
|
||||
type="enum",
|
||||
values=["user-profile", "job-posting", "candidate", "invoice"],
|
||||
),
|
||||
ParameterSlot(
|
||||
name="failure_mode",
|
||||
type="enum",
|
||||
values=["strict-reject", "coerce-when-safe"],
|
||||
),
|
||||
ParameterSlot(name="field_count", type="int_range", lo=4, hi=8),
|
||||
ParameterSlot(name="test_size", type="int_range", lo=4, hi=10),
|
||||
],
|
||||
rubric_anchors=RubricAnchors(
|
||||
expected_edit_count_band=(3, 30),
|
||||
expected_min_test_runs=2,
|
||||
expected_error_fix_cycles_band=(0, 5),
|
||||
notes="All slot draws land in the same engineering band.",
|
||||
),
|
||||
starter_files={
|
||||
"README.md": (
|
||||
"# Schema Guardrail\n\nImplement `guardrail.py`:\n"
|
||||
"- `validate(payload: dict) -> dict | Fallback`\n"
|
||||
"- required-field checks, failure policy, fallback emission\n"
|
||||
),
|
||||
"guardrail.py": "def validate(payload: dict):\n raise NotImplementedError\n",
|
||||
"test_guardrail.py": "def test_placeholder():\n assert True\n",
|
||||
},
|
||||
test_command="pytest -q",
|
||||
),
|
||||
"tpl-rag-chunker": TaskTemplate(
|
||||
id="tpl-rag-chunker",
|
||||
competency_id="stack-orchestration-c005",
|
||||
title="RAG Chunking Strategy",
|
||||
statement_skeleton=(
|
||||
"Implement a document chunker for {doc_type} retrieval. Support "
|
||||
"{strategy} chunking with a target size of ~{chunk_size} tokens, "
|
||||
"preserve {invariant} across chunk boundaries, and evaluate overlap "
|
||||
"quality with at least {test_size} fixture documents."
|
||||
),
|
||||
slots=[
|
||||
ParameterSlot(
|
||||
name="doc_type",
|
||||
type="enum",
|
||||
values=["technical-docs", "legal-contracts", "transcripts"],
|
||||
),
|
||||
ParameterSlot(
|
||||
name="strategy",
|
||||
type="enum",
|
||||
values=["fixed-window", "semantic-boundary", "hybrid"],
|
||||
),
|
||||
ParameterSlot(
|
||||
name="invariant",
|
||||
type="enum",
|
||||
values=["code-block-integrity", "section-headers", "sentence-completeness"],
|
||||
),
|
||||
ParameterSlot(name="chunk_size", type="int_range", lo=200, hi=800),
|
||||
ParameterSlot(name="test_size", type="int_range", lo=3, hi=6),
|
||||
],
|
||||
rubric_anchors=RubricAnchors(
|
||||
expected_edit_count_band=(4, 35),
|
||||
expected_min_test_runs=2,
|
||||
expected_error_fix_cycles_band=(0, 6),
|
||||
notes="Strategy draw changes implementation shape, not depth.",
|
||||
),
|
||||
starter_files={
|
||||
"README.md": (
|
||||
"# RAG Chunker\n\nImplement `chunker.py`:\n"
|
||||
"- `chunk(text: str) -> list[str]`\n- invariant preserved\n- tests green\n"
|
||||
),
|
||||
"chunker.py": "def chunk(text: str) -> list[str]:\n raise NotImplementedError\n",
|
||||
"test_chunker.py": "def test_placeholder():\n assert True\n",
|
||||
},
|
||||
test_command="pytest -q",
|
||||
),
|
||||
# -- REQ-5-005 design environment (D-044): artifact editing with a
|
||||
# -- validator/renderer harness - same fabric, typed starter contents.
|
||||
"tpl-conversation-flow-design": TaskTemplate(
|
||||
id="tpl-conversation-flow-design",
|
||||
competency_id="stack-designer-c001",
|
||||
title="Conversational Flow Artifact",
|
||||
statement_skeleton=(
|
||||
"Design a conversational flow for a {persona} assistant helping "
|
||||
"users accomplish {goal}. Author the flow as a structured artifact "
|
||||
"with at least {turn_count} conversation turns, explicit fallback "
|
||||
"paths for misunderstandings, and an AI-transparency disclosure "
|
||||
"pattern. The flow must render validly (the harness validates "
|
||||
"structure) and read naturally end to end."
|
||||
),
|
||||
slots=[
|
||||
ParameterSlot(
|
||||
name="persona",
|
||||
type="enum",
|
||||
values=["travel-planner", "homework-tutor", "fitness-coach", "recipe-guide"],
|
||||
),
|
||||
ParameterSlot(
|
||||
name="goal",
|
||||
type="enum",
|
||||
values=["book-a-trip", "master-a-concept", "start-a-routine", "cook-a-meal"],
|
||||
),
|
||||
ParameterSlot(name="turn_count", type="int_range", lo=6, hi=12),
|
||||
],
|
||||
rubric_anchors=RubricAnchors(
|
||||
expected_edit_count_band=(3, 20),
|
||||
expected_min_test_runs=1,
|
||||
expected_error_fix_cycles_band=(0, 3),
|
||||
notes="Design kind: artifact quality + iteration cadence, not code depth.",
|
||||
),
|
||||
starter_files={
|
||||
"README.md": (
|
||||
"# Conversational Flow Design\n\n"
|
||||
"Edit flow.md - the structured flow artifact. python3 "
|
||||
"validate_flow.py checks structure (turn headings, fallback "
|
||||
"sections, a transparency disclosure) and reports issues.\n"
|
||||
),
|
||||
"flow.md": (
|
||||
"# Flow: your persona here\n\n"
|
||||
"## Turn 1\n- **AI:** (opening)\n- **User (expected):** ...\n\n"
|
||||
"## Fallback\n- (misunderstanding handling)\n\n"
|
||||
"## AI Transparency Disclosure\n- (disclosure pattern)\n"
|
||||
),
|
||||
"validate_flow.py": (
|
||||
"import re, sys\n"
|
||||
"text = open('flow.md').read()\n"
|
||||
"issues = []\n"
|
||||
"turns = len(re.findall(r'^## Turn', text, re.M))\n"
|
||||
"if turns < 3:\n"
|
||||
" issues.append(f'expected at least 3 turn sections, found {turns}')\n"
|
||||
"if not re.search(r'^## Fallback', text, re.M):\n"
|
||||
" issues.append('missing Fallback section')\n"
|
||||
"if not re.search(r'^## AI Transparency', text, re.M):\n"
|
||||
" issues.append('missing AI Transparency Disclosure')\n"
|
||||
"print('VALID' if not issues else 'ISSUES: ' + '; '.join(issues))\n"
|
||||
"sys.exit(0 if not issues else 1)\n"
|
||||
),
|
||||
},
|
||||
test_command="python3 validate_flow.py",
|
||||
environment="design",
|
||||
harness_command="python3 validate_flow.py",
|
||||
),
|
||||
# -- REQ-5-005 simulation environment (D-044): parameterized benchmark
|
||||
# -- harness with dataset generation.
|
||||
"tpl-sensor-benchmark": TaskTemplate(
|
||||
id="tpl-sensor-benchmark",
|
||||
competency_id="stack-orchestration-c011",
|
||||
title="Sensor Data Simulation Harness",
|
||||
statement_skeleton=(
|
||||
"Build a simulation harness for {sensor} readings over {duration_min} "
|
||||
"minutes at {sample_hz} Hz. Generate a synthetic dataset with a "
|
||||
"realistic noise profile, run the analysis pipeline, and print a "
|
||||
"metrics summary (mean, p95, anomaly count at {anomaly_sigma} sigma). "
|
||||
"The harness must be reproducible from the committed seed."
|
||||
),
|
||||
slots=[
|
||||
ParameterSlot(
|
||||
name="sensor",
|
||||
type="enum",
|
||||
values=["temperature", "vibration", "luminosity", "pressure"],
|
||||
),
|
||||
ParameterSlot(name="duration_min", type="int_range", lo=5, hi=60),
|
||||
ParameterSlot(name="sample_hz", type="int_range", lo=1, hi=10),
|
||||
ParameterSlot(name="anomaly_sigma", type="int_range", lo=2, hi=4),
|
||||
],
|
||||
rubric_anchors=RubricAnchors(
|
||||
expected_edit_count_band=(3, 25),
|
||||
expected_min_test_runs=2,
|
||||
expected_error_fix_cycles_band=(0, 3),
|
||||
notes="Simulation kind: pipeline correctness + reproducibility.",
|
||||
),
|
||||
starter_files={
|
||||
"README.md": (
|
||||
"# Sensor Simulation Harness\n\n"
|
||||
"Edit simulate.py - python3 simulate.py runs the full pipeline: "
|
||||
"generate, analyze, print metrics. pytest covers the analysis "
|
||||
"functions.\n"
|
||||
),
|
||||
"simulate.py": (
|
||||
"import random, statistics\n\n"
|
||||
"def generate(n=300, seed=42):\n"
|
||||
" rng = random.Random(seed)\n"
|
||||
" return [rng.gauss(20.0, 1.5) for _ in range(n)]\n\n"
|
||||
"def analyze(samples, sigma=3):\n"
|
||||
" mean = statistics.fmean(samples)\n"
|
||||
" stdev = statistics.pstdev(samples)\n"
|
||||
" anomalies = [s for s in samples if abs(s - mean) > sigma * stdev]\n"
|
||||
" p95 = sorted(samples)[int(0.95 * len(samples))]\n"
|
||||
" return {'mean': mean, 'p95': p95, 'anomalies': len(anomalies)}\n\n"
|
||||
"if __name__ == '__main__':\n"
|
||||
" print(analyze(generate()))\n"
|
||||
),
|
||||
"test_simulate.py": (
|
||||
"from simulate import generate, analyze\n\n"
|
||||
"def test_reproducible():\n"
|
||||
" assert generate() == generate()\n\n"
|
||||
"def test_metrics_shape():\n"
|
||||
" m = analyze(generate())\n"
|
||||
" assert set(m) == {'mean', 'p95', 'anomalies'}\n"
|
||||
),
|
||||
},
|
||||
test_command="pytest -q",
|
||||
environment="simulation",
|
||||
harness_command="python3 simulate.py",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
_KNOWN_COMPETENCY_IDS: set[str] = {
|
||||
# D-021: mirrored from ai_service/corpus/learner_context.py — the Python
|
||||
# source of truth for stack-orchestration competencies used by v0.2 agents.
|
||||
"stack-orchestration-c001",
|
||||
"stack-orchestration-c002",
|
||||
"stack-orchestration-c003",
|
||||
"stack-orchestration-c004",
|
||||
"stack-orchestration-c005",
|
||||
"stack-orchestration-c007",
|
||||
"stack-orchestration-c008",
|
||||
"stack-orchestration-c011",
|
||||
"stack-designer-c001",
|
||||
"stack-designer-c002",
|
||||
"stack-safety-c021",
|
||||
}
|
||||
|
||||
|
||||
def get_template(template_id: str) -> TaskTemplate | None:
|
||||
return TEMPLATES.get(template_id)
|
||||
|
||||
|
||||
def template_for_competency(competency_id: str) -> list[TaskTemplate]:
|
||||
return [t for t in TEMPLATES.values() if t.competency_id == competency_id]
|
||||
|
||||
|
||||
def validate_competency_binding() -> None:
|
||||
"""All templates must bind to known D-021 corpus competency IDs."""
|
||||
for t in TEMPLATES.values():
|
||||
if t.competency_id not in _KNOWN_COMPETENCY_IDS:
|
||||
raise ValueError(
|
||||
f"template {t.id!r} binds unknown competency {t.competency_id!r}"
|
||||
)
|
||||
|
||||
|
||||
def slots_pattern_ok(skeleton: str, slots: list[ParameterSlot]) -> bool:
|
||||
"""Every {placeholder} in the skeleton has a matching slot and vice versa."""
|
||||
placeholders = set(re.findall(r"\{([a-z_][a-z0-9_]*)\}", skeleton))
|
||||
slot_names = {s.name for s in slots}
|
||||
return placeholders == slot_names
|
||||
@@ -0,0 +1,35 @@
|
||||
"""VoiceProvider protocol (D-030, REQ-3-006) — mirrors the LLMProvider seam.
|
||||
|
||||
Two implementations in v0.3:
|
||||
- MockVoiceProvider — deterministic canned transcripts + canned tone WAV
|
||||
chunks + scripted failure modes (tests + no-key default; tests NEVER call
|
||||
a real voice API).
|
||||
- browser descriptor — not a provider but a FALLBACK HINT: the web client
|
||||
selects browser-native SpeechRecognition/speechSynthesis when the server
|
||||
reports no real voice backend.
|
||||
|
||||
OpenAIAudioProvider (real server STT/TTS over OpenAI-compatible
|
||||
/audio/transcriptions + /audio/speech) is INTENTIONALLY NOT BUILT in v0.3 —
|
||||
deferred to v0.4 with KYC, when there is a real key and real users
|
||||
(GRILL CUT-1 / G-7). This protocol is its future drop-in seam.
|
||||
|
||||
Boundary: `voice/` never imports `agents/` or `api/`.
|
||||
"""
|
||||
|
||||
from ai_service.voice.base import (
|
||||
TranscriptSegment,
|
||||
VoiceDescriptor,
|
||||
VoiceProvider,
|
||||
)
|
||||
from ai_service.voice.browser import BROWSER_FALLBACK_DESCRIPTOR
|
||||
from ai_service.voice.factory import voice_provider_from_settings
|
||||
from ai_service.voice.mock import MockVoiceProvider
|
||||
|
||||
__all__ = [
|
||||
"BROWSER_FALLBACK_DESCRIPTOR",
|
||||
"MockVoiceProvider",
|
||||
"TranscriptSegment",
|
||||
"VoiceDescriptor",
|
||||
"VoiceProvider",
|
||||
"voice_provider_from_settings",
|
||||
]
|
||||
@@ -0,0 +1,62 @@
|
||||
"""VoiceProvider protocol + shared voice contracts (D-030, REQ-3-006).
|
||||
|
||||
Mirrors the LLMProvider seam (D-014 pattern): a narrow protocol the Examiner
|
||||
agent and the defense API compose via DI, with a deterministic mock and a
|
||||
browser-fallback descriptor. No network in this module — concrete providers
|
||||
live in their own modules and are selected by factory/config.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Literal, Protocol, runtime_checkable
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
VoiceRole = Literal["examiner", "learner"]
|
||||
|
||||
|
||||
class TranscriptSegment(BaseModel):
|
||||
"""One STT result: the transcribed text + timing metadata."""
|
||||
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
text: str = Field(min_length=1)
|
||||
language: str = "en"
|
||||
duration_ms: int | None = None
|
||||
confidence: float | None = Field(default=None, ge=0.0, le=1.0)
|
||||
|
||||
|
||||
class VoiceDescriptor(BaseModel):
|
||||
"""Capability descriptor served to the web client (D-030).
|
||||
|
||||
The assessment UI reads this to decide HOW the learner speaks/hears:
|
||||
- `mode="server"` → server-side STT/TTS (openai-audio, live since v0.5)
|
||||
- `mode="browser"` → browser-native SpeechRecognition/speechSynthesis
|
||||
- `mode="mock"` → deterministic no-op path (tests / no-key dev)
|
||||
The descriptor never contains secrets — only capability hints.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
mode: Literal["server", "browser", "mock"]
|
||||
sr_available: bool
|
||||
tts_available: bool
|
||||
hint: str = ""
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class VoiceProvider(Protocol):
|
||||
"""The voice port (D-030): STT in, TTS out. Never imports agents/api."""
|
||||
|
||||
async def transcribe(self, audio: bytes, fmt: str) -> TranscriptSegment:
|
||||
"""STT: audio bytes (fmt: 'wav' | 'webm' | 'mp3') → transcript."""
|
||||
...
|
||||
|
||||
def synthesize(self, text: str, voice: str = "default") -> AsyncIterator[bytes]:
|
||||
"""TTS: text -> async byte chunks (audio stream).
|
||||
|
||||
Implementations may be async generators (async-def + yield) — the
|
||||
consumer contract is `async for chunk in provider.synthesize(text)`.
|
||||
"""
|
||||
...
|
||||
@@ -0,0 +1,33 @@
|
||||
"""Browser-native fallback descriptor (D-030, CUT-1 / G-7, REQ-3-006).
|
||||
|
||||
Browser-native SR/TTS is the no-key CLIENT-side path. When the factory
|
||||
selects `browser` mode, the defense endpoints return this descriptor and the
|
||||
WEB CLIENT performs SpeechRecognition + speechSynthesis natively; the server
|
||||
persists text turns as usual. Real server STT/TTS (openai-audio) is live
|
||||
since v0.5 — this descriptor is the no-key fallback.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from .base import VoiceDescriptor
|
||||
|
||||
BROWSER_FALLBACK_DESCRIPTOR = VoiceDescriptor(
|
||||
mode="browser",
|
||||
sr_available=True,
|
||||
tts_available=True,
|
||||
hint=(
|
||||
"No server voice backend configured. Use browser-native "
|
||||
"SpeechRecognition for STT and speechSynthesis for TTS; send the "
|
||||
"transcribed text to POST /v1/defense/{id}/answer ({text} form)."
|
||||
),
|
||||
)
|
||||
|
||||
MOCK_DESCRIPTOR = VoiceDescriptor(
|
||||
mode="mock",
|
||||
sr_available=True,
|
||||
tts_available=True,
|
||||
hint=(
|
||||
"Deterministic mock voice (tests / no-key dev). Real server "
|
||||
"STT/TTS is live since v0.5 (AI_VOICE_PROVIDER=openai-audio)."
|
||||
),
|
||||
)
|
||||
@@ -0,0 +1,506 @@
|
||||
"""DefenseStore — oral-defense persistence: protocol + SQLite impl (REQ-3-006, D-027).
|
||||
|
||||
FOURTH protocol-wrapped store of the D-027 family and the first spanning
|
||||
TWO related tables: `defense_record` (the defense session + integrity
|
||||
signals) and `defense_turn` (the ordered examiner/learner transcript,
|
||||
FK → defense_record.id).
|
||||
|
||||
Postgres-migration-ready (D-027): the protocol is the only surface the
|
||||
Examiner pipeline (task 5-2-01) and the defense endpoints (task 5-3-01)
|
||||
touch; swapping SQLiteDefenseStore for a Postgres implementation must
|
||||
not change call sites. Both tables use only portable column types
|
||||
(str / int / datetime / JSON), so the same SQLModel schema stands up
|
||||
unchanged on Postgres.
|
||||
|
||||
Save semantics — where this sits among the D-027 stores (each has a
|
||||
deliberately different contract):
|
||||
TraceStore.append dedup-keep-first; IntegrityError SWALLOWED
|
||||
(at-least-once event ingest).
|
||||
GradeStore.save upsert-latest-wins (a regrade is latest-state).
|
||||
VariantStore.save insert-only first-wins; IntegrityError RAISED
|
||||
(reproducibility; a duplicate is a bug).
|
||||
DefenseStore a LIFECYCLE store:
|
||||
start() insert-only; a duplicate id raises
|
||||
(a defense id is minted once per session).
|
||||
append_turn() insert-only per (defense_id, seq); a duplicate
|
||||
seq raises AND an unknown defense_id raises (FK
|
||||
enforced) — a transcript turn must never silently
|
||||
vanish (it is the integrity/grading input) nor
|
||||
attach to a defense that does not exist.
|
||||
finalize() targeted UPDATE (status → finished; finished_at +
|
||||
integrity_signals JSON). Unknown id → None
|
||||
(documented below). Re-finalize overwrites
|
||||
signals + finished_at — latest-wins, mirroring
|
||||
GradeStore.save: a recomputed verdict replaces
|
||||
the previous one wholesale.
|
||||
|
||||
append_turn does NOT police status (turns after finalize are a
|
||||
sequencing bug for the endpoints to prevent, task 5-3-01): the store
|
||||
enforces DATA integrity (FK + PK + non-empty), not workflow.
|
||||
|
||||
integrity_signals (A-109): JSON dict on the record — long pauses,
|
||||
off-scope cadence markers and friends, computed by the Examiner over
|
||||
turn metadata and persisted by finalize for the Proctor/Mentor feed.
|
||||
An empty dict is legal (defense not finished, or a clean defense).
|
||||
|
||||
Concurrency (a-3): WAL + synchronous=NORMAL + busy timeout at
|
||||
connection time (mirrors the other D-027 stores), PLUS foreign_keys=ON
|
||||
— this is the family's first real foreign key and it is actually
|
||||
enforced on SQLite, matching Postgres's native behavior (D-027 parity).
|
||||
|
||||
`created_at` / `ts` contract: callers stamp UTC (datetime.now(UTC));
|
||||
SQLite stores them naive and read paths re-label tz-aware UTC (same
|
||||
boundary normalization as TelemetryEvent.ts / GradeRecord.created_at,
|
||||
so the contract holds on any backend).
|
||||
|
||||
Boundary (D-027): `voice/` never imports `agents/` / `api/`; this
|
||||
module imports config only.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal, Protocol
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy import JSON, Index, String
|
||||
from sqlalchemy.orm import validates
|
||||
from sqlmodel import Field, Session, SQLModel, create_engine, select
|
||||
|
||||
from ..config import Settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DefenseStatus = Literal["in_progress", "finished"]
|
||||
_DEFENSE_STATUSES: frozenset[str] = frozenset(DefenseStatus.__args__)
|
||||
|
||||
TurnRole = Literal["examiner", "learner"]
|
||||
_TURN_ROLES: frozenset[str] = frozenset(TurnRole.__args__)
|
||||
|
||||
|
||||
class DefenseRecord(SQLModel, table=True):
|
||||
"""A persisted oral-defense session; id is the PK.
|
||||
|
||||
Written by the defense endpoints (task 5-3-01) through the
|
||||
DefenseStore protocol; read back by the endpoints, the Examiner
|
||||
pipeline and the Proctor/Mentor feeds. Constraint enforcement
|
||||
mirrors TelemetryEvent / GradeRecord / VariantRecord: sqlmodel
|
||||
0.0.42's metaclass drops pydantic constraints on table models, so
|
||||
SQLAlchemy `@validates` hooks enforce instead and the column types
|
||||
stay Postgres-ready (D-027).
|
||||
|
||||
Field contract:
|
||||
id — non-empty defense identifier, minted once
|
||||
per session (a duplicate start raises).
|
||||
learner_id — non-empty learner identifier (same id space
|
||||
as traces, grades and variants).
|
||||
task_id — non-empty task identifier; the defense
|
||||
defends the submitted work for this trace
|
||||
key ((learner_id, task_id) joins to the
|
||||
trace/grade/variant the defense is about).
|
||||
status — in_progress | finished; the STORE owns the
|
||||
transition: start() forces in_progress,
|
||||
finalize() sets finished. Validated.
|
||||
integrity_signals — A-109 signal dict (long pauses, off-scope
|
||||
cadence markers, ...); {} until finalize;
|
||||
JSON column. An empty dict is legal.
|
||||
created_at — UTC start timestamp.
|
||||
finished_at — UTC finalize timestamp; None while in
|
||||
progress.
|
||||
|
||||
`turns` (property): the seq-ordered DefenseTurn transcript, attached
|
||||
ONLY by DefenseStore.get(); records from list_for_learner carry
|
||||
turns == [] — call get() for a full transcript.
|
||||
"""
|
||||
|
||||
__tablename__ = "defense_record"
|
||||
# The id PK covers point lookups; this secondary index covers
|
||||
# list_for_learner ordered by created_at without a sort step
|
||||
# (Postgres migration target D-027).
|
||||
__table_args__ = (
|
||||
Index("ix_defense_record_learner_created", "learner_id", "created_at"),
|
||||
)
|
||||
|
||||
id: str = Field(primary_key=True)
|
||||
learner_id: str
|
||||
task_id: str
|
||||
# Bare Literal annotations crash sqlmodel<=0.0.42's column inference
|
||||
# (issubclass(TypeAlias, Enum)); an explicit sa_type + the validates
|
||||
# hook below give the same contract: VARCHAR column, Literal-rejected
|
||||
# values (same pattern as TelemetryEvent.kind).
|
||||
status: DefenseStatus = Field(default="in_progress", sa_type=String)
|
||||
# JSON column: stored as TEXT on SQLite, native JSONB on Postgres (D-027).
|
||||
integrity_signals: dict[str, Any] = Field(default_factory=dict, sa_type=JSON)
|
||||
created_at: datetime
|
||||
finished_at: datetime | None = Field(default=None)
|
||||
|
||||
@property
|
||||
def turns(self) -> list["DefenseTurn"]:
|
||||
"""Seq-ordered transcript; [] unless attached by get().
|
||||
|
||||
Table models reject ad-hoc attributes (pydantic __setattr__
|
||||
raises on non-fields), so the store stashes the detached turn
|
||||
list via object.__setattr__ and this read-only property surfaces
|
||||
it. The returned list is a copy — caller mutations cannot
|
||||
corrupt the stash.
|
||||
"""
|
||||
return list(self.__dict__.get("_turns", []))
|
||||
|
||||
@validates("id", "learner_id", "task_id")
|
||||
def _ids_non_empty(self, key: str, value: str) -> str:
|
||||
if not value:
|
||||
raise ValueError(f"{key} must be a non-empty identifier")
|
||||
return value
|
||||
|
||||
@validates("status")
|
||||
def _status_is_known(self, key: str, value: str) -> str:
|
||||
if value not in _DEFENSE_STATUSES:
|
||||
raise ValueError(f"unknown defense status: {value!r}")
|
||||
return value
|
||||
|
||||
|
||||
class DefenseTurn(SQLModel, table=True):
|
||||
"""One examiner/learner dialogue turn; (defense_id, seq) is the PK.
|
||||
|
||||
Rows are append-only transcript entries written through
|
||||
DefenseStore.append_turn. seq numbers the dialogue within one
|
||||
defense starting at 0; monotonic assignment is the endpoints' job
|
||||
(task 5-3-01), this model only rejects negatives — the same split
|
||||
as TelemetryEvent.seq (model rejects < 0, store owns ordering).
|
||||
|
||||
Field contract:
|
||||
defense_id — non-empty; FK → defense_record.id. ENFORCED on
|
||||
SQLite via foreign_keys=ON (first real FK in the
|
||||
D-027 family; Postgres enforces FKs natively, so
|
||||
this keeps the backends equivalent, D-027).
|
||||
seq — turn index within the defense, >= 0. (defense_id,
|
||||
seq) is the PK: a duplicate raises instead of
|
||||
silently overwriting — the transcript is the
|
||||
integrity/grading input, a vanishing turn is
|
||||
audit corruption.
|
||||
role — examiner | learner (who spoke). Validated.
|
||||
text — non-empty utterance text (examiner question, or
|
||||
STT output for learner answers).
|
||||
ts — UTC utterance timestamp.
|
||||
latency_ms — per-turn pipeline latency in ms (STT + LLM TTFT +
|
||||
TTS, A-109); int or None. Populated by the
|
||||
endpoints (task 5-4-01); None allowed here — the
|
||||
store persists, it does not measure.
|
||||
created_at — UTC row-write timestamp.
|
||||
"""
|
||||
|
||||
__tablename__ = "defense_turn"
|
||||
# The composite PK (defense_id, seq) doubles as the covering index
|
||||
# for the per-defense seq-ordered read in get() — no secondary index
|
||||
# needed (contrast defense_record's learner-listing index).
|
||||
|
||||
defense_id: str = Field(foreign_key="defense_record.id", primary_key=True)
|
||||
seq: int = Field(primary_key=True)
|
||||
role: TurnRole = Field(sa_type=String)
|
||||
text: str
|
||||
ts: datetime
|
||||
latency_ms: int | None = Field(default=None)
|
||||
created_at: datetime
|
||||
|
||||
@validates("defense_id")
|
||||
def _defense_id_non_empty(self, key: str, value: str) -> str:
|
||||
if not value:
|
||||
raise ValueError(f"{key} must be a non-empty identifier")
|
||||
return value
|
||||
|
||||
@validates("seq")
|
||||
def _seq_non_negative(self, key: str, value: int) -> int:
|
||||
if value < 0:
|
||||
raise ValueError("seq must be >= 0 (ordering is the endpoints' job)")
|
||||
return value
|
||||
|
||||
@validates("role")
|
||||
def _role_is_known(self, key: str, value: str) -> str:
|
||||
if value not in _TURN_ROLES:
|
||||
raise ValueError(f"unknown turn role: {value!r}")
|
||||
return value
|
||||
|
||||
@validates("text")
|
||||
def _text_non_empty(self, key: str, value: str) -> str:
|
||||
if not value:
|
||||
raise ValueError(f"{key} must be a non-empty utterance string")
|
||||
return value
|
||||
|
||||
@validates("latency_ms")
|
||||
def _latency_non_negative(self, key: str, value: int | None) -> int | None:
|
||||
# None is legal (not yet instrumented); a NEGATIVE latency is
|
||||
# nonsense and surfaces as a construction error.
|
||||
if value is not None and value < 0:
|
||||
raise ValueError("latency_ms must be >= 0 or None")
|
||||
return value
|
||||
|
||||
|
||||
class DefenseStore(Protocol):
|
||||
"""Persistence contract for oral-defense sessions + transcripts.
|
||||
|
||||
Implemented by SQLiteDefenseStore (v0.3, D-027); a Postgres
|
||||
implementation must satisfy the same surface.
|
||||
"""
|
||||
|
||||
def start(self, defense: DefenseRecord) -> DefenseRecord:
|
||||
"""Insert a new defense. INSERT-ONLY: a duplicate id raises
|
||||
sqlalchemy.exc.IntegrityError (a defense id is minted once per
|
||||
session — surfacing, not swallowing, mirrors VariantStore).
|
||||
The store owns the lifecycle: status is forced to "in_progress"
|
||||
and finished_at to None, whatever the caller passed — only
|
||||
finalize() may move a defense to finished. Returns the stored
|
||||
record, detached from any DB session.
|
||||
"""
|
||||
...
|
||||
|
||||
def append_turn(self, defense_id: str, turn: DefenseTurn) -> DefenseTurn:
|
||||
"""Insert one transcript turn, ordered by (defense_id, seq).
|
||||
turn.defense_id MUST equal the defense_id argument — a mismatch
|
||||
raises ValueError (the defense identity must never be
|
||||
ambiguous). A duplicate (defense_id, seq) raises
|
||||
IntegrityError; an unknown defense_id raises IntegrityError
|
||||
(FK enforced). Does NOT police status — sequencing turns vs
|
||||
finalize is the endpoints' job (task 5-3-01). Returns the
|
||||
stored turn, detached.
|
||||
"""
|
||||
...
|
||||
|
||||
def finalize(
|
||||
self, defense_id: str, integrity_signals: dict[str, Any]
|
||||
) -> DefenseRecord | None:
|
||||
"""Seal the defense: status → "finished", finished_at = now(UTC),
|
||||
integrity_signals stored as JSON. UNKNOWN defense_id → None
|
||||
(documented choice: the API layer maps it to 404 without an
|
||||
exception dance; contrast start/append_turn where IntegrityError
|
||||
IS the contract — those are inserts, this is an update on a key
|
||||
the caller may legitimately not hold). Re-finalize overwrites
|
||||
signals + finished_at: latest-wins, mirroring GradeStore.save
|
||||
(a recomputed verdict replaces the previous one wholesale).
|
||||
Returns the updated record, detached, WITHOUT turns — get() is
|
||||
the with-turns path.
|
||||
"""
|
||||
...
|
||||
|
||||
def get(self, defense_id: str) -> DefenseRecord | None:
|
||||
"""The defense with its FULL transcript (turns in seq order,
|
||||
detached) and integrity signals; None when it does not exist.
|
||||
Safe to pass across layers — no open-session ORM magic.
|
||||
"""
|
||||
...
|
||||
|
||||
def list_for_learner(self, learner_id: str) -> list[DefenseRecord]:
|
||||
"""All stored defenses for the learner, ordered by created_at
|
||||
ascending (chronological; id breaks same-instant ties), WITHOUT
|
||||
turns — records carry turns == []; call get() for a transcript.
|
||||
Empty list when the learner has none.
|
||||
"""
|
||||
...
|
||||
|
||||
def close(self) -> None:
|
||||
"""Release DB connections. Store must not be used after close."""
|
||||
...
|
||||
|
||||
|
||||
def _sqlite_connect(dbapi_connection: sqlite3.Connection, _: object) -> None:
|
||||
"""Per-connection pragma setup (a-3). Mirrors the other D-027 stores.
|
||||
|
||||
journal_mode=WAL — readers never block the single writer.
|
||||
synchronous=NORMAL — safe in WAL mode, avoids full fsync-per-commit.
|
||||
busy_timeout=5000 — retry briefly under contention instead of
|
||||
`OperationalError: database is locked`.
|
||||
foreign_keys=ON — NEW vs the family: defense_turn is the first
|
||||
real FK among the D-027 stores; SQLite leaves
|
||||
FKs OFF by default while Postgres enforces them
|
||||
natively, so the pragma keeps the backends
|
||||
equivalent (D-027 parity).
|
||||
"""
|
||||
cursor = dbapi_connection.cursor()
|
||||
cursor.execute("PRAGMA journal_mode=WAL")
|
||||
cursor.execute("PRAGMA synchronous=NORMAL")
|
||||
cursor.execute("PRAGMA busy_timeout=5000")
|
||||
cursor.execute("PRAGMA foreign_keys=ON")
|
||||
cursor.close()
|
||||
|
||||
|
||||
def _as_utc(ts: datetime) -> datetime:
|
||||
"""Timestamps travel as naive datetime on SQLite, tz-aware elsewhere.
|
||||
|
||||
SQLite (via SQLModel) drops tzinfo; Postgres TIMESTAMP WITH TIME ZONE
|
||||
keeps it. Normalizing on the read path makes the store's contract
|
||||
tz-aware UTC regardless of the backend (D-027).
|
||||
"""
|
||||
if ts.tzinfo is None:
|
||||
return ts.replace(tzinfo=UTC) # naive UTC read-side: label as UTC
|
||||
return ts.astimezone(UTC)
|
||||
|
||||
|
||||
class SQLiteDefenseStore:
|
||||
"""SQLite-backed DefenseStore (SQLModel). Fourth protocol-wrapped
|
||||
store of the D-027 family (first: SQLiteTraceStore, second:
|
||||
SQLiteGradeStore, third: SQLiteVariantStore) and the first spanning
|
||||
two related tables.
|
||||
"""
|
||||
|
||||
def __init__(self, db_path: Path | None = None) -> None:
|
||||
self._db_path: Path = db_path if db_path is not None else Settings().db_path
|
||||
self._engine = create_engine(f"sqlite:///{self._db_path}")
|
||||
sa.event.listen(self._engine, "connect", _sqlite_connect)
|
||||
SQLModel.metadata.create_all(self._engine)
|
||||
|
||||
@contextmanager
|
||||
def _session(self) -> Iterator[Session]:
|
||||
# expire_on_commit=False: identical session behavior to the other
|
||||
# D-027 stores. start/append_turn return the caller's instance
|
||||
# after commit and get/finalize return rows expunged mid-session;
|
||||
# a uniform flag across the family keeps their detachment
|
||||
# guarantees from diverging.
|
||||
with Session(self._engine, expire_on_commit=False) as session:
|
||||
yield session
|
||||
|
||||
def start(self, defense: DefenseRecord) -> DefenseRecord:
|
||||
# The store owns the lifecycle: a defense is BORN in_progress and
|
||||
# only finalize() may move it to finished. A smuggled "finished"
|
||||
# status is normalized away, not rejected — the insert itself
|
||||
# stays insert-only, and a duplicate id raises to the caller
|
||||
# (mirroring VariantStore: the id is minted once per session).
|
||||
defense.status = "in_progress"
|
||||
defense.finished_at = None
|
||||
with self._session() as session:
|
||||
try:
|
||||
session.add(defense)
|
||||
session.commit()
|
||||
except sa.exc.IntegrityError:
|
||||
session.rollback()
|
||||
logger.debug("defense start rejected (id already stored): %s", defense.id)
|
||||
raise
|
||||
logger.debug(
|
||||
"defense started: %s learner=%s task=%s",
|
||||
defense.id,
|
||||
defense.learner_id,
|
||||
defense.task_id,
|
||||
)
|
||||
return defense
|
||||
|
||||
def append_turn(self, defense_id: str, turn: DefenseTurn) -> DefenseTurn:
|
||||
# The explicit defense_id argument is the defense identity for
|
||||
# this write; a turn object claiming another defense is a
|
||||
# programming error — surface it before touching the DB.
|
||||
if turn.defense_id != defense_id:
|
||||
raise ValueError(
|
||||
f"turn.defense_id {turn.defense_id!r} does not match the "
|
||||
f"defense_id argument {defense_id!r}"
|
||||
)
|
||||
with self._session() as session:
|
||||
try:
|
||||
session.add(turn)
|
||||
session.commit()
|
||||
except sa.exc.IntegrityError:
|
||||
# Two possible causes, both surfaced, neither swallowed:
|
||||
# duplicate (defense_id, seq) PK — a transcript turn must
|
||||
# never silently vanish; unknown defense_id — the FK
|
||||
# (foreign_keys=ON) rejects the orphan.
|
||||
session.rollback()
|
||||
logger.debug(
|
||||
"defense turn rejected (duplicate (defense_id, seq) "
|
||||
"or unknown defense_id): defense=%s seq=%s",
|
||||
defense_id,
|
||||
turn.seq,
|
||||
)
|
||||
raise
|
||||
logger.debug(
|
||||
"defense turn appended: %s seq=%d role=%s",
|
||||
defense_id,
|
||||
turn.seq,
|
||||
turn.role,
|
||||
)
|
||||
return turn
|
||||
|
||||
def finalize(
|
||||
self, defense_id: str, integrity_signals: dict[str, Any]
|
||||
) -> DefenseRecord | None:
|
||||
# A None signals blob would break the read contract (signals are
|
||||
# a dict, {} until finalize); reject before writing.
|
||||
if not isinstance(integrity_signals, dict):
|
||||
raise ValueError(
|
||||
"integrity_signals must be a JSON-object dict, got "
|
||||
f"{type(integrity_signals).__name__}"
|
||||
)
|
||||
with self._session() as session:
|
||||
record = session.get(DefenseRecord, defense_id)
|
||||
if record is None:
|
||||
# Documented unknown-id behavior: None, not a raise — the
|
||||
# defense endpoints map this to 404. Contrast start() /
|
||||
# append_turn(), where IntegrityError IS the contract.
|
||||
return None
|
||||
# Latest-wins re-finalize, mirroring GradeStore.save: a
|
||||
# recomputed verdict (fresh signals) replaces the stored one
|
||||
# wholesale; status just stays finished.
|
||||
record.status = "finished"
|
||||
record.finished_at = datetime.now(UTC)
|
||||
record.integrity_signals = integrity_signals
|
||||
session.commit()
|
||||
record.created_at = _as_utc(record.created_at)
|
||||
if record.finished_at is not None:
|
||||
record.finished_at = _as_utc(record.finished_at)
|
||||
# Detach from the session: callers must not depend on
|
||||
# open-session ORM magic (lazy loads fail once it closes).
|
||||
session.expunge(record)
|
||||
logger.debug(
|
||||
"defense finalized: %s signals=%s", defense_id, sorted(integrity_signals)
|
||||
)
|
||||
return record
|
||||
|
||||
def get(self, defense_id: str) -> DefenseRecord | None:
|
||||
with self._session() as session:
|
||||
record = session.get(DefenseRecord, defense_id)
|
||||
if record is None:
|
||||
return None
|
||||
record.created_at = _as_utc(record.created_at)
|
||||
if record.finished_at is not None:
|
||||
record.finished_at = _as_utc(record.finished_at)
|
||||
stmt = (
|
||||
select(DefenseTurn)
|
||||
.where(DefenseTurn.defense_id == defense_id)
|
||||
.order_by(DefenseTurn.seq)
|
||||
)
|
||||
turns = session.exec(stmt).all()
|
||||
for turn in turns:
|
||||
turn.ts = _as_utc(turn.ts)
|
||||
turn.created_at = _as_utc(turn.created_at)
|
||||
# Detach each turn: the transcript must be usable once
|
||||
# the session closes (no lazy-load magic).
|
||||
session.expunge(turn)
|
||||
session.expunge(record)
|
||||
# Table models reject ad-hoc attributes (pydantic __setattr__
|
||||
# raises on non-fields), so the seq-ordered transcript is
|
||||
# stashed via object.__setattr__ and surfaced through the
|
||||
# read-only `turns` property. Rows are detached either way —
|
||||
# safe to pass across layers.
|
||||
object.__setattr__(record, "_turns", list(turns))
|
||||
return record
|
||||
|
||||
def list_for_learner(self, learner_id: str) -> list[DefenseRecord]:
|
||||
with self._session() as session:
|
||||
stmt = (
|
||||
select(DefenseRecord)
|
||||
.where(DefenseRecord.learner_id == learner_id)
|
||||
# Chronological; id is a deterministic tie-break for
|
||||
# defenses stamped within the same instant.
|
||||
.order_by(DefenseRecord.created_at, DefenseRecord.id)
|
||||
)
|
||||
results = session.exec(stmt).all()
|
||||
for row in results:
|
||||
row.created_at = _as_utc(row.created_at)
|
||||
if row.finished_at is not None:
|
||||
row.finished_at = _as_utc(row.finished_at)
|
||||
# Turns are deliberately NOT loaded here: the list feed
|
||||
# (Proctor/Mentor) needs session headers, not full
|
||||
# transcripts — get() is the with-turns path.
|
||||
session.expunge(row)
|
||||
return list(results)
|
||||
|
||||
def close(self) -> None:
|
||||
self._engine.dispose()
|
||||
@@ -0,0 +1,67 @@
|
||||
"""Voice provider factory (D-030; REQ-5-001 real path, D-040).
|
||||
|
||||
`AI_VOICE_PROVIDER = mock | browser | openai-audio` (default: mock — the
|
||||
no-key path is first-class). `openai-audio` requires voice_base_url +
|
||||
voice_api_key: the factory raises `UnknownVoiceProviderError` with an
|
||||
actionable message for direct callers (tests), while the lifespan in
|
||||
main.py CATCHES it and falls back to mock with a loud log — a typo'd env
|
||||
must never crash the unattended boot (G-11), and the mock provider's
|
||||
descriptor then honestly reports mode='mock' so the UI badge cannot lie.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import httpx
|
||||
|
||||
from ..config import Settings
|
||||
from .base import VoiceProvider
|
||||
from .mock import MockVoiceProvider
|
||||
from .openai_audio import OpenAIAudioProvider
|
||||
|
||||
|
||||
class UnknownVoiceProviderError(ValueError):
|
||||
"""Raised for a provider name outside the contract, or a real provider
|
||||
selected without its required configuration."""
|
||||
|
||||
|
||||
def voice_provider_from_settings(
|
||||
settings: Settings, http_client: httpx.AsyncClient | None = None
|
||||
) -> VoiceProvider:
|
||||
"""Select the voice provider by settings (env `AI_VOICE_PROVIDER`).
|
||||
|
||||
`http_client` is required for the `openai-audio` branch (D-017 shared
|
||||
pool); mock/browser ignore it.
|
||||
"""
|
||||
name = (settings.voice_provider or "mock").strip().lower()
|
||||
if name == "mock":
|
||||
return MockVoiceProvider()
|
||||
if name == "browser":
|
||||
# Browser mode is a CLIENT-side capability: the server composes the
|
||||
# same MockVoiceProvider (typed fallback answers still work; the UI
|
||||
# uses the descriptor for mic/speech). See browser.py.
|
||||
return MockVoiceProvider()
|
||||
if name in ("openai-audio", "openai", "server"):
|
||||
if not settings.voice_base_url or not settings.voice_api_key:
|
||||
raise UnknownVoiceProviderError(
|
||||
"AI_VOICE_PROVIDER=openai-audio requires AI_VOICE_BASE_URL "
|
||||
"and AI_VOICE_API_KEY — set both, or use 'mock'/'browser'. "
|
||||
"(main.py falls back to mock when these are missing; the "
|
||||
"voice badge then honestly reports mock — G-11)"
|
||||
)
|
||||
if http_client is None:
|
||||
raise UnknownVoiceProviderError(
|
||||
"openai-audio requires the shared httpx client "
|
||||
"(voice_provider_from_settings(settings, http_client))"
|
||||
)
|
||||
return OpenAIAudioProvider(
|
||||
http_client=http_client,
|
||||
base_url=settings.voice_base_url,
|
||||
api_key=settings.voice_api_key,
|
||||
stt_model=settings.voice_stt_model,
|
||||
tts_model=settings.voice_tts_model,
|
||||
tts_voice=settings.voice_tts_voice,
|
||||
tts_format=settings.voice_tts_format,
|
||||
)
|
||||
raise UnknownVoiceProviderError(
|
||||
f"unknown AI_VOICE_PROVIDER {name!r}: use 'mock', 'browser', or 'openai-audio'"
|
||||
)
|
||||
@@ -0,0 +1,96 @@
|
||||
"""Deterministic MockVoiceProvider (D-030, REQ-3-006).
|
||||
|
||||
Canned transcripts (scripted per test via queue) + canned 1kHz-tone WAV bytes
|
||||
+ scripted failure modes. Two identical transcribe calls yield identical
|
||||
segments; tests NEVER touch a real voice API (conftest cloud-free rule).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import io
|
||||
import math
|
||||
import struct
|
||||
import wave
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from .base import TranscriptSegment
|
||||
|
||||
|
||||
def _tone_wav(duration_ms: int = 250, freq_hz: float = 1000.0) -> bytes:
|
||||
"""A small, deterministic 16-bit mono WAV: a sine tone (stdlib only)."""
|
||||
rate = 8000
|
||||
n_samples = max(1, int(rate * duration_ms / 1000))
|
||||
buf = io.BytesIO()
|
||||
with wave.open(buf, "wb") as w:
|
||||
w.setnchannels(1)
|
||||
w.setsampwidth(2)
|
||||
w.setframerate(rate)
|
||||
for i in range(n_samples):
|
||||
sample = int(12000 * math.sin(2 * math.pi * freq_hz * i / rate))
|
||||
w.writeframes(struct.pack("<h", sample))
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
class MockVoiceFailure(RuntimeError):
|
||||
"""Scripted failure mode for tests."""
|
||||
|
||||
|
||||
class MockVoiceProvider:
|
||||
"""Deterministic voice provider: scripted STT, canned-tone TTS.
|
||||
|
||||
- `transcribe`: pops the next scripted transcript from a queue (or a
|
||||
default); two identical calls with the same queue state are identical.
|
||||
Failure mode: raise MockVoiceFailure when the queue holds a failure
|
||||
marker (the string "FAIL") or `audio` is empty.
|
||||
- `synthesize`: yields the canned tone WAV in fixed-size chunks; failure
|
||||
mode: empty text raises MockVoiceFailure.
|
||||
"""
|
||||
|
||||
def __init__(self, transcripts: list[str] | None = None) -> None:
|
||||
self._transcripts = list(transcripts or [])
|
||||
self._cursor = 0
|
||||
self.transcribe_calls = 0
|
||||
self.synthesize_calls = 0
|
||||
|
||||
def script(self, transcripts: list[str]) -> None:
|
||||
"""Replace the scripted queue (tests set expectations up front)."""
|
||||
self._transcripts = list(transcripts)
|
||||
self._cursor = 0
|
||||
|
||||
async def transcribe(self, audio: bytes, fmt: str) -> TranscriptSegment:
|
||||
self.transcribe_calls += 1
|
||||
if not audio:
|
||||
raise MockVoiceFailure("no audio bytes provided")
|
||||
if not self._transcripts:
|
||||
raise MockVoiceFailure("transcript queue exhausted — script it")
|
||||
item = self._transcripts[self._cursor]
|
||||
self._cursor = (self._cursor + 1) % len(self._transcripts)
|
||||
if item == "FAIL":
|
||||
raise MockVoiceFailure("scripted STT failure")
|
||||
return TranscriptSegment(
|
||||
text=item,
|
||||
duration_ms=max(1, len(audio) // 32), # deterministic pseudo-duration
|
||||
)
|
||||
|
||||
async def synthesize(self, text: str, voice: str = "default") -> AsyncIterator[bytes]: # noqa: ASYNC109 (protocol parity)
|
||||
# NOTE: protocol parity matters more than the async-generator purity
|
||||
# lint; the real provider (openai_audio.py) streams over HTTP.
|
||||
self.synthesize_calls += 1
|
||||
if not text:
|
||||
raise MockVoiceFailure("cannot synthesize empty text")
|
||||
wav = _tone_wav(duration_ms=min(2000, max(120, len(text) * 12)))
|
||||
for i in range(0, len(wav), 1024):
|
||||
yield wav[i : i + 1024]
|
||||
await asyncio.sleep(0) # yield to the loop like a network stream
|
||||
|
||||
|
||||
# Protocol-shape parity guard (mock must satisfy the D-030 port).
|
||||
from .base import VoiceProvider # noqa: E402
|
||||
|
||||
|
||||
def _assert_protocol() -> None:
|
||||
assert isinstance(MockVoiceProvider(), VoiceProvider)
|
||||
|
||||
|
||||
_assert_protocol()
|
||||
@@ -0,0 +1,133 @@
|
||||
"""OpenAI-compatible audio provider — real server STT/TTS (D-040, REQ-5-001).
|
||||
|
||||
One implementation serves any OpenAI-compatible audio endpoint (base_url is
|
||||
config; A-301 endpoint-agnostic by config, D-014 pattern). Raw httpx on the
|
||||
shared lifespan client (D-017; read=300s tolerates multi-minute clips).
|
||||
|
||||
Boundary rules (mirror llm/openai_compat.py):
|
||||
- voice/ imports nothing from agents/ or api/
|
||||
- api_key NEVER appears in exceptions, logs, or error messages
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Literal
|
||||
|
||||
import httpx
|
||||
|
||||
from .base import TranscriptSegment, VoiceDescriptor
|
||||
|
||||
|
||||
class OpenAIAudioProvider:
|
||||
"""Server STT (`/audio/transcriptions`) + TTS (`/audio/speech`)."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
http_client: httpx.AsyncClient,
|
||||
base_url: str,
|
||||
api_key: str,
|
||||
stt_model: str = "whisper-1",
|
||||
tts_model: str = "tts-1",
|
||||
tts_voice: str = "alloy",
|
||||
tts_format: Literal["mp3", "wav", "opus"] = "mp3",
|
||||
) -> None:
|
||||
self._client = http_client
|
||||
self._base_url = base_url.rstrip("/")
|
||||
self._api_key = api_key
|
||||
self._stt_model = stt_model
|
||||
self._tts_model = tts_model
|
||||
self._tts_voice = tts_voice
|
||||
self._tts_format = tts_format
|
||||
# a-15: the descriptor is what defense.py prefers; a missing one
|
||||
# would badge the real server path as "mock".
|
||||
self.descriptor = VoiceDescriptor(
|
||||
mode="server",
|
||||
sr_available=True,
|
||||
tts_available=True,
|
||||
hint="server STT/TTS via AI_VOICE_BASE_URL",
|
||||
)
|
||||
|
||||
def _headers(self) -> dict[str, str]:
|
||||
headers: dict[str, str] = {}
|
||||
if self._api_key:
|
||||
headers["Authorization"] = f"Bearer {self._api_key}"
|
||||
return headers
|
||||
|
||||
def _sanitize(self, exc: Exception) -> RuntimeError:
|
||||
text = str(exc)
|
||||
if self._api_key and self._api_key in text:
|
||||
text = text.replace(self._api_key, "[REDACTED]")
|
||||
return RuntimeError(f"voice provider error: {text}")
|
||||
|
||||
async def transcribe(self, audio: bytes, fmt: str) -> TranscriptSegment:
|
||||
"""STT: multipart upload (`file` + `model`) → TranscriptSegment.
|
||||
|
||||
`fmt` is a bare extension ('wav' | 'webm' | 'mp3') — the defense
|
||||
route strips codec params before this call (D-041).
|
||||
"""
|
||||
files = {"file": (f"answer.{fmt}", audio, f"audio/{fmt}")}
|
||||
data = {"model": self._stt_model, "response_format": "json"}
|
||||
try:
|
||||
resp = await self._client.post(
|
||||
f"{self._base_url}/audio/transcriptions",
|
||||
files=files,
|
||||
data=data,
|
||||
headers=self._headers(),
|
||||
)
|
||||
resp.raise_for_status()
|
||||
body = resp.json()
|
||||
except httpx.HTTPError as exc:
|
||||
raise self._sanitize(exc) from exc
|
||||
if not isinstance(body, dict):
|
||||
# P2 (verifier): a non-dict 200 body is a contract break —
|
||||
# context-wrap instead of a raw AttributeError.
|
||||
raise RuntimeError(
|
||||
"voice provider error: unexpected transcription response shape"
|
||||
)
|
||||
text = str(body.get("text", "")).strip()
|
||||
if not text:
|
||||
# 200 with an empty transcript is a provider contract break —
|
||||
# TranscriptSegment(min_length=1) would raise a bare pydantic
|
||||
# error; wrap it with provider context instead.
|
||||
raise RuntimeError("voice provider error: empty transcription")
|
||||
return TranscriptSegment(text=text)
|
||||
|
||||
def synthesize(
|
||||
self, text: str, voice: str = "default"
|
||||
) -> AsyncIterator[bytes]:
|
||||
"""TTS: JSON body → raw audio byte stream.
|
||||
|
||||
OpenAI's TTS caps `input` at 4096 chars; examiner questions are
|
||||
short, but enforce the guard so a long question fails loudly at the
|
||||
seam instead of as an opaque provider 400.
|
||||
"""
|
||||
return self._synthesize_stream(text, voice)
|
||||
|
||||
async def _synthesize_stream(
|
||||
self, text: str, voice: str
|
||||
) -> AsyncIterator[bytes]:
|
||||
if len(text) > 4096:
|
||||
raise RuntimeError(
|
||||
f"voice provider error: TTS input exceeds 4096 chars ({len(text)})"
|
||||
)
|
||||
payload = {
|
||||
"model": self._tts_model,
|
||||
"input": text,
|
||||
"voice": voice if voice != "default" else self._tts_voice,
|
||||
"response_format": self._tts_format,
|
||||
}
|
||||
try:
|
||||
async with self._client.stream(
|
||||
"POST",
|
||||
f"{self._base_url}/audio/speech",
|
||||
content=json.dumps(payload),
|
||||
headers={**self._headers(), "Content-Type": "application/json"},
|
||||
) as resp:
|
||||
resp.raise_for_status()
|
||||
async for chunk in resp.aiter_bytes():
|
||||
if chunk:
|
||||
yield chunk
|
||||
except httpx.HTTPError as exc:
|
||||
raise self._sanitize(exc) from exc
|
||||
@@ -18,6 +18,9 @@ dependencies = [
|
||||
"sqlalchemy>=2.0,<2.1",
|
||||
"websockets>=13,<16",
|
||||
"aiofiles>=24.1,<26",
|
||||
# POST /v1/defense/{id}/answer multipart audio (REQ-3-006): FastAPI
|
||||
# form/File parsing requires python-multipart at runtime.
|
||||
"python-multipart>=0.0.32,<0.1",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
|
||||
@@ -1,28 +1,73 @@
|
||||
#!/usr/bin/env bash
|
||||
# Idempotent bootstrap: create venv + install deps.
|
||||
# Handles Debian systems without python3-venv/ensurepip via --without-pip + get-pip.
|
||||
# Handles Debian/Ubuntu systems without python3-venv/ensurepip via --without-pip + get-pip.
|
||||
# v2 (v0.3.5): recovers from a poisoned partial .venv left by a failed earlier
|
||||
# attempt, cleans before each retry, and dies with a distro-specific fix hint
|
||||
# when venv creation is impossible (e.g. missing python3.XX-venv package).
|
||||
set -euo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
APP_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
VENV="$APP_DIR/.venv"
|
||||
|
||||
venv_usable() {
|
||||
[ -x "$VENV/bin/python3" ]
|
||||
}
|
||||
|
||||
rm_broken_venv() {
|
||||
echo "bootstrap: removing broken partial .venv from a failed earlier attempt" >&2
|
||||
rm -rf "$VENV"
|
||||
}
|
||||
|
||||
mkdir -p "$HOME/.cache/ciagent"
|
||||
|
||||
if [ ! -x "$VENV/bin/python3" ]; then
|
||||
if python3 -m venv "$VENV" 2>/dev/null; then
|
||||
if venv_usable && [ ! -x "$VENV/bin/pip" ]; then
|
||||
# A usable python3 without pip means the --without-pip fallback half-ran and
|
||||
# the get-pip step never completed: start over cleanly.
|
||||
rm_broken_venv
|
||||
fi
|
||||
|
||||
if ! venv_usable; then
|
||||
if [ -d "$VENV" ]; then
|
||||
# Directory exists but no working python3: remains of a crashed venv create.
|
||||
rm_broken_venv
|
||||
fi
|
||||
if python3 -m venv "$VENV" 2>/tmp/venv-create.err; then
|
||||
:
|
||||
else
|
||||
# No ensurepip available — create bare venv and bootstrap pip separately.
|
||||
python3 -m venv --without-pip "$VENV"
|
||||
rm -rf "$VENV"
|
||||
if python3 -m venv --without-pip "$VENV" 2>>/tmp/venv-create.err; then
|
||||
:
|
||||
else
|
||||
rm -rf "$VENV"
|
||||
PYVER="$(python3 -c 'import sys; print("%d.%d" % sys.version_info[:2])' 2>/dev/null || true)"
|
||||
PKG="python3-venv"
|
||||
[ -n "$PYVER" ] && PKG="python${PYVER}-venv"
|
||||
echo "bootstrap: could not create a virtual environment." >&2
|
||||
echo " python3 reported:" >&2
|
||||
sed 's/^/ /' /tmp/venv-create.err >&2 || true
|
||||
echo " fix (Debian/Ubuntu): install the venv support package, then re-run nextcraft bootstrap:" >&2
|
||||
echo " apt install $PKG" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ ! -x "$VENV/bin/pip" ]; then
|
||||
GET_PIP="$HOME/.cache/ciagent/get-pip.py"
|
||||
if [ ! -f "$GET_PIP" ]; then
|
||||
curl -sSf --max-time 60 https://bootstrap.pypa.io/get-pip.py -o "$GET_PIP"
|
||||
if ! curl -sSf --max-time 60 https://bootstrap.pypa.io/get-pip.py -o "$GET_PIP"; then
|
||||
rm -rf "$VENV"
|
||||
echo "bootstrap: get-pip.py download failed (no network?)." >&2
|
||||
echo " fix: restore network access and re-run nextcraft bootstrap" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
if ! "$VENV/bin/python3" "$GET_PIP" --quiet; then
|
||||
rm -rf "$VENV"
|
||||
echo "bootstrap: pip installation into the venv failed." >&2
|
||||
echo " fix: re-run nextcraft bootstrap (the venv was cleaned; this retry is safe)" >&2
|
||||
exit 1
|
||||
fi
|
||||
"$VENV/bin/python3" "$GET_PIP" --quiet
|
||||
fi
|
||||
|
||||
"$VENV/bin/pip" install --quiet --upgrade pip
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
# Dev server: export secrets (if present) then run uvicorn on :8420.
|
||||
# Dev server: export secrets (if present) then run uvicorn.
|
||||
# Binds 0.0.0.0 by default so the stack is reachable from other machines
|
||||
# (v0.3.5 network mode) — set AI_HOST=127.0.0.1 in .env to revert to loopback.
|
||||
set -euo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
APP_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
@@ -7,7 +9,7 @@ REPO_ROOT="$(cd "$APP_DIR/../.." && pwd)"
|
||||
VENV="$APP_DIR/.venv"
|
||||
|
||||
if [ ! -x "$VENV/bin/uvicorn" ]; then
|
||||
echo "venv missing — run scripts/bootstrap.sh first" >&2
|
||||
echo "venv missing — run nextcraft bootstrap first (or: bash scripts/bootstrap.sh)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -22,5 +24,17 @@ if [ -f "$SECRETS" ]; then
|
||||
done < "$SECRETS"
|
||||
fi
|
||||
|
||||
ENV_FILE="$APP_DIR/.env"
|
||||
if [ -f "$ENV_FILE" ]; then
|
||||
while IFS='=' read -r key value; do
|
||||
case "$key" in
|
||||
AI_HOST|AI_PORT|AI_CORS_ORIGINS) export "$key=$value" ;;
|
||||
esac
|
||||
done < "$ENV_FILE"
|
||||
fi
|
||||
|
||||
HOST="${AI_HOST:-0.0.0.0}"
|
||||
PORT="${AI_PORT:-8420}"
|
||||
|
||||
cd "$APP_DIR"
|
||||
exec "$VENV/bin/uvicorn" ai_service.main:app --reload --port 8420
|
||||
exec "$VENV/bin/uvicorn" ai_service.main:app --reload --host "$HOST" --port "$PORT"
|
||||
@@ -0,0 +1,803 @@
|
||||
#!/usr/bin/env python3
|
||||
"""sandbox-agent — stdlib-only in-sandbox telemetry capture agent (REQ-3-003).
|
||||
|
||||
D-031: this file is copied into the sandbox namespace and runs against the
|
||||
system Python — no third-party packages are importable there, so this module
|
||||
depends on the standard library ONLY (the test suite enforces this with an
|
||||
AST scan of the file).
|
||||
|
||||
What it does:
|
||||
* wraps a non-interactive `/bin/sh` REPL: each stdin line is executed via
|
||||
`sh -c` inside the workspace and reported as `stdin` -> `command` ->
|
||||
`stdout` -> `run_result`/`test_result` events;
|
||||
* polls the workspace tree (~250 ms) and emits `file_diff` events
|
||||
(created/modified/deleted with unified diffs) plus periodic `activity`
|
||||
heartbeats;
|
||||
* streams events to ai-service as TelemetryEvent-shaped JSON frames over a
|
||||
raw-socket RFC 6455 WebSocket client (no `websockets` package exists in
|
||||
the namespace — the client handshake + frame codec is implemented here);
|
||||
* at-least-once delivery (D-026): every event is appended to an fsync'd
|
||||
JSONL spool file inside the workdir BEFORE any send attempt; on
|
||||
disconnect the spool grows; after reconnect (exponential backoff) the
|
||||
spool is flushed oldest-first. The server dedups on (learner, task, seq)
|
||||
so replayed duplicates are harmless — loss is not tolerated.
|
||||
|
||||
Configured entirely through env baked at spawn time:
|
||||
NC_LEARNER_ID / NC_TASK_ID / NC_INGEST_URL / NC_SANDBOX_ID (required)
|
||||
NC_WORKSPACE workspace root to watch/run in (default: cwd)
|
||||
NC_SPOOL spool path (default: <workspace>/.nc-agent/spool.jsonl)
|
||||
NC_POLL_INTERVAL_S / NC_ACTIVITY_INTERVAL_S / NC_COMMAND_TIMEOUT_S
|
||||
NC_BACKOFF_BASE_S / NC_BACKOFF_MAX_S (optional knobs)
|
||||
|
||||
Sequencing survives process restarts (incl. SIGKILL): on boot the spool is
|
||||
replayed into the pending queue and `seq` resumes at max(spooled seq) + 1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import difflib
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import secrets
|
||||
import socket
|
||||
import ssl
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import urllib.parse
|
||||
from collections import deque
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
_WS_GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
|
||||
_EVENT_KINDS = frozenset(
|
||||
{"command", "file_diff", "run_result", "test_result", "activity", "stdin", "stdout"}
|
||||
)
|
||||
_AGENT_DIR_PREFIX = ".nc-" # agent-private paths (spool) are excluded from watching
|
||||
_MAX_DIFF_BYTES = 64 * 1024 # files larger than this are reported truncated, no diff
|
||||
_MAX_OUTPUT_CHARS = 64 * 1024 # captured stdout/stderr tail cap per command
|
||||
_HANDSHAKE_MAX_BYTES = 64 * 1024
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- config
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AgentConfig:
|
||||
"""Runtime configuration, normally built from `NC_*` env baked at spawn."""
|
||||
|
||||
learner_id: str
|
||||
task_id: str
|
||||
ingest_url: str
|
||||
sandbox_id: str
|
||||
workspace: Path
|
||||
spool_path: Path
|
||||
poll_interval_s: float = 0.25
|
||||
activity_interval_s: float = 5.0
|
||||
command_timeout_s: float = 30.0
|
||||
backoff_base_s: float = 0.25
|
||||
backoff_max_s: float = 8.0
|
||||
#: Spool bound (G-14): explicit cap where none existed. Worst case
|
||||
#: ~64KB/line (diff cap) * SPOOL_MAX_LINES must stay well under the
|
||||
#: G-2 512MB workdir sweep: 4096 * 64KB = 256MB (half the budget).
|
||||
spool_max_lines: int = 4096
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
for name in ("learner_id", "task_id", "ingest_url", "sandbox_id"):
|
||||
if not getattr(self, name):
|
||||
raise ValueError(f"missing required config: NC_{name.upper()}")
|
||||
|
||||
@classmethod
|
||||
def from_env(cls, env: Mapping[str, str] | None = None) -> AgentConfig:
|
||||
src = os.environ if env is None else env
|
||||
workspace = Path(src.get("NC_WORKSPACE") or os.getcwd()).resolve()
|
||||
return cls(
|
||||
learner_id=src.get("NC_LEARNER_ID", ""),
|
||||
task_id=src.get("NC_TASK_ID", ""),
|
||||
ingest_url=src.get("NC_INGEST_URL", ""),
|
||||
sandbox_id=src.get("NC_SANDBOX_ID", ""),
|
||||
workspace=workspace,
|
||||
spool_path=Path(
|
||||
src.get("NC_SPOOL") or (workspace / ".nc-agent" / "spool.jsonl")
|
||||
),
|
||||
poll_interval_s=float(src.get("NC_POLL_INTERVAL_S", "0.25")),
|
||||
activity_interval_s=float(src.get("NC_ACTIVITY_INTERVAL_S", "5.0")),
|
||||
command_timeout_s=float(src.get("NC_COMMAND_TIMEOUT_S", "30.0")),
|
||||
backoff_base_s=float(src.get("NC_BACKOFF_BASE_S", "0.25")),
|
||||
backoff_max_s=float(src.get("NC_BACKOFF_MAX_S", "8.0")),
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- spool
|
||||
|
||||
|
||||
class Spool:
|
||||
"""Append-only JSONL spool with per-append fsync (survives SIGKILL).
|
||||
|
||||
`rewrite` swaps in a compacted file atomically (tmp file + os.replace).
|
||||
Lines are stored without trailing newlines in memory, one per line on disk.
|
||||
"""
|
||||
|
||||
def __init__(self, path: Path) -> None:
|
||||
self._path = path
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@property
|
||||
def path(self) -> Path:
|
||||
return self._path
|
||||
|
||||
def append(self, line: str) -> None:
|
||||
with self._path.open("a", encoding="utf-8") as fh:
|
||||
fh.write(line + "\n")
|
||||
fh.flush()
|
||||
os.fsync(fh.fileno())
|
||||
|
||||
def read_all(self) -> list[str]:
|
||||
if not self._path.exists():
|
||||
return []
|
||||
with self._path.open("r", encoding="utf-8") as fh:
|
||||
return [line.rstrip("\n") for line in fh if line.strip()]
|
||||
|
||||
def rewrite(self, lines: list[str]) -> None:
|
||||
tmp = self._path.with_name(self._path.name + ".tmp")
|
||||
with tmp.open("w", encoding="utf-8") as fh:
|
||||
for line in lines:
|
||||
fh.write(line + "\n")
|
||||
fh.flush()
|
||||
os.fsync(fh.fileno())
|
||||
os.replace(tmp, self._path)
|
||||
|
||||
|
||||
def _line_seq(line: str) -> int | None:
|
||||
"""Best-effort seq extraction from a spool line (None when unparseable).
|
||||
|
||||
The event's seq is a top-level wire field (`_next_event`). Used only for
|
||||
ack trimming; an unparseable line is retained (never dropped by the ack
|
||||
path — the overflow bound is the only dropper).
|
||||
"""
|
||||
try:
|
||||
seq = json.loads(line).get("seq")
|
||||
return seq if isinstance(seq, int) else None
|
||||
except (ValueError, AttributeError):
|
||||
return None
|
||||
|
||||
|
||||
# --------------------------------------------------------------- websocket codec
|
||||
|
||||
|
||||
def _encode_frame(opcode: int, payload: bytes) -> bytes:
|
||||
"""RFC 6455 client frame: FIN set, always masked (servers require it)."""
|
||||
header = bytearray([0x80 | opcode])
|
||||
n = len(payload)
|
||||
if n < 126:
|
||||
header.append(0x80 | n)
|
||||
elif n < 65536:
|
||||
header.append(0x80 | 126)
|
||||
header += struct.pack("!H", n)
|
||||
else:
|
||||
header.append(0x80 | 127)
|
||||
header += struct.pack("!Q", n)
|
||||
mask = secrets.token_bytes(4)
|
||||
header += mask
|
||||
masked = bytes(b ^ mask[i % 4] for i, b in enumerate(payload))
|
||||
return bytes(header) + masked
|
||||
|
||||
|
||||
class WsConnection:
|
||||
"""Minimal blocking RFC 6455 client over a raw socket (stdlib only)."""
|
||||
|
||||
def __init__(self, sock: socket.socket) -> None:
|
||||
self._sock = sock
|
||||
self._write_lock = threading.Lock()
|
||||
|
||||
@classmethod
|
||||
def connect(cls, url: str, timeout_s: float = 5.0) -> WsConnection:
|
||||
parts = urllib.parse.urlsplit(url)
|
||||
if parts.scheme not in ("ws", "wss"):
|
||||
raise ValueError(f"unsupported scheme in NC_INGEST_URL: {parts.scheme!r}")
|
||||
host = parts.hostname or "localhost"
|
||||
port = parts.port or (443 if parts.scheme == "wss" else 80)
|
||||
path = parts.path or "/"
|
||||
if parts.query:
|
||||
path += "?" + parts.query
|
||||
|
||||
sock = socket.create_connection((host, port), timeout=timeout_s)
|
||||
if parts.scheme == "wss":
|
||||
sock = ssl.create_default_context().wrap_socket(sock, server_hostname=host)
|
||||
|
||||
key = base64.b64encode(secrets.token_bytes(16)).decode("ascii")
|
||||
request = (
|
||||
f"GET {path} HTTP/1.1\r\n"
|
||||
f"Host: {host}:{port}\r\n"
|
||||
"Upgrade: websocket\r\n"
|
||||
"Connection: Upgrade\r\n"
|
||||
f"Sec-WebSocket-Key: {key}\r\n"
|
||||
"Sec-WebSocket-Version: 13\r\n\r\n"
|
||||
)
|
||||
sock.sendall(request.encode("ascii"))
|
||||
response = cls._read_http_response(sock)
|
||||
cls._validate_handshake(response, key)
|
||||
return cls(sock)
|
||||
|
||||
@staticmethod
|
||||
def _read_http_response(sock: socket.socket) -> bytes:
|
||||
buf = b""
|
||||
while b"\r\n\r\n" not in buf:
|
||||
chunk = sock.recv(4096)
|
||||
if not chunk:
|
||||
raise ConnectionError("server closed during WebSocket handshake")
|
||||
buf += chunk
|
||||
if len(buf) > _HANDSHAKE_MAX_BYTES:
|
||||
raise ConnectionError("handshake response exceeded size cap")
|
||||
return buf.split(b"\r\n\r\n", 1)[0]
|
||||
|
||||
@staticmethod
|
||||
def _validate_handshake(response: bytes, key: str) -> None:
|
||||
head = response.decode("latin-1")
|
||||
lines = head.split("\r\n")
|
||||
if not lines or " 101" not in lines[0]:
|
||||
raise ConnectionError(f"handshake rejected: {lines[0] if lines else '<empty>'}")
|
||||
headers = {}
|
||||
for line in lines[1:]:
|
||||
if ":" in line:
|
||||
name, _, value = line.partition(":")
|
||||
headers[name.strip().lower()] = value.strip()
|
||||
expect = base64.b64encode(
|
||||
hashlib.sha1((key + _WS_GUID).encode("ascii")).digest()
|
||||
).decode("ascii")
|
||||
if headers.get("sec-websocket-accept") != expect:
|
||||
raise ConnectionError("bad Sec-WebSocket-Accept in handshake response")
|
||||
|
||||
# -- send ------------------------------------------------------------
|
||||
def send_text(self, text: str) -> None:
|
||||
with self._write_lock:
|
||||
self._sock.sendall(_encode_frame(0x1, text.encode("utf-8")))
|
||||
|
||||
def _send_frame(self, opcode: int, payload: bytes) -> None:
|
||||
with self._write_lock:
|
||||
self._sock.sendall(_encode_frame(opcode, payload))
|
||||
|
||||
# -- receive ---------------------------------------------------------
|
||||
def recv_message(self, timeout_s: float) -> tuple[int, bytes] | None:
|
||||
"""Return (opcode, payload) for a data/close frame, or None on timeout.
|
||||
|
||||
Ping frames are answered with pong internally and never surfaced;
|
||||
pongs are swallowed. Fragmented messages are reassembled. Raises
|
||||
ConnectionError/OSError when the socket breaks.
|
||||
"""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
fragments = bytearray()
|
||||
frag_opcode = 0
|
||||
while True:
|
||||
frame = self._recv_one_frame(deadline)
|
||||
if frame is None:
|
||||
return None
|
||||
fin, opcode, payload = frame
|
||||
if opcode == 0x9: # ping
|
||||
self._send_frame(0xA, payload)
|
||||
continue
|
||||
if opcode == 0xA: # pong
|
||||
continue
|
||||
if opcode == 0x0: # continuation
|
||||
fragments += payload
|
||||
else:
|
||||
fragments = bytearray(payload)
|
||||
frag_opcode = opcode
|
||||
if fin:
|
||||
return frag_opcode, bytes(fragments)
|
||||
|
||||
def _recv_one_frame(self, deadline: float) -> tuple[bool, int, bytes] | None:
|
||||
header = self._read_exact(2, deadline)
|
||||
if header is None:
|
||||
return None
|
||||
b0, b1 = header[0], header[1]
|
||||
fin = bool(b0 & 0x80)
|
||||
opcode = b0 & 0x0F
|
||||
length = b1 & 0x7F
|
||||
if length == 126:
|
||||
ext = self._read_exact(2, deadline)
|
||||
if ext is None:
|
||||
return None
|
||||
length = struct.unpack("!H", ext)[0]
|
||||
elif length == 127:
|
||||
ext = self._read_exact(8, deadline)
|
||||
if ext is None:
|
||||
return None
|
||||
length = struct.unpack("!Q", ext)[0]
|
||||
mask = self._read_exact(4, deadline) if (b1 & 0x80) else b""
|
||||
if mask is None:
|
||||
return None
|
||||
payload = self._read_exact(length, deadline) if length else b""
|
||||
if payload is None:
|
||||
return None
|
||||
if mask:
|
||||
payload = bytes(b ^ mask[i % 4] for i, b in enumerate(payload))
|
||||
return fin, opcode, payload
|
||||
|
||||
def _read_exact(self, n: int, deadline: float) -> bytes | None:
|
||||
buf = bytearray()
|
||||
while len(buf) < n:
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
return None
|
||||
self._sock.settimeout(remaining)
|
||||
try:
|
||||
chunk = self._sock.recv(n - len(buf))
|
||||
except TimeoutError:
|
||||
return None
|
||||
if not chunk:
|
||||
raise ConnectionError("peer closed the WebSocket connection")
|
||||
buf += chunk
|
||||
return bytes(buf)
|
||||
|
||||
def close(self) -> None:
|
||||
try:
|
||||
self._sock.close()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- agent
|
||||
|
||||
|
||||
class Agent:
|
||||
"""Wires capture (shell + workspace watcher) to the framed event stream."""
|
||||
|
||||
def __init__(self, config: AgentConfig) -> None:
|
||||
self.config = config
|
||||
self._spool = Spool(config.spool_path)
|
||||
self._pending: deque[str] = deque()
|
||||
self._seq = 0
|
||||
# RLock (D-3 verifier fix): emit() holds this across sends; a send
|
||||
# failure drops the conn, and _drop_conn -> replay_margin re-enters
|
||||
# the same lock. A plain Lock deadlocked the emitting thread there.
|
||||
self._emit_lock = threading.RLock() # serializes seq + spool + flush
|
||||
self._conn_lock = threading.Lock() # guards _conn swaps
|
||||
self._conn: WsConnection | None = None
|
||||
self._last_sent: str | None = None # one-line replay margin, see below
|
||||
self._stop = threading.Event()
|
||||
self._threads: list[threading.Thread] = []
|
||||
self._baseline: dict[str, tuple[int, int, str | None]] = {}
|
||||
# Spool bound (G-14): explicit cap where none existed. Worst case
|
||||
# ~64KB/line (diff cap) x SPOOL_MAX_LINES must stay well under the
|
||||
# G-2 512MB workdir sweep; overflow drops OLDEST lines with a
|
||||
# counter — by-design gap creation, so the trace goes ungradable
|
||||
# (G-4) instead of silently truncated-but-gradable.
|
||||
self._dropped_overflow = 0
|
||||
self._enforce_spool_bound_locked()
|
||||
self._resume_from_spool()
|
||||
|
||||
# -- durability ------------------------------------------------------
|
||||
def _resume_from_spool(self) -> None:
|
||||
highest = -1
|
||||
for line in self._spool.read_all():
|
||||
self._pending.append(line)
|
||||
try:
|
||||
seq = int(json.loads(line).get("seq", -1))
|
||||
except (ValueError, AttributeError):
|
||||
continue
|
||||
highest = max(highest, seq)
|
||||
self._seq = highest + 1
|
||||
|
||||
# -- event construction ---------------------------------------------
|
||||
def _wire_frame(self, spooled_line: str) -> str:
|
||||
"""Spool format -> wire format: strip URL-owned identity fields.
|
||||
|
||||
The spool keeps full events (local durability + restart recovery).
|
||||
The ingest endpoint binds identity at the WS handshake (query params)
|
||||
and rejects frames carrying learner_id/task_id (`extra="forbid"`
|
||||
anti-spoofing), so the wire frame carries only seq/kind/payload/ts.
|
||||
"""
|
||||
import json as _json
|
||||
|
||||
full = _json.loads(spooled_line)
|
||||
wire = {
|
||||
k: full[k]
|
||||
for k in ("seq", "kind", "payload", "ts")
|
||||
}
|
||||
if full.get("sandbox_id"):
|
||||
wire["sandbox_id"] = full["sandbox_id"]
|
||||
return _json.dumps(wire)
|
||||
|
||||
def _next_event(self, kind: str, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
if kind not in _EVENT_KINDS:
|
||||
raise ValueError(f"unknown event kind: {kind!r}")
|
||||
event = {
|
||||
"learner_id": self.config.learner_id,
|
||||
"task_id": self.config.task_id,
|
||||
"seq": self._seq,
|
||||
"kind": kind,
|
||||
"payload": payload,
|
||||
"ts": datetime.now(UTC).isoformat(),
|
||||
"sandbox_id": self.config.sandbox_id,
|
||||
}
|
||||
self._seq += 1
|
||||
return event
|
||||
|
||||
# -- emission / flush (D-026) ----------------------------------------
|
||||
def emit(self, kind: str, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Spool-then-send. Never blocks on reconnect; loss is impossible."""
|
||||
with self._emit_lock:
|
||||
line = json.dumps(self._next_event(kind, payload))
|
||||
self._spool.append(line) # durable BEFORE any send attempt
|
||||
self._pending.append(line)
|
||||
self._enforce_spool_bound_locked()
|
||||
self._flush_locked()
|
||||
return json.loads(line)
|
||||
|
||||
def _enforce_spool_bound_locked(self) -> None:
|
||||
"""Drop OLDEST spooled lines past `spool_max_lines` (G-14).
|
||||
|
||||
Overflow is by-design gap creation: the dropped seqs become
|
||||
permanent gaps server-side, the gap path flags the trace, and the
|
||||
grader refuses it (G-4) — never a silently-truncated-but-gradable
|
||||
trace. Caller must hold `_emit_lock`. `_dropped_overflow` is the
|
||||
observable counter (surfaced in the final stop status event).
|
||||
"""
|
||||
lines = self._spool.read_all()
|
||||
overflow = len(lines) - self.config.spool_max_lines
|
||||
if overflow <= 0:
|
||||
return
|
||||
self._dropped_overflow += overflow
|
||||
self._spool.rewrite(lines[overflow:])
|
||||
# Pending may reference dropped lines; they replay as no-ops (server
|
||||
# dedup) but trimming them keeps the replay window honest.
|
||||
dropped = set(lines[:overflow])
|
||||
self._pending = deque(ln for ln in self._pending if ln not in dropped)
|
||||
|
||||
def _flush_locked(self) -> None:
|
||||
conn = self._current_conn()
|
||||
while self._pending and conn is not None:
|
||||
line = self._pending[0]
|
||||
try:
|
||||
conn.send_text(self._wire_frame(line))
|
||||
except (ConnectionError, OSError):
|
||||
self._drop_conn()
|
||||
return
|
||||
self._pending.popleft()
|
||||
self._last_sent = line
|
||||
# D-045/D-1 (verifier fix): the spool is NEVER compacted below the
|
||||
# unacked window. A send into a silently-dead socket "succeeds" at
|
||||
# TCP level — those lines may be lost in flight — so they stay in
|
||||
# the spool until the server's seq_ack proves durable storage
|
||||
# (trim_to_ack is the ONLY spool shrinker besides the overflow
|
||||
# bound). Replays are harmless: the server dedups on
|
||||
# (learner, task, seq).
|
||||
|
||||
def replay_margin(self) -> None:
|
||||
"""Requeue every unacked spooled line after a detected disconnect.
|
||||
|
||||
D-045/D-1 (verifier fix): the pre-ack one-line margin could not cover
|
||||
a multi-frame in-flight window — a burst accepted by a dying socket
|
||||
popped N lines from pending while the spool had been compacted to the
|
||||
last one, permanently losing lines 1..N-1. The spool now retains
|
||||
everything unacked, so replay requeues the full unacked window;
|
||||
server-side dedup absorbs the duplicates.
|
||||
"""
|
||||
with self._emit_lock:
|
||||
if self._pending:
|
||||
return # mid-flush caller holds the pending queue intact
|
||||
self._pending = deque(self._spool.read_all())
|
||||
self._last_sent = None
|
||||
|
||||
def trim_to_ack(self, ack_seq: int) -> None:
|
||||
"""Drop every spooled/pending line with seq <= ack_seq (D-045).
|
||||
|
||||
Advisory server hint: `seq_ack` carries the durable latest_seq, so
|
||||
everything up to and including it is stored server-side and dedup
|
||||
absorbs nothing on replay. Runs under `_emit_lock` — ordered against
|
||||
concurrent emit()/flush, and the rewrite is atomic (Spool.rewrite).
|
||||
Lines without a parseable seq are retained; the overflow bound is
|
||||
the only dropper of unparseable lines.
|
||||
"""
|
||||
with self._emit_lock:
|
||||
spooled = self._spool.read_all()
|
||||
kept_pending = [
|
||||
ln for ln in self._pending if (s := _line_seq(ln)) is None or s > ack_seq
|
||||
]
|
||||
kept_spooled = [
|
||||
ln for ln in spooled if (s := _line_seq(ln)) is None or s > ack_seq
|
||||
]
|
||||
if len(kept_pending) != len(self._pending) or len(kept_spooled) != len(spooled):
|
||||
self._pending = deque(kept_pending)
|
||||
self._spool.rewrite(kept_spooled)
|
||||
if self._last_sent is not None:
|
||||
s = _line_seq(self._last_sent)
|
||||
if s is not None and s <= ack_seq:
|
||||
self._last_sent = None
|
||||
|
||||
# -- connection supervision ------------------------------------------
|
||||
def _current_conn(self) -> WsConnection | None:
|
||||
with self._conn_lock:
|
||||
return self._conn
|
||||
|
||||
def _set_conn(self, conn: WsConnection | None) -> None:
|
||||
with self._conn_lock:
|
||||
self._conn = conn
|
||||
|
||||
def _drop_conn(self) -> None:
|
||||
conn = self._current_conn()
|
||||
self._set_conn(None)
|
||||
if conn is not None:
|
||||
conn.close()
|
||||
self.replay_margin()
|
||||
|
||||
def is_connected(self) -> bool:
|
||||
return self._current_conn() is not None
|
||||
|
||||
def wait_connected(self, timeout_s: float) -> bool:
|
||||
return self._wait_for(lambda: self.is_connected(), timeout_s)
|
||||
|
||||
def wait_disconnected(self, timeout_s: float) -> bool:
|
||||
return self._wait_for(lambda: not self.is_connected(), timeout_s)
|
||||
|
||||
def _wait_for(self, pred: Any, timeout_s: float) -> bool:
|
||||
deadline = time.monotonic() + timeout_s
|
||||
while time.monotonic() < deadline:
|
||||
if pred():
|
||||
return True
|
||||
time.sleep(0.02)
|
||||
return pred()
|
||||
|
||||
def _supervisor_loop(self) -> None:
|
||||
"""Maintain the WS connection: connect, flush backlog, read, backoff."""
|
||||
backoff = self.config.backoff_base_s
|
||||
while not self._stop.is_set():
|
||||
if self._current_conn() is None:
|
||||
try:
|
||||
conn = WsConnection.connect(self.config.ingest_url)
|
||||
except (ConnectionError, OSError, ValueError, TimeoutError):
|
||||
self._stop.wait(backoff)
|
||||
backoff = min(self.config.backoff_max_s, backoff * 2)
|
||||
continue
|
||||
self._set_conn(conn)
|
||||
self._last_sent = None
|
||||
backoff = self.config.backoff_base_s
|
||||
with self._emit_lock: # ordered against concurrent emit()s
|
||||
self._flush_locked()
|
||||
else:
|
||||
conn = self._current_conn()
|
||||
if conn is None:
|
||||
continue
|
||||
try:
|
||||
frame = conn.recv_message(timeout_s=1.0)
|
||||
except (ConnectionError, OSError):
|
||||
self._drop_conn()
|
||||
continue
|
||||
if frame is None:
|
||||
continue
|
||||
opcode, payload = frame
|
||||
if opcode == 0x1: # server text frame — parse advisory envelopes
|
||||
self._handle_server_text(payload)
|
||||
continue
|
||||
if opcode == 0x8: # server close frame
|
||||
self._drop_conn()
|
||||
|
||||
def _handle_server_text(self, payload: bytes) -> None:
|
||||
"""Consume server->agent envelopes (advisory; never fatal).
|
||||
|
||||
`seq_ack` (D-045): the server's durable latest_seq — trims the
|
||||
spool/pending to `seq > ack`, bounding the replay margin to the
|
||||
in-flight window (REQ-5-007). Unknown/malformed frames are ignored:
|
||||
acks are hints; gap detection and flood semantics stay authoritative
|
||||
server-side.
|
||||
"""
|
||||
try:
|
||||
envelope = json.loads(payload.decode("utf-8"))
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
return
|
||||
if not isinstance(envelope, dict):
|
||||
return
|
||||
if envelope.get("type") == "seq_ack":
|
||||
ack_seq = envelope.get("seq")
|
||||
if isinstance(ack_seq, int) and ack_seq >= 0:
|
||||
self.trim_to_ack(ack_seq)
|
||||
|
||||
# -- workspace watcher ------------------------------------------------
|
||||
def _snapshot_workspace(self) -> dict[str, tuple[int, int, str | None]]:
|
||||
"""Map rel path -> (mtime_ns, size, text-or-None-if-too-large)."""
|
||||
snap: dict[str, tuple[int, int, str | None]] = {}
|
||||
root = self.config.workspace
|
||||
if not root.is_dir():
|
||||
return snap
|
||||
for dirpath, dirnames, filenames in os.walk(root):
|
||||
dirnames[:] = sorted(
|
||||
d for d in dirnames if not d.startswith(_AGENT_DIR_PREFIX)
|
||||
)
|
||||
for name in sorted(filenames):
|
||||
if name.startswith(_AGENT_DIR_PREFIX):
|
||||
continue
|
||||
path = Path(dirpath) / name
|
||||
try:
|
||||
st = path.stat()
|
||||
except OSError:
|
||||
continue
|
||||
rel = path.relative_to(root).as_posix()
|
||||
text: str | None = None
|
||||
if st.st_size <= _MAX_DIFF_BYTES:
|
||||
try:
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
except OSError:
|
||||
pass
|
||||
snap[rel] = (st.st_mtime_ns, st.st_size, text)
|
||||
return snap
|
||||
|
||||
def _file_diff_payload(self, rel: str, change: str, old: str | None, new: str | None) -> dict:
|
||||
payload: dict[str, Any] = {"path": rel, "change": change}
|
||||
if old is None and new is None:
|
||||
payload["truncated"] = True
|
||||
return payload
|
||||
diff = "".join(
|
||||
difflib.unified_diff(
|
||||
(old or "").splitlines(keepends=True),
|
||||
(new or "").splitlines(keepends=True),
|
||||
fromfile=f"a/{rel}",
|
||||
tofile=f"b/{rel}",
|
||||
)
|
||||
)
|
||||
payload["diff"] = diff
|
||||
payload["size"] = len(new or "")
|
||||
return payload
|
||||
|
||||
def _watcher_loop(self) -> None:
|
||||
# Baseline is taken in start() before it returns, so any change made
|
||||
# after start() completes is guaranteed to be observed.
|
||||
baseline = self._baseline
|
||||
last_heartbeat = time.monotonic()
|
||||
while not self._stop.wait(self.config.poll_interval_s):
|
||||
current = self._snapshot_workspace()
|
||||
for rel in sorted(current.keys() | baseline.keys()):
|
||||
if rel not in baseline and rel in current:
|
||||
self.emit(
|
||||
"file_diff",
|
||||
self._file_diff_payload(rel, "created", None, current[rel][2]),
|
||||
)
|
||||
elif rel in baseline and rel not in current:
|
||||
self.emit(
|
||||
"file_diff",
|
||||
self._file_diff_payload(rel, "deleted", baseline[rel][2], None),
|
||||
)
|
||||
else:
|
||||
old_stat, new_stat = baseline[rel], current[rel]
|
||||
if old_stat[:2] != new_stat[:2] and old_stat[2] != new_stat[2]:
|
||||
self.emit(
|
||||
"file_diff",
|
||||
self._file_diff_payload(
|
||||
rel, "modified", old_stat[2], new_stat[2]
|
||||
),
|
||||
)
|
||||
baseline = current
|
||||
if time.monotonic() - last_heartbeat >= self.config.activity_interval_s:
|
||||
self.emit("activity", {"state": "idle", "spooled": len(self._pending)})
|
||||
last_heartbeat = time.monotonic()
|
||||
|
||||
# -- shell wrapper ------------------------------------------------------
|
||||
@staticmethod
|
||||
def _is_test_command(cmd: str) -> bool:
|
||||
return "test" in cmd.lower()
|
||||
|
||||
def run_command(self, line: str) -> dict[str, Any] | None:
|
||||
"""Run one REPL line; emits stdin/command/stdout/run|test_result."""
|
||||
line = line.strip()
|
||||
if not line:
|
||||
return None
|
||||
self.emit("stdin", {"line": line})
|
||||
self.emit("activity", {"state": "command", "spooled": len(self._pending)})
|
||||
self.emit("command", {"cmd": line})
|
||||
started = time.monotonic()
|
||||
timed_out = False
|
||||
exit_code: int | None = None
|
||||
out: str | bytes = ""
|
||||
err: str | bytes = ""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
["sh", "-c", line],
|
||||
cwd=self.config.workspace,
|
||||
capture_output=True,
|
||||
timeout=self.config.command_timeout_s,
|
||||
text=True,
|
||||
errors="replace",
|
||||
)
|
||||
exit_code, out, err = proc.returncode, proc.stdout, proc.stderr
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
timed_out = True
|
||||
# TimeoutExpired output attrs are always bytes (even in text mode).
|
||||
out = exc.stdout or b""
|
||||
err = exc.stderr or b""
|
||||
duration = time.monotonic() - started
|
||||
for stream, data in (("stdout", out), ("stderr", err)):
|
||||
if isinstance(data, bytes):
|
||||
data = data.decode(errors="replace")
|
||||
if data:
|
||||
self.emit("stdout", {"stream": stream, "data": data[-_MAX_OUTPUT_CHARS:]})
|
||||
kind = "test_result" if self._is_test_command(line) else "run_result"
|
||||
result = self.emit(
|
||||
kind,
|
||||
{
|
||||
"cmd": line,
|
||||
"exit_code": exit_code,
|
||||
"duration_s": round(duration, 6),
|
||||
"timed_out": timed_out,
|
||||
},
|
||||
)
|
||||
self.emit("activity", {"state": "idle", "spooled": len(self._pending)})
|
||||
return result
|
||||
|
||||
# -- lifecycle ----------------------------------------------------------
|
||||
def start(self) -> None:
|
||||
self.config.workspace.mkdir(parents=True, exist_ok=True)
|
||||
self._baseline = self._snapshot_workspace()
|
||||
self.emit("activity", {"state": "starting", "spooled": len(self._pending)})
|
||||
self._threads = [
|
||||
threading.Thread(target=self._supervisor_loop, daemon=True, name="nc-ws"),
|
||||
threading.Thread(target=self._watcher_loop, daemon=True, name="nc-watch"),
|
||||
]
|
||||
for thread in self._threads:
|
||||
thread.start()
|
||||
|
||||
def stop(self) -> None:
|
||||
if self._stop.is_set():
|
||||
return
|
||||
try:
|
||||
self.emit(
|
||||
"activity",
|
||||
{
|
||||
"state": "stopped",
|
||||
"spooled": len(self._pending),
|
||||
"dropped_overflow": self._dropped_overflow,
|
||||
},
|
||||
)
|
||||
finally:
|
||||
self._stop.set()
|
||||
self._drop_conn()
|
||||
for thread in self._threads:
|
||||
thread.join(timeout=3)
|
||||
for thread in self._threads:
|
||||
thread.join(timeout=3)
|
||||
with self._emit_lock:
|
||||
self._spool.rewrite(list(self._pending))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
config = AgentConfig.from_env()
|
||||
except ValueError as exc:
|
||||
print(f"sandbox-agent: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
agent = Agent(config)
|
||||
agent.start()
|
||||
try:
|
||||
# REPL mode: each stdin line is executed and reported (interactive use).
|
||||
# Daemon mode: when stdin is closed/absent (the sandbox backend spawns
|
||||
# the agent with stdin=DEVNULL), keep streaming workspace diffs +
|
||||
# activity until SIGTERM/SIGINT so the agent's lifecycle is tied to
|
||||
# the sandbox (destroy() reaps it) rather than to stdin EOF.
|
||||
if sys.stdin is None or sys.stdin.closed: # pragma: no cover - defensive
|
||||
agent._stop.wait() # noqa: SLF001 - daemon block
|
||||
else:
|
||||
line = sys.stdin.readline()
|
||||
while line:
|
||||
agent.run_command(line)
|
||||
line = sys.stdin.readline()
|
||||
if not agent._stop.is_set() and not sys.stdin.isatty(): # noqa: SLF001
|
||||
# EOF on a pipe (DEVNULL): daemonize — watch + stream until killed.
|
||||
import signal
|
||||
|
||||
signal.signal(signal.SIGTERM, lambda *_: agent._stop.set()) # noqa: SLF001
|
||||
agent._stop.wait() # noqa: SLF001
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
agent.stop()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,103 +1,118 @@
|
||||
"""Assessor agent tests — structured rubric scores (REQ-2-008).
|
||||
"""Assessor agent tests — live-grade coaching contract (REQ-3-007).
|
||||
|
||||
The Assessor is the structured-output showcase: tests use ScriptedJSONProvider
|
||||
for valid payloads and exercise the 4-layer defense failure modes.
|
||||
v0.3 re-grounding: the Assessor renders coaching FROM the stored grade
|
||||
(GradeRecord) — it never invents scores (the grading engine owns that).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import pytest
|
||||
|
||||
from ai_service.agents.assessor import AssessorAgent, RubricScore
|
||||
from ai_service.agents.structured import StructuredOutputError
|
||||
from ai_service.agents.assessor import AssessorAgent, GradeCoaching
|
||||
from ai_service.config import Settings
|
||||
from ai_service.corpus.artifacts import (
|
||||
get_artifact_bundle,
|
||||
get_transcript_for_artifact,
|
||||
render_rubric,
|
||||
from ai_service.grading.store import GradeRecord, SQLiteGradeStore
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
COACHING_JSON = json.dumps(
|
||||
{
|
||||
"summary": "Solid iterative build; tests drove the fixes.",
|
||||
"strengths": ["Ran tests after each change."],
|
||||
"gaps": ["Did not cover the empty-input case."],
|
||||
"next_steps": ["Add one edge-case test."],
|
||||
}
|
||||
)
|
||||
from ai_service.corpus.learner_context import get_learner_context
|
||||
from ai_service.llm.mock import MockProvider, ScriptedJSONProvider
|
||||
|
||||
VALID_SCORE = {
|
||||
"rubric_id": "rubric-orchestration-c002",
|
||||
"artifact_id": "art-eval-research-assistant",
|
||||
"competency_id": "stack-orchestration-c002",
|
||||
"scores": [
|
||||
{"criterion_id": "rc-architecture", "name": "Agent architecture soundness",
|
||||
"score": 92, "evidence": "Explicit state schema with planner-only write access"},
|
||||
{"criterion_id": "rc-communication", "name": "Inter-agent communication design",
|
||||
"score": 88, "evidence": "Typed ToolMessage responses with retry flags"},
|
||||
{"criterion_id": "rc-reliability", "name": "Reliability engineering",
|
||||
"score": 85, "evidence": "3-retry loop with degradation path"},
|
||||
{"criterion_id": "rc-process", "name": "Process trace quality",
|
||||
"score": 90, "evidence": "Iterative saves with passing test checkpoints"},
|
||||
],
|
||||
"strengths": ["Clean state boundaries", "Failure-aware tool wrapping"],
|
||||
"gaps": ["No reviewer node yet", "Graph diagram only in README"],
|
||||
"verdict": "mastered",
|
||||
}
|
||||
|
||||
|
||||
def make_assessor(provider=None) -> AssessorAgent:
|
||||
return AssessorAgent(provider or MockProvider(), Settings(provider="mock"))
|
||||
class ScriptedProvider(MockProvider):
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.requests: list[list[Message]] = []
|
||||
self.replies: list[str] = []
|
||||
|
||||
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
self.requests.append([Message(role=m.role, content=m.content) for m in messages])
|
||||
if self.replies:
|
||||
return self.replies.pop(0)
|
||||
return COACHING_JSON
|
||||
|
||||
|
||||
def get_bundle(artifact_id="art-eval-research-assistant"):
|
||||
bundle = get_artifact_bundle(artifact_id)
|
||||
assert bundle is not None
|
||||
return bundle
|
||||
|
||||
|
||||
async def test_evaluate_returns_validated_rubric_score():
|
||||
provider = ScriptedJSONProvider(VALID_SCORE)
|
||||
assessor = make_assessor(provider)
|
||||
artifact, rubric = get_bundle()
|
||||
transcript = get_transcript_for_artifact(artifact.artifact_id)
|
||||
result = await assessor.evaluate(artifact, rubric, transcript)
|
||||
assert isinstance(result, RubricScore)
|
||||
assert result.verdict == "mastered"
|
||||
assert len(result.scores) == 4
|
||||
assert result.weighted_total(rubric) == pytest.approx(
|
||||
92 * 0.3 + 88 * 0.3 + 85 * 0.25 + 90 * 0.15
|
||||
def _grade() -> GradeRecord:
|
||||
return GradeRecord(
|
||||
learner_id="assessor-learner",
|
||||
task_id="assessor-task",
|
||||
variant_seed=None,
|
||||
digest={"error_fix_cycles": 2, "final_test_status": "pass"},
|
||||
scores={
|
||||
"criteria": {
|
||||
"process_quality": 4,
|
||||
"correctness": 3,
|
||||
"debugging_discipline": 4,
|
||||
"test_usage": 3,
|
||||
},
|
||||
"strengths": ["s"],
|
||||
"gaps": ["g"],
|
||||
"verdict": "developing",
|
||||
},
|
||||
verdict="GRADED",
|
||||
model="gemma4:31b",
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
|
||||
|
||||
async def test_evaluate_rejects_invalid_schema_after_retry():
|
||||
"""Plain MockProvider returns non-rubric JSON → 4-layer defense exhausts
|
||||
its single retry and raises StructuredOutputError."""
|
||||
assessor = make_assessor(MockProvider())
|
||||
artifact, rubric = get_bundle()
|
||||
transcript = get_transcript_for_artifact(artifact.artifact_id)
|
||||
with pytest.raises(StructuredOutputError):
|
||||
await assessor.evaluate(artifact, rubric, transcript)
|
||||
@pytest.fixture()
|
||||
def provider() -> ScriptedProvider:
|
||||
return ScriptedProvider()
|
||||
|
||||
|
||||
def test_build_evaluation_input_carries_all_inputs():
|
||||
assessor = make_assessor()
|
||||
artifact, rubric = get_bundle()
|
||||
transcript = get_transcript_for_artifact(artifact.artifact_id)
|
||||
text = assessor.build_evaluation_input(artifact, rubric, transcript)
|
||||
assert artifact.name in text
|
||||
assert artifact.evidence_excerpt in text
|
||||
assert "rc-architecture" in text # rubric rendered
|
||||
assert "examiner:" in text # transcript rendered
|
||||
@pytest.fixture()
|
||||
def agent(provider) -> AssessorAgent:
|
||||
return AssessorAgent(provider, Settings(provider="mock"))
|
||||
|
||||
|
||||
def test_build_evaluation_input_without_transcript():
|
||||
assessor = make_assessor()
|
||||
artifact, rubric = get_bundle()
|
||||
text = assessor.build_evaluation_input(artifact, rubric, None)
|
||||
assert artifact.name in text
|
||||
assert "examiner:" not in text
|
||||
class TestCoachGrade:
|
||||
async def test_prompt_contains_stored_grade_not_learner_id(self, agent, provider) -> None:
|
||||
await agent.coach_grade(_grade())
|
||||
all_text = "\n".join(
|
||||
m.content for request in provider.requests for m in request
|
||||
)
|
||||
assert "process_quality" in all_text # stored scores rendered
|
||||
assert "GRADED" in all_text
|
||||
assert "assessor-learner" not in all_text # D-028 anonymity
|
||||
|
||||
async def test_coaching_validates_via_d020(self, agent, provider) -> None:
|
||||
coaching = await agent.coach_grade(_grade())
|
||||
assert isinstance(coaching, GradeCoaching)
|
||||
assert coaching.summary
|
||||
assert coaching.next_steps
|
||||
|
||||
async def test_malformed_then_good_exercises_retry(self, agent, provider) -> None:
|
||||
provider.replies = ["garbage", COACHING_JSON]
|
||||
coaching = await agent.coach_grade(_grade())
|
||||
assert coaching.summary
|
||||
assert len(provider.requests) == 2
|
||||
|
||||
async def test_no_corpus_artifact_imports(self) -> None:
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
py = Path(__file__).parents[2] / "ai_service" / "agents" / "assessor.py"
|
||||
tree = ast.parse(py.read_text())
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.ImportFrom) and node.module:
|
||||
assert "corpus.artifacts" not in node.module
|
||||
assert "corpus.telemetry" not in node.module
|
||||
|
||||
|
||||
def test_system_prompt_names_assessor_persona():
|
||||
prompt = make_assessor().system_prompt(get_learner_context())
|
||||
assert "Assessor" in prompt
|
||||
assert "ONLY" in prompt # JSON-only instruction
|
||||
|
||||
|
||||
def test_rubric_render_in_prompt_is_complete():
|
||||
"""The rubric passed to the model lists every criterion (fair grading)."""
|
||||
artifact, rubric = get_bundle()
|
||||
text = render_rubric(rubric)
|
||||
assert text.count("rc-") == len(rubric.criteria)
|
||||
class TestStoreRoundtrip:
|
||||
def test_grade_store_roundtrip(self, tmp_path) -> None:
|
||||
store = SQLiteGradeStore(db_path=tmp_path / "g.db")
|
||||
record = _grade()
|
||||
store.save(record)
|
||||
fetched = store.get("assessor-learner", "assessor-task")
|
||||
assert fetched is not None
|
||||
assert fetched.scores["criteria"]["process_quality"] == 4
|
||||
store.close()
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
"""Examiner agent tests (Task 5-2-01, REQ-3-006)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from ai_service.agents.examiner import DefenseVerdict, ExaminerAgent
|
||||
from ai_service.agents.registry import AgentRegistry, register_builtin_agents
|
||||
from ai_service.config import Settings
|
||||
from ai_service.grading.features import compute_digest
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
VERDICT_JSON = json.dumps(
|
||||
{
|
||||
"verdict": "developing",
|
||||
"understanding": "Explains the retry loop clearly.",
|
||||
"process_justification": "Justifies the edit-then-test cadence from the digest.",
|
||||
"communication": "Answers are specific and on-topic.",
|
||||
"strengths": ["Grounded the fix in a failed test."],
|
||||
"gaps": ["Did not justify the chunk-size choice."],
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class RecordingProvider(MockProvider):
|
||||
"""Mock provider that records every message list (prompt assertions)."""
|
||||
|
||||
def __init__(self, replies: list[str] | None = None) -> None:
|
||||
super().__init__()
|
||||
self.replies = list(replies or [])
|
||||
self.requests: list[list[Message]] = []
|
||||
|
||||
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
self.requests.append([Message(role=m.role, content=m.content) for m in messages])
|
||||
if self.replies:
|
||||
return self.replies.pop(0)
|
||||
return "Tell me about your build."
|
||||
|
||||
|
||||
def _digest():
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
from ai_service.telemetry.models import TelemetryEvent
|
||||
|
||||
t0 = datetime(2026, 9, 12, tzinfo=UTC)
|
||||
events = [
|
||||
TelemetryEvent(
|
||||
learner_id="examiner-learner",
|
||||
task_id="examiner-task",
|
||||
seq=n,
|
||||
kind=kind,
|
||||
payload=payload,
|
||||
ts=t0 + timedelta(seconds=n * 10),
|
||||
sandbox_id="sbx-examiner",
|
||||
)
|
||||
for n, (kind, payload) in enumerate(
|
||||
[
|
||||
("file_diff", {"path": "a.py"}),
|
||||
("command", {"cmd": "pytest -q"}),
|
||||
("test_result", {"passed": False, "exit_code": 1}),
|
||||
("file_diff", {"path": "a.py"}),
|
||||
("test_result", {"passed": True, "exit_code": 0}),
|
||||
]
|
||||
)
|
||||
]
|
||||
return compute_digest(events)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def provider() -> RecordingProvider:
|
||||
return RecordingProvider()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def settings() -> Settings:
|
||||
return Settings(provider="mock")
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def examiner(provider, settings) -> ExaminerAgent:
|
||||
return ExaminerAgent(provider, settings)
|
||||
|
||||
|
||||
class TestNextQuestion:
|
||||
async def test_prompt_contains_digest_but_no_learner_id(
|
||||
self, examiner, provider
|
||||
) -> None:
|
||||
await examiner.next_question(
|
||||
history=[Message(role="assistant", content="First question?")],
|
||||
trace_digest=_digest(),
|
||||
variant_statement="Build a chunker.",
|
||||
)
|
||||
all_content = "\n".join(
|
||||
m.content for request in provider.requests for m in request
|
||||
)
|
||||
assert "error_fix_cycles" in all_content # digest JSON grounded
|
||||
assert "examiner-learner" not in all_content # D-028 anonymity
|
||||
assert "Build a chunker." in all_content # variant statement grounded
|
||||
assert all_content.count('"examiner-learner"') == 0
|
||||
|
||||
async def test_question_returned_from_provider(self, examiner) -> None:
|
||||
question = await examiner.next_question(
|
||||
history=[], trace_digest=_digest()
|
||||
)
|
||||
assert isinstance(question, str)
|
||||
|
||||
|
||||
class TestFinalVerdict:
|
||||
async def test_verdict_validates_via_d020(self, examiner, provider) -> None:
|
||||
provider.replies = [VERDICT_JSON]
|
||||
verdict = await examiner.final_verdict(
|
||||
history=[Message(role="assistant", content="Q?")],
|
||||
trace_digest=_digest(),
|
||||
)
|
||||
assert isinstance(verdict, DefenseVerdict)
|
||||
assert verdict.verdict == "developing"
|
||||
assert verdict.strengths and verdict.gaps
|
||||
|
||||
async def test_malformed_then_good_exercises_retry(self, examiner, provider) -> None:
|
||||
provider.replies = ["not json", VERDICT_JSON]
|
||||
verdict = await examiner.final_verdict(history=[], trace_digest=_digest())
|
||||
assert verdict.verdict == "developing"
|
||||
assert len(provider.requests) == 2 # D-020 bounded retry
|
||||
|
||||
|
||||
class TestRegistry:
|
||||
def test_all_seven_agents_resolve(self, provider, settings) -> None:
|
||||
registry = AgentRegistry()
|
||||
register_builtin_agents(registry)
|
||||
assert registry.names() == [
|
||||
"assessor",
|
||||
"coach",
|
||||
"examiner",
|
||||
"lab",
|
||||
"mentor",
|
||||
"proctor",
|
||||
"tutor",
|
||||
]
|
||||
agent = registry.get(provider, settings, "examiner")
|
||||
assert isinstance(agent, ExaminerAgent)
|
||||
|
||||
|
||||
class TestBoundary:
|
||||
def test_examiner_never_imports_voice_or_api(self) -> None:
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
py = Path(__file__).parents[2] / "ai_service" / "agents" / "examiner.py"
|
||||
tree = ast.parse(py.read_text())
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.ImportFrom) and node.module:
|
||||
assert "voice" not in node.module, "examiner must not import voice/"
|
||||
assert not node.module.startswith("ai_service.api")
|
||||
if isinstance(node, ast.Import):
|
||||
for alias in node.names:
|
||||
assert alias.name != "fastapi"
|
||||
@@ -97,10 +97,11 @@ def test_proctor_and_mentor_resolve_via_registry():
|
||||
assert isinstance(mentor, MentorAgent)
|
||||
|
||||
|
||||
def test_registry_resolves_all_six_agents():
|
||||
"""Must-Have (Phase 5): the full roster — coach/tutor/lab/assessor/proctor/mentor."""
|
||||
def test_registry_resolves_all_seven_agents():
|
||||
"""Must-Have (Phase 5): the full roster — six tutors + the Examiner."""
|
||||
from ai_service.agents.assessor import AssessorAgent
|
||||
from ai_service.agents.coach import CoachAgent
|
||||
from ai_service.agents.examiner import ExaminerAgent
|
||||
from ai_service.agents.lab import LabAgent
|
||||
from ai_service.agents.mentor import MentorAgent
|
||||
from ai_service.agents.proctor import ProctorAgent
|
||||
@@ -108,7 +109,9 @@ def test_registry_resolves_all_six_agents():
|
||||
|
||||
registry = AgentRegistry()
|
||||
register_builtin_agents(registry)
|
||||
assert registry.names() == ["assessor", "coach", "lab", "mentor", "proctor", "tutor"]
|
||||
assert registry.names() == [
|
||||
"assessor", "coach", "examiner", "lab", "mentor", "proctor", "tutor",
|
||||
]
|
||||
settings = Settings(provider="mock")
|
||||
expected = {
|
||||
"coach": CoachAgent,
|
||||
@@ -117,6 +120,7 @@ def test_registry_resolves_all_six_agents():
|
||||
"assessor": AssessorAgent,
|
||||
"proctor": ProctorAgent,
|
||||
"mentor": MentorAgent,
|
||||
"examiner": ExaminerAgent,
|
||||
}
|
||||
for name, cls in expected.items():
|
||||
agent = registry.get(MockProvider(), settings, name)
|
||||
|
||||
@@ -0,0 +1,207 @@
|
||||
"""Live re-grounding tests: Lab, Assessor, Proctor on REAL inputs (REQ-3-007)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import tempfile
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from ai_service.agents.assessor import AssessorAgent, GradeCoaching
|
||||
from ai_service.agents.lab import LabAgent
|
||||
from ai_service.agents.proctor import ProctorAgent, ProctorAssessment
|
||||
from ai_service.config import Settings
|
||||
from ai_service.grading.store import GradeRecord, SQLiteGradeStore
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.llm.types import Message
|
||||
from ai_service.telemetry.models import TelemetryEvent
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
|
||||
COACHING_JSON = json.dumps(
|
||||
{
|
||||
"summary": "Iterative build with test discipline.",
|
||||
"strengths": ["Tested after changes."],
|
||||
"gaps": ["Missing edge cases."],
|
||||
"next_steps": ["Add an edge-case test."],
|
||||
}
|
||||
)
|
||||
PROCTOR_JSON = json.dumps(
|
||||
{
|
||||
"signals": [
|
||||
{"signal_type": "idle_gap", "severity": "low", "note": "One 400s gap."}
|
||||
],
|
||||
"intervention": "Offer a short break.",
|
||||
"summary": "Healthy session overall.",
|
||||
}
|
||||
)
|
||||
T0 = datetime(2026, 9, 12, tzinfo=UTC)
|
||||
|
||||
|
||||
class RecordingProvider(MockProvider):
|
||||
def __init__(self, structured_json: str) -> None:
|
||||
super().__init__()
|
||||
self._structured_json = structured_json
|
||||
self.requests: list[list[Message]] = []
|
||||
|
||||
def _reply_for(self, messages, response_format):
|
||||
self.requests.append([Message(role=m.role, content=m.content) for m in messages])
|
||||
if response_format is not None and response_format.get("type") == "json_object":
|
||||
return self._structured_json
|
||||
return "Coaching feedback referencing your latest test run."
|
||||
|
||||
|
||||
def _event(
|
||||
seq: int, kind: str, payload: dict, offset_s: float,
|
||||
learner="live-learner", task="live-task",
|
||||
):
|
||||
return TelemetryEvent(
|
||||
learner_id=learner,
|
||||
task_id=task,
|
||||
seq=seq,
|
||||
kind=kind,
|
||||
payload=payload,
|
||||
ts=T0 + timedelta(seconds=offset_s),
|
||||
sandbox_id="sbx-live",
|
||||
)
|
||||
|
||||
|
||||
def _seed_trace(store: SQLiteTraceStore) -> None:
|
||||
events = [
|
||||
_event(0, "file_diff", {"path": "a.py"}, 0),
|
||||
_event(1, "command", {"cmd": "pytest -q"}, 10),
|
||||
_event(2, "test_result", {"passed": False, "exit_code": 1}, 15),
|
||||
_event(3, "file_diff", {"path": "a.py"}, 30),
|
||||
_event(4, "test_result", {"passed": True, "exit_code": 0}, 45),
|
||||
_event(5, "activity", {"state": "idle"}, 500), # >120s gap -> idle
|
||||
]
|
||||
for e in events:
|
||||
store.append(e)
|
||||
|
||||
|
||||
class TestLabLive:
|
||||
async def test_lab_prompt_contains_digest_not_corpus(self, tmp_path) -> None:
|
||||
store = SQLiteTraceStore(db_path=tmp_path / "t.db")
|
||||
_seed_trace(store)
|
||||
provider = RecordingProvider("feedback")
|
||||
agent = LabAgent(provider, Settings(provider="mock"))
|
||||
|
||||
from ai_service.grading.features import compute_digest
|
||||
|
||||
digest = compute_digest(store.get_trace("live-learner", "live-task"))
|
||||
tokens = [t async for t in agent.stream_feedback(digest)]
|
||||
assert tokens
|
||||
all_text = "\n".join(
|
||||
m.content for request in provider.requests for m in request
|
||||
)
|
||||
assert "error_fix_cycles" in all_text
|
||||
assert "lab-scenario" not in all_text # no corpus fixture ids
|
||||
store.close()
|
||||
|
||||
def test_no_corpus_telemetry_import_in_lab(self) -> None:
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
py = Path(__file__).parents[2] / "ai_service" / "agents" / "lab.py"
|
||||
tree = ast.parse(py.read_text())
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.ImportFrom) and node.module:
|
||||
assert "corpus.telemetry" not in node.module
|
||||
|
||||
|
||||
class TestAssessorLive:
|
||||
async def test_assessor_prompt_contains_stored_scores(self) -> None:
|
||||
provider = RecordingProvider(COACHING_JSON)
|
||||
agent = AssessorAgent(provider, Settings(provider="mock"))
|
||||
grade = GradeRecord(
|
||||
learner_id="live-learner",
|
||||
task_id="live-task",
|
||||
variant_seed=None,
|
||||
digest={"error_fix_cycles": 1},
|
||||
scores={"criteria": {"process_quality": 3}},
|
||||
verdict="GRADED",
|
||||
model="mock",
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
coaching = await agent.coach_grade(grade)
|
||||
assert isinstance(coaching, GradeCoaching)
|
||||
all_text = "\n".join(
|
||||
m.content for request in provider.requests for m in request
|
||||
)
|
||||
assert "process_quality" in all_text
|
||||
assert "live-learner" not in all_text
|
||||
|
||||
def test_no_corpus_artifact_import_in_assessor(self) -> None:
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
py = Path(__file__).parents[2] / "ai_service" / "agents" / "assessor.py"
|
||||
tree = ast.parse(py.read_text())
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.ImportFrom) and node.module:
|
||||
assert "corpus.artifacts" not in node.module
|
||||
assert "corpus.telemetry" not in node.module
|
||||
|
||||
|
||||
class TestProctorLive:
|
||||
async def test_proctor_receives_real_digest_and_defense_signals(self) -> None:
|
||||
provider = RecordingProvider(PROCTOR_JSON)
|
||||
agent = ProctorAgent(provider, Settings(provider="mock"))
|
||||
from ai_service.grading.features import compute_digest
|
||||
|
||||
store = SQLiteTraceStore(db_path=Path(tempfile.mkdtemp()) / "proctor-t.db")
|
||||
_seed_trace(store)
|
||||
digest = compute_digest(store.get_trace("live-learner", "live-task"))
|
||||
store.close()
|
||||
|
||||
assessment = await agent.assess(
|
||||
digest,
|
||||
defense_signals={"long_pauses": [{"turn": 3, "latency_ms": 30000}]},
|
||||
variant_context={"template_id": "tpl-llm-judge", "seed": "cafe", "params": {}},
|
||||
)
|
||||
assert isinstance(assessment, ProctorAssessment)
|
||||
all_text = "\n".join(
|
||||
m.content for request in provider.requests for m in request
|
||||
)
|
||||
assert "idle_gap" in all_text or "idle_gap_count" in all_text
|
||||
assert "long_pauses" in all_text
|
||||
assert "tpl-llm-judge" in all_text
|
||||
assert "live-learner" not in all_text
|
||||
|
||||
def test_no_corpus_scenario_import_in_proctor(self) -> None:
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
py = Path(__file__).parents[2] / "ai_service" / "agents" / "proctor.py"
|
||||
tree = ast.parse(py.read_text())
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.ImportFrom) and node.module:
|
||||
assert "corpus.telemetry" not in node.module
|
||||
|
||||
|
||||
class TestProctorEndpoint:
|
||||
def test_signals_endpoint_serves_real_inputs(self, tmp_path) -> None:
|
||||
from ai_service.main import create_app
|
||||
from ai_service.telemetry.ingest import TraceIntegrityMap
|
||||
from ai_service.variants.store import SQLiteVariantStore
|
||||
from ai_service.voice.defense_store import SQLiteDefenseStore
|
||||
|
||||
app = create_app(Settings(provider="mock"))
|
||||
provider = RecordingProvider(PROCTOR_JSON)
|
||||
app.state.provider = provider
|
||||
app.state.trace_store = SQLiteTraceStore(db_path=tmp_path / "t.db")
|
||||
app.state.grade_store = SQLiteGradeStore(db_path=tmp_path / "g.db")
|
||||
app.state.variant_store = SQLiteVariantStore(db_path=tmp_path / "v.db")
|
||||
app.state.defense_store = SQLiteDefenseStore(db_path=tmp_path / "d.db")
|
||||
app.state.trace_integrity = TraceIntegrityMap()
|
||||
_seed_trace(app.state.trace_store)
|
||||
with TestClient(app) as client:
|
||||
resp = client.post(
|
||||
"/v1/proctor/signals",
|
||||
json={"learner_id": "live-learner", "task_id": "live-task"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["signals"] is not None
|
||||
assert "intervention" in body
|
||||
@@ -1,79 +1,87 @@
|
||||
"""Assessment evaluate endpoint tests — validated JSON, 404s (REQ-2-008)."""
|
||||
"""Assessment API tests — stored-grade coaching contract (REQ-3-007)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from ai_service.config import Settings
|
||||
from ai_service.grading.store import GradeRecord, SQLiteGradeStore
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.main import create_app
|
||||
from ai_service.telemetry.ingest import TraceIntegrityMap
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
|
||||
COACHING_JSON = json.dumps(
|
||||
{
|
||||
"summary": "Good iterative work.",
|
||||
"strengths": ["Tests after changes."],
|
||||
"gaps": ["Missing edge cases."],
|
||||
"next_steps": ["Add an edge-case test."],
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
from ai_service.agents.assessor import RubricScore
|
||||
from ai_service.llm.mock import ScriptedJSONProvider
|
||||
|
||||
VALID_SCORE = {
|
||||
"rubric_id": "rubric-orchestration-c002",
|
||||
"artifact_id": "art-eval-research-assistant",
|
||||
"competency_id": "stack-orchestration-c002",
|
||||
"scores": [
|
||||
{"criterion_id": "rc-architecture", "name": "Agent architecture soundness",
|
||||
"score": 92, "evidence": "Explicit state schema"},
|
||||
{"criterion_id": "rc-communication", "name": "Inter-agent communication design",
|
||||
"score": 88, "evidence": "Typed ToolMessage responses"},
|
||||
{"criterion_id": "rc-reliability", "name": "Reliability engineering",
|
||||
"score": 85, "evidence": "3-retry loop"},
|
||||
{"criterion_id": "rc-process", "name": "Process trace quality",
|
||||
"score": 90, "evidence": "Iterative checkpoints"},
|
||||
],
|
||||
"strengths": ["Clean state boundaries", "Failure-aware tools"],
|
||||
"gaps": ["No reviewer node", "Diagram only in README"],
|
||||
"verdict": "mastered",
|
||||
}
|
||||
class CoachingMock(MockProvider):
|
||||
def _reply_for(self, messages, response_format):
|
||||
if response_format is not None and response_format.get("type") == "json_object":
|
||||
return COACHING_JSON
|
||||
return super()._reply_for(messages, response_format)
|
||||
|
||||
|
||||
def test_evaluate_returns_validated_rubric_json(client):
|
||||
# Swap the app provider for a scripted-JSON provider for this test
|
||||
original = client.app.state.provider
|
||||
client.app.state.provider = ScriptedJSONProvider(VALID_SCORE)
|
||||
try:
|
||||
response = client.post(
|
||||
"/v1/assessment/evaluate",
|
||||
json={"artifact_id": "art-eval-research-assistant"},
|
||||
@pytest.fixture()
|
||||
def client(tmp_path) -> TestClient:
|
||||
app = create_app(Settings(provider="mock"))
|
||||
app.state.provider = CoachingMock()
|
||||
app.state.trace_store = SQLiteTraceStore(db_path=tmp_path / "t.db")
|
||||
app.state.grade_store = SQLiteGradeStore(db_path=tmp_path / "g.db")
|
||||
app.state.trace_integrity = TraceIntegrityMap()
|
||||
with TestClient(app) as c:
|
||||
yield c
|
||||
|
||||
|
||||
def _seed_grade(client: TestClient) -> None:
|
||||
|
||||
app = client.app
|
||||
store: SQLiteGradeStore = app.state.grade_store
|
||||
store.save(
|
||||
GradeRecord(
|
||||
learner_id="api-learner",
|
||||
task_id="api-task",
|
||||
variant_seed=None,
|
||||
digest={"error_fix_cycles": 1},
|
||||
scores={"criteria": {"process_quality": 3}, "verdict": "developing"},
|
||||
verdict="GRADED",
|
||||
model="gemma4:31b",
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
finally:
|
||||
client.app.state.provider = original
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
validated = RubricScore.model_validate(data) # response contract holds
|
||||
assert validated.verdict == "mastered"
|
||||
assert len(validated.scores) == 4
|
||||
|
||||
|
||||
def test_unknown_artifact_404(client):
|
||||
response = client.post("/v1/assessment/evaluate", json={"artifact_id": "ghost"})
|
||||
assert response.status_code == 404
|
||||
assert "ghost" in response.json()["detail"]
|
||||
|
||||
|
||||
def test_unparseable_provider_502(client):
|
||||
"""Plain MockProvider yields non-rubric JSON → structured defense exhausts
|
||||
retry → endpoint translates to 502 (bad gateway to the model)."""
|
||||
# default mock already returns non-rubric JSON
|
||||
response = client.post(
|
||||
"/v1/assessment/evaluate",
|
||||
json={"artifact_id": "art-eval-research-assistant"},
|
||||
)
|
||||
assert response.status_code == 502
|
||||
assert "failed" in response.json()["detail"].lower()
|
||||
|
||||
|
||||
def test_missing_artifact_id_422(client):
|
||||
response = client.post("/v1/assessment/evaluate", json={})
|
||||
assert response.status_code == 422
|
||||
def test_evaluate_renders_stored_grade_as_coaching(client) -> None:
|
||||
_seed_grade(client)
|
||||
resp = client.post(
|
||||
"/v1/assessment/evaluate",
|
||||
json={"learner_id": "api-learner", "task_id": "api-task"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["grade_verdict"] == "GRADED"
|
||||
assert body["coaching"]["summary"]
|
||||
|
||||
|
||||
def test_second_artifact_also_evaluates(client):
|
||||
original = client.app.state.provider
|
||||
payload = dict(VALID_SCORE, artifact_id="art-eval-rag-dashboard")
|
||||
client.app.state.provider = ScriptedJSONProvider(payload)
|
||||
try:
|
||||
response = client.post(
|
||||
"/v1/assessment/evaluate", json={"artifact_id": "art-eval-rag-dashboard"}
|
||||
)
|
||||
finally:
|
||||
client.app.state.provider = original
|
||||
assert response.status_code == 200
|
||||
assert response.json()["artifact_id"] == "art-eval-rag-dashboard"
|
||||
def test_evaluate_without_grade_404(client) -> None:
|
||||
resp = client.post(
|
||||
"/v1/assessment/evaluate",
|
||||
json={"learner_id": "nobody", "task_id": "nothing"},
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
assert "grade first" in resp.json()["detail"]
|
||||
|
||||
|
||||
def test_missing_fields_422(client) -> None:
|
||||
resp = client.post("/v1/assessment/evaluate", json={"learner_id": "x"})
|
||||
assert resp.status_code == 422
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
"""CORS policy tests (A-008, D-038 network mode).
|
||||
|
||||
v0.3 initially shipped `allow_methods` WITHOUT "PUT" while the learner
|
||||
build surface writes workspace files with PUT (engine-client writeFile) —
|
||||
every cross-origin Save failed preflight. These tests pin the policy so a
|
||||
future method-list edit fails loudly instead of silently breaking the
|
||||
headline flow.
|
||||
|
||||
v0.3.5 network mode (D-038): the default AI_CORS_ORIGINS='*' admits any
|
||||
origin (safe ONLY because credentials are never enabled); an explicit list
|
||||
restricts. Both modes are pinned here:
|
||||
- wildcard: remote origin gets the grant; credentials still never sent;
|
||||
- explicit: unlisted origins get no grant.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
ALLOWED_ORIGIN = "http://localhost:3000"
|
||||
REMOTE_ORIGIN = "http://nextcraft-1:3000"
|
||||
ALL_CLIENT_METHODS = ("GET", "POST", "PUT", "DELETE")
|
||||
|
||||
|
||||
def _allow_origin(resp) -> str | None:
|
||||
return resp.headers.get("access-control-allow-origin")
|
||||
|
||||
|
||||
def test_preflight_allows_every_method_the_web_client_uses(client: TestClient) -> None:
|
||||
for method in ALL_CLIENT_METHODS:
|
||||
resp = client.options(
|
||||
"/v1/sandboxes",
|
||||
headers={
|
||||
"Origin": ALLOWED_ORIGIN,
|
||||
"Access-Control-Request-Method": method,
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200, f"preflight {method} failed: {resp.status_code}"
|
||||
assert _allow_origin(resp) in ("*", ALLOWED_ORIGIN)
|
||||
allowed = resp.headers["access-control-allow-methods"].split(", ")
|
||||
assert method in allowed, f"{method} missing from CORS methods: {allowed}"
|
||||
|
||||
|
||||
def test_cross_origin_get_echoes_allow_origin(client: TestClient) -> None:
|
||||
resp = client.get("/v1/sandboxes", headers={"Origin": ALLOWED_ORIGIN})
|
||||
assert resp.status_code == 200
|
||||
assert _allow_origin(resp) in ("*", ALLOWED_ORIGIN)
|
||||
|
||||
|
||||
def test_wildcard_mode_grants_remote_origins(client: TestClient) -> None:
|
||||
"""D-038 default: '*' grants any origin — remote browsers work zero-config."""
|
||||
resp = client.get("/v1/sandboxes", headers={"Origin": REMOTE_ORIGIN})
|
||||
assert resp.status_code == 200
|
||||
assert _allow_origin(resp) in ("*", REMOTE_ORIGIN)
|
||||
|
||||
|
||||
def test_explicit_list_mode_denies_unlisted_origins(
|
||||
settings, monkeypatch, tmp_path
|
||||
) -> None:
|
||||
"""Explicit AI_CORS_ORIGINS restricts to the listed origins only."""
|
||||
from fastapi.testclient import TestClient as TC
|
||||
|
||||
from ai_service.main import create_app
|
||||
|
||||
restricted = settings.model_copy(update={"cors_origins": "http://localhost:3000"})
|
||||
app = create_app(restricted)
|
||||
with TC(app) as c:
|
||||
resp = c.get("/v1/sandboxes", headers={"Origin": "https://evil.example"})
|
||||
assert resp.status_code == 200 # non-CORS requests still serve
|
||||
assert resp.headers.get("access-control-allow-origin") is None
|
||||
|
||||
|
||||
def test_credentials_never_allowed(client: TestClient) -> None:
|
||||
resp = client.options(
|
||||
"/v1/sandboxes",
|
||||
headers={
|
||||
"Origin": ALLOWED_ORIGIN,
|
||||
"Access-Control-Request-Method": "PUT",
|
||||
"Access-Control-Request-Headers": "Content-Type",
|
||||
},
|
||||
)
|
||||
assert resp.headers.get("access-control-allow-credentials") != "true"
|
||||
@@ -0,0 +1,377 @@
|
||||
"""Defense endpoint tests (Task 5-3-01, REQ-3-006) — mock voice + mock LLM."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from ai_service.agents.examiner import ExaminerAgent
|
||||
from ai_service.config import Settings
|
||||
from ai_service.grading.store import SQLiteGradeStore
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.main import create_app
|
||||
from ai_service.telemetry.ingest import TraceIntegrityMap
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
from ai_service.variants.store import SQLiteVariantStore
|
||||
from ai_service.voice.defense_store import SQLiteDefenseStore
|
||||
from ai_service.voice.mock import MockVoiceProvider
|
||||
|
||||
VERDICT = {
|
||||
"verdict": "developing",
|
||||
"understanding": "Explains the build clearly.",
|
||||
"process_justification": "Justifies choices.",
|
||||
"communication": "Clear and specific.",
|
||||
"strengths": ["Grounded answers in the digest."],
|
||||
"gaps": ["Did not address the edge cases."],
|
||||
}
|
||||
|
||||
|
||||
class ScriptedLLM(MockProvider):
|
||||
"""Question-mode calls get a question; verdict-mode calls get D-020 JSON.
|
||||
|
||||
Discriminator: the verdict prompt contains "final verdict JSON" — the
|
||||
question prompt says "next question".
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
|
||||
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
all_text = "\n".join(m.content for m in messages)
|
||||
if "final verdict JSON" in all_text:
|
||||
return json.dumps(VERDICT)
|
||||
return "Why did you structure the fix that way?"
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def app(tmp_path: Path):
|
||||
from ai_service.identity.store import SQLiteIdentityStore
|
||||
|
||||
from ..conftest import SUITE_LEARNERS, seed_verified_identity
|
||||
|
||||
settings = Settings(
|
||||
provider="mock",
|
||||
voice_provider="mock",
|
||||
learner_allowlist=SUITE_LEARNERS,
|
||||
)
|
||||
application = create_app(settings)
|
||||
identity_store = SQLiteIdentityStore(db_path=tmp_path / "identity.db")
|
||||
seed_verified_identity(identity_store)
|
||||
application.state.identity_store = identity_store # G-9
|
||||
llm = ScriptedLLM()
|
||||
application.state.provider = llm
|
||||
application.state.trace_store = SQLiteTraceStore(db_path=tmp_path / "t.db")
|
||||
application.state.variant_store = SQLiteVariantStore(db_path=tmp_path / "v.db")
|
||||
application.state.grade_store = SQLiteGradeStore(db_path=tmp_path / "g.db")
|
||||
application.state.trace_integrity = TraceIntegrityMap()
|
||||
application.state.defense_store = SQLiteDefenseStore(db_path=tmp_path / "d.db")
|
||||
application.state.voice_provider = MockVoiceProvider(
|
||||
["the fix was in the retry loop"]
|
||||
)
|
||||
settings = Settings(provider="mock", voice_provider="mock")
|
||||
application.state.examiner_agent = ExaminerAgent(llm, settings)
|
||||
return application
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client(app) -> TestClient:
|
||||
with TestClient(app) as c:
|
||||
yield c
|
||||
|
||||
|
||||
def _start(client: TestClient) -> dict:
|
||||
resp = client.post(
|
||||
"/v1/defense/start", json={"learner_id": "defense-learner", "task_id": "defense-task"}
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
return resp.json()
|
||||
|
||||
|
||||
class TestStart:
|
||||
def test_start_returns_first_question_and_descriptor(self, client) -> None:
|
||||
body = _start(client)
|
||||
assert body["first_question"]
|
||||
assert body["defense_id"]
|
||||
assert body["voice_descriptor"]["mode"] == "mock"
|
||||
assert body["trace_complete"] is True
|
||||
stored = client.get(f"/v1/defense/{body['defense_id']}")
|
||||
assert stored.status_code == 200
|
||||
turns = stored.json()["turns"]
|
||||
assert turns and turns[0]["role"] == "examiner"
|
||||
|
||||
def test_start_with_unknown_trace_is_complete_flag(self, client) -> None:
|
||||
body = _start(client)
|
||||
assert body["trace_complete"] is True
|
||||
|
||||
def test_same_millisecond_starts_do_not_collide(self, client) -> None:
|
||||
"""v0.4.7 (P1): the old `dfn-{ms:x}` mint collided on same-ms starts —
|
||||
the insert-only store's PK then 500'd the second start PERMANENTLY
|
||||
(the identity-resubmit P0-C1 class). Nonce+counter keeps ids unique
|
||||
within a process regardless of clock granularity."""
|
||||
first = _start(client)
|
||||
second = _start(client)
|
||||
third = _start(client)
|
||||
ids = {first["defense_id"], second["defense_id"], third["defense_id"]}
|
||||
assert len(ids) == 3, f"defense ids collided: {ids}"
|
||||
for did in ids:
|
||||
stored = client.get(f"/v1/defense/{did}")
|
||||
assert stored.status_code == 200, stored.text
|
||||
|
||||
|
||||
class TestBrowserFallback:
|
||||
def test_browser_mode_serves_browser_descriptor(self, tmp_path: Path) -> None:
|
||||
"""Must-Have #6: AI_VOICE_PROVIDER=browser → start returns the
|
||||
browser-native SR/TTS fallback descriptor (D-030), not 'mock'."""
|
||||
from ai_service.identity.store import SQLiteIdentityStore
|
||||
|
||||
from ..conftest import SUITE_LEARNERS, seed_verified_identity
|
||||
|
||||
application = create_app(
|
||||
Settings(
|
||||
provider="mock",
|
||||
voice_provider="browser",
|
||||
learner_allowlist=SUITE_LEARNERS,
|
||||
)
|
||||
)
|
||||
identity_store = SQLiteIdentityStore(db_path=tmp_path / "identity-b.db")
|
||||
seed_verified_identity(identity_store)
|
||||
application.state.identity_store = identity_store # G-9
|
||||
application.state.provider = ScriptedLLM()
|
||||
application.state.trace_store = SQLiteTraceStore(db_path=tmp_path / "t.db")
|
||||
application.state.variant_store = SQLiteVariantStore(db_path=tmp_path / "v.db")
|
||||
application.state.grade_store = SQLiteGradeStore(db_path=tmp_path / "g.db")
|
||||
application.state.trace_integrity = TraceIntegrityMap()
|
||||
application.state.defense_store = SQLiteDefenseStore(db_path=tmp_path / "d.db")
|
||||
application.state.voice_provider = MockVoiceProvider(["answer"])
|
||||
llm = ScriptedLLM()
|
||||
application.state.examiner_agent = ExaminerAgent(
|
||||
llm, Settings(provider="mock", voice_provider="browser")
|
||||
)
|
||||
with TestClient(application) as c:
|
||||
body = _start(c)
|
||||
assert body["voice_descriptor"]["mode"] == "browser"
|
||||
assert body["voice_descriptor"]["sr_available"]
|
||||
assert "SpeechRecognition" in body["voice_descriptor"]["hint"]
|
||||
|
||||
|
||||
class TestAnswer:
|
||||
def test_typed_answer_yields_followup_with_latency(self, client) -> None:
|
||||
defense_id = _start(client)["defense_id"]
|
||||
resp = client.post(
|
||||
f"/v1/defense/{defense_id}/answer", data={"text": "I fixed the loop."}
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["question"]
|
||||
assert body["turn_latency"]["llm_ms"] is not None
|
||||
|
||||
def test_audio_answer_transcribed_and_recorded(self, client) -> None:
|
||||
defense_id = _start(client)["defense_id"]
|
||||
wav_bytes = b"RIFF" + b"\x00" * 64
|
||||
resp = client.post(
|
||||
f"/v1/defense/{defense_id}/answer",
|
||||
files={"audio": ("answer.wav", wav_bytes, "audio/wav")},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
stored = client.get(f"/v1/defense/{defense_id}").json()
|
||||
learner_turns = [t for t in stored["turns"] if t["role"] == "learner"]
|
||||
assert learner_turns, "learner turn missing after audio answer"
|
||||
assert learner_turns[0]["text"] == "the fix was in the retry loop"
|
||||
|
||||
def test_empty_audio_is_422_not_500(self, client) -> None:
|
||||
"""Zero-byte upload must 422 before the provider call (a real
|
||||
provider would raise the same way the mock does — validate first)."""
|
||||
defense_id = _start(client)["defense_id"]
|
||||
resp = client.post(
|
||||
f"/v1/defense/{defense_id}/answer",
|
||||
files={"audio": ("answer.wav", b"", "audio/wav")},
|
||||
)
|
||||
assert resp.status_code == 422, resp.text
|
||||
stored = client.get(f"/v1/defense/{defense_id}").json()
|
||||
assert len(stored["turns"]) == 1 # nothing appended
|
||||
|
||||
def test_answer_after_finish_is_409(self, client) -> None:
|
||||
"""A sealed transcript is append-only-no-more: the endpoints own
|
||||
turn-vs-finalize sequencing (defense_store contract)."""
|
||||
defense_id = _start(client)["defense_id"]
|
||||
client.post(f"/v1/defense/{defense_id}/answer", data={"text": "a"})
|
||||
assert client.post(f"/v1/defense/{defense_id}/finish").status_code == 200
|
||||
resp = client.post(f"/v1/defense/{defense_id}/answer", data={"text": "late"})
|
||||
assert resp.status_code == 409, resp.text
|
||||
stored = client.get(f"/v1/defense/{defense_id}").json()
|
||||
assert len(stored["turns"]) == 3 # ex, lrn, ex — no post-finish turns
|
||||
|
||||
def test_neither_text_nor_audio_422(self, client) -> None:
|
||||
defense_id = _start(client)["defense_id"]
|
||||
resp = client.post(f"/v1/defense/{defense_id}/answer")
|
||||
assert resp.status_code == 422
|
||||
|
||||
def test_unknown_defense_404(self, client) -> None:
|
||||
resp = client.post("/v1/defense/dfn-nope/answer", data={"text": "hi"})
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
class TestAudioEndpoint:
|
||||
def test_examiner_turn_streams_wav(self, client) -> None:
|
||||
defense_id = _start(client)["defense_id"]
|
||||
resp = client.get(f"/v1/defense/{defense_id}/audio/0")
|
||||
assert resp.status_code == 200
|
||||
assert resp.content
|
||||
assert resp.headers["content-type"].startswith("audio/")
|
||||
|
||||
def test_unknown_turn_404(self, client) -> None:
|
||||
defense_id = _start(client)["defense_id"]
|
||||
assert client.get(f"/v1/defense/{defense_id}/audio/42").status_code == 404
|
||||
|
||||
|
||||
class TestFinishAndGet:
|
||||
def test_full_loop_verdict_and_signals(self, client) -> None:
|
||||
defense_id = _start(client)["defense_id"]
|
||||
client.post(f"/v1/defense/{defense_id}/answer", data={"text": "answer one"})
|
||||
finish = client.post(f"/v1/defense/{defense_id}/finish")
|
||||
assert finish.status_code == 200, finish.text
|
||||
body = finish.json()
|
||||
assert body["verdict"]["verdict"] == "developing"
|
||||
assert body["integrity_signals"]["pause_threshold_ms"]
|
||||
stored = client.get(f"/v1/defense/{defense_id}").json()
|
||||
assert stored["status"] == "finished"
|
||||
assert stored["integrity_signals"]
|
||||
# Must-Have #1: "verdict + transcript persisted" — the verdict must
|
||||
# be retrievable from GET after finish, not only in the finish body.
|
||||
assert stored["integrity_signals"]["verdict"]["verdict"] == "developing"
|
||||
|
||||
def test_long_pause_flagged(self, client, app) -> None:
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from ai_service.voice.defense_store import DefenseTurn
|
||||
|
||||
defense_id = _start(client)["defense_id"]
|
||||
# inject a slow learner turn directly (simulated latency)
|
||||
store = app.state.defense_store
|
||||
store.append_turn(
|
||||
defense_id,
|
||||
DefenseTurn(
|
||||
defense_id=defense_id,
|
||||
seq=99,
|
||||
role="learner",
|
||||
text="slow reply",
|
||||
ts=datetime.now(UTC),
|
||||
latency_ms=30_000,
|
||||
created_at=datetime.now(UTC),
|
||||
),
|
||||
)
|
||||
finish = client.post(f"/v1/defense/{defense_id}/finish")
|
||||
assert finish.status_code == 200
|
||||
signals = finish.json()["integrity_signals"]
|
||||
assert any(p["turn"] == 99 for p in signals["long_pauses"])
|
||||
|
||||
def test_unknown_defense_404_on_all(self, client) -> None:
|
||||
assert client.post("/v1/defense/dfn-nope/finish").status_code == 404
|
||||
assert client.get("/v1/defense/dfn-nope").status_code == 404
|
||||
|
||||
|
||||
class TestServerVoiceRouteFixes:
|
||||
"""MH-2c (D-041/G-16/G-12): codec-strip, size guard, format-aware TTS."""
|
||||
|
||||
def test_webm_codec_params_stripped_for_provider(self, client, app) -> None:
|
||||
"""MediaRecorder sends 'audio/webm;codecs=opus' — the provider must
|
||||
see the bare 'webm' (D-041), else a real STT endpoint 400s."""
|
||||
received_fmts: list[str] = []
|
||||
|
||||
class ProbeVoice(MockVoiceProvider):
|
||||
async def transcribe(self, audio: bytes, fmt: str):
|
||||
received_fmts.append(fmt)
|
||||
return await super().transcribe(audio, fmt)
|
||||
|
||||
app.state.voice_provider = ProbeVoice(["clean fmt seen"])
|
||||
defense_id = _start(client)["defense_id"]
|
||||
resp = client.post(
|
||||
f"/v1/defense/{defense_id}/answer",
|
||||
files={"audio": ("answer.webm", b"\x1a\x45\xa3\xdf", "audio/webm;codecs=opus")},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert received_fmts == ["webm"], (
|
||||
f"codec params leaked to the provider: {received_fmts}"
|
||||
)
|
||||
assert resp.json()["question"]
|
||||
|
||||
def test_oversize_audio_413_before_provider_call(self, client, app) -> None:
|
||||
"""G-12/D-041: the guard fires before any provider call — the client
|
||||
renders an honest re-record prompt."""
|
||||
called = {"n": 0}
|
||||
|
||||
class ProbeVoice(MockVoiceProvider):
|
||||
async def transcribe(self, audio: bytes, fmt: str):
|
||||
called["n"] += 1
|
||||
return await super().transcribe(audio, fmt)
|
||||
|
||||
app.state.voice_provider = ProbeVoice(["x"])
|
||||
defense_id = _start(client)["defense_id"]
|
||||
settings = Settings()
|
||||
too_big = b"\x00" * (settings.voice_max_audio_mb * 1024 * 1024 + 1)
|
||||
resp = client.post(
|
||||
f"/v1/defense/{defense_id}/answer",
|
||||
files={"audio": ("answer.webm", too_big, "audio/webm")},
|
||||
)
|
||||
assert resp.status_code == 413
|
||||
assert "re-record" in resp.json()["detail"]
|
||||
assert called["n"] == 0, "provider must not be called for oversize audio"
|
||||
|
||||
def test_tts_media_type_maps_from_settings_enum(self, tmp_path: Path) -> None:
|
||||
"""G-16: media_type follows voice_tts_format (was hardcoded wav)."""
|
||||
from ai_service.identity.store import SQLiteIdentityStore
|
||||
|
||||
from ..conftest import SUITE_LEARNERS, seed_verified_identity
|
||||
|
||||
application = create_app(
|
||||
Settings(
|
||||
provider="mock",
|
||||
voice_provider="mock",
|
||||
voice_tts_format="opus",
|
||||
learner_allowlist=SUITE_LEARNERS,
|
||||
)
|
||||
)
|
||||
identity_store = SQLiteIdentityStore(db_path=tmp_path / "identity-opus.db")
|
||||
seed_verified_identity(identity_store)
|
||||
application.state.identity_store = identity_store # G-9
|
||||
application.state.provider = ScriptedLLM()
|
||||
application.state.trace_store = SQLiteTraceStore(db_path=tmp_path / "t.db")
|
||||
application.state.variant_store = SQLiteVariantStore(db_path=tmp_path / "v.db")
|
||||
application.state.grade_store = SQLiteGradeStore(db_path=tmp_path / "g.db")
|
||||
application.state.trace_integrity = TraceIntegrityMap()
|
||||
application.state.defense_store = SQLiteDefenseStore(db_path=tmp_path / "d.db")
|
||||
application.state.examiner_agent = ExaminerAgent(
|
||||
application.state.provider,
|
||||
Settings(provider="mock", voice_provider="mock"),
|
||||
)
|
||||
with TestClient(application) as c:
|
||||
defense_id = _start(c)["defense_id"]
|
||||
resp = c.get(f"/v1/defense/{defense_id}/audio/0")
|
||||
assert resp.status_code == 200
|
||||
assert resp.headers["content-type"].startswith("audio/opus"), (
|
||||
resp.headers["content-type"]
|
||||
)
|
||||
|
||||
def test_stt_provider_failure_is_502_never_500(self, client, app) -> None:
|
||||
"""Cross-phase P0 regression (final review): a provider failure on
|
||||
the audio-answer path (real endpoint outage — or the DEFAULT mock
|
||||
provider's unscripted queue, which every default-configured
|
||||
deployment hits on its first audio answer) must surface as an
|
||||
honest 502 per the assessment/proctor house pattern — never an
|
||||
unhandled 500. The transcript stays unaffected (typed answers
|
||||
still work)."""
|
||||
# Unscripted mock = exactly what create_app wires on default settings.
|
||||
app.state.voice_provider = MockVoiceProvider()
|
||||
defense_id = _start(client)["defense_id"]
|
||||
resp = client.post(
|
||||
f"/v1/defense/{defense_id}/answer",
|
||||
files={"audio": ("answer.webm", b"\x1a\x45\xa3\xdf" * 64, "audio/webm")},
|
||||
)
|
||||
assert resp.status_code == 502, resp.text
|
||||
assert "transcription failed" in resp.json()["detail"]
|
||||
# the defense is still alive for typed answers (no poisoned state)
|
||||
typed = client.post(f"/v1/defense/{defense_id}/answer", data={"text": "typed"})
|
||||
assert typed.status_code == 200, typed.text
|
||||
@@ -0,0 +1,227 @@
|
||||
"""Full credential-flow E2E over real engines (Task 6-5-01, REQ-3-007/008).
|
||||
|
||||
Endpoint-level end-to-end with mock LLM/voice providers (G-2 precedent:
|
||||
real engine plumbing over real endpoints; provider choice is
|
||||
service-internal): variant -> telemetry-wired sandbox -> real in-sandbox
|
||||
exec -> trace -> grade -> oral defense -> verdict/signals -> proctor.
|
||||
No corpus fixture anywhere in the flow.
|
||||
|
||||
Probe-guarded for user namespaces (the in-sandbox exec needs them).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import json
|
||||
import socket
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import uvicorn
|
||||
|
||||
from ai_service.config import Settings
|
||||
from ai_service.grading.store import SQLiteGradeStore
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.main import create_app
|
||||
from ai_service.telemetry.ingest import TraceIntegrityMap
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
from ai_service.variants.store import SQLiteVariantStore
|
||||
from ai_service.voice.defense_store import SQLiteDefenseStore
|
||||
from tests.sandbox.test_isolation import USERSNS_AVAILABLE
|
||||
|
||||
COACHING_JSON = json.dumps(
|
||||
{
|
||||
"summary": "Strong iteration.",
|
||||
"strengths": ["Tests early."],
|
||||
"gaps": ["One edge case missing."],
|
||||
"next_steps": ["Add it."],
|
||||
}
|
||||
)
|
||||
VERDICT_JSON = json.dumps(
|
||||
{
|
||||
"verdict": "developing",
|
||||
"understanding": "Explains the build clearly.",
|
||||
"process_justification": "Choices defended.",
|
||||
"communication": "Clear.",
|
||||
"strengths": ["Grounded in the digest."],
|
||||
"gaps": ["Missed one edge case."],
|
||||
}
|
||||
)
|
||||
PROCTOR_JSON = json.dumps(
|
||||
{
|
||||
"signals": [
|
||||
{"signal_type": "idle_gap", "severity": "low", "note": "A short pause."}
|
||||
],
|
||||
"intervention": "Keep momentum.",
|
||||
"summary": "Healthy session.",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class FlowLLM(MockProvider):
|
||||
"""Prompt-discriminated: verdict vs coaching vs question vs proctor JSON."""
|
||||
|
||||
def _reply_for(self, messages, response_format):
|
||||
all_text = "\n".join(m.content for m in messages)
|
||||
if response_format is not None and response_format.get("type") == "json_object":
|
||||
if "final verdict JSON" in all_text:
|
||||
return VERDICT_JSON
|
||||
if "rubric" in all_text.lower() and "Score this build session" in all_text:
|
||||
return json.dumps(
|
||||
{
|
||||
"criteria": {
|
||||
"process_quality": 4,
|
||||
"correctness": 3,
|
||||
"debugging_discipline": 3,
|
||||
"test_usage": 4,
|
||||
},
|
||||
"strengths": ["Iterated with tests."],
|
||||
"gaps": ["One edge case missing."],
|
||||
"verdict": "developing",
|
||||
}
|
||||
)
|
||||
if "Explain it as coaching" in all_text:
|
||||
return COACHING_JSON
|
||||
if "integrity signals supportively" in all_text:
|
||||
return PROCTOR_JSON
|
||||
return json.dumps(
|
||||
{
|
||||
"statement": (
|
||||
"Build a judge for code-review answers scoring factual "
|
||||
"accuracy with 3 edge cases and 5 test examples."
|
||||
)
|
||||
}
|
||||
)
|
||||
return "Walk me through your last fix — what changed and why?"
|
||||
|
||||
|
||||
def _free_port() -> int:
|
||||
with socket.socket() as s:
|
||||
s.bind(("127.0.0.1", 0))
|
||||
return s.getsockname()[1]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_full_credential_flow(tmp_path: Path) -> None:
|
||||
if not USERSNS_AVAILABLE:
|
||||
pytest.skip("user namespaces unavailable on this host (probe)")
|
||||
|
||||
import httpx
|
||||
|
||||
port = _free_port()
|
||||
from ai_service.identity.store import SQLiteIdentityStore
|
||||
from tests.conftest import SUITE_LEARNERS, seed_verified_identity
|
||||
|
||||
settings = Settings(
|
||||
provider="mock",
|
||||
voice_provider="mock",
|
||||
port=port,
|
||||
learner_allowlist=SUITE_LEARNERS,
|
||||
)
|
||||
app = create_app(settings)
|
||||
identity_store = SQLiteIdentityStore(db_path=tmp_path / "identity.db")
|
||||
seed_verified_identity(identity_store)
|
||||
app.state.identity_store = identity_store # G-9
|
||||
app.state.provider = FlowLLM()
|
||||
app.state.trace_store = SQLiteTraceStore(db_path=tmp_path / "t.db")
|
||||
app.state.grade_store = SQLiteGradeStore(db_path=tmp_path / "g.db")
|
||||
app.state.variant_store = SQLiteVariantStore(db_path=tmp_path / "v.db")
|
||||
app.state.defense_store = SQLiteDefenseStore(db_path=tmp_path / "d.db")
|
||||
app.state.trace_integrity = TraceIntegrityMap()
|
||||
|
||||
server = uvicorn.Server(uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning"))
|
||||
serve_task = asyncio.get_running_loop().create_task(server.serve())
|
||||
try:
|
||||
for _ in range(100):
|
||||
if server.started:
|
||||
break
|
||||
await asyncio.sleep(0.1)
|
||||
assert server.started
|
||||
|
||||
base = f"http://127.0.0.1:{port}"
|
||||
async with httpx.AsyncClient(base_url=base, timeout=30.0) as client:
|
||||
# 1. Variant (real seeded generation, mock-rendered).
|
||||
var = (await client.post("/v1/variants", json={
|
||||
"learner_id": "pilot-learner", "competency_id": "stack-orchestration-c007",
|
||||
})).json()
|
||||
assert var["task_id"] and var["statement"] and var["starter_files"]
|
||||
task_id = var["task_id"]
|
||||
|
||||
# 2. Telemetry-wired sandbox (real namespaces; agent joins via loopback).
|
||||
sbx = (await client.post("/v1/sandboxes", json={
|
||||
"learner_id": "pilot-learner", "task_id": task_id,
|
||||
}))
|
||||
assert sbx.status_code == 201, sbx.text
|
||||
sandbox_id = sbx.json()["id"]
|
||||
|
||||
# 3. Real in-sandbox exec: run the starter test (capture agent streams
|
||||
# the workspace effects into the trace).
|
||||
exec_resp = await client.post(f"/v1/sandboxes/{sandbox_id}/exec", json={
|
||||
"cmd": ["pytest", "-q"],
|
||||
})
|
||||
assert exec_resp.status_code == 200, exec_resp.text
|
||||
|
||||
# 4. Trace: events landed in order (the capture agent runs async).
|
||||
trace: list = []
|
||||
deadline = time.monotonic() + 20.0
|
||||
while time.monotonic() < deadline:
|
||||
tr = await client.get(f"/v1/telemetry/traces/pilot-learner/{task_id}")
|
||||
if tr.status_code == 200:
|
||||
trace = tr.json().get("events", [])
|
||||
if trace:
|
||||
break
|
||||
await asyncio.sleep(0.25)
|
||||
assert trace, "no telemetry events arrived from the real sandbox"
|
||||
seqs = [e["seq"] for e in trace]
|
||||
assert seqs == sorted(seqs)
|
||||
|
||||
# 5. Grade: rubric from the real digest (G-4 gate passed: no gaps).
|
||||
grade = (await client.post("/v1/assessment/grade", json={
|
||||
"learner_id": "pilot-learner", "task_id": task_id,
|
||||
})).json()
|
||||
assert grade["verdict"] == "GRADED", grade
|
||||
assert grade["scores"]["criteria"]["process_quality"] == 4
|
||||
assert grade["variant_seed"] == var["seed"] # D-029 stamped
|
||||
|
||||
# 6. Assessor coaching FROM the stored grade.
|
||||
coaching = (await client.post("/v1/assessment/evaluate", json={
|
||||
"learner_id": "pilot-learner", "task_id": task_id,
|
||||
})).json()
|
||||
assert coaching["coaching"]["summary"]
|
||||
|
||||
# 7. Oral defense: start -> typed answers -> finish (mock voice).
|
||||
defense = (await client.post("/v1/defense/start", json={
|
||||
"learner_id": "pilot-learner", "task_id": task_id,
|
||||
})).json()
|
||||
assert defense["first_question"]
|
||||
did = defense["defense_id"]
|
||||
ans = await client.post(f"/v1/defense/{did}/answer", data={"text": "I fixed the loop."})
|
||||
assert ans.status_code == 200, ans.text
|
||||
finish = (await client.post(f"/v1/defense/{did}/finish")).json()
|
||||
assert finish["verdict"]["verdict"] == "developing"
|
||||
assert finish["integrity_signals"]
|
||||
|
||||
# 8. Proctor over the real digest + defense signals.
|
||||
proctor = (await client.post("/v1/proctor/signals", json={
|
||||
"learner_id": "pilot-learner", "task_id": task_id,
|
||||
})).json()
|
||||
assert proctor["intervention"]
|
||||
|
||||
# 9. Sandbox destroyed; no leaks.
|
||||
destroy = await client.delete(f"/v1/sandboxes/{sandbox_id}")
|
||||
assert destroy.status_code == 204
|
||||
listed = (await client.get("/v1/sandboxes")).json()
|
||||
assert all(s["id"] != sandbox_id for s in (listed.get("sandboxes") or []))
|
||||
|
||||
# 10. No corpus fixtures anywhere in this flow's payloads.
|
||||
corpus_markers = ("lab-scenario", "proctor-scenario", "artifact-")
|
||||
for payload in (var, grade, defense, finish, proctor):
|
||||
assert not any(
|
||||
m in json.dumps(payload) for m in corpus_markers
|
||||
), "corpus fixture leaked into the learner path"
|
||||
finally:
|
||||
server.should_exit = True
|
||||
with contextlib.suppress(Exception):
|
||||
await asyncio.wait_for(serve_task, timeout=10.0)
|
||||
@@ -0,0 +1,397 @@
|
||||
"""Assessment grade endpoint tests — grading engine over HTTP (Task 3-3-01).
|
||||
|
||||
Contract under test (api/assessment.py, REQ-3-004):
|
||||
|
||||
POST /v1/assessment/grade {learner_id, task_id}
|
||||
GRADED → 200, rubric scores + digest summary
|
||||
UNGRADABLE_TRACE_INCOMPLETE → 200, gate record (missing_seqs /
|
||||
integrity_flag surfaced in scores)
|
||||
UNGRADABLE_EMPTY_TRACE → 200, gate record (documented choice: an
|
||||
unknown task is ALSO an empty pair; the
|
||||
engine cannot distinguish, and the gate
|
||||
outcome is a durable first-class result)
|
||||
StructuredOutputError → 502 (provider exhausted the D-020 budget)
|
||||
GET /v1/assessment/grade/{learner_id}/{task_id}
|
||||
stored latest grade → 200 (same scores as the POST)
|
||||
never-graded pair → 404
|
||||
|
||||
Wiring: per-test tmp-path SQLite stores + a pre-set GradingEngine
|
||||
(state-injection override — the lifespan adopts trace_store/grade_store/
|
||||
trace_integrity/grading_engine from app.state instead of constructing
|
||||
them; same pattern as test_telemetry_ingest.py / test_sandboxes.py). The
|
||||
engine binds a scripted provider so each test controls the LLM exactly,
|
||||
and counts calls so gate tests can assert the LLM was never reached
|
||||
(G-4 holds through the whole HTTP stack).
|
||||
|
||||
Zero network: providers are MockProvider subclasses only (conftest rule).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from ai_service.config import Settings
|
||||
from ai_service.corpus.trace_fixtures import STRONG_BUILDER
|
||||
from ai_service.grading.engine import GradingEngine
|
||||
from ai_service.grading.store import SQLiteGradeStore
|
||||
from ai_service.llm.mock import MockProvider, ScriptedJSONProvider
|
||||
from ai_service.main import create_app
|
||||
from ai_service.telemetry.ingest import TraceIntegrityMap
|
||||
from ai_service.telemetry.models import TelemetryEvent
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
|
||||
LEARNER = "grade-learner"
|
||||
TASK = "task-grade-1"
|
||||
T0 = datetime(2026, 9, 12, 3, 0, 0, tzinfo=UTC)
|
||||
|
||||
#: Canonical well-formed rubric payload (matches grading.engine.RubricScore).
|
||||
RUBRIC_PAYLOAD: dict = {
|
||||
"criteria": {
|
||||
"process_quality": 4,
|
||||
"correctness": 4,
|
||||
"debugging_discipline": 3,
|
||||
"test_usage": 4,
|
||||
},
|
||||
"strengths": ["tight edit-test loops throughout"],
|
||||
"gaps": ["final commit discipline loose"],
|
||||
"verdict": "mastered",
|
||||
}
|
||||
|
||||
|
||||
class CountingScriptedProvider(ScriptedJSONProvider):
|
||||
"""ScriptedJSONProvider that counts chat() calls.
|
||||
|
||||
Gate tests assert calls == 0 (the LLM is never reached through the
|
||||
whole HTTP stack — G-4); happy-path tests sanity-check calls >= 1.
|
||||
"""
|
||||
|
||||
def __init__(self, payload: dict) -> None:
|
||||
super().__init__(payload)
|
||||
self.calls = 0
|
||||
|
||||
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
self.calls += 1
|
||||
return await super().chat(
|
||||
messages, model=model, temperature=temperature, response_format=response_format
|
||||
)
|
||||
|
||||
|
||||
def _event(seq: int, kind: str, payload: dict | None = None, offset_s: float = 0.0):
|
||||
return TelemetryEvent(
|
||||
learner_id=LEARNER,
|
||||
task_id=TASK,
|
||||
seq=seq,
|
||||
kind=kind,
|
||||
payload=payload or {},
|
||||
ts=T0 + timedelta(seconds=offset_s),
|
||||
sandbox_id="sbx-grade",
|
||||
)
|
||||
|
||||
|
||||
def _complete_trace() -> list[TelemetryEvent]:
|
||||
"""Contiguous seq 0..6 — a gradeable trace (gap-free, unflagged)."""
|
||||
return [
|
||||
_event(0, "activity", {"state": "starting"}, 0.0),
|
||||
_event(1, "file_diff", {"path": "a.py", "added": 12}, 10.0),
|
||||
_event(2, "command", {"cmd": "pytest -q"}, 20.0),
|
||||
_event(3, "test_result", {"passed": False, "exit_code": 1}, 25.0),
|
||||
_event(4, "file_diff", {"path": "a.py", "added": 4, "removed": 2}, 40.0),
|
||||
_event(5, "command", {"cmd": "pytest -q"}, 60.0),
|
||||
_event(6, "test_result", {"passed": True, "exit_code": 0}, 65.0),
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def trace_store(tmp_path: Path) -> Iterator[SQLiteTraceStore]:
|
||||
store = SQLiteTraceStore(db_path=tmp_path / "traces.db")
|
||||
yield store
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def grade_store(tmp_path: Path) -> Iterator[SQLiteGradeStore]:
|
||||
store = SQLiteGradeStore(db_path=tmp_path / "grades.db")
|
||||
yield store
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def integrity() -> TraceIntegrityMap:
|
||||
return TraceIntegrityMap()
|
||||
|
||||
|
||||
def _make_client(
|
||||
tmp_path: Path,
|
||||
trace_store: SQLiteTraceStore,
|
||||
grade_store: SQLiteGradeStore,
|
||||
integrity: TraceIntegrityMap,
|
||||
provider: MockProvider,
|
||||
) -> TestClient:
|
||||
"""App + TestClient with stores, integrity map and a pre-set engine.
|
||||
|
||||
The lifespan adopts every pre-set service (state-injection override);
|
||||
the engine binds OUR provider, so the fixture — not Settings — scripts
|
||||
the LLM. Cloud-free guard: the provider must be a MockProvider family
|
||||
member (conftest rule, enforced here because this module builds its
|
||||
own client rather than consuming the conftest one).
|
||||
"""
|
||||
assert isinstance(provider, MockProvider)
|
||||
settings = Settings(
|
||||
provider="mock",
|
||||
db_path=tmp_path / "grading-test.db",
|
||||
sandbox_dir=tmp_path / "sandboxes",
|
||||
)
|
||||
app = create_app(settings)
|
||||
app.state.trace_store = trace_store
|
||||
app.state.grade_store = grade_store
|
||||
app.state.trace_integrity = integrity
|
||||
app.state.grading_engine = GradingEngine(
|
||||
trace_store, grade_store, integrity, provider, model="gemma4:31b"
|
||||
)
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _seed(store: SQLiteTraceStore, events: list[TelemetryEvent]) -> None:
|
||||
for e in events:
|
||||
store.append(e)
|
||||
|
||||
|
||||
def _post(client: TestClient, learner: str = LEARNER, task: str = TASK):
|
||||
return client.post("/v1/assessment/grade", json={"learner_id": learner, "task_id": task})
|
||||
|
||||
|
||||
def _get(client: TestClient, learner: str = LEARNER, task: str = TASK):
|
||||
return client.get(f"/v1/assessment/grade/{learner}/{task}")
|
||||
|
||||
|
||||
# -- happy path: complete trace → GRADED -----------------------------------------
|
||||
|
||||
|
||||
class TestGraded:
|
||||
def test_post_complete_trace_returns_rubric_scores_and_digest(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
_seed(trace_store, _complete_trace())
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _post(client)
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["verdict"] == "GRADED"
|
||||
assert body["learner_id"] == LEARNER
|
||||
assert body["task_id"] == TASK
|
||||
assert body["scores"] == RUBRIC_PAYLOAD # rubric verdict rides in scores
|
||||
assert body["scores"]["verdict"] == "mastered"
|
||||
assert body["variant_seed"] is None # null until P4 (D-029)
|
||||
assert body["model"] == "gemma4:31b" # provenance travels
|
||||
assert provider.calls == 1 # happy path: exactly one LLM call
|
||||
# digest summary rides along (D-028 reproducible input)
|
||||
assert body["digest"]["event_count"] == 7
|
||||
assert body["digest"]["final_test_status"] == "pass"
|
||||
|
||||
def test_corpus_fixture_trace_grades(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
"""The strong-builder corpus fixture (real calibration trace) is
|
||||
gradeable over HTTP end to end."""
|
||||
_seed(trace_store, STRONG_BUILDER.events)
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _post(
|
||||
client, learner=STRONG_BUILDER.learner_id, task=STRONG_BUILDER.task_id
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["verdict"] == "GRADED"
|
||||
assert body["digest"]["event_count"] == len(STRONG_BUILDER.event_specs)
|
||||
|
||||
def test_get_after_post_returns_same_stored_scores(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
_seed(trace_store, _complete_trace())
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
posted = _post(client)
|
||||
assert posted.status_code == 200
|
||||
|
||||
fetched = _get(client)
|
||||
assert fetched.status_code == 200
|
||||
stored = fetched.json()
|
||||
|
||||
assert stored["scores"] == posted.json()["scores"]
|
||||
assert stored["verdict"] == "GRADED"
|
||||
assert stored["digest"] == posted.json()["digest"]
|
||||
assert stored["created_at"] == posted.json()["created_at"]
|
||||
|
||||
def test_post_regrade_upserts_get_returns_latest(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
"""POST twice → one row holding the LATEST grade (GradeStore upsert)."""
|
||||
_seed(trace_store, _complete_trace())
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
first = _post(client)
|
||||
assert first.status_code == 200
|
||||
assert first.json()["scores"]["verdict"] == "mastered"
|
||||
|
||||
# the same provider now scripts a different (weaker) rubric
|
||||
updated = {
|
||||
**RUBRIC_PAYLOAD,
|
||||
"criteria": {**RUBRIC_PAYLOAD["criteria"], "process_quality": 1},
|
||||
"verdict": "not_yet",
|
||||
}
|
||||
provider.payload = updated
|
||||
second = _post(client)
|
||||
assert second.status_code == 200
|
||||
assert second.json()["scores"] == updated
|
||||
|
||||
# GET returns the LATEST grade, not the first
|
||||
fetched = _get(client)
|
||||
assert fetched.json()["scores"] == updated
|
||||
assert fetched.json()["scores"]["verdict"] == "not_yet"
|
||||
|
||||
# one row, latest wins (upsert, not append)
|
||||
grades = grade_store.list_for_learner(LEARNER)
|
||||
assert len(grades) == 1
|
||||
assert grades[0].scores == updated
|
||||
|
||||
|
||||
# -- G-4 gate outcomes over HTTP: 200 with the ungradable record -------------------
|
||||
|
||||
|
||||
class TestGateOutcomes:
|
||||
def test_gapped_trace_200_ungradable_incomplete_with_missing_seqs(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
_seed(trace_store, [e for e in _complete_trace() if e.seq != 2]) # seq 2 missing
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _post(client)
|
||||
|
||||
assert response.status_code == 200 # ungradable IS a valid result — not 5xx
|
||||
body = response.json()
|
||||
assert body["verdict"] == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
assert body["scores"]["missing_seqs"] == [2]
|
||||
assert body["scores"]["integrity_flag"] is None
|
||||
assert body["digest"] == {} # nothing was graded
|
||||
assert body["model"] == "none" # no LLM involved (honest provenance)
|
||||
assert provider.calls == 0, "LLM was called despite a gapped trace (G-4)"
|
||||
|
||||
def test_flooded_trace_200_ungradable_incomplete_with_integrity_reason(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
"""Integrity-flagged trace (G-3 INCOMPLETE_FLOODED) surfaces the flag
|
||||
reason in scores — the same verdict as gaps, different detail."""
|
||||
_seed(trace_store, _complete_trace())
|
||||
integrity.mark(LEARNER, TASK, "INCOMPLETE_FLOODED")
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _post(client)
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["verdict"] == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
assert body["scores"]["integrity_flag"] == "INCOMPLETE_FLOODED"
|
||||
assert body["scores"]["missing_seqs"] == [] # rows complete but untrusted
|
||||
assert provider.calls == 0, "LLM was called despite an integrity flag (G-4)"
|
||||
|
||||
def test_empty_trace_200_ungradable_empty(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
"""Zero events for the pair (including a task that never had a
|
||||
trace) → UNGRADABLE_EMPTY_TRACE, persisted like any grade."""
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _post(client, learner="ghost-learner", task="never-started")
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["verdict"] == "UNGRADABLE_EMPTY_TRACE"
|
||||
assert body["scores"] == {"integrity_flag": None, "missing_seqs": []}
|
||||
assert body["digest"] == {}
|
||||
assert body["model"] == "none"
|
||||
assert provider.calls == 0
|
||||
|
||||
# the gate record is durable: GET now returns it (no longer 404)
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
fetched = _get(client, learner="ghost-learner", task="never-started")
|
||||
assert fetched.status_code == 200
|
||||
assert fetched.json()["verdict"] == "UNGRADABLE_EMPTY_TRACE"
|
||||
|
||||
|
||||
# -- provider failure → 502 --------------------------------------------------------
|
||||
|
||||
|
||||
class TestProviderFailure:
|
||||
def test_persistent_malformed_llm_output_502(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
"""Stock MockProvider: its json_object reply is wrong-shaped, so the
|
||||
D-020 defense exhausts both attempts → endpoint maps to 502 and
|
||||
nothing is persisted."""
|
||||
_seed(trace_store, _complete_trace())
|
||||
provider = MockProvider() # always malformed for the rubric schema
|
||||
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _post(client)
|
||||
|
||||
assert response.status_code == 502
|
||||
assert "grading failed" in response.json()["detail"]
|
||||
# no fabricated/partial record was persisted
|
||||
assert grade_store.get(LEARNER, TASK) is None
|
||||
|
||||
def test_502_leaves_earlier_grade_intact(
|
||||
self, tmp_path, trace_store, grade_store, integrity
|
||||
):
|
||||
"""A failed regrade must not clobber the previously stored grade."""
|
||||
_seed(trace_store, _complete_trace())
|
||||
scripted = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, scripted) as client:
|
||||
assert _post(client).status_code == 200
|
||||
|
||||
# regrade attempt hits a provider that now always fails
|
||||
broken = MockProvider()
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, broken) as client:
|
||||
assert _post(client).status_code == 502
|
||||
fetched = _get(client)
|
||||
|
||||
assert fetched.status_code == 200 # earlier grade still readable
|
||||
assert fetched.json()["scores"] == RUBRIC_PAYLOAD
|
||||
|
||||
|
||||
# -- stored-grade reads ------------------------------------------------------------
|
||||
|
||||
|
||||
class TestStoredGradeReads:
|
||||
def test_get_unknown_pair_404(self, tmp_path, trace_store, grade_store, integrity):
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
response = _get(client, learner="nobody", task="never-graded")
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "no stored grade" in response.json()["detail"]
|
||||
|
||||
def test_missing_body_fields_422(self, tmp_path, trace_store, grade_store, integrity):
|
||||
provider = CountingScriptedProvider(RUBRIC_PAYLOAD)
|
||||
with _make_client(tmp_path, trace_store, grade_store, integrity, provider) as client:
|
||||
assert client.post("/v1/assessment/grade", json={}).status_code == 422
|
||||
assert (
|
||||
client.post(
|
||||
"/v1/assessment/grade", json={"learner_id": LEARNER}
|
||||
).status_code
|
||||
== 422
|
||||
)
|
||||
@@ -1,47 +1,86 @@
|
||||
"""Lab feedback endpoint tests — SSE envelope with agent=lab (REQ-2-007)."""
|
||||
"""Lab endpoint tests — LIVE trace contract (REQ-3-007).
|
||||
|
||||
import json
|
||||
v0.3 re-grounding: POST /v1/lab/feedback takes {learner_id, task_id}; the
|
||||
digest is computed from the learner's real TraceStore events. No corpus
|
||||
scenarios; empty trace is a valid "no telemetry yet" coaching path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from ai_service.config import Settings
|
||||
from ai_service.grading.store import SQLiteGradeStore
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.main import create_app
|
||||
from ai_service.telemetry.ingest import TraceIntegrityMap
|
||||
from ai_service.telemetry.models import TelemetryEvent
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
|
||||
T0 = datetime(2026, 9, 12, tzinfo=UTC)
|
||||
|
||||
|
||||
def stream_events(client, payload) -> list[dict]:
|
||||
with client.stream("POST", "/v1/lab/feedback", json=payload) as response:
|
||||
assert response.status_code == 200
|
||||
events = []
|
||||
for line in response.iter_lines():
|
||||
if line.startswith("data:"):
|
||||
d = line.removeprefix("data:").strip()
|
||||
if d == "[DONE]":
|
||||
events.append({"type": "[DONE]"})
|
||||
else:
|
||||
events.append(json.loads(d))
|
||||
return events
|
||||
def _event(seq: int, kind: str, payload: dict, offset_s: float):
|
||||
return TelemetryEvent(
|
||||
learner_id="lab-learner",
|
||||
task_id="lab-task",
|
||||
seq=seq,
|
||||
kind=kind,
|
||||
payload=payload,
|
||||
ts=T0 + timedelta(seconds=offset_s),
|
||||
sandbox_id="sbx-lab",
|
||||
)
|
||||
|
||||
|
||||
def test_lab_feedback_streams_full_envelope(client):
|
||||
events = stream_events(client, {"scenario_id": "lab-scenario-strong"})
|
||||
assert events[0]["type"] == "meta"
|
||||
assert events[0]["agent"] == "lab"
|
||||
assert events[0]["scenario_id"] == "lab-scenario-strong"
|
||||
deltas = [e for e in events if e["type"] == "delta"]
|
||||
assert len(deltas) >= 1
|
||||
assert any(e["type"] == "done" for e in events)
|
||||
assert events[-1]["type"] == "[DONE]"
|
||||
class StreamingMock(MockProvider):
|
||||
"""Deterministic token stream for the SSE path."""
|
||||
|
||||
def _reply_for(self, messages, response_format):
|
||||
return "Feedback grounded in your live session digest."
|
||||
|
||||
|
||||
def test_unknown_scenario_404(client):
|
||||
response = client.post("/v1/lab/feedback", json={"scenario_id": "nope"})
|
||||
assert response.status_code == 404
|
||||
assert "nope" in response.json()["detail"]
|
||||
@pytest.fixture()
|
||||
def client(tmp_path: Path) -> TestClient:
|
||||
app = create_app(Settings(provider="mock"))
|
||||
app.state.provider = StreamingMock()
|
||||
app.state.trace_store = SQLiteTraceStore(db_path=tmp_path / "t.db")
|
||||
app.state.grade_store = SQLiteGradeStore(db_path=tmp_path / "g.db")
|
||||
app.state.trace_integrity = TraceIntegrityMap()
|
||||
with TestClient(app) as c:
|
||||
yield c
|
||||
|
||||
|
||||
def test_distinct_scenarios_distinct_replies(client):
|
||||
strong = stream_events(client, {"scenario_id": "lab-scenario-strong"})
|
||||
struggling = stream_events(client, {"scenario_id": "lab-scenario-struggling"})
|
||||
strong_text = "".join(e["content"] for e in strong if e["type"] == "delta")
|
||||
struggling_text = "".join(e["content"] for e in struggling if e["type"] == "delta")
|
||||
assert strong_text != struggling_text
|
||||
def test_live_trace_streams_full_envelope(client: TestClient) -> None:
|
||||
store: SQLiteTraceStore = client.app.state.trace_store
|
||||
for e in [
|
||||
_event(0, "file_diff", {"path": "x.py"}, 0),
|
||||
_event(1, "command", {"cmd": "pytest -q"}, 10),
|
||||
_event(2, "test_result", {"passed": False, "exit_code": 1}, 20),
|
||||
_event(3, "test_result", {"passed": True, "exit_code": 0}, 40),
|
||||
]:
|
||||
store.append(e)
|
||||
with client.stream(
|
||||
"POST", "/v1/lab/feedback", json={"learner_id": "lab-learner", "task_id": "lab-task"}
|
||||
) as resp:
|
||||
assert resp.status_code == 200
|
||||
body = "".join(chunk.decode() for chunk in resp.iter_raw())
|
||||
assert '"agent": "lab"' in body or '"agent":"lab"' in body
|
||||
assert '"task_id": "lab-task"' in body
|
||||
assert '"type": "delta"' in body or '"type":"delta"' in body
|
||||
|
||||
|
||||
def test_missing_scenario_id_422(client):
|
||||
response = client.post("/v1/lab/feedback", json={})
|
||||
assert response.status_code == 422
|
||||
def test_empty_trace_coaches_the_baseline(client: TestClient) -> None:
|
||||
"""No telemetry is NOT an error — Lab coaches 'run the starter test'."""
|
||||
with client.stream(
|
||||
"POST", "/v1/lab/feedback", json={"learner_id": "lab-learner", "task_id": "no-events"}
|
||||
) as resp:
|
||||
assert resp.status_code == 200
|
||||
|
||||
|
||||
def test_missing_fields_422(client: TestClient) -> None:
|
||||
resp = client.post("/v1/lab/feedback", json={"learner_id": "x"})
|
||||
assert resp.status_code == 422
|
||||
|
||||
@@ -1,49 +1,121 @@
|
||||
"""Proctor signals endpoint tests — validated JSON, 404s (REQ-2-009)."""
|
||||
"""Proctor signals endpoint tests — REAL inputs contract (REQ-3-007).
|
||||
|
||||
from ai_service.agents.proctor import ProctorAssessment
|
||||
from ai_service.llm.mock import ScriptedJSONProvider
|
||||
v0.3 re-grounding: POST /v1/proctor/signals takes {learner_id, task_id}
|
||||
and gathers digest + defense signals + variant context server-side.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from ai_service.config import Settings
|
||||
from ai_service.grading.store import SQLiteGradeStore
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.main import create_app
|
||||
from ai_service.telemetry.ingest import TraceIntegrityMap
|
||||
from ai_service.telemetry.models import TelemetryEvent
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
from ai_service.variants.store import SQLiteVariantStore
|
||||
from ai_service.voice.defense_store import SQLiteDefenseStore
|
||||
|
||||
VALID = {
|
||||
"scenario_id": "proctor-scenario-distracted",
|
||||
"signals": [
|
||||
{"signal_type": "context_switch", "severity": "low",
|
||||
"note": "Docs tab at t+120s is normal"},
|
||||
{"signal_type": "idle_gap", "severity": "medium",
|
||||
"note": "5-minute idle at t+300s"},
|
||||
{"signal_type": "idle_gap", "severity": "low", "note": "One long pause."},
|
||||
],
|
||||
"intervention": "Offer a short break, then restate the plan",
|
||||
"summary": "Coaching-shaped session note",
|
||||
}
|
||||
|
||||
|
||||
def test_signals_returns_validated_json(client):
|
||||
original = client.app.state.provider
|
||||
client.app.state.provider = ScriptedJSONProvider(VALID)
|
||||
try:
|
||||
response = client.post(
|
||||
"/v1/proctor/signals", json={"scenario_id": "proctor-scenario-distracted"}
|
||||
)
|
||||
finally:
|
||||
client.app.state.provider = original
|
||||
assert response.status_code == 200
|
||||
validated = ProctorAssessment.model_validate(response.json())
|
||||
assert validated.scenario_id == "proctor-scenario-distracted"
|
||||
assert validated.intervention
|
||||
T0 = datetime(2026, 9, 12, tzinfo=UTC)
|
||||
|
||||
|
||||
def test_unknown_scenario_404(client):
|
||||
response = client.post("/v1/proctor/signals", json={"scenario_id": "ghost"})
|
||||
assert response.status_code == 404
|
||||
class ProctorJSON(MockProvider):
|
||||
def _reply_for(self, messages, response_format):
|
||||
if response_format is not None and response_format.get("type") == "json_object":
|
||||
return json.dumps(VALID)
|
||||
return super()._reply_for(messages, response_format)
|
||||
|
||||
|
||||
def test_unparseable_provider_502(client):
|
||||
response = client.post(
|
||||
"/v1/proctor/signals", json={"scenario_id": "proctor-scenario-healthy"}
|
||||
class BrokenJSON(MockProvider):
|
||||
def _reply_for(self, messages, response_format):
|
||||
if response_format is not None and response_format.get("type") == "json_object":
|
||||
return "not json ever"
|
||||
return super()._reply_for(messages, response_format)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client(tmp_path: Path) -> TestClient:
|
||||
app = create_app(Settings(provider="mock"))
|
||||
app.state.trace_store = SQLiteTraceStore(db_path=tmp_path / "t.db")
|
||||
app.state.grade_store = SQLiteGradeStore(db_path=tmp_path / "g.db")
|
||||
app.state.variant_store = SQLiteVariantStore(db_path=tmp_path / "v.db")
|
||||
app.state.defense_store = SQLiteDefenseStore(db_path=tmp_path / "d.db")
|
||||
app.state.trace_integrity = TraceIntegrityMap()
|
||||
with TestClient(app) as c:
|
||||
yield c
|
||||
|
||||
|
||||
def _seed_trace(client: TestClient) -> None:
|
||||
store: SQLiteTraceStore = client.app.state.trace_store
|
||||
for e in [
|
||||
TelemetryEvent(
|
||||
learner_id="p-learner",
|
||||
task_id="p-task",
|
||||
seq=0,
|
||||
kind="activity",
|
||||
payload={"state": "idle"},
|
||||
ts=T0,
|
||||
sandbox_id="sbx-p",
|
||||
),
|
||||
TelemetryEvent(
|
||||
learner_id="p-learner",
|
||||
task_id="p-task",
|
||||
seq=1,
|
||||
kind="activity",
|
||||
payload={"state": "idle"},
|
||||
ts=T0 + timedelta(seconds=400),
|
||||
sandbox_id="sbx-p",
|
||||
),
|
||||
]:
|
||||
store.append(e)
|
||||
|
||||
|
||||
def test_signals_returns_validated_json(client: TestClient) -> None:
|
||||
client.app.state.provider = ProctorJSON()
|
||||
_seed_trace(client)
|
||||
resp = client.post(
|
||||
"/v1/proctor/signals", json={"learner_id": "p-learner", "task_id": "p-task"}
|
||||
)
|
||||
assert response.status_code == 502
|
||||
assert "failed" in response.json()["detail"].lower()
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["intervention"]
|
||||
assert body["signals"][0]["signal_type"] == "idle_gap"
|
||||
|
||||
|
||||
def test_missing_scenario_id_422(client):
|
||||
response = client.post("/v1/proctor/signals", json={})
|
||||
assert response.status_code == 422
|
||||
def test_empty_trace_is_valid_not_404(client: TestClient) -> None:
|
||||
"""No telemetry → the proctor still assesses (nothing to flag)."""
|
||||
client.app.state.provider = ProctorJSON()
|
||||
resp = client.post(
|
||||
"/v1/proctor/signals", json={"learner_id": "nobody", "task_id": "nothing"}
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
|
||||
|
||||
def test_unparseable_provider_502(client: TestClient) -> None:
|
||||
client.app.state.provider = BrokenJSON()
|
||||
_seed_trace(client)
|
||||
resp = client.post(
|
||||
"/v1/proctor/signals", json={"learner_id": "p-learner", "task_id": "p-task"}
|
||||
)
|
||||
assert resp.status_code == 502
|
||||
|
||||
|
||||
def test_missing_fields_422(client: TestClient) -> None:
|
||||
client.app.state.provider = ProctorJSON()
|
||||
resp = client.post("/v1/proctor/signals", json={"learner_id": "x"})
|
||||
assert resp.status_code == 422
|
||||
|
||||
@@ -43,9 +43,23 @@ BASE_SETTINGS: dict = {
|
||||
"sandbox_max_concurrent": 5,
|
||||
"sandbox_max_per_learner": 1,
|
||||
"sandbox_creates_per_min": 10,
|
||||
# G-9: allowlist widened to the suite roster (identity records seeded
|
||||
# per-app below); the gate tests live in test_identity.py.
|
||||
"learner_allowlist": __import__("tests.conftest", fromlist=["SUITE_LEARNERS"]).SUITE_LEARNERS,
|
||||
}
|
||||
|
||||
|
||||
def _seed_identity(app, tmp_path: Path) -> None:
|
||||
"""G-9: every ad-hoc app in this module gets the suite's verified
|
||||
identity store (allowlist widened via BASE_SETTINGS)."""
|
||||
from ai_service.identity.store import SQLiteIdentityStore
|
||||
from tests.conftest import seed_verified_identity
|
||||
|
||||
store = SQLiteIdentityStore(db_path=tmp_path / f"identity-{id(app):x}.db")
|
||||
seed_verified_identity(store)
|
||||
app.state.identity_store = store
|
||||
|
||||
|
||||
class StubBackend:
|
||||
"""Structural SandboxBackend: lays out the workdir, spawns nothing.
|
||||
|
||||
@@ -103,6 +117,7 @@ def client(
|
||||
monkeypatch.setenv("AI_SANDBOX_CREATES_PER_MIN", "150") # shared-window headroom
|
||||
settings = Settings(**{**BASE_SETTINGS, "sandbox_dir": tmp_path / "sandboxes"})
|
||||
app = create_app(settings)
|
||||
_seed_identity(app, tmp_path)
|
||||
app.state.sandbox_manager = SandboxManager(backend=stub_backend, settings=settings)
|
||||
with TestClient(app) as c:
|
||||
yield c
|
||||
@@ -190,6 +205,7 @@ def test_pool_full_returns_503(tmp_path: Path, stub_backend: StubBackend) -> Non
|
||||
sandbox_creates_per_min=150,
|
||||
)
|
||||
app = create_app(settings)
|
||||
_seed_identity(app, tmp_path)
|
||||
app.state.sandbox_manager = SandboxManager(backend=stub_backend, settings=settings)
|
||||
with TestClient(app) as client:
|
||||
ids = [
|
||||
@@ -227,6 +243,7 @@ def test_second_active_sandbox_for_same_learner_429(
|
||||
sandbox_creates_per_min=150,
|
||||
)
|
||||
app = create_app(settings)
|
||||
_seed_identity(app, tmp_path)
|
||||
app.state.sandbox_manager = SandboxManager(backend=stub_backend, settings=settings)
|
||||
with TestClient(app) as client:
|
||||
first = client.post("/v1/sandboxes", json={"learner_id": PILOT})
|
||||
@@ -252,6 +269,7 @@ def test_burst_over_global_create_rate_429(tmp_path: Path, stub_backend: StubBac
|
||||
sandbox_creates_per_min=3,
|
||||
)
|
||||
app = create_app(settings)
|
||||
_seed_identity(app, tmp_path)
|
||||
manager = SandboxManager(backend=stub_backend, settings=settings)
|
||||
app.state.sandbox_manager = manager
|
||||
with TestClient(app) as client:
|
||||
@@ -309,6 +327,7 @@ def test_lifespan_start_and_shutdown_destroy(
|
||||
)
|
||||
manager = SandboxManager(backend=stub_backend, settings=settings)
|
||||
app = create_app(settings)
|
||||
_seed_identity(app, tmp_path)
|
||||
app.state.sandbox_manager = manager
|
||||
with TestClient(app) as client:
|
||||
assert not orphan_root.exists() # startup reaper ran during lifespan boot
|
||||
@@ -325,6 +344,7 @@ def test_lifespan_constructs_real_manager_when_not_overridden(tmp_path: Path) ->
|
||||
"""No override → the lifespan builds the production UnshareBackend manager."""
|
||||
settings = Settings(provider="mock", sandbox_dir=tmp_path / "sandboxes")
|
||||
app = create_app(settings)
|
||||
_seed_identity(app, tmp_path)
|
||||
with TestClient(app):
|
||||
manager = app.state.sandbox_manager
|
||||
assert isinstance(manager, SandboxManager)
|
||||
@@ -347,6 +367,7 @@ def test_real_backend_create_path_runs(tmp_path: Path) -> None:
|
||||
sandbox_creates_per_min=150,
|
||||
)
|
||||
app = create_app(settings) # no override → lifespan wires UnshareBackend
|
||||
_seed_identity(app, tmp_path)
|
||||
try:
|
||||
with TestClient(app) as client:
|
||||
created = client.post("/v1/sandboxes", json={"learner_id": PILOT})
|
||||
@@ -359,3 +380,91 @@ def test_real_backend_create_path_runs(tmp_path: Path) -> None:
|
||||
assert (sandbox_root / sandbox_id / "workspace").is_dir()
|
||||
finally:
|
||||
shutil.rmtree(sandbox_root, ignore_errors=True)
|
||||
|
||||
|
||||
class TestFilesAndExecRoutes:
|
||||
"""Workspace CRUD + Run/Test exec (Phase 6, REQ-3-008, CUT-2)."""
|
||||
|
||||
def test_file_write_read_list_roundtrip(self, client):
|
||||
handle = client.post(
|
||||
"/v1/sandboxes", json={"learner_id": "pilot-learner"}
|
||||
).json()
|
||||
sbx = handle["id"]
|
||||
put = client.put(
|
||||
f"/v1/sandboxes/{sbx}/files/main.py",
|
||||
json={"path": "main.py", "content": "print('hi')"},
|
||||
)
|
||||
assert put.status_code == 200, put.text
|
||||
got = client.get(f"/v1/sandboxes/{sbx}/files/main.py")
|
||||
assert got.status_code == 200
|
||||
assert "print('hi')" in got.json()["content"]
|
||||
listed = client.get(f"/v1/sandboxes/{sbx}/files")
|
||||
assert "main.py" in listed.json()["files"]
|
||||
client.delete(f"/v1/sandboxes/{sbx}")
|
||||
|
||||
def test_traversal_rejected(self, client):
|
||||
handle = client.post(
|
||||
"/v1/sandboxes", json={"learner_id": "pilot-learner"}
|
||||
).json()
|
||||
sbx = handle["id"]
|
||||
bad = client.put(
|
||||
f"/v1/sandboxes/{sbx}/files/..%2Fescape.txt",
|
||||
json={"path": "../escape.txt", "content": "x"},
|
||||
)
|
||||
assert bad.status_code == 422
|
||||
client.delete(f"/v1/sandboxes/{sbx}")
|
||||
|
||||
def test_unknown_file_404(self, client):
|
||||
handle = client.post(
|
||||
"/v1/sandboxes", json={"learner_id": "pilot-learner"}
|
||||
).json()
|
||||
sbx = handle["id"]
|
||||
assert client.get(f"/v1/sandboxes/{sbx}/files/ghost.py").status_code == 404
|
||||
client.delete(f"/v1/sandboxes/{sbx}")
|
||||
|
||||
def test_exec_unknown_sandbox_404(self, client):
|
||||
resp = client.post(
|
||||
"/v1/sandboxes/sbx-nope/exec", json={"cmd": ["echo", "hi"]}
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
def test_unknown_sandbox_file_routes_404_not_500(self, client):
|
||||
"""P7: read/write on an unknown sandbox must 404 (SandboxNotFoundError
|
||||
previously escaped _workspace_dir as an unhandled 500)."""
|
||||
assert (
|
||||
client.get("/v1/sandboxes/sbx-nope/files/whatever.py").status_code == 404
|
||||
)
|
||||
put = client.put(
|
||||
"/v1/sandboxes/sbx-nope/files/whatever.py",
|
||||
json={"path": "whatever.py", "content": "x"},
|
||||
)
|
||||
assert put.status_code == 404
|
||||
|
||||
def test_symlink_escape_rejected(self, client):
|
||||
"""P7: an exec-planted symlink in the workspace must not let the
|
||||
file routes read/write OUTSIDE the bind (lexical traversal checks
|
||||
cannot see symlinks — resolve + containment re-check is the gate)."""
|
||||
handle = client.post(
|
||||
"/v1/sandboxes", json={"learner_id": "pilot-learner"}
|
||||
).json()
|
||||
sbx = handle["id"]
|
||||
workspace = Path(handle["workdir"]) / "workspace"
|
||||
outside = workspace.parent / "secret.txt"
|
||||
outside.write_text("host secret") # a host file OUTSIDE the bind
|
||||
try:
|
||||
(workspace / "leak.txt").symlink_to(outside)
|
||||
read = client.get(f"/v1/sandboxes/{sbx}/files/leak.txt")
|
||||
assert read.status_code == 422, (
|
||||
f"symlink escape read must 422, got {read.status_code}: {read.text}"
|
||||
)
|
||||
write = client.put(
|
||||
f"/v1/sandboxes/{sbx}/files/leak.txt",
|
||||
json={"path": "leak.txt", "content": "pwned"},
|
||||
)
|
||||
assert write.status_code == 422, (
|
||||
f"symlink escape write must 422, got {write.status_code}: {write.text}"
|
||||
)
|
||||
assert outside.read_text() == "host secret" # untouched
|
||||
finally:
|
||||
client.delete(f"/v1/sandboxes/{sbx}")
|
||||
outside.unlink(missing_ok=True)
|
||||
|
||||
@@ -0,0 +1,561 @@
|
||||
"""WS telemetry ingest API tests (REQ-3-003, D-026, G-3).
|
||||
|
||||
Frame contract under test (telemetry/ingest.py):
|
||||
|
||||
WS /v1/telemetry/ingest?learner_id=L&task_id=T[&sandbox_id=S]
|
||||
client → server: one event per JSON text frame
|
||||
{"seq", "kind", "payload", "ts", "sandbox_id"?}
|
||||
server → client: {"type": "gap_warning"|"event_rejected"|"flooded"|"ack_total"}
|
||||
close: 1000 clean flush · 1008 policy violation (G-3 flood)
|
||||
|
||||
Each test runs against a per-test SQLite file in tmp_path via the same
|
||||
state-injection override the sandbox API tests use: build a real
|
||||
SQLiteTraceStore, pre-set `app.state.trace_store`, let the lifespan adopt it.
|
||||
|
||||
Covered:
|
||||
- connect → send 3 events → GET trace returns them ordered
|
||||
- resend event 2 → deduped (no double-store; at-least-once contract)
|
||||
- skip to seq 5 → gap_warning frame + GET /gaps reports missing seqs
|
||||
- burst past a lowered cap → 1008 close + INCOMPLETE_FLOODED +
|
||||
is_incomplete(...) true
|
||||
- queue-overflow path → same 1008 + INCOMPLETE_FLOODED
|
||||
- reconnect after flood still cannot append (flag is terminal)
|
||||
- unknown trace GET → 404 (traces + gaps)
|
||||
- missing identity query params → handshake rejected
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from starlette.websockets import WebSocketDisconnect
|
||||
|
||||
from ai_service.config import Settings
|
||||
from ai_service.main import create_app
|
||||
from ai_service.telemetry import ingest as ingest_mod
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
from ai_service.telemetry.tokens import IngestToken
|
||||
|
||||
LEARNER = "learner-9"
|
||||
TASK = "task-9"
|
||||
_BASE_TS = datetime(2026, 9, 11, 12, 0, 0, tzinfo=UTC)
|
||||
|
||||
KNOWN_KINDS = ("command", "file_diff", "run_result", "test_result", "activity")
|
||||
|
||||
#: Shared test token — minted into every test URL, verified by the endpoint
|
||||
#: (v0.4.7 P1: untokened dials are refused; tests prove both directions).
|
||||
TEST_TOKEN = IngestToken("test-secret-not-random")
|
||||
|
||||
|
||||
def _frame(seq: int, kind: str = "command", sandbox_id: str = "") -> str:
|
||||
return json.dumps(
|
||||
{
|
||||
"seq": seq,
|
||||
"kind": kind,
|
||||
"payload": {"n": seq},
|
||||
"ts": _BASE_TS.isoformat(),
|
||||
**({"sandbox_id": sandbox_id} if sandbox_id else {}),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _drain(ws) -> None:
|
||||
"""Give the portal a beat to deliver queued frames.
|
||||
|
||||
Starlette's TestClient runs the app on a portal thread; frames produced in
|
||||
the app loop still need a portal round-trip to reach the test thread. A
|
||||
real (tiny) sleep flushes them deterministically without hardcoding
|
||||
protocol timing.
|
||||
"""
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _recv_status(ws) -> dict:
|
||||
"""Next JSON status frame, skipping keepalive pings; re-raise close.
|
||||
|
||||
TestClient delivers close as a `websocket.close` message dict, not by
|
||||
itself raising — `_recv_status` must surface it as WebSocketDisconnect so
|
||||
`pytest.raises(WebSocketDisconnect)` around a read loop terminates.
|
||||
"""
|
||||
while True:
|
||||
msg = ws.receive()
|
||||
if msg.get("bytes") is not None: # keepalive ping — not a status frame
|
||||
continue
|
||||
if msg["type"] == "websocket.close":
|
||||
raise WebSocketDisconnect(msg.get("code", 1000), msg.get("reason", ""))
|
||||
frame = json.loads(msg["text"])
|
||||
if frame.get("type") == "seq_ack": # D-045 advisory ack per append
|
||||
continue
|
||||
return frame
|
||||
|
||||
|
||||
def _ingest_url(
|
||||
learner_id: str = LEARNER,
|
||||
task_id: str = TASK,
|
||||
sandbox_id: str = "",
|
||||
token: str | None = None,
|
||||
) -> str:
|
||||
"""Ingest URL with a VALID token by default (None = mint for identity);
|
||||
pass an explicit token string to test invalid/absent-token paths."""
|
||||
if token is None:
|
||||
token = TEST_TOKEN.mint(learner_id, task_id, sandbox_id)
|
||||
url = (
|
||||
f"/v1/telemetry/ingest?learner_id={learner_id}&task_id={task_id}"
|
||||
f"&token={token}"
|
||||
)
|
||||
return f"{url}&sandbox_id={sandbox_id}" if sandbox_id else url
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def store(tmp_path: Path) -> Iterator[SQLiteTraceStore]:
|
||||
s = SQLiteTraceStore(db_path=tmp_path / "telemetry-test.db")
|
||||
yield s
|
||||
s.close()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def app(store: SQLiteTraceStore, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterator:
|
||||
# Keep the keepalive interval comfortably above the per-test read window:
|
||||
# a ping that fires mid-assertion interleaves with status frames and the
|
||||
# reader must skip it; at 2s a single test's reads all land ping-free
|
||||
# while a *blocked* read still fails fast on the next ping tick.
|
||||
monkeypatch.setattr(ingest_mod, "PING_INTERVAL_S", 2.0)
|
||||
settings = Settings(
|
||||
provider="mock",
|
||||
db_path=tmp_path / "telemetry-test.db",
|
||||
sandbox_dir=tmp_path / "sandboxes",
|
||||
telemetry_max_events_per_task=5, # lowered G-3 cap for the flood test
|
||||
)
|
||||
application = create_app(settings)
|
||||
application.state.trace_store = store # state-injection override
|
||||
# v0.4.7 (P1): known token for tests (pre-boot-random would be untestable);
|
||||
# the endpoint refuses dials without a valid HMAC.
|
||||
application.state.ingest_token = TEST_TOKEN
|
||||
yield application
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client(app) -> Iterator[TestClient]:
|
||||
with TestClient(app) as c:
|
||||
yield c
|
||||
|
||||
|
||||
# -- seq-ack (D-045, REQ-5-007) ------------------------------------------------
|
||||
|
||||
|
||||
def test_each_append_emits_seq_ack_with_durable_latest(
|
||||
client: TestClient, store: SQLiteTraceStore
|
||||
) -> None:
|
||||
"""MH-1a: every successful append acks the post-append durable latest_seq
|
||||
(on dedup'd replays too — a-6)."""
|
||||
with client.websocket_connect(_ingest_url()) as ws:
|
||||
ws.send_text(_frame(0))
|
||||
ws.send_text(_frame(1))
|
||||
ws.send_text(_frame(2))
|
||||
ws.send_text(_frame(2)) # replay → dedup, still acked (a-6)
|
||||
acks: list[int] = []
|
||||
while len(acks) < 4:
|
||||
msg = ws.receive()
|
||||
if msg.get("bytes") is not None:
|
||||
continue
|
||||
if msg["type"] == "websocket.close":
|
||||
raise WebSocketDisconnect(msg.get("code", 1000))
|
||||
frame = json.loads(msg["text"])
|
||||
if frame.get("type") == "seq_ack":
|
||||
assert isinstance(frame["seq"], int)
|
||||
acks.append(frame["seq"])
|
||||
assert acks == [0, 1, 2, 2], f"expected per-append acks incl. dedup, got {acks}"
|
||||
assert store.latest_seq(LEARNER, TASK) == 2
|
||||
|
||||
|
||||
# -- happy path: ordered trace retrieval --------------------------------------
|
||||
|
||||
|
||||
def test_connect_send_three_get_trace_ordered(client: TestClient) -> None:
|
||||
with client.websocket_connect(_ingest_url()) as ws:
|
||||
# Arrive out of arrival order? No — seq arrival order here; ordering is
|
||||
# by seq at read. Send 0,1,2 then close cleanly.
|
||||
for seq in (0, 1, 2):
|
||||
ws.send_text(_frame(seq))
|
||||
|
||||
response = client.get(f"/v1/telemetry/traces/{LEARNER}/{TASK}")
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert [e["seq"] for e in body["events"]] == [0, 1, 2]
|
||||
assert [e["kind"] for e in body["events"]] == ["command"] * 3
|
||||
assert all(e["learner_id"] == LEARNER for e in body["events"])
|
||||
assert body["incomplete"] is False
|
||||
assert body["integrity_reason"] is None
|
||||
|
||||
|
||||
def test_different_kinds_and_sandbox_id_roundtrip(client: TestClient) -> None:
|
||||
with client.websocket_connect(_ingest_url(sandbox_id="sbx-42")) as ws:
|
||||
for seq, kind in enumerate(KNOWN_KINDS[:3]):
|
||||
ws.send_text(_frame(seq, kind))
|
||||
|
||||
body = client.get(f"/v1/telemetry/traces/{LEARNER}/{TASK}").json()
|
||||
assert [e["kind"] for e in body["events"]] == list(KNOWN_KINDS[:3])
|
||||
# Frame had no sandbox_id → the connection's query-param sandbox_id applies.
|
||||
assert all(e["sandbox_id"] == "sbx-42" for e in body["events"])
|
||||
|
||||
|
||||
# -- idempotent dedup (at-least-once, server-side) ------------------------------
|
||||
|
||||
|
||||
def test_resend_event_deduped_no_double_store(client: TestClient) -> None:
|
||||
with client.websocket_connect(_ingest_url()) as ws:
|
||||
for seq in (0, 1, 2):
|
||||
ws.send_text(_frame(seq))
|
||||
ws.send_text(_frame(1, kind="file_diff")) # at-least-once retry of seq 1
|
||||
|
||||
body = client.get(f"/v1/telemetry/traces/{LEARNER}/{TASK}").json()
|
||||
assert [e["seq"] for e in body["events"]] == [0, 1, 2] # stored exactly once
|
||||
# First write wins — the retry's different body must NOT overwrite.
|
||||
assert body["events"][1]["kind"] == "command"
|
||||
|
||||
|
||||
# -- gap detection --------------------------------------------------------------
|
||||
|
||||
|
||||
def test_skip_to_seq_5_reports_gap_warning_and_gaps_endpoint(
|
||||
client: TestClient,
|
||||
) -> None:
|
||||
with client.websocket_connect(_ingest_url()) as ws:
|
||||
for seq in (0, 1, 2):
|
||||
ws.send_text(_frame(seq))
|
||||
ws.send_text(_frame(5)) # skips 3,4 → gap_warning frame
|
||||
_drain(ws)
|
||||
frames = [_recv_status(ws)]
|
||||
assert frames == [{"type": "gap_warning", "missing_seqs": [3, 4]}]
|
||||
|
||||
gaps = client.get(f"/v1/telemetry/gaps/{LEARNER}/{TASK}")
|
||||
assert gaps.status_code == 200
|
||||
assert gaps.json()["gaps"] == [3, 4]
|
||||
assert gaps.json()["incomplete"] is False
|
||||
|
||||
|
||||
# -- G-3 flood control: cap breach → 1008 + INCOMPLETE_FLOODED --------------------
|
||||
|
||||
|
||||
def test_burst_past_cap_closes_1008_and_marks_incomplete_flooded(
|
||||
client: TestClient, app, store: SQLiteTraceStore
|
||||
) -> None:
|
||||
# Cap is 5 (fixture); the 6th event trips cap_exceeded.
|
||||
with client.websocket_connect(_ingest_url()) as ws:
|
||||
for seq in range(6):
|
||||
ws.send_text(_frame(seq))
|
||||
_drain(ws) # let the app flush the flooded frame + 1008 close
|
||||
frames: list[dict] = []
|
||||
with pytest.raises(WebSocketDisconnect) as excinfo:
|
||||
while True:
|
||||
frames.append(_recv_status(ws))
|
||||
assert excinfo.value.code == 1008
|
||||
assert frames == [{"type": "flooded", "reason": "cap_exceeded", "count": 6}]
|
||||
|
||||
# Trace is durably intact up to the cap — no silent drop corrupted it.
|
||||
assert [e.seq for e in store.get_trace(LEARNER, TASK)] == [0, 1, 2, 3, 4]
|
||||
|
||||
# The integrity signal Proctor/Phase-3 grader consume (G-4).
|
||||
assert app.state.trace_integrity.is_incomplete(LEARNER, TASK) is True
|
||||
assert app.state.trace_integrity.reason(LEARNER, TASK) == "INCOMPLETE_FLOODED"
|
||||
|
||||
# ...and it is readable over HTTP (trace with events + flag → 200, not 404).
|
||||
body = client.get(f"/v1/telemetry/traces/{LEARNER}/{TASK}").json()
|
||||
assert body["incomplete"] is True
|
||||
assert body["integrity_reason"] == "INCOMPLETE_FLOODED"
|
||||
|
||||
|
||||
def test_queue_overflow_also_floods(
|
||||
client: TestClient, app, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""G-3: inbound-queue overflow → same 1008 + INCOMPLETE_FLOODED (never
|
||||
drop-oldest). Lower the queue bound so one burst trips it without needing
|
||||
256 frames."""
|
||||
monkeypatch.setattr(ingest_mod, "INBOUND_QUEUE_MAX", 1)
|
||||
with client.websocket_connect(_ingest_url(learner_id="L-q", task_id="T-q")) as ws:
|
||||
for seq in range(64): # receiver out-drains the drainer with bound=1
|
||||
ws.send_text(_frame(seq))
|
||||
_drain(ws)
|
||||
with pytest.raises(WebSocketDisconnect) as excinfo:
|
||||
while True:
|
||||
_recv_status(ws)
|
||||
assert excinfo.value.code == 1008
|
||||
assert app.state.trace_integrity.is_incomplete("L-q", "T-q") is True
|
||||
assert app.state.trace_integrity.reason("L-q", "T-q") == "INCOMPLETE_FLOODED"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_queue_overflow_flood_session_task_terminates(tmp_path, monkeypatch):
|
||||
"""P7 regression: the queue-overflow flood path must not LEAK the
|
||||
session coroutine. v0.3's receiver returned from its QueueFull branch
|
||||
without the disconnect sentinel, so the drainer parked on an empty queue
|
||||
forever and IngestSession.run() never returned — one leaked
|
||||
(pinger+drainer) task-set per flooded trace, unbounded over a long-lived
|
||||
process. A REAL uvicorn server (TestClient teardown hides the leak) is
|
||||
stopped after the flood; the session tasks must be gone shortly after.
|
||||
"""
|
||||
import asyncio
|
||||
import contextlib
|
||||
import socket as socket_mod
|
||||
|
||||
import uvicorn
|
||||
|
||||
from ai_service.main import create_app
|
||||
from ai_service.telemetry.ingest import TraceIntegrityMap
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
|
||||
monkeypatch.setattr(ingest_mod, "INBOUND_QUEUE_MAX", 1)
|
||||
store = SQLiteTraceStore(db_path=tmp_path / "leak.db")
|
||||
app = create_app(
|
||||
Settings(
|
||||
provider="mock",
|
||||
db_path=tmp_path / "leak.db",
|
||||
sandbox_dir=tmp_path / "sandboxes",
|
||||
)
|
||||
)
|
||||
app.state.trace_store = store
|
||||
app.state.trace_integrity = TraceIntegrityMap()
|
||||
# v0.4.7: real-server dial needs a token the endpoint accepts
|
||||
app.state.ingest_token = TEST_TOKEN
|
||||
|
||||
with socket_mod.socket() as s:
|
||||
s.bind(("127.0.0.1", 0))
|
||||
port = s.getsockname()[1]
|
||||
server = uvicorn.Server(
|
||||
uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning")
|
||||
)
|
||||
serve_task = asyncio.get_running_loop().create_task(server.serve())
|
||||
leaked = True
|
||||
try:
|
||||
for _ in range(100):
|
||||
if server.started:
|
||||
break
|
||||
await asyncio.sleep(0.1)
|
||||
assert server.started
|
||||
|
||||
import websockets
|
||||
|
||||
uri = (
|
||||
f"ws://127.0.0.1:{port}/v1/telemetry/ingest"
|
||||
f"?learner_id=L-leak&task_id=T-leak"
|
||||
f"&token={TEST_TOKEN.mint('L-leak', 'T-leak', '')}"
|
||||
)
|
||||
async with websockets.connect(uri) as ws:
|
||||
for seq in range(64): # bound=1 → guaranteed overflow
|
||||
await ws.send(_frame(seq, sandbox_id=""))
|
||||
# The flood close (1008) reaches the client.
|
||||
try:
|
||||
await asyncio.wait_for(ws.recv(), timeout=10.0)
|
||||
await asyncio.wait_for(ws.recv(), timeout=10.0)
|
||||
except (websockets.exceptions.ConnectionClosed, TimeoutError, OSError):
|
||||
pass
|
||||
assert app.state.trace_integrity.is_incomplete("L-leak", "T-leak")
|
||||
|
||||
# The session's run() must have returned: no lingering nc-* tasks
|
||||
# holding the socket open. Poll briefly — teardown is async.
|
||||
deadline = asyncio.get_running_loop().time() + 5.0
|
||||
while asyncio.get_running_loop().time() < deadline:
|
||||
names = {
|
||||
t.get_name()
|
||||
for t in asyncio.all_tasks()
|
||||
if t is not asyncio.current_task()
|
||||
}
|
||||
if not any("ingest" in n.lower() for n in names):
|
||||
leaked = False
|
||||
break
|
||||
await asyncio.sleep(0.1)
|
||||
finally:
|
||||
server.should_exit = True
|
||||
with contextlib.suppress(Exception):
|
||||
await asyncio.wait_for(serve_task, timeout=10.0)
|
||||
store.close()
|
||||
assert not leaked, "IngestSession task set leaked after queue-overflow flood"
|
||||
|
||||
|
||||
def test_reconnect_after_flood_cannot_resurrect_trace(
|
||||
client: TestClient, app, store: SQLiteTraceStore
|
||||
) -> None:
|
||||
"""INCOMPLETE_FLOODED is terminal: a fresh connection for the same trace
|
||||
is immediately closed 1008 — intake never resumes on a flagged trace."""
|
||||
with client.websocket_connect(_ingest_url()) as ws:
|
||||
for seq in range(6):
|
||||
ws.send_text(_frame(seq))
|
||||
_drain(ws)
|
||||
with pytest.raises(WebSocketDisconnect):
|
||||
while True:
|
||||
_recv_status(ws)
|
||||
assert store.latest_seq(LEARNER, TASK) == 4
|
||||
|
||||
with client.websocket_connect(_ingest_url()) as ws:
|
||||
ws.send_text(_frame(5))
|
||||
_drain(ws)
|
||||
with pytest.raises(WebSocketDisconnect) as excinfo:
|
||||
while True:
|
||||
_recv_status(ws)
|
||||
assert excinfo.value.code == 1008
|
||||
assert store.latest_seq(LEARNER, TASK) == 4 # nothing appended post-flag
|
||||
|
||||
|
||||
# -- frame validation ------------------------------------------------------------
|
||||
|
||||
|
||||
def test_frame_with_learner_id_in_body_rejected(client: TestClient) -> None:
|
||||
"""Identity lives in the URL (extra=forbid): a frame carrying learner_id
|
||||
is a contract violation → event_rejected, nothing stored."""
|
||||
with client.websocket_connect(_ingest_url()) as ws:
|
||||
bad = json.loads(_frame(0))
|
||||
bad["learner_id"] = "spoofed"
|
||||
ws.send_text(json.dumps(bad))
|
||||
_drain(ws)
|
||||
frames = [_recv_status(ws)]
|
||||
assert frames[0]["type"] == "event_rejected"
|
||||
assert client.get(f"/v1/telemetry/traces/{LEARNER}/{TASK}").status_code == 404
|
||||
|
||||
|
||||
def test_unknown_event_kind_rejected_nothing_stored(client: TestClient) -> None:
|
||||
with client.websocket_connect(_ingest_url()) as ws:
|
||||
ws.send_text(_frame(0, kind="rm_rf_everything"))
|
||||
_drain(ws)
|
||||
frames = [_recv_status(ws)]
|
||||
assert frames[0]["type"] == "event_rejected"
|
||||
assert "unknown event kind" in frames[0]["detail"]
|
||||
assert client.get(f"/v1/telemetry/traces/{LEARNER}/{TASK}").status_code == 404
|
||||
|
||||
|
||||
# -- unknown trace reads ---------------------------------------------------------
|
||||
|
||||
|
||||
def test_unknown_trace_get_returns_404(client: TestClient) -> None:
|
||||
assert client.get("/v1/telemetry/traces/nobody/never").status_code == 404
|
||||
assert client.get("/v1/telemetry/gaps/nobody/never").status_code == 404
|
||||
|
||||
|
||||
# -- handshake contract ------------------------------------------------------------
|
||||
|
||||
|
||||
def test_missing_identity_query_params_rejected_at_handshake(
|
||||
client: TestClient,
|
||||
) -> None:
|
||||
with pytest.raises(WebSocketDisconnect) as excinfo:
|
||||
with client.websocket_connect("/v1/telemetry/ingest"):
|
||||
pass
|
||||
assert excinfo.value.code == 1008
|
||||
|
||||
|
||||
def test_browser_origin_rejected_in_explicit_list_mode(
|
||||
settings, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""D-038: with an explicit AI_CORS_ORIGINS list, a page loaded in the
|
||||
learner's browser (unlisted Origin) must not be able to open the ingest
|
||||
socket and poison/flood the trace. The stdlib capture agent sends no
|
||||
Origin and is unaffected (see the no-origin test below)."""
|
||||
from fastapi.testclient import TestClient as TC
|
||||
|
||||
restricted = settings.model_copy(update={"cors_origins": "http://localhost:3000"})
|
||||
app = create_app(restricted)
|
||||
with TC(app) as c:
|
||||
with pytest.raises(WebSocketDisconnect) as excinfo:
|
||||
with c.websocket_connect(
|
||||
_ingest_url(), headers={"Origin": "https://evil.example"}
|
||||
):
|
||||
pass
|
||||
assert excinfo.value.code == 1008
|
||||
|
||||
|
||||
def test_wildcard_mode_admits_any_browser_origin(client: TestClient) -> None:
|
||||
"""D-038 default ('*'): remote-browser origins open the ingest socket —
|
||||
the remote build surface streams telemetry from the learner's browser."""
|
||||
with client.websocket_connect(
|
||||
_ingest_url(), headers={"Origin": "http://nextcraft-1:3000"}
|
||||
) as ws:
|
||||
ws.send_text(_frame(0))
|
||||
|
||||
|
||||
def test_dev_origin_and_no_origin_both_allowed(client: TestClient) -> None:
|
||||
"""The same-origin dev page (Next.js :3000) opens fine, and so does the
|
||||
capture-agent path (no Origin header at all)."""
|
||||
for headers in ({"Origin": "http://localhost:3000"}, {}):
|
||||
with client.websocket_connect(_ingest_url(), headers=headers) as ws:
|
||||
ws.send_text(_frame(0))
|
||||
body = client.get(f"/v1/telemetry/traces/{LEARNER}/{TASK}").json()
|
||||
assert [e["seq"] for e in body["events"]] == [0]
|
||||
# Unique trace per iteration would collide on (LEARNER, TASK) PK —
|
||||
# seq 0 re-sent is deduped, so one row is the invariant either way.
|
||||
assert len(body["events"]) == 1
|
||||
|
||||
|
||||
# -- keepalive ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_server_sends_periodic_ping(
|
||||
client: TestClient, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Keepalive (D-026): the server emits a ping on PING_INTERVAL_S so the
|
||||
stdlib capture agent's auto-pong keeps the path warm. Frame content is
|
||||
opaque to the agent; the contract is that a ping arrives while idle."""
|
||||
monkeypatch.setattr(ingest_mod, "PING_INTERVAL_S", 0.05)
|
||||
with client.websocket_connect(_ingest_url()) as ws:
|
||||
ping = ws.receive_bytes() # first server→client frame while idle
|
||||
assert ping # opaque payload; contract is "a ping frame arrives"
|
||||
|
||||
|
||||
# -- ingest-token auth (v0.4.7, P1) -------------------------------------------------
|
||||
|
||||
|
||||
def test_untokened_dial_refused(client: TestClient, store: SQLiteTraceStore) -> None:
|
||||
"""P1 flag: the LAN-reachable grading input refuses identity params
|
||||
without a token — self-asserted learner/task could poison traces."""
|
||||
with pytest.raises(WebSocketDisconnect) as excinfo:
|
||||
with client.websocket_connect(
|
||||
f"/v1/telemetry/ingest?learner_id={LEARNER}&task_id={TASK}"
|
||||
):
|
||||
pass
|
||||
assert excinfo.value.code == 1008
|
||||
assert store.latest_seq(LEARNER, TASK) == -1 # nothing stored
|
||||
|
||||
|
||||
def test_bad_token_refused(client: TestClient, store: SQLiteTraceStore) -> None:
|
||||
with pytest.raises(WebSocketDisconnect) as excinfo:
|
||||
with client.websocket_connect(
|
||||
_ingest_url(token="deadbeef" * 8)
|
||||
):
|
||||
pass
|
||||
assert excinfo.value.code == 1008
|
||||
assert store.latest_seq(LEARNER, TASK) == -1
|
||||
|
||||
|
||||
def test_cross_identity_token_refused(client: TestClient) -> None:
|
||||
"""A token minted for (A, T1) cannot dial as (B, T1) or (A, T2) — the
|
||||
HMAC binds the identity, so a leaked URL poisons only its own trace."""
|
||||
other = "learner-evil"
|
||||
token = TEST_TOKEN.mint(other, TASK, "")
|
||||
with pytest.raises(WebSocketDisconnect) as excinfo:
|
||||
with client.websocket_connect(_ingest_url(token=token)):
|
||||
pass
|
||||
assert excinfo.value.code == 1008
|
||||
# same learner, different task
|
||||
token2 = TEST_TOKEN.mint(LEARNER, "task-other", "")
|
||||
with pytest.raises(WebSocketDisconnect) as excinfo:
|
||||
with client.websocket_connect(_ingest_url(token=token2)):
|
||||
pass
|
||||
assert excinfo.value.code == 1008
|
||||
|
||||
|
||||
def test_manager_urls_carry_valid_tokens(client: TestClient) -> None:
|
||||
"""The sandbox manager's NC_INGEST_URL mints a token the endpoint
|
||||
accepts — mint/verify agree through the shared app.state instance."""
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
manager = client.app.state.sandbox_manager
|
||||
env = manager._capture_env("sbx-tok-1", "learner-tok", "task-tok")
|
||||
parsed = urlparse(env["NC_INGEST_URL"])
|
||||
qs = parse_qs(parsed.query)
|
||||
assert qs["token"][0] == client.app.state.ingest_token.mint(
|
||||
"learner-tok", "task-tok", "sbx-tok-1"
|
||||
)
|
||||
@@ -0,0 +1,284 @@
|
||||
"""Variant API tests — generation, cache, distinctness over HTTP (Task 4-3-01).
|
||||
|
||||
Contract under test (api/variants.py, REQ-3-005):
|
||||
|
||||
POST /v1/variants {learner_id, template_id}
|
||||
first request → 200 generated variant (task_id, seed, params,
|
||||
statement, starter_files, competency_id)
|
||||
repeat request → 200 the SAME cached variant with ZERO LLM
|
||||
calls (D-029 reproducibility through the
|
||||
whole HTTP stack)
|
||||
unknown template → 404
|
||||
POST /v1/variants {learner_id, competency_id}
|
||||
bound competency → 200 the first template for that competency
|
||||
unknown competency → 404
|
||||
neither id given → 422
|
||||
GET /v1/variants/{task_id}
|
||||
generated earlier → 200 the stored variant (generate→get roundtrip)
|
||||
unknown task → 404
|
||||
GET /v1/variants?learner_id=...
|
||||
→ 200 that learner's variants only (scoping); [] for a learner
|
||||
with none.
|
||||
|
||||
Distinctness (REQ-3-005, API level): two learners POSTing the same
|
||||
template receive distinct statements, seeds and task_ids, and GET by
|
||||
task_id hands each back their own variant.
|
||||
|
||||
Wiring: per-test tmp-path SQLiteVariantStore + a pre-set VariantGenerator
|
||||
(state-injection override — the lifespan adopts variant_store /
|
||||
variant_generator from app.state instead of constructing them; same
|
||||
pattern as test_grading.py / test_telemetry_ingest.py). The generator
|
||||
binds a ScriptedRenderProvider so each test controls the render LLM
|
||||
exactly, and counts calls so the cache test can assert the LLM was never
|
||||
reached on the second POST.
|
||||
|
||||
Zero network: providers are MockProvider family members only (conftest
|
||||
rule, enforced in _make_client).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from ai_service.config import Settings
|
||||
from ai_service.identity.store import SQLiteIdentityStore
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.main import create_app
|
||||
from ai_service.variants.generator import VariantGenerator
|
||||
from ai_service.variants.store import SQLiteVariantStore
|
||||
from ai_service.variants.templates import get_template
|
||||
|
||||
from ..conftest import SUITE_LEARNERS, seed_verified_identity
|
||||
|
||||
LEARNER_A = "variant-learner-a"
|
||||
LEARNER_B = "variant-learner-b"
|
||||
TEMPLATE = "tpl-llm-judge"
|
||||
COMPETENCY = "stack-orchestration-c007" # tpl-llm-judge's D-021 binding
|
||||
|
||||
|
||||
class ScriptedRenderProvider(MockProvider):
|
||||
"""Deterministic render whose statement embeds the params (distinct per
|
||||
draw); counts calls so the cache test asserts zero LLM calls on the
|
||||
second POST. Mirrors tests/variants/test_generator.py."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.calls = 0
|
||||
|
||||
async def chat(self, messages, *, model, temperature=0.7, response_format=None): # noqa: ANN001
|
||||
self.calls += 1
|
||||
import json
|
||||
|
||||
user = next(m.content for m in reversed(messages) if m.role == "user")
|
||||
# Distinct per distinct params: hash the seeded slot lines.
|
||||
fingerprint = abs(hash(user)) % 10_000
|
||||
return json.dumps({"statement": f"Scripted variant #{fingerprint} — build it."})
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def store(tmp_path: Path) -> Iterator[SQLiteVariantStore]:
|
||||
s = SQLiteVariantStore(db_path=tmp_path / "variants.db")
|
||||
yield s
|
||||
s.close()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def provider() -> ScriptedRenderProvider:
|
||||
return ScriptedRenderProvider()
|
||||
|
||||
|
||||
def _make_client(
|
||||
tmp_path: Path, store: SQLiteVariantStore, provider: MockProvider
|
||||
) -> TestClient:
|
||||
"""App + TestClient with a pre-set store + generator.
|
||||
|
||||
The lifespan adopts both (state-injection override); the generator
|
||||
binds OUR provider, so the fixture — not Settings — scripts the LLM.
|
||||
Cloud-free guard: the provider must be a MockProvider family member
|
||||
(conftest rule, enforced here because this module builds its own
|
||||
client rather than consuming the conftest one).
|
||||
"""
|
||||
assert isinstance(provider, MockProvider)
|
||||
settings = Settings(
|
||||
provider="mock",
|
||||
db_path=tmp_path / "variant-test.db",
|
||||
sandbox_dir=tmp_path / "sandboxes",
|
||||
learner_allowlist=SUITE_LEARNERS,
|
||||
)
|
||||
app = create_app(settings)
|
||||
identity_store = SQLiteIdentityStore(db_path=tmp_path / "identity.db")
|
||||
seed_verified_identity(identity_store)
|
||||
app.state.identity_store = identity_store # G-9
|
||||
app.state.variant_store = store
|
||||
if getattr(app.state, "variant_generator", None) is None:
|
||||
app.state.variant_generator = VariantGenerator(
|
||||
store, provider, model="gemma4:31b"
|
||||
)
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _post(
|
||||
client: TestClient,
|
||||
learner_id: str,
|
||||
template_id: str | None = TEMPLATE,
|
||||
competency_id: str | None = None,
|
||||
):
|
||||
return client.post(
|
||||
"/v1/variants",
|
||||
json={
|
||||
"learner_id": learner_id,
|
||||
**({"template_id": template_id} if template_id is not None else {}),
|
||||
**({"competency_id": competency_id} if competency_id is not None else {}),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# -- POST: generation + response shape --------------------------------------------
|
||||
|
||||
|
||||
class TestGenerate:
|
||||
def test_post_returns_full_variant_payload(self, tmp_path, store, provider):
|
||||
with _make_client(tmp_path, store, provider) as client:
|
||||
response = _post(client, LEARNER_A)
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["learner_id"] == LEARNER_A
|
||||
assert body["template_id"] == TEMPLATE
|
||||
assert body["competency_id"] == COMPETENCY # enrichment from get_template
|
||||
assert body["task_id"].startswith("task-") and len(body["task_id"]) == len("task-") + 16
|
||||
assert body["seed"] # non-empty D-029 seed
|
||||
assert body["statement"] # rendered, non-empty
|
||||
assert body["starter_files"] == get_template(TEMPLATE).starter_files
|
||||
assert body["params"] # seeded slot values
|
||||
assert body["created_at"] # ISO timestamp travels
|
||||
assert provider.calls == 1 # first POST renders exactly once
|
||||
|
||||
def test_post_then_get_roundtrip(self, tmp_path, store, provider):
|
||||
with _make_client(tmp_path, store, provider) as client:
|
||||
posted = _post(client, LEARNER_A)
|
||||
assert posted.status_code == 200
|
||||
|
||||
fetched = client.get(f"/v1/variants/{posted.json()['task_id']}")
|
||||
|
||||
assert fetched.status_code == 200
|
||||
assert fetched.json() == posted.json() # identical stored variant
|
||||
|
||||
def test_post_regenerate_is_cache_hit_no_llm_call(self, tmp_path, store, provider):
|
||||
with _make_client(tmp_path, store, provider) as client:
|
||||
first = _post(client, LEARNER_A)
|
||||
assert first.status_code == 200
|
||||
assert provider.calls == 1
|
||||
|
||||
second = _post(client, LEARNER_A) # same (learner, template) → cache
|
||||
assert second.status_code == 200
|
||||
assert second.json() == first.json() # D-029: the SAME variant
|
||||
assert provider.calls == 1, "cached regenerate must not render again"
|
||||
|
||||
|
||||
# -- POST: distinctness (REQ-3-005, API level) ------------------------------------
|
||||
|
||||
|
||||
class TestDistinctLearners:
|
||||
def test_two_learners_distinct_statements_and_task_ids(
|
||||
self, tmp_path, store, provider
|
||||
):
|
||||
with _make_client(tmp_path, store, provider) as client:
|
||||
a = _post(client, LEARNER_A)
|
||||
b = _post(client, LEARNER_B)
|
||||
assert a.status_code == 200 and b.status_code == 200
|
||||
|
||||
a_body, b_body = a.json(), b.json()
|
||||
|
||||
# ...and each learner GETs back exactly their own variant
|
||||
a_fetched = client.get(f"/v1/variants/{a_body['task_id']}")
|
||||
b_fetched = client.get(f"/v1/variants/{b_body['task_id']}")
|
||||
|
||||
assert a_body["statement"] != b_body["statement"]
|
||||
assert a_body["seed"] != b_body["seed"]
|
||||
assert a_body["task_id"] != b_body["task_id"]
|
||||
assert a_body["competency_id"] == b_body["competency_id"] # same bar (a-5)
|
||||
assert a_fetched.status_code == 200 and a_fetched.json() == a_body
|
||||
assert b_fetched.status_code == 200 and b_fetched.json() == b_body
|
||||
|
||||
|
||||
# -- POST: template / competency resolution ---------------------------------------
|
||||
|
||||
|
||||
class TestTemplateResolution:
|
||||
def test_unknown_template_404(self, tmp_path, store, provider):
|
||||
with _make_client(tmp_path, store, provider) as client:
|
||||
response = _post(client, LEARNER_A, template_id="tpl-does-not-exist")
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "no task template" in response.json()["detail"]
|
||||
|
||||
def test_competency_lookup_uses_first_bound_template(self, tmp_path, store, provider):
|
||||
with _make_client(tmp_path, store, provider) as client:
|
||||
response = _post(
|
||||
client, LEARNER_A, template_id=None, competency_id=COMPETENCY
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["template_id"] == TEMPLATE # the competency's first template
|
||||
assert body["competency_id"] == COMPETENCY
|
||||
|
||||
def test_unknown_competency_404(self, tmp_path, store, provider):
|
||||
with _make_client(tmp_path, store, provider) as client:
|
||||
response = _post(
|
||||
client, LEARNER_A, template_id=None, competency_id="stack-none-c999"
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "no task template for competency" in response.json()["detail"]
|
||||
|
||||
def test_neither_id_given_422(self, tmp_path, store, provider):
|
||||
with _make_client(tmp_path, store, provider) as client:
|
||||
response = _post(client, LEARNER_A, template_id=None, competency_id=None)
|
||||
|
||||
assert response.status_code == 422
|
||||
|
||||
|
||||
# -- GET: stored reads ------------------------------------------------------------
|
||||
|
||||
|
||||
class TestStoredReads:
|
||||
def test_get_unknown_task_404(self, tmp_path, store, provider):
|
||||
with _make_client(tmp_path, store, provider) as client:
|
||||
response = client.get("/v1/variants/task-0000000000000000")
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "no stored variant" in response.json()["detail"]
|
||||
|
||||
def test_list_scoped_to_one_learner(self, tmp_path, store, provider):
|
||||
with _make_client(tmp_path, store, provider) as client:
|
||||
assert _post(client, LEARNER_A).status_code == 200
|
||||
assert _post(
|
||||
client, LEARNER_A, template_id="tpl-guardrail-schema"
|
||||
).status_code == 200
|
||||
assert _post(client, LEARNER_B).status_code == 200
|
||||
|
||||
listed = client.get("/v1/variants", params={"learner_id": LEARNER_A})
|
||||
empty = client.get("/v1/variants", params={"learner_id": "nobody"})
|
||||
|
||||
assert listed.status_code == 200
|
||||
variants = listed.json()["variants"]
|
||||
assert len(variants) == 2 # LEARNER_A's two, LEARNER_B's excluded
|
||||
assert {v["learner_id"] for v in variants} == {LEARNER_A}
|
||||
assert {v["template_id"] for v in variants} == {TEMPLATE, "tpl-guardrail-schema"}
|
||||
# chronological ordering; both carry the competency enrichment
|
||||
assert all(v["competency_id"] for v in variants)
|
||||
|
||||
assert empty.status_code == 200
|
||||
assert empty.json()["variants"] == []
|
||||
|
||||
def test_list_missing_learner_param_422(self, tmp_path, store, provider):
|
||||
with _make_client(tmp_path, store, provider) as client:
|
||||
response = client.get("/v1/variants")
|
||||
|
||||
assert response.status_code == 422
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Single-port static-UI mount tests (v0.3.6).
|
||||
|
||||
When AI_WEB_STATIC_DIR is set, the app serves the exported Next.js app
|
||||
(apps/web/out) at / — same origin as the API, so the site answers on one
|
||||
port behind HAProxy (the only reachable port). Pins:
|
||||
|
||||
- default (unset): NO mount — / stays the plain FastAPI 404 (dev/tests);
|
||||
- set: / serves index.html, API routes win over static, unknown paths
|
||||
get the export's 404.html with status 404;
|
||||
- set-but-missing dir: startup fails loudly (misconfig, never a silent
|
||||
mount-less app).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from ai_service.config import Settings
|
||||
from ai_service.main import create_app
|
||||
|
||||
|
||||
def _export_dir(tmp_path: Path) -> Path:
|
||||
"""Minimal fake of a Next.js static export."""
|
||||
out = tmp_path / "out"
|
||||
out.mkdir()
|
||||
(out / "index.html").write_text("<html>nextcraft home</html>")
|
||||
(out / "404.html").write_text("<html>not found page</html>")
|
||||
nested = out / "dashboard"
|
||||
nested.mkdir()
|
||||
(nested / "index.html").write_text("<html>dashboard page</html>")
|
||||
return out
|
||||
|
||||
|
||||
def _settings(tmp_path: Path, web_dir: Path | None) -> Settings:
|
||||
return Settings(
|
||||
provider="mock",
|
||||
model="gemma4:31b",
|
||||
port=8421,
|
||||
db_path=tmp_path / "test.db",
|
||||
sandbox_dir=tmp_path / "sandboxes",
|
||||
web_static_dir=web_dir or Path(""),
|
||||
)
|
||||
|
||||
|
||||
def test_default_no_mount_root_is_api_404(client: TestClient) -> None:
|
||||
"""Default web_static_dir='' → no mount; / is FastAPI's JSON 404."""
|
||||
resp = client.get("/")
|
||||
assert resp.status_code == 404
|
||||
assert resp.json() == {"detail": "Not Found"}
|
||||
|
||||
|
||||
def test_no_mount_boot_is_loud(tmp_path: Path, caplog: pytest.LogCaptureFixture) -> None:
|
||||
"""v0.4.6: a mount-less boot must WARN naming the fix — the silent
|
||||
mount-less boot is the root cause of every domain-root 404 outage
|
||||
(nothing detected it; HAProxy happily proxies a JSON 404)."""
|
||||
import logging
|
||||
|
||||
settings = _settings(tmp_path, None)
|
||||
with caplog.at_level(logging.WARNING, logger="ai_service.main"):
|
||||
create_app(settings)
|
||||
assert "AI_WEB_STATIC_DIR is unset" in caplog.text
|
||||
assert "pnpm build" in caplog.text
|
||||
|
||||
|
||||
def test_mount_serves_index_and_api_routes_win(tmp_path: Path) -> None:
|
||||
out = _export_dir(tmp_path)
|
||||
app = create_app(_settings(tmp_path, out))
|
||||
with TestClient(app) as c:
|
||||
# / serves the export's index.html
|
||||
resp = c.get("/")
|
||||
assert resp.status_code == 200
|
||||
assert "nextcraft home" in resp.text
|
||||
# API routes take precedence over the static mount
|
||||
assert c.get("/health").status_code == 200
|
||||
# trailingSlash-style directory index resolves
|
||||
assert "dashboard page" in c.get("/dashboard/").text
|
||||
|
||||
|
||||
def test_mount_unknown_path_serves_export_404_page(tmp_path: Path) -> None:
|
||||
out = _export_dir(tmp_path)
|
||||
app = create_app(_settings(tmp_path, out))
|
||||
with TestClient(app) as c:
|
||||
resp = c.get("/definitely-not-a-page/")
|
||||
assert resp.status_code == 404
|
||||
assert "not found page" in resp.text
|
||||
|
||||
|
||||
def test_mount_missing_dir_fails_loudly(tmp_path: Path) -> None:
|
||||
settings = _settings(tmp_path, tmp_path / "does-not-exist")
|
||||
with pytest.raises(RuntimeError, match="AI_WEB_STATIC_DIR"):
|
||||
create_app(settings)
|
||||
@@ -1,6 +1,9 @@
|
||||
"""Test suite — conftest: mock provider only, zero network (enforced)."""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
@@ -9,16 +12,99 @@ from ai_service.config import Settings
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.main import create_app
|
||||
|
||||
# Sandbox workdirs need a filesystem that supports bind mounts from a user
|
||||
# namespace: pytest's default tmpdir (/tmp) is overlayfs here and mount(2)
|
||||
# fails with ENODEV ("special device ... does not exist"). Home is real ext4.
|
||||
_HOME_ROOT = Path.home() / ".nextcraft" / "test-sandboxes"
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def settings() -> Settings:
|
||||
def sandbox_dir(tmp_path: Path) -> Path:
|
||||
"""Per-test sandbox workdir root on a bind-mount-safe filesystem."""
|
||||
_HOME_ROOT.mkdir(parents=True, exist_ok=True)
|
||||
d = Path(tempfile.mkdtemp(prefix=f"nc-test-{tmp_path.name}-", dir=_HOME_ROOT))
|
||||
yield d
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
|
||||
|
||||
# G-9 (grill): the v0.5 identity gates land on variants/sandboxes/defense —
|
||||
# routes every API suite exercises. The suite-wide learner roster + a
|
||||
# seeded verified identity keep pre-existing tests green while the gate
|
||||
# tests (test_identity.py) prove the 403/CTA composition on unverified ids.
|
||||
SUITE_LEARNERS = [
|
||||
"pilot-learner",
|
||||
"pilot-learner-2",
|
||||
"api-learner",
|
||||
"defense-learner",
|
||||
"grade-learner",
|
||||
"lab-learner",
|
||||
"p-learner",
|
||||
"variant-learner-a",
|
||||
"variant-learner-b",
|
||||
"learner-001",
|
||||
"lat-learner",
|
||||
"ghost-learner",
|
||||
]
|
||||
|
||||
|
||||
def seed_verified_identity(store, learner_ids=SUITE_LEARNERS) -> None:
|
||||
"""Insert a verified 18+ identity record per learner id (mock-marked).
|
||||
|
||||
A-304: records carry mock=True — the seed never masquerades as
|
||||
production verification.
|
||||
"""
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from ai_service.identity.store import IdentityRecord
|
||||
|
||||
for lid in learner_ids:
|
||||
try:
|
||||
store.insert(
|
||||
IdentityRecord(
|
||||
id=f"seed-{lid}",
|
||||
learner_id=lid,
|
||||
status="verified",
|
||||
provider="mock",
|
||||
verdict={"status": "verified", "age_band": "18+", "mock": True},
|
||||
age_band="18+",
|
||||
submitted_at=datetime.now(UTC),
|
||||
verified_at=datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
pass # already seeded (shared store)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def settings(tmp_path: Path) -> Settings:
|
||||
os.environ["AI_PROVIDER"] = "mock"
|
||||
return Settings(provider="mock", model="gemma4:31b", port=8421)
|
||||
# Hermetic stores (v0.3.6): db_path/sandbox_dir default to ~/.nextcraft —
|
||||
# tests must never read or write real state; pin to the pytest tmp dir.
|
||||
return Settings(
|
||||
provider="mock",
|
||||
model="gemma4:31b",
|
||||
port=8421,
|
||||
db_path=tmp_path / "nextcraft-test.db",
|
||||
sandbox_dir=tmp_path / "sandboxes",
|
||||
learner_allowlist=SUITE_LEARNERS,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def app(settings: Settings):
|
||||
return create_app(settings)
|
||||
def identity_store(tmp_path: Path):
|
||||
from ai_service.identity.store import SQLiteIdentityStore
|
||||
|
||||
store = SQLiteIdentityStore(db_path=tmp_path / "identity-test.db")
|
||||
seed_verified_identity(store)
|
||||
yield store
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def app(settings: Settings, identity_store):
|
||||
application = create_app(settings)
|
||||
application.state.identity_store = identity_store # state-injection (G-9)
|
||||
return application
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""Process-trace grading tests (REQ-3-004)."""
|
||||
@@ -0,0 +1,361 @@
|
||||
"""Grading calibration CONTRACT test (Task 3-2-02, REQ-3-004, D-021).
|
||||
|
||||
HONEST SCOPE — what this test is and is NOT:
|
||||
|
||||
This is a CALIBRATION CONTRACT test, not an LLM quality test. The mock
|
||||
provider scripts archetype-mapped rubric scores for each fixture; the
|
||||
assertions verify that:
|
||||
|
||||
1. the ENGINE pipeline (gate → digest → prompt → D-020 → validate →
|
||||
persist) runs each archetype end-to-end and preserves the score
|
||||
ordering the scripts impose;
|
||||
2. the fixtures' digest FEATURES actually separate the archetypes
|
||||
deterministically (the non-LLM half of the calibration: paste-and-run
|
||||
vs iterative debugging vs never-passing MUST produce observably
|
||||
different digests — otherwise no rubric could ever separate them);
|
||||
3. fixture IDs are valid corpus-aligned strings (D-021).
|
||||
|
||||
It does NOT prove the real LLM scores archetypes in this order — that
|
||||
requires live-model evaluation, which is out of scope for v0.3 tests
|
||||
(tests NEVER call the cloud; conftest enforces). What the contract pins
|
||||
is the ORDERING the rubric + digest must eventually enforce:
|
||||
|
||||
strong.process_quality >= lazy.process_quality
|
||||
strong.correctness > struggling.correctness
|
||||
|
||||
If a future prompt/digest change makes the scripted mapping unachievable
|
||||
by ANY model (e.g. the digest stops separating the archetypes), test 2
|
||||
fails — that is the calibration signal.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from ai_service.corpus.telemetry import LAB_SCENARIOS
|
||||
from ai_service.corpus.trace_fixtures import (
|
||||
CALIBRATION_LEARNER_ID,
|
||||
LAZY_BUILDER,
|
||||
STRONG_BUILDER,
|
||||
STRUGGLING_BUILDER,
|
||||
TRACE_FIXTURES,
|
||||
digest_of,
|
||||
)
|
||||
from ai_service.grading.engine import GradingEngine
|
||||
from ai_service.grading.store import SQLiteGradeStore
|
||||
from ai_service.telemetry.ingest import TraceIntegrityMap
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
|
||||
#: Archetype-mapped rubric scripts (the ORDERING CONTRACT in numbers).
|
||||
#: strong: iterative verified building — highest process quality.
|
||||
#: lazy: ends pass with zero iteration — the a-4 negative (churn with no
|
||||
#: test progress caps process_quality at 1; here it's paste-churn +
|
||||
#: one late test, so 1). correctness lands high (final pass, fast).
|
||||
#: struggling: never passes — lowest correctness.
|
||||
ARCHETYPE_SCORES: dict[str, dict] = {
|
||||
"strong": {
|
||||
"criteria": {
|
||||
"process_quality": 4,
|
||||
"correctness": 4,
|
||||
"debugging_discipline": 4,
|
||||
"test_usage": 4,
|
||||
},
|
||||
"strengths": ["tight edit-test loops; every failure cycle closed"],
|
||||
"gaps": ["session ended without a final full-suite run"],
|
||||
"verdict": "mastered",
|
||||
},
|
||||
"lazy": {
|
||||
"criteria": {
|
||||
"process_quality": 1, # a-4: bulk paste + churn, one late test
|
||||
"correctness": 3,
|
||||
"debugging_discipline": 1,
|
||||
"test_usage": 1,
|
||||
},
|
||||
"strengths": ["final tests pass on the first run"],
|
||||
"gaps": ["one bulk paste with zero verified increments"],
|
||||
"verdict": "developing",
|
||||
},
|
||||
"struggling": {
|
||||
"criteria": {
|
||||
"process_quality": 2,
|
||||
"correctness": 0, # never passes
|
||||
"debugging_discipline": 1, # cycles never close
|
||||
"test_usage": 3,
|
||||
},
|
||||
"strengths": ["persisted through repeated failures"],
|
||||
"gaps": ["the same failure recurred three times"],
|
||||
"verdict": "not_yet",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class ArchetypeProvider:
|
||||
"""Mock provider scripting archetype-mapped scores keyed by fixture.
|
||||
|
||||
Keying on the digest embedded in the user turn (not on call order)
|
||||
keeps the mapping robust to gate-path noise and asserts the prompt
|
||||
really carries the fixture's digest. Records the last user turn so
|
||||
tests can re-assert D-028 (digest in prompt; raw events out).
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.digest_to_archetype: dict[str, str] = {}
|
||||
self.calls = 0
|
||||
self.last_user_prompt: str = ""
|
||||
|
||||
def register(self, fixture) -> None:
|
||||
self.digest_to_archetype[digest_of(fixture).model_dump_json()] = (
|
||||
fixture.archetype
|
||||
)
|
||||
|
||||
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
self.calls += 1
|
||||
user = next((m for m in reversed(messages) if m.role == "user"), None)
|
||||
assert user is not None
|
||||
self.last_user_prompt = user.content
|
||||
archetype = None
|
||||
for digest_json, arch in self.digest_to_archetype.items():
|
||||
if digest_json in user.content:
|
||||
archetype = arch
|
||||
break
|
||||
assert archetype is not None, "fixture digest not found in prompt"
|
||||
return json.dumps(ARCHETYPE_SCORES[archetype])
|
||||
|
||||
async def stream_chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
yield await self.chat(
|
||||
messages, model=model, temperature=temperature, response_format=response_format
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def trace_store(tmp_path: Path) -> SQLiteTraceStore:
|
||||
store = SQLiteTraceStore(db_path=tmp_path / "traces.db")
|
||||
yield store
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def grade_store(tmp_path: Path) -> SQLiteGradeStore:
|
||||
store = SQLiteGradeStore(db_path=tmp_path / "grades.db")
|
||||
yield store
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def provider() -> ArchetypeProvider:
|
||||
p = ArchetypeProvider()
|
||||
for fixture in (STRONG_BUILDER, LAZY_BUILDER, STRUGGLING_BUILDER):
|
||||
p.register(fixture)
|
||||
return p
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def engine(trace_store, grade_store, provider) -> GradingEngine:
|
||||
return GradingEngine(
|
||||
trace_store, grade_store, TraceIntegrityMap(), provider, model="gemma4:31b"
|
||||
)
|
||||
|
||||
|
||||
async def _grade_fixture(engine, trace_store, fixture) -> dict:
|
||||
for event in fixture.events:
|
||||
trace_store.append(event)
|
||||
record = await engine.grade(fixture.learner_id, fixture.task_id)
|
||||
assert record.verdict == "GRADED", (
|
||||
f"fixture {fixture.fixture_id} did not grade cleanly: {record.verdict}"
|
||||
)
|
||||
return record.scores
|
||||
|
||||
|
||||
class TestFixtureAlignment:
|
||||
"""D-021: fixture IDs align with the v0.2 corpus scenario IDs."""
|
||||
|
||||
def test_every_fixture_id_is_corpus_scenario_scoped(self):
|
||||
for fixture in TRACE_FIXTURES.values():
|
||||
assert fixture.fixture_id.count("::") == 1
|
||||
scenario_part, trace_part = fixture.fixture_id.split("::")
|
||||
assert scenario_part in LAB_SCENARIOS, (
|
||||
f"{scenario_part!r} is not a v0.2 lab scenario id"
|
||||
)
|
||||
assert trace_part.endswith("-trace")
|
||||
# double-brace-free single slug
|
||||
assert "-" in trace_part
|
||||
|
||||
def test_every_fixture_aligns_to_an_existing_scenario(self):
|
||||
assert set(f.aligned_scenario_id for f in TRACE_FIXTURES.values()) <= set(
|
||||
LAB_SCENARIOS
|
||||
)
|
||||
|
||||
def test_fixture_ids_are_valid_strings(self):
|
||||
for fixture in TRACE_FIXTURES.values():
|
||||
assert fixture.fixture_id and isinstance(fixture.fixture_id, str)
|
||||
assert fixture.learner_id == CALIBRATION_LEARNER_ID
|
||||
assert fixture.learner_id.startswith("learner-00")
|
||||
|
||||
def test_lazy_alignment_is_the_flagged_scenario(self):
|
||||
# v0.2's paste-and-run archetype is the flagged lab scenario (D-021).
|
||||
assert LAZY_BUILDER.aligned_scenario_id == "lab-scenario-flagged"
|
||||
assert LAZY_BUILDER.archetype == "lazy"
|
||||
|
||||
def test_all_three_archetypes_present(self):
|
||||
assert {f.archetype for f in TRACE_FIXTURES.values()} == {
|
||||
"strong",
|
||||
"lazy",
|
||||
"struggling",
|
||||
}
|
||||
|
||||
def test_events_are_valid_telemetry_events(self):
|
||||
from ai_service.telemetry.models import TelemetryEvent
|
||||
|
||||
for fixture in TRACE_FIXTURES.values():
|
||||
assert len(fixture.events) > 0
|
||||
for e in fixture.events:
|
||||
assert isinstance(e, TelemetryEvent)
|
||||
assert e.seq >= 0
|
||||
# contiguous seqs: a grading trace must clear the G-4 gate
|
||||
seqs = [e.seq for e in fixture.events]
|
||||
assert seqs == list(range(len(seqs))), (
|
||||
f"{fixture.fixture_id} has seq gaps — it would trip the gate"
|
||||
)
|
||||
|
||||
|
||||
class TestDigestSeparatesArchetypes:
|
||||
"""The non-LLM half of calibration: digests differ in the RIGHT direction.
|
||||
|
||||
If these deterministic assertions ever fail, no rubric — however good —
|
||||
could separate the archetypes: the digest would be lossy in the wrong
|
||||
places. These are the features the a-4 advisory and the rubric anchors
|
||||
speak about, computed over REAL trace fixtures.
|
||||
"""
|
||||
|
||||
def test_strong_shows_iterative_cycles_lazy_shows_none(self):
|
||||
strong = digest_of(STRONG_BUILDER)
|
||||
lazy = digest_of(LAZY_BUILDER)
|
||||
assert strong.error_fix_cycles >= 1
|
||||
assert lazy.error_fix_cycles == 0
|
||||
|
||||
def test_strong_tests_early_lazy_tests_late(self):
|
||||
strong = digest_of(STRONG_BUILDER)
|
||||
lazy = digest_of(LAZY_BUILDER)
|
||||
assert strong.first_test_pass_offset_s is not None
|
||||
assert lazy.first_test_pass_offset_s is not None
|
||||
assert strong.first_test_pass_offset_s < lazy.first_test_pass_offset_s
|
||||
|
||||
def test_struggling_never_passes(self):
|
||||
struggling = digest_of(STRUGGLING_BUILDER)
|
||||
assert struggling.final_test_status == "fail"
|
||||
assert struggling.test_pass_count == 0
|
||||
assert struggling.test_fail_count >= 3
|
||||
|
||||
def test_strong_and_struggling_both_end(self):
|
||||
strong = digest_of(STRONG_BUILDER)
|
||||
assert strong.final_test_status == "pass"
|
||||
assert strong.test_fail_count >= 1 # honest failure, then closed
|
||||
|
||||
def test_lazy_churns_without_test_progress(self):
|
||||
"""a-4 anchor: bulk edit + command churn with (almost) no test signal."""
|
||||
lazy = digest_of(LAZY_BUILDER)
|
||||
assert lazy.edit_count >= 2
|
||||
assert lazy.test_fail_count == 0
|
||||
assert lazy.command_count >= 2
|
||||
assert lazy.final_test_status == "pass" # ...and still one late test
|
||||
|
||||
|
||||
class TestOrderingContract:
|
||||
"""The scripted ORDERING CONTRACT through the full engine pipeline."""
|
||||
|
||||
async def test_strong_at_least_lazy_on_process_quality(
|
||||
self, engine, trace_store
|
||||
):
|
||||
strong = await _grade_fixture(engine, trace_store, STRONG_BUILDER)
|
||||
lazy = await _grade_fixture(engine, trace_store, LAZY_BUILDER)
|
||||
assert (
|
||||
strong["criteria"]["process_quality"]
|
||||
>= lazy["criteria"]["process_quality"]
|
||||
)
|
||||
|
||||
async def test_strong_above_struggling_on_correctness(self, engine, trace_store):
|
||||
strong = await _grade_fixture(engine, trace_store, STRONG_BUILDER)
|
||||
struggling = await _grade_fixture(engine, trace_store, STRUGGLING_BUILDER)
|
||||
assert strong["criteria"]["correctness"] > struggling["criteria"]["correctness"]
|
||||
|
||||
async def test_full_ordering_across_all_three_archetypes(
|
||||
self, engine, trace_store, grade_store
|
||||
):
|
||||
scores = {
|
||||
arch: await _grade_fixture(engine, trace_store, f)
|
||||
for arch, f in (
|
||||
("strong", STRONG_BUILDER),
|
||||
("lazy", LAZY_BUILDER),
|
||||
("struggling", STRUGGLING_BUILDER),
|
||||
)
|
||||
}
|
||||
# the required ordering contract (task 3-2-02):
|
||||
assert (
|
||||
scores["strong"]["criteria"]["process_quality"]
|
||||
>= scores["lazy"]["criteria"]["process_quality"]
|
||||
)
|
||||
assert (
|
||||
scores["strong"]["criteria"]["correctness"]
|
||||
> scores["struggling"]["criteria"]["correctness"]
|
||||
)
|
||||
# Deliberate NON-ordering, asserted to document the calibration
|
||||
# intent: struggling > lazy on process quality. Process quality is
|
||||
# not outcome — the struggling builder iterated (edits between
|
||||
# runs, three test runs), the lazy builder only pasted once and
|
||||
# verified once (a-4 caps that at 1). A rubric that scores a
|
||||
# never-passing iterator BELOW a paste-and-runner on process is
|
||||
# miscalibrated on its own anchors.
|
||||
assert (
|
||||
scores["struggling"]["criteria"]["process_quality"]
|
||||
> scores["lazy"]["criteria"]["process_quality"]
|
||||
)
|
||||
# verdicts also ordered: mastered >= developing > not_yet
|
||||
assert scores["strong"]["verdict"] == "mastered"
|
||||
assert scores["lazy"]["verdict"] == "developing"
|
||||
assert scores["struggling"]["verdict"] == "not_yet"
|
||||
|
||||
# all three persisted under the calibration learner, latest-state
|
||||
grades = grade_store.list_for_learner(CALIBRATION_LEARNER_ID)
|
||||
assert len(grades) == 3
|
||||
assert {g.task_id for g in grades} == {
|
||||
"task-calibration-strong",
|
||||
"task-calibration-lazy",
|
||||
"task-calibration-struggling",
|
||||
}
|
||||
|
||||
async def test_prompt_carries_fixture_digest_not_raw_events(
|
||||
self, engine, trace_store, provider
|
||||
):
|
||||
"""D-028 on the calibration path: digest in the prompt, raw events
|
||||
never — the lazy fixture plants distinctive payload strings
|
||||
("eval.py", the bulk 240-line paste) that must NOT appear."""
|
||||
for event in LAZY_BUILDER.events:
|
||||
trace_store.append(event)
|
||||
await engine.grade(LAZY_BUILDER.learner_id, LAZY_BUILDER.task_id)
|
||||
|
||||
assert provider.calls == 1
|
||||
prompt = provider.last_user_prompt
|
||||
# the fixture's digest reached the LLM...
|
||||
assert "PROCESS TRACE DIGEST (JSON):" in prompt
|
||||
assert '"error_fix_cycles"' in prompt
|
||||
# ...but no raw trace material or identity did (D-028)
|
||||
assert "eval.py" not in prompt
|
||||
assert LAZY_BUILDER.learner_id not in prompt
|
||||
assert LAZY_BUILDER.task_id not in prompt
|
||||
|
||||
async def test_struggling_digests_reach_engine_without_gate(
|
||||
self, engine, trace_store, provider
|
||||
):
|
||||
for event in STRUGGLING_BUILDER.events:
|
||||
trace_store.append(event)
|
||||
record = await engine.grade(
|
||||
STRUGGLING_BUILDER.learner_id, STRUGGLING_BUILDER.task_id
|
||||
)
|
||||
assert record.verdict == "GRADED" # honest struggle is gradable
|
||||
assert record.scores["criteria"]["correctness"] == 0
|
||||
# a never-passing session still yields a real verdict — the grade
|
||||
# is low, not missing
|
||||
assert record.scores["verdict"] == "not_yet"
|
||||
@@ -0,0 +1,528 @@
|
||||
"""GradingEngine tests (Task 3-2-01, REQ-3-004, G-4 binding).
|
||||
|
||||
Contract under test:
|
||||
- G-4 gate FIRST: seq gaps OR integrity-flagged trace →
|
||||
verdict=UNGRADABLE_TRACE_INCOMPLETE with the gap list surfaced in
|
||||
scores; the LLM is NEVER called on those paths (asserted). Empty
|
||||
trace → UNGRADABLE_EMPTY_TRACE. All are first-class GradeRecords
|
||||
(returned + persisted), not exceptions.
|
||||
- Complete trace: prompt carries the digest JSON but NO raw trace
|
||||
material (D-028) — a distinctive marker planted in a command payload
|
||||
must be absent from every message the provider received.
|
||||
- Malformed LLM JSON → the D-020 retry path recovers (scripted bad
|
||||
first, good second) → graded.
|
||||
- Re-grade overwrites the stored record (GradeStore upsert).
|
||||
|
||||
Zero network: everything runs against the mock provider (conftest rule).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from pydantic import BaseModel, ValidationError
|
||||
|
||||
from ai_service.grading.engine import GradingEngine, RubricScore
|
||||
from ai_service.grading.store import SQLiteGradeStore
|
||||
from ai_service.llm.types import Message
|
||||
from ai_service.telemetry.ingest import TraceIntegrityMap
|
||||
from ai_service.telemetry.models import TelemetryEvent
|
||||
from ai_service.telemetry.store import SQLiteTraceStore
|
||||
from ai_service.variants.store import SQLiteVariantStore
|
||||
|
||||
T0 = datetime(2026, 9, 12, 1, 0, 0, tzinfo=UTC)
|
||||
RAW_MARKER = "SECRET-COMMAND-MARKER-7f3a"
|
||||
|
||||
|
||||
class RubricJSON:
|
||||
"""Canonical well-formed rubric payload (matches engine.RubricScore)."""
|
||||
|
||||
PAYLOAD: dict = {
|
||||
"criteria": {
|
||||
"process_quality": 3,
|
||||
"correctness": 4,
|
||||
"debugging_discipline": 3,
|
||||
"test_usage": 4,
|
||||
},
|
||||
"strengths": ["tight edit-test loops throughout"],
|
||||
"gaps": ["final commit discipline loose"],
|
||||
"verdict": "mastered",
|
||||
}
|
||||
|
||||
|
||||
class RecordingProvider:
|
||||
"""Mock LLMProvider: scripted replies + captured requests (no network).
|
||||
|
||||
Subclass-composes ai_service.llm.mock.MockProvider so the conftest
|
||||
cloud-free guard philosophy holds — but records every message list
|
||||
and counts calls, which the stock mocks don't do.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
from ai_service.llm.mock import MockProvider
|
||||
|
||||
self._mock = MockProvider()
|
||||
self.replies: list[str] = [] # consumed left-to-right; last repeats
|
||||
self.requests: list[list[Message]] = []
|
||||
self.calls = 0
|
||||
|
||||
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
self.calls += 1
|
||||
self.requests.append([Message(role=m.role, content=m.content) for m in messages])
|
||||
if self.replies:
|
||||
reply = self.replies.pop(0)
|
||||
else:
|
||||
reply = json.dumps(RubricJSON.PAYLOAD)
|
||||
return reply
|
||||
|
||||
async def stream_chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
yield await self.chat(
|
||||
messages, model=model, temperature=temperature, response_format=response_format
|
||||
)
|
||||
|
||||
|
||||
def _event(
|
||||
seq: int,
|
||||
kind: str,
|
||||
payload: dict | None = None,
|
||||
offset_s: float = 0.0,
|
||||
*,
|
||||
learner: str = "engine-learner",
|
||||
task: str = "engine-task",
|
||||
) -> TelemetryEvent:
|
||||
return TelemetryEvent(
|
||||
learner_id=learner,
|
||||
task_id=task,
|
||||
seq=seq,
|
||||
kind=kind,
|
||||
payload=payload or {},
|
||||
ts=T0 + timedelta(seconds=offset_s),
|
||||
sandbox_id="sbx-engine",
|
||||
)
|
||||
|
||||
|
||||
def _complete_trace(marker_command: bool = True) -> list[TelemetryEvent]:
|
||||
"""A healthy iterative trace; seq 0..N contiguous (no gaps).
|
||||
|
||||
`marker_command` plants a distinctive string inside a command payload
|
||||
— the D-028 assertion target: raw trace material must never reach the
|
||||
provider, so the marker must not appear in any captured request.
|
||||
"""
|
||||
if marker_command:
|
||||
cmd_payload = {"cmd": f"echo {RAW_MARKER} && pytest -q"}
|
||||
else:
|
||||
cmd_payload = {"cmd": "pytest -q"}
|
||||
return [
|
||||
_event(0, "activity", {"state": "starting"}, 0.0),
|
||||
_event(1, "file_diff", {"path": "a.py", "added": 12}, 10.0),
|
||||
_event(2, "command", cmd_payload, 20.0),
|
||||
_event(3, "test_result", {"passed": False, "exit_code": 1}, 25.0),
|
||||
_event(4, "file_diff", {"path": "a.py", "added": 4, "removed": 2}, 40.0),
|
||||
_event(5, "command", {"cmd": "pytest -q"}, 60.0),
|
||||
_event(6, "test_result", {"passed": True, "exit_code": 0}, 65.0),
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def trace_store(tmp_path: Path) -> SQLiteTraceStore:
|
||||
store = SQLiteTraceStore(db_path=tmp_path / "traces.db")
|
||||
yield store
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def grade_store(tmp_path: Path) -> SQLiteGradeStore:
|
||||
store = SQLiteGradeStore(db_path=tmp_path / "grades.db")
|
||||
yield store
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def integrity() -> TraceIntegrityMap:
|
||||
return TraceIntegrityMap()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def provider() -> RecordingProvider:
|
||||
return RecordingProvider()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def engine(trace_store, grade_store, integrity, provider) -> GradingEngine:
|
||||
return GradingEngine(
|
||||
trace_store, grade_store, integrity, provider, model="gemma4:31b"
|
||||
)
|
||||
|
||||
|
||||
def _ingest(store: SQLiteTraceStore, events: list[TelemetryEvent]) -> None:
|
||||
for e in events:
|
||||
store.append(e)
|
||||
|
||||
|
||||
class TestRubricScoreModel:
|
||||
"""The D-020 schema contract — engine-side validation rules."""
|
||||
|
||||
def _rubric(self, **overrides) -> dict:
|
||||
payload = json.loads(json.dumps(RubricJSON.PAYLOAD)) # deep copy
|
||||
payload.update(overrides)
|
||||
return payload
|
||||
|
||||
def test_well_formed_payload_validates(self):
|
||||
score = RubricScore.model_validate(RubricJSON.PAYLOAD)
|
||||
assert score.criteria["process_quality"] == 3
|
||||
assert score.verdict == "mastered"
|
||||
|
||||
def test_unknown_criterion_rejected(self):
|
||||
bad = self._rubric()
|
||||
bad["criteria"]["extra_criterion"] = 2
|
||||
with pytest.raises(ValidationError, match="criteria keys must be exactly"):
|
||||
RubricScore.model_validate(bad)
|
||||
|
||||
def test_missing_criterion_rejected(self):
|
||||
bad = self._rubric()
|
||||
del bad["criteria"]["test_usage"]
|
||||
with pytest.raises(ValidationError, match="criteria keys must be exactly"):
|
||||
RubricScore.model_validate(bad)
|
||||
|
||||
def test_out_of_range_score_rejected(self):
|
||||
bad = self._rubric()
|
||||
bad["criteria"]["process_quality"] = 5
|
||||
with pytest.raises(ValidationError, match="must be within 0-4"):
|
||||
RubricScore.model_validate(bad)
|
||||
|
||||
def test_unknown_verdict_rejected(self):
|
||||
bad = self._rubric()
|
||||
bad["verdict"] = "excellent"
|
||||
with pytest.raises(ValidationError, match="verdict must be one of"):
|
||||
RubricScore.model_validate(bad)
|
||||
|
||||
|
||||
class TestCompleteTraceGrading:
|
||||
async def test_graded_returns_validated_rubric_persisted_and_returned(
|
||||
self, engine, trace_store, grade_store, provider
|
||||
):
|
||||
_ingest(trace_store, _complete_trace())
|
||||
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "GRADED"
|
||||
assert record.model == "gemma4:31b"
|
||||
assert record.scores == RubricJSON.PAYLOAD
|
||||
assert record.scores["criteria"]["process_quality"] == 3
|
||||
assert record.digest.get("error_fix_cycles") == 1
|
||||
assert record.digest.get("final_test_status") == "pass"
|
||||
# digest persisted for auditability (D-028 reproducible input)
|
||||
assert record.digest.get("event_count") == 7
|
||||
|
||||
stored = grade_store.get("engine-learner", "engine-task")
|
||||
assert stored is not None
|
||||
assert stored.scores == record.scores
|
||||
assert stored.verdict == "GRADED"
|
||||
|
||||
async def test_prompt_contains_digest_but_no_raw_trace(self, engine, trace_store, provider):
|
||||
"""D-028: the provider saw the digest JSON but NEVER the raw trace.
|
||||
|
||||
The marker was planted inside a command payload (seq 2); it must be
|
||||
absent from every message of every request, while the digest marker
|
||||
line + a JSON object with the digest's field names must be present.
|
||||
"""
|
||||
_ingest(trace_store, _complete_trace(marker_command=True))
|
||||
|
||||
await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert provider.calls == 1 # happy path: exactly one LLM call
|
||||
all_messages = [m for req in provider.requests for m in req]
|
||||
prompt_text = "\n".join(m.content for m in all_messages)
|
||||
|
||||
assert RAW_MARKER not in prompt_text, (
|
||||
"raw trace material reached the LLM prompt (D-028 violation)"
|
||||
)
|
||||
# digest presence: marker line + a JSON object carrying digest fields
|
||||
assert "PROCESS TRACE DIGEST (JSON):" in prompt_text
|
||||
assert '"error_fix_cycles"' in prompt_text
|
||||
assert '"final_test_status"' in prompt_text
|
||||
# learner anonymity: identity strings never reach the LLM either
|
||||
assert "engine-learner" not in prompt_text
|
||||
assert "engine-task" not in prompt_text
|
||||
|
||||
async def test_system_prompt_has_rubric_and_a4_advisory(
|
||||
self, engine, trace_store, provider
|
||||
):
|
||||
_ingest(trace_store, _complete_trace())
|
||||
|
||||
await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
system = provider.requests[0][0]
|
||||
assert system.role == "system"
|
||||
assert "process_quality" in system.content
|
||||
assert "correctness" in system.content
|
||||
assert "debugging_discipline" in system.content
|
||||
assert "test_usage" in system.content
|
||||
# a-4: churn-without-test-progress is a process-quality NEGATIVE
|
||||
assert "churn is not" in system.content
|
||||
# 0-4 level anchors present for each criterion
|
||||
assert "ADVISORY" in system.content
|
||||
|
||||
|
||||
class TestGateFirst:
|
||||
"""G-4 binding: the gate fires BEFORE any grading work."""
|
||||
|
||||
async def test_gapped_trace_ungradable_llm_not_called(
|
||||
self, engine, trace_store, provider, grade_store
|
||||
):
|
||||
# seqs 0,1,3 stored — seq 2 missing (mirrors the task spec example)
|
||||
events = _complete_trace()
|
||||
gapped = [e for e in events if e.seq != 2]
|
||||
_ingest(trace_store, gapped)
|
||||
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
assert provider.calls == 0, "LLM was called despite a gapped trace (G-4)"
|
||||
# gap list surfaced in scores
|
||||
assert record.scores["missing_seqs"] == [2]
|
||||
assert record.scores["integrity_flag"] is None
|
||||
# first-class record: persisted, retrievable, empty digest
|
||||
stored = grade_store.get("engine-learner", "engine-task")
|
||||
assert stored is not None
|
||||
assert stored.verdict == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
assert stored.scores["missing_seqs"] == [2]
|
||||
assert stored.digest == {}
|
||||
assert stored.model == "none"
|
||||
|
||||
async def test_integrity_flagged_trace_ungradable_llm_not_called(
|
||||
self, engine, trace_store, integrity, provider
|
||||
):
|
||||
_ingest(trace_store, _complete_trace())
|
||||
integrity.mark("engine-learner", "engine-task", "INCOMPLETE_FLOODED")
|
||||
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
assert provider.calls == 0, "LLM was called despite integrity flag (G-4)"
|
||||
assert record.scores["integrity_flag"] == "INCOMPLETE_FLOODED"
|
||||
assert record.scores["missing_seqs"] == [] # complete rows, but untrusted
|
||||
|
||||
async def test_flag_wins_even_with_gaps(self, engine, trace_store, integrity, provider):
|
||||
"""Both gate signals set: the flag reason is surfaced (both listed)."""
|
||||
events = _complete_trace()
|
||||
_ingest(trace_store, [e for e in events if e.seq != 2])
|
||||
integrity.mark("engine-learner", "engine-task", "INCOMPLETE_FLOODED")
|
||||
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
assert provider.calls == 0
|
||||
assert record.scores["integrity_flag"] == "INCOMPLETE_FLOODED"
|
||||
assert record.scores["missing_seqs"] == [2]
|
||||
|
||||
async def test_empty_trace_ungradable(self, engine, provider, grade_store):
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "UNGRADABLE_EMPTY_TRACE"
|
||||
assert provider.calls == 0, "LLM was called for an empty trace"
|
||||
assert record.scores == {"integrity_flag": None, "missing_seqs": []}
|
||||
assert record.digest == {}
|
||||
stored = grade_store.get("engine-learner", "engine-task")
|
||||
assert stored is not None
|
||||
assert stored.verdict == "UNGRADABLE_EMPTY_TRACE"
|
||||
|
||||
|
||||
class TestD020RetryPath:
|
||||
async def test_malformed_first_response_recovers_via_retry(
|
||||
self, engine, trace_store, provider
|
||||
):
|
||||
"""First reply invalid (wrong shape, fenced), second valid → graded.
|
||||
|
||||
This exercises the D-020 layer-4 path INSIDE the engine through the
|
||||
real agents/structured.py — the engine does not reimplement it.
|
||||
"""
|
||||
_ingest(trace_store, _complete_trace())
|
||||
provider.replies = [
|
||||
'```json\n{"summary": "wrong shape"}\n```', # layer 3 rejects
|
||||
json.dumps(RubricJSON.PAYLOAD), # retry recovers
|
||||
]
|
||||
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert provider.calls == 2 # bounded to exactly one retry (layer 4)
|
||||
assert record.verdict == "GRADED"
|
||||
assert record.scores == RubricJSON.PAYLOAD
|
||||
# the retry feedback must carry the validation error back
|
||||
retry_request = provider.requests[1]
|
||||
retry_text = " ".join(m.content for m in retry_request)
|
||||
assert "previous response was invalid" in retry_text
|
||||
|
||||
async def test_persistently_malformed_raises_after_bounded_retry(
|
||||
self, engine, trace_store, provider
|
||||
):
|
||||
from ai_service.agents.structured import StructuredOutputError
|
||||
|
||||
_ingest(trace_store, _complete_trace())
|
||||
provider.replies = ["not json at all", "still not json"]
|
||||
|
||||
with pytest.raises(StructuredOutputError):
|
||||
await engine.grade("engine-learner", "engine-task")
|
||||
assert provider.calls == 2 # bounded: never more than one retry
|
||||
|
||||
|
||||
class TestRegrade:
|
||||
async def test_regrade_overwrites_stored_record(
|
||||
self, engine, trace_store, grade_store, provider
|
||||
):
|
||||
_ingest(trace_store, _complete_trace())
|
||||
|
||||
first = await engine.grade("engine-learner", "engine-task")
|
||||
assert first.verdict == "GRADED"
|
||||
|
||||
# second grade: the provider now scripts a DIFFERENT rubric outcome
|
||||
updated = json.loads(json.dumps(RubricJSON.PAYLOAD))
|
||||
updated["criteria"]["process_quality"] = 1
|
||||
updated["verdict"] = "not_yet"
|
||||
provider.replies = [json.dumps(updated)]
|
||||
|
||||
second = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert second.scores == updated
|
||||
assert second.created_at >= first.created_at
|
||||
# upsert, not append: exactly one row, holding the LATEST grade
|
||||
grades = grade_store.list_for_learner("engine-learner")
|
||||
assert len(grades) == 1
|
||||
assert grades[0].scores == updated
|
||||
assert grades[0].verdict == "GRADED"
|
||||
|
||||
async def test_regrade_after_gate_outcome_replaces_it(
|
||||
self, engine, trace_store, grade_store, provider, integrity
|
||||
):
|
||||
"""A later successful regrade wholesale-replaces a gate record —
|
||||
the GradeStore upsert contract applied across verdict kinds."""
|
||||
# first: gapped → gate record persisted
|
||||
events = _complete_trace()
|
||||
_ingest(trace_store, [e for e in events if e.seq != 2])
|
||||
gated = await engine.grade("engine-learner", "engine-task")
|
||||
assert gated.verdict == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
|
||||
# gap healed (late delivery): same pair regrades cleanly
|
||||
trace_store.append(next(e for e in events if e.seq == 2))
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "GRADED"
|
||||
stored = grade_store.get("engine-learner", "engine-task")
|
||||
assert stored is not None
|
||||
assert stored.verdict == "GRADED"
|
||||
assert stored.scores == RubricJSON.PAYLOAD
|
||||
assert len(grade_store.list_for_learner("engine-learner")) == 1
|
||||
|
||||
|
||||
class TestBoundary:
|
||||
def test_engine_module_imports_no_fastapi(self):
|
||||
"""D-027: the engine module must not import fastapi (transitively
|
||||
beyond the sanctioned agents/structured + llm + telemetry + prompts)."""
|
||||
import sys
|
||||
|
||||
import ai_service.grading.engine as engine_mod
|
||||
|
||||
code = open(engine_mod.__file__).read()
|
||||
assert "fastapi" not in code
|
||||
assert "from ..api" not in code and "from ..api." not in code
|
||||
# the one sanctioned agents import is the module-direct structured defense
|
||||
assert "from ..agents.structured import" in code
|
||||
assert "from ..agents import" not in code
|
||||
# sanity: the module really is loaded (no import cycle surprises)
|
||||
assert engine_mod.__name__ in sys.modules
|
||||
|
||||
|
||||
class TestModelValidationSmoke:
|
||||
"""RubricScore as a plain pydantic model (D-020 layer-3 target type)."""
|
||||
|
||||
def test_rubric_score_is_frozen_shape_for_scores_dict(self):
|
||||
score = RubricScore.model_validate(RubricJSON.PAYLOAD)
|
||||
dumped = score.model_dump()
|
||||
assert set(dumped["criteria"]) == {
|
||||
"process_quality",
|
||||
"correctness",
|
||||
"debugging_discipline",
|
||||
"test_usage",
|
||||
}
|
||||
assert isinstance(score, BaseModel)
|
||||
|
||||
|
||||
class TestVariantAnchorsShipment:
|
||||
"""Phase 4 MH#4: variant anchors + seed ship into grading (a-5 same bar)."""
|
||||
|
||||
@pytest.fixture
|
||||
def variant_engine(self, trace_store, grade_store, integrity, provider, tmp_path):
|
||||
store = SQLiteVariantStore(db_path=tmp_path / "variants.db")
|
||||
yield GradingEngine(
|
||||
trace_store,
|
||||
grade_store,
|
||||
integrity,
|
||||
provider,
|
||||
model="gemma4:31b",
|
||||
variant_store=store,
|
||||
), store
|
||||
store.close()
|
||||
|
||||
@pytest.fixture
|
||||
def plain_engine(self, trace_store, grade_store, integrity, provider):
|
||||
return GradingEngine(
|
||||
trace_store, grade_store, integrity, provider, model="gemma4:31b"
|
||||
)
|
||||
|
||||
async def test_variant_task_grades_with_anchors_and_seed(
|
||||
self, variant_engine, trace_store, provider
|
||||
):
|
||||
engine, vstore = variant_engine
|
||||
_ingest(trace_store, _complete_trace())
|
||||
|
||||
# A stored variant whose task_id matches the graded trace.
|
||||
from datetime import UTC
|
||||
from datetime import datetime as dt
|
||||
|
||||
from ai_service.variants.store import VariantRecord
|
||||
|
||||
vstore.save(
|
||||
VariantRecord(
|
||||
learner_id="engine-learner",
|
||||
task_id="engine-task",
|
||||
template_id="tpl-llm-judge",
|
||||
seed="cafe" * 16,
|
||||
params={"domain": "tutoring"},
|
||||
statement="Scripted statement long enough to be legal.",
|
||||
starter_files={"README.md": "x"},
|
||||
created_at=dt.now(UTC),
|
||||
)
|
||||
)
|
||||
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "GRADED"
|
||||
assert record.variant_seed == "cafe" * 16 # D-029 stamped
|
||||
user_msgs = [m.content for req in provider.requests for m in req if m.role == "user"]
|
||||
assert any("Expected effort envelope" in u for u in user_msgs)
|
||||
assert any("tpl-llm-judge" in u for u in user_msgs)
|
||||
assert any("expected_min_test_runs" in u for u in user_msgs)
|
||||
|
||||
async def test_non_variant_task_has_no_anchors(
|
||||
self, variant_engine, trace_store, provider
|
||||
):
|
||||
engine, _ = variant_engine
|
||||
_ingest(trace_store, _complete_trace())
|
||||
|
||||
record = await engine.grade("engine-learner", "engine-task")
|
||||
|
||||
assert record.verdict == "GRADED"
|
||||
assert record.variant_seed is None
|
||||
user_msgs = [m.content for req in provider.requests for m in req if m.role == "user"]
|
||||
assert not any("Expected effort envelope" in u for u in user_msgs)
|
||||
|
||||
async def test_plain_engine_stays_variant_blind(
|
||||
self, plain_engine, trace_store
|
||||
):
|
||||
_ingest(trace_store, _complete_trace())
|
||||
record = await plain_engine.grade("engine-learner", "engine-task")
|
||||
assert record.verdict == "GRADED"
|
||||
assert record.variant_seed is None
|
||||
@@ -0,0 +1,170 @@
|
||||
"""Trace digest tests (Task 3-1-01, REQ-3-004).
|
||||
|
||||
Contract: `compute_digest` is deterministic, bounded (< 4 KB), and carries
|
||||
NO raw trace material (commands, file contents, payload strings) — the raw
|
||||
trace never reaches the LLM (D-028).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
import pytest
|
||||
|
||||
from ai_service.grading.features import TraceDigest, compute_digest
|
||||
from ai_service.telemetry.models import TelemetryEvent
|
||||
|
||||
T0 = datetime(2026, 9, 12, 1, 0, 0, tzinfo=UTC)
|
||||
|
||||
|
||||
def _event(
|
||||
seq: int, kind: str, payload: dict | None = None, offset_s: float = 0.0
|
||||
) -> TelemetryEvent:
|
||||
return TelemetryEvent(
|
||||
learner_id="features-learner",
|
||||
task_id="features-task",
|
||||
seq=seq,
|
||||
kind=kind,
|
||||
payload=payload or {},
|
||||
ts=T0 + timedelta(seconds=offset_s),
|
||||
sandbox_id="sbx-features",
|
||||
)
|
||||
|
||||
|
||||
def _paste_and_run_trace() -> list[TelemetryEvent]:
|
||||
"""One big file dump, a single passing test at the very end (no cycles)."""
|
||||
return [
|
||||
_event(0, "activity", {"state": "starting"}, 0.0),
|
||||
_event(1, "file_diff", {"path": "main.py", "added": 180, "removed": 0}, 5.0),
|
||||
_event(2, "command", {"cmd": "python -m pytest -q"}, 30.0),
|
||||
_event(3, "test_result", {"passed": True, "exit_code": 0}, 45.0),
|
||||
_event(4, "activity", {"state": "idle"}, 50.0),
|
||||
]
|
||||
|
||||
|
||||
def _iterative_trace() -> list[TelemetryEvent]:
|
||||
"""Many small edits; failed runs interleaved; eventual pass (>= 2 cycles)."""
|
||||
events: list[TelemetryEvent] = [
|
||||
_event(0, "activity", {"state": "starting"}, 0.0),
|
||||
_event(1, "file_diff", {"path": "a.py", "added": 12}, 10.0),
|
||||
_event(2, "command", {"cmd": "pytest -q"}, 20.0),
|
||||
_event(3, "test_result", {"passed": False, "exit_code": 1}, 25.0),
|
||||
_event(4, "file_diff", {"path": "a.py", "added": 4, "removed": 2}, 40.0),
|
||||
_event(5, "file_diff", {"path": "b.py", "added": 6}, 50.0),
|
||||
_event(6, "command", {"cmd": "pytest -q"}, 60.0),
|
||||
_event(7, "test_result", {"passed": False, "exit_code": 2}, 65.0),
|
||||
_event(8, "file_diff", {"path": "a.py", "added": 3, "removed": 1}, 80.0),
|
||||
_event(9, "command", {"cmd": "pytest -q tests/"}, 90.0),
|
||||
_event(10, "test_result", {"passed": True, "exit_code": 0}, 95.0),
|
||||
_event(11, "activity", {"state": "idle"}, 100.0),
|
||||
]
|
||||
return events
|
||||
|
||||
|
||||
def test_paste_and_run_vs_iterative_produce_observably_different_digests() -> None:
|
||||
paste = compute_digest(_paste_and_run_trace())
|
||||
iterative = compute_digest(_iterative_trace())
|
||||
assert paste.error_fix_cycles == 0
|
||||
assert iterative.error_fix_cycles == 2
|
||||
assert paste.test_fail_count == 0
|
||||
assert iterative.test_fail_count == 2
|
||||
assert paste.edit_count < iterative.edit_count
|
||||
assert iterative.first_test_pass_offset_s is not None
|
||||
assert paste.first_test_pass_offset_s is not None
|
||||
# Iterative debugs longer before the first pass.
|
||||
assert iterative.first_test_pass_offset_s > paste.first_test_pass_offset_s
|
||||
|
||||
|
||||
def test_digest_is_deterministic() -> None:
|
||||
trace = _iterative_trace()
|
||||
assert compute_digest(trace) == compute_digest(list(reversed(trace))) # seq sort normalizes
|
||||
|
||||
|
||||
def test_empty_trace_yields_valid_zeroed_digest() -> None:
|
||||
digest = compute_digest([])
|
||||
assert digest.event_count == 0
|
||||
assert digest.final_test_status == "none"
|
||||
assert digest.first_test_pass_offset_s is None
|
||||
assert digest.mean_fix_latency_s is None
|
||||
assert isinstance(digest, TraceDigest)
|
||||
|
||||
|
||||
def test_digest_json_is_bounded_under_4kb() -> None:
|
||||
big = [
|
||||
_event(i, "command", {"cmd": f"grep PATTERN-{i} file-{i}.py"}, i * 1.0)
|
||||
for i in range(200)
|
||||
]
|
||||
serialized = compute_digest(big).model_dump_json()
|
||||
assert len(serialized.encode()) < 4096, f"digest too large: {len(serialized)}B"
|
||||
|
||||
|
||||
def test_no_raw_command_string_leaks_into_digest() -> None:
|
||||
marker = "SECRET-COMMAND-MARKER-7f3a"
|
||||
trace = [
|
||||
_event(0, "command", {"cmd": f"echo {marker} && cat /etc/hostname"}, 0.0),
|
||||
_event(1, "file_diff", {"path": marker + ".py"}, 1.0),
|
||||
_event(2, "run_result", {"exit_code": 0, "stdout": marker}, 2.0),
|
||||
]
|
||||
digest_json = compute_digest(trace).model_dump_json()
|
||||
assert marker not in digest_json, "raw payload material leaked into digest"
|
||||
|
||||
|
||||
def test_idle_gaps_computed_over_threshold() -> None:
|
||||
trace = [
|
||||
_event(0, "activity", {"state": "starting"}, 0.0),
|
||||
_event(1, "activity", {"state": "idle"}, 400.0), # > 120s gap
|
||||
_event(2, "activity", {"state": "idle"}, 500.0), # 100s gap (below)
|
||||
_event(3, "activity", {"state": "stopped"}, 800.0), # > 120s gap
|
||||
]
|
||||
digest = compute_digest(trace)
|
||||
assert digest.idle_gap_count == 2
|
||||
assert digest.idle_gap_total_s == pytest.approx(400.0 + 300.0, rel=1e-6)
|
||||
|
||||
|
||||
def test_command_category_histogram() -> None:
|
||||
trace = [
|
||||
_event(0, "command", {"cmd": "npm run build"}, 0.0),
|
||||
_event(1, "command", {"cmd": "pytest -q"}, 1.0),
|
||||
_event(2, "command", {"cmd": "ls -la"}, 2.0),
|
||||
_event(3, "command", {"cmd": "rm -rf build/"}, 3.0),
|
||||
_event(4, "command", {"cmd": "curl localhost:8420/health"}, 4.0),
|
||||
_event(5, "command", {"cmd": "python mystery.py"}, 5.0),
|
||||
]
|
||||
digest = compute_digest(trace)
|
||||
assert digest.command_categories == {
|
||||
"build": 1,
|
||||
"debug": 1,
|
||||
"file": 1,
|
||||
"nav": 1,
|
||||
"other": 1,
|
||||
"test": 1,
|
||||
}
|
||||
|
||||
|
||||
def test_daemon_topology_mix_is_tolerated() -> None:
|
||||
"""P2-verify P1: live traces carry activity+file_diff only — no crash."""
|
||||
trace = [
|
||||
_event(0, "activity", {"state": "starting"}, 0.0),
|
||||
_event(1, "file_diff", {"path": "made-by-exec.txt", "added": 1}, 2.0),
|
||||
_event(2, "activity", {"state": "idle"}, 3.0),
|
||||
]
|
||||
digest = compute_digest(trace)
|
||||
assert digest.final_test_status == "none"
|
||||
assert digest.edit_count == 1
|
||||
assert digest.test_pass_count == 0
|
||||
assert digest.kind_histogram.get("file_diff") == 1
|
||||
|
||||
|
||||
def test_run_result_exit_codes_fall_back_for_test_status() -> None:
|
||||
"""No test_result events: run_result exit codes decide pass/fail."""
|
||||
trace = [
|
||||
_event(0, "file_diff", {"path": "x.py"}, 0.0),
|
||||
_event(1, "run_result", {"exit_code": 1}, 10.0),
|
||||
_event(2, "file_diff", {"path": "x.py"}, 20.0),
|
||||
_event(3, "run_result", {"exit_code": 0}, 30.0),
|
||||
]
|
||||
digest = compute_digest(trace)
|
||||
assert digest.test_fail_count == 1
|
||||
assert digest.test_pass_count == 1
|
||||
assert digest.final_test_status == "pass"
|
||||
assert digest.error_fix_cycles == 1
|
||||
@@ -0,0 +1,268 @@
|
||||
"""SQLiteGradeStore tests (REQ-3-004, D-027).
|
||||
|
||||
Each test gets its own tmp-path SQLite file — no shared disk state. Covers:
|
||||
- save / get / list_for_learner roundtrip (all fields survive,
|
||||
including nested JSON dicts and the tz-aware created_at contract)
|
||||
- upsert-on-regrade: a second save with the same (learner_id, task_id)
|
||||
REPLACES the row wholesale — scores, verdict, created_at, digest,
|
||||
model and variant_seed all reflect the latest save (documented
|
||||
contract; deliberately opposite of TraceStore.append's dedup)
|
||||
- unknown (learner, task) pair -> None; unknown learner -> empty list
|
||||
- scoping: grades for other learners/tasks are never returned
|
||||
- variant_seed stays None until P4 (D-029) and survives a roundtrip
|
||||
- rows are detached: usable after the store is closed
|
||||
- WAL + synchronous=NORMAL pragmas actually applied to the DB file
|
||||
- concurrent writer + reader against the same DB file (a-3 smoke test)
|
||||
"""
|
||||
|
||||
import concurrent.futures
|
||||
import threading
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
import sqlalchemy as sa
|
||||
|
||||
from ai_service.grading.store import GradeRecord, SQLiteGradeStore
|
||||
|
||||
_BASE_TS = datetime(2026, 9, 12, 12, 0, 0, tzinfo=UTC)
|
||||
|
||||
|
||||
def make_grade(
|
||||
task_id: str = "task-1",
|
||||
learner_id: str = "learner-1",
|
||||
variant_seed: str | None = None,
|
||||
digest: dict[str, Any] | None = None,
|
||||
scores: dict[str, Any] | None = None,
|
||||
verdict: str = "STRONG",
|
||||
model: str = "gemma4:31b",
|
||||
created_at: datetime | None = None,
|
||||
) -> GradeRecord:
|
||||
"""Canonical kwargs builder — tests override only what they assert on."""
|
||||
return GradeRecord(
|
||||
learner_id=learner_id,
|
||||
task_id=task_id,
|
||||
variant_seed=variant_seed,
|
||||
digest=digest
|
||||
if digest is not None
|
||||
else {"error_fix_cycles": 3, "command_categories": {"build": 2}},
|
||||
scores=scores
|
||||
if scores is not None
|
||||
else {
|
||||
"criteria": {"process_quality": 3, "correctness": 4},
|
||||
"strengths": ["iterative debugging"],
|
||||
"gaps": ["no final test pass"],
|
||||
},
|
||||
verdict=verdict,
|
||||
model=model,
|
||||
created_at=created_at if created_at is not None else _BASE_TS,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def store(tmp_path: Path) -> SQLiteGradeStore:
|
||||
s = SQLiteGradeStore(db_path=tmp_path / "grades.db")
|
||||
yield s
|
||||
s.close()
|
||||
|
||||
|
||||
def test_save_and_get_roundtrip(store: SQLiteGradeStore) -> None:
|
||||
grade = make_grade()
|
||||
store.save(grade)
|
||||
|
||||
fetched = store.get("learner-1", "task-1")
|
||||
assert fetched is not None
|
||||
assert fetched.learner_id == "learner-1"
|
||||
assert fetched.task_id == "task-1"
|
||||
assert fetched.variant_seed is None # null until P4 (D-029)
|
||||
assert fetched.digest == grade.digest
|
||||
assert fetched.scores == grade.scores
|
||||
assert fetched.verdict == "STRONG"
|
||||
assert fetched.model == "gemma4:31b"
|
||||
assert fetched.created_at == _BASE_TS
|
||||
assert fetched.created_at.tzinfo is UTC # tz-normalized on read
|
||||
|
||||
|
||||
def test_get_unknown_pair_returns_none(store: SQLiteGradeStore) -> None:
|
||||
store.save(make_grade())
|
||||
|
||||
assert store.get("learner-1", "task-missing") is None
|
||||
assert store.get("learner-missing", "task-1") is None
|
||||
assert store.get("nobody", "nothing") is None
|
||||
|
||||
|
||||
def test_list_for_learner_roundtrip(store: SQLiteGradeStore) -> None:
|
||||
# Created out of insertion order; list must come back chronological.
|
||||
store.save(make_grade(task_id="task-c", created_at=_BASE_TS + timedelta(hours=2)))
|
||||
store.save(make_grade(task_id="task-a", created_at=_BASE_TS))
|
||||
store.save(make_grade(task_id="task-b", created_at=_BASE_TS + timedelta(hours=1)))
|
||||
|
||||
grades = store.list_for_learner("learner-1")
|
||||
assert [g.task_id for g in grades] == ["task-a", "task-b", "task-c"]
|
||||
assert all(g.learner_id == "learner-1" for g in grades)
|
||||
hours = (timedelta(hours=0), timedelta(hours=1), timedelta(hours=2))
|
||||
assert all(
|
||||
g.created_at == _BASE_TS + offset
|
||||
for g, offset in zip(grades, hours, strict=True)
|
||||
)
|
||||
assert all(g.created_at.tzinfo is UTC for g in grades)
|
||||
|
||||
|
||||
def test_list_for_learner_unknown_learner_returns_empty_list(
|
||||
store: SQLiteGradeStore,
|
||||
) -> None:
|
||||
assert store.list_for_learner("nobody") == []
|
||||
|
||||
|
||||
def test_lists_are_scoped_to_the_learner(store: SQLiteGradeStore) -> None:
|
||||
store.save(make_grade(learner_id="learner-1", task_id="task-1"))
|
||||
store.save(make_grade(learner_id="learner-2", task_id="task-1"))
|
||||
|
||||
assert [g.task_id for g in store.list_for_learner("learner-1")] == ["task-1"]
|
||||
assert [g.learner_id for g in store.list_for_learner("learner-2")] == ["learner-2"]
|
||||
# (learner-2, task-1) is a distinct row: same task_id, different grade.
|
||||
grades_2 = store.list_for_learner("learner-2")
|
||||
assert len(grades_2) == 1
|
||||
assert grades_2[0].learner_id == "learner-2"
|
||||
|
||||
|
||||
def test_regrade_overwrites_the_stored_row(store: SQLiteGradeStore) -> None:
|
||||
"""THE contract of this store (upsert on the PK pair, latest wins).
|
||||
|
||||
The grading engine re-grades a task as its rubric or input evolves;
|
||||
the second save replaces scores, verdict, created_at, digest, model
|
||||
and variant_seed wholesale — exactly one row survives per pair.
|
||||
"""
|
||||
first = make_grade(
|
||||
verdict="DEVELOPING",
|
||||
model="gemma4:31b",
|
||||
scores={"criteria": {"process_quality": 1}},
|
||||
digest={"error_fix_cycles": 0},
|
||||
created_at=_BASE_TS,
|
||||
)
|
||||
store.save(first)
|
||||
|
||||
second = make_grade(
|
||||
verdict="EXEMPLARY",
|
||||
model="gemma4:31b-p2",
|
||||
variant_seed="seed-77",
|
||||
scores={"criteria": {"process_quality": 4}},
|
||||
digest={"error_fix_cycles": 6},
|
||||
created_at=_BASE_TS + timedelta(hours=1),
|
||||
)
|
||||
store.save(second)
|
||||
|
||||
fetched = store.get("learner-1", "task-1")
|
||||
assert fetched is not None
|
||||
# The regrade replaced every field of the first save.
|
||||
assert fetched.verdict == "EXEMPLARY"
|
||||
assert fetched.model == "gemma4:31b-p2"
|
||||
assert fetched.variant_seed == "seed-77"
|
||||
assert fetched.scores == {"criteria": {"process_quality": 4}}
|
||||
assert fetched.digest == {"error_fix_cycles": 6}
|
||||
assert fetched.created_at == _BASE_TS + timedelta(hours=1)
|
||||
|
||||
# Latest-wins also holds in list_for_learner — one row, not two.
|
||||
grades = store.list_for_learner("learner-1")
|
||||
assert len(grades) == 1
|
||||
assert grades[0].verdict == "EXEMPLARY"
|
||||
|
||||
|
||||
def test_regrade_preserves_other_pairs(store: SQLiteGradeStore) -> None:
|
||||
# An upsert on (learner-1, task-1) must not touch (learner-1, task-2).
|
||||
store.save(make_grade(task_id="task-1", verdict="STRONG"))
|
||||
store.save(make_grade(task_id="task-2", verdict="DEVELOPING"))
|
||||
store.save(make_grade(task_id="task-1", verdict="EXEMPLARY"))
|
||||
|
||||
other = store.get("learner-1", "task-2")
|
||||
assert other is not None
|
||||
assert other.verdict == "DEVELOPING" # untouched by the task-1 regrade
|
||||
assert len(store.list_for_learner("learner-1")) == 2
|
||||
|
||||
|
||||
def test_empty_scores_dict_roundtrips(store: SQLiteGradeStore) -> None:
|
||||
# Legal shape: an UNGRADABLE_TRACE_INCOMPLETE record carries a verdict
|
||||
# but no scores (and here, no digest either).
|
||||
store.save(make_grade(verdict="UNGRADABLE_TRACE_INCOMPLETE", scores={}, digest={}))
|
||||
|
||||
fetched = store.get("learner-1", "task-1")
|
||||
assert fetched is not None
|
||||
assert fetched.scores == {}
|
||||
assert fetched.digest == {}
|
||||
assert fetched.verdict == "UNGRADABLE_TRACE_INCOMPLETE"
|
||||
|
||||
|
||||
def test_rows_are_detached_after_save(store: SQLiteGradeStore, tmp_path: Path) -> None:
|
||||
# The engine hands GradeRecords across layers; rows must survive the
|
||||
# store that produced them being closed (no open-session ORM magic).
|
||||
store.save(make_grade())
|
||||
fetched = store.get("learner-1", "task-1")
|
||||
store.close()
|
||||
|
||||
assert fetched is not None
|
||||
assert fetched.verdict == "STRONG"
|
||||
assert fetched.scores["criteria"]["process_quality"] == 3
|
||||
|
||||
# A fresh store on the same file sees the same row (durability).
|
||||
reopened = SQLiteGradeStore(db_path=tmp_path / "grades.db")
|
||||
try:
|
||||
again = reopened.get("learner-1", "task-1")
|
||||
assert again is not None
|
||||
assert again.verdict == "STRONG"
|
||||
finally:
|
||||
reopened.close()
|
||||
|
||||
|
||||
def test_pragmas_are_applied(store: SQLiteGradeStore) -> None:
|
||||
# Pragmas are per-connection; query through the store's engine so the
|
||||
# connect hook (not a default sqlite3 connection) is what we inspect.
|
||||
with store._engine.connect() as conn:
|
||||
(journal_mode,) = conn.execute(sa.text("PRAGMA journal_mode")).one()
|
||||
(synchronous,) = conn.execute(sa.text("PRAGMA synchronous")).one()
|
||||
assert journal_mode == "wal"
|
||||
# synchronous=NORMAL is 1 in SQLite's pragma numbering.
|
||||
assert synchronous == 1
|
||||
|
||||
|
||||
def test_concurrent_writer_and_reader_no_database_is_locked(tmp_path: Path) -> None:
|
||||
"""One thread saves while another reads in a tight loop (a-3).
|
||||
|
||||
Without WAL + busy_timeout this pattern reliably produces
|
||||
`OperationalError: database is locked` on SQLite. The assertion is
|
||||
that every reader call completes and the latest write lands intact.
|
||||
"""
|
||||
db_path = tmp_path / "grades.db"
|
||||
n_pairs = 60 # distinct tasks -> distinct PK pairs, one regrade each
|
||||
stop_writing = threading.Event()
|
||||
|
||||
writer = SQLiteGradeStore(db_path=db_path)
|
||||
reader = SQLiteGradeStore(db_path=db_path)
|
||||
try:
|
||||
|
||||
def write_grades() -> None:
|
||||
for seq in range(n_pairs):
|
||||
writer.save(
|
||||
make_grade(
|
||||
task_id=f"task-{seq}",
|
||||
created_at=_BASE_TS + timedelta(seconds=seq),
|
||||
)
|
||||
)
|
||||
stop_writing.set()
|
||||
|
||||
def read_grades() -> None:
|
||||
while not stop_writing.is_set():
|
||||
reader.list_for_learner("learner-1")
|
||||
# Final read after the writer is done.
|
||||
assert len(reader.list_for_learner("learner-1")) == n_pairs
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool:
|
||||
futures = [pool.submit(write_grades), pool.submit(read_grades)]
|
||||
for future in futures:
|
||||
future.result(timeout=30)
|
||||
|
||||
# Every save was an upsert on its own pair: nothing lost, none doubled.
|
||||
assert len(writer.list_for_learner("learner-1")) == n_pairs
|
||||
finally:
|
||||
reader.close()
|
||||
writer.close()
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user