From a4afcccd7fc9db89aa1e508e619a31b9df726c94 Mon Sep 17 00:00:00 2001 From: coreci Date: Tue, 29 Sep 2026 16:06:47 +0000 Subject: [PATCH 1/5] =?UTF-8?q?docs(init):=20validate=20specification=20?= =?UTF-8?q?=E2=80=94=20v0.3=20hosted=20milestone?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ---ci--- project: researchx phase: 0 milestone: v0.3 status: specify requirements: covered: [] partial: [] message: >- v0.3 'hosted' scope established (owner-directed 2026-09-29): auth + sessions (user/pass, admin-managed, OAuth deferred), per-user model API keys (encrypted at rest, resolution chain user→server→keyless), live-bound user-scoped surfaces (library, settings, suggestions, refine, graph/inbox, profile), backend hosting (SSE-safe), real inference as product default. Requirement set REQ-69..85 written; single-user/no-auth constraint superseded; config.json milestone_name set; ROADMAP v0.3 section added (P0..P5). ---/ci--- --- .ciagent/CHECKPOINT.json | 12 +++++------ .ciagent/PROJECT.md | 13 +++++++----- .ciagent/REQUIREMENTS.md | 31 +++++++++++++++++++++++++++ .ciagent/ROADMAP.md | 46 ++++++++++++++++++++++++++++++++++++++++ .ciagent/config.json | 4 ++-- 5 files changed, 93 insertions(+), 13 deletions(-) diff --git a/.ciagent/CHECKPOINT.json b/.ciagent/CHECKPOINT.json index 0f58587..9f3a004 100644 --- a/.ciagent/CHECKPOINT.json +++ b/.ciagent/CHECKPOINT.json @@ -1,8 +1,8 @@ { - "phase": 5, - "stage": "complete", - "milestone": "v0.2", - "phase_role": "final", + "phase": 0, + "stage": "specify", + "milestone": "v0.3", + "phase_role": "pre_execution", "attempts": 0, - "updated_at": "2026-09-27T16:30:00Z" -} + "updated_at": "2026-09-29T16:00:00Z" +} \ No newline at end of file diff --git a/.ciagent/PROJECT.md b/.ciagent/PROJECT.md index 2ceb7c7..88ef7e0 100644 --- a/.ciagent/PROJECT.md +++ b/.ciagent/PROJECT.md @@ -6,9 +6,10 @@ ResearchX (working name; rename deferred — naming collision audit on file, see ## Core Value A deep-research studio where you bring your own model, storage, and memory; choose a research methodology before a single query fires; and every claim in the final report is anchored to a visited, verified, quotable source. Simple enough to run in three clicks, transparent enough to audit line by line. -## Objective (two-milestone plan, owner-directed 2026-09-27) -1. **Milestone v0.1 — Prototype**: UI/UX only, mock data, the production Next.js frontend with all data behind a typed `ApiClient` interface backed by `MockApiClient`. Surfaces: Ask, Choose (all 8 methodology cards), Plan & Run, Reading Room v0, v2 surface mockups (project graph, task inbox). **Ships only after owner approval — no MVP work begins until approved.** -2. **Milestone v0.2 — MVP**: gated on prototype approval. FastAPI backend, custom async engine, adapter layer, trust loop, and integration (MockApiClient swapped for the OpenAPI-generated client). +## Objective (three-milestone plan, owner-directed; v0.3 scope directed 2026-09-29) +1. **Milestone v0.1 — Prototype** (COMPLETE, approved 2026-09-27, v0.1.8): UI/UX with mock data — production Next.js frontend behind a typed `ApiClient`/`MockApiClient` boundary. +2. **Milestone v0.2 — MVP** (COMPLETE, approved 2026-09-27, v0.2.5): FastAPI backend, async engine, adapter layer, trust loop, OpenAPI client swap. Live mode = core research loop only (G-002/G-009 ratified). +3. **Milestone v0.3 — Hosted** (owner-directed 2026-09-29): the hosted multi-user product — auth + sessions (user/pass, admin-managed accounts, OAuth deferred), per-user settings incl. model API keys (encrypted at rest; resolution chain user key → server key → keyless localhost), all prototype surfaces live-bound and user-scoped (library, settings persistence, suggestions, refine, graph/inbox, profile), backend hosting for the LXC deployment (SSE-safe), and real inference as the product default (mock demoted to dev/test fixture). The remaining 6 methodologies stay v0.4+ scope. ## Requirements (validated) 1. Five pluggable adapter slots: model (OpenAI-compatible, planner/writer split), storage (local FS), search (Tavily default, DuckDuckGo keyless fallback), memory (off — stubbed), secrets (env — OpenBao later) @@ -28,13 +29,15 @@ A deep-research studio where you bring your own model, storage, and memory; choo - Memory scopes (semantic caching), S3/git storage adapters, Exa/Perplexity/Google search, OpenBao secrets, Docker packaging ## Constraints -- Prototype must be approved before MVP work begins (owner directive) +- ~~Prototype must be approved before MVP work begins~~ (v0.1 approved 2026-09-27; v0.2 approved 2026-09-27) - Versioning starts at v0.1; MAJOR remains 0 until an owner-approved v1 release; all pre-v1 ships are MINOR/PATCH only (owner directive) +- ~~Single-user, local-first tool — no auth/multi-tenancy in MVP~~ **SUPERSEDED v0.3 (owner-directed 2026-09-29)**: multi-user hosted product — user/pass auth, admin-managed accounts (first registered user is admin), OAuth deferred to v0.4+; runs/library/settings partitioned by user +- Per-user model keys resolve per run: user's own key → server env-file key → keyless localhost fallback (owner directive) - Not a general-purpose chatbot; not a model host; not a closed content index (v1) - Paywalled academic access out of scope v1 — open APIs (ArXiv/PubMed) only -- Single-user, local-first tool — no auth/multi-tenancy in MVP - Ollama/vLLM supported via OpenAI-compatible endpoints only (no vendor-specific adapters) - Secrets never in app config or git; model capability honesty (warn when selected model is weak at structured outputs) +- No deploys without owner directive (D-019 carries); milestone merge to main halts at the owner approval gate (D-018 carries) ## Out of Scope - Paywalled academic sources; model hosting; multi-tenant/auth; native Ollama API adapter; domain registration/rename decision diff --git a/.ciagent/REQUIREMENTS.md b/.ciagent/REQUIREMENTS.md index d644847..882ff93 100644 --- a/.ciagent/REQUIREMENTS.md +++ b/.ciagent/REQUIREMENTS.md @@ -80,6 +80,33 @@ - **D-013/D-020**: openapi-typescript + openapi-fetch generated client bound behind ApiClient; RESEARCHX_API_MODE=mock|live, mock default until P3 - **D-014..D-017**: sse-starlette SSE + pydantic event bus; difflib quote anchoring ≥ 0.85; HTTP liveness live/stale/dead; JSON durable state atomically rewritten per transition +## Hosted (Milestone v0.3 — owner-directed 2026-09-29) + +### Auth & Sessions +- **REQ-69**: User/pass auth: user model with argon2id password hashing, register/login/logout/me routes, httpOnly cookie sessions (server-side session store), CSRF-safe same-origin posture; OAuth explicitly deferred to v0.4+ +- **REQ-70**: Admin-managed accounts: first registered user becomes admin; further accounts created (or invited) by an admin only — no open registration; admin can list/create/deactivate accounts +- **REQ-71**: User scoping: every run, stored report, and setting is partitioned by owner — the engine, storage paths, and library read-model are user-scoped; no user can read or act on another user's data through the API + +### Per-User Settings & Model Keys +- **REQ-72**: Per-user settings persistence (PUT/GET /api/config user-scoped): methodology defaults, storage directory preference, appearance slot; the existing read-only GET /api/config adapter status remains for the resolved-key view +- **REQ-73**: Per-user model API keys: masked entry in Settings (never returned in full — masked status only), encrypted at rest (per-user key store, key-encryption-key from env-file secrets scope, never in git/plaintext) +- **REQ-74**: Per-run model credential resolution chain: user's own key → server env-file key → keyless localhost:11434 fallback (existing fallback semantics preserved); the capability probe runs against the resolved key; config status surfaces which tier resolved +- **REQ-75**: Real inference as the product default: RESEARCHX_API_MODE=live is the default for the hosted instance; mock mode demoted to a dev/test fixture (REQ-31 zero-key fakes stay the CI discipline) + +### Live Surfaces (G-002 boundary lifted, user-scoped) +- **REQ-76**: Library backend: GET /api/library — folder-structure listing over the user's runs (auto-categorization by topic + methodology, recency/title sorting); report cards resolve to real runs +- **REQ-77**: Cross-report connections + project graph: GET /api/library/connections and GET /api/graph — shared-source and topical edges computed from the user's ledgers and run metadata +- **REQ-78**: Task inbox: GET /api/tasks — open questions from the user's runs as checkable follow-ups +- **REQ-79**: Suggestions: GET /api/suggestions — follow-up query suggestions derived from the user's unanswered questions (open inbox items joined with source runs); click-fills-the-box semantics preserved +- **REQ-80**: Refine: POST /api/refine/questions + /api/refine/compose — planner-model-driven clarifying exchange composing a refined question + suggested context (scripted prototype dialogue replaced by real model calls) +- **REQ-81**: User profile: GET/PUT /api/profile — local profile (display name, email) surfaced in the avatar user menu; no account deletion in v0.3 +- **REQ-82**: Live Reading Room fidelity: the live report renders structured sections with page-anchored citations (hover shows the stored quoted passage) and the ledger panel from the citation ledger — parity with the mock Reading Room v0 + +### Hosting +- **REQ-83**: Backend container: Dockerfile for the FastAPI backend (single uvicorn worker, non-root), two-service deployment (frontend + backend) via deploy.sh extension +- **REQ-84**: SSE-safe reverse proxy: HAProxy path routing /api → backend with streaming timeouts that do not buffer or cut long-lived SSE connections; frontend served with NEXT_PUBLIC_API_BASE_URL pointing at the public origin +- **REQ-85**: Hosted live mode: the deployed instance runs real inference in live mode (per REQ-75), per-user keys honored; auth guard covers the backend surface (no unauthenticated API access through the public URL) + ## V2 (Post-MVP backlog) ### Methodology Catalog Extension @@ -134,4 +161,8 @@ | REQ-25, REQ-26, REQ-33 (API, client, tooling) | v0.2 | P3 | complete (v0.2.3) | | REQ-17, REQ-32, REQ-34 (adversarial, E2E, round-trip) | v0.2 | P4 | complete (v0.2.4) | | All REQ-01..34 (milestone gate) | v0.2 | P5 | complete (v0.2.5 — **owner approval recorded 2026-09-27**, G-009 ratified; milestone merged to main) | +| REQ-69..71 (auth, sessions, scoping) | v0.3 | P1 | pending | +| REQ-72..74 (settings, per-user keys, resolution chain) | v0.3 | P2 | pending | +| REQ-75..82 (live default + live surfaces, user-scoped) | v0.3 | P3 | pending | +| REQ-83..85 (hosting; live default deployed) | v0.3 | P4 | pending | | REQ-35..51 | v2 backlog (post-MVP milestones) | — | pending | diff --git a/.ciagent/ROADMAP.md b/.ciagent/ROADMAP.md index 490ac75..76dc954 100644 --- a/.ciagent/ROADMAP.md +++ b/.ciagent/ROADMAP.md @@ -129,6 +129,52 @@ library/settings/suggestions/refine/graph stay mock-backed until v0.3+. Milestone branch merged to main and deleted — tags v0.2.0–v0.2.5 preserve the full history. Next: v0.3 planning. +## Milestone v0.3 — Hosted (feature) — owner-directed 2026-09-29 + +Scope (owner session 2026-09-29): one milestone combining the G-009-ratified live surfaces + hosting with three owner additions — auth/sessions (user/pass, admin-managed, OAuth deferred), per-user model API keys, and real inference as the product default. Tags run on the v0.3.x line; MAJOR stays 0; feature milestone — each phase ships a patch, the final phase's patch IS the milestone release. + +### Phase 0 — Pre-Execution (v0.3) +- **Description**: SPECIFY (v0.3 requirement set REQ-69..85; single-user constraint superseded), CLARIFY (auth/session mechanics, key-encryption, partitioning, mock's fate, Reading-Room fidelity), RESEARCH (FastAPI session/auth patterns, encrypted key stores, SSE-proxying, storage read-model), PLAN (vertical slices), GRILL, MVP/UX check. +- **Requirements covered**: none (planning artifacts) +- **Success criteria**: .ciagent/ files refreshed for v0.3; PLAN.md carries User-Facing Surface + Happy Path + UX Acceptance Criteria; PERSONAS.md v0.3 roster active; grill verdict recorded with binding decisions +- **Depends on**: v0.2 approval (recorded) +- **Status**: pending + +### Phase 1 — Auth + Sessions + User Scoping +- **Description**: User model (argon2id), register/login/logout/me, httpOnly cookie sessions (server-side store), first-user-is-admin + admin-managed accounts (REQ-70), auth guard on every API endpoint, runs/settings partitioned by owner (engine + storage paths). Frontend: login screen, real profile in user menu, logout. +- **Requirements covered**: REQ-69, REQ-70, REQ-71 +- **Success criteria**: unauthenticated API access returns 401; two users cannot see each other's runs; admin can create/deactivate accounts; existing engine tests green under an authenticated harness +- **Depends on**: 0 +- **Status**: pending + +### Phase 2 — Per-User Settings + Model Keys +- **Description**: Encrypted per-user key store (KEK from env-file secrets scope), per-user settings persistence (PUT/GET /api/config), per-run credential resolution chain (user key → server key → keyless localhost; REQ-74) with probe per resolved key, settings UI live-bound with masked keys. +- **Requirements covered**: REQ-72, REQ-73, REQ-74 +- **Success criteria**: a keyed user gets fully real inference end-to-end; keys never appear in API responses (masked only), logs, or traces; resolution tier surfaced in config status; zero-key tests stay green via REQ-31 fakes +- **Depends on**: 1 +- **Status**: pending + +### Phase 3 — Live Surfaces (user-scoped) +- **Description**: Library read-model (GET /api/library), connections + graph (GET /api/library/connections, GET /api/graph), task inbox (GET /api/tasks), suggestions (GET /api/suggestions), refine (POST /api/refine/*, real planner calls), profile (GET/PUT /api/profile), live Reading Room fidelity (structured report, anchored citations, ledger panel). G-002 boundary lifted; frontend live swaps with zero component rewrites (REQ-54). +- **Requirements covered**: REQ-76, REQ-77, REQ-78, REQ-79, REQ-80, REQ-81, REQ-82 +- **Success criteria**: every prototype surface works against real user data in live mode; suggestions/refine driven by real model calls; Reading Room parity with mock; ≤3-click path still green +- **Depends on**: 2 +- **Status**: pending + +### Phase 4 — Hosting + Real-Inference Default +- **Description**: Backend Dockerfile (single worker, non-root), two-service deployment via deploy.sh extension, SSE-safe HAProxy path routing (/api → backend, streaming timeouts), hosted instance runs live mode by default (mock demoted to dev/test fixture), auth guard verified through the public URL. +- **Requirements covered**: REQ-75, REQ-83, REQ-84, REQ-85 +- **Success criteria**: one-command deploy brings up both services; SSE streams unbuffered through the proxy; unauthenticated curl to the public /api returns 401; live-mode default proven on the staged instance +- **Depends on**: 3 +- **Status**: pending + +### Phase 5 — Final Review + Ship (FINAL) +- **Description**: Multi-persona review of the full milestone diff, audit, tag v0.3.5 = milestone release; pipeline HALTS at the merge_to_main owner approval gate (D-018 carries); deployment to research.coreci.dev only on owner directive (D-019 carries). +- **Requirements covered**: milestone gate (REQ-69..85) +- **Success criteria**: review PASS (P0s fixed in-phase); audit PASS; v0.3.5 tagged + release; pipeline halted for the recorded owner decision +- **Depends on**: 4 +- **Status**: pending + ## Milestone Releases - **v0.1.4 = extended-prototype candidate** (superseded by the P5/P6 extension cycle) - **v0.1.5 = app shell + research library + settings + avatar** (P5 patch) diff --git a/.ciagent/config.json b/.ciagent/config.json index 58a998f..cfb7297 100644 --- a/.ciagent/config.json +++ b/.ciagent/config.json @@ -3,13 +3,13 @@ "projects": [], "active_project": "researchx", "milestone": "v0.3", - "milestone_name": "next-milestone-planning", + "milestone_name": "hosted", "milestone_type": "feature", "versioning": { "major_locked": true, "major_lock_note": "MAJOR remains 0 until an owner-approved v1 release; all pre-v1 ships are MINOR/PATCH only", "tag_line": "v0.3.x", - "next_milestone": "v0.3 scope definition pending owner direction (post-MVP backlog: live-bound library/settings/suggestions/refine, backend hosting, remaining 6 methodologies)" + "next_milestone": "v0.4 candidates (owner-directed post-v0.3): remaining 6 methodologies (REQ-35..40), OAuth deferred from v0.3 auth, Exa/Perplexity search adapters, S3/git storage, memory scopes" }, "autonomy": { "level": "full", -- 2.43.0 From 33f4fa003a5f77aa6e5a23031aa72f2cd5b65388 Mon Sep 17 00:00:00 2001 From: coreci Date: Tue, 29 Sep 2026 16:12:36 +0000 Subject: [PATCH 2/5] =?UTF-8?q?decision(P00):=20clarification=20=E2=80=94?= =?UTF-8?q?=20v0.3=20hosted=20auth,=20key=20store,=20partitioning,=20live?= =?UTF-8?q?=20surfaces?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ---ci--- phase: 0 milestone: v0.3 status: clarify decisions: - id: D-101 decision: "Sessions: server-side store (in-process dict + durable sessions.json), httpOnly SameSite=Lax cookies, opaque tokens, 14-day sliding expiry, immediate revocation" rationale: "Single uvicorn worker makes in-process sound; server-side is the only posture satisfying REQ-70 deactivation" confidence: 0.90 - id: D-102 decision: "argon2id via argon2-cffi, verify via asyncio.to_thread" rationale: "REQ-69 names argon2id; standard maintained implementation" confidence: 0.95 - id: D-103 decision: "401 guard on all /api/* except login/register; CSRF = SameSite=Lax + custom header on non-GET; /healthz outside /api" rationale: "Same-origin proxy + JSON fetch + custom header kills cross-site CSRF without token round-trips" confidence: 0.82 - id: D-104 decision: "Register open only while user store empty (first user = admin); then admin-created accounts with admin-set passwords" rationale: "Only mechanism without CLI/mail; race-safe under single asyncio worker" confidence: 0.85 - id: D-105 decision: "user_id = 8-char slug, email login, no email verification" rationale: "House ID style; filesystem-safe partition key" confidence: 0.75 - id: D-106 decision: "User-prefixed storage runs/// (supersedes D-010); owner_id in RunState; no migration" rationale: "Defense in depth; per-user library = single directory scan" confidence: 0.88 - id: D-107 decision: "Data root = RESEARCHX_DATA_DIR (dev backend/var/researchx; container /var/lib/researchx)" rationale: "cwd-dependent storage breaks under container workdirs" confidence: 0.85 - id: D-108 decision: "KEK = RESEARCHX_KEK (env-file 'keys' scope), AES-256-GCM with user_id as AAD; disabled honestly when unconfigured" rationale: "REQ-73 mandate; graceful degradation keeps zero-key dev green" confidence: 0.83 - id: D-109 decision: "Key store scope = model key only in v0.3" rationale: "REQ-73/74 name model keys; minimal encrypted surface" confidence: 0.80 - id: D-110 decision: "Model adapter constructed per run via factory resolving user→server→keyless; probe per resolved tier" rationale: "Shared adapter mutates _active_base globally (verified bug); per-user keys impossible on shared instance" confidence: 0.90 - id: D-111 decision: "GET /api/config stays read-only status; new GET/PUT /api/config/settings for per-user settings; storage path display-only hosted" rationale: "Two read models already split in the ApiClient" confidence: 0.87 - id: D-112 decision: "Code default stays mock; hosted image built with NEXT_PUBLIC_RESEARCHX_API_MODE=live; no live delegation to mock after P3" rationale: "Build-time inlining of NEXT_PUBLIC vars; zero-key discipline preserved" confidence: 0.85 - id: D-113 decision: "REQ-82 in scope as backend work: persist report.json + evidence.json at DONE; endpoints enriched; live exports use real data" rationale: "Evidence is in-memory only today; hover quotes impossible without persisted artifacts" confidence: 0.90 - id: D-114 decision: "Open questions = plan sub-questions answered-iff DONE; no synthetic suggestions; empty state for new users" rationale: "Real, persisted, user-scoped, honest, zero extra model cost" confidence: 0.65 - id: D-115 decision: "Read-models read disk artifacts, not RunRegistry; in-flight runs marked cancelled on restart; rehydration out of v0.3 scope" rationale: "RunRegistry is in-process; disk-backed read-models mandatory for hosted restarts" confidence: 0.80 requirements: covered: [] partial: [] ---/ci--- REQUIREMENTS.md REQ-69..85 refined with the decision outcomes; PROJECT.md carries the D-101..D-115 decision log. --- .ciagent/CHECKPOINT.json | 4 ++-- .ciagent/PROJECT.md | 12 ++++++++++-- .ciagent/REQUIREMENTS.md | 22 +++++++++++----------- 3 files changed, 23 insertions(+), 15 deletions(-) diff --git a/.ciagent/CHECKPOINT.json b/.ciagent/CHECKPOINT.json index 9f3a004..bbd1ee7 100644 --- a/.ciagent/CHECKPOINT.json +++ b/.ciagent/CHECKPOINT.json @@ -1,8 +1,8 @@ { "phase": 0, - "stage": "specify", + "stage": "clarify", "milestone": "v0.3", "phase_role": "pre_execution", "attempts": 0, - "updated_at": "2026-09-29T16:00:00Z" + "updated_at": "2026-09-29T16:10:00Z" } \ No newline at end of file diff --git a/.ciagent/PROJECT.md b/.ciagent/PROJECT.md index 88ef7e0..5f3ac63 100644 --- a/.ciagent/PROJECT.md +++ b/.ciagent/PROJECT.md @@ -59,8 +59,16 @@ Vision finalized and locked 2026-09-26: v1 methodology composition LOCKED (8 cor ## Key Decisions (v0.2 clarify — D-007..D-020, 2026-09-27) - Backend layout: `backend/` repo-root package, src-layout `researchx` (config, methodologies, adapters, engine, evidence, trust, api per ARCHITECTURE §Components); uv + uv-managed Python 3.12 (D-007/D-008) -- Ports/binding: backend 127.0.0.1:8000, frontend :3000 via NEXT_PUBLIC_API_BASE_URL; CORS localhost:3000 (D-009); run IDs = 8-char slugs; storage = /runs// (D-010) +- Ports/binding: backend 127.0.0.1:8000, frontend :3000 via NEXT_PUBLIC_API_BASE_URL; CORS localhost:3000 (D-009); run IDs = 8-char slugs; storage = /runs// (D-010 — **superseded by D-106 for v0.3+: runs///**) - Adapters: Ollama base URLs + key via env-file secrets scopes (OLLAMA_BASE_URL, OLLAMA_API_KEY); DDG keyless fallback = html.duckduckgo.com/html conservative scrape (D-011/D-012) - Integration: openapi-typescript + openapi-fetch → frontend/src/lib/api/generated, bound behind ApiClient; RESEARCHX_API_MODE=mock|live, mock default (D-013/D-020) - Transport/trust: sse-starlette SSE + pydantic typed event bus (D-014); difflib quote anchoring ≥ 0.85 (D-015); HTTP liveness live/stale/dead (D-016); JSON durable state, atomic rewrite per transition (D-017) -- Run termination: pipeline ships P1..P4 + P5 final patch, then HALTS at the owner approval gate before milestone → main (merge_to_main hook, v0.1 precedent); no deploys without owner directive (D-018/D-019) \ No newline at end of file +- Run termination: pipeline ships P1..P4 + P5 final patch, then HALTS at the owner approval gate before milestone → main (merge_to_main hook, v0.1 precedent); no deploys without owner directive (D-018/D-019) +## Key Decisions (v0.3 clarify — D-101..D-115, 2026-09-29) +- Sessions: server-side store (in-process dict + durable sessions.json, D-017 idiom), httpOnly SameSite=Lax cookies, opaque tokens, 14-day sliding expiry, immediate revocation (D-101); argon2id via argon2-cffi, verified via asyncio.to_thread (D-102) +- Auth posture: 401 on all /api/* except login/register (register open only while user store empty — first user is admin, then admin-managed; D-103/D-104); CSRF = SameSite=Lax + custom header on non-GET (EventSource GET exempt); /healthz outside /api for proxy checks +- Identity/partitioning: user_id = 8-char slug, email login (D-105); storage runs/// (D-106 supersedes D-010); data root = RESEARCHX_DATA_DIR (backend/var/researchx dev, /var/lib/researchx container; D-107) +- Key store: AES-256-GCM with RESEARCHX_KEK from the env-file 'keys' scope, user_id as AAD; disabled honestly when no KEK (D-108); scope = model key only (D-109); model adapter constructed per run (fixes shared-adapter fallback bug; D-110) +- Settings: GET /api/config stays read-only adapter status (user-contextualized resolved tier); new GET/PUT /api/config/settings for per-user settings; storage path display-only hosted (D-111) +- Mock fate: code default stays mock (zero-key dev/CI); hosted image built with NEXT_PUBLIC_RESEARCHX_API_MODE=live; no live delegation to mock after P3 (D-112) +- REQ-82 is backend work: engine persists report.json + evidence.json at DONE; endpoints enriched (D-113); open questions = plan sub-questions answered-iff DONE (D-114); read-models read disk artifacts, not RunRegistry; in-flight runs marked cancelled on restart; rehydration out of v0.3 scope (D-115) diff --git a/.ciagent/REQUIREMENTS.md b/.ciagent/REQUIREMENTS.md index 882ff93..c4b411d 100644 --- a/.ciagent/REQUIREMENTS.md +++ b/.ciagent/REQUIREMENTS.md @@ -83,27 +83,27 @@ ## Hosted (Milestone v0.3 — owner-directed 2026-09-29) ### Auth & Sessions -- **REQ-69**: User/pass auth: user model with argon2id password hashing, register/login/logout/me routes, httpOnly cookie sessions (server-side session store), CSRF-safe same-origin posture; OAuth explicitly deferred to v0.4+ -- **REQ-70**: Admin-managed accounts: first registered user becomes admin; further accounts created (or invited) by an admin only — no open registration; admin can list/create/deactivate accounts -- **REQ-71**: User scoping: every run, stored report, and setting is partitioned by owner — the engine, storage paths, and library read-model are user-scoped; no user can read or act on another user's data through the API +- **REQ-69**: User/pass auth: user model with argon2id password hashing (argon2-cffi, verified off the event loop), register/login/logout/me routes, httpOnly cookie sessions (server-side session store: in-process dict + durable JSON under the data root, immediate revocation on logout/deactivation), CSRF-safe same-origin posture (SameSite=Lax cookie + custom-header check on mutating requests); OAuth explicitly deferred to v0.4+ +- **REQ-70**: Admin-managed accounts: registration is open only while the user store is empty; the first registered user becomes admin; subsequently accounts are created by an admin only (admin-set initial password, no email infrastructure); admin can list/create/deactivate accounts (deactivation kills sessions; data retained) +- **REQ-71**: User scoping: every run, stored report, and setting is partitioned by owner — the engine, storage paths, and library read-model are user-scoped; storage layout becomes runs/// (supersedes D-010); owner_id is recorded in run state; no migration (pre-hosted artifacts disposable); no user can read or act on another user's data through the API ### Per-User Settings & Model Keys -- **REQ-72**: Per-user settings persistence (PUT/GET /api/config user-scoped): methodology defaults, storage directory preference, appearance slot; the existing read-only GET /api/config adapter status remains for the resolved-key view -- **REQ-73**: Per-user model API keys: masked entry in Settings (never returned in full — masked status only), encrypted at rest (per-user key store, key-encryption-key from env-file secrets scope, never in git/plaintext) -- **REQ-74**: Per-run model credential resolution chain: user's own key → server env-file key → keyless localhost:11434 fallback (existing fallback semantics preserved); the capability probe runs against the resolved key; config status surfaces which tier resolved -- **REQ-75**: Real inference as the product default: RESEARCHX_API_MODE=live is the default for the hosted instance; mock mode demoted to a dev/test fixture (REQ-31 zero-key fakes stay the CI discipline) +- **REQ-72**: Per-user settings persistence (PUT/GET /api/config/settings): methodology defaults, resolved dials, appearance slot, masked key status; storage directory is server-managed per-user — display-only in hosted mode (the free-path picker applies to the local single-user profile only) +- **REQ-73**: Per-user model API keys: masked entry in Settings (never returned in full — masked status only), encrypted at rest (per-user key store; KEK = RESEARCHX_KEK from the env-file secrets 'keys' scope; cipher AES-256-GCM with user_id as AAD; without a configured KEK the key store is disabled with honest status — chain falls to server/keyless); scope = model key (OLLAMA_API_KEY) only in v0.3 +- **REQ-74**: Per-run model credential resolution chain: user's own key → server env-file key → keyless localhost:11434 fallback (existing fallback semantics preserved); the model adapter is constructed per run (fixes the shared-adapter fallback bug); the capability probe runs against the resolved tier; config status surfaces which tier resolved (user|server|keyless) for the current user +- **REQ-75**: Real inference as the product default: the code default of RESEARCHX_API_MODE stays mock (zero-key dev/CI per REQ-31); the hosted frontend is built with NEXT_PUBLIC_RESEARCHX_API_MODE=live (build-time inlining); after v0.3 P3 no live-mode surface delegates to the mock; mock demoted to a dev/test fixture ### Live Surfaces (G-002 boundary lifted, user-scoped) - **REQ-76**: Library backend: GET /api/library — folder-structure listing over the user's runs (auto-categorization by topic + methodology, recency/title sorting); report cards resolve to real runs - **REQ-77**: Cross-report connections + project graph: GET /api/library/connections and GET /api/graph — shared-source and topical edges computed from the user's ledgers and run metadata -- **REQ-78**: Task inbox: GET /api/tasks — open questions from the user's runs as checkable follow-ups -- **REQ-79**: Suggestions: GET /api/suggestions — follow-up query suggestions derived from the user's unanswered questions (open inbox items joined with source runs); click-fills-the-box semantics preserved +- **REQ-78**: Task inbox: GET /api/tasks — open questions from the user's runs as checkable follow-ups; open questions = plan sub-questions of the user's runs, answered-iff the run reached DONE +- **REQ-79**: Suggestions: GET /api/suggestions — follow-up query suggestions derived from the user's unanswered questions (open inbox items joined with source runs); click-fills-the-box semantics preserved; empty state for users with no runs (no synthetic suggestions) - **REQ-80**: Refine: POST /api/refine/questions + /api/refine/compose — planner-model-driven clarifying exchange composing a refined question + suggested context (scripted prototype dialogue replaced by real model calls) - **REQ-81**: User profile: GET/PUT /api/profile — local profile (display name, email) surfaced in the avatar user menu; no account deletion in v0.3 -- **REQ-82**: Live Reading Room fidelity: the live report renders structured sections with page-anchored citations (hover shows the stored quoted passage) and the ledger panel from the citation ledger — parity with the mock Reading Room v0 +- **REQ-82**: Live Reading Room fidelity: the live report renders structured sections with page-anchored citations (hover shows the stored quoted passage) and the ledger panel from the citation ledger — parity with the mock Reading Room v0; the engine persists report.json (structured sections + anchored citations with quotes) and evidence.json (passage texts) at DONE; report/ledger endpoints expose them; live exports use real entry data (no fabricated fields) ### Hosting -- **REQ-83**: Backend container: Dockerfile for the FastAPI backend (single uvicorn worker, non-root), two-service deployment (frontend + backend) via deploy.sh extension +- **REQ-83**: Backend container: Dockerfile for the FastAPI backend (single uvicorn worker, non-root), two-service deployment (frontend + backend) via deploy.sh extension; RESEARCHX_DATA_DIR volume (default /var/lib/researchx); unauthenticated /healthz outside /api for proxy health checks - **REQ-84**: SSE-safe reverse proxy: HAProxy path routing /api → backend with streaming timeouts that do not buffer or cut long-lived SSE connections; frontend served with NEXT_PUBLIC_API_BASE_URL pointing at the public origin - **REQ-85**: Hosted live mode: the deployed instance runs real inference in live mode (per REQ-75), per-user keys honored; auth guard covers the backend surface (no unauthenticated API access through the public URL) -- 2.43.0 From 6f5c4d5aa181d4bd8b0b975b2fa47454e65ad74a Mon Sep 17 00:00:00 2001 From: coreci Date: Tue, 29 Sep 2026 16:49:12 +0000 Subject: [PATCH 3/5] =?UTF-8?q?docs(P00):=20research=20findings=20?= =?UTF-8?q?=E2=80=94=20v0.3=20hosted=20auth,=20key=20store,=20partitioning?= =?UTF-8?q?,=20hosting?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ---ci--- project: researchx phase: 0 milestone: v0.3 status: research requirements: covered: [] partial: [] message: >- RESEARCH-v03.md: 12 topics, 20 conclusions (C3-01..C3-20), 14 verified multi-user pitfalls (P-1..P-14). Key findings: shared-adapter fallback-mutation bug confirmed (ollama.py:73-79) — per-run factory mandatory; config_status reads private runs._actors (first multi-user breakage); all read endpoints die on restart — disk read-models are load-bearing for hosting; CORS lacks allow_credentials; module-level create_app() would serve fakes in production — new researchx.api.main entry required; frontend marker regex forces [p-N]→[^pN] normalization at the report.json boundary. ARCHITECTURE.md rewritten for v0.3 (new components: Identity & Sessions, Per-User Key Store, Model Adapter Factory; deployment topology). PERSONAS.md v0.3 roster: security- engineer phase-specific P1/P2/P4. config.json gains the 'keys' secrets scope (RESEARCHX_KEK). ---/ci--- --- .ciagent/ARCHITECTURE.md | 73 ++--- .ciagent/CHECKPOINT.json | 4 +- .ciagent/PERSONAS.md | 34 +-- .ciagent/RESEARCH-v03.md | 555 +++++++++++++++++++++++++++++++++++++++ .ciagent/config.json | 3 + 5 files changed, 619 insertions(+), 50 deletions(-) create mode 100644 .ciagent/RESEARCH-v03.md diff --git a/.ciagent/ARCHITECTURE.md b/.ciagent/ARCHITECTURE.md index e7396a3..504b3a7 100644 --- a/.ciagent/ARCHITECTURE.md +++ b/.ciagent/ARCHITECTURE.md @@ -1,7 +1,7 @@ # Architecture ## Overview -Two-milestone build (owner-directed 2026-09-27). Milestone v0.1 builds the UI/UX prototype as the production Next.js frontend with all data access behind a typed `ApiClient` interface backed by `MockApiClient` — no backend, zero throwaway work. Milestone v0.2 (gated on prototype approval) adds the FastAPI backend, the custom async research engine, and the adapter layer, then swaps `MockApiClient` for the OpenAPI-generated client. The engine is a hand-rolled asyncio state machine; every run persists its resolved 12-dial configuration and full event trace for reproducibility and auditability. +Three-milestone build (owner-directed). Milestone v0.1 (COMPLETE) built the UI/UX prototype as the production Next.js frontend behind a typed `ApiClient`/`MockApiClient` boundary. Milestone v0.2 (COMPLETE) added the FastAPI backend, custom async research engine, adapter layer, and trust loop, then swapped MockApiClient for the OpenAPI-generated client. Milestone v0.3 — **Hosted** — converts the product to a hosted multi-user service: auth + server-side sessions, per-user encrypted model keys, user-scoped storage and surfaces, and a two-service deployment behind an SSE-safe reverse proxy. The engine keeps its hand-rolled asyncio state machine; every run persists resolved dials, full trace, and structured report/evidence artifacts for reproducibility and auditability. ## Components @@ -13,61 +13,72 @@ Typed pydantic schema for the 12 dials (source universe, search routing, query s ### 2. Adapter Layer (`src/researchx/adapters/`) Five pluggable slots with versioned contracts: -- **Model**: any OpenAI-compatible endpoint (covers Ollama cloud/local and vLLM); separate planner/writer roles; capability probe on connect (context window estimate, conservative request cap = min(model claim, 8192), embedding batch 8) surfaced as UI overrides; structured-output capability warning. Default: ollama.com/v1 cloud-first with localhost:11434/v1 fallback. +- **Model**: any OpenAI-compatible endpoint (covers Ollama cloud/local and vLLM); separate planner/writer roles; capability probe on connect (context window estimate, conservative request cap = min(model claim, 8192), embedding batch 8) surfaced as UI overrides; structured-output capability warning. **v0.3:** adapters are constructed **per run** by the Model Adapter Factory (`model_factory.py`) — no shared instance; the per-instance fallback (cloud → localhost) preserves REQ-08 semantics without cross-run mutation; optional `api_key` constructor param carries the resolved per-user credential. - **Search**: Tavily default; DuckDuckGo keyless fallback; interface ready for Exa/Perplexity later. -- **Storage**: local filesystem adapter; reports as plain Markdown/JSON under user-chosen directory; run state as JSON after each transition. -- **Memory**: off in v1.0 — interface stubbed for semantic caching. -- **Secrets**: env-file provider in v1.0 — interface stubbed for OpenBao. +- **Storage**: local filesystem adapter; reports as plain Markdown/JSON under a user-chosen local directory; run state as JSON after each transition. **v0.3:** owner-scoped `LocalStorageAdapter(base_dir, owner_id)` instances — run layout `runs///` (supersedes D-010); data root via `RESEARCHX_DATA_DIR` (`researchx.config.paths.data_root()`); protocol gains `append_run_artifact`. +- **Memory**: off — interface stubbed for semantic caching. +- **Secrets**: env-file provider — interface stubbed for OpenBao. **v0.3:** the scopes map gains a `'keys'` scope (`RESEARCHX_KEK`) consumed by the per-user key store. - **Boundaries:** all external I/O lives here; engine never imports vendor SDKs directly. - **DependsOn:** config (dial schemas only) -### 3. Research Engine (`src/researchx/engine/`) -Custom asyncio state machine: PLANNING → AWAITING_APPROVAL → SEARCHING → EXTRACTING → REFLECTING → (depth loop) → SYNTHESIZING → VERIFYING → DONE. One RunActor per run; durable JSON state persisted to storage after each transition (pause/cancel/resume); typed event bus feeding SSE live trace; plan (sub-questions + source universe + outline) halts at AWAITING_APPROVAL until approved/edited. +### 3. Identity & Sessions (`src/researchx/auth/`) — NEW v0.3 +User model (8-char slug `user_id`, email login, argon2id password hashing via argon2-cffi, all hashing off the event loop), server-side session store (opaque `token_urlsafe(32)` httpOnly SameSite=Lax cookies; in-process dict + durable `sessions.json` at the data root; 14-day sliding expiry; immediate revocation on logout/deactivation; token rotation at login), auth guard as a router-level dependency (`Depends(require_user)`) covering all `/api/*` except login/register, CSRF check (custom header on non-GET), first-user-is-admin + admin-managed accounts (`/api/admin/users`), user store `users.json` at the data root. + +- **Boundaries:** owns user/session data + password hashing; no run knowledge. +- **DependsOn:** adapters (secrets for the KEK scope), storage (data-root persistence) + +### 4. Per-User Key Store (`src/researchx/auth/key_store.py`) — NEW v0.3 +AES-256-GCM envelope keyed by `RESEARCHX_KEK` (env-file 'keys' scope, 32 bytes b64/hex), `user_id` as AAD (ciphertext is bound to its owner — swapped files fail `InvalidTag`), `os.urandom(12)` nonce per write, `keys/.json` (v1/entries format, atomic rewrite, 0600). API surface is masked-only (`configured` + last-4). Without a configured KEK the store is disabled with honest status; the resolution chain falls to server/keyless. + +- **Boundaries:** owns key encryption/decryption; never returns key material. +- **DependsOn:** Identity, adapters/secrets + +### 5. Model Adapter Factory (`src/researchx/adapters/model_factory.py`) — NEW v0.3 +Per-run credential resolution (REQ-74): user's own key → server env-file key → keyless localhost:11434; constructs one model adapter per run and returns the resolved tier (`user|server|keyless`); the capability probe runs against the resolved tier; `GET /api/config` resolves the factory for the current user without creating a run (fingerprint-cached probe). Fixes the v0.2 shared-adapter fallback-mutation bug by construction. + +- **Boundaries:** resolves credentials and constructs adapters; no engine knowledge. +- **DependsOn:** Identity (user), Key Store, adapters (model, secrets) + +### 6. Research Engine (`src/researchx/engine/`) +Custom asyncio state machine: PLANNING → AWAITING_APPROVAL → SEARCHING → EXTRACTING → REFLECTING → (depth loop) → SYNTHESIZING → VERIFYING → DONE. One RunActor per run; durable JSON state persisted to storage after each transition (pause/cancel/resume); typed event bus feeding SSE live trace; plan (sub-questions + source universe + outline) halts at AWAITING_APPROVAL until approved/edited. **v0.3:** `RunState` carries `owner_id`; at DONE the engine persists two additional artifacts — `report.json` (structured sections + anchored citations with quotes) and `evidence.json` (passage texts) — written in the verify phase before the evidence store is discarded; citation markers are normalized `[p-N]` → `[^pN]` at the report.json boundary (report.md keeps the trust-loop form); startup sweep marks in-flight runs CANCELLED on process restart (rehydration out of v0.3 scope). - **Boundaries:** orchestrates; contains no HTTP handling and no vendor I/O. - **DependsOn:** adapters, methodologies, trust, evidence -### 4. Evidence Store (`src/researchx/evidence/`) -Visited-page store with stable passage IDs; passage extraction from fetched pages. The only source of writer context — the visited-only citation constraint is enforced by construction: the writer sees nothing else. +### 7. Evidence Store (`src/researchx/evidence/`) +Visited-page store with stable passage IDs; passage extraction from fetched pages. The only source of writer context — the visited-only citation constraint is enforced by construction: the writer sees nothing else. **v0.3:** serialized to `evidence.json` at DONE (passage texts survive the actor). - **Boundaries:** owns evidence data model. - **DependsOn:** adapters (search/storage only) -### 5. Trust Loop (`src/researchx/trust/`) +### 8. Trust Loop (`src/researchx/trust/`) Runs inside every methodology: generation-time visited-only constraint [E1]; post-run URL liveness pass with live/stale/dead classification [E1]; claim-to-passage quote anchoring (fuzzy match against stored passages) [E1b]; citation ledger JSON per run (URL, visited_at, passages, claims supported, health, authority tier); verification checklist export. - **Boundaries:** read side of evidence store; verification reports feed the ledger. - **DependsOn:** evidence, adapters (model for verification passes) -### 6. API Layer (`src/researchx/api/`) -FastAPI routes: POST /api/runs, GET /api/methodologies, GET /api/runs/{id}, approve/pause/cancel/resume, GET /api/runs/{id}/events (SSE), GET /api/config. OpenAPI schema generates the typed frontend client. +### 9. API Layer (`src/researchx/api/`) +FastAPI routes — **v0.3 inventory**: `/api/auth/{register,login,logout,me}` (public router; register open only while the user store is empty), `/api/admin/users*` (admin-guarded), `POST /api/runs`, `GET /api/methodologies`, `GET /api/runs/{id}`, approve/pause/cancel/resume, `GET /api/runs/{id}/events` (SSE), `GET /api/runs/{id}/report` (enriched: markdown + structured report + evidence), `GET /api/runs/{id}/ledger`, `GET /api/config` (read-only, user-contextualized: resolved tier, key-store status, per-user cost counters, probe), `GET/PUT /api/config/settings` (per-user settings), `GET /api/library`, `GET /api/library/connections`, `GET /api/graph`, `GET /api/tasks`, `GET /api/suggestions`, `POST /api/refine/{questions,compose}`, `GET/PUT /api/profile`, and `/healthz` outside `/api` (unauthenticated, for proxy health checks). Read-model routes (library/connections/graph/tasks/suggestions/profile/settings) read disk artifacts only — never the RunRegistry; the registry is owner-keyed `(owner_id, run_id)` and pruned of terminal runs. OpenAPI schema generates the typed frontend client. - **Boundaries:** transport only — no business logic. -- **DependsOn:** engine, methodologies, config +- **DependsOn:** engine, methodologies, config, auth, key store, model factory -### 7. Frontend (Next.js App Router, TypeScript strict) -App shell (root layout, P5): persistent sidebar (primary nav — New research / Library / Settings; adapter connection-status dots as a v2 preview) + top bar (⌘K navigation palette — screens + stored reports; avatar button → user menu with the local single-user profile). Screens: Ask (question + optional context; open-question suggestions below the form — follow-ups from unanswered questions of past runs, click-fills-the-box, REQ-67; opt-in Refine dialogue — inline drawer, scripted clarifying exchange, composed refined question + suggested context, REQ-68), Choose (methodology cards with ETA/cost/why-recommended; dials behind "advanced"), Plan & Run (approve/edit plan, live SSE trace, pause/cancel), Reading Room v0 (rendered report, hover-a-citation → exact quoted passage, ledger panel, Markdown/JSON export), Library (P5: folder-structure report storage — auto-categorized folders, recency/title sorting, report cards into the Reading Room; cross-report Connections panel — shared-sources ⇄ / related-topic → edges; project graph + task inbox as v2-preview panels), Settings (P5: masked API-key status, storage directory, default methodology + resolved 12-dial display, appearance preview, about/version). +### 10. Frontend (Next.js App Router, TypeScript strict) +App shell (persistent sidebar + top bar with ⌘K slot + avatar). **v0.3 additions:** login screen; session-aware shell (avatar menu → real profile, logout); live binding of ALL surfaces (the G-002 mock-delegation boundary is lifted — `GeneratedApiClient` becomes a standalone `ApiClient` implementation with no `MockApiClient` inheritance); auth + CSRF fetch wrapper (`credentials: "include"`, `X-ResearchX-Request` on mutations); EventSource `withCredentials`; two build profiles (mock dev default; hosted build inlines `NEXT_PUBLIC_RESEARCHX_API_MODE=live` + empty same-origin base URL). Screens: Ask (question + context; suggestions below the form; opt-in refine dialogue — now real planner calls), Choose (methodology cards + dials drawer), Plan & Run (approve/edit, live SSE trace, pause/cancel/resume), Reading Room (structured report with anchored hover citations from report.json/evidence.json, ledger panel, exports), Library (folder structure, report cards, connections), Settings (per-user: masked model key, methodology defaults, model names, display-only storage), profile. -- **Boundaries:** consumes the typed `ApiClient` interface only — components never import the mock client directly. In v0.1 the interface is bound to `MockApiClient` (fixture data + scripted event replay); in v0.2 phase 3 the binding swaps to the OpenAPI-generated client with no component changes. -- **DependsOn:** ApiClient contract (v0.1: mock; v0.2: API layer) +- **Boundaries:** consumes the typed `ApiClient` interface only — components never import the mock or generated client directly; the boundary selects the binding by build profile. +- **DependsOn:** ApiClient contract ## Data Flow -v0.1 (prototype): Ask (question + context) → Choose (methodology slug + dial overrides) → MockApiClient returns scripted plan → Plan & Run (scripted live trace replay, pause/cancel) → Reading Room (mock report with anchored citations, ledger, real file exports). All flows exercise the exact component paths the MVP will use. +v0.3 (hosted): browser → HAProxy (TLS, `/api` + `/healthz` → backend :8000, SSE-safe: no response buffering, no API-backend compression, 3600s timeouts) → FastAPI (session guard resolves user → per-user settings + key store → model factory resolves credential tier user→server→keyless → owner-scoped engine run) → durable artifacts under `runs///` → read-models scan disk → Library/Graph/Tasks/Suggestions render → Reading Room renders report.json with anchored citations (hover quotes from evidence.json) → exports round-trip real data. Engine internals (state machine, evidence, trust loop) unchanged from v0.2. -v0.2 (MVP): Ask (question + context) → Choose (methodology slug + dial overrides) → POST /api/runs → engine PLANNING (planner model via model adapter, dial bundle resolved) → AWAITING_APPROVAL (plan persisted; SSE events; user approves/edits) → SEARCHING (search adapter, routing per dials) → EXTRACTING (fetch pages → evidence store, stable passage IDs) → REFLECTING (planner model; depth loop per dials) → SYNTHESIZING (writer model; context = evidence passages only, per-claim citation markers) → VERIFYING (liveness pass, quote anchoring, confidence labels) → DONE (report + ledger + checklist persisted to storage; Reading Room renders; export). +## Deployment Topology (NEW v0.3) +Two containers on the LXC host behind the existing HAProxy/TLS front (research.coreci.dev): `researchx` (Next.js frontend, :3000, live build args) and `researchx-api` (FastAPI backend, :8000 host-published, non-root uid 10001, `uvicorn --workers 1` — mandatory: in-memory registry + session store), `researchx-data` volume at `/var/lib/researchx` (RESEARCHX_DATA_DIR), runtime secrets via `--env-file` only. Rollback pairs `researchx-prev` / `researchx-api-prev`. deploy.sh builds both images at the tag, health-checks `/healthz` (backend) + `/ask` (frontend), and asserts live mode in the smoke test. ## Build Order -Milestone v0.1 (prototype): 1. Frontend scaffold + ApiClient/MockApiClient boundary → 2. Ask + Choose → 3. Plan & Run + Reading Room v0 → 4. v2 mockups + polish + 3-click validation → 5. review + owner approval gate (owner returned gate feedback → extension cycle: 5. app shell + research library + settings + avatar → 6. final review + approval re-presentation (2nd owner gate feedback: 7. ask suggestions + refine dialogue → 8. final re-review + ship)). +Milestone v0.1 (prototype — COMPLETE): frontend scaffold → Ask/Choose → Plan & Run/Reading Room → mockups/3-click → owner-gate extensions (app shell, library, settings, avatar; suggestions, refine) → approval. +Milestone v0.2 (MVP — COMPLETE): methodology framework → adapters → evidence → trust → engine → API → client swap → adversarial + E2E → approval. +Milestone v0.3 (hosted): 1. Auth + sessions + user scoping (storage partitioning, registry owner-keying, auth suite) → 2. Per-user settings + encrypted key store + model factory (credential chain, probe-per-tier) → 3. Live surfaces (read-models on disk artifacts, report.json/evidence.json + Reading Room fidelity, refine/suggestions real calls, frontend live swaps, mock-delegation ends) → 4. Hosting (backend Dockerfile + production entrypoint, two-service deploy.sh, SSE-safe HAProxy, live-default hosted build) → 5. Final review + owner gate. -Milestone v0.2 (MVP, gated on v0.1 approval): -1. Methodology framework (dial schema + 2 seed methodologies + registry) -2. Adapter layer (model/search/storage contracts + local implementations; fake adapters for tests) -3. Evidence store (visited pages, passage extraction) -4. Trust loop (liveness, anchoring, ledger) -5. Engine (state machine, event bus, durable state, pause/resume) -6. API layer (routes + SSE) -7. Integration (swap MockApiClient → OpenAPI-generated client) -8. Adversarial test suite (hallucinated-URL detection, quote-mismatch detection, pause/resume mid-depth-loop) + E2E happy path - -Build order is dependency-driven: trust verification requires evidence; engine requires adapters + trust; API requires engine; integration requires API; frontend components carry over unchanged from v0.1. \ No newline at end of file +Build order is dependency-driven: auth precedes key store (identity); key store precedes the factory (credential chain); read-models precede hosting (disk-backed reads must exist before redeploys); hosting precedes the gate (the owner reviews the deployed thing). \ No newline at end of file diff --git a/.ciagent/CHECKPOINT.json b/.ciagent/CHECKPOINT.json index bbd1ee7..54b148f 100644 --- a/.ciagent/CHECKPOINT.json +++ b/.ciagent/CHECKPOINT.json @@ -1,8 +1,8 @@ { "phase": 0, - "stage": "clarify", + "stage": "research", "milestone": "v0.3", "phase_role": "pre_execution", "attempts": 0, - "updated_at": "2026-09-29T16:10:00Z" + "updated_at": "2026-09-29T16:25:00Z" } \ No newline at end of file diff --git a/.ciagent/PERSONAS.md b/.ciagent/PERSONAS.md index cfbf0e3..63fa6cd 100644 --- a/.ciagent/PERSONAS.md +++ b/.ciagent/PERSONAS.md @@ -1,4 +1,4 @@ -# Personas — Milestone v0.2 (MVP) +# Personas — Milestone v0.3 (Hosted) --- active: @@ -8,8 +8,8 @@ active: - name: ci-lead-developer phase_specific: - name: security-engineer - phases: [2] -reason: "v0.2 is the MVP: FastAPI backend + engine + adapters + trust loop (backend/data domains activate), the frontend consumes the OpenAPI-generated client in P3 (frontend stays active for the binding swap + P4 E2E), and the trust loop in P2 needs a phase-specific security-engineer for secrets handling and liveness verification. Lead-developer coordinates." + phases: [1, 2, 4] +reason: "v0.3 is the hosted multi-user milestone: P1 lands auth/sessions/CSRF/user-scoping, P2 lands the encrypted key store + credential chain, and P4 exposes the whole surface through a public proxy — security review is needed at all three, but not at P3 (live surfaces are read-model + binding work with no new secret handling) or P5 (the review phase runs the full multi-persona pass anyway)." --- ## Roster @@ -18,8 +18,8 @@ reason: "v0.2 is the MVP: FastAPI backend + engine + adapters + trust loop (back - **Domain**: backend - **Active**: true - **Phase-specific**: false -- **Frameworks**: fastapi, pydantic v2, uv, uvicorn, sse-starlette, httpx, pytest, pytest-asyncio -- **Constraints**: src-layout package `researchx` in backend/; all external I/O lives in adapters — engine imports no vendor SDKs (httpx only inside adapters); no blocking calls in async paths; durable state via atomic tmp+os.replace; never swallow CancelledError; API layer is transport only (no business logic); single uvicorn worker (single-user) +- **Frameworks**: fastapi, pydantic v2, argon2-cffi, cryptography, uv, uvicorn, sse-starlette, httpx, pytest, pytest-asyncio +- **Constraints**: single uvicorn worker mandatory (in-memory registry + session store); all external I/O lives in adapters — engine imports no vendor SDKs; no blocking calls in async paths (argon2/AES via `asyncio.to_thread`); durable JSON via atomic tmp+os.replace; auth guard is router-dependency-based (`Depends(require_user)`), never BaseHTTPMiddleware (SSE safety); RunRegistry owner-keyed `(owner_id, run_id)` + pruned; engine contracts unchanged (per-run adapter injection); `create_app(auth_enabled=True)` default with parameter-only dev override — never an env var readable in production images - **Territory**: `backend/**` ### data-engineer @@ -27,30 +27,30 @@ reason: "v0.2 is the MVP: FastAPI backend + engine + adapters + trust loop (back - **Active**: true - **Phase-specific**: false - **Frameworks**: pydantic v2 (schemas), pytest -- **Constraints**: owns the pydantic schema contracts: dial schema (REQ-01), run state, evidence store models (stable passage IDs), citation ledger JSON schema (REQ-21); pure data + validation — no I/O in schema modules; every run persists resolved dial values (REQ-04) -- **Territory**: `backend/src/researchx/config/**`, `backend/src/researchx/evidence/**` +- **Constraints**: owns the new pydantic schema contracts: User/Session, per-user Settings, key-store envelope file, `RunState.owner_id` (serialization-compatible), report.json/evidence.json (marker normalization `p-N`→`pN`, frontend Report/Citation compatible), read-model payloads (library/tasks/suggestions/graph/profile); pure data + validation — no I/O in schema modules; every run persists resolved dial values (REQ-04 carries) +- **Territory**: `backend/src/researchx/config/**`, `backend/src/researchx/evidence/**` (+ schema review on `backend/src/researchx/auth/**` models) ### frontend-engineer - **Domain**: frontend - **Active**: true - **Phase-specific**: false -- **Frameworks**: next.js (App Router), react 19, typescript (strict), tailwind, vitest, openapi-fetch, openapi-typescript -- **Constraints**: components never import the mock client or the generated client directly — only the ApiClient boundary does; RESEARCHX_API_MODE=mock|live selects the binding, mock default; no secrets or env keys in the frontend (NEXT_PUBLIC_API_BASE_URL only); zero component rewrites in the P3 swap (REQ-54/REQ-26) +- **Frameworks**: next.js (App Router), react 19, typescript (strict), tailwind, vitest, openapi-fetch, eventsource +- **Constraints**: components never import the mock or generated client directly — only the ApiClient boundary does; PLANNED interface additions (login/logout/getSession, admin hooks) documented before P1; fetch `credentials:"include"` + `X-ResearchX-Request` on all mutations; EventSource `withCredentials: true`; build-time NEXT_PUBLIC_* inlining profiles (mock dev default / live hosted with empty same-origin base); zero component rewrites in live swaps (REQ-54); no secrets or env keys in the frontend - **Territory**: `frontend/**` -### security-engineer (phase-specific: P2) +### security-engineer (phase-specific: P1, P2, P4) - **Domain**: security -- **Active**: false (P2 shipped — removed per phase-specific policy) +- **Active**: true while P1/P2/P4 execute; deactivated between and after P4 - **Phase-specific**: true -- **Frameworks**: httpx, pytest, STRIDE analysis -- **Constraints**: secrets resolve from env-file scopes only (OLLAMA_API_KEY, TAVILY_API_KEY, GITEA_TOKEN via .ciagent/.env.secrets) — never shell env, never logged, never in git; liveness verification and trust-loop code must not leak keys in errors or traces; SSRF guard for fetched URLs (private-address refusal); authority tiering must not phone home -- **Territory**: `backend/src/researchx/trust/**`, `backend/src/researchx/adapters/secrets*/**` -- **Removal**: deactivate after P2 ships +- **Frameworks**: argon2-cffi, cryptography (AESGCM), httpx, pytest, STRIDE analysis +- **Constraints**: KEK from env-file 'keys' scope only — never shell env, never logged, never in git, never in API responses (masked status only); AAD binds ciphertext to user_id; opaque tokens 32-byte urlsafe; SameSite=Lax + custom-header CSRF verified end-to-end; cross-user 404s provable by test; SSRF guard carries to any new fetch path; hosted exposure audit (401 on public /api, /healthz the only unauthenticated route); no `auth_enabled` env-var backdoor in production images +- **Territory**: `backend/src/researchx/auth/**` (P1/P2), `.ciagent/release/**` + hosting config review (P4) +- **Removal**: deactivate after P4 ships ### ci-lead-developer - **Domain**: coordination - **Active**: true - **Phase-specific**: false - **Frameworks**: ciagent workflows, git discipline -- **Constraints**: enforce persona territories; adapter/engine boundary review (I/O only in adapters); commit discipline with ---ci--- blocks; phase-boundary file updates only (no .ciagent/ writes during task execution); gates green before every ship -- **Territory**: `.ciagent/**`, repo root config files (README, VERSIONING, .gitignore) \ No newline at end of file +- **Constraints**: enforce persona territories; adapter/engine boundary review (I/O only in adapters); commit discipline with ---ci--- blocks; phase-boundary file updates only (no .ciagent/ writes during task execution); gates green before every ship; deploy.sh two-service extension (build both images, paired rollback containers, health on /healthz + /ask, live-mode build args asserted in the smoke test); secrets only from .ciagent/.env.secrets; no deploys without owner directive (D-019) and merge-to-main halts at the owner gate (D-018) +- **Territory**: `.ciagent/**`, repo root config files (README, VERSIONING, .gitignore, Dockerfiles/deploy scripts) \ No newline at end of file diff --git a/.ciagent/RESEARCH-v03.md b/.ciagent/RESEARCH-v03.md new file mode 100644 index 0000000..e23a38b --- /dev/null +++ b/.ciagent/RESEARCH-v03.md @@ -0,0 +1,555 @@ +# Research — v0.3 Hosted (Phase 0) + +## Run: 2026-09-29 (stage: research) + +Phase-0 research findings for milestone v0.3 (Hosted): auth + server-side +sessions, per-user encrypted model keys, user-scoped storage, live surfaces +backed by disk artifacts, and two-service hosting behind an SSE-safe proxy. +Each topic records concrete code/config shapes and pitfalls that PLAN +consumes. All library claims were verified against the pinned versions in +the repo (starlette 0.46.2, fastapi 0.115.14, sse-starlette, eventsource +polyfill 5.1.2). + +## 1. FastAPI auth/session layer + +**Recommendation:** +- New package `backend/src/researchx/auth/`: `users.py` (user model + + argon2id), `sessions.py` (server-side session store), `guard.py` + (session + CSRF dependency), `routes.py` (register/login/logout/me). +- **Route dependencies, not middleware, as the enforcement point.** Group + all protected routes under `APIRouter(dependencies=[Depends(require_user)])` + inside `create_app`. The dependency resolves the session cookie, loads + the user, runs the CSRF header check on non-GET, and returns the `User`: + ```python + async def require_user(request: Request) -> User: + token = request.cookies.get(SESSION_COOKIE) # "researchx_session" + session = sessions.get(token) # None if absent/expired/revoked + if session is None: + raise HTTPException(status_code=401, detail="authentication required") + if request.method != "GET" and "x-researchx-request" not in request.headers: + raise HTTPException(status_code=403, detail="missing request header") + user = users.get(session.user_id) + if user is None or not user.active: + raise HTTPException(status_code=401, detail="account disabled") + sessions.touch(session) # sliding expiry + return user + ``` + Rationale vs global middleware: the dependency appears in OpenAPI (401 + documented), is overridable per-test via `app.dependency_overrides` + (critical for the zero-key harness, topic 9), and avoids the pure-ASGI + middleware rewrite needed to keep SSE streaming safe. A public router + (`/api/auth/*`, `/healthz`) + protected router (everything else) + enforces "all /api/* except login/register" by construction. +- **Session store (D-101):** opaque token = `secrets.token_urlsafe(32)`; + in-process `dict[token, Session]` + durable `sessions.json` at the data + root, written with the atomic tmp+`os.replace` idiom (D-017). + `Session = {token_hash, user_id, created_at, last_seen, expires_at}`. + 14-day sliding expiry via `touch()`; persist only when >60s since last + persist. Immediate revocation = delete token + persist. Logout deletes + the presented token; deactivation deletes all of the user's tokens. + Rotate the token at login (kills session fixation by construction). +- **Cookie:** `researchx_session=`; `httponly=True, samesite="lax", + path="/"`, `secure=True` only in hosted/TLS deploys (config flag), no + `Domain` attribute (host-only). +- **argon2-cffi (D-102):** `PasswordHasher()` defaults are argon2id with + the RFC 9106 low-memory profile — keep the defaults. All calls off the + event loop via `asyncio.to_thread`; `VerifyMismatchError` → uniform 401 + "invalid email or password"; `check_needs_rehash` → rehash on login. +- **CORS ordering (verified):** the LAST-added middleware is OUTERMOST + (starlette 0.46.2 `build_middleware_stack`). Add `CORSMiddleware` AFTER + any auth middleware so preflight OPTIONS is answered without a 401. + Cookie auth requires `allow_credentials=True` + explicit localhost + origins (browsers reject wildcard origins with credentials). +- **First-user-is-admin (D-104):** `POST /api/auth/register` checks + `users.empty()` then inserts `role="admin"`; no `await` between check + and insert except the argon2 hash — the single uvicorn worker makes + this race-free by construction. After the first user, register 403s; + `POST /api/admin/users` (admin-guarded) creates accounts. +- **User store:** `users.json` at the data root: + `{"users": [{"user_id", "email", "password_hash", "display_name", + "role": "admin"|"user", "active": true, "created_at"}]}`; `user_id` = + 8-char slug from the run-ID alphabet + validation regex `^[a-z0-9]{8}$` + (D-105 — filesystem-safe partition key). + +**Pitfalls:** +- Do NOT use `BaseHTTPMiddleware` for the guard — it wraps response bodies + and has a history of SSE/streaming buffering bugs; router dependencies + sidestep the class. +- SameSite site-computation is scheme+registrable domain, NOT port — + `localhost:3000 → localhost:8000` is same-site (Lax cookie flows with + `credentials:"include"`), but `localhost:3000 → 127.0.0.1:8000` is + CROSS-site (Lax cookie silently dropped). Dev docs must pin both sides + to `localhost` (`NEXT_PUBLIC_API_BASE_URL=http://localhost:8000`). +- argon2 verify at default params takes ~50–100ms CPU — `to_thread` + mandatory for hash and verify. +- `EnvFileSecretsProvider` loads at construction; user/session/key stores + are mutable and must reload-or-rewrite atomically instead. + +## 2. CSRF posture (D-103) + +**Recommendation:** +- Combined defense, no token round-trips: (1) `SameSite=Lax` session + cookie blocks cross-site cookie attachment on background POSTs; (2) + every non-GET `/api` request must carry `X-ResearchX-Request: 1` + (checked in `require_user`). A cross-site attacker can neither attach + the Lax cookie to a background POST nor set custom headers without + CORS preflight approval. +- Frontend: the live client sends the header on every mutating call — + a small `#req()` wrapper or openapi-fetch middleware in + `GeneratedApiClient`: `headers: { "X-ResearchX-Request": "1" }`. +- **EventSource is exempt by design:** GET-only, cannot set custom + headers — the guard skips GETs, so `/api/runs/{id}/events` + authenticates by cookie alone. The eventsource polyfill (5.1.2) used + in E2E supports `withCredentials` and a fetch override. +- Login/register POSTs also require the header (uniform rule). +- Dev split-origin: `fetch(..., {credentials: "include"})` everywhere + + CORS `allow_credentials=True` + same-hostname origins (§1 pitfall). + +**Pitfalls:** +- If dev ever needs cross-SITE cookies (localhost vs 127.0.0.1), Lax + fails silently — the fix is same-hostname origins, not `SameSite=None`. +- `Secure` must NOT be set in plain-HTTP dev (browsers drop Secure + cookies on http); MUST be set on the TLS-fronted host. + +## 3. Per-user encrypted key store (D-108, D-109) + +**Recommendation:** +- New `backend/src/researchx/auth/key_store.py` using + `cryptography.hazmat.primitives.ciphers.aead.AESGCM` + (`AESGCM(key).encrypt(nonce, data, aad)` → ciphertext||16-byte tag; + `decrypt` raises `InvalidTag` on mismatch; nonce = 12 bytes; NEVER + reuse a nonce under the same KEK). +- **KEK (D-108):** `RESEARCHX_KEK` from the env-file **'keys' scope** + (`config.json secrets.scopes` gains `"keys": ["RESEARCHX_KEK"]`), + accepted as urlsafe-base64 or hex of exactly 32 bytes; decode errors + raise `AdapterError("RESEARCHX_KEK is not valid b64/hex")`; wrong + length raises (`must decode to 32 bytes (AES-256)`). +- **Envelope:** AAD = `user_id.encode()` (a copied file under another + user's directory fails `InvalidTag`); fresh `os.urandom(12)` nonce per + write; scope = `OLLAMA_API_KEY` only (D-109). +- **File format** `keys/.json` under the data root, atomic + rewrite, `0600`: + ```json + { + "v": 1, + "alg": "AES-256-GCM", + "updated_at": "2026-09-29T12:00:00Z", + "entries": { + "OLLAMA_API_KEY": {"nonce": "", "ct": "", "updated_at": "..."} + } + } + ``` +- **API never returns key material** (REQ-73): PUT accepts a raw key, GET + returns `{"configured": true, "masked": "…3f2a"}` (last 4 chars). + Crypto in `asyncio.to_thread` for discipline. +- Honest degradation: no KEK → PUT returns `{"key_store": "disabled"}`; + resolution chain proceeds server/keyless; corrupt entries (InvalidTag) + surface as error status, not silent tier drop. + +**Pitfalls:** +- GCM nonce reuse under a fixed KEK is catastrophic — always `os.urandom`. +- KEK rotation is out of v0.3 scope; `v` + `entries` leave room later. +- A changed KEK requires process restart (EnvFileSecretsProvider loads + once) — document it. + +## 4. Per-run model adapter factory (D-110) + +**Recommendation:** +- Verified shared-adapter bug (`ollama.py:73-79`): a cloud failure + permanently flips `self._active_base` on the SHARED instance — every + subsequent run by every user silently uses localhost. Fix = per-run + adapter instances. +- New `backend/src/researchx/adapters/model_factory.py`: + `ModelAdapterFactory(secrets, key_store, defaults)` with + `async resolve(user, settings) -> (OllamaModelAdapter, tier)` resolving + the chain user → server → keyless (REQ-74) and constructing one + adapter per run. +- `OllamaModelAdapter` constructor gains optional `api_key: str | None` + taking precedence over `secrets.resolve(...)`; the secrets path stays + for back-compat. Fallback mutation stays — now per-instance. +- **Threading through:** `create_app` constructs the factory once; + `create_run` does `model, tier = await factory.resolve(user, settings)` + and passes `model` into `RunActor` — **engine contracts unchanged**; + only the instance origin changes. `RunRegistry` keys become + `(owner_id, run_id)`. `state.json` records `owner_id` and + `credential_tier` for auditability. +- **Probe-per-run:** the existing `supports_probe` + `probe_model_adapter` + + `actor.attach_probe(...)` flow operates on the adapter passed to + `create_run` — per-tier automatically. `GET /api/config` resolves the + factory for the current user WITHOUT creating a run; cache in-process + keyed by `(user_id, tier, key_fingerprint)` with short TTL. + +**Pitfalls:** +- `config_status` reads `runs._actors` (private) to steal the last run's + probe (`app.py:174-177`) — breaks with zero runs and leaks another + user's probe; replace with the factory-resolved probe. +- Don't cache adapters across runs — per-run lifetime keeps + `_last_fallback_reason` honest. + +## 5. Storage partitioning (D-106, D-107) — exact touchpoints + +**Recommendation:** +- **Owner-scoped adapter instances, not owner-in-signatures.** + `LocalStorageAdapter(base_dir, owner_id)` binds the owner at + construction; `run_dir(run_id) -> base_dir/"runs"/owner_id/run_id`. + The `StorageAdapter` protocol stays byte-for-byte unchanged; only + constructors gain `owner_id` (validated `^[a-z0-9]{8}$`, D-105). +- App wiring: routes construct the owner-scoped adapter from the auth + dependency per request. `RunState` gains `owner_id: str | None = None` + (serialization-compatible; no migration per D-106). + `make_run_config(...)` gains `owner_id` and stamps it on state. +- `RunRegistry` keys `_actors[(owner_id, run_id)]`; cross-owner access + is a 404. +- **D-107 data root:** new `researchx.config.paths.data_root()`: + `os.environ["RESEARCHX_DATA_DIR"]` else `backend/var/researchx` (dev); + container `/var/lib/researchx`. Replaces the cwd-dependent + `Path.cwd()/".researchx"` (`app.py:150`). Layout: + ``` + $RESEARCHX_DATA_DIR/ + users.json sessions.json + runs///{state.json, trace.jsonl, report.md, + report.json, evidence.json, ledger.json, checklist.md} + keys/.json + settings/.json + ``` +- **Exact touchpoint list (verified in-tree):** + 1. `adapters/local_storage.py` — `__init__(owner_id)`, `run_dir`, + + `list_run_ids() -> list[str]` (read-models scan). + 2. `adapters/storage.py` — protocol unchanged; add `append_run_artifact` + (actor uses it, protocol lacks it). + 3. `engine/run_state.py` — `owner_id` field. + 4. `engine/actor.py` — owner-scoped storage; `make_run_config(owner_id)`; + artifacts at DONE (topic 6). + 5. `api/app.py` — per-request storage, registry keys, config display = + data root, `create_app(..., auth_enabled: bool = True)`. + 6. `api/dev_bridge.py` — owner + auth off. + 7. `engine/demo.py` — storage constructor gains owner. + 8. Tests constructing storage/actor: `test_local_storage.py` + (layout assertion + traversal), `test_engine.py`, + `test_probe_wiring.py`, `test_adversarial.py`, `test_resume.py`, + `test_integration_p1.py`, `test_p0_regressions.py`, `test_api.py` — + mechanical `owner_id="u1"`-style constructor args. + 9. Frontend untouched by partitioning (ids stay run-scoped). + +**Pitfalls:** +- `owner_id` must be REQUIRED in `LocalStorageAdapter` — an optional + default invites silent unpartitioned use. +- `read_run_artifact`'s error message embeds `runs/{run_id}/{name}` — + update to owner path; tests may pin it. +- Consolidate the duplicate run-id generators (`local_storage` vs + `run_config.new_run_id` — different alphabets) when touching both. + +## 6. New persisted artifacts: report.json + evidence.json (D-113) + +**Recommendation:** +- Persist both at DONE inside `_verify_phase` (actor.py:315-332), right + after `report.md`/`ledger.json` — deterministic, no model calls. +- **Marker normalization (frontend-compat crux):** the writer emits + `[p-N]` (trust loop regex `\[p-(\d+)\]`); the Reading Room parses + `[^pN]` with `[A-Za-z0-9]+` only (`report-body.ts:18` — `[^p-1]` does + NOT match). Normalize at persistence: in `report.json` section bodies + `[p-3]` → `[^p3]`, ids `p-3` → `p3`. `report.md` keeps the raw form. +- **report.json**: `{v, run_id, owner_id, title, question, methodology_slug, + generated_at, sections: [{heading, body}], citations: [{passage_id, url, + title, quote, health, authority_tier}]}`. Sections split from + report.md headings; title = first H1 else `"Deep Report — {question}"`. +- **evidence.json**: `{v, run_id, passages: [{passage_id, url, title, + quote, text}]}` — serialized from `EvidenceStore` (ids renumbered `pN`). +- **Endpoint enrichment:** `GET /api/runs/{id}/report` returns + `{run_id, markdown, report: {...}, evidence: {...}}` (markdown retained, + back-compat); ledger unchanged. Live `exportReport` switches to real + fields — kills the fabricated `claims_supported: 1` and empty quotes + (`live-client.ts:329-384`). +- Live-client `getReport` maps report.json → frontend `Report` + (citations → `Citation` with real quotes; tier map official→1, + reference→2, else 3). + +**Pitfalls:** +- Do NOT change `report.md`'s marker syntax — `_claims_per_passage`, + `extract_report_claims`, `_cited_urls` regex on `\[p-(\d+)\]`. +- Old DONE runs lack report.json — the endpoint must fall back to + markdown-only gracefully (pre-hosted artifacts disposable per D-106). +- `EvidenceStore` is per-actor and GC'd after the run — evidence.json + MUST be written inside `_verify_phase` (D-115 disk-reads later). + +## 7. Live read-models on disk artifacts (D-114, D-115) + +**Recommendation:** +- Read-models read disk, never RunRegistry (D-115): a small + `api/read_models.py` scans `runs//*/` loading `state.json` + (+ report.json/ledger.json where needed). +- **Restart sweep:** on `create_app` startup, rewrite every non-terminal + `state.json` as CANCELLED (legal from every non-terminal state, + verified `states.py`); in-flight actors die with the process; + rehydration out of scope (D-115). +- **Registry pruning:** `_actors`/`_buses` are never pruned today (slow + multi-user leak); drop terminal entries once no live SSE subscriber + (refcount in the events route); events 404 after prune/restart is + documented, trace persists on disk. +- **Open-question semantics (D-114):** open questions = plan + sub-questions of the user's runs, answered-iff `state == DONE`; no + synthetic suggestions; empty states for new users. +- Route shapes (owner-scoped via the auth dependency): + - `GET /api/library?sort=recent|title` → folders (auto-categorized by + methodology + deterministic topic tokens from plan sub-questions) + + `ReportSummary[]`. + - `GET /api/library/connections` → shared-source edges from ledger URL + overlap (≥1 shared URL); related-topic edges on methodology + + category overlap. + - `GET /api/graph` → same edges in the frontend `ProjectGraph` + node/edge shape. + - `GET /api/tasks` → `InboxItem[] {id: "#", question, + sourceRunId, status}` (done-iff-run-DONE else open). + - `GET /api/suggestions` → open items joined with run titles: + `SuggestedQuery[] {id, question, sourceRunId, sourceReportTitle}`. + - `POST /api/refine/questions {draft}` → planner call (2-3 clarifying + questions); `POST /api/refine/compose {draft, answers}` → planner + composes refined question + context. Non-GET ⇒ CSRF header applies. + - `GET/PUT /api/profile` → display_name/email. + - `GET/PUT /api/config/settings` (D-111): per-user settings + (`settings/.json`): model names, methodology defaults, + masked key status, storage path display-only. + - `GET /api/config` stays read-only, user-contextualized: + `resolved_tier: user|server|keyless`, `key_store: enabled|disabled`, + per-user cost counters (owner-keyed), probe (§4). +- Directory scans skip dotfiles/tmp leftovers defensively. +- Refine endpoints spend planner tokens — same credential chain; a + keyless-only user with no local Ollama gets an honest 503, not a hang. + +## 8. Hosting: backend Dockerfile + two-service deploy (REQ-83) + +**Recommendation:** +- `backend/Dockerfile` (uv idiom, non-root uid 10001, single worker): + `python:3.12-slim` + uv from ghcr image; `uv sync --frozen --no-dev`; + `useradd --system --uid 10001 researchx`; `mkdir -p /var/lib/researchx` + + chown; `USER researchx`; + `CMD ["uvicorn", "researchx.api.main:app", "--host", "0.0.0.0", + "--port", "8000", "--workers", "1", "--proxy-headers"]`. + `--workers 1` is a hard requirement (in-memory registry + sessions). +- **New production entrypoint `researchx.api.main`:** today `app.py`'s + module-level `app = create_app()` serves **FakeModelAdapter + + cwd-based storage** — hosting that would silently ship fakes. + `main.py` wires `create_app(data_root=RESEARCHX_DATA_DIR, factory= + , auth_enabled=True)`. `app.py` + stays the test factory. +- Secrets at runtime only: `RESEARCHX_KEK`, `OLLAMA_API_KEY`, + `OLLAMA_BASE_URL`, `TAVILY_API_KEY` via `--env-file` — never baked + into the image, never in git. +- **Plain docker-run pairs, not compose** — matches the existing + deploy.sh idiom and avoids a compose dependency on the LXC host: + `researchx-net` bridge network; `researchx-api` backend (env-file, + data volume, `-p 127.0.0.1:8000:8000` if HAProxy is host-side); + `researchx` frontend (`-p 3000:3000`, live build args). Keep + `researchx-api-prev` + `researchx-prev` for rollback. +- **deploy.sh extension:** rsync now includes `backend/`; build both + images on the host; health check BOTH `/healthz` (backend) and `/ask` + (frontend); update the HAProxy backend section idempotently. +- Data volume: pre-create + `chown 10001` host-side (belt-and-braces). +- New pyproject deps: `argon2-cffi>=23.1,<26`, `cryptography>=43`; no + `python-multipart` (JSON login bodies only). + +**Pitfalls:** +- The volume must be writable by uid 10001 — a root-owned bind mount + makes every artifact write EACCES. +- uvicorn graceful shutdown default lets in-flight runs exercise + `_persist_cancelled` on redeploy; `docker rm -f` SIGKILLs — the + startup sweep (§7) is the real safety net. + +## 9. HAProxy SSE configuration (REQ-84) + +**Recommendation:** +- The host ALREADY runs HAProxy 2.8 with TLS for research.coreci.dev — + this is a front-config extension, not a new component: + ``` + frontend https-in + acl is_api path_beg /api + acl is_health path /healthz + use_backend backend_api if is_api or is_health + default_backend frontend_next + + backend backend_api + option httpchk GET /healthz + default-server inter 5s fall 3 rise 2 + server api1 127.0.0.1:8000 check + timeout server 3600s + timeout tunnel 3600s + + backend frontend_next + timeout server 300s + server next1 127.0.0.1:3000 check + ``` +- HAProxy 2.x does not buffer response bodies by default in HTTP mode — + SSE streams natively. The three buffering hazards: NO + `option http-buffer-response`, NO compression on the API backend + (gzip buffers text/event-stream), keep default http-keep-alive. +- Verified: `sse-starlette` sends `Cache-Control: no-store` + + `X-Accel-Buffering: no` and emits `: ping` every **15s** + (`DEFAULT_PING_INTERVAL = 15`) — defeats idle timeouts. +- `timeout client` in global/defaults must exceed the 15s ping interval. +- Health check on `/healthz` (unauthenticated, outside /api per D-103): + `GET /healthz → 200 {"status":"ok"}`. + +**Pitfalls:** +- Enabling global compression "to save bandwidth" is the classic way SSE + silently degrades — scope it off for the API backend. +- Don't disable the sse-starlette ping. + +## 10. Next.js build-time env + EventSource credentials (D-112) + +**Recommendation:** +- `NEXT_PUBLIC_*` vars are inlined at BUILD time — two build profiles: + - **dev/test (default):** no env; mode resolves mock; zero keys + (REQ-31 preserved, D-112). + - **hosted:** `pnpm build` with + `NEXT_PUBLIC_RESEARCHX_API_MODE=live NEXT_PUBLIC_API_BASE_URL=""` — + empty base = same-origin through HAProxy's `/api` path route. + `baseUrl()` treats `""` as same-origin (openapi-fetch accepts a + relative base; EventSource accepts relative URLs). Pin dev live-mode + docs to `http://localhost:8000` (§1 hostname rule). +- `deploy.sh` frontend build gains the build args; the Dockerfile build + stage passes them (`ARG`/`ENV` before `pnpm build`). +- EventSource: `new EventSource(url, { withCredentials: true })` — the + polyfill supports the same init dict. Same-origin hosted mode doesn't + strictly need it; set it anyway. +- **Delegation ends (D-112):** `GeneratedApiClient extends MockApiClient` + today; in P3 every method is overridden against real endpoints and the + inheritance is dropped (a standalone class implementing `ApiClient`); + mock remains standalone. OpenAPI snapshot regen (G-005): the new + auth/settings/library/refine/admin routes change `openapi.json` — + commit the snapshot, regen the generated schema, add the ApiClient + interface methods (`login/logout/getSession`, admin hooks) as PLANNED + extensions (the v0.2 G-002 lesson: document every addition up front). + +**Pitfalls:** +- Forgetting the build args yields a hosted instance silently running + mock mode — add a deploy.sh smoke assertion: hosted `/api/config` + returns 401 + a live-mode marker. +- A non-empty `NEXT_PUBLIC_API_BASE_URL` in the hosted build breaks + same-origin cookies — assert `""` at build. + +## 11. Dev/E2E harness under auth + +**Recommendation:** +- `create_app(..., auth_enabled: bool = True)`. When False, the guard + resolves a synthetic dev user (`user_id="dev00000"`, role="admin") — + every route behaves exactly as authenticated, owner-scoped, zero keys. +- Existing suites keep running unchanged: backend API tests + (`test_api.py`, `test_adversarial.py`) and the real-stack E2E + (`dev_bridge.py`) construct with `auth_enabled=False`. Engine/trust + suites only need the `owner_id` constructor args (topic 5). +- **New auth suite (`test_auth.py`) with `auth_enabled=True`:** + first-user-admin (+403 after), wrong password 401, logout revocation, + 401 on protected routes, CSRF header rejection, cross-user 404, + admin create/deactivate (kills sessions), expiry, key-store + round-trip + AAD tamper rejection + masked-only responses. + +**Pitfalls:** +- Don't gate `auth_enabled` on an env var readable in production images + (a baked `RESEARCHX_AUTH=off` violates REQ-85); parameter-only, and + `main.py` never passes it. +- The E2E custom fetchImpl forwards headers verbatim — once the live + client adds the CSRF header, E2E keeps working via GETs + dev-disabled + guard; note for the plan, don't build now. + +## 12. Pitfall scan — current code that breaks under multi-user + +1. **`config_status` reads `runs._actors`** (`app.py:174-181`) — + multi-user probe leak + zero-run breakage. → factory probe (§4). [P1] +2. **Shared `model` instance across runs** (`app.py:152`) — the verified + `_active_base` fallback mutation (`ollama.py:73-79`) flips every user + to localhost after one cloud 401. → per-run factory (§4). [P1] +3. **`app = create_app()` module global** (`app.py:359`) — serves fakes + + cwd storage in production. → `researchx.api.main` (§8). [P2/P4] +4. **CORS lacks `allow_credentials`** (`app.py:143-148`) — cookie auth + cannot work cross-origin in dev. → §1. [P1] +5. **`dev_bridge.py` no auth** — correct for E2E; must stay explicitly + `auth_enabled=False`, never the production entry. [verify P1] +6. **`RunRegistry` unbounded growth** — prune terminal runs. → §7. [P1] +7. **`RunState` no `owner_id`** — cross-user authorization impossible + today. → topic 5. [P1] +8. **Read endpoints require the registry** (`runs.has(run_id)` guards) + — every read dies on restart; a hosted redeploy 404s the whole + library. → disk read-models (D-115). [P1/P3] +9. **`demo.py`** constructs storage directly — mechanical owner fix. [P1] +10. **`test_local_storage.test_run_dir_layout`** asserts unpartitioned + path — changes with D-106 (intentional). [P1] +11. **Dead code** duplicate `return EventSourceResponse` (app.py:331) — + remove while touching the file. [P1] +12. **`exportReport` fabricates data** (`live-client.ts:350-389`) — + fixed by report.json/evidence.json. [P3] +13. **Global `cost_counters`** (`app.py:110`) — per-user (owner-keyed). [P1/P2] +14. **Duplicate run-id generators** (different alphabets) — consolidate. [P1] + +## Research Conclusions + +- **C3-01** Auth enforcement = router-level `Depends(require_user)` in + `create_app` (public router `/api/auth/*` + `/healthz`; protected the + rest), NOT middleware: OpenAPI-visible, dependency_overrides testable, + no BaseHTTPMiddleware/SSE interaction. +- **C3-02** Sessions: opaque `token_urlsafe(32)` cookies + (`researchx_session`, httpOnly, SameSite=Lax, Secure hosted-only), + server-side dict + atomic sessions.json, 14-day sliding expiry with + >60s persist throttle, token rotation at login, immediate revocation. +- **C3-03** argon2id via argon2-cffi `PasswordHasher()` defaults; hash, + verify, rehash-on-login all in `asyncio.to_thread`; uniform 401. +- **C3-04** CORS: `allow_credentials=True` with explicit localhost + origins, added so it wraps auth (last-added = outermost in starlette + 0.46.2); dev origins must share the hostname `localhost` (port is not + part of same-site computation). +- **C3-05** CSRF = SameSite=Lax + `X-ResearchX-Request` header on all + non-GET /api (checked in the auth dependency); EventSource GET exempt; + fetch `credentials:"include"` throughout. +- **C3-06** Key store: AES-256-GCM via cryptography AESGCM, KEK = + 32-byte RESEARCHX_KEK (b64/hex) from the env-file 'keys' scope, AAD = + user_id, `os.urandom(12)` nonce per write, `keys/.json` + (v1/entries format, atomic, 0600); masked-only API; honest disabled + status; InvalidTag surfaced. +- **C3-07** Per-run model factory: `resolve(user) → (adapter, tier)`, + chain user→server→keyless; OllamaModelAdapter gains optional `api_key` + constructor param; engine contracts unchanged; probe-per-tier falls + out of the existing flow; GET /api/config = current user's tier + + fingerprint-cached probe. +- **C3-08** Partitioning = owner-scoped `LocalStorageAdapter(base_dir, + owner_id)` (protocol unchanged), `runs///`, owner_id + in RunState, registry keyed (owner_id, run_id), per-request adapters; + ~10 enumerated touchpoints incl. 8 test files + demo.py + dev_bridge. +- **C3-09** Data root = RESEARCHX_DATA_DIR (dev backend/var/researchx, + container /var/lib/researchx) via a new `paths.data_root()`; the + module-level create_app() cwd default + fake adapters are replaced by + a production `researchx.api.main` entry. +- **C3-10** report.json + evidence.json persisted at DONE in + _verify_phase; markers normalized `[p-N]`→`[^pN]`, ids `p-N`→`pN` at + the persistence boundary so the Reading Room parses with zero + frontend changes; report.md keeps `[p-N]` for the trust loop. +- **C3-11** Read-models read disk artifacts only; open questions = plan + sub-questions answered-iff DONE; startup sweep marks non-terminal + runs CANCELLED; rehydration out of scope. +- **C3-12** Registry pruning: drop terminal entries once no SSE + subscriber; events 404 after prune/restart documented; trace persists. +- **C3-13** Backend container: uv-based non-root Dockerfile (uid 10001, + chowned volume), `uvicorn --workers 1` mandatory, runtime-only secrets + via --env-file, plain docker-run pairs (no compose dependency), + researchx-api-prev rollback, health on both /healthz and /ask. +- **C3-14** HAProxy: path route /api + /healthz → backend (2.8 on the + LXC host, already TLS-fronting); timeout server/tunnel 3600s, NO + http-buffer-response, NO compression on the API backend; sse-starlette + 15s pings + X-Accel-Buffering keep streams alive; httpchk GET /healthz. +- **C3-15** Two build profiles: code default mock; hosted image built + with NEXT_PUBLIC_RESEARCHX_API_MODE=live + + NEXT_PUBLIC_API_BASE_URL="" (same-origin); deploy smoke asserts live + mode; EventSource withCredentials; delegation to MockApiClient ends in + P3 (standalone GeneratedApiClient). +- **C3-16** Auth harness: create_app(auth_enabled=True) default with a + parameter-only dev/test override resolving a synthetic dev user; + existing zero-key suites run with it off; dedicated auth suite on. +- **C3-17** New pyproject deps: argon2-cffi>=23.1,<26, cryptography>=43; + no python-multipart; config.json secrets scopes gain + "keys": ["RESEARCHX_KEK"]. +- **C3-18** ApiClient interface additions are PLANNED (G-002 lesson): + login/logout/getSession + admin hooks documented in PLAN before P1; + openapi.json snapshot regen is a P1 gate (G-005 discipline carries). +- **C3-19** Pitfalls P-1..P-14 (§12) each closed by the phase touching + its file: P-1..P-11, P-13, P-14 in P1; P-3 in P2/P4; P-12 in P3; + P-5 verified by the auth suite. +- **C3-20** All decisions D-101..D-115 confirmed workable against the + actual code; refinements folded into PLAN as constraints. \ No newline at end of file diff --git a/.ciagent/config.json b/.ciagent/config.json index cfb7297..13f4c20 100644 --- a/.ciagent/config.json +++ b/.ciagent/config.json @@ -51,6 +51,9 @@ ], "search": [ "TAVILY_API_KEY" + ], + "keys": [ + "RESEARCHX_KEK" ] } } -- 2.43.0 From dca86a74adb26f2bd2015fa6424ed0888f2fb0c9 Mon Sep 17 00:00:00 2001 From: coreci Date: Tue, 29 Sep 2026 16:55:40 +0000 Subject: [PATCH 4/5] =?UTF-8?q?docs(P00):=20create=20phase=20plans=20?= =?UTF-8?q?=E2=80=94=20v0.3=20hosted=20P1..P4=20+=20P5=20note?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ---ci--- project: researchx phase: 0 milestone: v0.3 status: plan requirements: covered: [] partial: [] message: >- PLAN.md: vertical-slice plans for P1 (auth+sessions+scoping, T-1.1.. T-1.12), P2 (per-user settings+model keys+factory, T-2.1..T-2.11), P3 (live surfaces, read-models, report.json/evidence.json, standalone live client, T-3.1..T-3.13), P4 (hosting, live default, smoke+exposure audit, T-4.1..T-4.9); P5 final-review note. MVP/UX sections written (User-Facing Surface, Happy Path, UX Acceptance Criteria). C3-18 ApiClient extension list documented exhaustively before P1 (G-002 lesson). Pitfall ledger P-1..P-14 assigned and closable by v0.3.4. Carrying gates: G-005 snapshot+drift per route-touching phase, G-007 owner-keyed counters, G-008 DDG fixtures, D-018/D-019 halt+deploy. ---/ci--- --- .ciagent/CHECKPOINT.json | 4 +- .ciagent/PLAN.md | 261 +++++++++++++++++++++++++++++++++++---- 2 files changed, 238 insertions(+), 27 deletions(-) diff --git a/.ciagent/CHECKPOINT.json b/.ciagent/CHECKPOINT.json index 54b148f..fa6525b 100644 --- a/.ciagent/CHECKPOINT.json +++ b/.ciagent/CHECKPOINT.json @@ -1,8 +1,8 @@ { "phase": 0, - "stage": "research", + "stage": "plan", "milestone": "v0.3", "phase_role": "pre_execution", "attempts": 0, - "updated_at": "2026-09-29T16:25:00Z" + "updated_at": "2026-09-29T16:45:00Z" } \ No newline at end of file diff --git a/.ciagent/PLAN.md b/.ciagent/PLAN.md index 059a03d..9191f29 100644 --- a/.ciagent/PLAN.md +++ b/.ciagent/PLAN.md @@ -1,44 +1,255 @@ -# PLAN — Milestone v0.2 Phase 5 (FINAL): Review + Audit + Ship +# PLAN — Milestone v0.3 (Hosted): Working Plan for Phases 1–4 --- -phase: 5 -milestone: v0.2 -status: executed +phase: 0 +milestone: v0.3 +status: planned must_haves: - - "Multi-persona full-milestone review: 4 P0 + 15 P1 + 15 P2 found; all 4 P0s fixed in-phase with attack-proofs (A1 tautological anchoring → real writer-quote extraction; C2 gate denominator → cited-URL scoped; C1 zombie runs → shape-validated plans + catch-all FAILED persist; S1 fetch SSRF → per-hop re-guarding); 6 cheap P1s fixed (approve-with-plan, FAILED status, length caps, probe wiring, README env-var + adapter honesty)" - - "Audit: PASS-WITH-NOTES — reconstruction PASS (all REQ-01..34 covered across v0.2.0–v0.2.4), secrets clean (zero key values in history), branch hygiene exact; critical notes fixed in-phase (REQUIREMENTS traceability un-staled, P5 PLAN written, ROADMAP P5 status, security-engineer persona deactivated per its removal note)" - - "Milestone ship: v0.2.5 tagged as the MVP release candidate — the pipeline then HALTS at the merge_to_main owner approval gate (D-018)" - - "G-009 escalation presented at the gate: the owner ratifies that live mode covers the core research loop only (G-002); remaining prototype surfaces stay mock-backed to v0.3+" + - "Engine contracts unchanged: all v0.3 work lands outside the engine — per-run adapter injection, owner-scoped storage constructors, owner_id stamping via make_run_config; no methodology/engine protocol rewrites" + - "No secret values in git, logs, traces, or API responses: model keys masked (last 4 chars) at the API boundary and encrypted at rest (AES-256-GCM, KEK from the env-file 'keys' scope), passwords argon2id-hashed and never emitted" + - "User scoping provable end to end: runs/// partitioning, owner-keyed RunRegistry, disk read-models — cross-user access 404s, pinned by test in every phase that touches the data flow" + - "Every phase ships a patch (v0.3.1..v0.3.4; P5 tags v0.3.5 = the milestone release); gates green before every ship; pipeline halts at the merge_to_main owner approval gate (D-018); no deploys without owner directive (D-019)" + - "openapi.json snapshot committed and the generated frontend client regenerated from the snapshot (never a live server) in every phase that touches routes; schema-drift check wired into the gates (G-005)" + - "Zero-key test discipline: every CI suite runs on REQ-31 fakes with auth via create_app(auth_enabled=False) or dependency_overrides; keyed/live tiers are opt-in, skipped by default" + - "Single uvicorn worker in every serving mode — the in-memory RunRegistry and session store are single-process by design" + - "Mock demoted to dev/test fixture: the code default of RESEARCHX_API_MODE stays mock; after P3 no live-mode surface delegates to MockApiClient; hosted builds inline NEXT_PUBLIC_RESEARCHX_API_MODE=live (REQ-75)" personas: + - backend-engineer + - data-engineer + - frontend-engineer + - "security-engineer (phase-specific P1/P2/P4)" - ci-lead-developer -requirements: [REQ-01, REQ-02, REQ-03, REQ-04, REQ-05, REQ-06, REQ-07, REQ-08, REQ-09, REQ-10, REQ-11, REQ-12, REQ-13, REQ-14, REQ-15, REQ-16, REQ-17, REQ-18, REQ-19, REQ-20, REQ-21, REQ-22, REQ-23, REQ-24, REQ-25, REQ-26, REQ-31, REQ-32, REQ-33, REQ-34] +requirements: [REQ-69, REQ-70, REQ-71, REQ-72, REQ-73, REQ-74, REQ-75, REQ-76, REQ-77, REQ-78, REQ-79, REQ-80, REQ-81, REQ-82, REQ-83, REQ-84, REQ-85] --- ## User-Facing Surface -The MVP release candidate: the approved v0.1 prototype frontend (Ask → Choose → Plan & Run → Reading Room, all 8 cards, shell, library, settings) now backed by a real FastAPI research engine in live mode — methodology registry (Rapid Answer + Deep Report), 12-dial framework, asyncio state machine with durable pause/cancel/resume, visited-only evidence store, attack-proven trust loop (quote anchoring, SSRF-guarded liveness, citation ledger, ≥99% export gate with zero-unanchored requirement), SSE live trace, adversarial suite (14 attacks), E2E happy path on the real stack. Releases v0.2.0–v0.2.5. +The hosted multi-user product: the full approved frontend — a login screen; Ask → Choose → Plan & Run → Reading Room; the app shell with research library, project graph, task inbox, follow-up suggestions, refine dialogue, settings with masked model-key entry and resolved credential tier, avatar user menu with real profile — now live-bound end to end against the FastAPI backend with **zero live-mode delegation to the mock** (D-112), user-scoped (`runs///`), with per-user encrypted model keys resolving user → server → keyless per run, and deployed as a two-service stack (uvicorn single worker + Next.js live build) behind an SSE-safe HAProxy with TLS — real inference as the product default. Releases v0.3.0–v0.3.5. ## Happy Path (owner review script) -1. `cd backend && uv sync && uv run pytest` — 129 tests green, zero API keys (live tier: `uv run pytest -m live`, skips cleanly) -2. `cd frontend && corepack pnpm install && corepack pnpm test` — 97 tests green (90 component + 5 live-binding + 2 real-stack E2E) -3. `uv run uvicorn researchx.api.app:app --port 8000` + `NEXT_PUBLIC_RESEARCHX_API_MODE=live corepack pnpm dev` → http://localhost:3000 → Ask a question → Research this → pick Deep Report → Start research → approve the plan (3 clicks) → live SSE trace → report with anchored citations → ledger → export Markdown (citation links) / JSON (backend-schema round-trip) -4. `uv run python -m researchx.engine.demo` — full pipeline on fakes, artifacts in a temp runs dir -5. Owner decision: approve → milestone merge to main + v0.2.5 published as the MVP release; or direct changes → binding fix cycle +1. `cd backend && uv sync && uv run pytest` — all suites green with zero API keys: auth suite (`test_auth.py`), key-store tamper tests, read-model tests, restart-sweep test, engine/trust/adversarial suites under the `auth_enabled=False` harness; `uv run pytest -m live` skips cleanly. +2. `cd frontend && corepack pnpm install && corepack pnpm test` — component + live-binding suites green; typecheck + lint green. +3. Local live-mode full loop: `uv run uvicorn researchx.api.main:app --port 8000` (env-file with `RESEARCHX_KEK` + server model key) + `NEXT_PUBLIC_RESEARCHX_API_MODE=live NEXT_PUBLIC_API_BASE_URL=http://localhost:8000 corepack pnpm dev` → http://localhost:3000 → register (first user = admin) → login → Settings: enter model key (shows masked status only), config shows resolved tier → Ask a question → Research this → pick Deep Report → Start research → approve the plan (≤3 clicks on defaults) → live SSE trace → Reading Room: hover a citation shows the stored quoted passage; ledger panel; export Markdown/JSON with real fields → Library shows the run in its auto-categorized folder; graph/tasks/suggestions surface the run's open questions; refine dialogue runs on the real planner. +4. Multi-user isolation: admin creates a second user; that user sees zero of the first user's runs, graph, tasks, or suggestions; the first user's key never affects the second user's resolved tier. +5. Hosted: `deploy.sh` → both services up + healthy; `curl https:///api/runs` → 401 unauthenticated; `curl https:///healthz` → 200; login through the public URL; SSE trace streams unbuffered through the proxy; per-user key honored end to end (public deployment per owner directive, D-019). +6. Owner decision at the P5 gate: approve → v0.3.5 = milestone release, merge to main; or direct changes → binding fix cycle. ## UX Acceptance Criteria -- ≤3 clicks from cold start to initiated research, validated on the real stack (REQ-17) -- Every trust claim attack-proven: hallucinated citations fail the export gate; fabricated quotes on real passage IDs flag unanchored; SSRF redirects die; pause/resume/cancel survive the loop -- Honest scope: live mode = core research loop (G-002); library/settings/suggestions/refine stay mock-backed until v0.3+ — ratified at this gate (G-009) -- No secret value in any artifact, trace, or log; zero-key test discipline throughout +- The ≤3-click research path (Ask → Choose → Run on defaults) is preserved for an authenticated user; login is a once-per-session step (14-day sliding) adding zero clicks to the path. +- Reading Room parity with the approved mock: page-anchored citations whose hover shows the exact stored quoted passage; a real ledger panel; exports with real entry data — no fabricated fields anywhere. +- Honesty in live mode: resolved credential tier visible (user|server|keyless), key-store-disabled stated honestly, no synthetic suggestions, honest empty states for fresh users. +- Suggestions click-fills-the-box (never auto-starts); refine is opt-in and model-driven. +- Keys masked everywhere: no secret value is ever rendered, logged, or exported. +- Cross-user isolation is total and invisible: no surface leaks another user's runs, questions, or key status. -## What this phase did +## Planned ApiClient Extensions (C3-18 — documented before P1; this list is exhaustive) -1. **Review** (multi-persona, probe-verified): 4 P0 + 15 P1 + 15 P2. The 4 P0s broke the milestone's flagship guarantees and were all fixed with attack-proofs: quote anchoring extracted the stored passage's own quote (always 1.0) instead of the writer's text; the export-gate denominator counted unprobed visited pages as dead; malformed planner JSON left zombie runs; fetch adapters followed redirects without re-guarding. Six cheap P1s also fixed. -2. **Audit**: PASS-WITH-NOTES — reconstruction, secrets sweep (clean), branch/tag hygiene exact; both critical notes (stale traceability rows, missing P5 plan) fixed in-phase. -3. **Ship (this commit sequence)**: PR #14 (phase/05 → milestone), squash-merge, tag v0.2.5 — the MVP release candidate, release with the full milestone summary. Per D-018 and the merge_to_main escalation hook, the milestone merge to main does NOT proceed autonomously — the pipeline halts for the recorded owner decision, with G-009 (live-mode scope) presented at the same gate. +- `login(email, password): Promise` — POST /api/auth/login (sets the session cookie) +- `logout(): Promise` — POST /api/auth/logout +- `getSession(): Promise` — GET /api/auth/me (401 → null) +- Admin hooks (API-first in v0.3; no dedicated admin screen): `listUsers()`, `createUser(input)`, `deactivateUser(id)` — GET /api/admin/users, POST /api/admin/users, POST /api/admin/users/{id}/deactivate +- MockApiClient gains matching no-op/local implementations (standalone mock preserved — REQ-31/REQ-54); live implements against the routes. The v0.2 pause/resume extensions (G-003) carry unchanged. No other interface changes are planned for v0.3. -## Scope Guard +## Phase 1 — Auth + Sessions + User Scoping (ships v0.3.1) -Milestone merge to main, v0.2.5 publication, and any deploy wait on the owner approval gate (D-018/G-009). The pipeline does not proceed autonomously past this commit. \ No newline at end of file +**Personas** (lead-developer decomposition by territory): backend-engineer (lead — auth package, storage partitioning, engine/app wiring, sweep/pruning), data-engineer (auth + state pydantic contracts), security-engineer (phase-specific — auth adversarial suite + STRIDE), frontend-engineer (session surface, snapshot gate), ci-lead-developer (ship). + +**Requirements covered**: REQ-69, REQ-70, REQ-71 + +**Task waves**: + +*Wave 1 — contracts + stores (parallel)* +- **T-1.1 (backend-engineer)**: Data root + user store — `researchx.config.paths.data_root()` (RESEARCHX_DATA_DIR → `backend/var/researchx` dev; D-107); `auth/users.py`: User model (user_id = 8-char slug, validation `^[a-z0-9]{8}$`, D-105; email; argon2id via argon2-cffi `PasswordHasher()` defaults, hash/verify/rehash all in `asyncio.to_thread`, D-102; display_name; role; active; created_at), `users.json` at the data root, atomic rewrite (D-017 idiom). Pyproject gains `argon2-cffi>=23.1,<26` (no python-multipart). +- **T-1.2 (backend-engineer)**: Session store — `auth/sessions.py`: opaque `token_urlsafe(32)` tokens, in-process dict + durable `sessions.json` (atomic), `Session {token_hash, user_id, created_at, last_seen, expires_at}`, 14-day sliding expiry via `touch()` with >60s persist throttle, token rotation at login, immediate revocation (logout deletes the presented token; deactivation deletes all the user's tokens) (D-101). +- **T-1.3 (data-engineer)**: Auth + state contracts — pydantic User/Session schemas, auth route request/response models, `RunState.owner_id: str | None = None` (serialization-compatible, no migration per D-106); old state files must deserialize. + +*Wave 2 — enforcement (depends W1)* +- **T-1.4 (backend-engineer)**: Guard + auth routes + CORS — `auth/guard.py` `require_user` dependency (session resolve → 401; `X-ResearchX-Request` header required on non-GET → 403; inactive user → 401; sliding touch); `auth/routes.py` register/login/logout/me with httpOnly `researchx_session` cookie (SameSite=Lax, path=/, Secure hosted-only flag; no Domain attribute); uniform 401 "invalid email or password"; register open only while the user store is empty — first user `role="admin"`, then 403 (no await between check and insert except the argon2 hash; single worker = race-free, D-104); public router (`/api/auth/*`, `/healthz` outside /api per D-103) + protected router (everything else) via `APIRouter(dependencies=[Depends(require_user)])` — **router dependencies, not middleware** (C3-01; no BaseHTTPMiddleware near SSE); CORS: `allow_credentials=True` + explicit localhost origins, ordered so preflight OPTIONS answers without 401 (C3-04); login/register POSTs also require the CSRF header (C3-05). +- **T-1.5 (backend-engineer)**: Admin routes — POST /api/admin/users (admin-guarded create with admin-set initial password), GET /api/admin/users (list), deactivate (kills all sessions; data retained) — REQ-70. +- **T-1.6 (backend-engineer)**: Storage partitioning — `LocalStorageAdapter(base_dir, owner_id)` with **required** owner_id (validated), `run_dir → base_dir/runs///`, + `list_run_ids()` (read-models scan later); `StorageAdapter` protocol gains `append_run_artifact` (the actor already calls it); `read_run_artifact` error message embeds the owner path; consolidate the duplicate run-id generators (P-14). Protocol otherwise byte-for-byte unchanged (D-106). + +*Wave 3 — engine/app wiring (depends W2)* +- **T-1.7 (backend-engineer)**: Owner-scoped app wiring — `make_run_config(..., owner_id)` stamps `state.owner_id`; RunActor untouched (engine contracts unchanged); `RunRegistry` keyed `(owner_id, run_id)` — cross-owner access 404; routes construct the owner-scoped storage per request from the auth dependency; `create_app(*, auth_enabled=True)` default with **parameter-only** dev override resolving a synthetic dev user (`user_id="dev00000"`, role admin — C3-16, never an env var readable in production); `dev_bridge.py` stays explicitly `auth_enabled=False` (P-5); `engine/demo.py` owner arg (P-9); remove the dead duplicate `return EventSourceResponse` (app.py:331, P-11); mechanical `owner_id` constructor args across the 8 test files (RESEARCH §5 touchpoint list). +- **T-1.8 (backend-engineer)**: Startup sweep + registry pruning — on `create_app` startup, rewrite every non-terminal `state.json` under the data root as CANCELLED (D-115; legal from every non-terminal state; in-flight actors die with the process); registry drops terminal `(actor, bus)` entries once no live SSE subscriber (refcount in the events route — P-6); events-404-after-prune/restart documented (trace persists on disk). + +*Wave 4 — adversarial + frontend + snapshot (depends W3)* +- **T-1.9 (security-engineer + backend-engineer)**: Auth suite — `backend/tests/test_auth.py` with `auth_enabled=True` (RESEARCH §11): first-user-admin (+403 register after), wrong-password 401 uniform, logout revocation, 401 on all protected routes, CSRF header rejection, cross-user 404 (all run endpoints), admin create/deactivate (deactivation kills sessions), session expiry, cookie flags, no password-hash material in any response; pins P-5 and the authorization half of P-8. +- **T-1.10 (frontend-engineer)**: Session surface — ApiClient extensions per C3-18 (mock no-ops; live against routes); login screen (email + password, error states); live-mode route guard redirecting unauthenticated users to /login; user menu shows the real profile + logout; fetch discipline: `credentials:"include"` + `X-ResearchX-Request` header on every mutation (C3-05); dev docs pin same-hostname origins (`localhost` both sides — port is not part of same-site computation). +- **T-1.11 (frontend-engineer)**: Snapshot regen gate — export + commit `backend/openapi.json` (auth/admin routes included), regenerate `frontend/src/lib/api/generated/` from the snapshot only, wire the schema-drift check into the gates, operation-ids stable (G-005). + +*Wave 5 — ship* +- **T-1.12 (ci-lead-developer)**: Phase ship — all gates green (backend ruff/mypy/pytest incl. `test_auth.py`; frontend lint/typecheck/vitest; drift gate), tag **v0.3.1** + release; REQUIREMENTS/ROADMAP traceability rows updated. + +**Must-haves (verification-oriented)**: +- Unauthenticated access to any `/api` route except register/login → 401; register 403s once the user store is non-empty — pinned by `test_auth.py`. +- First registered user is admin; all subsequent accounts admin-created; deactivation kills sessions and retains data. +- No user can read or act on another user's data: owner-keyed registry → cross-owner 404 on run GET/lifecycle/report/ledger/events — pinned by test. +- Storage layout is `runs///`; `owner_id` recorded in every persisted run state. +- Sessions are server-side with httpOnly SameSite=Lax cookies, immediate revocation, sliding expiry; CSRF header enforced on non-GET (EventSource GET exempt). +- Engine contracts unchanged; existing zero-key suites green with only mechanical `owner_id` constructor changes. +- `openapi.json` snapshot committed; generated client regenerated from it; drift gate fails on intentional drift (proof). +- No run survives a restart in a non-terminal state (sweep test); terminal registry entries prune (leak test). +- No password hash or key material in any API response or log. + +**Pitfall closures (RESEARCH §12)**: **P-4** (CORS `allow_credentials` + ordering), **P-5** (dev_bridge explicitly auth-off, pinned by the suite), **P-6** (registry pruning), **P-7** (owner_id + partitioning), **P-9** (demo.py owner arg), **P-10** (layout test re-pinned to the partitioned path), **P-11** (dead code removed), **P-14** (run-id generators consolidated). P-8's authorization half (owner-keyed 404s) closes here; its registry-independence half closes in P3. + +**Success criteria (testable)**: +- `test_auth.py` green covering the full §11 list; two-user isolation test proves zero visibility across every run endpoint. +- Register → admin; second register → 403; admin creates + deactivates (sessions die) — all route-tested. +- Non-GET without the header → 403; with header + session → 2xx. +- Disk layout test proves `runs///`; restart-sweep test proves non-terminal → CANCELLED across an app restart; pruning test proves terminal entries drop with zero SSE subscribers. +- Drift gate: intentional schema change fails the gate; regen green. +- Gates green; patch v0.3.1 shipped. + +## Phase 2 — Per-User Settings + Model Keys (ships v0.3.2) + +**Personas**: backend-engineer (lead — key store, settings store, model factory, config rework, production entrypoint), data-engineer (settings + key-envelope schemas), security-engineer (phase-specific — key-store adversarial suite), frontend-engineer (settings live-binding, snapshot), ci-lead-developer (ship). + +**Requirements covered**: REQ-72, REQ-73, REQ-74 + +**Task waves**: + +*Wave 1 — contracts* +- **T-2.1 (data-engineer)**: Settings + key-envelope schemas — per-user Settings payload (model names, methodology defaults + resolved dials, appearance slot, masked key status, storage path display-only — D-111); key-store envelope `{v, alg, updated_at, entries: {OLLAMA_API_KEY: {nonce, ct, updated_at}}}` (D-108/D-109); KEK decode rules (urlsafe-b64 or hex, exactly 32 bytes; decode/length errors are typed AdapterErrors). + +*Wave 2 — stores (depends T-2.1)* +- **T-2.2 (backend-engineer)**: Key store — `auth/key_store.py`: AES-256-GCM via `cryptography` AESGCM (pyproject gains `cryptography>=43`); KEK = `RESEARCHX_KEK` from the env-file **'keys'** scope (`config.json` secrets.scopes gained `"keys": ["RESEARCHX_KEK"]` in P0; provisioned into `.ciagent/.env.secrets` only — never git); AAD = `user_id` (a copied file under another user fails `InvalidTag`); fresh `os.urandom(12)` nonce per write (never reused under the KEK); `keys/.json`, atomic, 0600; crypto in `asyncio.to_thread`; honest degradation: no KEK → disabled (PUT rejected with `{"key_store": "disabled"}`), corrupt entries surface as errors, never a silent tier drop (C3-06). KEK change requires process restart (documented). +- **T-2.3 (backend-engineer)**: Settings persistence — `settings/.json` store, atomic rewrite, defaults for fresh users, GET/PUT payload shapes (D-111). + +*Wave 3 — routes + factory (depends W2)* +- **T-2.4 (backend-engineer)**: Settings + key routes — GET/PUT `/api/config/settings`; model-key PUT accepts a raw key (scope = OLLAMA_API_KEY only, D-109) → encrypted at rest; every response returns configured + masked (last 4) only — the raw key never appears in any response, log, or trace (REQ-73). +- **T-2.5 (backend-engineer)**: Per-run model factory — `adapters/model_factory.py`: `ModelAdapterFactory(secrets, key_store, defaults).resolve(user, settings) → (adapter, tier)` resolving user → server → keyless (REQ-74); `OllamaModelAdapter` constructor gains optional `api_key` (precedence over secrets resolve; secrets path back-compat); **one adapter instance per run** — the `_active_base` fallback mutation is per-instance, never shared (P-2 closure); `create_run` resolves + injects (engine contracts unchanged); state records `owner_id` + `credential_tier`. +- **T-2.6 (backend-engineer)**: Config status rework — GET `/api/config` resolves the **current user's** tier via the factory without creating a run (kills the `runs._actors` private read — P-1 closure; no cross-user probe leak, no zero-run breakage); probe cached in-process keyed `(user_id, tier, key_fingerprint)` with short TTL, running against the resolved tier; per-user cost counters owner-keyed (P-13 closure; G-007 carries); `resolved_tier` + `key_store: enabled|disabled` surfaced. + +*Wave 4 — production + adversarial + frontend (depends W3)* +- **T-2.7 (backend-engineer)**: Production entrypoint — new `researchx.api.main` wiring `create_app(data_root=RESEARCHX_DATA_DIR, factory=, auth_enabled=True)` (P-3 backend closure — the `app.py:359` module global serving fakes + cwd storage leaves production); `app.py` stays the test factory; `main.py` never passes `auth_enabled=False`. +- **T-2.8 (security-engineer + backend-engineer)**: Key-store adversarial suite — tamper tests: AAD binding (key file copied under another user_id → `InvalidTag` surfaced), nonce uniqueness across writes, masked-only responses (no raw key material in response/log/trace), disabled-honesty without KEK, corrupt-entry error surfacing; opt-in keyed tier (skipped by default; zero-key CI unchanged) proving a keyed user gets real inference end to end through the REQ-74 chain. +- **T-2.9 (frontend-engineer)**: Settings live-binding — `GeneratedApiClient` getSettings/saveSettings → `/api/config/settings`; masked key entry (password-type input → PUT; masked status display; never echoed); getConnectionStatus maps GET `/api/config` (resolved tier user|server|keyless, key_store status, probe, per-user counters); storage directory display-only in live/hosted mode (D-111 — the free-path picker is the local single-user profile only); mock parity preserved standalone. +- **T-2.10 (frontend-engineer)**: Snapshot regen — settings/key routes change `openapi.json` → snapshot committed, client regenerated, drift gate green (G-005). + +*Wave 5 — ship* +- **T-2.11 (ci-lead-developer)**: Phase ship — gates green; tag **v0.3.2** + release; traceability updated. + +**Must-haves**: +- Keys never appear in full in any API response, log, or trace — masked only, pinned by test. +- Key store: AES-256-GCM, AAD=user_id, fresh nonce per write, 0600, atomic — tamper-proof pinned by test. +- Without a KEK: key store disabled honestly; resolution proceeds server/keyless; never a silent tier drop. +- Per-run credential chain user → server → keyless; probe runs against the resolved tier; config status surfaces the current user's tier without creating a run and without reading private registry state. +- Shared-adapter fallback bug dead: one adapter per run; `_active_base` mutation can never cross users. +- Cost counters owner-keyed. +- Engine contracts unchanged; zero-key suites green (fakes, `auth_enabled=False`). +- Production entrypoint live-wired; the module-global fake app serves nothing. + +**Pitfall closures**: **P-1** (config_status → factory-resolved probe), **P-2** (per-run factory kills the shared `_active_base` mutation), **P-13** (owner-keyed cost counters), **P-3 backend half** (`researchx.api.main`; the hosting half closes in P4). + +**Success criteria**: +- Opt-in keyed tier: a keyed user gets fully real inference end to end; `resolved_tier="user"` in config status. +- Zero-key CI: full suite green with no KEK configured — key store reports disabled, tier falls to server/keyless honestly. +- Tamper suite green: AAD tamper → InvalidTag surfaced; nonces unique; masked-only responses. +- Config status correct with zero runs and per-user isolated (no probe leak). +- Drift gate green; gates green; patch v0.3.2 shipped. + +## Phase 3 — Live Surfaces, user-scoped (ships v0.3.3) + +**Personas**: backend-engineer (lead — artifacts, read-models, all new routes), data-engineer (artifact + read-model payload schemas), frontend-engineer (standalone client, all surface swaps, Reading Room fidelity), ci-lead-developer (ship). Security-engineer inactive this phase (no new secret handling — roster reason). + +**Requirements covered**: REQ-76, REQ-77, REQ-78, REQ-79, REQ-80, REQ-81, REQ-82 (+ REQ-75 delegation-end half: after this phase no live-mode surface delegates to the mock) + +**Task waves**: + +*Wave 1 — artifacts + read-models (parallel)* +- **T-3.1 (backend-engineer + data-engineer)**: report.json + evidence.json at DONE — persisted inside `_verify_phase` right after ledger/checklist (EvidenceStore is GC'd with the actor — this is the only write point; D-113): `report.json {v, run_id, owner_id, title, question, methodology_slug, generated_at, sections:[{heading, body}], citations:[{passage_id, url, title, quote, health, authority_tier}]}`, `evidence.json {v, run_id, passages:[{passage_id, url, title, quote, text}]}`; **marker normalization at the persistence boundary only**: section bodies `[p-N]` → `[^pN]`, ids `p-N` → `pN` (the Reading Room parses with zero frontend changes); `report.md` keeps raw `[p-N]` — trust-loop regexes untouched (C3-10). +- **T-3.2 (backend-engineer + data-engineer)**: Read-models — `api/read_models.py` scans `runs//` loading `state.json` (+ report.json/ledger.json where needed); skips dotfiles/tmp defensively (D-115); payload schemas shaped to the existing frontend types: library folders (auto-categorization by methodology + deterministic topic tokens from plan sub-questions) + ReportSummary, ReportConnection, ProjectGraph nodes/edges, InboxItem `{id: "#", question, sourceRunId, status}`, SuggestedQuery. +- **T-3.3 (backend-engineer)**: Enriched report endpoint — GET `/api/runs/{id}/report` → `{run_id, markdown, report, evidence}` (markdown retained, back-compat); **disk-read, registry-independent** (P-8 full closure); graceful markdown-only fallback for pre-artifact DONE runs (old artifacts disposable per D-106, but never a 500); ledger endpoint likewise disk-read. + +*Wave 2 — routes (depends W1)* +- **T-3.4 (backend-engineer)**: Library + connections + graph — GET `/api/library?sort=recent|title` (folders + ReportSummary[] resolving to real runs); GET `/api/library/connections` (shared-source edges ≥1 ledger-URL overlap + topical edges); GET `/api/graph` (same edges in ProjectGraph shape) — REQ-76/77, D-114. +- **T-3.5 (backend-engineer)**: Tasks + suggestions — GET `/api/tasks` (open questions = plan sub-questions, answered-iff run reached DONE); GET `/api/suggestions` (open items joined with source run titles; no synthetic suggestions; empty state for fresh users) — REQ-78/79, D-114. +- **T-3.6 (backend-engineer)**: Refine + profile — POST `/api/refine/questions {draft}` → 2–3 planner clarifying questions; POST `/api/refine/compose {draft, answers}` → composed refined question + suggested context; planner calls ride the **same per-user credential chain** (P2 factory), honest 503 for a keyless user with no local endpoint — never a hang; GET/PUT `/api/profile` (display_name, email; no deletion) — REQ-80/81. All owner-scoped via the auth dependency; non-GET ⇒ CSRF header applies. + +*Wave 3 — frontend standalone (depends routes + snapshot)* +- **T-3.7 (frontend-engineer)**: Snapshot regen — all new routes into the committed `openapi.json` snapshot; client regenerated; drift gate green (G-005). +- **T-3.8 (frontend-engineer)**: GeneratedApiClient standalone — every ApiClient method overridden against real endpoints; `extends MockApiClient` **dropped** (a standalone class implementing ApiClient — D-112); request wrapper: `credentials:"include"` + `X-ResearchX-Request` on every mutation; `EventSource` constructed with `{withCredentials: true}`; mock remains a standalone dev/test fixture. +- **T-3.9 (frontend-engineer)**: Surface swaps — getReportLibrary, getReportConnections, getProjectGraph, getTaskInbox, getSuggestedQueries, getRefinementQuestions, composeRefinedQuestion, getUserProfile, plus the P2 settings/config methods — all live, zero component rewrites (REQ-54); suggestions click-fills never auto-starts; the refine drawer drives the real exchange. +- **T-3.10 (frontend-engineer)**: Reading Room fidelity + honest exports — getReport maps report.json → Report (sections + citations with **real quotes**; tier map official→1, reference→2, else 3); getLedger real entries; exportReport built from real entry data (kills the fabricated `claims_supported: 1` and empty quotes — P-12 closure, REQ-82); hover shows the stored quoted passage — parity with the mock Reading Room v0 (REQ-82); exports round-trip through the backend-schema gate (REQ-34 carries) with citation links intact. + +*Wave 4 — tests + ship (depends W3)* +- **T-3.11 (backend-engineer + data-engineer)**: Read-model test suite — `backend/tests/test_read_models.py`: a fixture disk tree (two users; DONE, CANCELLED, in-flight, FAILED runs) asserting library folder categorization + recency/title sorting; shared-source edges (≥1 ledger URL overlap) + topical edges; graph node/edge shape; tasks answered-iff DONE; suggestions = open items joined with run titles, empty for a fresh user, never synthetic; dotfile/tmp skip; **registry-independence test**: all read endpoints correct against a fresh `create_app` over an existing data root with zero in-memory actors (P-8 full-closure proof); artifact tests: report.json/evidence.json written at DONE with normalized markers (`[^pN]` in report.json, raw `[p-N]` in report.md), markdown-only fallback for pre-artifact DONE runs (never a 500). +- **T-3.12 (frontend-engineer)**: Live-surface regression suite — component tests run unchanged against the live binding (zero component rewrites proven, REQ-54); ≤3-click path pinned green in live mode; suggestions click-fills-the-box never auto-starts; refine exchange drives the real endpoint on fakes; drift gate green. +- **T-3.13 (ci-lead-developer)**: Phase ship — all gates green; tag **v0.3.3** + release; traceability rows updated. + +**Must-haves**: +- Every prototype surface (library, connections, graph, tasks, suggestions, refine, profile, settings, Reading Room) works against real user data in live mode — `GeneratedApiClient` is standalone, `extends MockApiClient` is gone (D-112), and no live-mode method delegates to the mock. +- Read-models read disk artifacts only, never RunRegistry: every read endpoint is correct after a restart with zero in-memory state; non-terminal runs post-sweep appear as CANCELLED, terminal runs stay readable. +- Reading Room parity with the mock: structured sections, page-anchored citations whose hover shows the stored quoted passage, ledger panel from real ledger.json; exports carry real quotes/claims/health/tier — no fabricated fields. +- Engine persists report.json + evidence.json at DONE inside `_verify_phase`; `report.md` marker syntax untouched (trust-loop regexes unchanged). +- All new routes owner-scoped via the auth dependency — cross-user 404s pinned; non-GET routes CSRF-header-guarded (refine POSTs included). +- ≤3-click path green; suggestions never auto-start; honest empty states; no synthetic suggestions. +- Snapshot committed, client regenerated from the snapshot only, drift gate green (G-005). +- Zero-key discipline intact: refine tests on REQ-31 fakes; keyless refine surfaces an honest 503, never a hang. + +**Pitfall closures**: **P-8 full closure** (registry-independent disk reads — auth half closed in P1, read half here, pinned by the restart test), **P-12** (export fabrication killed — real quotes/claims from report.json/evidence.json). Marker-normalization hazard (§6) closed by construction: normalization at the persistence boundary only, `report.md` raw form pinned by test. + +**Success criteria**: +- Read-model suite green over the multi-user fixture tree; fresh-app-over-existing-data-root test proves every read endpoint without the registry. +- Live mode: all surfaces live-bound, component tests unchanged, 3-click green, no delegation to mock anywhere in the live client. +- Export JSON payloads contain real quotes, real claims_supported, real health/tier — the fabricated `claims_supported: 1` / empty quotes are gone and a test pins their absence. +- Fresh user: empty library, empty tasks, empty suggestions — honest, no synthetic content. +- Drift gate green; gates green; patch v0.3.3 shipped. + +## Phase 4 — Hosting + Real-Inference Default (ships v0.3.4) + +**Personas**: ci-lead-developer (lead — deploy.sh, HAProxy config, hosting per its repo-root territory), backend-engineer (Dockerfile, production wiring completion, health endpoint), frontend-engineer (live build profile, build args), security-engineer (phase-specific — hosted exposure audit through the public URL). + +**Requirements covered**: REQ-75, REQ-83, REQ-84, REQ-85 + +**Task waves**: + +*Wave 1 — container + production wiring* +- **T-4.1 (backend-engineer)**: Backend Dockerfile — `python:3.12-slim` + uv from the ghcr image; `uv sync --frozen --no-dev`; `useradd --system --uid 10001 researchx`; `mkdir -p /var/lib/researchx` + chown; `USER researchx`; `CMD ["uvicorn", "researchx.api.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1", "--proxy-headers"]` (REQ-83, C3-13). Secrets are runtime-only via `--env-file` (`RESEARCHX_KEK`, `OLLAMA_API_KEY`, `OLLAMA_BASE_URL`, `TAVILY_API_KEY`) — never baked into the image, never in git. +- **T-4.2 (backend-engineer)**: Production wiring completion — `researchx.api.main` verified end to end: `data_root=RESEARCHX_DATA_DIR` (container default `/var/lib/researchx`), live model factory from env-file secrets, `auth_enabled=True` hardwired — **P-3 full closure**: no production path can serve FakeModelAdapter or cwd-based storage; parameter-only auth override confirmed absent from the image (no `RESEARCHX_AUTH`-style env readable anywhere); uvicorn graceful-shutdown behavior documented (in-flight runs exercise `_persist_cancelled` on SIGTERM; `docker rm -f` SIGKILLs — the P1 startup sweep is the safety net). + +*Wave 2 — frontend live build profile* +- **T-4.3 (frontend-engineer)**: Live build profile — frontend Dockerfile build stage gains `ARG`/`ENV` before `pnpm build`: `NEXT_PUBLIC_RESEARCHX_API_MODE=live` + `NEXT_PUBLIC_API_BASE_URL=""` (empty = same-origin through HAProxy's `/api` path route); `baseUrl()` treats `""` as same-origin (openapi-fetch relative base, EventSource relative URL); build asserts the empty base (a non-empty base would break same-origin cookies); dev live-mode docs pin `http://localhost:8000` on both sides (hostname rule, C3-15). + +*Wave 3 — deploy + proxy* +- **T-4.4 (ci-lead-developer)**: deploy.sh two-service extension — rsync now includes `backend/`; both images built on the host; plain docker-run pairs on the existing LXC idiom (no compose dependency): `researchx-net` bridge network; `researchx-api` backend (env-file, data volume bind, `-p 127.0.0.1:8000:8000` with host-side HAProxy, `--restart` policy); `researchx` frontend (`-p 3000:3000`, live build args); paired rollback containers `researchx-api-prev` + `researchx-prev` both retained; data volume pre-created + `chown 10001` host-side (belt-and-braces against EACCES); health check on BOTH `/healthz` (backend) and `/ask` (frontend) before the swap; idempotent re-run. +- **T-4.5 (ci-lead-developer, security-engineer review)**: HAProxy front config — documented + idempotent update to the host's existing HAProxy 2.8 TLS frontend: `acl is_api path_beg /api`, `acl is_health path /healthz` → `backend_api`; `default_backend frontend_next`; `backend_api`: `option httpchk GET /healthz`, `default-server inter 5s fall 3 rise 2`, `server api1 127.0.0.1:8000 check`, `timeout server 3600s`, `timeout tunnel 3600s`; **NO `option http-buffer-response`, NO compression on the API backend** (gzip buffers text/event-stream); `backend frontend_next`: `timeout server 300s`, `server next1 127.0.0.1:3000 check` (REQ-84, C3-14); `timeout client` exceeds the sse-starlette 15s ping interval; the config file is committed as documentation; application to the live host only on owner deploy directive (D-019). +- **T-4.6 (backend-engineer)**: `/healthz` verification — unauthenticated `GET /healthz → 200 {"status":"ok"}` (public router from P1, outside `/api` per D-103) proven through the proxy path route. + +*Wave 4 — smoke + exposure audit + ship* +- **T-4.7 (ci-lead-developer)**: Hosted smoke assertions — scripted against a staged instance: (1) unauthenticated `curl` to public `/api/runs` → **401**; (2) `/healthz` → **200**; (3) **live-mode marker**: authenticated `/api/config` resolves a real tier and the smoke asserts the build inlined live mode (empty base + live mode, not mock); (4) **SSE-through-proxy smoke**: an authenticated EventSource against the public origin streams a full run's events unbuffered end to end — no buffering, no mid-stream cut, 15s pings hold the connection through `timeout server 3600s` (REQ-84/85). +- **T-4.8 (security-engineer)**: Hosted exposure audit — through the public URL: every `/api` route 401s unauthenticated (401-on-public-/api pinned); `/healthz` is the ONLY unauthenticated route; no `auth_enabled` backdoor in the production image (parameter-only, `main.py` never passes False — verified); cookie `Secure` flag set on the TLS-fronted host; secrets sweep over the built image layers and deploy artifacts (zero key values anywhere); STRIDE pass on the proxy surface. +- **T-4.9 (ci-lead-developer)**: Phase ship — all gates green; tag **v0.3.4** + release; traceability updated; deployment to the live host staged only on owner directive (D-019) — the smoke suite is written to run against a staged deploy. + +**Must-haves**: +- One-command deploy brings up both services; paired rollback containers exist for BOTH services (REQ-83). +- Backend container: non-root uid 10001, single uvicorn worker (`--workers 1` mandatory — in-memory registry + session store), secrets runtime-only via `--env-file`, data volume writable by uid 10001. +- The hosted instance runs real inference in live mode by default (build-time inlined; REQ-75) with per-user keys honored through the full chain — while the code default stays mock (zero-key dev/CI preserved, REQ-31/D-112). +- SSE streams unbuffered through the proxy: no http-buffer-response, no API compression, timeout server/tunnel 3600s, pings unbroken — verified end to end. +- Unauthenticated public `/api` → 401 on every route; `/healthz` 200 the only unauthenticated route (REQ-85). +- No secret values baked into images, committed, or logged; volume chown 10001. +- HAProxy changes documented + idempotent; applied only on owner directive (D-019). + +**Pitfall closures**: **P-3 full closure** (hosting half — the container serves `researchx.api.main` with the live factory; nothing can host fakes). RESEARCH §8/§9 hazards encoded and tested: volume-ownership EACCES (chown 10001 + smoke write), SSE-compression hazard (explicitly absent from the API backend config, asserted in review), ping keepalive (SSE smoke holds ≥ several ping intervals), SIGKILL-on-redeploy (startup sweep as the net, covered by the P1 sweep test). + +**Success criteria**: +- Both images build on the host via the extended deploy.sh; staged smoke suite passes all four assertions (401 / healthz 200 / live-mode marker / SSE unbuffered). +- A keyed user on the staged instance gets fully real inference end to end through the proxy (`resolved_tier="user"` visible in the authenticated UI). +- Image-layer secrets sweep clean; no auth-disable path exists in the production image. +- Gates green; patch v0.3.4 shipped. + +## Phase 5 — Final Review + Ship (FINAL — not planned in detail here) + +Per ROADMAP: multi-persona review of the full milestone diff (REQ-69..85), audit (reconstruction, secrets sweep, branch/tag hygiene), then tag **v0.3.5 = the milestone release**. The pipeline HALTS at the merge_to_main owner approval gate (D-018 carries); deployment to research.coreci.dev proceeds only on owner directive (D-019 carries). The security-engineer's removal note (deactivate after P4) executes at P4 ship; the P5 review runs the standard full multi-persona pass. Detailed planning lands as the P5 PLAN when P4 completes, per the v0.2 precedent. + +## Cross-Phase Notes + +- **Versioning**: patches v0.3.1 (P1) → v0.3.2 (P2) → v0.3.3 (P3) → v0.3.4 (P4) → v0.3.5 (P5, the milestone release); tags on the v0.3.x line; MAJOR stays 0. +- **Carrying gates**: G-005 snapshot discipline applies in every route-touching phase (P1, P2, P3); G-007 cost counters survive as owner-keyed (P-13, P2); G-008 DDG fixture discipline carries untouched; D-018/D-019 halt + deploy gates bind at P4/P5. +- **REQ coverage assert**: each of REQ-69..85 maps to ≥1 test or contract proof in its phase (auth suite, key-store tamper suite, read-model suite, hosted smoke) — the audit reconstructs the milestone from these. +- **Interface additions**: the C3-18 list is exhaustive; any other ApiClient change requires a PLAN amendment before implementation (the G-002 lesson, restated). +- **Pitfall ledger (C3-19)**: P-1, P-2, P-13 close in P2; P-4, P-5, P-6, P-7, P-9, P-10, P-11, P-14 + P-8 (auth half) close in P1; P-8 (full) + P-12 close in P3; P-3 closes across P2 (backend half) and P4 (hosting half). All fourteen verified closed by v0.3.4. +- **Engine contracts unchanged throughout**: the only engine-file touches in the milestone are additive — `RunState.owner_id` stamping (P1) and the `_verify_phase` artifact writes (P3, D-113); no methodology, state-machine, or protocol rewrites anywhere. \ No newline at end of file -- 2.43.0 From 3ea96c09846d4633ff8138b9987ea8a86e3ba10d Mon Sep 17 00:00:00 2001 From: coreci Date: Tue, 29 Sep 2026 17:12:23 +0000 Subject: [PATCH 5/5] =?UTF-8?q?docs(grill):=20adversarial=20review=20?= =?UTF-8?q?=E2=80=94=2011=20challenges,=2010=20binding=20decisions,=201=20?= =?UTF-8?q?escalation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ---ci--- project: researchx phase: 0 milestone: v0.3 status: grill decisions: - id: G-101 decision: "Startup sweep handles PLANNING runs: C3-11's 'verified' CANCELLED-legality claim is false (states.py PLANNING to {AWAITING_APPROVAL,FAILED}); add PLANNING to CANCELLED in _LEGAL_TRANSITIONS or document a bypass; pinned by sweep test in T-1.8" rationale: "A crash mid-PLANNING leaves a persisted state the sweep cannot legally cancel — restart-path corruption at the first hosted redeploy" confidence: 0.90 - id: G-102 decision: "Stale out-of-scope rows fixed (REQUIREMENTS 'Auth/multi-tenancy' + PROJECT 'multi-tenant/auth') to match the recorded supersession" rationale: "Same-file contradictions with the owner-recorded v0.3 supersession" confidence: 0.95 - id: G-103 decision: "REQ-75 split bound (delegation-end P3, hosted-default P4); traceability row fixed; PLAN's pitfall-phase mapping authoritative over C3-19" rationale: "G-001-class triple-mapping contradiction across three documents" confidence: 0.85 - id: G-104 decision: "Exposure hardening: password min length 12 at register + admin-create; per-IP login rate limit at HAProxy front; audit asserts throttle + uniform 401s" rationale: "Public URL + argon2 ~50-100ms CPU per verify = DoS amplifier; no policy existed anywhere" confidence: 0.80 - id: G-105 decision: "KEK-loss posture stated: loss = per-user keys unrecoverable, honest degradation, re-entry under re-provisioned KEK; owner backs up off-repo; rotation out of v0.3 (recorded)" rationale: "KEK becomes load-bearing at P2; loss posture was stated nowhere" confidence: 0.85 - id: G-106 decision: "deploy.sh: free-disk assert + dangling-image prune + documented 4-container host budget" rationale: "P4 doubles image families on a single LXC host; current script has neither" confidence: 0.75 - id: G-107 decision: "Staged deploy defined: candidate pair on the host, health green, four smoke assertions host-locally; public cutover is a discrete owner-directive act with -prev armed" rationale: "P4's DoD used an undefined term; the host runs one container today" confidence: 0.75 - id: G-108 decision: "P3 pressure valve (G-006 precedent): may ship as P3a backend + P3b frontend patches without scope change" rationale: "P3 is the heaviest phase (7 REQs + standalone client + all swaps) with no pre-agreed valve" confidence: 0.70 - id: G-109 decision: "Task-inbox honesty: live-mode status derived (answered-iff DONE); checkbox display-honest in live mode; persisted check state is v0.4+; pinned by test" rationale: "REQ-78 promises 'checkable' but the derived GET-only read-model silently reverts user checks" confidence: 0.75 - id: G-110 decision: "Cost posture stated: server-tier runs bill the owner's key; bound = admin-managed accounts; per-user counters surface usage; no hard cap in v0.3" rationale: "Server-key-as-fallback (owner-picked) with the cost ceiling stated nowhere" confidence: 0.70 escalations: - "[G-111] P4's DoD (staged smoke on the LXC host) requires a host deployment D-019 forbids without a directive — owner must pre-authorize staged deploys for P4 DoD (no public cutover) or accept host proofs deferred to the gate" requirements: covered: [] partial: [] ---/ci--- Verdict: Proceed (0.68). 11/12 sampled research claims verified exact; the one false claim (sweep legality) bound as G-101 with a test pin. G-102/G-103 applied to REQUIREMENTS/PROJECT/PLAN in this commit; G-101/104..110 encoded as PLAN must-haves; G-111 visible via ciagent audit. --- .ciagent/CHECKPOINT.json | 4 +- .ciagent/GRILL.md | 215 ++++++++++++++++++++++++++++++++++++++- .ciagent/PLAN.md | 8 ++ .ciagent/PROJECT.md | 2 +- .ciagent/REQUIREMENTS.md | 6 +- 5 files changed, 228 insertions(+), 7 deletions(-) diff --git a/.ciagent/CHECKPOINT.json b/.ciagent/CHECKPOINT.json index fa6525b..58f28d9 100644 --- a/.ciagent/CHECKPOINT.json +++ b/.ciagent/CHECKPOINT.json @@ -1,8 +1,8 @@ { "phase": 0, - "stage": "plan", + "stage": "grill", "milestone": "v0.3", "phase_role": "pre_execution", "attempts": 0, - "updated_at": "2026-09-29T16:45:00Z" + "updated_at": "2026-09-29T16:55:00Z" } \ No newline at end of file diff --git a/.ciagent/GRILL.md b/.ciagent/GRILL.md index c81d89a..c79aba5 100644 --- a/.ciagent/GRILL.md +++ b/.ciagent/GRILL.md @@ -216,4 +216,217 @@ The plan is sound with fixable gaps. The evidence base is unusually strong for a ### Escalations - [G-009] Axis 2 — Owner ratification required at the P5 gate (D-018 halt): the v0.2 MVP demonstrates the **live core research loop only**; the remaining approved-prototype surfaces (Library, Settings persistence, Suggestions, Refine, graph/inbox previews, profile) remain mock-bound until v0.3+, and the live-mode gate demo requires a local backend + keys (or runs in mock mode) since D-019 forbids deploys. This redefines what "MVP done" looks like at the demo and must not be discovered by the owner at the gate — it must be ratified in advance. Engineering-side handling is bound in G-002/G-003; the ratification itself is unresolved until the owner records it. - - **RESOLVED 2026-09-27: RATIFIED.** Owner approval recorded via /ci-run 'APPROVED' at the v0.2.5 gate (decision commit 4721c2a), following the owner-directed UAT staging at https://research.coreci.dev. Live-mode scope = core research loop only is the accepted MVP boundary; mock-bound surfaces move to v0.3+. \ No newline at end of file + - **RESOLVED 2026-09-27: RATIFIED.** Owner approval recorded via /ci-run 'APPROVED' at the v0.2.5 gate (decision commit 4721c2a), following the owner-directed UAT staging at https://research.coreci.dev. Live-mode scope = core research loop only is the accepted MVP boundary; mock-bound surfaces move to v0.3+. +# CIAgent Grill Report + +## Run: 2026-09-29 17:05 (mode: mechanical, focus: all) + +### Verdict: Proceed (confidence: 0.68) + +The plan survives adversarial re-verification better than any artifact this project has produced: the grill independently re-checked the research file's load-bearing claims against the tree and found them exact — the shared-adapter fallback mutation (`ollama.py:73-79`), the `config_status` private-registry read (`app.py:174-177`), the CORS-credentials gap (`app.py:143-148`), the fake-serving module global (`app.py:359`), the missing `append_run_artifact` protocol method, the `[p-N]`/`[^pN]` marker split (`actor.py:337` vs `report-body.ts:18`), the export fabrication (`live-client.ts:296-304/335/376`), and the 8-test-file partitioning blast radius. The auth supersession is traceable end to end (a4afccc constraint rewrite + D-101..D-115 @ 33f4fa0 + ROADMAP session note — matching the owner's "one milestone / admin-managed / server-key-fallback" session picks). Engine-contract preservation is a real design discipline, not a slogan. But the grill found **one "verified" research claim that is false** (the restart sweep's CANCELLED-from-any-state legality — `states.py` forbids PLANNING→CANCELLED), **three phase/bookkeeping contradictions** (REQ-75 triple-mapped across three documents; two stale out-of-scope rows contradicting the same-file supersession; C3-19's pitfall-phase mapping vs PLAN), **one unaddressed prototype-fidelity gap** (the task-inbox checkbox cannot survive the derived live read-model), and **an honestly-unbudgeted public attack surface** (no password policy, no login rate limit on a TLS-fronted public URL). All are absorbed as binding decisions G-101..G-110; one item is genuinely owner-required (G-111: P4's definition-of-done depends on a staged host deployment that D-019 forbids without a directive). The milestone is heavier than v0.2 — the P3 pressure valve absent from the plan is bound here (G-108) as the G-006 precedent. Proceed. + +### Axis 1 — Business Case +- **Q1**: What problem does this actually solve, and is that problem still the top priority? + - Evidence: v0.2.5 approved and merged 2026-09-27 (8db45e4, b11b202); v0.3 scope owner-directed the next day, 2026-09-29 (a4afccc; ROADMAP "owner session 2026-09-29"); DEPLOY-RECORD.md itself records "live mode additionally needs the backend API on the host, which is v0.3+ deployment work" — the deployment gap is acknowledged in the ship record, not invented by the pipeline. + - Answer: Real and owner-named: convert the ratified mock-bound surfaces + single-user local loop into a hosted multi-user product with real inference. The same-day sequencing after v0.2.5 approval is the strongest available priority signal. + - Confidence: 0.80 +- **Q2**: Who is the named executive sponsor, and when did they last make a decision under pressure? + - Evidence: Owner pressure-tested through v0.1 (two gate rejections → binding extensions, commits 57c5231, 3fd0e10), G-009 ratified at the v0.2 gate (4721c2a); the v0.3 direction itself is a decision under pressure — superseding a recorded constraint ("single-user, local-first") the same day the MVP shipped. + - Answer: Sponsor engaged and decisive; the supersession is a recorded constraint change, not scope drift. + - Confidence: 0.85 +- **Q3**: What happens to the business if the project is cancelled? + - Evidence: v0.2.5 live mode = core research loop only, mock-default frontend, no hosting (G-009 ratification, ROADMAP v0.2 complete note); the owner-stated v0.3 purpose is the hosted product. + - Answer: Cancelling strands the G-009 promise ("mock-bound surfaces move to v0.3+") and leaves the product undeployable in its owner-intended form. Not a zombie — the payoff milestone. + - Confidence: 0.75 +- **Q4**: Is the ROI calculated against a counterfactual? + - Evidence: MISSING — no formal ROI anywhere; G-007 counters carried advisory-only, owner-keyed in P2 (T-2.6, P-13). v0.3 adds real costs v0.2 did not carry: public infrastructure exposure and the server-tier key serving any admin-created user's runs. + - Answer: Still owner-judgment — tolerable for a sponsor-beneficiary product with an admin-bounded user set, but the server-key cost posture is now a stated-nowhere exposure; bound as G-110. + - Confidence: 0.55 + - Decision: G-110 — document the cost posture: server-tier runs bill the owner's key; the bound is admin-managed accounts; per-user counters surface server-tier usage; hard caps deferred v0.4+. + +### Axis 2 — Scope and Requirements +- **Q1**: Is the scope expanding, contracting, or genuinely stable? + - Evidence: Scope grew by owner directive (ROADMAP 2026-09-29 note); REQ-69..85 frozen with per-phase traceability; PLAN frontmatter carries all 17. But the G-001-class phase contradictions persisting: REQ-75 is mapped P3 by REQUIREMENTS traceability, P4-only by ROADMAP P3/P4 lists, and split by PLAN (delegation-end in P3, hosted in P4). C3-19 likewise maps P-1/P-2/P-13 to P1 while PLAN closes them in P2. + - Answer: Formal scope stable and owner-frozen; the traceability layer is where the drift lives — three documents disagree about one requirement. Bound in G-103. + - Confidence: 0.60 + - Decision: G-103 — bind the REQ-75 split explicitly; fix the traceability row notation; declare PLAN's pitfall-closure mapping authoritative over C3-19's compressed prose. +- **Q2**: Who owns the requirements, and have they been frozen? + - Evidence: Owner-directed supersession traceable (a4afccc strikethrough + rationale; D-101..D-115 clarify commit 33f4fa0 at confidence 0.75–0.95). + - Answer: Owner owns; pipeline freezes and traces. The supersession discipline is exemplary — the constraint change is visible, dated, and strikethrough-preserved rather than silently rewritten. + - Confidence: 0.85 +- **Q3**: What is explicitly out of scope? + - Evidence: **CONTRADICTION FOUND** — REQUIREMENTS.md Out-of-Scope table still carries "Auth / multi-tenancy | Single-user local-first MVP"; PROJECT.md Out of Scope still lists "multi-tenant/auth" — while PROJECT.md:34 records the supersession in the same file and REQ-69..71 define the now-in-scope auth. The honest deferrals (OAuth v0.4+, 6 methodologies v0.4+, KEK rotation, account deletion, rehydration) are all recorded. + - Answer: Mostly honest, with two stale rows that directly contradict a recorded decision — exactly the spec-comment drift class the v0.2 grill flagged. Bound in G-102. + - Confidence: 0.85 + - Decision: G-102 — update both stale out-of-scope rows to match the supersession. +- **Q4**: Are there hidden requirements only disclosed late in delivery? + - Evidence: **FOUND — three.** (a) REQ-78 promises "checkable follow-ups"; `library-view.tsx:430-471` ships a working checkbox; the live read-model (D-114) derives status answered-iff-DONE over a GET-only route — a live user's check silently reverts on refetch; PLAN never addresses it. (b) No password minimum length exists anywhere (grep-verified). (c) No login rate limiting anywhere (grep-verified) — argon2 verify is ~50-100ms CPU, a public flood is a DoS amplifier on a single worker. + - Answer: The v0.2 G-002 lesson recurs once (the toggle); the exposure items are new-hosting hidden requirements. All three bound (G-109, G-104). + - Confidence: 0.75 + +### Axis 3 — Architecture and Technical Feasibility +- **Q1**: Has the architecture been validated by the people who will build and operate it? + - Evidence: RESEARCH-v03 claims re-verified by this grill: 11 of 12 sampled claims exact. **One claim is FALSE**: RESEARCH §7/C3-11 "rewrite every non-terminal state.json as CANCELLED (legal from every non-terminal state, verified states.py)" — `states.py` `_LEGAL_TRANSITIONS[PLANNING] = {AWAITING_APPROVAL, FAILED}`: no CANCELLED. A crash mid-PLANNING leaves a persisted PLANNING state the sweep cannot legally cancel. + - Answer: Design-side validation is the strongest yet — but the single false "verified" claim is the precise failure mode mechanical research invites: verification asserted, not re-run. Bound in G-101. + - Confidence: 0.85 + - Decision: G-101 — startup sweep must handle PLANNING-state runs: add PLANNING→CANCELLED to `_LEGAL_TRANSITIONS` or document the sweep as an explicit transition-validation bypass; pinned by test in T-1.8. +- **Q2**: What is the integration surface, and where does it quietly double in cost? + - Evidence: Five seams: auth dependency threading (P1), per-run factory + probe-per-tier (P2), disk read-models (P3), snapshot regen ×3 (G-005 wired into P1/P2/P3), HAProxy front extension (P4). The cost-doubler candidate is SSE-through-proxy: the only seam whose live configuration is deferred to deploy time. + - Answer: Residual risk is bounded, not eliminated: the P4 smoke asserts a full unbuffered SSE run through the proxy path with the `-prev` rollback pair retained — worst case is one deploy iteration, not milestone rework. Accepted with the G-111 governance condition. + - Confidence: 0.65 +- **Q3**: Is there an existing system being replaced? + - Evidence: Greenfield hosting; pre-hosted artifacts explicitly disposable (D-106, owner-recorded); the "replaced system" is the mock delegation layer (live-client.ts:61) — deletion is D-112/T-3.8 with component tests proving zero rewrites (T-3.12). Volume model: admin-managed accounts (REQ-70) bound the user set; single worker mandatory. + - Answer: No migration risk; the swap is designed as a deletion with regression pins. The one fidelity trap (task-inbox toggle) is found and bound (G-109). + - Confidence: 0.75 +- **Q4**: What is the technical debt being inherited, and is it budgeted for? + - Evidence: The 14-item pitfall ledger is budgeted per-phase (PLAN cross-phase notes) — with C3-19's compressed phase mapping contradicting PLAN (folded into G-103). Marker normalization: one-directional at the report.json boundary only; report.md raw form pinned by test; trust-loop regexes untouched. + - Answer: If a future writer emits `[^pN]` natively, `_cited_urls` matches nothing → ledger empties → the G-004 export gate fails visibly at runtime, and the raw-form pin catches it in CI first. Silent corruption is not reachable. + - Confidence: 0.70 + +### Axis 4 — People, Skills, and Organization +- **Q1**: Which 2–3 people, if they left, would the project fail? + - Evidence: One agent pipeline + one owner; gates owner-halt by design (D-018). The security-engineer persona is phase-specific P1/P2/P4 with an explicit removal note (PERSONAS.md). + - Answer: Total key-person dependency remains structural and accepted; the P5 full multi-persona review covers the security persona's absence. + - Confidence: 0.60 +- **Q2**: Are resources actually allocated at the percentages claimed? + - Evidence: v0.1 shipped ~9 phases in ~1 day; v0.2 shipped P0..P5 in one day; v0.3 P0 commits same-day (a4afccc→dca86a7). + - Answer: Velocity verified by artifact trail. v0.3's four phases are each multi-domain — the evidence supports capacity, with the G-108 valve for the heaviest phase. + - Confidence: 0.80 +- **Q3**: Is there a product owner with actual authority to prioritize? + - Evidence: Owner-directed supersession recorded same-day as MVP ship (a4afccc); G-009 ratified at gate (4721c2a); D-018/D-019 carry. + - Answer: Unambiguous single owner, exercised. + - Confidence: 0.90 +- **Q4**: Is the team building capability it doesn't have, hoping to learn as they go? + - Evidence: argon2-cffi, AESGCM envelope encryption, server-side sessions, SSE-through-proxy are all new to the codebase; mitigations are research-verified concrete shapes + adversarial suites specced + a dedicated security-engineer persona on exactly the phases that carry the new capability. + - Answer: Yes — learning-while-doing on a security-critical public surface, the highest-stakes version of this risk. The mitigation structure is proportionate. Acceptable; the G-104 hardening items close the gaps the research did not self-identify. + - Confidence: 0.60 + +### Axis 5 — Timeline and Estimates +- **Q1**: Was the deadline set before or after scope and approach were understood? + - Evidence: No wall-clock deadlines anywhere; dependency-driven phases; owner gates as the only cadence. + - Answer: Healthiest pattern, unchanged from v0.2. + - Confidence: 0.85 +- **Q2**: What is the critical path, and what single thing would push it by 3+ months? + - Evidence: P1→P2→P3→P4 sequential. P3 is the heaviest (7 REQs + standalone client + every surface swap + Reading Room fidelity + delegation-end); P1 is the widest (auth + partitioning across the 10-touchpoint list). The v0.2 pusher-class risk (integration drift at the swap) is pre-defused by G-005 wired into all three route-touching phases + the C3-18 exhaustive interface list. + - Answer: The pusher is P3 overrun with no pre-agreed valve — v0.2 had G-006; v0.3's plan had none. Bound in G-108. + - Confidence: 0.70 + - Decision: G-108 — P3 may ship as two patches (P3a backend; P3b frontend) or run mid-phase wave gates, without scope change; tags extend on the v0.3.x line. +- **Q3**: Are the estimates evidence-based? + - Evidence: v0.1/v0.2 velocity does not transfer 1:1 (auth/crypto/hosting are new domains); no per-phase sizing exists. The honest mitigation remains the phase-patch mechanism — now with the P3 seam named in advance. + - Answer: No estimates to be wrong; the valve replaces fake numbers. Endorsed, per the v0.2 precedent. + - Confidence: 0.60 +- **Q4**: Is there a working definition of done? + - Evidence: Per-phase must-haves are concretely testable (401 pins, two-user isolation, tamper pins, sweep test, drift gate, four smoke assertions); P5 owner gate. **One DoD term is undefined**: T-4.7/T-4.9 run the smoke "against a staged deploy" — "staged" is never defined, and the host runs one container today. + - Answer: DoD has teeth everywhere except the P4 smoke's target. Bound in G-107; its owner-contingency half escalates (G-111). + - Confidence: 0.75 + - Decision: G-107 — "staged deploy" defined: candidate pair built + run on the LXC host, health green, the four smoke assertions executed host-locally against the candidates; public TLS-front cutover remains a discrete owner-directive act (D-019) with post-cutover public assertions run with the -prev pair armed. + +### Axis 6 — Budget and Financial Realism +- **Q1**: What percentage of budget is spent vs. remaining? + - Evidence: MISSING — no cost ledger (v0.2 finding, carried). New v0.3 spends: second container family, argon2-cffi + cryptography (OSS), HAProxy front extension (existing instance), owner's Ollama key serving server-tier users. + - Answer: Untracked, small in absolute terms, tolerable for sponsor-beneficiary scope — but the server-tier exposure is a stated-nowhere cost posture (G-110). + - Confidence: 0.60 +- **Q2**: Are there predictable cost drivers not in the original budget? + - Evidence: (a) Server-key tier: any admin-created user's keyless runs bill the owner (REQ-74 chain; G-007 counters advisory-only). (b) Host disk: deploy.sh (212 lines, verified) builds one image today with no free-space assert and no image prune; P4 doubles image families and runs 4 containers. + - Answer: Both are real; (a) is a policy posture to state (G-110), (b) is deploy hygiene to add (G-106). + - Confidence: 0.70 + - Decision: G-106 — deploy.sh gains a pre-build free-disk assertion and a post-swap dangling-image prune; the two-image/rollback-pair footprint is documented as the host budget. +- **Q3**: Burn rate and runway? + - Evidence: Keyless localhost + DDG fallback keep the zero-marginal-cost degraded posture (G-008 carries); per-user keys shift burn to users; refine endpoints ride the same chain with honest 503. + - Answer: Runway effectively unbounded in degraded mode; hosting costs are fixed and small. Sound. + - Confidence: 0.70 +- **Q4**: Is the budget contingent on something that hasn't happened yet? + - Evidence: Infra exists (LXC + HAProxy already TLS-fronting research.coreci.dev); keys exist in scopes. One new prerequisite: RESEARCHX_KEK does not exist yet (provisioned into `.ciagent/.env.secrets` at P2 — never git). + - Answer: Not contingent in any structural sense; the KEK is a trivial owner action — which is precisely why its loss posture must be stated before it becomes load-bearing (G-105). + - Confidence: 0.75 + - Decision: G-105 — KEK-loss posture stated honestly: loss = per-user encrypted keys unrecoverable, chain degrades to server/keyless honestly, users re-enter keys under a re-provisioned KEK; the owner MUST back up the KEK off-repo; rotation tooling stays out of v0.3 (recorded, not silently dropped). + +### Axis 7 — Risks, Assumptions, and Dependencies +- **Q1**: Top 3 assumptions, and evidence for each? + - Evidence: (a) *HAProxy live-host config accepts the SSE-safe front extension* — version and TLS front evidenced by two prior deploys; global/default sections unread until P4. Medium evidence. (b) *Single-worker in-memory registry/sessions suffice* — bounded by admin-managed accounts; restart sweep + pruning + disk read-models remove restart-fragility. Good evidence. (c) *Zero-component-rewrite surface swaps hold* — evidenced by the v0.2 P3 precedent + G-005 + T-3.12 regression suite; the task-toggle is the one discovered exception (G-109). + - Answer: Two well-evidenced; (a) is the residual, bounded by the P4 smoke + rollback pair. + - Confidence: 0.60 +- **Q2**: External dependencies outside the team? + - Evidence: HAProxy host state (owner-held, D-019-gated), Ollama cloud/local + Tavily/DDG (carried v0.2, G-008 carries), Docker host disk (G-106 binds the missing hygiene). + - Answer: All acknowledged with degradation or rollback paths; the proxy is the only one without a code-level fallback, and it has a process-level one. + - Confidence: 0.65 +- **Q3**: The single risk that kills the project, and is there a mitigation? + - Evidence: Authorization-class auth bugs on a public URL (cross-user leakage) — mitigated by construction: owner-keyed registry 404s + partitioned storage + per-request owner-scoped adapters + the two-user isolation test pinned in P1/P3 + the P4 exposure audit + admin-managed accounts bounding the victim set. The sweep-legality bug (G-101) is the found instance of the adjacent class. + - Answer: Kill risk is real but the mitigation stack is the plan's strongest suit; the found bug proves the stack needs the grill, and now it has the fix. + - Confidence: 0.70 +- **Q4**: Has a pre-mortem been done? + - Evidence: MISSING as a formal artifact; this grill is the proxy. + - Answer: "12 months out, v0.3 failed" — derivable causes: (1) the sweep bug corrupting restarts at first redeploy (bound: G-101); (2) P3 fidelity parity slips — one found (toggle, G-109), the rest pinned by parity tests; (3) proxy/SSE iteration thrash post-cutover (bounded: smoke + rollback); (4) KEK lost with no documented posture (bound: G-105); (5) public brute-force/DoS on a fresh auth surface (bound: G-104). All five now bound or bounded-by-design. + - Confidence: 0.65 + +### Axis 8 — Governance, Decision-Making, and Communication +- **Q1**: Who is the decision-maker when two parties disagree? + - Evidence: Single owner; D-018/D-019 carry; escalation hooks codified (config.json). + - Answer: Unambiguous; no dual-executive stall possible. + - Confidence: 0.90 +- **Q2**: Governance cadence and escalation pattern — useful in a crisis? + - Evidence: Per-phase ship + gates; the pipeline halted itself at the v0.2 gate correctly; deploy is owner-directive-only. **New v0.3 wrinkle**: P4's phase-ship DoD requires a host deployment that D-019 forbids without a directive — the first time a phase's success criteria are contingent on a mid-milestone owner action. T-4.9 acknowledges it but does not resolve the contingency. + - Answer: The escalation pattern is crisis-proven, but this dependency is new and unresolved — if the owner declines the directive at P4, REQ-83..85 ship unproven-on-host and the milestone gate presents hosted requirements on faith. Escalated as G-111. + - Confidence: 0.70 +- **Q3**: What is being omitted from status reports? + - Evidence: `---ci---` honesty verified historically. The channel where optimism leaked this cycle is the research file: C3-11 asserts "verified states.py" for a claim that is false — asserted verification that was not re-run. This grill re-ran it. + - Answer: Status reporting remains honest; asserted-but-unre-run verification is the leak channel, and the fix is cultural (G-101's test pin makes the claim checkable). + - Confidence: 0.80 +- **Q4**: Is there a stop-the-project trigger? + - Evidence: Owner rejection precedent (twice in v0.1); D-019 is the hosting-scope-specific stop trigger — no public cutover without directive; merge halts at the gate (D-018). + - Answer: Yes, and the hosting design adds a second one: the staged/public split (G-107) means exposure is a discrete owner decision, not a gradual slide. + - Confidence: 0.85 + +### Axis 9 — Change, Adoption, and Operational Readiness +- **Q1**: Who will use this, how does their work change, and what's in it for them? + - Evidence: Admin-managed accounts (REQ-70) — the owner creates the user set; work change = login + hosted real inference with per-user keys vs. local setup; UX Acceptance Criteria preserve the ≤3-click path with login as a once-per-session step. + - Answer: Small, known, sponsor-adjacent user set; adoption politics minimal, value concrete. + - Confidence: 0.70 +- **Q2**: Is ops involved now, or handed a finished product at go-live? + - Evidence: The pipeline is ops; deploy.sh extends with paired rollback for both services (T-4.4); the startup sweep is the redeploy safety net (T-1.8); volume ownership handled (chown 10001, T-4.4). Gaps found: no disk assert/prune (G-106), undefined "staged" (G-107), KEK operational posture undocumented (G-105). + - Answer: Ops is built-in; three hygiene gaps bound. + - Confidence: 0.65 +- **Q3**: What is the rollback plan? + - Evidence: Paired `-prev` containers for BOTH services (T-4.4); mock-dev default preserved at the code level (REQ-75/D-112); pre-hosted artifacts disposable (D-106); sessions durable across restarts (D-101). + - Answer: Multi-layer and codified — the strongest single dimension of this plan. + - Confidence: 0.80 +- **Q4**: Have success criteria been validated with the people who will judge success? + - Evidence: ROADMAP v0.3 success criteria are concrete (401 curl, SSE unbuffered, two-user isolation, live-mode default on the staged instance); owner gate precedent enforces criteria. **Found**: REQ-78's own text promises "checkable follow-ups" while D-114/T-3.5 derive answered-iff-DONE over a GET-only route — the approved prototype's working checkbox (`library-view.tsx:463-471`) cannot hold live. + - Answer: One criterion-vs-implementation mismatch found; bound as honest degradation (G-109) rather than silent reversion. + - Confidence: 0.65 + - Decision: G-109 — live-mode task status is derived (open unless run DONE); the inbox checkbox becomes display-honest in live mode (disabled/annotated; mock keeps the free toggle for dev); persisted per-question check state is v0.4+ scope (REQ-45 backlog); pinned by a live-mode component test. + +### Meta — Closing Review +- **Q1**: If you were the auditor, what would you flag? + - Evidence: The false "verified" claim (C3-11 sweep legality vs `states.py` PLANNING); the REQ-75 triple mapping; the two stale out-of-scope rows contradicting the same-file supersession; C3-19 vs PLAN pitfall-phase mapping. + - Answer: Flag #1 is the sweep bug — a restart-path correctness claim that would have detonated at the first hosted redeploy, the exact moment the milestone's value proposition is under evaluation. Flags #2-4 are the recurring phase-mapping/bookkeeping class (G-001's relatives), now bound (G-102/G-103). + - Confidence: 0.80 +- **Q2**: What is the project not doing that it should? + - Evidence: Found across axes: exposure hardening (password policy, login rate limit — G-104); KEK-loss posture (G-105); deploy disk hygiene (G-106); staged-deploy definition (G-107); P3 pressure valve (G-108); task-toggle honesty (G-109); cost posture statement (G-110). + - Answer: Seven omissions, all bound. Nothing found suggests the project is doing something it should not — engine-contract preservation holds everywhere checked. + - Confidence: 0.75 +- **Q3**: What is the simplest version that delivers 80% of the value? + - Evidence: The owner explicitly picked "one milestone"; deferring live surfaces would re-create the G-009 mock-bound state the owner just moved past — the lever is not deferral. + - Answer: The pre-agreed lever is the P3 split (G-108), decided now while it is cheap — not surface deferral, which would contradict the owner's directive. + - Confidence: 0.65 +- **Q4**: What must be true for this to succeed in the next 90 days, and is it true today? + - Evidence: (a) D-101..D-115 workable against the tree — re-verified (11/12 sampled exact, 1 false and bound). (b) Owner engagement — evidenced same-day. (c) The KEK provisioned by P2 — trivial owner action, posture bound (G-105). (d) The P4 staged host deployment authorized — currently NOT true; D-019 requires a directive the plan does not pre-arrange (G-111). + - Answer: (a)(b)(c) true or evidenced; (d) becomes true when the owner records the pre-authorization. That escalation is the gating action of this grill. + - Confidence: 0.70 + +### Binding Decisions +| ID | Axis | Decision | Confidence | +|----|------|----------|-----------| +| G-101 | 3/7 | Startup sweep must handle PLANNING-state runs: RESEARCH §7/C3-11's "CANCELLED legal from every non-terminal state" is false (`states.py`: PLANNING → {AWAITING_APPROVAL, FAILED}). Add PLANNING→CANCELLED to `_LEGAL_TRANSITIONS` or document the sweep as an explicit transition-validation bypass; pin with a PLANNING-fixture→CANCELLED sweep test in T-1.8. | 0.90 | +| G-102 | 2 | Fix the stale out-of-scope rows contradicting the recorded supersession: REQUIREMENTS.md Out-of-Scope ("Auth / multi-tenancy") and PROJECT.md Out of Scope ("multi-tenant/auth") — replaced with the honest v0.3 posture (auth/multi-user in scope; open self-serve registration and OAuth out; admin-managed accounts only, OAuth v0.4+). | 0.95 | +| G-103 | 2/5 | Phase-mapping integrity (G-001 precedent): bind REQ-75's split — delegation-end half closes in P3 (pinned by the standalone-client/no-mock-delegation tests), hosted-live-default half in P4 (pinned by the live-mode smoke marker); fix the traceability row (REQ-75..82 → REQ-76..82 + REQ-75 split note); declare PLAN's pitfall-closure mapping authoritative over C3-19's compressed phase prose. | 0.85 | +| G-104 | 2/3 | Public-exposure hardening, budgeted honestly: (a) minimum password length ≥12 enforced at register + admin-create (P1 T-1.4/T-1.5, uniform error messages preserved); (b) per-IP login rate limit at the HAProxy front (stick-table on /api/auth/*, P4 T-4.5) bounding argon2 CPU burn on the single worker; (c) the T-4.8 exposure audit asserts the throttle and confirms uniform 401s (no user enumeration). | 0.80 | +| G-105 | 6/7 | KEK-loss posture stated honestly (PLAN T-2.2 + owner docs): RESEARCHX_KEK's only copy is the env-file (never git); loss = per-user encrypted keys unrecoverable (AES-256-GCM + AAD), chain degrades to server/keyless honestly, users re-enter keys under a re-provisioned KEK; the owner MUST back up the KEK off-repo; rotation/re-wrap tooling stays out of v0.3 (recorded, not silently dropped). | 0.85 | +| G-106 | 6 | deploy.sh two-image hygiene: pre-build free-disk assertion, post-swap dangling-image prune, and a documented host budget (2 image families × live+prev, 4 containers) — the current script has neither, and P4 doubles the footprint on a single LXC host. | 0.75 | +| G-107 | 5/9 | "Staged deploy" defined: candidate pair (researchx + researchx-api) built + run on the LXC host, health checks green, the four smoke assertions executed host-locally against the candidates; public TLS-front cutover remains a discrete owner-directive act (D-019) with post-cutover public assertions (public 401, SSE through proxy) run with the -prev pair armed. | 0.75 | +| G-108 | 5 | G-006 precedent applied to P3: may ship as two patches (P3a backend artifacts + read-models + routes + read-model suite; P3b frontend standalone client + surface swaps + Reading Room fidelity + delegation-end) or run mid-phase wave gates, without scope change; tags extend on the v0.3.x line; the milestone release remains the final phase's patch. | 0.70 | +| G-109 | 9 | Task-inbox honesty (G-002 lesson): live-mode status is derived (answered-iff run DONE, D-114); the inbox checkbox (`library-view.tsx:463-471`) becomes display-honest in live mode — disabled/annotated, never a silent reversion; mock keeps the free toggle for dev; persisted per-question check state recorded as v0.4+ (REQ-45 backlog); pinned by a live-mode component test in T-3.12. | 0.75 | +| G-110 | 1/6 | Cost posture stated in PLAN/PROJECT: server-tier runs bill the owner's OLLAMA_API_KEY; the bound is the admin-managed account set (registration closed, REQ-70); per-user counters (P-13, T-2.6) surface server-tier usage per user for owner audit; no hard spend cap in v0.3 — hard caps are v0.4+ scope if the user set grows; keyless localhost remains the zero-cost path. | 0.70 | + +### Escalations +- [G-111] Axis 5/8 — Owner pre-authorization required before P4 executes: P4's own definition of done ("staged smoke suite passes all four assertions", ROADMAP/PLAN T-4.7/T-4.9) requires deploying the candidate container pair to the LXC host — an action D-019 forbids without an owner directive. Without pre-authorization, P4 ships with REQ-83..85 unproven on the real host (the HAProxy 2.8 live config is unread until then) and the milestone gate would present hosting requirements on faith. The owner must either (a) pre-authorize staged host deployments for P4 DoD purposes (candidate pair + host-local smoke, NO public cutover — cutover stays a separate owner directive per G-107), or (b) accept that P4's phase ship carries hosting proven only on a local docker pair, with the host proofs deferred to the gate. This is the first phase in the project whose success criteria are contingent on a mid-milestone owner action; it must not be discovered at P4 ship time. Engineering-side handling is bound in G-107; the authorization itself is unresolved until the owner records it. diff --git a/.ciagent/PLAN.md b/.ciagent/PLAN.md index 9191f29..49a9cda 100644 --- a/.ciagent/PLAN.md +++ b/.ciagent/PLAN.md @@ -13,6 +13,14 @@ must_haves: - "Zero-key test discipline: every CI suite runs on REQ-31 fakes with auth via create_app(auth_enabled=False) or dependency_overrides; keyed/live tiers are opt-in, skipped by default" - "Single uvicorn worker in every serving mode — the in-memory RunRegistry and session store are single-process by design" - "Mock demoted to dev/test fixture: the code default of RESEARCHX_API_MODE stays mock; after P3 no live-mode surface delegates to MockApiClient; hosted builds inline NEXT_PUBLIC_RESEARCHX_API_MODE=live (REQ-75)" + - "G-101: startup sweep handles PLANNING-state runs — add PLANNING→CANCELLED to _LEGAL_TRANSITIONS (or document the sweep as an explicit validation bypass); pinned by a PLANNING-fixture→CANCELLED sweep test in T-1.8" + - "G-104: public-exposure hardening — password minimum length ≥12 at register + admin-create (T-1.4/T-1.5); per-IP login rate limit at the HAProxy front (stick-table on /api/auth/*, T-4.5); exposure audit asserts the throttle + uniform 401s (T-4.8)" + - "G-105: KEK-loss posture documented (T-2.2 + owner docs): loss = per-user keys unrecoverable, honest degradation to server/keyless, re-entry under a re-provisioned KEK; owner backs up the KEK off-repo; rotation tooling out of v0.3 (recorded)" + - "G-106: deploy.sh hygiene — pre-build free-disk assertion + post-swap dangling-image prune + documented host budget (2 image families × live+prev)" + - "G-107: 'staged deploy' = candidate pair built + run on the LXC host, health green, four smoke assertions host-locally; public TLS-front cutover stays a discrete owner-directive act with -prev armed" + - "G-108: P3 pressure valve (G-006 precedent) — may ship as P3a (backend artifacts + read-models + routes + suite) and P3b (standalone client + swaps + fidelity + delegation-end), tags extending on the v0.3.x line, without scope change" + - "G-109: task-inbox honesty — live-mode status is derived (answered-iff run DONE); the checkbox is display-honest in live mode (disabled/annotated; mock keeps the free toggle); persisted check state is v0.4+ (REQ-45 backlog); pinned by a live-mode component test in T-3.12" + - "G-110: cost posture stated — server-tier runs bill the owner's OLLAMA_API_KEY; the bound is the admin-managed account set; per-user counters surface server-tier usage; no hard spend cap in v0.3 (hard caps v0.4+); keyless localhost remains the zero-cost path" personas: - backend-engineer - data-engineer diff --git a/.ciagent/PROJECT.md b/.ciagent/PROJECT.md index 5f3ac63..2b34811 100644 --- a/.ciagent/PROJECT.md +++ b/.ciagent/PROJECT.md @@ -40,7 +40,7 @@ A deep-research studio where you bring your own model, storage, and memory; choo - No deploys without owner directive (D-019 carries); milestone merge to main halts at the owner approval gate (D-018 carries) ## Out of Scope -- Paywalled academic sources; model hosting; multi-tenant/auth; native Ollama API adapter; domain registration/rename decision +- Paywalled academic sources; model hosting; open self-serve registration / OAuth (admin-managed accounts only, OAuth v0.4+); native Ollama API adapter; domain registration/rename decision ## Context Vision finalized and locked 2026-09-26: v1 methodology composition LOCKED (8 core), product named ResearchX, OpenBao retained as architectural secrets component, criticism matrix and evidence base [E1]–[E15] on file. Owner revisions 2026-09-27: (1) prototype-first — UI/UX prototype with mock data must be approved before MVP work begins; (2) version starts at v0.1, MAJOR remains 0 until MVP released; (3) the prototype is its own milestone. Milestone plan: v0.1 prototype (this milestone) → v0.2 MVP (gated). diff --git a/.ciagent/REQUIREMENTS.md b/.ciagent/REQUIREMENTS.md index c4b411d..2b0cf73 100644 --- a/.ciagent/REQUIREMENTS.md +++ b/.ciagent/REQUIREMENTS.md @@ -139,7 +139,7 @@ |---|---| | Paywalled academic access | Unsolved in v1 — routed around via open APIs (vision §8) | | General-purpose chatbot / model host / closed content index | Explicit non-goals (vision §8) | -| Auth / multi-tenancy | Single-user local-first MVP | +| Auth / open self-serve registration / OAuth | v0.3 ships user/pass auth with admin-managed accounts (owner-directed 2026-09-29); open registration and OAuth stay out (OAuth v0.4+) | | Native Ollama API adapter | Ollama supported via OpenAI-compatible endpoint only | | Domain registration / product rename | Deferred by owner (vision §11) | | v1.0.0 release | Owner-approved release only; not created autonomously by the pipeline | @@ -163,6 +163,6 @@ | All REQ-01..34 (milestone gate) | v0.2 | P5 | complete (v0.2.5 — **owner approval recorded 2026-09-27**, G-009 ratified; milestone merged to main) | | REQ-69..71 (auth, sessions, scoping) | v0.3 | P1 | pending | | REQ-72..74 (settings, per-user keys, resolution chain) | v0.3 | P2 | pending | -| REQ-75..82 (live default + live surfaces, user-scoped) | v0.3 | P3 | pending | -| REQ-83..85 (hosting; live default deployed) | v0.3 | P4 | pending | +| REQ-76..82 (live surfaces, user-scoped) | v0.3 | P3 | pending | +| REQ-75 (delegation-end half) + REQ-83..85 (hosting; live default hosted) | v0.3 | P3/P4 (REQ-75 split per G-103: delegation-end P3, hosted-default P4) | pending | | REQ-35..51 | v2 backlog (post-MVP milestones) | — | pending | -- 2.43.0