researchx/.ciagent/PROJECT.md

ResearchX

Name

ResearchX (working name; rename deferred — naming collision audit on file, see vision Decisions Log)

Core Value

A deep-research studio where you bring your own model, storage, and memory; choose a research methodology before a single query fires; and every claim in the final report is anchored to a visited, verified, quotable source. Simple enough to run in three clicks, transparent enough to audit line by line.

Objective (two-milestone plan, owner-directed 2026-09-27)

1. Milestone v0.1 — Prototype: UI/UX only, mock data, the production Next.js frontend with all data behind a typed ApiClient interface backed by MockApiClient. Surfaces: Ask, Choose (all 8 methodology cards), Plan & Run, Reading Room v0, v2 surface mockups (project graph, task inbox). Ships only after owner approval — no MVP work begins until approved.
2. Milestone v0.2 — MVP: gated on prototype approval. FastAPI backend, custom async engine, adapter layer, trust loop, and integration (MockApiClient swapped for the OpenAPI-generated client).

Requirements (validated)

1. Five pluggable adapter slots: model (OpenAI-compatible, planner/writer split), storage (local FS), search (Tavily default, DuckDuckGo keyless fallback), memory (off — stubbed), secrets (env — OpenBao later)
2. Hardware-aware defaults: probe backend, cap requests conservatively (8192 context default, embedding batch 8), surface overrides in UI, degrade gracefully
3. 12-dial methodology framework (typed schema, every run persists resolved dial values for reproducibility/auditability)
4. Methodology registry shipping Rapid Answer (depth 1/breadth 1/memo/per-claim citations) and Deep Report (depth 3/breadth 2/executive report/full verification); 8-methodology catalog locked, phased rollout
5. Custom async engine: state machine PLANNING → AWAITING_APPROVAL → SEARCHING → EXTRACTING → REFLECTING → SYNTHESIZING → VERIFYING → DONE; durable JSON state; pause/cancel/resume; typed event bus → SSE live trace
6. Trust loop in every methodology: visited-only constraint at generation time; URL liveness pass with live/stale/dead classification; claim-to-passage quote anchoring; citation ledger JSON per run; exportable verification checklist
7. API surface: POST /api/runs, GET /api/methodologies, run lifecycle (approve/pause/cancel/resume), SSE events, config status; OpenAPI-generated typed frontend client
8. Frontend: Ask, Choose, Plan & Run screens + Reading Room v0 (page-anchored citations with hover quotes, ledger panel, Markdown/JSON export)
9. Testing: fake model/search adapters enabling zero-API-key engine tests; adversarial hallucinated-URL and quote-mismatch detection tests targeting ≥99% citation liveness
10. Tech stack: FastAPI (Python 3.12, uv, pydantic) + Next.js App Router (TypeScript strict)

Requirements (active, later milestones)

• Remaining 6 methodologies: SOTA Survey, Systematic Review (PRISMA-like), Fact-Check Dossier, Multi-Perspective Dossier, Competitive Landscape, Corpus Interrogation
• Reading Room full: project graph, global search, task inbox, confidence labels, BibTeX/PDF export
• Memory scopes (semantic caching), S3/git storage adapters, Exa/Perplexity/Google search, OpenBao secrets, Docker packaging

Constraints

• Prototype must be approved before MVP work begins (owner directive)
• Versioning starts at v0.1; MAJOR remains 0 until an owner-approved v1 release; all pre-v1 ships are MINOR/PATCH only (owner directive)
• Not a general-purpose chatbot; not a model host; not a closed content index (v1)
• Paywalled academic access out of scope v1 — open APIs (ArXiv/PubMed) only
• Single-user, local-first tool — no auth/multi-tenancy in MVP
• Ollama/vLLM supported via OpenAI-compatible endpoints only (no vendor-specific adapters)
• Secrets never in app config or git; model capability honesty (warn when selected model is weak at structured outputs)

Out of Scope

• Paywalled academic sources; model hosting; multi-tenant/auth; native Ollama API adapter; domain registration/rename decision

Context

Vision finalized and locked 2026-09-26: v1 methodology composition LOCKED (8 core), product named ResearchX, OpenBao retained as architectural secrets component, criticism matrix and evidence base [E1]–[E15] on file. Owner revisions 2026-09-27: (1) prototype-first — UI/UX prototype with mock data must be approved before MVP work begins; (2) version starts at v0.1, MAJOR remains 0 until MVP released; (3) the prototype is its own milestone. Milestone plan: v0.1 prototype (this milestone) → v0.2 MVP (gated).

Key Decisions

• Methodology-first selection is the core differentiator, not raw model quality — converts ad-hoc prompt-crafting into structured, reproducible selection
• Visited-only citation constraint + mandatory verification pass in all methodologies — deep-research agents hallucinate URLs at the highest measured rates; verification cuts failures to <1% [E1]
• Progressive disclosure resolves granular-vs-simple tension — defaults on the surface, dials on demand
• Custom async engine (hand-rolled asyncio state machine) over LangGraph — full control over pause/resume, durable state, trace streaming; framework lock-in avoided
• NotebookLM-style source grounding kept; source-blindness removed via quality annotation
• Model adapter default: Ollama cloud-first (ollama.com/v1) with keyless localhost:11434/v1 fallback on auth/connection failure
• Prototype-first gate (owner): UI/UX prototype with mock data, approved before MVP work; prototype is its own milestone (v0.1)
• Versioning (owner): starts at v0.1; MAJOR locked at 0 until owner-approved v1 release; MVP may span multiple 0.x minor milestones
• Prototype = production Next.js frontend with typed ApiClient/MockApiClient split — zero throwaway work; phase 3 of v0.2 swaps the mock for the OpenAPI client
• Tag-line mapping: 0.x milestones run patches on their own minor line (v0.1.x); supersedes the greenfield deviation noted at init

Key Decisions (v0.2 clarify — D-007..D-020, 2026-09-27)

• Backend layout: backend/ repo-root package, src-layout researchx (config, methodologies, adapters, engine, evidence, trust, api per ARCHITECTURE §Components); uv + uv-managed Python 3.12 (D-007/D-008)
• Ports/binding: backend 127.0.0.1:8000, frontend :3000 via NEXT_PUBLIC_API_BASE_URL; CORS localhost:3000 (D-009); run IDs = 8-char slugs; storage = /runs// (D-010)
• Adapters: Ollama base URLs + key via env-file secrets scopes (OLLAMA_BASE_URL, OLLAMA_API_KEY); DDG keyless fallback = html.duckduckgo.com/html conservative scrape (D-011/D-012)
• Integration: openapi-typescript + openapi-fetch → frontend/src/lib/api/generated, bound behind ApiClient; RESEARCHX_API_MODE=mock|live, mock default (D-013/D-020)
• Transport/trust: sse-starlette SSE + pydantic typed event bus (D-014); difflib quote anchoring ≥ 0.85 (D-015); HTTP liveness live/stale/dead (D-016); JSON durable state, atomic rewrite per transition (D-017)
• Run termination: pipeline ships P1..P4 + P5 final patch, then HALTS at the owner approval gate before milestone → main (merge_to_main hook, v0.1 precedent); no deploys without owner directive (D-018/D-019)