Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 coreci-bot
|
75479a9bc0 | ||
 coreci
|
65f9140ed8 | ||
|
coreci-bot
|
558f21a851 | ||
|
coreci
|
968eeb9102 | ||
|
coreci-bot
|
dfb738d54e | ||
|
coreci
|
056c286b57 | ||
|
coreci-bot
|
bbf80ce318 |
@@ -1,8 +1,8 @@
|
||||
{
|
||||
"phase": 5,
|
||||
"stage": "complete",
|
||||
"phase": 3,
|
||||
"stage": "execute",
|
||||
"milestone": "v0.2",
|
||||
"phase_role": "final",
|
||||
"phase_role": "execution",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-09-27T16:30:00Z"
|
||||
"updated_at": "2026-09-27T11:50:00Z"
|
||||
}
|
||||
|
||||
+1
-2
@@ -215,5 +215,4 @@ The plan is sound with fixable gaps. The evidence base is unusually strong for a
|
||||
| G-010 | Meta | Pre-agreed contingency lever: if P2/P4 slip beyond one extension cycle, Deep Report (depth 3) defers to v0.3 and the MVP ships Rapid Answer end-to-end; pulling the lever requires owner sign-off at a gate. | 0.65 |
|
||||
|
||||
### Escalations
|
||||
- [G-009] Axis 2 — Owner ratification required at the P5 gate (D-018 halt): the v0.2 MVP demonstrates the **live core research loop only**; the remaining approved-prototype surfaces (Library, Settings persistence, Suggestions, Refine, graph/inbox previews, profile) remain mock-bound until v0.3+, and the live-mode gate demo requires a local backend + keys (or runs in mock mode) since D-019 forbids deploys. This redefines what "MVP done" looks like at the demo and must not be discovered by the owner at the gate — it must be ratified in advance. Engineering-side handling is bound in G-002/G-003; the ratification itself is unresolved until the owner records it.
|
||||
- **RESOLVED 2026-09-27: RATIFIED.** Owner approval recorded via /ci-run 'APPROVED' at the v0.2.5 gate (decision commit 4721c2a), following the owner-directed UAT staging at https://research.coreci.dev. Live-mode scope = core research loop only is the accepted MVP boundary; mock-bound surfaces move to v0.3+.
|
||||
- [G-009] Axis 2 — Owner ratification required at the P5 gate (D-018 halt): the v0.2 MVP demonstrates the **live core research loop only**; the remaining approved-prototype surfaces (Library, Settings persistence, Suggestions, Refine, graph/inbox previews, profile) remain mock-bound until v0.3+, and the live-mode gate demo requires a local backend + keys (or runs in mock mode) since D-019 forbids deploys. This redefines what "MVP done" looks like at the demo and must not be discovered by the owner at the gate — it must be ratified in advance. Engineering-side handling is bound in G-002/G-003; the ratification itself is unresolved until the owner records it.
|
||||
@@ -40,7 +40,7 @@ reason: "v0.2 is the MVP: FastAPI backend + engine + adapters + trust loop (back
|
||||
|
||||
### security-engineer (phase-specific: P2)
|
||||
- **Domain**: security
|
||||
- **Active**: false (P2 shipped — removed per phase-specific policy)
|
||||
- **Active**: true (P2 only)
|
||||
- **Phase-specific**: true
|
||||
- **Frameworks**: httpx, pytest, STRIDE analysis
|
||||
- **Constraints**: secrets resolve from env-file scopes only (OLLAMA_API_KEY, TAVILY_API_KEY, GITEA_TOKEN via .ciagent/.env.secrets) — never shell env, never logged, never in git; liveness verification and trust-loop code must not leak keys in errors or traces; SSRF guard for fetched URLs (private-address refusal); authority tiering must not phone home
|
||||
|
||||
+50
-26
@@ -1,44 +1,68 @@
|
||||
# PLAN — Milestone v0.2 Phase 5 (FINAL): Review + Audit + Ship
|
||||
# PLAN — Milestone v0.2 Phase 3: API Layer + Integration
|
||||
|
||||
---
|
||||
phase: 5
|
||||
phase: 3
|
||||
milestone: v0.2
|
||||
status: executed
|
||||
status: planned
|
||||
must_haves:
|
||||
- "Multi-persona full-milestone review: 4 P0 + 15 P1 + 15 P2 found; all 4 P0s fixed in-phase with attack-proofs (A1 tautological anchoring → real writer-quote extraction; C2 gate denominator → cited-URL scoped; C1 zombie runs → shape-validated plans + catch-all FAILED persist; S1 fetch SSRF → per-hop re-guarding); 6 cheap P1s fixed (approve-with-plan, FAILED status, length caps, probe wiring, README env-var + adapter honesty)"
|
||||
- "Audit: PASS-WITH-NOTES — reconstruction PASS (all REQ-01..34 covered across v0.2.0–v0.2.4), secrets clean (zero key values in history), branch hygiene exact; critical notes fixed in-phase (REQUIREMENTS traceability un-staled, P5 PLAN written, ROADMAP P5 status, security-engineer persona deactivated per its removal note)"
|
||||
- "Milestone ship: v0.2.5 tagged as the MVP release candidate — the pipeline then HALTS at the merge_to_main owner approval gate (D-018)"
|
||||
- "G-009 escalation presented at the gate: the owner ratifies that live mode covers the core research loop only (G-002); remaining prototype surfaces stay mock-backed to v0.3+"
|
||||
- "FastAPI app with routes: POST /api/runs (create), GET /api/methodologies (cards with ETA + cost estimate + why-recommended), GET /api/runs/{id}, POST approve/pause/cancel/resume, GET /api/runs/{id}/events (SSE), GET /api/config (adapter status + probe results + cost counters per G-007)"
|
||||
- "In-memory run registry (single-user, single uvicorn worker) mapping run_id → RunActor; runs execute as asyncio tasks"
|
||||
- "SSE transport: sse-starlette EventSourceResponse streaming the typed event bus (state replay on connect); client disconnect cancels only the subscription, never the run"
|
||||
- "openapi.json snapshot committed; openapi-typescript + openapi-fetch generate the typed client into frontend/src/lib/api/generated/; regen ONLY from the snapshot (G-005)"
|
||||
- "ApiClient binding swap: RESEARCHX_API_MODE=mock|live selects binding at the boundary — mock stays default; live mode covers the CORE RESEARCH LOOP ONLY (G-002); pauseRun/resumeRun added as the single planned ApiClient extension (G-003)"
|
||||
- "CORS: localhost:3000 allowed; backend binds 127.0.0.1"
|
||||
- "ETA/cost heuristics for methodology cards (depth × breadth based)"
|
||||
- "Backend route tests via httpx ASGI transport — zero API keys, fakes injected"
|
||||
personas:
|
||||
- backend-engineer
|
||||
- frontend-engineer
|
||||
- ci-lead-developer
|
||||
requirements: [REQ-01, REQ-02, REQ-03, REQ-04, REQ-05, REQ-06, REQ-07, REQ-08, REQ-09, REQ-10, REQ-11, REQ-12, REQ-13, REQ-14, REQ-15, REQ-16, REQ-17, REQ-18, REQ-19, REQ-20, REQ-21, REQ-22, REQ-23, REQ-24, REQ-25, REQ-26, REQ-31, REQ-32, REQ-33, REQ-34]
|
||||
requirements: [REQ-25, REQ-26, REQ-33]
|
||||
---
|
||||
|
||||
## User-Facing Surface
|
||||
|
||||
The MVP release candidate: the approved v0.1 prototype frontend (Ask → Choose → Plan & Run → Reading Room, all 8 cards, shell, library, settings) now backed by a real FastAPI research engine in live mode — methodology registry (Rapid Answer + Deep Report), 12-dial framework, asyncio state machine with durable pause/cancel/resume, visited-only evidence store, attack-proven trust loop (quote anchoring, SSRF-guarded liveness, citation ledger, ≥99% export gate with zero-unanchored requirement), SSE live trace, adversarial suite (14 attacks), E2E happy path on the real stack. Releases v0.2.0–v0.2.5.
|
||||
The MVP goes live: with the backend running (`uv run uvicorn researchx.api.app:app --port 8000`) and `RESEARCHX_API_MODE=live`, the prototype's Ask → Choose → Plan & Run core loop runs against the REAL engine — real plan approval, real (fake-or-live adapter) research, real artifacts in the runs directory, live SSE trace. Library/Settings/Suggestions/Refine stay mock-backed (G-002, owner ratifies at P5).
|
||||
|
||||
## Happy Path (owner review script)
|
||||
## Happy Path (P3 — written before execute)
|
||||
|
||||
1. `cd backend && uv sync && uv run pytest` — 129 tests green, zero API keys (live tier: `uv run pytest -m live`, skips cleanly)
|
||||
2. `cd frontend && corepack pnpm install && corepack pnpm test` — 97 tests green (90 component + 5 live-binding + 2 real-stack E2E)
|
||||
3. `uv run uvicorn researchx.api.app:app --port 8000` + `NEXT_PUBLIC_RESEARCHX_API_MODE=live corepack pnpm dev` → http://localhost:3000 → Ask a question → Research this → pick Deep Report → Start research → approve the plan (3 clicks) → live SSE trace → report with anchored citations → ledger → export Markdown (citation links) / JSON (backend-schema round-trip)
|
||||
4. `uv run python -m researchx.engine.demo` — full pipeline on fakes, artifacts in a temp runs dir
|
||||
5. Owner decision: approve → milestone merge to main + v0.2.5 published as the MVP release; or direct changes → binding fix cycle
|
||||
1. `cd backend && uv run uvicorn researchx.api.app:app --port 8000` → health at `/api/config` (adapter status JSON)
|
||||
2. `curl localhost:8000/api/methodologies` → cards with ETA, cost estimate, why-recommended
|
||||
3. `curl -X POST localhost:8000/api/runs -d '{"question":"...","methodology":"rapid-answer"}'` → run id, state PLANNING→AWAITING_APPROVAL
|
||||
4. `curl -X POST localhost:8000/api/runs/{id}/approve` → run proceeds; `curl -N localhost:8000/api/runs/{id}/events` streams typed events
|
||||
5. `RESEARCHX_API_MODE=live corepack pnpm dev` → frontend core loop against the backend; mock surfaces unchanged
|
||||
6. `corepack pnpm test` — ApiClient boundary tests green in both modes; `uv run pytest` — route tests green
|
||||
|
||||
## UX Acceptance Criteria
|
||||
## UX Acceptance Criteria (P3)
|
||||
|
||||
- ≤3 clicks from cold start to initiated research, validated on the real stack (REQ-17)
|
||||
- Every trust claim attack-proven: hallucinated citations fail the export gate; fabricated quotes on real passage IDs flag unanchored; SSRF redirects die; pause/resume/cancel survive the loop
|
||||
- Honest scope: live mode = core research loop (G-002); library/settings/suggestions/refine stay mock-backed until v0.3+ — ratified at this gate (G-009)
|
||||
- No secret value in any artifact, trace, or log; zero-key test discipline throughout
|
||||
- Zero component rewrites: only the ApiClient boundary file changes (REQ-54/REQ-26 promise kept)
|
||||
- Live mode is opt-in and obvious; mock remains the default — nobody accidentally spends tokens
|
||||
- Methodology cards show honest ETA/cost estimates (heuristics labeled as estimates)
|
||||
- SSE stream survives page reloads (state replay on connect); closing the tab never kills a run
|
||||
- 3-click path (Ask → Choose → approve plan) unchanged
|
||||
|
||||
## What this phase did
|
||||
## Tasks (waves)
|
||||
|
||||
1. **Review** (multi-persona, probe-verified): 4 P0 + 15 P1 + 15 P2. The 4 P0s broke the milestone's flagship guarantees and were all fixed with attack-proofs: quote anchoring extracted the stored passage's own quote (always 1.0) instead of the writer's text; the export-gate denominator counted unprobed visited pages as dead; malformed planner JSON left zombie runs; fetch adapters followed redirects without re-guarding. Six cheap P1s also fixed.
|
||||
2. **Audit**: PASS-WITH-NOTES — reconstruction, secrets sweep (clean), branch/tag hygiene exact; both critical notes (stale traceability rows, missing P5 plan) fixed in-phase.
|
||||
3. **Ship (this commit sequence)**: PR #14 (phase/05 → milestone), squash-merge, tag v0.2.5 — the MVP release candidate, release with the full milestone summary. Per D-018 and the merge_to_main escalation hook, the milestone merge to main does NOT proceed autonomously — the pipeline halts for the recorded owner decision, with G-009 (live-mode scope) presented at the same gate.
|
||||
### Wave 1 — API layer (backend-engineer)
|
||||
- **T-3.1** FastAPI app + in-memory run registry + CORS: `researchx/api/app.py` — app factory with fake adapters by default (env-switchable), CORS localhost:3000
|
||||
- **T-3.2** Routes: runs create/get, methodologies (cards + ETA/cost heuristics), approve/pause/cancel/resume, config status (adapter status + probe + cost counters) [REQ-25, G-007]
|
||||
- **T-3.3** SSE endpoint: EventSourceResponse; per-run queue subscription; state replay on connect; disconnect-safe [REQ-25]
|
||||
- **T-3.4** Route tests: httpx ASGI transport; full lifecycle walk over HTTP; SSE event stream assertion; zero keys
|
||||
|
||||
## Scope Guard
|
||||
### Wave 2 — Client swap (frontend-engineer)
|
||||
- **T-3.5** openapi.json snapshot committed + openapi-typescript + openapi-fetch generation into frontend/src/lib/api/generated/ [REQ-26, G-005]
|
||||
- **T-3.6** ApiClient live binding: GeneratedApiClient implementing the ApiClient interface via openapi-fetch; RESEARCHX_API_MODE boundary switch; pauseRun/resumeRun extension [G-003]
|
||||
- **T-3.7** Frontend tests: boundary binding both modes; live-binding contract tests against mocked fetch; component suite stays green (no component changes)
|
||||
|
||||
Milestone merge to main, v0.2.5 publication, and any deploy wait on the owner approval gate (D-018/G-009). The pipeline does not proceed autonomously past this commit.
|
||||
### Wave 3 — Integration proofs (ci-lead-developer)
|
||||
- **T-3.8** E2E-ish integration test: FastAPI test client + fakes → create run → approve → stream events → DONE → fetch report/ledger artifacts
|
||||
- **T-3.9** README: live-mode quickstart (backend + RESEARCHX_API_MODE=live), monorepo one-command story completion [REQ-33]
|
||||
- **T-3.10** Gates + REQ mapping check: REQ-25/26/33 coverage table in verify commit
|
||||
|
||||
## Out of scope for P3
|
||||
- Adversarial suite expansion + real-stack 3-click validation (P4, REQ-17/32/34); UI for library/settings live data (v0.3+, G-002)
|
||||
|
||||
## Verification hooks
|
||||
- Gates: backend ruff/mypy/pytest (route tests); frontend typecheck/lint/test/build
|
||||
- REQ coverage: REQ-25 (routes + SSE + config), REQ-26 (generated client + swap), REQ-33 (one-command story)
|
||||
- Security: CORS limited; no secrets in snapshots; run-id validation at the API boundary
|
||||
@@ -130,8 +130,8 @@
|
||||
| REQ-63, REQ-64, REQ-65, REQ-66 (app shell, library, settings, avatar) | v0.1 | P5 | complete |
|
||||
| REQ-67, REQ-68 (ask suggestions, refine dialogue) | v0.1 | P7 | complete |
|
||||
| REQ-01..06, REQ-08..11, REQ-31 (framework, adapters) | v0.2 | P1 | complete (v0.2.1; REQ-07/12 contracts proven, live probing in P2) |
|
||||
| REQ-07, REQ-12, REQ-13..24 (probing, engine, trust) | v0.2 | P2 | complete (v0.2.2) |
|
||||
| REQ-25, REQ-26, REQ-33 (API, client, tooling) | v0.2 | P3 | complete (v0.2.3) |
|
||||
| REQ-17, REQ-32, REQ-34 (adversarial, E2E, round-trip) | v0.2 | P4 | complete (v0.2.4) |
|
||||
| All REQ-01..34 (milestone gate) | v0.2 | P5 | complete (v0.2.5 — **owner approval recorded 2026-09-27**, G-009 ratified; milestone merged to main) |
|
||||
| REQ-07, REQ-12, REQ-13..24 (probing, engine, trust) | v0.2 | P2 | blocked |
|
||||
| REQ-25, REQ-26, REQ-33 (API, client, tooling) | v0.2 | P3 | blocked |
|
||||
| REQ-17, REQ-32, REQ-34 (adversarial, E2E, round-trip) | v0.2 | P4 | blocked |
|
||||
| All REQ-01..34 (milestone gate) | v0.2 | P5 | blocked |
|
||||
| REQ-35..51 | v2 backlog (post-MVP milestones) | — | pending |
|
||||
|
||||
+8
-25
@@ -75,7 +75,7 @@ prototype release is **v0.1.8** (final phase patch). Milestone branch
|
||||
merged to main and deleted — tags v0.1.0–v0.1.8 preserve the full
|
||||
history. Next milestone: v0.2 MVP (unblocked).
|
||||
|
||||
## Milestone v0.2 — MVP (feature) — COMPLETE (approved 2026-09-27)
|
||||
## Milestone v0.2 — MVP (feature) — UNBLOCKED (prototype approved 2026-09-27)
|
||||
|
||||
Phases renumbered from the original v1.0 plan (1-based within the milestone). Tags run on the v0.2.x line; MAJOR stays 0.
|
||||
|
||||
@@ -98,36 +98,19 @@ Phases renumbered from the original v1.0 plan (1-based within the milestone). Ta
|
||||
- **Requirements covered**: REQ-07, REQ-12, REQ-13..24
|
||||
- **Status**: complete (PR #11 squash-merged 558f21a, tag v0.2.2, release HTTP 201, phase branch deleted; verify found 2 P0 + 9 P1, all fixed in-phase with regression pins; SSE transport itself lands in P3 — the bus + trace artifacts shipped here)
|
||||
|
||||
### Phase 3 — API Layer + Integration (complete — shipped v0.2.3)
|
||||
### Phase 3 — API Layer + Integration
|
||||
- **Description**: FastAPI routes (runs, methodologies, lifecycle, SSE events, config status); OpenAPI-generated typed client; **swap MockApiClient → generated client** (the ApiClient interface absorbs any prototype-era API drift); ETA/cost heuristics for methodology cards.
|
||||
- **Requirements covered**: REQ-25, REQ-26
|
||||
- **Status**: complete (PR #12 squash-merged 75479a9, tag v0.2.3, release HTTP 201, phase branch deleted; verify found 1 P0 mode-flag + 4 P1, all fixed in-phase; REQ-33 one-command story completed here per plan)
|
||||
- **Status**: blocked → next (sequential phases; unblock as each prior phase ships)
|
||||
|
||||
### Phase 4 — Hardening + Adversarial Suite + E2E (complete — shipped v0.2.4)
|
||||
- **Description**: Adversarial test suite (hallucinated-URL writer, quote-anchor mismatch, pause/resume mid-depth-loop), E2E happy-path (real stack via live binding + dev bridge), monorepo tooling polish (one-command install/dev, lint/typecheck/test green).
|
||||
### Phase 4 — Hardening + Adversarial Suite + E2E
|
||||
- **Description**: Adversarial test suite (hallucinated-URL writer, quote-anchor mismatch, pause/resume mid-depth-loop), E2E happy-path (Playwright), monorepo tooling polish (one-command install/dev, lint/typecheck/test green).
|
||||
- **Requirements covered**: REQ-17, REQ-32, REQ-33, REQ-34
|
||||
- **Status**: complete (PR #13 squash-merged dcf2475, tag v0.2.4, release HTTP 201, phase branch deleted; verify found 5 P1 honesty gaps in the suite itself — export gate anchored, weak band pinned, fabricated reports fail export; E2E validated on the real stack, defaults only)
|
||||
- **Status**: blocked → next (sequential phases; unblock as each prior phase ships)
|
||||
|
||||
### Phase 5 — Final Review + Ship (FINAL)
|
||||
- **Description**: Review + audit + milestone ship; final patch tag IS the MVP release. MAJOR stays 0 — the MVP ships as a 0.x release. v1.0.0 comes only after an owner-approved release decision.
|
||||
- **Status**: complete (PR #14 squash-merged 023667e, tag v0.2.5, release HTTP 201, phase branch deleted; review 4 P0 + 15 P1 — P0s attack-proven fixed; audit PASS-WITH-NOTES — criticals fixed; **owner approval RECORDED 2026-09-27** — decision commit 4721c2a, G-009 scope ratified; milestone merged to main, v0.2.5 = the MVP release)
|
||||
|
||||
## Milestone v0.2 — COMPLETE (approved 2026-09-27)
|
||||
|
||||
All requirements complete: REQ-01..26, REQ-31..34 (the MVP set). The
|
||||
approved MVP release is **v0.2.5** (final phase patch). Delivered: the
|
||||
FastAPI research engine (asyncio state machine, durable
|
||||
pause/cancel/resume), methodology framework (12 dials, Rapid Answer +
|
||||
Deep Report), five-adapter layer (Ollama cloud-first fallback,
|
||||
Tavily/DDG, storage, memory, secrets), the attack-proven trust loop
|
||||
(real quote anchoring, cited-scoped ≥99% liveness export gate, SSRF-
|
||||
guarded fetching, citation ledger, verification checklist), SSE live
|
||||
trace, OpenAPI-generated typed client behind the unchanged ApiClient
|
||||
boundary, a 14-attack adversarial suite, and the 3-click E2E on the
|
||||
real stack. G-009 ratified: live mode = core research loop only;
|
||||
library/settings/suggestions/refine/graph stay mock-backed until
|
||||
v0.3+. Milestone branch merged to main and deleted — tags
|
||||
v0.2.0–v0.2.5 preserve the full history. Next: v0.3 planning.
|
||||
- **Status**: blocked → next (sequential phases; unblock as each prior phase ships)
|
||||
|
||||
## Milestone Releases
|
||||
- **v0.1.4 = extended-prototype candidate** (superseded by the P5/P6 extension cycle)
|
||||
@@ -135,7 +118,7 @@ v0.2.0–v0.2.5 preserve the full history. Next: v0.3 planning.
|
||||
- **v0.1.6 = extended-prototype candidate** (superseded by the P7/P8 extension cycle)
|
||||
- **v0.1.7 = ask suggestions + refine dialogue** (P7 patch)
|
||||
- **v0.1.8 = milestone release** (P8 final phase patch — the approved extended prototype release, pending owner decision)
|
||||
- **v0.2.5 = milestone release** (P5 final phase patch — the approved MVP; owner approval recorded 2026-09-27, G-009 ratified)
|
||||
- **v0.2.x = MVP release** (MAJOR still 0; ship only if all v0.2 phases complete)
|
||||
- If MVP scope extends: additional 0.x minor milestones (v0.3, …), each a full milestone cycle
|
||||
- **v1.0.0**: owner-approved release only — not created by the pipeline autonomously
|
||||
|
||||
|
||||
+10
-22
@@ -2,23 +2,18 @@
|
||||
"ci_version": "0.7.0",
|
||||
"projects": [],
|
||||
"active_project": "researchx",
|
||||
"milestone": "v0.3",
|
||||
"milestone_name": "next-milestone-planning",
|
||||
"milestone": "v0.2",
|
||||
"milestone_name": "mvp",
|
||||
"milestone_type": "feature",
|
||||
"versioning": {
|
||||
"major_locked": true,
|
||||
"major_lock_note": "MAJOR remains 0 until an owner-approved v1 release; all pre-v1 ships are MINOR/PATCH only",
|
||||
"tag_line": "v0.3.x",
|
||||
"next_milestone": "v0.3 scope definition pending owner direction (post-MVP backlog: live-bound library/settings/suggestions/refine, backend hosting, remaining 6 methodologies)"
|
||||
"major_lock_note": "MAJOR remains 0 until an owner-approved v1 release; the MVP may span multiple 0.x minor milestones; all pre-v1 ships are MINOR/PATCH only",
|
||||
"tag_line": "v0.2.x",
|
||||
"next_milestone": "v0.3+ (post-MVP backlog milestones)"
|
||||
},
|
||||
"autonomy": {
|
||||
"level": "full",
|
||||
"escalation_hooks": [
|
||||
"deploy",
|
||||
"delete_data",
|
||||
"merge_to_main",
|
||||
"prototype_approval"
|
||||
],
|
||||
"escalation_hooks": ["deploy", "delete_data", "merge_to_main", "prototype_approval"],
|
||||
"decision_confidence_threshold": 0.6,
|
||||
"max_revision_iterations": 3,
|
||||
"clarify_budget": 10
|
||||
@@ -42,16 +37,9 @@
|
||||
"secrets": {
|
||||
"env_file": ".ciagent/.env.secrets",
|
||||
"scopes": {
|
||||
"release": [
|
||||
"GITEA_TOKEN"
|
||||
],
|
||||
"model": [
|
||||
"OLLAMA_API_KEY",
|
||||
"OLLAMA_BASE_URL"
|
||||
],
|
||||
"search": [
|
||||
"TAVILY_API_KEY"
|
||||
]
|
||||
"release": ["GITEA_TOKEN"],
|
||||
"model": ["OLLAMA_API_KEY", "OLLAMA_BASE_URL"],
|
||||
"search": ["TAVILY_API_KEY"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,8 +1,8 @@
|
||||
# Deploy Record — UAT stagings and releases
|
||||
# Deploy Record — UAT stagings of the v0.1 prototype
|
||||
|
||||
Owner-directed deployments, recorded per commit discipline. Release
|
||||
deploys carry the owner approval (v0.1.8); UAT variants stage review
|
||||
builds without minting release tags.
|
||||
Owner-directed UAT deployments, recorded per commit discipline. The
|
||||
approval gate (REQ-62) stays closed; UAT variants stage review builds
|
||||
without minting release tags.
|
||||
|
||||
| Date | Version | Commit | Directive | Result |
|
||||
|---|---|---|---|---|
|
||||
@@ -11,8 +11,6 @@ builds without minting release tags.
|
||||
| 2026-09-27 | v0.1.7-uat | 432d2c0 (P7 ship + boundary docs) | owner cadence: deploy after P7 and P8 | healthy — http://10.0.0.114:3000 |
|
||||
| 2026-09-27 | v0.1.8-uat | 93a7b57 (P8 ship — release candidate) | owner cadence: deploy after P7 and P8 | healthy — http://10.0.0.114:3000 |
|
||||
| 2026-09-27 | **v0.1.8 (release)** | 93a7b57 (tagged) | **owner approval recorded — milestone release published** | healthy — http://10.0.0.114:3000 |
|
||||
| 2026-09-27 | v0.2.5-uat | c28a97e (v0.2 milestone HEAD + v0.2.5 version strings) | owner: "deploy and validate against https://research.coreci.dev" | healthy — https://research.coreci.dev (HAProxy/SSL) |
|
||||
| 2026-09-27 | **v0.2.5 (release)** | b11b202 (main merge; tag v0.2.5) | **owner approval recorded (/ci-run APPROVED) — milestone release published; G-009 ratified** | serving — same container (v0.2.5-uat build = release content); no redeploy needed |
|
||||
|
||||
## v0.1.6-uat (extended prototype for owner review)
|
||||
|
||||
@@ -26,25 +24,4 @@ builds without minting release tags.
|
||||
- Rollback: `ssh root@10.0.0.114 'docker stop researchx; docker rename
|
||||
researchx-prev researchx; docker start researchx'`
|
||||
- The approval gate remains closed: milestone merge to main still halts
|
||||
for the recorded owner decision (REQ-62)
|
||||
|
||||
## v0.2.5-uat (MVP release candidate, owner-directed)
|
||||
|
||||
- Deployed via `bash .ciagent/release/deploy.sh v0.2.5-uat` — the v0.2
|
||||
milestone HEAD (c28a97e: tag v0.2.5 + v0.2.5 About/package.json
|
||||
version strings), per the owner's direct request to deploy and
|
||||
validate against the public URL
|
||||
- Gates re-run at deploy time from the milestone HEAD: typecheck,
|
||||
lint, test (97/97), build — green before sync
|
||||
- Image `researchx:v0.2.5-uat` + `latest` on the host; container
|
||||
swapped, `researchx-prev` (v0.1.8) kept for rollback
|
||||
- Health: 2 checks; validated through the HAProxy SSL front:
|
||||
https://research.coreci.dev — /ask /library /settings /choose all
|
||||
200, root 307→/ask, TLS verified; served settings chunk carries
|
||||
"v0.2.5 · MVP release candidate"; the RESEARCHX_API_MODE switch is
|
||||
present in the served bundle (mock default — correct for a
|
||||
frontend-only container; live mode additionally needs the backend
|
||||
API on the host, which is v0.3+ deployment work)
|
||||
- The merge gate (D-018) remains closed: milestone → main still halts
|
||||
for the recorded owner decision, with G-009 (live-mode scope)
|
||||
presented at the same gate
|
||||
for the recorded owner decision (REQ-62)
|
||||
@@ -1,87 +0,0 @@
|
||||
# UAT Script — v0.2.5 (MVP release candidate)
|
||||
|
||||
Deployment: https://research.coreci.dev (HAProxy → LXC container
|
||||
`researchx:v0.2.5-uat`, commit c28a97e, mock-backed frontend).
|
||||
|
||||
Scope note (G-009): this container serves the v0.2.5 frontend with
|
||||
scripted mock data. The v0.2 engine (FastAPI, adapters, trust loop)
|
||||
is validated by the automated suites (129 backend + 97 frontend
|
||||
tests, 14 adversarial attacks, real-stack E2E) and runs via the
|
||||
README live-mode locally; hosting the backend is v0.3+ work. This
|
||||
UAT validates the release-candidate build of the product surface.
|
||||
|
||||
## 1. Shell & navigation (REQ-63, REQ-66)
|
||||
|
||||
1. Open https://research.coreci.dev → lands on **/ask** inside the app shell
|
||||
2. Sidebar: **New research / Library / Settings** nav present on every screen
|
||||
3. Top bar: press **⌘K** → navigation palette (screens + stored reports)
|
||||
4. Click the **avatar** (top right) → user menu: local profile, Settings, About
|
||||
5. Pass: shell persistent everywhere, honest preview-marked future slots
|
||||
|
||||
## 2. Version check (this deploy)
|
||||
|
||||
1. Sidebar → **Settings** → scroll to *About & version*
|
||||
2. Pass: reads **"v0.2.5 · MVP release candidate"** (confirms the deployed build)
|
||||
|
||||
## 3. The ≤3-click happy path (REQ-61 / REQ-17 — core criterion)
|
||||
|
||||
1. **/ask**: type a question → click **Research this** (click 1)
|
||||
2. **/choose**: pick a card (Deep Report is the recommended default) → **Start research** (click 2)
|
||||
3. **Plan & Run**: review the plan → **Approve plan** (click 3)
|
||||
4. Pass: exactly 3 clicks, defaults only, research running with a live trace
|
||||
|
||||
## 4. Plan & Run (REQ-57)
|
||||
|
||||
1. Plan approval UI shows sub-questions, source universe, outline
|
||||
2. Watch the live trace stream (scripted replay)
|
||||
3. Press **pause** → trace halts; **resume** → continues; **cancel** → terminal state
|
||||
4. Pass: all three controls behave; no lost events
|
||||
|
||||
## 5. Reading Room + exports (REQ-58, REQ-59, REQ-34)
|
||||
|
||||
1. Open the finished report from the run screen
|
||||
2. Hover/focus any **[n]** citation marker → the exact quoted passage appears
|
||||
3. Open the **citation ledger** panel: URL, quote, health, authority tier
|
||||
4. **Export Markdown** and **Export JSON** → real files download; citations survive as links
|
||||
5. Pass: quotes match exactly; files open; links intact
|
||||
|
||||
## 6. Ask intelligence (REQ-67, REQ-68)
|
||||
|
||||
1. **New research** → below the form: open-question suggestions appear
|
||||
2. Click one → the question box fills; **never auto-starts**
|
||||
3. Type a draft → **Refine question** → answer the 3 clarifying questions → **Use this** → refined question + context fill the form
|
||||
4. Try **Keep original** on a second draft → untouched; keyboard (Enter) works through both flows
|
||||
5. Pass: both helpers are opt-in, keyboard-accessible, drafts preserved on dismiss
|
||||
|
||||
## 7. Library (REQ-64)
|
||||
|
||||
1. Sidebar → **Library**: folder tree, auto-categorized (topic + methodology), sortable (recency/title)
|
||||
2. Open a stored report → its own Reading Room view
|
||||
3. **Connections** panel: shared-source ⇄ and topical → edges
|
||||
4. v2 previews: project graph + task inbox panels render
|
||||
5. Pass: folders, sorting, connections, previews all render from the fixture data
|
||||
|
||||
## 8. Settings (REQ-65)
|
||||
|
||||
1. API keys: masked (never raw); storage directory field
|
||||
2. Methodology defaults: resolved 12-dial display
|
||||
3. Appearance preview; About & version (step 2)
|
||||
4. Pass: everything honest and functional on mock data
|
||||
|
||||
## Optional: hands-on the live engine (not on this URL)
|
||||
|
||||
```bash
|
||||
# locally, two terminals:
|
||||
cd backend && uv sync && uv run uvicorn researchx.api.app:app --port 8000
|
||||
cd frontend && NEXT_PUBLIC_RESEARCHX_API_MODE=live corepack pnpm dev
|
||||
# then repeat steps 3-5 against the real engine (fake adapters, zero keys)
|
||||
```
|
||||
|
||||
## Decision rubric (the gate this UAT feeds)
|
||||
|
||||
- **Approve**: recorded via /ci-run → milestone merges to main,
|
||||
v0.2.5 publishes as the MVP release, v0.3 planning unblocks
|
||||
(with G-009 scope ratified: library/settings/suggestions/refine/
|
||||
graph stay mock-backed until v0.3+)
|
||||
- **Request changes**: feedback becomes binding decisions for a
|
||||
fix cycle (v0.1 precedent: two feedback cycles before approval)
|
||||
@@ -9,7 +9,7 @@ as established by the CIAgent branch strategy and `config.json`.
|
||||
| Milestone | Tags | Deployed by |
|
||||
|---|---|---|
|
||||
| v0.1 prototype | v0.1.0 … v0.1.8 | `bash .ciagent/release/deploy.sh v0.1.8` |
|
||||
| v0.2 MVP | v0.2.0 … v0.2.5 | `bash .ciagent/release/deploy.sh v0.2.5-uat` (staged) — see DEPLOY-RECORD.md |
|
||||
| v0.2 MVP | v0.2.0 … | `bash .ciagent/release/deploy.sh v0.2.x` |
|
||||
|
||||
## Rules
|
||||
|
||||
|
||||
@@ -1,157 +0,0 @@
|
||||
# .gitea/workflows/ci.yml - ResearchX phase-4 CI gate (researchx#24).
|
||||
#
|
||||
# This file IS the gate. The pre-code clause (coreci/company#105, c14057):
|
||||
# a code phase opens only when the product repo's gating workflow file has
|
||||
# dispatched at least one run on `main` and every job in that file is green
|
||||
# in that run. Until this file exists, coreci/researchx has no workflow and
|
||||
# no run at all - that is the case the clause was amended for (researchx#24,
|
||||
# c14093). A half-green pipeline does not open the phase either.
|
||||
#
|
||||
# Two jobs:
|
||||
# backend - uv + Python 3.12: ruff, mypy, pytest
|
||||
# frontend - Node 22 + pnpm 12.6.0: typecheck, lint, test, build
|
||||
#
|
||||
# Runner label: `ubuntu-latest` - the org CI fleet's published label whose
|
||||
# image is docker.gitea.com/runner-images:ubuntu-latest (label map read at
|
||||
# source: coreci/nextcraft deploy/ci/docker-compose.yaml + deploy/ci/README.md,
|
||||
# R-3). The previous string `employee-runner` was a fleet NAME, not a label:
|
||||
# the employee fleet's labels are `employee-py` and `employee-coder` (read
|
||||
# from a live job log of this seat's own home), and the CI fleet's are node /
|
||||
# ubuntu-latest / python / release - no runner carried the old label, so a
|
||||
# merged file asking for it could still buy no run.
|
||||
#
|
||||
# Toolchain: measured, not assumed. From the runner's own logs (job 56393,
|
||||
# coreci/nova-platform, 2026-10-26T23:17Z, and job 57005, coreci/inboxos-v1,
|
||||
# 2026-10-27T03:01Z): the image carries python 3.12.3, pip, git, node (the
|
||||
# actions run in it), pypi REACHABLE and github REACHABLE; actions/checkout@v4,
|
||||
# actions/setup-python@v5 and actions/setup-go@v5 all run inside it. It does
|
||||
# NOT carry uv or pnpm, so this file installs them: uv from pypi (pip), pnpm
|
||||
# 12.6.0 through corepack - the repo's own documented way (README command map).
|
||||
# The gates are the repo's documented commands: backend `uv run ruff check src
|
||||
# tests`, `uv run mypy src`, `uv run pytest`; frontend `pnpm typecheck`,
|
||||
# `pnpm lint`, `pnpm test`, `pnpm build`. The frontend lane mirrors the
|
||||
# release image's own recipe (frontend/Dockerfile: node:22-alpine, corepack,
|
||||
# pnpm install --frozen-lockfile, pnpm build), so the gate tests what the
|
||||
# product ships.
|
||||
#
|
||||
# Owner of a red leg (the clause asks for the owner of the fix next to the
|
||||
# owner of the product line): toolchain / image / dispatch problems ->
|
||||
# coreci-sre (employee-runner-fleet + gitea-instance); product-line problems
|
||||
# -> coreci-researchx-po. Approval for this file rides the merge request
|
||||
# (merge = approval, constitution v1.3).
|
||||
|
||||
name: ci
|
||||
|
||||
on:
|
||||
push:
|
||||
pull_request:
|
||||
branches: ["main"]
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
backend:
|
||||
name: "backend (uv, Python 3.12: ruff/mypy/pytest)"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Python 3.12
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
# uv is not in the CI image; pypi is reachable from it (measured). The
|
||||
# install is a toolchain leg, not a product gate: the gates still run
|
||||
# from the repo's own committed lockfile (backend/uv.lock).
|
||||
- name: Install uv (from pypi)
|
||||
run: |
|
||||
set -eu
|
||||
python -m pip install --upgrade uv
|
||||
uv --version
|
||||
|
||||
- name: Install dependencies (uv, from the committed lockfile)
|
||||
working-directory: backend
|
||||
run: uv sync --frozen
|
||||
|
||||
- name: Lint (ruff)
|
||||
working-directory: backend
|
||||
run: uv run ruff check src tests
|
||||
|
||||
- name: Typecheck (mypy)
|
||||
working-directory: backend
|
||||
run: uv run mypy src
|
||||
|
||||
- name: Test (pytest)
|
||||
working-directory: backend
|
||||
run: uv run pytest
|
||||
|
||||
frontend:
|
||||
name: "frontend (Next.js on pnpm: typecheck/lint/test/build)"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 25
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
# Node 22 to mirror the release image (frontend/Dockerfile: node:22).
|
||||
- name: Setup Node 22
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "22"
|
||||
|
||||
- name: Setup Python 3.12 (for the real-stack e2e bridge)
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Install uv (from pypi)
|
||||
run: |
|
||||
set -eu
|
||||
python -m pip install --upgrade uv
|
||||
|
||||
# pnpm 12.6.0 is pinned by frontend/package.json `packageManager`;
|
||||
# corepack reads it from there. If the image carries no corepack, npm
|
||||
# installs the same pinned version.
|
||||
- name: pnpm 12.6.0 (corepack, pinned by frontend/package.json)
|
||||
working-directory: frontend
|
||||
run: |
|
||||
set -eu
|
||||
node --version
|
||||
corepack enable pnpm 2>/dev/null || true
|
||||
if ! pnpm --version >/dev/null 2>&1; then
|
||||
npm i -g pnpm@12.6.0
|
||||
fi
|
||||
test "$(pnpm --version)" = "12.6.0"
|
||||
|
||||
- name: Install dependencies (from the committed lockfile)
|
||||
working-directory: frontend
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
# frontend/src/lib/api/e2e-real-stack.test.ts drives the real FastAPI
|
||||
# stack: it spawns `uv run python -m researchx.api.dev_bridge` from
|
||||
# ../backend (loopback :8000, fakes only, zero API keys). Without a
|
||||
# synced backend venv that test cannot come up, so the lane provisions
|
||||
# it. Provisioning only - the backend gates stay in the backend job.
|
||||
- name: Backend venv for the real-stack e2e bridge
|
||||
working-directory: backend
|
||||
run: uv sync --frozen
|
||||
|
||||
- name: Typecheck
|
||||
working-directory: frontend
|
||||
run: pnpm typecheck
|
||||
|
||||
- name: Lint
|
||||
working-directory: frontend
|
||||
run: pnpm lint
|
||||
|
||||
- name: Test
|
||||
working-directory: frontend
|
||||
run: pnpm test
|
||||
|
||||
- name: Build
|
||||
working-directory: frontend
|
||||
env:
|
||||
NEXT_TELEMETRY_DISABLED: "1"
|
||||
run: pnpm build
|
||||
@@ -1,15 +1,8 @@
|
||||
# ResearchX
|
||||
|
||||
**TL;DR** — ResearchX turns one question into a cited research report: pick a method, approve the plan, and the engine visits real web sources, with every claim anchored to a quoted passage from a page it actually read. The engine milestone (v0.2.5) is shipped and owner-approved; hosting goes live next (v0.3, in progress).
|
||||
|
||||
- **What** — FastAPI backend (planning, search, extraction, verification, export gates) plus a Next.js frontend; the full test suite runs with zero API keys.
|
||||
- **Why** — Readers can check every citation back to its source passage; runs are resumable from durable state.
|
||||
- **Action** — no action — FYI. Developers: see the Quickstart below.
|
||||
- **Details** — `frontend/README.md` (web app); `.ciagent/` (pipeline state and roadmap).
|
||||
|
||||
A deep-research studio where you bring your own model, storage, and memory; choose a research methodology before a single query fires; and every claim in the final report is anchored to a visited, verified, quotable source.
|
||||
|
||||
**Status: MVP shipped (v0.2.5, owner-approved 2026-09-27)** — the full research engine (FastAPI + asyncio state machine + adapter layer + attack-proven trust loop) behind the v0.1 frontend. Live mode covers the core research loop (G-002/G-009): the engine runs locally per the quickstart; library/settings/suggestions/refine stay mock-backed until v0.3+.
|
||||
**Status: MVP in development (milestone v0.2)** — the approved v0.1 prototype frontend (mock-backed) plus the FastAPI backend, research engine, adapter layer, and trust loop. The frontend runs standalone on mock data; live backend mode arrives with v0.2 phase 3.
|
||||
|
||||
## Quickstart
|
||||
|
||||
@@ -55,25 +48,6 @@ Run artifacts per run: `state.json`, `report.md`, `ledger.json`, `checklist.md`.
|
||||
|
||||
Try it on fakes (no keys): `uv run python -m researchx.engine.demo`
|
||||
|
||||
## Adversarial guarantees (v0.2 P4)
|
||||
|
||||
Every trust-loop claim is attacked by the adversarial suite (`backend/tests/test_adversarial.py`) — these are tests that try to fool the verifier, and their passing proves the defenses:
|
||||
|
||||
| Attack | Defense (proven) |
|
||||
|---|---|
|
||||
| Writer hallucinates a citation to a passage that was never stored | Unknown passage IDs count as **unanchored claims** in the ledger; the export gate fails the report |
|
||||
| Fabricated quote (matches nothing) | Anchoring verdict: no match, flagged — never silently accepted |
|
||||
| Near-miss quote (0.80–0.85 similarity) | Flagged `weak_match` in the ledger — surfaced, not accepted |
|
||||
| Liveness probe redirected to a private/metadata IP | Per-hop redirect re-guarding classifies it **dead** (SSRF refusal) |
|
||||
| Redirect chain ending at a gone page (410) | Classified **dead** |
|
||||
| Oversized page (5MB) response | 2MB read cap — classifies without unbounded memory |
|
||||
| Pause during the depth loop | Durable state persists; resume completes the run over the HTTP API |
|
||||
| Cancel mid-loop | Terminal CANCELLED state persisted, never swallowed |
|
||||
|
||||
REQ-23 (≥99% citation liveness) is a per-run **export gate** enforced by the verification checklist, plus an opt-in keyed live tier (`cd backend && uv run pytest -m live`) — the zero-key CI suite never asserts it vacuously.
|
||||
|
||||
The E2E happy path (≤3 clicks: ask, choose, approve) is validated against the real FastAPI + engine stack, defaults only — not the mock (`frontend/src/lib/api/e2e-real-stack.test.ts`).
|
||||
|
||||
## Adapter configuration (v0.2)
|
||||
|
||||
All keys resolve from env files only (`.env`, `.env.secrets`, `.env.*` — never the shell environment, never git):
|
||||
@@ -91,13 +65,11 @@ The test suite needs **no** keys: fake model/search adapters (REQ-31) drive ever
|
||||
The frontend runs mock-backed by default (works standalone, zero keys). To run the core research loop against the real backend:
|
||||
|
||||
```bash
|
||||
# terminal 1 — backend (fake adapters by default; real Ollama/Tavily adapters
|
||||
# are used by the engine in v0.2.5+ wiring — the served API currently ships
|
||||
# fake-adapter runs for demos and tests; see Adapter configuration below)
|
||||
# terminal 1 — backend (fake adapters by default; live adapters via env-file keys)
|
||||
cd backend && uv run uvicorn researchx.api.app:app --reload --port 8000
|
||||
|
||||
# terminal 2 — frontend in live mode
|
||||
cd frontend && NEXT_PUBLIC_RESEARCHX_API_MODE=live corepack pnpm dev
|
||||
cd frontend && RESEARCHX_API_MODE=live corepack pnpm dev
|
||||
```
|
||||
|
||||
Live mode covers the core loop (Ask → Choose → approve plan → live trace → report + ledger); the library, settings persistence, suggestions, refine, and graph surfaces stay mock-backed until v0.3. The API surface: `POST /api/runs`, `GET /api/methodologies`, run lifecycle (`approve/pause/cancel/resume`), `GET /api/runs/{id}/events` (SSE), `GET /api/config`, report + ledger artifacts.
|
||||
|
||||
@@ -41,8 +41,5 @@ python_version = "3.12"
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
pythonpath = ["src"]
|
||||
markers = [
|
||||
"live: opt-in keyed tests against real adapters (skipped without keys)",
|
||||
]
|
||||
asyncio_mode = "auto"
|
||||
testpaths = ["tests"]
|
||||
|
||||
@@ -3,6 +3,7 @@ import json
|
||||
import httpx
|
||||
|
||||
from researchx.adapters.model import ModelAdapter
|
||||
from researchx.adapters.ollama import OllamaModelAdapter
|
||||
from researchx.adapters.probe import (
|
||||
DEFAULT_EMBEDDING_BATCH,
|
||||
CapabilityProbe,
|
||||
@@ -12,25 +13,19 @@ from researchx.adapters.probe import (
|
||||
from researchx.adapters.types import ModelMessage
|
||||
|
||||
|
||||
def supports_probe(adapter: ModelAdapter) -> bool:
|
||||
return getattr(adapter, "probing_supported", False)
|
||||
|
||||
|
||||
async def probe_model_adapter(adapter: ModelAdapter) -> CapabilityProbe:
|
||||
async def probe_model_adapter(adapter: OllamaModelAdapter) -> CapabilityProbe:
|
||||
models = await adapter.list_models()
|
||||
smoke_ok = await _structured_output_smoke_test(adapter)
|
||||
writer = adapter.roles().writer
|
||||
warning = structured_output_warning(smoke_ok, writer)
|
||||
endpoint = getattr(adapter, "active_base_url", "unknown")
|
||||
fallback_reason = getattr(adapter, "last_fallback_reason", None)
|
||||
return CapabilityProbe(
|
||||
endpoint=str(endpoint),
|
||||
endpoint=adapter.active_base_url,
|
||||
models=models,
|
||||
context_window=cap_context_window(None),
|
||||
embedding_batch=DEFAULT_EMBEDDING_BATCH,
|
||||
structured_outputs_supported=smoke_ok,
|
||||
structured_output_warning=warning,
|
||||
fallback_reason=fallback_reason,
|
||||
fallback_reason=adapter.last_fallback_reason,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -13,7 +13,6 @@ from researchx.adapters.url_guard import url_allowed
|
||||
_DDG_URL = "https://html.duckduckgo.com/html"
|
||||
_SEARCH_TIMEOUT = 10.0
|
||||
_FETCH_TIMEOUT = 10.0
|
||||
_MAX_FETCH_REDIRECTS = 5
|
||||
_BACKOFF_SECONDS = 1.0
|
||||
_USER_AGENT = "researchx/0.2 (local research studio)"
|
||||
_RESULT_RE = re.compile(
|
||||
@@ -94,34 +93,21 @@ class DuckDuckGoSearchAdapter:
|
||||
return re.sub(r"\s+", " ", text).strip()
|
||||
|
||||
async def fetch(self, url: str) -> PageContent:
|
||||
from urllib.parse import urljoin
|
||||
|
||||
current = url
|
||||
if not url_allowed(url):
|
||||
raise AdapterError("fetch refused: url not allowed (private or non-http scheme)")
|
||||
async with httpx.AsyncClient(
|
||||
timeout=_FETCH_TIMEOUT,
|
||||
follow_redirects=False,
|
||||
follow_redirects=True,
|
||||
headers={"User-Agent": _USER_AGENT},
|
||||
) as client:
|
||||
for _ in range(_MAX_FETCH_REDIRECTS + 1):
|
||||
if not url_allowed(current):
|
||||
raise AdapterError(
|
||||
"fetch refused: url not allowed (private or non-http scheme)"
|
||||
)
|
||||
response = await client.get(current)
|
||||
if 300 <= response.status_code < 400:
|
||||
location = response.headers.get("location")
|
||||
if not location:
|
||||
raise AdapterError("fetch failed: redirect without location")
|
||||
current = urljoin(current, location)
|
||||
continue
|
||||
response.raise_for_status()
|
||||
return PageContent(
|
||||
url=url,
|
||||
title=extract_title(response.text),
|
||||
text=extract_text(response.text),
|
||||
fetched_at=datetime.now(UTC),
|
||||
)
|
||||
raise AdapterError("fetch failed: too many redirects")
|
||||
response = await client.get(url)
|
||||
response.raise_for_status()
|
||||
return PageContent(
|
||||
url=url,
|
||||
title=extract_title(response.text),
|
||||
text=extract_text(response.text),
|
||||
fetched_at=datetime.now(UTC),
|
||||
)
|
||||
|
||||
@property
|
||||
def last_degradation(self) -> str | None:
|
||||
|
||||
@@ -12,8 +12,6 @@ _REQUEST_TIMEOUT = 60.0
|
||||
|
||||
|
||||
class OllamaModelAdapter:
|
||||
probing_supported = True
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
secrets: EnvFileSecretsProvider,
|
||||
|
||||
@@ -10,7 +10,6 @@ from researchx.adapters.url_guard import url_allowed
|
||||
_TAVILY_URL = "https://api.tavily.com/search"
|
||||
_SEARCH_TIMEOUT = 10.0
|
||||
_FETCH_TIMEOUT = 10.0
|
||||
_MAX_FETCH_REDIRECTS = 5
|
||||
_USER_AGENT = "researchx/0.2 (local research studio)"
|
||||
_TITLE_RE = re.compile(r"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)
|
||||
_BLOCK_RE = re.compile(r"<(script|style)[^>]*>.*?</\1>", re.IGNORECASE | re.DOTALL)
|
||||
@@ -57,31 +56,18 @@ class TavilySearchAdapter:
|
||||
][:max_results]
|
||||
|
||||
async def fetch(self, url: str) -> PageContent:
|
||||
from urllib.parse import urljoin
|
||||
|
||||
current = url
|
||||
if not url_allowed(url):
|
||||
raise AdapterError("fetch refused: url not allowed (private or non-http scheme)")
|
||||
async with httpx.AsyncClient(
|
||||
timeout=_FETCH_TIMEOUT,
|
||||
follow_redirects=False,
|
||||
follow_redirects=True,
|
||||
headers={"User-Agent": _USER_AGENT},
|
||||
) as client:
|
||||
for _ in range(_MAX_FETCH_REDIRECTS + 1):
|
||||
if not url_allowed(current):
|
||||
raise AdapterError(
|
||||
"fetch refused: url not allowed (private or non-http scheme)"
|
||||
)
|
||||
response = await client.get(current)
|
||||
if 300 <= response.status_code < 400:
|
||||
location = response.headers.get("location")
|
||||
if not location:
|
||||
raise AdapterError("fetch failed: redirect without location")
|
||||
current = urljoin(current, location)
|
||||
continue
|
||||
response.raise_for_status()
|
||||
return PageContent(
|
||||
url=url,
|
||||
title=extract_title(response.text),
|
||||
text=extract_text(response.text),
|
||||
fetched_at=datetime.now(UTC),
|
||||
)
|
||||
raise AdapterError("fetch failed: too many redirects")
|
||||
response = await client.get(url)
|
||||
response.raise_for_status()
|
||||
return PageContent(
|
||||
url=url,
|
||||
title=extract_title(response.text),
|
||||
text=extract_text(response.text),
|
||||
fetched_at=datetime.now(UTC),
|
||||
)
|
||||
@@ -8,13 +8,12 @@ from fastapi.middleware.cors import CORSMiddleware
|
||||
from pydantic import BaseModel, Field, ValidationError
|
||||
from sse_starlette.sse import EventSourceResponse
|
||||
|
||||
from researchx.adapters.capability_probe import probe_model_adapter, supports_probe
|
||||
from researchx.adapters.fakes import CORPUS, FakeModelAdapter, FakeSearchAdapter
|
||||
from researchx.adapters.local_storage import LocalStorageAdapter
|
||||
from researchx.adapters.types import AdapterError, ChatResult
|
||||
from researchx.engine.actor import RunActor, make_run_config
|
||||
from researchx.engine.events import EventBus, RunEvent
|
||||
from researchx.engine.run_state import Plan, RunState
|
||||
from researchx.engine.run_state import RunState
|
||||
from researchx.engine.states import RunStateName
|
||||
from researchx.methodologies import registry
|
||||
|
||||
@@ -29,8 +28,8 @@ _CANNED_WRITER = (
|
||||
|
||||
|
||||
class CreateRunBody(BaseModel):
|
||||
question: str = Field(min_length=1, max_length=2000)
|
||||
context: str = Field(default="", max_length=8000)
|
||||
question: str = Field(min_length=1)
|
||||
context: str = ""
|
||||
methodology: str
|
||||
dial_overrides: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
@@ -77,7 +76,7 @@ _STATUS_MAP = {
|
||||
RunStateName.VERIFYING: "verifying",
|
||||
RunStateName.DONE: "done",
|
||||
RunStateName.CANCELLED: "cancelled",
|
||||
RunStateName.FAILED: "failed",
|
||||
RunStateName.FAILED: "cancelled",
|
||||
}
|
||||
|
||||
_ETA_TABLE = {"rapid-answer": (2, "low"), "deep-report": (12, "medium")}
|
||||
@@ -171,15 +170,11 @@ def create_app(
|
||||
|
||||
@app.get("/api/config")
|
||||
async def config_status() -> ConfigStatus:
|
||||
latest_probe = None
|
||||
for run_id in list(runs._actors)[-1:]:
|
||||
actor_probe = runs.actor(run_id).state.probe
|
||||
latest_probe = actor_probe
|
||||
return ConfigStatus(
|
||||
model_adapter=type(model).__name__,
|
||||
search_adapter=type(search).__name__,
|
||||
storage_dir=str(storage.run_dir("00000000").parent),
|
||||
probe=latest_probe,
|
||||
probe=None,
|
||||
cost_counters=runs.cost_counters,
|
||||
)
|
||||
|
||||
@@ -226,22 +221,14 @@ def create_app(
|
||||
storage=storage,
|
||||
registry=registry,
|
||||
event_bus=bus,
|
||||
on_complete=runs.note_completed,
|
||||
)
|
||||
probe_data: dict[str, Any] | None = None
|
||||
if supports_probe(model):
|
||||
try:
|
||||
probe = await probe_model_adapter(model)
|
||||
probe_data = probe.model_dump(mode="json")
|
||||
await actor.attach_probe(probe_data)
|
||||
except AdapterError:
|
||||
probe_data = None
|
||||
|
||||
task = asyncio.create_task(actor.run())
|
||||
runs.add(run_id, actor, task, bus)
|
||||
|
||||
async def _watch() -> None:
|
||||
await task
|
||||
state = await task
|
||||
if state.state == RunStateName.DONE:
|
||||
runs.note_completed()
|
||||
|
||||
asyncio.create_task(_watch())
|
||||
return _summary(run_id)
|
||||
@@ -257,13 +244,7 @@ def create_app(
|
||||
if not runs.has(run_id):
|
||||
raise HTTPException(status_code=404, detail="run not found")
|
||||
actor = runs.actor(run_id)
|
||||
edited_plan: Plan | None = None
|
||||
if body and body.plan:
|
||||
try:
|
||||
edited_plan = Plan.model_validate(body.plan)
|
||||
except ValidationError:
|
||||
raise HTTPException(status_code=422, detail="invalid plan")
|
||||
await actor.approve(edited_plan)
|
||||
await actor.approve()
|
||||
return _summary(run_id)
|
||||
|
||||
@app.post("/api/runs/{run_id}/pause")
|
||||
|
||||
@@ -1,85 +0,0 @@
|
||||
"""Dev bridge: serve the FastAPI app on a loopback port for E2E tests.
|
||||
|
||||
Prints READY on stdout once the server is listening; SIGTERM shuts
|
||||
down cleanly. Single-user, loopback-only, disposable — for tests and
|
||||
local dev. Uses an inexhaustible scripted model so many sequential
|
||||
runs work without restocking canned responses.
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import uvicorn
|
||||
|
||||
from researchx.adapters.fakes import CORPUS, FakeSearchAdapter
|
||||
from researchx.adapters.types import ChatResult, ModelMessage
|
||||
from researchx.api.app import create_app
|
||||
|
||||
_HOST = "127.0.0.1"
|
||||
_PORT = 8000
|
||||
|
||||
_PLAN = json.dumps(
|
||||
{
|
||||
"sub_questions": ["verification passes"],
|
||||
"source_universe": ["web"],
|
||||
"outline": ["approach", "findings"],
|
||||
}
|
||||
)
|
||||
_WRITER = (
|
||||
"# Research Memo\n\n"
|
||||
"The verified core finding [p-1]. A corroborated observation [p-2].\n"
|
||||
)
|
||||
|
||||
|
||||
class EndlessModelAdapter:
|
||||
def __init__(self) -> None:
|
||||
self.calls = 0
|
||||
|
||||
async def chat(
|
||||
self,
|
||||
messages: list[ModelMessage],
|
||||
*,
|
||||
model: str,
|
||||
temperature: float = 0.2,
|
||||
json_mode: bool = False,
|
||||
) -> ChatResult:
|
||||
del messages, temperature
|
||||
self.calls += 1
|
||||
if json_mode:
|
||||
return ChatResult(content=_PLAN, model=model)
|
||||
return ChatResult(content=_WRITER, model=model)
|
||||
|
||||
async def list_models(self) -> list[str]:
|
||||
return ["endless-model"]
|
||||
|
||||
def roles(self) -> Any:
|
||||
from researchx.adapters.model import ModelRoles
|
||||
|
||||
return ModelRoles(planner="endless-model", writer="endless-model")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
app = create_app(
|
||||
storage_dir=Path(tmp),
|
||||
model_adapter=EndlessModelAdapter(),
|
||||
search_adapter=FakeSearchAdapter(CORPUS),
|
||||
)
|
||||
|
||||
@app.on_event("startup")
|
||||
def _announce() -> None:
|
||||
print(f"READY port={_PORT}", flush=True)
|
||||
|
||||
config = uvicorn.Config(app, host=_HOST, port=_PORT, log_level="warning")
|
||||
server = uvicorn.Server(config)
|
||||
with contextlib.suppress(KeyboardInterrupt):
|
||||
server.run()
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,5 +1,4 @@
|
||||
import asyncio
|
||||
from collections.abc import Callable
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
@@ -35,7 +34,6 @@ class RunActor:
|
||||
storage: LocalStorageAdapter,
|
||||
registry: MethodologyRegistry,
|
||||
event_bus: EventBus,
|
||||
on_complete: Callable[[], None] | None = None,
|
||||
) -> None:
|
||||
self._config = run_config
|
||||
self._model = model
|
||||
@@ -43,7 +41,6 @@ class RunActor:
|
||||
self._storage = storage
|
||||
self._registry = registry
|
||||
self._bus = event_bus
|
||||
self._on_complete = on_complete
|
||||
self._store = EvidenceStore()
|
||||
self.state = RunState(
|
||||
run_id=run_config.run_id,
|
||||
@@ -86,8 +83,6 @@ class RunActor:
|
||||
async def _checkpoint(self, target: RunStateName) -> None:
|
||||
if self.state.state != target:
|
||||
self.state.transition(target)
|
||||
if target is RunStateName.DONE and self._on_complete is not None:
|
||||
self._on_complete()
|
||||
await self._persist()
|
||||
|
||||
async def _wait_if_paused(self) -> None:
|
||||
@@ -117,15 +112,6 @@ class RunActor:
|
||||
except AdapterError as exc:
|
||||
await self._fail(str(exc))
|
||||
raise EngineError(str(exc)) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
await self._fail(f"network error ({type(exc).__name__})")
|
||||
raise EngineError(f"network error ({type(exc).__name__})") from exc
|
||||
except EngineError:
|
||||
await self._fail("engine error")
|
||||
raise
|
||||
except Exception as exc:
|
||||
await self._fail(f"unexpected engine failure ({type(exc).__name__})")
|
||||
raise EngineError(f"unexpected engine failure ({type(exc).__name__})") from exc
|
||||
return self.state
|
||||
|
||||
async def attach_probe(self, probe_data: dict[str, Any]) -> None:
|
||||
@@ -191,21 +177,10 @@ class RunActor:
|
||||
data = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise AdapterError("planner returned non-JSON plan") from exc
|
||||
if not isinstance(data, dict):
|
||||
raise AdapterError("planner plan is not a JSON object")
|
||||
sub_questions = data.get("sub_questions", [])
|
||||
source_universe = data.get("source_universe", [])
|
||||
outline = data.get("outline", [])
|
||||
if not (
|
||||
isinstance(sub_questions, list)
|
||||
and isinstance(source_universe, list)
|
||||
and isinstance(outline, list)
|
||||
):
|
||||
raise AdapterError("planner plan has malformed lists")
|
||||
return Plan(
|
||||
sub_questions=[str(q) for q in sub_questions if isinstance(q, str)],
|
||||
source_universe=[str(u) for u in source_universe if isinstance(u, str)],
|
||||
outline=[str(o) for o in outline if isinstance(o, str)],
|
||||
sub_questions=[str(q) for q in data.get("sub_questions", [])],
|
||||
source_universe=[str(u) for u in data.get("source_universe", [])],
|
||||
outline=[str(o) for o in data.get("outline", [])],
|
||||
)
|
||||
|
||||
async def _approval_phase(self) -> None:
|
||||
|
||||
@@ -49,41 +49,12 @@ def anchor_quote(quote: str, passages: list[Passage]) -> AnchorVerdict:
|
||||
)
|
||||
|
||||
|
||||
_MARKER_RE = re.compile(r"\[p-(\d+)\]")
|
||||
_HEADER_RE = re.compile(r"^#{1,6}\s.*$", re.MULTILINE)
|
||||
|
||||
|
||||
def _claim_fragment(before: str) -> str:
|
||||
cleaned = _HEADER_RE.sub("", before).rstrip()
|
||||
fragment_start = max(
|
||||
cleaned.rfind("."), cleaned.rfind("!"), cleaned.rfind("?"), cleaned.rfind("\n")
|
||||
)
|
||||
quote_text = cleaned[fragment_start + 1 :].strip(" \t>\"'*-")
|
||||
if not quote_text:
|
||||
quote_text = cleaned[-200:].strip(" \t>\"'*-")
|
||||
return quote_text
|
||||
|
||||
|
||||
def extract_report_claims(report_text: str) -> list[tuple[str, str]]:
|
||||
"""Extract (passage_id, quote_text) claims from the report.
|
||||
|
||||
The quote text is what the WRITER actually wrote adjacent to the
|
||||
citation marker — the last sentence fragment before each [p-N]
|
||||
marker — NOT the stored passage's own text. This is the adversarial
|
||||
core of REQ-20: a fabricated quote riding a real passage ID must
|
||||
be visible to the anchorer.
|
||||
"""
|
||||
claims: list[tuple[str, str]] = []
|
||||
for match in _MARKER_RE.finditer(report_text):
|
||||
passage_id = f"p-{match.group(1)}"
|
||||
quote_text = _claim_fragment(report_text[: match.start()])
|
||||
claims.append((passage_id, quote_text))
|
||||
return claims
|
||||
|
||||
|
||||
def extract_quotes_from_report(report_text: str, evidence: EvidenceStore) -> list[str]:
|
||||
return [quote for _pid, quote in extract_report_claims(report_text) if evidence.get(_pid)]
|
||||
|
||||
|
||||
def extract_cited_passage_ids(report_text: str) -> list[str]:
|
||||
return [f"p-{m.group(1)}" for m in _MARKER_RE.finditer(report_text)]
|
||||
marker_re = re.compile(r"\[(p-\d+)\]")
|
||||
quotes: list[str] = []
|
||||
for match in marker_re.finditer(report_text):
|
||||
passage_id = match.group(1)
|
||||
passage = evidence.get(passage_id)
|
||||
if passage:
|
||||
quotes.append(passage.quote)
|
||||
return quotes
|
||||
@@ -4,12 +4,7 @@ from datetime import datetime
|
||||
from pydantic import BaseModel
|
||||
|
||||
from researchx.evidence.store import EvidenceStore, Passage
|
||||
from researchx.trust.anchoring import (
|
||||
anchor_quote,
|
||||
extract_cited_passage_ids,
|
||||
extract_report_claims,
|
||||
normalize,
|
||||
)
|
||||
from researchx.trust.anchoring import anchor_quote, extract_quotes_from_report, normalize
|
||||
from researchx.trust.liveness import LivenessStatus
|
||||
|
||||
_AUTHORITY_TIERS: dict[str, str] = {
|
||||
@@ -80,22 +75,15 @@ def build_ledger(
|
||||
) -> CitationLedger:
|
||||
claims = _claims_per_passage(report_text)
|
||||
all_passages = evidence.writer_context()
|
||||
report_claims = extract_report_claims(report_text)
|
||||
cited_ids = extract_cited_passage_ids(report_text)
|
||||
known_ids = {p.passage_id for p in all_passages}
|
||||
cited_urls: list[str] = []
|
||||
report_quotes = extract_quotes_from_report(report_text, evidence)
|
||||
entries: list[CitationEntry] = []
|
||||
anchored = 0
|
||||
unanchored = 0
|
||||
weak = 0
|
||||
confidence: dict[str, str] = {}
|
||||
|
||||
for passage_id, writer_quote in report_claims:
|
||||
passage = evidence.get(passage_id)
|
||||
if passage is None:
|
||||
unanchored += 1
|
||||
continue
|
||||
verdict = anchor_quote(writer_quote, [passage])
|
||||
for quote in report_quotes:
|
||||
verdict = anchor_quote(quote, all_passages)
|
||||
if verdict.matched:
|
||||
anchored += 1
|
||||
elif verdict.weak:
|
||||
@@ -103,16 +91,7 @@ def build_ledger(
|
||||
else:
|
||||
unanchored += 1
|
||||
|
||||
for cited_id in cited_ids:
|
||||
if cited_id not in known_ids:
|
||||
unanchored += 1
|
||||
passage = evidence.get(cited_id)
|
||||
if passage is not None and passage.url not in cited_urls:
|
||||
cited_urls.append(passage.url)
|
||||
|
||||
for page in evidence.pages():
|
||||
if page.url not in cited_urls:
|
||||
continue
|
||||
passages = evidence.passages_for_url(page.url)
|
||||
passage_ids = [p.passage_id for p in passages]
|
||||
page_claims = sum(claims.get(pid, 0) for pid in passage_ids)
|
||||
@@ -169,9 +148,7 @@ LIVENESS_EXPORT_GATE = 0.99
|
||||
|
||||
def liveness_export_gate(ledger: CitationLedger) -> tuple[bool, list[str]]:
|
||||
failures = [e.url for e in ledger.entries if e.health != "live"]
|
||||
liveness_ok = ledger.checklist.liveness_ratio >= LIVENESS_EXPORT_GATE
|
||||
anchoring_ok = ledger.checklist.unanchored_claims == 0
|
||||
return liveness_ok and anchoring_ok, failures
|
||||
return ledger.checklist.liveness_ratio >= LIVENESS_EXPORT_GATE, failures
|
||||
|
||||
|
||||
def export_checklist(ledger: CitationLedger) -> str:
|
||||
@@ -186,10 +163,6 @@ def export_checklist(ledger: CitationLedger) -> str:
|
||||
f"Liveness ratio: {ledger.checklist.liveness_ratio} (export gate >= {LIVENESS_EXPORT_GATE}: "
|
||||
f"{'PASS' if gate_passed else 'FAIL'})"
|
||||
),
|
||||
(
|
||||
f"Anchoring gate: {'PASS' if ledger.checklist.unanchored_claims == 0 else 'FAIL'} "
|
||||
f"({ledger.checklist.unanchored_claims} unanchored)"
|
||||
),
|
||||
f"Anchored claims: {ledger.checklist.anchored_claims}",
|
||||
f"Unanchored claims: {ledger.checklist.unanchored_claims}",
|
||||
f"Weak matches (flagged, not accepted): {ledger.checklist.weak_matches}",
|
||||
|
||||
@@ -1,359 +0,0 @@
|
||||
"""Adversarial suite (REQ-32): attacks on the trust loop, proven caught.
|
||||
|
||||
Every test here is an attempt to fool the verifier. If one passes
|
||||
without being flagged, the trust loop is broken — the suite fails.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
from researchx.adapters.fakes import CORPUS, FakeModelAdapter, FakeSearchAdapter
|
||||
from researchx.adapters.types import AdapterError, ChatResult
|
||||
from researchx.api.app import create_app
|
||||
from researchx.evidence.store import EvidenceStore
|
||||
from researchx.trust.anchoring import anchor_quote
|
||||
from researchx.trust.ledger import build_ledger, export_checklist, liveness_export_gate
|
||||
from researchx.trust.liveness import LivenessStatus, check_liveness
|
||||
|
||||
|
||||
def test_attack_hallucinated_url_fails_export_gate() -> None:
|
||||
store = EvidenceStore()
|
||||
store.add_passage("https://real.example.org/a", "A real stored passage about the topic " * 5)
|
||||
report = (
|
||||
"# Report\n\n"
|
||||
"Real claim [p-1].\n\n"
|
||||
"Supporting detail from a hallucinated source [p-2].\n"
|
||||
)
|
||||
ledger = build_ledger(
|
||||
run_id="attack001",
|
||||
report_text=report,
|
||||
evidence=store,
|
||||
liveness={"https://real.example.org/a": LivenessStatus.LIVE},
|
||||
visited_at=datetime.now(UTC),
|
||||
)
|
||||
claim_counts = _claim_counts(ledger)
|
||||
assert claim_counts.get("p-1", 0) >= 1
|
||||
assert "p-2" in report
|
||||
assert not any("p-2" in pids for pids in [e.passages for e in ledger.entries]), (
|
||||
"hallucinated passage id must not appear in any ledger entry"
|
||||
)
|
||||
unanchored = ledger.checklist.unanchored_claims
|
||||
assert unanchored >= 1, "hallucinated citation must be flagged as unanchored"
|
||||
|
||||
|
||||
def _claim_counts(ledger) -> dict[str, int]:
|
||||
counts: dict[str, int] = {}
|
||||
for entry in ledger.entries:
|
||||
for pid in entry.passages:
|
||||
counts[pid] = counts.get(pid, 0) + entry.claims_supported
|
||||
return counts
|
||||
|
||||
|
||||
def test_attack_fabricated_quote_flagged_not_accepted() -> None:
|
||||
store = EvidenceStore()
|
||||
for page in CORPUS:
|
||||
store.add_page(page.url, page.title, page.text)
|
||||
passages = store.writer_context()
|
||||
|
||||
fabricated = "This sentence appears in exactly zero stored passages anywhere."
|
||||
verdict = anchor_quote(fabricated, passages)
|
||||
assert verdict.matched is False
|
||||
assert verdict.weak is False
|
||||
|
||||
near_miss = passages[0].text[: len(passages[0].text) // 2] + " And a divergent clause."
|
||||
near_verdict = anchor_quote(near_miss, passages)
|
||||
assert near_verdict.matched is False or near_verdict.weak is True
|
||||
|
||||
exact = anchor_quote(passages[0].quote, passages)
|
||||
assert exact.matched is True
|
||||
|
||||
|
||||
def test_attack_quote_from_other_passage_counts_as_corroboration_not_anchor() -> None:
|
||||
from researchx.adapters.fakes import SHARED_SENTENCE
|
||||
|
||||
store = EvidenceStore()
|
||||
first = store.add_passage("https://a.example.org", SHARED_SENTENCE)
|
||||
second = store.add_passage("https://b.example.org", SHARED_SENTENCE + " Extra tail text.")
|
||||
verdict_first = anchor_quote(second.quote, [first])
|
||||
assert verdict_first.matched is True or verdict_first.ratio > 0.5
|
||||
|
||||
|
||||
async def test_attack_liveness_private_redirect_dies(monkeypatch) -> None:
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(302, headers={"location": "http://169.254.169.254/x"})
|
||||
|
||||
transport = httpx.MockTransport(handler)
|
||||
async with httpx.AsyncClient(transport=transport) as client:
|
||||
status = await check_liveness("https://example.org/redirect", client)
|
||||
assert status == LivenessStatus.DEAD
|
||||
|
||||
|
||||
async def test_attack_liveness_redirect_chain_to_dead(monkeypatch) -> None:
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if "hop1" in str(request.url):
|
||||
return httpx.Response(302, headers={"location": "https://example.org/hop2"})
|
||||
if "hop2" in str(request.url):
|
||||
return httpx.Response(302, headers={"location": "https://example.org/hop3"})
|
||||
return httpx.Response(410)
|
||||
|
||||
transport = httpx.MockTransport(handler)
|
||||
async with httpx.AsyncClient(transport=transport) as client:
|
||||
status = await check_liveness("https://example.org/hop1", client)
|
||||
assert status == LivenessStatus.DEAD
|
||||
|
||||
|
||||
async def test_attack_oversized_response_capped(monkeypatch) -> None:
|
||||
huge = "x" * (5 * 1024 * 1024)
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(200, text=huge)
|
||||
|
||||
transport = httpx.MockTransport(handler)
|
||||
async with httpx.AsyncClient(transport=transport) as client:
|
||||
status = await check_liveness("https://example.org/huge", client)
|
||||
assert status == LivenessStatus.LIVE
|
||||
|
||||
|
||||
async def test_attack_pause_resume_mid_depth_loop_over_http(tmp_path: Path) -> None:
|
||||
plan_template = json.dumps(
|
||||
{
|
||||
"sub_questions": ["verification passes", "quote anchoring"],
|
||||
"source_universe": ["web"],
|
||||
"outline": ["intro", "findings"],
|
||||
}
|
||||
)
|
||||
writer = "# Report\n\nClaim [p-1] and claim [p-2].\n"
|
||||
canned = [
|
||||
ChatResult(content=plan_template, model="fake-planner"),
|
||||
ChatResult(content=plan_template, model="fake-planner"),
|
||||
ChatResult(content=plan_template, model="fake-planner"),
|
||||
ChatResult(content=writer, model="fake-writer"),
|
||||
]
|
||||
app = create_app(
|
||||
storage_dir=tmp_path,
|
||||
model_adapter=FakeModelAdapter(canned=canned),
|
||||
search_adapter=FakeSearchAdapter(CORPUS),
|
||||
)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app), base_url="http://testserver"
|
||||
) as client:
|
||||
created = await client.post(
|
||||
"/api/runs",
|
||||
json={"question": "q", "methodology": "deep-report"},
|
||||
)
|
||||
assert created.status_code == 201
|
||||
run_id = created.json()["id"]
|
||||
|
||||
for _ in range(300):
|
||||
status = (await client.get(f"/api/runs/{run_id}")).json()["status"]
|
||||
if status == "awaiting_approval":
|
||||
break
|
||||
await asyncio.sleep(0.01)
|
||||
assert status == "awaiting_approval"
|
||||
|
||||
await client.post(f"/api/runs/{run_id}/approve")
|
||||
|
||||
paused_seen = False
|
||||
for _ in range(600):
|
||||
status = (await client.get(f"/api/runs/{run_id}")).json()["status"]
|
||||
if status in ("reflecting", "searching", "extracting"):
|
||||
await client.post(f"/api/runs/{run_id}/pause")
|
||||
await client.post(f"/api/runs/{run_id}/resume")
|
||||
paused_seen = True
|
||||
break
|
||||
if status == "done":
|
||||
break
|
||||
await asyncio.sleep(0.005)
|
||||
|
||||
for _ in range(600):
|
||||
status = (await client.get(f"/api/runs/{run_id}")).json()["status"]
|
||||
if status == "done":
|
||||
break
|
||||
await asyncio.sleep(0.005)
|
||||
assert status == "done"
|
||||
assert paused_seen, "mid-loop pause/resume must have been exercised"
|
||||
|
||||
report = await client.get(f"/api/runs/{run_id}/report")
|
||||
assert report.status_code == 200
|
||||
|
||||
|
||||
async def test_attack_cancel_mid_loop_persists(tmp_path: Path) -> None:
|
||||
plan = json.dumps(
|
||||
{
|
||||
"sub_questions": ["verification passes", "quote anchoring"],
|
||||
"source_universe": ["web"],
|
||||
"outline": ["intro"],
|
||||
}
|
||||
)
|
||||
app = create_app(
|
||||
storage_dir=tmp_path,
|
||||
model_adapter=FakeModelAdapter(
|
||||
canned=[ChatResult(content=plan, model="fake-planner") for _ in range(10)]
|
||||
+ [ChatResult(content="# never reached", model="fake-writer")]
|
||||
),
|
||||
search_adapter=FakeSearchAdapter(CORPUS),
|
||||
)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app), base_url="http://testserver"
|
||||
) as client:
|
||||
created = await client.post(
|
||||
"/api/runs", json={"question": "q", "methodology": "deep-report"}
|
||||
)
|
||||
run_id = created.json()["id"]
|
||||
for _ in range(300):
|
||||
if (await client.get(f"/api/runs/{run_id}")).json()["status"] == "awaiting_approval":
|
||||
break
|
||||
await asyncio.sleep(0.01)
|
||||
await client.post(f"/api/runs/{run_id}/approve")
|
||||
for _ in range(600):
|
||||
status = (await client.get(f"/api/runs/{run_id}")).json()["status"]
|
||||
if status in ("searching", "extracting", "reflecting"):
|
||||
await client.post(f"/api/runs/{run_id}/cancel")
|
||||
break
|
||||
await asyncio.sleep(0.005)
|
||||
for _ in range(300):
|
||||
status = (await client.get(f"/api/runs/{run_id}")).json()["status"]
|
||||
if status == "cancelled":
|
||||
break
|
||||
await asyncio.sleep(0.01)
|
||||
assert status == "cancelled"
|
||||
|
||||
def test_attack_weak_band_pinned_real_input() -> None:
|
||||
store = EvidenceStore()
|
||||
passage_text = (
|
||||
"The verification pipeline re-anchored every quoted claim to its stored "
|
||||
"passage before the report was allowed to ship to the reading room."
|
||||
)
|
||||
passage = store.add_passage("https://example.org/weak", passage_text)
|
||||
words = passage_text.split()
|
||||
near_miss = " ".join(words[:11]) + " released to the audience."
|
||||
verdict = anchor_quote(near_miss, [passage])
|
||||
assert 0.80 <= verdict.ratio < 0.85, f"expected weak band, got {verdict.ratio}"
|
||||
assert verdict.matched is False
|
||||
assert verdict.weak is True, f"near-threshold must flag weak, got {verdict.ratio}"
|
||||
|
||||
|
||||
def test_attack_fully_fabricated_report_fails_export() -> None:
|
||||
store = EvidenceStore()
|
||||
real = store.add_passage("https://example.org/real", "A genuine stored passage " * 10)
|
||||
fabricated_report = (
|
||||
"# Report\n\nClaim [p-1].\n\nClaim [p-7]. Claim [p-42].\n"
|
||||
)
|
||||
ledger = build_ledger(
|
||||
run_id="attack002",
|
||||
report_text=fabricated_report,
|
||||
evidence=store,
|
||||
liveness={"https://example.org/real": LivenessStatus.LIVE},
|
||||
visited_at=datetime.now(UTC),
|
||||
)
|
||||
passed, failures = liveness_export_gate(ledger)
|
||||
assert passed is False, "fully-fabricated report must fail the export gate"
|
||||
assert failures == []
|
||||
assert ledger.checklist.unanchored_claims >= 2
|
||||
del real
|
||||
|
||||
|
||||
def test_export_round_trip_json_validates_into_backend_model() -> None:
|
||||
from researchx.trust.ledger import CitationLedger
|
||||
|
||||
store = EvidenceStore()
|
||||
stored_text = "A passage with real content for the round-trip proof."
|
||||
store.add_passage("https://example.org/a", stored_text)
|
||||
report = f"# Report\n\n{stored_text} [p-1].\n"
|
||||
ledger = build_ledger(
|
||||
run_id="attack003",
|
||||
report_text=report,
|
||||
evidence=store,
|
||||
liveness={"https://example.org/a": LivenessStatus.LIVE},
|
||||
visited_at=datetime.now(UTC),
|
||||
)
|
||||
exported = ledger.model_dump_json()
|
||||
restored = CitationLedger.model_validate_json(exported)
|
||||
assert restored == ledger
|
||||
checklist_md = export_checklist(ledger)
|
||||
assert "Anchoring gate" in checklist_md
|
||||
assert "PASS" in checklist_md
|
||||
|
||||
|
||||
def test_attack_fabricated_quote_on_real_passage_id_fails() -> None:
|
||||
store = EvidenceStore()
|
||||
stored = store.add_passage(
|
||||
"https://example.org/real",
|
||||
"The genuine finding about verification pipelines is that anchoring works. " * 3,
|
||||
)
|
||||
report = (
|
||||
"# Report\n\n"
|
||||
f"THIS FABRICATED QUOTE WAS NEVER STORED ANYWHERE AT ALL [{stored.passage_id}]\n"
|
||||
)
|
||||
ledger = build_ledger(
|
||||
run_id="attack004",
|
||||
report_text=report,
|
||||
evidence=store,
|
||||
liveness={"https://example.org/real": LivenessStatus.LIVE},
|
||||
visited_at=datetime.now(UTC),
|
||||
)
|
||||
assert ledger.checklist.unanchored_claims >= 1, (
|
||||
"a fabricated quote riding a real passage ID must be flagged (REQ-20)"
|
||||
)
|
||||
passed, _ = liveness_export_gate(ledger)
|
||||
assert passed is False
|
||||
|
||||
|
||||
def test_attack_real_verbatim_quote_on_real_passage_anchors() -> None:
|
||||
store = EvidenceStore()
|
||||
stored_text = (
|
||||
"The genuine finding about verification pipelines is that anchoring works."
|
||||
)
|
||||
stored = store.add_passage("https://example.org/real", stored_text + " " + stored_text)
|
||||
report = f"# Report\n\n{stored_text} [{stored.passage_id}]\n"
|
||||
ledger = build_ledger(
|
||||
run_id="attack005",
|
||||
report_text=report,
|
||||
evidence=store,
|
||||
liveness={"https://example.org/real": LivenessStatus.LIVE},
|
||||
visited_at=datetime.now(UTC),
|
||||
)
|
||||
assert ledger.checklist.anchored_claims == 1
|
||||
assert ledger.checklist.unanchored_claims == 0
|
||||
passed, _ = liveness_export_gate(ledger)
|
||||
assert passed is True
|
||||
|
||||
|
||||
async def test_attack_search_result_redirecting_to_private_ip_refused(monkeypatch) -> None:
|
||||
import tempfile
|
||||
from pathlib import Path as PathMod
|
||||
|
||||
import httpx as httpx_mod
|
||||
|
||||
from researchx.adapters.secrets import EnvFileSecretsProvider
|
||||
from researchx.adapters.tavily import TavilySearchAdapter
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
env_file = PathMod(tmpdir) / ".env.secrets"
|
||||
env_file.write_text("TAVILY_API_KEY=t-key\n", encoding="utf-8")
|
||||
secrets = EnvFileSecretsProvider(paths=[env_file])
|
||||
adapter = TavilySearchAdapter(secrets)
|
||||
|
||||
def handler(request: httpx_mod.Request) -> httpx_mod.Response:
|
||||
if "boobytrap" in str(request.url):
|
||||
return httpx_mod.Response(
|
||||
302, headers={"location": "http://169.254.169.254/latest/meta-data"}
|
||||
)
|
||||
return httpx_mod.Response(200, text="<html><body>ok</body></html>")
|
||||
|
||||
transport = httpx_mod.MockTransport(handler)
|
||||
original = httpx_mod.AsyncClient
|
||||
monkeypatch.setattr(
|
||||
httpx_mod, "AsyncClient", lambda **kw: original(transport=transport, **kw)
|
||||
)
|
||||
|
||||
try:
|
||||
await adapter.fetch("https://example.org/boobytrap")
|
||||
except AdapterError as exc:
|
||||
assert "refused" in str(exc) or "not allowed" in str(exc)
|
||||
else:
|
||||
raise AssertionError("fetch of a private-redirecting URL must be refused")
|
||||
@@ -1,64 +0,0 @@
|
||||
"""Opt-in live smoke tier (G-004): real-adapter probes, keyed only.
|
||||
|
||||
These tests NEVER run in the zero-key CI suite: they are marked
|
||||
`live` and skip cleanly when OLLAMA_API_KEY / TAVILY_API_KEY are
|
||||
absent from the secrets scope. Run explicitly with:
|
||||
|
||||
uv run pytest -m live
|
||||
"""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from researchx.adapters.capability_probe import probe_model_adapter
|
||||
from researchx.adapters.ollama import OllamaModelAdapter
|
||||
from researchx.adapters.secrets import EnvFileSecretsProvider
|
||||
from researchx.adapters.tavily import TavilySearchAdapter
|
||||
from researchx.trust.liveness import LivenessStatus, check_liveness
|
||||
|
||||
pytestmark = pytest.mark.live
|
||||
|
||||
_SECRETS_PATHS = [
|
||||
Path.cwd() / ".ciagent" / ".env.secrets",
|
||||
Path.cwd().parents[1] / ".ciagent" / ".env.secrets",
|
||||
]
|
||||
|
||||
|
||||
def _secrets() -> EnvFileSecretsProvider:
|
||||
return EnvFileSecretsProvider(paths=_SECRETS_PATHS)
|
||||
|
||||
|
||||
def _needs_key(key: str) -> str:
|
||||
value = _secrets().resolve(key) or os.environ.get(key)
|
||||
if not value:
|
||||
pytest.skip(f"{key} not present — live tier is opt-in")
|
||||
return value
|
||||
|
||||
|
||||
async def test_live_ollama_probe_and_chat() -> None:
|
||||
_needs_key("OLLAMA_API_KEY")
|
||||
adapter = OllamaModelAdapter(
|
||||
_secrets(), planner_model="gpt-oss:20b", writer_model="gpt-oss:20b"
|
||||
)
|
||||
probe = await probe_model_adapter(adapter)
|
||||
assert probe.endpoint in ("https://ollama.com/v1", "http://localhost:11434/v1")
|
||||
assert probe.context_window <= 8192
|
||||
assert probe.embedding_batch == 8
|
||||
|
||||
|
||||
async def test_live_tavily_search() -> None:
|
||||
_needs_key("TAVILY_API_KEY")
|
||||
adapter = TavilySearchAdapter(_secrets())
|
||||
results = await adapter.search("deep research agent verification", max_results=3)
|
||||
assert 1 <= len(results) <= 3
|
||||
assert all(r.url.startswith("http") for r in results)
|
||||
|
||||
|
||||
async def test_live_liveness_against_real_page() -> None:
|
||||
status = await check_liveness("https://example.org/")
|
||||
assert status in (LivenessStatus.LIVE, LivenessStatus.STALE, LivenessStatus.DEAD)
|
||||
void_httpx = httpx.AsyncClient
|
||||
del void_httpx
|
||||
@@ -190,11 +190,10 @@ def test_ledger_export_gate() -> None:
|
||||
from researchx.trust.liveness import LivenessStatus
|
||||
|
||||
store = EvidenceStore()
|
||||
stored_text = "The verified sentence used for the export gate test."
|
||||
store.add_passage("https://a", stored_text)
|
||||
store.add_passage("https://a", "passage one " * 20)
|
||||
ledger = build_ledger(
|
||||
run_id="abcd2345",
|
||||
report_text=f"{stored_text} [p-1]",
|
||||
report_text="[p-1]",
|
||||
evidence=store,
|
||||
liveness={"https://a": LivenessStatus.LIVE},
|
||||
visited_at=datetime.now(UTC),
|
||||
@@ -205,7 +204,7 @@ def test_ledger_export_gate() -> None:
|
||||
|
||||
ledger_dead = build_ledger(
|
||||
run_id="abcd2345",
|
||||
report_text=f"{stored_text} [p-1]",
|
||||
report_text="[p-1]",
|
||||
evidence=store,
|
||||
liveness={"https://a": LivenessStatus.DEAD},
|
||||
visited_at=datetime.now(UTC),
|
||||
|
||||
@@ -97,10 +97,7 @@ def _store_with_report() -> tuple[EvidenceStore, str]:
|
||||
store = EvidenceStore()
|
||||
for page in CORPUS[:3]:
|
||||
store.add_page(page.url, page.title, page.text)
|
||||
first_passage = store.pages()[0].passage_ids[0]
|
||||
stored = store.get(first_passage)
|
||||
sentence = (stored.text if stored else "").split(".")[0] + "."
|
||||
report = f"# Report\n\n{sentence} [{first_passage}]\n\nMore context [{first_passage}]\n"
|
||||
report = f"# Report\n\nClaim one [{store.pages()[0].passage_ids[0]}] and claim two [p-1].\n"
|
||||
return store, report
|
||||
|
||||
|
||||
@@ -124,10 +121,9 @@ async def test_ledger_build_and_checklist_export(monkeypatch) -> None:
|
||||
visited_at=datetime.now(UTC),
|
||||
)
|
||||
assert ledger.run_id == "abcd2345"
|
||||
cited_url = store.pages()[0].url
|
||||
assert [e.url for e in ledger.entries] == [cited_url]
|
||||
assert ledger.checklist.citations_checked == 1
|
||||
assert ledger.checklist.live == 1
|
||||
assert len(ledger.entries) == len(store.pages())
|
||||
assert ledger.checklist.citations_checked == len(store.pages())
|
||||
assert ledger.checklist.live == len(store.pages())
|
||||
assert ledger.checklist.dead == 0
|
||||
assert ledger.checklist.anchored_claims >= 1
|
||||
assert all(
|
||||
|
||||
@@ -1,12 +1,5 @@
|
||||
# ResearchX Frontend
|
||||
|
||||
**TL;DR** — This is the ResearchX web interface: ask a question, choose a research method, approve the plan, watch the live trace, read the cited report. The prototype runs entirely on scripted sample data — no API keys, no backend, no network — so the full product journey is tryable today.
|
||||
|
||||
- **What** — A Next.js app (ask, choose, run, report, library, settings) rendering mock data through one typed client interface.
|
||||
- **Why** — The journey is checkable end-to-end before live inference exists.
|
||||
- **Action** — no action — FYI. Developers: Quickstart below (Node 20+, pnpm).
|
||||
- **Details** — Root `README.md`: backend engine, live mode, architecture.
|
||||
|
||||
The prototype frontend for ResearchX — methodology-driven deep research
|
||||
with a trust loop. This milestone (v0.1) runs **entirely on mock data**:
|
||||
no API keys, no env vars, no network calls beyond static assets.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@researchx/frontend",
|
||||
"version": "0.2.5",
|
||||
"version": "0.1.8",
|
||||
"private": true,
|
||||
"packageManager": "pnpm@12.6.0",
|
||||
"scripts": {
|
||||
@@ -30,7 +30,6 @@
|
||||
"@vitejs/plugin-react": "4.7.0",
|
||||
"eslint": "^9.39.5",
|
||||
"eslint-config-next": "15.5.6",
|
||||
"eventsource": "^5.1.2",
|
||||
"globals": "^17.0.0",
|
||||
"jsdom": "26.1.0",
|
||||
"openapi-fetch": "^0.13.8",
|
||||
|
||||
Generated
-17
@@ -212,9 +212,6 @@ importers:
|
||||
eslint-config-next:
|
||||
specifier: 15.5.6
|
||||
version: 15.5.6(eslint@9.39.5(jiti@2.7.0)(supports-color@10.2.2))(supports-color@10.2.2)(typescript@5.9.3)
|
||||
eventsource:
|
||||
specifier: ^5.1.2
|
||||
version: 5.1.2
|
||||
globals:
|
||||
specifier: ^17.0.0
|
||||
version: 17.12.0
|
||||
@@ -1858,14 +1855,6 @@ packages:
|
||||
resolution: {integrity: sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g==}
|
||||
engines: {node: '>=0.10.0'}
|
||||
|
||||
eventsource-parser@4.1.1:
|
||||
resolution: {integrity: sha512-zaWNqeSLeyzEAAt2Q6aDBRDRk/VnPmsKqvToyGvv6aahvLv71A4Jw7jG93Hckf11zXurbqgsffvpSgKPoeU1Qw==}
|
||||
engines: {node: '>=22.12'}
|
||||
|
||||
eventsource@5.1.2:
|
||||
resolution: {integrity: sha512-VJnwqqp+fOT3WDkjnUy49Zuyh48+kH0JdJApZkslrRQyx35zIZJeJYJl0+a+LrTzh5TFv6MdmXnBDv71UXFyWQ==}
|
||||
engines: {node: '>=22.12.0'}
|
||||
|
||||
expect-type@1.4.0:
|
||||
resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==}
|
||||
engines: {node: '>=12.0.0'}
|
||||
@@ -4658,12 +4647,6 @@ snapshots:
|
||||
|
||||
esutils@2.0.3: {}
|
||||
|
||||
eventsource-parser@4.1.1: {}
|
||||
|
||||
eventsource@5.1.2:
|
||||
dependencies:
|
||||
eventsource-parser: 4.1.1
|
||||
|
||||
expect-type@1.4.0: {}
|
||||
|
||||
fast-deep-equal@3.1.3: {}
|
||||
|
||||
@@ -107,6 +107,6 @@ describe("SettingsView (REQ-65)", () => {
|
||||
expect(screen.getByText("preview", { exact: true })).toBeInTheDocument();
|
||||
|
||||
expect(screen.getByText(/About & version|About and version/i)).toBeInTheDocument();
|
||||
expect(screen.getByText(/v0\.2\.5 · MVP release candidate/i)).toBeInTheDocument();
|
||||
expect(screen.getByText(/v0\.1\.8 · prototype/i)).toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -244,9 +244,8 @@ export function SettingsView() {
|
||||
visited-only citations, three clicks to a trustworthy report.
|
||||
</p>
|
||||
<p className="mt-2 text-caption text-ink-faint">
|
||||
v0.2.5 · MVP release candidate — mock-backed by default; the
|
||||
engine, adapters, and trust loop ship with v0.2 (live mode:
|
||||
NEXT_PUBLIC_RESEARCHX_API_MODE=live).
|
||||
v0.1.8 · prototype — runs on scripted mock data; the engine,
|
||||
adapters, and trust loop arrive in v0.2.
|
||||
</p>
|
||||
</section>
|
||||
|
||||
|
||||
@@ -1,215 +0,0 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*
|
||||
* E2E happy path on the REAL stack (REQ-17): new user → research
|
||||
* running in ≤3 clicks using defaults only.
|
||||
*
|
||||
* The live ApiClient binding talks to a real FastAPI app (engine +
|
||||
* fakes) through httpx's ASGI transport wired in as fetch. This is
|
||||
* the full boundary → API → engine → trust-loop loop — no mocks in
|
||||
* the data path. Click budget: submit question (1), pick methodology
|
||||
* (2), approve plan (3).
|
||||
*/
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
|
||||
import { EventSource as EventSourcePolyfill } from "eventsource";
|
||||
|
||||
const g = globalThis as unknown as { EventSource?: unknown };
|
||||
if (!g.EventSource) {
|
||||
g.EventSource = EventSourcePolyfill;
|
||||
}
|
||||
|
||||
import { GeneratedApiClient } from "@/lib/api/live-client";
|
||||
|
||||
type Lifecycle = { pauseRun(id: string): Promise<unknown> };
|
||||
|
||||
let client: GeneratedApiClient & Lifecycle;
|
||||
let server: {
|
||||
ready: Promise<void>;
|
||||
close: () => Promise<void>;
|
||||
app: unknown;
|
||||
};
|
||||
|
||||
beforeAll(async () => {
|
||||
const { spawn } = await import("node:child_process");
|
||||
const { resolve } = await import("node:path");
|
||||
|
||||
const backendDir = resolve(process.cwd(), "..", "backend");
|
||||
const child = spawn(
|
||||
"/bin/bash",
|
||||
["-lc", "uv run python -m researchx.api.dev_bridge"],
|
||||
{
|
||||
cwd: backendDir,
|
||||
stdio: ["pipe", "pipe", "inherit"],
|
||||
},
|
||||
);
|
||||
|
||||
const ready = new Promise<void>((resolve, reject) => {
|
||||
const timer = setTimeout(() => reject(new Error("bridge not ready in 60s")), 60_000);
|
||||
let buf = "";
|
||||
child.stdout!.on("data", (chunk: Buffer) => {
|
||||
buf += chunk.toString();
|
||||
if (buf.includes("READY")) {
|
||||
clearTimeout(timer);
|
||||
resolve();
|
||||
}
|
||||
});
|
||||
child.on("error", (err) => {
|
||||
clearTimeout(timer);
|
||||
reject(err);
|
||||
});
|
||||
});
|
||||
|
||||
server = {
|
||||
app: child,
|
||||
ready,
|
||||
close: async () => {
|
||||
child.kill("SIGTERM");
|
||||
},
|
||||
};
|
||||
await ready;
|
||||
|
||||
const waitForPort = async (): Promise<void> => {
|
||||
const deadline = Date.now() + 30_000;
|
||||
while (Date.now() < deadline) {
|
||||
try {
|
||||
const probe = await fetch("http://127.0.0.1:8000/api/config");
|
||||
if (probe.ok) {
|
||||
return;
|
||||
}
|
||||
} catch {
|
||||
// not listening yet
|
||||
}
|
||||
await new Promise((r) => setTimeout(r, 250));
|
||||
}
|
||||
throw new Error("backend bridge never listened on 8000");
|
||||
};
|
||||
await waitForPort();
|
||||
|
||||
const fetchImpl = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||
const req = input instanceof Request ? input : new Request(input, init);
|
||||
const body = req.body ? await new Response(req.body).text() : undefined;
|
||||
const headers: Record<string, string> = {};
|
||||
req.headers.forEach((v, k) => {
|
||||
headers[k] = v;
|
||||
});
|
||||
const response = await fetch("http://127.0.0.1:8000" + new URL(req.url).pathname, {
|
||||
method: req.method,
|
||||
headers,
|
||||
...(body ? { body } : {}),
|
||||
});
|
||||
return response;
|
||||
}) as typeof globalThis.fetch;
|
||||
|
||||
client = new GeneratedApiClient(fetchImpl) as GeneratedApiClient & Lifecycle;
|
||||
}, 90_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await server?.close?.();
|
||||
});
|
||||
|
||||
describe("E2E happy path — real stack, defaults only (REQ-17)", () => {
|
||||
it("runs research in ≤3 user actions", async () => {
|
||||
// Action budget: 1) submit question on Ask, 2) pick methodology on
|
||||
// Choose (defaults preselected — the plan requires zero dial fiddling),
|
||||
// 3) approve plan on Plan & Run.
|
||||
let userActions = 0;
|
||||
|
||||
const methodologies = await client.getMethodologies(); // catalog load — not a click
|
||||
expect(methodologies.length).toBeGreaterThanOrEqual(2);
|
||||
|
||||
const question = "How do verification pipelines keep citations honest?";
|
||||
userActions += 1; // Ask: submit question
|
||||
const run = await client.createRun({
|
||||
question,
|
||||
methodologySlug: "rapid-answer",
|
||||
});
|
||||
expect(run.status).toBeOneOf(["planning", "awaiting_approval"]);
|
||||
|
||||
userActions += 2; // Choose: pick methodology + Plan & Run: approve
|
||||
let current = await client.getRun(run.id);
|
||||
const deadline = Date.now() + 30_000;
|
||||
while (current.status === "planning" && Date.now() < deadline) {
|
||||
current = await client.getRun(run.id);
|
||||
}
|
||||
expect(current.plan.subQuestions.length).toBeGreaterThanOrEqual(1);
|
||||
|
||||
const approved = await client.approvePlan(run.id);
|
||||
expect(approved.status).not.toBe("planning");
|
||||
|
||||
let done = false;
|
||||
const events: string[] = [];
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
const timer = setTimeout(() => reject(new Error("run timed out")), 60_000);
|
||||
const subscription = client.getRunEvents(
|
||||
run.id,
|
||||
(event) => {
|
||||
events.push(event.type);
|
||||
},
|
||||
() => {
|
||||
done = true;
|
||||
clearTimeout(timer);
|
||||
resolve();
|
||||
},
|
||||
);
|
||||
// Safety valve — cancel the subscription if the run hangs.
|
||||
setTimeout(() => {
|
||||
if (!done) {
|
||||
subscription.cancel();
|
||||
clearTimeout(timer);
|
||||
reject(new Error("no run_done within window"));
|
||||
}
|
||||
}, 55_000);
|
||||
});
|
||||
expect(done).toBe(true);
|
||||
expect(events).toContain("status");
|
||||
|
||||
const final = await client.getRun(run.id);
|
||||
expect(final.status).toBe("done");
|
||||
expect(userActions).toBeLessThanOrEqual(3);
|
||||
}, 90_000);
|
||||
|
||||
it("exports round-trip: markdown carries citation links, JSON validates (REQ-34)", async () => {
|
||||
const run = await client.createRun({
|
||||
question: "Export round-trip proof",
|
||||
methodologySlug: "rapid-answer",
|
||||
});
|
||||
const deadline = Date.now() + 30_000;
|
||||
let current = await client.getRun(run.id);
|
||||
while (current.status === "planning" && Date.now() < deadline) {
|
||||
current = await client.getRun(run.id);
|
||||
}
|
||||
await client.approvePlan(run.id);
|
||||
const doneDeadline = Date.now() + 60_000;
|
||||
while (current.status !== "done" && Date.now() < doneDeadline) {
|
||||
current = await client.getRun(run.id);
|
||||
}
|
||||
expect(current.status).toBe("done");
|
||||
|
||||
const markdown = await client.exportReport(run.id, "markdown");
|
||||
expect(markdown.filename.endsWith(".md")).toBe(true);
|
||||
expect(markdown.content).toContain("[p-");
|
||||
const linkRe = /\[https?:\/\/[^\]]+\]\(https?:\/\/[^)]+\)/;
|
||||
expect(markdown.content).toMatch(linkRe);
|
||||
|
||||
const json = await client.exportReport(run.id, "json");
|
||||
expect(json.filename.endsWith(".json")).toBe(true);
|
||||
const parsed = JSON.parse(json.content) as {
|
||||
run_id: string;
|
||||
entries: Array<{
|
||||
url: string;
|
||||
visited_at: string;
|
||||
passages: string[];
|
||||
health: string;
|
||||
authority_tier: string | number;
|
||||
}>;
|
||||
};
|
||||
expect(parsed.run_id).toBe(run.id);
|
||||
expect(parsed.entries.length).toBeGreaterThanOrEqual(1);
|
||||
for (const entry of parsed.entries) {
|
||||
expect(entry.url).toMatch(/^https?:\/\//);
|
||||
expect(entry.health).toBeOneOf(["live", "stale", "dead"]);
|
||||
expect(Array.isArray(entry.passages)).toBe(true);
|
||||
}
|
||||
}, 90_000);
|
||||
});
|
||||
@@ -351,35 +351,19 @@ export class GeneratedApiClient extends MockApiClient {
|
||||
id: string,
|
||||
format: "markdown" | "json",
|
||||
): Promise<{ filename: string; content: string }> {
|
||||
const [report, ledger] = await Promise.all([this.getReport(id), this.getLedger(id)]);
|
||||
if (format === "markdown") {
|
||||
const report = await this.getReport(id);
|
||||
const body = report.sections.map((s) => `## ${s.heading}\n\n${s.body}`).join("\n\n");
|
||||
const citationsBlock = ledger.citations
|
||||
.map(
|
||||
(c, i) =>
|
||||
`${i + 1}. [${c.url}](${c.url}) — health: ${c.verification}, tier: ${c.authorityTier}`,
|
||||
)
|
||||
.join("\n");
|
||||
return {
|
||||
filename: `researchx-${id}.md`,
|
||||
content:
|
||||
`# Research Report\n\n${body}\n\n## Citations\n\n${citationsBlock || "(no citations)"}\n`,
|
||||
};
|
||||
return { filename: `researchx-${id}.md`, content: `# Research Report\n\n${body}\n` };
|
||||
}
|
||||
const [report, ledger] = await Promise.all([this.getReport(id), this.getLedger(id)]);
|
||||
const payload = {
|
||||
run_id: id,
|
||||
generated_at: ledger.generatedAt,
|
||||
entries: ledger.citations.map((c) => ({
|
||||
url: c.url,
|
||||
visited_at: ledger.generatedAt,
|
||||
passages: c.passageId ? [c.passageId] : [],
|
||||
claims_supported: 1,
|
||||
health: c.verification,
|
||||
authority_tier: c.authorityTier === 1 ? "official" : c.authorityTier === 2 ? "reference" : "general",
|
||||
})),
|
||||
checklist: {
|
||||
run_id: id,
|
||||
citations_checked: ledger.citations.length,
|
||||
runId: id,
|
||||
report: report.sections,
|
||||
ledger: {
|
||||
runId: ledger.runId,
|
||||
generatedAt: ledger.generatedAt,
|
||||
citations: ledger.citations,
|
||||
},
|
||||
};
|
||||
return {
|
||||
|
||||
Reference in New Issue
Block a user