ship: v0.2 P5 final review + milestone ship (#14)
This commit was merged in pull request #14.
This commit is contained in:
@@ -40,7 +40,7 @@ reason: "v0.2 is the MVP: FastAPI backend + engine + adapters + trust loop (back
|
||||
|
||||
### security-engineer (phase-specific: P2)
|
||||
- **Domain**: security
|
||||
- **Active**: true (P2 only)
|
||||
- **Active**: false (P2 shipped — removed per phase-specific policy)
|
||||
- **Phase-specific**: true
|
||||
- **Frameworks**: httpx, pytest, STRIDE analysis
|
||||
- **Constraints**: secrets resolve from env-file scopes only (OLLAMA_API_KEY, TAVILY_API_KEY, GITEA_TOKEN via .ciagent/.env.secrets) — never shell env, never logged, never in git; liveness verification and trust-loop code must not leak keys in errors or traces; SSRF guard for fetched URLs (private-address refusal); authority tiering must not phone home
|
||||
|
||||
+26
-43
@@ -1,61 +1,44 @@
|
||||
# PLAN — Milestone v0.2 Phase 4: Hardening + Adversarial Suite + E2E
|
||||
# PLAN — Milestone v0.2 Phase 5 (FINAL): Review + Audit + Ship
|
||||
|
||||
---
|
||||
phase: 4
|
||||
phase: 5
|
||||
milestone: v0.2
|
||||
status: planned
|
||||
status: executed
|
||||
must_haves:
|
||||
- "Adversarial suite (REQ-32): hallucinated-URL writer caught by the verifier (report cites a URL never visited/anchored → flagged in ledger, export gate fails it); quote-anchor mismatch detection (near-miss and fabricated quotes flagged, never silently accepted); pause/resume mid-depth-loop over the HTTP API"
|
||||
- "E2E happy path (REQ-17): new user → research running in ≤3 clicks using defaults only, validated against the REAL stack (FastAPI + engine via ASGI, live ApiClient binding at the boundary) — not the mock"
|
||||
- "Export round-trip (REQ-34): report + ledger + citations survive as real links — markdown export carries citation URLs, JSON export round-trips into pydantic models"
|
||||
- "Live-adapter smoke test tier (opt-in, G-004): keyed tests for Ollama + Tavily marked skippable when keys absent — zero-key CI unaffected"
|
||||
- "Monorepo tooling polish (REQ-33 completion): one-command install/dev documented + root convenience scripts"
|
||||
- "Multi-persona full-milestone review: 4 P0 + 15 P1 + 15 P2 found; all 4 P0s fixed in-phase with attack-proofs (A1 tautological anchoring → real writer-quote extraction; C2 gate denominator → cited-URL scoped; C1 zombie runs → shape-validated plans + catch-all FAILED persist; S1 fetch SSRF → per-hop re-guarding); 6 cheap P1s fixed (approve-with-plan, FAILED status, length caps, probe wiring, README env-var + adapter honesty)"
|
||||
- "Audit: PASS-WITH-NOTES — reconstruction PASS (all REQ-01..34 covered across v0.2.0–v0.2.4), secrets clean (zero key values in history), branch hygiene exact; critical notes fixed in-phase (REQUIREMENTS traceability un-staled, P5 PLAN written, ROADMAP P5 status, security-engineer persona deactivated per its removal note)"
|
||||
- "Milestone ship: v0.2.5 tagged as the MVP release candidate — the pipeline then HALTS at the merge_to_main owner approval gate (D-018)"
|
||||
- "G-009 escalation presented at the gate: the owner ratifies that live mode covers the core research loop only (G-002); remaining prototype surfaces stay mock-backed to v0.3+"
|
||||
personas:
|
||||
- backend-engineer
|
||||
- frontend-engineer
|
||||
- ci-lead-developer
|
||||
requirements: [REQ-17, REQ-32, REQ-34]
|
||||
requirements: [REQ-01, REQ-02, REQ-03, REQ-04, REQ-05, REQ-06, REQ-07, REQ-08, REQ-09, REQ-10, REQ-11, REQ-12, REQ-13, REQ-14, REQ-15, REQ-16, REQ-17, REQ-18, REQ-19, REQ-20, REQ-21, REQ-22, REQ-23, REQ-24, REQ-25, REQ-26, REQ-31, REQ-32, REQ-33, REQ-34]
|
||||
---
|
||||
|
||||
## User-Facing Surface
|
||||
|
||||
The adversarial suite is the deliverable: every trust-loop claim is now attacked by tests that try to fool it. The owner-facing proof is a README section documenting the adversarial guarantees + the E2E validation script. No UI changes.
|
||||
The MVP release candidate: the approved v0.1 prototype frontend (Ask → Choose → Plan & Run → Reading Room, all 8 cards, shell, library, settings) now backed by a real FastAPI research engine in live mode — methodology registry (Rapid Answer + Deep Report), 12-dial framework, asyncio state machine with durable pause/cancel/resume, visited-only evidence store, attack-proven trust loop (quote anchoring, SSRF-guarded liveness, citation ledger, ≥99% export gate with zero-unanchored requirement), SSE live trace, adversarial suite (14 attacks), E2E happy path on the real stack. Releases v0.2.0–v0.2.5.
|
||||
|
||||
## Happy Path (P4 — written before execute)
|
||||
## Happy Path (owner review script)
|
||||
|
||||
1. `cd backend && uv run pytest` — suite now includes adversarial tests: hallucinated URL caught, quote mismatch flagged, mid-loop pause/resume over HTTP
|
||||
2. `uv run pytest -m live` (keys present) or default skip — live smoke tier
|
||||
3. Frontend E2E validation: the live-binding test suite walks Ask → Choose → approve → done with the REAL backend through the ASGI transport — ≤3 user actions
|
||||
4. Export round-trip: export markdown from a finished run, verify citation URLs present; export JSON, re-validate into the ledger model
|
||||
5. All gates green: backend ruff/mypy/pytest, frontend typecheck/lint/test/build
|
||||
1. `cd backend && uv sync && uv run pytest` — 129 tests green, zero API keys (live tier: `uv run pytest -m live`, skips cleanly)
|
||||
2. `cd frontend && corepack pnpm install && corepack pnpm test` — 97 tests green (90 component + 5 live-binding + 2 real-stack E2E)
|
||||
3. `uv run uvicorn researchx.api.app:app --port 8000` + `NEXT_PUBLIC_RESEARCHX_API_MODE=live corepack pnpm dev` → http://localhost:3000 → Ask a question → Research this → pick Deep Report → Start research → approve the plan (3 clicks) → live SSE trace → report with anchored citations → ledger → export Markdown (citation links) / JSON (backend-schema round-trip)
|
||||
4. `uv run python -m researchx.engine.demo` — full pipeline on fakes, artifacts in a temp runs dir
|
||||
5. Owner decision: approve → milestone merge to main + v0.2.5 published as the MVP release; or direct changes → binding fix cycle
|
||||
|
||||
## UX Acceptance Criteria (P4)
|
||||
## UX Acceptance Criteria
|
||||
|
||||
- Trust claims are adversarially proven, not asserted: the suite demonstrates attacks being caught
|
||||
- The 3-click criterion is measured on the real stack, defaults only
|
||||
- Zero-key test discipline holds; live tier is opt-in and never blocks CI
|
||||
- ≤3 clicks from cold start to initiated research, validated on the real stack (REQ-17)
|
||||
- Every trust claim attack-proven: hallucinated citations fail the export gate; fabricated quotes on real passage IDs flag unanchored; SSRF redirects die; pause/resume/cancel survive the loop
|
||||
- Honest scope: live mode = core research loop (G-002); library/settings/suggestions/refine stay mock-backed until v0.3+ — ratified at this gate (G-009)
|
||||
- No secret value in any artifact, trace, or log; zero-key test discipline throughout
|
||||
|
||||
## Tasks (waves)
|
||||
## What this phase did
|
||||
|
||||
### Wave 1 — Adversarial suite (backend-engineer)
|
||||
- **T-4.1** Hallucinated-URL attack: fake writer emits a citation to an unvisited URL; verify the ledger flags it (no anchor, dead health, export gate fails) [REQ-32]
|
||||
- **T-4.2** Quote-anchor attacks: fabricated quotes (no match), near-miss quotes (weak_match), exact quotes (match) — all three verdicts proven against a multi-page corpus [REQ-32]
|
||||
- **T-4.3** Pause/resume mid-depth-loop over HTTP: deep-report run (depth 3) paused during REFLECTING, resumed, completes; also cancel mid-loop [REQ-32]
|
||||
- **T-4.4** Liveness attack surface: private-IP redirect (must die), redirect chain to dead URL, oversized response cap
|
||||
1. **Review** (multi-persona, probe-verified): 4 P0 + 15 P1 + 15 P2. The 4 P0s broke the milestone's flagship guarantees and were all fixed with attack-proofs: quote anchoring extracted the stored passage's own quote (always 1.0) instead of the writer's text; the export-gate denominator counted unprobed visited pages as dead; malformed planner JSON left zombie runs; fetch adapters followed redirects without re-guarding. Six cheap P1s also fixed.
|
||||
2. **Audit**: PASS-WITH-NOTES — reconstruction, secrets sweep (clean), branch/tag hygiene exact; both critical notes (stale traceability rows, missing P5 plan) fixed in-phase.
|
||||
3. **Ship (this commit sequence)**: PR #14 (phase/05 → milestone), squash-merge, tag v0.2.5 — the MVP release candidate, release with the full milestone summary. Per D-018 and the merge_to_main escalation hook, the milestone merge to main does NOT proceed autonomously — the pipeline halts for the recorded owner decision, with G-009 (live-mode scope) presented at the same gate.
|
||||
|
||||
### Wave 2 — E2E + round-trip (frontend-engineer + backend-engineer)
|
||||
- **T-4.5** E2E happy path on the real stack: full-loop test through the live binding (ASGI transport as fetch backend) — create run with defaults → awaiting approval → approve → done → report+ledger fetch; click-budget assertion (3 user actions: submit question, pick methodology, approve plan) [REQ-17]
|
||||
- **T-4.6** Export round-trip: markdown export contains citation URLs as links; JSON export validates back into the ledger model [REQ-34]
|
||||
## Scope Guard
|
||||
|
||||
### Wave 3 — Live tier + polish (ci-lead-developer)
|
||||
- **T-4.7** Opt-in live smoke tests: pytest marker `live`; skip cleanly without OLLAMA_API_KEY/TAVILY_API_KEY; probe + one search when present [G-004]
|
||||
- **T-4.8** README adversarial-guarantees section + root one-command story (scripts + docs) [REQ-33]
|
||||
- **T-4.9** REQ mapping table for verify: REQ-17/32/34 → tests
|
||||
|
||||
## Out of scope for P4
|
||||
- Any UI change; new API routes; methodology additions (G-010 lever untouched — Deep Report ships in this milestone)
|
||||
|
||||
## Verification hooks
|
||||
- Gates: backend ruff/mypy/pytest (adversarial suite included, zero-key); frontend typecheck/lint/test/build + live-binding E2E
|
||||
- REQ coverage: REQ-17 (E2E 3-click), REQ-32 (adversarial), REQ-34 (round-trip)
|
||||
- Security: attacks stay in tests; no live endpoints hit without the marker
|
||||
Milestone merge to main, v0.2.5 publication, and any deploy wait on the owner approval gate (D-018/G-009). The pipeline does not proceed autonomously past this commit.
|
||||
@@ -130,8 +130,8 @@
|
||||
| REQ-63, REQ-64, REQ-65, REQ-66 (app shell, library, settings, avatar) | v0.1 | P5 | complete |
|
||||
| REQ-67, REQ-68 (ask suggestions, refine dialogue) | v0.1 | P7 | complete |
|
||||
| REQ-01..06, REQ-08..11, REQ-31 (framework, adapters) | v0.2 | P1 | complete (v0.2.1; REQ-07/12 contracts proven, live probing in P2) |
|
||||
| REQ-07, REQ-12, REQ-13..24 (probing, engine, trust) | v0.2 | P2 | blocked |
|
||||
| REQ-25, REQ-26, REQ-33 (API, client, tooling) | v0.2 | P3 | blocked |
|
||||
| REQ-17, REQ-32, REQ-34 (adversarial, E2E, round-trip) | v0.2 | P4 | blocked |
|
||||
| All REQ-01..34 (milestone gate) | v0.2 | P5 | blocked |
|
||||
| REQ-07, REQ-12, REQ-13..24 (probing, engine, trust) | v0.2 | P2 | complete (v0.2.2) |
|
||||
| REQ-25, REQ-26, REQ-33 (API, client, tooling) | v0.2 | P3 | complete (v0.2.3) |
|
||||
| REQ-17, REQ-32, REQ-34 (adversarial, E2E, round-trip) | v0.2 | P4 | complete (v0.2.4) |
|
||||
| All REQ-01..34 (milestone gate) | v0.2 | P5 | complete (v0.2.5 — all requirements covered; owner gate pending per D-018) |
|
||||
| REQ-35..51 | v2 backlog (post-MVP milestones) | — | pending |
|
||||
|
||||
+1
-1
@@ -110,7 +110,7 @@ Phases renumbered from the original v1.0 plan (1-based within the milestone). Ta
|
||||
|
||||
### Phase 5 — Final Review + Ship (FINAL)
|
||||
- **Description**: Review + audit + milestone ship; final patch tag IS the MVP release. MAJOR stays 0 — the MVP ships as a 0.x release. v1.0.0 comes only after an owner-approved release decision.
|
||||
- **Status**: blocked → next (sequential phases; unblock as each prior phase ships)
|
||||
- **Status**: in progress (final phase — review complete, audit complete, ship v0.2.5 next; milestone merge HALTS at the owner approval gate per D-018)
|
||||
|
||||
## Milestone Releases
|
||||
- **v0.1.4 = extended-prototype candidate** (superseded by the P5/P6 extension cycle)
|
||||
|
||||
@@ -84,11 +84,13 @@ The test suite needs **no** keys: fake model/search adapters (REQ-31) drive ever
|
||||
The frontend runs mock-backed by default (works standalone, zero keys). To run the core research loop against the real backend:
|
||||
|
||||
```bash
|
||||
# terminal 1 — backend (fake adapters by default; live adapters via env-file keys)
|
||||
# terminal 1 — backend (fake adapters by default; real Ollama/Tavily adapters
|
||||
# are used by the engine in v0.2.5+ wiring — the served API currently ships
|
||||
# fake-adapter runs for demos and tests; see Adapter configuration below)
|
||||
cd backend && uv run uvicorn researchx.api.app:app --reload --port 8000
|
||||
|
||||
# terminal 2 — frontend in live mode
|
||||
cd frontend && RESEARCHX_API_MODE=live corepack pnpm dev
|
||||
cd frontend && NEXT_PUBLIC_RESEARCHX_API_MODE=live corepack pnpm dev
|
||||
```
|
||||
|
||||
Live mode covers the core loop (Ask → Choose → approve plan → live trace → report + ledger); the library, settings persistence, suggestions, refine, and graph surfaces stay mock-backed until v0.3. The API surface: `POST /api/runs`, `GET /api/methodologies`, run lifecycle (`approve/pause/cancel/resume`), `GET /api/runs/{id}/events` (SSE), `GET /api/config`, report + ledger artifacts.
|
||||
|
||||
@@ -3,7 +3,6 @@ import json
|
||||
import httpx
|
||||
|
||||
from researchx.adapters.model import ModelAdapter
|
||||
from researchx.adapters.ollama import OllamaModelAdapter
|
||||
from researchx.adapters.probe import (
|
||||
DEFAULT_EMBEDDING_BATCH,
|
||||
CapabilityProbe,
|
||||
@@ -13,19 +12,25 @@ from researchx.adapters.probe import (
|
||||
from researchx.adapters.types import ModelMessage
|
||||
|
||||
|
||||
async def probe_model_adapter(adapter: OllamaModelAdapter) -> CapabilityProbe:
|
||||
def supports_probe(adapter: ModelAdapter) -> bool:
|
||||
return getattr(adapter, "probing_supported", False)
|
||||
|
||||
|
||||
async def probe_model_adapter(adapter: ModelAdapter) -> CapabilityProbe:
|
||||
models = await adapter.list_models()
|
||||
smoke_ok = await _structured_output_smoke_test(adapter)
|
||||
writer = adapter.roles().writer
|
||||
warning = structured_output_warning(smoke_ok, writer)
|
||||
endpoint = getattr(adapter, "active_base_url", "unknown")
|
||||
fallback_reason = getattr(adapter, "last_fallback_reason", None)
|
||||
return CapabilityProbe(
|
||||
endpoint=adapter.active_base_url,
|
||||
endpoint=str(endpoint),
|
||||
models=models,
|
||||
context_window=cap_context_window(None),
|
||||
embedding_batch=DEFAULT_EMBEDDING_BATCH,
|
||||
structured_outputs_supported=smoke_ok,
|
||||
structured_output_warning=warning,
|
||||
fallback_reason=adapter.last_fallback_reason,
|
||||
fallback_reason=fallback_reason,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ from researchx.adapters.url_guard import url_allowed
|
||||
_DDG_URL = "https://html.duckduckgo.com/html"
|
||||
_SEARCH_TIMEOUT = 10.0
|
||||
_FETCH_TIMEOUT = 10.0
|
||||
_MAX_FETCH_REDIRECTS = 5
|
||||
_BACKOFF_SECONDS = 1.0
|
||||
_USER_AGENT = "researchx/0.2 (local research studio)"
|
||||
_RESULT_RE = re.compile(
|
||||
@@ -93,21 +94,34 @@ class DuckDuckGoSearchAdapter:
|
||||
return re.sub(r"\s+", " ", text).strip()
|
||||
|
||||
async def fetch(self, url: str) -> PageContent:
|
||||
if not url_allowed(url):
|
||||
raise AdapterError("fetch refused: url not allowed (private or non-http scheme)")
|
||||
from urllib.parse import urljoin
|
||||
|
||||
current = url
|
||||
async with httpx.AsyncClient(
|
||||
timeout=_FETCH_TIMEOUT,
|
||||
follow_redirects=True,
|
||||
follow_redirects=False,
|
||||
headers={"User-Agent": _USER_AGENT},
|
||||
) as client:
|
||||
response = await client.get(url)
|
||||
response.raise_for_status()
|
||||
return PageContent(
|
||||
url=url,
|
||||
title=extract_title(response.text),
|
||||
text=extract_text(response.text),
|
||||
fetched_at=datetime.now(UTC),
|
||||
)
|
||||
for _ in range(_MAX_FETCH_REDIRECTS + 1):
|
||||
if not url_allowed(current):
|
||||
raise AdapterError(
|
||||
"fetch refused: url not allowed (private or non-http scheme)"
|
||||
)
|
||||
response = await client.get(current)
|
||||
if 300 <= response.status_code < 400:
|
||||
location = response.headers.get("location")
|
||||
if not location:
|
||||
raise AdapterError("fetch failed: redirect without location")
|
||||
current = urljoin(current, location)
|
||||
continue
|
||||
response.raise_for_status()
|
||||
return PageContent(
|
||||
url=url,
|
||||
title=extract_title(response.text),
|
||||
text=extract_text(response.text),
|
||||
fetched_at=datetime.now(UTC),
|
||||
)
|
||||
raise AdapterError("fetch failed: too many redirects")
|
||||
|
||||
@property
|
||||
def last_degradation(self) -> str | None:
|
||||
|
||||
@@ -12,6 +12,8 @@ _REQUEST_TIMEOUT = 60.0
|
||||
|
||||
|
||||
class OllamaModelAdapter:
|
||||
probing_supported = True
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
secrets: EnvFileSecretsProvider,
|
||||
|
||||
@@ -10,6 +10,7 @@ from researchx.adapters.url_guard import url_allowed
|
||||
_TAVILY_URL = "https://api.tavily.com/search"
|
||||
_SEARCH_TIMEOUT = 10.0
|
||||
_FETCH_TIMEOUT = 10.0
|
||||
_MAX_FETCH_REDIRECTS = 5
|
||||
_USER_AGENT = "researchx/0.2 (local research studio)"
|
||||
_TITLE_RE = re.compile(r"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)
|
||||
_BLOCK_RE = re.compile(r"<(script|style)[^>]*>.*?</\1>", re.IGNORECASE | re.DOTALL)
|
||||
@@ -56,18 +57,31 @@ class TavilySearchAdapter:
|
||||
][:max_results]
|
||||
|
||||
async def fetch(self, url: str) -> PageContent:
|
||||
if not url_allowed(url):
|
||||
raise AdapterError("fetch refused: url not allowed (private or non-http scheme)")
|
||||
from urllib.parse import urljoin
|
||||
|
||||
current = url
|
||||
async with httpx.AsyncClient(
|
||||
timeout=_FETCH_TIMEOUT,
|
||||
follow_redirects=True,
|
||||
follow_redirects=False,
|
||||
headers={"User-Agent": _USER_AGENT},
|
||||
) as client:
|
||||
response = await client.get(url)
|
||||
response.raise_for_status()
|
||||
return PageContent(
|
||||
url=url,
|
||||
title=extract_title(response.text),
|
||||
text=extract_text(response.text),
|
||||
fetched_at=datetime.now(UTC),
|
||||
)
|
||||
for _ in range(_MAX_FETCH_REDIRECTS + 1):
|
||||
if not url_allowed(current):
|
||||
raise AdapterError(
|
||||
"fetch refused: url not allowed (private or non-http scheme)"
|
||||
)
|
||||
response = await client.get(current)
|
||||
if 300 <= response.status_code < 400:
|
||||
location = response.headers.get("location")
|
||||
if not location:
|
||||
raise AdapterError("fetch failed: redirect without location")
|
||||
current = urljoin(current, location)
|
||||
continue
|
||||
response.raise_for_status()
|
||||
return PageContent(
|
||||
url=url,
|
||||
title=extract_title(response.text),
|
||||
text=extract_text(response.text),
|
||||
fetched_at=datetime.now(UTC),
|
||||
)
|
||||
raise AdapterError("fetch failed: too many redirects")
|
||||
@@ -8,12 +8,13 @@ from fastapi.middleware.cors import CORSMiddleware
|
||||
from pydantic import BaseModel, Field, ValidationError
|
||||
from sse_starlette.sse import EventSourceResponse
|
||||
|
||||
from researchx.adapters.capability_probe import probe_model_adapter, supports_probe
|
||||
from researchx.adapters.fakes import CORPUS, FakeModelAdapter, FakeSearchAdapter
|
||||
from researchx.adapters.local_storage import LocalStorageAdapter
|
||||
from researchx.adapters.types import AdapterError, ChatResult
|
||||
from researchx.engine.actor import RunActor, make_run_config
|
||||
from researchx.engine.events import EventBus, RunEvent
|
||||
from researchx.engine.run_state import RunState
|
||||
from researchx.engine.run_state import Plan, RunState
|
||||
from researchx.engine.states import RunStateName
|
||||
from researchx.methodologies import registry
|
||||
|
||||
@@ -28,8 +29,8 @@ _CANNED_WRITER = (
|
||||
|
||||
|
||||
class CreateRunBody(BaseModel):
|
||||
question: str = Field(min_length=1)
|
||||
context: str = ""
|
||||
question: str = Field(min_length=1, max_length=2000)
|
||||
context: str = Field(default="", max_length=8000)
|
||||
methodology: str
|
||||
dial_overrides: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
@@ -76,7 +77,7 @@ _STATUS_MAP = {
|
||||
RunStateName.VERIFYING: "verifying",
|
||||
RunStateName.DONE: "done",
|
||||
RunStateName.CANCELLED: "cancelled",
|
||||
RunStateName.FAILED: "cancelled",
|
||||
RunStateName.FAILED: "failed",
|
||||
}
|
||||
|
||||
_ETA_TABLE = {"rapid-answer": (2, "low"), "deep-report": (12, "medium")}
|
||||
@@ -170,11 +171,15 @@ def create_app(
|
||||
|
||||
@app.get("/api/config")
|
||||
async def config_status() -> ConfigStatus:
|
||||
latest_probe = None
|
||||
for run_id in list(runs._actors)[-1:]:
|
||||
actor_probe = runs.actor(run_id).state.probe
|
||||
latest_probe = actor_probe
|
||||
return ConfigStatus(
|
||||
model_adapter=type(model).__name__,
|
||||
search_adapter=type(search).__name__,
|
||||
storage_dir=str(storage.run_dir("00000000").parent),
|
||||
probe=None,
|
||||
probe=latest_probe,
|
||||
cost_counters=runs.cost_counters,
|
||||
)
|
||||
|
||||
@@ -222,6 +227,15 @@ def create_app(
|
||||
registry=registry,
|
||||
event_bus=bus,
|
||||
)
|
||||
probe_data: dict[str, Any] | None = None
|
||||
if supports_probe(model):
|
||||
try:
|
||||
probe = await probe_model_adapter(model)
|
||||
probe_data = probe.model_dump(mode="json")
|
||||
await actor.attach_probe(probe_data)
|
||||
except AdapterError:
|
||||
probe_data = None
|
||||
|
||||
task = asyncio.create_task(actor.run())
|
||||
runs.add(run_id, actor, task, bus)
|
||||
|
||||
@@ -244,7 +258,13 @@ def create_app(
|
||||
if not runs.has(run_id):
|
||||
raise HTTPException(status_code=404, detail="run not found")
|
||||
actor = runs.actor(run_id)
|
||||
await actor.approve()
|
||||
edited_plan: Plan | None = None
|
||||
if body and body.plan:
|
||||
try:
|
||||
edited_plan = Plan.model_validate(body.plan)
|
||||
except ValidationError:
|
||||
raise HTTPException(status_code=422, detail="invalid plan")
|
||||
await actor.approve(edited_plan)
|
||||
return _summary(run_id)
|
||||
|
||||
@app.post("/api/runs/{run_id}/pause")
|
||||
|
||||
@@ -112,6 +112,15 @@ class RunActor:
|
||||
except AdapterError as exc:
|
||||
await self._fail(str(exc))
|
||||
raise EngineError(str(exc)) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
await self._fail(f"network error ({type(exc).__name__})")
|
||||
raise EngineError(f"network error ({type(exc).__name__})") from exc
|
||||
except EngineError:
|
||||
await self._fail("engine error")
|
||||
raise
|
||||
except Exception as exc:
|
||||
await self._fail(f"unexpected engine failure ({type(exc).__name__})")
|
||||
raise EngineError(f"unexpected engine failure ({type(exc).__name__})") from exc
|
||||
return self.state
|
||||
|
||||
async def attach_probe(self, probe_data: dict[str, Any]) -> None:
|
||||
@@ -177,10 +186,21 @@ class RunActor:
|
||||
data = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise AdapterError("planner returned non-JSON plan") from exc
|
||||
if not isinstance(data, dict):
|
||||
raise AdapterError("planner plan is not a JSON object")
|
||||
sub_questions = data.get("sub_questions", [])
|
||||
source_universe = data.get("source_universe", [])
|
||||
outline = data.get("outline", [])
|
||||
if not (
|
||||
isinstance(sub_questions, list)
|
||||
and isinstance(source_universe, list)
|
||||
and isinstance(outline, list)
|
||||
):
|
||||
raise AdapterError("planner plan has malformed lists")
|
||||
return Plan(
|
||||
sub_questions=[str(q) for q in data.get("sub_questions", [])],
|
||||
source_universe=[str(u) for u in data.get("source_universe", [])],
|
||||
outline=[str(o) for o in data.get("outline", [])],
|
||||
sub_questions=[str(q) for q in sub_questions if isinstance(q, str)],
|
||||
source_universe=[str(u) for u in source_universe if isinstance(u, str)],
|
||||
outline=[str(o) for o in outline if isinstance(o, str)],
|
||||
)
|
||||
|
||||
async def _approval_phase(self) -> None:
|
||||
|
||||
@@ -49,17 +49,41 @@ def anchor_quote(quote: str, passages: list[Passage]) -> AnchorVerdict:
|
||||
)
|
||||
|
||||
|
||||
def extract_quotes_from_report(report_text: str, evidence: EvidenceStore) -> list[str]:
|
||||
marker_re = re.compile(r"\[p-(\d+)\]")
|
||||
quotes: list[str] = []
|
||||
for match in marker_re.finditer(report_text):
|
||||
_MARKER_RE = re.compile(r"\[p-(\d+)\]")
|
||||
_HEADER_RE = re.compile(r"^#{1,6}\s.*$", re.MULTILINE)
|
||||
|
||||
|
||||
def _claim_fragment(before: str) -> str:
|
||||
cleaned = _HEADER_RE.sub("", before).rstrip()
|
||||
fragment_start = max(
|
||||
cleaned.rfind("."), cleaned.rfind("!"), cleaned.rfind("?"), cleaned.rfind("\n")
|
||||
)
|
||||
quote_text = cleaned[fragment_start + 1 :].strip(" \t>\"'*-")
|
||||
if not quote_text:
|
||||
quote_text = cleaned[-200:].strip(" \t>\"'*-")
|
||||
return quote_text
|
||||
|
||||
|
||||
def extract_report_claims(report_text: str) -> list[tuple[str, str]]:
|
||||
"""Extract (passage_id, quote_text) claims from the report.
|
||||
|
||||
The quote text is what the WRITER actually wrote adjacent to the
|
||||
citation marker — the last sentence fragment before each [p-N]
|
||||
marker — NOT the stored passage's own text. This is the adversarial
|
||||
core of REQ-20: a fabricated quote riding a real passage ID must
|
||||
be visible to the anchorer.
|
||||
"""
|
||||
claims: list[tuple[str, str]] = []
|
||||
for match in _MARKER_RE.finditer(report_text):
|
||||
passage_id = f"p-{match.group(1)}"
|
||||
passage = evidence.get(passage_id)
|
||||
if passage:
|
||||
quotes.append(passage.quote)
|
||||
return quotes
|
||||
quote_text = _claim_fragment(report_text[: match.start()])
|
||||
claims.append((passage_id, quote_text))
|
||||
return claims
|
||||
|
||||
|
||||
def extract_quotes_from_report(report_text: str, evidence: EvidenceStore) -> list[str]:
|
||||
return [quote for _pid, quote in extract_report_claims(report_text) if evidence.get(_pid)]
|
||||
|
||||
|
||||
def extract_cited_passage_ids(report_text: str) -> list[str]:
|
||||
marker_re = re.compile(r"\[p-(\d+)\]")
|
||||
return [f"p-{m.group(1)}" for m in marker_re.finditer(report_text)]
|
||||
return [f"p-{m.group(1)}" for m in _MARKER_RE.finditer(report_text)]
|
||||
@@ -7,7 +7,7 @@ from researchx.evidence.store import EvidenceStore, Passage
|
||||
from researchx.trust.anchoring import (
|
||||
anchor_quote,
|
||||
extract_cited_passage_ids,
|
||||
extract_quotes_from_report,
|
||||
extract_report_claims,
|
||||
normalize,
|
||||
)
|
||||
from researchx.trust.liveness import LivenessStatus
|
||||
@@ -80,17 +80,22 @@ def build_ledger(
|
||||
) -> CitationLedger:
|
||||
claims = _claims_per_passage(report_text)
|
||||
all_passages = evidence.writer_context()
|
||||
report_quotes = extract_quotes_from_report(report_text, evidence)
|
||||
report_claims = extract_report_claims(report_text)
|
||||
cited_ids = extract_cited_passage_ids(report_text)
|
||||
known_ids = {p.passage_id for p in all_passages}
|
||||
cited_urls: list[str] = []
|
||||
entries: list[CitationEntry] = []
|
||||
anchored = 0
|
||||
unanchored = 0
|
||||
weak = 0
|
||||
confidence: dict[str, str] = {}
|
||||
|
||||
for quote in report_quotes:
|
||||
verdict = anchor_quote(quote, all_passages)
|
||||
for passage_id, writer_quote in report_claims:
|
||||
passage = evidence.get(passage_id)
|
||||
if passage is None:
|
||||
unanchored += 1
|
||||
continue
|
||||
verdict = anchor_quote(writer_quote, [passage])
|
||||
if verdict.matched:
|
||||
anchored += 1
|
||||
elif verdict.weak:
|
||||
@@ -101,8 +106,13 @@ def build_ledger(
|
||||
for cited_id in cited_ids:
|
||||
if cited_id not in known_ids:
|
||||
unanchored += 1
|
||||
passage = evidence.get(cited_id)
|
||||
if passage is not None and passage.url not in cited_urls:
|
||||
cited_urls.append(passage.url)
|
||||
|
||||
for page in evidence.pages():
|
||||
if page.url not in cited_urls:
|
||||
continue
|
||||
passages = evidence.passages_for_url(page.url)
|
||||
passage_ids = [p.passage_id for p in passages]
|
||||
page_claims = sum(claims.get(pid, 0) for pid in passage_ids)
|
||||
|
||||
@@ -12,7 +12,7 @@ from pathlib import Path
|
||||
import httpx
|
||||
|
||||
from researchx.adapters.fakes import CORPUS, FakeModelAdapter, FakeSearchAdapter
|
||||
from researchx.adapters.types import ChatResult
|
||||
from researchx.adapters.types import AdapterError, ChatResult
|
||||
from researchx.api.app import create_app
|
||||
from researchx.evidence.store import EvidenceStore
|
||||
from researchx.trust.anchoring import anchor_quote
|
||||
@@ -261,8 +261,9 @@ def test_export_round_trip_json_validates_into_backend_model() -> None:
|
||||
from researchx.trust.ledger import CitationLedger
|
||||
|
||||
store = EvidenceStore()
|
||||
store.add_passage("https://example.org/a", "A passage with real content " * 6)
|
||||
report = "# Report\n\nClaim [p-1].\n"
|
||||
stored_text = "A passage with real content for the round-trip proof."
|
||||
store.add_passage("https://example.org/a", stored_text)
|
||||
report = f"# Report\n\n{stored_text} [p-1].\n"
|
||||
ledger = build_ledger(
|
||||
run_id="attack003",
|
||||
report_text=report,
|
||||
@@ -276,3 +277,83 @@ def test_export_round_trip_json_validates_into_backend_model() -> None:
|
||||
checklist_md = export_checklist(ledger)
|
||||
assert "Anchoring gate" in checklist_md
|
||||
assert "PASS" in checklist_md
|
||||
|
||||
|
||||
def test_attack_fabricated_quote_on_real_passage_id_fails() -> None:
|
||||
store = EvidenceStore()
|
||||
stored = store.add_passage(
|
||||
"https://example.org/real",
|
||||
"The genuine finding about verification pipelines is that anchoring works. " * 3,
|
||||
)
|
||||
report = (
|
||||
"# Report\n\n"
|
||||
f"THIS FABRICATED QUOTE WAS NEVER STORED ANYWHERE AT ALL [{stored.passage_id}]\n"
|
||||
)
|
||||
ledger = build_ledger(
|
||||
run_id="attack004",
|
||||
report_text=report,
|
||||
evidence=store,
|
||||
liveness={"https://example.org/real": LivenessStatus.LIVE},
|
||||
visited_at=datetime.now(UTC),
|
||||
)
|
||||
assert ledger.checklist.unanchored_claims >= 1, (
|
||||
"a fabricated quote riding a real passage ID must be flagged (REQ-20)"
|
||||
)
|
||||
passed, _ = liveness_export_gate(ledger)
|
||||
assert passed is False
|
||||
|
||||
|
||||
def test_attack_real_verbatim_quote_on_real_passage_anchors() -> None:
|
||||
store = EvidenceStore()
|
||||
stored_text = (
|
||||
"The genuine finding about verification pipelines is that anchoring works."
|
||||
)
|
||||
stored = store.add_passage("https://example.org/real", stored_text + " " + stored_text)
|
||||
report = f"# Report\n\n{stored_text} [{stored.passage_id}]\n"
|
||||
ledger = build_ledger(
|
||||
run_id="attack005",
|
||||
report_text=report,
|
||||
evidence=store,
|
||||
liveness={"https://example.org/real": LivenessStatus.LIVE},
|
||||
visited_at=datetime.now(UTC),
|
||||
)
|
||||
assert ledger.checklist.anchored_claims == 1
|
||||
assert ledger.checklist.unanchored_claims == 0
|
||||
passed, _ = liveness_export_gate(ledger)
|
||||
assert passed is True
|
||||
|
||||
|
||||
async def test_attack_search_result_redirecting_to_private_ip_refused(monkeypatch) -> None:
|
||||
import tempfile
|
||||
from pathlib import Path as PathMod
|
||||
|
||||
import httpx as httpx_mod
|
||||
|
||||
from researchx.adapters.secrets import EnvFileSecretsProvider
|
||||
from researchx.adapters.tavily import TavilySearchAdapter
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
env_file = PathMod(tmpdir) / ".env.secrets"
|
||||
env_file.write_text("TAVILY_API_KEY=t-key\n", encoding="utf-8")
|
||||
secrets = EnvFileSecretsProvider(paths=[env_file])
|
||||
adapter = TavilySearchAdapter(secrets)
|
||||
|
||||
def handler(request: httpx_mod.Request) -> httpx_mod.Response:
|
||||
if "boobytrap" in str(request.url):
|
||||
return httpx_mod.Response(
|
||||
302, headers={"location": "http://169.254.169.254/latest/meta-data"}
|
||||
)
|
||||
return httpx_mod.Response(200, text="<html><body>ok</body></html>")
|
||||
|
||||
transport = httpx_mod.MockTransport(handler)
|
||||
original = httpx_mod.AsyncClient
|
||||
monkeypatch.setattr(
|
||||
httpx_mod, "AsyncClient", lambda **kw: original(transport=transport, **kw)
|
||||
)
|
||||
|
||||
try:
|
||||
await adapter.fetch("https://example.org/boobytrap")
|
||||
except AdapterError as exc:
|
||||
assert "refused" in str(exc) or "not allowed" in str(exc)
|
||||
else:
|
||||
raise AssertionError("fetch of a private-redirecting URL must be refused")
|
||||
|
||||
@@ -190,10 +190,11 @@ def test_ledger_export_gate() -> None:
|
||||
from researchx.trust.liveness import LivenessStatus
|
||||
|
||||
store = EvidenceStore()
|
||||
store.add_passage("https://a", "passage one " * 20)
|
||||
stored_text = "The verified sentence used for the export gate test."
|
||||
store.add_passage("https://a", stored_text)
|
||||
ledger = build_ledger(
|
||||
run_id="abcd2345",
|
||||
report_text="[p-1]",
|
||||
report_text=f"{stored_text} [p-1]",
|
||||
evidence=store,
|
||||
liveness={"https://a": LivenessStatus.LIVE},
|
||||
visited_at=datetime.now(UTC),
|
||||
@@ -204,7 +205,7 @@ def test_ledger_export_gate() -> None:
|
||||
|
||||
ledger_dead = build_ledger(
|
||||
run_id="abcd2345",
|
||||
report_text="[p-1]",
|
||||
report_text=f"{stored_text} [p-1]",
|
||||
evidence=store,
|
||||
liveness={"https://a": LivenessStatus.DEAD},
|
||||
visited_at=datetime.now(UTC),
|
||||
|
||||
@@ -97,7 +97,10 @@ def _store_with_report() -> tuple[EvidenceStore, str]:
|
||||
store = EvidenceStore()
|
||||
for page in CORPUS[:3]:
|
||||
store.add_page(page.url, page.title, page.text)
|
||||
report = f"# Report\n\nClaim one [{store.pages()[0].passage_ids[0]}] and claim two [p-1].\n"
|
||||
first_passage = store.pages()[0].passage_ids[0]
|
||||
stored = store.get(first_passage)
|
||||
sentence = (stored.text if stored else "").split(".")[0] + "."
|
||||
report = f"# Report\n\n{sentence} [{first_passage}]\n\nMore context [{first_passage}]\n"
|
||||
return store, report
|
||||
|
||||
|
||||
@@ -121,9 +124,10 @@ async def test_ledger_build_and_checklist_export(monkeypatch) -> None:
|
||||
visited_at=datetime.now(UTC),
|
||||
)
|
||||
assert ledger.run_id == "abcd2345"
|
||||
assert len(ledger.entries) == len(store.pages())
|
||||
assert ledger.checklist.citations_checked == len(store.pages())
|
||||
assert ledger.checklist.live == len(store.pages())
|
||||
cited_url = store.pages()[0].url
|
||||
assert [e.url for e in ledger.entries] == [cited_url]
|
||||
assert ledger.checklist.citations_checked == 1
|
||||
assert ledger.checklist.live == 1
|
||||
assert ledger.checklist.dead == 0
|
||||
assert ledger.checklist.anchored_claims >= 1
|
||||
assert all(
|
||||
|
||||
Reference in New Issue
Block a user