Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 coreci
|
8b92b23a3f | ||
|
coreci
|
357bd22ff2 | ||
|
coreci
|
547a7b812d |
@@ -0,0 +1,48 @@
|
||||
# ARCHITECTURE — CoreCI Broker
|
||||
|
||||
## Overview
|
||||
|
||||
One Open WebUI workspace tool, implemented as a single Python file (`coreci_broker.py`), embedding a stdlib HTTP server (`ThreadingHTTPServer`) bound to `BROKER_BIND` (default `127.0.0.1:8765`). It serves a JSON API, a four-page admin web UI (static HTML/CSS/JS from `coreci_broker_ui/`), and a public static client module. SQLite (`broker.db`, mode 0600) is the source of truth; an append-only JSONL file (`broker.audit.log`, mode 0600) records every read and denied attempt. The broker is the trust root with no upstream auth; it reads the `creds-coreci-coo` KB through a pluggable adapter and never writes to it.
|
||||
|
||||
Two runtime modes share the same file: (a) Open WebUI workspace tool — a `Tools` class starts the server in a daemon thread at construction (`__init__`), behind a once-per-process idempotency guard (the OWUI loader has no activate() hook and re-execs modules on edit; paths resolve from env, never `__file__`); (b) standalone — `python3 coreci_broker.py` for development, testing, and direct operation.
|
||||
|
||||
## Components
|
||||
|
||||
| Name | Description | Boundaries | Depends on |
|
||||
|------|-------------|-----------|------------|
|
||||
| Config bootstrap | Reads/writes the broker `.env` (spec §5.1 contract + documented additions); first-boot generation of `BROKER_INSTANCE_PEPPER` and `BROKER_ADMIN_PASS` (printed to the startup log once, written to `.env` and SQLite) | Owns the `.env` file and the `broker_config` table; never prints secrets after first boot | SQLite store |
|
||||
| SQLite store | Tables: `broker_config`, `app_registrations`, `user_allowlist`, `agent_registrations`, `admin_sessions` + indexes (spec §5.2); mode 0600 | Sole writer of `broker.db`; all SQL lives here | — |
|
||||
| Audit writer/reader | Appends one JSON line per action (register/read/denied/login/logout/add_app/remove_app/add_user/remove_user); filtered query endpoint (since/until/user_id/uuid/action/limit) | Owns `broker.audit.log`; append-only; no rotation in v0.1 | — |
|
||||
| Hasher | Dual-mode slow hash: bcrypt when importable, else stdlib `hashlib.scrypt`; stored hashes format-prefixed (`$2b$…` / `scrypt$…`) so verification is format-aware | The only component with an optional import | — |
|
||||
| Token engine | Server-side derivation: `token = base64(sha256(pepper:UUID:user_id:agent_id:','.join(sorted(tags))))` | Pure function of (pepper, inputs); never stores plaintext tokens | Hasher, Config |
|
||||
| HTTP server + router | stdlib threaded HTTP; JSON body/param handling; `{"error, message}` contract with 400/401/403/404/409/500; static file serving (UI + client module) | No business logic; routes to admin/tool planes | All handlers |
|
||||
| Admin plane | Session auth (cookie `broker_session` + HTTP Basic); login/logout/whoami; app_registrations CRUD (add/revoke/reinstate); user_allowlist CRUD; audit query | Admin-only; audited | Store, Audit |
|
||||
| Tool auth plane | `POST /api/broker/agents/register` (allowlist gates, idempotent re-derive, TTL, pepper rule); App-auth verification (`Authorization: App <UUID> <token>` — UUID-scoped slow-hash scan + in-memory verified-token cache); `GET /agents/<id>`; state GET/PUT with identity enforcement | Register is callable without a valid token (gates are the allowlists); the cache is never authoritative | Store, Token engine, Hasher |
|
||||
| Secrets plane | `GET /api/broker/secrets/<path>`; scope check `path == "coreci/" + agent_id`; creds-source adapter (FileCredsSource default \| OwuiApiCredsSource) | The only reader of `creds-coreci-coo`; read-only | Tool auth plane, Creds adapter |
|
||||
| Agent-state store | Backs `GET/PUT /api/broker/agents/<id>/state`; toggle `BROKER_AGENT_STATE_BACKEND=internal` (rows in `broker.db`) `\| owui` (real per-agent KBs via OWUI API) | Broker-owned; tool-facing; identity-checked | Tool auth plane |
|
||||
| Static UI | Four pages (login, app registrations, user allowlist, audit) + dashboard + persistent nav; server-rendered HTML, no build step | Read-only against the admin API | Admin plane |
|
||||
| Client module | `coreci_broker_client.py` — dependency-free (stdlib urllib) auth client: ensure-registered, state get/put, secret read with 401→re-register→retry-once; served unauthenticated at `/static/` | Contains no secrets; importable by any OWUI tool | Broker API (external) |
|
||||
| Reference tools | `reference_tools/` — OWUI tool classes showcasing auth-first-then-use (e.g., proxmox reference: register → fetch `coreci/proxmox` → downstream call, safe dry-run default) | Templates for tool authors; not wired to live infra in tests | Client module |
|
||||
|
||||
## DataFlow
|
||||
|
||||
1. **Admin**: CEO logs in via `/ui/login` (or HTTP Basic) → session row in `admin_sessions` → manages allowlists via UI → every admin action audited.
|
||||
2. **Registration**: tool calls `POST /api/broker/agents/register` with `{agent_id, app_registration_uuid, user_id, tags}` → broker checks `app_registrations` (active) and `user_allowlist` (active) → 403 on unknown/revoked → broker derives the token from the pepper + inputs → stores slow-hash in `agent_registrations`, sets TTL expiry → returns token (+ pepper when the row is created or the caller presented no valid token) → tool persists via `PUT /agents/<id>/state`.
|
||||
3. **Secret read**: tool calls `GET /api/broker/secrets/coreci/<agent_id>` with `Authorization: App <UUID> <token>` → broker parses UUID, scans that tool's registration rows with the slow hash (in-memory cache after first verify) → 401 on failure → scope check `path == coreci/<agent_id>` → 403 on mismatch → creds adapter reads `creds-coreci-coo` → audit line (ok/denied/error) → JSON response.
|
||||
4. **Expiry**: after `BROKER_TOKEN_TTL` the token verifies but the row is expired → 401 → tool re-registers (step 2) → new token overwrites `token_hash` → audit `register`.
|
||||
|
||||
## BuildOrder
|
||||
|
||||
1. Config bootstrap + SQLite store + audit writer (P1)
|
||||
2. Hasher + token engine (P1)
|
||||
3. HTTP server + router + error contract + OWUI/standalone mounts (P1)
|
||||
4. Admin plane + audit query + static UI (P2)
|
||||
5. Tool auth plane + agent-state store (P3)
|
||||
6. Secrets plane + creds adapter + client module + reference tools (P4)
|
||||
7. Acceptance matrix, security review, README (P5)
|
||||
|
||||
## Notes
|
||||
|
||||
- The in-memory verified-token cache maps `(uuid, token) -> registration row`; it is populated on first successful slow-hash verification and cleared on restart. SQLite remains authoritative; cache entries are re-validated against row status/expiry on every hit.
|
||||
- Creds adapter contract: lookup by path string (`coreci/<agent_id>`) → JSON object. FileCredsSource reads a single JSON document mapping paths to secret objects (default for dev/tests, `BROKER_CREDS_FILE`). OwuiApiCredsSource calls the Open WebUI HTTP API with `BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY` and is exercised in production; its exact KB-file layout is resolved in P4 research.
|
||||
- This file is the init-time snapshot; it is refined at phase boundaries per ci-files discipline.
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"phase": 1,
|
||||
"stage": "verify",
|
||||
"milestone": "v0.1",
|
||||
"phase_role": "execution",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-10-09T23:58:00Z"
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
# CIAgent Grill Report
|
||||
|
||||
## Run: 2026-10-09 23:10 (mode: interactive, focus: all)
|
||||
|
||||
### Verdict: Proceed (confidence: 0.85)
|
||||
|
||||
7 of 9 axes resolved on evidence; the four escalated items (Axes 7, 9) were resolved interactively by the owner on 2026-10-09 — decisions G-005..G-008, zero open escalations.
|
||||
|
||||
### Axis 1 — Business Case
|
||||
- **Q1**: Is the per-tool credential problem real and top-priority?
|
||||
- Evidence: PROJECT.md:9,47 (four live creds via KB/env/0600 JSON today; proxmox tool exists now — no revocation, no audit, no allowlist); ROADMAP.md:5 (owner-directed 2026-10-09)
|
||||
- Answer: real and current — the estate's path has no revocation or audit; owner directed same-day. (0.85)
|
||||
- **Q2**: Named sponsor who has decided under pressure?
|
||||
- Evidence: D-007 owner-confirmed §13 defaults @0.95 (ffdc539); D-012..D-015 forced under clarify pressure (53d99e8)
|
||||
- Answer: CEO-owner is sole sponsor and decision-maker; pressure decisions on the record. (0.85)
|
||||
- **Q3**: What breaks if cancelled — and is ROI evidence-based?
|
||||
- Evidence: PROJECT.md:9 (counterfactual explicit); no financial deck exists; cost is tokens only
|
||||
- Answer: cancellation returns the estate to unaudited per-tool reads (capability loss, not continuity); ROI is operational, measured against the documented counterfactual. (0.80)
|
||||
|
||||
### Axis 2 — Scope
|
||||
- **Q1**: Stable, frozen, owned?
|
||||
- Evidence: REQ-01..REQ-20 frozen from spec v0.5, full traceability (REQUIREMENTS.md:5,78-99); D-008; "No requirement is deferred" (PROJECT.md:27); owner coreci-coo (REQUIREMENTS.md:3)
|
||||
- Answer: frozen and traced; one acknowledged in-scope unknown (KB file layout → P4, ARCHITECTURE.md:47) with candidate flows pre-verified (RESEARCH.md:55). (0.85)
|
||||
- **Q2**: Explicitly out of scope?
|
||||
- Evidence: 12-row table, each row with reason + spec source (REQUIREMENTS.md:57-74)
|
||||
- Answer: exhaustive; covers the usual silent creep targets (rotation, HA, MFA, federation). (0.90)
|
||||
- **Q3**: Hidden requirements (security, ops, accessibility)?
|
||||
- Evidence: security reviews P3-06/P5-02 (PLAN.md:113,145); traversal research (RESEARCH.md:62); D-013 cookie hardening; REQ-15 0600; ops README P5-04
|
||||
- Answer: security/ops surfaced pre-code; accessibility unstated — immaterial for a single-admin internal UI in v0.1. (0.75)
|
||||
|
||||
### Axis 3 — Architecture
|
||||
- **Q1**: Validated by the builders, not the sellers?
|
||||
- Evidence: RESEARCH.md §4 source-verified against OWUI source at main (no activate() hook, Valves rule, `__file__` temp-delete; KB endpoints from knowledge.py/files.py); plan-checker PASS_WITH_NOTES (b8122be)
|
||||
- Answer: yes — the executing pipeline authored and checked the plan against source. (0.80)
|
||||
- **Q2**: Integration surface bounded?
|
||||
- Evidence: exactly two external surfaces — OWUI Tools mount (RESEARCH.md:30-38) + OWUI KB read API (RESEARCH.md:40-55); gitea push per sibling convention (RESEARCH.md §3)
|
||||
- Answer: bounded and source-verified; no third integration exists. (0.80)
|
||||
- **Q3**: Replaced-system edge cases; inherited debt?
|
||||
- Evidence: four creds, single host, no volume concern (PROJECT.md:47); spec defects 1-4 found and resolved pre-code (RESEARCH.md §2; D-004/D-005); greenfield repo (acf5557)
|
||||
- Answer: edge cases (pepper chicken-and-egg, KB-loss, re-reg KeyError) resolved by decision; external debt = OWUI lifecycle quirks, mitigated by the once-per-process guard (ARCHITECTURE.md:7); 800-line budget fails fast via AC14 at P1-08. (0.75)
|
||||
|
||||
### Axis 4 — People
|
||||
- **Q1**: Key-person dependency — acceptable?
|
||||
- Evidence: D-007 CEO sole approver, binding (PROJECT.md:61); tokens re-register on TTL; allowlist changes are the only CEO-gated act
|
||||
- Answer: accepted — CEO absence stalls new onboarding only, not broker continuity; by design for a v0.1 credential broker on a trusted single host. (0.75)
|
||||
- **Q2**: Resources allocated; PO with authority?
|
||||
- Evidence: agent pipeline, sequential, parallelization off (config.json:33-37); guided autonomy, escalation_hooks ["*"] (config.json:5-9)
|
||||
- Answer: no BAU contention applies; owner-CEO is PO with absolute authority. (0.80)
|
||||
- **Q3**: Building capability they don't have?
|
||||
- Evidence: sibling precedent on same pipeline + forge (RESEARCH.md §3); the one novel surface (OWUI mount) is source-researched
|
||||
- Answer: capability precedented; unproven only in that zero code exists — P1 is the cheap falsifier. (0.70)
|
||||
|
||||
### Axis 5 — Timeline
|
||||
- **Q1**: Deadline set before or after scope?
|
||||
- Evidence: spec v0.5 (ffdc539, 22:37) precedes plan (b8122be, 23:02); no external deadline exists anywhere in the corpus
|
||||
- Answer: spec-first; no reverse-engineered date to fail against. (0.85)
|
||||
- **Q2**: Critical path; single 3-month pusher?
|
||||
- Evidence: P1→P5 sequential (ROADMAP.md:24-57); only live-environment unknown is OwuiApiCredsSource KB layout (P4), bounded by FileCredsSource default + CEO-controlled KB structure
|
||||
- Answer: no 3-month pusher; worst case is bounded P4 adapter rework. (0.75)
|
||||
- **Q3**: Evidence-based estimates; definition of done?
|
||||
- Evidence: 30 one-commit tasks, tests ship with task (PLAN.md:12); must_haves frontmatter (PLAN.md:2-7); AC1-AC14 + per-phase exits (ROADMAP)
|
||||
- Answer: slice-sized with checker sign-off; done is test-enforced, not demo-defined. (0.80)
|
||||
|
||||
### Axis 6 — Budget
|
||||
- **Q1**: Cost profile and burn?
|
||||
- Evidence: token/compute only; phase 0 complete in ~30 min wall-clock (5 commits); no cash budget exists
|
||||
- Answer: no runway question; pipeline runs on owned infra. (0.75)
|
||||
- **Q2**: Predictable drivers outside the budget?
|
||||
- Evidence: stdlib-only eliminates licensing (D-009); forge org exists (RESEARCH.md:9); OWUI self-hosted; no pip, no npm
|
||||
- Answer: none uncovered — the design removes the usual cost drivers by construction. (0.80)
|
||||
- **Q3**: Contingent on something unhappened?
|
||||
- Evidence: forge repo creation untested (assumption d), but release_blocking false (config.json:24) and manual forge-UI fallback is trivial
|
||||
- Answer: no financial contingency; the sole contingency is operational (→ Axis 7). (0.70)
|
||||
|
||||
### Axis 7 — Risks
|
||||
- **Q1**: Top assumptions, evidence per item?
|
||||
- (a) bcrypt ships in OWUI runtime + daemon-thread tools allowed: RESEARCH.md §4 source+docs — but validated at `main`, deployed version unpinned; absorbed by dual-mode scrypt fallback (0.70)
|
||||
- (b) stdlib dev parity with the OWUI runtime: D-001 dual-mode covers both environments (RESEARCH.md:8) (0.80)
|
||||
- (c) FileCredsSource makes tests hermetic: tempdir + env-var state (RESEARCH.md §6; PLAN P1-01/P4-01) — plan-level, proven at P1 (0.70)
|
||||
- (d) forge repo creatable with current token: token valid, org exists, repo 404 — creation UNTESTED (RESEARCH.md:9). **→ ESCALATION**
|
||||
- **Q2**: Dependencies outside the team?
|
||||
- Evidence: OWUI (host + API + process perms), gitea — all in daily estate use (PROJECT.md:47; RESEARCH.md §3)
|
||||
- Answer: bounded and precedented. (0.75)
|
||||
- **Q3**: Single killer risk; pre-mortem?
|
||||
- Evidence: worst case = mount or adapter fails on the live host → standalone mode is functionally equivalent (ARCHITECTURE.md:7); defect table (RESEARCH.md §2) is the pre-mortem substance
|
||||
- Answer: no project-killing risk found on evidence — but NO plan task validates against the live OWUI host before ship. **→ ESCALATION** (0.65)
|
||||
|
||||
### Axis 8 — Governance
|
||||
- **Q1**: Decision-maker under disagreement?
|
||||
- Evidence: single owner-CEO; guided autonomy escalates every gate (ROADMAP.md:9; config.json:6)
|
||||
- Answer: no committee exists to deadlock; owner is arbiter. (0.85)
|
||||
- **Q2**: Cadence, escalation, omissions from status?
|
||||
- Evidence: ---ci--- blocks with confidence on all four commits (ffdc539..b8122be); 15 decisions recorded; checker notes published, not buried
|
||||
- Answer: decision log is exemplary; the mild contradiction is RESEARCH.md:82 "no genuine blockers" sitting beside untested assumption (d). (0.70)
|
||||
- **Q3**: Stop-the-project trigger?
|
||||
- Evidence: MISSING as a formal trigger — but sponsor = approver = decision-maker is one person
|
||||
- Answer: implicit owner prerogative suffices where one person holds all three roles. (0.70)
|
||||
|
||||
### Axis 9 — Adoption
|
||||
- **Q1**: Who uses it; what's in it for them?
|
||||
- Evidence: CEO gets allowlist + audit UI replacing blind KB edits; tool authors get a zero-dep client replacing hand-rolled 0600 JSON (REQ-19; PROJECT.md:9)
|
||||
- Answer: both user classes shed concrete work — not a "train them and pray" case. (0.75)
|
||||
- **Q2**: Ops handoff; rollback plan?
|
||||
- Evidence: ops = owner; README + backup guidance P5-04 (PLAN.md:150); full state = broker.db + broker.audit.log + .env (PROJECT.md:37-39); unmigrated tools never depend on the broker
|
||||
- Answer: rollback = stop process, restore three files; migrated proxmox tool reverts to direct reads. (0.75)
|
||||
- **Q3**: Success criteria validated with the judges?
|
||||
- Evidence: judge = owner via AC1-AC14 (spec §12, owner-owned); but every OWUI-facing test is stub-driven (PLAN P3-04, P4-01), no task touches the live host, and proxmox migration is "post-broker", dateless (PROJECT.md:47)
|
||||
- Answer: acceptance is owner-validated; live readiness and the adoption criterion were escalated and are now resolved as G-005/G-008. (0.85)
|
||||
|
||||
### Meta
|
||||
- **Auditor's flag**: no live-host validation before ship; forge repo creation untested; deployed OWUI version unpinned; adoption criterion undefined. All cheap to fix; none fatal.
|
||||
- **Not being done**: live smoke test on the OWUI host; OWUI version pin; a risk register; post-ship adoption criteria.
|
||||
- **Simplest 80% version**: drop the admin UI + reference tools (CLI allowlists, file creds only) saves ~one phase — but the UI is the CEO's core value and OwuiApiCredsSource is the production path; at 30 tasks / 800 lines, the over-scope charge does not stick.
|
||||
- **90-day must-be-true**: broker live on the OWUI host; proxmox tool on the client module; allowlist + audit in CEO daily use. Not true today (no code) — but every gating factor is internal.
|
||||
|
||||
### Binding Decisions
|
||||
| ID | Axis | Decision | Confidence |
|
||||
|----|------|----------|------------|
|
||||
| G-001 | 4 | Key-person dependency accepted for v0.1: sole-approver CEO absence stalls new onboarding only, not continuity (D-007 by design) | 0.75 |
|
||||
| G-002 | 7 | Assumptions (a)-(c) accepted as mitigated-by-design (dual-mode hashing, hermetic test plan); (d) + live validation escalate | 0.70 |
|
||||
| G-003 | 9 | Rollback accepted: full state = broker.db + broker.audit.log + .env; stop-and-restore; tools revert to direct reads | 0.75 |
|
||||
| G-004 | 3 | 800-line budget risk accepted: AC14 enforced from P1-08 (fails fast); spec §12.14 amendment is the owner's pressure valve | 0.70 |
|
||||
| G-005 | 9 | Post-ship live-OWUI smoke checklist (owner-run: mount, register, read) is the plan of record; v0.1 ships on standalone + stub tests; no real tool migrates before the checklist passes; rollback = disable the tool | 0.85 |
|
||||
| G-006 | 7 | Deployed OWUI pinned at v0.11.3 (owner-stated 2026-10-09) — same-era as the researched `main` surface, no re-research required; smoke checklist item 1 re-confirms the two KB read endpoints on v0.11.3 | 0.85 |
|
||||
| G-007 | 7 | Forge repo creation is tested at the P0 ship (POST /orgs/coreci/repos with the existing token), not deferred to P5-06; on scope failure: local-only fallback + auto-escalation, and the owner's 30-second manual forge-UI creation is the recorded fallback | 0.9 |
|
||||
| G-008 | 9 | 90-day adoption criterion: proxmox_readonly_tool.py runs on coreci_broker_client.py with direct KB reads removed by 2027-01-09; owner owns the date; recorded in ROADMAP as a post-milestone note (out of v0.1 scope) | 0.8 |
|
||||
|
||||
### Escalations
|
||||
- (none — all four escalated challenges resolved interactively by the owner on 2026-10-09; decisions G-005..G-008 above)
|
||||
@@ -0,0 +1,60 @@
|
||||
# PERSONAS — CoreCI Broker
|
||||
|
||||
RESEARCH-stage roster (2026-10-09, milestone v0.1). Pure stdlib Python 3.11 project: single-file broker, no package.json, no JS framework, no src/ tree; the only UI is server-rendered static HTML/CSS/JS from `coreci_broker_ui/` (P2). Execution is sequential (parallelization off), so phase-specific personas activate only at their phase boundaries per this roster.
|
||||
|
||||
---
|
||||
name: backend-engineer
|
||||
domain: backend
|
||||
active: true
|
||||
reason: Primary execution persona — the entire v0.1 product is Python on one file: P1 core skeleton, P2 admin-plane API, P3 tool auth plane, P4 secrets adapter + public client module + reference tools. Owns every security-critical code path (token derivation, slow-hash verification, allowlist gates, audit trail, path-scoped reads) and every test that gates a phase.
|
||||
frameworks: [python3.11, http.server, sqlite3, hashlib.scrypt, unittest]
|
||||
constraints: ["coreci_broker.py stays one file under 800 lines (REQ-17, AC14)", "stdlib-only runtime; bcrypt is the sole optional import; no pip, no npm (D-001/D-009)", "all errors use the {error, message} JSON contract on 400/401/403/404/409/500", "broker.db and broker.audit.log are created mode 0600 (REQ-15)", "the broker never writes to creds-coreci-coo (REQ-08)"]
|
||||
territory: [coreci_broker.py, tests/**, coreci_broker_client.py, reference_tools/**]
|
||||
subagent_type: backend-engineer
|
||||
---
|
||||
|
||||
---
|
||||
name: frontend-engineer
|
||||
domain: frontend
|
||||
active: true
|
||||
phase_specific: true
|
||||
reason: P2 only (REQ-13 admin UI). Inactive outside P2 — no UI work exists in P1/P3/P4/P5; deactivate at the P2 exit boundary and do not dispatch outside P2.
|
||||
frameworks: [html, css, vanilla-js]
|
||||
constraints: ["server-rendered static pages from coreci_broker_ui/ on the same port as the API; no build step, no client framework, no npm (REQ-13)", "pages: /ui/login, /ui/app_registrations, /ui/user_allowlist, /ui/audit plus /ui/dashboard and a persistent nav header with Logout", "pages refresh via the corresponding admin API after each action; the API contract is owned by backend-engineer — frontend adapts", "session cookie broker_session is HttpOnly + SameSite=Lax (D-013); no direct DB or file access — admin API only"]
|
||||
territory: [coreci_broker_ui/**]
|
||||
subagent_type: frontend-engineer
|
||||
---
|
||||
|
||||
---
|
||||
name: security-engineer
|
||||
domain: security
|
||||
active: true
|
||||
phase_specific: true
|
||||
reason: Custom persona activated for P3 (tool auth plane: register gates, App-token verification, identity enforcement) and P5 (security review pass before ship); inactive in P1/P2/P4 unless the lead escalates a finding.
|
||||
frameworks: [python3.11, unittest, STRIDE, CWE]
|
||||
constraints: ["review-only: coreci_broker.py and tests stay owned by backend-engineer; findings route through lead-developer, never direct edits", "findings are STRIDE-classified and CWE-first (cite CWE ids)", "focus: path traversal on /secrets/<path> (URL-decode before validation), authz (allowlist gates, path scoping, identity match), audit completeness, pepper never re-printed, 0600 modes, token scoping"]
|
||||
territory: [coreci_broker.py, tests/**]
|
||||
subagent_type: ci-security-auditor
|
||||
---
|
||||
|
||||
---
|
||||
name: lead-developer
|
||||
domain: coordination
|
||||
active: true
|
||||
reason: Active across all execution phases (P1–P5): decomposes plans into vertical slices per persona domain, arbitrates territory conflicts, revises the plan at phase boundaries, and sequences tasks across persona dependencies. No direct code writes.
|
||||
frameworks: [none]
|
||||
constraints: ["no direct modifications to code files — coordination and conflict resolution only", "arbitration: backend-engineer owns files in cross-persona conflicts and defines the API contract; frontend adapts; security findings flow through backend", "enforce boundaries: no direct database access from the UI, no UI logic inside coreci_broker.py; battle-tested defaults over novel approaches"]
|
||||
territory: [.ciagent/PLAN.md, .ciagent/*.md]
|
||||
subagent_type: lead-developer
|
||||
---
|
||||
|
||||
---
|
||||
name: data-engineer
|
||||
domain: data
|
||||
active: false
|
||||
reason: Deactivated — no database tier exists beyond sqlite3 embedded in the single file; the spec §5.2 schema and all SQL live inside coreci_broker.py and are owned by backend-engineer; no pipelines, ETL, or migrations in v0.1 scope.
|
||||
frameworks: [sqlite3]
|
||||
constraints: ["stays inactive for the whole milestone; any residual data work folds into backend-engineer"]
|
||||
territory: []
|
||||
subagent_type: backend-engineer
|
||||
---
|
||||
@@ -0,0 +1,158 @@
|
||||
---
|
||||
must_haves:
|
||||
- "python3 -m unittest discover -s tests passes (all phases, cumulative)"
|
||||
- "coreci_broker.py single file, line count < 800 (AC14)"
|
||||
- "python3 -m py_compile coreci_broker.py coreci_broker_client.py"
|
||||
- "broker boots standalone on an ephemeral port; DB + audit log created mode 0600; admin pass printed exactly once"
|
||||
- "all AC1-AC14 acceptance criteria green by end of P5"
|
||||
---
|
||||
|
||||
# PLAN — CoreCI Broker (Milestone v0.1)
|
||||
|
||||
Execution conventions: sequential waves (within a wave, tasks run in listed order); each task = one subagent commit `feat(P##): …` carrying a `---ci---` block (phase, milestone: v0.1, status: execute, persona, task, requirements.covered). Tests ship with their task — never deferred. Personas per PERSONAS.md: backend-engineer (all phases), frontend-engineer (P2 only), security-engineer (P3+P5, review-only — findings route via lead-developer, never direct edits), lead-developer (coordination, no code writes). Verify commands: `python3 -m py_compile <files>`; `python3 -m unittest discover -s tests`; AC14 line-count via test. Test harness: in-process `ThreadingHTTPServer` on an ephemeral port + `tempfile.TemporaryDirectory()` state, driven with stdlib `urllib` (RESEARCH §6).
|
||||
|
||||
## User-Facing Surface
|
||||
|
||||
- **CEO admin UI** — server-rendered from `coreci_broker_ui/`, same port as the API, persistent nav with Logout: `/ui/login` (the CEO's bookmark), `/ui/app_registrations` (table + add-modal + revoke/reinstate), `/ui/user_allowlist` (same, optional email), `/ui/audit` (filterable, read-only, no export), `/ui/dashboard`.
|
||||
- **Tool-side client module** — `coreci_broker_client.py`: `ensure_registered()`, `get_state()`, `put_state()`, `read_secret()`; stdlib-urllib-only, importable with zero dependencies in any OWUI tool env; fetched unauthenticated at `/static/coreci_broker_client.py`.
|
||||
- **Standalone CLI** — `python3 coreci_broker.py`: binds `127.0.0.1:8765`; first boot prints the generated admin pass exactly once.
|
||||
- **OWUI Tools mount** — `Tools` class (same file): daemon-thread server behind a once-per-process guard, `self.valves`, `_`-prefixed helpers.
|
||||
|
||||
## Happy Path
|
||||
|
||||
1. First boot in a fresh directory: broker writes `.env` (32-byte base64 pepper + 32+-char admin pass), prints the pass to the startup log exactly once (AC1); a second boot prints nothing.
|
||||
2. CEO opens the `/ui/login` bookmark, logs in with the printed pass (session cookie `broker_session`), lands on the dashboard.
|
||||
3. CEO adds the tool's app UUID on `/ui/app_registrations` and the user_id on `/ui/user_allowlist`; each page reflects the persisted row right after the action (AC2/AC3); audit gets `add_app`/`add_user`.
|
||||
4. Tool author fetches `/static/coreci_broker_client.py` (unauthenticated) into the OWUI workspace and imports it — no third-party deps.
|
||||
5. Tool calls `ensure_registered()`: broker gates on the allowlists, derives the token server-side, returns `{token, token_expires_at, pepper}` (row-create); the client persists via `PUT /agents/<id>/state`.
|
||||
6. Tool reads `coreci/proxmox` with `Authorization: App <UUID> <token>` → 200 secret payload (AC5); audit line `read`/`ok`.
|
||||
7. Same tool requests `coreci/gitea` → scope mismatch → 403 (AC6); audit `read`/`denied`.
|
||||
8. Token expires (`BROKER_TOKEN_TTL`): next call → 401 (AC10) → client re-registers → new token → retry succeeds; audit `register`.
|
||||
9. CEO reviews `/ui/audit`: login/logout (AC9), allowlist adds, registers, reads ok/denied — every step audited.
|
||||
|
||||
## UX Acceptance Criteria
|
||||
|
||||
- **AC1** — first boot prints the admin pass exactly once; the CEO logs in at `/ui/login` with it; second boot prints nothing.
|
||||
- **AC2/AC3** — app registrations and user-allowlist entries are added and removed via the UI; the page reflects persisted state after every action (no stale views).
|
||||
- **AC9** — every login and logout writes an audit line (`login`/`logout`).
|
||||
- **UI** — server-rendered static HTML/CSS/JS from `coreci_broker_ui/` on the same port: no build step, no framework, no npm.
|
||||
- **Audit page** — read-only with filters; NO export button or download control of any kind.
|
||||
- **Client module** — `coreci_broker_client.py` imports cleanly with zero deps (stdlib urllib only) in any OWUI tool env; contains no secrets.
|
||||
|
||||
## Phase 1 — Core Skeleton
|
||||
|
||||
Branch `phase/01-core-skeleton` → tag `v0.1.1`. Requirements: REQ-03, REQ-11, REQ-14, REQ-15, REQ-17, REQ-18.
|
||||
Exit: broker boots standalone on an ephemeral port; `broker.db` + `broker.audit.log` created mode 0600; admin pass printed exactly once (silent second boot); §6 token derivation byte-exact; source < 800 lines.
|
||||
|
||||
### Wave 1 — persistence layer (independent)
|
||||
|
||||
- **P1-01** · backend-engineer · `coreci_broker.py`, `tests/test_config.py` · `.env` reader/writer at `BROKER_ENV_PATH` (spec §5.1 contract + documented additions); first-boot generation of `BROKER_INSTANCE_PEPPER` (32 random bytes, base64, synced to SQLite when the store boots — sync lands with P1-02's schema) and `BROKER_ADMIN_PASS` (32+ random chars), written to `.env`, printed to the startup log exactly once — never again · REQ-14 · test: first boot writes `.env` + prints pass once; second boot silent; `BROKER_*` overrides honored.
|
||||
- **P1-02** · backend-engineer · `coreci_broker.py`, `tests/test_store.py` · SQLite schema per spec §5.2 (`broker_config`, `app_registrations`, `user_allowlist`, `agent_registrations`, `admin_sessions` + indexes); DB created mode 0600 via `os.open(..., 0o600)` (umask-independent); sole writer of `broker.db`, all SQL in-file · REQ-15 · test: boot creates `broker.db` at 0600 with all tables + indexes.
|
||||
- **P1-03** · backend-engineer · `coreci_broker.py`, `tests/test_audit.py` · append-only JSONL audit writer at `BROKER_AUDIT_LOG_PATH`: one JSON object per line, fields `ts, user_id, app_registration_uuid, agent_id, action, path, result` (admin actions carry null user_id/uuid); mode 0600; flush + fsync per write; no rotation (D-007) · REQ-11 · test: each write appends one valid JSON line at mode 0600; all action verbs accepted.
|
||||
|
||||
### Wave 2 — crypto primitives
|
||||
|
||||
- **P1-04** · backend-engineer · `coreci_broker.py`, `tests/test_hasher.py` · dual-mode hasher (D-001): bcrypt when importable, else stdlib `hashlib.scrypt`; stored hashes format-prefixed (`$2b$…` / `scrypt$…`); verification format-aware · REQ-03 · test: scrypt hash/verify roundtrip + prefix dispatch on both formats (bcrypt path via format contract — no bcrypt locally).
|
||||
- **P1-05** · backend-engineer · `coreci_broker.py`, `tests/test_token.py` · token engine: `base64(sha256(pepper:UUID:user_id:agent_id:','.join(sorted(tags))))` — 44-char ASCII, no prefix; pure function of (pepper, inputs); never stores plaintext tokens · REQ-03 · test: derivation matches the §6 formula byte-for-byte; 44-char assertion; tag order normalized.
|
||||
|
||||
### Wave 3 — transport + mounts (listed in dependency order)
|
||||
|
||||
- **P1-06** · backend-engineer · `coreci_broker.py`, `tests/test_router.py` · stdlib `ThreadingHTTPServer` + JSON router: body/param parsing, `{"error, message}` contract on 400/401/403/404/409/500, static-file serving hook for `/ui/` + `/static/` (fixed root, traversal-rejecting); no business logic in the router · REQ-18 · test: ephemeral-port boot; every error code returns the contract shape; static hook serves a fixture.
|
||||
- **P1-07** · backend-engineer · `coreci_broker.py`, `tests/test_mounts.py` · OWUI `Tools` mount: `__init__` starts the daemon-thread server behind a once-per-process guard (no activate() hook exists — RESEARCH §4), `self.valves = self.Valves()`, `_`-prefixed helpers, paths resolved from env never `__file__`; standalone `__main__` entry · REQ-18 · test: `Tools()` double-instantiation starts exactly one server; standalone boot serves a 200.
|
||||
|
||||
### Wave 4 — phase acceptance
|
||||
|
||||
- **P1-08** · backend-engineer · `tests/test_acceptance_p1.py` · AC14 line-count test (`coreci_broker.py` < 800 lines, wc-style, no comment stripping) + boot acceptance: 0600 on DB + audit log, admin pass printed exactly once (silent second boot), restart persistence of DB rows + audit history · REQ-17, REQ-16 (per REQUIREMENTS traceability), REQ-15 · test: is the acceptance file.
|
||||
|
||||
## Phase 2 — Admin Plane
|
||||
|
||||
Branch `phase/02-admin-plane` → tag `v0.1.2`. Requirements: REQ-01, REQ-05, REQ-06, REQ-12, REQ-13.
|
||||
Exit: AC1/AC2/AC3/AC9 green; admin-usable (login → manage allowlists → query audit, all via UI).
|
||||
|
||||
### Wave 1 — admin auth
|
||||
|
||||
- **P2-01** · backend-engineer · `coreci_broker.py`, `tests/test_admin_auth.py` · `POST /api/broker/auth/login|logout|whoami`: verifies `BROKER_ADMIN_USER`/`BROKER_ADMIN_PASS`, creates `admin_sessions` row (TTL `BROKER_SESSION_TTL`, default 300s); cookie `broker_session` HttpOnly + SameSite=Lax (D-013); HTTP Basic accepted on all admin API endpoints; lazy expiry + boot cleanup of stale sessions · REQ-01, REQ-11 · test: login sets cookie; wrong pass → 401; expired session → 401; logout invalidates + audits (AC9 `login`/`logout` lines).
|
||||
|
||||
### Wave 2 — admin API
|
||||
|
||||
- **P2-02** · backend-engineer · `coreci_broker.py`, `tests/test_apps.py` · app_registrations CRUD: `GET/POST /api/broker/app_registrations`, `DELETE .../<uuid>` (marks revoked — new registrations rejected, existing tokens honored until TTL), `POST .../reinstate`; UUID format validation → 400; duplicate → 409; audited `add_app`/`remove_app` · REQ-05 · test: add/list/revoke/reinstate matrix + 400/409 + revoke-keeps-tokens + audit lines.
|
||||
- **P2-03** · backend-engineer · `coreci_broker.py`, `tests/test_allowlist.py` · user_allowlist CRUD: same shape (`GET/POST`, `DELETE .../<user_id>` revoked semantics, `POST .../reinstate`), optional `email`, duplicate → 409; audited `add_user`/`remove_user` · REQ-06 · test: add/remove/reinstate + 409 + audit lines.
|
||||
- **P2-04** · backend-engineer · `coreci_broker.py`, `tests/test_audit_query.py` · admin-only `GET /api/broker/audit`: filters `since`, `until`, `user_id`, `app_registration_uuid`, `action`; `limit` default 100 / max 1000; `next_since` cursor; streams the file line-by-line · REQ-12 · test: seeded audit file → filter combos, limit clamp, cursor pagination.
|
||||
|
||||
### Wave 3 — admin UI (frontend-engineer; pages refresh via the admin API after each action; no direct DB/file access)
|
||||
|
||||
- **P2-05** · frontend-engineer · `coreci_broker_ui/` (login page, dashboard, shared nav/CSS) · `/ui/login` (the CEO's bookmark) + `/ui/dashboard`; persistent nav header with Logout on every page · REQ-13, REQ-01 · verify: pages served 200 on the API port, post-login redirect to dashboard; full AC drive lands in P2-08.
|
||||
- **P2-06** · frontend-engineer · `coreci_broker_ui/` (app_registrations + user_allowlist pages) · table + add-modal + revoke/reinstate per row on both pages; after each action the page re-fetches via the corresponding API and reflects persisted state · REQ-13, REQ-05, REQ-06 · verify: AC2/AC3 flows (tests in P2-08); interim smoke — pages boot + served.
|
||||
- **P2-07** · frontend-engineer · `coreci_broker_ui/` (audit page) · filterable audit view (since/until/user_id/uuid/action) wired to `GET /api/broker/audit`; strictly read-only — no export button or control of any kind · REQ-13, REQ-12 · verify: page renders query results; no export element in the markup.
|
||||
|
||||
### Wave 4 — phase acceptance
|
||||
|
||||
- **P2-08** · backend-engineer · `tests/test_acceptance_p2.py` · AC1 (first-boot login at `/ui/login` with the printed pass), AC2/AC3 (add + remove app/user through the UI; `app_registrations`/`user_allowlist` persist; page reflects state after each action), AC9 (login/logout audit lines) · REQ-01, REQ-05, REQ-06, REQ-12, REQ-13 · test: is the acceptance file.
|
||||
|
||||
## Phase 3 — Tool Auth Plane
|
||||
|
||||
Branch `phase/03-tool-auth-plane` → tag `v0.1.3`. Requirements: REQ-02, REQ-04, REQ-09, REQ-10.
|
||||
Exit: AC4/AC7/AC8/AC10/AC12 green; tool-auth usable (register → App-auth calls → state persistence).
|
||||
|
||||
### Wave 1 — registration
|
||||
|
||||
- **P3-01** · backend-engineer · `coreci_broker.py`, `tests/test_register.py` · `POST /api/broker/agents/register` with `{agent_id, app_registration_uuid, user_id, tags}`: allowlist gates → 403 unknown/revoked UUID or user; malformed → 400; idempotent re-derive (D-005) — overwrite stored hash on input change, same derived token when unexpired + unchanged; TTL from `BROKER_TOKEN_TTL` (default 86400, D-007); pepper returned on row-create or tokenless register (KB-loss recovery); audited `register` · REQ-02, REQ-03 · test: gate matrix (AC7/AC8), idempotency/overwrite/same-token, pepper rule, 400s.
|
||||
|
||||
### Wave 2 — App-auth + agent surface (listed in dependency order)
|
||||
|
||||
- **P3-02** · backend-engineer · `coreci_broker.py`, `tests/test_app_auth.py` · App-auth verification (D-004): parse `Authorization: App <UUID> <token>`; UUID-scoped slow-hash scan over that tool's rows; in-memory verified-token cache populated on first success, never authoritative — every hit re-checks row status + expiry; 401 missing/malformed/failed · REQ-04 · test: cache-miss scan + cache-hit re-check (revoke + expiry), both hash formats, 401 matrix; revoked-app token honored until TTL.
|
||||
- **P3-03** · backend-engineer · `coreci_broker.py`, `tests/test_agent_meta.py` · `GET /api/broker/agents/<agent_id>` (D-015): App-auth + identity match; metadata-only response (no token, no hash) · REQ-04, REQ-02 · test: correct-identity 200 with metadata-only payload; identity mismatch → 403; no token material in response.
|
||||
- **P3-04** · backend-engineer · `coreci_broker.py`, `tests/test_state.py` · `GET/PUT /api/broker/agents/<agent_id>/state` (App-auth): broker enforces calling UUID + token-derived user_id match the stored identity; 404 when no state exists; state schema per spec §8 (agent_id, app_registration_uuid, user_id, tags, pepper, current_token, token_expires_at, first_registered_at, last_seen_at); backend toggle `BROKER_AGENT_STATE_BACKEND=internal` (default, rows in `broker.db`) `| owui` (thin OWUI-API adapter, stub-verified) (D-003) · REQ-09, REQ-10 · test: state roundtrip + identity enforcement + 404; owui backend against a stub server.
|
||||
|
||||
### Wave 3 — phase acceptance
|
||||
|
||||
- **P3-05** · backend-engineer · `tests/test_acceptance_p3.py` · AC4 (broker- and tool-side derivation byte-identical; stored as slow-hash), AC7 (revoked/unknown UUID → 403), AC8 (unknown user → 403), AC10 (TTL expiry → 401 → re-register → new token, audited), AC12 (state read/write with identity enforcement) · REQ-02, REQ-04, REQ-09, REQ-10 · test: is the acceptance file.
|
||||
|
||||
### Wave 4 — security review (review-only)
|
||||
|
||||
- **P3-06** · security-engineer · review of `coreci_broker.py`, `tests/**` (no edits — findings STRIDE-classified, CWE-cited, routed via lead-developer) · tool-auth security pass: register gates, UUID-scoped scan + cache authority, identity enforcement, TTL/expiry handling, audit completeness · REQ-02, REQ-04, REQ-09 · verify: findings recorded in the task commit's `---ci---` block; remediation (if any) = follow-up backend commit dispatched by lead-developer.
|
||||
|
||||
## Phase 4 — Secrets + Client Ecosystem
|
||||
|
||||
Branch `phase/04-secrets-client-ecosystem` → tag `v0.1.4`. Requirements: REQ-07, REQ-08, REQ-19, REQ-20.
|
||||
Exit: AC5/AC6 green; full product — client module fetched from `/static/` drives a reference tool end-to-end against the test broker.
|
||||
|
||||
### Wave 1 — secrets plane (listed in dependency order)
|
||||
|
||||
- **P4-01** · backend-engineer · `coreci_broker.py`, `tests/test_creds_source.py` · creds-source adapter (D-002): `FileCredsSource` default (JSON at `BROKER_CREDS_FILE`, default `data/creds.json` — D-014) + `OwuiApiCredsSource` (`BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY`; RESEARCH §4 flow: KB lookup → file map → `/api/v1/files/{id}/data/content`); response `{path, data, version, updated_at}` (version synthesized broker-side if absent); broker never writes the KB · REQ-08 · test: file-source hit/miss; owui source against a stub HTTP server.
|
||||
- **P4-02** · backend-engineer · `coreci_broker.py`, `tests/test_secrets.py` · `GET /api/broker/secrets/<path>` (App-auth): scope check `path == "coreci/" + agent_id` → 403 mismatch, 404 absent, 401 auth failure; every attempt audited (`read` ok/denied/error); traversal defense — URL-decode BEFORE validation, reject `..`, backslash, NUL, absolute paths (decoded form feeds the scope check) · REQ-07 · test: ok/401/403/404 matrix + URL-encoded traversal attempts rejected + audit per attempt.
|
||||
|
||||
### Wave 2 — client ecosystem (listed in dependency order)
|
||||
|
||||
- **P4-03** · backend-engineer · `coreci_broker_client.py`, `tests/test_client.py`, `coreci_broker.py` (static route) · public client module: stdlib `urllib` only (no httpx, no bcrypt — importable in any OWUI tool env); `ensure_registered()` idempotent; `get_state()`/`put_state()`; `read_secret()` with 401 → re-register → retry-once; `data.get("pepper")` (never assumes the key — RESEARCH §2 defect 3); served unauthenticated at `/static/coreci_broker_client.py` via the P1-06 hook; contains no secrets · REQ-19 · test: zero third-party imports asserted; full flow against the in-process broker; 401→re-register→retry path.
|
||||
- **P4-04** · backend-engineer · `reference_tools/`, `tests/test_reference_tools.py` · reference OWUI tools (D-006): (a) proxmox-style §11 flow — first call registers + persists state, subsequent calls read the token from state and fetch `coreci/proxmox`, downstream call defaults to a safe dry-run; (b) minimal auth-first-then-use-a-second-tool showcase · REQ-20 · test: both tools drive the test broker (FileCredsSource); dry-run default asserted; no live infra.
|
||||
|
||||
### Wave 3 — phase acceptance
|
||||
|
||||
- **P4-05** · backend-engineer · `tests/test_acceptance_p4.py` · AC5 (valid token reads the secret; tampered token → 401), AC6 (proxmox-registered tool reading `coreci/gitea` → 403); end-to-end: fetch client from `/static/` unauthenticated → import → register → state → secret read · REQ-07, REQ-19, REQ-20 · test: is the acceptance file.
|
||||
|
||||
## Phase 5 — Final Review + Ship
|
||||
|
||||
Branch `phase/05-final-review-ship` → tag `v0.1.5` (final phase patch = milestone release, D-008). Requirements: REQ-01..REQ-20 (acceptance).
|
||||
Exit: AC1–AC14 all green; security review PASS; README complete with deviations; owner approval recorded; milestone merged to main, tagged `v0.1.5`, released on gitea.
|
||||
|
||||
### Wave 1 — acceptance matrix
|
||||
|
||||
- **P5-01** · backend-engineer · `tests/test_acceptance_full.py` · complete §12 matrix: AC1–AC13 as behavioral tests (consolidating the per-phase AC files and completing any remaining §12 criteria — 0600 modes, restart persistence, client-module fetch) + AC14 line-count capstone; cumulative suite green via `python3 -m unittest discover -s tests` · REQ-01..REQ-20 · test: is the acceptance suite.
|
||||
|
||||
### Wave 2 — security review (review-only)
|
||||
|
||||
- **P5-02** · security-engineer · review of `coreci_broker.py`, `coreci_broker_client.py`, `reference_tools/`, `tests/**` (no edits — STRIDE/CWE findings via lead-developer) · pre-ship security pass: path traversal incl. URL-encoded forms, pepper never re-printed (second boot, re-register), 0600 modes, token scoping + identity enforcement, audit completeness, static-serving roots · REQ-07, REQ-11, REQ-14, REQ-15 (+ all) · verify: PASS or CWE-cited findings; remediation follows in P5-03.
|
||||
|
||||
### Wave 3 — remediation + docs
|
||||
|
||||
- **P5-03** · backend-engineer · `coreci_broker.py`, `tests/**` · remediate P5-02 findings (no-op skip if PASS) · affected REQs per finding · test: full suite re-green.
|
||||
- **P5-04** · backend-engineer · `README.md` · run/deploy notes: standalone + OWUI mount, env/config contract (spec §5.1 + documented additions), first-boot pass discipline, backup guidance (SQLite is the source of truth), client-module bootstrap; documented deviations distilled from D-001..D-015 incl. the spec defects + resolutions (PROJECT.md §Context) · REQ-14, REQ-18 (+ traceability) · test: README covers run/deploy/deviations; suite still green.
|
||||
|
||||
### Wave 4 — review + reconstruction
|
||||
|
||||
- **P5-05** · lead-developer · `.ciagent/*.md` review notes (no code writes) · multi-persona review of the milestone diff (backend, frontend P2 artifacts, security sign-offs); audit/reconstruction: every REQ-01..REQ-20 maps to green tests + traceable commits · REQ-01..REQ-20 · verify: sign-offs + reconstruction recorded in `---ci---` blocks.
|
||||
|
||||
### Wave 5 — ship
|
||||
|
||||
- **P5-06** · lead-developer · no code writes · owner approval gate recorded; squash-merge `milestone/v0.1-coreci-broker` → `main`; tag `v0.1.5`; first push to gitea `coreci/broker` (public, default branch main — forge repo exists before push, D-010/D-012) + release · REQ-01..REQ-20 · verify: `v0.1.5` on `main`; release visible on the forge.
|
||||
@@ -0,0 +1,69 @@
|
||||
# PROJECT — CoreCI Broker
|
||||
|
||||
## Name
|
||||
|
||||
CoreCI Broker
|
||||
|
||||
## Core Value
|
||||
|
||||
A single canonical credential flow for autonomous agent tools. The broker mediates access to the `creds-coreci-coo` knowledge base: tools authenticate by a derived token (not a shared secret), the CEO manages a UI allowlist of approved tool UUIDs and user IDs, every read is audited to an append-only file, and per-tool scope is enforced by path (`coreci/<agent_id>` only). It replaces per-tool credential resolution (KB files, env vars, mode-0600 JSON) with one small, readable implementation surface: a single Python file, one SQLite DB, a four-page web UI.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Validated
|
||||
|
||||
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09; supersedes v0.4). The full v0.5 spec is distilled into REQ-01..REQ-20 in REQUIREMENTS.md, with acceptance criteria drawn from the spec's §12 list. Clarification decisions D-001..D-011 (recorded in the init commit) plus D-012..D-015 (recorded in the clarify stage) resolve the spec's ambiguities and residual defaults:
|
||||
|
||||
- Dual-mode token hashing (bcrypt when importable, stdlib scrypt fallback; format-prefixed hashes)
|
||||
- Pluggable creds-source adapter (file-based default; Open WebUI HTTP API for production)
|
||||
- Agent-state backend toggle (`internal` default | `owui`)
|
||||
- UUID-scoped token verification + in-memory verified-token cache (spec §7.5's token_hash index is impossible with bcrypt)
|
||||
- Register/pepper semantics including the KB-loss recovery path
|
||||
- A public, dependency-free client module served as a static asset so agents can bootstrap their own OWUI tools
|
||||
- Spec §13 defaults: no audit rotation, single instance, 24h token TTL, CEO sole approver
|
||||
|
||||
### Active
|
||||
|
||||
Milestone v0.1 (`milestone/v0.1-coreci-broker`) implements the entire v0.5 spec: REQ-01..REQ-20. No requirement is deferred.
|
||||
|
||||
### Out of Scope
|
||||
|
||||
See REQUIREMENTS.md §Out of Scope — federation, secret versioning, high availability, UI beyond the four pages + dashboard, audit rotation, multiple broker instances, approver agents, key rotation ceremony, HSM, rate limiting, admin MFA, notifications.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Single-file broker**: `coreci_broker.py` under 800 lines (excluding tests, UI HTML, client module, reference tools) — spec §12.14
|
||||
- **Stdlib-only runtime**: no pip in the target dev environment; bcrypt is the sole optional import (dual-mode hashing); tests use unittest
|
||||
- **SQLite is the source of truth**: single instance, no HA; back it up
|
||||
- **Bind default** `127.0.0.1:8765`; the bind address controls reachability
|
||||
- **Mode 0600** on `broker.db` and `broker.audit.log`
|
||||
- **The broker never writes to `creds-coreci-coo`** — the CEO writes that KB directly through Open WebUI's UI
|
||||
- **Config contract** = spec §5.1 plus documented additions: `BROKER_ENV_PATH`, `BROKER_CREDS_SOURCE`, `BROKER_CREDS_FILE`, `BROKER_OWUI_BASE_URL`, `BROKER_OWUI_API_KEY`, `BROKER_AGENT_STATE_BACKEND`
|
||||
- **Token form**: `base64(sha256(pepper:UUID:user_id:agent_id:sorted_tags))` — 44-char ASCII, no prefix; `BROKER_TOKEN_TTL` default 86400s
|
||||
- **Open WebUI runtime**: bcrypt and httpx exist there; the client module must not depend on either (stdlib urllib) so any tool can import it
|
||||
|
||||
## Context
|
||||
|
||||
CoreCI estate: sibling projects researchx and nextcraft live on gitea (`git.coreci.dev/coreci/*`) and use this same ciagent pipeline. The broker runs on the Open WebUI host as one workspace tool (embedded HTTP server in a daemon thread; also runnable standalone). The `creds-coreci-coo` KB currently mediates four credentials (`coreci/proxmox`, `coreci/gitea`, `coreci/executor` TBD, plus the COO's own login — independent of the broker's `BROKER_ADMIN_PASS`). The existing `proxmox_readonly_tool.py` is updated post-broker to use the public client module; it lives outside this repo, so this repo ships the reference implementation of the tool-side bootstrap instead.
|
||||
|
||||
Known spec defects resolved by clarification (recorded for traceability): §7.5's in-memory `token_hash` index is impossible with bcrypt salted hashes; §6 claims the tool derives a token every call while §9's flow uses the broker-returned token from the agent KB (the broker derives authoritatively; the tool stores); §9's snippet raises `KeyError` on `data["pepper"]` for re-registrations; the pepper chicken-and-egg (a token is needed to read agent state, but the pepper is needed to derive a token) is fixed by returning the pepper when the caller presents no valid token.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
| ID | Decision | Rationale | Outcome |
|
||||
|----|----------|-----------|---------|
|
||||
| D-001 | Dual-mode bcrypt/scrypt hashing, format-prefixed hashes | Spec §6 requires bcrypt; dev env is stdlib-only; OWUI runtime ships bcrypt | To be validated in P1 |
|
||||
| D-002 | Creds-source adapter: file (default) \| OWUI API | Spec never specified the physical read mechanism; both environments must work | To be validated in P4 |
|
||||
| D-003 | Agent-state backend toggle, internal default | §9.1 endpoints are the only interface; internal storage is functionally identical without the admin-API dependency | To be validated in P3 |
|
||||
| D-004 | UUID-scoped slow-hash scan + in-memory verified-token cache | §7.5's bcrypt token_hash index is impossible; header UUID bounds the scan; SQLite stays authoritative | To be validated in P3 |
|
||||
| D-005 | Register re-derives; pepper returned on row-create or tokenless register | Broker holds pepper + all inputs; tokenless register is the only KB-loss recovery path | To be validated in P3 |
|
||||
| D-006 | Public dependency-free client module at /static/ + reference tools | Owner directive: agents bootstrap their own tools within the OWUI workspace | To be validated in P4 |
|
||||
| D-007 | Spec §13 defaults accepted | Owner confirmed: no rotation, single instance, 24h TTL, CEO sole approver | Binding for v0.1 |
|
||||
| D-008 | Milestone v0.1; "v0.5" remains the spec revision label | computeMilestoneTag greenfield convention (researchx, nextcraft precedent) | Binding for v0.1 |
|
||||
| D-009 | Stdlib-only runtime; unittest for tests | No pip in environment; dual-mode hashing is the sole optional import | Binding for v0.1 |
|
||||
| D-010 | Single-project mode; gitea release target coreci/broker | Sibling convention on the same forge; first push deferred to a ship phase | Binding for v0.1 |
|
||||
| D-011 | Empty root commit on main precedes the branch hierarchy | Branches cannot be created from an unborn HEAD; nextcraft precedent | Executed at init |
|
||||
| D-012 | Forge repo coreci/broker public, default branch main | Sibling convention (researchx/nextcraft both public); code ships no runtime secrets — pepper/admin pass generated at boot, .env.secrets gitignored | Binding for v0.1 |
|
||||
| D-013 | Session cookie `broker_session` uses HttpOnly + SameSite=Lax | Spec silent on cookie attributes; HttpOnly blocks JS exfiltration, Lax preserves bookmark-login UX | Binding for v0.1 |
|
||||
| D-014 | `BROKER_CREDS_FILE` defaults to `data/creds.json` | FileCredsSource dev/test default location; `data/` already gitignored | Binding for v0.1 |
|
||||
| D-015 | `GET /api/broker/agents/<agent_id>`: App-auth + identity match, metadata-only response (no token, no hash) | §7.4 auth/response unspecified; mirrors state-endpoint identity enforcement; token material never leaves the broker | Binding for v0.1 |
|
||||
@@ -0,0 +1,99 @@
|
||||
# Requirements — CoreCI Broker
|
||||
|
||||
## Overview
|
||||
|
||||
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09; supersedes the v0.4 in-memory draft). "v0.5" is the spec's revision label; the product milestone is v0.1. REQ-01..REQ-20 constitute the full v0.1 scope — nothing is deferred. Acceptance criteria reference the spec's §12 list (AC1–AC14). Clarification decisions D-001..D-011 live in the init commit's `---ci---` block and are distilled in PROJECT.md.
|
||||
|
||||
## v1 (Milestone v0.1)
|
||||
|
||||
### Auth & Identity
|
||||
|
||||
- **REQ-01 Admin authentication.** `BROKER_ADMIN_USER`/`BROKER_ADMIN_PASS` from the broker `.env`. Session cookie `broker_session` for the UI; HTTP Basic accepted on all admin API endpoints. `POST /api/broker/auth/login` creates a session (SQLite `admin_sessions`, TTL from `BROKER_SESSION_TTL`, default 300s), `logout` invalidates it, `whoami` returns the session info or 401. Admin sessions expire lazily and are cleaned on boot.
|
||||
- **REQ-02 Tool self-registration.** `POST /api/broker/agents/register` with `{agent_id, app_registration_uuid, user_id, tags}`. The broker rejects unknown or revoked UUIDs and user_ids with 403 (allowlists are the gate); malformed input → 400. Re-registration is idempotent: the broker re-derives from the request inputs and overwrites the stored hash when inputs changed; an unexpired row re-issues the same derived token. The response returns the token, `token_expires_at`, and the pepper when the row is created or the caller presented no valid token (KB-loss recovery). Register actions are audited.
|
||||
- **REQ-03 Derived token.** `token = base64(sha256(pepper:APP_UUID:user_id:AGENT_ID:','.join(sorted(tags))))` — a 44-char ASCII string, no prefix, no challenge protocol. The broker derives the same token a compliant tool would derive. Stored as a slow hash in `agent_registrations.token_hash`: bcrypt when importable, else stdlib scrypt; hashes are format-prefixed (`$2b$…` / `scrypt$…`) so verification is format-aware.
|
||||
- **REQ-04 Token verification & TTL.** Tool endpoints require `Authorization: App <UUID> <token>`. Verification: look up rows scoped to the presented UUID, check the slow hash (in-memory verified-token cache after first success; cache is never authoritative — rows re-checked for status/expiry). Missing/malformed/failed → 401. Tokens live `BROKER_TOKEN_TTL` seconds (default 86400); expiry → 401 → the tool re-registers. Revoked apps keep working until TTL expiry by design.
|
||||
|
||||
### Allowlist Administration
|
||||
|
||||
- **REQ-05 App registrations.** Admin-only `GET/POST /api/broker/app_registrations`, `DELETE /api/broker/app_registrations/<uuid>` (marks revoked; new registrations rejected; existing tokens honored until TTL), `POST .../reinstate`. UUID format validated (400 on invalid, 409 on duplicate). All actions audited as `add_app`/`remove_app`.
|
||||
- **REQ-06 User allowlist.** Admin-only `GET/POST /api/broker/user_allowlist`, `DELETE .../<user_id>` (revoked semantics), `POST .../reinstate`. Optional `email` field. 409 on duplicate. Audited as `add_user`/`remove_user`.
|
||||
|
||||
### Secret Mediation
|
||||
|
||||
- **REQ-07 Path-scoped secret reads.** `GET /api/broker/secrets/<path>`. Scope is the path: a registration may read exactly `coreci/<agent_id>` — cross-path → 403, missing secret → 404, auth failure → 401. Every attempt (ok/denied/error) is audited with `user_id`, `app_registration_uuid`, `agent_id`, `action: read`, `path`, `result`.
|
||||
- **REQ-08 Creds-source adapter.** The broker reads `creds-coreci-coo` exclusively through a pluggable adapter, `BROKER_CREDS_SOURCE=file|owui`: `FileCredsSource` reads a JSON document at `BROKER_CREDS_FILE` mapping paths to secret objects (default — dev/tests); `OwuiApiCredsSource` reads via the Open WebUI HTTP API using `BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY` (production). The broker never writes to the KB. Response shape: `{path, data, version, updated_at}`.
|
||||
|
||||
### Agent State
|
||||
|
||||
- **REQ-09 State endpoints.** `GET/PUT /api/broker/agents/<agent_id>/state` (App-auth). The broker enforces that the calling UUID and the token-derived user_id match the stored identity; 404 when no state exists. State schema per spec §8 (agent_id, app_registration_uuid, user_id, tags, pepper, current_token, token_expires_at, first_registered_at, last_seen_at). This is the tool's only persistence channel — the broker is the durability layer.
|
||||
- **REQ-10 State backend toggle.** `BROKER_AGENT_STATE_BACKEND=internal` (default: rows in `broker.db` behind the state endpoints) `| owui` (real per-agent Open WebUI KBs via the OWUI API). The tool cannot tell the difference — the state endpoints are the only interface.
|
||||
|
||||
### Audit
|
||||
|
||||
- **REQ-11 Append-only audit log.** One JSON object per line at `BROKER_AUDIT_LOG_PATH` (mode 0600, append-only, no rotation in v0.1). Fields: `ts`, `user_id`, `app_registration_uuid`, `agent_id`, `action` (register/read/denied/login/logout/add_app/remove_app/add_user/remove_user), `path`, `result` (ok/denied/error); admin actions carry null user_id/uuid.
|
||||
- **REQ-12 Audit query.** Admin-only `GET /api/broker/audit` with `since`, `until`, `user_id`, `app_registration_uuid`, `action` filters and `limit` (default 100, max 1000); returns entries plus `next_since` cursor. The broker streams the file line-by-line with filtering.
|
||||
|
||||
### Web UI
|
||||
|
||||
- **REQ-13 Four pages + dashboard.** Server-rendered HTML from `coreci_broker_ui/`, no build step, same port as the API. `/ui/login` (the CEO's bookmark), `/ui/app_registrations` (table + add-modal + revoke/reinstate per row), `/ui/user_allowlist` (same), `/ui/audit` (filterable, read-only, no export), plus `/ui/dashboard` and a persistent nav header with Logout. After each action the page refreshes via the corresponding API.
|
||||
|
||||
### Ops & Bootstrap
|
||||
|
||||
- **REQ-14 `.env` config + first boot.** The broker reads its `.env` at `BROKER_ENV_PATH` on boot (spec §5.1 contract plus the documented additions). On first boot it generates `BROKER_INSTANCE_PEPPER` (32 random bytes, base64 — synced to SQLite) and `BROKER_ADMIN_PASS` (32+ random chars), writes them to `.env`, and prints the admin pass to the startup log exactly once — never again.
|
||||
- **REQ-15 Mode 0600.** `broker.db` and `broker.audit.log` are created mode 0600, owned by the Open WebUI process user. `.env.secrets` discipline (ciagent) mirrors this.
|
||||
- **REQ-16 Persistence across restarts.** SQLite and the audit file are the durable state. Restarts preserve registrations, allowlist entries, sessions' data, and audit history; in-memory caches (verified-token cache) rebuild lazily; existing tokens still verify.
|
||||
- **REQ-17 Single-file constraint.** `coreci_broker.py` is one Python file under 800 lines, excluding tests, UI HTML, the client module, and reference tools. A line-count test enforces it (AC14).
|
||||
- **REQ-18 Runtime modes.** Importable as an Open WebUI workspace tool (daemon-thread server + a `Tools` class surface) and runnable standalone (`python3 coreci_broker.py`). Stdlib-only at runtime; bcrypt is the sole optional import (D-001/D-009). Tests use unittest against an in-process server on an ephemeral port.
|
||||
|
||||
### Client Ecosystem
|
||||
|
||||
- **REQ-19 Public client module.** `coreci_broker_client.py` — dependency-free (stdlib urllib; no httpx, no bcrypt) — implements the tool-side bootstrap of spec §9: ensure-registered (idempotent), state get/put, secret read with 401 → re-register → retry-once, and never assumes the pepper key is present in register responses (`data.get("pepper")`). Served unauthenticated at `/static/coreci_broker_client.py` so any OWUI tool author can fetch it into their workspace and import it. It contains no secrets.
|
||||
- **REQ-20 Reference tools.** `reference_tools/` ships at least: (a) a proxmox-style reference implementing §11's flow — first call registers and persists state, subsequent calls read the token from state and fetch `coreci/proxmox`, downstream call defaults to a safe dry-run; (b) a minimal showcase of auth-first-then-use-a-second-tool. These are templates for tool authors, wired to the test broker in tests.
|
||||
|
||||
## v2 (Deferred)
|
||||
|
||||
None defined. Future candidates held as out-of-scope decisions, not requirements: audit rotation with retention, multiple broker instances, approver-agent delegation, OWUI-KB credential sync.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
| Feature | Reason |
|
||||
|---------|--------|
|
||||
| Multi-instance federation | Spec §3 — one broker on the Open WebUI host |
|
||||
| Secret versioning / history | Spec §3 — PUT overwrites; broker is read-only on the KB |
|
||||
| High availability | Spec §3 — SQLite is the source of truth; back it up |
|
||||
| Web UI beyond the four pages + dashboard | Spec §3, §10 |
|
||||
| Broker writes to `creds-coreci-coo` | Spec §3 — the CEO writes the KB via Open WebUI's UI |
|
||||
| Audit log rotation | Spec §13 Q1 — CEO default: never rotate; manual `mv` when large |
|
||||
| Multiple broker instances | Spec §13 Q2 — confirmed out of scope for v0.5 |
|
||||
| Approver agents (delegated allowlist adds) | Spec §13 Q4 — CEO is sole approver |
|
||||
| Key rotation ceremony | Spec §14 — pepper and tool UUIDs are stable |
|
||||
| HSM integration for the pepper | Spec §14 |
|
||||
| Rate limiting on auth endpoints | Spec §14 — assumes trusted network |
|
||||
| Multi-factor admin auth | Spec §14 — single password sufficient |
|
||||
| Email/notification on allowlist changes | Spec §14 |
|
||||
| bcrypt as a hard dependency; pip-installable packages | D-001/D-009 — environment has no pip; stdlib-only runtime |
|
||||
|
||||
## Traceability
|
||||
|
||||
| Requirement | Phase | Status |
|
||||
|-------------|-------|--------|
|
||||
| REQ-01 | 2 | pending |
|
||||
| REQ-02 | 3 | pending |
|
||||
| REQ-03 | 1 | complete |
|
||||
| REQ-04 | 3 | pending |
|
||||
| REQ-05 | 2 | pending |
|
||||
| REQ-06 | 2 | pending |
|
||||
| REQ-07 | 4 | pending |
|
||||
| REQ-08 | 4 | pending |
|
||||
| REQ-09 | 3 | pending |
|
||||
| REQ-10 | 3 | pending |
|
||||
| REQ-11 | 1 | complete |
|
||||
| REQ-12 | 2 | pending |
|
||||
| REQ-13 | 2 | pending |
|
||||
| REQ-14 | 1 | complete |
|
||||
| REQ-15 | 1 | complete |
|
||||
| REQ-16 | 1 | complete |
|
||||
| REQ-17 | 1 | complete |
|
||||
| REQ-18 | 1 | complete |
|
||||
| REQ-19 | 4 | pending |
|
||||
| REQ-20 | 4 | pending |
|
||||
@@ -0,0 +1,82 @@
|
||||
# RESEARCH — CoreCI Broker
|
||||
|
||||
Date: 2026-10-09 · Milestone: v0.1 · Phase: 0 (pre-execution) · Status: complete
|
||||
Sources: `.ciagent/` PROJECT/REQUIREMENTS/ARCHITECTURE/ROADMAP, git log (init ffdc5392, clarify 53d99e80), docs.openwebui.com + open-webui source at `main` (fetched 2026-10-09). No scope changes; D-001..D-015 reconfirmed; no secrets read or printed.
|
||||
|
||||
## 1. Environment Constraints (verified facts — do not re-verify)
|
||||
|
||||
- Python 3.11.2 on the dev host; NO pip, NO bcrypt locally → D-001 dual-mode hashing active: stdlib `hashlib.scrypt` in dev/tests, bcrypt in the OWUI runtime; hashes format-prefixed (`$2b$…` / `scrypt$…`). Tests = stdlib unittest (D-009). Pure-Python repo, no package.json.
|
||||
- Forge: gitea `git.coreci.dev`, org `coreci`; release target `coreci/broker` — repo created at first push/ship, public, default branch `main` (D-010/D-012). `GITEA_TOKEN` resolves from `.ciagent/.env.secrets` only (gitignored, mode 0600, never printed).
|
||||
- Stdlib components carrying the runtime: `http.server.ThreadingHTTPServer`, `sqlite3`, `hashlib` (scrypt/sha256), `secrets`, `json`, `base64`, `urllib` (client module), `threading`.
|
||||
- Repo state verified at research time: branches `main`, `milestone/v0.1-coreci-broker`, `phase/00-pre-execution`; `.gitignore` at root; `.ciagent/` holds PROJECT/REQUIREMENTS/ARCHITECTURE/ROADMAP/config.json/CHECKPOINT.json + `.env.secrets` (unread, 0600).
|
||||
|
||||
## 2. Spec Defects & Resolutions (PROJECT.md §Context; decisions in init/clarify commits)
|
||||
|
||||
| # | Defect | Resolution | Test implication |
|
||||
|---|--------|------------|------------------|
|
||||
| 1 | §7.5 in-memory `token_hash` index impossible with bcrypt (salted hashes neither comparable nor indexable) | D-004: UUID-scoped slow-hash scan + in-memory verified-token cache; SQLite authoritative, cache never authoritative | Cover both paths: cache-miss scan and cache-hit re-check of row status/expiry; both hash formats |
|
||||
| 2 | §6 (tool derives every call) vs §9 (tool stores broker-returned token) contradiction | Broker derives authoritatively; tool stores via state endpoints (ARCHITECTURE.md DataFlow 2–4) | AC4: broker- and tool-side derivation byte-identical (§6 formula); state GET/PUT roundtrip |
|
||||
| 3 | §9 snippet `KeyError` on `data["pepper"]` for re-registrations | D-005: pepper returned only on row-create or tokenless register; REQ-19 mandates `data.get("pepper")` in the client | Re-registration without a pepper key must not crash the client; register still audited |
|
||||
| 4 | Pepper chicken-and-egg (token needed to read state; pepper needed to derive) | Tokenless register returns the pepper — the only KB-loss recovery path (D-005) | Recovery scenario: no valid token → register → pepper returned → tool can re-derive |
|
||||
|
||||
## 3. Sibling Conventions (data provided — not fetched)
|
||||
|
||||
- researchx: root = `.ciagent/`, `.gitignore`, `README.md`, `backend/`, `frontend/` — public repo, default branch main.
|
||||
- nextcraft: root = `.ciagent/`, `apps/`, `packages/`, `deploy/`, `scripts/`, pnpm workspace, `Dockerfile`, `.gitea/` — public repo, default branch main.
|
||||
- Implications: flat single-project `.ciagent/` (matches D-010); `README.md` at repo root expected at ship (P5); per-phase tags on the v0.1.x line (D-008); `---ci---` block in every commit; ships via gitea releases once the forge repo exists.
|
||||
|
||||
## 4. Open WebUI Integration Surface (researched 2026-10-09; docs live, post-v0.10 era)
|
||||
|
||||
### Workspace Tools — `Tools` class surface (docs: /features/extensibility/plugin/tools/development)
|
||||
- Single Python file; frontmatter docstring keys: `title`, `author`, `author_url`, `git_url`, `description`, `required_open_webui_version`, `requirements`, `version`, `licence`. Broker ships `requirements:` empty — stdlib-only + bcrypt already present in the OWUI runtime (PROJECT.md Constraints).
|
||||
- `Tools` class rules: `__init__` MUST set `self.valves = self.Valves()` (stored admin valves load only onto an instance that has a `valves` attribute); optional `Valves`/`UserValves` (pydantic BaseModel); every public callable is exposed to the model → helpers must be `_`-prefixed; method docstrings (reST `:param name:` form) + type hints generate the JSON schema; return str/dict/list; async methods preferred.
|
||||
- **Lifecycle: there is NO `activate()`/`deactivate()` hook for Tools.** The loader (`backend/open_webui/utils/plugin.py`) execs the source into a `tool_{id}` module in `sys.modules`, instantiates `module.Tools()` at load (create/update) and caches it on `app.state.TOOLS`; re-exec occurs on source change; module-top-level side effects persist, and disabling a tool does NOT undo them. ⇒ ARCHITECTURE.md's "mounts the server at activation" maps to constructor/module-load time: start the daemon-thread `ThreadingHTTPServer` in `__init__` (or module top level) behind a once-per-process idempotency guard.
|
||||
- Loader sets the module's `__file__` to a temp file deleted after exec ⇒ the broker must resolve DB/audit/env paths from config/env vars, never from `__file__`.
|
||||
- User context: OWUI provides no `self.user` attribute; the per-call reserved arg `__user__` is a dict (`id`, `email`, `name`, `role`, plus `valves` when UserValves defined) (docs: /features/extensibility/plugin/development/reserved-args). Broker admin auth (`BROKER_ADMIN_PASS`) is independent of OWUI identity; ARCHITECTURE.md's "self.user supplies admin-agnostic context" is the broker's own convention — the broker's Tools class can stash `__user__` or ignore it. Nuance only, no architecture change.
|
||||
- Background infra: OWUI's documented pattern is `asyncio.create_task` (Under the Hood §Patterns 4); the broker's stdlib analog is `threading.Thread(target=serve_forever, daemon=True)` — same once-per-process guard required (module re-exec on edits).
|
||||
- Asset serving: a tool may register FastAPI routes via `app.add_api_route` with an idempotency guard (Under the Hood §Patterns 3) — but such routes are unauthenticated unless gated. The broker deliberately serves UI + `/static/` from its OWN `ThreadingHTTPServer` on `127.0.0.1:8765` — no FastAPI coupling; `/static/coreci_broker_client.py` is intentionally unauthenticated (REQ-19).
|
||||
- Security posture: workspace tools execute unsandboxed Python in the server process (docs Plugin Security Warning) — the broker mount inherits that trust boundary.
|
||||
|
||||
### Knowledge Bases API — read path (source-verified: `backend/open_webui/routers/knowledge.py`, `files.py` at main, 2026-10-09)
|
||||
- Auth: `Authorization: Bearer <API_KEY>` (sk-… key from Settings→Account; JWT also accepted; `x-api-key` alt header via `CUSTOM_API_KEY_HEADER`) — feeds `BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY` (REQ-08/D-002). Exact layout deferred to P4 per ARCHITECTURE.md §Notes; candidates below.
|
||||
- Router `/api/v1/knowledge` (all endpoints `get_verified_user`):
|
||||
- `GET /api/v1/knowledge/?page=N` — paginated KB list (30/page) → `{items:[{id,name,description,write_access,…}], total}`
|
||||
- `GET /api/v1/knowledge/search?query=&page=` — same shape; candidate for finding `creds-coreci-coo` by name
|
||||
- `GET /api/v1/knowledge/{id}` — KB detail + `files[]` (per-file metadata) + `write_access`
|
||||
- `GET /api/v1/knowledge/{id}/files?query=&include_content=&page=` — file list; `include_content=true` embeds extracted content (docs flag it as expensive)
|
||||
- `GET /api/v1/knowledge/{id}/files/pending`, `GET /api/v1/knowledge/{id}/export` (zip) — auxiliary
|
||||
- Write ops exist (`/create`, `/{id}/update`, `/{id}/file/add|update|remove|move`, `/{id}/files/batch/add`, `/{id}/delete`, `/{id}/reset`, `/{id}/sync/*`, `/{id}/dirs/*`) — the broker never calls them (read-only, REQ-08).
|
||||
- Router `/api/v1/files`:
|
||||
- `GET /api/v1/files/{id}` → FileModel; `data.content` holds extracted text
|
||||
- `GET /api/v1/files/{id}/data/content` → `{"content": "<extracted text>"}` — JSON form, best fit for credential JSON files
|
||||
- `GET /api/v1/files/{id}/content` → raw bytes (FileResponse)
|
||||
- `GET /api/v1/files/{id}/process/status` → `{"status": "pending"|"completed"|"failed"}`
|
||||
- `GET /api/v1/files/?page=` (`content=true` default), `GET /api/v1/files/search?filename=<glob>`
|
||||
- Candidate P4 flow: search KB by name → map path→file (filename convention TBD in P4) → `GET /api/v1/files/{file_id}/data/content` → parse JSON → `{path, data, version, updated_at}`. Note: OWUI exposes `meta.size`/`updated_at` but no version field → `version` may be synthesized broker-side (P4 detail; REQ-08 response shape unchanged).
|
||||
|
||||
## 5. Security Prior Art
|
||||
|
||||
- Peppered derivation + slow-hash storage: the token derives from a 256-bit random pepper (`BROKER_INSTANCE_PEPPER`, 32 bytes), so an attacker holding `token_hash` faces an offline brute force over the pepper — infeasible. Online guessing is rate-limit-free (spec §14: trusted network + `127.0.0.1` bind), but each guess costs one scrypt/bcrypt op, and the UUID-scoped scan bounds verification work (D-004). Storing only the salted slow hash (never the plaintext token) follows OWASP password-storage guidance; format prefixes keep verification format-aware (D-001).
|
||||
- Append-only JSONL audit: one JSON object per line; flush-per-write (+ `os.fsync` for the process-restart case) is affordable at the broker's write volume (register/read/admin actions) and preserves the no-loss guarantee; file mode 0600; no rotation — manual `mv` (D-007, spec §13 Q1). Exact flush/fsync choice is a P1 plan decision, not research.
|
||||
- Session cookie `broker_session`: HttpOnly + SameSite=Lax (D-013); no Secure flag (loopback HTTP by default); Path scoped to the broker.
|
||||
- Path traversal on `/secrets/<path>`: URL-decode (`urllib.parse.unquote`) BEFORE validation; reject `..` segments, absolute paths, backslashes, NUL; the scope check `path == "coreci/" + agent_id` runs on the decoded value (REQ-07). Static serving of `/static/` and UI must likewise resolve against a fixed root and reject traversal.
|
||||
- OWUI-runtime boundary: broker binds `127.0.0.1:8765` — bind address controls reachability (PROJECT.md Constraints). The OWUI process hosts both the tool (broker mount) and the KB the broker reads; `BROKER_OWUI_API_KEY` is the only OWUI-side credential the broker holds, and it lives in the broker `.env` (0600 discipline, REQ-15).
|
||||
|
||||
## 6. Testing Prior Art
|
||||
|
||||
- In-process server harness (stdlib, matches D-009/REQ-18): `ThreadingHTTPServer(("127.0.0.1", 0), Handler)` for an ephemeral port in `setUp`; `threading.Thread(target=server.serve_forever, daemon=True)`; drive requests with `urllib.request` against `http://127.0.0.1:{port}`; `shutdown()` + `server_close()` via `addCleanup`.
|
||||
- Temp state: `tempfile.TemporaryDirectory()` per test for `broker.db`, `broker.audit.log`, `.env`, `data/creds.json`; point the broker at them via env vars (`BROKER_ENV_PATH`, `BROKER_CREDS_FILE`, audit path).
|
||||
- 0600 modes: `os.stat(path).st_mode & 0o777 == 0o600` after first boot; create with `os.open(path, O_CREAT|O_WRONLY, 0o600)` (umask-independent) in the broker, so the test just asserts.
|
||||
- AC14 line-count test (REQ-17): `coreci_broker.py` is a single file — tests/UI HTML/client module/reference tools are separate files, so a plain line count of that one file is the spec's measure (< 800, spec §12.14); test reads the file and asserts `sum(1 for _ in f) <= 800` (wc -l-style, no comment stripping).
|
||||
- TTL tests: set `BROKER_TOKEN_TTL` to 1–2 s and `time.sleep`, or inject an expired `token_expires_at` directly in SQLite (faster, deterministic).
|
||||
- Dual-hash tests: both `scrypt$…` and `$2b$…` verification paths; CI can only execute the scrypt path (no bcrypt locally) — the bcrypt path's format-awareness is tested via the prefix and mocked/verified against the format contract.
|
||||
|
||||
## 7. Research Conclusions
|
||||
|
||||
- P1 (skeleton): bootstrap `.env` + first-boot generation (pepper/admin pass printed exactly once — log once, never again on second boot); SQLite schema per spec §5.2 at 0600; audit writer (JSONL, 0600); dual-mode hasher with format prefixes; token engine pure-function of (pepper, inputs); ThreadingHTTPServer + `{"error, message}` router; OWUI mount = `Tools.__init__` starts the daemon-thread server behind a once-per-process guard (no activate() hook exists), `self.valves = self.Valves()` required, `_`-prefixed helpers; standalone `__main__`; path resolution from env, never `__file__`; keep the 800-line budget from the first commit (UI/client/reference are separate files).
|
||||
- P2 (admin plane): cookie `broker_session` HttpOnly+SameSite=Lax; HTTP Basic + session cookie both accepted on all admin API endpoints; four pages + dashboard served from `coreci_broker_ui/` on the same port; audit query streams the file line-by-line with `next_since` cursor (limit default 100/max 1000).
|
||||
- P3 (tool auth): register semantics per D-005 (idempotent re-derive, overwrite on input change, pepper on row-create-or-tokenless); verification per D-004 (UUID-scoped slow-hash scan; cache hit re-checks row status/expiry); `GET /agents/<id>` per D-015 (metadata-only); state endpoints with identity enforcement; `BROKER_AGENT_STATE_BACKEND=internal` default.
|
||||
- P4 (secrets + ecosystem): `OwuiApiCredsSource` builds on the KB read endpoints in §4 (Bearer key; `/{id}` + `/{id}/files` for mapping, `/api/v1/files/{id}/data/content` for content); `FileCredsSource` default `data/creds.json` (D-014); client module stdlib-urllib-only, `data.get("pepper")`, served unauthenticated at `/static/`; reference tools with safe dry-run default.
|
||||
- P5 (review + ship): AC1–AC13 behavioral + AC14 line-count; security checklist (traversal incl. URL-decoded forms, pepper never re-printed, 0600 modes, token scoping, audit completeness); README with documented deviations; ship to gitea (repo exists before first push).
|
||||
- ARCHITECTURE.md verification: **no assumption changes required.** The adapter-contract note (OWUI KB-file layout resolved in P4, ARCHITECTURE.md §Notes) stands — candidate endpoints captured in §4. One nuance for P1, not a defect: "mounts the server at activation" maps to constructor-time in the actual OWUI loader (no activate()/deactivate() lifecycle for Tools), and OWUI supplies `__user__` per call rather than a `self.user` attribute — the broker's Tools class owns that convention.
|
||||
- No genuine blockers found. Environment, OWUI surface, and forge conventions all support the v0.1 scope as specified.
|
||||
@@ -0,0 +1,60 @@
|
||||
# Roadmap — Milestone v0.1 (coreci-broker)
|
||||
|
||||
## Overview
|
||||
|
||||
One milestone, owner-directed 2026-10-09: implement Requirements Specification v0.5 in full. **Milestone v0.1 — coreci-broker** (feature type): the single-file broker, its SQLite store, audit log, four-page admin UI, tool-auth plane, secrets mediation, public client module, and reference tools, with the spec's §12 acceptance matrix (AC1–AC14) green. The spec's "v0.5" is its requirements-doc revision label, not the product version (D-008).
|
||||
|
||||
Versioning (greenfield deviation, researchx/nextcraft precedent): the milestone branch is `milestone/v0.1-coreci-broker`; phase tags run on the milestone's own line — phase 0 ships `v0.1.0`, execution phases ship `v0.1.1`..`v0.1.4`, and the final phase's patch (`v0.1.5`) IS the milestone release; no separate minor tag. MAJOR remains 0 until an owner-approved v1 release. Ships go to gitea (`git.coreci.dev/coreci/broker`); the forge repo must exist before the first push.
|
||||
|
||||
Autonomy: **guided** — escalate on every decision gate; threshold 0.85; 1 revision iteration; clarify budget 20.
|
||||
|
||||
## Phase 0 — Pre-Execution
|
||||
|
||||
- **Description**: Specification parsing, clarification (decisions D-001..D-011 recorded in the init commit), `.ciagent/` reference files, and the pre-execution pipeline stages: specify → clarify → research → plan → grill. Init has landed the reference files; `/ci-run` phase 0 completes RESEARCH.md (environment constraints, spec defects, sibling conventions), PLAN.md (task-level breakdown), PERSONAS.md, and the GRILL verdict before any code.
|
||||
- **Requirements covered**: none (planning artifacts)
|
||||
- **Success criteria**: `.ciagent/` files reflect the 6-phase structure; REQ-01..REQ-20 defined and traceable; milestone v0.1 established in config.json and PROJECT.md; PLAN.md + PERSONAS.md committed; grill verdict recorded; phase tagged `v0.1.0`
|
||||
- **Depends on**: —
|
||||
- **Status**: complete
|
||||
|
||||
## Phase 1 — Core Skeleton
|
||||
|
||||
- **Description**: `coreci_broker.py` foundation: `.env` read/write + first-boot generation (pepper + admin pass, printed once), SQLite schema + indexes per spec §5.2, mode-0600 on DB and audit log, append-only JSONL audit writer per §5.3, dual-mode bcrypt/scrypt hasher (D-001), server-side token derivation per §6, stdlib `ThreadingHTTPServer` + JSON router + `{"error, message}` contract per §7, Open WebUI `Tools`-class mount (daemon thread) + standalone `__main__` entry.
|
||||
- **Requirements covered**: REQ-03, REQ-11, REQ-14, REQ-15, REQ-16, REQ-17, REQ-18
|
||||
- **Success criteria**: broker boots standalone on an ephemeral port, creates DB + audit log at 0600, prints the generated admin pass exactly once (not on second boot); token derivation matches the §6 formula byte-for-byte; source under 800 lines
|
||||
- **Depends on**: 0
|
||||
- **Status**: complete
|
||||
|
||||
## Phase 2 — Admin Plane
|
||||
|
||||
- **Description**: Session auth (cookie `broker_session` + HTTP Basic both accepted), `POST /api/broker/auth/login`, `logout`, `whoami` (§7.1); app_registrations list/add/revoke/reinstate (§7.2, UUID format validation, 409 on duplicate, revoke keeps existing tokens until TTL); user_allowlist list/add/revoke/reinstate (§7.3); audit query with filters + `next_since` cursor (§7.6); the four UI pages + dashboard + persistent nav (§10) served from `coreci_broker_ui/`; session cleanup.
|
||||
- **Requirements covered**: REQ-01, REQ-05, REQ-06, REQ-12, REQ-13
|
||||
- **Success criteria**: AC1 (first-boot login with the printed pass), AC2 (app registration add/remove via UI, persisted in `app_registrations`, UI updates after each action), AC3 (user allowlist same), AC9's login/logout audit lines
|
||||
- **Depends on**: 1
|
||||
- **Status**: not_started
|
||||
|
||||
## Phase 3 — Tool Auth Plane
|
||||
|
||||
- **Description**: `POST /api/broker/agents/register` (§7.4): allowlist gates → 403 on unknown/revoked UUID or user; idempotent re-registration re-derives from request inputs and overwrites on input change (D-005); pepper returned on row-create or when the caller presents no valid token; TTL set from `BROKER_TOKEN_TTL`. App-auth verification: parse `Authorization: App <UUID> <token>`, UUID-scoped slow-hash scan + in-memory verified-token cache (D-004). `GET /api/broker/agents/<agent_id>` (§7.4). State endpoints `GET/PUT /api/broker/agents/<id>/state` (§9.1) with identity enforcement (calling UUID + token-derived user_id must match stored identity) and the backend toggle `BROKER_AGENT_STATE_BACKEND=internal` (default) `| owui` (D-003).
|
||||
- **Requirements covered**: REQ-02, REQ-04, REQ-09, REQ-10
|
||||
- **Success criteria**: AC4 (token derived + stored as slow-hash, same value computed both sides), AC7 (revoked/unknown UUID → 403), AC8 (unknown user → 403), AC10 (TTL expiry → 401 → re-register → new token, audited), AC12 (state read/write with identity enforcement)
|
||||
- **Depends on**: 2
|
||||
- **Status**: not_started
|
||||
|
||||
## Phase 4 — Secrets + Client Ecosystem
|
||||
|
||||
- **Description**: Creds-source adapter (D-002): `FileCredsSource` (default; JSON document at `BROKER_CREDS_FILE` mapping paths to secret objects) and `OwuiApiCredsSource` (`BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY`). `GET /api/broker/secrets/<path>` (§7.5): App-auth, scope check `path == "coreci/" + agent_id` → 403 on mismatch, 404 when absent, audit every attempt. Public client module `coreci_broker_client.py` (stdlib urllib only): ensure-registered, state get/put, secret read with 401 → re-register → retry-once; served unauthenticated at `/static/coreci_broker_client.py`. Reference OWUI tools (`reference_tools/`): proxmox reference + a minimal auth-then-use showcase, safe dry-run default (§9, §11; D-006).
|
||||
- **Requirements covered**: REQ-07, REQ-08, REQ-19, REQ-20
|
||||
- **Success criteria**: AC5 (valid token reads the secret; tampered token → 401), AC6 (proxmox-registered tool reading `coreci/gitea` → 403); client module fetchable unauthenticated and drives a reference tool end-to-end against a test broker
|
||||
- **Depends on**: 3
|
||||
- **Status**: not_started
|
||||
|
||||
## Phase 5 — Final Review + Ship
|
||||
|
||||
- **Description**: Full §12 acceptance matrix as tests (AC1–AC13 as behavioral tests, AC14 as a line-count test); security review pass (path traversal on `<path>`, pepper never re-printed, 0600 modes, token scoping, audit completeness); README with run/deploy notes and documented spec deviations, including the owner-run post-ship live-OWUI smoke checklist (mount/register/read on the deployed v0.11.3 host — G-005/G-006); multi-persona review of the milestone diff; owner approval gate; milestone merge to main and tag `v0.1.5` (final phase patch = milestone release); first push to gitea once the forge repo exists.
|
||||
- **Requirements covered**: REQ-01..REQ-20 (acceptance)
|
||||
- **Success criteria**: AC1–AC14 all green (`python3 -m unittest discover -s tests`); security review PASS; README complete with deviations; owner approval recorded; milestone squash-merged to main, tagged `v0.1.5`, release created
|
||||
- **Depends on**: 4
|
||||
- **Status**: not_started
|
||||
## Post-Milestone Notes (owner-promoted from GRILL G-008)
|
||||
|
||||
- **90-day adoption criterion**: `proxmox_readonly_tool.py` runs on `coreci_broker_client.py` with direct KB reads removed by **2027-01-09**; the owner owns the date. No real tool migrates to the broker before the post-ship OWUI smoke checklist (G-005) passes on the deployed v0.11.3 host.
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"version": "0.8.0",
|
||||
"autonomy": {
|
||||
"level": "guided",
|
||||
"decision_confidence_threshold": 0.85,
|
||||
"escalation_hooks": ["*"],
|
||||
"max_revision_iterations": 1,
|
||||
"clarify_budget": 20
|
||||
},
|
||||
"release": {
|
||||
"forge": "gitea",
|
||||
"base_url": "https://git.coreci.dev",
|
||||
"owner": "coreci",
|
||||
"repo": "broker"
|
||||
},
|
||||
"secrets": {
|
||||
"scopes": ["gitea"],
|
||||
"env_files": [".env", ".env.secrets", ".env.*"]
|
||||
},
|
||||
"ship": {
|
||||
"per_phase": true,
|
||||
"allow_skip": false,
|
||||
"max_release_retries": 3,
|
||||
"release_blocking": false
|
||||
},
|
||||
"verification": {
|
||||
"bdd_default": false
|
||||
},
|
||||
"personas": {
|
||||
"enabled": true,
|
||||
"territory_enforcement": "warn"
|
||||
},
|
||||
"parallelization": {
|
||||
"enabled": false,
|
||||
"max_concurrent_agents": 1
|
||||
},
|
||||
"ideation": {
|
||||
"max_ideas": 20,
|
||||
"categories": ["security", "quality", "architecture", "coverage", "improvement", "spec", "chaos", "bdd"]
|
||||
},
|
||||
"security": {
|
||||
"bash_allowlist": {
|
||||
"blocked_env_vars": ["GITEA_TOKEN", "GITHUB_TOKEN", "GITLAB_TOKEN"]
|
||||
}
|
||||
},
|
||||
"projects": [],
|
||||
"active_project": null,
|
||||
"milestone": {
|
||||
"version": "v0.1",
|
||||
"name": "coreci-broker",
|
||||
"type": "feature",
|
||||
"branch": "milestone/v0.1-coreci-broker"
|
||||
}
|
||||
}
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# Secrets (ciagent credential discipline)
|
||||
.env
|
||||
.env.secrets
|
||||
.env.*
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# Broker runtime state
|
||||
data/
|
||||
*.db
|
||||
*.log
|
||||
@@ -0,0 +1,618 @@
|
||||
"""title: CoreCI Broker
|
||||
author: coreci
|
||||
description: Single-file credential mediation broker for CoreCI agent tools.
|
||||
version: 0.1.0
|
||||
requirements: []
|
||||
licence: MIT
|
||||
"""
|
||||
import base64
|
||||
import datetime
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import os
|
||||
import secrets
|
||||
import sqlite3
|
||||
import threading
|
||||
import time
|
||||
import urllib.parse
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
try:
|
||||
import bcrypt
|
||||
except ImportError:
|
||||
bcrypt = None
|
||||
|
||||
ENV_PATH_DEFAULT = ".env"
|
||||
|
||||
ENV_KEYS = (
|
||||
"BROKER_ENV_PATH",
|
||||
"BROKER_BIND",
|
||||
"BROKER_ADMIN_USER",
|
||||
"BROKER_ADMIN_PASS",
|
||||
"BROKER_INSTANCE_PEPPER",
|
||||
"BROKER_TOKEN_TTL",
|
||||
"BROKER_SESSION_TTL",
|
||||
"BROKER_DB_PATH",
|
||||
"BROKER_AUDIT_LOG_PATH",
|
||||
"BROKER_UI_ROOT",
|
||||
"BROKER_STATIC_ROOT",
|
||||
)
|
||||
|
||||
ENV_DEFAULTS = {
|
||||
"BROKER_BIND": "127.0.0.1:8765",
|
||||
"BROKER_ADMIN_USER": "admin",
|
||||
"BROKER_TOKEN_TTL": "86400",
|
||||
"BROKER_SESSION_TTL": "300",
|
||||
"BROKER_DB_PATH": "broker.db",
|
||||
"BROKER_AUDIT_LOG_PATH": "broker.audit.log",
|
||||
"BROKER_UI_ROOT": "coreci_broker_ui",
|
||||
"BROKER_STATIC_ROOT": ".",
|
||||
}
|
||||
|
||||
AUDIT_ACTIONS = (
|
||||
"register",
|
||||
"read",
|
||||
"denied",
|
||||
"login",
|
||||
"logout",
|
||||
"add_app",
|
||||
"remove_app",
|
||||
"add_user",
|
||||
"remove_user",
|
||||
)
|
||||
|
||||
AUDIT_RESULTS = ("ok", "denied", "error")
|
||||
|
||||
BCRYPT_PREFIXES = ("$2a$", "$2b$", "$2y$")
|
||||
SCRYPT_PREFIX = "scrypt$"
|
||||
SCRYPT_SALT_BYTES = 16
|
||||
SCRYPT_N = 2 ** 14
|
||||
SCRYPT_R = 8
|
||||
SCRYPT_P = 1
|
||||
|
||||
STATIC_PREFIXES = (
|
||||
("ui", "ui_root"),
|
||||
("static", "static_root"),
|
||||
)
|
||||
|
||||
STATIC_CONTENT_TYPES = {
|
||||
".html": "text/html; charset=utf-8",
|
||||
".css": "text/css; charset=utf-8",
|
||||
".js": "application/javascript; charset=utf-8",
|
||||
".py": "text/x-python; charset=utf-8",
|
||||
".txt": "text/plain; charset=utf-8",
|
||||
".json": "application/json; charset=utf-8",
|
||||
}
|
||||
|
||||
SCHEMA_SQL = """
|
||||
CREATE TABLE IF NOT EXISTS broker_config (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS app_registrations (
|
||||
uuid TEXT PRIMARY KEY,
|
||||
name TEXT,
|
||||
status TEXT DEFAULT 'active',
|
||||
created_at TEXT,
|
||||
revoked_at TEXT
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS user_allowlist (
|
||||
user_id TEXT PRIMARY KEY,
|
||||
email TEXT,
|
||||
status TEXT DEFAULT 'active',
|
||||
created_at TEXT,
|
||||
revoked_at TEXT
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS agent_registrations (
|
||||
agent_id TEXT,
|
||||
app_registration_uuid TEXT,
|
||||
user_id TEXT,
|
||||
tags TEXT,
|
||||
token_hash TEXT,
|
||||
token_expires_at TEXT,
|
||||
first_registered_at TEXT,
|
||||
last_seen_at TEXT,
|
||||
PRIMARY KEY (agent_id, app_registration_uuid)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_registrations_app_uuid
|
||||
ON agent_registrations (app_registration_uuid);
|
||||
CREATE TABLE IF NOT EXISTS admin_sessions (
|
||||
session_id TEXT PRIMARY KEY,
|
||||
created_at TEXT,
|
||||
expires_at TEXT
|
||||
);
|
||||
"""
|
||||
|
||||
|
||||
def _parse_env_file(path):
|
||||
entries = []
|
||||
if not os.path.isfile(path):
|
||||
return entries
|
||||
with open(path, "r", encoding="utf-8") as handle:
|
||||
for raw in handle:
|
||||
line = raw.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, _, value = line.partition("=")
|
||||
key = key.strip()
|
||||
value = value.strip()
|
||||
if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
|
||||
value = value[1:-1]
|
||||
if key:
|
||||
entries.append((key, value))
|
||||
return entries
|
||||
|
||||
|
||||
def _append_env_entries(path, entries):
|
||||
needs_newline = False
|
||||
if os.path.exists(path) and os.path.getsize(path) > 0:
|
||||
with open(path, "rb") as handle:
|
||||
handle.seek(-1, os.SEEK_END)
|
||||
needs_newline = handle.read(1) != b"\n"
|
||||
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o600)
|
||||
try:
|
||||
os.fchmod(fd, 0o600)
|
||||
with os.fdopen(fd, "a", encoding="utf-8") as handle:
|
||||
if needs_newline:
|
||||
handle.write("\n")
|
||||
for key, value in entries:
|
||||
handle.write(f"{key}={value}\n")
|
||||
except BaseException:
|
||||
os.close(fd)
|
||||
raise
|
||||
|
||||
|
||||
def _generate_pepper():
|
||||
return base64.b64encode(secrets.token_bytes(32)).decode("ascii")
|
||||
|
||||
|
||||
def _generate_admin_pass():
|
||||
return secrets.token_urlsafe(32)
|
||||
|
||||
|
||||
def _utc_now_iso():
|
||||
return datetime.datetime.now(datetime.timezone.utc).isoformat()
|
||||
|
||||
|
||||
def _b64url_encode(raw):
|
||||
return base64.urlsafe_b64encode(raw).decode("ascii")
|
||||
|
||||
|
||||
def _b64url_decode(text):
|
||||
return base64.urlsafe_b64decode(text.encode("ascii"))
|
||||
|
||||
|
||||
def hash_token(token):
|
||||
token_bytes = token.encode("utf-8")
|
||||
if bcrypt is not None:
|
||||
return bcrypt.hashpw(token_bytes, bcrypt.gensalt()).decode("ascii")
|
||||
salt = secrets.token_bytes(SCRYPT_SALT_BYTES)
|
||||
digest = hashlib.scrypt(token_bytes, salt=salt, n=SCRYPT_N, r=SCRYPT_R, p=SCRYPT_P)
|
||||
return f"{SCRYPT_PREFIX}{_b64url_encode(salt)}${_b64url_encode(digest)}"
|
||||
|
||||
|
||||
def verify_token(token, stored):
|
||||
if not isinstance(stored, str) or not stored:
|
||||
return False
|
||||
token_bytes = token.encode("utf-8")
|
||||
if stored.startswith(BCRYPT_PREFIXES):
|
||||
if bcrypt is None:
|
||||
return False
|
||||
try:
|
||||
return bcrypt.checkpw(token_bytes, stored.encode("ascii"))
|
||||
except (ValueError, TypeError):
|
||||
return False
|
||||
if stored.startswith(SCRYPT_PREFIX):
|
||||
fields = stored[len(SCRYPT_PREFIX):].split("$")
|
||||
if len(fields) != 2:
|
||||
return False
|
||||
try:
|
||||
salt = _b64url_decode(fields[0])
|
||||
expected = _b64url_decode(fields[1])
|
||||
actual = hashlib.scrypt(token_bytes, salt=salt, n=SCRYPT_N, r=SCRYPT_R, p=SCRYPT_P)
|
||||
except (ValueError, TypeError):
|
||||
return False
|
||||
return hmac.compare_digest(actual, expected)
|
||||
return False
|
||||
|
||||
|
||||
def derive_token(pepper, app_uuid, user_id, agent_id, tags):
|
||||
material = f"{pepper}:{app_uuid}:{user_id}:{agent_id}:{','.join(sorted(tags))}"
|
||||
return base64.b64encode(hashlib.sha256(material.encode("utf-8")).digest()).decode("ascii")
|
||||
|
||||
|
||||
def load_config(env=None):
|
||||
if env is None:
|
||||
env = os.environ
|
||||
env_path = str(env.get("BROKER_ENV_PATH") or ENV_PATH_DEFAULT)
|
||||
file_values = dict(_parse_env_file(env_path))
|
||||
generated = []
|
||||
if "BROKER_INSTANCE_PEPPER" not in env and "BROKER_INSTANCE_PEPPER" not in file_values:
|
||||
generated.append(("BROKER_INSTANCE_PEPPER", _generate_pepper()))
|
||||
if "BROKER_ADMIN_PASS" not in env and "BROKER_ADMIN_PASS" not in file_values:
|
||||
generated.append(("BROKER_ADMIN_PASS", _generate_admin_pass()))
|
||||
if generated:
|
||||
_append_env_entries(env_path, generated)
|
||||
file_values.update(generated)
|
||||
config = {"BROKER_ENV_PATH": env_path}
|
||||
for key in ENV_KEYS:
|
||||
if key == "BROKER_ENV_PATH":
|
||||
continue
|
||||
config[key] = str(env.get(key) or file_values.get(key) or ENV_DEFAULTS.get(key, ""))
|
||||
for key, value in generated:
|
||||
if key == "BROKER_ADMIN_PASS":
|
||||
print(f"BROKER_ADMIN_PASS={value}", flush=True)
|
||||
return config
|
||||
|
||||
|
||||
class Store:
|
||||
def __init__(self, path):
|
||||
self.path = path
|
||||
fd = os.open(path, os.O_CREAT | os.O_WRONLY, 0o600)
|
||||
try:
|
||||
os.fchmod(fd, 0o600)
|
||||
finally:
|
||||
os.close(fd)
|
||||
self.conn = sqlite3.connect(path)
|
||||
self.conn.executescript(SCHEMA_SQL)
|
||||
|
||||
def set_config(self, key, value):
|
||||
self.conn.execute(
|
||||
"INSERT OR REPLACE INTO broker_config (key, value) VALUES (?, ?)",
|
||||
(key, str(value)),
|
||||
)
|
||||
self.conn.commit()
|
||||
|
||||
def get_config(self, key):
|
||||
row = self.conn.execute(
|
||||
"SELECT value FROM broker_config WHERE key = ?", (key,)
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
return row[0]
|
||||
|
||||
def close(self):
|
||||
self.conn.close()
|
||||
|
||||
|
||||
class Audit:
|
||||
def __init__(self, path):
|
||||
self.path = path
|
||||
self._lock = threading.Lock()
|
||||
self._handle = None
|
||||
fd = os.open(path, os.O_CREAT | os.O_WRONLY, 0o600)
|
||||
try:
|
||||
os.fchmod(fd, 0o600)
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
def write(
|
||||
self,
|
||||
action,
|
||||
path="",
|
||||
result="ok",
|
||||
user_id=None,
|
||||
app_registration_uuid=None,
|
||||
agent_id=None,
|
||||
):
|
||||
if action not in AUDIT_ACTIONS:
|
||||
raise ValueError(f"unknown audit action: {action!r}")
|
||||
if result not in AUDIT_RESULTS:
|
||||
raise ValueError(f"unknown audit result: {result!r}")
|
||||
entry = {
|
||||
"ts": _utc_now_iso(),
|
||||
"user_id": user_id,
|
||||
"app_registration_uuid": app_registration_uuid,
|
||||
"agent_id": agent_id,
|
||||
"action": action,
|
||||
"path": path,
|
||||
"result": result,
|
||||
}
|
||||
line = json.dumps(entry, separators=(",", ":")) + "\n"
|
||||
with self._lock:
|
||||
if self._handle is None:
|
||||
self._handle = open(self.path, "a", encoding="utf-8")
|
||||
self._handle.write(line)
|
||||
self._handle.flush()
|
||||
os.fsync(self._handle.fileno())
|
||||
|
||||
def close(self):
|
||||
with self._lock:
|
||||
if self._handle is not None:
|
||||
self._handle.close()
|
||||
self._handle = None
|
||||
|
||||
|
||||
class Broker:
|
||||
def __init__(self, config, store, audit):
|
||||
self.config = config
|
||||
self.store = store
|
||||
self.audit = audit
|
||||
self.ui_root = config.get("BROKER_UI_ROOT") or "coreci_broker_ui"
|
||||
self.static_root = config.get("BROKER_STATIC_ROOT") or "."
|
||||
self.router = Router()
|
||||
self.router.add("GET", "/healthz", _healthz)
|
||||
|
||||
|
||||
def boot(env=None):
|
||||
config = load_config(env)
|
||||
store = Store(config["BROKER_DB_PATH"])
|
||||
store.set_config("BROKER_INSTANCE_PEPPER", config["BROKER_INSTANCE_PEPPER"])
|
||||
audit = Audit(config["BROKER_AUDIT_LOG_PATH"])
|
||||
return Broker(config, store, audit)
|
||||
|
||||
|
||||
class _BadRequest(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def _healthz(request):
|
||||
request.send_json(200, {"ok": True})
|
||||
|
||||
|
||||
def _parse_query(query):
|
||||
values = urllib.parse.parse_qs(query, keep_blank_values=True)
|
||||
return {key: value[0] for key, value in values.items()}
|
||||
|
||||
|
||||
def _match_pattern(pattern, path):
|
||||
pattern_parts = pattern.split("/")
|
||||
path_parts = path.split("/")
|
||||
if len(pattern_parts) != len(path_parts):
|
||||
return None
|
||||
params = {}
|
||||
for expected, actual in zip(pattern_parts, path_parts):
|
||||
if len(expected) > 2 and expected.startswith("<") and expected.endswith(">"):
|
||||
if not actual:
|
||||
return None
|
||||
params[expected[1:-1]] = actual
|
||||
elif expected != actual:
|
||||
return None
|
||||
return params
|
||||
|
||||
|
||||
def _static_content_type(name):
|
||||
extension = os.path.splitext(name)[1].lower()
|
||||
return STATIC_CONTENT_TYPES.get(extension, "application/octet-stream")
|
||||
|
||||
|
||||
def _broker_protected_paths(broker):
|
||||
protected = set()
|
||||
for key in ("BROKER_ENV_PATH", "BROKER_DB_PATH", "BROKER_AUDIT_LOG_PATH"):
|
||||
value = broker.config.get(key) or ""
|
||||
if not value:
|
||||
continue
|
||||
base = os.path.realpath(value)
|
||||
protected.add(base)
|
||||
for suffix in ("-journal", "-wal", "-shm"):
|
||||
protected.add(base + suffix)
|
||||
return protected
|
||||
|
||||
|
||||
def _resolve_static_file(root, rest, protected=()):
|
||||
if "\x00" in rest or "\\" in rest or rest.startswith("/"):
|
||||
return None
|
||||
parts = rest.split("/")
|
||||
if ".." in parts or any(part.startswith(".") for part in parts):
|
||||
return None
|
||||
stripped = rest.strip("/")
|
||||
names = []
|
||||
if stripped:
|
||||
names.append(stripped)
|
||||
if "." not in os.path.basename(stripped):
|
||||
names.append(stripped + ".html")
|
||||
else:
|
||||
names.append("index.html")
|
||||
root_real = os.path.realpath(root)
|
||||
for name in names:
|
||||
target = os.path.realpath(os.path.join(root_real, name))
|
||||
if not target.startswith(root_real + os.sep):
|
||||
continue
|
||||
if target in protected:
|
||||
continue
|
||||
if os.path.isfile(target):
|
||||
return target
|
||||
return None
|
||||
|
||||
|
||||
class Router:
|
||||
def __init__(self):
|
||||
self._routes = []
|
||||
|
||||
def add(self, method, pattern, handler):
|
||||
self._routes.append((method.upper(), pattern, handler))
|
||||
|
||||
def resolve(self, method, path):
|
||||
for route_method, pattern, handler in self._routes:
|
||||
if route_method != method:
|
||||
continue
|
||||
params = _match_pattern(pattern, path)
|
||||
if params is not None:
|
||||
return handler, params
|
||||
return None, None
|
||||
|
||||
|
||||
class BrokerRequestHandler(BaseHTTPRequestHandler):
|
||||
broker = None
|
||||
responded = False
|
||||
path_params = None
|
||||
query = None
|
||||
body = None
|
||||
|
||||
def log_message(self, format, *args):
|
||||
pass
|
||||
|
||||
def do_GET(self):
|
||||
self._dispatch("GET")
|
||||
|
||||
def do_POST(self):
|
||||
self._dispatch("POST")
|
||||
|
||||
def do_PUT(self):
|
||||
self._dispatch("PUT")
|
||||
|
||||
def do_DELETE(self):
|
||||
self._dispatch("DELETE")
|
||||
|
||||
def send_json(self, status, payload):
|
||||
if self.responded:
|
||||
return
|
||||
self.responded = True
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", "application/json; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def send_error_json(self, status, message):
|
||||
self.send_json(status, {"error": status, "message": message})
|
||||
|
||||
def _read_body(self):
|
||||
header = self.headers.get("Content-Length") or "0"
|
||||
try:
|
||||
length = int(header)
|
||||
except ValueError:
|
||||
raise _BadRequest("invalid Content-Length")
|
||||
if length <= 0:
|
||||
return b""
|
||||
return self.rfile.read(length)
|
||||
|
||||
def _dispatch(self, method):
|
||||
self.path_params = {}
|
||||
self.query = {}
|
||||
self.body = None
|
||||
try:
|
||||
parsed = urllib.parse.urlsplit(self.path)
|
||||
path = urllib.parse.unquote(parsed.path)
|
||||
self.query = _parse_query(parsed.query)
|
||||
raw = self._read_body() if method in ("POST", "PUT") else b""
|
||||
if self._serve_static(method, path):
|
||||
return
|
||||
handler, params = self.broker.router.resolve(method, path)
|
||||
if handler is None:
|
||||
self.send_error_json(404, "not found")
|
||||
return
|
||||
if raw:
|
||||
try:
|
||||
self.body = json.loads(raw)
|
||||
except ValueError:
|
||||
self.send_error_json(400, "invalid JSON body")
|
||||
return
|
||||
self.path_params = params
|
||||
handler(self)
|
||||
except _BadRequest as error:
|
||||
self.send_error_json(400, str(error))
|
||||
except Exception:
|
||||
if not self.responded:
|
||||
self.send_error_json(500, "internal error")
|
||||
|
||||
def _serve_static(self, method, path):
|
||||
if method != "GET":
|
||||
return False
|
||||
for prefix, attr in STATIC_PREFIXES:
|
||||
if path.startswith("/" + prefix + "/"):
|
||||
self._send_static_file(getattr(self.broker, attr), path[len(prefix) + 2:])
|
||||
return True
|
||||
return False
|
||||
|
||||
def _send_static_file(self, root, rest):
|
||||
target = _resolve_static_file(
|
||||
root, rest, _broker_protected_paths(self.broker)
|
||||
)
|
||||
if target is None:
|
||||
self.send_error_json(404, "not found")
|
||||
return
|
||||
with open(target, "rb") as handle:
|
||||
data = handle.read()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", _static_content_type(target))
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
|
||||
def serve(host, port, broker):
|
||||
class BoundHandler(BrokerRequestHandler):
|
||||
pass
|
||||
|
||||
BoundHandler.broker = broker
|
||||
server = ThreadingHTTPServer((host, port), BoundHandler)
|
||||
server.broker = broker
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
return server
|
||||
|
||||
|
||||
_server_lock = threading.Lock()
|
||||
_started_server = None
|
||||
|
||||
|
||||
def _parse_bind(bind):
|
||||
host, _, port = str(bind).rpartition(":")
|
||||
if not port.isdigit():
|
||||
raise ValueError(f"BROKER_BIND must be host:port, got {bind!r}")
|
||||
return host or "127.0.0.1", int(port)
|
||||
|
||||
|
||||
def _start_guarded_server(env=None):
|
||||
global _started_server
|
||||
with _server_lock:
|
||||
if _started_server is not None:
|
||||
return _started_server
|
||||
broker = boot(env)
|
||||
host, port = _parse_bind(broker.config["BROKER_BIND"])
|
||||
_started_server = serve(host, port, broker)
|
||||
return _started_server
|
||||
|
||||
|
||||
def _reset_server_guard():
|
||||
global _started_server
|
||||
with _server_lock:
|
||||
server = _started_server
|
||||
_started_server = None
|
||||
if server is None:
|
||||
return None
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
server.broker.audit.close()
|
||||
server.broker.store.close()
|
||||
return server
|
||||
|
||||
|
||||
class Tools:
|
||||
class Valves:
|
||||
pass
|
||||
|
||||
def __init__(self):
|
||||
self.valves = self.Valves()
|
||||
self._server = _start_guarded_server()
|
||||
|
||||
def status(self) -> str:
|
||||
"""Report the broker server state as a one-line string."""
|
||||
host, port = self._server.server_address[:2]
|
||||
return f"coreci-broker serving on http://{host}:{port}"
|
||||
|
||||
|
||||
def run_standalone(env=None):
|
||||
broker = boot(env)
|
||||
host, port = _parse_bind(broker.config["BROKER_BIND"])
|
||||
server = serve(host, port, broker)
|
||||
bound_host, bound_port = server.server_address[:2]
|
||||
print(f"coreci-broker listening on http://{bound_host}:{bound_port}", flush=True)
|
||||
try:
|
||||
while True:
|
||||
time.sleep(1)
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
broker.audit.close()
|
||||
broker.store.close()
|
||||
print("coreci-broker shut down", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run_standalone()
|
||||
@@ -0,0 +1,114 @@
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.request
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
|
||||
class AcceptanceP1Tests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
}
|
||||
|
||||
def _boot(self):
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
broker = coreci_broker.boot(dict(self.env))
|
||||
return broker, buffer.getvalue()
|
||||
|
||||
def test_ac14_coreci_broker_line_count_under_800(self):
|
||||
source_path = os.path.join(os.path.dirname(__file__), "..", "coreci_broker.py")
|
||||
with open(source_path, "r", encoding="utf-8") as handle:
|
||||
line_count = sum(1 for _ in handle)
|
||||
self.assertLess(line_count, 800)
|
||||
|
||||
def test_first_boot_prints_pass_once_second_boot_silent_with_0600_state(self):
|
||||
first = io.StringIO()
|
||||
with contextlib.redirect_stdout(first):
|
||||
broker = coreci_broker.boot(dict(self.env))
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
db_path = self.env["BROKER_DB_PATH"]
|
||||
audit_path = self.env["BROKER_AUDIT_LOG_PATH"]
|
||||
env_path = self.env["BROKER_ENV_PATH"]
|
||||
self.assertTrue(os.path.isfile(db_path))
|
||||
self.assertTrue(os.path.isfile(audit_path))
|
||||
self.assertTrue(os.path.isfile(env_path))
|
||||
self.assertEqual(0o600, os.stat(db_path).st_mode & 0o777)
|
||||
self.assertEqual(0o600, os.stat(audit_path).st_mode & 0o777)
|
||||
admin_pass = broker.config["BROKER_ADMIN_PASS"]
|
||||
pepper = broker.config["BROKER_INSTANCE_PEPPER"]
|
||||
self.assertIn(admin_pass, first.getvalue())
|
||||
self.assertEqual(1, first.getvalue().count(admin_pass))
|
||||
self.assertNotIn(pepper, first.getvalue())
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
broker.audit.close()
|
||||
broker.store.close()
|
||||
second = io.StringIO()
|
||||
with contextlib.redirect_stdout(second):
|
||||
rebooted = coreci_broker.boot(dict(self.env))
|
||||
self.assertEqual("", second.getvalue())
|
||||
rebooted.audit.close()
|
||||
rebooted.store.close()
|
||||
|
||||
def test_restart_preserves_allowlist_row_and_audit_history(self):
|
||||
broker, _ = self._boot()
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
user_id = "acceptance-restart-user"
|
||||
broker.store.conn.execute(
|
||||
"INSERT INTO user_allowlist (user_id, email, status, created_at) VALUES (?, ?, ?, ?)",
|
||||
(user_id, "ceo@coreci.dev", "active", coreci_broker._utc_now_iso()),
|
||||
)
|
||||
broker.store.conn.commit()
|
||||
broker.audit.write("add_user", user_id=user_id)
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
broker.audit.close()
|
||||
broker.store.close()
|
||||
rebooted, _ = self._boot()
|
||||
self.addCleanup(rebooted.audit.close)
|
||||
self.addCleanup(rebooted.store.close)
|
||||
row = rebooted.store.conn.execute(
|
||||
"SELECT email, status FROM user_allowlist WHERE user_id = ?", (user_id,)
|
||||
).fetchone()
|
||||
self.assertEqual(("ceo@coreci.dev", "active"), row)
|
||||
audit_path = self.env["BROKER_AUDIT_LOG_PATH"]
|
||||
with open(audit_path, "r", encoding="utf-8") as handle:
|
||||
lines = [line for line in handle if line.strip()]
|
||||
self.assertEqual(1, len(lines))
|
||||
entry = json.loads(lines[0])
|
||||
self.assertEqual("add_user", entry["action"])
|
||||
self.assertEqual(user_id, entry["user_id"])
|
||||
self.assertEqual(0o600, os.stat(audit_path).st_mode & 0o777)
|
||||
|
||||
def test_healthz_end_to_end(self):
|
||||
broker, _ = self._boot()
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
self.addCleanup(broker.audit.close)
|
||||
self.addCleanup(broker.store.close)
|
||||
self.addCleanup(server.server_close)
|
||||
self.addCleanup(server.shutdown)
|
||||
port = server.server_address[1]
|
||||
with urllib.request.urlopen(
|
||||
"http://127.0.0.1:%d/healthz" % port, timeout=5
|
||||
) as response:
|
||||
self.assertEqual(200, response.status)
|
||||
payload = json.loads(response.read().decode("utf-8"))
|
||||
self.assertEqual({"ok": True}, payload)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,213 @@
|
||||
import contextlib
|
||||
import datetime
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
AUDIT_FIELDS = [
|
||||
"ts",
|
||||
"user_id",
|
||||
"app_registration_uuid",
|
||||
"agent_id",
|
||||
"action",
|
||||
"path",
|
||||
"result",
|
||||
]
|
||||
|
||||
ALL_ACTIONS = [
|
||||
"register",
|
||||
"read",
|
||||
"denied",
|
||||
"login",
|
||||
"logout",
|
||||
"add_app",
|
||||
"remove_app",
|
||||
"add_user",
|
||||
"remove_user",
|
||||
]
|
||||
|
||||
ADMIN_ACTIONS = [
|
||||
"login",
|
||||
"logout",
|
||||
"add_app",
|
||||
"remove_app",
|
||||
"add_user",
|
||||
"remove_user",
|
||||
]
|
||||
|
||||
|
||||
class AuditTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.env_path = os.path.join(tmp.name, ".env")
|
||||
self.db_path = os.path.join(tmp.name, "broker.db")
|
||||
self.audit_path = os.path.join(tmp.name, "broker.audit.log")
|
||||
|
||||
def _boot(self, extra_env=None):
|
||||
env = {
|
||||
"BROKER_ENV_PATH": self.env_path,
|
||||
"BROKER_DB_PATH": self.db_path,
|
||||
"BROKER_AUDIT_LOG_PATH": self.audit_path,
|
||||
}
|
||||
if extra_env:
|
||||
env.update(extra_env)
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
broker = coreci_broker.boot(env)
|
||||
self.addCleanup(broker.store.close)
|
||||
self.addCleanup(broker.audit.close)
|
||||
return broker
|
||||
|
||||
def _raw_lines(self):
|
||||
with open(self.audit_path, "r", encoding="utf-8") as handle:
|
||||
return handle.read().splitlines()
|
||||
|
||||
def _entries(self):
|
||||
return [json.loads(raw) for raw in self._raw_lines()]
|
||||
|
||||
def test_boot_wires_ready_audit_at_mode_0600(self):
|
||||
broker = self._boot()
|
||||
self.assertIsInstance(broker.audit, coreci_broker.Audit)
|
||||
self.assertEqual(self.audit_path, broker.audit.path)
|
||||
self.assertTrue(os.path.isfile(self.audit_path))
|
||||
self.assertEqual(0o600, os.stat(self.audit_path).st_mode & 0o777)
|
||||
|
||||
def test_audit_mode_0600_independent_of_umask(self):
|
||||
old_umask = os.umask(0o777)
|
||||
try:
|
||||
broker = self._boot()
|
||||
finally:
|
||||
os.umask(old_umask)
|
||||
broker.audit.write("login", path="/ui/login")
|
||||
self.assertEqual(0o600, os.stat(self.audit_path).st_mode & 0o777)
|
||||
|
||||
def test_each_write_appends_exactly_one_valid_json_line(self):
|
||||
broker = self._boot()
|
||||
self.assertEqual(0, len(self._raw_lines()))
|
||||
broker.audit.write("login", path="/api/broker/auth/login")
|
||||
broker.audit.write(
|
||||
"read",
|
||||
path="coreci/proxmox",
|
||||
user_id="staff-7",
|
||||
app_registration_uuid="0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a",
|
||||
agent_id="proxmox",
|
||||
)
|
||||
broker.audit.write("denied", path="coreci/gitea", result="denied")
|
||||
lines = self._raw_lines()
|
||||
self.assertEqual(3, len(lines))
|
||||
for line in lines:
|
||||
self.assertTrue(line)
|
||||
entries = self._entries()
|
||||
for entry in entries:
|
||||
self.assertEqual(AUDIT_FIELDS, list(entry))
|
||||
self.assertEqual(["login", "read", "denied"], [e["action"] for e in entries])
|
||||
|
||||
def test_all_nine_action_verbs_accepted_and_written(self):
|
||||
broker = self._boot()
|
||||
for action in ALL_ACTIONS:
|
||||
broker.audit.write(action, path="/x")
|
||||
entries = self._entries()
|
||||
self.assertEqual(9, len(entries))
|
||||
self.assertEqual(sorted(ALL_ACTIONS), sorted(e["action"] for e in entries))
|
||||
|
||||
def test_unknown_action_verb_raises_and_appends_nothing(self):
|
||||
broker = self._boot()
|
||||
for bad in ("bogus", "", "LOGIN", None, "rotate"):
|
||||
with self.assertRaises(ValueError):
|
||||
broker.audit.write(bad, path="/x")
|
||||
with self.assertRaises(ValueError):
|
||||
broker.audit.write("read", path="/x", result="bogus")
|
||||
self.assertEqual(0, len(self._raw_lines()))
|
||||
|
||||
def test_admin_actions_carry_null_identity_fields(self):
|
||||
broker = self._boot()
|
||||
for action in ADMIN_ACTIONS:
|
||||
broker.audit.write(action, path="/api/broker/app_registrations")
|
||||
entries = self._entries()
|
||||
self.assertEqual(len(ADMIN_ACTIONS), len(entries))
|
||||
for entry in entries:
|
||||
self.assertIsNone(entry["user_id"])
|
||||
self.assertIsNone(entry["app_registration_uuid"])
|
||||
self.assertIsNone(entry["agent_id"])
|
||||
broker.audit.write(
|
||||
"register",
|
||||
path="/api/broker/agents/register",
|
||||
user_id="staff-7",
|
||||
app_registration_uuid="0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a",
|
||||
agent_id="proxmox",
|
||||
)
|
||||
entry = self._entries()[-1]
|
||||
self.assertEqual("staff-7", entry["user_id"])
|
||||
self.assertEqual("0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a", entry["app_registration_uuid"])
|
||||
self.assertEqual("proxmox", entry["agent_id"])
|
||||
|
||||
def test_append_only_preserves_every_line_across_many_writes(self):
|
||||
broker = self._boot()
|
||||
for i in range(50):
|
||||
broker.audit.write("read", path=f"coreci/agent-{i}")
|
||||
entries = self._entries()
|
||||
self.assertEqual(50, len(entries))
|
||||
self.assertEqual("coreci/agent-0", entries[0]["path"])
|
||||
self.assertEqual("coreci/agent-49", entries[-1]["path"])
|
||||
self.assertEqual(
|
||||
[f"coreci/agent-{i}" for i in range(50)],
|
||||
[e["path"] for e in entries],
|
||||
)
|
||||
|
||||
def test_reboot_appends_without_truncating_history(self):
|
||||
first = self._boot()
|
||||
first.audit.write("login", path="/ui/login")
|
||||
first.store.close()
|
||||
first.audit.close()
|
||||
second = self._boot()
|
||||
second.audit.write("logout", path="/ui/logout")
|
||||
entries = self._entries()
|
||||
self.assertEqual(["login", "logout"], [e["action"] for e in entries])
|
||||
self.assertEqual(0o600, os.stat(self.audit_path).st_mode & 0o777)
|
||||
|
||||
def test_ts_parses_as_iso8601_utc(self):
|
||||
broker = self._boot()
|
||||
broker.audit.write("read", path="coreci/proxmox", user_id="staff-7")
|
||||
for entry in self._entries():
|
||||
parsed = datetime.datetime.fromisoformat(entry["ts"])
|
||||
self.assertEqual(datetime.timedelta(0), parsed.utcoffset())
|
||||
|
||||
def test_result_values_written(self):
|
||||
broker = self._boot()
|
||||
broker.audit.write("read", path="coreci/proxmox", result="ok")
|
||||
broker.audit.write("read", path="coreci/gitea", result="denied")
|
||||
broker.audit.write("read", path="coreci/boom", result="error")
|
||||
self.assertEqual(["ok", "denied", "error"], [e["result"] for e in self._entries()])
|
||||
|
||||
def test_concurrent_writes_are_serialized(self):
|
||||
broker = self._boot()
|
||||
threads = []
|
||||
for t in range(4):
|
||||
def worker(t=t):
|
||||
for i in range(10):
|
||||
broker.audit.write("read", path=f"coreci/t{t}-i{i}")
|
||||
thread = threading.Thread(target=worker)
|
||||
threads.append(thread)
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
for thread in threads:
|
||||
thread.join()
|
||||
entries = self._entries()
|
||||
self.assertEqual(40, len(entries))
|
||||
self.assertEqual(
|
||||
sorted(f"coreci/t{t}-i{i}" for t in range(4) for i in range(10)),
|
||||
sorted(e["path"] for e in entries),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,105 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import io
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
|
||||
class LoadConfigTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.env_path = os.path.join(tmp.name, ".env")
|
||||
|
||||
def _boot(self, env):
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
config = coreci_broker.load_config(env)
|
||||
return config, buffer.getvalue()
|
||||
|
||||
def _read_env_file(self):
|
||||
values = {}
|
||||
with open(self.env_path, "r", encoding="utf-8") as handle:
|
||||
for raw in handle:
|
||||
line = raw.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, _, value = line.partition("=")
|
||||
values[key.strip()] = value.strip()
|
||||
return values
|
||||
|
||||
def test_first_boot_writes_env_and_prints_pass_once(self):
|
||||
config, stdout = self._boot({"BROKER_ENV_PATH": self.env_path})
|
||||
self.assertTrue(os.path.isfile(self.env_path))
|
||||
stored = self._read_env_file()
|
||||
admin_pass = stored["BROKER_ADMIN_PASS"]
|
||||
pepper = stored["BROKER_INSTANCE_PEPPER"]
|
||||
self.assertGreaterEqual(len(admin_pass), 32)
|
||||
self.assertEqual(32, len(base64.b64decode(pepper)))
|
||||
self.assertEqual(admin_pass, config["BROKER_ADMIN_PASS"])
|
||||
self.assertEqual(pepper, config["BROKER_INSTANCE_PEPPER"])
|
||||
self.assertEqual(1, stdout.count(admin_pass))
|
||||
self.assertNotIn(pepper, stdout)
|
||||
self.assertEqual(0o600, os.stat(self.env_path).st_mode & 0o777)
|
||||
|
||||
def test_second_boot_prints_nothing(self):
|
||||
first, first_stdout = self._boot({"BROKER_ENV_PATH": self.env_path})
|
||||
second, second_stdout = self._boot({"BROKER_ENV_PATH": self.env_path})
|
||||
self.assertEqual(1, first_stdout.count(first["BROKER_ADMIN_PASS"]))
|
||||
self.assertEqual("", second_stdout)
|
||||
self.assertEqual(first["BROKER_ADMIN_PASS"], second["BROKER_ADMIN_PASS"])
|
||||
self.assertEqual(first["BROKER_INSTANCE_PEPPER"], second["BROKER_INSTANCE_PEPPER"])
|
||||
|
||||
def test_env_overrides_dotenv(self):
|
||||
with open(self.env_path, "w", encoding="utf-8") as handle:
|
||||
handle.write("BROKER_BIND=10.1.2.3:9999\n")
|
||||
handle.write('BROKER_ADMIN_USER="ceo"\n')
|
||||
handle.write("BROKER_ADMIN_PASS=oldpass\n")
|
||||
handle.write("BROKER_TOKEN_TTL=100\n")
|
||||
env = {
|
||||
"BROKER_ENV_PATH": self.env_path,
|
||||
"BROKER_ADMIN_PASS": "envpass",
|
||||
"BROKER_TOKEN_TTL": "55",
|
||||
"BROKER_DB_PATH": "/tmp/elsewhere/broker.db",
|
||||
}
|
||||
config, stdout = self._boot(env)
|
||||
self.assertEqual("envpass", config["BROKER_ADMIN_PASS"])
|
||||
self.assertEqual("55", config["BROKER_TOKEN_TTL"])
|
||||
self.assertEqual("/tmp/elsewhere/broker.db", config["BROKER_DB_PATH"])
|
||||
self.assertEqual("10.1.2.3:9999", config["BROKER_BIND"])
|
||||
self.assertEqual("ceo", config["BROKER_ADMIN_USER"])
|
||||
self.assertEqual("300", config["BROKER_SESSION_TTL"])
|
||||
self.assertEqual("", stdout)
|
||||
stored = self._read_env_file()
|
||||
self.assertEqual("oldpass", stored["BROKER_ADMIN_PASS"])
|
||||
self.assertIn("BROKER_INSTANCE_PEPPER", stored)
|
||||
|
||||
def test_defaults_applied(self):
|
||||
config, _ = self._boot({"BROKER_ENV_PATH": self.env_path})
|
||||
self.assertEqual(self.env_path, config["BROKER_ENV_PATH"])
|
||||
self.assertEqual("127.0.0.1:8765", config["BROKER_BIND"])
|
||||
self.assertEqual("admin", config["BROKER_ADMIN_USER"])
|
||||
self.assertEqual("86400", config["BROKER_TOKEN_TTL"])
|
||||
self.assertEqual("300", config["BROKER_SESSION_TTL"])
|
||||
self.assertEqual("broker.db", config["BROKER_DB_PATH"])
|
||||
self.assertEqual("broker.audit.log", config["BROKER_AUDIT_LOG_PATH"])
|
||||
|
||||
def test_reads_process_environ_by_default(self):
|
||||
env = {"BROKER_ENV_PATH": self.env_path, "BROKER_BIND": "127.0.0.1:9000"}
|
||||
with mock.patch.dict(os.environ, env, clear=True):
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
config = coreci_broker.load_config()
|
||||
self.assertEqual("127.0.0.1:9000", config["BROKER_BIND"])
|
||||
self.assertEqual(self.env_path, config["BROKER_ENV_PATH"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,207 @@
|
||||
import base64
|
||||
import hashlib
|
||||
import os
|
||||
import sys
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
SCRYPT_N = 2 ** 14
|
||||
SCRYPT_R = 8
|
||||
SCRYPT_P = 1
|
||||
|
||||
BCRYPT_SHAPED = "$2b$12$K7L9O1P3QRSTUVWXYZabcdefghijk012345678901234567890123456789"
|
||||
|
||||
|
||||
def _scrypt_stored(token, salt):
|
||||
digest = hashlib.scrypt(
|
||||
token.encode("utf-8"), salt=salt, n=SCRYPT_N, r=SCRYPT_R, p=SCRYPT_P
|
||||
)
|
||||
return "{}${}${}".format(
|
||||
"scrypt",
|
||||
base64.urlsafe_b64encode(salt).decode("ascii"),
|
||||
base64.urlsafe_b64encode(digest).decode("ascii"),
|
||||
)
|
||||
|
||||
|
||||
class _FakeBcryptModule:
|
||||
def __init__(self):
|
||||
self.gensalt_calls = []
|
||||
self.hashpw_calls = []
|
||||
self.checkpw_calls = []
|
||||
|
||||
def gensalt(self):
|
||||
self.gensalt_calls.append(True)
|
||||
return b"$2b$12$fakesaltfakesalt"
|
||||
|
||||
def hashpw(self, password, salt):
|
||||
self.hashpw_calls.append((password, salt))
|
||||
return b"$2b$12$storedhashvalue"
|
||||
|
||||
def checkpw(self, password, hashed):
|
||||
self.checkpw_calls.append((password, hashed))
|
||||
return password == b"mock-token"
|
||||
|
||||
|
||||
class HashFormatTests(unittest.TestCase):
|
||||
def test_bcrypt_is_not_importable_on_this_host(self):
|
||||
self.assertIsNone(coreci_broker.bcrypt)
|
||||
with self.assertRaises(ImportError):
|
||||
import bcrypt
|
||||
|
||||
def test_hash_output_starts_with_scrypt_prefix(self):
|
||||
stored = coreci_broker.hash_token("format-token")
|
||||
self.assertTrue(stored.startswith("scrypt$"))
|
||||
|
||||
def test_hash_output_carries_salt_and_digest_b64url_fields(self):
|
||||
stored = coreci_broker.hash_token("format-token")
|
||||
fields = stored.split("$")
|
||||
self.assertEqual(3, len(fields))
|
||||
self.assertEqual("scrypt", fields[0])
|
||||
salt = base64.urlsafe_b64decode(fields[1].encode("ascii"))
|
||||
digest = base64.urlsafe_b64decode(fields[2].encode("ascii"))
|
||||
self.assertEqual(16, len(salt))
|
||||
self.assertEqual(64, len(digest))
|
||||
|
||||
|
||||
class RoundtripTests(unittest.TestCase):
|
||||
def test_hash_then_verify_roundtrip(self):
|
||||
stored = coreci_broker.hash_token("roundtrip-token")
|
||||
self.assertTrue(coreci_broker.verify_token("roundtrip-token", stored))
|
||||
|
||||
def test_wrong_token_fails(self):
|
||||
stored = coreci_broker.hash_token("roundtrip-token")
|
||||
self.assertFalse(coreci_broker.verify_token("wrong-token", stored))
|
||||
|
||||
def test_two_hashes_of_same_token_differ(self):
|
||||
first = coreci_broker.hash_token("same-token")
|
||||
second = coreci_broker.hash_token("same-token")
|
||||
self.assertNotEqual(first, second)
|
||||
self.assertTrue(coreci_broker.verify_token("same-token", first))
|
||||
self.assertTrue(coreci_broker.verify_token("same-token", second))
|
||||
|
||||
|
||||
class ScryptParseTests(unittest.TestCase):
|
||||
def test_hand_built_stored_value_verifies(self):
|
||||
stored = _scrypt_stored("parse-token", bytes(range(16)))
|
||||
self.assertTrue(stored.startswith("scrypt$"))
|
||||
self.assertTrue(coreci_broker.verify_token("parse-token", stored))
|
||||
|
||||
def test_hand_built_stored_value_rejects_other_token(self):
|
||||
stored = _scrypt_stored("parse-token", bytes(range(16)))
|
||||
self.assertFalse(coreci_broker.verify_token("parse-token-x", stored))
|
||||
|
||||
def test_tampered_digest_field_fails(self):
|
||||
salt = bytes(range(16))
|
||||
digest = hashlib.scrypt(
|
||||
b"parse-token", salt=salt, n=SCRYPT_N, r=SCRYPT_R, p=SCRYPT_P
|
||||
)
|
||||
tampered = digest[:-1] + bytes([digest[-1] ^ 0x01])
|
||||
stored = "{}${}${}".format(
|
||||
"scrypt",
|
||||
base64.urlsafe_b64encode(salt).decode("ascii"),
|
||||
base64.urlsafe_b64encode(tampered).decode("ascii"),
|
||||
)
|
||||
self.assertFalse(coreci_broker.verify_token("parse-token", stored))
|
||||
|
||||
def test_malformed_scrypt_values_fail_cleanly(self):
|
||||
for stored in (
|
||||
"scrypt$",
|
||||
"scrypt$abc",
|
||||
"scrypt$abc$def$ghi",
|
||||
"scrypt$$",
|
||||
"scrypt$!@#$%",
|
||||
):
|
||||
with self.subTest(stored=stored):
|
||||
self.assertFalse(coreci_broker.verify_token("x", stored))
|
||||
|
||||
|
||||
class UnknownPrefixTests(unittest.TestCase):
|
||||
def test_unknown_prefixes_return_false_without_raising(self):
|
||||
for stored in (
|
||||
"",
|
||||
"plaintext",
|
||||
"$2c$12$abc",
|
||||
"$2x$12$abc",
|
||||
"argon2$abc$def",
|
||||
"SCRYPT$abc$def",
|
||||
"scrypt2$abc$def",
|
||||
"0123456789",
|
||||
):
|
||||
with self.subTest(stored=stored):
|
||||
self.assertFalse(coreci_broker.verify_token("x", stored))
|
||||
|
||||
def test_non_string_stored_returns_false_cleanly(self):
|
||||
self.assertFalse(coreci_broker.verify_token("x", None))
|
||||
self.assertFalse(coreci_broker.verify_token("x", 5))
|
||||
self.assertFalse(coreci_broker.verify_token("x", ["scrypt$abc$def"]))
|
||||
|
||||
|
||||
class BcryptDispatchContractTests(unittest.TestCase):
|
||||
def test_bcrypt_prefix_returns_false_cleanly_with_bcrypt_absent(self):
|
||||
self.assertIsNone(coreci_broker.bcrypt)
|
||||
with self.assertRaises(ImportError):
|
||||
import bcrypt
|
||||
self.assertFalse(coreci_broker.verify_token("x", BCRYPT_SHAPED))
|
||||
|
||||
def test_all_bcrypt_prefixes_return_false_cleanly_with_bcrypt_absent(self):
|
||||
self.assertIsNone(coreci_broker.bcrypt)
|
||||
for prefix in ("$2a$", "$2b$", "$2y$"):
|
||||
with self.subTest(prefix=prefix):
|
||||
self.assertFalse(coreci_broker.verify_token("x", prefix + "12$storedhashvalue"))
|
||||
|
||||
|
||||
class BcryptBranchTests(unittest.TestCase):
|
||||
def test_hash_token_uses_bcrypt_branch_when_importable(self):
|
||||
fake = _FakeBcryptModule()
|
||||
with mock.patch.object(coreci_broker, "bcrypt", fake):
|
||||
stored = coreci_broker.hash_token("mock-token")
|
||||
self.assertEqual("$2b$12$storedhashvalue", stored)
|
||||
self.assertEqual([True], fake.gensalt_calls)
|
||||
self.assertEqual([(b"mock-token", b"$2b$12$fakesaltfakesalt")], fake.hashpw_calls)
|
||||
|
||||
def test_verify_dispatches_bcrypt_prefixes_to_bcrypt_branch(self):
|
||||
fake = _FakeBcryptModule()
|
||||
expected = [
|
||||
(b"mock-token", (prefix + "12$storedhashvalue").encode("ascii"))
|
||||
for prefix in ("$2a$", "$2b$", "$2y$")
|
||||
]
|
||||
with mock.patch.object(coreci_broker, "bcrypt", fake):
|
||||
for password, hashed in expected:
|
||||
with self.subTest(hashed=hashed):
|
||||
self.assertTrue(coreci_broker.verify_token(password.decode("utf-8"), hashed.decode("ascii")))
|
||||
self.assertEqual(expected, fake.checkpw_calls)
|
||||
|
||||
def test_bcrypt_checkpw_rejection_returns_false(self):
|
||||
fake = _FakeBcryptModule()
|
||||
with mock.patch.object(coreci_broker, "bcrypt", fake):
|
||||
self.assertFalse(coreci_broker.verify_token("other-token", "$2b$12$storedhashvalue"))
|
||||
self.assertEqual([(b"other-token", b"$2b$12$storedhashvalue")], fake.checkpw_calls)
|
||||
|
||||
def test_bcrypt_checkpw_error_returns_false_cleanly(self):
|
||||
fake = _FakeBcryptModule()
|
||||
|
||||
def broken(password, hashed):
|
||||
raise ValueError("invalid salt")
|
||||
|
||||
fake.checkpw = broken
|
||||
with mock.patch.object(coreci_broker, "bcrypt", fake):
|
||||
self.assertFalse(coreci_broker.verify_token("mock-token", "$2b$garbage"))
|
||||
|
||||
def test_scrypt_and_unknown_stored_never_dispatch_to_bcrypt(self):
|
||||
fake = _FakeBcryptModule()
|
||||
stored = coreci_broker.hash_token("mock-token")
|
||||
self.assertTrue(stored.startswith("scrypt$"))
|
||||
with mock.patch.object(coreci_broker, "bcrypt", fake):
|
||||
self.assertTrue(coreci_broker.verify_token("mock-token", stored))
|
||||
self.assertFalse(coreci_broker.verify_token("other", stored))
|
||||
self.assertFalse(coreci_broker.verify_token("mock-token", "unknown$prefix"))
|
||||
self.assertEqual([], fake.checkpw_calls)
|
||||
self.assertEqual([], fake.hashpw_calls)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,148 @@
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.request
|
||||
from http.server import ThreadingHTTPServer
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
|
||||
class MountsTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
coreci_broker._reset_server_guard()
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_BIND": "127.0.0.1:0",
|
||||
}
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.addCleanup(coreci_broker._reset_server_guard)
|
||||
|
||||
def _patched_environ(self):
|
||||
return mock.patch.dict(os.environ, self.env, clear=True)
|
||||
|
||||
def _read_env_values(self):
|
||||
values = {}
|
||||
with open(self.env["BROKER_ENV_PATH"], "r", encoding="utf-8") as handle:
|
||||
for raw in handle:
|
||||
line = raw.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, _, value = line.partition("=")
|
||||
values[key.strip()] = value.strip()
|
||||
return values
|
||||
|
||||
def _healthz(self, port):
|
||||
url = "http://127.0.0.1:%d/healthz" % port
|
||||
with urllib.request.urlopen(url, timeout=5) as response:
|
||||
return response.status, json.loads(response.read().decode("utf-8"))
|
||||
|
||||
def test_owui_docstring_frontmatter(self):
|
||||
doc = coreci_broker.__doc__ or ""
|
||||
self.assertTrue(doc.startswith("title: CoreCI Broker"))
|
||||
for line in (
|
||||
"author: coreci",
|
||||
"description: ",
|
||||
"version: 0.1.0",
|
||||
"requirements: []",
|
||||
"licence: MIT",
|
||||
):
|
||||
self.assertIn(line, doc)
|
||||
|
||||
def test_source_never_references_dunder_file(self):
|
||||
with open(coreci_broker.__file__, "r", encoding="utf-8") as handle:
|
||||
source = handle.read()
|
||||
self.assertNotIn("__file__", source)
|
||||
|
||||
def test_tools_double_instantiation_starts_exactly_one_server(self):
|
||||
serve_calls = []
|
||||
real_serve = coreci_broker.serve
|
||||
|
||||
def counting_serve(host, port, broker):
|
||||
serve_calls.append((host, port))
|
||||
return real_serve(host, port, broker)
|
||||
|
||||
with self._patched_environ():
|
||||
with mock.patch.object(coreci_broker, "serve", counting_serve):
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
first = coreci_broker.Tools()
|
||||
port_after_first = coreci_broker._started_server.server_address[1]
|
||||
second = coreci_broker.Tools()
|
||||
guard = coreci_broker._started_server
|
||||
self.assertEqual(1, len(serve_calls))
|
||||
self.assertIsInstance(guard, ThreadingHTTPServer)
|
||||
self.assertIs(coreci_broker._started_server, first._server)
|
||||
self.assertIs(first._server, second._server)
|
||||
self.assertEqual(port_after_first, guard.server_address[1])
|
||||
self.assertNotEqual(0, guard.server_address[1])
|
||||
self.assertIsInstance(first.valves, coreci_broker.Tools.Valves)
|
||||
self.assertIsInstance(second.valves, coreci_broker.Tools.Valves)
|
||||
status, payload = self._healthz(port_after_first)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, payload)
|
||||
|
||||
def test_tools_status_returns_non_empty_string(self):
|
||||
with self._patched_environ():
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
tools = coreci_broker.Tools()
|
||||
status = tools.status()
|
||||
port = tools._server.server_address[1]
|
||||
self.assertIsInstance(status, str)
|
||||
self.assertTrue(status.strip())
|
||||
self.assertIn(str(port), status)
|
||||
|
||||
def test_first_boot_print_discipline_through_tools_boot(self):
|
||||
with self._patched_environ():
|
||||
first = io.StringIO()
|
||||
with contextlib.redirect_stdout(first):
|
||||
coreci_broker.Tools()
|
||||
second = io.StringIO()
|
||||
with contextlib.redirect_stdout(second):
|
||||
coreci_broker.Tools()
|
||||
stored = self._read_env_values()
|
||||
admin_pass = stored["BROKER_ADMIN_PASS"]
|
||||
pepper = stored["BROKER_INSTANCE_PEPPER"]
|
||||
self.assertEqual(1, first.getvalue().count(admin_pass))
|
||||
self.assertNotIn(pepper, first.getvalue())
|
||||
self.assertEqual("", second.getvalue())
|
||||
|
||||
def test_run_standalone_boots_serves_and_shuts_down_cleanly(self):
|
||||
captured = {}
|
||||
real_serve = coreci_broker.serve
|
||||
|
||||
def capturing_serve(host, port, broker):
|
||||
server = real_serve(host, port, broker)
|
||||
captured["server"] = server
|
||||
return server
|
||||
|
||||
def fake_sleep(seconds):
|
||||
server = captured["server"]
|
||||
status, payload = self._healthz(server.server_address[1])
|
||||
captured["healthz"] = (status, payload)
|
||||
raise KeyboardInterrupt
|
||||
|
||||
stdout = io.StringIO()
|
||||
with mock.patch.object(coreci_broker, "serve", capturing_serve):
|
||||
with mock.patch.object(coreci_broker.time, "sleep", fake_sleep):
|
||||
with contextlib.redirect_stdout(stdout):
|
||||
coreci_broker.run_standalone(dict(self.env))
|
||||
out = stdout.getvalue()
|
||||
self.assertEqual((200, {"ok": True}), captured["healthz"])
|
||||
self.assertEqual(1, out.count("BROKER_ADMIN_PASS="))
|
||||
self.assertIn("listening on http://127.0.0.1:", out)
|
||||
self.assertIn("shut down", out)
|
||||
self.assertEqual(-1, captured["server"].socket.fileno())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,242 @@
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from http.server import ThreadingHTTPServer
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
|
||||
def _echo(request):
|
||||
request.send_json(
|
||||
200,
|
||||
{"body": request.body, "query": request.query, "params": request.path_params},
|
||||
)
|
||||
|
||||
|
||||
def _unauthorized(request):
|
||||
request.send_error_json(401, "unauthorized")
|
||||
|
||||
|
||||
def _forbidden(request):
|
||||
request.send_error_json(403, "forbidden")
|
||||
|
||||
|
||||
def _conflict(request):
|
||||
request.send_error_json(409, "conflict")
|
||||
|
||||
|
||||
def _explode(request):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
|
||||
class RouterTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.ui_root = os.path.join(self.tmp, "ui")
|
||||
self.static_root = os.path.join(self.tmp, "static")
|
||||
os.makedirs(self.ui_root)
|
||||
os.makedirs(self.static_root)
|
||||
env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_UI_ROOT": self.ui_root,
|
||||
"BROKER_STATIC_ROOT": self.static_root,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(env)
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.broker.router.add("POST", "/test/echo/<name>", _echo)
|
||||
self.broker.router.add("GET", "/test/401", _unauthorized)
|
||||
self.broker.router.add("GET", "/test/403", _forbidden)
|
||||
self.broker.router.add("GET", "/test/409", _conflict)
|
||||
self.broker.router.add("GET", "/test/500", _explode)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
|
||||
def _request(self, method, path, data=None):
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=5) as response:
|
||||
return response.status, response.read(), response.headers
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read(), error.headers
|
||||
|
||||
def _request_json(self, method, path, payload=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
status, body, headers = self._request(method, path, data=data)
|
||||
return status, json.loads(body.decode("utf-8")), headers
|
||||
|
||||
def _write(self, directory, name, content):
|
||||
with open(os.path.join(directory, name), "w", encoding="utf-8") as handle:
|
||||
handle.write(content)
|
||||
|
||||
def test_serve_returns_started_threading_daemon_server(self):
|
||||
self.assertIsInstance(self.server, ThreadingHTTPServer)
|
||||
self.assertTrue(self.server.daemon_threads)
|
||||
self.assertEqual(self.broker, self.server.broker)
|
||||
|
||||
def test_healthz(self):
|
||||
status, payload, headers = self._request_json("GET", "/healthz")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, payload)
|
||||
self.assertEqual("application/json; charset=utf-8", headers.get("Content-Type"))
|
||||
|
||||
def test_error_contract_for_every_code(self):
|
||||
cases = [
|
||||
("GET", "/test/401", None, 401),
|
||||
("GET", "/test/403", None, 403),
|
||||
("GET", "/test/409", None, 409),
|
||||
("GET", "/test/500", None, 500),
|
||||
("GET", "/no/such/route", None, 404),
|
||||
("POST", "/test/echo/alice", b"{not json", 400),
|
||||
]
|
||||
for method, path, data, code in cases:
|
||||
with self.subTest(code=code):
|
||||
status, body, headers = self._request(method, path, data=data)
|
||||
self.assertEqual(code, status)
|
||||
self.assertEqual(
|
||||
"application/json; charset=utf-8", headers.get("Content-Type")
|
||||
)
|
||||
payload = json.loads(body.decode("utf-8"))
|
||||
self.assertEqual({"error", "message"}, set(payload))
|
||||
self.assertEqual(code, payload["error"])
|
||||
self.assertIsInstance(payload["message"], str)
|
||||
self.assertTrue(payload["message"])
|
||||
|
||||
def test_json_body_query_and_params_roundtrip(self):
|
||||
payload = {"k": [1, 2, {"z": True}], "s": "værdi", "n": None}
|
||||
status, body, _ = self._request_json(
|
||||
"POST", "/test/echo/alice?x=1&tag=a&tag=b", payload
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(
|
||||
{
|
||||
"body": payload,
|
||||
"query": {"x": "1", "tag": "a"},
|
||||
"params": {"name": "alice"},
|
||||
},
|
||||
body,
|
||||
)
|
||||
|
||||
def test_post_with_empty_body_yields_null_body(self):
|
||||
status, body, _ = self._request_json("POST", "/test/echo/alice")
|
||||
self.assertEqual(200, status)
|
||||
self.assertIsNone(body["body"])
|
||||
|
||||
def test_unknown_route_is_404_even_with_valid_json_body(self):
|
||||
status, body, _ = self._request_json("POST", "/api/broker/nope", {"a": 1})
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual({"error", "message"}, set(body))
|
||||
self.assertEqual(404, body["error"])
|
||||
|
||||
def test_method_mismatch_is_404(self):
|
||||
status, body, _ = self._request_json("DELETE", "/healthz")
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual(404, body["error"])
|
||||
|
||||
def test_static_content_types(self):
|
||||
fixtures = {
|
||||
"index.html": ("<html><body>ok</body></html>", "text/html; charset=utf-8"),
|
||||
"app.css": ("body{margin:0}", "text/css; charset=utf-8"),
|
||||
"app.js": ("console.log(1)", "application/javascript; charset=utf-8"),
|
||||
"mod.py": ("import os\n", "text/x-python; charset=utf-8"),
|
||||
"notes.txt": ("hello world", "text/plain; charset=utf-8"),
|
||||
"data.json": ("{\"k\": 1}", "application/json; charset=utf-8"),
|
||||
}
|
||||
for name, (content, _) in fixtures.items():
|
||||
self._write(self.ui_root, name, content)
|
||||
for name, (content, content_type) in fixtures.items():
|
||||
with self.subTest(name=name):
|
||||
status, body, headers = self._request("GET", "/ui/" + name)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(content.encode("utf-8"), body)
|
||||
self.assertEqual(content_type, headers.get("Content-Type"))
|
||||
|
||||
def test_static_serves_client_module(self):
|
||||
self._write(
|
||||
self.static_root, "coreci_broker_client.py", "import urllib.request\n"
|
||||
)
|
||||
status, body, headers = self._request("GET", "/static/coreci_broker_client.py")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(b"import urllib.request\n", body)
|
||||
self.assertEqual("text/x-python; charset=utf-8", headers.get("Content-Type"))
|
||||
|
||||
def test_ui_index_and_extensionless_page(self):
|
||||
self._write(self.ui_root, "index.html", "<html>index</html>")
|
||||
self._write(self.ui_root, "login.html", "<html>login</html>")
|
||||
status, body, _ = self._request("GET", "/ui/")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(b"<html>index</html>", body)
|
||||
status, body, _ = self._request("GET", "/ui/login")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(b"<html>login</html>", body)
|
||||
|
||||
def test_static_missing_file_is_404_contract(self):
|
||||
status, body, headers = self._request("GET", "/ui/absent.html")
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual("application/json; charset=utf-8", headers.get("Content-Type"))
|
||||
self.assertEqual(
|
||||
{"error": 404, "message": "not found"}, json.loads(body.decode("utf-8"))
|
||||
)
|
||||
|
||||
def test_static_prefix_rejects_non_get(self):
|
||||
status, body, _ = self._request("POST", "/ui/index.html", data=b"{}")
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual(404, json.loads(body.decode("utf-8"))["error"])
|
||||
|
||||
def test_ui_root_attribute_is_configurable(self):
|
||||
alt = os.path.join(self.tmp, "ui_alt")
|
||||
os.makedirs(alt)
|
||||
self._write(alt, "x.html", "alt root")
|
||||
self.broker.ui_root = alt
|
||||
status, body, _ = self._request("GET", "/ui/x.html")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(b"alt root", body)
|
||||
|
||||
def test_traversal_attempts_rejected(self):
|
||||
attempts = [
|
||||
"/ui/../etc/passwd",
|
||||
"/static/%2e%2e/etc/passwd",
|
||||
"/static/..%2fetc%2fpasswd",
|
||||
"/static/..%5cetc%5cpasswd",
|
||||
"/ui/..%5c..%5cetc%5cpasswd",
|
||||
"/static/%00etc/passwd",
|
||||
"/static//etc/passwd",
|
||||
"/ui/./.././../etc/passwd",
|
||||
]
|
||||
for path in attempts:
|
||||
with self.subTest(path=path):
|
||||
status, body, _ = self._request("GET", path)
|
||||
self.assertIn(status, (400, 404))
|
||||
payload = json.loads(body.decode("utf-8"))
|
||||
self.assertEqual({"error", "message"}, set(payload))
|
||||
self.assertEqual(status, payload["error"])
|
||||
|
||||
def test_symlink_escape_rejected(self):
|
||||
outside = os.path.join(self.tmp, "outside")
|
||||
os.makedirs(outside)
|
||||
self._write(outside, "secret.txt", "top secret")
|
||||
os.symlink(outside, os.path.join(self.ui_root, "alias"))
|
||||
status, body, _ = self._request("GET", "/ui/alias/secret.txt")
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual(
|
||||
{"error": 404, "message": "not found"}, json.loads(body.decode("utf-8"))
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,114 @@
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
|
||||
class StaticDisclosureGuardTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_UI_ROOT": self.tmp,
|
||||
"BROKER_STATIC_ROOT": self.tmp,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(dict(self.env))
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
with open(
|
||||
os.path.join(self.tmp, "coreci_broker_client.py"), "w", encoding="utf-8"
|
||||
) as handle:
|
||||
handle.write("import urllib.request\n")
|
||||
with open(
|
||||
os.path.join(self.tmp, ".hidden.css"), "w", encoding="utf-8"
|
||||
) as handle:
|
||||
handle.write("body{display:none}")
|
||||
os.symlink(self.env["BROKER_ENV_PATH"], os.path.join(self.tmp, "env_alias"))
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
|
||||
def _get(self, path):
|
||||
try:
|
||||
with urllib.request.urlopen(self.base + path, timeout=5) as response:
|
||||
return response.status, response.read()
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read()
|
||||
|
||||
def test_state_files_never_served_when_roots_overlap_state_dir(self):
|
||||
for path in (
|
||||
"/static/.env",
|
||||
"/static/broker.db",
|
||||
"/static/broker.db-journal",
|
||||
"/static/broker.db-wal",
|
||||
"/static/broker.audit.log",
|
||||
"/ui/.env",
|
||||
"/ui/broker.db",
|
||||
"/ui/broker.audit.log",
|
||||
):
|
||||
with self.subTest(path=path):
|
||||
status, body = self._get(path)
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual(
|
||||
{"error": 404, "message": "not found"}, json.loads(body.decode("utf-8"))
|
||||
)
|
||||
|
||||
def test_symlink_alias_to_env_rejected(self):
|
||||
status, _ = self._get("/static/env_alias")
|
||||
self.assertEqual(404, status)
|
||||
|
||||
def test_dotfiles_rejected_even_with_known_extension(self):
|
||||
for path in ("/static/.hidden.css", "/ui/.hidden.css"):
|
||||
with self.subTest(path=path):
|
||||
status, _ = self._get(path)
|
||||
self.assertEqual(404, status)
|
||||
|
||||
def test_public_files_still_served(self):
|
||||
status, body = self._get("/static/coreci_broker_client.py")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(b"import urllib.request\n", body)
|
||||
|
||||
def test_default_config_relative_paths_do_not_disclose_state(self):
|
||||
cwd = os.getcwd()
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
os.chdir(tmp.name)
|
||||
try:
|
||||
env = {"BROKER_ENV_PATH": os.path.join(tmp.name, ".env")}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
broker = coreci_broker.boot(dict(env))
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
base = "http://127.0.0.1:%d" % server.server_address[1]
|
||||
for path in ("/static/.env", "/static/broker.db", "/static/broker.audit.log"):
|
||||
with self.subTest(path=path):
|
||||
try:
|
||||
with urllib.request.urlopen(base + path, timeout=5) as response:
|
||||
status = response.status
|
||||
except urllib.error.HTTPError as error:
|
||||
status = error.code
|
||||
self.assertEqual(404, status)
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
broker.audit.close()
|
||||
broker.store.close()
|
||||
finally:
|
||||
os.chdir(cwd)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,155 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import io
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
TABLE_COLUMNS = {
|
||||
"broker_config": ["key", "value"],
|
||||
"app_registrations": ["uuid", "name", "status", "created_at", "revoked_at"],
|
||||
"user_allowlist": ["user_id", "email", "status", "created_at", "revoked_at"],
|
||||
"agent_registrations": [
|
||||
"agent_id",
|
||||
"app_registration_uuid",
|
||||
"user_id",
|
||||
"tags",
|
||||
"token_hash",
|
||||
"token_expires_at",
|
||||
"first_registered_at",
|
||||
"last_seen_at",
|
||||
],
|
||||
"admin_sessions": ["session_id", "created_at", "expires_at"],
|
||||
}
|
||||
|
||||
|
||||
class StoreTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.env_path = os.path.join(tmp.name, ".env")
|
||||
self.db_path = os.path.join(tmp.name, "broker.db")
|
||||
|
||||
def _boot(self, extra_env=None):
|
||||
env = {"BROKER_ENV_PATH": self.env_path, "BROKER_DB_PATH": self.db_path}
|
||||
if extra_env:
|
||||
env.update(extra_env)
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
broker = coreci_broker.boot(env)
|
||||
self.addCleanup(broker.store.close)
|
||||
return broker, buffer.getvalue()
|
||||
|
||||
def _connect(self):
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
self.addCleanup(conn.close)
|
||||
return conn
|
||||
|
||||
def test_boot_creates_db_with_mode_0600(self):
|
||||
broker, stdout = self._boot()
|
||||
self.assertTrue(os.path.isfile(self.db_path))
|
||||
self.assertEqual(0o600, os.stat(self.db_path).st_mode & 0o777)
|
||||
self.assertEqual(1, stdout.count(broker.config["BROKER_ADMIN_PASS"]))
|
||||
|
||||
def test_db_mode_0600_independent_of_umask(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
env = {
|
||||
"BROKER_ENV_PATH": os.path.join(tmp.name, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(tmp.name, "broker.db"),
|
||||
}
|
||||
old_umask = os.umask(0o777)
|
||||
try:
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
broker = coreci_broker.boot(env)
|
||||
self.addCleanup(broker.store.close)
|
||||
self.assertEqual(0o600, os.stat(env["BROKER_DB_PATH"]).st_mode & 0o777)
|
||||
self.assertEqual(0o600, os.stat(env["BROKER_ENV_PATH"]).st_mode & 0o777)
|
||||
finally:
|
||||
os.umask(old_umask)
|
||||
|
||||
def test_all_five_tables_present_with_columns(self):
|
||||
self._boot()
|
||||
conn = self._connect()
|
||||
tables = {
|
||||
row[0]
|
||||
for row in conn.execute("SELECT name FROM sqlite_master WHERE type = 'table'")
|
||||
}
|
||||
self.assertTrue(set(TABLE_COLUMNS).issubset(tables))
|
||||
for table, expected in TABLE_COLUMNS.items():
|
||||
actual = [row[1] for row in conn.execute("PRAGMA table_info(%s)" % table)]
|
||||
self.assertEqual(expected, actual)
|
||||
|
||||
def test_index_on_agent_registrations_app_registration_uuid(self):
|
||||
self._boot()
|
||||
conn = self._connect()
|
||||
names = [
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type = 'index' AND tbl_name = 'agent_registrations'"
|
||||
)
|
||||
]
|
||||
matching = []
|
||||
for name in names:
|
||||
columns = [row[2] for row in conn.execute("PRAGMA index_info(%s)" % name)]
|
||||
if columns and columns[0] == "app_registration_uuid":
|
||||
matching.append(name)
|
||||
self.assertTrue(matching)
|
||||
|
||||
def test_pepper_synced_into_broker_config(self):
|
||||
broker, _ = self._boot()
|
||||
conn = self._connect()
|
||||
row = conn.execute(
|
||||
"SELECT value FROM broker_config WHERE key = 'BROKER_INSTANCE_PEPPER'"
|
||||
).fetchone()
|
||||
self.assertIsNotNone(row)
|
||||
self.assertEqual(broker.config["BROKER_INSTANCE_PEPPER"], row[0])
|
||||
self.assertEqual(32, len(base64.b64decode(row[0])))
|
||||
|
||||
def test_pepper_env_override_synced(self):
|
||||
self._boot({"BROKER_INSTANCE_PEPPER": "Zm9vYmFy"})
|
||||
conn = self._connect()
|
||||
row = conn.execute(
|
||||
"SELECT value FROM broker_config WHERE key = 'BROKER_INSTANCE_PEPPER'"
|
||||
).fetchone()
|
||||
self.assertEqual("Zm9vYmFy", row[0])
|
||||
|
||||
def test_second_boot_silent_and_idempotent(self):
|
||||
first, first_stdout = self._boot()
|
||||
second, second_stdout = self._boot()
|
||||
self.assertEqual(1, first_stdout.count(first.config["BROKER_ADMIN_PASS"]))
|
||||
self.assertEqual("", second_stdout)
|
||||
self.assertEqual(
|
||||
first.config["BROKER_INSTANCE_PEPPER"],
|
||||
second.config["BROKER_INSTANCE_PEPPER"],
|
||||
)
|
||||
self.assertEqual(
|
||||
first.config["BROKER_INSTANCE_PEPPER"],
|
||||
second.store.get_config("BROKER_INSTANCE_PEPPER"),
|
||||
)
|
||||
self.assertEqual(0o600, os.stat(self.db_path).st_mode & 0o777)
|
||||
|
||||
def test_restart_preserves_rows(self):
|
||||
broker, _ = self._boot()
|
||||
broker.store.conn.execute(
|
||||
"INSERT INTO app_registrations (uuid, name) VALUES (?, ?)",
|
||||
("0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a", "proxmox-tool"),
|
||||
)
|
||||
broker.store.conn.commit()
|
||||
broker.store.close()
|
||||
conn = self._connect()
|
||||
row = conn.execute(
|
||||
"SELECT name FROM app_registrations WHERE uuid = ?",
|
||||
("0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a",),
|
||||
).fetchone()
|
||||
self.assertEqual("proxmox-tool", row[0])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,182 @@
|
||||
import base64
|
||||
import hashlib
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
PEPPER = "ZpQ1hP3rLmX7wYc2vB4nT6kS9dF5gR8j"
|
||||
APP_UUID = "3f2a9c1e-77d0-4f0a-9b2e-6c1d5a8e4b3c"
|
||||
USER_ID = "user-42"
|
||||
AGENT_ID = "proxmox"
|
||||
|
||||
B64_ALPHABET = set(
|
||||
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/="
|
||||
)
|
||||
|
||||
|
||||
def _independent_token(pepper, app_uuid, user_id, agent_id, tags):
|
||||
payload = "{}:{}:{}:{}:{}".format(
|
||||
pepper,
|
||||
app_uuid,
|
||||
user_id,
|
||||
agent_id,
|
||||
",".join(sorted(tags)),
|
||||
)
|
||||
digest = hashlib.sha256(payload.encode("utf-8")).digest()
|
||||
return base64.b64encode(digest).decode("ascii")
|
||||
|
||||
|
||||
class FormulaTests(unittest.TestCase):
|
||||
def test_matches_independent_computation(self):
|
||||
token = coreci_broker.derive_token(
|
||||
PEPPER, APP_UUID, USER_ID, AGENT_ID, ["infra", "proxmox"]
|
||||
)
|
||||
expected = _independent_token(
|
||||
PEPPER, APP_UUID, USER_ID, AGENT_ID, ["infra", "proxmox"]
|
||||
)
|
||||
self.assertEqual(expected, token)
|
||||
|
||||
def test_exact_spec_formula_byte_for_byte(self):
|
||||
expected = base64.b64encode(
|
||||
hashlib.sha256(
|
||||
"{}:{}:{}:{}:{}".format(
|
||||
PEPPER, APP_UUID, USER_ID, AGENT_ID, "a,b"
|
||||
).encode("utf-8")
|
||||
).digest()
|
||||
).decode("ascii")
|
||||
actual = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a", "b"])
|
||||
self.assertEqual(expected, actual)
|
||||
|
||||
def test_empty_tags_match_independent_computation(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, [])
|
||||
self.assertEqual(
|
||||
_independent_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, []),
|
||||
token,
|
||||
)
|
||||
|
||||
def test_deterministic_for_same_inputs(self):
|
||||
first = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
second = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertEqual(first, second)
|
||||
|
||||
|
||||
class ShapeTests(unittest.TestCase):
|
||||
def test_token_is_exactly_44_chars(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertEqual(44, len(token))
|
||||
|
||||
def test_token_is_pure_ascii(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertTrue(token.isascii())
|
||||
token.encode("ascii")
|
||||
|
||||
def test_token_carries_base64_padding_char(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertTrue(token.endswith("="))
|
||||
self.assertEqual(1, token.count("="))
|
||||
|
||||
def test_token_uses_only_base64_alphabet(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertTrue(set(token) <= B64_ALPHABET)
|
||||
|
||||
def test_token_decodes_to_32_byte_sha256_digest(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertEqual(32, len(base64.b64decode(token)))
|
||||
|
||||
def test_token_has_no_prefix(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
for prefix in ("App", "Bearer", "scrypt$", "$2", "token", "tk_", "coreci"):
|
||||
with self.subTest(prefix=prefix):
|
||||
self.assertFalse(token.startswith(prefix))
|
||||
self.assertNotIn(" ", token)
|
||||
self.assertNotIn(":", token)
|
||||
self.assertNotIn("$", token)
|
||||
|
||||
|
||||
class TagNormalizationTests(unittest.TestCase):
|
||||
def test_tag_order_does_not_change_token(self):
|
||||
first = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["b", "a"])
|
||||
second = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a", "b"])
|
||||
self.assertEqual(first, second)
|
||||
|
||||
def test_accepts_tuple_and_set_iterables(self):
|
||||
expected = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a", "b"])
|
||||
self.assertEqual(
|
||||
expected,
|
||||
coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ("a", "b")),
|
||||
)
|
||||
self.assertEqual(
|
||||
expected,
|
||||
coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, {"a", "b"}),
|
||||
)
|
||||
|
||||
|
||||
class InputSensitivityTests(unittest.TestCase):
|
||||
def test_pepper_changes_token(self):
|
||||
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
other = coreci_broker.derive_token("other-pepper", APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertNotEqual(base, other)
|
||||
|
||||
def test_app_uuid_changes_token(self):
|
||||
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
other = coreci_broker.derive_token(
|
||||
PEPPER, "00000000-0000-0000-0000-000000000000", USER_ID, AGENT_ID, ["a"]
|
||||
)
|
||||
self.assertNotEqual(base, other)
|
||||
|
||||
def test_user_id_changes_token(self):
|
||||
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
other = coreci_broker.derive_token(PEPPER, APP_UUID, "user-99", AGENT_ID, ["a"])
|
||||
self.assertNotEqual(base, other)
|
||||
|
||||
def test_agent_id_changes_token(self):
|
||||
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
other = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, "gitea", ["a"])
|
||||
self.assertNotEqual(base, other)
|
||||
|
||||
def test_tags_change_token(self):
|
||||
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
other = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a", "c"])
|
||||
self.assertNotEqual(base, other)
|
||||
|
||||
|
||||
class PurityTests(unittest.TestCase):
|
||||
def test_derivation_writes_nothing_to_disk(self):
|
||||
cwd = os.getcwd()
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
os.chdir(tmp)
|
||||
try:
|
||||
coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertEqual([], os.listdir(tmp))
|
||||
finally:
|
||||
os.chdir(cwd)
|
||||
|
||||
def test_derivation_isolated_from_store_and_audit(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
db_path = os.path.join(tmp, "broker.db")
|
||||
audit_path = os.path.join(tmp, "broker.audit.log")
|
||||
store = coreci_broker.Store(db_path)
|
||||
audit = coreci_broker.Audit(audit_path)
|
||||
try:
|
||||
token = coreci_broker.derive_token(
|
||||
PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"]
|
||||
)
|
||||
rows = store.conn.execute(
|
||||
"SELECT * FROM agent_registrations"
|
||||
).fetchall()
|
||||
self.assertEqual([], rows)
|
||||
finally:
|
||||
store.close()
|
||||
audit.close()
|
||||
with open(audit_path, "r", encoding="utf-8") as handle:
|
||||
self.assertEqual([], handle.readlines())
|
||||
self.assertNotIn(token, open(db_path, "rb").read().decode("latin-1"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user