223 lines
11 KiB
Markdown
223 lines
11 KiB
Markdown
# CoreCI Broker
|
||
|
||
**TL;DR** — CoreCI Broker — single-file credential broker for autonomous agent tools (Open WebUI workspace tool / standalone). This file explains how to stand it up and who to contact; the machinery detail stays lower under Details.
|
||
|
||
|
||
A single-file credential broker for autonomous agent tools on the Open WebUI
|
||
host. Tools authenticate with derived tokens instead of shared secrets, the
|
||
CEO gates access through UI-managed allowlists of app UUIDs and user IDs,
|
||
every read lands in an append-only audit log, and secret reads are path-scoped
|
||
to `coreci/<agent_id>` — one Python file, one SQLite DB, a four-page admin UI,
|
||
zero pip installs, no build step.
|
||
|
||
## Quickstart (standalone)
|
||
|
||
```bash
|
||
python3 coreci_broker.py
|
||
```
|
||
|
||
- Stdlib-only runtime, Python 3.11+. Zero pip installs: bcrypt when
|
||
importable, stdlib scrypt otherwise (format-prefixed hashes).
|
||
- First boot generates `.env`: a 32-byte base64 pepper plus a 32+-char admin
|
||
password. The password is printed to the startup log as `BROKER_ADMIN_PASS=…`
|
||
**exactly once** — save it immediately; it is never printed again. Lost pass:
|
||
remove `BROKER_ADMIN_PASS` from `.env` and reboot — a new one prints once.
|
||
- The server binds `127.0.0.1:8765`. Open `http://127.0.0.1:8765/ui/login` and
|
||
log in with the printed pass.
|
||
- Stop with Ctrl+C. Durable state is exactly three files: `broker.db`,
|
||
`broker.audit.log`, `.env` — back them up.
|
||
|
||
## Configuration
|
||
|
||
Read on boot from the `.env` at `BROKER_ENV_PATH`. Precedence:
|
||
**process env → `.env` → defaults**.
|
||
|
||
| Variable | Default | Purpose |
|
||
|----------|---------|---------|
|
||
| `BROKER_BIND` | `127.0.0.1:8765` | Listen address; the bind controls reachability |
|
||
| `BROKER_ADMIN_USER` | `admin` | Admin login user name |
|
||
| `BROKER_ADMIN_PASS` | generated first boot | Admin login password (32+ random chars) |
|
||
| `BROKER_INSTANCE_PEPPER` | generated first boot | 32-byte base64 pepper; token derivation input, synced to SQLite |
|
||
| `BROKER_ENV_PATH` | `.env` | Path of the `.env` file the broker reads/writes |
|
||
| `BROKER_TOKEN_TTL` | `86400` | Tool token lifetime, seconds (24 h) |
|
||
| `BROKER_SESSION_TTL` | `300` | Admin UI session cookie lifetime, seconds |
|
||
| `BROKER_DB_PATH` | `broker.db` | SQLite database (mode 0600) |
|
||
| `BROKER_AUDIT_LOG_PATH` | `broker.audit.log` | Append-only JSONL audit log (mode 0600) |
|
||
| `BROKER_CREDS_SOURCE` | `file` | Credential source adapter: `file` (dev/tests) or `owui` (production) |
|
||
| `BROKER_CREDS_FILE` | `data/creds.json` | `FileCredsSource` JSON document: paths → secret objects |
|
||
| `BROKER_OWUI_BASE_URL` | — | Open WebUI base URL for the OWUI creds source |
|
||
| `BROKER_OWUI_API_KEY` | — | Open WebUI API key (Bearer) for KB read access |
|
||
| `BROKER_AGENT_STATE_BACKEND` | `internal` | Agent-state storage: `internal` (rows in broker.db) or `owui` |
|
||
| `BROKER_MAX_AGENTS_PER_APP` | `50` | Cap on agent registrations per app UUID |
|
||
| `BROKER_UI_ROOT` | `coreci_broker_ui` | Directory served for `/ui/*` pages |
|
||
| `BROKER_STATIC_ROOT` | `.` | Directory served for `/static/*` (client module) |
|
||
|
||
Production: `BROKER_CREDS_SOURCE=owui` with `BROKER_OWUI_BASE_URL` +
|
||
`BROKER_OWUI_API_KEY` (0600 `.env` discipline). Keep `BROKER_BIND` on
|
||
loopback unless you deliberately widen reachability.
|
||
|
||
## Running as an Open WebUI workspace tool
|
||
|
||
Paste `coreci_broker.py` into Open WebUI as a workspace Tool (frontmatter
|
||
`requirements:` stays empty — the runtime already ships everything). Notes
|
||
that matter:
|
||
|
||
- The server starts in `Tools.__init__` behind a once-per-process guard; the
|
||
OWUI loader has no activate/deactivate hook, and disabling a tool does not
|
||
stop a previously started server.
|
||
- **After editing the broker tool source, restart Open WebUI.** The loader
|
||
re-execs the module on source change, but the previously loaded server keeps
|
||
its port — the new instance fails loud with a bind-error message telling you
|
||
to restart, rather than silently serving old code.
|
||
- Paths (DB, audit log, `.env`, UI roots) resolve from environment config,
|
||
never from `__file__` — the OWUI loader deletes the temp file it execs.
|
||
- Durable state: `broker.db` + `broker.audit.log` + `.env` — **back those
|
||
three up**; SQLite is the source of truth, no HA, restore is a file copy.
|
||
- `Tools.status()` reports the serving address; call it from a chat to verify
|
||
the mount, or hit `/healthz` on the bind.
|
||
|
||
## Admin UI
|
||
|
||
Four pages + dashboard, server-rendered on the same port as the API:
|
||
|
||
- `/ui/login` — the CEO's bookmark. First-boot pass discipline: the printed
|
||
pass is shown once; record it in your secret store on first login.
|
||
- `/ui/app_registrations` — allowlist of tool app UUIDs: add, revoke, reinstate.
|
||
- `/ui/user_allowlist` — allowlist of user IDs (optional email): same actions.
|
||
- `/ui/audit` — filterable, read-only view of the audit log (no export in v0.1).
|
||
- `/ui/dashboard` — landing page with persistent nav header and Logout.
|
||
|
||
Allowlist semantics: **revoke keeps existing tokens working until TTL expiry —
|
||
by design.** Revocation immediately blocks new registrations; already-issued
|
||
tokens live at most one `BROKER_TOKEN_TTL` (24 h by default).
|
||
|
||
## Tool authors
|
||
|
||
Fetch the public client module — unauthenticated, from
|
||
`http://<broker>/static/coreci_broker_client.py` — into your workspace and
|
||
import it. Flow: `ensure_registered(state)` → `put_state(state)` →
|
||
`read_secret("coreci/<agent_id>", state)`. Stdlib-urllib-only (no httpx, no
|
||
bcrypt), so any tool can import it; on a 401 it re-registers and retries
|
||
once. State endpoints are the tool's only persistence channel — keep `state`
|
||
there, not in local files. See `reference_tools/` for the proxmox-style §11
|
||
flow and a minimal auth-then-use showcase; downstream calls default to a
|
||
safe dry-run.
|
||
|
||
## Security model
|
||
|
||
- Derived token: `base64(sha256(pepper:UUID:user_id:agent_id:sorted_tags))` —
|
||
44 ASCII chars, no prefix, no challenge protocol. Stored only as a slow
|
||
hash (`$2b$…` / `scrypt$…`), never in plaintext.
|
||
- Allowlists are the gate: unknown or revoked app UUIDs and unknown users get
|
||
403 at registration; the token is the consequence, not the authority.
|
||
- Every secret-read attempt is audited (ok/denied/error) with user, UUID,
|
||
agent, and path.
|
||
- Secret reads are path-scoped: a registration reads exactly
|
||
`coreci/<agent_id>` — cross-path → 403, absent → 404.
|
||
- The broker never writes to `creds-coreci-coo`; the CEO maintains that KB
|
||
through the Open WebUI UI.
|
||
- `broker.db`, `broker.audit.log`, and `.env` are mode 0600.
|
||
- Binds `127.0.0.1` by default; trusted-network posture — no rate limiting by
|
||
design (spec §14).
|
||
|
||
## Documented spec deviations
|
||
|
||
Honest list; full detail in `.ciagent/PROJECT.md`:
|
||
|
||
- **Line-cap amendments (D-016/D-017/D-018)**: the single-file constraint grew
|
||
800 → 1000 → 1200 → 1400 lines via the owner-approved §12.14 amendment
|
||
pattern (readability over dense cramming). Current source: 1390 lines; AC14
|
||
enforces the cap.
|
||
- **Dual-mode hashing (D-001)**: spec §6 mandates bcrypt; the runtime is
|
||
stdlib-only, so scrypt fallback with format-prefixed hashes.
|
||
- **Token verification (D-004)**: spec §7.5's in-memory `token_hash` index is
|
||
impossible with salted slow hashes; the broker scans rows scoped to the
|
||
presented UUID and keeps a non-authoritative verified-token cache.
|
||
- **Pepper rule (D-005)**: the pepper is returned only on row creation or
|
||
tokenless registration (the KB-loss recovery path); spec §9's snippet would
|
||
raise `KeyError` on re-registration.
|
||
- **Spec defects resolved**: §7.5 index, §6-vs-§9 (the broker derives
|
||
authoritatively; the tool stores), §9's `KeyError` — see PROJECT.md §Context.
|
||
- `BROKER_ADMIN_USER` defaults to `admin` (spec silent on a default).
|
||
- Register cap `BROKER_MAX_AGENTS_PER_APP` (50) — a documented addition.
|
||
- Reinstate actions audit as `add_app`/`add_user` (no separate audit action).
|
||
- In-scope but absent secrets audit as `read` with `result: error` (404s are
|
||
audited, not silent).
|
||
|
||
## Post-ship live-OWUI smoke checklist (owner-run)
|
||
|
||
Before migrating any real tool (G-005; deployed host pinned v0.11.3, G-006):
|
||
|
||
1. Confirm the deployed Open WebUI version (expected **v0.11.3**) and that the
|
||
KB read endpoints match the researched surface: `/api/v1/knowledge/{id}` and
|
||
`/api/v1/files/{id}/data/content`.
|
||
2. Add the broker as a workspace tool; check `/healthz` on the bind.
|
||
3. CEO first-boot login at `/ui/login` with the printed pass.
|
||
4. Add one app UUID and one user to the allowlists via the UI.
|
||
5. In a scratch tool: import the client module from `/static/` →
|
||
`ensure_registered` → read `coreci/proxmox` via the OWUI creds source.
|
||
6. Verify every step in `/ui/audit`.
|
||
7. Rollback = disable the tool + restore the three state files.
|
||
|
||
No real tool migrates before this passes; the 90-day adoption criterion
|
||
(proxmox tool on the client, direct KB reads removed) is due 2027-01-09.
|
||
|
||
## Tests
|
||
|
||
```bash
|
||
timeout 300 python3 -m unittest discover -s tests
|
||
```
|
||
|
||
295 tests, stdlib unittest only (~2 min; one skip where bcrypt is not
|
||
importable). The AC1–AC14 acceptance matrix is green.
|
||
|
||
## Repository layout
|
||
|
||
| Path | What it is |
|
||
|------|------------|
|
||
| `coreci_broker.py` | The broker: single file — server, store, audit, auth, UI serving |
|
||
| `coreci_broker_client.py` | Public stdlib-only client module (served at `/static/`) |
|
||
| `coreci_broker_ui/` | Server-rendered HTML for the four pages + dashboard |
|
||
| `reference_tools/` | Tool-author templates (proxmox-style §11 flow + minimal showcase) |
|
||
| `tests/` | Stdlib-unittest suite incl. the AC1–AC14 acceptance matrix |
|
||
| `.ciagent/` | CI pipeline artifacts (project, requirements, roadmap, decisions) |
|
||
## CLI and Docker (v0.2)
|
||
|
||
From v0.2.1 onward every gitea release attaches the broker packaged as a
|
||
self-contained CLI zipapp: `coreci-broker-vX.Y.Z.pyz` (its sha256 is in
|
||
the release notes).
|
||
|
||
**CLI install + usage:**
|
||
|
||
```bash
|
||
# from the release page assets:
|
||
chmod +x coreci-broker-vX.Y.Z.pyz
|
||
./coreci-broker-vX.Y.Z.pyz --version # prints the release version
|
||
./coreci-broker-vX.Y.Z.pyz # boots the standalone broker
|
||
# or: python3 coreci-broker-vX.Y.Z.pyz (Python 3.11+)
|
||
```
|
||
|
||
The artifact bundles the broker, the admin UI pages, and the client
|
||
module — one file serves everything (UI at `/ui/`, client at
|
||
`/static/coreci_broker_client.py`). First boot prints
|
||
`BROKER_ADMIN_PASS=...` exactly once and creates `.env`, `broker.db`,
|
||
and `broker.audit.log` mode 0600 in the state dir; state is anchored by
|
||
`BROKER_ENV_PATH` (relative paths stay CWD-relative). The full
|
||
`BROKER_*` env contract is in the Configuration section above.
|
||
|
||
**Docker:**
|
||
|
||
```bash
|
||
docker pull git.coreci.dev/coreci/broker:<tag> # also :latest
|
||
docker run -p 8765:8765 -v broker-state:/state \
|
||
git.coreci.dev/coreci/broker:<tag>
|
||
```
|
||
|
||
State lives on the `/state` volume (`WORKDIR /state`; first boot
|
||
generates `.env`, `broker.db`, and `broker.audit.log there) — mount a
|
||
persistent volume or the admin pass and any registrations are lost with
|
||
the container. The image is built and published once per milestone at
|
||
the milestone release (see `scripts/publish_image.py` — the daemonless
|
||
registry client; the `Dockerfile` documents the docker-user path and
|
||
its equivalence mapping).
|