Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 coreci
|
60a9811ba4 | ||
|
coreci
|
e650681755 | ||
|
coreci
|
7f7868f7cf |
@@ -0,0 +1,79 @@
|
||||
# ARCHITECTURE — CoreCI Broker
|
||||
|
||||
## Overview
|
||||
|
||||
One Open WebUI workspace tool, implemented as a single Python file (`coreci_broker.py`), embedding a stdlib HTTP server (`ThreadingHTTPServer`) bound to `BROKER_BIND` (default `127.0.0.1:8765`). It serves a JSON API, a four-page admin web UI (static HTML/CSS/JS from `coreci_broker_ui/`), and a public static client module. SQLite (`broker.db`, mode 0600) is the source of truth; an append-only JSONL file (`broker.audit.log`, mode 0600) records every read and denied attempt. The broker is the trust root with no upstream auth; it reads the `creds-coreci-coo` KB through a pluggable adapter and never writes to it.
|
||||
|
||||
Two runtime modes share the same file: (a) Open WebUI workspace tool — a `Tools` class starts the server in a daemon thread at construction (`__init__`), behind a once-per-process idempotency guard (the OWUI loader has no activate() hook and re-execs modules on edit; paths resolve from env, never `__file__`); (b) standalone — `python3 coreci_broker.py` for development, testing, and direct operation.
|
||||
|
||||
## Components
|
||||
|
||||
| Name | Description | Boundaries | Depends on |
|
||||
|------|-------------|-----------|------------|
|
||||
| Config bootstrap | Reads/writes the broker `.env` (spec §5.1 contract + documented additions); first-boot generation of `BROKER_INSTANCE_PEPPER` and `BROKER_ADMIN_PASS` (printed to the startup log once, written to `.env` and SQLite) | Owns the `.env` file and the `broker_config` table; never prints secrets after first boot | SQLite store |
|
||||
| SQLite store | Tables: `broker_config`, `app_registrations`, `user_allowlist`, `agent_registrations`, `admin_sessions` + indexes (spec §5.2); mode 0600 | Sole writer of `broker.db`; all SQL lives here | — |
|
||||
| Audit writer/reader | Appends one JSON line per action (register/read/denied/login/logout/add_app/remove_app/add_user/remove_user); filtered query endpoint (since/until/user_id/uuid/action/limit) | Owns `broker.audit.log`; append-only; no rotation in v0.1 | — |
|
||||
| Hasher | Dual-mode slow hash: bcrypt when importable, else stdlib `hashlib.scrypt`; stored hashes format-prefixed (`$2b$…` / `scrypt$…`) so verification is format-aware | The only component with an optional import | — |
|
||||
| Token engine | Server-side derivation: `token = base64(sha256(pepper:UUID:user_id:agent_id:','.join(sorted(tags))))` | Pure function of (pepper, inputs); never stores plaintext tokens | Hasher, Config |
|
||||
| HTTP server + router | stdlib threaded HTTP; JSON body/param handling; `{"error, message}` contract with 400/401/403/404/409/500; static file serving (UI + client module) | No business logic; routes to admin/tool planes | All handlers |
|
||||
| Admin plane | Session auth (cookie `broker_session` + HTTP Basic); login/logout/whoami; app_registrations CRUD (add/revoke/reinstate); user_allowlist CRUD; audit query | Admin-only; audited | Store, Audit |
|
||||
| Tool auth plane | `POST /api/broker/agents/register` (allowlist gates, idempotent re-derive, TTL, pepper rule); App-auth verification (`Authorization: App <UUID> <token>` — UUID-scoped slow-hash scan + in-memory verified-token cache); `GET /agents/<id>`; state GET/PUT with identity enforcement | Register is callable without a valid token (gates are the allowlists); the cache is never authoritative | Store, Token engine, Hasher |
|
||||
| Secrets plane | `GET /api/broker/secrets/<path>`; scope check `path == "coreci/" + agent_id`; creds-source adapter (FileCredsSource default \| OwuiApiCredsSource) | The only reader of `creds-coreci-coo`; read-only | Tool auth plane, Creds adapter |
|
||||
| Agent-state store | Backs `GET/PUT /api/broker/agents/<id>/state`; toggle `BROKER_AGENT_STATE_BACKEND=internal` (rows in `broker.db`) `\| owui` (real per-agent KBs via OWUI API) | Broker-owned; tool-facing; identity-checked | Tool auth plane |
|
||||
| Static UI | Four pages (login, app registrations, user allowlist, audit) + dashboard + persistent nav; server-rendered HTML, no build step | Read-only against the admin API | Admin plane |
|
||||
| Client module | `coreci_broker_client.py` — dependency-free (stdlib urllib) auth client: ensure-registered, state get/put, secret read with 401→re-register→retry-once; served unauthenticated at `/static/` | Contains no secrets; importable by any OWUI tool | Broker API (external) |
|
||||
| Reference tools | `reference_tools/` — OWUI tool classes showcasing auth-first-then-use (e.g., proxmox reference: register → fetch `coreci/proxmox` → downstream call, safe dry-run default) | Templates for tool authors; not wired to live infra in tests | Client module |
|
||||
|
||||
## DataFlow
|
||||
|
||||
1. **Admin**: CEO logs in via `/ui/login` (or HTTP Basic) → session row in `admin_sessions` → manages allowlists via UI → every admin action audited.
|
||||
2. **Registration**: tool calls `POST /api/broker/agents/register` with `{agent_id, app_registration_uuid, user_id, tags}` → broker checks `app_registrations` (active) and `user_allowlist` (active) → 403 on unknown/revoked → broker derives the token from the pepper + inputs → stores slow-hash in `agent_registrations`, sets TTL expiry → returns token (+ pepper when the row is created or the caller presented no valid token) → tool persists via `PUT /agents/<id>/state`.
|
||||
3. **Secret read**: tool calls `GET /api/broker/secrets/coreci/<agent_id>` with `Authorization: App <UUID> <token>` → broker parses UUID, scans that tool's registration rows with the slow hash (in-memory cache after first verify) → 401 on failure → scope check `path == coreci/<agent_id>` → 403 on mismatch → creds adapter reads `creds-coreci-coo` → audit line (ok/denied/error) → JSON response.
|
||||
4. **Expiry**: after `BROKER_TOKEN_TTL` the token verifies but the row is expired → 401 → tool re-registers (step 2) → new token overwrites `token_hash` → audit `register`.
|
||||
|
||||
## BuildOrder
|
||||
|
||||
1. Config bootstrap + SQLite store + audit writer (P1)
|
||||
2. Hasher + token engine (P1)
|
||||
3. HTTP server + router + error contract + OWUI/standalone mounts (P1)
|
||||
4. Admin plane + audit query + static UI (P2)
|
||||
5. Tool auth plane + agent-state store (P3)
|
||||
6. Secrets plane + creds adapter + client module + reference tools (P4)
|
||||
7. Acceptance matrix, security review, README (P5)
|
||||
|
||||
## Notes
|
||||
|
||||
- The in-memory verified-token cache maps `(uuid, token) -> registration row`; it is populated on first successful slow-hash verification and cleared on restart. SQLite remains authoritative; cache entries are re-validated against row status/expiry on every hit.
|
||||
- Creds adapter contract: lookup by path string (`coreci/<agent_id>`) → JSON object. FileCredsSource reads a single JSON document mapping paths to secret objects (default for dev/tests, `BROKER_CREDS_FILE`). OwuiApiCredsSource calls the Open WebUI HTTP API with `BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY` and is exercised in production; its exact KB-file layout is resolved in P4 research.
|
||||
- This file is the init-time snapshot; it is refined at phase boundaries per ci-files discipline.
|
||||
|
||||
## Distribution (v0.2 — cli-packaging)
|
||||
|
||||
v0.2 adds a distribution layer around the unchanged v0.1 broker: no broker-line changes (`run_standalone`, `boot`, `load_config`, `_parse_env_file` are consumed as-is). Three new components sit outside `coreci_broker.py` and are exercised by the same stdlib-unittest harness (in-process stub servers on ephemeral ports; subprocess boot tests against the built artifact).
|
||||
|
||||
### Components (v0.2)
|
||||
|
||||
| Name | Description | Boundaries | Depends on |
|
||||
|------|-------------|-----------|------------|
|
||||
| CLI artifact build (`scripts/build_cli.py` + `cli/__main__.py`) | Stages `coreci_broker.py` + `coreci_broker_client.py` + `coreci_broker_ui/` + the boot shim + a generated `_cli_version.py`, builds `coreci-broker.pyz` with stdlib `zipapp` (`--compress`, shebang `#!/usr/bin/env python3`), then a deterministic normalization pass (sorted entries, fixed timestamps) for byte-reproducible output; emits `dist/coreci-broker-vX.Y.Z.pyz` + sha256 | Produces the artifact only; never publishes; owns no state; the shim materializes bundled UI/static assets into `<state>/.coreci_broker_assets/<version>/{ui,static}` at boot (temp+`os.replace`, overwrite-per-boot) and sets `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` only when neither process env nor the user `.env` defines them; `--version` is answered from the generated `_cli_version.py` before any broker import | Broker module surface (`run_standalone`, `_parse_env_file`) — import-only, zero edits |
|
||||
| Release-asset pipeline (`scripts/release.py` + `scripts/_env_secrets.py`) | Builds release notes (milestone summary, phases, verification, CLI install/usage, docker usage, artifact sha256), create-or-get the gitea release by tag, attaches `coreci-broker-vX.Y.Z.pyz` via `POST /api/v1/repos/{owner}/{repo}/releases/{id}/assets` (multipart field `attachment`, `Authorization: token <T>`); idempotent re-attach; bounded retry (3) → `release_pending` escalation on exhaustion — never blocks tag+merge | Owns the forge API conversation for release assets only; token resolved exclusively from `.env*` files via the file-reading loader (never shell env; assert non-empty before any call — empty token → gitea 404 masking, v1.8); does not build the artifact, does not push images | CLI artifact build (attaches its output); gitea API (`/api/v1/...`) |
|
||||
| Daemonless registry publish client (`scripts/publish_image.py`) | Once per milestone: pull the pinned `python:3.11-slim` base from Docker Hub (anonymous token exchange → OCI index → resolve linux/amd64 child manifest → config + layer blobs), append one deterministic broker layer carrying `/app/coreci-broker.pyz` (tar+gzip, fixed metadata, `mtime=0`), build a new config blob (diff_id appended; Entrypoint `["python3","/app/coreci-broker.pyz"]`, Cmd `[]`, WorkingDir `/state`), and push `git.coreci.dev/coreci/broker:<milestone-release-tag>` + `latest` to the gitea container registry (Basic-auth token exchange at `/v2/token`, `Bearer` JWT on API calls, HEAD-skip + monolithic blob uploads, OCI image manifest PUT; explicit `Location`-following — stdlib urllib does not follow 307/308 on PUT) | No container runtime (D-019); base layer digests reused verbatim, base blobs re-uploaded (no cross-registry mount); records resolved base digests as provenance; failure follows bounded-retry → `release_pending` escalation — never blocks the milestone merge | CLI artifact build (ships its output in the layer); Docker Hub + gitea registry HTTP APIs (OCI dist-spec v2) |
|
||||
| Dockerfile (documentation) | Self-contained docker-build path for users WITH docker: same base, in-image `zipapp` build of the same sources + shim, same entrypoint/`/state` anchoring | Documentation + alternative build path only; the daemonless client is the executable source of truth for what the pipeline publishes; equivalence with the published image is required, byte-identity is not | CLI artifact build (same staging content, same shim) |
|
||||
|
||||
### DataFlow (v0.2 distribution)
|
||||
|
||||
1. **CLI boot (packaged)**: `./coreci-broker.pyz` → shim resolves the state dir (`BROKER_ENV_PATH`-anchored; CWD-relative defaults preserved) → materializes UI/static assets into the versioned assets dir → sets `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` (user env/`.env` overrides win) → `coreci_broker.run_standalone()` → first-boot `.env` generation, admin pass printed once, `broker.db`/`broker.audit.log` 0600 in the state dir, UI served at `/ui/`, client at `/static/coreci_broker_client.py` — behavior byte-identical to the unpackaged standalone mode.
|
||||
2. **Release attachment (per phase ship from the CLI-building phase)**: `scripts/build_cli.py` → reproducible `dist/coreci-broker-vX.Y.Z.pyz` (+sha256) → `scripts/release.py` → gitea create-or-get release by tag → multipart attach → notes body (CLI install/usage + docker usage + digest). Bounded retry; `release_pending` escalation on exhaustion; tag+merge is the completion gate.
|
||||
3. **Image publish (once per milestone, at the final ship)**: `scripts/publish_image.py` → anonymous Docker Hub pull (`python:3.11-slim` OCI index → amd64 child + blobs) → deterministic broker layer (contains the pyz) + new config → gitea registry auth (`/v2/token`, Basic `<user>:<GITEA_TOKEN>`) → blob upload (HEAD-skip, monolithic POST, explicit Location-following) → manifest PUT `<milestone-release-tag>` then `latest` → provenance digests recorded. Bounded retry; never blocks the milestone merge.
|
||||
|
||||
### BuildOrder (v0.2 — appended to the v0.1 order above)
|
||||
|
||||
8. CLI build pipeline + boot shim + packaging/boot-parity tests (P1 — REQ-21)
|
||||
9. Release-asset uploader + daemonless publish client + Dockerfile + stub tests (P2 — REQ-22/REQ-23)
|
||||
10. Cumulative acceptance (v0.1 suite through the packaged artifact), security review, milestone ship: attach pyz to `v0.2.3`, publish image, merge (P3)
|
||||
|
||||
### Notes (v0.2)
|
||||
|
||||
- Distribution components hold no secrets: the archive bundles only broker + UI + client code; the image ships no state; forge tokens live in `.env*` files loaded only by `scripts/_env_secrets.py` (blocked from shell env by `config.security.bash_allowlist`).
|
||||
- All three pipeline scripts take injectable base URLs (stub-testable); live-forge interactions are read-only in tests and attempted-only at ship per the bounded-retry contract.
|
||||
- Verified environment facts backing this section live in RESEARCH.md v0.2 (gitea 1.26.1 API shapes, OCI index resolution for `python:3.11-slim`, gitea `/v2/token` Basic-auth exchange, zipapp normalization, urllib 307/308 behavior).
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"phase": 0,
|
||||
"stage": "plan",
|
||||
"milestone": "v0.2",
|
||||
"phase_role": "pre_execution",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-10-10T16:50:00Z"
|
||||
}
|
||||
@@ -0,0 +1,254 @@
|
||||
# CIAgent Grill Report
|
||||
|
||||
## Run: 2026-10-10 (mode: interactive, focus: v0.2 plan red-team — cli-packaging)
|
||||
|
||||
**Supersedes the v0.1 grill in this file** (v0.1 run of 2026-10-09; its full text and
|
||||
verdicts G-001..G-008 live in git history at `7f7868f` and remain binding where the plan
|
||||
depends on them: G-005's owner-run post-ship OWUI smoke checklist and G-008's 90-day
|
||||
adoption criterion carry forward in ROADMAP.md §Post-Milestone Notes; G-007's
|
||||
forge-repo-creation test was executed at v0.1's P0 ship and is closed). New verdicts
|
||||
number G-009 onward.
|
||||
|
||||
### Verdict (summary): GO-WITH-CONDITIONS (confidence: 0.82)
|
||||
|
||||
The plan rests on genuinely strong research — live-verified forge/registry wire shapes, a
|
||||
working zipapp prototype booted against the real broker sources, zero-broker-lines
|
||||
verified against the actual file (1390/1400). Red-team findings are concrete and
|
||||
fixable by task/ship-step edits: one **unexecutable mechanism** (the v0.2.1 attachment
|
||||
bridge cites a harness path RESEARCH itself proved absent from this environment),
|
||||
one **ship-sequence inversion** (P3-05 attaches the release before the tag exists —
|
||||
gitea would mint `v0.2.3` at the wrong commit), one **false-green determinism gap**
|
||||
(the normalize pass "preserves" umask-derived zip attributes, so must-have 4 passes
|
||||
while the "clean checkout" exit criterion silently fails cross-machine), and a set of
|
||||
test-realism holes (subprocess harness with no timeout discipline, missing creds env
|
||||
in the packaged secret-read, no live read-only validation before the one live publish).
|
||||
Three forks were resolved interactively with the owner this session (G-011, G-013,
|
||||
G-014); everything else is bound on evidence below.
|
||||
|
||||
---
|
||||
|
||||
## Surface 1 — Determinism claims (must-have 4; P1-01)
|
||||
|
||||
**Forcing question**: Is two-build sha256 equality actually achievable and testable, and
|
||||
what does the normalize pass fix vs. miss?
|
||||
|
||||
- **Evidence** (stdlib source, read this session): `/usr/lib/python3.11/zipapp.py`
|
||||
builds via `for child in source.rglob('*'): z.write(child, arcname.as_posix())` —
|
||||
entry order = scandir order, entry mtime = source mtime (`ZipInfo.from_file`), and
|
||||
`external_attr` = the staging file's stat mode `<< 16`. PLAN P1-01's normalize pass
|
||||
fixes order (sorted), `date_time=(1980,1,1,0,0,0)`, `ZIP_DEFLATED` — but specifies
|
||||
"**preserved** `external_attr`". Preserved from what zipapp wrote, i.e. derived from
|
||||
staging file modes — which are `0666 & ~umask` if staged via plain copy, or
|
||||
checkout-mode if via `copy2`. Umask 0022 vs 0002 across machines/clones → different
|
||||
`external_attr` → different sha256, **while must-have 4 (two consecutive builds, same
|
||||
machine) still passes**. A silent false green against the phase exit criterion
|
||||
("builds reproducibly from a clean checkout").
|
||||
- **gzip is not in the pyz path** (zipfile DEFLATE only). Image-layer gzip is
|
||||
deterministic on-platform: `gzip.compress(..., mtime=0)` routes to
|
||||
`zlib.compress(wbits=31)` on 3.11 (verified in gzip.py source) — fixed header, fixed
|
||||
OS byte, stable per zlib build. Scope claim: byte-identity is **same-environment**
|
||||
(same machine, same zlib); cross-zlib-version identity is neither claimed nor
|
||||
required by must-have 4.
|
||||
- **Answer**: achievable and testable as scoped — provided the normalize pass also
|
||||
fixes what it currently preserves. → **G-009**.
|
||||
|
||||
## Surface 2 — Boot parity via subprocess (P1-02)
|
||||
|
||||
**Forcing question**: What makes the subprocess acceptance flaky, and what's missing?
|
||||
|
||||
- **Evidence**: zero v0.1 tests use `subprocess`/`Popen` (grep over `tests/`: no hits) —
|
||||
P1-02 introduces the repo's first subprocess-boot harness. The plan's env enumeration
|
||||
(`BROKER_BIND`, `BROKER_ENV_PATH/DB/AUDIT`) omits `BROKER_CREDS_FILE`/
|
||||
`BROKER_CREDS_SOURCE` while requiring "secret-read reuse the v0.1 acceptance request
|
||||
patterns" — the v0.1 precedent (`tests/test_acceptance_full.py:76-77`) sets both;
|
||||
`FileCredsSource` on a missing file raises `_SecretNotFound` → 404
|
||||
(`coreci_broker.py:440-441`), so the packaged secret-read would fail. `Popen(["python3",
|
||||
...], env={minimal})` leaves the child without `PATH` (works today via execvp's
|
||||
confstr fallback — by accident). No timeout discipline on "read stdout lines until
|
||||
`coreci-broker listening on`" (RESEARCH §4.3) — a failed boot never prints the line and
|
||||
hangs the suite. "Pass printed exactly once" is asserted against a *still-running*
|
||||
server; shutdown-path output is unread unless drained after terminate. And every
|
||||
planned boot pins absolute state paths — the owner's literal happy path (download →
|
||||
`python3 *.pyz` in some directory → `.env`/db/audit materialize in the CWD) is never
|
||||
tested.
|
||||
- **Answer**: harden the harness — the plan's claims are all testable once the harness
|
||||
is honest about them. → **G-010**.
|
||||
|
||||
## Surface 3 — One-shot lead upload bridge (P1-03)
|
||||
|
||||
**Forcing question**: Is the v0.2.1 ad-hoc attachment a discipline violation, and is it
|
||||
even executable as specified?
|
||||
|
||||
- **Evidence**: PLAN P1-03 attaches via "the same harness mechanism v0.1 used for its
|
||||
`broker-vX.Y.Z.tar.gz` assets" — but RESEARCH §2.2 (the plan's own research base)
|
||||
found that mechanism is the ship harness's `coreci build/package/release upload` CLI,
|
||||
**"not installed in this dev environment"**; v0.1's archives were produced
|
||||
container-side, ad-hoc, with no repo-owned script. The bridge as written is therefore
|
||||
unexecutable via the cited mechanism, and its realistic substitute is an untested
|
||||
hand-rolled live-forge write (multipart encoding, token handling, idempotency) at
|
||||
first live use — performed by lead-developer, the one persona barred from code
|
||||
writes, using token discipline that `scripts/_env_secrets.py` (which does not exist
|
||||
until P2) is supposed to enforce. This is precisely the practice REQ-22 formalizes
|
||||
away.
|
||||
- **Owner decision (this session)**: restructure — release automation moves into P1;
|
||||
the bridge is deleted. → **G-011**. The hard gate question dissolves: v0.2.1 ships
|
||||
script-driven; there is no bridge to become permanent.
|
||||
|
||||
## Surface 4 — Daemonless OCI publish (P2-02, P3-05)
|
||||
|
||||
**Forcing question**: Does the milestone ship sequence hold up when publish fails after
|
||||
the tag exists — and does it even run in the right order?
|
||||
|
||||
- **Evidence — sequence inversion**: P3-05 as written runs `release.py` attach →
|
||||
publish image → squash-merge → `main` → tag `v0.2.3` → push. But `POST /api/v1/repos/
|
||||
{o}/{r}/releases` with a missing tag creates the tag at `target_commitish`
|
||||
(default-branch head) — gitea's documented CreateReleaseOption semantics (schema in
|
||||
RESEARCH §2.1). A create-first flow pre-merge would mint `v0.2.3` pinned to
|
||||
**pre-merge main**, then collide with (or silently diverge from) the harness's own tag
|
||||
push. The plan's own P2-04 sequences correctly (tag → merge → push → release) — P3-05
|
||||
alone inverts it.
|
||||
- **Evidence — wire discipline**: bounded retry(3) is specified but no per-request
|
||||
timeout anywhere; a hung 29.8 MB monolithic POST through a throttled gitea turns
|
||||
"bounded retry" into an unbounded hang. The 307/308 stub test proves only
|
||||
Location-following, and gitea's `Location` may be **relative** (RESEARCH §3.3) — the
|
||||
harder case is untested. Upstream `3.11-slim` drift is handled (resolve-at-publish +
|
||||
`--base-manifest` + provenance digests; RESEARCH §3.7) and the anonymous-200 token
|
||||
trap fails loudly per spec.
|
||||
- **Evidence — failure story**: "publish never blocks merge" is honored, but when
|
||||
publish fails after `v0.2.3` exists, the plan records only `release_pending` — no
|
||||
named post-milestone re-run/closure step. The ship contract makes the escalation
|
||||
auto-resolution; somebody must own the retry.
|
||||
- **Answer**: reorder P3-05, add timeouts + relative-Location coverage, and make the
|
||||
escalation carry its own closure runbook. → **G-012**, **G-019**, **G-020**.
|
||||
|
||||
## Surface 5 — Scope discipline (P2-03, REQ-22/23)
|
||||
|
||||
**Forcing question**: Is anything decorative, and is anything REQ-22/23 implies missing?
|
||||
|
||||
- **Evidence — decorative risk**: P2-03's verify is presence-only ("`Dockerfile` +
|
||||
README sections present") and the dev environment cannot build the Dockerfile (no
|
||||
runtime) — a syntactically or semantically broken Dockerfile passes the gate. The
|
||||
"equivalence mapping" header is documentation with no enforcement.
|
||||
- **Evidence — missing**: no `.dockerignore` anywhere in the plan — `docker build .`
|
||||
ships the whole repo (`.git/`, any local `.env*`, `data/`, `*.db`, `dist/`,
|
||||
`__pycache__/`, `.ciagent/`) as build context, adjacent to a security review that
|
||||
asserts "no secrets in any image layer". REQ-22's notes model is covered (§2.3, stub
|
||||
asserts sections present); the sha256's *source* is unspecified (hand-pasted vs.
|
||||
computed); `latest` semantics are coherent (once per milestone at final ship —
|
||||
v0.2.3 is the first image ever, D-021); the state-dir volume contract is documented
|
||||
(`WORKDIR /state`, README docker run).
|
||||
- **Answer**: make the Dockerfile contract testable, add `.dockerignore`, pin digest
|
||||
provenance. → **G-015**, **G-016**, **G-017**.
|
||||
|
||||
## Surface 6 — Frozen broker / shim dependency risk (P1-01)
|
||||
|
||||
**Forcing question**: Does the shim need broker cooperation, and what is the coupling
|
||||
risk?
|
||||
|
||||
- **Evidence**: RESEARCH §1.2's prototype booted from the real sources with zero broker
|
||||
cooperation — re-verified against the file this session: `run_standalone`
|
||||
(`coreci_broker.py:1369`), `_parse_env_file` (:154), `boot` (:598), `load_config`
|
||||
(:240) are module-level; `load_config` precedence (process env → `.env` → defaults,
|
||||
:246-257) makes the shim's pre-set `os.environ` approach contract-identical. The
|
||||
broker's `.env` resolution is CWD-relative by default (`ENV_PATH_DEFAULT=".env"`,
|
||||
`ENV_DEFAULTS` :53-66) — packaged boots with unset `BROKER_ENV_PATH` anchor state to
|
||||
the CWD, which is exactly the documented happy path and is currently untested (folded
|
||||
into G-010). Residual risk: the shim calls `_parse_env_file` — an underscore-private
|
||||
symbol. Within v0.2 the broker is frozen so the risk is nil; across future milestones
|
||||
a rename breaks the shim with a loud ImportError (acceptable), but only if somebody
|
||||
knows to look.
|
||||
- **Answer**: no cooperation needed (verified twice); the private-API dependency must
|
||||
be documented and review-tracked, and the default-anchoring boot must be tested. →
|
||||
**G-018** (anchor case bound under G-010).
|
||||
|
||||
## Surface 7 — Test realism: stub fidelity (P2-01/P2-02 tests)
|
||||
|
||||
**Forcing question**: What would a stub MISS that the real forge does — and is the
|
||||
as-built client ever validated against the real forge before its one live write?
|
||||
|
||||
- **Evidence**: the stub coverage is unusually good (anonymous-200 trap, index→amd64
|
||||
resolution with Accept check, HEAD-skip, monolithic+two-step uploads, 307-following,
|
||||
digest and diff_id assertions, both tags; multipart field/`?name=` byte assertions,
|
||||
idempotent re-attach, 500×3 marker, token discipline). Gaps: fixtures are frozen at
|
||||
research-time shapes (2026-10-10) while upstream index and forge auth can move
|
||||
mid-milestone; the 307 test doesn't pin the relative-`Location` case (bound in
|
||||
G-019); gitea's 4 MB attachment cap is out-of-band (pyz ≈30 KB — immaterial, noted);
|
||||
and **no step anywhere validates the as-built scripts against the real
|
||||
`git.coreci.dev` in read-only mode** — the first live run of `publish_image.py` is
|
||||
the P3-05 milestone publish itself, the one shot per milestone.
|
||||
- **Owner decision (this session)**: add an env-gated, read-only live smoke as a
|
||||
P3-05 pre-ship verify step. → **G-013**.
|
||||
|
||||
## Surface 8 — Wave ordering & effort (P2 packing)
|
||||
|
||||
**Forcing question**: Is P2 overloaded, and are there hidden dependencies that break
|
||||
the wave order?
|
||||
|
||||
- **Evidence**: original P2 packs `_env_secrets` + `release.py` + `publish_image.py` +
|
||||
`Dockerfile` + README + 2 test files into 3 waves — the overload is real, and the
|
||||
bridge finding (Surface 3) is the reason it dissolves: release automation belongs in
|
||||
P1 (G-011), leaving P2 = publish client → Dockerfile/docs → ship. Hidden
|
||||
dependencies check out: `publish_image.py` imports `_env_secrets` (lands earlier in
|
||||
P1 now — order preserved transitively); the Dockerfile task consumes P1-01's staging
|
||||
allowlist (G-016 formalizes the contract as a test). `publish_image.py` remains the
|
||||
plan's biggest single commit even after the restructure.
|
||||
- **Owner decision (this session)**: keep it one vertical task — the pull/push seam is
|
||||
artificial, the stub fixtures are shared end-to-end, RESEARCH §3.4 is near-pseudocode,
|
||||
and a split would not isolate the actual risk (first live write — covered by G-013).
|
||||
→ **G-014**.
|
||||
|
||||
---
|
||||
|
||||
## Binding Decisions
|
||||
|
||||
| ID | Surface | Decision | Binds | Confidence |
|
||||
|----|---------|----------|-------|------------|
|
||||
| G-009 | 1 | Normalize pass must SET constant `external_attr` (files `0o100644<<16`, dir entries `0o40755<<16`), not "preserve" staging-derived modes; staging is an explicit allowlist (`coreci_broker.py`, `coreci_broker_client.py`, `coreci_broker_ui/**`, shim staged as root `__main__.py`, generated `_cli_version.py`) — never a repo rglob, `__pycache__`/`dist/` excluded by construction. Byte-identity scope = same environment (same zlib/machine); cross-zlib identity is not claimed. | P1-01 task edit | 0.85 |
|
||||
| G-010 | 2 | P1-02 harness hardening: subprocess boots use `sys.executable` (absolute); child env adds `BROKER_CREDS_SOURCE=file` + `BROKER_CREDS_FILE=<tmp fixture>` (v0.1 precedent test_acceptance_full.py:76-77); deadline-bounded wait for the listening line (~30s → fail with captured stdout+stderr, terminate child); pass-once/silent-second-boot assertions completed by draining stdout after terminate; ADD a default-anchoring boot case (`cwd=<tmp>`, no state overrides beyond `BROKER_BIND`) asserting `.env`/`broker.db`/`broker.audit.log` in the CWD — the owner's literal happy path. | P1-02 task edit | 0.85 |
|
||||
| G-011 | 3 | **Restructure (owner, this session)**: move `scripts/_env_secrets.py` + `scripts/release.py` + `tests/test_release_upload.py` into Phase 1 as new wave 3 (P1-03 · release-asset automation); old P1-03 ship renumbers P1-04 and ships v0.2.1 script-driven (`build_cli.py v0.2.1` → `release.py v0.2.1 <pyz>`); the one-shot lead-upload note is DELETED (its cited mechanism is absent from this environment — RESEARCH §2.2). P2 renumbers: publish_image.py → P2-01 (wave 1), Dockerfile+docs → P2-02 (wave 2), ship → P2-03 (wave 3). Four-test-file count unchanged. | PLAN edit (before P1-01 executes) | 0.90 |
|
||||
| G-012 | 4 | P3-05 sequence corrected: squash-merge → `main` → tag `v0.2.3` → push → THEN `release.py` attach (GET-by-tag finds the real tag; no pre-merge release create) → publish image → closeout. Mirrors P2's own correct ship order; eliminates the gitea create-tag-at-target_commitish hazard. | P3-05 ship-step edit | 0.80 |
|
||||
| G-013 | 7 | **Env-gated live read-only smoke (owner, this session)**: pre-publish P3-05 verify step — gitea `GET /v2/` 401 challenge shape, `GET .../releases/tags/{tag}`, `GET /v2/token` Basic → non-empty JWT, Docker Hub anonymous token + `3.11-slim` index→amd64 resolve. Implemented as a `--check`/verify mode on the scripts or `scripts/verify_endpoints.py`; GET-only, never in the default unittest run; tokens only via `_env_secrets`. | P3-05 verify + task edit | 0.85 |
|
||||
| G-014 | 8 | **publish_image.py stays one vertical task (owner, this session)**: shared stub fixtures, RESEARCH §3.4 near-pseudocode; a split would not isolate the real risk (first live write — G-013 covers it). | PLAN shape (no edit) | 0.75 |
|
||||
| G-015 | 5 | `.dockerignore` added to the Dockerfile task: excludes `.git`, `.env*`, `data/`, `*.db`, `*.log`, `dist/`, `__pycache__`, `tests/`, `.ciagent/`. Build-context hygiene; P3-02 scope gains "no secrets in the build context" alongside "none in any layer". | P2-02 task edit | 0.85 |
|
||||
| G-016 | 5 | Dockerfile equivalence made testable: the Dockerfile task adds a consistency test (into `tests/test_build_cli.py`) asserting COPY source set == `build_cli.py` staging allowlist, identical zipapp flags (`--python "/usr/bin/env python3" --compress`), identical `_cli_version.py` generation format, `ENTRYPOINT`/`WORKDIR` == publish_image's `config.Entrypoint`/`config.WorkingDir`. Presence-only verification no longer gates a possibly-broken Dockerfile. | P2-02 task edit | 0.80 |
|
||||
| G-017 | 5 | Digest discipline: the sha256 in release notes is computed by `release.py` from the artifact file itself — never a CLI argument, never hand-pasted. Digest/asset mismatch becomes structurally impossible. | P1-03 task edit | 0.85 |
|
||||
| G-018 | 6 | Shim private-API dependency tracked: `cli/__main__.py` header documents the coupling to `coreci_broker._parse_env_file` + `run_standalone` ("private-API dependency — any future broker refactor must re-run the v0.2 packaged acceptance"); P3-02 review scope lists it. Loud-ImportError breakage accepted. | P1-01 task edit + P3-02 scope | 0.75 |
|
||||
| G-019 | 4 | Publish-client wire hardening: explicit per-request timeouts on every registry call (generous on blob uploads, ~120s); blob-level bounded retry with HEAD-skip between attempts (a timed-out 29.8 MB POST resumes cheap); the 307 stub test uses a RELATIVE `Location` at least once (gitea may answer relative — RESEARCH §3.3; absolute is the easy case). | P2-01 task edit | 0.80 |
|
||||
| G-020 | 4 | Publish-failure closure is a named step, not an open escalation: the `release_pending` entry records the exact re-run command (`python3 scripts/publish_image.py` on `main` @ `v0.2.3`; `--base-manifest` optional pin) and its close condition (both tags visible in gitea packages). "Never blocks merge" unchanged. | P3-05 ship-step edit | 0.80 |
|
||||
| G-021 | 4 | The `v0.2.0` (phase-0) release carries NO `.pyz` and its notes must not promise one — D-020 starts at `v0.2.1` (build tooling first exists in P1). Prevents a D-020 misread at the phase-0 ship immediately following this grill. | Phase-0 ship note | 0.85 |
|
||||
|
||||
## Escalations
|
||||
|
||||
- (none open — the three material forks were resolved interactively with the owner this
|
||||
session: G-011 restructure, G-013 live smoke, G-014 task packing. G-001..G-008 from
|
||||
v0.1 remain binding per git history; G-005/G-008 carry forward in ROADMAP post-milestone
|
||||
notes unchanged.)
|
||||
|
||||
## Promotion note (explicit, not performed by the grill)
|
||||
|
||||
G-011 shifts REQ-22's attach automation into Phase 1; ROADMAP.md's Phase 2 description
|
||||
("release-asset automation" wording) and Phase 1's requirements line need a one-line
|
||||
touch-up when the PLAN restructure is promoted — by the orchestrator/owner via the
|
||||
normal plan-revision path, not by this file. Nothing else in PROJECT/REQUIREMENTS/ROADMAP
|
||||
is affected by these verdicts.
|
||||
|
||||
## Overall Verdict
|
||||
|
||||
**GO-WITH-CONDITIONS** — confidence 0.82.
|
||||
|
||||
The plan is research-grounded to an unusual degree (live wire shapes, working prototype,
|
||||
zero-broker-lines verified against the file), and every finding is actionable by task or
|
||||
ship-step edit. It is GO only with all of the following applied:
|
||||
|
||||
1. **G-011 restructure applied to PLAN.md before P1-01 executes** — the bridge is
|
||||
unexecutable as written; v0.2.1 must ship script-driven.
|
||||
2. **G-009 + G-010 applied to P1-01/P1-02** — determinism must fix (not preserve)
|
||||
entry attributes; the subprocess harness needs interpreter-path, creds-env, timeout,
|
||||
drain, and default-anchoring discipline.
|
||||
3. **G-012 + G-020 applied before P3-05** — attach-after-tag ordering and the
|
||||
publish-failure closure runbook are ship-contract corrections, not optional polish.
|
||||
4. **G-013 live smoke wired as the P3-05 pre-ship verify step** — the one live publish
|
||||
must not also be the first live wire contact of the as-built client.
|
||||
5. **G-015/016/017/018/019 applied with their tasks** — context hygiene, testable
|
||||
Dockerfile contract, digest provenance, tracked private-API coupling, wire timeouts.
|
||||
6. **G-021 noted at the phase-0 ship** — v0.2.0 carries no `.pyz`.
|
||||
@@ -0,0 +1,75 @@
|
||||
# PERSONAS — CoreCI Broker
|
||||
|
||||
Execution-stage roster (2026-10-10, milestone v0.2 cli-packaging). Pure stdlib Python 3.11 project: no package.json, no JS framework, no pip, no npm, no container runtime in the dev environment — frameworks are stdlib modules; tests are stdlib unittest against in-process stub HTTP servers and subprocess boots. The v0.1 broker is FROZEN this milestone: `coreci_broker.py` stands at 1390/1400 lines (REQ-17/AC14) and v0.2 adds zero broker lines (verified, RESEARCH §4.2), so no persona holds `coreci_broker.py` — or the v0.1 UI/client/reference sources — as writable territory; a P0 defect there is a lead-developer-dispatched exception only. All v0.2 work is new files: `cli/` boot shim, `scripts/` build+release+publish tooling, `Dockerfile`, four new test files, additive README sections, `.gitignore` additions. The custom release-engineer persona owns the whole distribution pipeline; backend-engineer — v0.1's primary persona — is deactivated with the P0-dispatch exception documented below. Execution is sequential (parallelization off), so phase-specific personas activate only at their phase boundaries per this roster.
|
||||
|
||||
---
|
||||
name: release-engineer
|
||||
domain: distribution
|
||||
active: true
|
||||
phase_specific: false
|
||||
reason: Primary execution persona — v0.2 is distribution engineering on a frozen broker, a domain distinct from v0.1's broker application work; hence a custom persona rather than a fold into backend-engineer, whose identity, constraints, and territory are the broker's runtime contract — a poor fit for build/release/publish tooling and a structural guarantee that the frozen file is owned by nobody. Owns the full pipeline — P1 zipapp build + materialization boot shim (REQ-21), P2 gitea release-asset automation + daemonless OCI registry publish + Dockerfile (REQ-22/23), P3 ship — and every security-critical surface this milestone: secrets discipline (file-only token loader), artifact integrity (no secrets in the .pyz or image layers, reproducible digests), registry auth flows.
|
||||
frameworks: [python3.11, zipapp, zipfile, tarfile, gzip, hashlib.sha256, urllib.request, json, subprocess, http.server, unittest]
|
||||
constraints: ["coreci_broker.py is FROZEN — zero lines added (1390/1400, REQ-17/AC14); broker sources (coreci_broker.py, coreci_broker_client.py, coreci_broker_ui/**, reference_tools/**) are read-only staging inputs to the build, never edited; a P0 defect there routes through lead-developer dispatch", "stdlib-only build path — python3 -m zipapp --compress; no pip, no npm, no wheel/PyInstaller, no container runtime (D-009/D-019)", "secrets discipline — GITEA_TOKEN resolves ONLY via scripts/_env_secrets.py reading .env/.env.secrets/.env.* files, never os.environ (blocked_env_vars: GITEA_TOKEN); assert non-empty before the first API call (v1.8 lesson: empty token → gitea 404 masks auth failure); no secrets ever enter the .pyz or an image layer", "digest determinism — zip entries sorted with date_time (1980,1,1,0,0,0) via the normalize post-pass; tar entries sorted with fixed TarInfo metadata; gzip.compress(mtime=0); json.dumps(sort_keys=True, separators=(',',':')); two-build sha256 equality gates the artifact (REQ-21)", "OCI end-to-end — manifest application/vnd.oci.image.manifest.v1+json, config application/vnd.oci.image.config.v1+json, layers application/vnd.oci.image.layer.v1.tar+gzip; base layer descriptors reused verbatim; never push an index; urllib does NOT follow 307/308 on PUT/POST — implement explicit Location-following; gitea's anonymous-200 /v2/token answer must fail loudly; HEAD-before-upload makes re-runs idempotent", "bounded retry (3 attempts) on release attach and publish; on exhaustion exit non-zero with a machine-readable marker → ship harness records release_pending escalation; never blocks tag+merge", "stub-testability — scripts take base URLs as parameters (no hardcoded hosts); stubs bind ('127.0.0.1', 0) per the v0.1 _StubOwuiHandler precedent; no live-forge writes in tests; v0.1 test files are frozen reference — v0.2 adds test_cli_packaging.py, test_build_cli.py, test_release_upload.py, test_publish_image.py only, and the cumulative suite stays green", "dist/ is build output — gitignored (dist/, *.pyz), never committed; README.md edits are additive sections (CLI install/usage, docker usage) with v0.1 content intact"]
|
||||
territory: [cli/**, scripts/**, tests/**, Dockerfile, README.md, .gitignore]
|
||||
subagent_type: backend-engineer
|
||||
---
|
||||
|
||||
---
|
||||
name: security-engineer
|
||||
domain: security
|
||||
active: true
|
||||
phase_specific: true
|
||||
reason: Review-only pass activated for P3 (milestone security review — artifact integrity, no secrets in the archive/image, token discipline in upload paths, registry auth flows, boot parity of materialized assets); inactive in P1/P2 unless the lead escalates a finding — same posture as v0.1.
|
||||
frameworks: [python3.11, unittest, STRIDE, CWE, oci-dist-spec]
|
||||
constraints: ["review-only — every v0.2 file stays owned by release-engineer; findings route through lead-developer, never direct edits; coreci_broker.py is in review scope for zero-delta verification (AC14) only and is writable by nobody this milestone", "findings are STRIDE-classified and CWE-first (cite CWE ids)", "focus — secrets discipline (file-only token path via scripts/_env_secrets.py, never shell env, non-empty assert before first use), artifact integrity (no secrets in the .pyz or any image layer; sha256 provenance recorded in release notes), registry auth flows (gitea anonymous-200 /v2/token quirk, Basic username-ignored shape, 307/308 handling), attachment idempotency and bounded-retry escalation paths"]
|
||||
territory: [coreci_broker.py, cli/**, scripts/**, Dockerfile, tests/**]
|
||||
subagent_type: ci-security-auditor
|
||||
---
|
||||
|
||||
---
|
||||
name: lead-developer
|
||||
domain: coordination
|
||||
active: true
|
||||
phase_specific: false
|
||||
reason: Active across all execution phases (P1–P3): decomposes plans into vertical slices along the build → attach → publish pipeline, arbitrates the frozen-broker rule (no persona writes coreci_broker.py — P0 fixes are lead-dispatched exceptions), routes security findings, and sequences tasks across phase boundaries. No direct code writes.
|
||||
frameworks: [none]
|
||||
constraints: ["no direct modifications to code files — coordination and conflict resolution only", "sole gate for frozen-broker exceptions — any P0 fix to coreci_broker.py (or the frozen UI/client/reference sources) is a lead-dispatched, defect-scoped task, never a persona territory claim", "arbitration — release-engineer owns every v0.2 file in cross-persona conflicts and defines the artifact/registry/release contracts; security findings flow lead → release-engineer; battle-tested defaults over novel approaches", "enforce — zero broker-line delta (AC14), secrets only via the .env* file loader, no secrets in artifacts, bounded-retry release_pending escalations never block tag+merge"]
|
||||
territory: [.ciagent/PLAN.md, .ciagent/*.md]
|
||||
subagent_type: lead-developer
|
||||
---
|
||||
|
||||
---
|
||||
name: backend-engineer
|
||||
domain: backend
|
||||
active: false
|
||||
phase_specific: false
|
||||
reason: Deactivated — the broker is frozen in v0.2; RESEARCH §4.2 verified run_standalone/boot/load_config/_parse_env_file are importable as-is, 1390/1400 lines leaves 10 lines of headroom, and zero-change is a hard constraint. All v0.2 engineering is distribution tooling owned by release-engineer. Reactivates only as a lead-developer-dispatched, defect-scoped P0 fix to coreci_broker.py — never via its own territory.
|
||||
frameworks: [python3.11, http.server, sqlite3, hashlib.scrypt, unittest]
|
||||
constraints: ["stays inactive for all planned v0.2 work; any broker change is a lead-developer-dispatched exception scoped to the P0 defect", "if dispatched — the 1400-line AC14 cap and every v0.1 runtime constraint (REQ-14..REQ-18, {error, message} JSON contract, 0600 modes, never write to creds-coreci-coo) remain binding"]
|
||||
territory: []
|
||||
subagent_type: backend-engineer
|
||||
---
|
||||
|
||||
---
|
||||
name: frontend-engineer
|
||||
domain: frontend
|
||||
active: false
|
||||
phase_specific: false
|
||||
reason: Deactivated — no UI work in v0.2; REQ-21..23 touch no page. The v0.1 UI (coreci_broker_ui/) is complete, frozen, and consumed as a read-only staging input by the zipapp build, then materialized verbatim at boot; any UI defect surfaced through the packaged path is a lead-dispatched exception.
|
||||
frameworks: [html, css, vanilla-js]
|
||||
constraints: ["stays inactive for the whole milestone; coreci_broker_ui/** is a read-only build input owned by no persona"]
|
||||
territory: []
|
||||
subagent_type: frontend-engineer
|
||||
---
|
||||
|
||||
---
|
||||
name: data-engineer
|
||||
domain: data
|
||||
active: false
|
||||
phase_specific: false
|
||||
reason: Deactivated (carried from v0.1) — no database tier beyond sqlite3 embedded in the frozen broker; no schema, pipeline, ETL, or migration work in v0.2. Image layers and release assets are build outputs owned by release-engineer, not data surfaces.
|
||||
frameworks: [sqlite3]
|
||||
constraints: ["stays inactive for the whole milestone; any residual data work folds into release-engineer or a lead-dispatched broker fix"]
|
||||
territory: []
|
||||
subagent_type: backend-engineer
|
||||
---
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
must_haves:
|
||||
- "python3 -m unittest discover -s tests passes (cumulative — the v0.1 suite plus the v0.2 test files: test_build_cli, test_cli_packaging, test_release_upload, test_publish_image, test_dockerfile_consistency, test_acceptance_v02)"
|
||||
- "python3 -m py_compile cli/__main__.py scripts/build_cli.py scripts/release.py scripts/publish_image.py scripts/_env_secrets.py"
|
||||
- "broker line count unchanged: coreci_broker.py stays at its v0.1.5 count of 1390 lines (AC14, D-016..D-018 cap 1400) — zero broker edits"
|
||||
- "two consecutive builds via scripts/build_cli.py produce byte-identical artifacts (sha256 equal) — same-environment scope (G-009)"
|
||||
- "the built .pyz boots with v0.1 acceptance parity (admin pass printed exactly once, broker.db + broker.audit.log 0600, UI + client served, silent second boot)"
|
||||
- "stub-registry publish roundtrip green (scripts/publish_image.py vs a local stub, including a relative-Location 307 case, G-019) and stub release upload green (scripts/release.py vs a local stub)"
|
||||
---
|
||||
|
||||
# PLAN — CoreCI Broker (Milestone v0.2 — cli-packaging)
|
||||
|
||||
Execution conventions: sequential waves (within a wave, tasks run in listed order); each task = one subagent commit `feat(P##): …` carrying a `---ci---` block (phase, milestone: v0.2, status: execute, persona, task, requirements.covered). Tests ship with their task — never deferred. Personas per PERSONAS.md: release-engineer (ALL v0.2 code — `cli/**`, `scripts/**`, the new test files, `Dockerfile`, `.dockerignore`, additive README, `.gitignore`), security-engineer (P3 only, review-only — findings route via lead-developer, never direct edits), lead-developer (coordination + ship steps, no code writes; sole gate for frozen-broker exceptions). The broker is FROZEN this milestone (1390/1400, REQ-17/AC14): no task edits `coreci_broker.py`, `coreci_broker_ui/`, `coreci_broker_client.py`, or `reference_tools/` — a P0 defect there is a lead-developer-dispatched exception only. Verify commands: `python3 -m py_compile <files>`; `python3 -m unittest discover -s tests`; broker zero-delta via the AC14 line-count test. Test harness (v0.1 §6 conventions extended, RESEARCH §4.3): `ThreadingHTTPServer` stubs on `("127.0.0.1", 0)` + `tempfile.TemporaryDirectory()` state + subprocess boots of the built artifact driven with stdlib `urllib`; scripts take base URLs as parameters (stub-testable); NEVER live-forge writes in tests (the only live-forge contact is the G-013 smoke gate + the ships themselves). Secrets resolve only via `scripts/_env_secrets.py` reading `.env`/`.env.secrets`/`.env.*` files — never shell env (`GITEA_TOKEN` is in `blocked_env_vars`); non-empty asserted before the first API call. `dist/` + `*.pyz` are gitignored — build artifacts are never committed. Release attach + image publish failures follow bounded retry (3) → `release_pending` escalation — never blocking tag+merge. Ships: phase 0 → `v0.2.0`, P1 → `v0.2.1`, P2 → `v0.2.2`, P3 → `v0.2.3` (final phase patch = milestone release, D-008 convention). **G-021: the `v0.2.0` phase-0 release carries NO `.pyz` and its notes must not promise one — D-020 starts at `v0.2.1`.** Release-tooling split (G-011): the sanctioned secrets loader + release attacher land in P1 (Wave 3) so every release from `v0.2.1` is script-attached; P2 owns the image publish client + Dockerfile.
|
||||
|
||||
## User-Facing Surface
|
||||
|
||||
- **Downloadable CLI artifact** — `coreci-broker-vX.Y.Z.pyz` on the gitea release page (every release from `v0.2.1`, D-020), alongside the existing source archive; its sha256 recorded in the release notes (computed by `scripts/release.py` from the artifact file — G-017).
|
||||
- **CLI boot** — `python3 coreci-broker-vX.Y.Z.pyz` or `./coreci-broker-vX.Y.Z.pyz` after `chmod +x` (shebang `#!/usr/bin/env python3`); boots the standalone broker — full admin UI at `/ui/`, client module at `/static/coreci_broker_client.py`, bound address per `BROKER_BIND` (default `127.0.0.1:8765`).
|
||||
- **`--version`** — prints the release version, exits 0.
|
||||
- **Docker image** — `docker pull git.coreci.dev/coreci/broker:<tag>` (+ `latest`); `docker run` with a volume for state.
|
||||
- **Release notes** — on the release page: milestone summary, phases, verification, CLI install + usage, docker usage.
|
||||
- The v0.1 surface (four-page UI + dashboard, client module, audit) unchanged — served through the packaged path.
|
||||
|
||||
## Happy Path
|
||||
|
||||
1. Owner downloads `coreci-broker-v0.2.3.pyz` from the `v0.2.3` release → `python3 coreci-broker-v0.2.3.pyz` (or `chmod +x` + `./…`) → first boot generates `.env` and prints the admin pass exactly once; `broker.db` + `broker.audit.log` created mode 0600 in the state dir → UI at the bound address (`/ui/login` bookmark works).
|
||||
2. Second boot: silent — pass never re-printed; state persists.
|
||||
3. Tool authors fetch `coreci_broker_client.py` from `/static/coreci_broker_client.py` exactly as in v0.1 — the packaged artifact is the only input.
|
||||
4. `--version` reports the release version without booting.
|
||||
5. Separately: `docker pull git.coreci.dev/coreci/broker:v0.2.3` → `docker run -p 8765:8765 -v broker-state:/state …` → identical first-boot/0600/UI behavior inside the container, state on the volume.
|
||||
|
||||
## UX Acceptance Criteria
|
||||
|
||||
- **Self-contained artifact** — the `.pyz` bundles broker + UI pages + client module; only runtime state lives outside it (`.env`, `broker.db`, `broker.audit.log`, materialized assets under `<state>/.coreci_broker_assets/`); one file serves the full admin UI at `/ui/` and the client at `/static/`.
|
||||
- **`--version`** — prints the release version (from the build-time `_cli_version.py`), exits 0, no broker import on that path.
|
||||
- **First-boot parity** — behavior through the packaged artifact is byte-exact with v0.1 standalone: admin pass printed exactly once, `broker.db` + `broker.audit.log` mode 0600, UI + client served, second boot silent.
|
||||
- **Per-release artifact** — every release from `v0.2.1` (D-020) carries `coreci-broker-vX.Y.Z.pyz` + release notes (CLI install/usage + sha256, docker usage); attachment failure escalates `release_pending`, never blocks tag+merge. `v0.2.0` carries none (G-021).
|
||||
- **Image published once per milestone** — at the final ship: `git.coreci.dev/coreci/broker:v0.2.3` + `latest` (D-021); publish failure escalates per the same contract with the G-020 closure runbook, never blocking the milestone merge.
|
||||
|
||||
## Phase 1 — CLI Packaging + Release Tooling
|
||||
|
||||
Branch `phase/01-cli-packaging` → tag `v0.2.1`. Requirements: REQ-21, REQ-22.
|
||||
Exit: artifact builds reproducibly from a clean checkout (two consecutive same-environment builds → byte-identical sha256); the built `.pyz` boots with v0.1 acceptance parity (pass printed once, 0600 state files, UI + client served from materialized assets, silent second boot); `--version` wired; release-asset automation stub-verified; REQ-21 + REQ-22 consolidated green; the `v0.2.1` release carries the `.pyz` attached by `scripts/release.py`.
|
||||
|
||||
### Wave 1 — build pipeline + boot shim
|
||||
|
||||
- **P1-01** · release-engineer · `scripts/build_cli.py`, `cli/__main__.py`, `.gitignore`, `tests/test_build_cli.py` · build pipeline + boot shim (RESEARCH §1.2/§1.3; G-009/G-018): `build_cli.py <version>` stages an explicit allowlist tree — `coreci_broker.py`, `coreci_broker_client.py`, `coreci_broker_ui/**`, `cli/__main__.py`, generated `_cli_version.py` (`VERSION = "<tag>"`) — then `python3 -m zipapp <staging> -o dist/coreci-broker-vX.Y.Z.pyz --python "/usr/bin/env python3" --compress` (zipapp defaults to ZIP_STORED — `--compress` mandatory), then a normalize post-pass making same-environment rebuilds byte-identical (G-009: zip entries sorted by name, `date_time=(1980,1,1,0,0,0)`, `ZIP_DEFLATED`, **`external_attr` SET to constants** — files `0o100644 << 16`, dirs `0o40755 << 16` — never preserved/umask-derived; shebang re-prepended + exec bit), and prints the artifact sha256 (digest-in-notes default; no sidecar asset); `cli/__main__.py` = the five-step shim: (1) `--version` first — print `_cli_version.VERSION`, exit 0, no broker import; (2) state-dir resolution identical to the broker (`BROKER_ENV_PATH`-anchored, relative paths stay CWD-relative — zero path changes); (3) materialize assets from `zipfile.ZipFile(sys.path[0])` → `<state>/.coreci_broker_assets/<VERSION>/{ui,static}` (temp + `os.replace`, versioned dirs, same-version overwrite self-repair, no cleanup); (4) override precedence user-wins — `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` set only when absent from process env AND the user `.env` (parsed via `coreci_broker._parse_env_file`); (5) handoff `coreci_broker.run_standalone()`; an `os.path.isdir(sys.path[0])` guard falls back to repo-relative roots when run unzipped (zipapp-only is the documented mode); the shim header documents its private-API coupling to the broker (`_parse_env_file`, `run_standalone` — G-018: loud `ImportError` breakage accepted and tracked in P3-02); `.gitignore` += `dist/`, `*.pyz`; broker sources are read-only staging inputs — zero broker lines · REQ-21 · test: `test_build_cli.py` — two consecutive same-environment builds → byte-identical sha256; `_cli_version.VERSION` embedded; archive enumerates exactly the staging allowlist (broker + client + UI pages + shim); constant `external_attr` asserted (no umask bleed); `py_compile` of the shim green; built artifact `--version` prints the version, exits 0.
|
||||
|
||||
### Wave 2 — packaged boot acceptance
|
||||
|
||||
- **P1-02** · release-engineer · `tests/test_cli_packaging.py` · end-to-end boot acceptance through the built artifact (RESEARCH §4.3 pattern; G-010 harness hardening; consolidates REQ-21): build into a `TemporaryDirectory` → `Popen([sys.executable, "<pyz>"], env={"BROKER_BIND": "127.0.0.1:0", BROKER_ENV_PATH/DB/AUDIT/CREDS_FILE/CREDS_SOURCE: tmp paths})` → parse the `coreci-broker listening on http://…` line for the ephemeral base URL under an explicit deadline loop; first boot prints `BROKER_ADMIN_PASS=` exactly once (stdout fully drained after `terminate()` before the second-boot assertion); `.env` + `broker.db` + `broker.audit.log` created mode 0600 in the state dir (independent of CWD); `/healthz` 200, `/ui/login` + `/ui/dashboard` 200 from materialized assets, `/static/coreci_broker_client.py` byte-exact; admin login + agent register + state + secret-read reuse the v0.1 acceptance request patterns (v0.1 parity through the packaged path); second boot silent (pass never re-printed); `--version` exits 0 with the version string; materialized assets verified under `<state>/.coreci_broker_assets/<VERSION>/`; user override honored (`.env`-set `BROKER_UI_ROOT` wins over the shim default); plus one un-overridden default-anchoring boot (state in CWD of a temp dir — the owner's literal happy path, G-010) · REQ-21 · test: is the acceptance file.
|
||||
|
||||
### Wave 3 — release-asset automation (G-011: lands in P1)
|
||||
|
||||
- **P1-03** · release-engineer · `scripts/_env_secrets.py`, `scripts/release.py`, `tests/test_release_upload.py` · sanctioned secrets loader + release attacher (REQ-22; RESEARCH §2): `_env_secrets.py` parses `.env`/`.env.secrets`/`.env.*` from the repo root + `.ciagent/` (mirroring the broker's `_parse_env_file` semantics, quote-stripping) — `GITEA_TOKEN` resolves ONLY from these files, never `os.environ` (structurally blocked by `blocked_env_vars`), asserted non-empty before the first API call (v1.8 lesson: empty token → gitea 404, masking auth failure); `release.py <tag> <artifact>` (gitea base URL + owner/repo parameterized): **computes the artifact sha256 from the file itself — never an argument, never hand-pasted (G-017)** → builds the notes body per the RESEARCH §2.3 model (milestone summary, phases, verification, CLI install + usage with the artifact sha256, docker usage) → create-or-get by tag (`GET /api/v1/repos/{o}/{r}/releases/tags/{tag}`; 404 → `POST .../releases` with the notes body) → idempotent attach (`GET .../releases/{id}/assets` — same name + size → success, no duplicate on bounded-retry re-runs) → `POST .../releases/{id}/assets` multipart (field name `attachment`, `?name=coreci-broker-vX.Y.Z.pyz`, `Authorization: token <T>`) → bounded retry (3 attempts, backoff) → exhaustion exits non-zero with a machine-readable marker for the `release_pending` escalation; never blocks tag+merge · REQ-22 · test: `test_release_upload.py` — `_StubGiteaHandler` on `("127.0.0.1", 0)` (v0.1 `_StubOwuiHandler` precedent, fake token, no live-forge writes): create-or-get in both orders (release exists / 404 → create), multipart parsed (field name + `?name=` + body bytes asserted), sha256-from-file asserted (passing a wrong digest argument is ignored/refused), idempotent re-attach uploads nothing twice, 500×3 → the `release_pending` marker, token discipline (file-only resolution + non-empty assert before the first call), notes sections present.
|
||||
|
||||
### Wave 4 — phase ship
|
||||
|
||||
- **P1-04** · lead-developer · no code writes (the release-engineer's built artifact + scripts are consumed read-only) · phase ship: tag `v0.2.1` → squash-merge `phase/01-cli-packaging` → `milestone/v0.2-cli-packaging` → push tag + branch → gitea release via `scripts/release.py` (D-020 starts here: `coreci-broker-v0.2.1.pyz` + its sha256 in the notes, attached by the repo script — no ad-hoc bridge, G-011) · REQ-21, REQ-22 (ship) · verify: `v0.2.1` tag + release on the forge with the `.pyz` attached; attachment failure → bounded retry (3) → `release_pending` escalation, never blocking tag+merge.
|
||||
|
||||
## Phase 2 — Docker + Image Publish
|
||||
|
||||
Branch `phase/02-docker-release-integration` → tag `v0.2.2`. Requirements: REQ-23 (+ REQ-21/REQ-22 docs refinement).
|
||||
Exit: daemonless publish client stub-verified end-to-end (no live-forge writes, no container runtime); Dockerfile + .dockerignore + README document the docker-user path and stay consistency-tested against the daemonless image; the `v0.2.2` release carries the `.pyz` attached by `scripts/release.py` with CLI + docker notes.
|
||||
|
||||
### Wave 1 — daemonless OCI publish client
|
||||
|
||||
- **P2-01** · release-engineer · `scripts/publish_image.py`, `tests/test_publish_image.py` · daemonless OCI publish client (D-019, G-014: one vertical task; all endpoints parameterized; G-019 wire hardening): pull — anonymous Docker Hub token (`auth.docker.io/token`, scope `repository:library/python:pull`) → `python:3.11-slim` manifest (D-022) with the full Accept set → branch on `"manifests"` vs `"layers"` (index → resolve the `linux/amd64` child manifest by digest) → fetch child + config + 4 layer blobs; record resolved digests (`--base-manifest` override for exact re-publishes); auth — gitea `GET /v2/token?service=container_registry&scope=repository:coreci/broker:pull,push` with `Authorization: Basic <user>:<GITEA_TOKEN>` (token via `_env_secrets`; username ignored by the forge) — insist on Basic credentials present + issued JWT non-empty (the endpoint answers 200 anonymously with an empty-scope JWT — fail loudly, never treat 200 as authorization); push — **per-request timeouts (~120s on blob POSTs, G-019)** + **blob-level retry with HEAD-skip** (HEAD each blob first: 200 → skip; base layers ~45.7 MB re-upload once per milestone, retries resume cheap) → monolithic `POST /v2/coreci/broker/blobs/uploads/?digest=sha256:<hex>` (fallback: two-step POST → 202 `Location` → PUT) with explicit Location-following on 307/308 (stdlib urllib does NOT follow method-preserving redirects on PUT/POST — verified; **the stub test covers a relative `Location` at least once, G-019**) → deterministic broker layer (tar entries sorted, fixed TarInfo metadata uid/gid 0, `gzip.compress(..., mtime=0, compresslevel=9)`) carrying `/app/coreci-broker.pyz`; config copied from the base + diff_id appended + `Entrypoint ["python3", "/app/coreci-broker.pyz"]`, `Cmd []`, `WorkingDir /state`, optional version Label, serialized `json.dumps(sort_keys=True, separators=(",",":"))`; manifest PUT with OCI media types end-to-end (manifest `application/vnd.oci.image.manifest.v1+json`, config `…image.config.v1+json`, layers `…image.layer.v1.tar+gzip`; base layer descriptors reused verbatim; never push an index) under `<version>` then `latest` (D-021); provenance digests recorded for the release notes · REQ-23 · test: `test_publish_image.py` — `_StubRegistryHandler` on `("127.0.0.1", 0)` (fake token via temp `.env` files): anonymous-token trap (200 with empty-scope JWT → loud failure), HEAD-skip on retry, monolithic upload with 307 → relative `Location` followed explicitly, manifest PUT with correct media types + tag sequence (version then `latest`), deterministic layer/config digests stable across two runs (gzip mtime=0), base manifest/layer handling against a fixture index, timeout wiring (a stalled endpoint → client aborts, does not hang).
|
||||
|
||||
### Wave 2 — Dockerfile + .dockerignore + docs
|
||||
|
||||
- **P2-02** · release-engineer · `Dockerfile`, `.dockerignore`, `README.md` (additive section only — no v0.1 content rewrites), `tests/test_dockerfile_consistency.py` · docker-user path + usage docs (G-015/G-016): `Dockerfile` self-contained (no pre-built artifact in the build context): `FROM python:3.11-slim` → `COPY coreci_broker.py coreci_broker_client.py coreci_broker_ui/ cli/ /app/src/` → `ARG VERSION` → generated `_cli_version.py` from the build arg → `RUN python3 -m zipapp /app/src -o /app/coreci-broker.pyz --python "/usr/bin/env python3" --compress` → `WORKDIR /state` → `ENTRYPOINT ["python3", "/app/coreci-broker.pyz"]` → `EXPOSE 8765`; header documents the equivalence mapping to the daemonless-published image (same base family, entrypoint, pyz content, `/state` anchoring; differences: docker-build layering vs verbatim base layers + one broker layer, build-arg vs build-time version provenance; the daemonless client is the source of truth, byte-identity not required — RESEARCH §3.5); `.dockerignore` (G-015): `.git`, `.env`, `.env.*`, `data/`, `*.db`, `dist/`, `__pycache__/`, `tests/`, `.ciagent/` — no secrets, state, or build artifacts can enter an image build context; README additive section: CLI install + usage (release-page download, `chmod +x` / `python3`, `--version`, env-contract pointer, first-boot pass discipline, `BROKER_ENV_PATH` state anchoring, Python 3.11+ note) + docker usage (`docker pull git.coreci.dev/coreci/broker:<tag>` + `latest`; `docker run -p 8765:8765 -v broker-state:/state …`; state-on-volume contract) · REQ-23, REQ-21/REQ-22 (docs) · test: `test_dockerfile_consistency.py` (G-016 — equivalence made testable): Dockerfile COPY set == `build_cli.py` staging allowlist; zipapp flags identical (`--python`/`--compress`); `ARG VERSION` → `_cli_version` format parity; `ENTRYPOINT`/`WORKDIR` == the daemonless-published config values; `.dockerignore` excludes every forbidden context entry.
|
||||
|
||||
### Wave 3 — phase ship
|
||||
|
||||
- **P2-03** · lead-developer · no code writes · phase ship: tag `v0.2.2` → squash-merge `phase/02-docker-release-integration` → `milestone/v0.2-cli-packaging` → push tag + branch → release via `scripts/release.py` (notes per the §2.3 model now including docker usage, `coreci-broker-v0.2.2.pyz` + sha256 attached alongside the source archive) · REQ-23 (phase completion) · verify: `v0.2.2` tag + release on the forge with the `.pyz` + notes; attachment failure → bounded retry → `release_pending`, never blocks tag+merge.
|
||||
|
||||
## Phase 3 — Final Review + Ship
|
||||
|
||||
Branch `phase/03-final-review-ship` → tag `v0.2.3` (final phase patch = milestone release, D-008 convention). Requirements: REQ-21, REQ-22, REQ-23 (acceptance).
|
||||
Exit: cumulative v0.1 + v0.2 acceptance green; broker zero-delta verified (1390/1400, AC14); security review PASS (or findings remediated); `v0.2.3` on `main` with the `.pyz` attached + full notes; image published as a gitea package (or escalated per the bounded-retry contract + G-020 closure runbook — never blocking merge); REQUIREMENTS/ROADMAP closed out; checkpoint cleared.
|
||||
|
||||
### Wave 1 — acceptance consolidation
|
||||
|
||||
- **P3-01** · release-engineer · `tests/test_acceptance_v02.py` · v0.2 acceptance matrix: cumulative suite green via `python3 -m unittest discover -s tests` (all v0.1 files + the v0.2 test files); packaging acceptance consolidated — boot parity through the artifact (first-boot pass printed once, `broker.db` + `broker.audit.log` 0600, UI + client served, silent second boot, register/read flows), reproducibility (two consecutive same-environment builds → identical sha256), self-containment (only runtime state external to the `.pyz`: env/db/audit + materialized assets), stub publish roundtrip + stub release upload green; broker zero-delta — `coreci_broker.py` unchanged at 1390 lines vs `v0.1.5` (AC14 under the D-016..D-018 cap of 1400) · REQ-21, REQ-22, REQ-23 (acceptance) · test: is the acceptance file.
|
||||
|
||||
### Wave 2 — security review (review-only)
|
||||
|
||||
- **P3-02** · security-engineer · review of `coreci_broker.py` (zero-delta verification only — writable by nobody this milestone), `cli/**`, `scripts/**`, `Dockerfile`, `.dockerignore`, `tests/**` (no edits — findings STRIDE-classified, CWE-cited, routed via lead-developer) · v0.2 security pass: secrets in artifacts (no secrets in the `.pyz` or any image layer; `.env` never inside the archive; pepper/admin pass generated at boot; no state baked into the image; build-context hygiene per `.dockerignore`, G-015), digest determinism (zip normalize pass with constant `external_attr` — G-009; sorted tar + fixed TarInfo; `gzip mtime=0`; sorted-key JSON), token discipline (file-only `_env_secrets` path, non-empty assert before first use, never shell env; gitea's anonymous-200 `/v2/token` quirk fails loudly), 307/308 Location handling incl. relative `Location`, traversal + disclosure over materialized asset roots (dot-dir roots vs request-path dot filtering; protected-paths realpaths wherever the state dir sits), shim private-API coupling tracked (G-018), attachment idempotency + bounded-retry escalation paths · REQ-21, REQ-22, REQ-23 (+ all) · verify: findings recorded in the task commit's `---ci---` block; remediation (if any) = the follow-up release-engineer commit dispatched by lead-developer (P3-03).
|
||||
|
||||
### Wave 3 — remediation
|
||||
|
||||
- **P3-03** · release-engineer · `cli/**`, `scripts/**`, `Dockerfile`, `.dockerignore`, `tests/**` (never broker sources — a P0 broker defect routes through lead-developer dispatch only) · remediate P3-02 findings (no-op skip if PASS) · affected REQs per finding · test: full cumulative suite re-green.
|
||||
|
||||
### Wave 4 — review + reconstruction
|
||||
|
||||
- **P3-04** · lead-developer · `.ciagent/*.md` review notes (no code writes) · multi-persona review of the milestone diff (release-engineer artifacts across P1/P2, security sign-off from P3-02/P3-03); audit/reconstruction: every REQ-21..REQ-23 maps to green tests + traceable commits; broker zero-delta verified (1390/1400, AC14); escalation ledger clean or `release_pending` entries documented · REQ-21, REQ-22, REQ-23 · verify: sign-offs + reconstruction recorded in `---ci---` blocks.
|
||||
|
||||
### Wave 5 — milestone ship
|
||||
|
||||
- **P3-05** · lead-developer · no code writes (release-engineer scripts invoked, never edited) · milestone ship, in this exact order (G-012: merge→tag→push BEFORE release attach — a pre-merge release create would mint `v0.2.3` at pre-merge main via gitea's target_commitish semantics): build the final `.pyz` (`python3 scripts/build_cli.py v0.2.3`) → squash-merge `milestone/v0.2-cli-packaging` → `main` → tag `v0.2.3` → push main + tag → **G-013 live smoke (env-gated `PUBLISH_SMOKE=1`, read-only, never in the default suite): real gitea `/v2/` auth challenge, token→JWT exchange, `GET` release-by-tag, Docker Hub index resolve — all must pass before the one live publish** → attach to the `v0.2.3` release with full notes (milestone summary, phases, verification, CLI install/usage + sha256, docker usage) via `scripts/release.py` → daemonless image publish ONCE (`python3 scripts/publish_image.py` → `git.coreci.dev/coreci/broker:v0.2.3` + `latest`, D-019/D-021/D-022) → branch cleanup → REQUIREMENTS/ROADMAP closeout → checkpoint cleared. Publish failure: bounded retry (3) → `release_pending` escalation **with the G-020 closure runbook committed alongside** (exact re-run command on `main` @ `v0.2.3`: build + smoke + `publish_image.py`; close condition = both tags visible as a gitea package) — never blocks the merge, never left open · REQ-21, REQ-22, REQ-23 · verify: `v0.2.3` on `main`; the release carries the `.pyz` + full notes; the image is visible as a gitea package (or escalated per contract + runbook, never blocking); closeout + checkpoint-clear commits recorded.
|
||||
@@ -0,0 +1,87 @@
|
||||
# PROJECT — CoreCI Broker
|
||||
|
||||
## Name
|
||||
|
||||
CoreCI Broker
|
||||
|
||||
## Core Value
|
||||
|
||||
A single canonical credential flow for autonomous agent tools. The broker mediates access to the `creds-coreci-coo` knowledge base: tools authenticate by a derived token (not a shared secret), the CEO manages a UI allowlist of approved tool UUIDs and user IDs, every read is audited to an append-only file, and per-tool scope is enforced by path (`coreci/<agent_id>` only). It replaces per-tool credential resolution (KB files, env vars, mode-0600 JSON) with one small, readable implementation surface: a single Python file, one SQLite DB, a four-page web UI.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Validated
|
||||
|
||||
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09; supersedes v0.4) — distilled into REQ-01..REQ-20 (shipped in v0.1). v0.2 additions validated from owner directives (2026-10-10) — distilled into REQ-21..REQ-23: CLI zipapp packaging, per-release artifact attachment with notes on gitea, once-per-milestone docker publication to the gitea container/package registry. The v0.5 spec's ambiguities were resolved by D-001..D-015 (init + clarify commits, v0.1); v0.2's packaging/distribution ambiguities are resolved as D-019..D-022 (this milestone's clarify stage):
|
||||
|
||||
- Dual-mode token hashing (bcrypt when importable, stdlib scrypt fallback; format-prefixed hashes)
|
||||
- Pluggable creds-source adapter (file-based default; Open WebUI HTTP API for production)
|
||||
- Agent-state backend toggle (`internal` default | `owui`)
|
||||
- UUID-scoped token verification + in-memory verified-token cache (spec §7.5's token_hash index is impossible with bcrypt)
|
||||
- Register/pepper semantics including the KB-loss recovery path
|
||||
- A public, dependency-free client module served as a static asset so agents can bootstrap their own OWUI tools
|
||||
- Spec §13 defaults: no audit rotation, single instance, 24h token TTL, CEO sole approver
|
||||
|
||||
### Active
|
||||
|
||||
Milestone v0.2 (`milestone/v0.2-cli-packaging`) — **CLI + distribution only**, no functional changes to the broker: REQ-21 (zipapp CLI packaging), REQ-22 (release assets attached + notes on gitea), REQ-23 (docker container built once per milestone, published as a gitea package). Owner-locked inputs: zipapp `.pyz` form factor; docker publication starts with v0.2 (no retroactive v0.1 image); scope excludes deferred v2 candidates.
|
||||
|
||||
### Out of Scope
|
||||
|
||||
See REQUIREMENTS.md §Out of Scope — federation, secret versioning, high availability, UI beyond the four pages + dashboard, audit rotation, multiple broker instances, approver agents, key rotation ceremony, HSM, rate limiting, admin MFA, notifications.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Single-file broker**: `coreci_broker.py` under 1400 lines (amended from 800 via owner decisions D-016..D-018; excluding tests, UI HTML, client module, reference tools) — spec §12.14 as amended
|
||||
- **Stdlib-only runtime**: no pip in the target dev environment; bcrypt is the sole optional import (dual-mode hashing); tests use unittest
|
||||
- **SQLite is the source of truth**: single instance, no HA; back it up
|
||||
- **Bind default** `127.0.0.1:8765`; the bind address controls reachability
|
||||
- **Mode 0600** on `broker.db` and `broker.audit.log`
|
||||
- **The broker never writes to `creds-coreci-coo`** — the CEO writes that KB directly through Open WebUI's UI
|
||||
- **Config contract** = spec §5.1 plus documented additions: `BROKER_ENV_PATH`, `BROKER_CREDS_SOURCE`, `BROKER_CREDS_FILE`, `BROKER_OWUI_BASE_URL`, `BROKER_OWUI_API_KEY`, `BROKER_AGENT_STATE_BACKEND`
|
||||
- **Token form**: `base64(sha256(pepper:UUID:user_id:agent_id:sorted_tags))` — 44-char ASCII, no prefix; `BROKER_TOKEN_TTL` default 86400s
|
||||
- **Open WebUI runtime**: bcrypt and httpx exist there; the client module must not depend on either (stdlib urllib) so any tool can import it
|
||||
|
||||
v0.2 (packaging/distribution) constraints:
|
||||
|
||||
- **No pip / stdlib-only build path**: the CLI artifact is a stdlib `zipapp` (`.pyz`); no wheel, no PyInstaller, no pip at build or runtime (D-009)
|
||||
- **No container runtime in the dev environment**: no docker/podman/buildah available — the once-per-milestone publish runs daemonless via a stdlib registry client (D-019)
|
||||
- **Self-contained artifact**: one `.pyz` bundles broker + UI assets + client module; asset roots resolved from env, never `__file__`
|
||||
- **Broker behavior frozen**: first-boot semantics, API surface, and all v0.1 acceptance criteria remain byte-identical when driven through the packaged CLI
|
||||
- **Release secrets discipline**: forge tokens resolved only from `.env*` files via the secrets-loader — never shell environment (v1.8 lesson)
|
||||
- **`broker.db`/`broker.audit.log` remain state-directory-local** — the CLI boots in a runtime state dir (docker: a volume), never writes inside its own archive
|
||||
|
||||
## Context
|
||||
|
||||
CoreCI estate: sibling projects researchx and nextcraft live on gitea (`git.coreci.dev/coreci/*`) and use this same ciagent pipeline. The broker runs on the Open WebUI host as one workspace tool (embedded HTTP server in a daemon thread; also runnable standalone). The `creds-coreci-coo` KB currently mediates four credentials (`coreci/proxmox`, `coreci/gitea`, `coreci/executor` TBD, plus the COO's own login — independent of the broker's `BROKER_ADMIN_PASS`). The existing `proxmox_readonly_tool.py` is updated post-broker to use the public client module; it lives outside this repo, so this repo ships the reference implementation of the tool-side bootstrap instead.
|
||||
|
||||
v0.2 context: milestone v0.1 shipped complete (all REQ-01..REQ-20, `v0.1.5` on main, six gitea releases each carrying a `broker-vX.Y.Z.tar.gz` source archive, zero escalations). Owner directives for v0.2 (2026-10-10): package the broker as a CLI; attach the CLI to every release alongside notes on gitea; build a docker container once per milestone and publish it as a package in the gitea registry. The v0.1 release-attachment mechanism (sanctioned `.env*` token path, per-release asset upload) is the proven base REQ-22 extends.
|
||||
|
||||
Known spec defects resolved by clarification (recorded for traceability): §7.5's in-memory `token_hash` index is impossible with bcrypt salted hashes; §6 claims the tool derives a token every call while §9's flow uses the broker-returned token from the agent KB (the broker derives authoritatively; the tool stores); §9's snippet raises `KeyError` on `data["pepper"]` for re-registrations; the pepper chicken-and-egg (a token is needed to read agent state, but the pepper is needed to derive a token) is fixed by returning the pepper when the caller presents no valid token.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
| ID | Decision | Rationale | Outcome |
|
||||
|----|----------|-----------|---------|
|
||||
| D-001 | Dual-mode bcrypt/scrypt hashing, format-prefixed hashes | Spec §6 requires bcrypt; dev env is stdlib-only; OWUI runtime ships bcrypt | To be validated in P1 |
|
||||
| D-002 | Creds-source adapter: file (default) \| OWUI API | Spec never specified the physical read mechanism; both environments must work | To be validated in P4 |
|
||||
| D-003 | Agent-state backend toggle, internal default | §9.1 endpoints are the only interface; internal storage is functionally identical without the admin-API dependency | To be validated in P3 |
|
||||
| D-004 | UUID-scoped slow-hash scan + in-memory verified-token cache | §7.5's bcrypt token_hash index is impossible; header UUID bounds the scan; SQLite stays authoritative | To be validated in P3 |
|
||||
| D-005 | Register re-derives; pepper returned on row-create or tokenless register | Broker holds pepper + all inputs; tokenless register is the only KB-loss recovery path | To be validated in P3 |
|
||||
| D-006 | Public dependency-free client module at /static/ + reference tools | Owner directive: agents bootstrap their own tools within the OWUI workspace | To be validated in P4 |
|
||||
| D-007 | Spec §13 defaults accepted | Owner confirmed: no rotation, single instance, 24h TTL, CEO sole approver | Binding for v0.1 |
|
||||
| D-008 | Milestone v0.1; "v0.5" remains the spec revision label | computeMilestoneTag greenfield convention (researchx, nextcraft precedent) | Binding for v0.1 |
|
||||
| D-009 | Stdlib-only runtime; unittest for tests | No pip in environment; dual-mode hashing is the sole optional import | Binding for v0.1 |
|
||||
| D-010 | Single-project mode; gitea release target coreci/broker | Sibling convention on the same forge; first push deferred to a ship phase | Binding for v0.1 |
|
||||
| D-011 | Empty root commit on main precedes the branch hierarchy | Branches cannot be created from an unborn HEAD; nextcraft precedent | Executed at init |
|
||||
| D-012 | Forge repo coreci/broker public, default branch main | Sibling convention (researchx/nextcraft both public); code ships no runtime secrets — pepper/admin pass generated at boot, .env.secrets gitignored | Binding for v0.1 |
|
||||
| D-013 | Session cookie `broker_session` uses HttpOnly + SameSite=Lax | Spec silent on cookie attributes; HttpOnly blocks JS exfiltration, Lax preserves bookmark-login UX | Binding for v0.1 |
|
||||
| D-014 | `BROKER_CREDS_FILE` defaults to `data/creds.json` | FileCredsSource dev/test default location; `data/` already gitignored | Binding for v0.1 |
|
||||
| D-015 | `GET /api/broker/agents/<agent_id>`: App-auth + identity match, metadata-only response (no token, no hash) | §7.4 auth/response unspecified; mirrors state-endpoint identity enforcement; token material never leaves the broker | Binding for v0.1 |
|
||||
| D-016 | AC14 line cap amended: coreci_broker.py < 1000 lines (was < 800) | Owner-approved §12.14 amendment at P2: 11 lines of headroom vs ~275 lines of remaining handler surface; preserves the readable single-file intent without dense-cramming | Binding for v0.1 |
|
||||
| D-017 | D-016 extended: AC14 cap < 1200 lines | Second §12.14 amendment at P3 (owner's D-016 pattern applied with new data): P3-01 lands at 990/1000 with ~150-180 lines of remaining broker surface (App-auth cache, agent state, creds adapter, secrets endpoint); mild compaction reclaims only ~50 — same readability-over-cramming tradeoff the owner chose at D-016; subject to review at the P5 owner approval gate | Binding for v0.1 |
|
||||
| D-018 | D-017 extended: AC14 cap < 1400 lines — final projected amendment | Third §12.14 amendment, pre-P4 per the P3 verify planning signal: 1178/1200 after P3 with 21 lines headroom vs P4's ~105-140 line need (creds adapter + secrets endpoint, the LAST broker-surface additions); P5 adds only excluded artifacts (tests, README). Same owner pattern (D-016/D-017), flagged for the P5 owner approval gate | Binding for v0.1 |
|
||||
| D-019 | Daemonless container publish: stdlib registry client (pull pinned base layers, append broker layer, push to gitea registry) | No docker/podman/buildah in the dev environment; gitea's package registry speaks OCI registry HTTP API v2 — fully automated once-per-milestone publish without a container runtime; stub-testable | Binding for v0.2 |
|
||||
| D-020 | Attach `coreci-broker-vX.Y.Z.pyz` to every release from `v0.2.1` onward | Consistent with v0.1's per-release `broker-vX.Y.Z.tar.gz` asset convention; every tagged release stays independently deployable | Binding for v0.2 |
|
||||
| D-021 | Image tags `git.coreci.dev/coreci/broker:<milestone-release-tag>` + `latest`, pushed once per milestone | Version tag pins the release; `latest` gives docker users a sane default; once-per-milestone honors the owner directive | Binding for v0.2 |
|
||||
| D-022 | Base image `python:3.11-slim` (pinned minor line) | Dev environment runs Python 3.11.2 — pinned minor matches tested stdlib behavior and stays reproducible | Binding for v0.2 |
|
||||
@@ -0,0 +1,113 @@
|
||||
# Requirements — CoreCI Broker
|
||||
|
||||
## Overview
|
||||
|
||||
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09). "v0.5" is the spec's revision label. REQ-01..REQ-20 (milestone v0.1) shipped complete as `v0.1.5`. **Milestone v0.2 (cli-packaging)** adds REQ-21..REQ-23 from owner directives (2026-10-10): package the broker as a CLI, attach it to every gitea release with notes, and publish a docker container once per milestone as a gitea package. Scope is CLI + distribution only — no functional changes to the broker. Clarification decisions D-001..D-015 (v0.1) remain binding; D-016+ resolve v0.2's packaging/distribution ambiguities.
|
||||
|
||||
## v1 (Milestone v0.1 — complete, shipped as v0.1.5)
|
||||
|
||||
### Auth & Identity
|
||||
|
||||
- **REQ-01 Admin authentication.** `BROKER_ADMIN_USER`/`BROKER_ADMIN_PASS` from the broker `.env`. Session cookie `broker_session` for the UI; HTTP Basic accepted on all admin API endpoints. `POST /api/broker/auth/login` creates a session (SQLite `admin_sessions`, TTL from `BROKER_SESSION_TTL`, default 300s), `logout` invalidates it, `whoami` returns the session info or 401. Admin sessions expire lazily and are cleaned on boot.
|
||||
- **REQ-02 Tool self-registration.** `POST /api/broker/agents/register` with `{agent_id, app_registration_uuid, user_id, tags}`. The broker rejects unknown or revoked UUIDs and user_ids with 403 (allowlists are the gate); malformed input → 400. Re-registration is idempotent: the broker re-derives from the request inputs and overwrites the stored hash when inputs changed; an unexpired row re-issues the same derived token. The response returns the token, `token_expires_at`, and the pepper when the row is created or the caller presented no valid token (KB-loss recovery). Register actions are audited.
|
||||
- **REQ-03 Derived token.** `token = base64(sha256(pepper:APP_UUID:user_id:AGENT_ID:','.join(sorted(tags))))` — a 44-char ASCII string, no prefix, no challenge protocol. The broker derives the same token a compliant tool would derive. Stored as a slow hash in `agent_registrations.token_hash`: bcrypt when importable, else stdlib scrypt; hashes are format-prefixed (`$2b$…` / `scrypt$…`) so verification is format-aware.
|
||||
- **REQ-04 Token verification & TTL.** Tool endpoints require `Authorization: App <UUID> <token>`. Verification: look up rows scoped to the presented UUID, check the slow hash (in-memory verified-token cache after first success; cache is never authoritative — rows re-checked for status/expiry). Missing/malformed/failed → 401. Tokens live `BROKER_TOKEN_TTL` seconds (default 86400); expiry → 401 → the tool re-registers. Revoked apps keep working until TTL expiry by design.
|
||||
|
||||
### Allowlist Administration
|
||||
|
||||
- **REQ-05 App registrations.** Admin-only `GET/POST /api/broker/app_registrations`, `DELETE /api/broker/app_registrations/<uuid>` (marks revoked; new registrations rejected; existing tokens honored until TTL), `POST .../reinstate`. UUID format validated (400 on invalid, 409 on duplicate). All actions audited as `add_app`/`remove_app`.
|
||||
- **REQ-06 User allowlist.** Admin-only `GET/POST /api/broker/user_allowlist`, `DELETE .../<user_id>` (revoked semantics), `POST .../reinstate`. Optional `email` field. 409 on duplicate. Audited as `add_user`/`remove_user`.
|
||||
|
||||
### Secret Mediation
|
||||
|
||||
- **REQ-07 Path-scoped secret reads.** `GET /api/broker/secrets/<path>`. Scope is the path: a registration may read exactly `coreci/<agent_id>` — cross-path → 403, missing secret → 404, auth failure → 401. Every attempt (ok/denied/error) is audited with `user_id`, `app_registration_uuid`, `agent_id`, `action: read`, `path`, `result`.
|
||||
- **REQ-08 Creds-source adapter.** The broker reads `creds-coreci-coo` exclusively through a pluggable adapter, `BROKER_CREDS_SOURCE=file|owui`: `FileCredsSource` reads a JSON document at `BROKER_CREDS_FILE` mapping paths to secret objects (default — dev/tests); `OwuiApiCredsSource` reads via the Open WebUI HTTP API using `BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY` (production). The broker never writes to the KB. Response shape: `{path, data, version, updated_at}`.
|
||||
|
||||
### Agent State
|
||||
|
||||
- **REQ-09 State endpoints.** `GET/PUT /api/broker/agents/<agent_id>/state` (App-auth). The broker enforces that the calling UUID and the token-derived user_id match the stored identity; 404 when no state exists. State schema per spec §8 (agent_id, app_registration_uuid, user_id, tags, pepper, current_token, token_expires_at, first_registered_at, last_seen_at). This is the tool's only persistence channel — the broker is the durability layer.
|
||||
- **REQ-10 State backend toggle.** `BROKER_AGENT_STATE_BACKEND=internal` (default: rows in `broker.db` behind the state endpoints) `| owui` (real per-agent Open WebUI KBs via the OWUI API). The tool cannot tell the difference — the state endpoints are the only interface.
|
||||
|
||||
### Audit
|
||||
|
||||
- **REQ-11 Append-only audit log.** One JSON object per line at `BROKER_AUDIT_LOG_PATH` (mode 0600, append-only, no rotation in v0.1). Fields: `ts`, `user_id`, `app_registration_uuid`, `agent_id`, `action` (register/read/denied/login/logout/add_app/remove_app/add_user/remove_user), `path`, `result` (ok/denied/error); admin actions carry null user_id/uuid.
|
||||
- **REQ-12 Audit query.** Admin-only `GET /api/broker/audit` with `since`, `until`, `user_id`, `app_registration_uuid`, `action` filters and `limit` (default 100, max 1000); returns entries plus `next_since` cursor. The broker streams the file line-by-line with filtering.
|
||||
|
||||
### Web UI
|
||||
|
||||
- **REQ-13 Four pages + dashboard.** Server-rendered HTML from `coreci_broker_ui/`, no build step, same port as the API. `/ui/login` (the CEO's bookmark), `/ui/app_registrations` (table + add-modal + revoke/reinstate per row), `/ui/user_allowlist` (same), `/ui/audit` (filterable, read-only, no export), plus `/ui/dashboard` and a persistent nav header with Logout. After each action the page refreshes via the corresponding API.
|
||||
|
||||
### Ops & Bootstrap
|
||||
|
||||
- **REQ-14 `.env` config + first boot.** The broker reads its `.env` at `BROKER_ENV_PATH` on boot (spec §5.1 contract plus the documented additions). On first boot it generates `BROKER_INSTANCE_PEPPER` (32 random bytes, base64 — synced to SQLite) and `BROKER_ADMIN_PASS` (32+ random chars), writes them to `.env`, and prints the admin pass to the startup log exactly once — never again.
|
||||
- **REQ-15 Mode 0600.** `broker.db` and `broker.audit.log` are created mode 0600, owned by the Open WebUI process user. `.env.secrets` discipline (ciagent) mirrors this.
|
||||
- **REQ-16 Persistence across restarts.** SQLite and the audit file are the durable state. Restarts preserve registrations, allowlist entries, sessions' data, and audit history; in-memory caches (verified-token cache) rebuild lazily; existing tokens still verify.
|
||||
- **REQ-17 Single-file constraint.** `coreci_broker.py` is one Python file under **1400** lines (amended 800→1000 D-016, →1200 D-017, →1400 D-018 — owner-approved §12.14 amendment pattern, final projected), excluding tests, UI HTML, the client module, and reference tools. A line-count test enforces it (AC14).
|
||||
- **REQ-18 Runtime modes.** Importable as an Open WebUI workspace tool (daemon-thread server + a `Tools` class surface) and runnable standalone (`python3 coreci_broker.py`). Stdlib-only at runtime; bcrypt is the sole optional import (D-001/D-009). Tests use unittest against an in-process server on an ephemeral port.
|
||||
|
||||
### Client Ecosystem
|
||||
|
||||
- **REQ-19 Public client module.** `coreci_broker_client.py` — dependency-free (stdlib urllib; no httpx, no bcrypt) — implements the tool-side bootstrap of spec §9: ensure-registered (idempotent), state get/put, secret read with 401 → re-register → retry-once, and never assumes the pepper key is present in register responses (`data.get("pepper")`). Served unauthenticated at `/static/coreci_broker_client.py` so any OWUI tool author can fetch it into their workspace and import it. It contains no secrets.
|
||||
- **REQ-20 Reference tools.** `reference_tools/` ships at least: (a) a proxmox-style reference implementing §11's flow — first call registers and persists state, subsequent calls read the token from state and fetch `coreci/proxmox`, downstream call defaults to a safe dry-run; (b) a minimal showcase of auth-first-then-use-a-second-tool. These are templates for tool authors, wired to the test broker in tests.
|
||||
|
||||
## v2 (Milestone v0.2)
|
||||
|
||||
### Packaging
|
||||
|
||||
- **REQ-21 CLI packaging (zipapp).** The broker ships as a single-file executable CLI `coreci-broker.pyz` built with stdlib `python3 -m zipapp` from a clean checkout — no pip, no third-party build tooling (D-009 preserved). The archive is self-contained: it bundles `coreci_broker.py`, the `coreci_broker_ui/` pages, and `coreci_broker_client.py`, materializing the UI/static roots into the runtime state directory on boot (paths resolved from env, never `__file__`), so one file serves the full admin UI at `/ui/` and the client module at `/static/coreci_broker_client.py`. Running it boots the standalone server with first-boot semantics identical to REQ-14/REQ-18 (`.env` generation, admin pass printed exactly once, `broker.db` + `broker.audit.log` at mode 0600); `--version` reports the release version. A build script regenerates the artifact reproducibly, and tests boot the built artifact end-to-end.
|
||||
|
||||
### Distribution
|
||||
|
||||
- **REQ-22 Release assets.** Every gitea release from the phase that first builds the CLI (`v0.2.1`, D-020) onward attaches the built artifact `coreci-broker-vX.Y.Z.pyz` alongside the existing source archive. Uploads resolve the forge token only via the sanctioned secrets path (`.env*` files — never shell environment). Release notes document the CLI: artifact download + install (`chmod +x` / `python3`), the CLI surface (boot, `--version`, env contract), and docker usage per REQ-23. Attachment failure follows the ship workflow's bounded-retry then `release_pending` escalation — it never blocks tag+merge.
|
||||
- **REQ-23 Docker package.** The repo ships a `Dockerfile` and a **daemonless stdlib registry client** (D-019): once per milestone, at the final ship, the client pulls the pinned `python:3.11-slim` base image manifest + layers from the public registry (HTTP API v2), appends a layer carrying the CLI artifact, and pushes the image `git.coreci.dev/coreci/broker:<milestone-release-tag>` plus the `latest` alias (D-021/D-022) to the gitea container/package registry — no container runtime required. The container runs the CLI standalone with state on a volume. Registry auth, manifest, layer, and digest handling are stub-tested; publish failure follows bounded-retry escalation — it never blocks the milestone merge.
|
||||
|
||||
## Deferred Beyond v0.2
|
||||
|
||||
None defined. Future candidates held as out-of-scope decisions, not requirements: audit rotation with retention, multiple broker instances, approver-agent delegation, OWUI-KB credential sync.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
| Feature | Reason |
|
||||
|---------|--------|
|
||||
| Multi-instance federation | Spec §3 — one broker on the Open WebUI host |
|
||||
| Secret versioning / history | Spec §3 — PUT overwrites; broker is read-only on the KB |
|
||||
| High availability | Spec §3 — SQLite is the source of truth; back it up |
|
||||
| Web UI beyond the four pages + dashboard | Spec §3, §10 |
|
||||
| Broker writes to `creds-coreci-coo` | Spec §3 — the CEO writes the KB via Open WebUI's UI |
|
||||
| Audit log rotation | Spec §13 Q1 — CEO default: never rotate; manual `mv` when large |
|
||||
| Multiple broker instances | Spec §13 Q2 — confirmed out of scope for v0.5 |
|
||||
| Approver agents (delegated allowlist adds) | Spec §13 Q4 — CEO is sole approver |
|
||||
| Key rotation ceremony | Spec §14 — pepper and tool UUIDs are stable |
|
||||
| HSM integration for the pepper | Spec §14 |
|
||||
| Rate limiting on auth endpoints | Spec §14 — assumes trusted network |
|
||||
| Multi-factor admin auth | Spec §14 — single password sufficient |
|
||||
| Email/notification on allowlist changes | Spec §14 |
|
||||
| bcrypt as a hard dependency; pip-installable packages | D-001/D-009 — environment has no pip; stdlib-only runtime |
|
||||
|
||||
## Traceability
|
||||
|
||||
| Requirement | Phase | Status |
|
||||
|-------------|-------|--------|
|
||||
| REQ-01 | 2 | complete |
|
||||
| REQ-02 | 3 | complete |
|
||||
| REQ-03 | 1 | complete |
|
||||
| REQ-04 | 3 | complete |
|
||||
| REQ-05 | 2 | complete |
|
||||
| REQ-06 | 2 | complete |
|
||||
| REQ-07 | 4 | complete |
|
||||
| REQ-08 | 4 | complete |
|
||||
| REQ-09 | 3 | complete |
|
||||
| REQ-10 | 3 | complete |
|
||||
| REQ-11 | 1 | complete |
|
||||
| REQ-12 | 2 | complete |
|
||||
| REQ-13 | 2 | complete |
|
||||
| REQ-14 | 1 | complete |
|
||||
| REQ-15 | 1 | complete |
|
||||
| REQ-16 | 1 | complete |
|
||||
| REQ-17 | 1 | complete |
|
||||
| REQ-18 | 1 | complete |
|
||||
| REQ-19 | 4 | complete |
|
||||
| REQ-20 | 4 | complete |
|
||||
| REQ-21 | 1 | pending |
|
||||
| REQ-22 | 1 | pending |
|
||||
| REQ-23 | 2 | pending |
|
||||
@@ -0,0 +1,191 @@
|
||||
# RESEARCH — CoreCI Broker
|
||||
|
||||
Date: 2026-10-10 · Milestone: v0.2 (cli-packaging) · Phase: 0 (pre-execution) · Status: complete
|
||||
Sources: `.ciagent/` PROJECT/REQUIREMENTS/ARCHITECTURE/ROADMAP/PLAN, `coreci_broker.py` (1390 lines, read in full), git history/tags v0.1.0..v0.1.5, live gitea 1.26.1 API (`git.coreci.dev` swagger.v1.json + releases — read-only, token never printed), live Docker Hub registry + auth.docker.io (anonymous pulls, verified end-to-end), live gitea container-registry auth probes (read-only, no upload sessions created), stdlib source (`/usr/lib/python3.11/zipapp.py`, `urllib/request.py`), and an **end-to-end zipapp prototype booted from the real broker sources** (built + booted as subprocess + driven with urllib; `/tmp` scratch, repo untouched).
|
||||
|
||||
## 0. Carried-Forward v0.1 Findings (still binding)
|
||||
|
||||
- **Environment (v0.1 §1, re-confirmed)**: Python 3.11.2, NO pip anywhere, NO docker/podman/buildah runtime; stdlib-only runtime + build path (D-009 preserved by REQ-21/D-019); tests = stdlib unittest; forge gitea `git.coreci.dev`, owner `coreci`, repo `broker`, public, default branch `main`; `GITEA_TOKEN` resolves from `.ciagent/.env.secrets` only (mode 0600, never printed, `config.security.bash_allowlist.blocked_env_vars` includes GITEA_TOKEN → shell-env tokens are structurally blocked).
|
||||
- **Test harness (v0.1 §6)**: in-process `ThreadingHTTPServer(("127.0.0.1", 0), Handler)` + daemon thread + `tempfile.TemporaryDirectory()` state + stdlib `urllib.request` client + `addCleanup`; 0600 assertions via `st_mode & 0o777 == 0o600`; stub-HTTP precedent: `_StubOwuiHandler` in `tests/test_creds_source.py` (BaseHTTPRequestHandler with class-level fixtures + hit recording) — this is the template for v0.2 stub registry/gitea servers.
|
||||
- **Ship contract (`ci/workflows/ship.md` + config.json)**: every phase ships; release creation best-effort with bounded retries (`ship.max_release_retries: 3`); on exhaustion commit escalation `resolution: "auto"`, `type: "release_pending"`, phase still complete — tag+merge is the gate, `release_blocking: false`; **all secrets resolved ONLY from `.env`/`.env.secrets`/`.env.*` files via the secrets-loader, never shell env** (v1.8 root cause: empty token → gitea answers 404, not 401, masking auth failure).
|
||||
- **OWUI integration surface (v0.1 §4)**: unchanged by v0.2 — the CLI does not touch the `Tools` mount; path resolution from env (never `__file__`) is the v0.1 property that makes zipapp packaging possible at all.
|
||||
- **Static-serving rules that the packaged CLI must preserve**: `_resolve_static_file` rejects `\x00`, `\`, leading `/`, `..` segments, and any *request* path part starting with `.`; roots are real-pathed (`os.path.realpath`) so a dot-named *root* directory is fine (materialized assets live under a dot-dir; only request paths are dot-filtered). `_broker_protected_paths` protects `.env`/db/audit/creds (+ `-journal/-wal/-shm` suffixes) from static reads.
|
||||
|
||||
## 1. zipapp CLI Packaging (REQ-21)
|
||||
|
||||
### 1.1 stdlib mechanics (all VERIFIED empirically on Python 3.11.2)
|
||||
|
||||
- Command: `python3 -m zipapp <srcdir> -o coreci-broker.pyz --python "/usr/bin/env python3" --compress`.
|
||||
- **`__main__.py` is mandatory** — `zipapp` refuses with `ZipAppError("Archive has no entry point")` without it (the `-m module:fn` form is the only alternative; the shipped `cli/__main__.py` source is cleaner and testable). Files staged: `coreci_broker.py`, `coreci_broker_client.py`, `coreci_broker_ui/**`, `cli/__main__.py`, generated `_cli_version.py`.
|
||||
- **Compression: `zipapp` defaults to ZIP_STORED** (stdlib source line 136: `zipfile.ZIP_DEFLATED if compressed else zipfile.ZIP_STORED`). `--compress` is required for DEFLATED. Measured: ~95 KB content → 27.6 KB pyz compressed vs ~95 KB stored. Use `--compress`.
|
||||
- **Shebang + exec bit**: with `--python`, zipapp writes `#!/usr/bin/env python3\n` as a literal first line, then the zip (zip readers locate EOCD from the file end — leading bytes tolerated), and `chmod +x`es the target (`target.chmod(... | S_IEXEC)`). Both `./coreci-broker.pyz` and `python3 coreci-broker.pyz` verified working; `sys.argv[0]` = the archive path as invoked; `python3 -m zipapp --info` reads the interpreter back.
|
||||
- **Data files are NOT `open()`-able inside the archive** (verified `FileNotFoundError` for `assets/page.html`): zipimport only serves `.py` imports; the broker serves UI/static via **filesystem paths + `open()`** → asset materialization to disk at boot is *mandatory*, not optional. `importlib.resources` cannot provide a directory for `BROKER_UI_ROOT` — zipfile extraction is the only design that fits the broker's config contract. (Not a close call.)
|
||||
- **`__main__` module collision — none**: the archive's `__main__.py` IS the `__main__` module; `import coreci_broker` inside it executes the broker's module-level code once and its `if __name__ == "__main__"` block does NOT fire (verified: side-effect counter runs once, standalone print block silent). Call `coreci_broker.run_standalone()` — no re-execution of module code, no subprocess re-invocation needed.
|
||||
- **zipimport limits**: none bite here — the archive contains only pure-Python repo files; stdlib (`sqlite3`, `http.server`, C extensions) resolves from the host interpreter, and the broker's sole optional import (bcrypt) stays optional.
|
||||
|
||||
### 1.2 Boot shim design (prototype built, booted, verified — recommended as-is)
|
||||
|
||||
`cli/__main__.py` (staged into the pyz), five steps:
|
||||
|
||||
1. **`--version` first, before importing the broker**: parse `sys.argv[1:]`; if `--version`, `import _cli_version; print(_cli_version.VERSION); sys.exit(0)`. Version comes from a build-time-generated `_cli_version.py` (`VERSION = "vX.Y.Z"` written by `scripts/build_cli.py` from the tag) — chosen over a zip-comment (opaque, fragile, still needs a parser). No broker import on the `--version` path.
|
||||
2. **Resolve the state dir exactly like the broker**: `env_path = os.environ.get("BROKER_ENV_PATH") or ".env"`; `state_dir = dirname(abspath(env_path)) or cwd`. Broker path resolution (read from code, authoritative): `BROKER_ENV_PATH` default `.env`, `BROKER_DB_PATH` `broker.db`, `BROKER_AUDIT_LOG_PATH` `broker.audit.log`, `BROKER_CREDS_FILE` `data/creds.json`, `BROKER_UI_ROOT` `coreci_broker_ui`, `BROKER_STATIC_ROOT` `.` — **all CWD-relative when relative**; process env beats `.env` beats defaults. The CLI therefore needs no path changes at all: relative paths land in the CWD (docker: `WORKDIR /state` + volume).
|
||||
3. **Materialize assets**: `zipfile.ZipFile(sys.path[0])` (inside a zipapp, `sys.path[0]` is the `.pyz` itself — verified) → extract `coreci_broker_ui/**` → `<state>/.coreci_broker_assets/<VERSION>/ui/` and `coreci_broker_client.py` → `<...>/static/`. Write temp + `os.replace` (atomic vs a concurrently-serving older instance). **Versioned asset dir**: upgrades materialize a new dir; old dirs are left in place (another instance may still serve them; ~30 KB each — cleanup explicitly NOT done in v0.2). Overwrite-same-version on every boot (self-repair, assets are immutable inputs; extraction happens before this instance's server starts).
|
||||
4. **Override precedence — user wins**: `import coreci_broker`; parse the user `.env` with the broker's own `coreci_broker._parse_env_file(env_path)`; set `os.environ["BROKER_UI_ROOT"]` / `["BROKER_STATIC_ROOT"]` only when the key is in **neither** `os.environ` nor the `.env` values. Net precedence: process env → `.env` → shim defaults → broker defaults — identical to the broker's own contract, achieved with **zero broker changes**.
|
||||
5. **Hand off**: `coreci_broker.run_standalone()` (foreground loop + KeyboardInterrupt shutdown; prints `coreci-broker listening on http://<host>:<port>` with the *actual bound address* — `BROKER_BIND=127.0.0.1:0` gives an ephemeral port via `_parse_bind` (`"0".isdigit()` → True) and the harness parses the printed URL).
|
||||
|
||||
**Verified end-to-end prototype results** (real broker sources, subprocess, urllib): first boot printed `BROKER_ADMIN_PASS=…` exactly once then the listening line; `/healthz` 200; `/ui/login` + `/ui/dashboard` 200 (served from materialized assets); `/static/coreci_broker_client.py` 200 byte-exact (6180 B); `.env` + `broker.db` + `broker.audit.log` created mode 0600 in the state dir; second boot silent (no pass re-print); state dir fully independent of CWD (env paths pin it).
|
||||
|
||||
### 1.3 Determinism / reproducibility
|
||||
|
||||
- **Raw `zipapp` is NOT reliably byte-reproducible**: zip entries carry the source files' mtimes (`z.write` → `ZipInfo.from_file`), no SOURCE_DATE_EPOCH handling exists anywhere in zipapp, and entry order is `rglob('*')` = `os.scandir` order (unsorted, filesystem-dependent). My two identical staging-dir builds hashed equal *only because* mtimes and scandir order happened to match — a fresh `git clone` (checkout-time mtimes) or a different filesystem breaks it.
|
||||
- **Fix — normalization post-pass (verified byte-identical across rebuilds)**: after zipapp builds the artifact, the build script reads it (shebang line + zip body), rewrites the zip with entries **sorted by name**, `date_time=(1980,1,1,0,0,0)` (zip epoch minimum), `ZIP_DEFLATED`, preserved `external_attr`, and re-prepends the shebang (+`chmod +x`). Verified: two builds → identical sha256; normalized pyz still executes (`--version` OK). Alternative (rejected): `os.utime`-ing the staging tree pre-build — fixes mtimes but not entry order; the post-pass fixes both and keeps `python3 -m zipapp` as the format mechanism (the normalized file is still "shebang + zip", the zipapp format).
|
||||
- Reproducibility test: `test_build_cli.py` builds twice into separate dirs, asserts identical sha256, and asserts the embedded `_cli_version.VERSION`.
|
||||
|
||||
### 1.4 Defects / risks (zipapp)
|
||||
|
||||
- **RISK** — stale-asset drift if a *user* sets `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` in `.env`: honored (by design, documented in notes) — the archive still carries the assets, they're just unused. Tests cover the override path.
|
||||
- **RISK** — first-boot extraction torn reads if two instances share a state dir and the same version: mitigated by temp+`os.replace`; multi-instance sharing one SQLite remains unsupported (as v0.1).
|
||||
- **NOTE** — shebang `env python3` may resolve to a non-3.11 python on exotic hosts; the broker needs 3.7+ features in practice but is tested on 3.11 only — release notes state "Python 3.11+ required" (D-022 alignment).
|
||||
- **NOTE** — `sys.path[0]` is the archive only when run as a zipapp; the shim is not meant to run from an unzipped checkout (guard: it can `os.path.isdir` check and fall back to repo-relative roots, or simply document zipapp-only — planner's choice, zero-cost either way).
|
||||
- **NOTE** — assets land under a dot-directory (`.coreci_broker_assets`) — hidden-dir hygiene in the state dir; broker dot-filtering applies to request paths, not roots (verified `_resolve_static_file`).
|
||||
- **NOTE** — AC14 (line count) counts `coreci_broker.py` ONLY (REQ-17); the shim/build scripts are separate files → v0.2 needs **zero broker lines**; broker is 1390/1400 — only 10 lines of headroom anyway, so zero-change is also a hard constraint, and it is satisfied (verified: `run_standalone`, `_parse_env_file`, `boot` are importable and sufficient as-is).
|
||||
|
||||
## 2. Gitea Release Assets (REQ-22)
|
||||
|
||||
### 2.1 Verified API shapes (live gitea 1.26.1, `git.coreci.dev/swagger.v1.json` + working calls)
|
||||
|
||||
- **List**: `GET /api/v1/repos/coreci/broker/releases?limit=20` → `[{id, tag_name, name, body, assets: [{id, name, size, download_count, browser_download_url}], draft, prerelease, created_at}]`. **By tag**: `GET /api/v1/repos/coreci/broker/releases/tags/{tag}` (200 or 404). **Latest**: `GET .../releases/latest`.
|
||||
- **Create**: `POST /api/v1/repos/{owner}/{repo}/releases` — JSON `CreateReleaseOption` `{tag_name, name, body, draft, prerelease, target_commitish, tag_message}` (schema verified from swagger).
|
||||
- **Attach asset**: `POST /api/v1/repos/{owner}/{repo}/releases/{id}/assets` — **consumes `multipart/form-data`** (or `application/octet-stream`); params verified from swagger: `id` (path, int64), `name` (query), **`attachment` (formData, type file)** — the multipart field name is `attachment`. Response: attachment JSON with `id`, `name`, `size`, `browser_download_url`.
|
||||
- **Auth**: `Authorization: token <GITEA_TOKEN>` (verified working on live forge; gitea also accepts Basic `user:token` — use the `token` form).
|
||||
- **Asset size**: gitea `[attachment] MAX_SIZE` default is 4 MB — the pyz is ~30 KB (deflated) → no issue; NOTE if server policy changes, attachment failure escalates through the bounded-retry path rather than blocking.
|
||||
- **Multipart with stdlib** (no `requests`): hand-rolled body, ~15 lines — boundary string, `Content-Disposition: form-data; name="attachment"; filename="coreci-broker-vX.Y.Z.pyz"`, `Content-Type: application/octet-stream`, `?name=` query param; gitea parses with standard Go multipart — boundary is ours to choose. Test against a stub gitea (§2.4).
|
||||
|
||||
### 2.2 v0.1 attachment forensics (how it was actually done)
|
||||
|
||||
- All six releases v0.1.0..v0.1.5 carry `broker-vX.Y.Z.tar.gz` (29 KB → 99 KB, growing with the codebase). Downloaded v0.1.5's asset: **57 members = the complete working tree including `.ciagent/`** (53 git-tracked files + 4 dir entries; matches `git ls-tree -r v0.1.5` exactly, plus directory entries), **no top-level prefix**, uniform `mtime 1791646214` (= the release timestamp), `uid 0/uname root`, modes 0664/0775.
|
||||
- Interpretation: the archive was produced by the **ship harness's packaging infrastructure** (ship.md step "Create distribution packages" via the `coreci build/package/release upload` CLI — **not installed in this dev environment**; the root-owned uniform timestamps indicate a container-side packaging step) — i.e. **ad-hoc from the repo's perspective: no repo-owned script exists**. The only repo-visible traces are ship.md's generic instructions.
|
||||
- Conclusion for v0.2: REQ-22 formalizes exactly this — a repo-owned `scripts/release.py` (plus `scripts/build_cli.py`) replaces the absent harness path. The v0.1 *conventions* carry over: per-release asset naming (now `coreci-broker-vX.Y.Z.pyz`, D-020), notes model (§2.3), and the bounded-retry contract (ship.md). The existing source-archive `broker-vX.Y.Z.tar.gz` convention continues to be the harness's own product (best-effort); the repo script owns the `.pyz` (+ digest).
|
||||
|
||||
### 2.3 Release notes content model (v0.1.5 body verified; v0.2 sections)
|
||||
|
||||
v0.1.5 body sections: `Milestone summary` → `Phases` (P0..P5 with tags) → `Verification` (test counts, security review, audit) → `Deployment`. v0.2 releases add the REQ-22-mandated CLI + docker documentation; recommended body structure (build-able by `scripts/release.py` from PLAN/ROADMAP/requirements data):
|
||||
|
||||
1. `Milestone summary` — one paragraph (carries the v0.1 style).
|
||||
2. `Phases` — phase list with tags (auto from `git log vX.Y.Z-1..vX.Y.Z`/ROADMAP).
|
||||
3. `Verification` — test count, security review verdict, escalations.
|
||||
4. `CLI install + usage` — download `coreci-broker-vX.Y.Z.pyz` from this release's assets; `chmod +x` then `./coreci-broker-vX.Y.Z.pyz` or `python3 coreci-broker-vX.Y.Z.pyz`; `--version`; env contract table pointer (README); first-boot pass discipline (printed once, 0600 state files); state-dir anchoring (`BROKER_ENV_PATH`); **sha256 of the .pyz embedded** (plus optional `.sha256` sidecar asset — planner's call; lean default: digest in notes only).
|
||||
5. `Docker usage` (REQ-23/D-021) — `docker pull git.coreci.dev/coreci/broker:vX.Y.Z` (and `latest`); `docker run -p 8765:8765 -v broker-state:/state …`; state-on-volume contract.
|
||||
|
||||
### 2.4 `scripts/release.py` design (stdlib, ~200 lines)
|
||||
|
||||
- **Secrets**: token via a shared file-reading loader `scripts/_env_secrets.py` (~20 lines: parse `.ciagent/.env.secrets` `KEY=VALUE`, strip quotes — mirror the broker's `_parse_env_file` semantics). NEVER `os.environ` (GITEA_TOKEN is in `blocked_env_vars`). **Assert token non-empty before any API call** (v1.8 lesson: empty token → gitea 404, masking auth failure).
|
||||
- Flow: (1) resolve tag → `GET /releases/tags/{tag}`; 404 → `POST /releases` (create with notes body, `draft: false`, `prerelease: false`); (2) `GET .../releases/{id}/assets` — if `coreci-broker-vX.Y.Z.pyz` already present with matching size → **idempotent success** (no duplicate asset on bounded-retry re-run); (3) POST multipart attach (field `attachment`, `?name=`); (4) bounded retry: 3 attempts with backoff on URLError/5xx/HTTPError; on exhaustion → exit non-zero with a machine-readable marker → ship harness commits `release_pending` escalation; **never blocks tag+merge**.
|
||||
- **Stub-test** (`tests/test_release_upload.py`): `_StubGiteaHandler(BaseHTTPRequestHandler)` on `ThreadingHTTPServer(("127.0.0.1", 0))` (v0.1 `_StubOwuiHandler` precedent): routes `GET /api/v1/repos/{o}/{r}/releases/tags/{tag}` (200 fixture / 404), `POST .../releases` (assert body JSON), `POST .../releases/{id}/assets` (parse multipart with a stdlib parser — e.g. `email` or manual boundary split — assert field name `attachment` + `?name=` + body bytes), plus failure-modes: 500×3 → release_pending path; existing asset → no second upload.
|
||||
|
||||
### 2.5 Defects / risks (release assets)
|
||||
|
||||
- **RISK** — multipart hand-rolling bugs (boundary quoting, CRLF): mitigated by stub-gitea test with byte-exact assertions.
|
||||
- **RISK** — release may not exist when the attacher runs (ship harness creates it best-effort): create-or-get flow covers both orders; idempotent re-attach covers partial failure.
|
||||
- **NOTE** — 4 MB attachment cap (default) — pyz far under; failure would surface through bounded-retry, not silently.
|
||||
- **NOTE** — `GET /releases?limit=50` paging: prefer `/releases/tags/{tag}` (exact, no paging).
|
||||
|
||||
## 3. Daemonless OCI Build + Push (REQ-23, D-019/D-021/D-022)
|
||||
|
||||
### 3.1 PULL from Docker Hub (verified live, anonymous, end-to-end)
|
||||
|
||||
- `GET https://registry-1.docker.io/v2/` → 401 `WWW-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"`; anonymous rate limit header `ratelimit-limit: 100;w=3600` (100 req/h) — ample for once-per-milestone.
|
||||
- `GET https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/python:pull` → 200 `{token, access_token, expires_in, issued_at}` — anonymous OK (verified).
|
||||
- `GET /v2/library/python/manifests/3.11-slim` with `Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json` → **200 `Content-Type: application/vnd.oci.image.index.v1+json` — `python:3.11-slim` IS an OCI image index (16 children: 8 platform manifests + 8 attestation manifests `unknown/unknown`)**. The client MUST resolve the `linux/amd64` child manifest by digest and fetch it separately.
|
||||
- Verified nuance: Docker Hub currently returns the index **even without Accept** (and even Accepting only schema2) — but the client still sends the full Accept set (contract correctness) and branches on the response `"manifests"` key (index) vs `"layers"` (manifest), not on hope.
|
||||
- amd64 child (verified, digest drifts upstream — resolve at publish time and record it): `application/vnd.oci.image.manifest.v1+json`, config `application/vnd.oci.image.config.v1+json` (`Cmd: ["python3"]`, Env PATH/LANG/GPG_KEY/`PYTHON_VERSION=3.11.17`/PYTHON_SHA256 — base runs 3.11.17 vs dev 3.11.2, same pinned minor line per D-022), **4 layers** `application/vnd.oci.image.layer.v1.tar+gzip` (≈29.8 MB + 1.3 MB + 14.5 MB + 250 B ≈ 45.7 MB total).
|
||||
- `GET /v2/library/python/blobs/<digest>` → blob bytes (layers + config). All flows urllib-only.
|
||||
|
||||
### 3.2 Auth to the gitea registry (verified live; username+token shapes pinned)
|
||||
|
||||
- `GET https://git.coreci.dev/v2/` → 401 `WWW-Authenticate: Bearer realm="https://git.coreci.dev/v2/token",service="container_registry",scope="*"`.
|
||||
- Token endpoint: `GET https://git.coreci.dev/v2/token?service=container_registry&scope=repository:coreci/broker:pull,push` with **`Authorization: Basic base64(<username>:<GITEA_TOKEN>)`** → 200 `{"token": "<JWT>"}` (HS256; `exp-nbf = 86400` s = 24 h).
|
||||
- **Verified**: the Basic **username is ignored** — `coreci`, `nosuchuser123`, and empty all return the same JWT (UserID 57, scopes `write:activitypub,…,write:package,…` — the existing GITEA_TOKEN has package-write). The token (as password) identifies the user. Docker login convention: username = the gitea user, password = the token; the client sends any non-empty username.
|
||||
- **Verified quirk (must handle)**: the token endpoint answers **200 even anonymously** with an unusable JWT (`UserID: -1`, empty scope). A 200 from `/v2/token` proves nothing about push rights — the client must (a) insist on Basic credentials being present, (b) check the issued token is non-empty, and let real operations fail loudly.
|
||||
- Verified: `Authorization: Bearer <JWT>` on `/v2/` → 200; `HEAD /v2/coreci/broker/blobs/sha256:0…0` → 404 (existence checks work); `GET /v2/coreci/broker/tags/list` → 404 `NAME_UNKNOWN` (the container package doesn't exist until first push — expected today).
|
||||
|
||||
### 3.3 PUSH flow + media-type choice
|
||||
|
||||
- **Manifest media type: OCI image manifest end-to-end** — `application/vnd.oci.image.manifest.v1+json`, config `application/vnd.oci.image.config.v1+json`, layers `application/vnd.oci.image.layer.v1.tar+gzip`. Rationale: (1) the pulled base child manifest already uses exactly these descriptor types — reusing its 4 layer digests **verbatim** keeps descriptor/mediaType/blob consistency; (2) gitea 1.26 accepts docker schema2 and OCI manifests — OCI is the modern default and what upstream itself serves; (3) Docker CLI ≥ 18.06 and podman/buildah pull OCI manifests from any v2 registry. **Do NOT push an index/manifest-list**: one linux/amd64 manifest per tag; docker pulls it directly.
|
||||
- **Blob uploads (OCI dist-spec, gitea-conformant)**: primary = **monolithic** `POST /v2/coreci/broker/blobs/uploads/?digest=sha256:<hex>` with the blob as body → 201 (single round trip); fallback = two-step `POST /v2/<name>/blobs/uploads/` → 202 + `Location` → `PUT <Location>?digest=…` → 201. Always `HEAD /v2/<name>/blobs/<digest>` first — 200 → skip upload (base layers re-run-proof: a partially-failed publish resumes cheaply). **Base layers must be re-uploaded to gitea** (~45.7 MB once per milestone) — gitea cannot mount blobs from Docker Hub; acceptable per D-019's once-per-milestone cadence.
|
||||
- **Redirect trap (verified in CPython 3.11 source)**: `HTTPRedirectHandler.redirect_request` only follows GET/HEAD (301/302/303 POST→GET); **307/308 on a PUT/POST raise HTTPError** — stdlib urllib does NOT follow method-preserving redirects. The client must implement explicit Location-following (read `Location`, `urljoin` against the registry base — gitea's `Location` may be relative or absolute — re-issue the request with method+body). Gitea default local storage answers directly, but the client is correct against any conformance-grade registry.
|
||||
- **Manifest PUT**: `PUT /v2/coreci/broker/manifests/<tag>` with `Content-Type: application/vnd.oci.image.manifest.v1+json` → 201. Tags: `<milestone-release-tag>` (e.g. `v0.2.3`) **and** `latest` — two PUTs of the same manifest (D-021).
|
||||
|
||||
### 3.4 Layer + config + manifest construction (exact recipe)
|
||||
|
||||
1. **Broker layer**: deterministic tar (`tarfile`, entries sorted, fixed `TarInfo` metadata — uid/gid 0, uname/gname fixed, `mtime=0`/1980-epoch, file mode 0644) containing `/app/coreci-broker.pyz` (the built artifact). Gzip with `gzip.compress(data, compresslevel=9, mtime=0)` — **`mtime=0` is mandatory** (default `None` = wall clock → nondeterministic digests); fix the compresslevel in code (stable per build).
|
||||
2. Digests: **layer digest** = sha256 of the gzip blob; **diff_id** = sha256 of the *uncompressed* tar.
|
||||
3. **New config blob**: copy the pulled base config; append the new diff_id to `rootfs.diff_ids`; set `config.Entrypoint = ["python3", "/app/coreci-broker.pyz"]`, `config.Cmd = []` (so `docker run image --version` appends `--version` through the entrypoint), `config.WorkingDir = "/state"`, keep base `Env`; optional `config.Labels = {"org.opencontainers.image.version": "<tag>"}`. Serialize deterministically (`json.dumps(..., sort_keys=True, separators=(",", ":"))`) → reproducible config digest.
|
||||
4. **Push order**: HEAD all blobs (4 base layers, new layer, new config) → monolithic POST each missing one → PUT manifest referencing the base's 4 layer descriptors **verbatim** (original digests + mediaTypes) + the new layer descriptor + the new config descriptor. Record the resolved base index digest + amd64 child digest + all blob digests in the release notes (provenance; also `--base-manifest` override argument for exact re-publishes, default = tag resolution with documented upstream drift).
|
||||
5. Image name: `git.coreci.dev/coreci/broker` (gitea registry = `owner/name`; the package links to the repo by owner+name). gitea creates the package at first push (verified `NAME_UNKNOWN` until then).
|
||||
|
||||
### 3.5 Dockerfile role + mapping limits (documentation; daemonless client = source of truth)
|
||||
|
||||
- The Dockerfile is for users WITH docker; the daemonless client is what the pipeline publishes. **Equivalence required, byte-identity not.** Self-contained Dockerfile (no pre-built artifact in build context): `FROM python:3.11-slim`; `COPY coreci_broker.py coreci_broker_client.py coreci_broker_ui/ cli/ /app/src/`; `ARG VERSION=dev`; generated `_cli_version.py` via build arg; `RUN python3 -m zipapp /app/src -o /app/coreci-broker.pyz --python "/usr/bin/env python3" --compress`; `WORKDIR /state`; `ENTRYPOINT ["python3", "/app/coreci-broker.pyz"]`; `EXPOSE 8765`.
|
||||
- Mapping: same base family (tag drift upstream ≠ the pinned digests the client resolved at publish time), same entrypoint, same pyz content, same `/state` state anchoring; differences: docker-build layering (COPY+RUN blobs vs verbatim base layers + one broker layer) and version string provenance (build-arg vs build-time injection). Document both in the Dockerfile header.
|
||||
|
||||
### 3.6 Stub-test plan (`tests/test_publish_image.py`; v0.1 stub precedent)
|
||||
|
||||
- `_StubRegistryHandler` on an ephemeral port: `GET /v2/` → 401 + `WWW-Authenticate` (realm = stub `/token`); `GET /token` → `{"token":"stub"}` (plus the anonymous-empty-token variant → client must fail cleanly); `GET /v2/<name>/manifests/<ref>` → fixture index (Accept-checked) / child manifest; `GET /v2/<name>/blobs/<digest>` → fixture bytes; `HEAD` blob → 200/404; `POST /v2/<name>/blobs/uploads/` (+`?digest=`) → 201 monolithic (assert body sha256 == digest param) or 202+`Location`; `PUT .../uploads/<uuid>?digest=` → 201; `PUT /v2/<name>/manifests/<tag>` → 201 (assert `Content-Type` == OCI manifest, all referenced digests already uploaded, diff_ids count == layers count).
|
||||
- Real-manifest fixture: tiny synthetic OCI index/manifest/config/1-2 tiny layers built in-test with the same tar+gzip code (NOT the 45 MB real base — suite stays fast); plus one "live-shape" assertion test using the digests/mediaTypes captured in §3.1 as constants.
|
||||
- End-to-end offline: stub Docker Hub + stub gitea → run the publish flow → assert both tags PUT, base layer bytes passed through, broker layer contains the pyz at `/app/`.
|
||||
- Redirect edge: stub answers 307 on the two-step upload PUT → client follows Location explicitly (urllib would raise otherwise).
|
||||
|
||||
### 3.7 Defects / risks (daemonless)
|
||||
|
||||
- **RISK** — upstream tag drift (`3.11-slim` moves): pin at publish time by recording digests + `--base-manifest` override; notes carry provenance.
|
||||
- **RISK** — 307/308 not followed by urllib (verified): explicit Location-following required; stub-tested.
|
||||
- **RISK** — ~46 MB upload per milestone (base layers re-uploaded; no cross-registry mount): bounded, once-per-milestone; HEAD-skip makes retries cheap.
|
||||
- **RISK** — gitea's anonymous-200 token quirk (verified): empty-scope JWT must not be treated as auth success.
|
||||
- **NOTE** — monolithic upload limits: gitea handles multi-MB bodies; the config/manifest are KB-scale, the largest blob is the 29.8 MB base layer — fine for monolithic POST (dist-spec; gitea conformance-tested upstream).
|
||||
- **NOTE** — publish failure follows the same bounded-retry → `release_pending` escalation as releases; never blocks the milestone merge (ROADMAP P3).
|
||||
|
||||
## 4. Cross-Cutting: Layout, Security, Line Count
|
||||
|
||||
### 4.1 Recommended repo layout (new files only)
|
||||
|
||||
```
|
||||
cli/__main__.py # boot shim (staged into the pyz; NOT coreci_broker.py → outside AC14)
|
||||
scripts/build_cli.py # stage + zipapp + normalize; emits dist/coreci-broker-vX.Y.Z.pyz (+ sha256)
|
||||
scripts/release.py # notes builder + create-or-get release + asset attach + bounded retry
|
||||
scripts/publish_image.py # daemonless registry client (pull base → append layer → push, D-019)
|
||||
scripts/_env_secrets.py # shared .env-file secrets loader (~20 lines, one discipline point)
|
||||
Dockerfile # documentation + docker-user path (§3.5)
|
||||
dist/ # build output (gitignored)
|
||||
tests/test_cli_packaging.py # build in tmp + boot subprocess + boot-parity flows via urllib
|
||||
tests/test_build_cli.py # reproducibility: two builds → identical sha256; --version wiring
|
||||
tests/test_release_upload.py # stub-gitea attachment + retry/escalation paths
|
||||
tests/test_publish_image.py # stub-registry pull/push flows + manifest assembly
|
||||
```
|
||||
|
||||
- `scripts/_env_secrets.py` shared rather than copy-pasted per script: one place asserts the "file-only, non-empty" token discipline; `python3 scripts/release.py` runs with `sys.path[0] == scripts/` so the sibling import works without path setup.
|
||||
- **.gitignore additions**: `dist/`, `*.pyz` (root-level strays). Existing patterns already cover `__pycache__`, `*.db`, `*.log`, `.env*`, `data/`.
|
||||
- Phase mapping (per ROADMAP): Phase 1 = `cli/` + `scripts/build_cli.py` + its tests (REQ-21); Phase 2 = `release.py` + `publish_image.py` + `Dockerfile` + stub tests (REQ-22/23).
|
||||
|
||||
### 4.2 Broker changes: ZERO (verified)
|
||||
|
||||
- `run_standalone()`, `boot()`, `load_config()`, `_parse_env_file()` are module-level and importable; the shim + scripts need nothing added to the broker. `coreci_broker.py` = 1390/1400 lines (D-018 cap) — v0.2 adds no lines; AC14 stays green by construction.
|
||||
|
||||
### 4.3 Test-harness conventions (carried v0.1 §6 + new packaging patterns)
|
||||
|
||||
- Packaging test pattern (verified in prototype): `TemporaryDirectory` → `build_cli.build(state=tmp)` → subprocess `Popen(["python3", "<pyz>"], env={BROKER_BIND: "127.0.0.1:0", BROKER_ENV_PATH/DB/AUDIT: tmp paths}, stdout=PIPE)` → read stdout lines until `coreci-broker listening on http://…` (parse base URL; the `BROKER_ADMIN_PASS=` line comes first and is captured for login) → drive `urllib` flows (healthz, login, register, secret read — reuse v0.1 acceptance request patterns) → `proc.terminate()` in cleanup. Boot cost ~1–2 s; suite impact acceptable (v0.1 suite ≈ 2 min).
|
||||
- Stub servers: extend the `_StubOwuiHandler` pattern (class-level fixtures + hit recording + `addCleanup`) to `_StubGiteaHandler` / `_StubRegistryHandler`; all stubs bind `("127.0.0.1", 0)` and the code under test takes a base URL parameter (no hardcoded hosts in scripts — required for stubbing anyway).
|
||||
|
||||
### 4.4 Security posture (v0.2-specific)
|
||||
|
||||
- **No secrets in the artifact**: the pyz bundles broker + UI + client only; the client module contains no credentials; the broker generates pepper/admin pass at first boot; `.env` is never inside the archive. Same for the image: no state baked in (`WORKDIR /state` empty at build; first boot generates).
|
||||
- **Token discipline**: `release.py` / `publish_image.py` read `GITEA_TOKEN` ONLY via the `.env*` file loader (never shell env — structurally blocked by `blocked_env_vars`); assert non-empty before first use (v1.8: empty token → gitea 404 masks auth failure). Stubs use fake tokens; no live-forge writes in tests.
|
||||
- **Artifact integrity**: sha256 of the `.pyz` computed at build, embedded in release notes; reproducible build makes the digest meaningful across rebuilds.
|
||||
- **Static disclosure**: v0.1's `_broker_protected_paths` behavior extends unchanged to materialized roots (assets contain no secrets; `.env`/db/audit realpaths remain protected wherever the state dir sits).
|
||||
|
||||
## 5. Research Conclusions (planner-ready)
|
||||
|
||||
- **Phase 1 (REQ-21)**: `cli/__main__.py` five-step shim (§1.2 — prototype verified end-to-end, lift it), `scripts/build_cli.py` (stage → zipapp `--compress` → normalize post-pass §1.3 → sha256), `tests/test_cli_packaging.py` + `tests/test_build_cli.py`. Zero broker lines.
|
||||
- **Phase 2 (REQ-22/23)**: `scripts/_env_secrets.py` + `release.py` (create-or-get by tag, idempotent attach, bounded retry → release_pending, notes model §2.3); `scripts/publish_image.py` (§3.1–3.4 flows; OCI manifest end-to-end; monolithic uploads + HEAD skip + explicit Location-following); `Dockerfile` (§3.5); stub tests §2.4/§3.6.
|
||||
- **Phase 3 (acceptance)**: boot-parity matrix through the artifact (first-boot pass once, 0600 modes, silent second boot, UI + client served, register/read flows), reproducibility sha256, stub-releasability of both pipelines, security review (no secrets in artifact/image; token discipline), then milestone ship: build → attach to `v0.2.3` → publish image tags `v0.2.3` + `latest` → merge.
|
||||
- **No BLOCKERs found.** All environment facts were verified live (registry auth shapes, index resolution, gitea API, zipapp mechanics) and the packaging design is proven by a working prototype against the real broker sources. Remaining open items are NOTE/RISK grade and owned by the plan: sidecar `.sha256` asset vs digest-in-notes; shim fallback when run unzipped (trivial); `latest`-tag manifest PUT ordering (push version tag first, `latest` second — trivial).
|
||||
@@ -0,0 +1,45 @@
|
||||
# Roadmap — Milestone v0.2 (cli-packaging)
|
||||
|
||||
## Overview
|
||||
|
||||
One milestone, owner-directed 2026-10-10: package the completed v0.1 broker for distribution. **Milestone v0.2 — cli-packaging** (feature type): the broker as a single-file zipapp CLI (`coreci-broker.pyz`, self-contained), the CLI attached to every gitea release alongside release notes, and a docker container built **once per milestone** and published as a package in the gitea container registry. Scope is CLI + distribution only — no functional changes to the broker (REQ-21..REQ-23). Docker publication starts with v0.2; no retroactive v0.1 image.
|
||||
|
||||
Versioning (greenfield convention, D-008): the milestone branch is `milestone/v0.2-cli-packaging`; phase tags run on the milestone's own line — phase 0 ships `v0.2.0`, execution phases ship `v0.2.1`..`v0.2.2`, and the final phase's patch (`v0.2.3`) IS the milestone release; no separate minor tag. MAJOR remains 0 until an owner-approved v1 release. Ships go to gitea (`git.coreci.dev/coreci/broker`); releases are best-effort with bounded retry, never blocking tag+merge.
|
||||
|
||||
Autonomy: **guided** — escalate on every decision gate; threshold 0.85; 1 revision iteration; clarify budget 20.
|
||||
|
||||
## Phase 0 — Pre-Execution
|
||||
|
||||
- **Description**: Specification parsing (REQ-21..REQ-23), clarification (packaging/distribution decisions D-019..D-022), research (zipapp mechanics, gitea release-asset API, gitea container registry + OCI registry HTTP API v2, Docker Hub anonymous pulls), plan (task-level breakdown with waves), grill, MVP/UX check. `/ci-run` phase 0 completes REQUIREMENTS.md v2 section, RESEARCH.md, PLAN.md, PERSONAS.md, and the grill verdict before any code.
|
||||
- **Requirements covered**: none (planning artifacts)
|
||||
- **Success criteria**: REQ-21..REQ-23 defined and traceable; milestone v0.2 established in config.json and PROJECT.md; D-016+ decisions recorded; PLAN.md + PERSONAS.md committed; grill verdict recorded; MVP/UX sections present; phase tagged `v0.2.0`
|
||||
- **Depends on**: —
|
||||
- **Status**: in_progress
|
||||
|
||||
## Phase 1 — CLI Packaging + Release Tooling
|
||||
|
||||
- **Description**: `coreci-broker.pyz` build pipeline: stdlib `python3 -m zipapp` build script (no pip) with G-009 normalize pass (sorted entries, fixed timestamps, constant external_attr); self-contained archive bundling `coreci_broker.py` + `coreci_broker_ui/` + `coreci_broker_client.py` with asset materialization at boot (env-resolved roots, never `__file__`); standalone boot parity with v0.1 (first-boot `.env` generation, admin pass printed once, 0600 modes, full admin UI + client module served); `--version` flag; release-asset automation (G-011: `scripts/_env_secrets.py` + `scripts/release.py` land here so the v0.2.1 ship attach is script-driven — no ad-hoc bridge); tests boot the built artifact end-to-end and stub-verify the gitea upload.
|
||||
- **Requirements covered**: REQ-21, REQ-22
|
||||
- **Success criteria**: artifact builds reproducibly from a clean checkout (two consecutive same-environment builds → byte-identical sha256); `python3 coreci-broker.pyz` boots with behavior byte-identical to `python3 coreci_broker.py` (v0.1 suite green through the packaged path); UI pages + client module served from the archive; first-boot semantics preserved on second boot; the `v0.2.1` release carries the `.pyz` attached by `scripts/release.py` (D-020 starts here; `v0.2.0` carries none — G-021)
|
||||
- **Depends on**: 0
|
||||
- **Status**: not_started
|
||||
|
||||
## Phase 2 — Docker + Image Publish
|
||||
|
||||
- **Description**: Daemonless publish client per D-019 (once-per-milestone pull pinned base layers / append broker layer / push to `git.coreci.dev/coreci/broker` with version + `latest` tags, D-021/D-022; G-019 wire hardening: per-request timeouts, blob retry with HEAD-skip, explicit 307/308 relative-Location handling); `Dockerfile` + `.dockerignore` (G-015) + consistency-tested equivalence mapping (G-016); README additive section (CLI + docker usage); the `v0.2.2` release attaches the `.pyz` via `scripts/release.py` with CLI + docker notes.
|
||||
- **Requirements covered**: REQ-23 (+ REQ-21/REQ-22 docs)
|
||||
- **Success criteria**: daemonless publish client stub-verified end-to-end (no live-forge writes, no container runtime); Dockerfile/.dockerignore consistency tests green; this phase's release (`v0.2.2`) carries the `.pyz` attached alongside the source archive with notes; bounded-retry → `release_pending` paths covered by tests
|
||||
- **Depends on**: 1
|
||||
- **Status**: not_started
|
||||
|
||||
## Phase 3 — Final Review + Ship
|
||||
|
||||
- **Description**: Full acceptance: cumulative v0.1 suite green plus v0.2 packaging acceptance (boot parity through the artifact, self-containment, daemonless publish flow); security review (artifact integrity, no secrets in the archive/image, token discipline in upload paths, build-context hygiene); multi-persona review of the milestone diff; audit/reconstruction (REQ-21..23 traceable); milestone ship in the G-012-corrected order — build CLI → squash-merge `milestone/v0.2-cli-packaging` → `main` → tag `v0.2.3` → push → G-013 live smoke (read-only, env-gated) → attach `.pyz` + full notes via `scripts/release.py` → daemonless image publish once (D-019..D-022, with the G-020 closure runbook on failure) → branch cleanup → closeout.
|
||||
- **Requirements covered**: REQ-21, REQ-22, REQ-23 (acceptance)
|
||||
- **Success criteria**: all v0.1 + v0.2 acceptance green; security review PASS; `v0.2.3` on `main` with the `.pyz` attached + notes on the gitea release; docker image published as a gitea package (or escalated per the bounded-retry contract + closure runbook, never blocking merge); REQUIREMENTS/ROADMAP closed out; checkpoint cleared
|
||||
- **Depends on**: 2
|
||||
- **Status**: not_started
|
||||
|
||||
## Post-Milestone Notes (carry-forward from v0.1, GRILL G-008)
|
||||
|
||||
- **90-day adoption criterion**: `proxmox_readonly_tool.py` runs on `coreci_broker_client.py` with direct KB reads removed by **2027-01-09**; the owner owns the date. No real tool migrates to the broker before the owner-run post-ship OWUI smoke checklist (G-005) passes on the deployed v0.11.3 host. These v0.1 post-ship obligations remain open after the v0.1 ship (2026-10-10) and carry forward unchanged.
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"version": "0.8.0",
|
||||
"autonomy": {
|
||||
"level": "guided",
|
||||
"decision_confidence_threshold": 0.85,
|
||||
"escalation_hooks": ["*"],
|
||||
"max_revision_iterations": 1,
|
||||
"clarify_budget": 20
|
||||
},
|
||||
"release": {
|
||||
"forge": "gitea",
|
||||
"base_url": "https://git.coreci.dev",
|
||||
"owner": "coreci",
|
||||
"repo": "broker"
|
||||
},
|
||||
"secrets": {
|
||||
"scopes": ["gitea"],
|
||||
"env_files": [".env", ".env.secrets", ".env.*"]
|
||||
},
|
||||
"ship": {
|
||||
"per_phase": true,
|
||||
"allow_skip": false,
|
||||
"max_release_retries": 3,
|
||||
"release_blocking": false
|
||||
},
|
||||
"verification": {
|
||||
"bdd_default": false
|
||||
},
|
||||
"personas": {
|
||||
"enabled": true,
|
||||
"territory_enforcement": "warn"
|
||||
},
|
||||
"parallelization": {
|
||||
"enabled": false,
|
||||
"max_concurrent_agents": 1
|
||||
},
|
||||
"ideation": {
|
||||
"max_ideas": 20,
|
||||
"categories": ["security", "quality", "architecture", "coverage", "improvement", "spec", "chaos", "bdd"]
|
||||
},
|
||||
"security": {
|
||||
"bash_allowlist": {
|
||||
"blocked_env_vars": ["GITEA_TOKEN", "GITHUB_TOKEN", "GITLAB_TOKEN"]
|
||||
}
|
||||
},
|
||||
"projects": [],
|
||||
"active_project": null,
|
||||
"milestone": {
|
||||
"version": "v0.2",
|
||||
"name": "cli-packaging",
|
||||
"type": "feature",
|
||||
"branch": "milestone/v0.2-cli-packaging"
|
||||
}
|
||||
}
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# Secrets (ciagent credential discipline)
|
||||
.env
|
||||
.env.secrets
|
||||
.env.*
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# Broker runtime state
|
||||
data/
|
||||
*.db
|
||||
*.log
|
||||
@@ -0,0 +1,180 @@
|
||||
# CoreCI Broker
|
||||
|
||||
A single-file credential broker for autonomous agent tools on the Open WebUI
|
||||
host. Tools authenticate with derived tokens instead of shared secrets, the
|
||||
CEO gates access through UI-managed allowlists of app UUIDs and user IDs,
|
||||
every read lands in an append-only audit log, and secret reads are path-scoped
|
||||
to `coreci/<agent_id>` — one Python file, one SQLite DB, a four-page admin UI,
|
||||
zero pip installs, no build step.
|
||||
|
||||
## Quickstart (standalone)
|
||||
|
||||
```bash
|
||||
python3 coreci_broker.py
|
||||
```
|
||||
|
||||
- Stdlib-only runtime, Python 3.11+. Zero pip installs: bcrypt when
|
||||
importable, stdlib scrypt otherwise (format-prefixed hashes).
|
||||
- First boot generates `.env`: a 32-byte base64 pepper plus a 32+-char admin
|
||||
password. The password is printed to the startup log as `BROKER_ADMIN_PASS=…`
|
||||
**exactly once** — save it immediately; it is never printed again. Lost pass:
|
||||
remove `BROKER_ADMIN_PASS` from `.env` and reboot — a new one prints once.
|
||||
- The server binds `127.0.0.1:8765`. Open `http://127.0.0.1:8765/ui/login` and
|
||||
log in with the printed pass.
|
||||
- Stop with Ctrl+C. Durable state is exactly three files: `broker.db`,
|
||||
`broker.audit.log`, `.env` — back them up.
|
||||
|
||||
## Configuration
|
||||
|
||||
Read on boot from the `.env` at `BROKER_ENV_PATH`. Precedence:
|
||||
**process env → `.env` → defaults**.
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `BROKER_BIND` | `127.0.0.1:8765` | Listen address; the bind controls reachability |
|
||||
| `BROKER_ADMIN_USER` | `admin` | Admin login user name |
|
||||
| `BROKER_ADMIN_PASS` | generated first boot | Admin login password (32+ random chars) |
|
||||
| `BROKER_INSTANCE_PEPPER` | generated first boot | 32-byte base64 pepper; token derivation input, synced to SQLite |
|
||||
| `BROKER_ENV_PATH` | `.env` | Path of the `.env` file the broker reads/writes |
|
||||
| `BROKER_TOKEN_TTL` | `86400` | Tool token lifetime, seconds (24 h) |
|
||||
| `BROKER_SESSION_TTL` | `300` | Admin UI session cookie lifetime, seconds |
|
||||
| `BROKER_DB_PATH` | `broker.db` | SQLite database (mode 0600) |
|
||||
| `BROKER_AUDIT_LOG_PATH` | `broker.audit.log` | Append-only JSONL audit log (mode 0600) |
|
||||
| `BROKER_CREDS_SOURCE` | `file` | Credential source adapter: `file` (dev/tests) or `owui` (production) |
|
||||
| `BROKER_CREDS_FILE` | `data/creds.json` | `FileCredsSource` JSON document: paths → secret objects |
|
||||
| `BROKER_OWUI_BASE_URL` | — | Open WebUI base URL for the OWUI creds source |
|
||||
| `BROKER_OWUI_API_KEY` | — | Open WebUI API key (Bearer) for KB read access |
|
||||
| `BROKER_AGENT_STATE_BACKEND` | `internal` | Agent-state storage: `internal` (rows in broker.db) or `owui` |
|
||||
| `BROKER_MAX_AGENTS_PER_APP` | `50` | Cap on agent registrations per app UUID |
|
||||
| `BROKER_UI_ROOT` | `coreci_broker_ui` | Directory served for `/ui/*` pages |
|
||||
| `BROKER_STATIC_ROOT` | `.` | Directory served for `/static/*` (client module) |
|
||||
|
||||
Production: `BROKER_CREDS_SOURCE=owui` with `BROKER_OWUI_BASE_URL` +
|
||||
`BROKER_OWUI_API_KEY` (0600 `.env` discipline). Keep `BROKER_BIND` on
|
||||
loopback unless you deliberately widen reachability.
|
||||
|
||||
## Running as an Open WebUI workspace tool
|
||||
|
||||
Paste `coreci_broker.py` into Open WebUI as a workspace Tool (frontmatter
|
||||
`requirements:` stays empty — the runtime already ships everything). Notes
|
||||
that matter:
|
||||
|
||||
- The server starts in `Tools.__init__` behind a once-per-process guard; the
|
||||
OWUI loader has no activate/deactivate hook, and disabling a tool does not
|
||||
stop a previously started server.
|
||||
- **After editing the broker tool source, restart Open WebUI.** The loader
|
||||
re-execs the module on source change, but the previously loaded server keeps
|
||||
its port — the new instance fails loud with a bind-error message telling you
|
||||
to restart, rather than silently serving old code.
|
||||
- Paths (DB, audit log, `.env`, UI roots) resolve from environment config,
|
||||
never from `__file__` — the OWUI loader deletes the temp file it execs.
|
||||
- Durable state: `broker.db` + `broker.audit.log` + `.env` — **back those
|
||||
three up**; SQLite is the source of truth, no HA, restore is a file copy.
|
||||
- `Tools.status()` reports the serving address; call it from a chat to verify
|
||||
the mount, or hit `/healthz` on the bind.
|
||||
|
||||
## Admin UI
|
||||
|
||||
Four pages + dashboard, server-rendered on the same port as the API:
|
||||
|
||||
- `/ui/login` — the CEO's bookmark. First-boot pass discipline: the printed
|
||||
pass is shown once; record it in your secret store on first login.
|
||||
- `/ui/app_registrations` — allowlist of tool app UUIDs: add, revoke, reinstate.
|
||||
- `/ui/user_allowlist` — allowlist of user IDs (optional email): same actions.
|
||||
- `/ui/audit` — filterable, read-only view of the audit log (no export in v0.1).
|
||||
- `/ui/dashboard` — landing page with persistent nav header and Logout.
|
||||
|
||||
Allowlist semantics: **revoke keeps existing tokens working until TTL expiry —
|
||||
by design.** Revocation immediately blocks new registrations; already-issued
|
||||
tokens live at most one `BROKER_TOKEN_TTL` (24 h by default).
|
||||
|
||||
## Tool authors
|
||||
|
||||
Fetch the public client module — unauthenticated, from
|
||||
`http://<broker>/static/coreci_broker_client.py` — into your workspace and
|
||||
import it. Flow: `ensure_registered(state)` → `put_state(state)` →
|
||||
`read_secret("coreci/<agent_id>", state)`. Stdlib-urllib-only (no httpx, no
|
||||
bcrypt), so any tool can import it; on a 401 it re-registers and retries
|
||||
once. State endpoints are the tool's only persistence channel — keep `state`
|
||||
there, not in local files. See `reference_tools/` for the proxmox-style §11
|
||||
flow and a minimal auth-then-use showcase; downstream calls default to a
|
||||
safe dry-run.
|
||||
|
||||
## Security model
|
||||
|
||||
- Derived token: `base64(sha256(pepper:UUID:user_id:agent_id:sorted_tags))` —
|
||||
44 ASCII chars, no prefix, no challenge protocol. Stored only as a slow
|
||||
hash (`$2b$…` / `scrypt$…`), never in plaintext.
|
||||
- Allowlists are the gate: unknown or revoked app UUIDs and unknown users get
|
||||
403 at registration; the token is the consequence, not the authority.
|
||||
- Every secret-read attempt is audited (ok/denied/error) with user, UUID,
|
||||
agent, and path.
|
||||
- Secret reads are path-scoped: a registration reads exactly
|
||||
`coreci/<agent_id>` — cross-path → 403, absent → 404.
|
||||
- The broker never writes to `creds-coreci-coo`; the CEO maintains that KB
|
||||
through the Open WebUI UI.
|
||||
- `broker.db`, `broker.audit.log`, and `.env` are mode 0600.
|
||||
- Binds `127.0.0.1` by default; trusted-network posture — no rate limiting by
|
||||
design (spec §14).
|
||||
|
||||
## Documented spec deviations
|
||||
|
||||
Honest list; full detail in `.ciagent/PROJECT.md`:
|
||||
|
||||
- **Line-cap amendments (D-016/D-017/D-018)**: the single-file constraint grew
|
||||
800 → 1000 → 1200 → 1400 lines via the owner-approved §12.14 amendment
|
||||
pattern (readability over dense cramming). Current source: 1390 lines; AC14
|
||||
enforces the cap.
|
||||
- **Dual-mode hashing (D-001)**: spec §6 mandates bcrypt; the runtime is
|
||||
stdlib-only, so scrypt fallback with format-prefixed hashes.
|
||||
- **Token verification (D-004)**: spec §7.5's in-memory `token_hash` index is
|
||||
impossible with salted slow hashes; the broker scans rows scoped to the
|
||||
presented UUID and keeps a non-authoritative verified-token cache.
|
||||
- **Pepper rule (D-005)**: the pepper is returned only on row creation or
|
||||
tokenless registration (the KB-loss recovery path); spec §9's snippet would
|
||||
raise `KeyError` on re-registration.
|
||||
- **Spec defects resolved**: §7.5 index, §6-vs-§9 (the broker derives
|
||||
authoritatively; the tool stores), §9's `KeyError` — see PROJECT.md §Context.
|
||||
- `BROKER_ADMIN_USER` defaults to `admin` (spec silent on a default).
|
||||
- Register cap `BROKER_MAX_AGENTS_PER_APP` (50) — a documented addition.
|
||||
- Reinstate actions audit as `add_app`/`add_user` (no separate audit action).
|
||||
- In-scope but absent secrets audit as `read` with `result: error` (404s are
|
||||
audited, not silent).
|
||||
|
||||
## Post-ship live-OWUI smoke checklist (owner-run)
|
||||
|
||||
Before migrating any real tool (G-005; deployed host pinned v0.11.3, G-006):
|
||||
|
||||
1. Confirm the deployed Open WebUI version (expected **v0.11.3**) and that the
|
||||
KB read endpoints match the researched surface: `/api/v1/knowledge/{id}` and
|
||||
`/api/v1/files/{id}/data/content`.
|
||||
2. Add the broker as a workspace tool; check `/healthz` on the bind.
|
||||
3. CEO first-boot login at `/ui/login` with the printed pass.
|
||||
4. Add one app UUID and one user to the allowlists via the UI.
|
||||
5. In a scratch tool: import the client module from `/static/` →
|
||||
`ensure_registered` → read `coreci/proxmox` via the OWUI creds source.
|
||||
6. Verify every step in `/ui/audit`.
|
||||
7. Rollback = disable the tool + restore the three state files.
|
||||
|
||||
No real tool migrates before this passes; the 90-day adoption criterion
|
||||
(proxmox tool on the client, direct KB reads removed) is due 2027-01-09.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
timeout 300 python3 -m unittest discover -s tests
|
||||
```
|
||||
|
||||
295 tests, stdlib unittest only (~2 min; one skip where bcrypt is not
|
||||
importable). The AC1–AC14 acceptance matrix is green.
|
||||
|
||||
## Repository layout
|
||||
|
||||
| Path | What it is |
|
||||
|------|------------|
|
||||
| `coreci_broker.py` | The broker: single file — server, store, audit, auth, UI serving |
|
||||
| `coreci_broker_client.py` | Public stdlib-only client module (served at `/static/`) |
|
||||
| `coreci_broker_ui/` | Server-rendered HTML for the four pages + dashboard |
|
||||
| `reference_tools/` | Tool-author templates (proxmox-style §11 flow + minimal showcase) |
|
||||
| `tests/` | Stdlib-unittest suite incl. the AC1–AC14 acceptance matrix |
|
||||
| `.ciagent/` | CI pipeline artifacts (project, requirements, roadmap, decisions) |
|
||||
+1390
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,161 @@
|
||||
"""Public stdlib-only client for the CoreCI Broker: idempotent tool registration, agent-state persistence, and path-scoped secret reads."""
|
||||
import json
|
||||
import time
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
SECRETS_PREFIX = "/api/broker/secrets/"
|
||||
CLIENT_MODULE_PATH = "/static/coreci_broker_client.py"
|
||||
|
||||
|
||||
class BrokerError(Exception):
|
||||
def __init__(self, status, message):
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
self.message = message
|
||||
|
||||
|
||||
def _broker_error(status, body):
|
||||
message = "broker returned HTTP %s" % status
|
||||
if isinstance(body, dict) and body.get("message"):
|
||||
message = str(body["message"])
|
||||
return BrokerError(status, message)
|
||||
|
||||
|
||||
def _epoch_from_iso(value):
|
||||
try:
|
||||
year, month, day = int(value[0:4]), int(value[5:7]), int(value[8:10])
|
||||
hour, minute, second = int(value[11:13]), int(value[14:16]), int(value[17:19])
|
||||
rest = value[19:]
|
||||
micro = 0
|
||||
if rest[:1] == ".":
|
||||
digits = ""
|
||||
for char in rest[1:]:
|
||||
if not char.isdigit():
|
||||
break
|
||||
digits += char
|
||||
micro = int((digits + "000000")[:6])
|
||||
rest = rest[1 + len(digits):]
|
||||
offset = 0
|
||||
if rest[:1] in ("+", "-"):
|
||||
sign = 1 if rest[0] == "+" else -1
|
||||
offset = sign * (int(rest[1:3]) * 3600 + int(rest[4:6]) * 60)
|
||||
if not 1 <= month <= 12 or not 1 <= day <= 31:
|
||||
return None
|
||||
if hour > 23 or minute > 59 or second > 59:
|
||||
return None
|
||||
year -= month <= 2
|
||||
era = (year if year >= 0 else year - 399) // 400
|
||||
yoe = year - era * 400
|
||||
doy = (153 * (month - 3 if month > 2 else month + 9) + 2) // 5 + day - 1
|
||||
doe = yoe * 365 + yoe // 4 - yoe // 100 + doy
|
||||
return (era * 146097 + doe - 719468) * 86400 + hour * 3600 + minute * 60 + second - offset + micro / 1000000.0
|
||||
except (ValueError, TypeError, IndexError):
|
||||
return None
|
||||
|
||||
|
||||
def _token_live(state):
|
||||
token = state.get("current_token")
|
||||
expires = _epoch_from_iso(state.get("token_expires_at"))
|
||||
return isinstance(token, str) and bool(token) and expires is not None and expires > time.time()
|
||||
|
||||
|
||||
class BrokerClient:
|
||||
def __init__(self, base_url, agent_id, app_uuid, user_id, tags=None, timeout=10.0):
|
||||
self.base_url = str(base_url).rstrip("/")
|
||||
self.agent_id = agent_id
|
||||
self.app_uuid = app_uuid
|
||||
self.user_id = user_id
|
||||
self.tags = list(tags or [])
|
||||
self.timeout = timeout
|
||||
|
||||
def _app_headers(self, state=None):
|
||||
token = (state or {}).get("current_token")
|
||||
if not isinstance(token, str) or not token:
|
||||
return {}
|
||||
return {"Authorization": "App %s %s" % (self.app_uuid, token)}
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(self.base_url + path, data=data, method=method)
|
||||
if data is not None:
|
||||
request.add_header("Content-Type", "application/json")
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=self.timeout) as response:
|
||||
status, raw = response.status, response.read()
|
||||
except urllib.request.HTTPError as error:
|
||||
status, raw = error.code, error.read()
|
||||
body = None
|
||||
text = raw.decode("utf-8", "replace")
|
||||
if text:
|
||||
try:
|
||||
body = json.loads(text)
|
||||
except ValueError:
|
||||
body = None
|
||||
return status, body
|
||||
|
||||
def _authorized_request(self, method, path, state, payload=None):
|
||||
status, body = self._request(method, path, payload, self._app_headers(state))
|
||||
if status == 401:
|
||||
self.register(state)
|
||||
status, body = self._request(method, path, payload, self._app_headers(state))
|
||||
return status, body
|
||||
|
||||
def _state_path(self):
|
||||
return "/api/broker/agents/%s/state" % urllib.parse.quote(self.agent_id, safe="")
|
||||
|
||||
def register(self, state):
|
||||
if state is None:
|
||||
state = {}
|
||||
payload = {"agent_id": self.agent_id, "app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id, "tags": list(self.tags)}
|
||||
status, data = self._request("POST", REGISTER_PATH, payload, self._app_headers(state))
|
||||
if status not in (200, 201) or not isinstance(data, dict):
|
||||
raise _broker_error(status, data)
|
||||
state["agent_id"] = self.agent_id
|
||||
state["app_registration_uuid"] = self.app_uuid
|
||||
state["user_id"] = self.user_id
|
||||
state["tags"] = list(self.tags)
|
||||
state["current_token"] = data["token"]
|
||||
state["token_expires_at"] = data["token_expires_at"]
|
||||
pepper = data.get("pepper")
|
||||
if pepper is not None:
|
||||
state["pepper"] = pepper
|
||||
return state
|
||||
|
||||
def ensure_registered(self, state):
|
||||
if state is None:
|
||||
state = {}
|
||||
if _token_live(state):
|
||||
return state
|
||||
return self.register(state)
|
||||
|
||||
def get_state(self, state):
|
||||
status, body = self._authorized_request("GET", self._state_path(), state)
|
||||
if status == 200:
|
||||
return body
|
||||
if status == 404:
|
||||
return None
|
||||
raise _broker_error(status, body)
|
||||
|
||||
def put_state(self, state):
|
||||
status, body = self._authorized_request("PUT", self._state_path(), state, state)
|
||||
if status == 200:
|
||||
return True
|
||||
raise _broker_error(status, body)
|
||||
|
||||
def read_secret(self, path, state):
|
||||
suffix = urllib.parse.quote(str(path), safe="/")
|
||||
status, body = self._authorized_request("GET", SECRETS_PREFIX + suffix, state)
|
||||
if status == 200:
|
||||
return body
|
||||
raise _broker_error(status, body)
|
||||
|
||||
|
||||
def fetch_client_module(base_url, timeout=10.0):
|
||||
url = str(base_url).rstrip("/") + CLIENT_MODULE_PATH
|
||||
with urllib.request.urlopen(url, timeout=timeout) as response:
|
||||
return response.read().decode("utf-8")
|
||||
@@ -0,0 +1,271 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>App Registrations — CoreCI Broker</title>
|
||||
<link rel="stylesheet" href="/ui/shared.css">
|
||||
<style>
|
||||
.page-head {
|
||||
display: flex;
|
||||
align-items: flex-start;
|
||||
justify-content: space-between;
|
||||
gap: 16px;
|
||||
flex-wrap: wrap;
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.page-sub {
|
||||
color: var(--muted);
|
||||
font-size: 14px;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.table-empty {
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.cell-actions {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.btn-danger {
|
||||
color: var(--danger);
|
||||
background: transparent;
|
||||
border: 1px solid rgba(240, 122, 114, 0.45);
|
||||
}
|
||||
|
||||
.badge-revoked {
|
||||
color: var(--danger);
|
||||
border-color: rgba(240, 122, 114, 0.45);
|
||||
background: rgba(240, 122, 114, 0.08);
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="nav"></div>
|
||||
<main class="wrap">
|
||||
<section class="card">
|
||||
<div class="page-head">
|
||||
<div>
|
||||
<h1>App Registrations</h1>
|
||||
<p class="page-sub">Tool app UUIDs allowed to register against the broker.</p>
|
||||
</div>
|
||||
<button type="button" class="btn btn-primary" id="add-open">Add registration</button>
|
||||
</div>
|
||||
<p class="form-error" id="page-error" role="alert" hidden></p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>UUID</th>
|
||||
<th>Name</th>
|
||||
<th>Status</th>
|
||||
<th>Created</th>
|
||||
<th>Revoked</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="rows"></tbody>
|
||||
</table>
|
||||
</section>
|
||||
</main>
|
||||
<div class="modal-backdrop" id="modal" hidden role="dialog" aria-modal="true" aria-labelledby="modal-title">
|
||||
<div class="modal">
|
||||
<h2 id="modal-title">Add registration</h2>
|
||||
<form id="add-form">
|
||||
<div class="field">
|
||||
<label for="uuid">App UUID</label>
|
||||
<input id="uuid" name="uuid" type="text" class="mono" placeholder="00000000-0000-0000-0000-000000000000" required>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="name">Name (optional)</label>
|
||||
<input id="name" name="name" type="text" placeholder="proxmox tool">
|
||||
</div>
|
||||
<p class="form-error" id="add-error" role="alert" hidden></p>
|
||||
<div class="modal-actions">
|
||||
<button type="button" class="btn btn-subtle" id="add-cancel">Cancel</button>
|
||||
<button type="submit" class="btn btn-primary" id="add-submit">Add</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
<script src="/ui/nav.js"></script>
|
||||
<script>
|
||||
(function () {
|
||||
var API = "/api/broker/app_registrations";
|
||||
var rows = document.getElementById("rows");
|
||||
var pageError = document.getElementById("page-error");
|
||||
var modal = document.getElementById("modal");
|
||||
var form = document.getElementById("add-form");
|
||||
var uuidInput = document.getElementById("uuid");
|
||||
var nameInput = document.getElementById("name");
|
||||
var addError = document.getElementById("add-error");
|
||||
var submitButton = document.getElementById("add-submit");
|
||||
|
||||
function showPageError(message) {
|
||||
pageError.textContent = message;
|
||||
pageError.hidden = false;
|
||||
}
|
||||
|
||||
function clearPageError() {
|
||||
pageError.hidden = true;
|
||||
pageError.textContent = "";
|
||||
}
|
||||
|
||||
function messageRow(text) {
|
||||
rows.replaceChildren();
|
||||
var tr = document.createElement("tr");
|
||||
var td = document.createElement("td");
|
||||
td.colSpan = 6;
|
||||
td.className = "table-empty";
|
||||
td.textContent = text;
|
||||
tr.appendChild(td);
|
||||
rows.appendChild(tr);
|
||||
}
|
||||
|
||||
function textCell(value, mono) {
|
||||
var td = document.createElement("td");
|
||||
if (mono) {
|
||||
td.className = "mono";
|
||||
}
|
||||
td.textContent = value === null || value === undefined || value === "" ? "—" : value;
|
||||
return td;
|
||||
}
|
||||
|
||||
function statusBadge(status) {
|
||||
var span = document.createElement("span");
|
||||
span.className = "badge " + (status === "active" ? "badge-active" : "badge-revoked");
|
||||
span.textContent = status;
|
||||
return span;
|
||||
}
|
||||
|
||||
function actionButton(label, className, disabled, handler) {
|
||||
var button = document.createElement("button");
|
||||
button.type = "button";
|
||||
button.className = "btn " + className;
|
||||
button.textContent = label;
|
||||
button.disabled = disabled;
|
||||
button.addEventListener("click", handler);
|
||||
return button;
|
||||
}
|
||||
|
||||
function renderRow(row) {
|
||||
var tr = document.createElement("tr");
|
||||
tr.appendChild(textCell(row.uuid, true));
|
||||
tr.appendChild(textCell(row.name));
|
||||
var statusCell = document.createElement("td");
|
||||
statusCell.appendChild(statusBadge(row.status));
|
||||
tr.appendChild(statusCell);
|
||||
tr.appendChild(textCell(row.created_at));
|
||||
tr.appendChild(textCell(row.revoked_at));
|
||||
var actionsCell = document.createElement("td");
|
||||
var actions = document.createElement("div");
|
||||
actions.className = "cell-actions";
|
||||
var key = encodeURIComponent(row.uuid);
|
||||
actions.appendChild(actionButton("Revoke", "btn-danger", row.status !== "active", function () {
|
||||
rowAction("DELETE", API + "/" + key, "Revoke app registration " + row.uuid + "?", "Revoke failed", row.uuid);
|
||||
}));
|
||||
actions.appendChild(actionButton("Reinstate", "btn-subtle", row.status === "active", function () {
|
||||
rowAction("POST", API + "/" + key + "/reinstate", "Reinstate app registration " + row.uuid + "?", "Reinstate failed", row.uuid);
|
||||
}));
|
||||
actionsCell.appendChild(actions);
|
||||
tr.appendChild(actionsCell);
|
||||
return tr;
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
messageRow("Loading registrations…");
|
||||
var result = await api(API);
|
||||
if (!result.ok) {
|
||||
messageRow(result.status === 0 ? "Broker unreachable." : "Could not load registrations.");
|
||||
return;
|
||||
}
|
||||
var list = (result.data && result.data.registrations) || [];
|
||||
if (!list.length) {
|
||||
messageRow("No app registrations yet.");
|
||||
return;
|
||||
}
|
||||
rows.replaceChildren();
|
||||
list.forEach(function (row) {
|
||||
rows.appendChild(renderRow(row));
|
||||
});
|
||||
}
|
||||
|
||||
async function rowAction(method, path, question, label, uuid) {
|
||||
if (!confirm(question)) {
|
||||
return;
|
||||
}
|
||||
clearPageError();
|
||||
var result = await api(path, {method: method});
|
||||
if (!result.ok) {
|
||||
if (result.status === 0) {
|
||||
showPageError(label + ": broker unreachable.");
|
||||
} else {
|
||||
var detail = result.data && result.data.message ? ": " + result.data.message : ".";
|
||||
showPageError(label + " " + uuid + detail);
|
||||
}
|
||||
}
|
||||
await refresh();
|
||||
}
|
||||
|
||||
function openModal() {
|
||||
form.reset();
|
||||
addError.hidden = true;
|
||||
addError.textContent = "";
|
||||
submitButton.disabled = false;
|
||||
modal.hidden = false;
|
||||
uuidInput.focus();
|
||||
}
|
||||
|
||||
function closeModal() {
|
||||
modal.hidden = true;
|
||||
}
|
||||
|
||||
form.addEventListener("submit", async function (event) {
|
||||
event.preventDefault();
|
||||
addError.hidden = true;
|
||||
addError.textContent = "";
|
||||
submitButton.disabled = true;
|
||||
var payload = {uuid: uuidInput.value.trim()};
|
||||
var name = nameInput.value.trim();
|
||||
if (name) {
|
||||
payload.name = name;
|
||||
}
|
||||
var result = await api(API, {method: "POST", body: payload});
|
||||
submitButton.disabled = false;
|
||||
if (result.ok) {
|
||||
closeModal();
|
||||
clearPageError();
|
||||
await refresh();
|
||||
return;
|
||||
}
|
||||
if (result.status === 0) {
|
||||
addError.textContent = "Broker unreachable.";
|
||||
} else if (result.data && result.data.message) {
|
||||
addError.textContent = result.data.message;
|
||||
} else {
|
||||
addError.textContent = "Could not add registration.";
|
||||
}
|
||||
addError.hidden = false;
|
||||
await refresh();
|
||||
});
|
||||
|
||||
document.getElementById("add-open").addEventListener("click", openModal);
|
||||
document.getElementById("add-cancel").addEventListener("click", closeModal);
|
||||
modal.addEventListener("click", function (event) {
|
||||
if (event.target === modal) {
|
||||
closeModal();
|
||||
}
|
||||
});
|
||||
document.addEventListener("keydown", function (event) {
|
||||
if (event.key === "Escape" && !modal.hidden) {
|
||||
closeModal();
|
||||
}
|
||||
});
|
||||
refresh();
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,280 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Audit — CoreCI Broker</title>
|
||||
<link rel="stylesheet" href="/ui/shared.css">
|
||||
<style>
|
||||
.page-sub {
|
||||
color: var(--muted);
|
||||
font-size: 14px;
|
||||
margin: 0 0 16px;
|
||||
}
|
||||
|
||||
.table-empty {
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.filter-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(170px, 1fr));
|
||||
gap: 12px;
|
||||
align-items: end;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
.filter-grid .field {
|
||||
margin-bottom: 0;
|
||||
}
|
||||
|
||||
.filter-apply .btn {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.table-scroll {
|
||||
overflow-x: auto;
|
||||
}
|
||||
|
||||
.count-line {
|
||||
color: var(--muted);
|
||||
font-size: 14px;
|
||||
margin: 12px 0 0;
|
||||
}
|
||||
|
||||
.load-more {
|
||||
margin-top: 14px;
|
||||
}
|
||||
|
||||
.badge-denied {
|
||||
color: var(--danger);
|
||||
border-color: rgba(240, 122, 114, 0.45);
|
||||
background: rgba(240, 122, 114, 0.08);
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="nav"></div>
|
||||
<main class="wrap">
|
||||
<section class="card">
|
||||
<h1>Audit</h1>
|
||||
<p class="page-sub">Read-only trail of broker activity. Set filters and apply to query the audit log.</p>
|
||||
<p class="form-error" id="page-error" role="alert" hidden></p>
|
||||
<form id="filters" class="filter-grid">
|
||||
<div class="field">
|
||||
<label for="since">Since</label>
|
||||
<input type="datetime-local" id="since" name="since">
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="until">Until</label>
|
||||
<input type="datetime-local" id="until" name="until">
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="user_id">User ID</label>
|
||||
<input type="text" id="user_id" name="user_id" class="mono" autocomplete="off">
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="app_registration_uuid">App UUID</label>
|
||||
<input type="text" id="app_registration_uuid" name="app_registration_uuid" class="mono" autocomplete="off" placeholder="00000000-0000-0000-0000-000000000000">
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="action">Action</label>
|
||||
<select id="action" name="action">
|
||||
<option value="">Any action</option>
|
||||
<option value="register">register</option>
|
||||
<option value="read">read</option>
|
||||
<option value="denied">denied</option>
|
||||
<option value="login">login</option>
|
||||
<option value="logout">logout</option>
|
||||
<option value="add_app">add_app</option>
|
||||
<option value="remove_app">remove_app</option>
|
||||
<option value="add_user">add_user</option>
|
||||
<option value="remove_user">remove_user</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="limit">Limit</label>
|
||||
<input type="number" id="limit" name="limit" min="1" max="1000" value="100">
|
||||
</div>
|
||||
<div class="filter-apply">
|
||||
<button type="submit" class="btn btn-primary" id="apply">Apply</button>
|
||||
</div>
|
||||
</form>
|
||||
<div class="table-scroll">
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Timestamp</th>
|
||||
<th>User ID</th>
|
||||
<th>App UUID</th>
|
||||
<th>Agent ID</th>
|
||||
<th>Action</th>
|
||||
<th>Path</th>
|
||||
<th>Result</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="rows"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
<p class="count-line" id="count" hidden></p>
|
||||
<div class="load-more">
|
||||
<button type="button" class="btn btn-subtle" id="more" hidden>Load more</button>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
<script src="/ui/nav.js"></script>
|
||||
<script>
|
||||
(function () {
|
||||
var API = "/api/broker/audit";
|
||||
var FILTERS = ["since", "until", "user_id", "app_registration_uuid", "action"];
|
||||
var rows = document.getElementById("rows");
|
||||
var pageError = document.getElementById("page-error");
|
||||
var filterForm = document.getElementById("filters");
|
||||
var limitInput = document.getElementById("limit");
|
||||
var countLine = document.getElementById("count");
|
||||
var moreButton = document.getElementById("more");
|
||||
var state = {filters: {}, cursor: null, shown: 0};
|
||||
|
||||
function showPageError(message) {
|
||||
pageError.textContent = message;
|
||||
pageError.hidden = false;
|
||||
}
|
||||
|
||||
function clearPageError() {
|
||||
pageError.hidden = true;
|
||||
pageError.textContent = "";
|
||||
}
|
||||
|
||||
function messageRow(text) {
|
||||
rows.replaceChildren();
|
||||
var tr = document.createElement("tr");
|
||||
var td = document.createElement("td");
|
||||
td.colSpan = 7;
|
||||
td.className = "table-empty";
|
||||
td.textContent = text;
|
||||
tr.appendChild(td);
|
||||
rows.appendChild(tr);
|
||||
}
|
||||
|
||||
function textCell(value, mono) {
|
||||
var td = document.createElement("td");
|
||||
if (mono) {
|
||||
td.className = "mono";
|
||||
}
|
||||
td.textContent = value === null || value === undefined || value === "" ? "—" : String(value);
|
||||
return td;
|
||||
}
|
||||
|
||||
function resultBadge(result) {
|
||||
var span = document.createElement("span");
|
||||
span.className = "badge";
|
||||
if (result === "ok") {
|
||||
span.className += " badge-active";
|
||||
} else if (result === "denied" || result === "error") {
|
||||
span.className += " badge-denied";
|
||||
}
|
||||
span.textContent = result === null || result === undefined || result === "" ? "—" : String(result);
|
||||
return span;
|
||||
}
|
||||
|
||||
function renderRow(entry) {
|
||||
var tr = document.createElement("tr");
|
||||
tr.appendChild(textCell(entry.ts, true));
|
||||
tr.appendChild(textCell(entry.user_id, true));
|
||||
tr.appendChild(textCell(entry.app_registration_uuid, true));
|
||||
tr.appendChild(textCell(entry.agent_id, true));
|
||||
tr.appendChild(textCell(entry.action, true));
|
||||
tr.appendChild(textCell(entry.path, true));
|
||||
var resultCell = document.createElement("td");
|
||||
resultCell.appendChild(resultBadge(entry.result));
|
||||
tr.appendChild(resultCell);
|
||||
return tr;
|
||||
}
|
||||
|
||||
function readFilters() {
|
||||
var filters = {};
|
||||
FILTERS.forEach(function (name) {
|
||||
var value = document.getElementById(name).value.trim();
|
||||
if (value) {
|
||||
filters[name] = value;
|
||||
}
|
||||
});
|
||||
var limit = parseInt(limitInput.value, 10);
|
||||
filters.limit = Number.isFinite(limit) && limit > 0 ? limit : 100;
|
||||
return filters;
|
||||
}
|
||||
|
||||
function buildQuery(filters, cursor) {
|
||||
var params = [];
|
||||
FILTERS.forEach(function (name) {
|
||||
if (name === "since" && cursor !== null) {
|
||||
return;
|
||||
}
|
||||
if (filters[name]) {
|
||||
params.push(encodeURIComponent(name) + "=" + encodeURIComponent(filters[name]));
|
||||
}
|
||||
});
|
||||
if (cursor !== null) {
|
||||
params.push("since=" + encodeURIComponent(cursor));
|
||||
}
|
||||
params.push("limit=" + encodeURIComponent(String(filters.limit)));
|
||||
return "?" + params.join("&");
|
||||
}
|
||||
|
||||
async function load(reset) {
|
||||
clearPageError();
|
||||
moreButton.hidden = true;
|
||||
if (reset) {
|
||||
state.cursor = null;
|
||||
state.shown = 0;
|
||||
countLine.hidden = true;
|
||||
countLine.textContent = "";
|
||||
messageRow("Loading audit entries…");
|
||||
}
|
||||
var result = await api(API + buildQuery(state.filters, state.cursor));
|
||||
if (!result.ok) {
|
||||
var label = result.status === 0 ? "Broker unreachable." : (result.data && result.data.message) || "Could not load audit entries.";
|
||||
if (reset) {
|
||||
messageRow(label);
|
||||
} else {
|
||||
showPageError(label);
|
||||
moreButton.hidden = false;
|
||||
}
|
||||
return;
|
||||
}
|
||||
var data = result.data || {};
|
||||
var entries = Array.isArray(data.entries) ? data.entries : [];
|
||||
state.cursor = data.next_since === undefined ? null : data.next_since;
|
||||
if (reset) {
|
||||
rows.replaceChildren();
|
||||
}
|
||||
entries.forEach(function (entry) {
|
||||
rows.appendChild(renderRow(entry));
|
||||
});
|
||||
state.shown += entries.length;
|
||||
if (!state.shown) {
|
||||
messageRow("No audit entries match the current filters.");
|
||||
countLine.hidden = true;
|
||||
return;
|
||||
}
|
||||
countLine.textContent = state.shown + (state.shown === 1 ? " entry shown." : " entries shown.");
|
||||
countLine.hidden = false;
|
||||
moreButton.hidden = state.cursor === null;
|
||||
}
|
||||
|
||||
filterForm.addEventListener("submit", function (event) {
|
||||
event.preventDefault();
|
||||
state.filters = readFilters();
|
||||
load(true);
|
||||
});
|
||||
|
||||
moreButton.addEventListener("click", function () {
|
||||
load(false);
|
||||
});
|
||||
|
||||
state.filters = readFilters();
|
||||
load(true);
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,64 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Dashboard — CoreCI Broker</title>
|
||||
<link rel="stylesheet" href="/ui/shared.css">
|
||||
</head>
|
||||
<body>
|
||||
<div id="nav"></div>
|
||||
<main class="wrap">
|
||||
<section class="card">
|
||||
<h1>Dashboard</h1>
|
||||
<p>Signed in as <strong id="who">…</strong></p>
|
||||
</section>
|
||||
<section class="card status-strip">
|
||||
<span class="dot" id="status-dot" aria-hidden="true"></span>
|
||||
<span id="status-text" aria-live="polite">Broker status: checking…</span>
|
||||
</section>
|
||||
<section class="card">
|
||||
<h2>Quick links</h2>
|
||||
<div class="link-grid">
|
||||
<a class="link-card" href="/ui/app_registrations">
|
||||
<strong>App Registrations</strong>
|
||||
<span>Add, revoke, and reinstate tool app UUIDs</span>
|
||||
</a>
|
||||
<a class="link-card" href="/ui/user_allowlist">
|
||||
<strong>User Allowlist</strong>
|
||||
<span>Manage the user ids allowed to register</span>
|
||||
</a>
|
||||
<a class="link-card" href="/ui/audit">
|
||||
<strong>Audit Log</strong>
|
||||
<span>Review broker activity, read-only</span>
|
||||
</a>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
<script src="/ui/nav.js"></script>
|
||||
<script>
|
||||
(async function () {
|
||||
var who = document.getElementById("who");
|
||||
var dot = document.getElementById("status-dot");
|
||||
var text = document.getElementById("status-text");
|
||||
var session = await api("/api/broker/auth/whoami");
|
||||
if (session.ok && session.data) {
|
||||
who.textContent = session.data.username || "unknown";
|
||||
} else {
|
||||
who.textContent = "unavailable";
|
||||
}
|
||||
var health = await api("/healthz");
|
||||
if (health.ok) {
|
||||
dot.className = "dot dot-ok";
|
||||
text.textContent = "Broker status: healthy";
|
||||
} else if (health.status === 0) {
|
||||
dot.className = "dot dot-down";
|
||||
text.textContent = "Broker status: unreachable";
|
||||
} else {
|
||||
dot.className = "dot dot-down";
|
||||
text.textContent = "Broker status: unhealthy";
|
||||
}
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,13 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>CoreCI Broker</title>
|
||||
<meta http-equiv="refresh" content="0; url=/ui/login">
|
||||
</head>
|
||||
<body>
|
||||
<p><a href="/ui/login">Continue to CoreCI Broker sign-in</a></p>
|
||||
<script>location.replace("/ui/login");</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,78 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>CoreCI Broker</title>
|
||||
<link rel="stylesheet" href="/ui/shared.css">
|
||||
</head>
|
||||
<body class="login-body">
|
||||
<main class="login-card card">
|
||||
<h1>CoreCI Broker</h1>
|
||||
<p class="login-sub">Admin sign-in</p>
|
||||
<form id="login-form">
|
||||
<div class="field">
|
||||
<label for="username">Username</label>
|
||||
<input id="username" name="username" type="text" autocomplete="username" autofocus required>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="password">Password</label>
|
||||
<input id="password" name="password" type="password" autocomplete="current-password" required>
|
||||
</div>
|
||||
<p class="form-error" id="login-error" role="alert" hidden></p>
|
||||
<button type="submit" class="btn btn-primary btn-block" id="login-button">Sign in</button>
|
||||
</form>
|
||||
</main>
|
||||
<script>
|
||||
(function () {
|
||||
var form = document.getElementById("login-form");
|
||||
var error = document.getElementById("login-error");
|
||||
var button = document.getElementById("login-button");
|
||||
|
||||
form.addEventListener("input", function () {
|
||||
error.hidden = true;
|
||||
error.textContent = "";
|
||||
});
|
||||
|
||||
form.addEventListener("submit", async function (event) {
|
||||
event.preventDefault();
|
||||
error.hidden = true;
|
||||
error.textContent = "";
|
||||
button.disabled = true;
|
||||
var status = 0;
|
||||
var data = null;
|
||||
try {
|
||||
var response = await fetch("/api/broker/auth/login", {
|
||||
method: "POST",
|
||||
credentials: "same-origin",
|
||||
headers: {"Content-Type": "application/json"},
|
||||
body: JSON.stringify({
|
||||
username: document.getElementById("username").value,
|
||||
password: document.getElementById("password").value
|
||||
})
|
||||
});
|
||||
status = response.status;
|
||||
try {
|
||||
data = await response.json();
|
||||
} catch (parseError) {
|
||||
data = null;
|
||||
}
|
||||
} catch (networkError) {
|
||||
status = 0;
|
||||
}
|
||||
if (status === 200) {
|
||||
location.href = "/ui/dashboard";
|
||||
return;
|
||||
}
|
||||
if (status === 0) {
|
||||
error.textContent = "Sign-in failed: broker unreachable.";
|
||||
} else {
|
||||
error.textContent = (data && data.message) || "Sign-in failed. Please try again.";
|
||||
}
|
||||
error.hidden = false;
|
||||
button.disabled = false;
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,84 @@
|
||||
var NAV_ITEMS = [
|
||||
{label: "Dashboard", href: "/ui/dashboard"},
|
||||
{label: "App Registrations", href: "/ui/app_registrations"},
|
||||
{label: "Users", href: "/ui/user_allowlist"},
|
||||
{label: "Audit", href: "/ui/audit"}
|
||||
];
|
||||
|
||||
async function api(path, opts) {
|
||||
opts = opts || {};
|
||||
var init = {method: opts.method || "GET", credentials: "same-origin", headers: {}};
|
||||
if (opts.body !== undefined) {
|
||||
init.headers["Content-Type"] = "application/json";
|
||||
init.body = typeof opts.body === "string" ? opts.body : JSON.stringify(opts.body);
|
||||
}
|
||||
if (opts.headers) {
|
||||
for (var name in opts.headers) {
|
||||
init.headers[name] = opts.headers[name];
|
||||
}
|
||||
}
|
||||
var response = null;
|
||||
try {
|
||||
response = await fetch(path, init);
|
||||
} catch (networkError) {
|
||||
return {ok: false, status: 0, data: null};
|
||||
}
|
||||
var data = null;
|
||||
try {
|
||||
data = await response.json();
|
||||
} catch (parseError) {
|
||||
data = null;
|
||||
}
|
||||
if (response.status === 401) {
|
||||
location.href = "/ui/login";
|
||||
}
|
||||
return {ok: response.ok, status: response.status, data: data};
|
||||
}
|
||||
|
||||
async function navLogout() {
|
||||
try {
|
||||
await api("/api/broker/auth/logout", {method: "POST"});
|
||||
} catch (error) {}
|
||||
location.href = "/ui/login";
|
||||
}
|
||||
|
||||
function navRender() {
|
||||
var mount = document.getElementById("nav");
|
||||
if (!mount) {
|
||||
return;
|
||||
}
|
||||
var current = location.pathname.replace(/\.html$/, "").replace(/\/+$/, "");
|
||||
var header = document.createElement("header");
|
||||
header.className = "nav";
|
||||
var brand = document.createElement("a");
|
||||
brand.className = "nav-brand";
|
||||
brand.href = "/ui/dashboard";
|
||||
brand.textContent = "CoreCI Broker";
|
||||
header.appendChild(brand);
|
||||
var links = document.createElement("nav");
|
||||
links.setAttribute("aria-label", "Primary");
|
||||
NAV_ITEMS.forEach(function (item) {
|
||||
var link = document.createElement("a");
|
||||
link.href = item.href;
|
||||
link.textContent = item.label;
|
||||
if (current === item.href) {
|
||||
link.className = "active";
|
||||
link.setAttribute("aria-current", "page");
|
||||
}
|
||||
links.appendChild(link);
|
||||
});
|
||||
header.appendChild(links);
|
||||
var logout = document.createElement("button");
|
||||
logout.type = "button";
|
||||
logout.className = "btn btn-subtle nav-logout";
|
||||
logout.textContent = "Logout";
|
||||
logout.addEventListener("click", navLogout);
|
||||
header.appendChild(logout);
|
||||
mount.appendChild(header);
|
||||
}
|
||||
|
||||
if (document.readyState === "loading") {
|
||||
document.addEventListener("DOMContentLoaded", navRender);
|
||||
} else {
|
||||
navRender();
|
||||
}
|
||||
@@ -0,0 +1,369 @@
|
||||
:root {
|
||||
color-scheme: dark;
|
||||
--bg: #131820;
|
||||
--card: #1a212b;
|
||||
--card-alt: #232d3a;
|
||||
--border: #2d3947;
|
||||
--text: #e7edf4;
|
||||
--muted: #96a3b4;
|
||||
--accent: #2dd4bf;
|
||||
--accent-ink: #062421;
|
||||
--accent-soft: rgba(45, 212, 191, 0.12);
|
||||
--danger: #f07a72;
|
||||
--radius: 10px;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
[hidden] {
|
||||
display: none !important;
|
||||
}
|
||||
|
||||
html,
|
||||
body {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
body {
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
font: 15px/1.55 system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
|
||||
}
|
||||
|
||||
h1 {
|
||||
font-size: 24px;
|
||||
line-height: 1.25;
|
||||
margin: 0 0 12px;
|
||||
}
|
||||
|
||||
h2 {
|
||||
font-size: 17px;
|
||||
line-height: 1.25;
|
||||
margin: 0 0 12px;
|
||||
}
|
||||
|
||||
h3 {
|
||||
font-size: 15px;
|
||||
line-height: 1.25;
|
||||
margin: 0 0 10px;
|
||||
}
|
||||
|
||||
p {
|
||||
margin: 0 0 10px;
|
||||
}
|
||||
|
||||
a {
|
||||
color: var(--accent);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
a:hover {
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
a:focus-visible,
|
||||
button:focus-visible,
|
||||
input:focus-visible,
|
||||
select:focus-visible {
|
||||
outline: 2px solid var(--accent);
|
||||
outline-offset: 2px;
|
||||
}
|
||||
|
||||
.wrap {
|
||||
max-width: 1040px;
|
||||
margin: 0 auto;
|
||||
padding: 24px;
|
||||
}
|
||||
|
||||
.card {
|
||||
background: var(--card);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
padding: 20px 24px;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
.nav {
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 10;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
flex-wrap: wrap;
|
||||
gap: 8px 20px;
|
||||
padding: 10px 24px;
|
||||
background: var(--card);
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.nav-brand {
|
||||
color: var(--text);
|
||||
font-size: 16px;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.nav-brand:hover {
|
||||
color: var(--accent);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.nav nav {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
flex-wrap: wrap;
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
.nav nav a {
|
||||
color: var(--muted);
|
||||
font-size: 14px;
|
||||
padding: 6px 10px;
|
||||
border-radius: 6px;
|
||||
}
|
||||
|
||||
.nav nav a:hover {
|
||||
color: var(--text);
|
||||
background: var(--card-alt);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.nav nav a.active {
|
||||
color: var(--accent);
|
||||
background: var(--accent-soft);
|
||||
}
|
||||
|
||||
.nav-logout {
|
||||
margin-left: auto;
|
||||
}
|
||||
|
||||
.btn {
|
||||
display: inline-block;
|
||||
font: inherit;
|
||||
font-size: 14px;
|
||||
color: var(--text);
|
||||
background: var(--card-alt);
|
||||
border: 1px solid transparent;
|
||||
border-radius: 8px;
|
||||
padding: 9px 16px;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.btn:hover {
|
||||
filter: brightness(1.12);
|
||||
}
|
||||
|
||||
.btn:disabled {
|
||||
opacity: 0.6;
|
||||
cursor: default;
|
||||
}
|
||||
|
||||
.btn-primary {
|
||||
color: var(--accent-ink);
|
||||
background: var(--accent);
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.btn-subtle {
|
||||
background: transparent;
|
||||
border-color: var(--border);
|
||||
}
|
||||
|
||||
.btn-block {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.field {
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
|
||||
.field label {
|
||||
display: block;
|
||||
color: var(--muted);
|
||||
font-size: 13px;
|
||||
margin-bottom: 6px;
|
||||
}
|
||||
|
||||
.field input,
|
||||
.field select {
|
||||
width: 100%;
|
||||
font: inherit;
|
||||
color: var(--text);
|
||||
background: var(--bg);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
padding: 10px 12px;
|
||||
}
|
||||
|
||||
.field input:focus,
|
||||
.field select:focus {
|
||||
border-color: var(--accent);
|
||||
}
|
||||
|
||||
.form-error {
|
||||
color: var(--danger);
|
||||
background: rgba(240, 122, 114, 0.08);
|
||||
border: 1px solid rgba(240, 122, 114, 0.35);
|
||||
border-radius: 8px;
|
||||
padding: 10px 12px;
|
||||
font-size: 14px;
|
||||
margin: 0 0 14px;
|
||||
}
|
||||
|
||||
.login-body {
|
||||
min-height: 100vh;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
padding: 24px;
|
||||
}
|
||||
|
||||
.login-card {
|
||||
width: 100%;
|
||||
max-width: 380px;
|
||||
margin-bottom: 0;
|
||||
}
|
||||
|
||||
.login-sub {
|
||||
color: var(--muted);
|
||||
font-size: 14px;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
.status-strip {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
padding: 12px 24px;
|
||||
font-size: 14px;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.dot {
|
||||
width: 10px;
|
||||
height: 10px;
|
||||
border-radius: 50%;
|
||||
background: var(--muted);
|
||||
}
|
||||
|
||||
.dot-ok {
|
||||
background: var(--accent);
|
||||
}
|
||||
|
||||
.dot-down {
|
||||
background: var(--danger);
|
||||
}
|
||||
|
||||
.link-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
|
||||
gap: 16px;
|
||||
}
|
||||
|
||||
.link-card {
|
||||
display: block;
|
||||
background: var(--card-alt);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
padding: 16px;
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.link-card:hover {
|
||||
border-color: var(--accent);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.link-card strong {
|
||||
display: block;
|
||||
font-size: 15px;
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
|
||||
.link-card span {
|
||||
color: var(--muted);
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
th {
|
||||
text-align: left;
|
||||
color: var(--muted);
|
||||
font-size: 12px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.06em;
|
||||
padding: 8px 12px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
|
||||
td {
|
||||
padding: 10px 12px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
|
||||
tbody tr:hover {
|
||||
background: var(--card-alt);
|
||||
}
|
||||
|
||||
.mono {
|
||||
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.badge {
|
||||
display: inline-block;
|
||||
color: var(--muted);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 999px;
|
||||
padding: 2px 10px;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.badge-active {
|
||||
color: var(--accent);
|
||||
border-color: var(--accent);
|
||||
background: var(--accent-soft);
|
||||
}
|
||||
|
||||
.modal-backdrop {
|
||||
position: fixed;
|
||||
inset: 0;
|
||||
background: rgba(0, 0, 0, 0.55);
|
||||
display: grid;
|
||||
place-items: center;
|
||||
padding: 24px;
|
||||
z-index: 20;
|
||||
}
|
||||
|
||||
.modal {
|
||||
width: 100%;
|
||||
max-width: 460px;
|
||||
background: var(--card);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
padding: 20px 24px;
|
||||
margin-bottom: 0;
|
||||
}
|
||||
|
||||
.modal-actions {
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
gap: 10px;
|
||||
margin-top: 20px;
|
||||
}
|
||||
|
||||
@media (max-width: 900px) {
|
||||
.wrap {
|
||||
padding: 16px;
|
||||
}
|
||||
|
||||
.nav {
|
||||
padding: 10px 16px;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,271 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>User Allowlist — CoreCI Broker</title>
|
||||
<link rel="stylesheet" href="/ui/shared.css">
|
||||
<style>
|
||||
.page-head {
|
||||
display: flex;
|
||||
align-items: flex-start;
|
||||
justify-content: space-between;
|
||||
gap: 16px;
|
||||
flex-wrap: wrap;
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.page-sub {
|
||||
color: var(--muted);
|
||||
font-size: 14px;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.table-empty {
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.cell-actions {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.btn-danger {
|
||||
color: var(--danger);
|
||||
background: transparent;
|
||||
border: 1px solid rgba(240, 122, 114, 0.45);
|
||||
}
|
||||
|
||||
.badge-revoked {
|
||||
color: var(--danger);
|
||||
border-color: rgba(240, 122, 114, 0.45);
|
||||
background: rgba(240, 122, 114, 0.08);
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="nav"></div>
|
||||
<main class="wrap">
|
||||
<section class="card">
|
||||
<div class="page-head">
|
||||
<div>
|
||||
<h1>User Allowlist</h1>
|
||||
<p class="page-sub">User ids allowed to register agents against the broker.</p>
|
||||
</div>
|
||||
<button type="button" class="btn btn-primary" id="add-open">Add user</button>
|
||||
</div>
|
||||
<p class="form-error" id="page-error" role="alert" hidden></p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>User ID</th>
|
||||
<th>Email</th>
|
||||
<th>Status</th>
|
||||
<th>Created</th>
|
||||
<th>Revoked</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="rows"></tbody>
|
||||
</table>
|
||||
</section>
|
||||
</main>
|
||||
<div class="modal-backdrop" id="modal" hidden role="dialog" aria-modal="true" aria-labelledby="modal-title">
|
||||
<div class="modal">
|
||||
<h2 id="modal-title">Add user</h2>
|
||||
<form id="add-form">
|
||||
<div class="field">
|
||||
<label for="user_id">User ID</label>
|
||||
<input id="user_id" name="user_id" type="text" class="mono" required>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="email">Email (optional)</label>
|
||||
<input id="email" name="email" type="text" placeholder="user@example.com">
|
||||
</div>
|
||||
<p class="form-error" id="add-error" role="alert" hidden></p>
|
||||
<div class="modal-actions">
|
||||
<button type="button" class="btn btn-subtle" id="add-cancel">Cancel</button>
|
||||
<button type="submit" class="btn btn-primary" id="add-submit">Add</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
<script src="/ui/nav.js"></script>
|
||||
<script>
|
||||
(function () {
|
||||
var API = "/api/broker/user_allowlist";
|
||||
var rows = document.getElementById("rows");
|
||||
var pageError = document.getElementById("page-error");
|
||||
var modal = document.getElementById("modal");
|
||||
var form = document.getElementById("add-form");
|
||||
var userInput = document.getElementById("user_id");
|
||||
var emailInput = document.getElementById("email");
|
||||
var addError = document.getElementById("add-error");
|
||||
var submitButton = document.getElementById("add-submit");
|
||||
|
||||
function showPageError(message) {
|
||||
pageError.textContent = message;
|
||||
pageError.hidden = false;
|
||||
}
|
||||
|
||||
function clearPageError() {
|
||||
pageError.hidden = true;
|
||||
pageError.textContent = "";
|
||||
}
|
||||
|
||||
function messageRow(text) {
|
||||
rows.replaceChildren();
|
||||
var tr = document.createElement("tr");
|
||||
var td = document.createElement("td");
|
||||
td.colSpan = 6;
|
||||
td.className = "table-empty";
|
||||
td.textContent = text;
|
||||
tr.appendChild(td);
|
||||
rows.appendChild(tr);
|
||||
}
|
||||
|
||||
function textCell(value, mono) {
|
||||
var td = document.createElement("td");
|
||||
if (mono) {
|
||||
td.className = "mono";
|
||||
}
|
||||
td.textContent = value === null || value === undefined || value === "" ? "—" : value;
|
||||
return td;
|
||||
}
|
||||
|
||||
function statusBadge(status) {
|
||||
var span = document.createElement("span");
|
||||
span.className = "badge " + (status === "active" ? "badge-active" : "badge-revoked");
|
||||
span.textContent = status;
|
||||
return span;
|
||||
}
|
||||
|
||||
function actionButton(label, className, disabled, handler) {
|
||||
var button = document.createElement("button");
|
||||
button.type = "button";
|
||||
button.className = "btn " + className;
|
||||
button.textContent = label;
|
||||
button.disabled = disabled;
|
||||
button.addEventListener("click", handler);
|
||||
return button;
|
||||
}
|
||||
|
||||
function renderRow(row) {
|
||||
var tr = document.createElement("tr");
|
||||
tr.appendChild(textCell(row.user_id, true));
|
||||
tr.appendChild(textCell(row.email));
|
||||
var statusCell = document.createElement("td");
|
||||
statusCell.appendChild(statusBadge(row.status));
|
||||
tr.appendChild(statusCell);
|
||||
tr.appendChild(textCell(row.created_at));
|
||||
tr.appendChild(textCell(row.revoked_at));
|
||||
var actionsCell = document.createElement("td");
|
||||
var actions = document.createElement("div");
|
||||
actions.className = "cell-actions";
|
||||
var key = encodeURIComponent(row.user_id);
|
||||
actions.appendChild(actionButton("Revoke", "btn-danger", row.status !== "active", function () {
|
||||
rowAction("DELETE", API + "/" + key, "Revoke allowlisted user " + row.user_id + "?", "Revoke failed", row.user_id);
|
||||
}));
|
||||
actions.appendChild(actionButton("Reinstate", "btn-subtle", row.status === "active", function () {
|
||||
rowAction("POST", API + "/" + key + "/reinstate", "Reinstate allowlisted user " + row.user_id + "?", "Reinstate failed", row.user_id);
|
||||
}));
|
||||
actionsCell.appendChild(actions);
|
||||
tr.appendChild(actionsCell);
|
||||
return tr;
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
messageRow("Loading allowlist…");
|
||||
var result = await api(API);
|
||||
if (!result.ok) {
|
||||
messageRow(result.status === 0 ? "Broker unreachable." : "Could not load allowlist.");
|
||||
return;
|
||||
}
|
||||
var list = (result.data && result.data.allowlist) || [];
|
||||
if (!list.length) {
|
||||
messageRow("No allowlisted users yet.");
|
||||
return;
|
||||
}
|
||||
rows.replaceChildren();
|
||||
list.forEach(function (row) {
|
||||
rows.appendChild(renderRow(row));
|
||||
});
|
||||
}
|
||||
|
||||
async function rowAction(method, path, question, label, userId) {
|
||||
if (!confirm(question)) {
|
||||
return;
|
||||
}
|
||||
clearPageError();
|
||||
var result = await api(path, {method: method});
|
||||
if (!result.ok) {
|
||||
if (result.status === 0) {
|
||||
showPageError(label + ": broker unreachable.");
|
||||
} else {
|
||||
var detail = result.data && result.data.message ? ": " + result.data.message : ".";
|
||||
showPageError(label + " " + userId + detail);
|
||||
}
|
||||
}
|
||||
await refresh();
|
||||
}
|
||||
|
||||
function openModal() {
|
||||
form.reset();
|
||||
addError.hidden = true;
|
||||
addError.textContent = "";
|
||||
submitButton.disabled = false;
|
||||
modal.hidden = false;
|
||||
userInput.focus();
|
||||
}
|
||||
|
||||
function closeModal() {
|
||||
modal.hidden = true;
|
||||
}
|
||||
|
||||
form.addEventListener("submit", async function (event) {
|
||||
event.preventDefault();
|
||||
addError.hidden = true;
|
||||
addError.textContent = "";
|
||||
submitButton.disabled = true;
|
||||
var payload = {user_id: userInput.value.trim()};
|
||||
var email = emailInput.value.trim();
|
||||
if (email) {
|
||||
payload.email = email;
|
||||
}
|
||||
var result = await api(API, {method: "POST", body: payload});
|
||||
submitButton.disabled = false;
|
||||
if (result.ok) {
|
||||
closeModal();
|
||||
clearPageError();
|
||||
await refresh();
|
||||
return;
|
||||
}
|
||||
if (result.status === 0) {
|
||||
addError.textContent = "Broker unreachable.";
|
||||
} else if (result.data && result.data.message) {
|
||||
addError.textContent = result.data.message;
|
||||
} else {
|
||||
addError.textContent = "Could not add user.";
|
||||
}
|
||||
addError.hidden = false;
|
||||
await refresh();
|
||||
});
|
||||
|
||||
document.getElementById("add-open").addEventListener("click", openModal);
|
||||
document.getElementById("add-cancel").addEventListener("click", closeModal);
|
||||
modal.addEventListener("click", function (event) {
|
||||
if (event.target === modal) {
|
||||
closeModal();
|
||||
}
|
||||
});
|
||||
document.addEventListener("keydown", function (event) {
|
||||
if (event.key === "Escape" && !modal.hidden) {
|
||||
closeModal();
|
||||
}
|
||||
});
|
||||
refresh();
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,25 @@
|
||||
# Reference Tools
|
||||
|
||||
Templates for Open WebUI tool authors wiring tools to the CoreCI Broker.
|
||||
They contain no secrets and no live-infrastructure client.
|
||||
|
||||
- `proxmox_readonly_reference.py` - the full broker flow (spec section 11):
|
||||
first call registers and persists agent state, then fetches
|
||||
`coreci/proxmox`; later calls load the persisted token via `get_state`.
|
||||
- `minimal_auth_showcase.py` - the minimal pattern: authenticate with the
|
||||
broker, then use a second tool.
|
||||
|
||||
The flow both templates follow:
|
||||
|
||||
1. The CEO allowlists the tool's app UUID and user id in the broker UI.
|
||||
2. Fetch the client from `/static/coreci_broker_client.py` and import it.
|
||||
3. `ensure_registered(state)` registers (idempotent) and merges the token
|
||||
into your state dict; `put_state(state)` persists it as the tool's
|
||||
only durable state channel.
|
||||
4. `read_secret("coreci/<agent_id>", state)` fetches the credential; on a
|
||||
401 the client re-registers and retries once.
|
||||
|
||||
Downstream calls default to a safe dry-run with no live credential use;
|
||||
replace the dry-run block with real calls in your own tool.
|
||||
|
||||
Client module: `coreci_broker_client.py` (stdlib urllib only, zero deps).
|
||||
@@ -0,0 +1,45 @@
|
||||
"""title: Minimal Auth Showcase [CoreCI]
|
||||
author: coreci
|
||||
description: Minimal Open WebUI tool template showing the CoreCI Broker pattern - authenticate with the broker first, then use a second tool.
|
||||
version: 0.1.0
|
||||
requirements: []
|
||||
licence: MIT
|
||||
"""
|
||||
import coreci_broker_client
|
||||
|
||||
|
||||
class Tools:
|
||||
class Valves:
|
||||
base_url = ""
|
||||
agent_id = ""
|
||||
app_uuid = ""
|
||||
user_id = ""
|
||||
|
||||
def __init__(self):
|
||||
self.valves = self.Valves()
|
||||
self._state = {}
|
||||
|
||||
def _client(self):
|
||||
return coreci_broker_client.BrokerClient(
|
||||
self.valves.base_url, self.valves.agent_id,
|
||||
self.valves.app_uuid, self.valves.user_id, ["coreci"],
|
||||
)
|
||||
|
||||
def demo(self, context=None) -> dict:
|
||||
"""Demonstrate broker auth before any downstream tool call.
|
||||
|
||||
Ensures the agent is registered with the broker (idempotent), then
|
||||
reads the agent's own secret through the client: the credential
|
||||
fetch itself proves the auth path works before any second tool
|
||||
would use the credential. The secret material never appears in the
|
||||
returned marker.
|
||||
|
||||
:param context: optional Open WebUI call context, unused
|
||||
:return: marker dict with registered and secret_read flags
|
||||
"""
|
||||
client = self._client()
|
||||
client.ensure_registered(self._state)
|
||||
client.read_secret("coreci/%s" % self.valves.agent_id, self._state)
|
||||
return {"registered": True, "secret_read": True,
|
||||
"path": "coreci/%s" % self.valves.agent_id,
|
||||
"note": "auth first, then use"}
|
||||
@@ -0,0 +1,104 @@
|
||||
"""title: Proxmox Read-Only Reference [CoreCI]
|
||||
author: coreci
|
||||
description: Open WebUI tool template for the CoreCI Broker flow (spec section 11) - register, persist agent state, read coreci/proxmox, then act; downstream Proxmox calls default to a safe dry-run.
|
||||
version: 0.1.0
|
||||
requirements: []
|
||||
licence: MIT
|
||||
"""
|
||||
import datetime
|
||||
|
||||
import coreci_broker_client
|
||||
|
||||
SECRET_PATH = "coreci/proxmox"
|
||||
DRY_RUN_NOTE = (
|
||||
"reference dry-run: the broker credential was fetched but stays unused; "
|
||||
"replace this block with a real read-only Proxmox call to go live"
|
||||
)
|
||||
|
||||
|
||||
class Tools:
|
||||
class Valves:
|
||||
base_url = ""
|
||||
agent_id = ""
|
||||
app_uuid = ""
|
||||
user_id = ""
|
||||
|
||||
def __init__(self):
|
||||
self.valves = self.Valves()
|
||||
self._state = {}
|
||||
|
||||
def _client(self):
|
||||
return coreci_broker_client.BrokerClient(
|
||||
self.valves.base_url, self.valves.agent_id,
|
||||
self.valves.app_uuid, self.valves.user_id, ["proxmox", "readonly"],
|
||||
)
|
||||
|
||||
def _persist(self, client):
|
||||
now = datetime.datetime.now(datetime.timezone.utc).isoformat()
|
||||
self._state.setdefault("first_registered_at", now)
|
||||
self._state["last_seen_at"] = now
|
||||
client.put_state(self._state)
|
||||
|
||||
def _load_persisted(self, client):
|
||||
document = client.get_state(self._state)
|
||||
if document is not None:
|
||||
for key in document:
|
||||
self._state.setdefault(key, document[key])
|
||||
|
||||
def _credential(self):
|
||||
client = self._client()
|
||||
first_call = self._state.get("current_token") is None
|
||||
fingerprint = (self._state.get("current_token"),
|
||||
self._state.get("token_expires_at"))
|
||||
if first_call:
|
||||
client.ensure_registered(self._state)
|
||||
self._persist(client)
|
||||
else:
|
||||
self._load_persisted(client)
|
||||
document = client.read_secret(SECRET_PATH, self._state)
|
||||
current = (self._state.get("current_token"),
|
||||
self._state.get("token_expires_at"))
|
||||
if not first_call and current != fingerprint:
|
||||
self._persist(client)
|
||||
return document
|
||||
|
||||
def _dry_run(self, action, document):
|
||||
return {
|
||||
"dry_run": True,
|
||||
"action": action,
|
||||
"credential_present": isinstance(document, dict) and "data" in document,
|
||||
"note": DRY_RUN_NOTE,
|
||||
}
|
||||
|
||||
def list_nodes(self, context=None) -> dict:
|
||||
"""List the Proxmox cluster nodes as a safe dry-run by default.
|
||||
|
||||
First call registers with the broker (idempotent) and persists the
|
||||
agent state; subsequent calls load the persisted token and reuse
|
||||
it. The credential is fetched from coreci/proxmox before any
|
||||
downstream action, and on a 401 the client re-registers with a
|
||||
single retry after which the new token is persisted again. The
|
||||
reference template then returns a dry-run marker and performs no
|
||||
live Proxmox call: replace the dry-run block with a real read-only
|
||||
API call when wiring live infrastructure; this template ships no
|
||||
live-credential use path.
|
||||
|
||||
:param context: optional Open WebUI call context, unused
|
||||
:return: dry-run response dict with credential presence
|
||||
"""
|
||||
return self._dry_run("list_nodes", self._credential())
|
||||
|
||||
def vm_status(self, vm_id: str, context=None) -> dict:
|
||||
"""Report one Proxmox VM status as a safe dry-run by default.
|
||||
|
||||
Follows the same broker flow as list_nodes: register and persist
|
||||
the agent state on the first call, load the persisted token on
|
||||
later calls, fetch the coreci/proxmox credential, then act. The
|
||||
dry-run block is the default and no live Proxmox client is wired
|
||||
in - replace it with a real read-only status call for live use.
|
||||
|
||||
:param vm_id: Proxmox VM identifier to look up
|
||||
:param context: optional Open WebUI call context, unused
|
||||
:return: dry-run response dict with credential presence
|
||||
"""
|
||||
return self._dry_run("vm_status", self._credential())
|
||||
@@ -0,0 +1,345 @@
|
||||
import ast
|
||||
import base64
|
||||
import contextlib
|
||||
import datetime
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
|
||||
BROKER_SOURCE_PATH = os.path.join(REPO_ROOT, "coreci_broker.py")
|
||||
CLIENT_SOURCE_PATH = os.path.join(REPO_ROOT, "coreci_broker_client.py")
|
||||
CLIENT_STATIC_PATH = "/static/coreci_broker_client.py"
|
||||
LOGIN_PATH = "/api/broker/auth/login"
|
||||
LOGOUT_PATH = "/api/broker/auth/logout"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
AUDIT_PATH = "/api/broker/audit"
|
||||
SECRETS_PREFIX = "/api/broker/secrets/"
|
||||
PROXMOX_PATH = "coreci/proxmox"
|
||||
GITEA_PATH = "coreci/gitea"
|
||||
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
|
||||
GITEA_SECRET = {"url": "https://git.coreci.dev", "token": "git-tok"}
|
||||
|
||||
AC_PIN_IDS = {
|
||||
"AC1": "test_acceptance_p2.AcceptanceP2Tests.test_ac1_first_boot_printed_pass_login_via_ui_login_page",
|
||||
"AC2": "test_acceptance_p2.AcceptanceP2Tests.test_ac2_app_registration_lifecycle_via_session_cookie",
|
||||
"AC3": "test_acceptance_p2.AcceptanceP2Tests.test_ac3_user_allowlist_lifecycle_via_session_cookie",
|
||||
"AC4": "test_acceptance_p3.AcceptanceP3Tests.test_ac4_derivation_identical_both_sides_stored_as_slow_hash",
|
||||
"AC5": "test_acceptance_p4.AcceptanceP4Tests.test_ac5_valid_token_reads_secret_tampered_token_is_401",
|
||||
"AC6": "test_acceptance_p4.AcceptanceP4Tests.test_ac6_cross_path_read_is_403_with_denied_audit",
|
||||
"AC7": "test_acceptance_p3.AcceptanceP3Tests.test_ac7_revoked_or_unknown_uuid_denied_existing_token_honored",
|
||||
"AC8": "test_acceptance_p3.AcceptanceP3Tests.test_ac8_unknown_or_revoked_user_denied",
|
||||
"AC9": "test_acceptance_p2.AcceptanceP2Tests.test_ac9_login_and_logout_audit_lines_with_null_identity",
|
||||
"AC10": "test_acceptance_p3.AcceptanceP3Tests.test_ac10_expiry_401_reregister_refreshes_expiry_and_audits",
|
||||
"AC11": "test_acceptance_full.AcceptanceFullTests.test_ac11_fresh_boot_state_files_0600_umask_independent",
|
||||
"AC12": "test_acceptance_p3.AcceptanceP3Tests.test_ac12_state_roundtrip_with_identity_enforcement",
|
||||
"AC13": "test_acceptance_full.AcceptanceFullTests.test_ac13_client_module_fetch_byte_identical_importable_standalone",
|
||||
"AC14": "test_acceptance_full.AcceptanceFullTests.test_ac14_coreci_broker_line_count_under_1400",
|
||||
}
|
||||
|
||||
|
||||
def _basic_header(password):
|
||||
raw = "%s:%s" % (ADMIN_USER, password)
|
||||
return {"Authorization": "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")}
|
||||
|
||||
|
||||
class AcceptanceFullTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.creds_path = os.path.join(self.tmp, "creds.json")
|
||||
with open(self.creds_path, "w", encoding="utf-8") as handle:
|
||||
handle.write(json.dumps({PROXMOX_PATH: PROXMOX_SECRET, GITEA_PATH: GITEA_SECRET}))
|
||||
|
||||
def _env(self, root=None, extra=None):
|
||||
root = self.tmp if root is None else root
|
||||
env = {
|
||||
"BROKER_ENV_PATH": os.path.join(root, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(root, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(root, "broker.audit.log"),
|
||||
"BROKER_CREDS_SOURCE": "file",
|
||||
"BROKER_CREDS_FILE": self.creds_path,
|
||||
"BROKER_UI_ROOT": os.path.join(REPO_ROOT, "coreci_broker_ui"),
|
||||
"BROKER_STATIC_ROOT": REPO_ROOT,
|
||||
}
|
||||
env.update(extra or {})
|
||||
return env
|
||||
|
||||
def _boot(self, env):
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
broker = coreci_broker.boot(dict(env))
|
||||
self.addCleanup(broker.audit.close)
|
||||
self.addCleanup(broker.store.close)
|
||||
return broker, buffer.getvalue()
|
||||
|
||||
def _serve(self, broker):
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
self.addCleanup(server.server_close)
|
||||
self.addCleanup(server.shutdown)
|
||||
return "http://127.0.0.1:%d" % server.server_address[1]
|
||||
|
||||
def _printed_pass(self, output):
|
||||
printed = [
|
||||
line.split("=", 1)[1]
|
||||
for line in output.splitlines()
|
||||
if line.startswith("BROKER_ADMIN_PASS=")
|
||||
]
|
||||
self.assertEqual(1, len(printed))
|
||||
return printed[0]
|
||||
|
||||
def _request(self, base, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
return response.status, response.read(), response.headers
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read(), error.headers
|
||||
|
||||
def _request_json(self, base, method, path, payload=None, headers=None):
|
||||
status, body, response_headers = self._request(base, method, path, payload, headers)
|
||||
return status, json.loads(body.decode("utf-8")), response_headers
|
||||
|
||||
def _assert_null_identity(self, entry):
|
||||
self.assertIsNone(entry["user_id"])
|
||||
self.assertIsNone(entry["app_registration_uuid"])
|
||||
self.assertIsNone(entry["agent_id"])
|
||||
|
||||
def _assert_tool_identity(self, entry, app_uuid):
|
||||
self.assertEqual("ceo", entry["user_id"])
|
||||
self.assertEqual(app_uuid, entry["app_registration_uuid"])
|
||||
self.assertEqual("proxmox", entry["agent_id"])
|
||||
|
||||
def _assert_state_files_0600(self, env):
|
||||
for key in ("BROKER_DB_PATH", "BROKER_AUDIT_LOG_PATH", "BROKER_ENV_PATH"):
|
||||
path = env[key]
|
||||
self.assertTrue(os.path.isfile(path), path)
|
||||
self.assertEqual(0o600, os.stat(path).st_mode & 0o777, path)
|
||||
|
||||
def test_ac11_fresh_boot_state_files_0600_umask_independent(self):
|
||||
env = self._env()
|
||||
self._boot(env)
|
||||
self._assert_state_files_0600(env)
|
||||
other = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(other.cleanup)
|
||||
other_env = self._env(root=other.name)
|
||||
previous_umask = os.umask(0o777)
|
||||
try:
|
||||
probe = os.path.join(other.name, "umask-probe")
|
||||
os.close(os.open(probe, os.O_CREAT | os.O_WRONLY, 0o666))
|
||||
self.assertEqual(0o000, os.stat(probe).st_mode & 0o777)
|
||||
self._boot(other_env)
|
||||
finally:
|
||||
os.umask(previous_umask)
|
||||
self._assert_state_files_0600(other_env)
|
||||
|
||||
def test_ac13_client_module_fetch_byte_identical_importable_standalone(self):
|
||||
broker, _ = self._boot(self._env())
|
||||
base = self._serve(broker)
|
||||
with open(CLIENT_SOURCE_PATH, "rb") as handle:
|
||||
expected = handle.read()
|
||||
with urllib.request.urlopen(base + CLIENT_STATIC_PATH, timeout=10) as response:
|
||||
self.assertEqual(200, response.status)
|
||||
fetched = response.read()
|
||||
self.assertEqual(expected, fetched)
|
||||
imports = set()
|
||||
for node in ast.walk(ast.parse(fetched.decode("utf-8"))):
|
||||
if isinstance(node, ast.Import):
|
||||
imports.update(alias.name for alias in node.names)
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
imports.add(node.module or "")
|
||||
self.assertTrue(imports)
|
||||
for name in imports:
|
||||
self.assertIn(name.split(".")[0], sys.stdlib_module_names, name)
|
||||
copy_path = os.path.join(self.tmp, "fetched_coreci_broker_client.py")
|
||||
with open(copy_path, "wb") as handle:
|
||||
handle.write(fetched)
|
||||
spec = importlib.util.spec_from_file_location("full_fetched_client", copy_path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
self.assertTrue(callable(module.BrokerClient))
|
||||
self.assertTrue(callable(module.fetch_client_module))
|
||||
for method in ("ensure_registered", "get_state", "put_state", "read_secret"):
|
||||
self.assertTrue(callable(getattr(module.BrokerClient, method)), method)
|
||||
|
||||
def test_ac14_coreci_broker_line_count_under_1400(self):
|
||||
with open(BROKER_SOURCE_PATH, "r", encoding="utf-8") as handle:
|
||||
line_count = sum(1 for _ in handle)
|
||||
self.assertLess(line_count, 1400)
|
||||
|
||||
def test_milestone_story_full_happy_path_consolidates_ac1_ac10_ac12(self):
|
||||
env = self._env(extra={"BROKER_TOKEN_TTL": "1"})
|
||||
first, output = self._boot(env)
|
||||
printed_pass = self._printed_pass(output)
|
||||
self.assertGreaterEqual(len(printed_pass), 32)
|
||||
self.assertEqual(1, output.count(printed_pass))
|
||||
pepper = None
|
||||
with open(env["BROKER_ENV_PATH"], "r", encoding="utf-8") as handle:
|
||||
for line in handle:
|
||||
if line.startswith("BROKER_INSTANCE_PEPPER="):
|
||||
pepper = line.split("=", 1)[1].strip()
|
||||
self.assertIsNotNone(pepper)
|
||||
self.assertNotIn(pepper, output)
|
||||
second, silent = self._boot(env)
|
||||
self.assertEqual("", silent)
|
||||
second.audit.close()
|
||||
second.store.close()
|
||||
base = self._serve(first)
|
||||
status, body, headers = self._request(base, "GET", "/ui/login")
|
||||
self.assertEqual(200, status)
|
||||
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
|
||||
self.assertIn('name="password"', body.decode("utf-8"))
|
||||
status, body, response_headers = self._request_json(
|
||||
base, "POST", LOGIN_PATH, {"username": ADMIN_USER, "password": printed_pass}
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
|
||||
raw_cookie = response_headers.get("Set-Cookie")
|
||||
self.assertTrue(raw_cookie)
|
||||
self.assertTrue(raw_cookie.startswith("broker_session="))
|
||||
self.assertIn("HttpOnly", raw_cookie)
|
||||
self.assertIn("SameSite=Lax", raw_cookie)
|
||||
cookie = {"Cookie": raw_cookie.split(";", 1)[0]}
|
||||
app_uuid = str(uuidlib.uuid4())
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", APPS_PATH,
|
||||
{"uuid": app_uuid, "name": "proxmox-tool"}, headers=cookie,
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", USERS_PATH,
|
||||
{"user_id": "ceo", "email": "ceo@coreci.dev"}, headers=cookie,
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
self.assertEqual({"user_id": "ceo", "status": "active"}, body)
|
||||
with urllib.request.urlopen(base + CLIENT_STATIC_PATH, timeout=10) as response:
|
||||
self.assertEqual(200, response.status)
|
||||
client_source = response.read()
|
||||
with open(CLIENT_SOURCE_PATH, "rb") as handle:
|
||||
self.assertEqual(handle.read(), client_source)
|
||||
copy_path = os.path.join(self.tmp, "story_coreci_broker_client.py")
|
||||
with open(copy_path, "wb") as handle:
|
||||
handle.write(client_source)
|
||||
spec = importlib.util.spec_from_file_location("full_story_client", copy_path)
|
||||
client_module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(client_module)
|
||||
client = client_module.BrokerClient(base, "proxmox", app_uuid, "ceo", ["coreci"])
|
||||
state = {}
|
||||
client.ensure_registered(state)
|
||||
self.assertEqual(44, len(state["current_token"]))
|
||||
self.assertTrue(state["current_token"].isascii())
|
||||
self.assertIn("pepper", state)
|
||||
self.assertEqual("proxmox", state["agent_id"])
|
||||
self.assertEqual(app_uuid, state["app_registration_uuid"])
|
||||
self.assertEqual("ceo", state["user_id"])
|
||||
self.assertTrue(client.put_state(state))
|
||||
snapshot = dict(state)
|
||||
self.assertEqual(snapshot, client.get_state(state))
|
||||
document = client.read_secret(PROXMOX_PATH, state)
|
||||
self.assertEqual({"path", "data", "version", "updated_at"}, set(document))
|
||||
self.assertEqual(PROXMOX_PATH, document["path"])
|
||||
self.assertEqual(PROXMOX_SECRET, document["data"])
|
||||
with self.assertRaises(client_module.BrokerError) as caught:
|
||||
client.read_secret(GITEA_PATH, state)
|
||||
self.assertEqual(403, caught.exception.status)
|
||||
expired_at = state["token_expires_at"]
|
||||
time.sleep(1.2)
|
||||
document = client.read_secret(PROXMOX_PATH, state)
|
||||
self.assertEqual(PROXMOX_SECRET, document["data"])
|
||||
self.assertGreater(
|
||||
datetime.datetime.fromisoformat(state["token_expires_at"]),
|
||||
datetime.datetime.fromisoformat(expired_at),
|
||||
)
|
||||
status, body, _ = self._request_json(base, "POST", LOGOUT_PATH, headers=cookie)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
status, body, _ = self._request_json(
|
||||
base, "GET", AUDIT_PATH + "?limit=1000", headers=_basic_header(printed_pass)
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertIsNone(body["next_since"])
|
||||
entries = body["entries"]
|
||||
|
||||
def select(action, result=None):
|
||||
return [
|
||||
entry for entry in entries
|
||||
if entry["action"] == action and (result is None or entry["result"] == result)
|
||||
]
|
||||
|
||||
logins = select("login", "ok")
|
||||
self.assertEqual(1, len(logins))
|
||||
self.assertEqual(LOGIN_PATH, logins[0]["path"])
|
||||
self._assert_null_identity(logins[0])
|
||||
add_apps = select("add_app", "ok")
|
||||
self.assertEqual(1, len(add_apps))
|
||||
self.assertEqual(APPS_PATH + "/" + app_uuid, add_apps[0]["path"])
|
||||
self._assert_null_identity(add_apps[0])
|
||||
add_users = select("add_user", "ok")
|
||||
self.assertEqual(1, len(add_users))
|
||||
self.assertEqual(USERS_PATH + "/ceo", add_users[0]["path"])
|
||||
self._assert_null_identity(add_users[0])
|
||||
registers = select("register", "ok")
|
||||
self.assertGreaterEqual(len(registers), 2)
|
||||
for entry in registers:
|
||||
self.assertEqual(REGISTER_PATH, entry["path"])
|
||||
self._assert_tool_identity(entry, app_uuid)
|
||||
ok_reads = select("read", "ok")
|
||||
self.assertGreaterEqual(len(ok_reads), 2)
|
||||
for entry in ok_reads:
|
||||
self.assertEqual(PROXMOX_PATH, entry["path"])
|
||||
self._assert_tool_identity(entry, app_uuid)
|
||||
denied_reads = select("read", "denied")
|
||||
self.assertEqual(1, len(denied_reads))
|
||||
self.assertEqual(GITEA_PATH, denied_reads[0]["path"])
|
||||
self._assert_tool_identity(denied_reads[0], app_uuid)
|
||||
auth_denied = [
|
||||
entry for entry in select("denied", "denied")
|
||||
if entry["path"] == SECRETS_PREFIX + PROXMOX_PATH
|
||||
]
|
||||
self.assertGreaterEqual(len(auth_denied), 1)
|
||||
self.assertEqual(app_uuid, auth_denied[0]["app_registration_uuid"])
|
||||
self.assertIsNone(auth_denied[0]["user_id"])
|
||||
self.assertIsNone(auth_denied[0]["agent_id"])
|
||||
logouts = select("logout", "ok")
|
||||
self.assertEqual(1, len(logouts))
|
||||
self.assertEqual(LOGOUT_PATH, logouts[0]["path"])
|
||||
self._assert_null_identity(logouts[0])
|
||||
|
||||
def test_coverage_tally_pins_ac1_through_ac14_across_suite(self):
|
||||
self.assertEqual(14, len(AC_PIN_IDS))
|
||||
loader = unittest.TestLoader()
|
||||
suite = loader.discover(os.path.dirname(os.path.abspath(__file__)))
|
||||
stack, ids = [suite], set()
|
||||
while stack:
|
||||
item = stack.pop()
|
||||
if isinstance(item, unittest.TestSuite):
|
||||
stack.extend(item)
|
||||
else:
|
||||
ids.add(item.id())
|
||||
for ac in sorted(AC_PIN_IDS):
|
||||
with self.subTest(ac=ac):
|
||||
self.assertIn(AC_PIN_IDS[ac], ids)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,114 @@
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.request
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
|
||||
class AcceptanceP1Tests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
}
|
||||
|
||||
def _boot(self):
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
broker = coreci_broker.boot(dict(self.env))
|
||||
return broker, buffer.getvalue()
|
||||
|
||||
def test_ac14_coreci_broker_line_count_under_1400(self):
|
||||
source_path = os.path.join(os.path.dirname(__file__), "..", "coreci_broker.py")
|
||||
with open(source_path, "r", encoding="utf-8") as handle:
|
||||
line_count = sum(1 for _ in handle)
|
||||
self.assertLess(line_count, 1400)
|
||||
|
||||
def test_first_boot_prints_pass_once_second_boot_silent_with_0600_state(self):
|
||||
first = io.StringIO()
|
||||
with contextlib.redirect_stdout(first):
|
||||
broker = coreci_broker.boot(dict(self.env))
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
db_path = self.env["BROKER_DB_PATH"]
|
||||
audit_path = self.env["BROKER_AUDIT_LOG_PATH"]
|
||||
env_path = self.env["BROKER_ENV_PATH"]
|
||||
self.assertTrue(os.path.isfile(db_path))
|
||||
self.assertTrue(os.path.isfile(audit_path))
|
||||
self.assertTrue(os.path.isfile(env_path))
|
||||
self.assertEqual(0o600, os.stat(db_path).st_mode & 0o777)
|
||||
self.assertEqual(0o600, os.stat(audit_path).st_mode & 0o777)
|
||||
admin_pass = broker.config["BROKER_ADMIN_PASS"]
|
||||
pepper = broker.config["BROKER_INSTANCE_PEPPER"]
|
||||
self.assertIn(admin_pass, first.getvalue())
|
||||
self.assertEqual(1, first.getvalue().count(admin_pass))
|
||||
self.assertNotIn(pepper, first.getvalue())
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
broker.audit.close()
|
||||
broker.store.close()
|
||||
second = io.StringIO()
|
||||
with contextlib.redirect_stdout(second):
|
||||
rebooted = coreci_broker.boot(dict(self.env))
|
||||
self.assertEqual("", second.getvalue())
|
||||
rebooted.audit.close()
|
||||
rebooted.store.close()
|
||||
|
||||
def test_restart_preserves_allowlist_row_and_audit_history(self):
|
||||
broker, _ = self._boot()
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
user_id = "acceptance-restart-user"
|
||||
broker.store.conn.execute(
|
||||
"INSERT INTO user_allowlist (user_id, email, status, created_at) VALUES (?, ?, ?, ?)",
|
||||
(user_id, "ceo@coreci.dev", "active", coreci_broker._utc_now_iso()),
|
||||
)
|
||||
broker.store.conn.commit()
|
||||
broker.audit.write("add_user", user_id=user_id)
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
broker.audit.close()
|
||||
broker.store.close()
|
||||
rebooted, _ = self._boot()
|
||||
self.addCleanup(rebooted.audit.close)
|
||||
self.addCleanup(rebooted.store.close)
|
||||
row = rebooted.store.conn.execute(
|
||||
"SELECT email, status FROM user_allowlist WHERE user_id = ?", (user_id,)
|
||||
).fetchone()
|
||||
self.assertEqual(("ceo@coreci.dev", "active"), row)
|
||||
audit_path = self.env["BROKER_AUDIT_LOG_PATH"]
|
||||
with open(audit_path, "r", encoding="utf-8") as handle:
|
||||
lines = [line for line in handle if line.strip()]
|
||||
self.assertEqual(1, len(lines))
|
||||
entry = json.loads(lines[0])
|
||||
self.assertEqual("add_user", entry["action"])
|
||||
self.assertEqual(user_id, entry["user_id"])
|
||||
self.assertEqual(0o600, os.stat(audit_path).st_mode & 0o777)
|
||||
|
||||
def test_healthz_end_to_end(self):
|
||||
broker, _ = self._boot()
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
self.addCleanup(broker.audit.close)
|
||||
self.addCleanup(broker.store.close)
|
||||
self.addCleanup(server.server_close)
|
||||
self.addCleanup(server.shutdown)
|
||||
port = server.server_address[1]
|
||||
with urllib.request.urlopen(
|
||||
"http://127.0.0.1:%d/healthz" % port, timeout=5
|
||||
) as response:
|
||||
self.assertEqual(200, response.status)
|
||||
payload = json.loads(response.read().decode("utf-8"))
|
||||
self.assertEqual({"ok": True}, payload)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,241 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
LOGIN_PATH = "/api/broker/auth/login"
|
||||
LOGOUT_PATH = "/api/broker/auth/logout"
|
||||
WHOAMI_PATH = "/api/broker/auth/whoami"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
AUDIT_PATH = "/api/broker/audit"
|
||||
|
||||
|
||||
class AcceptanceP2Tests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
}
|
||||
|
||||
def _boot_served(self, extra=None):
|
||||
env = dict(self.env)
|
||||
env.update(extra or {})
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
broker = coreci_broker.boot(env)
|
||||
self.addCleanup(broker.audit.close)
|
||||
self.addCleanup(broker.store.close)
|
||||
printed = [
|
||||
line.split("=", 1)[1]
|
||||
for line in buffer.getvalue().splitlines()
|
||||
if line.startswith("BROKER_ADMIN_PASS=")
|
||||
]
|
||||
self.assertEqual(1, len(printed))
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
self.addCleanup(server.server_close)
|
||||
self.addCleanup(server.shutdown)
|
||||
return "http://127.0.0.1:%d" % server.server_address[1], printed[0]
|
||||
|
||||
def _request(self, base, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=5) as response:
|
||||
return response.status, response.read(), response.headers
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read(), error.headers
|
||||
|
||||
def _request_json(self, base, method, path, payload=None, headers=None):
|
||||
status, body, response_headers = self._request(
|
||||
base, method, path, payload, headers
|
||||
)
|
||||
return status, json.loads(body.decode("utf-8")), response_headers
|
||||
|
||||
def _get_page(self, base, path):
|
||||
status, body, headers = self._request(base, "GET", path)
|
||||
self.assertEqual(200, status)
|
||||
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
|
||||
return body.decode("utf-8")
|
||||
|
||||
def _basic(self, password):
|
||||
raw = "%s:%s" % (ADMIN_USER, password)
|
||||
encoded = base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
return {"Authorization": "Basic " + encoded}
|
||||
|
||||
def _login(self, base, password):
|
||||
status, body, headers = self._request_json(
|
||||
base, "POST", LOGIN_PATH, {"username": ADMIN_USER, "password": password}
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
|
||||
raw = headers.get("Set-Cookie")
|
||||
self.assertTrue(raw)
|
||||
self.assertTrue(raw.startswith("broker_session="))
|
||||
self.assertIn("HttpOnly", raw)
|
||||
self.assertIn("SameSite=Lax", raw)
|
||||
return {"Cookie": raw.split(";", 1)[0]}
|
||||
|
||||
def _find(self, rows, key, value):
|
||||
for row in rows:
|
||||
if row[key] == value:
|
||||
return row
|
||||
self.fail("no row with %s=%r" % (key, value))
|
||||
|
||||
def _ok_entries(self, entries):
|
||||
return [entry for entry in entries if entry["result"] == "ok"]
|
||||
|
||||
def _assert_null_identity(self, entries):
|
||||
self.assertTrue(entries)
|
||||
for entry in entries:
|
||||
self.assertIsNone(entry["user_id"])
|
||||
self.assertIsNone(entry["app_registration_uuid"])
|
||||
self.assertIsNone(entry["agent_id"])
|
||||
|
||||
def test_ac1_first_boot_printed_pass_login_via_ui_login_page(self):
|
||||
base, printed_pass = self._boot_served()
|
||||
self._get_page(base, "/ui/login")
|
||||
cookie = self._login(base, printed_pass)
|
||||
status, body, _ = self._request_json(
|
||||
base, "GET", WHOAMI_PATH, headers=cookie
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(ADMIN_USER, body["username"])
|
||||
|
||||
def test_ac2_app_registration_lifecycle_via_session_cookie(self):
|
||||
base, printed_pass = self._boot_served()
|
||||
cookie = self._login(base, printed_pass)
|
||||
app_uuid = str(uuidlib.uuid4())
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", APPS_PATH,
|
||||
{"uuid": app_uuid, "name": "proxmox-tool"}, headers=cookie,
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
|
||||
status, body, _ = self._request_json(base, "GET", APPS_PATH, headers=cookie)
|
||||
self.assertEqual(200, status)
|
||||
row = self._find(body["registrations"], "uuid", app_uuid)
|
||||
self.assertEqual("proxmox-tool", row["name"])
|
||||
self.assertEqual("active", row["status"])
|
||||
self._get_page(base, "/ui/app_registrations")
|
||||
status, body, _ = self._request_json(
|
||||
base, "DELETE", APPS_PATH + "/" + app_uuid, headers=cookie
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("revoked", body["status"])
|
||||
status, body, _ = self._request_json(base, "GET", APPS_PATH, headers=cookie)
|
||||
self.assertEqual(
|
||||
"revoked", self._find(body["registrations"], "uuid", app_uuid)["status"]
|
||||
)
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", APPS_PATH + "/" + app_uuid + "/reinstate", headers=cookie
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("active", body["status"])
|
||||
status, body, _ = self._request_json(base, "GET", APPS_PATH, headers=cookie)
|
||||
self.assertEqual(
|
||||
"active", self._find(body["registrations"], "uuid", app_uuid)["status"]
|
||||
)
|
||||
|
||||
def test_ac3_user_allowlist_lifecycle_via_session_cookie(self):
|
||||
base, printed_pass = self._boot_served()
|
||||
cookie = self._login(base, printed_pass)
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", USERS_PATH,
|
||||
{"user_id": "ceo", "email": "ceo@coreci.dev"}, headers=cookie,
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
self.assertEqual({"user_id": "ceo", "status": "active"}, body)
|
||||
status, body, _ = self._request_json(base, "GET", USERS_PATH, headers=cookie)
|
||||
self.assertEqual(200, status)
|
||||
row = self._find(body["allowlist"], "user_id", "ceo")
|
||||
self.assertEqual("ceo@coreci.dev", row["email"])
|
||||
self.assertEqual("active", row["status"])
|
||||
self._get_page(base, "/ui/user_allowlist")
|
||||
status, body, _ = self._request_json(
|
||||
base, "DELETE", USERS_PATH + "/ceo", headers=cookie
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("revoked", body["status"])
|
||||
status, body, _ = self._request_json(base, "GET", USERS_PATH, headers=cookie)
|
||||
self.assertEqual(
|
||||
"revoked", self._find(body["allowlist"], "user_id", "ceo")["status"]
|
||||
)
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", USERS_PATH + "/ceo/reinstate", headers=cookie
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("active", body["status"])
|
||||
status, body, _ = self._request_json(base, "GET", USERS_PATH, headers=cookie)
|
||||
self.assertEqual(
|
||||
"active", self._find(body["allowlist"], "user_id", "ceo")["status"]
|
||||
)
|
||||
|
||||
def test_ac9_login_and_logout_audit_lines_with_null_identity(self):
|
||||
base, printed_pass = self._boot_served()
|
||||
cookie = self._login(base, printed_pass)
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", LOGOUT_PATH, headers=cookie
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
admin = self._basic(printed_pass)
|
||||
status, body, _ = self._request_json(
|
||||
base, "GET", AUDIT_PATH + "?action=login", headers=admin
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
logins = self._ok_entries(body["entries"])
|
||||
self.assertEqual(["login"], [entry["action"] for entry in logins])
|
||||
self._assert_null_identity(logins)
|
||||
status, body, _ = self._request_json(
|
||||
base, "GET", AUDIT_PATH + "?action=logout", headers=admin
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
logouts = self._ok_entries(body["entries"])
|
||||
self.assertEqual(["logout"], [entry["action"] for entry in logouts])
|
||||
self._assert_null_identity(logouts)
|
||||
|
||||
def test_ui_pages_served_and_static_hygiene(self):
|
||||
base, printed_pass = self._boot_served()
|
||||
dashboard = self._get_page(base, "/ui/dashboard")
|
||||
self.assertIn("Dashboard", dashboard)
|
||||
audit_page = self._get_page(base, "/ui/audit")
|
||||
self.assertNotIn("export", audit_page.lower())
|
||||
login_page = self._get_page(base, "/ui/login")
|
||||
self.assertIn('name="username"', login_page)
|
||||
self.assertIn('name="password"', login_page)
|
||||
|
||||
def test_session_ttl_expiry_end_to_end(self):
|
||||
base, printed_pass = self._boot_served({"BROKER_SESSION_TTL": "1"})
|
||||
cookie = self._login(base, printed_pass)
|
||||
status, _, _ = self._request(base, "GET", WHOAMI_PATH, headers=cookie)
|
||||
self.assertEqual(200, status)
|
||||
time.sleep(1.2)
|
||||
status, body, _ = self._request_json(
|
||||
base, "GET", WHOAMI_PATH, headers=cookie
|
||||
)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(401, body["error"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,309 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import datetime
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
AUDIT_PATH = "/api/broker/audit"
|
||||
AGENT_PATH = "/api/broker/agents/%s"
|
||||
STATE_PATH = "/api/broker/agents/%s/state"
|
||||
AUTH_401_BODY = {"error": 401, "message": "invalid app credentials"}
|
||||
REGISTER_403_BODY = {"error": 403, "message": "registration not allowed"}
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
def _derive(pepper, app_uuid, user_id, agent_id, tags):
|
||||
material = "%s:%s:%s:%s:%s" % (
|
||||
pepper, app_uuid, user_id, agent_id, ",".join(sorted(tags))
|
||||
)
|
||||
digest = hashlib.sha256(material.encode("utf-8")).digest()
|
||||
return base64.b64encode(digest).decode("ascii")
|
||||
|
||||
|
||||
class AcceptanceP3Tests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
}
|
||||
self.broker, self.base = self._boot()
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
self.app_uuid = str(uuidlib.uuid4())
|
||||
self.user_id = "ceo"
|
||||
self._seed(self.base, self.app_uuid, self.user_id)
|
||||
|
||||
def _boot(self, extra=None):
|
||||
env = dict(self.env)
|
||||
env.update(extra or {})
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
broker = coreci_broker.boot(env)
|
||||
self.addCleanup(broker.audit.close)
|
||||
self.addCleanup(broker.store.close)
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
self.addCleanup(server.shutdown)
|
||||
self.addCleanup(server.server_close)
|
||||
return broker, "http://127.0.0.1:%d" % server.server_address[1]
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None, base=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(
|
||||
(base or self.base) + path, data=data, method=method
|
||||
)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
raw = response.read().decode("utf-8")
|
||||
return response.status, json.loads(raw) if raw else None
|
||||
except urllib.error.HTTPError as error:
|
||||
raw = error.read().decode("utf-8")
|
||||
return error.code, json.loads(raw) if raw else None
|
||||
|
||||
def _seed(self, base, app_uuid, user_id):
|
||||
status, body = self._request(
|
||||
"POST", APPS_PATH, {"uuid": app_uuid}, headers=self.admin, base=base
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
status, body = self._request(
|
||||
"POST", USERS_PATH, {"user_id": user_id}, headers=self.admin, base=base
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def _register(self, agent_id, tags, base=None, app_uuid=None, user_id=None):
|
||||
payload = {
|
||||
"agent_id": agent_id,
|
||||
"app_registration_uuid": app_uuid or self.app_uuid,
|
||||
"user_id": user_id or self.user_id,
|
||||
"tags": list(tags),
|
||||
}
|
||||
return self._request("POST", REGISTER_PATH, payload, base=base)
|
||||
|
||||
def _app_header(self, token, app_uuid=None):
|
||||
return {"Authorization": "App %s %s" % (app_uuid or self.app_uuid, token)}
|
||||
|
||||
def _audit_entries(self, base, query):
|
||||
status, body = self._request(
|
||||
"GET", AUDIT_PATH + query, headers=self.admin, base=base
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
return body["entries"]
|
||||
|
||||
def test_ac4_derivation_identical_both_sides_stored_as_slow_hash(self):
|
||||
agent_id = "px-1"
|
||||
tags = ["zeta", "alpha"]
|
||||
status, body = self._register(agent_id, tags)
|
||||
self.assertEqual(201, status)
|
||||
self.assertTrue(body["created"])
|
||||
token = body["token"]
|
||||
pepper = self.broker.store.get_config("BROKER_INSTANCE_PEPPER")
|
||||
self.assertEqual(pepper, body["pepper"])
|
||||
broker_side = _derive(pepper, self.app_uuid, self.user_id, agent_id, tags)
|
||||
tool_side = _derive(body["pepper"], self.app_uuid, self.user_id, agent_id, tags)
|
||||
self.assertEqual(broker_side, token)
|
||||
self.assertEqual(tool_side, token)
|
||||
self.assertEqual(44, len(token))
|
||||
self.assertTrue(token.isascii())
|
||||
row = self.broker.store.get_agent(agent_id, self.app_uuid)
|
||||
stored = row[4]
|
||||
self.assertNotEqual(token, stored)
|
||||
if coreci_broker.bcrypt is None:
|
||||
self.assertTrue(stored.startswith("scrypt$"))
|
||||
else:
|
||||
self.assertTrue(stored.startswith(coreci_broker.BCRYPT_PREFIXES))
|
||||
self.assertTrue(coreci_broker.verify_token(token, stored))
|
||||
self.assertFalse(coreci_broker.verify_token(token, token))
|
||||
self.assertEqual(self.user_id, row[2])
|
||||
self.assertEqual(json.dumps(sorted(tags)), row[3])
|
||||
self.assertEqual(body["token_expires_at"], row[5])
|
||||
|
||||
def test_ac7_revoked_or_unknown_uuid_denied_existing_token_honored(self):
|
||||
status, first = self._register("tool-1", ["coreci"])
|
||||
self.assertEqual(201, status)
|
||||
token = first["token"]
|
||||
status, body = self._request(
|
||||
"DELETE", APPS_PATH + "/" + self.app_uuid, headers=self.admin
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"uuid": self.app_uuid, "status": "revoked"}, body)
|
||||
status, body = self._register("tool-2", ["coreci"])
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual(REGISTER_403_BODY, body)
|
||||
self.assertIsNone(self.broker.store.get_agent("tool-2", self.app_uuid))
|
||||
status, body = self._register("tool-3", ["coreci"], app_uuid=str(uuidlib.uuid4()))
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual(REGISTER_403_BODY, body)
|
||||
status, body = self._request(
|
||||
"GET", AGENT_PATH % "tool-1", headers=self._app_header(token)
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("tool-1", body["agent_id"])
|
||||
self.assertEqual(self.user_id, body["user_id"])
|
||||
self.assertEqual(self.app_uuid, body["app_registration_uuid"])
|
||||
self.assertEqual(["coreci"], body["tags"])
|
||||
|
||||
def test_ac8_unknown_or_revoked_user_denied(self):
|
||||
status, body = self._register("tool-1", ["coreci"], user_id="intern")
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual(REGISTER_403_BODY, body)
|
||||
status, body = self._request(
|
||||
"POST", USERS_PATH, {"user_id": "intern"}, headers=self.admin
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
status, body = self._register("tool-1", ["coreci"], user_id="intern")
|
||||
self.assertEqual(201, status)
|
||||
self.assertTrue(body["created"])
|
||||
status, body = self._request(
|
||||
"DELETE", USERS_PATH + "/intern", headers=self.admin
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"user_id": "intern", "status": "revoked"}, body)
|
||||
status, body = self._register("tool-1", ["coreci"], user_id="intern")
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual(REGISTER_403_BODY, body)
|
||||
|
||||
def test_ac10_expiry_401_reregister_refreshes_expiry_and_audits(self):
|
||||
broker, base = self._boot({
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "ttl.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "ttl.audit.log"),
|
||||
"BROKER_TOKEN_TTL": "1",
|
||||
})
|
||||
self._seed(base, self.app_uuid, self.user_id)
|
||||
agent_id = "ttl-tool"
|
||||
status, first = self._register(agent_id, ["coreci"], base=base)
|
||||
self.assertEqual(201, status)
|
||||
delta = (datetime.datetime.fromisoformat(first["token_expires_at"])
|
||||
- datetime.datetime.now(datetime.timezone.utc)).total_seconds()
|
||||
self.assertGreater(delta, 0)
|
||||
self.assertLess(delta, 3)
|
||||
status, body = self._request(
|
||||
"GET", AGENT_PATH % agent_id,
|
||||
headers=self._app_header(first["token"]), base=base,
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
time.sleep(1.2)
|
||||
status, body = self._request(
|
||||
"GET", AGENT_PATH % agent_id,
|
||||
headers=self._app_header(first["token"]), base=base,
|
||||
)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
status, body = self._register(agent_id, ["coreci"], base=base, user_id="ghost-user")
|
||||
self.assertEqual(403, status)
|
||||
status, second = self._register(agent_id, ["coreci"], base=base)
|
||||
self.assertEqual(200, status)
|
||||
self.assertFalse(second["created"])
|
||||
self.assertEqual(first["token"], second["token"])
|
||||
self.assertIn("pepper", second)
|
||||
self.assertGreater(
|
||||
datetime.datetime.fromisoformat(second["token_expires_at"]),
|
||||
datetime.datetime.fromisoformat(first["token_expires_at"]),
|
||||
)
|
||||
status, body = self._request(
|
||||
"GET", AGENT_PATH % agent_id,
|
||||
headers=self._app_header(second["token"]), base=base,
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
row = broker.store.get_agent(agent_id, self.app_uuid)
|
||||
self.assertEqual(second["token_expires_at"], row[5])
|
||||
self.assertTrue(coreci_broker.verify_token(second["token"], row[4]))
|
||||
entries = self._audit_entries(base, "?action=register")
|
||||
self.assertEqual(["ok", "denied", "ok"], [entry["result"] for entry in entries])
|
||||
for entry in entries:
|
||||
self.assertEqual(agent_id, entry["agent_id"])
|
||||
self.assertEqual(self.app_uuid, entry["app_registration_uuid"])
|
||||
self.assertEqual(REGISTER_PATH, entry["path"])
|
||||
self.assertEqual(self.user_id, entries[0]["user_id"])
|
||||
self.assertEqual("ghost-user", entries[1]["user_id"])
|
||||
self.assertEqual(self.user_id, entries[2]["user_id"])
|
||||
|
||||
def test_ac12_state_roundtrip_with_identity_enforcement(self):
|
||||
status, a = self._register("tool-a", ["coreci", "alpha"])
|
||||
self.assertEqual(201, status)
|
||||
status, b = self._register("tool-b", ["coreci"])
|
||||
self.assertEqual(201, status)
|
||||
state = {
|
||||
"agent_id": "tool-a",
|
||||
"app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id,
|
||||
"tags": ["alpha", "coreci"],
|
||||
"pepper": a["pepper"],
|
||||
"current_token": a["token"],
|
||||
"token_expires_at": a["token_expires_at"],
|
||||
"first_registered_at": "t0",
|
||||
"last_seen_at": "t1",
|
||||
"payload": {"counter": 7, "notes": ["x", "y"], "nested": {"k": None}},
|
||||
}
|
||||
status, body = self._request(
|
||||
"PUT", STATE_PATH % "tool-a", state, headers=self._app_header(a["token"])
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
status, body = self._request(
|
||||
"GET", STATE_PATH % "tool-a", headers=self._app_header(a["token"])
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(state, body)
|
||||
self.assertEqual(list(state), list(body))
|
||||
status, body = self._request(
|
||||
"GET", STATE_PATH % "tool-a", headers=self._app_header(b["token"])
|
||||
)
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual({"error": 403, "message": "agent identity mismatch"}, body)
|
||||
status, body = self._request(
|
||||
"PUT", STATE_PATH % "tool-a", {"intruder": True},
|
||||
headers=self._app_header(b["token"]),
|
||||
)
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual({"error": 403, "message": "agent identity mismatch"}, body)
|
||||
status, body = self._request(
|
||||
"PUT", STATE_PATH % "tool-b", {"mine": True},
|
||||
headers=self._app_header(b["token"]),
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
status, body = self._request(
|
||||
"GET", STATE_PATH % "tool-b", headers=self._app_header(b["token"])
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"mine": True}, body)
|
||||
status, body = self._request(
|
||||
"GET", STATE_PATH % "ghost-agent", headers=self._app_header(a["token"])
|
||||
)
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual({"error": 404, "message": "not found"}, body)
|
||||
status, body = self._request(
|
||||
"GET", STATE_PATH % "tool-a", headers=self._app_header(a["token"])
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(state, body)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,290 @@
|
||||
import ast
|
||||
import base64
|
||||
import contextlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
AUDIT_PATH = "/api/broker/audit"
|
||||
SECRETS_PREFIX = "/api/broker/secrets/"
|
||||
CLIENT_STATIC_PATH = "/static/coreci_broker_client.py"
|
||||
PROXMOX_PATH = "coreci/proxmox"
|
||||
GITEA_PATH = "coreci/gitea"
|
||||
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
|
||||
GITEA_SECRET = {"url": "https://git.coreci.dev", "token": "git-tok"}
|
||||
AUTH_401_BODY = {"error": 401, "message": "invalid app credentials"}
|
||||
GITEA_403_BODY = {"error": 403, "message": "forbidden"}
|
||||
REFERENCE_DIR = os.path.abspath(
|
||||
os.path.join(os.path.dirname(__file__), os.pardir, "reference_tools")
|
||||
)
|
||||
STANDALONE_CLIENT_IMPORTS = {"json", "time", "urllib.parse", "urllib.request"}
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
def _flip(token):
|
||||
replacement = "x" if token[21] != "x" else "y"
|
||||
return token[:21] + replacement + token[22:]
|
||||
|
||||
|
||||
class AcceptanceP4Tests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.creds_path = os.path.join(self.tmp, "creds.json")
|
||||
with open(self.creds_path, "w", encoding="utf-8") as handle:
|
||||
handle.write(json.dumps(
|
||||
{PROXMOX_PATH: PROXMOX_SECRET, GITEA_PATH: GITEA_SECRET}
|
||||
))
|
||||
env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
"BROKER_CREDS_SOURCE": "file",
|
||||
"BROKER_CREDS_FILE": self.creds_path,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(env)
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
self.app_uuid = str(uuidlib.uuid4())
|
||||
self.user_id = "ceo"
|
||||
self._seed()
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
raw = response.read().decode("utf-8")
|
||||
return response.status, json.loads(raw) if raw else None
|
||||
except urllib.error.HTTPError as error:
|
||||
raw = error.read().decode("utf-8")
|
||||
return error.code, json.loads(raw) if raw else None
|
||||
|
||||
def _seed(self):
|
||||
status, _ = self._request("POST", APPS_PATH, {"uuid": self.app_uuid}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
status, _ = self._request("POST", USERS_PATH, {"user_id": self.user_id}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def _register(self, tags):
|
||||
payload = {"agent_id": "proxmox", "app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id, "tags": list(tags)}
|
||||
status, body = self._request("POST", REGISTER_PATH, payload)
|
||||
self.assertEqual(201, status)
|
||||
return body
|
||||
|
||||
def _app_header(self, token):
|
||||
return {"Authorization": "App %s %s" % (self.app_uuid, token)}
|
||||
|
||||
def _secret(self, suffix, headers):
|
||||
return self._request("GET", SECRETS_PREFIX + suffix, headers=headers)
|
||||
|
||||
def _audit(self, action):
|
||||
query = urllib.parse.urlencode({"action": action, "limit": "1000"})
|
||||
status, body = self._request(
|
||||
"GET", AUDIT_PATH + "?" + query, headers=self.admin
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
return body["entries"]
|
||||
|
||||
def _fetch_client_source(self):
|
||||
with urllib.request.urlopen(
|
||||
self.base + CLIENT_STATIC_PATH, timeout=10
|
||||
) as response:
|
||||
self.assertEqual(200, response.status)
|
||||
return response.read().decode("utf-8")
|
||||
|
||||
def _import_module(self, name, source):
|
||||
path = os.path.join(self.tmp, name + ".py")
|
||||
with open(path, "w", encoding="utf-8") as handle:
|
||||
handle.write(source)
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
def _assert_standalone_imports(self, source):
|
||||
modules = set()
|
||||
for node in ast.walk(ast.parse(source)):
|
||||
if isinstance(node, ast.Import):
|
||||
modules.update(alias.name for alias in node.names)
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
modules.add(node.module or "")
|
||||
self.assertTrue(modules)
|
||||
for module in modules:
|
||||
self.assertIn(module, STANDALONE_CLIENT_IMPORTS, module)
|
||||
|
||||
def _bootstrapped_client(self):
|
||||
module = self._import_module(
|
||||
"p4_fetched_coreci_broker_client", self._fetch_client_source()
|
||||
)
|
||||
return module.BrokerClient(
|
||||
self.base, "proxmox", self.app_uuid, self.user_id, ["coreci", "bootstrap"]
|
||||
)
|
||||
|
||||
def _reference_tools(self):
|
||||
source_path = os.path.join(REFERENCE_DIR, "proxmox_readonly_reference.py")
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"p4_proxmox_readonly_reference", source_path
|
||||
)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
tools = module.Tools()
|
||||
tools.valves.base_url = self.base
|
||||
tools.valves.agent_id = "proxmox"
|
||||
tools.valves.app_uuid = self.app_uuid
|
||||
tools.valves.user_id = self.user_id
|
||||
return tools
|
||||
|
||||
def _persisted_state(self):
|
||||
return self.broker.store.get_agent_state("proxmox", self.app_uuid)
|
||||
|
||||
def _assert_tool_identity(self, entry):
|
||||
self.assertEqual(self.user_id, entry["user_id"])
|
||||
self.assertIsNotNone(entry["user_id"])
|
||||
self.assertEqual(self.app_uuid, entry["app_registration_uuid"])
|
||||
self.assertIsNotNone(entry["app_registration_uuid"])
|
||||
self.assertEqual("proxmox", entry["agent_id"])
|
||||
self.assertIsNotNone(entry["agent_id"])
|
||||
|
||||
def test_ac5_valid_token_reads_secret_tampered_token_is_401(self):
|
||||
registered = self._register(["coreci"])
|
||||
status, body = self._secret(
|
||||
"coreci/proxmox", self._app_header(registered["token"])
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"path", "data", "version", "updated_at"}, set(body))
|
||||
self.assertEqual(PROXMOX_PATH, body["path"])
|
||||
self.assertEqual(PROXMOX_SECRET, body["data"])
|
||||
self.assertRegex(body["version"], r"^[0-9a-f]{8}$")
|
||||
flipped = _flip(registered["token"])
|
||||
self.assertNotEqual(registered["token"], flipped)
|
||||
status, body = self._secret("coreci/proxmox", self._app_header(flipped))
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
|
||||
def test_ac6_cross_path_read_is_403_with_denied_audit(self):
|
||||
registered = self._register(["coreci"])
|
||||
status, body = self._secret("coreci/gitea", self._app_header(registered["token"]))
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual(GITEA_403_BODY, body)
|
||||
denied = [
|
||||
entry for entry in self._audit("read") if entry["result"] == "denied"
|
||||
]
|
||||
self.assertEqual(1, len(denied))
|
||||
self._assert_tool_identity(denied[0])
|
||||
self.assertEqual("read", denied[0]["action"])
|
||||
self.assertEqual(GITEA_PATH, denied[0]["path"])
|
||||
|
||||
def test_client_bootstrap_public_fetch_import_register_state_secret(self):
|
||||
source = self._fetch_client_source()
|
||||
self.assertIn("class BrokerClient", source)
|
||||
self._assert_standalone_imports(source)
|
||||
client = self._bootstrapped_client()
|
||||
state = {}
|
||||
client.ensure_registered(state)
|
||||
self.assertEqual(44, len(state["current_token"]))
|
||||
self.assertEqual("proxmox", state["agent_id"])
|
||||
self.assertEqual(self.app_uuid, state["app_registration_uuid"])
|
||||
self.assertEqual(self.user_id, state["user_id"])
|
||||
self.assertIn("pepper", state)
|
||||
self.assertTrue(client.put_state(state))
|
||||
document = client.read_secret(PROXMOX_PATH, state)
|
||||
self.assertEqual(PROXMOX_PATH, document["path"])
|
||||
self.assertEqual(PROXMOX_SECRET, document["data"])
|
||||
persisted = self._persisted_state()
|
||||
self.assertEqual(state["current_token"], persisted["current_token"])
|
||||
|
||||
def test_reference_tool_full_flow_single_register_line(self):
|
||||
tools = self._reference_tools()
|
||||
before = len(self._audit("register"))
|
||||
response = tools.list_nodes()
|
||||
self.assertIs(True, response["dry_run"])
|
||||
self.assertIs(True, response["credential_present"])
|
||||
registers = self._audit("register")
|
||||
self.assertEqual(1, len(registers) - before)
|
||||
self.assertEqual("ok", registers[-1]["result"])
|
||||
self._assert_tool_identity(registers[-1])
|
||||
persisted = self._persisted_state()
|
||||
self.assertEqual(tools._state["current_token"], persisted["current_token"])
|
||||
|
||||
def test_phase_audit_completeness_after_full_drive(self):
|
||||
registered = self._register(["coreci"])
|
||||
token = registered["token"]
|
||||
status, _ = self._secret("coreci/proxmox", self._app_header(token))
|
||||
self.assertEqual(200, status)
|
||||
status, _ = self._secret("coreci/proxmox", self._app_header(_flip(token)))
|
||||
self.assertEqual(401, status)
|
||||
status, _ = self._secret("coreci/gitea", self._app_header(token))
|
||||
self.assertEqual(403, status)
|
||||
client = self._bootstrapped_client()
|
||||
state = {}
|
||||
client.ensure_registered(state)
|
||||
self.assertTrue(client.put_state(state))
|
||||
self.assertEqual(PROXMOX_SECRET, client.read_secret(PROXMOX_PATH, state)["data"])
|
||||
before = len(self._audit("register"))
|
||||
tools = self._reference_tools()
|
||||
response = tools.list_nodes()
|
||||
self.assertIs(True, response["dry_run"])
|
||||
self.assertIs(True, response["credential_present"])
|
||||
registers = self._audit("register")
|
||||
self.assertEqual(1, len(registers) - before)
|
||||
self.assertEqual(3, len(registers))
|
||||
for entry in registers:
|
||||
self.assertEqual("ok", entry["result"])
|
||||
self.assertEqual(REGISTER_PATH, entry["path"])
|
||||
self._assert_tool_identity(entry)
|
||||
reads = self._audit("read")
|
||||
ok_reads = [entry for entry in reads if entry["result"] == "ok"]
|
||||
denied_reads = [entry for entry in reads if entry["result"] == "denied"]
|
||||
self.assertEqual(3, len(ok_reads))
|
||||
for entry in ok_reads:
|
||||
self.assertEqual(PROXMOX_PATH, entry["path"])
|
||||
self._assert_tool_identity(entry)
|
||||
self.assertEqual(1, len(denied_reads))
|
||||
self.assertEqual(GITEA_PATH, denied_reads[0]["path"])
|
||||
self._assert_tool_identity(denied_reads[0])
|
||||
denied_actions = self._audit("denied")
|
||||
self.assertEqual(1, len(denied_actions))
|
||||
self.assertEqual("denied", denied_actions[0]["action"])
|
||||
self.assertEqual("denied", denied_actions[0]["result"])
|
||||
self.assertEqual(SECRETS_PREFIX + "coreci/proxmox", denied_actions[0]["path"])
|
||||
self.assertEqual(self.app_uuid, denied_actions[0]["app_registration_uuid"])
|
||||
self.assertIsNone(denied_actions[0]["user_id"])
|
||||
self.assertIsNone(denied_actions[0]["agent_id"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,251 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import datetime
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
LOGIN_PATH = "/api/broker/auth/login"
|
||||
LOGOUT_PATH = "/api/broker/auth/logout"
|
||||
WHOAMI_PATH = "/api/broker/auth/whoami"
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
def _iso_offset(seconds):
|
||||
now = datetime.datetime.now(datetime.timezone.utc)
|
||||
return (now + datetime.timedelta(seconds=seconds)).isoformat()
|
||||
|
||||
|
||||
class AdminAuthTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
}
|
||||
self.broker = self._boot_broker(self.env)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
|
||||
def _boot_broker(self, env):
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
broker = coreci_broker.boot(dict(env))
|
||||
self.addCleanup(broker.audit.close)
|
||||
self.addCleanup(broker.store.close)
|
||||
return broker
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None, base=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(
|
||||
(base or self.base) + path, data=data, method=method
|
||||
)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=5) as response:
|
||||
return response.status, response.read(), response.headers
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read(), error.headers
|
||||
|
||||
def _request_json(self, method, path, payload=None, headers=None, base=None):
|
||||
status, body, response_headers = self._request(
|
||||
method, path, payload, headers, base
|
||||
)
|
||||
return status, json.loads(body.decode("utf-8")), response_headers
|
||||
|
||||
def _login(self, username=ADMIN_USER, password=ADMIN_PASS, base=None):
|
||||
return self._request_json(
|
||||
"POST",
|
||||
LOGIN_PATH,
|
||||
{"username": username, "password": password},
|
||||
base=base,
|
||||
)
|
||||
|
||||
def _login_cookie(self, base=None):
|
||||
status, body, headers = self._login(base=base)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
|
||||
raw = headers.get("Set-Cookie")
|
||||
self.assertTrue(raw)
|
||||
return raw
|
||||
|
||||
def _cookie_id(self, raw):
|
||||
return raw.split(";", 1)[0].split("=", 1)[1]
|
||||
|
||||
def _audit_entries(self):
|
||||
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
|
||||
return [json.loads(raw) for raw in handle if raw.strip()]
|
||||
|
||||
def _audit_find(self, action, result):
|
||||
for entry in self._audit_entries():
|
||||
if entry["action"] == action and entry["result"] == result:
|
||||
return entry
|
||||
return None
|
||||
|
||||
def _assert_null_identity(self, entry):
|
||||
self.assertIsNotNone(entry)
|
||||
self.assertIsNone(entry["user_id"])
|
||||
self.assertIsNone(entry["app_registration_uuid"])
|
||||
self.assertIsNone(entry["agent_id"])
|
||||
|
||||
def test_login_logout_whoami_flow_and_cookie_contract(self):
|
||||
raw = self._login_cookie()
|
||||
self.assertTrue(raw.startswith("broker_session="))
|
||||
self.assertIn("Path=/", raw)
|
||||
self.assertIn("HttpOnly", raw)
|
||||
self.assertIn("SameSite=Lax", raw)
|
||||
cookie = {"Cookie": "broker_session=" + self._cookie_id(raw)}
|
||||
status, body, _ = self._request_json("GET", WHOAMI_PATH, headers=cookie)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(ADMIN_USER, body["username"])
|
||||
created = datetime.datetime.fromisoformat(body["created_at"])
|
||||
expires = datetime.datetime.fromisoformat(body["expires_at"])
|
||||
self.assertEqual(datetime.timedelta(seconds=300), expires - created)
|
||||
status, body, _ = self._request_json("POST", LOGOUT_PATH, headers=cookie)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
status, body, _ = self._request_json("GET", WHOAMI_PATH, headers=cookie)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(401, body["error"])
|
||||
self.assertIsNone(self.broker.store.get_admin_session(self._cookie_id(raw)))
|
||||
self._assert_null_identity(self._audit_find("login", "ok"))
|
||||
self._assert_null_identity(self._audit_find("logout", "ok"))
|
||||
|
||||
def test_wrong_password_is_401_with_denied_audit(self):
|
||||
status, body, headers = self._login(password="not-the-pass")
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(401, body["error"])
|
||||
self.assertTrue(body["message"])
|
||||
self.assertIsNone(headers.get("Set-Cookie"))
|
||||
self._assert_null_identity(self._audit_find("login", "denied"))
|
||||
|
||||
def test_wrong_username_is_401(self):
|
||||
status, body, headers = self._login(username="mallory")
|
||||
self.assertEqual(401, status)
|
||||
self.assertIsNone(headers.get("Set-Cookie"))
|
||||
self.assertIsNotNone(self._audit_find("login", "denied"))
|
||||
|
||||
def test_login_with_missing_credentials_is_401(self):
|
||||
status, body, _ = self._request_json("POST", LOGIN_PATH, payload={})
|
||||
self.assertEqual(401, status)
|
||||
status, body, _ = self._request_json("POST", LOGIN_PATH)
|
||||
self.assertEqual(401, status)
|
||||
self.assertIsNotNone(self._audit_find("login", "denied"))
|
||||
|
||||
def test_expired_session_is_rejected_lazily(self):
|
||||
raw = self._login_cookie()
|
||||
session_id = self._cookie_id(raw)
|
||||
self.broker.store.conn.execute(
|
||||
"UPDATE admin_sessions SET expires_at = ? WHERE session_id = ?",
|
||||
(_iso_offset(-60), session_id),
|
||||
)
|
||||
self.broker.store.conn.commit()
|
||||
status, body, _ = self._request_json(
|
||||
"GET", WHOAMI_PATH, headers={"Cookie": "broker_session=" + session_id}
|
||||
)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(401, body["error"])
|
||||
|
||||
def test_boot_cleanup_deletes_stale_sessions_keeps_live(self):
|
||||
self.broker.store.conn.executemany(
|
||||
"INSERT INTO admin_sessions VALUES (?, ?, ?)",
|
||||
[
|
||||
("stale-session", _iso_offset(-600), _iso_offset(-60)),
|
||||
("live-session", _iso_offset(-600), _iso_offset(300)),
|
||||
],
|
||||
)
|
||||
self.broker.store.conn.commit()
|
||||
rebooted = self._boot_broker(self.env)
|
||||
self.assertIsNone(rebooted.store.get_admin_session("stale-session"))
|
||||
self.assertIsNotNone(rebooted.store.get_admin_session("live-session"))
|
||||
|
||||
def test_basic_valid_on_whoami(self):
|
||||
headers = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
status, body, _ = self._request_json("GET", WHOAMI_PATH, headers=headers)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(ADMIN_USER, body["username"])
|
||||
self.assertIsNone(body["created_at"])
|
||||
self.assertIsNone(body["expires_at"])
|
||||
|
||||
def test_basic_invalid_is_401(self):
|
||||
headers = {"Authorization": _basic_header(ADMIN_USER, "wrong-pass")}
|
||||
status, body, _ = self._request_json("GET", WHOAMI_PATH, headers=headers)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(401, body["error"])
|
||||
self.assertTrue(body["message"])
|
||||
|
||||
def test_whoami_without_credentials_is_401(self):
|
||||
status, body, _ = self._request_json("GET", WHOAMI_PATH)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(401, body["error"])
|
||||
|
||||
def test_valid_session_cookie_wins_over_invalid_basic(self):
|
||||
raw = self._login_cookie()
|
||||
headers = {
|
||||
"Cookie": "broker_session=" + self._cookie_id(raw),
|
||||
"Authorization": _basic_header(ADMIN_USER, "wrong-pass"),
|
||||
}
|
||||
status, body, _ = self._request_json("GET", WHOAMI_PATH, headers=headers)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(ADMIN_USER, body["username"])
|
||||
|
||||
def test_logout_without_credentials_is_401(self):
|
||||
status, body, _ = self._request_json("POST", LOGOUT_PATH)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(401, body["error"])
|
||||
self.assertIsNone(self._audit_find("logout", "ok"))
|
||||
|
||||
def test_logout_via_basic_without_session_is_ok(self):
|
||||
headers = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
status, body, _ = self._request_json("POST", LOGOUT_PATH, headers=headers)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
self._assert_null_identity(self._audit_find("logout", "ok"))
|
||||
|
||||
def test_session_ttl_is_configurable(self):
|
||||
env = dict(self.env)
|
||||
env["BROKER_DB_PATH"] = os.path.join(self.tmp, "ttl.db")
|
||||
env["BROKER_AUDIT_LOG_PATH"] = os.path.join(self.tmp, "ttl.audit.log")
|
||||
env["BROKER_SESSION_TTL"] = "60"
|
||||
broker = self._boot_broker(env)
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
self.addCleanup(server.shutdown)
|
||||
self.addCleanup(server.server_close)
|
||||
base = "http://127.0.0.1:%d" % server.server_address[1]
|
||||
raw = self._login_cookie(base=base)
|
||||
status, body, _ = self._request_json(
|
||||
"GET",
|
||||
WHOAMI_PATH,
|
||||
headers={"Cookie": "broker_session=" + self._cookie_id(raw)},
|
||||
base=base,
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
created = datetime.datetime.fromisoformat(body["created_at"])
|
||||
expires = datetime.datetime.fromisoformat(body["expires_at"])
|
||||
self.assertEqual(datetime.timedelta(seconds=60), expires - created)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,134 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
META_KEYS = {"agent_id", "app_registration_uuid", "user_id", "tags",
|
||||
"token_expires_at", "first_registered_at", "last_seen_at"}
|
||||
FORBIDDEN_RAW = ('"token"', '"token_hash"', '"pepper"')
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
class AgentMetaTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(dict(self.env))
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
self.app_uuid = str(uuidlib.uuid4())
|
||||
self.user_id = "ceo"
|
||||
self._seed(self.app_uuid, self.user_id)
|
||||
self.registered = self._register("agent-1", ["b", "a"])
|
||||
self.token = self.registered["token"]
|
||||
self.pepper = self.registered["pepper"]
|
||||
self.app = {"Authorization": "App %s %s" % (self.app_uuid, self.token)}
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
raw = response.read().decode("utf-8")
|
||||
return response.status, json.loads(raw), raw
|
||||
except urllib.error.HTTPError as error:
|
||||
raw = error.read().decode("utf-8")
|
||||
return error.code, json.loads(raw), raw
|
||||
|
||||
def _seed(self, app_uuid, user_id):
|
||||
status, _, _ = self._request("POST", APPS_PATH, {"uuid": app_uuid}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
status, _, _ = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def _register(self, agent_id, tags):
|
||||
payload = {"agent_id": agent_id, "app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id, "tags": tags}
|
||||
status, body, _ = self._request("POST", REGISTER_PATH, payload)
|
||||
self.assertEqual(201, status)
|
||||
return body
|
||||
|
||||
def _meta(self, agent_id, headers=None):
|
||||
return self._request(
|
||||
"GET", "/api/broker/agents/" + agent_id, headers=headers if headers is not None else self.app
|
||||
)
|
||||
|
||||
def test_identity_match_returns_metadata_only(self):
|
||||
status, body, raw = self._meta("agent-1")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(META_KEYS, set(body))
|
||||
self.assertEqual("agent-1", body["agent_id"])
|
||||
self.assertEqual(self.app_uuid, body["app_registration_uuid"])
|
||||
self.assertEqual(self.user_id, body["user_id"])
|
||||
self.assertEqual(self.registered["token_expires_at"], body["token_expires_at"])
|
||||
row = self.broker.store.get_agent("agent-1", self.app_uuid)
|
||||
self.assertEqual(row[6], body["first_registered_at"])
|
||||
self.assertEqual(row[7], body["last_seen_at"])
|
||||
for forbidden in FORBIDDEN_RAW:
|
||||
self.assertNotIn(forbidden, raw)
|
||||
self.assertNotIn(self.token, raw)
|
||||
self.assertNotIn(self.pepper, raw)
|
||||
|
||||
def test_tags_roundtrip_is_sorted_canonical_list(self):
|
||||
status, body, _ = self._meta("agent-1")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["a", "b"], body["tags"])
|
||||
|
||||
def test_url_agent_id_mismatch_is_403(self):
|
||||
self._register("agent-2", ["coreci"])
|
||||
status, body, raw = self._meta("agent-2")
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual(403, body["error"])
|
||||
self.assertTrue(body["message"])
|
||||
for forbidden in FORBIDDEN_RAW:
|
||||
self.assertNotIn(forbidden, raw)
|
||||
|
||||
def test_unknown_agent_id_is_404(self):
|
||||
status, body, _ = self._meta("ghost-agent")
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual({"error": 404, "message": "not found"}, body)
|
||||
|
||||
def test_missing_app_header_is_401(self):
|
||||
status, body, _ = self._meta("agent-1", headers={})
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual({"error": 401, "message": "invalid app credentials"}, body)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,177 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
BASE_PATH = "/api/broker/user_allowlist"
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
class UserAllowlistTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(dict(self.env))
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=5) as response:
|
||||
return response.status, json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, json.loads(error.read().decode("utf-8"))
|
||||
|
||||
def _add(self, user_id, email=None, headers=None):
|
||||
payload = {"user_id": user_id}
|
||||
if email is not None:
|
||||
payload["email"] = email
|
||||
return self._request("POST", BASE_PATH, payload, headers or self.admin)
|
||||
|
||||
def _row(self, user_id):
|
||||
status, body = self._request("GET", BASE_PATH, headers=self.admin)
|
||||
self.assertEqual(200, status)
|
||||
for row in body["allowlist"]:
|
||||
if row["user_id"] == user_id:
|
||||
return row
|
||||
return None
|
||||
|
||||
def _audit(self):
|
||||
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
|
||||
return [json.loads(raw) for raw in handle if raw.strip()]
|
||||
|
||||
def test_list_empty_then_add_then_list(self):
|
||||
status, body = self._request("GET", BASE_PATH, headers=self.admin)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"allowlist": []}, body)
|
||||
status, body = self._add("alice", "alice@coreci.dev")
|
||||
self.assertEqual(201, status)
|
||||
self.assertEqual({"user_id": "alice", "status": "active"}, body)
|
||||
row = self._row("alice")
|
||||
self.assertEqual("alice@coreci.dev", row["email"])
|
||||
self.assertEqual("active", row["status"])
|
||||
self.assertTrue(row["created_at"])
|
||||
self.assertIsNone(row["revoked_at"])
|
||||
|
||||
def test_add_without_email_stores_null_email(self):
|
||||
status, _ = self._add("bob")
|
||||
self.assertEqual(201, status)
|
||||
self.assertIsNone(self._row("bob")["email"])
|
||||
status, _ = self._request("POST", BASE_PATH, {"user_id": "bob2", "email": 42}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
self.assertIsNone(self._row("bob2")["email"])
|
||||
|
||||
def test_add_missing_or_empty_user_id_is_400(self):
|
||||
for payload in ({"email": "x@coreci.dev"}, {}, {"user_id": ""},
|
||||
{"user_id": None}, {"user_id": 12345},
|
||||
{"user_id": ["alice"]}, [1, 2]):
|
||||
status, body = self._request("POST", BASE_PATH, payload, self.admin)
|
||||
self.assertEqual(400, status, repr(payload))
|
||||
self.assertEqual(400, body["error"])
|
||||
|
||||
def test_add_duplicate_active_or_revoked_is_409(self):
|
||||
self._add("carol", "carol@coreci.dev")
|
||||
status, body = self._add("carol")
|
||||
self.assertEqual(409, status)
|
||||
self.assertEqual(409, body["error"])
|
||||
self.assertTrue(body["message"])
|
||||
status, _ = self._request("DELETE", BASE_PATH + "/carol", headers=self.admin)
|
||||
self.assertEqual(200, status)
|
||||
status, body = self._add("carol")
|
||||
self.assertEqual(409, status)
|
||||
|
||||
def test_revoke_revoke_again_reinstate_lifecycle(self):
|
||||
self._add("dave", "dave@coreci.dev")
|
||||
status, body = self._request("DELETE", BASE_PATH + "/dave", headers=self.admin)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("revoked", body["status"])
|
||||
revoked_at = self._row("dave")["revoked_at"]
|
||||
self.assertIsNotNone(revoked_at)
|
||||
self.assertEqual("revoked", self._row("dave")["status"])
|
||||
status, body = self._request("DELETE", BASE_PATH + "/dave", headers=self.admin)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(revoked_at, self._row("dave")["revoked_at"])
|
||||
status, body = self._request(
|
||||
"POST", BASE_PATH + "/dave/reinstate", headers=self.admin
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("active", body["status"])
|
||||
row = self._row("dave")
|
||||
self.assertEqual("active", row["status"])
|
||||
self.assertIsNone(row["revoked_at"])
|
||||
|
||||
def test_revoke_missing_is_404(self):
|
||||
status, body = self._request("DELETE", BASE_PATH + "/ghost", headers=self.admin)
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual(404, body["error"])
|
||||
|
||||
def test_reinstate_missing_is_404(self):
|
||||
status, body = self._request(
|
||||
"POST", BASE_PATH + "/ghost/reinstate", headers=self.admin
|
||||
)
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual(404, body["error"])
|
||||
|
||||
def test_admin_actions_audited_with_null_identity(self):
|
||||
self._add("erin", "erin@coreci.dev")
|
||||
self._request("DELETE", BASE_PATH + "/erin", headers=self.admin)
|
||||
self._request("POST", BASE_PATH + "/erin/reinstate", headers=self.admin)
|
||||
entries = self._audit()
|
||||
self.assertEqual(["add_user", "remove_user", "add_user"],
|
||||
[entry["action"] for entry in entries])
|
||||
for entry in entries:
|
||||
self.assertEqual("ok", entry["result"])
|
||||
self.assertIsNone(entry["user_id"])
|
||||
self.assertIsNone(entry["app_registration_uuid"])
|
||||
self.assertIsNone(entry["agent_id"])
|
||||
self.assertEqual("/api/broker/user_allowlist/erin", entry["path"])
|
||||
|
||||
def test_unauthenticated_requests_are_401(self):
|
||||
checks = (
|
||||
("GET", BASE_PATH, None),
|
||||
("POST", BASE_PATH, {"user_id": "zoe"}),
|
||||
("DELETE", BASE_PATH + "/zoe", None),
|
||||
("POST", BASE_PATH + "/zoe/reinstate", None),
|
||||
)
|
||||
for method, path, payload in checks:
|
||||
status, body = self._request(method, path, payload)
|
||||
self.assertEqual(401, status, method + " " + path)
|
||||
self.assertEqual(401, body["error"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,263 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import datetime
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
PROTECTED_PATH = "/test/app-auth"
|
||||
AUTH_401_BODY = {"error": 401, "message": "invalid app credentials"}
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
def _protected_echo(request):
|
||||
request.send_json(200, dict(request.app_auth))
|
||||
|
||||
|
||||
class AppAuthTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(dict(self.env))
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
self.app_uuid = str(uuidlib.uuid4())
|
||||
self.user_id = "ceo"
|
||||
self._seed(self.app_uuid, self.user_id)
|
||||
self.broker.router.add(
|
||||
"GET", PROTECTED_PATH, coreci_broker._require_app(_protected_echo)
|
||||
)
|
||||
self.token = self._register("agent-1")["token"]
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
return response.status, json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, json.loads(error.read().decode("utf-8"))
|
||||
|
||||
def _seed(self, app_uuid, user_id):
|
||||
status, _ = self._request("POST", APPS_PATH, {"uuid": app_uuid}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
status, _ = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def _register(self, agent_id):
|
||||
payload = {
|
||||
"agent_id": agent_id,
|
||||
"app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id,
|
||||
"tags": ["coreci"],
|
||||
}
|
||||
status, body = self._request("POST", REGISTER_PATH, payload)
|
||||
self.assertEqual(201, status)
|
||||
return body
|
||||
|
||||
def _auth(self, authorization):
|
||||
headers = {} if authorization is None else {"Authorization": authorization}
|
||||
return self._request("GET", PROTECTED_PATH, headers=headers)
|
||||
|
||||
def _app_header(self, token=None, app_uuid=None):
|
||||
return "App %s %s" % (app_uuid or self.app_uuid, token or self.token)
|
||||
|
||||
def _revoke_app(self):
|
||||
status, _ = self._request(
|
||||
"DELETE", APPS_PATH + "/" + self.app_uuid, headers=self.admin
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
|
||||
def _patch_column(self, agent_id, column, value):
|
||||
self.broker.store.conn.execute(
|
||||
"UPDATE agent_registrations SET %s = ? WHERE agent_id = ?" % column,
|
||||
(value, agent_id),
|
||||
)
|
||||
self.broker.store.conn.commit()
|
||||
|
||||
def _count_verify_calls(self):
|
||||
counter = {"calls": 0}
|
||||
original = coreci_broker.verify_token
|
||||
|
||||
def counted(token, stored):
|
||||
counter["calls"] += 1
|
||||
return original(token, stored)
|
||||
|
||||
coreci_broker.verify_token = counted
|
||||
self.addCleanup(setattr, coreci_broker, "verify_token", original)
|
||||
return counter
|
||||
|
||||
def _past(self):
|
||||
return (datetime.datetime.now(datetime.timezone.utc)
|
||||
- datetime.timedelta(seconds=60)).isoformat()
|
||||
|
||||
def _future(self):
|
||||
return (datetime.datetime.now(datetime.timezone.utc)
|
||||
+ datetime.timedelta(seconds=3600)).isoformat()
|
||||
|
||||
def test_valid_header_returns_context_and_updates_last_seen(self):
|
||||
past = self._past()
|
||||
self._patch_column("agent-1", "last_seen_at", past)
|
||||
status, body = self._auth(self._app_header())
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("agent-1", body["agent_id"])
|
||||
self.assertEqual(self.user_id, body["user_id"])
|
||||
self.assertEqual(self.app_uuid, body["app_registration_uuid"])
|
||||
row = self.broker.store.get_agent("agent-1", self.app_uuid)
|
||||
self.assertEqual(row[5], body["token_expires_at"])
|
||||
self.assertGreater(row[7], past)
|
||||
|
||||
def test_401_matrix_uniform_contract(self):
|
||||
cases = (
|
||||
(None),
|
||||
(_basic_header(ADMIN_USER, ADMIN_PASS)),
|
||||
("App"),
|
||||
("App %s" % self.app_uuid),
|
||||
("App %s %s extra" % (self.app_uuid, self.token)),
|
||||
(self._app_header(app_uuid=str(uuidlib.uuid4()))),
|
||||
(self._app_header(token="A" * 44)),
|
||||
("App not-a-uuid %s" % self.token),
|
||||
("App %s %s" % (self.app_uuid.upper(), self.token)),
|
||||
)
|
||||
for header in cases:
|
||||
with self.subTest(header=header):
|
||||
status, body = self._auth(header)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
|
||||
def test_cache_first_success_scans_then_identical_call_hits_cache(self):
|
||||
counter = self._count_verify_calls()
|
||||
status, body = self._auth(self._app_header())
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(1, counter["calls"])
|
||||
status, body = self._auth(self._app_header())
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("agent-1", body["agent_id"])
|
||||
self.assertEqual(1, counter["calls"])
|
||||
self.assertIn((self.app_uuid, self.token), self.broker.token_cache)
|
||||
|
||||
def test_clear_token_cache_forces_reverify(self):
|
||||
counter = self._count_verify_calls()
|
||||
self._auth(self._app_header())
|
||||
self.assertEqual(1, counter["calls"])
|
||||
self.broker.clear_token_cache()
|
||||
self.assertEqual({}, self.broker.token_cache)
|
||||
status, _ = self._auth(self._app_header())
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(2, counter["calls"])
|
||||
|
||||
def test_cache_hit_revoked_app_still_200_until_ttl(self):
|
||||
counter = self._count_verify_calls()
|
||||
self._auth(self._app_header())
|
||||
self._revoke_app()
|
||||
status, body = self._auth(self._app_header())
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("agent-1", body["agent_id"])
|
||||
self.assertEqual(1, counter["calls"])
|
||||
row = self.broker.store.get_row("app_registrations", "uuid", self.app_uuid)
|
||||
self.assertEqual("revoked", row[2])
|
||||
|
||||
def test_revoked_app_first_auth_still_200(self):
|
||||
self._revoke_app()
|
||||
counter = self._count_verify_calls()
|
||||
status, body = self._auth(self._app_header())
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("agent-1", body["agent_id"])
|
||||
self.assertEqual(1, counter["calls"])
|
||||
|
||||
def test_revoked_app_expired_row_is_401(self):
|
||||
self._revoke_app()
|
||||
self._patch_column("agent-1", "token_expires_at", self._past())
|
||||
status, body = self._auth(self._app_header())
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
|
||||
def test_cache_hit_expired_row_401_then_evicted_and_reverifies(self):
|
||||
counter = self._count_verify_calls()
|
||||
self._auth(self._app_header())
|
||||
self.assertEqual(1, counter["calls"])
|
||||
self._patch_column("agent-1", "token_expires_at", self._past())
|
||||
status, body = self._auth(self._app_header())
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
self.assertEqual(1, counter["calls"])
|
||||
self.assertEqual({}, self.broker.token_cache)
|
||||
self._patch_column("agent-1", "token_expires_at", self._future())
|
||||
status, body = self._auth(self._app_header())
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(2, counter["calls"])
|
||||
self.assertIn((self.app_uuid, self.token), self.broker.token_cache)
|
||||
|
||||
def test_cache_hit_row_absent_is_401(self):
|
||||
self._auth(self._app_header())
|
||||
self.broker.store.conn.execute(
|
||||
"DELETE FROM agent_registrations WHERE agent_id = ?", ("agent-1",)
|
||||
)
|
||||
self.broker.store.conn.commit()
|
||||
status, body = self._auth(self._app_header())
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
self.assertEqual({}, self.broker.token_cache)
|
||||
|
||||
def test_scrypt_hash_format_authenticates(self):
|
||||
token2 = self._register("agent-2")["token"]
|
||||
original_bcrypt = coreci_broker.bcrypt
|
||||
coreci_broker.bcrypt = None
|
||||
try:
|
||||
scrypt_hash = coreci_broker.hash_token(token2)
|
||||
finally:
|
||||
coreci_broker.bcrypt = original_bcrypt
|
||||
self.assertTrue(scrypt_hash.startswith("scrypt$"))
|
||||
self._patch_column("agent-2", "token_hash", scrypt_hash)
|
||||
status, body = self._auth(self._app_header(token=token2))
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("agent-2", body["agent_id"])
|
||||
|
||||
def test_bcrypt_hash_format_authenticates(self):
|
||||
if coreci_broker.bcrypt is None:
|
||||
self.skipTest("bcrypt not importable in this environment")
|
||||
token3 = self._register("agent-3")["token"]
|
||||
stored = self.broker.store.get_agent("agent-3", self.app_uuid)[4]
|
||||
self.assertTrue(stored.startswith("$2"))
|
||||
status, body = self._auth(self._app_header(token=token3))
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("agent-3", body["agent_id"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,187 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
BASE_PATH = "/api/broker/app_registrations"
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
class AppRegistrationsTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(dict(self.env))
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=5) as response:
|
||||
return response.status, json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, json.loads(error.read().decode("utf-8"))
|
||||
|
||||
def _add(self, app_uuid, name=None, headers=None):
|
||||
payload = {"uuid": app_uuid}
|
||||
if name is not None:
|
||||
payload["name"] = name
|
||||
return self._request("POST", BASE_PATH, payload, headers or self.admin)
|
||||
|
||||
def _row(self, app_uuid):
|
||||
status, body = self._request("GET", BASE_PATH, headers=self.admin)
|
||||
self.assertEqual(200, status)
|
||||
for row in body["registrations"]:
|
||||
if row["uuid"] == app_uuid:
|
||||
return row
|
||||
return None
|
||||
|
||||
def _audit(self):
|
||||
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
|
||||
return [json.loads(raw) for raw in handle if raw.strip()]
|
||||
|
||||
def test_list_empty_then_add_then_list(self):
|
||||
status, body = self._request("GET", BASE_PATH, headers=self.admin)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"registrations": []}, body)
|
||||
app_uuid = str(uuidlib.uuid4())
|
||||
status, body = self._add(app_uuid, "proxmox-tool")
|
||||
self.assertEqual(201, status)
|
||||
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
|
||||
row = self._row(app_uuid)
|
||||
self.assertEqual("proxmox-tool", row["name"])
|
||||
self.assertEqual("active", row["status"])
|
||||
self.assertTrue(row["created_at"])
|
||||
self.assertIsNone(row["revoked_at"])
|
||||
|
||||
def test_add_without_name_stores_null_name(self):
|
||||
app_uuid = str(uuidlib.uuid4())
|
||||
status, _ = self._add(app_uuid)
|
||||
self.assertEqual(201, status)
|
||||
self.assertIsNone(self._row(app_uuid)["name"])
|
||||
|
||||
def test_add_duplicate_active_or_revoked_is_409(self):
|
||||
app_uuid = str(uuidlib.uuid4())
|
||||
self._add(app_uuid)
|
||||
status, body = self._add(app_uuid)
|
||||
self.assertEqual(409, status)
|
||||
self.assertEqual(409, body["error"])
|
||||
self.assertTrue(body["message"])
|
||||
status, _ = self._request("DELETE", BASE_PATH + "/" + app_uuid, headers=self.admin)
|
||||
self.assertEqual(200, status)
|
||||
status, body = self._add(app_uuid)
|
||||
self.assertEqual(409, status)
|
||||
|
||||
def test_add_invalid_or_noncanonical_uuid_is_400(self):
|
||||
canonical = str(uuidlib.uuid4())
|
||||
for bad in ("not-a-uuid", "", "{" + canonical.upper() + "}", canonical.upper(),
|
||||
canonical.replace("-", ""), "urn:uuid:" + canonical):
|
||||
status, body = self._add(bad)
|
||||
self.assertEqual(400, status, bad)
|
||||
self.assertEqual(400, body["error"])
|
||||
status, _ = self._request("POST", BASE_PATH, {}, self.admin)
|
||||
self.assertEqual(400, status)
|
||||
status, _ = self._request("POST", BASE_PATH, {"uuid": 12345}, self.admin)
|
||||
self.assertEqual(400, status)
|
||||
status, _ = self._request("POST", BASE_PATH, {"name": "no-uuid"}, self.admin)
|
||||
self.assertEqual(400, status)
|
||||
|
||||
def test_revoke_revoke_again_reinstate_lifecycle(self):
|
||||
app_uuid = str(uuidlib.uuid4())
|
||||
self._add(app_uuid)
|
||||
status, body = self._request("DELETE", BASE_PATH + "/" + app_uuid, headers=self.admin)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("revoked", body["status"])
|
||||
revoked_at = self._row(app_uuid)["revoked_at"]
|
||||
self.assertIsNotNone(revoked_at)
|
||||
self.assertEqual("revoked", self._row(app_uuid)["status"])
|
||||
status, body = self._request("DELETE", BASE_PATH + "/" + app_uuid, headers=self.admin)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(revoked_at, self._row(app_uuid)["revoked_at"])
|
||||
status, body = self._request(
|
||||
"POST", BASE_PATH + "/" + app_uuid + "/reinstate", headers=self.admin
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("active", body["status"])
|
||||
row = self._row(app_uuid)
|
||||
self.assertEqual("active", row["status"])
|
||||
self.assertIsNone(row["revoked_at"])
|
||||
|
||||
def test_revoke_missing_is_404(self):
|
||||
status, body = self._request(
|
||||
"DELETE", BASE_PATH + "/" + str(uuidlib.uuid4()), headers=self.admin
|
||||
)
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual(404, body["error"])
|
||||
|
||||
def test_reinstate_missing_is_404(self):
|
||||
status, body = self._request(
|
||||
"POST", BASE_PATH + "/" + str(uuidlib.uuid4()) + "/reinstate", headers=self.admin
|
||||
)
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual(404, body["error"])
|
||||
|
||||
def test_admin_actions_audited_with_null_identity(self):
|
||||
app_uuid = str(uuidlib.uuid4())
|
||||
self._add(app_uuid)
|
||||
self._request("DELETE", BASE_PATH + "/" + app_uuid, headers=self.admin)
|
||||
self._request("POST", BASE_PATH + "/" + app_uuid + "/reinstate", headers=self.admin)
|
||||
entries = self._audit()
|
||||
self.assertEqual(["add_app", "remove_app", "add_app"], [e["action"] for e in entries])
|
||||
for entry in entries:
|
||||
self.assertEqual("ok", entry["result"])
|
||||
self.assertIsNone(entry["user_id"])
|
||||
self.assertIsNone(entry["app_registration_uuid"])
|
||||
self.assertIsNone(entry["agent_id"])
|
||||
self.assertEqual("/api/broker/app_registrations/" + app_uuid, entry["path"])
|
||||
|
||||
def test_unauthenticated_requests_are_401(self):
|
||||
checks = (
|
||||
("GET", BASE_PATH, None),
|
||||
("POST", BASE_PATH, {"uuid": str(uuidlib.uuid4())}),
|
||||
("DELETE", BASE_PATH + "/" + str(uuidlib.uuid4()), None),
|
||||
("POST", BASE_PATH + "/" + str(uuidlib.uuid4()) + "/reinstate", None),
|
||||
)
|
||||
for method, path, payload in checks:
|
||||
status, body = self._request(method, path, payload)
|
||||
self.assertEqual(401, status, method + " " + path)
|
||||
self.assertEqual(401, body["error"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,213 @@
|
||||
import contextlib
|
||||
import datetime
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
AUDIT_FIELDS = [
|
||||
"ts",
|
||||
"user_id",
|
||||
"app_registration_uuid",
|
||||
"agent_id",
|
||||
"action",
|
||||
"path",
|
||||
"result",
|
||||
]
|
||||
|
||||
ALL_ACTIONS = [
|
||||
"register",
|
||||
"read",
|
||||
"denied",
|
||||
"login",
|
||||
"logout",
|
||||
"add_app",
|
||||
"remove_app",
|
||||
"add_user",
|
||||
"remove_user",
|
||||
]
|
||||
|
||||
ADMIN_ACTIONS = [
|
||||
"login",
|
||||
"logout",
|
||||
"add_app",
|
||||
"remove_app",
|
||||
"add_user",
|
||||
"remove_user",
|
||||
]
|
||||
|
||||
|
||||
class AuditTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.env_path = os.path.join(tmp.name, ".env")
|
||||
self.db_path = os.path.join(tmp.name, "broker.db")
|
||||
self.audit_path = os.path.join(tmp.name, "broker.audit.log")
|
||||
|
||||
def _boot(self, extra_env=None):
|
||||
env = {
|
||||
"BROKER_ENV_PATH": self.env_path,
|
||||
"BROKER_DB_PATH": self.db_path,
|
||||
"BROKER_AUDIT_LOG_PATH": self.audit_path,
|
||||
}
|
||||
if extra_env:
|
||||
env.update(extra_env)
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
broker = coreci_broker.boot(env)
|
||||
self.addCleanup(broker.store.close)
|
||||
self.addCleanup(broker.audit.close)
|
||||
return broker
|
||||
|
||||
def _raw_lines(self):
|
||||
with open(self.audit_path, "r", encoding="utf-8") as handle:
|
||||
return handle.read().splitlines()
|
||||
|
||||
def _entries(self):
|
||||
return [json.loads(raw) for raw in self._raw_lines()]
|
||||
|
||||
def test_boot_wires_ready_audit_at_mode_0600(self):
|
||||
broker = self._boot()
|
||||
self.assertIsInstance(broker.audit, coreci_broker.Audit)
|
||||
self.assertEqual(self.audit_path, broker.audit.path)
|
||||
self.assertTrue(os.path.isfile(self.audit_path))
|
||||
self.assertEqual(0o600, os.stat(self.audit_path).st_mode & 0o777)
|
||||
|
||||
def test_audit_mode_0600_independent_of_umask(self):
|
||||
old_umask = os.umask(0o777)
|
||||
try:
|
||||
broker = self._boot()
|
||||
finally:
|
||||
os.umask(old_umask)
|
||||
broker.audit.write("login", path="/ui/login")
|
||||
self.assertEqual(0o600, os.stat(self.audit_path).st_mode & 0o777)
|
||||
|
||||
def test_each_write_appends_exactly_one_valid_json_line(self):
|
||||
broker = self._boot()
|
||||
self.assertEqual(0, len(self._raw_lines()))
|
||||
broker.audit.write("login", path="/api/broker/auth/login")
|
||||
broker.audit.write(
|
||||
"read",
|
||||
path="coreci/proxmox",
|
||||
user_id="staff-7",
|
||||
app_registration_uuid="0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a",
|
||||
agent_id="proxmox",
|
||||
)
|
||||
broker.audit.write("denied", path="coreci/gitea", result="denied")
|
||||
lines = self._raw_lines()
|
||||
self.assertEqual(3, len(lines))
|
||||
for line in lines:
|
||||
self.assertTrue(line)
|
||||
entries = self._entries()
|
||||
for entry in entries:
|
||||
self.assertEqual(AUDIT_FIELDS, list(entry))
|
||||
self.assertEqual(["login", "read", "denied"], [e["action"] for e in entries])
|
||||
|
||||
def test_all_nine_action_verbs_accepted_and_written(self):
|
||||
broker = self._boot()
|
||||
for action in ALL_ACTIONS:
|
||||
broker.audit.write(action, path="/x")
|
||||
entries = self._entries()
|
||||
self.assertEqual(9, len(entries))
|
||||
self.assertEqual(sorted(ALL_ACTIONS), sorted(e["action"] for e in entries))
|
||||
|
||||
def test_unknown_action_verb_raises_and_appends_nothing(self):
|
||||
broker = self._boot()
|
||||
for bad in ("bogus", "", "LOGIN", None, "rotate"):
|
||||
with self.assertRaises(ValueError):
|
||||
broker.audit.write(bad, path="/x")
|
||||
with self.assertRaises(ValueError):
|
||||
broker.audit.write("read", path="/x", result="bogus")
|
||||
self.assertEqual(0, len(self._raw_lines()))
|
||||
|
||||
def test_admin_actions_carry_null_identity_fields(self):
|
||||
broker = self._boot()
|
||||
for action in ADMIN_ACTIONS:
|
||||
broker.audit.write(action, path="/api/broker/app_registrations")
|
||||
entries = self._entries()
|
||||
self.assertEqual(len(ADMIN_ACTIONS), len(entries))
|
||||
for entry in entries:
|
||||
self.assertIsNone(entry["user_id"])
|
||||
self.assertIsNone(entry["app_registration_uuid"])
|
||||
self.assertIsNone(entry["agent_id"])
|
||||
broker.audit.write(
|
||||
"register",
|
||||
path="/api/broker/agents/register",
|
||||
user_id="staff-7",
|
||||
app_registration_uuid="0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a",
|
||||
agent_id="proxmox",
|
||||
)
|
||||
entry = self._entries()[-1]
|
||||
self.assertEqual("staff-7", entry["user_id"])
|
||||
self.assertEqual("0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a", entry["app_registration_uuid"])
|
||||
self.assertEqual("proxmox", entry["agent_id"])
|
||||
|
||||
def test_append_only_preserves_every_line_across_many_writes(self):
|
||||
broker = self._boot()
|
||||
for i in range(50):
|
||||
broker.audit.write("read", path=f"coreci/agent-{i}")
|
||||
entries = self._entries()
|
||||
self.assertEqual(50, len(entries))
|
||||
self.assertEqual("coreci/agent-0", entries[0]["path"])
|
||||
self.assertEqual("coreci/agent-49", entries[-1]["path"])
|
||||
self.assertEqual(
|
||||
[f"coreci/agent-{i}" for i in range(50)],
|
||||
[e["path"] for e in entries],
|
||||
)
|
||||
|
||||
def test_reboot_appends_without_truncating_history(self):
|
||||
first = self._boot()
|
||||
first.audit.write("login", path="/ui/login")
|
||||
first.store.close()
|
||||
first.audit.close()
|
||||
second = self._boot()
|
||||
second.audit.write("logout", path="/ui/logout")
|
||||
entries = self._entries()
|
||||
self.assertEqual(["login", "logout"], [e["action"] for e in entries])
|
||||
self.assertEqual(0o600, os.stat(self.audit_path).st_mode & 0o777)
|
||||
|
||||
def test_ts_parses_as_iso8601_utc(self):
|
||||
broker = self._boot()
|
||||
broker.audit.write("read", path="coreci/proxmox", user_id="staff-7")
|
||||
for entry in self._entries():
|
||||
parsed = datetime.datetime.fromisoformat(entry["ts"])
|
||||
self.assertEqual(datetime.timedelta(0), parsed.utcoffset())
|
||||
|
||||
def test_result_values_written(self):
|
||||
broker = self._boot()
|
||||
broker.audit.write("read", path="coreci/proxmox", result="ok")
|
||||
broker.audit.write("read", path="coreci/gitea", result="denied")
|
||||
broker.audit.write("read", path="coreci/boom", result="error")
|
||||
self.assertEqual(["ok", "denied", "error"], [e["result"] for e in self._entries()])
|
||||
|
||||
def test_concurrent_writes_are_serialized(self):
|
||||
broker = self._boot()
|
||||
threads = []
|
||||
for t in range(4):
|
||||
def worker(t=t):
|
||||
for i in range(10):
|
||||
broker.audit.write("read", path=f"coreci/t{t}-i{i}")
|
||||
thread = threading.Thread(target=worker)
|
||||
threads.append(thread)
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
for thread in threads:
|
||||
thread.join()
|
||||
entries = self._entries()
|
||||
self.assertEqual(40, len(entries))
|
||||
self.assertEqual(
|
||||
sorted(f"coreci/t{t}-i{i}" for t in range(4) for i in range(10)),
|
||||
sorted(e["path"] for e in entries),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,305 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
AUDIT_PATH = "/api/broker/audit"
|
||||
UUID_A = "0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a"
|
||||
UUID_B = "3f2d8a04-9c17-4b1e-8d2f-6a5b4c3d2e1f"
|
||||
|
||||
SEED = [
|
||||
{"action": "read", "path": "coreci/proxmox", "user_id": "staff-1",
|
||||
"app_registration_uuid": UUID_A, "agent_id": "proxmox"},
|
||||
{"action": "read", "path": "coreci/gitea", "user_id": "staff-2",
|
||||
"app_registration_uuid": UUID_A, "agent_id": "gitea"},
|
||||
{"action": "denied", "path": "coreci/gitea", "result": "denied",
|
||||
"user_id": "staff-2", "app_registration_uuid": UUID_B, "agent_id": "gitea"},
|
||||
{"action": "add_app", "path": "/api/broker/app_registrations"},
|
||||
{"action": "read", "path": "coreci/proxmox", "user_id": "staff-1",
|
||||
"app_registration_uuid": UUID_B, "agent_id": "proxmox"},
|
||||
{"action": "login", "path": "/api/broker/auth/login"},
|
||||
{"action": "remove_user", "path": "/api/broker/user_allowlist/staff-2"},
|
||||
]
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
class AuditQueryTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.audit_path = os.path.join(tmp.name, "broker.audit.log")
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot({
|
||||
"BROKER_ENV_PATH": os.path.join(tmp.name, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(tmp.name, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": self.audit_path,
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
})
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.auth = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
|
||||
def _request_json(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=5) as response:
|
||||
return response.status, json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, json.loads(error.read().decode("utf-8"))
|
||||
|
||||
def _get(self, params=None, headers=None):
|
||||
path = AUDIT_PATH
|
||||
if params:
|
||||
path += "?" + urllib.parse.urlencode(params)
|
||||
return self._request_json(
|
||||
"GET", path, headers=self.auth if headers is None else headers
|
||||
)
|
||||
|
||||
def _seed(self, specs):
|
||||
for spec in specs:
|
||||
self.broker.audit.write(**spec)
|
||||
time.sleep(0.005)
|
||||
|
||||
def _ts(self):
|
||||
values = []
|
||||
with open(self.audit_path, "r", encoding="utf-8") as handle:
|
||||
for raw in handle:
|
||||
try:
|
||||
values.append(json.loads(raw)["ts"])
|
||||
except (ValueError, TypeError, KeyError):
|
||||
continue
|
||||
return values
|
||||
|
||||
def _paths(self, body):
|
||||
return [entry["path"] for entry in body["entries"]]
|
||||
|
||||
def test_no_filters_returns_all_entries_with_null_cursor(self):
|
||||
self._seed(SEED)
|
||||
status, body = self._get()
|
||||
self.assertEqual(200, status)
|
||||
with open(self.audit_path, "r", encoding="utf-8") as handle:
|
||||
expected = [json.loads(raw) for raw in handle]
|
||||
self.assertEqual(expected, body["entries"])
|
||||
self.assertEqual(len(SEED), len(body["entries"]))
|
||||
self.assertIsNone(body["next_since"])
|
||||
|
||||
def test_user_id_filter_exact_match(self):
|
||||
self._seed(SEED)
|
||||
status, body = self._get({"user_id": "staff-1"})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["coreci/proxmox", "coreci/proxmox"], self._paths(body))
|
||||
self.assertEqual([UUID_A, UUID_B],
|
||||
[e["app_registration_uuid"] for e in body["entries"]])
|
||||
self.assertIsNone(body["next_since"])
|
||||
|
||||
def test_app_registration_uuid_filter_exact_match(self):
|
||||
self._seed(SEED)
|
||||
status, body = self._get({"app_registration_uuid": UUID_B})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["denied", "read"], [e["action"] for e in body["entries"]])
|
||||
self.assertEqual(["coreci/gitea", "coreci/proxmox"], self._paths(body))
|
||||
status, body = self._get({"app_registration_uuid": "no-such-uuid"})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual([], body["entries"])
|
||||
self.assertIsNone(body["next_since"])
|
||||
|
||||
def test_action_filter_exact_match(self):
|
||||
self._seed(SEED)
|
||||
status, body = self._get({"action": "denied"})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["coreci/gitea"], self._paths(body))
|
||||
status, body = self._get({"action": "read"})
|
||||
self.assertEqual(["coreci/proxmox", "coreci/gitea", "coreci/proxmox"],
|
||||
self._paths(body))
|
||||
status, body = self._get({"action": "rea"})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual([], body["entries"])
|
||||
|
||||
def test_since_filter_is_exclusive(self):
|
||||
self._seed(SEED)
|
||||
ts = self._ts()
|
||||
status, body = self._get({"since": ts[2]})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["add_app", "read", "login", "remove_user"],
|
||||
[e["action"] for e in body["entries"]])
|
||||
self.assertEqual(4, len(body["entries"]))
|
||||
self.assertIsNone(body["next_since"])
|
||||
status, body = self._get({"since": ts[-1]})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual([], body["entries"])
|
||||
self.assertIsNone(body["next_since"])
|
||||
|
||||
def test_until_filter_is_inclusive(self):
|
||||
self._seed(SEED)
|
||||
ts = self._ts()
|
||||
status, body = self._get({"until": ts[2]})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["read", "read", "denied"],
|
||||
[e["action"] for e in body["entries"]])
|
||||
self.assertEqual(3, len(body["entries"]))
|
||||
self.assertIsNone(body["next_since"])
|
||||
|
||||
def test_since_and_until_range_together(self):
|
||||
self._seed(SEED)
|
||||
ts = self._ts()
|
||||
status, body = self._get({"since": ts[1], "until": ts[4]})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["denied", "add_app", "read"], [e["action"] for e in body["entries"]])
|
||||
status, body = self._get({"since": ts[3], "until": ts[3]})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual([], body["entries"])
|
||||
|
||||
def test_combined_filters(self):
|
||||
self._seed(SEED)
|
||||
ts = self._ts()
|
||||
status, body = self._get({"user_id": "staff-1", "action": "read"})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["coreci/proxmox", "coreci/proxmox"], self._paths(body))
|
||||
status, body = self._get({"user_id": "staff-2", "action": "read"})
|
||||
self.assertEqual(["coreci/gitea"], self._paths(body))
|
||||
status, body = self._get(
|
||||
{"app_registration_uuid": UUID_B, "since": ts[2], "until": ts[6]}
|
||||
)
|
||||
self.assertEqual(["coreci/proxmox"], self._paths(body))
|
||||
status, body = self._get(
|
||||
{"user_id": "staff-1", "since": ts[0], "until": ts[4]}
|
||||
)
|
||||
self.assertEqual(["coreci/proxmox"], self._paths(body))
|
||||
status, body = self._get(
|
||||
{"user_id": "staff-1", "action": "denied",
|
||||
"app_registration_uuid": UUID_A, "since": ts[0], "until": ts[6]}
|
||||
)
|
||||
self.assertEqual([], body["entries"])
|
||||
|
||||
def test_limit_pagination_with_next_since_cursor(self):
|
||||
specs = [{"action": "read", "path": "coreci/agent-%d" % i} for i in range(5)]
|
||||
self._seed(specs)
|
||||
ts = self._ts()
|
||||
status, body = self._get({"limit": 2})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["coreci/agent-0", "coreci/agent-1"], self._paths(body))
|
||||
self.assertEqual(ts[1], body["next_since"])
|
||||
status, body = self._get({"since": body["next_since"], "limit": 2})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["coreci/agent-2", "coreci/agent-3"], self._paths(body))
|
||||
self.assertEqual(ts[3], body["next_since"])
|
||||
status, body = self._get({"since": ts[1]})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["coreci/agent-2", "coreci/agent-3", "coreci/agent-4"],
|
||||
self._paths(body))
|
||||
self.assertIsNone(body["next_since"])
|
||||
status, body = self._get({"since": ts[4]})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual([], body["entries"])
|
||||
self.assertIsNone(body["next_since"])
|
||||
|
||||
def test_limit_zero_and_negative_clamp_to_one(self):
|
||||
self._seed([{"action": "read", "path": "coreci/a"},
|
||||
{"action": "read", "path": "coreci/b"},
|
||||
{"action": "read", "path": "coreci/c"}])
|
||||
ts = self._ts()
|
||||
for bad in (0, -3):
|
||||
status, body = self._get({"limit": bad})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["coreci/a"], self._paths(body))
|
||||
self.assertEqual(ts[0], body["next_since"])
|
||||
|
||||
def test_limit_above_max_and_non_numeric_are_clamped(self):
|
||||
self._seed([{"action": "read", "path": "coreci/a"},
|
||||
{"action": "read", "path": "coreci/b"},
|
||||
{"action": "read", "path": "coreci/c"}])
|
||||
for value in (5000, "abc"):
|
||||
status, body = self._get({"limit": value})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(3, len(body["entries"]))
|
||||
self.assertIsNone(body["next_since"])
|
||||
|
||||
def test_default_limit_is_exactly_100(self):
|
||||
specs = [{"action": "read", "path": "coreci/agent-%d" % i} for i in range(100)]
|
||||
self._seed(specs)
|
||||
ts = self._ts()
|
||||
status, body = self._get()
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(100, len(body["entries"]))
|
||||
self.assertEqual("coreci/agent-99", body["entries"][-1]["path"])
|
||||
self.assertEqual(ts[99], body["next_since"])
|
||||
status, body = self._get({"since": ts[99]})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual([], body["entries"])
|
||||
self.assertIsNone(body["next_since"])
|
||||
|
||||
def test_invalid_since_or_until_is_400(self):
|
||||
self._seed(SEED)
|
||||
status, body = self._get({"since": "not-a-timestamp"})
|
||||
self.assertEqual(400, status)
|
||||
self.assertEqual(400, body["error"])
|
||||
self.assertTrue(body["message"])
|
||||
status, body = self._get({"until": "2026-02-30T00:00:00Z"})
|
||||
self.assertEqual(400, status)
|
||||
self.assertEqual(400, body["error"])
|
||||
self.assertTrue(body["message"])
|
||||
status, body = self._get({"since": ""})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(len(SEED), len(body["entries"]))
|
||||
|
||||
def test_malformed_lines_skipped(self):
|
||||
self._seed([{"action": "read", "path": "coreci/a"},
|
||||
{"action": "read", "path": "coreci/b"}])
|
||||
with open(self.audit_path, "a", encoding="utf-8") as handle:
|
||||
handle.write("{not json\n")
|
||||
handle.write("[1, 2, 3]\n")
|
||||
self._seed([{"action": "read", "path": "coreci/c"}])
|
||||
ts = self._ts()
|
||||
self.assertEqual(3, len(ts))
|
||||
status, body = self._get()
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["coreci/a", "coreci/b", "coreci/c"], self._paths(body))
|
||||
self.assertIsNone(body["next_since"])
|
||||
status, body = self._get({"limit": 2})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["coreci/a", "coreci/b"], self._paths(body))
|
||||
self.assertEqual(ts[1], body["next_since"])
|
||||
status, body = self._get({"since": ts[1], "action": "read"})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(["coreci/c"], self._paths(body))
|
||||
|
||||
def test_unauthenticated_is_401(self):
|
||||
self._seed(SEED)
|
||||
status, body = self._get(headers={})
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(401, body["error"])
|
||||
status, body = self._get(
|
||||
headers={"Authorization": _basic_header(ADMIN_USER, "wrong-pass")}
|
||||
)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(401, body["error"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,188 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import datetime
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
AUDIT_PATH = "/api/broker/audit"
|
||||
STATE_PATH = "/api/broker/agents/%s/state"
|
||||
AUTH_401_BODY = {"error": 401, "message": "invalid app credentials"}
|
||||
BAD_JSON_BODY = {"error": 400, "message": "invalid JSON body"}
|
||||
NEST_DEPTH = 50000
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
class AuthAuditTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(dict(self.env))
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
self.app_uuid = str(uuidlib.uuid4())
|
||||
self.user_id = "ceo"
|
||||
self.agent_id = "agent-1"
|
||||
self._seed(self.app_uuid, self.user_id)
|
||||
self.token = self._register()["token"]
|
||||
self.app = {"Authorization": "App %s %s" % (self.app_uuid, self.token)}
|
||||
|
||||
def _raw(self, method, path, data, headers):
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
return response.status, json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, json.loads(error.read().decode("utf-8"))
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
return self._raw(method, path, data, headers)
|
||||
|
||||
def _seed(self, app_uuid, user_id):
|
||||
status, _ = self._request("POST", APPS_PATH, {"uuid": app_uuid}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
status, _ = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def _register(self):
|
||||
payload = {"agent_id": self.agent_id, "app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id, "tags": ["coreci"]}
|
||||
status, body = self._request("POST", REGISTER_PATH, payload)
|
||||
self.assertEqual(201, status)
|
||||
return body
|
||||
|
||||
def _denied(self):
|
||||
status, body = self._request("GET", AUDIT_PATH + "?action=denied",
|
||||
headers=self.admin)
|
||||
self.assertEqual(200, status)
|
||||
return body["entries"]
|
||||
|
||||
def _assert_denied_line(self, entry, app_uuid, path):
|
||||
self.assertEqual("denied", entry["action"])
|
||||
self.assertEqual("denied", entry["result"])
|
||||
self.assertEqual(app_uuid, entry["app_registration_uuid"])
|
||||
self.assertEqual(path, entry["path"])
|
||||
self.assertIsNone(entry["user_id"])
|
||||
self.assertIsNone(entry["agent_id"])
|
||||
|
||||
def test_deep_nested_json_body_is_400_not_500(self):
|
||||
data = ("[" * NEST_DEPTH).encode("utf-8")
|
||||
self.assertLess(len(data), 1024 * 1024)
|
||||
status, body = self._raw("PUT", STATE_PATH % self.agent_id, data, self.app)
|
||||
self.assertEqual(400, status)
|
||||
self.assertEqual(BAD_JSON_BODY, body)
|
||||
status, body = self._request("GET", "/healthz")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
|
||||
def test_wrong_token_denied_audit_carries_uuid_and_path(self):
|
||||
path = STATE_PATH % self.agent_id
|
||||
status, body = self._request(
|
||||
"GET", path, headers={"Authorization": "App %s %s" % (self.app_uuid, "A" * 44)}
|
||||
)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
entries = self._denied()
|
||||
self.assertEqual(1, len(entries))
|
||||
self._assert_denied_line(entries[0], self.app_uuid, path)
|
||||
|
||||
def test_expired_or_absent_token_same_denied_audit(self):
|
||||
path = STATE_PATH % self.agent_id
|
||||
past = (datetime.datetime.now(datetime.timezone.utc)
|
||||
- datetime.timedelta(seconds=60)).isoformat()
|
||||
self.broker.store.conn.execute(
|
||||
"UPDATE agent_registrations SET token_expires_at = ? WHERE agent_id = ?",
|
||||
(past, self.agent_id),
|
||||
)
|
||||
self.broker.store.conn.commit()
|
||||
status, body = self._request("GET", path, headers=self.app)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
status, body = self._request(
|
||||
"GET", path, headers={"Authorization": "App %s" % self.app_uuid}
|
||||
)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
entries = self._denied()
|
||||
self.assertEqual(2, len(entries))
|
||||
for entry in entries:
|
||||
self._assert_denied_line(entry, self.app_uuid, path)
|
||||
|
||||
def test_missing_app_header_401_writes_no_denied_audit(self):
|
||||
self.assertEqual(0, len(self._denied()))
|
||||
path = STATE_PATH % self.agent_id
|
||||
status, body = self._request("GET", path, headers={})
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
status, body = self._request("GET", path, headers=self.admin)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
self.assertEqual(0, len(self._denied()))
|
||||
|
||||
def test_malformed_uuid_denied_audit_carries_null_uuid(self):
|
||||
path = STATE_PATH % self.agent_id
|
||||
status, body = self._request(
|
||||
"GET", path, headers={"Authorization": "App not-a-uuid %s" % self.token}
|
||||
)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
entries = self._denied()
|
||||
self.assertEqual(1, len(entries))
|
||||
self._assert_denied_line(entries[0], None, path)
|
||||
|
||||
def test_admin_audit_action_filter_returns_denied_lines(self):
|
||||
path = STATE_PATH % self.agent_id
|
||||
for header in ("App %s wrong" % self.app_uuid,
|
||||
"App not-a-uuid %s" % self.token):
|
||||
status, _ = self._request("GET", path, headers={"Authorization": header})
|
||||
self.assertEqual(401, status)
|
||||
query = urllib.parse.urlencode({"action": "denied"})
|
||||
status, body = self._request("GET", AUDIT_PATH + "?" + query,
|
||||
headers=self.admin)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(2, len(body["entries"]))
|
||||
self.assertEqual([self.app_uuid, None],
|
||||
[entry["app_registration_uuid"] for entry in body["entries"]])
|
||||
for entry in body["entries"]:
|
||||
self.assertEqual(path, entry["path"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,308 @@
|
||||
import ast
|
||||
import base64
|
||||
import contextlib
|
||||
import datetime
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
import coreci_broker_client
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
|
||||
GITEA_SECRET = {"url": "https://git.coreci.dev", "token": "git-tok"}
|
||||
CLIENT_SOURCE_PATH = os.path.abspath(
|
||||
os.path.join(os.path.dirname(__file__), os.pardir, "coreci_broker_client.py")
|
||||
)
|
||||
ALLOWED_CLIENT_IMPORTS = {"urllib.request", "urllib.parse", "json", "time"}
|
||||
SECRETS_PROXMOX_PATH = "/api/broker/secrets/coreci/proxmox"
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
class ClientHygieneTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
with open(CLIENT_SOURCE_PATH, "r", encoding="utf-8") as handle:
|
||||
self.source = handle.read()
|
||||
self.tree = ast.parse(self.source)
|
||||
|
||||
def test_imports_are_stdlib_only(self):
|
||||
modules = set()
|
||||
for node in ast.walk(self.tree):
|
||||
if isinstance(node, ast.Import):
|
||||
modules.update(alias.name for alias in node.names)
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
modules.add(node.module or "")
|
||||
self.assertIn("urllib.request", modules)
|
||||
for module in modules:
|
||||
self.assertIn(module, ALLOWED_CLIENT_IMPORTS, module)
|
||||
|
||||
def test_single_module_docstring_and_no_comments(self):
|
||||
docstring = ast.get_docstring(self.tree)
|
||||
self.assertIsNotNone(docstring)
|
||||
self.assertNotIn("\n", docstring)
|
||||
for node in ast.walk(self.tree):
|
||||
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
|
||||
self.assertIsNone(ast.get_docstring(node))
|
||||
for number, line in enumerate(self.source.splitlines(), 1):
|
||||
self.assertFalse(line.lstrip().startswith("#"), "comment on line %d" % number)
|
||||
|
||||
def test_broker_error_carries_status_and_message(self):
|
||||
error = coreci_broker_client.BrokerError(403, "forbidden")
|
||||
self.assertEqual(403, error.status)
|
||||
self.assertEqual("forbidden", error.message)
|
||||
self.assertEqual("forbidden", str(error))
|
||||
|
||||
|
||||
class _ClientCase(unittest.TestCase):
|
||||
extra_env = {}
|
||||
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.creds_path = os.path.join(self.tmp, "creds.json")
|
||||
with open(self.creds_path, "w", encoding="utf-8") as handle:
|
||||
handle.write(json.dumps(
|
||||
{"coreci/proxmox": PROXMOX_SECRET, "coreci/gitea": GITEA_SECRET}
|
||||
))
|
||||
env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
"BROKER_CREDS_SOURCE": "file",
|
||||
"BROKER_CREDS_FILE": self.creds_path,
|
||||
}
|
||||
env.update(self.extra_env)
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(env)
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
self.app_uuid = str(uuidlib.uuid4())
|
||||
self.user_id = "ceo"
|
||||
self._seed()
|
||||
|
||||
def _raw_request(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
raw = response.read().decode("utf-8")
|
||||
return response.status, json.loads(raw) if raw else None
|
||||
except urllib.error.HTTPError as error:
|
||||
raw = error.read().decode("utf-8")
|
||||
return error.code, json.loads(raw) if raw else None
|
||||
|
||||
def _seed(self):
|
||||
status, _ = self._raw_request("POST", APPS_PATH, {"uuid": self.app_uuid}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
status, _ = self._raw_request("POST", USERS_PATH, {"user_id": self.user_id}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def _client(self, agent_id="proxmox", tags=("coreci",)):
|
||||
return coreci_broker_client.BrokerClient(
|
||||
self.base, agent_id, self.app_uuid, self.user_id, list(tags)
|
||||
)
|
||||
|
||||
def _count_requests(self, client):
|
||||
calls = []
|
||||
original = client._request
|
||||
|
||||
def counting(method, path, payload=None, headers=None):
|
||||
calls.append((method, path))
|
||||
return original(method, path, payload, headers)
|
||||
|
||||
def restore():
|
||||
client._request = original
|
||||
|
||||
client._request = counting
|
||||
return calls, restore
|
||||
|
||||
|
||||
class ClientFlowTests(_ClientCase):
|
||||
def test_ensure_registered_from_empty_state_populates_token_fields(self):
|
||||
client = self._client()
|
||||
state = {}
|
||||
returned = client.ensure_registered(state)
|
||||
self.assertIs(state, returned)
|
||||
token = state["current_token"]
|
||||
self.assertEqual(44, len(token))
|
||||
self.assertTrue(token.isascii())
|
||||
expires = datetime.datetime.fromisoformat(state["token_expires_at"])
|
||||
self.assertGreater(expires, datetime.datetime.now(datetime.timezone.utc))
|
||||
self.assertEqual("proxmox", state["agent_id"])
|
||||
self.assertEqual(self.app_uuid, state["app_registration_uuid"])
|
||||
self.assertEqual(self.user_id, state["user_id"])
|
||||
self.assertEqual(["coreci"], state["tags"])
|
||||
status, body = self._raw_request(
|
||||
"GET", SECRETS_PROXMOX_PATH,
|
||||
headers={"Authorization": "App %s %s" % (self.app_uuid, token)},
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(PROXMOX_SECRET, body["data"])
|
||||
|
||||
def test_first_register_merges_pepper_into_state(self):
|
||||
client = self._client()
|
||||
state = client.ensure_registered({})
|
||||
self.assertIn("pepper", state)
|
||||
self.assertEqual(
|
||||
self.broker.store.get_config("BROKER_INSTANCE_PEPPER"), state["pepper"]
|
||||
)
|
||||
|
||||
def test_ensure_registered_with_live_token_makes_no_network_call(self):
|
||||
client = self._client()
|
||||
state = client.ensure_registered({})
|
||||
snapshot = dict(state)
|
||||
calls, restore = self._count_requests(client)
|
||||
try:
|
||||
returned = client.ensure_registered(state)
|
||||
finally:
|
||||
restore()
|
||||
self.assertIs(state, returned)
|
||||
self.assertEqual([], calls)
|
||||
self.assertEqual(snapshot, state)
|
||||
|
||||
def test_reregister_with_valid_token_pepperless_response_does_not_crash(self):
|
||||
client = self._client()
|
||||
state = client.ensure_registered({})
|
||||
pepper, token, expires = state["pepper"], state["current_token"], state["token_expires_at"]
|
||||
status, body = self._raw_request(
|
||||
"POST", coreci_broker_client.REGISTER_PATH,
|
||||
{"agent_id": "proxmox", "app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id, "tags": ["coreci"]},
|
||||
{"Authorization": "App %s %s" % (self.app_uuid, token)},
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertNotIn("pepper", body)
|
||||
returned = client.register(state)
|
||||
self.assertIs(state, returned)
|
||||
self.assertEqual(pepper, state["pepper"])
|
||||
self.assertEqual(token, state["current_token"])
|
||||
self.assertEqual(expires, state["token_expires_at"])
|
||||
|
||||
def test_put_then_get_state_roundtrip_through_broker(self):
|
||||
client = self._client()
|
||||
state = client.ensure_registered({})
|
||||
state["scratchpad"] = {"calls": 3, "notes": ["x"]}
|
||||
self.assertTrue(client.put_state(state))
|
||||
document = client.get_state(state)
|
||||
self.assertEqual(state, document)
|
||||
|
||||
def test_get_state_returns_none_before_first_put(self):
|
||||
client = self._client()
|
||||
state = client.ensure_registered({})
|
||||
self.assertIsNone(client.get_state(state))
|
||||
|
||||
def test_read_secret_returns_secret_document(self):
|
||||
client = self._client()
|
||||
state = client.ensure_registered({})
|
||||
document = client.read_secret("coreci/proxmox", state)
|
||||
self.assertEqual("coreci/proxmox", document["path"])
|
||||
self.assertEqual(PROXMOX_SECRET, document["data"])
|
||||
self.assertRegex(document["version"], r"^[0-9a-f]{8}$")
|
||||
self.assertIn("updated_at", document)
|
||||
|
||||
def test_cross_path_read_secret_raises_403_broker_error(self):
|
||||
client = self._client()
|
||||
state = client.ensure_registered({})
|
||||
with self.assertRaises(coreci_broker_client.BrokerError) as caught:
|
||||
client.read_secret("coreci/gitea", state)
|
||||
self.assertEqual(403, caught.exception.status)
|
||||
self.assertTrue(caught.exception.message)
|
||||
|
||||
def test_missing_secret_read_raises_404_broker_error(self):
|
||||
client = self._client("nonexistent")
|
||||
state = client.ensure_registered({})
|
||||
with self.assertRaises(coreci_broker_client.BrokerError) as caught:
|
||||
client.read_secret("coreci/nonexistent", state)
|
||||
self.assertEqual(404, caught.exception.status)
|
||||
|
||||
def test_read_secret_from_cold_state_registers_then_retries_once(self):
|
||||
client = self._client()
|
||||
state = {}
|
||||
calls, restore = self._count_requests(client)
|
||||
try:
|
||||
document = client.read_secret("coreci/proxmox", state)
|
||||
finally:
|
||||
restore()
|
||||
self.assertEqual(PROXMOX_SECRET, document["data"])
|
||||
self.assertEqual(
|
||||
[("GET", SECRETS_PROXMOX_PATH),
|
||||
("POST", coreci_broker_client.REGISTER_PATH),
|
||||
("GET", SECRETS_PROXMOX_PATH)],
|
||||
calls,
|
||||
)
|
||||
self.assertIn("current_token", state)
|
||||
|
||||
def test_get_state_with_foreign_agent_token_is_403_identity_enforced(self):
|
||||
intruder = self._client("proxmox")
|
||||
other = self._client("gitea-agent")
|
||||
other_state = other.ensure_registered({})
|
||||
intruder.ensure_registered({})
|
||||
with self.assertRaises(coreci_broker_client.BrokerError) as caught:
|
||||
intruder.get_state({"current_token": other_state["current_token"]})
|
||||
self.assertEqual(403, caught.exception.status)
|
||||
|
||||
|
||||
class TtlAutoReregisterTests(_ClientCase):
|
||||
extra_env = {"BROKER_TOKEN_TTL": "1"}
|
||||
|
||||
def test_expired_token_secret_read_auto_reregisters_and_retries(self):
|
||||
client = self._client()
|
||||
state = {}
|
||||
first = client.read_secret("coreci/proxmox", state)
|
||||
self.assertEqual(PROXMOX_SECRET, first["data"])
|
||||
first_expires = datetime.datetime.fromisoformat(state["token_expires_at"])
|
||||
time.sleep(1.2)
|
||||
second = client.read_secret("coreci/proxmox", state)
|
||||
self.assertEqual(PROXMOX_SECRET, second["data"])
|
||||
second_expires = datetime.datetime.fromisoformat(state["token_expires_at"])
|
||||
self.assertGreater(second_expires, first_expires)
|
||||
self.assertIn("pepper", state)
|
||||
|
||||
|
||||
class StaticDeliveryTests(_ClientCase):
|
||||
def test_fetch_client_module_returns_own_source(self):
|
||||
with open(CLIENT_SOURCE_PATH, "r", encoding="utf-8") as handle:
|
||||
source = handle.read()
|
||||
fetched = coreci_broker_client.fetch_client_module(self.base)
|
||||
self.assertEqual(source, fetched)
|
||||
self.assertIn("class BrokerClient", fetched)
|
||||
|
||||
def test_client_module_served_unauthenticated(self):
|
||||
with open(CLIENT_SOURCE_PATH, "r", encoding="utf-8") as handle:
|
||||
source = handle.read()
|
||||
with urllib.request.urlopen(
|
||||
self.base + "/static/coreci_broker_client.py", timeout=10
|
||||
) as response:
|
||||
self.assertEqual(200, response.status)
|
||||
self.assertEqual(source, response.read().decode("utf-8"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,105 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import io
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
|
||||
class LoadConfigTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.env_path = os.path.join(tmp.name, ".env")
|
||||
|
||||
def _boot(self, env):
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
config = coreci_broker.load_config(env)
|
||||
return config, buffer.getvalue()
|
||||
|
||||
def _read_env_file(self):
|
||||
values = {}
|
||||
with open(self.env_path, "r", encoding="utf-8") as handle:
|
||||
for raw in handle:
|
||||
line = raw.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, _, value = line.partition("=")
|
||||
values[key.strip()] = value.strip()
|
||||
return values
|
||||
|
||||
def test_first_boot_writes_env_and_prints_pass_once(self):
|
||||
config, stdout = self._boot({"BROKER_ENV_PATH": self.env_path})
|
||||
self.assertTrue(os.path.isfile(self.env_path))
|
||||
stored = self._read_env_file()
|
||||
admin_pass = stored["BROKER_ADMIN_PASS"]
|
||||
pepper = stored["BROKER_INSTANCE_PEPPER"]
|
||||
self.assertGreaterEqual(len(admin_pass), 32)
|
||||
self.assertEqual(32, len(base64.b64decode(pepper)))
|
||||
self.assertEqual(admin_pass, config["BROKER_ADMIN_PASS"])
|
||||
self.assertEqual(pepper, config["BROKER_INSTANCE_PEPPER"])
|
||||
self.assertEqual(1, stdout.count(admin_pass))
|
||||
self.assertNotIn(pepper, stdout)
|
||||
self.assertEqual(0o600, os.stat(self.env_path).st_mode & 0o777)
|
||||
|
||||
def test_second_boot_prints_nothing(self):
|
||||
first, first_stdout = self._boot({"BROKER_ENV_PATH": self.env_path})
|
||||
second, second_stdout = self._boot({"BROKER_ENV_PATH": self.env_path})
|
||||
self.assertEqual(1, first_stdout.count(first["BROKER_ADMIN_PASS"]))
|
||||
self.assertEqual("", second_stdout)
|
||||
self.assertEqual(first["BROKER_ADMIN_PASS"], second["BROKER_ADMIN_PASS"])
|
||||
self.assertEqual(first["BROKER_INSTANCE_PEPPER"], second["BROKER_INSTANCE_PEPPER"])
|
||||
|
||||
def test_env_overrides_dotenv(self):
|
||||
with open(self.env_path, "w", encoding="utf-8") as handle:
|
||||
handle.write("BROKER_BIND=10.1.2.3:9999\n")
|
||||
handle.write('BROKER_ADMIN_USER="ceo"\n')
|
||||
handle.write("BROKER_ADMIN_PASS=oldpass\n")
|
||||
handle.write("BROKER_TOKEN_TTL=100\n")
|
||||
env = {
|
||||
"BROKER_ENV_PATH": self.env_path,
|
||||
"BROKER_ADMIN_PASS": "envpass",
|
||||
"BROKER_TOKEN_TTL": "55",
|
||||
"BROKER_DB_PATH": "/tmp/elsewhere/broker.db",
|
||||
}
|
||||
config, stdout = self._boot(env)
|
||||
self.assertEqual("envpass", config["BROKER_ADMIN_PASS"])
|
||||
self.assertEqual("55", config["BROKER_TOKEN_TTL"])
|
||||
self.assertEqual("/tmp/elsewhere/broker.db", config["BROKER_DB_PATH"])
|
||||
self.assertEqual("10.1.2.3:9999", config["BROKER_BIND"])
|
||||
self.assertEqual("ceo", config["BROKER_ADMIN_USER"])
|
||||
self.assertEqual("300", config["BROKER_SESSION_TTL"])
|
||||
self.assertEqual("", stdout)
|
||||
stored = self._read_env_file()
|
||||
self.assertEqual("oldpass", stored["BROKER_ADMIN_PASS"])
|
||||
self.assertIn("BROKER_INSTANCE_PEPPER", stored)
|
||||
|
||||
def test_defaults_applied(self):
|
||||
config, _ = self._boot({"BROKER_ENV_PATH": self.env_path})
|
||||
self.assertEqual(self.env_path, config["BROKER_ENV_PATH"])
|
||||
self.assertEqual("127.0.0.1:8765", config["BROKER_BIND"])
|
||||
self.assertEqual("admin", config["BROKER_ADMIN_USER"])
|
||||
self.assertEqual("86400", config["BROKER_TOKEN_TTL"])
|
||||
self.assertEqual("300", config["BROKER_SESSION_TTL"])
|
||||
self.assertEqual("broker.db", config["BROKER_DB_PATH"])
|
||||
self.assertEqual("broker.audit.log", config["BROKER_AUDIT_LOG_PATH"])
|
||||
|
||||
def test_reads_process_environ_by_default(self):
|
||||
env = {"BROKER_ENV_PATH": self.env_path, "BROKER_BIND": "127.0.0.1:9000"}
|
||||
with mock.patch.dict(os.environ, env, clear=True):
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
config = coreci_broker.load_config()
|
||||
self.assertEqual("127.0.0.1:9000", config["BROKER_BIND"])
|
||||
self.assertEqual(self.env_path, config["BROKER_ENV_PATH"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,312 @@
|
||||
import contextlib
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
import urllib.parse
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
API_KEY = "sk-test-key"
|
||||
KB_NAME = "creds-coreci-coo"
|
||||
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
|
||||
PROXMOX_UPDATED = "2026-10-01T12:00:00Z"
|
||||
|
||||
|
||||
def _standard_fixtures():
|
||||
kbs = [{"id": "kb-1", "name": KB_NAME}]
|
||||
files = {"kb-1": [{"name": "coreci/proxmox", "id": "f1", "updated_at": PROXMOX_UPDATED}]}
|
||||
contents = {"f1": json.dumps(PROXMOX_SECRET)}
|
||||
return kbs, files, contents
|
||||
|
||||
|
||||
class _StubOwuiHandler(BaseHTTPRequestHandler):
|
||||
kbs = None
|
||||
files = None
|
||||
contents = None
|
||||
failures = None
|
||||
hits = None
|
||||
|
||||
def log_message(self, format, *args):
|
||||
pass
|
||||
|
||||
def _send(self, status, payload):
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def do_GET(self):
|
||||
parsed = urllib.parse.urlsplit(self.path)
|
||||
path = parsed.path
|
||||
self.hits.append((path, self.headers.get("Authorization")))
|
||||
if path in self.failures:
|
||||
self._send(self.failures[path], {"error": self.failures[path]})
|
||||
elif path == "/api/v1/knowledge/":
|
||||
page = max(1, int((urllib.parse.parse_qs(parsed.query).get("page") or ["1"])[0]))
|
||||
self._send(200, {"items": self.kbs[(page - 1) * 30:page * 30], "total": len(self.kbs)})
|
||||
elif path.startswith("/api/v1/knowledge/"):
|
||||
kb_id = path[len("/api/v1/knowledge/"):]
|
||||
kb = next((entry for entry in self.kbs if entry["id"] == kb_id), None)
|
||||
if kb is None:
|
||||
self._send(404, {"error": 404})
|
||||
else:
|
||||
self._send(200, dict(kb, files=self.files.get(kb_id, [])))
|
||||
elif path.startswith("/api/v1/files/") and path.endswith("/data/content"):
|
||||
file_id = path[len("/api/v1/files/"):-len("/data/content")]
|
||||
if file_id in self.contents:
|
||||
self._send(200, {"content": self.contents[file_id]})
|
||||
else:
|
||||
self._send(404, {"error": 404})
|
||||
else:
|
||||
self._send(404, {"error": 404})
|
||||
|
||||
|
||||
class StubOwuiServer:
|
||||
def __init__(self, kbs, files, contents):
|
||||
self.kbs, self.files, self.contents, self.failures = kbs, files, contents, {}
|
||||
self.hits = []
|
||||
handler = type("BoundStubHandler", (_StubOwuiHandler,), {
|
||||
"kbs": kbs, "files": files, "contents": contents,
|
||||
"failures": self.failures, "hits": self.hits,
|
||||
})
|
||||
self.server = ThreadingHTTPServer(("127.0.0.1", 0), handler)
|
||||
threading.Thread(target=self.server.serve_forever, daemon=True).start()
|
||||
|
||||
@property
|
||||
def base_url(self):
|
||||
return "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
|
||||
def hits_for(self, path):
|
||||
return [hit for hit in self.hits if hit[0] == path]
|
||||
|
||||
def close(self):
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
|
||||
|
||||
class FileCredsSourceTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.creds_path = os.path.join(tmp.name, "creds.json")
|
||||
self.document = {
|
||||
"coreci/proxmox": {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"},
|
||||
"coreci/gitea": {"url": "https://git.coreci.dev", "token": "git-tok"},
|
||||
}
|
||||
self._write(self.document)
|
||||
self.source = coreci_broker.FileCredsSource(self.creds_path)
|
||||
|
||||
def _write(self, document):
|
||||
with open(self.creds_path, "w", encoding="utf-8") as handle:
|
||||
json.dump(document, handle)
|
||||
|
||||
def test_fetches_both_paths_with_response_shape(self):
|
||||
for path, secret in self.document.items():
|
||||
result = self.source.get_secret(path)
|
||||
self.assertEqual(path, result["path"])
|
||||
self.assertEqual(secret, result["data"])
|
||||
self.assertIsNone(result["updated_at"])
|
||||
self.assertRegex(result["version"], r"^[0-9a-f]{8}$")
|
||||
|
||||
def test_version_is_short_sha256_of_serialized_data(self):
|
||||
result = self.source.get_secret("coreci/proxmox")
|
||||
serialized = json.dumps(result["data"], sort_keys=True, separators=(",", ":"))
|
||||
expected = hashlib.sha256(serialized.encode("utf-8")).hexdigest()[:8]
|
||||
self.assertEqual(expected, result["version"])
|
||||
|
||||
def test_equal_data_yields_equal_versions(self):
|
||||
first = self.source.get_secret("coreci/proxmox")
|
||||
second = self.source.get_secret("coreci/proxmox")
|
||||
self.assertEqual(first["version"], second["version"])
|
||||
|
||||
def test_missing_path_is_not_found(self):
|
||||
with self.assertRaises(coreci_broker._SecretNotFound):
|
||||
self.source.get_secret("coreci/ghost")
|
||||
|
||||
def test_missing_file_means_all_paths_not_found(self):
|
||||
os.remove(self.creds_path)
|
||||
for path in self.document:
|
||||
with self.assertRaises(coreci_broker._SecretNotFound):
|
||||
self.source.get_secret(path)
|
||||
|
||||
def test_invalid_json_is_source_error(self):
|
||||
with open(self.creds_path, "w", encoding="utf-8") as handle:
|
||||
handle.write("{not-json")
|
||||
with self.assertRaises(coreci_broker._CredsSourceError):
|
||||
self.source.get_secret("coreci/proxmox")
|
||||
|
||||
def test_non_object_document_is_source_error(self):
|
||||
self._write(["not", "a", "mapping"])
|
||||
with self.assertRaises(coreci_broker._CredsSourceError):
|
||||
self.source.get_secret("coreci/proxmox")
|
||||
|
||||
def test_fresh_read_sees_out_of_band_edits(self):
|
||||
first = self.source.get_secret("coreci/proxmox")
|
||||
self.document["coreci/proxmox"]["token"] = "rotated"
|
||||
self._write(self.document)
|
||||
second = self.source.get_secret("coreci/proxmox")
|
||||
self.assertEqual("pve-tok", first["data"]["token"])
|
||||
self.assertEqual("rotated", second["data"]["token"])
|
||||
self.assertNotEqual(first["version"], second["version"])
|
||||
|
||||
|
||||
class OwuiApiCredsSourceTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
kbs, files, contents = _standard_fixtures()
|
||||
self.stub = StubOwuiServer(kbs, files, contents)
|
||||
self.addCleanup(self.stub.close)
|
||||
self.source = coreci_broker.OwuiApiCredsSource(self.stub.base_url, API_KEY)
|
||||
|
||||
def test_roundtrip_returns_response_shape(self):
|
||||
result = self.source.get_secret("coreci/proxmox")
|
||||
self.assertEqual("coreci/proxmox", result["path"])
|
||||
self.assertEqual(PROXMOX_SECRET, result["data"])
|
||||
self.assertEqual(PROXMOX_UPDATED, result["updated_at"])
|
||||
self.assertRegex(result["version"], r"^[0-9a-f]{8}$")
|
||||
|
||||
def test_kb_and_file_map_cached_across_calls(self):
|
||||
self.source.get_secret("coreci/proxmox")
|
||||
self.source.get_secret("coreci/proxmox")
|
||||
self.assertEqual(1, len(self.stub.hits_for("/api/v1/knowledge/")))
|
||||
self.assertEqual(1, len(self.stub.hits_for("/api/v1/knowledge/kb-1")))
|
||||
self.assertEqual(2, len(self.stub.hits_for("/api/v1/files/f1/data/content")))
|
||||
|
||||
def test_bearer_header_present_on_every_upstream_call(self):
|
||||
self.source.get_secret("coreci/proxmox")
|
||||
self.assertTrue(self.stub.hits)
|
||||
for _, auth in self.stub.hits:
|
||||
self.assertEqual("Bearer " + API_KEY, auth)
|
||||
|
||||
def test_refresh_on_miss_picks_up_newly_added_files(self):
|
||||
self.source.get_secret("coreci/proxmox")
|
||||
self.stub.files["kb-1"].append(
|
||||
{"name": "coreci/gitea", "id": "f2", "updated_at": "2026-10-02T00:00:00Z"})
|
||||
self.stub.contents["f2"] = json.dumps({"user": "ceo"})
|
||||
result = self.source.get_secret("coreci/gitea")
|
||||
self.assertEqual({"user": "ceo"}, result["data"])
|
||||
self.assertEqual("2026-10-02T00:00:00Z", result["updated_at"])
|
||||
self.assertEqual(2, len(self.stub.hits_for("/api/v1/knowledge/")))
|
||||
|
||||
def test_unknown_path_is_not_found(self):
|
||||
with self.assertRaises(coreci_broker._SecretNotFound):
|
||||
self.source.get_secret("coreci/ghost")
|
||||
|
||||
def test_missing_kb_is_not_found(self):
|
||||
self.stub.kbs.clear()
|
||||
with self.assertRaises(coreci_broker._SecretNotFound):
|
||||
self.source.get_secret("coreci/proxmox")
|
||||
|
||||
def test_kb_list_404_is_not_found(self):
|
||||
self.stub.failures["/api/v1/knowledge/"] = 404
|
||||
with self.assertRaises(coreci_broker._SecretNotFound):
|
||||
self.source.get_secret("coreci/proxmox")
|
||||
|
||||
def test_kb_detail_404_is_not_found(self):
|
||||
self.stub.failures["/api/v1/knowledge/kb-1"] = 404
|
||||
with self.assertRaises(coreci_broker._SecretNotFound):
|
||||
self.source.get_secret("coreci/proxmox")
|
||||
|
||||
def test_file_content_404_is_not_found(self):
|
||||
del self.stub.contents["f1"]
|
||||
with self.assertRaises(coreci_broker._SecretNotFound):
|
||||
self.source.get_secret("coreci/proxmox")
|
||||
|
||||
def test_upstream_500_is_source_error_not_crash(self):
|
||||
self.stub.failures["/api/v1/files/f1/data/content"] = 500
|
||||
with self.assertRaises(coreci_broker._CredsSourceError):
|
||||
self.source.get_secret("coreci/proxmox")
|
||||
|
||||
def test_kb_list_500_is_source_error(self):
|
||||
self.stub.failures["/api/v1/knowledge/"] = 500
|
||||
with self.assertRaises(coreci_broker._CredsSourceError):
|
||||
self.source.get_secret("coreci/proxmox")
|
||||
|
||||
def test_unavailable_upstream_maps_to_distinct_source_error(self):
|
||||
source = coreci_broker.OwuiApiCredsSource("http://127.0.0.1:9", API_KEY)
|
||||
with self.assertRaises(coreci_broker._CredsSourceError) as caught:
|
||||
source.get_secret("coreci/proxmox")
|
||||
self.assertIn("unavailable", str(caught.exception))
|
||||
|
||||
def test_invalid_content_json_is_source_error(self):
|
||||
self.stub.contents["f1"] = "not-json"
|
||||
with self.assertRaises(coreci_broker._CredsSourceError):
|
||||
self.source.get_secret("coreci/proxmox")
|
||||
|
||||
def test_pagination_finds_kb_beyond_first_page(self):
|
||||
self.stub.kbs[:] = [{"id": "kb-%d" % i, "name": "kb-%d" % i} for i in range(30)]
|
||||
self.stub.kbs.append({"id": "kb-creds", "name": KB_NAME})
|
||||
self.stub.files["kb-creds"] = [
|
||||
{"name": "coreci/proxmox", "id": "f1", "updated_at": PROXMOX_UPDATED}]
|
||||
result = self.source.get_secret("coreci/proxmox")
|
||||
self.assertEqual(PROXMOX_SECRET, result["data"])
|
||||
self.assertEqual(2, len(self.stub.hits_for("/api/v1/knowledge/")))
|
||||
|
||||
|
||||
class CredsSourceFactoryTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
|
||||
def _boot(self, extra):
|
||||
env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
}
|
||||
env.update(extra)
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
broker = coreci_broker.boot(env)
|
||||
self.addCleanup(broker.audit.close)
|
||||
self.addCleanup(broker.store.close)
|
||||
return broker
|
||||
|
||||
def test_default_source_is_file_with_default_path(self):
|
||||
broker = self._boot({})
|
||||
self.assertIsInstance(broker.creds_source, coreci_broker.FileCredsSource)
|
||||
self.assertEqual("data/creds.json", broker.creds_source.path)
|
||||
|
||||
def test_explicit_file_source_with_custom_path(self):
|
||||
creds_path = os.path.join(self.tmp, "custom-creds.json")
|
||||
with open(creds_path, "w", encoding="utf-8") as handle:
|
||||
json.dump({"coreci/proxmox": PROXMOX_SECRET}, handle)
|
||||
broker = self._boot({
|
||||
"BROKER_CREDS_SOURCE": "file",
|
||||
"BROKER_CREDS_FILE": creds_path,
|
||||
})
|
||||
self.assertIsInstance(broker.creds_source, coreci_broker.FileCredsSource)
|
||||
self.assertEqual(creds_path, broker.creds_source.path)
|
||||
self.assertEqual(PROXMOX_SECRET, broker.creds_source.get_secret("coreci/proxmox")["data"])
|
||||
|
||||
def test_owui_source_selected_by_config(self):
|
||||
kbs, files, contents = _standard_fixtures()
|
||||
stub = StubOwuiServer(kbs, files, contents)
|
||||
self.addCleanup(stub.close)
|
||||
broker = self._boot({
|
||||
"BROKER_CREDS_SOURCE": "owui",
|
||||
"BROKER_OWUI_BASE_URL": stub.base_url,
|
||||
"BROKER_OWUI_API_KEY": API_KEY,
|
||||
})
|
||||
source = broker.creds_source
|
||||
self.assertIsInstance(source, coreci_broker.OwuiApiCredsSource)
|
||||
self.assertEqual(stub.base_url, source.base_url)
|
||||
self.assertEqual(API_KEY, source.api_key)
|
||||
self.assertEqual(PROXMOX_SECRET, source.get_secret("coreci/proxmox")["data"])
|
||||
|
||||
def test_unknown_source_value_raises_at_boot(self):
|
||||
with self.assertRaises(ValueError):
|
||||
self._boot({"BROKER_CREDS_SOURCE": "banana"})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,162 @@
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
THREAD_COUNT = 8
|
||||
WRITES_PER_THREAD = 16
|
||||
MAX_BODY_BYTES = 1024 * 1024
|
||||
|
||||
|
||||
class HardeningTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
}
|
||||
|
||||
def _boot_and_serve(self):
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
broker = coreci_broker.boot(dict(self.env))
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
self.addCleanup(broker.audit.close)
|
||||
self.addCleanup(broker.store.close)
|
||||
self.addCleanup(server.server_close)
|
||||
self.addCleanup(server.shutdown)
|
||||
return broker, server
|
||||
|
||||
def test_oversized_declared_content_length_post_rejected(self):
|
||||
broker, server = self._boot_and_serve()
|
||||
port = server.server_address[1]
|
||||
request = urllib.request.Request(
|
||||
"http://127.0.0.1:%d/healthz" % port,
|
||||
data=b"",
|
||||
headers={"Content-Length": str(MAX_BODY_BYTES + 1)},
|
||||
method="POST",
|
||||
)
|
||||
with self.assertRaises(urllib.error.HTTPError) as raised:
|
||||
urllib.request.urlopen(request, timeout=5)
|
||||
error = raised.exception
|
||||
self.assertEqual(400, error.code)
|
||||
payload = json.loads(error.read().decode("utf-8"))
|
||||
self.assertEqual({"error": 400, "message": "body too large"}, payload)
|
||||
|
||||
def test_handler_timeout_attribute_is_30(self):
|
||||
self.assertEqual(30, coreci_broker.BrokerRequestHandler.timeout)
|
||||
broker, server = self._boot_and_serve()
|
||||
self.assertEqual(30, server.RequestHandlerClass.timeout)
|
||||
|
||||
def test_concurrent_store_writes_from_eight_threads(self):
|
||||
db_path = os.path.join(self.tmp, "concurrent.db")
|
||||
store = coreci_broker.Store(db_path)
|
||||
self.addCleanup(store.close)
|
||||
errors = []
|
||||
barrier = threading.Barrier(THREAD_COUNT)
|
||||
|
||||
def worker(index):
|
||||
try:
|
||||
barrier.wait(timeout=5)
|
||||
for slot in range(WRITES_PER_THREAD):
|
||||
store.set_config(
|
||||
"key-%d-%d" % (index, slot), "value-%d-%d" % (index, slot)
|
||||
)
|
||||
store.get_config("key-%d-0" % index)
|
||||
except BaseException as error:
|
||||
errors.append(error)
|
||||
|
||||
threads = [
|
||||
threading.Thread(target=worker, args=(index,))
|
||||
for index in range(THREAD_COUNT)
|
||||
]
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
for thread in threads:
|
||||
thread.join(timeout=30)
|
||||
self.assertEqual([], errors)
|
||||
for index in range(THREAD_COUNT):
|
||||
for slot in range(WRITES_PER_THREAD):
|
||||
key = "key-%d-%d" % (index, slot)
|
||||
expected = "value-%d-%d" % (index, slot)
|
||||
self.assertEqual(expected, store.get_config(key))
|
||||
conn = sqlite3.connect(db_path)
|
||||
self.addCleanup(conn.close)
|
||||
count = conn.execute("SELECT COUNT(*) FROM broker_config").fetchone()[0]
|
||||
self.assertEqual(THREAD_COUNT * WRITES_PER_THREAD, count)
|
||||
|
||||
def test_first_boot_env_write_path_still_works(self):
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
config = coreci_broker.load_config(dict(self.env))
|
||||
env_path = self.env["BROKER_ENV_PATH"]
|
||||
self.assertTrue(os.path.isfile(env_path))
|
||||
self.assertEqual(0o600, os.stat(env_path).st_mode & 0o777)
|
||||
stored = {}
|
||||
with open(env_path, "r", encoding="utf-8") as handle:
|
||||
for raw in handle:
|
||||
line = raw.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, _, value = line.partition("=")
|
||||
stored[key.strip()] = value.strip()
|
||||
self.assertEqual(config["BROKER_ADMIN_PASS"], stored["BROKER_ADMIN_PASS"])
|
||||
self.assertEqual(
|
||||
config["BROKER_INSTANCE_PEPPER"], stored["BROKER_INSTANCE_PEPPER"]
|
||||
)
|
||||
self.assertEqual(1, buffer.getvalue().count(config["BROKER_ADMIN_PASS"]))
|
||||
second = io.StringIO()
|
||||
with contextlib.redirect_stdout(second):
|
||||
coreci_broker.load_config(dict(self.env))
|
||||
self.assertEqual("", second.getvalue())
|
||||
|
||||
def test_append_env_entries_write_error_propagates_with_single_close(self):
|
||||
env_path = os.path.join(self.tmp, ".env")
|
||||
with open(env_path, "w", encoding="utf-8") as handle:
|
||||
handle.write("seed=1\n")
|
||||
closes = []
|
||||
|
||||
class FailingHandle:
|
||||
def __init__(self, fd):
|
||||
self.fd = fd
|
||||
|
||||
def write(self, data):
|
||||
raise OSError("simulated write failure")
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_value, traceback):
|
||||
os.close(self.fd)
|
||||
closes.append(self.fd)
|
||||
return False
|
||||
|
||||
def fake_fdopen(fd, *args, **kwargs):
|
||||
return FailingHandle(fd)
|
||||
|
||||
with mock.patch.object(os, "fdopen", fake_fdopen):
|
||||
with self.assertRaises(OSError) as raised:
|
||||
coreci_broker._append_env_entries(env_path, [("NEW", "value")])
|
||||
self.assertEqual(1, len(closes))
|
||||
self.assertIn("simulated write failure", str(raised.exception))
|
||||
with open(env_path, "r", encoding="utf-8") as handle:
|
||||
self.assertEqual("seed=1\n", handle.read())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,207 @@
|
||||
import base64
|
||||
import hashlib
|
||||
import os
|
||||
import sys
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
SCRYPT_N = 2 ** 14
|
||||
SCRYPT_R = 8
|
||||
SCRYPT_P = 1
|
||||
|
||||
BCRYPT_SHAPED = "$2b$12$K7L9O1P3QRSTUVWXYZabcdefghijk012345678901234567890123456789"
|
||||
|
||||
|
||||
def _scrypt_stored(token, salt):
|
||||
digest = hashlib.scrypt(
|
||||
token.encode("utf-8"), salt=salt, n=SCRYPT_N, r=SCRYPT_R, p=SCRYPT_P
|
||||
)
|
||||
return "{}${}${}".format(
|
||||
"scrypt",
|
||||
base64.urlsafe_b64encode(salt).decode("ascii"),
|
||||
base64.urlsafe_b64encode(digest).decode("ascii"),
|
||||
)
|
||||
|
||||
|
||||
class _FakeBcryptModule:
|
||||
def __init__(self):
|
||||
self.gensalt_calls = []
|
||||
self.hashpw_calls = []
|
||||
self.checkpw_calls = []
|
||||
|
||||
def gensalt(self):
|
||||
self.gensalt_calls.append(True)
|
||||
return b"$2b$12$fakesaltfakesalt"
|
||||
|
||||
def hashpw(self, password, salt):
|
||||
self.hashpw_calls.append((password, salt))
|
||||
return b"$2b$12$storedhashvalue"
|
||||
|
||||
def checkpw(self, password, hashed):
|
||||
self.checkpw_calls.append((password, hashed))
|
||||
return password == b"mock-token"
|
||||
|
||||
|
||||
class HashFormatTests(unittest.TestCase):
|
||||
def test_bcrypt_is_not_importable_on_this_host(self):
|
||||
self.assertIsNone(coreci_broker.bcrypt)
|
||||
with self.assertRaises(ImportError):
|
||||
import bcrypt
|
||||
|
||||
def test_hash_output_starts_with_scrypt_prefix(self):
|
||||
stored = coreci_broker.hash_token("format-token")
|
||||
self.assertTrue(stored.startswith("scrypt$"))
|
||||
|
||||
def test_hash_output_carries_salt_and_digest_b64url_fields(self):
|
||||
stored = coreci_broker.hash_token("format-token")
|
||||
fields = stored.split("$")
|
||||
self.assertEqual(3, len(fields))
|
||||
self.assertEqual("scrypt", fields[0])
|
||||
salt = base64.urlsafe_b64decode(fields[1].encode("ascii"))
|
||||
digest = base64.urlsafe_b64decode(fields[2].encode("ascii"))
|
||||
self.assertEqual(16, len(salt))
|
||||
self.assertEqual(64, len(digest))
|
||||
|
||||
|
||||
class RoundtripTests(unittest.TestCase):
|
||||
def test_hash_then_verify_roundtrip(self):
|
||||
stored = coreci_broker.hash_token("roundtrip-token")
|
||||
self.assertTrue(coreci_broker.verify_token("roundtrip-token", stored))
|
||||
|
||||
def test_wrong_token_fails(self):
|
||||
stored = coreci_broker.hash_token("roundtrip-token")
|
||||
self.assertFalse(coreci_broker.verify_token("wrong-token", stored))
|
||||
|
||||
def test_two_hashes_of_same_token_differ(self):
|
||||
first = coreci_broker.hash_token("same-token")
|
||||
second = coreci_broker.hash_token("same-token")
|
||||
self.assertNotEqual(first, second)
|
||||
self.assertTrue(coreci_broker.verify_token("same-token", first))
|
||||
self.assertTrue(coreci_broker.verify_token("same-token", second))
|
||||
|
||||
|
||||
class ScryptParseTests(unittest.TestCase):
|
||||
def test_hand_built_stored_value_verifies(self):
|
||||
stored = _scrypt_stored("parse-token", bytes(range(16)))
|
||||
self.assertTrue(stored.startswith("scrypt$"))
|
||||
self.assertTrue(coreci_broker.verify_token("parse-token", stored))
|
||||
|
||||
def test_hand_built_stored_value_rejects_other_token(self):
|
||||
stored = _scrypt_stored("parse-token", bytes(range(16)))
|
||||
self.assertFalse(coreci_broker.verify_token("parse-token-x", stored))
|
||||
|
||||
def test_tampered_digest_field_fails(self):
|
||||
salt = bytes(range(16))
|
||||
digest = hashlib.scrypt(
|
||||
b"parse-token", salt=salt, n=SCRYPT_N, r=SCRYPT_R, p=SCRYPT_P
|
||||
)
|
||||
tampered = digest[:-1] + bytes([digest[-1] ^ 0x01])
|
||||
stored = "{}${}${}".format(
|
||||
"scrypt",
|
||||
base64.urlsafe_b64encode(salt).decode("ascii"),
|
||||
base64.urlsafe_b64encode(tampered).decode("ascii"),
|
||||
)
|
||||
self.assertFalse(coreci_broker.verify_token("parse-token", stored))
|
||||
|
||||
def test_malformed_scrypt_values_fail_cleanly(self):
|
||||
for stored in (
|
||||
"scrypt$",
|
||||
"scrypt$abc",
|
||||
"scrypt$abc$def$ghi",
|
||||
"scrypt$$",
|
||||
"scrypt$!@#$%",
|
||||
):
|
||||
with self.subTest(stored=stored):
|
||||
self.assertFalse(coreci_broker.verify_token("x", stored))
|
||||
|
||||
|
||||
class UnknownPrefixTests(unittest.TestCase):
|
||||
def test_unknown_prefixes_return_false_without_raising(self):
|
||||
for stored in (
|
||||
"",
|
||||
"plaintext",
|
||||
"$2c$12$abc",
|
||||
"$2x$12$abc",
|
||||
"argon2$abc$def",
|
||||
"SCRYPT$abc$def",
|
||||
"scrypt2$abc$def",
|
||||
"0123456789",
|
||||
):
|
||||
with self.subTest(stored=stored):
|
||||
self.assertFalse(coreci_broker.verify_token("x", stored))
|
||||
|
||||
def test_non_string_stored_returns_false_cleanly(self):
|
||||
self.assertFalse(coreci_broker.verify_token("x", None))
|
||||
self.assertFalse(coreci_broker.verify_token("x", 5))
|
||||
self.assertFalse(coreci_broker.verify_token("x", ["scrypt$abc$def"]))
|
||||
|
||||
|
||||
class BcryptDispatchContractTests(unittest.TestCase):
|
||||
def test_bcrypt_prefix_returns_false_cleanly_with_bcrypt_absent(self):
|
||||
self.assertIsNone(coreci_broker.bcrypt)
|
||||
with self.assertRaises(ImportError):
|
||||
import bcrypt
|
||||
self.assertFalse(coreci_broker.verify_token("x", BCRYPT_SHAPED))
|
||||
|
||||
def test_all_bcrypt_prefixes_return_false_cleanly_with_bcrypt_absent(self):
|
||||
self.assertIsNone(coreci_broker.bcrypt)
|
||||
for prefix in ("$2a$", "$2b$", "$2y$"):
|
||||
with self.subTest(prefix=prefix):
|
||||
self.assertFalse(coreci_broker.verify_token("x", prefix + "12$storedhashvalue"))
|
||||
|
||||
|
||||
class BcryptBranchTests(unittest.TestCase):
|
||||
def test_hash_token_uses_bcrypt_branch_when_importable(self):
|
||||
fake = _FakeBcryptModule()
|
||||
with mock.patch.object(coreci_broker, "bcrypt", fake):
|
||||
stored = coreci_broker.hash_token("mock-token")
|
||||
self.assertEqual("$2b$12$storedhashvalue", stored)
|
||||
self.assertEqual([True], fake.gensalt_calls)
|
||||
self.assertEqual([(b"mock-token", b"$2b$12$fakesaltfakesalt")], fake.hashpw_calls)
|
||||
|
||||
def test_verify_dispatches_bcrypt_prefixes_to_bcrypt_branch(self):
|
||||
fake = _FakeBcryptModule()
|
||||
expected = [
|
||||
(b"mock-token", (prefix + "12$storedhashvalue").encode("ascii"))
|
||||
for prefix in ("$2a$", "$2b$", "$2y$")
|
||||
]
|
||||
with mock.patch.object(coreci_broker, "bcrypt", fake):
|
||||
for password, hashed in expected:
|
||||
with self.subTest(hashed=hashed):
|
||||
self.assertTrue(coreci_broker.verify_token(password.decode("utf-8"), hashed.decode("ascii")))
|
||||
self.assertEqual(expected, fake.checkpw_calls)
|
||||
|
||||
def test_bcrypt_checkpw_rejection_returns_false(self):
|
||||
fake = _FakeBcryptModule()
|
||||
with mock.patch.object(coreci_broker, "bcrypt", fake):
|
||||
self.assertFalse(coreci_broker.verify_token("other-token", "$2b$12$storedhashvalue"))
|
||||
self.assertEqual([(b"other-token", b"$2b$12$storedhashvalue")], fake.checkpw_calls)
|
||||
|
||||
def test_bcrypt_checkpw_error_returns_false_cleanly(self):
|
||||
fake = _FakeBcryptModule()
|
||||
|
||||
def broken(password, hashed):
|
||||
raise ValueError("invalid salt")
|
||||
|
||||
fake.checkpw = broken
|
||||
with mock.patch.object(coreci_broker, "bcrypt", fake):
|
||||
self.assertFalse(coreci_broker.verify_token("mock-token", "$2b$garbage"))
|
||||
|
||||
def test_scrypt_and_unknown_stored_never_dispatch_to_bcrypt(self):
|
||||
fake = _FakeBcryptModule()
|
||||
stored = coreci_broker.hash_token("mock-token")
|
||||
self.assertTrue(stored.startswith("scrypt$"))
|
||||
with mock.patch.object(coreci_broker, "bcrypt", fake):
|
||||
self.assertTrue(coreci_broker.verify_token("mock-token", stored))
|
||||
self.assertFalse(coreci_broker.verify_token("other", stored))
|
||||
self.assertFalse(coreci_broker.verify_token("mock-token", "unknown$prefix"))
|
||||
self.assertEqual([], fake.checkpw_calls)
|
||||
self.assertEqual([], fake.hashpw_calls)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,148 @@
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.request
|
||||
from http.server import ThreadingHTTPServer
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
|
||||
class MountsTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
coreci_broker._reset_server_guard()
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_BIND": "127.0.0.1:0",
|
||||
}
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.addCleanup(coreci_broker._reset_server_guard)
|
||||
|
||||
def _patched_environ(self):
|
||||
return mock.patch.dict(os.environ, self.env, clear=True)
|
||||
|
||||
def _read_env_values(self):
|
||||
values = {}
|
||||
with open(self.env["BROKER_ENV_PATH"], "r", encoding="utf-8") as handle:
|
||||
for raw in handle:
|
||||
line = raw.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, _, value = line.partition("=")
|
||||
values[key.strip()] = value.strip()
|
||||
return values
|
||||
|
||||
def _healthz(self, port):
|
||||
url = "http://127.0.0.1:%d/healthz" % port
|
||||
with urllib.request.urlopen(url, timeout=5) as response:
|
||||
return response.status, json.loads(response.read().decode("utf-8"))
|
||||
|
||||
def test_owui_docstring_frontmatter(self):
|
||||
doc = coreci_broker.__doc__ or ""
|
||||
self.assertTrue(doc.startswith("title: CoreCI Broker"))
|
||||
for line in (
|
||||
"author: coreci",
|
||||
"description: ",
|
||||
"version: 0.1.0",
|
||||
"requirements: []",
|
||||
"licence: MIT",
|
||||
):
|
||||
self.assertIn(line, doc)
|
||||
|
||||
def test_source_never_references_dunder_file(self):
|
||||
with open(coreci_broker.__file__, "r", encoding="utf-8") as handle:
|
||||
source = handle.read()
|
||||
self.assertNotIn("__file__", source)
|
||||
|
||||
def test_tools_double_instantiation_starts_exactly_one_server(self):
|
||||
serve_calls = []
|
||||
real_serve = coreci_broker.serve
|
||||
|
||||
def counting_serve(host, port, broker):
|
||||
serve_calls.append((host, port))
|
||||
return real_serve(host, port, broker)
|
||||
|
||||
with self._patched_environ():
|
||||
with mock.patch.object(coreci_broker, "serve", counting_serve):
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
first = coreci_broker.Tools()
|
||||
port_after_first = coreci_broker._started_server.server_address[1]
|
||||
second = coreci_broker.Tools()
|
||||
guard = coreci_broker._started_server
|
||||
self.assertEqual(1, len(serve_calls))
|
||||
self.assertIsInstance(guard, ThreadingHTTPServer)
|
||||
self.assertIs(coreci_broker._started_server, first._server)
|
||||
self.assertIs(first._server, second._server)
|
||||
self.assertEqual(port_after_first, guard.server_address[1])
|
||||
self.assertNotEqual(0, guard.server_address[1])
|
||||
self.assertIsInstance(first.valves, coreci_broker.Tools.Valves)
|
||||
self.assertIsInstance(second.valves, coreci_broker.Tools.Valves)
|
||||
status, payload = self._healthz(port_after_first)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, payload)
|
||||
|
||||
def test_tools_status_returns_non_empty_string(self):
|
||||
with self._patched_environ():
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
tools = coreci_broker.Tools()
|
||||
status = tools.status()
|
||||
port = tools._server.server_address[1]
|
||||
self.assertIsInstance(status, str)
|
||||
self.assertTrue(status.strip())
|
||||
self.assertIn(str(port), status)
|
||||
|
||||
def test_first_boot_print_discipline_through_tools_boot(self):
|
||||
with self._patched_environ():
|
||||
first = io.StringIO()
|
||||
with contextlib.redirect_stdout(first):
|
||||
coreci_broker.Tools()
|
||||
second = io.StringIO()
|
||||
with contextlib.redirect_stdout(second):
|
||||
coreci_broker.Tools()
|
||||
stored = self._read_env_values()
|
||||
admin_pass = stored["BROKER_ADMIN_PASS"]
|
||||
pepper = stored["BROKER_INSTANCE_PEPPER"]
|
||||
self.assertEqual(1, first.getvalue().count(admin_pass))
|
||||
self.assertNotIn(pepper, first.getvalue())
|
||||
self.assertEqual("", second.getvalue())
|
||||
|
||||
def test_run_standalone_boots_serves_and_shuts_down_cleanly(self):
|
||||
captured = {}
|
||||
real_serve = coreci_broker.serve
|
||||
|
||||
def capturing_serve(host, port, broker):
|
||||
server = real_serve(host, port, broker)
|
||||
captured["server"] = server
|
||||
return server
|
||||
|
||||
def fake_sleep(seconds):
|
||||
server = captured["server"]
|
||||
status, payload = self._healthz(server.server_address[1])
|
||||
captured["healthz"] = (status, payload)
|
||||
raise KeyboardInterrupt
|
||||
|
||||
stdout = io.StringIO()
|
||||
with mock.patch.object(coreci_broker, "serve", capturing_serve):
|
||||
with mock.patch.object(coreci_broker.time, "sleep", fake_sleep):
|
||||
with contextlib.redirect_stdout(stdout):
|
||||
coreci_broker.run_standalone(dict(self.env))
|
||||
out = stdout.getvalue()
|
||||
self.assertEqual((200, {"ok": True}), captured["healthz"])
|
||||
self.assertEqual(1, out.count("BROKER_ADMIN_PASS="))
|
||||
self.assertIn("listening on http://127.0.0.1:", out)
|
||||
self.assertIn("shut down", out)
|
||||
self.assertEqual(-1, captured["server"].socket.fileno())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,249 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import datetime
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
PROTECTED_PATH = "/test/p3-remediation"
|
||||
AUTH_401_BODY = {"error": 401, "message": "invalid app credentials"}
|
||||
CAP_BODY = {"error": 400, "message": "too many agents for this app"}
|
||||
AGENT_ID_BODY = {"error": 400, "message": "invalid agent_id"}
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
def _protected_echo(request):
|
||||
request.send_json(200, dict(request.app_auth))
|
||||
|
||||
|
||||
class P3RemediationTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(dict(self.env))
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
self.app_uuid = str(uuidlib.uuid4())
|
||||
self.user_id = "ceo"
|
||||
self._seed(self.app_uuid, self.user_id)
|
||||
self.broker.router.add(
|
||||
"GET", PROTECTED_PATH, coreci_broker._require_app(_protected_echo)
|
||||
)
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None, base=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(
|
||||
(base or self.base) + path, data=data, method=method
|
||||
)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
return response.status, json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, json.loads(error.read().decode("utf-8"))
|
||||
|
||||
def _seed(self, app_uuid, user_id, base=None):
|
||||
status, _ = self._request("POST", APPS_PATH, {"uuid": app_uuid}, self.admin, base)
|
||||
self.assertEqual(201, status)
|
||||
status, _ = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin, base)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def _register(self, agent_id="agent-1", tags=("coreci",), app_uuid=None,
|
||||
user_id=None, base=None):
|
||||
payload = {
|
||||
"agent_id": agent_id,
|
||||
"app_registration_uuid": app_uuid or self.app_uuid,
|
||||
"user_id": user_id or self.user_id,
|
||||
"tags": list(tags),
|
||||
}
|
||||
return self._request("POST", REGISTER_PATH, payload, base=base)
|
||||
|
||||
def _auth(self, token, app_uuid=None):
|
||||
header = "App %s %s" % (app_uuid or self.app_uuid, token)
|
||||
return self._request("GET", PROTECTED_PATH, headers={"Authorization": header})
|
||||
|
||||
def _audit(self, action=None, result=None):
|
||||
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
|
||||
entries = [json.loads(raw) for raw in handle if raw.strip()]
|
||||
return [
|
||||
entry for entry in entries
|
||||
if (action is None or entry["action"] == action)
|
||||
and (result is None or entry["result"] == result)
|
||||
]
|
||||
|
||||
def _future(self, seconds=3600):
|
||||
return (datetime.datetime.now(datetime.timezone.utc)
|
||||
+ datetime.timedelta(seconds=seconds)).isoformat()
|
||||
|
||||
def test_agent_id_colon_slash_backslash_rejected(self):
|
||||
for bad in ("a:b", "a/b", "a\\b"):
|
||||
status, body = self._register(agent_id=bad)
|
||||
self.assertEqual(400, status, repr(bad))
|
||||
self.assertEqual(AGENT_ID_BODY, body)
|
||||
status, body = self._register(agent_id="a:b", app_uuid=str(uuidlib.uuid4()))
|
||||
self.assertEqual(400, status)
|
||||
self.assertEqual(AGENT_ID_BODY, body)
|
||||
self.assertEqual([], self._audit("register"))
|
||||
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
|
||||
|
||||
def test_tag_containing_colon_rejected(self):
|
||||
status, body = self._register(tags=("ok", "ba:d"))
|
||||
self.assertEqual(400, status)
|
||||
self.assertEqual({"error": 400, "message": "invalid tags"}, body)
|
||||
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
|
||||
status, body = self._register(tags=("ok",))
|
||||
self.assertEqual(201, status)
|
||||
self.assertTrue(body["created"])
|
||||
|
||||
def test_user_id_colon_rejected_at_allowlist_add(self):
|
||||
status, body = self._request("POST", USERS_PATH, {"user_id": "co:o"}, self.admin)
|
||||
self.assertEqual(400, status)
|
||||
self.assertEqual({"error": 400, "message": "invalid user_id"}, body)
|
||||
status, body = self._register(user_id="co:o")
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual({"error": 403, "message": "registration not allowed"}, body)
|
||||
self.assertEqual(1, len(self._audit("register", "denied")))
|
||||
status, body = self._request("POST", USERS_PATH, {"user_id": "coo"}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
status, body = self._register(agent_id="coo-agent", user_id="coo")
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def test_default_fifty_agent_cap_blocks_register(self):
|
||||
self.assertEqual("50", self.broker.config["BROKER_MAX_AGENTS_PER_APP"])
|
||||
now = datetime.datetime.now(datetime.timezone.utc).isoformat()
|
||||
seed_hash = coreci_broker.hash_token("seed")
|
||||
for index in range(50):
|
||||
self.broker.store.save_agent(
|
||||
"seeded-%d" % index, self.app_uuid, self.user_id, "[]",
|
||||
seed_hash, self._future(), now,
|
||||
)
|
||||
status, body = self._register(agent_id="overflow")
|
||||
self.assertEqual(400, status)
|
||||
self.assertEqual(CAP_BODY, body)
|
||||
status, body = self._register(agent_id="seeded-0")
|
||||
self.assertEqual(400, status)
|
||||
self.assertEqual(CAP_BODY, body)
|
||||
self.assertIsNone(self.broker.store.get_agent("overflow", self.app_uuid))
|
||||
self.assertEqual(50, len(self.broker.store.agents_by_uuid(self.app_uuid)))
|
||||
self.assertEqual([], self._audit("register"))
|
||||
|
||||
def test_max_agents_per_app_env_override_honored(self):
|
||||
env = dict(self.env)
|
||||
env["BROKER_DB_PATH"] = os.path.join(self.tmp, "cap.db")
|
||||
env["BROKER_AUDIT_LOG_PATH"] = os.path.join(self.tmp, "cap.audit.log")
|
||||
env["BROKER_MAX_AGENTS_PER_APP"] = "2"
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
broker = coreci_broker.boot(dict(env))
|
||||
self.addCleanup(broker.audit.close)
|
||||
self.addCleanup(broker.store.close)
|
||||
self.assertEqual("2", broker.config["BROKER_MAX_AGENTS_PER_APP"])
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
self.addCleanup(server.shutdown)
|
||||
self.addCleanup(server.server_close)
|
||||
base = "http://127.0.0.1:%d" % server.server_address[1]
|
||||
self._seed(self.app_uuid, self.user_id, base=base)
|
||||
for agent_id in ("agent-a", "agent-b"):
|
||||
status, body = self._register(agent_id=agent_id, base=base)
|
||||
self.assertEqual(201, status, agent_id)
|
||||
self.assertTrue(body["created"])
|
||||
status, body = self._register(agent_id="agent-c", base=base)
|
||||
self.assertEqual(400, status)
|
||||
self.assertEqual(CAP_BODY, body)
|
||||
self.assertIsNone(broker.store.get_agent("agent-c", self.app_uuid))
|
||||
self.assertEqual(2, len(broker.store.agents_by_uuid(self.app_uuid)))
|
||||
|
||||
def test_overwrite_evicts_uuid_cache_entries(self):
|
||||
_, first = self._register(tags=("red",))
|
||||
old_token = first["token"]
|
||||
status, _ = self._auth(old_token)
|
||||
self.assertEqual(200, status)
|
||||
self.assertIn((self.app_uuid, old_token), self.broker.token_cache)
|
||||
status, second = self._register(tags=("blue",))
|
||||
self.assertEqual(200, status)
|
||||
self.assertFalse(second["created"])
|
||||
new_token = second["token"]
|
||||
self.assertNotEqual(old_token, new_token)
|
||||
self.assertEqual([], [k for k in self.broker.token_cache if k[0] == self.app_uuid])
|
||||
status, body = self._auth(old_token)
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
self.assertNotIn((self.app_uuid, old_token), self.broker.token_cache)
|
||||
status, body = self._auth(new_token)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("agent-1", body["agent_id"])
|
||||
self.assertIn((self.app_uuid, new_token), self.broker.token_cache)
|
||||
row = self.broker.store.get_agent("agent-1", self.app_uuid)
|
||||
self.assertFalse(coreci_broker.verify_token(old_token, row[4]))
|
||||
self.assertTrue(coreci_broker.verify_token(new_token, row[4]))
|
||||
|
||||
def test_integrity_error_race_takes_idempotent_path(self):
|
||||
agent_id = "race-agent"
|
||||
tags = ["coreci"]
|
||||
pepper = self.broker.store.get_config("BROKER_INSTANCE_PEPPER")
|
||||
token = coreci_broker.derive_token(pepper, self.app_uuid, self.user_id, agent_id, tags)
|
||||
expires = self._future()
|
||||
now = datetime.datetime.now(datetime.timezone.utc).isoformat()
|
||||
self.broker.store.save_agent(
|
||||
agent_id, self.app_uuid, self.user_id, json.dumps(sorted(tags)),
|
||||
coreci_broker.hash_token(token), expires, now,
|
||||
)
|
||||
real_get_agent = self.broker.store.get_agent
|
||||
calls = []
|
||||
|
||||
def racing_get_agent(agent_id, app_uuid):
|
||||
calls.append(agent_id)
|
||||
if len(calls) == 1:
|
||||
return None
|
||||
return real_get_agent(agent_id, app_uuid)
|
||||
|
||||
self.broker.store.get_agent = racing_get_agent
|
||||
self.addCleanup(setattr, self.broker.store, "get_agent", real_get_agent)
|
||||
status, body = self._register(agent_id=agent_id, tags=tags)
|
||||
self.assertEqual(200, status)
|
||||
self.assertFalse(body["created"])
|
||||
self.assertEqual(token, body["token"])
|
||||
self.assertEqual(expires, body["token_expires_at"])
|
||||
self.assertIn("pepper", body)
|
||||
self.assertEqual(2, len(calls))
|
||||
row = real_get_agent(agent_id, self.app_uuid)
|
||||
self.assertTrue(coreci_broker.verify_token(token, row[4]))
|
||||
self.assertGreaterEqual(row[7], now)
|
||||
self.assertEqual(1, len(self._audit("register", "ok")))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,207 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import errno
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
AGENT_ID_BODY = {"error": 400, "message": "invalid agent_id"}
|
||||
TAGS_BODY = {"error": 400, "message": "invalid tags"}
|
||||
USER_ID_BODY = {"error": 400, "message": "invalid user_id"}
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
class P5RegisterValidationTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(dict(self.env))
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
self.app_uuid = str(uuidlib.uuid4())
|
||||
self.user_id = "ceo"
|
||||
self._seed()
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
return response.status, json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, json.loads(error.read().decode("utf-8"))
|
||||
|
||||
def _seed(self):
|
||||
status, _ = self._request("POST", APPS_PATH, {"uuid": self.app_uuid}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
status, _ = self._request("POST", USERS_PATH, {"user_id": self.user_id}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def _register(self, agent_id="agent-1", tags=("coreci",)):
|
||||
return self._request("POST", REGISTER_PATH, {
|
||||
"agent_id": agent_id,
|
||||
"app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id,
|
||||
"tags": list(tags),
|
||||
})
|
||||
|
||||
def _audit(self, action=None, result=None):
|
||||
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
|
||||
entries = [json.loads(raw) for raw in handle if raw.strip()]
|
||||
return [
|
||||
entry for entry in entries
|
||||
if (action is None or entry["action"] == action)
|
||||
and (result is None or entry["result"] == result)
|
||||
]
|
||||
|
||||
def test_agent_id_bare_dot_and_dotdot_rejected(self):
|
||||
for agent_id in (".", ".."):
|
||||
status, body = self._register(agent_id=agent_id)
|
||||
self.assertEqual(400, status, repr(agent_id))
|
||||
self.assertEqual(AGENT_ID_BODY, body)
|
||||
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
|
||||
self.assertEqual([], self._audit("register"))
|
||||
|
||||
def test_agent_id_control_chars_rejected(self):
|
||||
for agent_id in ("a\x00b", "a\nb", "a\x1fb", "a\x7fb"):
|
||||
status, body = self._register(agent_id=agent_id)
|
||||
self.assertEqual(400, status, repr(agent_id))
|
||||
self.assertEqual(AGENT_ID_BODY, body)
|
||||
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
|
||||
self.assertEqual([], self._audit("register"))
|
||||
|
||||
def test_agent_id_embedded_dots_still_accepted(self):
|
||||
status, body = self._register(agent_id="a.b")
|
||||
self.assertEqual(201, status)
|
||||
self.assertTrue(body["created"])
|
||||
status, body = self._register(agent_id="tool.v2")
|
||||
self.assertEqual(201, status)
|
||||
self.assertTrue(body["created"])
|
||||
self.assertIsNotNone(self.broker.store.get_agent("a.b", self.app_uuid))
|
||||
self.assertIsNotNone(self.broker.store.get_agent("tool.v2", self.app_uuid))
|
||||
self.assertEqual(2, len(self._audit("register", "ok")))
|
||||
|
||||
def test_tag_control_chars_rejected(self):
|
||||
for tags in (("ok", "ba\x00d"), ("ok", "ba\x7fd"), ("ok", "ba:d")):
|
||||
status, body = self._register(tags=tags)
|
||||
self.assertEqual(400, status, repr(tags))
|
||||
self.assertEqual(TAGS_BODY, body)
|
||||
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
|
||||
status, body = self._register(tags=("a.b", ".."))
|
||||
self.assertEqual(201, status)
|
||||
self.assertTrue(body["created"])
|
||||
row = self.broker.store.get_agent("agent-1", self.app_uuid)
|
||||
self.assertEqual(["..", "a.b"], json.loads(row[3]))
|
||||
|
||||
def test_agent_id_path_separators_still_rejected(self):
|
||||
for agent_id in ("a:b", "a/b", "a\\b"):
|
||||
status, body = self._register(agent_id=agent_id)
|
||||
self.assertEqual(400, status, repr(agent_id))
|
||||
self.assertEqual(AGENT_ID_BODY, body)
|
||||
self.assertEqual([], self._audit("register"))
|
||||
status, body = self._register(agent_id="plain-agent")
|
||||
self.assertEqual(201, status)
|
||||
self.assertTrue(body["created"])
|
||||
|
||||
def test_user_allowlist_add_shares_id_rules(self):
|
||||
for user_id in (".", "..", "a\x00b", "a:b", "a/b"):
|
||||
status, body = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
|
||||
self.assertEqual(400, status, repr(user_id))
|
||||
self.assertEqual(USER_ID_BODY, body)
|
||||
status, body = self._request("POST", USERS_PATH, {"user_id": "ceo.two"}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
|
||||
class P5ReexecGuardTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
coreci_broker._reset_server_guard()
|
||||
self.addCleanup(coreci_broker._reset_server_guard)
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
|
||||
def _env(self, bind):
|
||||
return {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_BIND": bind,
|
||||
}
|
||||
|
||||
def _exec_broker(self):
|
||||
with open(coreci_broker.__file__, "r", encoding="utf-8") as handle:
|
||||
source = handle.read()
|
||||
namespace = {"__name__": "coreci_broker_reexec"}
|
||||
exec(compile(source, "coreci_broker.py", "exec"), namespace)
|
||||
return namespace
|
||||
|
||||
def _healthz(self, port):
|
||||
url = "http://127.0.0.1:%d/healthz" % port
|
||||
with urllib.request.urlopen(url, timeout=5) as response:
|
||||
return response.status, json.loads(response.read().decode("utf-8"))
|
||||
|
||||
def test_fresh_namespace_tools_fails_loud_when_old_module_serves(self):
|
||||
first = self._exec_broker()
|
||||
with mock.patch.dict(os.environ, self._env("127.0.0.1:0"), clear=True):
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
first_tools = first["Tools"]()
|
||||
first_again = first["Tools"]()
|
||||
server = first["_started_server"]
|
||||
self.addCleanup(first["_reset_server_guard"])
|
||||
self.assertIs(server, first_tools._server)
|
||||
self.assertIs(first_tools._server, first_again._server)
|
||||
port = server.server_address[1]
|
||||
self.assertNotEqual(0, port)
|
||||
second = self._exec_broker()
|
||||
self.assertIsNone(second["_started_server"])
|
||||
with mock.patch.dict(os.environ, self._env("127.0.0.1:%d" % port), clear=True):
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
with self.assertRaises(RuntimeError) as caught:
|
||||
second["Tools"]()
|
||||
message = str(caught.exception).lower()
|
||||
self.assertIn("restart", message)
|
||||
self.assertIn("open webui", message)
|
||||
self.assertIsInstance(caught.exception.__cause__, OSError)
|
||||
self.assertEqual(errno.EADDRINUSE, caught.exception.__cause__.errno)
|
||||
self.assertIsNone(second["_started_server"])
|
||||
self.assertEqual((200, {"ok": True}), self._healthz(port))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,271 @@
|
||||
import ast
|
||||
import base64
|
||||
import contextlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
|
||||
REFERENCE_DIR = os.path.abspath(
|
||||
os.path.join(os.path.dirname(__file__), os.pardir, "reference_tools")
|
||||
)
|
||||
REFERENCE_TITLES = {
|
||||
"proxmox_readonly_reference": "Proxmox Read-Only Reference [CoreCI]",
|
||||
"minimal_auth_showcase": "Minimal Auth Showcase [CoreCI]",
|
||||
}
|
||||
ALLOWED_REFERENCE_IMPORTS = {"coreci_broker_client", "datetime"}
|
||||
FORBIDDEN_SOURCE_MARKERS = ("urllib", "httpx", "socket", "proxmoxer")
|
||||
BROKER_ONLY_AUDIT_ACTIONS = {"add_app", "add_user", "register", "read"}
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
def _load_reference(name):
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
name, os.path.join(REFERENCE_DIR, name + ".py")
|
||||
)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def _reference_source(name):
|
||||
with open(os.path.join(REFERENCE_DIR, name + ".py"), "r", encoding="utf-8") as handle:
|
||||
return handle.read()
|
||||
|
||||
|
||||
PROXMOX_REFERENCE = _load_reference("proxmox_readonly_reference")
|
||||
MINIMAL_SHOWCASE = _load_reference("minimal_auth_showcase")
|
||||
|
||||
|
||||
class _ReferenceCase(unittest.TestCase):
|
||||
extra_env = {}
|
||||
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.creds_path = os.path.join(self.tmp, "creds.json")
|
||||
with open(self.creds_path, "w", encoding="utf-8") as handle:
|
||||
handle.write(json.dumps({"coreci/proxmox": PROXMOX_SECRET}))
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
"BROKER_CREDS_SOURCE": "file",
|
||||
"BROKER_CREDS_FILE": self.creds_path,
|
||||
}
|
||||
self.env.update(self.extra_env)
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(self.env)
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
self.app_uuid = str(uuidlib.uuid4())
|
||||
self.user_id = "ceo"
|
||||
self._seed()
|
||||
|
||||
def _seed(self):
|
||||
status, _ = self._request("POST", APPS_PATH, {"uuid": self.app_uuid}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
status, _ = self._request("POST", USERS_PATH, {"user_id": self.user_id}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
raw = response.read().decode("utf-8")
|
||||
return response.status, json.loads(raw) if raw else None
|
||||
except urllib.error.HTTPError as error:
|
||||
raw = error.read().decode("utf-8")
|
||||
return error.code, json.loads(raw) if raw else None
|
||||
|
||||
def _audit(self, action=None, result=None):
|
||||
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
|
||||
entries = [json.loads(raw) for raw in handle if raw.strip()]
|
||||
return [
|
||||
entry for entry in entries
|
||||
if (action is None or entry["action"] == action)
|
||||
and (result is None or entry["result"] == result)
|
||||
]
|
||||
|
||||
def _tools(self, module, agent_id="proxmox"):
|
||||
tools = module.Tools()
|
||||
tools.valves.base_url = self.base
|
||||
tools.valves.agent_id = agent_id
|
||||
tools.valves.app_uuid = self.app_uuid
|
||||
tools.valves.user_id = self.user_id
|
||||
return tools
|
||||
|
||||
def _persisted_state(self, agent_id="proxmox"):
|
||||
return self.broker.store.get_agent_state(agent_id, self.app_uuid)
|
||||
|
||||
|
||||
class ProxmoxReferenceFlowTests(_ReferenceCase):
|
||||
def test_first_call_registers_persists_state_reads_secret_dry_runs(self):
|
||||
tools = self._tools(PROXMOX_REFERENCE)
|
||||
response = tools.list_nodes()
|
||||
self.assertIs(True, response["dry_run"])
|
||||
self.assertEqual("list_nodes", response["action"])
|
||||
self.assertIs(True, response["credential_present"])
|
||||
self.assertTrue(str(response["note"]).strip())
|
||||
self.assertNotIn("data", response)
|
||||
self.assertNotIn("pve-tok", json.dumps(response))
|
||||
self.assertEqual(1, len(self._audit(action="register")))
|
||||
document = self._persisted_state()
|
||||
self.assertIsNotNone(document)
|
||||
self.assertIn("current_token", document)
|
||||
self.assertEqual(tools._state["current_token"], document["current_token"])
|
||||
self.assertEqual([], self._audit(result="denied"))
|
||||
self.assertTrue(
|
||||
{entry["action"] for entry in self._audit()} <= BROKER_ONLY_AUDIT_ACTIONS
|
||||
)
|
||||
|
||||
def test_second_call_reuses_persisted_state_without_new_register(self):
|
||||
tools = self._tools(PROXMOX_REFERENCE)
|
||||
tools.list_nodes()
|
||||
token = tools._state["current_token"]
|
||||
second = tools.list_nodes()
|
||||
self.assertIs(True, second["dry_run"])
|
||||
self.assertEqual("list_nodes", second["action"])
|
||||
self.assertIs(True, second["credential_present"])
|
||||
self.assertEqual(1, len(self._audit(action="register")))
|
||||
self.assertEqual(token, tools._state["current_token"])
|
||||
document = self._persisted_state()
|
||||
self.assertIn("current_token", document)
|
||||
self.assertEqual(token, document["current_token"])
|
||||
|
||||
def test_vm_status_is_dry_run_with_credential_present(self):
|
||||
tools = self._tools(PROXMOX_REFERENCE)
|
||||
response = tools.vm_status("101")
|
||||
self.assertIs(True, response["dry_run"])
|
||||
self.assertEqual("vm_status", response["action"])
|
||||
self.assertIs(True, response["credential_present"])
|
||||
self.assertNotIn("pve-tok", json.dumps(response))
|
||||
self.assertEqual(1, len(self._audit(action="register")))
|
||||
|
||||
|
||||
class ProxmoxReferenceTtlTests(_ReferenceCase):
|
||||
extra_env = {"BROKER_TOKEN_TTL": "1"}
|
||||
|
||||
def test_expired_subsequent_call_reregisters_and_persists_new_expiry(self):
|
||||
tools = self._tools(PROXMOX_REFERENCE)
|
||||
self.assertTrue(tools.list_nodes()["credential_present"])
|
||||
first_expires = tools._state["token_expires_at"]
|
||||
time.sleep(1.2)
|
||||
response = tools.list_nodes()
|
||||
self.assertIs(True, response["dry_run"])
|
||||
self.assertIs(True, response["credential_present"])
|
||||
self.assertEqual(2, len(self._audit(action="register")))
|
||||
document = self._persisted_state()
|
||||
self.assertEqual(tools._state["current_token"], document["current_token"])
|
||||
self.assertEqual(tools._state["token_expires_at"], document["token_expires_at"])
|
||||
self.assertNotEqual(first_expires, document["token_expires_at"])
|
||||
|
||||
|
||||
class MinimalShowcaseTests(_ReferenceCase):
|
||||
def test_demo_authenticates_then_reads_secret(self):
|
||||
tools = self._tools(MINIMAL_SHOWCASE)
|
||||
response = tools.demo()
|
||||
self.assertEqual(
|
||||
{"registered": True, "secret_read": True,
|
||||
"path": "coreci/proxmox", "note": "auth first, then use"},
|
||||
response,
|
||||
)
|
||||
self.assertNotIn("pve-tok", json.dumps(response))
|
||||
self.assertEqual(1, len(self._audit(action="register")))
|
||||
self.assertEqual([], self._audit(result="denied"))
|
||||
|
||||
|
||||
class ReferenceHygieneTests(unittest.TestCase):
|
||||
def test_sources_carry_no_secret_literals(self):
|
||||
for name in REFERENCE_TITLES:
|
||||
source = _reference_source(name)
|
||||
self.assertNotRegex(source, r"(?im)^\s*(password|pepper)\s*=\s*[\"']", name)
|
||||
for value in PROXMOX_SECRET.values():
|
||||
self.assertNotIn(value, source, name)
|
||||
|
||||
def test_sources_use_no_direct_network_or_infra_clients(self):
|
||||
for name in REFERENCE_TITLES:
|
||||
source = _reference_source(name)
|
||||
for marker in FORBIDDEN_SOURCE_MARKERS:
|
||||
self.assertNotIn(marker, source, "%s: %s" % (name, marker))
|
||||
|
||||
def test_imports_limited_to_client_module_and_stdlib(self):
|
||||
for name in REFERENCE_TITLES:
|
||||
tree = ast.parse(_reference_source(name))
|
||||
modules = set()
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Import):
|
||||
modules.update(alias.name for alias in node.names)
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
modules.add(node.module or "")
|
||||
for module in modules:
|
||||
self.assertIn(module, ALLOWED_REFERENCE_IMPORTS, "%s: %s" % (name, module))
|
||||
|
||||
def test_owui_frontmatter_docstrings(self):
|
||||
for name, title in REFERENCE_TITLES.items():
|
||||
doc = ast.get_docstring(ast.parse(_reference_source(name)))
|
||||
self.assertTrue(doc.startswith("title: " + title), name)
|
||||
for line in ("author: coreci", "description: ", "version: 0.1.0",
|
||||
"requirements: []", "licence: MIT"):
|
||||
self.assertIn(line, doc, "%s: %s" % (name, line))
|
||||
|
||||
def test_tools_surface_valves_state_and_docstring_discipline(self):
|
||||
for name in REFERENCE_TITLES:
|
||||
module = _load_reference(name)
|
||||
tools = module.Tools()
|
||||
self.assertIsInstance(tools.valves, module.Tools.Valves)
|
||||
for attr in ("base_url", "agent_id", "app_uuid", "user_id"):
|
||||
self.assertEqual("", getattr(module.Tools.Valves, attr), attr)
|
||||
self.assertEqual("", getattr(tools.valves, attr), attr)
|
||||
self.assertEqual({}, tools._state)
|
||||
public = []
|
||||
for node in ast.walk(ast.parse(_reference_source(name))):
|
||||
if not isinstance(node, ast.FunctionDef):
|
||||
continue
|
||||
doc = ast.get_docstring(node)
|
||||
if node.name.startswith("_"):
|
||||
self.assertIsNone(doc, "%s.%s" % (name, node.name))
|
||||
else:
|
||||
self.assertIsNotNone(doc, "%s.%s" % (name, node.name))
|
||||
public.append(node.name)
|
||||
self.assertTrue(public, name)
|
||||
|
||||
def test_no_comments_in_sources(self):
|
||||
for name in REFERENCE_TITLES:
|
||||
for number, line in enumerate(_reference_source(name).splitlines(), 1):
|
||||
self.assertFalse(line.lstrip().startswith("#"), "%s:%d" % (name, number))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,347 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import datetime
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
WHOAMI_PATH = "/api/broker/auth/whoami"
|
||||
DEFAULT_TTL = 86400
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
def _seed_direct(broker, app_uuid, user_id):
|
||||
broker.store.add_row("app_registrations", app_uuid, None)
|
||||
broker.store.add_row("user_allowlist", user_id, None)
|
||||
|
||||
|
||||
class AgentRegisterTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(dict(self.env))
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
self.app_uuid = str(uuidlib.uuid4())
|
||||
self.user_id = "ceo"
|
||||
self._seed(self.app_uuid, self.user_id)
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None, base=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(
|
||||
(base or self.base) + path, data=data, method=method
|
||||
)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
return response.status, json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, json.loads(error.read().decode("utf-8"))
|
||||
|
||||
def _seed(self, app_uuid, user_id):
|
||||
status, _ = self._request("POST", APPS_PATH, {"uuid": app_uuid}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
status, _ = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def _register(self, agent_id="agent-1", tags=("coreci",), app_uuid=None,
|
||||
user_id=None, headers=None, payload=None):
|
||||
if payload is None:
|
||||
payload = {
|
||||
"agent_id": agent_id,
|
||||
"app_registration_uuid": app_uuid or self.app_uuid,
|
||||
"user_id": user_id or self.user_id,
|
||||
"tags": list(tags),
|
||||
}
|
||||
return self._request("POST", REGISTER_PATH, payload, headers)
|
||||
|
||||
def _agent_row(self, agent_id="agent-1", app_uuid=None):
|
||||
return self.broker.store.get_agent(agent_id, app_uuid or self.app_uuid)
|
||||
|
||||
def _audit(self, action=None, result=None):
|
||||
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
|
||||
entries = [json.loads(raw) for raw in handle if raw.strip()]
|
||||
return [
|
||||
entry for entry in entries
|
||||
if (action is None or entry["action"] == action)
|
||||
and (result is None or entry["result"] == result)
|
||||
]
|
||||
|
||||
def _pepper(self):
|
||||
return self.broker.store.get_config("BROKER_INSTANCE_PEPPER")
|
||||
|
||||
def _app_header(self, token, app_uuid=None):
|
||||
return {"Authorization": "App %s %s" % (app_uuid or self.app_uuid, token)}
|
||||
|
||||
def _ttl_delta_seconds(self, body):
|
||||
expires = datetime.datetime.fromisoformat(body["token_expires_at"])
|
||||
now = datetime.datetime.now(datetime.timezone.utc)
|
||||
return (expires - now).total_seconds()
|
||||
|
||||
def test_gate_matrix_unknown_and_revoked_uuid_and_user(self):
|
||||
unknown_uuid = str(uuidlib.uuid4())
|
||||
status, body = self._register(app_uuid=unknown_uuid)
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual(403, body["error"])
|
||||
self.assertTrue(body["message"])
|
||||
self._request("DELETE", APPS_PATH + "/" + self.app_uuid, headers=self.admin)
|
||||
status, body = self._register()
|
||||
self.assertEqual(403, status)
|
||||
self._request("POST", APPS_PATH + "/" + self.app_uuid + "/reinstate", headers=self.admin)
|
||||
status, body = self._register(user_id="stranger")
|
||||
self.assertEqual(403, status)
|
||||
self._request("DELETE", USERS_PATH + "/" + self.user_id, headers=self.admin)
|
||||
status, body = self._register()
|
||||
self.assertEqual(403, status)
|
||||
denied = self._audit("register", "denied")
|
||||
self.assertEqual(4, len(denied))
|
||||
for entry in denied:
|
||||
self.assertEqual(REGISTER_PATH, entry["path"])
|
||||
self.assertEqual("agent-1", entry["agent_id"])
|
||||
self.assertEqual(unknown_uuid, denied[0]["app_registration_uuid"])
|
||||
self.assertEqual(self.user_id, denied[0]["user_id"])
|
||||
self.assertEqual(self.app_uuid, denied[1]["app_registration_uuid"])
|
||||
self.assertIsNone(self._agent_row())
|
||||
self._request("POST", USERS_PATH + "/" + self.user_id + "/reinstate", headers=self.admin)
|
||||
status, body = self._register()
|
||||
self.assertEqual(201, status)
|
||||
self.assertTrue(body["created"])
|
||||
self.assertIsNotNone(self._agent_row())
|
||||
|
||||
def test_malformed_bodies_are_400(self):
|
||||
base = {"agent_id": "a", "app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id, "tags": ["x"]}
|
||||
variants = (
|
||||
{}, {"agent_id": "a"}, {"agent_id": "a", "app_registration_uuid": self.app_uuid},
|
||||
dict(base, agent_id=None), dict(base, agent_id=123), dict(base, agent_id=""),
|
||||
dict(base, app_registration_uuid=None), dict(base, app_registration_uuid=[]),
|
||||
dict(base, user_id=7), dict(base, user_id=""), dict(base, tags=None),
|
||||
dict(base, tags="x"), dict(base, tags=["a", 1]), dict(base, tags=[["nested"]]),
|
||||
dict(base, tags={"a": 1}), [1, 2, 3], "nope",
|
||||
)
|
||||
for payload in variants:
|
||||
status, body = self._request("POST", REGISTER_PATH, payload)
|
||||
self.assertEqual(400, status, repr(payload))
|
||||
self.assertEqual(400, body["error"])
|
||||
self.assertTrue(body["message"])
|
||||
self.assertIsNone(self._agent_row())
|
||||
self.assertEqual([], self._audit("register"))
|
||||
|
||||
def test_first_register_creates_row_pepper_and_default_ttl(self):
|
||||
status, body = self._register(agent_id="px-1", tags=("proxmox", "readonly"))
|
||||
self.assertEqual(201, status)
|
||||
self.assertTrue(body["created"])
|
||||
self.assertEqual(self._pepper(), body["pepper"])
|
||||
expected = coreci_broker.derive_token(
|
||||
self._pepper(), self.app_uuid, self.user_id, "px-1", ["proxmox", "readonly"]
|
||||
)
|
||||
self.assertEqual(expected, body["token"])
|
||||
self.assertEqual(44, len(body["token"]))
|
||||
self.assertTrue(body["token"].isascii())
|
||||
delta = self._ttl_delta_seconds(body)
|
||||
self.assertGreater(delta, DEFAULT_TTL - 300)
|
||||
self.assertLess(delta, DEFAULT_TTL + 300)
|
||||
row = self._agent_row("px-1")
|
||||
self.assertNotEqual(body["token"], row[4])
|
||||
self.assertTrue(coreci_broker.verify_token(body["token"], row[4]))
|
||||
self.assertEqual(self.user_id, row[2])
|
||||
self.assertEqual(["proxmox", "readonly"], json.loads(row[3]))
|
||||
self.assertEqual(body["token_expires_at"], row[5])
|
||||
self.assertTrue(row[6])
|
||||
self.assertTrue(row[7])
|
||||
entries = self._audit("register", "ok")
|
||||
self.assertEqual(1, len(entries))
|
||||
entry = entries[0]
|
||||
self.assertEqual(self.user_id, entry["user_id"])
|
||||
self.assertEqual(self.app_uuid, entry["app_registration_uuid"])
|
||||
self.assertEqual("px-1", entry["agent_id"])
|
||||
self.assertEqual(REGISTER_PATH, entry["path"])
|
||||
|
||||
def test_reregister_unchanged_reissues_same_token_without_rehash(self):
|
||||
_, first = self._register()
|
||||
stored_hash = self._agent_row()[4]
|
||||
seen_at = self._agent_row()[7]
|
||||
status, second = self._register()
|
||||
self.assertEqual(200, status)
|
||||
self.assertFalse(second["created"])
|
||||
self.assertEqual(first["token"], second["token"])
|
||||
self.assertEqual(first["token_expires_at"], second["token_expires_at"])
|
||||
row = self._agent_row()
|
||||
self.assertEqual(stored_hash, row[4])
|
||||
self.assertGreaterEqual(row[7], seen_at)
|
||||
self.assertTrue(coreci_broker.verify_token(first["token"], row[4]))
|
||||
self.assertEqual(2, len(self._audit("register", "ok")))
|
||||
|
||||
def test_reregister_pepper_rule_tokenless_vs_valid_app_token(self):
|
||||
_, first = self._register()
|
||||
status, tokenless = self._register()
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(first["token"], tokenless["token"])
|
||||
self.assertIn("pepper", tokenless)
|
||||
status, with_token = self._register(headers=self._app_header(first["token"]))
|
||||
self.assertEqual(200, status)
|
||||
self.assertNotIn("pepper", with_token)
|
||||
self.assertEqual(first["token"], with_token["token"])
|
||||
status, bad_token = self._register(
|
||||
headers={"Authorization": "App %s wrong-token" % self.app_uuid}
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertIn("pepper", bad_token)
|
||||
|
||||
def test_changed_tags_overwrite_hash_token_and_expiry(self):
|
||||
_, first = self._register(tags=("red",))
|
||||
status, second = self._register(tags=("red", "blue"))
|
||||
self.assertEqual(200, status)
|
||||
self.assertFalse(second["created"])
|
||||
self.assertNotEqual(first["token"], second["token"])
|
||||
row = self._agent_row()
|
||||
self.assertFalse(coreci_broker.verify_token(first["token"], row[4]))
|
||||
self.assertTrue(coreci_broker.verify_token(second["token"], row[4]))
|
||||
self.assertEqual(["blue", "red"], json.loads(row[3]))
|
||||
self.assertEqual(second["token_expires_at"], row[5])
|
||||
|
||||
def test_changed_user_overwrites_row_identity(self):
|
||||
self._request("POST", USERS_PATH, {"user_id": "coo"}, self.admin)
|
||||
_, first = self._register()
|
||||
status, second = self._register(user_id="coo")
|
||||
self.assertEqual(200, status)
|
||||
self.assertNotEqual(first["token"], second["token"])
|
||||
row = self._agent_row()
|
||||
self.assertEqual("coo", row[2])
|
||||
self.assertFalse(coreci_broker.verify_token(first["token"], row[4]))
|
||||
self.assertTrue(coreci_broker.verify_token(second["token"], row[4]))
|
||||
|
||||
def test_expired_row_reregister_rehashes_and_extends(self):
|
||||
_, first = self._register()
|
||||
row = self._agent_row()
|
||||
old_hash, first_registered = row[4], row[6]
|
||||
past = (datetime.datetime.now(datetime.timezone.utc)
|
||||
- datetime.timedelta(seconds=60)).isoformat()
|
||||
self.broker.store.conn.execute(
|
||||
"UPDATE agent_registrations SET token_expires_at = ? WHERE agent_id = ?",
|
||||
(past, "agent-1"),
|
||||
)
|
||||
self.broker.store.conn.commit()
|
||||
status, second = self._register()
|
||||
self.assertEqual(200, status)
|
||||
self.assertFalse(second["created"])
|
||||
self.assertEqual(first["token"], second["token"])
|
||||
delta = self._ttl_delta_seconds(second)
|
||||
self.assertGreater(delta, DEFAULT_TTL - 300)
|
||||
row = self._agent_row()
|
||||
self.assertNotEqual(old_hash, row[4])
|
||||
self.assertEqual(first_registered, row[6])
|
||||
self.assertTrue(coreci_broker.verify_token(first["token"], row[4]))
|
||||
self.assertEqual(second["token_expires_at"], row[5])
|
||||
self.assertIn("pepper", second)
|
||||
|
||||
def test_revoked_app_blocks_new_registrations_rows_untouched(self):
|
||||
_, first = self._register()
|
||||
row_before = self._agent_row()
|
||||
self._request("DELETE", APPS_PATH + "/" + self.app_uuid, headers=self.admin)
|
||||
status, body = self._register(agent_id="fresh-agent")
|
||||
self.assertEqual(403, status)
|
||||
status, body = self._register()
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual(2, len(self._audit("register", "denied")))
|
||||
row_after = self._agent_row()
|
||||
self.assertEqual(row_before[4], row_after[4])
|
||||
self.assertEqual(row_before[5], row_after[5])
|
||||
self.assertTrue(coreci_broker.verify_token(first["token"], row_after[4]))
|
||||
|
||||
def test_token_ttl_override_honored(self):
|
||||
env = dict(self.env)
|
||||
env["BROKER_DB_PATH"] = os.path.join(self.tmp, "ttl.db")
|
||||
env["BROKER_AUDIT_LOG_PATH"] = os.path.join(self.tmp, "ttl.audit.log")
|
||||
env["BROKER_TOKEN_TTL"] = "60"
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
broker = coreci_broker.boot(dict(env))
|
||||
self.addCleanup(broker.audit.close)
|
||||
self.addCleanup(broker.store.close)
|
||||
_seed_direct(broker, self.app_uuid, self.user_id)
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
self.addCleanup(server.shutdown)
|
||||
self.addCleanup(server.server_close)
|
||||
base = "http://127.0.0.1:%d" % server.server_address[1]
|
||||
payload = {"agent_id": "ttl-agent", "app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id, "tags": []}
|
||||
status, body = self._request("POST", REGISTER_PATH, payload, base=base)
|
||||
self.assertEqual(201, status)
|
||||
delta = self._ttl_delta_seconds(body)
|
||||
self.assertGreater(delta, 30)
|
||||
self.assertLess(delta, 90)
|
||||
status, body = self._request("POST", REGISTER_PATH, payload, base=base)
|
||||
self.assertEqual(200, status)
|
||||
delta = self._ttl_delta_seconds(body)
|
||||
self.assertGreater(delta, 30)
|
||||
self.assertLess(delta, 90)
|
||||
|
||||
def test_user_allowlist_rejects_slash_backslash_and_control_chars(self):
|
||||
for bad in ("a/b", "a\\b", "a\nb", "a\x00b", "a\x1fb", "a\x7fb"):
|
||||
status, body = self._request("POST", USERS_PATH, {"user_id": bad}, self.admin)
|
||||
self.assertEqual(400, status, repr(bad))
|
||||
self.assertEqual({"error": 400, "message": "invalid user_id"}, body)
|
||||
status, body = self._request("GET", USERS_PATH, headers=self.admin)
|
||||
self.assertEqual([self.user_id], [row["user_id"] for row in body["allowlist"]])
|
||||
|
||||
def test_bearer_scheme_with_valid_creds_is_401(self):
|
||||
header = "Bearer " + base64.b64encode(
|
||||
("%s:%s" % (ADMIN_USER, ADMIN_PASS)).encode("utf-8")
|
||||
).decode("ascii")
|
||||
status, body = self._request("GET", WHOAMI_PATH, headers={"Authorization": header})
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(401, body["error"])
|
||||
|
||||
def test_basic_without_colon_is_401(self):
|
||||
header = "Basic " + base64.b64encode(b"just-a-username").decode("ascii")
|
||||
status, body = self._request("GET", WHOAMI_PATH, headers={"Authorization": header})
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(401, body["error"])
|
||||
|
||||
def test_basic_garbage_rejected_lowercase_scheme_accepted(self):
|
||||
status, body = self._request(
|
||||
"GET", WHOAMI_PATH, headers={"Authorization": "Basic !!!not-base64!!!"}
|
||||
)
|
||||
self.assertEqual(401, status)
|
||||
header = _basic_header(ADMIN_USER, ADMIN_PASS).replace("Basic", "basic")
|
||||
status, body = self._request("GET", WHOAMI_PATH, headers={"Authorization": header})
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(ADMIN_USER, body["username"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,242 @@
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from http.server import ThreadingHTTPServer
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
|
||||
def _echo(request):
|
||||
request.send_json(
|
||||
200,
|
||||
{"body": request.body, "query": request.query, "params": request.path_params},
|
||||
)
|
||||
|
||||
|
||||
def _unauthorized(request):
|
||||
request.send_error_json(401, "unauthorized")
|
||||
|
||||
|
||||
def _forbidden(request):
|
||||
request.send_error_json(403, "forbidden")
|
||||
|
||||
|
||||
def _conflict(request):
|
||||
request.send_error_json(409, "conflict")
|
||||
|
||||
|
||||
def _explode(request):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
|
||||
class RouterTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.ui_root = os.path.join(self.tmp, "ui")
|
||||
self.static_root = os.path.join(self.tmp, "static")
|
||||
os.makedirs(self.ui_root)
|
||||
os.makedirs(self.static_root)
|
||||
env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_UI_ROOT": self.ui_root,
|
||||
"BROKER_STATIC_ROOT": self.static_root,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(env)
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.broker.router.add("POST", "/test/echo/<name>", _echo)
|
||||
self.broker.router.add("GET", "/test/401", _unauthorized)
|
||||
self.broker.router.add("GET", "/test/403", _forbidden)
|
||||
self.broker.router.add("GET", "/test/409", _conflict)
|
||||
self.broker.router.add("GET", "/test/500", _explode)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
|
||||
def _request(self, method, path, data=None):
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=5) as response:
|
||||
return response.status, response.read(), response.headers
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read(), error.headers
|
||||
|
||||
def _request_json(self, method, path, payload=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
status, body, headers = self._request(method, path, data=data)
|
||||
return status, json.loads(body.decode("utf-8")), headers
|
||||
|
||||
def _write(self, directory, name, content):
|
||||
with open(os.path.join(directory, name), "w", encoding="utf-8") as handle:
|
||||
handle.write(content)
|
||||
|
||||
def test_serve_returns_started_threading_daemon_server(self):
|
||||
self.assertIsInstance(self.server, ThreadingHTTPServer)
|
||||
self.assertTrue(self.server.daemon_threads)
|
||||
self.assertEqual(self.broker, self.server.broker)
|
||||
|
||||
def test_healthz(self):
|
||||
status, payload, headers = self._request_json("GET", "/healthz")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, payload)
|
||||
self.assertEqual("application/json; charset=utf-8", headers.get("Content-Type"))
|
||||
|
||||
def test_error_contract_for_every_code(self):
|
||||
cases = [
|
||||
("GET", "/test/401", None, 401),
|
||||
("GET", "/test/403", None, 403),
|
||||
("GET", "/test/409", None, 409),
|
||||
("GET", "/test/500", None, 500),
|
||||
("GET", "/no/such/route", None, 404),
|
||||
("POST", "/test/echo/alice", b"{not json", 400),
|
||||
]
|
||||
for method, path, data, code in cases:
|
||||
with self.subTest(code=code):
|
||||
status, body, headers = self._request(method, path, data=data)
|
||||
self.assertEqual(code, status)
|
||||
self.assertEqual(
|
||||
"application/json; charset=utf-8", headers.get("Content-Type")
|
||||
)
|
||||
payload = json.loads(body.decode("utf-8"))
|
||||
self.assertEqual({"error", "message"}, set(payload))
|
||||
self.assertEqual(code, payload["error"])
|
||||
self.assertIsInstance(payload["message"], str)
|
||||
self.assertTrue(payload["message"])
|
||||
|
||||
def test_json_body_query_and_params_roundtrip(self):
|
||||
payload = {"k": [1, 2, {"z": True}], "s": "værdi", "n": None}
|
||||
status, body, _ = self._request_json(
|
||||
"POST", "/test/echo/alice?x=1&tag=a&tag=b", payload
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(
|
||||
{
|
||||
"body": payload,
|
||||
"query": {"x": "1", "tag": "a"},
|
||||
"params": {"name": "alice"},
|
||||
},
|
||||
body,
|
||||
)
|
||||
|
||||
def test_post_with_empty_body_yields_null_body(self):
|
||||
status, body, _ = self._request_json("POST", "/test/echo/alice")
|
||||
self.assertEqual(200, status)
|
||||
self.assertIsNone(body["body"])
|
||||
|
||||
def test_unknown_route_is_404_even_with_valid_json_body(self):
|
||||
status, body, _ = self._request_json("POST", "/api/broker/nope", {"a": 1})
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual({"error", "message"}, set(body))
|
||||
self.assertEqual(404, body["error"])
|
||||
|
||||
def test_method_mismatch_is_404(self):
|
||||
status, body, _ = self._request_json("DELETE", "/healthz")
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual(404, body["error"])
|
||||
|
||||
def test_static_content_types(self):
|
||||
fixtures = {
|
||||
"index.html": ("<html><body>ok</body></html>", "text/html; charset=utf-8"),
|
||||
"app.css": ("body{margin:0}", "text/css; charset=utf-8"),
|
||||
"app.js": ("console.log(1)", "application/javascript; charset=utf-8"),
|
||||
"mod.py": ("import os\n", "text/x-python; charset=utf-8"),
|
||||
"notes.txt": ("hello world", "text/plain; charset=utf-8"),
|
||||
"data.json": ("{\"k\": 1}", "application/json; charset=utf-8"),
|
||||
}
|
||||
for name, (content, _) in fixtures.items():
|
||||
self._write(self.ui_root, name, content)
|
||||
for name, (content, content_type) in fixtures.items():
|
||||
with self.subTest(name=name):
|
||||
status, body, headers = self._request("GET", "/ui/" + name)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(content.encode("utf-8"), body)
|
||||
self.assertEqual(content_type, headers.get("Content-Type"))
|
||||
|
||||
def test_static_serves_client_module(self):
|
||||
self._write(
|
||||
self.static_root, "coreci_broker_client.py", "import urllib.request\n"
|
||||
)
|
||||
status, body, headers = self._request("GET", "/static/coreci_broker_client.py")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(b"import urllib.request\n", body)
|
||||
self.assertEqual("text/x-python; charset=utf-8", headers.get("Content-Type"))
|
||||
|
||||
def test_ui_index_and_extensionless_page(self):
|
||||
self._write(self.ui_root, "index.html", "<html>index</html>")
|
||||
self._write(self.ui_root, "login.html", "<html>login</html>")
|
||||
status, body, _ = self._request("GET", "/ui/")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(b"<html>index</html>", body)
|
||||
status, body, _ = self._request("GET", "/ui/login")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(b"<html>login</html>", body)
|
||||
|
||||
def test_static_missing_file_is_404_contract(self):
|
||||
status, body, headers = self._request("GET", "/ui/absent.html")
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual("application/json; charset=utf-8", headers.get("Content-Type"))
|
||||
self.assertEqual(
|
||||
{"error": 404, "message": "not found"}, json.loads(body.decode("utf-8"))
|
||||
)
|
||||
|
||||
def test_static_prefix_rejects_non_get(self):
|
||||
status, body, _ = self._request("POST", "/ui/index.html", data=b"{}")
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual(404, json.loads(body.decode("utf-8"))["error"])
|
||||
|
||||
def test_ui_root_attribute_is_configurable(self):
|
||||
alt = os.path.join(self.tmp, "ui_alt")
|
||||
os.makedirs(alt)
|
||||
self._write(alt, "x.html", "alt root")
|
||||
self.broker.ui_root = alt
|
||||
status, body, _ = self._request("GET", "/ui/x.html")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(b"alt root", body)
|
||||
|
||||
def test_traversal_attempts_rejected(self):
|
||||
attempts = [
|
||||
"/ui/../etc/passwd",
|
||||
"/static/%2e%2e/etc/passwd",
|
||||
"/static/..%2fetc%2fpasswd",
|
||||
"/static/..%5cetc%5cpasswd",
|
||||
"/ui/..%5c..%5cetc%5cpasswd",
|
||||
"/static/%00etc/passwd",
|
||||
"/static//etc/passwd",
|
||||
"/ui/./.././../etc/passwd",
|
||||
]
|
||||
for path in attempts:
|
||||
with self.subTest(path=path):
|
||||
status, body, _ = self._request("GET", path)
|
||||
self.assertIn(status, (400, 404))
|
||||
payload = json.loads(body.decode("utf-8"))
|
||||
self.assertEqual({"error", "message"}, set(payload))
|
||||
self.assertEqual(status, payload["error"])
|
||||
|
||||
def test_symlink_escape_rejected(self):
|
||||
outside = os.path.join(self.tmp, "outside")
|
||||
os.makedirs(outside)
|
||||
self._write(outside, "secret.txt", "top secret")
|
||||
os.symlink(outside, os.path.join(self.ui_root, "alias"))
|
||||
status, body, _ = self._request("GET", "/ui/alias/secret.txt")
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual(
|
||||
{"error": 404, "message": "not found"}, json.loads(body.decode("utf-8"))
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,233 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
AUDIT_PATH = "/api/broker/audit"
|
||||
SECRETS_PREFIX = "/api/broker/secrets/"
|
||||
AUTH_401_BODY = {"error": 401, "message": "invalid app credentials"}
|
||||
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
|
||||
GITEA_SECRET = {"url": "https://git.coreci.dev", "token": "git-tok"}
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
class _SecretsCase(unittest.TestCase):
|
||||
creds_json = json.dumps(
|
||||
{"coreci/proxmox": PROXMOX_SECRET, "coreci/gitea": GITEA_SECRET}
|
||||
)
|
||||
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.creds_path = os.path.join(self.tmp, "creds.json")
|
||||
with open(self.creds_path, "w", encoding="utf-8") as handle:
|
||||
handle.write(self.creds_json)
|
||||
env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
"BROKER_CREDS_SOURCE": "file",
|
||||
"BROKER_CREDS_FILE": self.creds_path,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(env)
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
self.app_uuid = str(uuidlib.uuid4())
|
||||
self.user_id = "ceo"
|
||||
self._seed()
|
||||
self.registered = self._register("proxmox", ["coreci"])
|
||||
self.app = {"Authorization": "App %s %s" % (self.app_uuid, self.registered["token"])}
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
raw = response.read().decode("utf-8")
|
||||
return response.status, json.loads(raw) if raw else None
|
||||
except urllib.error.HTTPError as error:
|
||||
raw = error.read().decode("utf-8")
|
||||
return error.code, json.loads(raw) if raw else None
|
||||
|
||||
def _seed(self):
|
||||
status, _ = self._request("POST", APPS_PATH, {"uuid": self.app_uuid}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
status, _ = self._request("POST", USERS_PATH, {"user_id": self.user_id}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def _register(self, agent_id, tags):
|
||||
payload = {"agent_id": agent_id, "app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id, "tags": list(tags)}
|
||||
status, body = self._request("POST", REGISTER_PATH, payload)
|
||||
self.assertEqual(201, status)
|
||||
return body
|
||||
|
||||
def _secret(self, suffix, headers=None):
|
||||
return self._request("GET", SECRETS_PREFIX + suffix,
|
||||
headers=self.app if headers is None else headers)
|
||||
|
||||
def _audit(self, action):
|
||||
query = urllib.parse.urlencode({"action": action, "limit": "1000"})
|
||||
status, body = self._request("GET", AUDIT_PATH + "?" + query, headers=self.admin)
|
||||
self.assertEqual(200, status)
|
||||
return body["entries"]
|
||||
|
||||
def _read_entries(self, result):
|
||||
return [entry for entry in self._audit("read") if entry["result"] == result]
|
||||
|
||||
def _assert_identity(self, entry, path, agent_id="proxmox"):
|
||||
self.assertEqual("read", entry["action"])
|
||||
self.assertEqual(path, entry["path"])
|
||||
self.assertEqual(self.user_id, entry["user_id"])
|
||||
self.assertEqual(self.app_uuid, entry["app_registration_uuid"])
|
||||
self.assertEqual(agent_id, entry["agent_id"])
|
||||
|
||||
def _ghost_headers(self, agent_id):
|
||||
ghost = self._register(agent_id, ["coreci"])
|
||||
return {"Authorization": "App %s %s" % (self.app_uuid, ghost["token"])}
|
||||
|
||||
|
||||
class SecretsReadTests(_SecretsCase):
|
||||
def test_valid_read_returns_document_and_ok_audit(self):
|
||||
status, body = self._secret("coreci/proxmox")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("coreci/proxmox", body["path"])
|
||||
self.assertEqual(PROXMOX_SECRET, body["data"])
|
||||
self.assertRegex(body["version"], r"^[0-9a-f]{8}$")
|
||||
self.assertIn("updated_at", body)
|
||||
entries = self._read_entries("ok")
|
||||
self.assertEqual(1, len(entries))
|
||||
self._assert_identity(entries[0], "coreci/proxmox")
|
||||
|
||||
def test_encoded_slash_path_resolves_to_same_secret(self):
|
||||
status, body = self._secret("coreci%2Fproxmox")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("coreci/proxmox", body["path"])
|
||||
self.assertEqual(PROXMOX_SECRET, body["data"])
|
||||
self.assertEqual(1, len(self._read_entries("ok")))
|
||||
|
||||
def test_missing_app_header_is_401(self):
|
||||
status, body = self._secret("coreci/proxmox", headers={})
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
self.assertEqual([], self._read_entries("ok"))
|
||||
self.assertEqual([], self._read_entries("denied"))
|
||||
|
||||
def test_tampered_token_is_401_with_denied_audit(self):
|
||||
token = self.registered["token"]
|
||||
twisted = ("x" if token[0] != "x" else "y") + token[1:]
|
||||
status, body = self._secret("coreci/proxmox", headers={
|
||||
"Authorization": "App %s %s" % (self.app_uuid, twisted)})
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual(AUTH_401_BODY, body)
|
||||
entries = self._audit("denied")
|
||||
self.assertEqual(1, len(entries))
|
||||
self.assertEqual("denied", entries[0]["action"])
|
||||
self.assertEqual("denied", entries[0]["result"])
|
||||
self.assertEqual(self.app_uuid, entries[0]["app_registration_uuid"])
|
||||
self.assertIsNone(entries[0]["user_id"])
|
||||
self.assertIsNone(entries[0]["agent_id"])
|
||||
|
||||
def test_cross_path_read_is_403_with_denied_audit(self):
|
||||
status, body = self._secret("coreci/gitea")
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual(403, body["error"])
|
||||
self.assertTrue(body["message"])
|
||||
entries = self._read_entries("denied")
|
||||
self.assertEqual(1, len(entries))
|
||||
self._assert_identity(entries[0], "coreci/gitea")
|
||||
|
||||
def test_out_of_scope_paths_are_403(self):
|
||||
suffixes = ("coreci/proxmox/extra", "other/proxmox", "coreci", "CORECI/proxmox")
|
||||
for suffix in suffixes:
|
||||
with self.subTest(suffix=suffix):
|
||||
status, body = self._secret(suffix)
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual(403, body["error"])
|
||||
self.assertEqual(len(suffixes), len(self._read_entries("denied")))
|
||||
|
||||
def test_absent_secret_is_404_with_error_audit(self):
|
||||
status, body = self._secret("coreci/nonexistent",
|
||||
headers=self._ghost_headers("nonexistent"))
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual({"error": 404, "message": "not found"}, body)
|
||||
entries = self._read_entries("error")
|
||||
self.assertEqual(1, len(entries))
|
||||
self._assert_identity(entries[0], "coreci/nonexistent", "nonexistent")
|
||||
|
||||
def test_traversal_payloads_never_reach_source(self):
|
||||
original = self.broker.creds_source.get_secret
|
||||
self.addCleanup(setattr, self.broker.creds_source, "get_secret", original)
|
||||
|
||||
def detonate(path):
|
||||
raise AssertionError("creds source reached for %r" % path)
|
||||
|
||||
self.broker.creds_source.get_secret = detonate
|
||||
attempts = ["%2e%2e/", "..%2f", "coreci%5cproxmox", "coreci%00proxmox",
|
||||
"//coreci/proxmox", "%2e%2e/coreci/proxmox", "coreci/..%2fproxmox"]
|
||||
for suffix in attempts:
|
||||
with self.subTest(suffix=suffix):
|
||||
status, body = self._secret(suffix)
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual(403, body["error"])
|
||||
self.assertEqual(len(attempts), len(self._read_entries("denied")))
|
||||
|
||||
def test_audit_completeness_action_read_covers_all_results(self):
|
||||
self._secret("coreci/proxmox")
|
||||
self._secret("coreci/gitea")
|
||||
self._secret("coreci/nonexistent", headers=self._ghost_headers("nonexistent"))
|
||||
entries = self._audit("read")
|
||||
self.assertEqual(3, len(entries))
|
||||
by_result = {entry["result"]: entry for entry in entries}
|
||||
self.assertEqual({"ok", "denied", "error"}, set(by_result))
|
||||
self._assert_identity(by_result["ok"], "coreci/proxmox")
|
||||
self._assert_identity(by_result["denied"], "coreci/gitea")
|
||||
self._assert_identity(by_result["error"], "coreci/nonexistent", "nonexistent")
|
||||
|
||||
|
||||
class SecretsSourceErrorTests(_SecretsCase):
|
||||
creds_json = "{not-json"
|
||||
|
||||
def test_source_failure_is_500_contract_with_error_audit(self):
|
||||
status, body = self._secret("coreci/proxmox")
|
||||
self.assertEqual(500, status)
|
||||
self.assertEqual({"error": 500, "message": "creds source unavailable"}, body)
|
||||
entries = self._read_entries("error")
|
||||
self.assertEqual(1, len(entries))
|
||||
self._assert_identity(entries[0], "coreci/proxmox")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,318 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
STATE_PATH = "/api/broker/agents/%s/state"
|
||||
STUB_PREFIX = "/api/v1/broker-agent-state/"
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
class _StubOwuiHandler(BaseHTTPRequestHandler):
|
||||
documents = None
|
||||
records = None
|
||||
|
||||
def log_message(self, format, *args):
|
||||
pass
|
||||
|
||||
def do_GET(self):
|
||||
agent_id = self.path[len(STUB_PREFIX):]
|
||||
self.records.append(("GET", self.path, self.headers.get("Authorization"), None))
|
||||
if agent_id not in self.documents:
|
||||
self._send(404, {"error": 404})
|
||||
return
|
||||
self._send(200, self.documents[agent_id])
|
||||
|
||||
def do_PUT(self):
|
||||
agent_id = self.path[len(STUB_PREFIX):]
|
||||
length = int(self.headers.get("Content-Length") or "0")
|
||||
raw = self.rfile.read(length)
|
||||
self.records.append(("PUT", self.path, self.headers.get("Authorization"), raw))
|
||||
self.documents[agent_id] = json.loads(raw.decode("utf-8"))
|
||||
self._send(200, {"ok": True})
|
||||
|
||||
def _send(self, status, payload):
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
|
||||
class StubOwuiServer:
|
||||
def __init__(self):
|
||||
self.documents, self.records = {}, []
|
||||
handler = type("BoundStubHandler", (_StubOwuiHandler,),
|
||||
{"documents": self.documents, "records": self.records})
|
||||
self.server = ThreadingHTTPServer(("127.0.0.1", 0), handler)
|
||||
threading.Thread(target=self.server.serve_forever, daemon=True).start()
|
||||
|
||||
@property
|
||||
def base_url(self):
|
||||
return "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
|
||||
def close(self):
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
|
||||
|
||||
class _StateCase(unittest.TestCase):
|
||||
backend = "internal"
|
||||
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
"BROKER_AGENT_STATE_BACKEND": self.backend,
|
||||
}
|
||||
self.env.update(getattr(self, "extra_env", {}))
|
||||
self._start_broker()
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
self.app_uuid = str(uuidlib.uuid4())
|
||||
self.user_id = "ceo"
|
||||
self._seed(self.app_uuid, self.user_id)
|
||||
self.registered = self._register("agent-1", ["b", "a"])
|
||||
self.app = {"Authorization": "App %s %s" % (self.app_uuid, self.registered["token"])}
|
||||
|
||||
def _start_broker(self):
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(dict(self.env))
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
raw = response.read().decode("utf-8")
|
||||
return response.status, json.loads(raw) if raw else None
|
||||
except urllib.error.HTTPError as error:
|
||||
raw = error.read().decode("utf-8")
|
||||
return error.code, json.loads(raw) if raw else None
|
||||
|
||||
def _state(self, method, agent_id, payload=None, headers=None):
|
||||
return self._request(method, STATE_PATH % agent_id, payload,
|
||||
self.app if headers is None else headers)
|
||||
|
||||
def _seed(self, app_uuid, user_id):
|
||||
status, _ = self._request("POST", APPS_PATH, {"uuid": app_uuid}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
status, _ = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def _register(self, agent_id, tags):
|
||||
payload = {"agent_id": agent_id, "app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id, "tags": list(tags)}
|
||||
status, body = self._request("POST", REGISTER_PATH, payload)
|
||||
self.assertEqual(201, status)
|
||||
return body
|
||||
|
||||
|
||||
class InternalStateTests(_StateCase):
|
||||
def test_put_then_get_roundtrip_is_document_faithful(self):
|
||||
state = {"agent_id": "agent-1", "app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id, "tags": ["b", "a"],
|
||||
"pepper": self.registered["pepper"],
|
||||
"current_token": self.registered["token"],
|
||||
"token_expires_at": self.registered["token_expires_at"],
|
||||
"first_registered_at": "t0", "last_seen_at": "t1",
|
||||
"nested": {"flags": [1, 2.5, None, True], "deep": {"k": None}}, "z": 1}
|
||||
status, body = self._state("PUT", "agent-1", state)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
status, body = self._state("GET", "agent-1")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(state, body)
|
||||
self.assertEqual(list(state), list(body))
|
||||
self.assertEqual(state, self.broker.store.get_agent_state("agent-1", self.app_uuid))
|
||||
|
||||
def test_state_survives_broker_restart(self):
|
||||
state = {"counter": 7, "notes": ["x"], "pepper": "keep-me"}
|
||||
status, _ = self._state("PUT", "agent-1", state)
|
||||
self.assertEqual(200, status)
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
self.broker.store.close()
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
second = coreci_broker.boot(dict(self.env))
|
||||
self.addCleanup(second.audit.close)
|
||||
self.addCleanup(second.store.close)
|
||||
server = coreci_broker.serve("127.0.0.1", 0, second)
|
||||
self.addCleanup(server.shutdown)
|
||||
self.addCleanup(server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % server.server_address[1]
|
||||
status, body = self._state("GET", "agent-1")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(state, body)
|
||||
row = second.store.get_agent("agent-1", self.app_uuid)
|
||||
self.assertTrue(coreci_broker.verify_token(self.registered["token"], row[4]))
|
||||
|
||||
def test_identity_mismatch_is_403_on_get_and_put(self):
|
||||
second = self._register("agent-2", ["coreci"])
|
||||
for method, payload in (("GET", None), ("PUT", {"who": "agent-2"})):
|
||||
status, body = self._state(method, "agent-2", payload)
|
||||
self.assertEqual(403, status, method)
|
||||
self.assertEqual(403, body["error"])
|
||||
self.assertTrue(body["message"])
|
||||
status, body = self._state("GET", "agent-2", headers={
|
||||
"Authorization": "App %s %s" % (self.app_uuid, second["token"])})
|
||||
self.assertEqual(404, status)
|
||||
|
||||
def test_unregistered_agent_id_is_404_on_get(self):
|
||||
status, body = self._state("GET", "ghost-agent")
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual({"error": 404, "message": "not found"}, body)
|
||||
|
||||
def test_get_without_state_is_404(self):
|
||||
status, body = self._state("GET", "agent-1")
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual({"error": 404, "message": "not found"}, body)
|
||||
|
||||
def test_put_non_dict_is_400(self):
|
||||
for payload in ([1, 2], "nope", 5, None):
|
||||
status, body = self._state("PUT", "agent-1", payload)
|
||||
self.assertEqual(400, status, repr(payload))
|
||||
self.assertEqual(400, body["error"])
|
||||
status, _ = self._state("GET", "agent-1")
|
||||
self.assertEqual(404, status)
|
||||
|
||||
def test_put_above_256kib_is_400(self):
|
||||
status, body = self._state("PUT", "agent-1", {"pad": "x" * (300 * 1024)})
|
||||
self.assertEqual(400, status)
|
||||
self.assertEqual(400, body["error"])
|
||||
status, _ = self._state("GET", "agent-1")
|
||||
self.assertEqual(404, status)
|
||||
|
||||
def test_put_at_exactly_256kib_is_stored(self):
|
||||
pad = "x" * (256 * 1024 - len(json.dumps({"pad": ""})))
|
||||
status, body = self._state("PUT", "agent-1", {"pad": pad})
|
||||
self.assertEqual(200, status)
|
||||
status, body = self._state("GET", "agent-1")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"pad": pad}, body)
|
||||
|
||||
def test_missing_app_header_is_401(self):
|
||||
status, body = self._state("GET", "agent-1", headers={})
|
||||
self.assertEqual(401, status)
|
||||
self.assertEqual({"error": 401, "message": "invalid app credentials"}, body)
|
||||
|
||||
|
||||
class OwuiStateTests(_StateCase):
|
||||
backend = "owui"
|
||||
|
||||
def setUp(self):
|
||||
self.stub = StubOwuiServer()
|
||||
self.addCleanup(self.stub.close)
|
||||
self.extra_env = {
|
||||
"BROKER_OWUI_BASE_URL": self.stub.base_url,
|
||||
"BROKER_OWUI_API_KEY": "test-key",
|
||||
}
|
||||
super().setUp()
|
||||
|
||||
def test_put_then_get_roundtrip_flows_through_stub(self):
|
||||
state = {"agent_id": "agent-1", "current_token": "tok", "n": 3}
|
||||
status, body = self._state("PUT", "agent-1", state)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
self.assertEqual(state, self.stub.documents["agent-1"])
|
||||
status, body = self._state("GET", "agent-1")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(state, body)
|
||||
puts = [record for record in self.stub.records if record[0] == "PUT"]
|
||||
gets = [record for record in self.stub.records if record[0] == "GET"]
|
||||
self.assertEqual(1, len(puts))
|
||||
self.assertEqual(1, len(gets))
|
||||
for _, path, auth, _ in self.stub.records:
|
||||
self.assertEqual("Bearer test-key", auth)
|
||||
self.assertEqual(STUB_PREFIX + "agent-1", puts[0][1])
|
||||
self.assertEqual(STUB_PREFIX + "agent-1", gets[0][1])
|
||||
self.assertEqual(state, json.loads(puts[0][3].decode("utf-8")))
|
||||
|
||||
def test_stub_miss_maps_to_broker_404(self):
|
||||
status, body = self._state("GET", "agent-1")
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual({"error": 404, "message": "not found"}, body)
|
||||
|
||||
def test_identity_enforced_before_backend_call(self):
|
||||
self._register("agent-2", ["coreci"])
|
||||
status, body = self._state("GET", "agent-2")
|
||||
self.assertEqual(403, status)
|
||||
status, body = self._state("PUT", "agent-2", {"x": 1})
|
||||
self.assertEqual(403, status)
|
||||
self.assertEqual([], self.stub.records)
|
||||
|
||||
|
||||
class BackendToggleTests(unittest.TestCase):
|
||||
def _boot(self, extra):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
env = {
|
||||
"BROKER_ENV_PATH": os.path.join(tmp.name, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(tmp.name, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(tmp.name, "broker.audit.log"),
|
||||
}
|
||||
env.update(extra)
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
broker = coreci_broker.boot(env)
|
||||
self.addCleanup(broker.audit.close)
|
||||
self.addCleanup(broker.store.close)
|
||||
return broker
|
||||
|
||||
def test_default_backend_is_internal(self):
|
||||
broker = self._boot({})
|
||||
self.assertIsInstance(broker.agent_state_backend, coreci_broker.InternalAgentStateBackend)
|
||||
|
||||
def test_owui_backend_selected_by_config(self):
|
||||
broker = self._boot({
|
||||
"BROKER_AGENT_STATE_BACKEND": "owui",
|
||||
"BROKER_OWUI_BASE_URL": "http://127.0.0.1:9999/",
|
||||
"BROKER_OWUI_API_KEY": "k",
|
||||
})
|
||||
backend = broker.agent_state_backend
|
||||
self.assertIsInstance(backend, coreci_broker.OwuiAgentStateBackend)
|
||||
self.assertEqual("http://127.0.0.1:9999", backend.base_url)
|
||||
self.assertEqual("k", backend.api_key)
|
||||
|
||||
def test_unknown_backend_value_falls_back_to_internal(self):
|
||||
broker = self._boot({"BROKER_AGENT_STATE_BACKEND": "banana"})
|
||||
self.assertIsInstance(broker.agent_state_backend, coreci_broker.InternalAgentStateBackend)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,181 @@
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
|
||||
class StaticDisclosureGuardTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_UI_ROOT": self.tmp,
|
||||
"BROKER_STATIC_ROOT": self.tmp,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(dict(self.env))
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
with open(
|
||||
os.path.join(self.tmp, "coreci_broker_client.py"), "w", encoding="utf-8"
|
||||
) as handle:
|
||||
handle.write("import urllib.request\n")
|
||||
with open(
|
||||
os.path.join(self.tmp, ".hidden.css"), "w", encoding="utf-8"
|
||||
) as handle:
|
||||
handle.write("body{display:none}")
|
||||
os.symlink(self.env["BROKER_ENV_PATH"], os.path.join(self.tmp, "env_alias"))
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
|
||||
def _get(self, path):
|
||||
try:
|
||||
with urllib.request.urlopen(self.base + path, timeout=5) as response:
|
||||
return response.status, response.read()
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read()
|
||||
|
||||
def test_state_files_never_served_when_roots_overlap_state_dir(self):
|
||||
for path in (
|
||||
"/static/.env",
|
||||
"/static/broker.db",
|
||||
"/static/broker.db-journal",
|
||||
"/static/broker.db-wal",
|
||||
"/static/broker.audit.log",
|
||||
"/ui/.env",
|
||||
"/ui/broker.db",
|
||||
"/ui/broker.audit.log",
|
||||
):
|
||||
with self.subTest(path=path):
|
||||
status, body = self._get(path)
|
||||
self.assertEqual(404, status)
|
||||
self.assertEqual(
|
||||
{"error": 404, "message": "not found"}, json.loads(body.decode("utf-8"))
|
||||
)
|
||||
|
||||
def test_symlink_alias_to_env_rejected(self):
|
||||
status, _ = self._get("/static/env_alias")
|
||||
self.assertEqual(404, status)
|
||||
|
||||
def test_dotfiles_rejected_even_with_known_extension(self):
|
||||
for path in ("/static/.hidden.css", "/ui/.hidden.css"):
|
||||
with self.subTest(path=path):
|
||||
status, _ = self._get(path)
|
||||
self.assertEqual(404, status)
|
||||
|
||||
def test_public_files_still_served(self):
|
||||
status, body = self._get("/static/coreci_broker_client.py")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(b"import urllib.request\n", body)
|
||||
|
||||
def test_default_config_relative_paths_do_not_disclose_state(self):
|
||||
cwd = os.getcwd()
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
os.chdir(tmp.name)
|
||||
try:
|
||||
env = {"BROKER_ENV_PATH": os.path.join(tmp.name, ".env")}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
broker = coreci_broker.boot(dict(env))
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
base = "http://127.0.0.1:%d" % server.server_address[1]
|
||||
for path in ("/static/.env", "/static/broker.db", "/static/broker.audit.log"):
|
||||
with self.subTest(path=path):
|
||||
try:
|
||||
with urllib.request.urlopen(base + path, timeout=5) as response:
|
||||
status = response.status
|
||||
except urllib.error.HTTPError as error:
|
||||
status = error.code
|
||||
self.assertEqual(404, status)
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
broker.audit.close()
|
||||
broker.store.close()
|
||||
finally:
|
||||
os.chdir(cwd)
|
||||
|
||||
|
||||
class CredsFileDisclosureGuardTests(unittest.TestCase):
|
||||
SECRET_JSON = json.dumps({"coreci/proxmox": {"token": "pve-tok"}})
|
||||
|
||||
def _get(self, base, path):
|
||||
try:
|
||||
with urllib.request.urlopen(base + path, timeout=5) as response:
|
||||
return response.status, response.read()
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read()
|
||||
|
||||
def _write_creds(self, path):
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "w", encoding="utf-8") as handle:
|
||||
handle.write(self.SECRET_JSON)
|
||||
|
||||
def test_default_layout_never_serves_creds_file(self):
|
||||
cwd = os.getcwd()
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
creds_path = os.path.join(tmp.name, "data", "creds.json")
|
||||
self._write_creds(creds_path)
|
||||
os.chdir(tmp.name)
|
||||
try:
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
broker = coreci_broker.boot({"BROKER_ENV_PATH": os.path.join(tmp.name, ".env")})
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
base = "http://127.0.0.1:%d" % server.server_address[1]
|
||||
for path in ("/static/data/creds.json", "/static/data%2Fcreds.json"):
|
||||
with self.subTest(path=path):
|
||||
status, body = self._get(base, path)
|
||||
self.assertEqual(404, status)
|
||||
self.assertNotIn(b"pve-tok", body)
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
broker.audit.close()
|
||||
broker.store.close()
|
||||
finally:
|
||||
os.chdir(cwd)
|
||||
|
||||
def test_creds_file_blocked_when_roots_overlap_creds_dir(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
creds_path = os.path.join(tmp.name, "creds.json")
|
||||
self._write_creds(creds_path)
|
||||
env = {
|
||||
"BROKER_ENV_PATH": os.path.join(tmp.name, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(tmp.name, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(tmp.name, "broker.audit.log"),
|
||||
"BROKER_UI_ROOT": tmp.name,
|
||||
"BROKER_STATIC_ROOT": tmp.name,
|
||||
"BROKER_CREDS_FILE": creds_path,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
broker = coreci_broker.boot(dict(env))
|
||||
self.addCleanup(broker.audit.close)
|
||||
self.addCleanup(broker.store.close)
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
self.addCleanup(server.server_close)
|
||||
self.addCleanup(server.shutdown)
|
||||
base = "http://127.0.0.1:%d" % server.server_address[1]
|
||||
for path in ("/static/creds.json", "/ui/creds.json"):
|
||||
with self.subTest(path=path):
|
||||
status, body = self._get(base, path)
|
||||
self.assertEqual(404, status)
|
||||
self.assertNotIn(b"pve-tok", body)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,155 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import io
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
TABLE_COLUMNS = {
|
||||
"broker_config": ["key", "value"],
|
||||
"app_registrations": ["uuid", "name", "status", "created_at", "revoked_at"],
|
||||
"user_allowlist": ["user_id", "email", "status", "created_at", "revoked_at"],
|
||||
"agent_registrations": [
|
||||
"agent_id",
|
||||
"app_registration_uuid",
|
||||
"user_id",
|
||||
"tags",
|
||||
"token_hash",
|
||||
"token_expires_at",
|
||||
"first_registered_at",
|
||||
"last_seen_at",
|
||||
],
|
||||
"admin_sessions": ["session_id", "created_at", "expires_at"],
|
||||
}
|
||||
|
||||
|
||||
class StoreTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.env_path = os.path.join(tmp.name, ".env")
|
||||
self.db_path = os.path.join(tmp.name, "broker.db")
|
||||
|
||||
def _boot(self, extra_env=None):
|
||||
env = {"BROKER_ENV_PATH": self.env_path, "BROKER_DB_PATH": self.db_path}
|
||||
if extra_env:
|
||||
env.update(extra_env)
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
broker = coreci_broker.boot(env)
|
||||
self.addCleanup(broker.store.close)
|
||||
return broker, buffer.getvalue()
|
||||
|
||||
def _connect(self):
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
self.addCleanup(conn.close)
|
||||
return conn
|
||||
|
||||
def test_boot_creates_db_with_mode_0600(self):
|
||||
broker, stdout = self._boot()
|
||||
self.assertTrue(os.path.isfile(self.db_path))
|
||||
self.assertEqual(0o600, os.stat(self.db_path).st_mode & 0o777)
|
||||
self.assertEqual(1, stdout.count(broker.config["BROKER_ADMIN_PASS"]))
|
||||
|
||||
def test_db_mode_0600_independent_of_umask(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
env = {
|
||||
"BROKER_ENV_PATH": os.path.join(tmp.name, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(tmp.name, "broker.db"),
|
||||
}
|
||||
old_umask = os.umask(0o777)
|
||||
try:
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
broker = coreci_broker.boot(env)
|
||||
self.addCleanup(broker.store.close)
|
||||
self.assertEqual(0o600, os.stat(env["BROKER_DB_PATH"]).st_mode & 0o777)
|
||||
self.assertEqual(0o600, os.stat(env["BROKER_ENV_PATH"]).st_mode & 0o777)
|
||||
finally:
|
||||
os.umask(old_umask)
|
||||
|
||||
def test_all_five_tables_present_with_columns(self):
|
||||
self._boot()
|
||||
conn = self._connect()
|
||||
tables = {
|
||||
row[0]
|
||||
for row in conn.execute("SELECT name FROM sqlite_master WHERE type = 'table'")
|
||||
}
|
||||
self.assertTrue(set(TABLE_COLUMNS).issubset(tables))
|
||||
for table, expected in TABLE_COLUMNS.items():
|
||||
actual = [row[1] for row in conn.execute("PRAGMA table_info(%s)" % table)]
|
||||
self.assertEqual(expected, actual)
|
||||
|
||||
def test_index_on_agent_registrations_app_registration_uuid(self):
|
||||
self._boot()
|
||||
conn = self._connect()
|
||||
names = [
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type = 'index' AND tbl_name = 'agent_registrations'"
|
||||
)
|
||||
]
|
||||
matching = []
|
||||
for name in names:
|
||||
columns = [row[2] for row in conn.execute("PRAGMA index_info(%s)" % name)]
|
||||
if columns and columns[0] == "app_registration_uuid":
|
||||
matching.append(name)
|
||||
self.assertTrue(matching)
|
||||
|
||||
def test_pepper_synced_into_broker_config(self):
|
||||
broker, _ = self._boot()
|
||||
conn = self._connect()
|
||||
row = conn.execute(
|
||||
"SELECT value FROM broker_config WHERE key = 'BROKER_INSTANCE_PEPPER'"
|
||||
).fetchone()
|
||||
self.assertIsNotNone(row)
|
||||
self.assertEqual(broker.config["BROKER_INSTANCE_PEPPER"], row[0])
|
||||
self.assertEqual(32, len(base64.b64decode(row[0])))
|
||||
|
||||
def test_pepper_env_override_synced(self):
|
||||
self._boot({"BROKER_INSTANCE_PEPPER": "Zm9vYmFy"})
|
||||
conn = self._connect()
|
||||
row = conn.execute(
|
||||
"SELECT value FROM broker_config WHERE key = 'BROKER_INSTANCE_PEPPER'"
|
||||
).fetchone()
|
||||
self.assertEqual("Zm9vYmFy", row[0])
|
||||
|
||||
def test_second_boot_silent_and_idempotent(self):
|
||||
first, first_stdout = self._boot()
|
||||
second, second_stdout = self._boot()
|
||||
self.assertEqual(1, first_stdout.count(first.config["BROKER_ADMIN_PASS"]))
|
||||
self.assertEqual("", second_stdout)
|
||||
self.assertEqual(
|
||||
first.config["BROKER_INSTANCE_PEPPER"],
|
||||
second.config["BROKER_INSTANCE_PEPPER"],
|
||||
)
|
||||
self.assertEqual(
|
||||
first.config["BROKER_INSTANCE_PEPPER"],
|
||||
second.store.get_config("BROKER_INSTANCE_PEPPER"),
|
||||
)
|
||||
self.assertEqual(0o600, os.stat(self.db_path).st_mode & 0o777)
|
||||
|
||||
def test_restart_preserves_rows(self):
|
||||
broker, _ = self._boot()
|
||||
broker.store.conn.execute(
|
||||
"INSERT INTO app_registrations (uuid, name) VALUES (?, ?)",
|
||||
("0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a", "proxmox-tool"),
|
||||
)
|
||||
broker.store.conn.commit()
|
||||
broker.store.close()
|
||||
conn = self._connect()
|
||||
row = conn.execute(
|
||||
"SELECT name FROM app_registrations WHERE uuid = ?",
|
||||
("0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a",),
|
||||
).fetchone()
|
||||
self.assertEqual("proxmox-tool", row[0])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,182 @@
|
||||
import base64
|
||||
import hashlib
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
PEPPER = "ZpQ1hP3rLmX7wYc2vB4nT6kS9dF5gR8j"
|
||||
APP_UUID = "3f2a9c1e-77d0-4f0a-9b2e-6c1d5a8e4b3c"
|
||||
USER_ID = "user-42"
|
||||
AGENT_ID = "proxmox"
|
||||
|
||||
B64_ALPHABET = set(
|
||||
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/="
|
||||
)
|
||||
|
||||
|
||||
def _independent_token(pepper, app_uuid, user_id, agent_id, tags):
|
||||
payload = "{}:{}:{}:{}:{}".format(
|
||||
pepper,
|
||||
app_uuid,
|
||||
user_id,
|
||||
agent_id,
|
||||
",".join(sorted(tags)),
|
||||
)
|
||||
digest = hashlib.sha256(payload.encode("utf-8")).digest()
|
||||
return base64.b64encode(digest).decode("ascii")
|
||||
|
||||
|
||||
class FormulaTests(unittest.TestCase):
|
||||
def test_matches_independent_computation(self):
|
||||
token = coreci_broker.derive_token(
|
||||
PEPPER, APP_UUID, USER_ID, AGENT_ID, ["infra", "proxmox"]
|
||||
)
|
||||
expected = _independent_token(
|
||||
PEPPER, APP_UUID, USER_ID, AGENT_ID, ["infra", "proxmox"]
|
||||
)
|
||||
self.assertEqual(expected, token)
|
||||
|
||||
def test_exact_spec_formula_byte_for_byte(self):
|
||||
expected = base64.b64encode(
|
||||
hashlib.sha256(
|
||||
"{}:{}:{}:{}:{}".format(
|
||||
PEPPER, APP_UUID, USER_ID, AGENT_ID, "a,b"
|
||||
).encode("utf-8")
|
||||
).digest()
|
||||
).decode("ascii")
|
||||
actual = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a", "b"])
|
||||
self.assertEqual(expected, actual)
|
||||
|
||||
def test_empty_tags_match_independent_computation(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, [])
|
||||
self.assertEqual(
|
||||
_independent_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, []),
|
||||
token,
|
||||
)
|
||||
|
||||
def test_deterministic_for_same_inputs(self):
|
||||
first = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
second = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertEqual(first, second)
|
||||
|
||||
|
||||
class ShapeTests(unittest.TestCase):
|
||||
def test_token_is_exactly_44_chars(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertEqual(44, len(token))
|
||||
|
||||
def test_token_is_pure_ascii(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertTrue(token.isascii())
|
||||
token.encode("ascii")
|
||||
|
||||
def test_token_carries_base64_padding_char(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertTrue(token.endswith("="))
|
||||
self.assertEqual(1, token.count("="))
|
||||
|
||||
def test_token_uses_only_base64_alphabet(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertTrue(set(token) <= B64_ALPHABET)
|
||||
|
||||
def test_token_decodes_to_32_byte_sha256_digest(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertEqual(32, len(base64.b64decode(token)))
|
||||
|
||||
def test_token_has_no_prefix(self):
|
||||
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
for prefix in ("App", "Bearer", "scrypt$", "$2", "token", "tk_", "coreci"):
|
||||
with self.subTest(prefix=prefix):
|
||||
self.assertFalse(token.startswith(prefix))
|
||||
self.assertNotIn(" ", token)
|
||||
self.assertNotIn(":", token)
|
||||
self.assertNotIn("$", token)
|
||||
|
||||
|
||||
class TagNormalizationTests(unittest.TestCase):
|
||||
def test_tag_order_does_not_change_token(self):
|
||||
first = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["b", "a"])
|
||||
second = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a", "b"])
|
||||
self.assertEqual(first, second)
|
||||
|
||||
def test_accepts_tuple_and_set_iterables(self):
|
||||
expected = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a", "b"])
|
||||
self.assertEqual(
|
||||
expected,
|
||||
coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ("a", "b")),
|
||||
)
|
||||
self.assertEqual(
|
||||
expected,
|
||||
coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, {"a", "b"}),
|
||||
)
|
||||
|
||||
|
||||
class InputSensitivityTests(unittest.TestCase):
|
||||
def test_pepper_changes_token(self):
|
||||
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
other = coreci_broker.derive_token("other-pepper", APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertNotEqual(base, other)
|
||||
|
||||
def test_app_uuid_changes_token(self):
|
||||
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
other = coreci_broker.derive_token(
|
||||
PEPPER, "00000000-0000-0000-0000-000000000000", USER_ID, AGENT_ID, ["a"]
|
||||
)
|
||||
self.assertNotEqual(base, other)
|
||||
|
||||
def test_user_id_changes_token(self):
|
||||
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
other = coreci_broker.derive_token(PEPPER, APP_UUID, "user-99", AGENT_ID, ["a"])
|
||||
self.assertNotEqual(base, other)
|
||||
|
||||
def test_agent_id_changes_token(self):
|
||||
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
other = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, "gitea", ["a"])
|
||||
self.assertNotEqual(base, other)
|
||||
|
||||
def test_tags_change_token(self):
|
||||
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
other = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a", "c"])
|
||||
self.assertNotEqual(base, other)
|
||||
|
||||
|
||||
class PurityTests(unittest.TestCase):
|
||||
def test_derivation_writes_nothing_to_disk(self):
|
||||
cwd = os.getcwd()
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
os.chdir(tmp)
|
||||
try:
|
||||
coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
|
||||
self.assertEqual([], os.listdir(tmp))
|
||||
finally:
|
||||
os.chdir(cwd)
|
||||
|
||||
def test_derivation_isolated_from_store_and_audit(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
db_path = os.path.join(tmp, "broker.db")
|
||||
audit_path = os.path.join(tmp, "broker.audit.log")
|
||||
store = coreci_broker.Store(db_path)
|
||||
audit = coreci_broker.Audit(audit_path)
|
||||
try:
|
||||
token = coreci_broker.derive_token(
|
||||
PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"]
|
||||
)
|
||||
rows = store.conn.execute(
|
||||
"SELECT * FROM agent_registrations"
|
||||
).fetchall()
|
||||
self.assertEqual([], rows)
|
||||
finally:
|
||||
store.close()
|
||||
audit.close()
|
||||
with open(audit_path, "r", encoding="utf-8") as handle:
|
||||
self.assertEqual([], handle.readlines())
|
||||
self.assertNotIn(token, open(db_path, "rb").read().decode("latin-1"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user