7 Commits

Author SHA1 Message Date
jonathan 6ecd7129f5 Merge pull request 'docs(verbiage): TL;DR prologues per comms-etiquette v1.0' (#2) from coo/verbiage-1 into main
Reviewed-on: #2
2026-10-19 23:09:52 +00:00
coreci-coo 8cbc76125c docs(verbiage): TL;DR prologues per comms-etiquette v1.0 (applies coreci/broker#1) 2026-10-19 21:57:55 +00:00
coreci d3251edc07 docs(milestone): clear checkpoint — v0.2 complete, next run starts a new milestone
---ci---
phase: 3
milestone: v0.2
status: complete
---/ci---
2026-10-11 00:32:44 +00:00
coreci 9b012b7606 docs(milestone): closeout — v0.2 requirements complete, roadmap complete
---ci---
phase: 3
milestone: v0.2
status: complete
requirements:
  covered: [REQ-21, REQ-22, REQ-23]
  partial: []
---/ci---
2026-10-11 00:32:44 +00:00
coreci da15bda6d2 docs(milestone): complete cli-packaging
---ci---
phase: 3
milestone: v0.2
status: complete
requirements:
  covered: [REQ-21, REQ-22, REQ-23]
  partial: []
---/ci---
2026-10-11 00:32:26 +00:00
coreci e650681755 docs(milestone): clear checkpoint — v0.1 complete, next run starts a new milestone
---ci---
phase: 5
milestone: v0.1
status: complete
---/ci---
2026-10-10 15:30:14 +00:00
coreci 7f7868f7cf docs(milestone): complete coreci-broker
---ci---
phase: 5
milestone: v0.1
status: complete
requirements:
  covered: [REQ-01, REQ-02, REQ-03, REQ-04, REQ-05, REQ-06, REQ-07, REQ-08, REQ-09, REQ-10, REQ-11, REQ-12, REQ-13, REQ-14, REQ-15, REQ-16, REQ-17, REQ-18, REQ-19, REQ-20]
  partial: []
---/ci---
2026-10-10 15:30:14 +00:00
66 changed files with 15512 additions and 0 deletions
+79
View File
@@ -0,0 +1,79 @@
# ARCHITECTURE — CoreCI Broker
## Overview
One Open WebUI workspace tool, implemented as a single Python file (`coreci_broker.py`), embedding a stdlib HTTP server (`ThreadingHTTPServer`) bound to `BROKER_BIND` (default `127.0.0.1:8765`). It serves a JSON API, a four-page admin web UI (static HTML/CSS/JS from `coreci_broker_ui/`), and a public static client module. SQLite (`broker.db`, mode 0600) is the source of truth; an append-only JSONL file (`broker.audit.log`, mode 0600) records every read and denied attempt. The broker is the trust root with no upstream auth; it reads the `creds-coreci-coo` KB through a pluggable adapter and never writes to it.
Two runtime modes share the same file: (a) Open WebUI workspace tool — a `Tools` class starts the server in a daemon thread at construction (`__init__`), behind a once-per-process idempotency guard (the OWUI loader has no activate() hook and re-execs modules on edit; paths resolve from env, never `__file__`); (b) standalone — `python3 coreci_broker.py` for development, testing, and direct operation.
## Components
| Name | Description | Boundaries | Depends on |
|------|-------------|-----------|------------|
| Config bootstrap | Reads/writes the broker `.env` (spec §5.1 contract + documented additions); first-boot generation of `BROKER_INSTANCE_PEPPER` and `BROKER_ADMIN_PASS` (printed to the startup log once, written to `.env` and SQLite) | Owns the `.env` file and the `broker_config` table; never prints secrets after first boot | SQLite store |
| SQLite store | Tables: `broker_config`, `app_registrations`, `user_allowlist`, `agent_registrations`, `admin_sessions` + indexes (spec §5.2); mode 0600 | Sole writer of `broker.db`; all SQL lives here | — |
| Audit writer/reader | Appends one JSON line per action (register/read/denied/login/logout/add_app/remove_app/add_user/remove_user); filtered query endpoint (since/until/user_id/uuid/action/limit) | Owns `broker.audit.log`; append-only; no rotation in v0.1 | — |
| Hasher | Dual-mode slow hash: bcrypt when importable, else stdlib `hashlib.scrypt`; stored hashes format-prefixed (`$2b$…` / `scrypt$…`) so verification is format-aware | The only component with an optional import | — |
| Token engine | Server-side derivation: `token = base64(sha256(pepper:UUID:user_id:agent_id:','.join(sorted(tags))))` | Pure function of (pepper, inputs); never stores plaintext tokens | Hasher, Config |
| HTTP server + router | stdlib threaded HTTP; JSON body/param handling; `{"error, message}` contract with 400/401/403/404/409/500; static file serving (UI + client module) | No business logic; routes to admin/tool planes | All handlers |
| Admin plane | Session auth (cookie `broker_session` + HTTP Basic); login/logout/whoami; app_registrations CRUD (add/revoke/reinstate); user_allowlist CRUD; audit query | Admin-only; audited | Store, Audit |
| Tool auth plane | `POST /api/broker/agents/register` (allowlist gates, idempotent re-derive, TTL, pepper rule); App-auth verification (`Authorization: App <UUID> <token>` — UUID-scoped slow-hash scan + in-memory verified-token cache); `GET /agents/<id>`; state GET/PUT with identity enforcement | Register is callable without a valid token (gates are the allowlists); the cache is never authoritative | Store, Token engine, Hasher |
| Secrets plane | `GET /api/broker/secrets/<path>`; scope check `path == "coreci/" + agent_id`; creds-source adapter (FileCredsSource default \| OwuiApiCredsSource) | The only reader of `creds-coreci-coo`; read-only | Tool auth plane, Creds adapter |
| Agent-state store | Backs `GET/PUT /api/broker/agents/<id>/state`; toggle `BROKER_AGENT_STATE_BACKEND=internal` (rows in `broker.db`) `\| owui` (real per-agent KBs via OWUI API) | Broker-owned; tool-facing; identity-checked | Tool auth plane |
| Static UI | Four pages (login, app registrations, user allowlist, audit) + dashboard + persistent nav; server-rendered HTML, no build step | Read-only against the admin API | Admin plane |
| Client module | `coreci_broker_client.py` — dependency-free (stdlib urllib) auth client: ensure-registered, state get/put, secret read with 401→re-register→retry-once; served unauthenticated at `/static/` | Contains no secrets; importable by any OWUI tool | Broker API (external) |
| Reference tools | `reference_tools/` — OWUI tool classes showcasing auth-first-then-use (e.g., proxmox reference: register → fetch `coreci/proxmox` → downstream call, safe dry-run default) | Templates for tool authors; not wired to live infra in tests | Client module |
## DataFlow
1. **Admin**: CEO logs in via `/ui/login` (or HTTP Basic) → session row in `admin_sessions` → manages allowlists via UI → every admin action audited.
2. **Registration**: tool calls `POST /api/broker/agents/register` with `{agent_id, app_registration_uuid, user_id, tags}` → broker checks `app_registrations` (active) and `user_allowlist` (active) → 403 on unknown/revoked → broker derives the token from the pepper + inputs → stores slow-hash in `agent_registrations`, sets TTL expiry → returns token (+ pepper when the row is created or the caller presented no valid token) → tool persists via `PUT /agents/<id>/state`.
3. **Secret read**: tool calls `GET /api/broker/secrets/coreci/<agent_id>` with `Authorization: App <UUID> <token>` → broker parses UUID, scans that tool's registration rows with the slow hash (in-memory cache after first verify) → 401 on failure → scope check `path == coreci/<agent_id>` → 403 on mismatch → creds adapter reads `creds-coreci-coo` → audit line (ok/denied/error) → JSON response.
4. **Expiry**: after `BROKER_TOKEN_TTL` the token verifies but the row is expired → 401 → tool re-registers (step 2) → new token overwrites `token_hash` → audit `register`.
## BuildOrder
1. Config bootstrap + SQLite store + audit writer (P1)
2. Hasher + token engine (P1)
3. HTTP server + router + error contract + OWUI/standalone mounts (P1)
4. Admin plane + audit query + static UI (P2)
5. Tool auth plane + agent-state store (P3)
6. Secrets plane + creds adapter + client module + reference tools (P4)
7. Acceptance matrix, security review, README (P5)
## Notes
- The in-memory verified-token cache maps `(uuid, token) -> registration row`; it is populated on first successful slow-hash verification and cleared on restart. SQLite remains authoritative; cache entries are re-validated against row status/expiry on every hit.
- Creds adapter contract: lookup by path string (`coreci/<agent_id>`) → JSON object. FileCredsSource reads a single JSON document mapping paths to secret objects (default for dev/tests, `BROKER_CREDS_FILE`). OwuiApiCredsSource calls the Open WebUI HTTP API with `BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY` and is exercised in production; its exact KB-file layout is resolved in P4 research.
- This file is the init-time snapshot; it is refined at phase boundaries per ci-files discipline.
## Distribution (v0.2 — cli-packaging)
v0.2 adds a distribution layer around the unchanged v0.1 broker: no broker-line changes (`run_standalone`, `boot`, `load_config`, `_parse_env_file` are consumed as-is). Three new components sit outside `coreci_broker.py` and are exercised by the same stdlib-unittest harness (in-process stub servers on ephemeral ports; subprocess boot tests against the built artifact).
### Components (v0.2)
| Name | Description | Boundaries | Depends on |
|------|-------------|-----------|------------|
| CLI artifact build (`scripts/build_cli.py` + `cli/__main__.py`) | Stages `coreci_broker.py` + `coreci_broker_client.py` + `coreci_broker_ui/` + the boot shim + a generated `_cli_version.py`, builds `coreci-broker.pyz` with stdlib `zipapp` (`--compress`, shebang `#!/usr/bin/env python3`), then a deterministic normalization pass (sorted entries, fixed timestamps) for byte-reproducible output; emits `dist/coreci-broker-vX.Y.Z.pyz` + sha256 | Produces the artifact only; never publishes; owns no state; the shim materializes bundled UI/static assets into `<state>/.coreci_broker_assets/<version>/{ui,static}` at boot (temp+`os.replace`, overwrite-per-boot) and sets `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` only when neither process env nor the user `.env` defines them; `--version` is answered from the generated `_cli_version.py` before any broker import | Broker module surface (`run_standalone`, `_parse_env_file`) — import-only, zero edits |
| Release-asset pipeline (`scripts/release.py` + `scripts/_env_secrets.py`) | Builds release notes (milestone summary, phases, verification, CLI install/usage, docker usage, artifact sha256), create-or-get the gitea release by tag, attaches `coreci-broker-vX.Y.Z.pyz` via `POST /api/v1/repos/{owner}/{repo}/releases/{id}/assets` (multipart field `attachment`, `Authorization: token <T>`); idempotent re-attach; bounded retry (3) → `release_pending` escalation on exhaustion — never blocks tag+merge | Owns the forge API conversation for release assets only; token resolved exclusively from `.env*` files via the file-reading loader (never shell env; assert non-empty before any call — empty token → gitea 404 masking, v1.8); does not build the artifact, does not push images | CLI artifact build (attaches its output); gitea API (`/api/v1/...`) |
| Daemonless registry publish client (`scripts/publish_image.py`) | Once per milestone: pull the pinned `python:3.11-slim` base from Docker Hub (anonymous token exchange → OCI index → resolve linux/amd64 child manifest → config + layer blobs), append one deterministic broker layer carrying `/app/coreci-broker.pyz` (tar+gzip, fixed metadata, `mtime=0`), build a new config blob (diff_id appended; Entrypoint `["python3","/app/coreci-broker.pyz"]`, Cmd `[]`, WorkingDir `/state`), and push `git.coreci.dev/coreci/broker:<milestone-release-tag>` + `latest` to the gitea container registry (Basic-auth token exchange at `/v2/token`, `Bearer` JWT on API calls, HEAD-skip + monolithic blob uploads, OCI image manifest PUT; explicit `Location`-following — stdlib urllib does not follow 307/308 on PUT) | No container runtime (D-019); base layer digests reused verbatim, base blobs re-uploaded (no cross-registry mount); records resolved base digests as provenance; failure follows bounded-retry → `release_pending` escalation — never blocks the milestone merge | CLI artifact build (ships its output in the layer); Docker Hub + gitea registry HTTP APIs (OCI dist-spec v2) |
| Dockerfile (documentation) | Self-contained docker-build path for users WITH docker: same base, in-image `zipapp` build of the same sources + shim, same entrypoint/`/state` anchoring | Documentation + alternative build path only; the daemonless client is the executable source of truth for what the pipeline publishes; equivalence with the published image is required, byte-identity is not | CLI artifact build (same staging content, same shim) |
### DataFlow (v0.2 distribution)
1. **CLI boot (packaged)**: `./coreci-broker.pyz` → shim resolves the state dir (`BROKER_ENV_PATH`-anchored; CWD-relative defaults preserved) → materializes UI/static assets into the versioned assets dir → sets `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` (user env/`.env` overrides win) → `coreci_broker.run_standalone()` → first-boot `.env` generation, admin pass printed once, `broker.db`/`broker.audit.log` 0600 in the state dir, UI served at `/ui/`, client at `/static/coreci_broker_client.py` — behavior byte-identical to the unpackaged standalone mode.
2. **Release attachment (per phase ship from the CLI-building phase)**: `scripts/build_cli.py` → reproducible `dist/coreci-broker-vX.Y.Z.pyz` (+sha256) → `scripts/release.py` → gitea create-or-get release by tag → multipart attach → notes body (CLI install/usage + docker usage + digest). Bounded retry; `release_pending` escalation on exhaustion; tag+merge is the completion gate.
3. **Image publish (once per milestone, at the final ship)**: `scripts/publish_image.py` → anonymous Docker Hub pull (`python:3.11-slim` OCI index → amd64 child + blobs) → deterministic broker layer (contains the pyz) + new config → gitea registry auth (`/v2/token`, Basic `<user>:<GITEA_TOKEN>`) → blob upload (HEAD-skip, monolithic POST, explicit Location-following) → manifest PUT `<milestone-release-tag>` then `latest` → provenance digests recorded. Bounded retry; never blocks the milestone merge.
### BuildOrder (v0.2 — appended to the v0.1 order above)
8. CLI build pipeline + boot shim + packaging/boot-parity tests (P1 — REQ-21)
9. Release-asset uploader + daemonless publish client + Dockerfile + stub tests (P2 — REQ-22/REQ-23)
10. Cumulative acceptance (v0.1 suite through the packaged artifact), security review, milestone ship: attach pyz to `v0.2.3`, publish image, merge (P3)
### Notes (v0.2)
- Distribution components hold no secrets: the archive bundles only broker + UI + client code; the image ships no state; forge tokens live in `.env*` files loaded only by `scripts/_env_secrets.py` (blocked from shell env by `config.security.bash_allowlist`).
- All three pipeline scripts take injectable base URLs (stub-testable); live-forge interactions are read-only in tests and attempted-only at ship per the bounded-retry contract.
- Verified environment facts backing this section live in RESEARCH.md v0.2 (gitea 1.26.1 API shapes, OCI index resolution for `python:3.11-slim`, gitea `/v2/token` Basic-auth exchange, zipapp normalization, urllib 307/308 behavior).
+254
View File
@@ -0,0 +1,254 @@
# CIAgent Grill Report
## Run: 2026-10-10 (mode: interactive, focus: v0.2 plan red-team — cli-packaging)
**Supersedes the v0.1 grill in this file** (v0.1 run of 2026-10-09; its full text and
verdicts G-001..G-008 live in git history at `7f7868f` and remain binding where the plan
depends on them: G-005's owner-run post-ship OWUI smoke checklist and G-008's 90-day
adoption criterion carry forward in ROADMAP.md §Post-Milestone Notes; G-007's
forge-repo-creation test was executed at v0.1's P0 ship and is closed). New verdicts
number G-009 onward.
### Verdict (summary): GO-WITH-CONDITIONS (confidence: 0.82)
The plan rests on genuinely strong research — live-verified forge/registry wire shapes, a
working zipapp prototype booted against the real broker sources, zero-broker-lines
verified against the actual file (1390/1400). Red-team findings are concrete and
fixable by task/ship-step edits: one **unexecutable mechanism** (the v0.2.1 attachment
bridge cites a harness path RESEARCH itself proved absent from this environment),
one **ship-sequence inversion** (P3-05 attaches the release before the tag exists —
gitea would mint `v0.2.3` at the wrong commit), one **false-green determinism gap**
(the normalize pass "preserves" umask-derived zip attributes, so must-have 4 passes
while the "clean checkout" exit criterion silently fails cross-machine), and a set of
test-realism holes (subprocess harness with no timeout discipline, missing creds env
in the packaged secret-read, no live read-only validation before the one live publish).
Three forks were resolved interactively with the owner this session (G-011, G-013,
G-014); everything else is bound on evidence below.
---
## Surface 1 — Determinism claims (must-have 4; P1-01)
**Forcing question**: Is two-build sha256 equality actually achievable and testable, and
what does the normalize pass fix vs. miss?
- **Evidence** (stdlib source, read this session): `/usr/lib/python3.11/zipapp.py`
builds via `for child in source.rglob('*'): z.write(child, arcname.as_posix())` —
entry order = scandir order, entry mtime = source mtime (`ZipInfo.from_file`), and
`external_attr` = the staging file's stat mode `<< 16`. PLAN P1-01's normalize pass
fixes order (sorted), `date_time=(1980,1,1,0,0,0)`, `ZIP_DEFLATED` — but specifies
"**preserved** `external_attr`". Preserved from what zipapp wrote, i.e. derived from
staging file modes — which are `0666 & ~umask` if staged via plain copy, or
checkout-mode if via `copy2`. Umask 0022 vs 0002 across machines/clones → different
`external_attr` → different sha256, **while must-have 4 (two consecutive builds, same
machine) still passes**. A silent false green against the phase exit criterion
("builds reproducibly from a clean checkout").
- **gzip is not in the pyz path** (zipfile DEFLATE only). Image-layer gzip is
deterministic on-platform: `gzip.compress(..., mtime=0)` routes to
`zlib.compress(wbits=31)` on 3.11 (verified in gzip.py source) — fixed header, fixed
OS byte, stable per zlib build. Scope claim: byte-identity is **same-environment**
(same machine, same zlib); cross-zlib-version identity is neither claimed nor
required by must-have 4.
- **Answer**: achievable and testable as scoped — provided the normalize pass also
fixes what it currently preserves. → **G-009**.
## Surface 2 — Boot parity via subprocess (P1-02)
**Forcing question**: What makes the subprocess acceptance flaky, and what's missing?
- **Evidence**: zero v0.1 tests use `subprocess`/`Popen` (grep over `tests/`: no hits) —
P1-02 introduces the repo's first subprocess-boot harness. The plan's env enumeration
(`BROKER_BIND`, `BROKER_ENV_PATH/DB/AUDIT`) omits `BROKER_CREDS_FILE`/
`BROKER_CREDS_SOURCE` while requiring "secret-read reuse the v0.1 acceptance request
patterns" — the v0.1 precedent (`tests/test_acceptance_full.py:76-77`) sets both;
`FileCredsSource` on a missing file raises `_SecretNotFound` → 404
(`coreci_broker.py:440-441`), so the packaged secret-read would fail. `Popen(["python3",
...], env={minimal})` leaves the child without `PATH` (works today via execvp's
confstr fallback — by accident). No timeout discipline on "read stdout lines until
`coreci-broker listening on`" (RESEARCH §4.3) — a failed boot never prints the line and
hangs the suite. "Pass printed exactly once" is asserted against a *still-running*
server; shutdown-path output is unread unless drained after terminate. And every
planned boot pins absolute state paths — the owner's literal happy path (download →
`python3 *.pyz` in some directory → `.env`/db/audit materialize in the CWD) is never
tested.
- **Answer**: harden the harness — the plan's claims are all testable once the harness
is honest about them. → **G-010**.
## Surface 3 — One-shot lead upload bridge (P1-03)
**Forcing question**: Is the v0.2.1 ad-hoc attachment a discipline violation, and is it
even executable as specified?
- **Evidence**: PLAN P1-03 attaches via "the same harness mechanism v0.1 used for its
`broker-vX.Y.Z.tar.gz` assets" — but RESEARCH §2.2 (the plan's own research base)
found that mechanism is the ship harness's `coreci build/package/release upload` CLI,
**"not installed in this dev environment"**; v0.1's archives were produced
container-side, ad-hoc, with no repo-owned script. The bridge as written is therefore
unexecutable via the cited mechanism, and its realistic substitute is an untested
hand-rolled live-forge write (multipart encoding, token handling, idempotency) at
first live use — performed by lead-developer, the one persona barred from code
writes, using token discipline that `scripts/_env_secrets.py` (which does not exist
until P2) is supposed to enforce. This is precisely the practice REQ-22 formalizes
away.
- **Owner decision (this session)**: restructure — release automation moves into P1;
the bridge is deleted. → **G-011**. The hard gate question dissolves: v0.2.1 ships
script-driven; there is no bridge to become permanent.
## Surface 4 — Daemonless OCI publish (P2-02, P3-05)
**Forcing question**: Does the milestone ship sequence hold up when publish fails after
the tag exists — and does it even run in the right order?
- **Evidence — sequence inversion**: P3-05 as written runs `release.py` attach →
publish image → squash-merge → `main` → tag `v0.2.3` → push. But `POST /api/v1/repos/
{o}/{r}/releases` with a missing tag creates the tag at `target_commitish`
(default-branch head) — gitea's documented CreateReleaseOption semantics (schema in
RESEARCH §2.1). A create-first flow pre-merge would mint `v0.2.3` pinned to
**pre-merge main**, then collide with (or silently diverge from) the harness's own tag
push. The plan's own P2-04 sequences correctly (tag → merge → push → release) — P3-05
alone inverts it.
- **Evidence — wire discipline**: bounded retry(3) is specified but no per-request
timeout anywhere; a hung 29.8 MB monolithic POST through a throttled gitea turns
"bounded retry" into an unbounded hang. The 307/308 stub test proves only
Location-following, and gitea's `Location` may be **relative** (RESEARCH §3.3) — the
harder case is untested. Upstream `3.11-slim` drift is handled (resolve-at-publish +
`--base-manifest` + provenance digests; RESEARCH §3.7) and the anonymous-200 token
trap fails loudly per spec.
- **Evidence — failure story**: "publish never blocks merge" is honored, but when
publish fails after `v0.2.3` exists, the plan records only `release_pending` — no
named post-milestone re-run/closure step. The ship contract makes the escalation
auto-resolution; somebody must own the retry.
- **Answer**: reorder P3-05, add timeouts + relative-Location coverage, and make the
escalation carry its own closure runbook. → **G-012**, **G-019**, **G-020**.
## Surface 5 — Scope discipline (P2-03, REQ-22/23)
**Forcing question**: Is anything decorative, and is anything REQ-22/23 implies missing?
- **Evidence — decorative risk**: P2-03's verify is presence-only ("`Dockerfile` +
README sections present") and the dev environment cannot build the Dockerfile (no
runtime) — a syntactically or semantically broken Dockerfile passes the gate. The
"equivalence mapping" header is documentation with no enforcement.
- **Evidence — missing**: no `.dockerignore` anywhere in the plan — `docker build .`
ships the whole repo (`.git/`, any local `.env*`, `data/`, `*.db`, `dist/`,
`__pycache__/`, `.ciagent/`) as build context, adjacent to a security review that
asserts "no secrets in any image layer". REQ-22's notes model is covered (§2.3, stub
asserts sections present); the sha256's *source* is unspecified (hand-pasted vs.
computed); `latest` semantics are coherent (once per milestone at final ship —
v0.2.3 is the first image ever, D-021); the state-dir volume contract is documented
(`WORKDIR /state`, README docker run).
- **Answer**: make the Dockerfile contract testable, add `.dockerignore`, pin digest
provenance. → **G-015**, **G-016**, **G-017**.
## Surface 6 — Frozen broker / shim dependency risk (P1-01)
**Forcing question**: Does the shim need broker cooperation, and what is the coupling
risk?
- **Evidence**: RESEARCH §1.2's prototype booted from the real sources with zero broker
cooperation — re-verified against the file this session: `run_standalone`
(`coreci_broker.py:1369`), `_parse_env_file` (:154), `boot` (:598), `load_config`
(:240) are module-level; `load_config` precedence (process env → `.env` → defaults,
:246-257) makes the shim's pre-set `os.environ` approach contract-identical. The
broker's `.env` resolution is CWD-relative by default (`ENV_PATH_DEFAULT=".env"`,
`ENV_DEFAULTS` :53-66) — packaged boots with unset `BROKER_ENV_PATH` anchor state to
the CWD, which is exactly the documented happy path and is currently untested (folded
into G-010). Residual risk: the shim calls `_parse_env_file` — an underscore-private
symbol. Within v0.2 the broker is frozen so the risk is nil; across future milestones
a rename breaks the shim with a loud ImportError (acceptable), but only if somebody
knows to look.
- **Answer**: no cooperation needed (verified twice); the private-API dependency must
be documented and review-tracked, and the default-anchoring boot must be tested. →
**G-018** (anchor case bound under G-010).
## Surface 7 — Test realism: stub fidelity (P2-01/P2-02 tests)
**Forcing question**: What would a stub MISS that the real forge does — and is the
as-built client ever validated against the real forge before its one live write?
- **Evidence**: the stub coverage is unusually good (anonymous-200 trap, index→amd64
resolution with Accept check, HEAD-skip, monolithic+two-step uploads, 307-following,
digest and diff_id assertions, both tags; multipart field/`?name=` byte assertions,
idempotent re-attach, 500×3 marker, token discipline). Gaps: fixtures are frozen at
research-time shapes (2026-10-10) while upstream index and forge auth can move
mid-milestone; the 307 test doesn't pin the relative-`Location` case (bound in
G-019); gitea's 4 MB attachment cap is out-of-band (pyz ≈30 KB — immaterial, noted);
and **no step anywhere validates the as-built scripts against the real
`git.coreci.dev` in read-only mode** — the first live run of `publish_image.py` is
the P3-05 milestone publish itself, the one shot per milestone.
- **Owner decision (this session)**: add an env-gated, read-only live smoke as a
P3-05 pre-ship verify step. → **G-013**.
## Surface 8 — Wave ordering & effort (P2 packing)
**Forcing question**: Is P2 overloaded, and are there hidden dependencies that break
the wave order?
- **Evidence**: original P2 packs `_env_secrets` + `release.py` + `publish_image.py` +
`Dockerfile` + README + 2 test files into 3 waves — the overload is real, and the
bridge finding (Surface 3) is the reason it dissolves: release automation belongs in
P1 (G-011), leaving P2 = publish client → Dockerfile/docs → ship. Hidden
dependencies check out: `publish_image.py` imports `_env_secrets` (lands earlier in
P1 now — order preserved transitively); the Dockerfile task consumes P1-01's staging
allowlist (G-016 formalizes the contract as a test). `publish_image.py` remains the
plan's biggest single commit even after the restructure.
- **Owner decision (this session)**: keep it one vertical task — the pull/push seam is
artificial, the stub fixtures are shared end-to-end, RESEARCH §3.4 is near-pseudocode,
and a split would not isolate the actual risk (first live write — covered by G-013).
→ **G-014**.
---
## Binding Decisions
| ID | Surface | Decision | Binds | Confidence |
|----|---------|----------|-------|------------|
| G-009 | 1 | Normalize pass must SET constant `external_attr` (files `0o100644<<16`, dir entries `0o40755<<16`), not "preserve" staging-derived modes; staging is an explicit allowlist (`coreci_broker.py`, `coreci_broker_client.py`, `coreci_broker_ui/**`, shim staged as root `__main__.py`, generated `_cli_version.py`) — never a repo rglob, `__pycache__`/`dist/` excluded by construction. Byte-identity scope = same environment (same zlib/machine); cross-zlib identity is not claimed. | P1-01 task edit | 0.85 |
| G-010 | 2 | P1-02 harness hardening: subprocess boots use `sys.executable` (absolute); child env adds `BROKER_CREDS_SOURCE=file` + `BROKER_CREDS_FILE=<tmp fixture>` (v0.1 precedent test_acceptance_full.py:76-77); deadline-bounded wait for the listening line (~30s → fail with captured stdout+stderr, terminate child); pass-once/silent-second-boot assertions completed by draining stdout after terminate; ADD a default-anchoring boot case (`cwd=<tmp>`, no state overrides beyond `BROKER_BIND`) asserting `.env`/`broker.db`/`broker.audit.log` in the CWD — the owner's literal happy path. | P1-02 task edit | 0.85 |
| G-011 | 3 | **Restructure (owner, this session)**: move `scripts/_env_secrets.py` + `scripts/release.py` + `tests/test_release_upload.py` into Phase 1 as new wave 3 (P1-03 · release-asset automation); old P1-03 ship renumbers P1-04 and ships v0.2.1 script-driven (`build_cli.py v0.2.1` → `release.py v0.2.1 <pyz>`); the one-shot lead-upload note is DELETED (its cited mechanism is absent from this environment — RESEARCH §2.2). P2 renumbers: publish_image.py → P2-01 (wave 1), Dockerfile+docs → P2-02 (wave 2), ship → P2-03 (wave 3). Four-test-file count unchanged. | PLAN edit (before P1-01 executes) | 0.90 |
| G-012 | 4 | P3-05 sequence corrected: squash-merge → `main` → tag `v0.2.3` → push → THEN `release.py` attach (GET-by-tag finds the real tag; no pre-merge release create) → publish image → closeout. Mirrors P2's own correct ship order; eliminates the gitea create-tag-at-target_commitish hazard. | P3-05 ship-step edit | 0.80 |
| G-013 | 7 | **Env-gated live read-only smoke (owner, this session)**: pre-publish P3-05 verify step — gitea `GET /v2/` 401 challenge shape, `GET .../releases/tags/{tag}`, `GET /v2/token` Basic → non-empty JWT, Docker Hub anonymous token + `3.11-slim` index→amd64 resolve. Implemented as a `--check`/verify mode on the scripts or `scripts/verify_endpoints.py`; GET-only, never in the default unittest run; tokens only via `_env_secrets`. | P3-05 verify + task edit | 0.85 |
| G-014 | 8 | **publish_image.py stays one vertical task (owner, this session)**: shared stub fixtures, RESEARCH §3.4 near-pseudocode; a split would not isolate the real risk (first live write — G-013 covers it). | PLAN shape (no edit) | 0.75 |
| G-015 | 5 | `.dockerignore` added to the Dockerfile task: excludes `.git`, `.env*`, `data/`, `*.db`, `*.log`, `dist/`, `__pycache__`, `tests/`, `.ciagent/`. Build-context hygiene; P3-02 scope gains "no secrets in the build context" alongside "none in any layer". | P2-02 task edit | 0.85 |
| G-016 | 5 | Dockerfile equivalence made testable: the Dockerfile task adds a consistency test (into `tests/test_build_cli.py`) asserting COPY source set == `build_cli.py` staging allowlist, identical zipapp flags (`--python "/usr/bin/env python3" --compress`), identical `_cli_version.py` generation format, `ENTRYPOINT`/`WORKDIR` == publish_image's `config.Entrypoint`/`config.WorkingDir`. Presence-only verification no longer gates a possibly-broken Dockerfile. | P2-02 task edit | 0.80 |
| G-017 | 5 | Digest discipline: the sha256 in release notes is computed by `release.py` from the artifact file itself — never a CLI argument, never hand-pasted. Digest/asset mismatch becomes structurally impossible. | P1-03 task edit | 0.85 |
| G-018 | 6 | Shim private-API dependency tracked: `cli/__main__.py` header documents the coupling to `coreci_broker._parse_env_file` + `run_standalone` ("private-API dependency — any future broker refactor must re-run the v0.2 packaged acceptance"); P3-02 review scope lists it. Loud-ImportError breakage accepted. | P1-01 task edit + P3-02 scope | 0.75 |
| G-019 | 4 | Publish-client wire hardening: explicit per-request timeouts on every registry call (generous on blob uploads, ~120s); blob-level bounded retry with HEAD-skip between attempts (a timed-out 29.8 MB POST resumes cheap); the 307 stub test uses a RELATIVE `Location` at least once (gitea may answer relative — RESEARCH §3.3; absolute is the easy case). | P2-01 task edit | 0.80 |
| G-020 | 4 | Publish-failure closure is a named step, not an open escalation: the `release_pending` entry records the exact re-run command (`python3 scripts/publish_image.py` on `main` @ `v0.2.3`; `--base-manifest` optional pin) and its close condition (both tags visible in gitea packages). "Never blocks merge" unchanged. | P3-05 ship-step edit | 0.80 |
| G-021 | 4 | The `v0.2.0` (phase-0) release carries NO `.pyz` and its notes must not promise one — D-020 starts at `v0.2.1` (build tooling first exists in P1). Prevents a D-020 misread at the phase-0 ship immediately following this grill. | Phase-0 ship note | 0.85 |
## Escalations
- (none open — the three material forks were resolved interactively with the owner this
session: G-011 restructure, G-013 live smoke, G-014 task packing. G-001..G-008 from
v0.1 remain binding per git history; G-005/G-008 carry forward in ROADMAP post-milestone
notes unchanged.)
## Promotion note (explicit, not performed by the grill)
G-011 shifts REQ-22's attach automation into Phase 1; ROADMAP.md's Phase 2 description
("release-asset automation" wording) and Phase 1's requirements line need a one-line
touch-up when the PLAN restructure is promoted — by the orchestrator/owner via the
normal plan-revision path, not by this file. Nothing else in PROJECT/REQUIREMENTS/ROADMAP
is affected by these verdicts.
## Overall Verdict
**GO-WITH-CONDITIONS** — confidence 0.82.
The plan is research-grounded to an unusual degree (live wire shapes, working prototype,
zero-broker-lines verified against the file), and every finding is actionable by task or
ship-step edit. It is GO only with all of the following applied:
1. **G-011 restructure applied to PLAN.md before P1-01 executes** — the bridge is
unexecutable as written; v0.2.1 must ship script-driven.
2. **G-009 + G-010 applied to P1-01/P1-02** — determinism must fix (not preserve)
entry attributes; the subprocess harness needs interpreter-path, creds-env, timeout,
drain, and default-anchoring discipline.
3. **G-012 + G-020 applied before P3-05** — attach-after-tag ordering and the
publish-failure closure runbook are ship-contract corrections, not optional polish.
4. **G-013 live smoke wired as the P3-05 pre-ship verify step** — the one live publish
must not also be the first live wire contact of the as-built client.
5. **G-015/016/017/018/019 applied with their tasks** — context hygiene, testable
Dockerfile contract, digest provenance, tracked private-API coupling, wire timeouts.
6. **G-021 noted at the phase-0 ship** — v0.2.0 carries no `.pyz`.
+75
View File
@@ -0,0 +1,75 @@
# PERSONAS — CoreCI Broker
Execution-stage roster (2026-10-10, milestone v0.2 cli-packaging). Pure stdlib Python 3.11 project: no package.json, no JS framework, no pip, no npm, no container runtime in the dev environment — frameworks are stdlib modules; tests are stdlib unittest against in-process stub HTTP servers and subprocess boots. The v0.1 broker is FROZEN this milestone: `coreci_broker.py` stands at 1390/1400 lines (REQ-17/AC14) and v0.2 adds zero broker lines (verified, RESEARCH §4.2), so no persona holds `coreci_broker.py` — or the v0.1 UI/client/reference sources — as writable territory; a P0 defect there is a lead-developer-dispatched exception only. All v0.2 work is new files: `cli/` boot shim, `scripts/` build+release+publish tooling, `Dockerfile`, four new test files, additive README sections, `.gitignore` additions. The custom release-engineer persona owns the whole distribution pipeline; backend-engineer — v0.1's primary persona — is deactivated with the P0-dispatch exception documented below. Execution is sequential (parallelization off), so phase-specific personas activate only at their phase boundaries per this roster.
---
name: release-engineer
domain: distribution
active: true
phase_specific: false
reason: Primary execution persona — v0.2 is distribution engineering on a frozen broker, a domain distinct from v0.1's broker application work; hence a custom persona rather than a fold into backend-engineer, whose identity, constraints, and territory are the broker's runtime contract — a poor fit for build/release/publish tooling and a structural guarantee that the frozen file is owned by nobody. Owns the full pipeline — P1 zipapp build + materialization boot shim (REQ-21), P2 gitea release-asset automation + daemonless OCI registry publish + Dockerfile (REQ-22/23), P3 ship — and every security-critical surface this milestone: secrets discipline (file-only token loader), artifact integrity (no secrets in the .pyz or image layers, reproducible digests), registry auth flows.
frameworks: [python3.11, zipapp, zipfile, tarfile, gzip, hashlib.sha256, urllib.request, json, subprocess, http.server, unittest]
constraints: ["coreci_broker.py is FROZEN — zero lines added (1390/1400, REQ-17/AC14); broker sources (coreci_broker.py, coreci_broker_client.py, coreci_broker_ui/**, reference_tools/**) are read-only staging inputs to the build, never edited; a P0 defect there routes through lead-developer dispatch", "stdlib-only build path — python3 -m zipapp --compress; no pip, no npm, no wheel/PyInstaller, no container runtime (D-009/D-019)", "secrets discipline — GITEA_TOKEN resolves ONLY via scripts/_env_secrets.py reading .env/.env.secrets/.env.* files, never os.environ (blocked_env_vars: GITEA_TOKEN); assert non-empty before the first API call (v1.8 lesson: empty token → gitea 404 masks auth failure); no secrets ever enter the .pyz or an image layer", "digest determinism — zip entries sorted with date_time (1980,1,1,0,0,0) via the normalize post-pass; tar entries sorted with fixed TarInfo metadata; gzip.compress(mtime=0); json.dumps(sort_keys=True, separators=(',',':')); two-build sha256 equality gates the artifact (REQ-21)", "OCI end-to-end — manifest application/vnd.oci.image.manifest.v1+json, config application/vnd.oci.image.config.v1+json, layers application/vnd.oci.image.layer.v1.tar+gzip; base layer descriptors reused verbatim; never push an index; urllib does NOT follow 307/308 on PUT/POST — implement explicit Location-following; gitea's anonymous-200 /v2/token answer must fail loudly; HEAD-before-upload makes re-runs idempotent", "bounded retry (3 attempts) on release attach and publish; on exhaustion exit non-zero with a machine-readable marker → ship harness records release_pending escalation; never blocks tag+merge", "stub-testability — scripts take base URLs as parameters (no hardcoded hosts); stubs bind ('127.0.0.1', 0) per the v0.1 _StubOwuiHandler precedent; no live-forge writes in tests; v0.1 test files are frozen reference — v0.2 adds test_cli_packaging.py, test_build_cli.py, test_release_upload.py, test_publish_image.py only, and the cumulative suite stays green", "dist/ is build output — gitignored (dist/, *.pyz), never committed; README.md edits are additive sections (CLI install/usage, docker usage) with v0.1 content intact"]
territory: [cli/**, scripts/**, tests/**, Dockerfile, README.md, .gitignore]
subagent_type: backend-engineer
---
---
name: security-engineer
domain: security
active: true
phase_specific: true
reason: Review-only pass activated for P3 (milestone security review — artifact integrity, no secrets in the archive/image, token discipline in upload paths, registry auth flows, boot parity of materialized assets); inactive in P1/P2 unless the lead escalates a finding — same posture as v0.1.
frameworks: [python3.11, unittest, STRIDE, CWE, oci-dist-spec]
constraints: ["review-only — every v0.2 file stays owned by release-engineer; findings route through lead-developer, never direct edits; coreci_broker.py is in review scope for zero-delta verification (AC14) only and is writable by nobody this milestone", "findings are STRIDE-classified and CWE-first (cite CWE ids)", "focus — secrets discipline (file-only token path via scripts/_env_secrets.py, never shell env, non-empty assert before first use), artifact integrity (no secrets in the .pyz or any image layer; sha256 provenance recorded in release notes), registry auth flows (gitea anonymous-200 /v2/token quirk, Basic username-ignored shape, 307/308 handling), attachment idempotency and bounded-retry escalation paths"]
territory: [coreci_broker.py, cli/**, scripts/**, Dockerfile, tests/**]
subagent_type: ci-security-auditor
---
---
name: lead-developer
domain: coordination
active: true
phase_specific: false
reason: Active across all execution phases (P1–P3): decomposes plans into vertical slices along the build → attach → publish pipeline, arbitrates the frozen-broker rule (no persona writes coreci_broker.py — P0 fixes are lead-dispatched exceptions), routes security findings, and sequences tasks across phase boundaries. No direct code writes.
frameworks: [none]
constraints: ["no direct modifications to code files — coordination and conflict resolution only", "sole gate for frozen-broker exceptions — any P0 fix to coreci_broker.py (or the frozen UI/client/reference sources) is a lead-dispatched, defect-scoped task, never a persona territory claim", "arbitration — release-engineer owns every v0.2 file in cross-persona conflicts and defines the artifact/registry/release contracts; security findings flow lead → release-engineer; battle-tested defaults over novel approaches", "enforce — zero broker-line delta (AC14), secrets only via the .env* file loader, no secrets in artifacts, bounded-retry release_pending escalations never block tag+merge"]
territory: [.ciagent/PLAN.md, .ciagent/*.md]
subagent_type: lead-developer
---
---
name: backend-engineer
domain: backend
active: false
phase_specific: false
reason: Deactivated — the broker is frozen in v0.2; RESEARCH §4.2 verified run_standalone/boot/load_config/_parse_env_file are importable as-is, 1390/1400 lines leaves 10 lines of headroom, and zero-change is a hard constraint. All v0.2 engineering is distribution tooling owned by release-engineer. Reactivates only as a lead-developer-dispatched, defect-scoped P0 fix to coreci_broker.py — never via its own territory.
frameworks: [python3.11, http.server, sqlite3, hashlib.scrypt, unittest]
constraints: ["stays inactive for all planned v0.2 work; any broker change is a lead-developer-dispatched exception scoped to the P0 defect", "if dispatched — the 1400-line AC14 cap and every v0.1 runtime constraint (REQ-14..REQ-18, {error, message} JSON contract, 0600 modes, never write to creds-coreci-coo) remain binding"]
territory: []
subagent_type: backend-engineer
---
---
name: frontend-engineer
domain: frontend
active: false
phase_specific: false
reason: Deactivated — no UI work in v0.2; REQ-21..23 touch no page. The v0.1 UI (coreci_broker_ui/) is complete, frozen, and consumed as a read-only staging input by the zipapp build, then materialized verbatim at boot; any UI defect surfaced through the packaged path is a lead-dispatched exception.
frameworks: [html, css, vanilla-js]
constraints: ["stays inactive for the whole milestone; coreci_broker_ui/** is a read-only build input owned by no persona"]
territory: []
subagent_type: frontend-engineer
---
---
name: data-engineer
domain: data
active: false
phase_specific: false
reason: Deactivated (carried from v0.1) — no database tier beyond sqlite3 embedded in the frozen broker; no schema, pipeline, ETL, or migration work in v0.2. Image layers and release assets are build outputs owned by release-engineer, not data surfaces.
frameworks: [sqlite3]
constraints: ["stays inactive for the whole milestone; any residual data work folds into release-engineer or a lead-dispatched broker fix"]
territory: []
subagent_type: backend-engineer
---
+101
View File
@@ -0,0 +1,101 @@
---
must_haves:
- "python3 -m unittest discover -s tests passes (cumulative — the v0.1 suite plus the v0.2 test files: test_build_cli, test_cli_packaging, test_release_upload, test_publish_image, test_dockerfile_consistency, test_acceptance_v02)"
- "python3 -m py_compile cli/__main__.py scripts/build_cli.py scripts/release.py scripts/publish_image.py scripts/_env_secrets.py"
- "broker line count unchanged: coreci_broker.py stays at its v0.1.5 count of 1390 lines (AC14, D-016..D-018 cap 1400) — zero broker edits"
- "two consecutive builds via scripts/build_cli.py produce byte-identical artifacts (sha256 equal) — same-environment scope (G-009)"
- "the built .pyz boots with v0.1 acceptance parity (admin pass printed exactly once, broker.db + broker.audit.log 0600, UI + client served, silent second boot)"
- "stub-registry publish roundtrip green (scripts/publish_image.py vs a local stub, including a relative-Location 307 case, G-019) and stub release upload green (scripts/release.py vs a local stub)"
---
# PLAN — CoreCI Broker (Milestone v0.2 — cli-packaging)
Execution conventions: sequential waves (within a wave, tasks run in listed order); each task = one subagent commit `feat(P##): …` carrying a `---ci---` block (phase, milestone: v0.2, status: execute, persona, task, requirements.covered). Tests ship with their task — never deferred. Personas per PERSONAS.md: release-engineer (ALL v0.2 code — `cli/**`, `scripts/**`, the new test files, `Dockerfile`, `.dockerignore`, additive README, `.gitignore`), security-engineer (P3 only, review-only — findings route via lead-developer, never direct edits), lead-developer (coordination + ship steps, no code writes; sole gate for frozen-broker exceptions). The broker is FROZEN this milestone (1390/1400, REQ-17/AC14): no task edits `coreci_broker.py`, `coreci_broker_ui/`, `coreci_broker_client.py`, or `reference_tools/` — a P0 defect there is a lead-developer-dispatched exception only. Verify commands: `python3 -m py_compile <files>`; `python3 -m unittest discover -s tests`; broker zero-delta via the AC14 line-count test. Test harness (v0.1 §6 conventions extended, RESEARCH §4.3): `ThreadingHTTPServer` stubs on `("127.0.0.1", 0)` + `tempfile.TemporaryDirectory()` state + subprocess boots of the built artifact driven with stdlib `urllib`; scripts take base URLs as parameters (stub-testable); NEVER live-forge writes in tests (the only live-forge contact is the G-013 smoke gate + the ships themselves). Secrets resolve only via `scripts/_env_secrets.py` reading `.env`/`.env.secrets`/`.env.*` files — never shell env (`GITEA_TOKEN` is in `blocked_env_vars`); non-empty asserted before the first API call. `dist/` + `*.pyz` are gitignored — build artifacts are never committed. Release attach + image publish failures follow bounded retry (3) → `release_pending` escalation — never blocking tag+merge. Ships: phase 0 → `v0.2.0`, P1 → `v0.2.1`, P2 → `v0.2.2`, P3 → `v0.2.3` (final phase patch = milestone release, D-008 convention). **G-021: the `v0.2.0` phase-0 release carries NO `.pyz` and its notes must not promise one — D-020 starts at `v0.2.1`.** Release-tooling split (G-011): the sanctioned secrets loader + release attacher land in P1 (Wave 3) so every release from `v0.2.1` is script-attached; P2 owns the image publish client + Dockerfile.
## User-Facing Surface
- **Downloadable CLI artifact** — `coreci-broker-vX.Y.Z.pyz` on the gitea release page (every release from `v0.2.1`, D-020), alongside the existing source archive; its sha256 recorded in the release notes (computed by `scripts/release.py` from the artifact file — G-017).
- **CLI boot** — `python3 coreci-broker-vX.Y.Z.pyz` or `./coreci-broker-vX.Y.Z.pyz` after `chmod +x` (shebang `#!/usr/bin/env python3`); boots the standalone broker — full admin UI at `/ui/`, client module at `/static/coreci_broker_client.py`, bound address per `BROKER_BIND` (default `127.0.0.1:8765`).
- **`--version`** — prints the release version, exits 0.
- **Docker image** — `docker pull git.coreci.dev/coreci/broker:<tag>` (+ `latest`); `docker run` with a volume for state.
- **Release notes** — on the release page: milestone summary, phases, verification, CLI install + usage, docker usage.
- The v0.1 surface (four-page UI + dashboard, client module, audit) unchanged — served through the packaged path.
## Happy Path
1. Owner downloads `coreci-broker-v0.2.3.pyz` from the `v0.2.3` release → `python3 coreci-broker-v0.2.3.pyz` (or `chmod +x` + `./…`) → first boot generates `.env` and prints the admin pass exactly once; `broker.db` + `broker.audit.log` created mode 0600 in the state dir → UI at the bound address (`/ui/login` bookmark works).
2. Second boot: silent — pass never re-printed; state persists.
3. Tool authors fetch `coreci_broker_client.py` from `/static/coreci_broker_client.py` exactly as in v0.1 — the packaged artifact is the only input.
4. `--version` reports the release version without booting.
5. Separately: `docker pull git.coreci.dev/coreci/broker:v0.2.3` → `docker run -p 8765:8765 -v broker-state:/state …` → identical first-boot/0600/UI behavior inside the container, state on the volume.
## UX Acceptance Criteria
- **Self-contained artifact** — the `.pyz` bundles broker + UI pages + client module; only runtime state lives outside it (`.env`, `broker.db`, `broker.audit.log`, materialized assets under `<state>/.coreci_broker_assets/`); one file serves the full admin UI at `/ui/` and the client at `/static/`.
- **`--version`** — prints the release version (from the build-time `_cli_version.py`), exits 0, no broker import on that path.
- **First-boot parity** — behavior through the packaged artifact is byte-exact with v0.1 standalone: admin pass printed exactly once, `broker.db` + `broker.audit.log` mode 0600, UI + client served, second boot silent.
- **Per-release artifact** — every release from `v0.2.1` (D-020) carries `coreci-broker-vX.Y.Z.pyz` + release notes (CLI install/usage + sha256, docker usage); attachment failure escalates `release_pending`, never blocks tag+merge. `v0.2.0` carries none (G-021).
- **Image published once per milestone** — at the final ship: `git.coreci.dev/coreci/broker:v0.2.3` + `latest` (D-021); publish failure escalates per the same contract with the G-020 closure runbook, never blocking the milestone merge.
## Phase 1 — CLI Packaging + Release Tooling
Branch `phase/01-cli-packaging` → tag `v0.2.1`. Requirements: REQ-21, REQ-22.
Exit: artifact builds reproducibly from a clean checkout (two consecutive same-environment builds → byte-identical sha256); the built `.pyz` boots with v0.1 acceptance parity (pass printed once, 0600 state files, UI + client served from materialized assets, silent second boot); `--version` wired; release-asset automation stub-verified; REQ-21 + REQ-22 consolidated green; the `v0.2.1` release carries the `.pyz` attached by `scripts/release.py`.
### Wave 1 — build pipeline + boot shim
- **P1-01** · release-engineer · `scripts/build_cli.py`, `cli/__main__.py`, `.gitignore`, `tests/test_build_cli.py` · build pipeline + boot shim (RESEARCH §1.2/§1.3; G-009/G-018): `build_cli.py <version>` stages an explicit allowlist tree — `coreci_broker.py`, `coreci_broker_client.py`, `coreci_broker_ui/**`, `cli/__main__.py`, generated `_cli_version.py` (`VERSION = "<tag>"`) — then `python3 -m zipapp <staging> -o dist/coreci-broker-vX.Y.Z.pyz --python "/usr/bin/env python3" --compress` (zipapp defaults to ZIP_STORED — `--compress` mandatory), then a normalize post-pass making same-environment rebuilds byte-identical (G-009: zip entries sorted by name, `date_time=(1980,1,1,0,0,0)`, `ZIP_DEFLATED`, **`external_attr` SET to constants** — files `0o100644 << 16`, dirs `0o40755 << 16` — never preserved/umask-derived; shebang re-prepended + exec bit), and prints the artifact sha256 (digest-in-notes default; no sidecar asset); `cli/__main__.py` = the five-step shim: (1) `--version` first — print `_cli_version.VERSION`, exit 0, no broker import; (2) state-dir resolution identical to the broker (`BROKER_ENV_PATH`-anchored, relative paths stay CWD-relative — zero path changes); (3) materialize assets from `zipfile.ZipFile(sys.path[0])` → `<state>/.coreci_broker_assets/<VERSION>/{ui,static}` (temp + `os.replace`, versioned dirs, same-version overwrite self-repair, no cleanup); (4) override precedence user-wins — `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` set only when absent from process env AND the user `.env` (parsed via `coreci_broker._parse_env_file`); (5) handoff `coreci_broker.run_standalone()`; an `os.path.isdir(sys.path[0])` guard falls back to repo-relative roots when run unzipped (zipapp-only is the documented mode); the shim header documents its private-API coupling to the broker (`_parse_env_file`, `run_standalone` — G-018: loud `ImportError` breakage accepted and tracked in P3-02); `.gitignore` += `dist/`, `*.pyz`; broker sources are read-only staging inputs — zero broker lines · REQ-21 · test: `test_build_cli.py` — two consecutive same-environment builds → byte-identical sha256; `_cli_version.VERSION` embedded; archive enumerates exactly the staging allowlist (broker + client + UI pages + shim); constant `external_attr` asserted (no umask bleed); `py_compile` of the shim green; built artifact `--version` prints the version, exits 0.
### Wave 2 — packaged boot acceptance
- **P1-02** · release-engineer · `tests/test_cli_packaging.py` · end-to-end boot acceptance through the built artifact (RESEARCH §4.3 pattern; G-010 harness hardening; consolidates REQ-21): build into a `TemporaryDirectory` → `Popen([sys.executable, "<pyz>"], env={"BROKER_BIND": "127.0.0.1:0", BROKER_ENV_PATH/DB/AUDIT/CREDS_FILE/CREDS_SOURCE: tmp paths})` → parse the `coreci-broker listening on http://…` line for the ephemeral base URL under an explicit deadline loop; first boot prints `BROKER_ADMIN_PASS=` exactly once (stdout fully drained after `terminate()` before the second-boot assertion); `.env` + `broker.db` + `broker.audit.log` created mode 0600 in the state dir (independent of CWD); `/healthz` 200, `/ui/login` + `/ui/dashboard` 200 from materialized assets, `/static/coreci_broker_client.py` byte-exact; admin login + agent register + state + secret-read reuse the v0.1 acceptance request patterns (v0.1 parity through the packaged path); second boot silent (pass never re-printed); `--version` exits 0 with the version string; materialized assets verified under `<state>/.coreci_broker_assets/<VERSION>/`; user override honored (`.env`-set `BROKER_UI_ROOT` wins over the shim default); plus one un-overridden default-anchoring boot (state in CWD of a temp dir — the owner's literal happy path, G-010) · REQ-21 · test: is the acceptance file.
### Wave 3 — release-asset automation (G-011: lands in P1)
- **P1-03** · release-engineer · `scripts/_env_secrets.py`, `scripts/release.py`, `tests/test_release_upload.py` · sanctioned secrets loader + release attacher (REQ-22; RESEARCH §2): `_env_secrets.py` parses `.env`/`.env.secrets`/`.env.*` from the repo root + `.ciagent/` (mirroring the broker's `_parse_env_file` semantics, quote-stripping) — `GITEA_TOKEN` resolves ONLY from these files, never `os.environ` (structurally blocked by `blocked_env_vars`), asserted non-empty before the first API call (v1.8 lesson: empty token → gitea 404, masking auth failure); `release.py <tag> <artifact>` (gitea base URL + owner/repo parameterized): **computes the artifact sha256 from the file itself — never an argument, never hand-pasted (G-017)** → builds the notes body per the RESEARCH §2.3 model (milestone summary, phases, verification, CLI install + usage with the artifact sha256, docker usage) → create-or-get by tag (`GET /api/v1/repos/{o}/{r}/releases/tags/{tag}`; 404 → `POST .../releases` with the notes body) → idempotent attach (`GET .../releases/{id}/assets` — same name + size → success, no duplicate on bounded-retry re-runs) → `POST .../releases/{id}/assets` multipart (field name `attachment`, `?name=coreci-broker-vX.Y.Z.pyz`, `Authorization: token <T>`) → bounded retry (3 attempts, backoff) → exhaustion exits non-zero with a machine-readable marker for the `release_pending` escalation; never blocks tag+merge · REQ-22 · test: `test_release_upload.py` — `_StubGiteaHandler` on `("127.0.0.1", 0)` (v0.1 `_StubOwuiHandler` precedent, fake token, no live-forge writes): create-or-get in both orders (release exists / 404 → create), multipart parsed (field name + `?name=` + body bytes asserted), sha256-from-file asserted (passing a wrong digest argument is ignored/refused), idempotent re-attach uploads nothing twice, 500×3 → the `release_pending` marker, token discipline (file-only resolution + non-empty assert before the first call), notes sections present.
### Wave 4 — phase ship
- **P1-04** · lead-developer · no code writes (the release-engineer's built artifact + scripts are consumed read-only) · phase ship: tag `v0.2.1` → squash-merge `phase/01-cli-packaging` → `milestone/v0.2-cli-packaging` → push tag + branch → gitea release via `scripts/release.py` (D-020 starts here: `coreci-broker-v0.2.1.pyz` + its sha256 in the notes, attached by the repo script — no ad-hoc bridge, G-011) · REQ-21, REQ-22 (ship) · verify: `v0.2.1` tag + release on the forge with the `.pyz` attached; attachment failure → bounded retry (3) → `release_pending` escalation, never blocking tag+merge.
## Phase 2 — Docker + Image Publish
Branch `phase/02-docker-release-integration` → tag `v0.2.2`. Requirements: REQ-23 (+ REQ-21/REQ-22 docs refinement).
Exit: daemonless publish client stub-verified end-to-end (no live-forge writes, no container runtime); Dockerfile + .dockerignore + README document the docker-user path and stay consistency-tested against the daemonless image; the `v0.2.2` release carries the `.pyz` attached by `scripts/release.py` with CLI + docker notes.
### Wave 1 — daemonless OCI publish client
- **P2-01** · release-engineer · `scripts/publish_image.py`, `tests/test_publish_image.py` · daemonless OCI publish client (D-019, G-014: one vertical task; all endpoints parameterized; G-019 wire hardening): pull — anonymous Docker Hub token (`auth.docker.io/token`, scope `repository:library/python:pull`) → `python:3.11-slim` manifest (D-022) with the full Accept set → branch on `"manifests"` vs `"layers"` (index → resolve the `linux/amd64` child manifest by digest) → fetch child + config + 4 layer blobs; record resolved digests (`--base-manifest` override for exact re-publishes); auth — gitea `GET /v2/token?service=container_registry&scope=repository:coreci/broker:pull,push` with `Authorization: Basic <user>:<GITEA_TOKEN>` (token via `_env_secrets`; username ignored by the forge) — insist on Basic credentials present + issued JWT non-empty (the endpoint answers 200 anonymously with an empty-scope JWT — fail loudly, never treat 200 as authorization); push — **per-request timeouts (~120s on blob POSTs, G-019)** + **blob-level retry with HEAD-skip** (HEAD each blob first: 200 → skip; base layers ~45.7 MB re-upload once per milestone, retries resume cheap) → monolithic `POST /v2/coreci/broker/blobs/uploads/?digest=sha256:<hex>` (fallback: two-step POST → 202 `Location` → PUT) with explicit Location-following on 307/308 (stdlib urllib does NOT follow method-preserving redirects on PUT/POST — verified; **the stub test covers a relative `Location` at least once, G-019**) → deterministic broker layer (tar entries sorted, fixed TarInfo metadata uid/gid 0, `gzip.compress(..., mtime=0, compresslevel=9)`) carrying `/app/coreci-broker.pyz`; config copied from the base + diff_id appended + `Entrypoint ["python3", "/app/coreci-broker.pyz"]`, `Cmd []`, `WorkingDir /state`, optional version Label, serialized `json.dumps(sort_keys=True, separators=(",",":"))`; manifest PUT with OCI media types end-to-end (manifest `application/vnd.oci.image.manifest.v1+json`, config `…image.config.v1+json`, layers `…image.layer.v1.tar+gzip`; base layer descriptors reused verbatim; never push an index) under `<version>` then `latest` (D-021); provenance digests recorded for the release notes · REQ-23 · test: `test_publish_image.py` — `_StubRegistryHandler` on `("127.0.0.1", 0)` (fake token via temp `.env` files): anonymous-token trap (200 with empty-scope JWT → loud failure), HEAD-skip on retry, monolithic upload with 307 → relative `Location` followed explicitly, manifest PUT with correct media types + tag sequence (version then `latest`), deterministic layer/config digests stable across two runs (gzip mtime=0), base manifest/layer handling against a fixture index, timeout wiring (a stalled endpoint → client aborts, does not hang).
### Wave 2 — Dockerfile + .dockerignore + docs
- **P2-02** · release-engineer · `Dockerfile`, `.dockerignore`, `README.md` (additive section only — no v0.1 content rewrites), `tests/test_dockerfile_consistency.py` · docker-user path + usage docs (G-015/G-016): `Dockerfile` self-contained (no pre-built artifact in the build context): `FROM python:3.11-slim` → `COPY coreci_broker.py coreci_broker_client.py coreci_broker_ui/ cli/ /app/src/` → `ARG VERSION` → generated `_cli_version.py` from the build arg → `RUN python3 -m zipapp /app/src -o /app/coreci-broker.pyz --python "/usr/bin/env python3" --compress` → `WORKDIR /state` → `ENTRYPOINT ["python3", "/app/coreci-broker.pyz"]` → `EXPOSE 8765`; header documents the equivalence mapping to the daemonless-published image (same base family, entrypoint, pyz content, `/state` anchoring; differences: docker-build layering vs verbatim base layers + one broker layer, build-arg vs build-time version provenance; the daemonless client is the source of truth, byte-identity not required — RESEARCH §3.5); `.dockerignore` (G-015): `.git`, `.env`, `.env.*`, `data/`, `*.db`, `dist/`, `__pycache__/`, `tests/`, `.ciagent/` — no secrets, state, or build artifacts can enter an image build context; README additive section: CLI install + usage (release-page download, `chmod +x` / `python3`, `--version`, env-contract pointer, first-boot pass discipline, `BROKER_ENV_PATH` state anchoring, Python 3.11+ note) + docker usage (`docker pull git.coreci.dev/coreci/broker:<tag>` + `latest`; `docker run -p 8765:8765 -v broker-state:/state …`; state-on-volume contract) · REQ-23, REQ-21/REQ-22 (docs) · test: `test_dockerfile_consistency.py` (G-016 — equivalence made testable): Dockerfile COPY set == `build_cli.py` staging allowlist; zipapp flags identical (`--python`/`--compress`); `ARG VERSION` → `_cli_version` format parity; `ENTRYPOINT`/`WORKDIR` == the daemonless-published config values; `.dockerignore` excludes every forbidden context entry.
### Wave 3 — phase ship
- **P2-03** · lead-developer · no code writes · phase ship: tag `v0.2.2` → squash-merge `phase/02-docker-release-integration` → `milestone/v0.2-cli-packaging` → push tag + branch → release via `scripts/release.py` (notes per the §2.3 model now including docker usage, `coreci-broker-v0.2.2.pyz` + sha256 attached alongside the source archive) · REQ-23 (phase completion) · verify: `v0.2.2` tag + release on the forge with the `.pyz` + notes; attachment failure → bounded retry → `release_pending`, never blocks tag+merge.
## Phase 3 — Final Review + Ship
Branch `phase/03-final-review-ship` → tag `v0.2.3` (final phase patch = milestone release, D-008 convention). Requirements: REQ-21, REQ-22, REQ-23 (acceptance).
Exit: cumulative v0.1 + v0.2 acceptance green; broker zero-delta verified (1390/1400, AC14); security review PASS (or findings remediated); `v0.2.3` on `main` with the `.pyz` attached + full notes; image published as a gitea package (or escalated per the bounded-retry contract + G-020 closure runbook — never blocking merge); REQUIREMENTS/ROADMAP closed out; checkpoint cleared.
### Wave 1 — acceptance consolidation
- **P3-01** · release-engineer · `tests/test_acceptance_v02.py` · v0.2 acceptance matrix: cumulative suite green via `python3 -m unittest discover -s tests` (all v0.1 files + the v0.2 test files); packaging acceptance consolidated — boot parity through the artifact (first-boot pass printed once, `broker.db` + `broker.audit.log` 0600, UI + client served, silent second boot, register/read flows), reproducibility (two consecutive same-environment builds → identical sha256), self-containment (only runtime state external to the `.pyz`: env/db/audit + materialized assets), stub publish roundtrip + stub release upload green; broker zero-delta — `coreci_broker.py` unchanged at 1390 lines vs `v0.1.5` (AC14 under the D-016..D-018 cap of 1400) · REQ-21, REQ-22, REQ-23 (acceptance) · test: is the acceptance file.
### Wave 2 — security review (review-only)
- **P3-02** · security-engineer · review of `coreci_broker.py` (zero-delta verification only — writable by nobody this milestone), `cli/**`, `scripts/**`, `Dockerfile`, `.dockerignore`, `tests/**` (no edits — findings STRIDE-classified, CWE-cited, routed via lead-developer) · v0.2 security pass: secrets in artifacts (no secrets in the `.pyz` or any image layer; `.env` never inside the archive; pepper/admin pass generated at boot; no state baked into the image; build-context hygiene per `.dockerignore`, G-015), digest determinism (zip normalize pass with constant `external_attr` — G-009; sorted tar + fixed TarInfo; `gzip mtime=0`; sorted-key JSON), token discipline (file-only `_env_secrets` path, non-empty assert before first use, never shell env; gitea's anonymous-200 `/v2/token` quirk fails loudly), 307/308 Location handling incl. relative `Location`, traversal + disclosure over materialized asset roots (dot-dir roots vs request-path dot filtering; protected-paths realpaths wherever the state dir sits), shim private-API coupling tracked (G-018), attachment idempotency + bounded-retry escalation paths · REQ-21, REQ-22, REQ-23 (+ all) · verify: findings recorded in the task commit's `---ci---` block; remediation (if any) = the follow-up release-engineer commit dispatched by lead-developer (P3-03).
### Wave 3 — remediation
- **P3-03** · release-engineer · `cli/**`, `scripts/**`, `Dockerfile`, `.dockerignore`, `tests/**` (never broker sources — a P0 broker defect routes through lead-developer dispatch only) · remediate P3-02 findings (no-op skip if PASS) · affected REQs per finding · test: full cumulative suite re-green.
### Wave 4 — review + reconstruction
- **P3-04** · lead-developer · `.ciagent/*.md` review notes (no code writes) · multi-persona review of the milestone diff (release-engineer artifacts across P1/P2, security sign-off from P3-02/P3-03); audit/reconstruction: every REQ-21..REQ-23 maps to green tests + traceable commits; broker zero-delta verified (1390/1400, AC14); escalation ledger clean or `release_pending` entries documented · REQ-21, REQ-22, REQ-23 · verify: sign-offs + reconstruction recorded in `---ci---` blocks.
### Wave 5 — milestone ship
- **P3-05** · lead-developer · no code writes (release-engineer scripts invoked, never edited) · milestone ship, in this exact order (G-012: merge→tag→push BEFORE release attach — a pre-merge release create would mint `v0.2.3` at pre-merge main via gitea's target_commitish semantics): build the final `.pyz` (`python3 scripts/build_cli.py v0.2.3`) → squash-merge `milestone/v0.2-cli-packaging` → `main` → tag `v0.2.3` → push main + tag → **G-013 live smoke (env-gated `PUBLISH_SMOKE=1`, read-only, never in the default suite): real gitea `/v2/` auth challenge, token→JWT exchange, `GET` release-by-tag, Docker Hub index resolve — all must pass before the one live publish** → attach to the `v0.2.3` release with full notes (milestone summary, phases, verification, CLI install/usage + sha256, docker usage) via `scripts/release.py` → daemonless image publish ONCE (`python3 scripts/publish_image.py` → `git.coreci.dev/coreci/broker:v0.2.3` + `latest`, D-019/D-021/D-022) → branch cleanup → REQUIREMENTS/ROADMAP closeout → checkpoint cleared. Publish failure: bounded retry (3) → `release_pending` escalation **with the G-020 closure runbook committed alongside** (exact re-run command on `main` @ `v0.2.3`: build + smoke + `publish_image.py`; close condition = both tags visible as a gitea package) — never blocks the merge, never left open · REQ-21, REQ-22, REQ-23 · verify: `v0.2.3` on `main`; the release carries the `.pyz` + full notes; the image is visible as a gitea package (or escalated per contract + runbook, never blocking); closeout + checkpoint-clear commits recorded.
+87
View File
@@ -0,0 +1,87 @@
# PROJECT — CoreCI Broker
## Name
CoreCI Broker
## Core Value
A single canonical credential flow for autonomous agent tools. The broker mediates access to the `creds-coreci-coo` knowledge base: tools authenticate by a derived token (not a shared secret), the CEO manages a UI allowlist of approved tool UUIDs and user IDs, every read is audited to an append-only file, and per-tool scope is enforced by path (`coreci/<agent_id>` only). It replaces per-tool credential resolution (KB files, env vars, mode-0600 JSON) with one small, readable implementation surface: a single Python file, one SQLite DB, a four-page web UI.
## Requirements
### Validated
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09; supersedes v0.4) — distilled into REQ-01..REQ-20 (shipped in v0.1). v0.2 additions validated from owner directives (2026-10-10) — distilled into REQ-21..REQ-23: CLI zipapp packaging, per-release artifact attachment with notes on gitea, once-per-milestone docker publication to the gitea container/package registry. The v0.5 spec's ambiguities were resolved by D-001..D-015 (init + clarify commits, v0.1); v0.2's packaging/distribution ambiguities are resolved as D-019..D-022 (this milestone's clarify stage):
- Dual-mode token hashing (bcrypt when importable, stdlib scrypt fallback; format-prefixed hashes)
- Pluggable creds-source adapter (file-based default; Open WebUI HTTP API for production)
- Agent-state backend toggle (`internal` default | `owui`)
- UUID-scoped token verification + in-memory verified-token cache (spec §7.5's token_hash index is impossible with bcrypt)
- Register/pepper semantics including the KB-loss recovery path
- A public, dependency-free client module served as a static asset so agents can bootstrap their own OWUI tools
- Spec §13 defaults: no audit rotation, single instance, 24h token TTL, CEO sole approver
### Active
Milestone v0.2 (`milestone/v0.2-cli-packaging`) — **CLI + distribution only**, no functional changes to the broker: REQ-21 (zipapp CLI packaging), REQ-22 (release assets attached + notes on gitea), REQ-23 (docker container built once per milestone, published as a gitea package). Owner-locked inputs: zipapp `.pyz` form factor; docker publication starts with v0.2 (no retroactive v0.1 image); scope excludes deferred v2 candidates.
### Out of Scope
See REQUIREMENTS.md §Out of Scope — federation, secret versioning, high availability, UI beyond the four pages + dashboard, audit rotation, multiple broker instances, approver agents, key rotation ceremony, HSM, rate limiting, admin MFA, notifications.
## Constraints
- **Single-file broker**: `coreci_broker.py` under 1400 lines (amended from 800 via owner decisions D-016..D-018; excluding tests, UI HTML, client module, reference tools) — spec §12.14 as amended
- **Stdlib-only runtime**: no pip in the target dev environment; bcrypt is the sole optional import (dual-mode hashing); tests use unittest
- **SQLite is the source of truth**: single instance, no HA; back it up
- **Bind default** `127.0.0.1:8765`; the bind address controls reachability
- **Mode 0600** on `broker.db` and `broker.audit.log`
- **The broker never writes to `creds-coreci-coo`** — the CEO writes that KB directly through Open WebUI's UI
- **Config contract** = spec §5.1 plus documented additions: `BROKER_ENV_PATH`, `BROKER_CREDS_SOURCE`, `BROKER_CREDS_FILE`, `BROKER_OWUI_BASE_URL`, `BROKER_OWUI_API_KEY`, `BROKER_AGENT_STATE_BACKEND`
- **Token form**: `base64(sha256(pepper:UUID:user_id:agent_id:sorted_tags))` — 44-char ASCII, no prefix; `BROKER_TOKEN_TTL` default 86400s
- **Open WebUI runtime**: bcrypt and httpx exist there; the client module must not depend on either (stdlib urllib) so any tool can import it
v0.2 (packaging/distribution) constraints:
- **No pip / stdlib-only build path**: the CLI artifact is a stdlib `zipapp` (`.pyz`); no wheel, no PyInstaller, no pip at build or runtime (D-009)
- **No container runtime in the dev environment**: no docker/podman/buildah available — the once-per-milestone publish runs daemonless via a stdlib registry client (D-019)
- **Self-contained artifact**: one `.pyz` bundles broker + UI assets + client module; asset roots resolved from env, never `__file__`
- **Broker behavior frozen**: first-boot semantics, API surface, and all v0.1 acceptance criteria remain byte-identical when driven through the packaged CLI
- **Release secrets discipline**: forge tokens resolved only from `.env*` files via the secrets-loader — never shell environment (v1.8 lesson)
- **`broker.db`/`broker.audit.log` remain state-directory-local** — the CLI boots in a runtime state dir (docker: a volume), never writes inside its own archive
## Context
CoreCI estate: sibling projects researchx and nextcraft live on gitea (`git.coreci.dev/coreci/*`) and use this same ciagent pipeline. The broker runs on the Open WebUI host as one workspace tool (embedded HTTP server in a daemon thread; also runnable standalone). The `creds-coreci-coo` KB currently mediates four credentials (`coreci/proxmox`, `coreci/gitea`, `coreci/executor` TBD, plus the COO's own login — independent of the broker's `BROKER_ADMIN_PASS`). The existing `proxmox_readonly_tool.py` is updated post-broker to use the public client module; it lives outside this repo, so this repo ships the reference implementation of the tool-side bootstrap instead.
v0.2 context: milestone v0.1 shipped complete (all REQ-01..REQ-20, `v0.1.5` on main, six gitea releases each carrying a `broker-vX.Y.Z.tar.gz` source archive, zero escalations). Owner directives for v0.2 (2026-10-10): package the broker as a CLI; attach the CLI to every release alongside notes on gitea; build a docker container once per milestone and publish it as a package in the gitea registry. The v0.1 release-attachment mechanism (sanctioned `.env*` token path, per-release asset upload) is the proven base REQ-22 extends.
Known spec defects resolved by clarification (recorded for traceability): §7.5's in-memory `token_hash` index is impossible with bcrypt salted hashes; §6 claims the tool derives a token every call while §9's flow uses the broker-returned token from the agent KB (the broker derives authoritatively; the tool stores); §9's snippet raises `KeyError` on `data["pepper"]` for re-registrations; the pepper chicken-and-egg (a token is needed to read agent state, but the pepper is needed to derive a token) is fixed by returning the pepper when the caller presents no valid token.
## Key Decisions
| ID | Decision | Rationale | Outcome |
|----|----------|-----------|---------|
| D-001 | Dual-mode bcrypt/scrypt hashing, format-prefixed hashes | Spec §6 requires bcrypt; dev env is stdlib-only; OWUI runtime ships bcrypt | To be validated in P1 |
| D-002 | Creds-source adapter: file (default) \| OWUI API | Spec never specified the physical read mechanism; both environments must work | To be validated in P4 |
| D-003 | Agent-state backend toggle, internal default | §9.1 endpoints are the only interface; internal storage is functionally identical without the admin-API dependency | To be validated in P3 |
| D-004 | UUID-scoped slow-hash scan + in-memory verified-token cache | §7.5's bcrypt token_hash index is impossible; header UUID bounds the scan; SQLite stays authoritative | To be validated in P3 |
| D-005 | Register re-derives; pepper returned on row-create or tokenless register | Broker holds pepper + all inputs; tokenless register is the only KB-loss recovery path | To be validated in P3 |
| D-006 | Public dependency-free client module at /static/ + reference tools | Owner directive: agents bootstrap their own tools within the OWUI workspace | To be validated in P4 |
| D-007 | Spec §13 defaults accepted | Owner confirmed: no rotation, single instance, 24h TTL, CEO sole approver | Binding for v0.1 |
| D-008 | Milestone v0.1; "v0.5" remains the spec revision label | computeMilestoneTag greenfield convention (researchx, nextcraft precedent) | Binding for v0.1 |
| D-009 | Stdlib-only runtime; unittest for tests | No pip in environment; dual-mode hashing is the sole optional import | Binding for v0.1 |
| D-010 | Single-project mode; gitea release target coreci/broker | Sibling convention on the same forge; first push deferred to a ship phase | Binding for v0.1 |
| D-011 | Empty root commit on main precedes the branch hierarchy | Branches cannot be created from an unborn HEAD; nextcraft precedent | Executed at init |
| D-012 | Forge repo coreci/broker public, default branch main | Sibling convention (researchx/nextcraft both public); code ships no runtime secrets — pepper/admin pass generated at boot, .env.secrets gitignored | Binding for v0.1 |
| D-013 | Session cookie `broker_session` uses HttpOnly + SameSite=Lax | Spec silent on cookie attributes; HttpOnly blocks JS exfiltration, Lax preserves bookmark-login UX | Binding for v0.1 |
| D-014 | `BROKER_CREDS_FILE` defaults to `data/creds.json` | FileCredsSource dev/test default location; `data/` already gitignored | Binding for v0.1 |
| D-015 | `GET /api/broker/agents/<agent_id>`: App-auth + identity match, metadata-only response (no token, no hash) | §7.4 auth/response unspecified; mirrors state-endpoint identity enforcement; token material never leaves the broker | Binding for v0.1 |
| D-016 | AC14 line cap amended: coreci_broker.py < 1000 lines (was < 800) | Owner-approved §12.14 amendment at P2: 11 lines of headroom vs ~275 lines of remaining handler surface; preserves the readable single-file intent without dense-cramming | Binding for v0.1 |
| D-017 | D-016 extended: AC14 cap < 1200 lines | Second §12.14 amendment at P3 (owner's D-016 pattern applied with new data): P3-01 lands at 990/1000 with ~150-180 lines of remaining broker surface (App-auth cache, agent state, creds adapter, secrets endpoint); mild compaction reclaims only ~50 — same readability-over-cramming tradeoff the owner chose at D-016; subject to review at the P5 owner approval gate | Binding for v0.1 |
| D-018 | D-017 extended: AC14 cap < 1400 lines — final projected amendment | Third §12.14 amendment, pre-P4 per the P3 verify planning signal: 1178/1200 after P3 with 21 lines headroom vs P4's ~105-140 line need (creds adapter + secrets endpoint, the LAST broker-surface additions); P5 adds only excluded artifacts (tests, README). Same owner pattern (D-016/D-017), flagged for the P5 owner approval gate | Binding for v0.1 |
| D-019 | Daemonless container publish: stdlib registry client (pull pinned base layers, append broker layer, push to gitea registry) | No docker/podman/buildah in the dev environment; gitea's package registry speaks OCI registry HTTP API v2 — fully automated once-per-milestone publish without a container runtime; stub-testable | Binding for v0.2 |
| D-020 | Attach `coreci-broker-vX.Y.Z.pyz` to every release from `v0.2.1` onward | Consistent with v0.1's per-release `broker-vX.Y.Z.tar.gz` asset convention; every tagged release stays independently deployable | Binding for v0.2 |
| D-021 | Image tags `git.coreci.dev/coreci/broker:<milestone-release-tag>` + `latest`, pushed once per milestone | Version tag pins the release; `latest` gives docker users a sane default; once-per-milestone honors the owner directive | Binding for v0.2 |
| D-022 | Base image `python:3.11-slim` (pinned minor line) | Dev environment runs Python 3.11.2 — pinned minor matches tested stdlib behavior and stays reproducible | Binding for v0.2 |
+113
View File
@@ -0,0 +1,113 @@
# Requirements — CoreCI Broker
## Overview
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09). "v0.5" is the spec's revision label. REQ-01..REQ-20 (milestone v0.1) shipped complete as `v0.1.5`. **Milestone v0.2 (cli-packaging)** adds REQ-21..REQ-23 from owner directives (2026-10-10): package the broker as a CLI, attach it to every gitea release with notes, and publish a docker container once per milestone as a gitea package. Scope is CLI + distribution only — no functional changes to the broker. Clarification decisions D-001..D-015 (v0.1) remain binding; D-016+ resolve v0.2's packaging/distribution ambiguities.
## v1 (Milestone v0.1 — complete, shipped as v0.1.5)
### Auth & Identity
- **REQ-01 Admin authentication.** `BROKER_ADMIN_USER`/`BROKER_ADMIN_PASS` from the broker `.env`. Session cookie `broker_session` for the UI; HTTP Basic accepted on all admin API endpoints. `POST /api/broker/auth/login` creates a session (SQLite `admin_sessions`, TTL from `BROKER_SESSION_TTL`, default 300s), `logout` invalidates it, `whoami` returns the session info or 401. Admin sessions expire lazily and are cleaned on boot.
- **REQ-02 Tool self-registration.** `POST /api/broker/agents/register` with `{agent_id, app_registration_uuid, user_id, tags}`. The broker rejects unknown or revoked UUIDs and user_ids with 403 (allowlists are the gate); malformed input → 400. Re-registration is idempotent: the broker re-derives from the request inputs and overwrites the stored hash when inputs changed; an unexpired row re-issues the same derived token. The response returns the token, `token_expires_at`, and the pepper when the row is created or the caller presented no valid token (KB-loss recovery). Register actions are audited.
- **REQ-03 Derived token.** `token = base64(sha256(pepper:APP_UUID:user_id:AGENT_ID:','.join(sorted(tags))))` — a 44-char ASCII string, no prefix, no challenge protocol. The broker derives the same token a compliant tool would derive. Stored as a slow hash in `agent_registrations.token_hash`: bcrypt when importable, else stdlib scrypt; hashes are format-prefixed (`$2b$…` / `scrypt$…`) so verification is format-aware.
- **REQ-04 Token verification & TTL.** Tool endpoints require `Authorization: App <UUID> <token>`. Verification: look up rows scoped to the presented UUID, check the slow hash (in-memory verified-token cache after first success; cache is never authoritative — rows re-checked for status/expiry). Missing/malformed/failed → 401. Tokens live `BROKER_TOKEN_TTL` seconds (default 86400); expiry → 401 → the tool re-registers. Revoked apps keep working until TTL expiry by design.
### Allowlist Administration
- **REQ-05 App registrations.** Admin-only `GET/POST /api/broker/app_registrations`, `DELETE /api/broker/app_registrations/<uuid>` (marks revoked; new registrations rejected; existing tokens honored until TTL), `POST .../reinstate`. UUID format validated (400 on invalid, 409 on duplicate). All actions audited as `add_app`/`remove_app`.
- **REQ-06 User allowlist.** Admin-only `GET/POST /api/broker/user_allowlist`, `DELETE .../<user_id>` (revoked semantics), `POST .../reinstate`. Optional `email` field. 409 on duplicate. Audited as `add_user`/`remove_user`.
### Secret Mediation
- **REQ-07 Path-scoped secret reads.** `GET /api/broker/secrets/<path>`. Scope is the path: a registration may read exactly `coreci/<agent_id>` — cross-path → 403, missing secret → 404, auth failure → 401. Every attempt (ok/denied/error) is audited with `user_id`, `app_registration_uuid`, `agent_id`, `action: read`, `path`, `result`.
- **REQ-08 Creds-source adapter.** The broker reads `creds-coreci-coo` exclusively through a pluggable adapter, `BROKER_CREDS_SOURCE=file|owui`: `FileCredsSource` reads a JSON document at `BROKER_CREDS_FILE` mapping paths to secret objects (default — dev/tests); `OwuiApiCredsSource` reads via the Open WebUI HTTP API using `BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY` (production). The broker never writes to the KB. Response shape: `{path, data, version, updated_at}`.
### Agent State
- **REQ-09 State endpoints.** `GET/PUT /api/broker/agents/<agent_id>/state` (App-auth). The broker enforces that the calling UUID and the token-derived user_id match the stored identity; 404 when no state exists. State schema per spec §8 (agent_id, app_registration_uuid, user_id, tags, pepper, current_token, token_expires_at, first_registered_at, last_seen_at). This is the tool's only persistence channel — the broker is the durability layer.
- **REQ-10 State backend toggle.** `BROKER_AGENT_STATE_BACKEND=internal` (default: rows in `broker.db` behind the state endpoints) `| owui` (real per-agent Open WebUI KBs via the OWUI API). The tool cannot tell the difference — the state endpoints are the only interface.
### Audit
- **REQ-11 Append-only audit log.** One JSON object per line at `BROKER_AUDIT_LOG_PATH` (mode 0600, append-only, no rotation in v0.1). Fields: `ts`, `user_id`, `app_registration_uuid`, `agent_id`, `action` (register/read/denied/login/logout/add_app/remove_app/add_user/remove_user), `path`, `result` (ok/denied/error); admin actions carry null user_id/uuid.
- **REQ-12 Audit query.** Admin-only `GET /api/broker/audit` with `since`, `until`, `user_id`, `app_registration_uuid`, `action` filters and `limit` (default 100, max 1000); returns entries plus `next_since` cursor. The broker streams the file line-by-line with filtering.
### Web UI
- **REQ-13 Four pages + dashboard.** Server-rendered HTML from `coreci_broker_ui/`, no build step, same port as the API. `/ui/login` (the CEO's bookmark), `/ui/app_registrations` (table + add-modal + revoke/reinstate per row), `/ui/user_allowlist` (same), `/ui/audit` (filterable, read-only, no export), plus `/ui/dashboard` and a persistent nav header with Logout. After each action the page refreshes via the corresponding API.
### Ops & Bootstrap
- **REQ-14 `.env` config + first boot.** The broker reads its `.env` at `BROKER_ENV_PATH` on boot (spec §5.1 contract plus the documented additions). On first boot it generates `BROKER_INSTANCE_PEPPER` (32 random bytes, base64 — synced to SQLite) and `BROKER_ADMIN_PASS` (32+ random chars), writes them to `.env`, and prints the admin pass to the startup log exactly once — never again.
- **REQ-15 Mode 0600.** `broker.db` and `broker.audit.log` are created mode 0600, owned by the Open WebUI process user. `.env.secrets` discipline (ciagent) mirrors this.
- **REQ-16 Persistence across restarts.** SQLite and the audit file are the durable state. Restarts preserve registrations, allowlist entries, sessions' data, and audit history; in-memory caches (verified-token cache) rebuild lazily; existing tokens still verify.
- **REQ-17 Single-file constraint.** `coreci_broker.py` is one Python file under **1400** lines (amended 800→1000 D-016, →1200 D-017, →1400 D-018 — owner-approved §12.14 amendment pattern, final projected), excluding tests, UI HTML, the client module, and reference tools. A line-count test enforces it (AC14).
- **REQ-18 Runtime modes.** Importable as an Open WebUI workspace tool (daemon-thread server + a `Tools` class surface) and runnable standalone (`python3 coreci_broker.py`). Stdlib-only at runtime; bcrypt is the sole optional import (D-001/D-009). Tests use unittest against an in-process server on an ephemeral port.
### Client Ecosystem
- **REQ-19 Public client module.** `coreci_broker_client.py` — dependency-free (stdlib urllib; no httpx, no bcrypt) — implements the tool-side bootstrap of spec §9: ensure-registered (idempotent), state get/put, secret read with 401 → re-register → retry-once, and never assumes the pepper key is present in register responses (`data.get("pepper")`). Served unauthenticated at `/static/coreci_broker_client.py` so any OWUI tool author can fetch it into their workspace and import it. It contains no secrets.
- **REQ-20 Reference tools.** `reference_tools/` ships at least: (a) a proxmox-style reference implementing §11's flow — first call registers and persists state, subsequent calls read the token from state and fetch `coreci/proxmox`, downstream call defaults to a safe dry-run; (b) a minimal showcase of auth-first-then-use-a-second-tool. These are templates for tool authors, wired to the test broker in tests.
## v2 (Milestone v0.2)
### Packaging
- **REQ-21 CLI packaging (zipapp).** The broker ships as a single-file executable CLI `coreci-broker.pyz` built with stdlib `python3 -m zipapp` from a clean checkout — no pip, no third-party build tooling (D-009 preserved). The archive is self-contained: it bundles `coreci_broker.py`, the `coreci_broker_ui/` pages, and `coreci_broker_client.py`, materializing the UI/static roots into the runtime state directory on boot (paths resolved from env, never `__file__`), so one file serves the full admin UI at `/ui/` and the client module at `/static/coreci_broker_client.py`. Running it boots the standalone server with first-boot semantics identical to REQ-14/REQ-18 (`.env` generation, admin pass printed exactly once, `broker.db` + `broker.audit.log` at mode 0600); `--version` reports the release version. A build script regenerates the artifact reproducibly, and tests boot the built artifact end-to-end.
### Distribution
- **REQ-22 Release assets.** Every gitea release from the phase that first builds the CLI (`v0.2.1`, D-020) onward attaches the built artifact `coreci-broker-vX.Y.Z.pyz` alongside the existing source archive. Uploads resolve the forge token only via the sanctioned secrets path (`.env*` files — never shell environment). Release notes document the CLI: artifact download + install (`chmod +x` / `python3`), the CLI surface (boot, `--version`, env contract), and docker usage per REQ-23. Attachment failure follows the ship workflow's bounded-retry then `release_pending` escalation — it never blocks tag+merge.
- **REQ-23 Docker package.** The repo ships a `Dockerfile` and a **daemonless stdlib registry client** (D-019): once per milestone, at the final ship, the client pulls the pinned `python:3.11-slim` base image manifest + layers from the public registry (HTTP API v2), appends a layer carrying the CLI artifact, and pushes the image `git.coreci.dev/coreci/broker:<milestone-release-tag>` plus the `latest` alias (D-021/D-022) to the gitea container/package registry — no container runtime required. The container runs the CLI standalone with state on a volume. Registry auth, manifest, layer, and digest handling are stub-tested; publish failure follows bounded-retry escalation — it never blocks the milestone merge.
## Deferred Beyond v0.2
None defined. Future candidates held as out-of-scope decisions, not requirements: audit rotation with retention, multiple broker instances, approver-agent delegation, OWUI-KB credential sync.
## Out of Scope
| Feature | Reason |
|---------|--------|
| Multi-instance federation | Spec §3 — one broker on the Open WebUI host |
| Secret versioning / history | Spec §3 — PUT overwrites; broker is read-only on the KB |
| High availability | Spec §3 — SQLite is the source of truth; back it up |
| Web UI beyond the four pages + dashboard | Spec §3, §10 |
| Broker writes to `creds-coreci-coo` | Spec §3 — the CEO writes the KB via Open WebUI's UI |
| Audit log rotation | Spec §13 Q1 — CEO default: never rotate; manual `mv` when large |
| Multiple broker instances | Spec §13 Q2 — confirmed out of scope for v0.5 |
| Approver agents (delegated allowlist adds) | Spec §13 Q4 — CEO is sole approver |
| Key rotation ceremony | Spec §14 — pepper and tool UUIDs are stable |
| HSM integration for the pepper | Spec §14 |
| Rate limiting on auth endpoints | Spec §14 — assumes trusted network |
| Multi-factor admin auth | Spec §14 — single password sufficient |
| Email/notification on allowlist changes | Spec §14 |
| bcrypt as a hard dependency; pip-installable packages | D-001/D-009 — environment has no pip; stdlib-only runtime |
## Traceability
| Requirement | Phase | Status |
|-------------|-------|--------|
| REQ-01 | 2 | complete |
| REQ-02 | 3 | complete |
| REQ-03 | 1 | complete |
| REQ-04 | 3 | complete |
| REQ-05 | 2 | complete |
| REQ-06 | 2 | complete |
| REQ-07 | 4 | complete |
| REQ-08 | 4 | complete |
| REQ-09 | 3 | complete |
| REQ-10 | 3 | complete |
| REQ-11 | 1 | complete |
| REQ-12 | 2 | complete |
| REQ-13 | 2 | complete |
| REQ-14 | 1 | complete |
| REQ-15 | 1 | complete |
| REQ-16 | 1 | complete |
| REQ-17 | 1 | complete |
| REQ-18 | 1 | complete |
| REQ-19 | 4 | complete |
| REQ-20 | 4 | complete |
| REQ-21 | 1 | complete |
| REQ-22 | 1 | complete |
| REQ-23 | 2 | complete |
+191
View File
@@ -0,0 +1,191 @@
# RESEARCH — CoreCI Broker
Date: 2026-10-10 · Milestone: v0.2 (cli-packaging) · Phase: 0 (pre-execution) · Status: complete
Sources: `.ciagent/` PROJECT/REQUIREMENTS/ARCHITECTURE/ROADMAP/PLAN, `coreci_broker.py` (1390 lines, read in full), git history/tags v0.1.0..v0.1.5, live gitea 1.26.1 API (`git.coreci.dev` swagger.v1.json + releases — read-only, token never printed), live Docker Hub registry + auth.docker.io (anonymous pulls, verified end-to-end), live gitea container-registry auth probes (read-only, no upload sessions created), stdlib source (`/usr/lib/python3.11/zipapp.py`, `urllib/request.py`), and an **end-to-end zipapp prototype booted from the real broker sources** (built + booted as subprocess + driven with urllib; `/tmp` scratch, repo untouched).
## 0. Carried-Forward v0.1 Findings (still binding)
- **Environment (v0.1 §1, re-confirmed)**: Python 3.11.2, NO pip anywhere, NO docker/podman/buildah runtime; stdlib-only runtime + build path (D-009 preserved by REQ-21/D-019); tests = stdlib unittest; forge gitea `git.coreci.dev`, owner `coreci`, repo `broker`, public, default branch `main`; `GITEA_TOKEN` resolves from `.ciagent/.env.secrets` only (mode 0600, never printed, `config.security.bash_allowlist.blocked_env_vars` includes GITEA_TOKEN → shell-env tokens are structurally blocked).
- **Test harness (v0.1 §6)**: in-process `ThreadingHTTPServer(("127.0.0.1", 0), Handler)` + daemon thread + `tempfile.TemporaryDirectory()` state + stdlib `urllib.request` client + `addCleanup`; 0600 assertions via `st_mode & 0o777 == 0o600`; stub-HTTP precedent: `_StubOwuiHandler` in `tests/test_creds_source.py` (BaseHTTPRequestHandler with class-level fixtures + hit recording) — this is the template for v0.2 stub registry/gitea servers.
- **Ship contract (`ci/workflows/ship.md` + config.json)**: every phase ships; release creation best-effort with bounded retries (`ship.max_release_retries: 3`); on exhaustion commit escalation `resolution: "auto"`, `type: "release_pending"`, phase still complete — tag+merge is the gate, `release_blocking: false`; **all secrets resolved ONLY from `.env`/`.env.secrets`/`.env.*` files via the secrets-loader, never shell env** (v1.8 root cause: empty token → gitea answers 404, not 401, masking auth failure).
- **OWUI integration surface (v0.1 §4)**: unchanged by v0.2 — the CLI does not touch the `Tools` mount; path resolution from env (never `__file__`) is the v0.1 property that makes zipapp packaging possible at all.
- **Static-serving rules that the packaged CLI must preserve**: `_resolve_static_file` rejects `\x00`, `\`, leading `/`, `..` segments, and any *request* path part starting with `.`; roots are real-pathed (`os.path.realpath`) so a dot-named *root* directory is fine (materialized assets live under a dot-dir; only request paths are dot-filtered). `_broker_protected_paths` protects `.env`/db/audit/creds (+ `-journal/-wal/-shm` suffixes) from static reads.
## 1. zipapp CLI Packaging (REQ-21)
### 1.1 stdlib mechanics (all VERIFIED empirically on Python 3.11.2)
- Command: `python3 -m zipapp <srcdir> -o coreci-broker.pyz --python "/usr/bin/env python3" --compress`.
- **`__main__.py` is mandatory** — `zipapp` refuses with `ZipAppError("Archive has no entry point")` without it (the `-m module:fn` form is the only alternative; the shipped `cli/__main__.py` source is cleaner and testable). Files staged: `coreci_broker.py`, `coreci_broker_client.py`, `coreci_broker_ui/**`, `cli/__main__.py`, generated `_cli_version.py`.
- **Compression: `zipapp` defaults to ZIP_STORED** (stdlib source line 136: `zipfile.ZIP_DEFLATED if compressed else zipfile.ZIP_STORED`). `--compress` is required for DEFLATED. Measured: ~95 KB content → 27.6 KB pyz compressed vs ~95 KB stored. Use `--compress`.
- **Shebang + exec bit**: with `--python`, zipapp writes `#!/usr/bin/env python3\n` as a literal first line, then the zip (zip readers locate EOCD from the file end — leading bytes tolerated), and `chmod +x`es the target (`target.chmod(... | S_IEXEC)`). Both `./coreci-broker.pyz` and `python3 coreci-broker.pyz` verified working; `sys.argv[0]` = the archive path as invoked; `python3 -m zipapp --info` reads the interpreter back.
- **Data files are NOT `open()`-able inside the archive** (verified `FileNotFoundError` for `assets/page.html`): zipimport only serves `.py` imports; the broker serves UI/static via **filesystem paths + `open()`** → asset materialization to disk at boot is *mandatory*, not optional. `importlib.resources` cannot provide a directory for `BROKER_UI_ROOT` — zipfile extraction is the only design that fits the broker's config contract. (Not a close call.)
- **`__main__` module collision — none**: the archive's `__main__.py` IS the `__main__` module; `import coreci_broker` inside it executes the broker's module-level code once and its `if __name__ == "__main__"` block does NOT fire (verified: side-effect counter runs once, standalone print block silent). Call `coreci_broker.run_standalone()` — no re-execution of module code, no subprocess re-invocation needed.
- **zipimport limits**: none bite here — the archive contains only pure-Python repo files; stdlib (`sqlite3`, `http.server`, C extensions) resolves from the host interpreter, and the broker's sole optional import (bcrypt) stays optional.
### 1.2 Boot shim design (prototype built, booted, verified — recommended as-is)
`cli/__main__.py` (staged into the pyz), five steps:
1. **`--version` first, before importing the broker**: parse `sys.argv[1:]`; if `--version`, `import _cli_version; print(_cli_version.VERSION); sys.exit(0)`. Version comes from a build-time-generated `_cli_version.py` (`VERSION = "vX.Y.Z"` written by `scripts/build_cli.py` from the tag) — chosen over a zip-comment (opaque, fragile, still needs a parser). No broker import on the `--version` path.
2. **Resolve the state dir exactly like the broker**: `env_path = os.environ.get("BROKER_ENV_PATH") or ".env"`; `state_dir = dirname(abspath(env_path)) or cwd`. Broker path resolution (read from code, authoritative): `BROKER_ENV_PATH` default `.env`, `BROKER_DB_PATH` `broker.db`, `BROKER_AUDIT_LOG_PATH` `broker.audit.log`, `BROKER_CREDS_FILE` `data/creds.json`, `BROKER_UI_ROOT` `coreci_broker_ui`, `BROKER_STATIC_ROOT` `.` — **all CWD-relative when relative**; process env beats `.env` beats defaults. The CLI therefore needs no path changes at all: relative paths land in the CWD (docker: `WORKDIR /state` + volume).
3. **Materialize assets**: `zipfile.ZipFile(sys.path[0])` (inside a zipapp, `sys.path[0]` is the `.pyz` itself — verified) → extract `coreci_broker_ui/**` → `<state>/.coreci_broker_assets/<VERSION>/ui/` and `coreci_broker_client.py` → `<...>/static/`. Write temp + `os.replace` (atomic vs a concurrently-serving older instance). **Versioned asset dir**: upgrades materialize a new dir; old dirs are left in place (another instance may still serve them; ~30 KB each — cleanup explicitly NOT done in v0.2). Overwrite-same-version on every boot (self-repair, assets are immutable inputs; extraction happens before this instance's server starts).
4. **Override precedence — user wins**: `import coreci_broker`; parse the user `.env` with the broker's own `coreci_broker._parse_env_file(env_path)`; set `os.environ["BROKER_UI_ROOT"]` / `["BROKER_STATIC_ROOT"]` only when the key is in **neither** `os.environ` nor the `.env` values. Net precedence: process env → `.env` → shim defaults → broker defaults — identical to the broker's own contract, achieved with **zero broker changes**.
5. **Hand off**: `coreci_broker.run_standalone()` (foreground loop + KeyboardInterrupt shutdown; prints `coreci-broker listening on http://<host>:<port>` with the *actual bound address* — `BROKER_BIND=127.0.0.1:0` gives an ephemeral port via `_parse_bind` (`"0".isdigit()` → True) and the harness parses the printed URL).
**Verified end-to-end prototype results** (real broker sources, subprocess, urllib): first boot printed `BROKER_ADMIN_PASS=…` exactly once then the listening line; `/healthz` 200; `/ui/login` + `/ui/dashboard` 200 (served from materialized assets); `/static/coreci_broker_client.py` 200 byte-exact (6180 B); `.env` + `broker.db` + `broker.audit.log` created mode 0600 in the state dir; second boot silent (no pass re-print); state dir fully independent of CWD (env paths pin it).
### 1.3 Determinism / reproducibility
- **Raw `zipapp` is NOT reliably byte-reproducible**: zip entries carry the source files' mtimes (`z.write` → `ZipInfo.from_file`), no SOURCE_DATE_EPOCH handling exists anywhere in zipapp, and entry order is `rglob('*')` = `os.scandir` order (unsorted, filesystem-dependent). My two identical staging-dir builds hashed equal *only because* mtimes and scandir order happened to match — a fresh `git clone` (checkout-time mtimes) or a different filesystem breaks it.
- **Fix — normalization post-pass (verified byte-identical across rebuilds)**: after zipapp builds the artifact, the build script reads it (shebang line + zip body), rewrites the zip with entries **sorted by name**, `date_time=(1980,1,1,0,0,0)` (zip epoch minimum), `ZIP_DEFLATED`, preserved `external_attr`, and re-prepends the shebang (+`chmod +x`). Verified: two builds → identical sha256; normalized pyz still executes (`--version` OK). Alternative (rejected): `os.utime`-ing the staging tree pre-build — fixes mtimes but not entry order; the post-pass fixes both and keeps `python3 -m zipapp` as the format mechanism (the normalized file is still "shebang + zip", the zipapp format).
- Reproducibility test: `test_build_cli.py` builds twice into separate dirs, asserts identical sha256, and asserts the embedded `_cli_version.VERSION`.
### 1.4 Defects / risks (zipapp)
- **RISK** — stale-asset drift if a *user* sets `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` in `.env`: honored (by design, documented in notes) — the archive still carries the assets, they're just unused. Tests cover the override path.
- **RISK** — first-boot extraction torn reads if two instances share a state dir and the same version: mitigated by temp+`os.replace`; multi-instance sharing one SQLite remains unsupported (as v0.1).
- **NOTE** — shebang `env python3` may resolve to a non-3.11 python on exotic hosts; the broker needs 3.7+ features in practice but is tested on 3.11 only — release notes state "Python 3.11+ required" (D-022 alignment).
- **NOTE** — `sys.path[0]` is the archive only when run as a zipapp; the shim is not meant to run from an unzipped checkout (guard: it can `os.path.isdir` check and fall back to repo-relative roots, or simply document zipapp-only — planner's choice, zero-cost either way).
- **NOTE** — assets land under a dot-directory (`.coreci_broker_assets`) — hidden-dir hygiene in the state dir; broker dot-filtering applies to request paths, not roots (verified `_resolve_static_file`).
- **NOTE** — AC14 (line count) counts `coreci_broker.py` ONLY (REQ-17); the shim/build scripts are separate files → v0.2 needs **zero broker lines**; broker is 1390/1400 — only 10 lines of headroom anyway, so zero-change is also a hard constraint, and it is satisfied (verified: `run_standalone`, `_parse_env_file`, `boot` are importable and sufficient as-is).
## 2. Gitea Release Assets (REQ-22)
### 2.1 Verified API shapes (live gitea 1.26.1, `git.coreci.dev/swagger.v1.json` + working calls)
- **List**: `GET /api/v1/repos/coreci/broker/releases?limit=20` → `[{id, tag_name, name, body, assets: [{id, name, size, download_count, browser_download_url}], draft, prerelease, created_at}]`. **By tag**: `GET /api/v1/repos/coreci/broker/releases/tags/{tag}` (200 or 404). **Latest**: `GET .../releases/latest`.
- **Create**: `POST /api/v1/repos/{owner}/{repo}/releases` — JSON `CreateReleaseOption` `{tag_name, name, body, draft, prerelease, target_commitish, tag_message}` (schema verified from swagger).
- **Attach asset**: `POST /api/v1/repos/{owner}/{repo}/releases/{id}/assets` — **consumes `multipart/form-data`** (or `application/octet-stream`); params verified from swagger: `id` (path, int64), `name` (query), **`attachment` (formData, type file)** — the multipart field name is `attachment`. Response: attachment JSON with `id`, `name`, `size`, `browser_download_url`.
- **Auth**: `Authorization: token <GITEA_TOKEN>` (verified working on live forge; gitea also accepts Basic `user:token` — use the `token` form).
- **Asset size**: gitea `[attachment] MAX_SIZE` default is 4 MB — the pyz is ~30 KB (deflated) → no issue; NOTE if server policy changes, attachment failure escalates through the bounded-retry path rather than blocking.
- **Multipart with stdlib** (no `requests`): hand-rolled body, ~15 lines — boundary string, `Content-Disposition: form-data; name="attachment"; filename="coreci-broker-vX.Y.Z.pyz"`, `Content-Type: application/octet-stream`, `?name=` query param; gitea parses with standard Go multipart — boundary is ours to choose. Test against a stub gitea (§2.4).
### 2.2 v0.1 attachment forensics (how it was actually done)
- All six releases v0.1.0..v0.1.5 carry `broker-vX.Y.Z.tar.gz` (29 KB → 99 KB, growing with the codebase). Downloaded v0.1.5's asset: **57 members = the complete working tree including `.ciagent/`** (53 git-tracked files + 4 dir entries; matches `git ls-tree -r v0.1.5` exactly, plus directory entries), **no top-level prefix**, uniform `mtime 1791646214` (= the release timestamp), `uid 0/uname root`, modes 0664/0775.
- Interpretation: the archive was produced by the **ship harness's packaging infrastructure** (ship.md step "Create distribution packages" via the `coreci build/package/release upload` CLI — **not installed in this dev environment**; the root-owned uniform timestamps indicate a container-side packaging step) — i.e. **ad-hoc from the repo's perspective: no repo-owned script exists**. The only repo-visible traces are ship.md's generic instructions.
- Conclusion for v0.2: REQ-22 formalizes exactly this — a repo-owned `scripts/release.py` (plus `scripts/build_cli.py`) replaces the absent harness path. The v0.1 *conventions* carry over: per-release asset naming (now `coreci-broker-vX.Y.Z.pyz`, D-020), notes model (§2.3), and the bounded-retry contract (ship.md). The existing source-archive `broker-vX.Y.Z.tar.gz` convention continues to be the harness's own product (best-effort); the repo script owns the `.pyz` (+ digest).
### 2.3 Release notes content model (v0.1.5 body verified; v0.2 sections)
v0.1.5 body sections: `Milestone summary` → `Phases` (P0..P5 with tags) → `Verification` (test counts, security review, audit) → `Deployment`. v0.2 releases add the REQ-22-mandated CLI + docker documentation; recommended body structure (build-able by `scripts/release.py` from PLAN/ROADMAP/requirements data):
1. `Milestone summary` — one paragraph (carries the v0.1 style).
2. `Phases` — phase list with tags (auto from `git log vX.Y.Z-1..vX.Y.Z`/ROADMAP).
3. `Verification` — test count, security review verdict, escalations.
4. `CLI install + usage` — download `coreci-broker-vX.Y.Z.pyz` from this release's assets; `chmod +x` then `./coreci-broker-vX.Y.Z.pyz` or `python3 coreci-broker-vX.Y.Z.pyz`; `--version`; env contract table pointer (README); first-boot pass discipline (printed once, 0600 state files); state-dir anchoring (`BROKER_ENV_PATH`); **sha256 of the .pyz embedded** (plus optional `.sha256` sidecar asset — planner's call; lean default: digest in notes only).
5. `Docker usage` (REQ-23/D-021) — `docker pull git.coreci.dev/coreci/broker:vX.Y.Z` (and `latest`); `docker run -p 8765:8765 -v broker-state:/state …`; state-on-volume contract.
### 2.4 `scripts/release.py` design (stdlib, ~200 lines)
- **Secrets**: token via a shared file-reading loader `scripts/_env_secrets.py` (~20 lines: parse `.ciagent/.env.secrets` `KEY=VALUE`, strip quotes — mirror the broker's `_parse_env_file` semantics). NEVER `os.environ` (GITEA_TOKEN is in `blocked_env_vars`). **Assert token non-empty before any API call** (v1.8 lesson: empty token → gitea 404, masking auth failure).
- Flow: (1) resolve tag → `GET /releases/tags/{tag}`; 404 → `POST /releases` (create with notes body, `draft: false`, `prerelease: false`); (2) `GET .../releases/{id}/assets` — if `coreci-broker-vX.Y.Z.pyz` already present with matching size → **idempotent success** (no duplicate asset on bounded-retry re-run); (3) POST multipart attach (field `attachment`, `?name=`); (4) bounded retry: 3 attempts with backoff on URLError/5xx/HTTPError; on exhaustion → exit non-zero with a machine-readable marker → ship harness commits `release_pending` escalation; **never blocks tag+merge**.
- **Stub-test** (`tests/test_release_upload.py`): `_StubGiteaHandler(BaseHTTPRequestHandler)` on `ThreadingHTTPServer(("127.0.0.1", 0))` (v0.1 `_StubOwuiHandler` precedent): routes `GET /api/v1/repos/{o}/{r}/releases/tags/{tag}` (200 fixture / 404), `POST .../releases` (assert body JSON), `POST .../releases/{id}/assets` (parse multipart with a stdlib parser — e.g. `email` or manual boundary split — assert field name `attachment` + `?name=` + body bytes), plus failure-modes: 500×3 → release_pending path; existing asset → no second upload.
### 2.5 Defects / risks (release assets)
- **RISK** — multipart hand-rolling bugs (boundary quoting, CRLF): mitigated by stub-gitea test with byte-exact assertions.
- **RISK** — release may not exist when the attacher runs (ship harness creates it best-effort): create-or-get flow covers both orders; idempotent re-attach covers partial failure.
- **NOTE** — 4 MB attachment cap (default) — pyz far under; failure would surface through bounded-retry, not silently.
- **NOTE** — `GET /releases?limit=50` paging: prefer `/releases/tags/{tag}` (exact, no paging).
## 3. Daemonless OCI Build + Push (REQ-23, D-019/D-021/D-022)
### 3.1 PULL from Docker Hub (verified live, anonymous, end-to-end)
- `GET https://registry-1.docker.io/v2/` → 401 `WWW-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"`; anonymous rate limit header `ratelimit-limit: 100;w=3600` (100 req/h) — ample for once-per-milestone.
- `GET https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/python:pull` → 200 `{token, access_token, expires_in, issued_at}` — anonymous OK (verified).
- `GET /v2/library/python/manifests/3.11-slim` with `Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json` → **200 `Content-Type: application/vnd.oci.image.index.v1+json` — `python:3.11-slim` IS an OCI image index (16 children: 8 platform manifests + 8 attestation manifests `unknown/unknown`)**. The client MUST resolve the `linux/amd64` child manifest by digest and fetch it separately.
- Verified nuance: Docker Hub currently returns the index **even without Accept** (and even Accepting only schema2) — but the client still sends the full Accept set (contract correctness) and branches on the response `"manifests"` key (index) vs `"layers"` (manifest), not on hope.
- amd64 child (verified, digest drifts upstream — resolve at publish time and record it): `application/vnd.oci.image.manifest.v1+json`, config `application/vnd.oci.image.config.v1+json` (`Cmd: ["python3"]`, Env PATH/LANG/GPG_KEY/`PYTHON_VERSION=3.11.17`/PYTHON_SHA256 — base runs 3.11.17 vs dev 3.11.2, same pinned minor line per D-022), **4 layers** `application/vnd.oci.image.layer.v1.tar+gzip` (≈29.8 MB + 1.3 MB + 14.5 MB + 250 B ≈ 45.7 MB total).
- `GET /v2/library/python/blobs/<digest>` → blob bytes (layers + config). All flows urllib-only.
### 3.2 Auth to the gitea registry (verified live; username+token shapes pinned)
- `GET https://git.coreci.dev/v2/` → 401 `WWW-Authenticate: Bearer realm="https://git.coreci.dev/v2/token",service="container_registry",scope="*"`.
- Token endpoint: `GET https://git.coreci.dev/v2/token?service=container_registry&scope=repository:coreci/broker:pull,push` with **`Authorization: Basic base64(<username>:<GITEA_TOKEN>)`** → 200 `{"token": "<JWT>"}` (HS256; `exp-nbf = 86400` s = 24 h).
- **Verified**: the Basic **username is ignored** — `coreci`, `nosuchuser123`, and empty all return the same JWT (UserID 57, scopes `write:activitypub,…,write:package,…` — the existing GITEA_TOKEN has package-write). The token (as password) identifies the user. Docker login convention: username = the gitea user, password = the token; the client sends any non-empty username.
- **Verified quirk (must handle)**: the token endpoint answers **200 even anonymously** with an unusable JWT (`UserID: -1`, empty scope). A 200 from `/v2/token` proves nothing about push rights — the client must (a) insist on Basic credentials being present, (b) check the issued token is non-empty, and let real operations fail loudly.
- Verified: `Authorization: Bearer <JWT>` on `/v2/` → 200; `HEAD /v2/coreci/broker/blobs/sha256:0…0` → 404 (existence checks work); `GET /v2/coreci/broker/tags/list` → 404 `NAME_UNKNOWN` (the container package doesn't exist until first push — expected today).
### 3.3 PUSH flow + media-type choice
- **Manifest media type: OCI image manifest end-to-end** — `application/vnd.oci.image.manifest.v1+json`, config `application/vnd.oci.image.config.v1+json`, layers `application/vnd.oci.image.layer.v1.tar+gzip`. Rationale: (1) the pulled base child manifest already uses exactly these descriptor types — reusing its 4 layer digests **verbatim** keeps descriptor/mediaType/blob consistency; (2) gitea 1.26 accepts docker schema2 and OCI manifests — OCI is the modern default and what upstream itself serves; (3) Docker CLI ≥ 18.06 and podman/buildah pull OCI manifests from any v2 registry. **Do NOT push an index/manifest-list**: one linux/amd64 manifest per tag; docker pulls it directly.
- **Blob uploads (OCI dist-spec, gitea-conformant)**: primary = **monolithic** `POST /v2/coreci/broker/blobs/uploads/?digest=sha256:<hex>` with the blob as body → 201 (single round trip); fallback = two-step `POST /v2/<name>/blobs/uploads/` → 202 + `Location` → `PUT <Location>?digest=…` → 201. Always `HEAD /v2/<name>/blobs/<digest>` first — 200 → skip upload (base layers re-run-proof: a partially-failed publish resumes cheaply). **Base layers must be re-uploaded to gitea** (~45.7 MB once per milestone) — gitea cannot mount blobs from Docker Hub; acceptable per D-019's once-per-milestone cadence.
- **Redirect trap (verified in CPython 3.11 source)**: `HTTPRedirectHandler.redirect_request` only follows GET/HEAD (301/302/303 POST→GET); **307/308 on a PUT/POST raise HTTPError** — stdlib urllib does NOT follow method-preserving redirects. The client must implement explicit Location-following (read `Location`, `urljoin` against the registry base — gitea's `Location` may be relative or absolute — re-issue the request with method+body). Gitea default local storage answers directly, but the client is correct against any conformance-grade registry.
- **Manifest PUT**: `PUT /v2/coreci/broker/manifests/<tag>` with `Content-Type: application/vnd.oci.image.manifest.v1+json` → 201. Tags: `<milestone-release-tag>` (e.g. `v0.2.3`) **and** `latest` — two PUTs of the same manifest (D-021).
### 3.4 Layer + config + manifest construction (exact recipe)
1. **Broker layer**: deterministic tar (`tarfile`, entries sorted, fixed `TarInfo` metadata — uid/gid 0, uname/gname fixed, `mtime=0`/1980-epoch, file mode 0644) containing `/app/coreci-broker.pyz` (the built artifact). Gzip with `gzip.compress(data, compresslevel=9, mtime=0)` — **`mtime=0` is mandatory** (default `None` = wall clock → nondeterministic digests); fix the compresslevel in code (stable per build).
2. Digests: **layer digest** = sha256 of the gzip blob; **diff_id** = sha256 of the *uncompressed* tar.
3. **New config blob**: copy the pulled base config; append the new diff_id to `rootfs.diff_ids`; set `config.Entrypoint = ["python3", "/app/coreci-broker.pyz"]`, `config.Cmd = []` (so `docker run image --version` appends `--version` through the entrypoint), `config.WorkingDir = "/state"`, keep base `Env`; optional `config.Labels = {"org.opencontainers.image.version": "<tag>"}`. Serialize deterministically (`json.dumps(..., sort_keys=True, separators=(",", ":"))`) → reproducible config digest.
4. **Push order**: HEAD all blobs (4 base layers, new layer, new config) → monolithic POST each missing one → PUT manifest referencing the base's 4 layer descriptors **verbatim** (original digests + mediaTypes) + the new layer descriptor + the new config descriptor. Record the resolved base index digest + amd64 child digest + all blob digests in the release notes (provenance; also `--base-manifest` override argument for exact re-publishes, default = tag resolution with documented upstream drift).
5. Image name: `git.coreci.dev/coreci/broker` (gitea registry = `owner/name`; the package links to the repo by owner+name). gitea creates the package at first push (verified `NAME_UNKNOWN` until then).
### 3.5 Dockerfile role + mapping limits (documentation; daemonless client = source of truth)
- The Dockerfile is for users WITH docker; the daemonless client is what the pipeline publishes. **Equivalence required, byte-identity not.** Self-contained Dockerfile (no pre-built artifact in build context): `FROM python:3.11-slim`; `COPY coreci_broker.py coreci_broker_client.py coreci_broker_ui/ cli/ /app/src/`; `ARG VERSION=dev`; generated `_cli_version.py` via build arg; `RUN python3 -m zipapp /app/src -o /app/coreci-broker.pyz --python "/usr/bin/env python3" --compress`; `WORKDIR /state`; `ENTRYPOINT ["python3", "/app/coreci-broker.pyz"]`; `EXPOSE 8765`.
- Mapping: same base family (tag drift upstream ≠ the pinned digests the client resolved at publish time), same entrypoint, same pyz content, same `/state` state anchoring; differences: docker-build layering (COPY+RUN blobs vs verbatim base layers + one broker layer) and version string provenance (build-arg vs build-time injection). Document both in the Dockerfile header.
### 3.6 Stub-test plan (`tests/test_publish_image.py`; v0.1 stub precedent)
- `_StubRegistryHandler` on an ephemeral port: `GET /v2/` → 401 + `WWW-Authenticate` (realm = stub `/token`); `GET /token` → `{"token":"stub"}` (plus the anonymous-empty-token variant → client must fail cleanly); `GET /v2/<name>/manifests/<ref>` → fixture index (Accept-checked) / child manifest; `GET /v2/<name>/blobs/<digest>` → fixture bytes; `HEAD` blob → 200/404; `POST /v2/<name>/blobs/uploads/` (+`?digest=`) → 201 monolithic (assert body sha256 == digest param) or 202+`Location`; `PUT .../uploads/<uuid>?digest=` → 201; `PUT /v2/<name>/manifests/<tag>` → 201 (assert `Content-Type` == OCI manifest, all referenced digests already uploaded, diff_ids count == layers count).
- Real-manifest fixture: tiny synthetic OCI index/manifest/config/1-2 tiny layers built in-test with the same tar+gzip code (NOT the 45 MB real base — suite stays fast); plus one "live-shape" assertion test using the digests/mediaTypes captured in §3.1 as constants.
- End-to-end offline: stub Docker Hub + stub gitea → run the publish flow → assert both tags PUT, base layer bytes passed through, broker layer contains the pyz at `/app/`.
- Redirect edge: stub answers 307 on the two-step upload PUT → client follows Location explicitly (urllib would raise otherwise).
### 3.7 Defects / risks (daemonless)
- **RISK** — upstream tag drift (`3.11-slim` moves): pin at publish time by recording digests + `--base-manifest` override; notes carry provenance.
- **RISK** — 307/308 not followed by urllib (verified): explicit Location-following required; stub-tested.
- **RISK** — ~46 MB upload per milestone (base layers re-uploaded; no cross-registry mount): bounded, once-per-milestone; HEAD-skip makes retries cheap.
- **RISK** — gitea's anonymous-200 token quirk (verified): empty-scope JWT must not be treated as auth success.
- **NOTE** — monolithic upload limits: gitea handles multi-MB bodies; the config/manifest are KB-scale, the largest blob is the 29.8 MB base layer — fine for monolithic POST (dist-spec; gitea conformance-tested upstream).
- **NOTE** — publish failure follows the same bounded-retry → `release_pending` escalation as releases; never blocks the milestone merge (ROADMAP P3).
## 4. Cross-Cutting: Layout, Security, Line Count
### 4.1 Recommended repo layout (new files only)
```
cli/__main__.py # boot shim (staged into the pyz; NOT coreci_broker.py → outside AC14)
scripts/build_cli.py # stage + zipapp + normalize; emits dist/coreci-broker-vX.Y.Z.pyz (+ sha256)
scripts/release.py # notes builder + create-or-get release + asset attach + bounded retry
scripts/publish_image.py # daemonless registry client (pull base → append layer → push, D-019)
scripts/_env_secrets.py # shared .env-file secrets loader (~20 lines, one discipline point)
Dockerfile # documentation + docker-user path (§3.5)
dist/ # build output (gitignored)
tests/test_cli_packaging.py # build in tmp + boot subprocess + boot-parity flows via urllib
tests/test_build_cli.py # reproducibility: two builds → identical sha256; --version wiring
tests/test_release_upload.py # stub-gitea attachment + retry/escalation paths
tests/test_publish_image.py # stub-registry pull/push flows + manifest assembly
```
- `scripts/_env_secrets.py` shared rather than copy-pasted per script: one place asserts the "file-only, non-empty" token discipline; `python3 scripts/release.py` runs with `sys.path[0] == scripts/` so the sibling import works without path setup.
- **.gitignore additions**: `dist/`, `*.pyz` (root-level strays). Existing patterns already cover `__pycache__`, `*.db`, `*.log`, `.env*`, `data/`.
- Phase mapping (per ROADMAP): Phase 1 = `cli/` + `scripts/build_cli.py` + its tests (REQ-21); Phase 2 = `release.py` + `publish_image.py` + `Dockerfile` + stub tests (REQ-22/23).
### 4.2 Broker changes: ZERO (verified)
- `run_standalone()`, `boot()`, `load_config()`, `_parse_env_file()` are module-level and importable; the shim + scripts need nothing added to the broker. `coreci_broker.py` = 1390/1400 lines (D-018 cap) — v0.2 adds no lines; AC14 stays green by construction.
### 4.3 Test-harness conventions (carried v0.1 §6 + new packaging patterns)
- Packaging test pattern (verified in prototype): `TemporaryDirectory` → `build_cli.build(state=tmp)` → subprocess `Popen(["python3", "<pyz>"], env={BROKER_BIND: "127.0.0.1:0", BROKER_ENV_PATH/DB/AUDIT: tmp paths}, stdout=PIPE)` → read stdout lines until `coreci-broker listening on http://…` (parse base URL; the `BROKER_ADMIN_PASS=` line comes first and is captured for login) → drive `urllib` flows (healthz, login, register, secret read — reuse v0.1 acceptance request patterns) → `proc.terminate()` in cleanup. Boot cost ~1–2 s; suite impact acceptable (v0.1 suite ≈ 2 min).
- Stub servers: extend the `_StubOwuiHandler` pattern (class-level fixtures + hit recording + `addCleanup`) to `_StubGiteaHandler` / `_StubRegistryHandler`; all stubs bind `("127.0.0.1", 0)` and the code under test takes a base URL parameter (no hardcoded hosts in scripts — required for stubbing anyway).
### 4.4 Security posture (v0.2-specific)
- **No secrets in the artifact**: the pyz bundles broker + UI + client only; the client module contains no credentials; the broker generates pepper/admin pass at first boot; `.env` is never inside the archive. Same for the image: no state baked in (`WORKDIR /state` empty at build; first boot generates).
- **Token discipline**: `release.py` / `publish_image.py` read `GITEA_TOKEN` ONLY via the `.env*` file loader (never shell env — structurally blocked by `blocked_env_vars`); assert non-empty before first use (v1.8: empty token → gitea 404 masks auth failure). Stubs use fake tokens; no live-forge writes in tests.
- **Artifact integrity**: sha256 of the `.pyz` computed at build, embedded in release notes; reproducible build makes the digest meaningful across rebuilds.
- **Static disclosure**: v0.1's `_broker_protected_paths` behavior extends unchanged to materialized roots (assets contain no secrets; `.env`/db/audit realpaths remain protected wherever the state dir sits).
## 5. Research Conclusions (planner-ready)
- **Phase 1 (REQ-21)**: `cli/__main__.py` five-step shim (§1.2 — prototype verified end-to-end, lift it), `scripts/build_cli.py` (stage → zipapp `--compress` → normalize post-pass §1.3 → sha256), `tests/test_cli_packaging.py` + `tests/test_build_cli.py`. Zero broker lines.
- **Phase 2 (REQ-22/23)**: `scripts/_env_secrets.py` + `release.py` (create-or-get by tag, idempotent attach, bounded retry → release_pending, notes model §2.3); `scripts/publish_image.py` (§3.1–3.4 flows; OCI manifest end-to-end; monolithic uploads + HEAD skip + explicit Location-following); `Dockerfile` (§3.5); stub tests §2.4/§3.6.
- **Phase 3 (acceptance)**: boot-parity matrix through the artifact (first-boot pass once, 0600 modes, silent second boot, UI + client served, register/read flows), reproducibility sha256, stub-releasability of both pipelines, security review (no secrets in artifact/image; token discipline), then milestone ship: build → attach to `v0.2.3` → publish image tags `v0.2.3` + `latest` → merge.
- **No BLOCKERs found.** All environment facts were verified live (registry auth shapes, index resolution, gitea API, zipapp mechanics) and the packaging design is proven by a working prototype against the real broker sources. Remaining open items are NOTE/RISK grade and owned by the plan: sidecar `.sha256` asset vs digest-in-notes; shim fallback when run unzipped (trivial); `latest`-tag manifest PUT ordering (push version tag first, `latest` second — trivial).
+45
View File
@@ -0,0 +1,45 @@
# Roadmap — Milestone v0.2 (cli-packaging)
## Overview
One milestone, owner-directed 2026-10-10: package the completed v0.1 broker for distribution. **Milestone v0.2 — cli-packaging** (feature type): the broker as a single-file zipapp CLI (`coreci-broker.pyz`, self-contained), the CLI attached to every gitea release alongside release notes, and a docker container built **once per milestone** and published as a package in the gitea container registry. Scope is CLI + distribution only — no functional changes to the broker (REQ-21..REQ-23). Docker publication starts with v0.2; no retroactive v0.1 image.
Versioning (greenfield convention, D-008): the milestone branch is `milestone/v0.2-cli-packaging`; phase tags run on the milestone's own line — phase 0 ships `v0.2.0`, execution phases ship `v0.2.1`..`v0.2.2`, and the final phase's patch (`v0.2.3`) IS the milestone release; no separate minor tag. MAJOR remains 0 until an owner-approved v1 release. Ships go to gitea (`git.coreci.dev/coreci/broker`); releases are best-effort with bounded retry, never blocking tag+merge.
Autonomy: **guided** — escalate on every decision gate; threshold 0.85; 1 revision iteration; clarify budget 20.
## Phase 0 — Pre-Execution
- **Description**: Specification parsing (REQ-21..REQ-23), clarification (packaging/distribution decisions D-019..D-022), research (zipapp mechanics, gitea release-asset API, gitea container registry + OCI registry HTTP API v2, Docker Hub anonymous pulls), plan (task-level breakdown with waves), grill, MVP/UX check. `/ci-run` phase 0 completes REQUIREMENTS.md v2 section, RESEARCH.md, PLAN.md, PERSONAS.md, and the grill verdict before any code.
- **Requirements covered**: none (planning artifacts)
- **Success criteria**: REQ-21..REQ-23 defined and traceable; milestone v0.2 established in config.json and PROJECT.md; D-016+ decisions recorded; PLAN.md + PERSONAS.md committed; grill verdict recorded; MVP/UX sections present; phase tagged `v0.2.0`
- **Depends on**: —
- **Status**: complete (shipped as `v0.2.0`, release id 970)
## Phase 1 — CLI Packaging + Release Tooling
- **Description**: `coreci-broker.pyz` build pipeline: stdlib `python3 -m zipapp` build script (no pip) with G-009 normalize pass (sorted entries, fixed timestamps, constant external_attr); self-contained archive bundling `coreci_broker.py` + `coreci_broker_ui/` + `coreci_broker_client.py` with asset materialization at boot (env-resolved roots, never `__file__`); standalone boot parity with v0.1 (first-boot `.env` generation, admin pass printed once, 0600 modes, full admin UI + client module served); `--version` flag; release-asset automation (G-011: `scripts/_env_secrets.py` + `scripts/release.py` land here so the v0.2.1 ship attach is script-driven — no ad-hoc bridge); tests boot the built artifact end-to-end and stub-verify the gitea upload.
- **Requirements covered**: REQ-21, REQ-22
- **Success criteria**: artifact builds reproducibly from a clean checkout (two consecutive same-environment builds → byte-identical sha256); `python3 coreci-broker.pyz` boots with behavior byte-identical to `python3 coreci_broker.py` (v0.1 suite green through the packaged path); UI pages + client module served from the archive; first-boot semantics preserved on second boot; the `v0.2.1` release carries the `.pyz` attached by `scripts/release.py` (D-020 starts here; `v0.2.0` carries none — G-021)
- **Depends on**: 0
- **Status**: complete (shipped as `v0.2.1` — 325 cumulative tests green, verify four-layer PASS, .pyz attached by script, asset id 396)
## Phase 2 — Docker + Image Publish
- **Description**: Daemonless publish client per D-019 (once-per-milestone pull pinned base layers / append broker layer / push to `git.coreci.dev/coreci/broker` with version + `latest` tags, D-021/D-022; G-019 wire hardening: per-request timeouts, blob retry with HEAD-skip, explicit 307/308 relative-Location handling); `Dockerfile` + `.dockerignore` (G-015) + consistency-tested equivalence mapping (G-016); README additive section (CLI + docker usage); the `v0.2.2` release attaches the `.pyz` via `scripts/release.py` with CLI + docker notes.
- **Requirements covered**: REQ-23 (+ REQ-21/REQ-22 docs)
- **Success criteria**: daemonless publish client stub-verified end-to-end (no live-forge writes, no container runtime); Dockerfile/.dockerignore consistency tests green; this phase's release (`v0.2.2`) carries the `.pyz` attached alongside the source archive with notes; bounded-retry → `release_pending` paths covered by tests
- **Depends on**: 1
- **Status**: complete (shipped as `v0.2.2` — 341 cumulative tests green, verify four-layer PASS, .pyz attached by script, asset id 397)
## Phase 3 — Final Review + Ship
- **Description**: Full acceptance: cumulative v0.1 suite green plus v0.2 packaging acceptance (boot parity through the artifact, self-containment, daemonless publish flow); security review (artifact integrity, no secrets in the archive/image, token discipline in upload paths, build-context hygiene); multi-persona review of the milestone diff; audit/reconstruction (REQ-21..23 traceable); milestone ship in the G-012-corrected order — build CLI → squash-merge `milestone/v0.2-cli-packaging` → `main` → tag `v0.2.3` → push → G-013 live smoke (read-only, env-gated) → attach `.pyz` + full notes via `scripts/release.py` → daemonless image publish once (D-019..D-022, with the G-020 closure runbook on failure) → branch cleanup → closeout.
- **Requirements covered**: REQ-21, REQ-22, REQ-23 (acceptance)
- **Success criteria**: all v0.1 + v0.2 acceptance green; security review PASS; `v0.2.3` on `main` with the `.pyz` attached + notes on the gitea release; docker image published as a gitea package (or escalated per the bounded-retry contract + closure runbook, never blocking merge); REQUIREMENTS/ROADMAP closed out; checkpoint cleared
- **Depends on**: 2
- **Status**: complete (shipped as `v0.2.3` — milestone release; 356 cumulative tests green, security review PASS-WITH-FINDINGS remediated, `.pyz` attached + notes, image published once)
## Post-Milestone Notes (carry-forward from v0.1, GRILL G-008)
- **90-day adoption criterion**: `proxmox_readonly_tool.py` runs on `coreci_broker_client.py` with direct KB reads removed by **2027-01-09**; the owner owns the date. No real tool migrates to the broker before the owner-run post-ship OWUI smoke checklist (G-005) passes on the deployed v0.11.3 host. These v0.1 post-ship obligations remain open after the v0.1 ship (2026-10-10) and carry forward unchanged.
+54
View File
@@ -0,0 +1,54 @@
{
"version": "0.8.0",
"autonomy": {
"level": "guided",
"decision_confidence_threshold": 0.85,
"escalation_hooks": ["*"],
"max_revision_iterations": 1,
"clarify_budget": 20
},
"release": {
"forge": "gitea",
"base_url": "https://git.coreci.dev",
"owner": "coreci",
"repo": "broker"
},
"secrets": {
"scopes": ["gitea"],
"env_files": [".env", ".env.secrets", ".env.*"]
},
"ship": {
"per_phase": true,
"allow_skip": false,
"max_release_retries": 3,
"release_blocking": false
},
"verification": {
"bdd_default": false
},
"personas": {
"enabled": true,
"territory_enforcement": "warn"
},
"parallelization": {
"enabled": false,
"max_concurrent_agents": 1
},
"ideation": {
"max_ideas": 20,
"categories": ["security", "quality", "architecture", "coverage", "improvement", "spec", "chaos", "bdd"]
},
"security": {
"bash_allowlist": {
"blocked_env_vars": ["GITEA_TOKEN", "GITHUB_TOKEN", "GITLAB_TOKEN"]
}
},
"projects": [],
"active_project": null,
"milestone": {
"version": "v0.2",
"name": "cli-packaging",
"type": "feature",
"branch": "milestone/v0.2-cli-packaging"
}
}
+17
View File
@@ -0,0 +1,17 @@
# Build-context hygiene (G-015): no secrets, state, or build artifacts
# may enter an image build context.
.git
.env
.env.*
data/
*.db
*.log
dist/
__pycache__/
tests/
.ciagent/
# Nested matches too (P3-03, CWE-693): the root-anchored `__pycache__/`
# above misses cli/__pycache__/*.pyc — build artifacts must stay out of
# the context wherever they sit in the tree.
**/__pycache__/
**/*.pyc
+17
View File
@@ -0,0 +1,17 @@
# Secrets (ciagent credential discipline)
.env
.env.secrets
.env.*
# Python
__pycache__/
*.pyc
# Broker runtime state
data/
*.db
*.log
# v0.2 build artifacts
dist/
*.pyz
+42
View File
@@ -0,0 +1,42 @@
# coreci-broker docker image (v0.2, REQ-23).
#
# Equivalence mapping to the daemonless-published image
# (scripts/publish_image.py — the pipeline's source of truth):
# same base family: python:3.11-slim (D-022 pinned minor)
# same artifact: coreci_broker.py + coreci_broker_client.py +
# coreci_broker_ui/ + cli/__main__.py + _cli_version.py,
# packed by the same `python3 -m zipapp` invocation
# (--python "/usr/bin/env python3" --compress)
# same entrypoint: ["python3", "/app/coreci-broker.pyz"]
# same state anchor: WORKDIR /state (volume-backed; first boot generates
# .env, broker.db, broker.audit.log there)
# Documented differences (equivalence required, byte-identity NOT):
# - docker-build layering (COPY + RUN) vs the daemonless client's
# verbatim base layers + one deterministic broker layer
# - version provenance: ARG VERSION here vs build-time _cli_version.py
# injection in scripts/build_cli.py
# - no normalize pass here (docker rebuilds are not digest-pinned)
FROM python:3.11-slim
# Staging mirrors scripts/build_cli.py EXACTLY (P3-03, CWE-636): Docker
# copies directory CONTENTS flattened, so the staging layout must come
# from the destination paths — root sources at /app/src/, UI pages under
# /app/src/coreci_broker_ui/ (cli/__main__.py:80-83 materializes only
# coreci_broker_ui/* members; a flattened copy leaves /ui/* 404ing), and
# the shim staged as the archive-root __main__.py (zipapp boots the
# root __main__.py; the file-to-file rename mirrors build_cli's
# STAGED_SHIM staging).
COPY coreci_broker.py coreci_broker_client.py /app/src/
COPY coreci_broker_ui/ /app/src/coreci_broker_ui/
COPY cli/__main__.py /app/src/__main__.py
ARG VERSION=dev
RUN printf 'VERSION = "%s"\n' "$VERSION" > /app/src/_cli_version.py \
&& python3 -m zipapp /app/src -o /app/coreci-broker.pyz \
--python "/usr/bin/env python3" --compress
WORKDIR /state
ENTRYPOINT ["python3", "/app/coreci-broker.pyz"]
EXPOSE 8765
+222
View File
@@ -0,0 +1,222 @@
# CoreCI Broker
**TL;DR** — CoreCI Broker — single-file credential broker for autonomous agent tools (Open WebUI workspace tool / standalone). This file explains how to stand it up and who to contact; the machinery detail stays lower under Details.
A single-file credential broker for autonomous agent tools on the Open WebUI
host. Tools authenticate with derived tokens instead of shared secrets, the
CEO gates access through UI-managed allowlists of app UUIDs and user IDs,
every read lands in an append-only audit log, and secret reads are path-scoped
to `coreci/<agent_id>` — one Python file, one SQLite DB, a four-page admin UI,
zero pip installs, no build step.
## Quickstart (standalone)
```bash
python3 coreci_broker.py
```
- Stdlib-only runtime, Python 3.11+. Zero pip installs: bcrypt when
importable, stdlib scrypt otherwise (format-prefixed hashes).
- First boot generates `.env`: a 32-byte base64 pepper plus a 32+-char admin
password. The password is printed to the startup log as `BROKER_ADMIN_PASS=…`
**exactly once** — save it immediately; it is never printed again. Lost pass:
remove `BROKER_ADMIN_PASS` from `.env` and reboot — a new one prints once.
- The server binds `127.0.0.1:8765`. Open `http://127.0.0.1:8765/ui/login` and
log in with the printed pass.
- Stop with Ctrl+C. Durable state is exactly three files: `broker.db`,
`broker.audit.log`, `.env` — back them up.
## Configuration
Read on boot from the `.env` at `BROKER_ENV_PATH`. Precedence:
**process env → `.env` → defaults**.
| Variable | Default | Purpose |
|----------|---------|---------|
| `BROKER_BIND` | `127.0.0.1:8765` | Listen address; the bind controls reachability |
| `BROKER_ADMIN_USER` | `admin` | Admin login user name |
| `BROKER_ADMIN_PASS` | generated first boot | Admin login password (32+ random chars) |
| `BROKER_INSTANCE_PEPPER` | generated first boot | 32-byte base64 pepper; token derivation input, synced to SQLite |
| `BROKER_ENV_PATH` | `.env` | Path of the `.env` file the broker reads/writes |
| `BROKER_TOKEN_TTL` | `86400` | Tool token lifetime, seconds (24 h) |
| `BROKER_SESSION_TTL` | `300` | Admin UI session cookie lifetime, seconds |
| `BROKER_DB_PATH` | `broker.db` | SQLite database (mode 0600) |
| `BROKER_AUDIT_LOG_PATH` | `broker.audit.log` | Append-only JSONL audit log (mode 0600) |
| `BROKER_CREDS_SOURCE` | `file` | Credential source adapter: `file` (dev/tests) or `owui` (production) |
| `BROKER_CREDS_FILE` | `data/creds.json` | `FileCredsSource` JSON document: paths → secret objects |
| `BROKER_OWUI_BASE_URL` | — | Open WebUI base URL for the OWUI creds source |
| `BROKER_OWUI_API_KEY` | — | Open WebUI API key (Bearer) for KB read access |
| `BROKER_AGENT_STATE_BACKEND` | `internal` | Agent-state storage: `internal` (rows in broker.db) or `owui` |
| `BROKER_MAX_AGENTS_PER_APP` | `50` | Cap on agent registrations per app UUID |
| `BROKER_UI_ROOT` | `coreci_broker_ui` | Directory served for `/ui/*` pages |
| `BROKER_STATIC_ROOT` | `.` | Directory served for `/static/*` (client module) |
Production: `BROKER_CREDS_SOURCE=owui` with `BROKER_OWUI_BASE_URL` +
`BROKER_OWUI_API_KEY` (0600 `.env` discipline). Keep `BROKER_BIND` on
loopback unless you deliberately widen reachability.
## Running as an Open WebUI workspace tool
Paste `coreci_broker.py` into Open WebUI as a workspace Tool (frontmatter
`requirements:` stays empty — the runtime already ships everything). Notes
that matter:
- The server starts in `Tools.__init__` behind a once-per-process guard; the
OWUI loader has no activate/deactivate hook, and disabling a tool does not
stop a previously started server.
- **After editing the broker tool source, restart Open WebUI.** The loader
re-execs the module on source change, but the previously loaded server keeps
its port — the new instance fails loud with a bind-error message telling you
to restart, rather than silently serving old code.
- Paths (DB, audit log, `.env`, UI roots) resolve from environment config,
never from `__file__` — the OWUI loader deletes the temp file it execs.
- Durable state: `broker.db` + `broker.audit.log` + `.env` — **back those
three up**; SQLite is the source of truth, no HA, restore is a file copy.
- `Tools.status()` reports the serving address; call it from a chat to verify
the mount, or hit `/healthz` on the bind.
## Admin UI
Four pages + dashboard, server-rendered on the same port as the API:
- `/ui/login` — the CEO's bookmark. First-boot pass discipline: the printed
pass is shown once; record it in your secret store on first login.
- `/ui/app_registrations` — allowlist of tool app UUIDs: add, revoke, reinstate.
- `/ui/user_allowlist` — allowlist of user IDs (optional email): same actions.
- `/ui/audit` — filterable, read-only view of the audit log (no export in v0.1).
- `/ui/dashboard` — landing page with persistent nav header and Logout.
Allowlist semantics: **revoke keeps existing tokens working until TTL expiry —
by design.** Revocation immediately blocks new registrations; already-issued
tokens live at most one `BROKER_TOKEN_TTL` (24 h by default).
## Tool authors
Fetch the public client module — unauthenticated, from
`http://<broker>/static/coreci_broker_client.py` — into your workspace and
import it. Flow: `ensure_registered(state)` → `put_state(state)` →
`read_secret("coreci/<agent_id>", state)`. Stdlib-urllib-only (no httpx, no
bcrypt), so any tool can import it; on a 401 it re-registers and retries
once. State endpoints are the tool's only persistence channel — keep `state`
there, not in local files. See `reference_tools/` for the proxmox-style §11
flow and a minimal auth-then-use showcase; downstream calls default to a
safe dry-run.
## Security model
- Derived token: `base64(sha256(pepper:UUID:user_id:agent_id:sorted_tags))` —
44 ASCII chars, no prefix, no challenge protocol. Stored only as a slow
hash (`$2b$…` / `scrypt$…`), never in plaintext.
- Allowlists are the gate: unknown or revoked app UUIDs and unknown users get
403 at registration; the token is the consequence, not the authority.
- Every secret-read attempt is audited (ok/denied/error) with user, UUID,
agent, and path.
- Secret reads are path-scoped: a registration reads exactly
`coreci/<agent_id>` — cross-path → 403, absent → 404.
- The broker never writes to `creds-coreci-coo`; the CEO maintains that KB
through the Open WebUI UI.
- `broker.db`, `broker.audit.log`, and `.env` are mode 0600.
- Binds `127.0.0.1` by default; trusted-network posture — no rate limiting by
design (spec §14).
## Documented spec deviations
Honest list; full detail in `.ciagent/PROJECT.md`:
- **Line-cap amendments (D-016/D-017/D-018)**: the single-file constraint grew
800 → 1000 → 1200 → 1400 lines via the owner-approved §12.14 amendment
pattern (readability over dense cramming). Current source: 1390 lines; AC14
enforces the cap.
- **Dual-mode hashing (D-001)**: spec §6 mandates bcrypt; the runtime is
stdlib-only, so scrypt fallback with format-prefixed hashes.
- **Token verification (D-004)**: spec §7.5's in-memory `token_hash` index is
impossible with salted slow hashes; the broker scans rows scoped to the
presented UUID and keeps a non-authoritative verified-token cache.
- **Pepper rule (D-005)**: the pepper is returned only on row creation or
tokenless registration (the KB-loss recovery path); spec §9's snippet would
raise `KeyError` on re-registration.
- **Spec defects resolved**: §7.5 index, §6-vs-§9 (the broker derives
authoritatively; the tool stores), §9's `KeyError` — see PROJECT.md §Context.
- `BROKER_ADMIN_USER` defaults to `admin` (spec silent on a default).
- Register cap `BROKER_MAX_AGENTS_PER_APP` (50) — a documented addition.
- Reinstate actions audit as `add_app`/`add_user` (no separate audit action).
- In-scope but absent secrets audit as `read` with `result: error` (404s are
audited, not silent).
## Post-ship live-OWUI smoke checklist (owner-run)
Before migrating any real tool (G-005; deployed host pinned v0.11.3, G-006):
1. Confirm the deployed Open WebUI version (expected **v0.11.3**) and that the
KB read endpoints match the researched surface: `/api/v1/knowledge/{id}` and
`/api/v1/files/{id}/data/content`.
2. Add the broker as a workspace tool; check `/healthz` on the bind.
3. CEO first-boot login at `/ui/login` with the printed pass.
4. Add one app UUID and one user to the allowlists via the UI.
5. In a scratch tool: import the client module from `/static/` →
`ensure_registered` → read `coreci/proxmox` via the OWUI creds source.
6. Verify every step in `/ui/audit`.
7. Rollback = disable the tool + restore the three state files.
No real tool migrates before this passes; the 90-day adoption criterion
(proxmox tool on the client, direct KB reads removed) is due 2027-01-09.
## Tests
```bash
timeout 300 python3 -m unittest discover -s tests
```
295 tests, stdlib unittest only (~2 min; one skip where bcrypt is not
importable). The AC1–AC14 acceptance matrix is green.
## Repository layout
| Path | What it is |
|------|------------|
| `coreci_broker.py` | The broker: single file — server, store, audit, auth, UI serving |
| `coreci_broker_client.py` | Public stdlib-only client module (served at `/static/`) |
| `coreci_broker_ui/` | Server-rendered HTML for the four pages + dashboard |
| `reference_tools/` | Tool-author templates (proxmox-style §11 flow + minimal showcase) |
| `tests/` | Stdlib-unittest suite incl. the AC1–AC14 acceptance matrix |
| `.ciagent/` | CI pipeline artifacts (project, requirements, roadmap, decisions) |
## CLI and Docker (v0.2)
From v0.2.1 onward every gitea release attaches the broker packaged as a
self-contained CLI zipapp: `coreci-broker-vX.Y.Z.pyz` (its sha256 is in
the release notes).
**CLI install + usage:**
```bash
# from the release page assets:
chmod +x coreci-broker-vX.Y.Z.pyz
./coreci-broker-vX.Y.Z.pyz --version # prints the release version
./coreci-broker-vX.Y.Z.pyz # boots the standalone broker
# or: python3 coreci-broker-vX.Y.Z.pyz (Python 3.11+)
```
The artifact bundles the broker, the admin UI pages, and the client
module — one file serves everything (UI at `/ui/`, client at
`/static/coreci_broker_client.py`). First boot prints
`BROKER_ADMIN_PASS=...` exactly once and creates `.env`, `broker.db`,
and `broker.audit.log` mode 0600 in the state dir; state is anchored by
`BROKER_ENV_PATH` (relative paths stay CWD-relative). The full
`BROKER_*` env contract is in the Configuration section above.
**Docker:**
```bash
docker pull git.coreci.dev/coreci/broker:<tag> # also :latest
docker run -p 8765:8765 -v broker-state:/state \
git.coreci.dev/coreci/broker:<tag>
```
State lives on the `/state` volume (`WORKDIR /state`; first boot
generates `.env`, `broker.db`, and `broker.audit.log there) — mount a
persistent volume or the admin pass and any registrations are lost with
the container. The image is built and published once per milestone at
the milestone release (see `scripts/publish_image.py` — the daemonless
registry client; the `Dockerfile` documents the docker-user path and
its equivalence mapping).
+126
View File
@@ -0,0 +1,126 @@
# coreci-broker CLI boot shim; staged by scripts/build_cli.py as the root
# __main__.py of the coreci-broker-vX.Y.Z.pyz zipapp (the documented mode —
# running this file unzipped from a source tree is a guarded fallback).
#
# Boot sequence: (1) --version prints _cli_version.VERSION and exits 0 before
# any broker import; (2) the state dir is resolved exactly like the broker
# resolves it (BROKER_ENV_PATH-anchored; relative default paths stay
# CWD-relative — the broker's path contract is inherited unchanged);
# (3) bundled assets are materialized from the running archive
# (zipfile.ZipFile(sys.path[0])) into
# <state>/.coreci_broker_assets/<VERSION>/{ui,static} — ui/ receives the
# bundled coreci_broker_ui/ pages, static/ receives the bundled
# coreci_broker_client.py — via a temp dir plus os.replace (versioned dirs;
# same-version overwrite self-repairs; other versions are never cleaned);
# (4) BROKER_UI_ROOT/BROKER_STATIC_ROOT are pointed at the materialized
# dirs only when the key is absent from both the process env and the user
# .env (user wins); (5) coreci_broker.run_standalone() boots the broker.
# Guard: when sys.path[0] is a directory (or empty) this shim is running
# unzipped from a source tree — materialization is skipped and the broker's
# repo-relative defaults apply.
#
# Private-API coupling (grill G-018, tracked in P3-02): this shim imports
# coreci_broker._parse_env_file and coreci_broker.run_standalone —
# module-level but private-surfaced broker APIs (coreci_broker.py is frozen
# at 1390 lines). Any future broker refactor that removes or renames them
# breaks packaged boots loudly (ImportError/AttributeError); that breakage
# is accepted — the fix is to update this shim and re-run the v0.2 packaged
# acceptance (tests/test_cli_packaging.py).
import os
import shutil
import sys
import tempfile
import zipfile
ASSET_DIR_NAME = ".coreci_broker_assets"
def _version_requested():
return "--version" in sys.argv[1:]
def _resolve_state_dir():
env_path = os.environ.get("BROKER_ENV_PATH") or ".env"
return os.path.dirname(os.path.abspath(env_path)) or os.getcwd()
def _safe_relative(name):
if not name or "\x00" in name or "\\" in name or name.startswith("/"):
return None
parts = name.split("/")
if ".." in parts or any(part.startswith(".") for part in parts):
return None
return name
def _write_asset(target_dir, relative, payload):
target = os.path.join(target_dir, relative)
os.makedirs(os.path.dirname(target), exist_ok=True)
with open(target, "wb") as handle:
handle.write(payload)
def _materialize_assets(archive_path, state_dir):
import _cli_version
base_dir = os.path.join(state_dir, ASSET_DIR_NAME, _cli_version.VERSION)
parent_dir = os.path.dirname(base_dir)
os.makedirs(parent_dir, exist_ok=True)
tmp_dir = tempfile.mkdtemp(dir=parent_dir, prefix=".tmp-")
try:
ui_dir = os.path.join(tmp_dir, "ui")
static_dir = os.path.join(tmp_dir, "static")
os.makedirs(ui_dir)
os.makedirs(static_dir)
with zipfile.ZipFile(archive_path) as archive:
for name in archive.namelist():
if name == "coreci_broker_client.py":
_write_asset(static_dir, name, archive.read(name))
elif name.startswith("coreci_broker_ui/") and not name.endswith("/"):
relative = _safe_relative(name[len("coreci_broker_ui/"):])
if relative is not None:
_write_asset(ui_dir, relative, archive.read(name))
if os.path.isdir(base_dir):
stale_dir = base_dir + ".stale"
shutil.rmtree(stale_dir, ignore_errors=True)
os.replace(base_dir, stale_dir)
try:
os.replace(tmp_dir, base_dir)
except OSError:
os.replace(stale_dir, base_dir)
raise
shutil.rmtree(stale_dir, ignore_errors=True)
else:
os.replace(tmp_dir, base_dir)
finally:
shutil.rmtree(tmp_dir, ignore_errors=True)
return {"ui": os.path.join(base_dir, "ui"), "static": os.path.join(base_dir, "static")}
def _apply_asset_roots(roots, env_path):
import coreci_broker
file_values = dict(coreci_broker._parse_env_file(env_path))
for key, name in (("BROKER_UI_ROOT", "ui"), ("BROKER_STATIC_ROOT", "static")):
if key not in os.environ and key not in file_values:
os.environ[key] = roots[name]
def main():
if _version_requested():
import _cli_version
print(_cli_version.VERSION)
return 0
archive_path = sys.path[0]
if archive_path and not os.path.isdir(archive_path):
env_path = os.environ.get("BROKER_ENV_PATH") or ".env"
roots = _materialize_assets(archive_path, _resolve_state_dir())
_apply_asset_roots(roots, env_path)
import coreci_broker
coreci_broker.run_standalone()
return 0
if __name__ == "__main__":
sys.exit(main())
+1390
View File
File diff suppressed because it is too large Load Diff
+161
View File
@@ -0,0 +1,161 @@
"""Public stdlib-only client for the CoreCI Broker: idempotent tool registration, agent-state persistence, and path-scoped secret reads."""
import json
import time
import urllib.parse
import urllib.request
REGISTER_PATH = "/api/broker/agents/register"
SECRETS_PREFIX = "/api/broker/secrets/"
CLIENT_MODULE_PATH = "/static/coreci_broker_client.py"
class BrokerError(Exception):
def __init__(self, status, message):
super().__init__(message)
self.status = status
self.message = message
def _broker_error(status, body):
message = "broker returned HTTP %s" % status
if isinstance(body, dict) and body.get("message"):
message = str(body["message"])
return BrokerError(status, message)
def _epoch_from_iso(value):
try:
year, month, day = int(value[0:4]), int(value[5:7]), int(value[8:10])
hour, minute, second = int(value[11:13]), int(value[14:16]), int(value[17:19])
rest = value[19:]
micro = 0
if rest[:1] == ".":
digits = ""
for char in rest[1:]:
if not char.isdigit():
break
digits += char
micro = int((digits + "000000")[:6])
rest = rest[1 + len(digits):]
offset = 0
if rest[:1] in ("+", "-"):
sign = 1 if rest[0] == "+" else -1
offset = sign * (int(rest[1:3]) * 3600 + int(rest[4:6]) * 60)
if not 1 <= month <= 12 or not 1 <= day <= 31:
return None
if hour > 23 or minute > 59 or second > 59:
return None
year -= month <= 2
era = (year if year >= 0 else year - 399) // 400
yoe = year - era * 400
doy = (153 * (month - 3 if month > 2 else month + 9) + 2) // 5 + day - 1
doe = yoe * 365 + yoe // 4 - yoe // 100 + doy
return (era * 146097 + doe - 719468) * 86400 + hour * 3600 + minute * 60 + second - offset + micro / 1000000.0
except (ValueError, TypeError, IndexError):
return None
def _token_live(state):
token = state.get("current_token")
expires = _epoch_from_iso(state.get("token_expires_at"))
return isinstance(token, str) and bool(token) and expires is not None and expires > time.time()
class BrokerClient:
def __init__(self, base_url, agent_id, app_uuid, user_id, tags=None, timeout=10.0):
self.base_url = str(base_url).rstrip("/")
self.agent_id = agent_id
self.app_uuid = app_uuid
self.user_id = user_id
self.tags = list(tags or [])
self.timeout = timeout
def _app_headers(self, state=None):
token = (state or {}).get("current_token")
if not isinstance(token, str) or not token:
return {}
return {"Authorization": "App %s %s" % (self.app_uuid, token)}
def _request(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(self.base_url + path, data=data, method=method)
if data is not None:
request.add_header("Content-Type", "application/json")
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=self.timeout) as response:
status, raw = response.status, response.read()
except urllib.request.HTTPError as error:
status, raw = error.code, error.read()
body = None
text = raw.decode("utf-8", "replace")
if text:
try:
body = json.loads(text)
except ValueError:
body = None
return status, body
def _authorized_request(self, method, path, state, payload=None):
status, body = self._request(method, path, payload, self._app_headers(state))
if status == 401:
self.register(state)
status, body = self._request(method, path, payload, self._app_headers(state))
return status, body
def _state_path(self):
return "/api/broker/agents/%s/state" % urllib.parse.quote(self.agent_id, safe="")
def register(self, state):
if state is None:
state = {}
payload = {"agent_id": self.agent_id, "app_registration_uuid": self.app_uuid,
"user_id": self.user_id, "tags": list(self.tags)}
status, data = self._request("POST", REGISTER_PATH, payload, self._app_headers(state))
if status not in (200, 201) or not isinstance(data, dict):
raise _broker_error(status, data)
state["agent_id"] = self.agent_id
state["app_registration_uuid"] = self.app_uuid
state["user_id"] = self.user_id
state["tags"] = list(self.tags)
state["current_token"] = data["token"]
state["token_expires_at"] = data["token_expires_at"]
pepper = data.get("pepper")
if pepper is not None:
state["pepper"] = pepper
return state
def ensure_registered(self, state):
if state is None:
state = {}
if _token_live(state):
return state
return self.register(state)
def get_state(self, state):
status, body = self._authorized_request("GET", self._state_path(), state)
if status == 200:
return body
if status == 404:
return None
raise _broker_error(status, body)
def put_state(self, state):
status, body = self._authorized_request("PUT", self._state_path(), state, state)
if status == 200:
return True
raise _broker_error(status, body)
def read_secret(self, path, state):
suffix = urllib.parse.quote(str(path), safe="/")
status, body = self._authorized_request("GET", SECRETS_PREFIX + suffix, state)
if status == 200:
return body
raise _broker_error(status, body)
def fetch_client_module(base_url, timeout=10.0):
url = str(base_url).rstrip("/") + CLIENT_MODULE_PATH
with urllib.request.urlopen(url, timeout=timeout) as response:
return response.read().decode("utf-8")
+271
View File
@@ -0,0 +1,271 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>App Registrations — CoreCI Broker</title>
<link rel="stylesheet" href="/ui/shared.css">
<style>
.page-head {
display: flex;
align-items: flex-start;
justify-content: space-between;
gap: 16px;
flex-wrap: wrap;
margin-bottom: 8px;
}
.page-sub {
color: var(--muted);
font-size: 14px;
margin: 0;
}
.table-empty {
color: var(--muted);
}
.cell-actions {
display: flex;
gap: 8px;
flex-wrap: wrap;
}
.btn-danger {
color: var(--danger);
background: transparent;
border: 1px solid rgba(240, 122, 114, 0.45);
}
.badge-revoked {
color: var(--danger);
border-color: rgba(240, 122, 114, 0.45);
background: rgba(240, 122, 114, 0.08);
}
</style>
</head>
<body>
<div id="nav"></div>
<main class="wrap">
<section class="card">
<div class="page-head">
<div>
<h1>App Registrations</h1>
<p class="page-sub">Tool app UUIDs allowed to register against the broker.</p>
</div>
<button type="button" class="btn btn-primary" id="add-open">Add registration</button>
</div>
<p class="form-error" id="page-error" role="alert" hidden></p>
<table>
<thead>
<tr>
<th>UUID</th>
<th>Name</th>
<th>Status</th>
<th>Created</th>
<th>Revoked</th>
<th>Actions</th>
</tr>
</thead>
<tbody id="rows"></tbody>
</table>
</section>
</main>
<div class="modal-backdrop" id="modal" hidden role="dialog" aria-modal="true" aria-labelledby="modal-title">
<div class="modal">
<h2 id="modal-title">Add registration</h2>
<form id="add-form">
<div class="field">
<label for="uuid">App UUID</label>
<input id="uuid" name="uuid" type="text" class="mono" placeholder="00000000-0000-0000-0000-000000000000" required>
</div>
<div class="field">
<label for="name">Name (optional)</label>
<input id="name" name="name" type="text" placeholder="proxmox tool">
</div>
<p class="form-error" id="add-error" role="alert" hidden></p>
<div class="modal-actions">
<button type="button" class="btn btn-subtle" id="add-cancel">Cancel</button>
<button type="submit" class="btn btn-primary" id="add-submit">Add</button>
</div>
</form>
</div>
</div>
<script src="/ui/nav.js"></script>
<script>
(function () {
var API = "/api/broker/app_registrations";
var rows = document.getElementById("rows");
var pageError = document.getElementById("page-error");
var modal = document.getElementById("modal");
var form = document.getElementById("add-form");
var uuidInput = document.getElementById("uuid");
var nameInput = document.getElementById("name");
var addError = document.getElementById("add-error");
var submitButton = document.getElementById("add-submit");
function showPageError(message) {
pageError.textContent = message;
pageError.hidden = false;
}
function clearPageError() {
pageError.hidden = true;
pageError.textContent = "";
}
function messageRow(text) {
rows.replaceChildren();
var tr = document.createElement("tr");
var td = document.createElement("td");
td.colSpan = 6;
td.className = "table-empty";
td.textContent = text;
tr.appendChild(td);
rows.appendChild(tr);
}
function textCell(value, mono) {
var td = document.createElement("td");
if (mono) {
td.className = "mono";
}
td.textContent = value === null || value === undefined || value === "" ? "—" : value;
return td;
}
function statusBadge(status) {
var span = document.createElement("span");
span.className = "badge " + (status === "active" ? "badge-active" : "badge-revoked");
span.textContent = status;
return span;
}
function actionButton(label, className, disabled, handler) {
var button = document.createElement("button");
button.type = "button";
button.className = "btn " + className;
button.textContent = label;
button.disabled = disabled;
button.addEventListener("click", handler);
return button;
}
function renderRow(row) {
var tr = document.createElement("tr");
tr.appendChild(textCell(row.uuid, true));
tr.appendChild(textCell(row.name));
var statusCell = document.createElement("td");
statusCell.appendChild(statusBadge(row.status));
tr.appendChild(statusCell);
tr.appendChild(textCell(row.created_at));
tr.appendChild(textCell(row.revoked_at));
var actionsCell = document.createElement("td");
var actions = document.createElement("div");
actions.className = "cell-actions";
var key = encodeURIComponent(row.uuid);
actions.appendChild(actionButton("Revoke", "btn-danger", row.status !== "active", function () {
rowAction("DELETE", API + "/" + key, "Revoke app registration " + row.uuid + "?", "Revoke failed", row.uuid);
}));
actions.appendChild(actionButton("Reinstate", "btn-subtle", row.status === "active", function () {
rowAction("POST", API + "/" + key + "/reinstate", "Reinstate app registration " + row.uuid + "?", "Reinstate failed", row.uuid);
}));
actionsCell.appendChild(actions);
tr.appendChild(actionsCell);
return tr;
}
async function refresh() {
messageRow("Loading registrations…");
var result = await api(API);
if (!result.ok) {
messageRow(result.status === 0 ? "Broker unreachable." : "Could not load registrations.");
return;
}
var list = (result.data && result.data.registrations) || [];
if (!list.length) {
messageRow("No app registrations yet.");
return;
}
rows.replaceChildren();
list.forEach(function (row) {
rows.appendChild(renderRow(row));
});
}
async function rowAction(method, path, question, label, uuid) {
if (!confirm(question)) {
return;
}
clearPageError();
var result = await api(path, {method: method});
if (!result.ok) {
if (result.status === 0) {
showPageError(label + ": broker unreachable.");
} else {
var detail = result.data && result.data.message ? ": " + result.data.message : ".";
showPageError(label + " " + uuid + detail);
}
}
await refresh();
}
function openModal() {
form.reset();
addError.hidden = true;
addError.textContent = "";
submitButton.disabled = false;
modal.hidden = false;
uuidInput.focus();
}
function closeModal() {
modal.hidden = true;
}
form.addEventListener("submit", async function (event) {
event.preventDefault();
addError.hidden = true;
addError.textContent = "";
submitButton.disabled = true;
var payload = {uuid: uuidInput.value.trim()};
var name = nameInput.value.trim();
if (name) {
payload.name = name;
}
var result = await api(API, {method: "POST", body: payload});
submitButton.disabled = false;
if (result.ok) {
closeModal();
clearPageError();
await refresh();
return;
}
if (result.status === 0) {
addError.textContent = "Broker unreachable.";
} else if (result.data && result.data.message) {
addError.textContent = result.data.message;
} else {
addError.textContent = "Could not add registration.";
}
addError.hidden = false;
await refresh();
});
document.getElementById("add-open").addEventListener("click", openModal);
document.getElementById("add-cancel").addEventListener("click", closeModal);
modal.addEventListener("click", function (event) {
if (event.target === modal) {
closeModal();
}
});
document.addEventListener("keydown", function (event) {
if (event.key === "Escape" && !modal.hidden) {
closeModal();
}
});
refresh();
})();
</script>
</body>
</html>
+280
View File
@@ -0,0 +1,280 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Audit — CoreCI Broker</title>
<link rel="stylesheet" href="/ui/shared.css">
<style>
.page-sub {
color: var(--muted);
font-size: 14px;
margin: 0 0 16px;
}
.table-empty {
color: var(--muted);
}
.filter-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(170px, 1fr));
gap: 12px;
align-items: end;
margin-bottom: 20px;
}
.filter-grid .field {
margin-bottom: 0;
}
.filter-apply .btn {
width: 100%;
}
.table-scroll {
overflow-x: auto;
}
.count-line {
color: var(--muted);
font-size: 14px;
margin: 12px 0 0;
}
.load-more {
margin-top: 14px;
}
.badge-denied {
color: var(--danger);
border-color: rgba(240, 122, 114, 0.45);
background: rgba(240, 122, 114, 0.08);
}
</style>
</head>
<body>
<div id="nav"></div>
<main class="wrap">
<section class="card">
<h1>Audit</h1>
<p class="page-sub">Read-only trail of broker activity. Set filters and apply to query the audit log.</p>
<p class="form-error" id="page-error" role="alert" hidden></p>
<form id="filters" class="filter-grid">
<div class="field">
<label for="since">Since</label>
<input type="datetime-local" id="since" name="since">
</div>
<div class="field">
<label for="until">Until</label>
<input type="datetime-local" id="until" name="until">
</div>
<div class="field">
<label for="user_id">User ID</label>
<input type="text" id="user_id" name="user_id" class="mono" autocomplete="off">
</div>
<div class="field">
<label for="app_registration_uuid">App UUID</label>
<input type="text" id="app_registration_uuid" name="app_registration_uuid" class="mono" autocomplete="off" placeholder="00000000-0000-0000-0000-000000000000">
</div>
<div class="field">
<label for="action">Action</label>
<select id="action" name="action">
<option value="">Any action</option>
<option value="register">register</option>
<option value="read">read</option>
<option value="denied">denied</option>
<option value="login">login</option>
<option value="logout">logout</option>
<option value="add_app">add_app</option>
<option value="remove_app">remove_app</option>
<option value="add_user">add_user</option>
<option value="remove_user">remove_user</option>
</select>
</div>
<div class="field">
<label for="limit">Limit</label>
<input type="number" id="limit" name="limit" min="1" max="1000" value="100">
</div>
<div class="filter-apply">
<button type="submit" class="btn btn-primary" id="apply">Apply</button>
</div>
</form>
<div class="table-scroll">
<table>
<thead>
<tr>
<th>Timestamp</th>
<th>User ID</th>
<th>App UUID</th>
<th>Agent ID</th>
<th>Action</th>
<th>Path</th>
<th>Result</th>
</tr>
</thead>
<tbody id="rows"></tbody>
</table>
</div>
<p class="count-line" id="count" hidden></p>
<div class="load-more">
<button type="button" class="btn btn-subtle" id="more" hidden>Load more</button>
</div>
</section>
</main>
<script src="/ui/nav.js"></script>
<script>
(function () {
var API = "/api/broker/audit";
var FILTERS = ["since", "until", "user_id", "app_registration_uuid", "action"];
var rows = document.getElementById("rows");
var pageError = document.getElementById("page-error");
var filterForm = document.getElementById("filters");
var limitInput = document.getElementById("limit");
var countLine = document.getElementById("count");
var moreButton = document.getElementById("more");
var state = {filters: {}, cursor: null, shown: 0};
function showPageError(message) {
pageError.textContent = message;
pageError.hidden = false;
}
function clearPageError() {
pageError.hidden = true;
pageError.textContent = "";
}
function messageRow(text) {
rows.replaceChildren();
var tr = document.createElement("tr");
var td = document.createElement("td");
td.colSpan = 7;
td.className = "table-empty";
td.textContent = text;
tr.appendChild(td);
rows.appendChild(tr);
}
function textCell(value, mono) {
var td = document.createElement("td");
if (mono) {
td.className = "mono";
}
td.textContent = value === null || value === undefined || value === "" ? "—" : String(value);
return td;
}
function resultBadge(result) {
var span = document.createElement("span");
span.className = "badge";
if (result === "ok") {
span.className += " badge-active";
} else if (result === "denied" || result === "error") {
span.className += " badge-denied";
}
span.textContent = result === null || result === undefined || result === "" ? "—" : String(result);
return span;
}
function renderRow(entry) {
var tr = document.createElement("tr");
tr.appendChild(textCell(entry.ts, true));
tr.appendChild(textCell(entry.user_id, true));
tr.appendChild(textCell(entry.app_registration_uuid, true));
tr.appendChild(textCell(entry.agent_id, true));
tr.appendChild(textCell(entry.action, true));
tr.appendChild(textCell(entry.path, true));
var resultCell = document.createElement("td");
resultCell.appendChild(resultBadge(entry.result));
tr.appendChild(resultCell);
return tr;
}
function readFilters() {
var filters = {};
FILTERS.forEach(function (name) {
var value = document.getElementById(name).value.trim();
if (value) {
filters[name] = value;
}
});
var limit = parseInt(limitInput.value, 10);
filters.limit = Number.isFinite(limit) && limit > 0 ? limit : 100;
return filters;
}
function buildQuery(filters, cursor) {
var params = [];
FILTERS.forEach(function (name) {
if (name === "since" && cursor !== null) {
return;
}
if (filters[name]) {
params.push(encodeURIComponent(name) + "=" + encodeURIComponent(filters[name]));
}
});
if (cursor !== null) {
params.push("since=" + encodeURIComponent(cursor));
}
params.push("limit=" + encodeURIComponent(String(filters.limit)));
return "?" + params.join("&");
}
async function load(reset) {
clearPageError();
moreButton.hidden = true;
if (reset) {
state.cursor = null;
state.shown = 0;
countLine.hidden = true;
countLine.textContent = "";
messageRow("Loading audit entries…");
}
var result = await api(API + buildQuery(state.filters, state.cursor));
if (!result.ok) {
var label = result.status === 0 ? "Broker unreachable." : (result.data && result.data.message) || "Could not load audit entries.";
if (reset) {
messageRow(label);
} else {
showPageError(label);
moreButton.hidden = false;
}
return;
}
var data = result.data || {};
var entries = Array.isArray(data.entries) ? data.entries : [];
state.cursor = data.next_since === undefined ? null : data.next_since;
if (reset) {
rows.replaceChildren();
}
entries.forEach(function (entry) {
rows.appendChild(renderRow(entry));
});
state.shown += entries.length;
if (!state.shown) {
messageRow("No audit entries match the current filters.");
countLine.hidden = true;
return;
}
countLine.textContent = state.shown + (state.shown === 1 ? " entry shown." : " entries shown.");
countLine.hidden = false;
moreButton.hidden = state.cursor === null;
}
filterForm.addEventListener("submit", function (event) {
event.preventDefault();
state.filters = readFilters();
load(true);
});
moreButton.addEventListener("click", function () {
load(false);
});
state.filters = readFilters();
load(true);
})();
</script>
</body>
</html>
+64
View File
@@ -0,0 +1,64 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Dashboard — CoreCI Broker</title>
<link rel="stylesheet" href="/ui/shared.css">
</head>
<body>
<div id="nav"></div>
<main class="wrap">
<section class="card">
<h1>Dashboard</h1>
<p>Signed in as <strong id="who">…</strong></p>
</section>
<section class="card status-strip">
<span class="dot" id="status-dot" aria-hidden="true"></span>
<span id="status-text" aria-live="polite">Broker status: checking…</span>
</section>
<section class="card">
<h2>Quick links</h2>
<div class="link-grid">
<a class="link-card" href="/ui/app_registrations">
<strong>App Registrations</strong>
<span>Add, revoke, and reinstate tool app UUIDs</span>
</a>
<a class="link-card" href="/ui/user_allowlist">
<strong>User Allowlist</strong>
<span>Manage the user ids allowed to register</span>
</a>
<a class="link-card" href="/ui/audit">
<strong>Audit Log</strong>
<span>Review broker activity, read-only</span>
</a>
</div>
</section>
</main>
<script src="/ui/nav.js"></script>
<script>
(async function () {
var who = document.getElementById("who");
var dot = document.getElementById("status-dot");
var text = document.getElementById("status-text");
var session = await api("/api/broker/auth/whoami");
if (session.ok && session.data) {
who.textContent = session.data.username || "unknown";
} else {
who.textContent = "unavailable";
}
var health = await api("/healthz");
if (health.ok) {
dot.className = "dot dot-ok";
text.textContent = "Broker status: healthy";
} else if (health.status === 0) {
dot.className = "dot dot-down";
text.textContent = "Broker status: unreachable";
} else {
dot.className = "dot dot-down";
text.textContent = "Broker status: unhealthy";
}
})();
</script>
</body>
</html>
+13
View File
@@ -0,0 +1,13 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>CoreCI Broker</title>
<meta http-equiv="refresh" content="0; url=/ui/login">
</head>
<body>
<p><a href="/ui/login">Continue to CoreCI Broker sign-in</a></p>
<script>location.replace("/ui/login");</script>
</body>
</html>
+78
View File
@@ -0,0 +1,78 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>CoreCI Broker</title>
<link rel="stylesheet" href="/ui/shared.css">
</head>
<body class="login-body">
<main class="login-card card">
<h1>CoreCI Broker</h1>
<p class="login-sub">Admin sign-in</p>
<form id="login-form">
<div class="field">
<label for="username">Username</label>
<input id="username" name="username" type="text" autocomplete="username" autofocus required>
</div>
<div class="field">
<label for="password">Password</label>
<input id="password" name="password" type="password" autocomplete="current-password" required>
</div>
<p class="form-error" id="login-error" role="alert" hidden></p>
<button type="submit" class="btn btn-primary btn-block" id="login-button">Sign in</button>
</form>
</main>
<script>
(function () {
var form = document.getElementById("login-form");
var error = document.getElementById("login-error");
var button = document.getElementById("login-button");
form.addEventListener("input", function () {
error.hidden = true;
error.textContent = "";
});
form.addEventListener("submit", async function (event) {
event.preventDefault();
error.hidden = true;
error.textContent = "";
button.disabled = true;
var status = 0;
var data = null;
try {
var response = await fetch("/api/broker/auth/login", {
method: "POST",
credentials: "same-origin",
headers: {"Content-Type": "application/json"},
body: JSON.stringify({
username: document.getElementById("username").value,
password: document.getElementById("password").value
})
});
status = response.status;
try {
data = await response.json();
} catch (parseError) {
data = null;
}
} catch (networkError) {
status = 0;
}
if (status === 200) {
location.href = "/ui/dashboard";
return;
}
if (status === 0) {
error.textContent = "Sign-in failed: broker unreachable.";
} else {
error.textContent = (data && data.message) || "Sign-in failed. Please try again.";
}
error.hidden = false;
button.disabled = false;
});
})();
</script>
</body>
</html>
+84
View File
@@ -0,0 +1,84 @@
var NAV_ITEMS = [
{label: "Dashboard", href: "/ui/dashboard"},
{label: "App Registrations", href: "/ui/app_registrations"},
{label: "Users", href: "/ui/user_allowlist"},
{label: "Audit", href: "/ui/audit"}
];
async function api(path, opts) {
opts = opts || {};
var init = {method: opts.method || "GET", credentials: "same-origin", headers: {}};
if (opts.body !== undefined) {
init.headers["Content-Type"] = "application/json";
init.body = typeof opts.body === "string" ? opts.body : JSON.stringify(opts.body);
}
if (opts.headers) {
for (var name in opts.headers) {
init.headers[name] = opts.headers[name];
}
}
var response = null;
try {
response = await fetch(path, init);
} catch (networkError) {
return {ok: false, status: 0, data: null};
}
var data = null;
try {
data = await response.json();
} catch (parseError) {
data = null;
}
if (response.status === 401) {
location.href = "/ui/login";
}
return {ok: response.ok, status: response.status, data: data};
}
async function navLogout() {
try {
await api("/api/broker/auth/logout", {method: "POST"});
} catch (error) {}
location.href = "/ui/login";
}
function navRender() {
var mount = document.getElementById("nav");
if (!mount) {
return;
}
var current = location.pathname.replace(/\.html$/, "").replace(/\/+$/, "");
var header = document.createElement("header");
header.className = "nav";
var brand = document.createElement("a");
brand.className = "nav-brand";
brand.href = "/ui/dashboard";
brand.textContent = "CoreCI Broker";
header.appendChild(brand);
var links = document.createElement("nav");
links.setAttribute("aria-label", "Primary");
NAV_ITEMS.forEach(function (item) {
var link = document.createElement("a");
link.href = item.href;
link.textContent = item.label;
if (current === item.href) {
link.className = "active";
link.setAttribute("aria-current", "page");
}
links.appendChild(link);
});
header.appendChild(links);
var logout = document.createElement("button");
logout.type = "button";
logout.className = "btn btn-subtle nav-logout";
logout.textContent = "Logout";
logout.addEventListener("click", navLogout);
header.appendChild(logout);
mount.appendChild(header);
}
if (document.readyState === "loading") {
document.addEventListener("DOMContentLoaded", navRender);
} else {
navRender();
}
+369
View File
@@ -0,0 +1,369 @@
:root {
color-scheme: dark;
--bg: #131820;
--card: #1a212b;
--card-alt: #232d3a;
--border: #2d3947;
--text: #e7edf4;
--muted: #96a3b4;
--accent: #2dd4bf;
--accent-ink: #062421;
--accent-soft: rgba(45, 212, 191, 0.12);
--danger: #f07a72;
--radius: 10px;
}
* {
box-sizing: border-box;
}
[hidden] {
display: none !important;
}
html,
body {
margin: 0;
padding: 0;
}
body {
background: var(--bg);
color: var(--text);
font: 15px/1.55 system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
}
h1 {
font-size: 24px;
line-height: 1.25;
margin: 0 0 12px;
}
h2 {
font-size: 17px;
line-height: 1.25;
margin: 0 0 12px;
}
h3 {
font-size: 15px;
line-height: 1.25;
margin: 0 0 10px;
}
p {
margin: 0 0 10px;
}
a {
color: var(--accent);
text-decoration: none;
}
a:hover {
text-decoration: underline;
}
a:focus-visible,
button:focus-visible,
input:focus-visible,
select:focus-visible {
outline: 2px solid var(--accent);
outline-offset: 2px;
}
.wrap {
max-width: 1040px;
margin: 0 auto;
padding: 24px;
}
.card {
background: var(--card);
border: 1px solid var(--border);
border-radius: var(--radius);
padding: 20px 24px;
margin-bottom: 20px;
}
.nav {
position: sticky;
top: 0;
z-index: 10;
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 8px 20px;
padding: 10px 24px;
background: var(--card);
border-bottom: 1px solid var(--border);
}
.nav-brand {
color: var(--text);
font-size: 16px;
font-weight: 700;
}
.nav-brand:hover {
color: var(--accent);
text-decoration: none;
}
.nav nav {
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 4px;
}
.nav nav a {
color: var(--muted);
font-size: 14px;
padding: 6px 10px;
border-radius: 6px;
}
.nav nav a:hover {
color: var(--text);
background: var(--card-alt);
text-decoration: none;
}
.nav nav a.active {
color: var(--accent);
background: var(--accent-soft);
}
.nav-logout {
margin-left: auto;
}
.btn {
display: inline-block;
font: inherit;
font-size: 14px;
color: var(--text);
background: var(--card-alt);
border: 1px solid transparent;
border-radius: 8px;
padding: 9px 16px;
cursor: pointer;
}
.btn:hover {
filter: brightness(1.12);
}
.btn:disabled {
opacity: 0.6;
cursor: default;
}
.btn-primary {
color: var(--accent-ink);
background: var(--accent);
font-weight: 600;
}
.btn-subtle {
background: transparent;
border-color: var(--border);
}
.btn-block {
width: 100%;
}
.field {
margin-bottom: 16px;
}
.field label {
display: block;
color: var(--muted);
font-size: 13px;
margin-bottom: 6px;
}
.field input,
.field select {
width: 100%;
font: inherit;
color: var(--text);
background: var(--bg);
border: 1px solid var(--border);
border-radius: 8px;
padding: 10px 12px;
}
.field input:focus,
.field select:focus {
border-color: var(--accent);
}
.form-error {
color: var(--danger);
background: rgba(240, 122, 114, 0.08);
border: 1px solid rgba(240, 122, 114, 0.35);
border-radius: 8px;
padding: 10px 12px;
font-size: 14px;
margin: 0 0 14px;
}
.login-body {
min-height: 100vh;
display: grid;
place-items: center;
padding: 24px;
}
.login-card {
width: 100%;
max-width: 380px;
margin-bottom: 0;
}
.login-sub {
color: var(--muted);
font-size: 14px;
margin-bottom: 20px;
}
.status-strip {
display: flex;
align-items: center;
gap: 10px;
padding: 12px 24px;
font-size: 14px;
color: var(--muted);
}
.dot {
width: 10px;
height: 10px;
border-radius: 50%;
background: var(--muted);
}
.dot-ok {
background: var(--accent);
}
.dot-down {
background: var(--danger);
}
.link-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
gap: 16px;
}
.link-card {
display: block;
background: var(--card-alt);
border: 1px solid var(--border);
border-radius: var(--radius);
padding: 16px;
color: var(--text);
}
.link-card:hover {
border-color: var(--accent);
text-decoration: none;
}
.link-card strong {
display: block;
font-size: 15px;
margin-bottom: 4px;
}
.link-card span {
color: var(--muted);
font-size: 13px;
}
table {
width: 100%;
border-collapse: collapse;
font-size: 14px;
}
th {
text-align: left;
color: var(--muted);
font-size: 12px;
text-transform: uppercase;
letter-spacing: 0.06em;
padding: 8px 12px;
border-bottom: 1px solid var(--border);
}
td {
padding: 10px 12px;
border-bottom: 1px solid var(--border);
}
tbody tr:hover {
background: var(--card-alt);
}
.mono {
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
font-size: 13px;
}
.badge {
display: inline-block;
color: var(--muted);
border: 1px solid var(--border);
border-radius: 999px;
padding: 2px 10px;
font-size: 12px;
}
.badge-active {
color: var(--accent);
border-color: var(--accent);
background: var(--accent-soft);
}
.modal-backdrop {
position: fixed;
inset: 0;
background: rgba(0, 0, 0, 0.55);
display: grid;
place-items: center;
padding: 24px;
z-index: 20;
}
.modal {
width: 100%;
max-width: 460px;
background: var(--card);
border: 1px solid var(--border);
border-radius: var(--radius);
padding: 20px 24px;
margin-bottom: 0;
}
.modal-actions {
display: flex;
justify-content: flex-end;
gap: 10px;
margin-top: 20px;
}
@media (max-width: 900px) {
.wrap {
padding: 16px;
}
.nav {
padding: 10px 16px;
}
}
+271
View File
@@ -0,0 +1,271 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>User Allowlist — CoreCI Broker</title>
<link rel="stylesheet" href="/ui/shared.css">
<style>
.page-head {
display: flex;
align-items: flex-start;
justify-content: space-between;
gap: 16px;
flex-wrap: wrap;
margin-bottom: 8px;
}
.page-sub {
color: var(--muted);
font-size: 14px;
margin: 0;
}
.table-empty {
color: var(--muted);
}
.cell-actions {
display: flex;
gap: 8px;
flex-wrap: wrap;
}
.btn-danger {
color: var(--danger);
background: transparent;
border: 1px solid rgba(240, 122, 114, 0.45);
}
.badge-revoked {
color: var(--danger);
border-color: rgba(240, 122, 114, 0.45);
background: rgba(240, 122, 114, 0.08);
}
</style>
</head>
<body>
<div id="nav"></div>
<main class="wrap">
<section class="card">
<div class="page-head">
<div>
<h1>User Allowlist</h1>
<p class="page-sub">User ids allowed to register agents against the broker.</p>
</div>
<button type="button" class="btn btn-primary" id="add-open">Add user</button>
</div>
<p class="form-error" id="page-error" role="alert" hidden></p>
<table>
<thead>
<tr>
<th>User ID</th>
<th>Email</th>
<th>Status</th>
<th>Created</th>
<th>Revoked</th>
<th>Actions</th>
</tr>
</thead>
<tbody id="rows"></tbody>
</table>
</section>
</main>
<div class="modal-backdrop" id="modal" hidden role="dialog" aria-modal="true" aria-labelledby="modal-title">
<div class="modal">
<h2 id="modal-title">Add user</h2>
<form id="add-form">
<div class="field">
<label for="user_id">User ID</label>
<input id="user_id" name="user_id" type="text" class="mono" required>
</div>
<div class="field">
<label for="email">Email (optional)</label>
<input id="email" name="email" type="text" placeholder="user@example.com">
</div>
<p class="form-error" id="add-error" role="alert" hidden></p>
<div class="modal-actions">
<button type="button" class="btn btn-subtle" id="add-cancel">Cancel</button>
<button type="submit" class="btn btn-primary" id="add-submit">Add</button>
</div>
</form>
</div>
</div>
<script src="/ui/nav.js"></script>
<script>
(function () {
var API = "/api/broker/user_allowlist";
var rows = document.getElementById("rows");
var pageError = document.getElementById("page-error");
var modal = document.getElementById("modal");
var form = document.getElementById("add-form");
var userInput = document.getElementById("user_id");
var emailInput = document.getElementById("email");
var addError = document.getElementById("add-error");
var submitButton = document.getElementById("add-submit");
function showPageError(message) {
pageError.textContent = message;
pageError.hidden = false;
}
function clearPageError() {
pageError.hidden = true;
pageError.textContent = "";
}
function messageRow(text) {
rows.replaceChildren();
var tr = document.createElement("tr");
var td = document.createElement("td");
td.colSpan = 6;
td.className = "table-empty";
td.textContent = text;
tr.appendChild(td);
rows.appendChild(tr);
}
function textCell(value, mono) {
var td = document.createElement("td");
if (mono) {
td.className = "mono";
}
td.textContent = value === null || value === undefined || value === "" ? "—" : value;
return td;
}
function statusBadge(status) {
var span = document.createElement("span");
span.className = "badge " + (status === "active" ? "badge-active" : "badge-revoked");
span.textContent = status;
return span;
}
function actionButton(label, className, disabled, handler) {
var button = document.createElement("button");
button.type = "button";
button.className = "btn " + className;
button.textContent = label;
button.disabled = disabled;
button.addEventListener("click", handler);
return button;
}
function renderRow(row) {
var tr = document.createElement("tr");
tr.appendChild(textCell(row.user_id, true));
tr.appendChild(textCell(row.email));
var statusCell = document.createElement("td");
statusCell.appendChild(statusBadge(row.status));
tr.appendChild(statusCell);
tr.appendChild(textCell(row.created_at));
tr.appendChild(textCell(row.revoked_at));
var actionsCell = document.createElement("td");
var actions = document.createElement("div");
actions.className = "cell-actions";
var key = encodeURIComponent(row.user_id);
actions.appendChild(actionButton("Revoke", "btn-danger", row.status !== "active", function () {
rowAction("DELETE", API + "/" + key, "Revoke allowlisted user " + row.user_id + "?", "Revoke failed", row.user_id);
}));
actions.appendChild(actionButton("Reinstate", "btn-subtle", row.status === "active", function () {
rowAction("POST", API + "/" + key + "/reinstate", "Reinstate allowlisted user " + row.user_id + "?", "Reinstate failed", row.user_id);
}));
actionsCell.appendChild(actions);
tr.appendChild(actionsCell);
return tr;
}
async function refresh() {
messageRow("Loading allowlist…");
var result = await api(API);
if (!result.ok) {
messageRow(result.status === 0 ? "Broker unreachable." : "Could not load allowlist.");
return;
}
var list = (result.data && result.data.allowlist) || [];
if (!list.length) {
messageRow("No allowlisted users yet.");
return;
}
rows.replaceChildren();
list.forEach(function (row) {
rows.appendChild(renderRow(row));
});
}
async function rowAction(method, path, question, label, userId) {
if (!confirm(question)) {
return;
}
clearPageError();
var result = await api(path, {method: method});
if (!result.ok) {
if (result.status === 0) {
showPageError(label + ": broker unreachable.");
} else {
var detail = result.data && result.data.message ? ": " + result.data.message : ".";
showPageError(label + " " + userId + detail);
}
}
await refresh();
}
function openModal() {
form.reset();
addError.hidden = true;
addError.textContent = "";
submitButton.disabled = false;
modal.hidden = false;
userInput.focus();
}
function closeModal() {
modal.hidden = true;
}
form.addEventListener("submit", async function (event) {
event.preventDefault();
addError.hidden = true;
addError.textContent = "";
submitButton.disabled = true;
var payload = {user_id: userInput.value.trim()};
var email = emailInput.value.trim();
if (email) {
payload.email = email;
}
var result = await api(API, {method: "POST", body: payload});
submitButton.disabled = false;
if (result.ok) {
closeModal();
clearPageError();
await refresh();
return;
}
if (result.status === 0) {
addError.textContent = "Broker unreachable.";
} else if (result.data && result.data.message) {
addError.textContent = result.data.message;
} else {
addError.textContent = "Could not add user.";
}
addError.hidden = false;
await refresh();
});
document.getElementById("add-open").addEventListener("click", openModal);
document.getElementById("add-cancel").addEventListener("click", closeModal);
modal.addEventListener("click", function (event) {
if (event.target === modal) {
closeModal();
}
});
document.addEventListener("keydown", function (event) {
if (event.key === "Escape" && !modal.hidden) {
closeModal();
}
});
refresh();
})();
</script>
</body>
</html>
+25
View File
@@ -0,0 +1,25 @@
# Reference Tools
Templates for Open WebUI tool authors wiring tools to the CoreCI Broker.
They contain no secrets and no live-infrastructure client.
- `proxmox_readonly_reference.py` - the full broker flow (spec section 11):
first call registers and persists agent state, then fetches
`coreci/proxmox`; later calls load the persisted token via `get_state`.
- `minimal_auth_showcase.py` - the minimal pattern: authenticate with the
broker, then use a second tool.
The flow both templates follow:
1. The CEO allowlists the tool's app UUID and user id in the broker UI.
2. Fetch the client from `/static/coreci_broker_client.py` and import it.
3. `ensure_registered(state)` registers (idempotent) and merges the token
into your state dict; `put_state(state)` persists it as the tool's
only durable state channel.
4. `read_secret("coreci/<agent_id>", state)` fetches the credential; on a
401 the client re-registers and retries once.
Downstream calls default to a safe dry-run with no live credential use;
replace the dry-run block with real calls in your own tool.
Client module: `coreci_broker_client.py` (stdlib urllib only, zero deps).
+45
View File
@@ -0,0 +1,45 @@
"""title: Minimal Auth Showcase [CoreCI]
author: coreci
description: Minimal Open WebUI tool template showing the CoreCI Broker pattern - authenticate with the broker first, then use a second tool.
version: 0.1.0
requirements: []
licence: MIT
"""
import coreci_broker_client
class Tools:
class Valves:
base_url = ""
agent_id = ""
app_uuid = ""
user_id = ""
def __init__(self):
self.valves = self.Valves()
self._state = {}
def _client(self):
return coreci_broker_client.BrokerClient(
self.valves.base_url, self.valves.agent_id,
self.valves.app_uuid, self.valves.user_id, ["coreci"],
)
def demo(self, context=None) -> dict:
"""Demonstrate broker auth before any downstream tool call.
Ensures the agent is registered with the broker (idempotent), then
reads the agent's own secret through the client: the credential
fetch itself proves the auth path works before any second tool
would use the credential. The secret material never appears in the
returned marker.
:param context: optional Open WebUI call context, unused
:return: marker dict with registered and secret_read flags
"""
client = self._client()
client.ensure_registered(self._state)
client.read_secret("coreci/%s" % self.valves.agent_id, self._state)
return {"registered": True, "secret_read": True,
"path": "coreci/%s" % self.valves.agent_id,
"note": "auth first, then use"}
@@ -0,0 +1,104 @@
"""title: Proxmox Read-Only Reference [CoreCI]
author: coreci
description: Open WebUI tool template for the CoreCI Broker flow (spec section 11) - register, persist agent state, read coreci/proxmox, then act; downstream Proxmox calls default to a safe dry-run.
version: 0.1.0
requirements: []
licence: MIT
"""
import datetime
import coreci_broker_client
SECRET_PATH = "coreci/proxmox"
DRY_RUN_NOTE = (
"reference dry-run: the broker credential was fetched but stays unused; "
"replace this block with a real read-only Proxmox call to go live"
)
class Tools:
class Valves:
base_url = ""
agent_id = ""
app_uuid = ""
user_id = ""
def __init__(self):
self.valves = self.Valves()
self._state = {}
def _client(self):
return coreci_broker_client.BrokerClient(
self.valves.base_url, self.valves.agent_id,
self.valves.app_uuid, self.valves.user_id, ["proxmox", "readonly"],
)
def _persist(self, client):
now = datetime.datetime.now(datetime.timezone.utc).isoformat()
self._state.setdefault("first_registered_at", now)
self._state["last_seen_at"] = now
client.put_state(self._state)
def _load_persisted(self, client):
document = client.get_state(self._state)
if document is not None:
for key in document:
self._state.setdefault(key, document[key])
def _credential(self):
client = self._client()
first_call = self._state.get("current_token") is None
fingerprint = (self._state.get("current_token"),
self._state.get("token_expires_at"))
if first_call:
client.ensure_registered(self._state)
self._persist(client)
else:
self._load_persisted(client)
document = client.read_secret(SECRET_PATH, self._state)
current = (self._state.get("current_token"),
self._state.get("token_expires_at"))
if not first_call and current != fingerprint:
self._persist(client)
return document
def _dry_run(self, action, document):
return {
"dry_run": True,
"action": action,
"credential_present": isinstance(document, dict) and "data" in document,
"note": DRY_RUN_NOTE,
}
def list_nodes(self, context=None) -> dict:
"""List the Proxmox cluster nodes as a safe dry-run by default.
First call registers with the broker (idempotent) and persists the
agent state; subsequent calls load the persisted token and reuse
it. The credential is fetched from coreci/proxmox before any
downstream action, and on a 401 the client re-registers with a
single retry after which the new token is persisted again. The
reference template then returns a dry-run marker and performs no
live Proxmox call: replace the dry-run block with a real read-only
API call when wiring live infrastructure; this template ships no
live-credential use path.
:param context: optional Open WebUI call context, unused
:return: dry-run response dict with credential presence
"""
return self._dry_run("list_nodes", self._credential())
def vm_status(self, vm_id: str, context=None) -> dict:
"""Report one Proxmox VM status as a safe dry-run by default.
Follows the same broker flow as list_nodes: register and persist
the agent state on the first call, load the persisted token on
later calls, fetch the coreci/proxmox credential, then act. The
dry-run block is the default and no live Proxmox client is wired
in - replace it with a real read-only status call for live use.
:param vm_id: Proxmox VM identifier to look up
:param context: optional Open WebUI call context, unused
:return: dry-run response dict with credential presence
"""
return self._dry_run("vm_status", self._credential())
+112
View File
@@ -0,0 +1,112 @@
#!/usr/bin/env python3
"""Sanctioned secrets loader for the v0.2 distribution scripts (stdlib only).
Secrets resolve ONLY from env files on disk — never from the process
environment (``GITEA_TOKEN`` sits in ``config.security.bash_allowlist
.blocked_env_vars``, so shell-env tokens are structurally blocked; the
v1.8 lesson: an empty or mis-sourced token makes gitea answer 404 rather
than 401, masking the auth failure).
Parsing mirrors ``coreci_broker._parse_env_file`` semantics exactly:
blank lines and ``#`` comments are skipped, keys/values are whitespace-
stripped, and a value wrapped in matching single or double quotes is
unquoted.
File precedence — later files win. Locations in order (``root``, then the
sanctioned ``.ciagent/`` dir under it); each location in ``config.json``
``secrets.env_files`` pattern order (``.env``, ``.env.secrets``, then the
``.env.*`` glob, sorted)::
<root>/.env, <root>/.env.secrets, <root>/.env.*
<root>/.ciagent/.env, <root>/.ciagent/.env.secrets, <root>/.ciagent/.env.*
``load()`` returns the merged mapping; ``get_token()`` enforces the
non-empty assert that callers MUST let happen before their first API
call. This module never prints secret values.
"""
import glob
import os
ENV_FILE_PATTERNS = (".env", ".env.secrets", ".env.*")
CIAGENT_SUBDIR = ".ciagent"
class SecretsError(RuntimeError):
"""A required secret is missing or empty in the sanctioned env files."""
def default_root():
"""Repo root: the parent directory of this scripts/ folder."""
return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
def _parse_env_file(path):
"""Mirror coreci_broker._parse_env_file (comments, blanks, quotes)."""
entries = []
if not os.path.isfile(path):
return entries
with open(path, "r", encoding="utf-8") as handle:
for raw in handle:
line = raw.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, value = (part.strip() for part in line.partition("="))
if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
value = value[1:-1]
if key:
entries.append((key, value))
return entries
def _candidate_files(root):
"""Ordered, deduplicated env-file paths; later files win."""
seen = set()
files = []
for location in (root, os.path.join(root, CIAGENT_SUBDIR)):
for pattern in ENV_FILE_PATTERNS:
for path in sorted(glob.glob(os.path.join(location, pattern))):
if not os.path.isfile(path):
continue
real = os.path.abspath(path)
if real in seen:
continue
seen.add(real)
files.append(real)
return files
def load(root=None):
"""Merge every sanctioned env file under ``root`` into one dict.
``root`` defaults to the repo root (the parent of scripts/).
"""
base = os.path.abspath(root) if root is not None else default_root()
values = {}
for path in _candidate_files(base):
for key, value in _parse_env_file(path):
values[key] = value
return values
def get_token(var="GITEA_TOKEN", root=None):
"""Return the non-empty value of ``var`` from the env files only.
Raises SecretsError when the value is missing, empty, or whitespace-
only — callers must surface this BEFORE any HTTP call (an empty token
makes gitea answer 404 and masks the auth failure — the v1.8 lesson).
"""
token = load(root).get(var, "")
if not token.strip():
base = os.path.abspath(root) if root is not None else default_root()
searched = ", ".join(
os.path.join(location, pattern)
for location in (base, os.path.join(base, CIAGENT_SUBDIR))
for pattern in ENV_FILE_PATTERNS
)
raise SecretsError(
"{} is missing or empty in the sanctioned env files "
"(searched: {}); it never resolves from the process "
"environment".format(var, searched)
)
return token.strip()
+141
View File
@@ -0,0 +1,141 @@
#!/usr/bin/env python3
"""Build the coreci-broker CLI zipapp artifact (stdlib only).
Usage: python3 scripts/build_cli.py <version>
Stages an explicit allowlist into a temp dir, packs it with
``python3 -m zipapp ... --python "/usr/bin/env python3" --compress``, then
runs a normalize post-pass (G-009) so same-environment rebuilds are
byte-identical: entries sorted by name, ``date_time=(1980,1,1,0,0,0)``,
``ZIP_DEFLATED``, ``external_attr`` set to constants (files
``0o100644 << 16``, dirs ``0o40755 << 16`` — never preserved/umask-derived),
shebang re-prepended, exec bit preserved. Prints the artifact sha256.
"""
import argparse
import hashlib
import io
import os
import shutil
import stat
import subprocess
import sys
import tempfile
import zipfile
ROOT_SOURCES = ("coreci_broker.py", "coreci_broker_client.py")
UI_SOURCE_DIR = "coreci_broker_ui"
SHIM_SOURCE = os.path.join("cli", "__main__.py")
STAGED_SHIM = "__main__.py"
VERSION_MODULE = "_cli_version.py"
FIXED_DATE_TIME = (1980, 1, 1, 0, 0, 0)
FILE_EXTERNAL_ATTR = 0o100644 << 16
DIR_EXTERNAL_ATTR = 0o40755 << 16
def repo_root():
return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
def artifact_name(version):
tag = version[1:] if version.startswith("v") else version
return "coreci-broker-v{}.pyz".format(tag)
def stage_sources(repo, staging):
for name in ROOT_SOURCES:
shutil.copyfile(os.path.join(repo, name), os.path.join(staging, name))
ui_root = os.path.join(repo, UI_SOURCE_DIR)
staged_ui = os.path.join(staging, UI_SOURCE_DIR)
for current, _dirs, files in os.walk(ui_root):
relative = os.path.relpath(current, ui_root)
target_dir = staged_ui if relative == "." else os.path.join(staged_ui, relative)
os.makedirs(target_dir, exist_ok=True)
for name in sorted(files):
shutil.copyfile(os.path.join(current, name), os.path.join(target_dir, name))
shutil.copyfile(os.path.join(repo, SHIM_SOURCE), os.path.join(staging, STAGED_SHIM))
def write_version_module(staging, version):
with open(os.path.join(staging, VERSION_MODULE), "w", encoding="utf-8") as handle:
handle.write('VERSION = "{}"\n'.format(version))
def run_zipapp(staging, artifact):
command = [
sys.executable, "-m", "zipapp", staging,
"-o", artifact,
"--python", "/usr/bin/env python3",
"--compress",
]
completed = subprocess.run(command, capture_output=True, text=True)
if completed.returncode != 0:
detail = (completed.stderr or completed.stdout).strip()
raise RuntimeError(
"zipapp failed with code {}: {}".format(completed.returncode, detail)
)
def normalize_archive(artifact):
with open(artifact, "rb") as handle:
raw = handle.read()
if not raw.startswith(b"#!"):
raise RuntimeError("zipapp artifact is missing its shebang prefix")
shebang = raw.split(b"\n", 1)[0] + b"\n"
entries = []
with zipfile.ZipFile(io.BytesIO(raw)) as archive:
for info in archive.infolist():
payload = b"" if info.is_dir() else archive.read(info)
entries.append((info.filename, payload))
entries.sort(key=lambda entry: entry[0])
body = io.BytesIO()
with zipfile.ZipFile(body, "w", compression=zipfile.ZIP_DEFLATED) as archive:
for name, payload in entries:
info = zipfile.ZipInfo(name, date_time=FIXED_DATE_TIME)
info.compress_type = zipfile.ZIP_DEFLATED
info.external_attr = DIR_EXTERNAL_ATTR if name.endswith("/") else FILE_EXTERNAL_ATTR
archive.writestr(info, payload)
mode = os.stat(artifact).st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH
temp_path = artifact + ".tmp"
with open(temp_path, "wb") as handle:
handle.write(shebang)
handle.write(body.getvalue())
os.replace(temp_path, artifact)
os.chmod(artifact, mode)
def sha256_file(path):
digest = hashlib.sha256()
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(65536), b""):
digest.update(chunk)
return digest.hexdigest()
def build(version, repo=None, dist_dir=None):
repo = os.path.abspath(repo or repo_root())
dist_dir = dist_dir or os.path.join(repo, "dist")
os.makedirs(dist_dir, exist_ok=True)
artifact = os.path.join(dist_dir, artifact_name(version))
with tempfile.TemporaryDirectory(prefix="coreci-broker-cli-") as staging:
stage_sources(repo, staging)
write_version_module(staging, version)
run_zipapp(staging, artifact)
normalize_archive(artifact)
return artifact
def main(argv=None):
parser = argparse.ArgumentParser(
description="Build the coreci-broker CLI zipapp artifact."
)
parser.add_argument("version", help="release version of the artifact, e.g. v0.2.1")
args = parser.parse_args(argv)
artifact = build(args.version)
print(artifact)
print("sha256: {}".format(sha256_file(artifact)))
return 0
if __name__ == "__main__":
sys.exit(main())
+844
View File
@@ -0,0 +1,844 @@
#!/usr/bin/env python3
"""Daemonless OCI image publish client (D-019; stdlib only).
Usage:
python3 scripts/publish_image.py <tag> [--smoke]
[--artifact PATH] [--hub-registry URL] [--hub-auth URL]
[--gitea-registry URL] [--env-root DIR] [--owner O] [--repo R]
[--base-manifest DIGEST] [--timeout SECS]
Flow (REQ-23, RESEARCH §3): pull the pinned ``python:3.11-slim`` base
from the public registry anonymously (token -> OCI index -> resolve the
linux/amd64 child manifest by digest -> config + layer blobs; digests
recorded for provenance; ``--base-manifest`` overrides the tag
resolution for exact re-publishes) -> authenticate to the gitea
container registry (``GET /v2/token`` with Basic credentials — the
gitea token comes only from the sanctioned env files via
``scripts/_env_secrets.py``; the endpoint answers 200 even anonymously
with an empty-scope JWT, so the issued token must be non-empty and
failures must be loud — never treat a 200 alone as authorization) ->
push: HEAD each blob first (200 -> skip — retries resume cheap; the
~45.7 MB base layers are re-uploaded once per milestone because the
registries cannot cross-mount) -> monolithic blob upload
``POST /v2/<name>/blobs/uploads/?digest=`` (202 + ``Location`` ->
two-step PUT) with explicit Location-following on 307/308 (stdlib
urllib does NOT method-preserve on PUT/POST redirects) -> deterministic
broker layer (sorted tar entries, fixed metadata, member at the stable
path ``/app/coreci-broker.pyz``, ``gzip.compress(..., mtime=0)``) ->
new config blob (base config + appended diff_id; Entrypoint
``["python3", "/app/coreci-broker.pyz"]``, ``Cmd []``, ``WorkingDir
"/state"``, version label; serialized with sort_keys) -> OCI manifest
PUT with the base layer descriptors reused verbatim under ``<tag>``
FIRST, then ``latest`` (D-021).
Exit codes: 0 only after both manifest PUTs confirm 201; 1 after a
remote failure — bounded retries (3 attempts, 2s/4s backoff) exhausted
OR a permanent registry answer — printing the machine-readable
``IMAGE_PENDING <image>:<tag>`` marker for the ship escalation (never
blocks tag+merge); 2 for local precondition errors (missing artifact,
token, or config — nothing attempted remotely, no marker). Token values
are never printed.
Smoke mode (``--smoke``; G-013) — read-only pre-ship verify. NEVER part
of the default publish flow (the flag dispatches an early return; the
publish path above never consults it): four GET-only checks against
the live endpoints, run in order and stopping at the first failure —
nothing is uploaded, no manifest is PUT, no remote state is mutated.
1. ``GET {gitea}/v2/`` must answer 401 with a Bearer challenge whose
realm is the gitea token endpoint (the realm is recorded for 2).
2. Token->JWT exchange at that recorded realm
(``?service=container_registry&scope=repository:{o}/{r}:pull,push``)
with Basic auth — the gitea token resolves ONLY from the sanctioned
env files via ``scripts/_env_secrets.py`` (never the shell
environment); an empty token in the 200 answer is the anonymous-200
trap and fails loudly.
3. ``GET {gitea}/api/v1/repos/{o}/{r}/releases/tags/<tag>`` — 200
(exists) and 404 (not yet created) are BOTH acceptable pre-ship and
the line says which; the release's asset names print as a fact, and
a hard assertion binds ONLY when the tag is exactly ``v0.2.0``
(G-021 forge-side confirmation: the phase-0 release predates the
CLI and must carry NO ``.pyz`` asset, so any ``.pyz`` in its asset
list fails the smoke — the structural half of the guarantee,
release.py taking the tag as a positional argument with no v0.2.0
code path, is already machine-checked in the suite).
4. Docker Hub anonymous index resolve of ``python:3.11-slim`` with the
full Accept set must yield an index whose linux/amd64 child is
resolved; the amd64 child digest is printed — the base-drift fact
the P3-05 ship compares against expectations.
Smoke exit codes mirror the publish flow: 0 all four PASS; 1 with one
clear FAILED line per failure (checks after the failed one never run);
2 for local preconditions (missing token — nothing attempted
remotely). Smoke never prints token values and never emits
``IMAGE_PENDING`` (nothing was attempted, so nothing is pending).
Live gating (G-013): the LIVE smoke executes only at the P3-05 ship
step, which sets ``PUBLISH_SMOKE=1`` in the environment before invoking
this mode — this script neither reads nor requires that variable (the
SHIP GATE owns the gating). The smoke's LOGIC is what the default
unittest suite exercises (stub-tested, zero network access); the live
smoke itself never runs in the default suite.
"""
import argparse
import base64
import gzip
import hashlib
import http.client
import io
import json
import os
import re
import sys
import tarfile
import time
import urllib.error
import urllib.parse
import urllib.request
_SCRIPTS_DIR = os.path.dirname(os.path.abspath(__file__))
if _SCRIPTS_DIR not in sys.path:
sys.path.insert(0, _SCRIPTS_DIR)
import _env_secrets # sibling import (scripts/ dir on sys.path)
METADATA_TIMEOUT_SECONDS = 30
BLOB_TIMEOUT_SECONDS = 120
MAX_ATTEMPTS = 3
BACKOFF_SECONDS = (2, 4)
MAX_REDIRECTS = 5
BASE_REPO = "library/python"
BASE_TAG = "3.11-slim"
HUB_AUTH_SERVICE = "registry.docker.io"
GITEA_AUTH_SERVICE = "container_registry"
# The smoke-mode G-021 forge-side no-.pyz assertion binds ONLY this
# tag: the phase-0 release predates the CLI (D-020 starts at v0.2.1).
SMOKE_G021_TAG = "v0.2.0"
OCI_INDEX = "application/vnd.oci.image.index.v1+json"
OCI_MANIFEST = "application/vnd.oci.image.manifest.v1+json"
OCI_CONFIG = "application/vnd.oci.image.config.v1+json"
OCI_LAYER = "application/vnd.oci.image.layer.v1.tar+gzip"
DOCKER_LIST = "application/vnd.docker.distribution.manifest.list.v2+json"
DOCKER_MANIFEST = "application/vnd.docker.distribution.manifest.v2+json"
ACCEPT_MANIFESTS = ", ".join(
(OCI_INDEX, DOCKER_LIST, OCI_MANIFEST, DOCKER_MANIFEST))
class PublishError(Exception):
"""Image-publisher failure base."""
class TransientFailure(PublishError):
"""Connection error or 5xx persisting past the bounded retries."""
class RegistryApiError(PublishError):
"""Permanent registry answer carrying the remote detail."""
class LocalError(PublishError):
"""Local precondition failure (nothing was attempted remotely)."""
def repo_root():
return os.path.dirname(_SCRIPTS_DIR)
def load_release_defaults(root=None):
"""Read the release block from .ciagent/config.json ({} if absent)."""
path = os.path.join(root or repo_root(), ".ciagent", "config.json")
try:
with open(path, "r", encoding="utf-8") as handle:
config = json.load(handle)
except (OSError, ValueError):
return {}
block = config.get("release")
return block if isinstance(block, dict) else {}
def _error_detail(body):
try:
payload = json.loads(body.decode("utf-8", "replace"))
except ValueError:
text = body.decode("utf-8", "replace").strip()
return text[:200] if text else "<empty body>"
if isinstance(payload, dict):
for key in ("errors", "message"):
if payload.get(key):
return json.dumps(payload[key])[:200]
return json.dumps(payload)[:200] if not isinstance(payload, str) else payload[:200]
def _header(headers, name):
"""Case-insensitive response-header lookup — ``_attempt_once``
returns a plain dict whose keys keep the wire casing, and registry
header casing is not guaranteed (e.g. WWW-Authenticate)."""
for key, value in headers.items():
if key.lower() == name.lower():
return value
return None
def _parse_json(body, context, status):
"""Parse a remote JSON body (token / manifest / config answers).
Malformed bodies raise RegistryApiError — never an uncaught
ValueError — so main()'s handler keeps the exit-1 + IMAGE_PENDING
marker contract for this failure class (CWE-754).
"""
try:
return json.loads(body.decode("utf-8", "replace"))
except ValueError as error:
raise RegistryApiError(
"{} answered malformed JSON ({})".format(context, error),
status, body)
def _attempt_once(method, url, headers, data, timeout):
"""One HTTP exchange; HTTPError becomes a completed answer."""
request = urllib.request.Request(url, data=data, method=method)
for key, value in headers.items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
body = b"" if method == "HEAD" else response.read()
return response.status, body, dict(response.headers.items())
except urllib.error.HTTPError as error:
body = b"" if method == "HEAD" else error.read()
return error.code, body, dict((error.headers or {}).items())
def _origin(url):
"""(scheme, netloc) of a URL — two URLs share an origin iff both match."""
parts = urllib.parse.urlsplit(url)
return (parts.scheme, parts.netloc)
def _attempt_following(method, url, headers, data, timeout):
"""One attempt, following 307/308 redirects explicitly (urllib does
NOT method-preserve on PUT/POST — the Location may be relative).
Cross-origin discipline (CWE-200): when a followed Location resolves
to a different origin than the ORIGINAL request's, the Authorization
header is stripped from the re-issued request — the registry JWT
must never reach another origin (blob-upload Locations are commonly
presigned external storage). Same-origin redirects keep it; once
stripped it is never re-attached."""
current_url = url
current_headers = dict(headers)
for _ in range(MAX_REDIRECTS + 1):
status, body, resp_headers = _attempt_once(
method, current_url, current_headers, data, timeout)
if status in (307, 308) and resp_headers.get("Location"):
next_url = urllib.parse.urljoin(
current_url, resp_headers["Location"])
if _origin(next_url) != _origin(url):
current_headers.pop("Authorization", None)
current_url = next_url
continue
return status, body, resp_headers
raise RegistryApiError(
"too many redirects following {} {}".format(method, url), 500, b"")
def _request(method, url, headers, data, timeout,
max_attempts=None, backoff=None):
"""Bounded-retry HTTP: retries connection errors and 5xx answers.
Returns (status, body, response_headers) for any answer below 500;
a 4xx returns immediately (permanent). Raises TransientFailure once
the retry budget is exhausted.
"""
attempts = MAX_ATTEMPTS if max_attempts is None else int(max_attempts)
delays = BACKOFF_SECONDS if backoff is None else tuple(backoff)
failure = "unknown failure"
for attempt in range(1, attempts + 1):
try:
status, body, resp_headers = _attempt_following(
method, url, headers, data, timeout)
except (OSError, http.client.HTTPException) as error:
status = None
failure = "connection error: {}: {}".format(
type(error).__name__, error)
else:
if status < 500:
return status, body, resp_headers
failure = "HTTP {}: {}".format(status, _error_detail(body))
if attempt < attempts and delays:
time.sleep(delays[min(attempt - 1, len(delays) - 1)])
raise TransientFailure(
"{} {} failed after {} attempt(s): {}".format(
method, url, attempts, failure))
def _hub_token(hub_auth, scope):
query = urllib.parse.urlencode(
{"service": HUB_AUTH_SERVICE, "scope": scope})
status, body, _ = _request(
"GET", "{}?{}".format(hub_auth, query), {}, None,
METADATA_TIMEOUT_SECONDS)
if status != 200:
raise RegistryApiError("hub token request failed", status, body)
payload = _parse_json(body, "hub token endpoint", status)
token = ""
if isinstance(payload, dict):
token = payload.get("token") or payload.get("access_token") or ""
if not token:
raise RegistryApiError("hub token endpoint returned no token",
status, body)
return token
def _fetch_manifest(registry, repository, ref, bearer, accept=ACCEPT_MANIFESTS):
url = "{}/v2/{}/manifests/{}".format(registry.rstrip("/"), repository, ref)
status, body, _ = _request(
"GET", url, {"Authorization": "Bearer {}".format(bearer),
"Accept": accept}, None, METADATA_TIMEOUT_SECONDS)
if status != 200:
raise RegistryApiError(
"manifest fetch {} {}".format(repository, ref), status, body)
return _parse_json(
body, "manifest fetch {} {}".format(repository, ref), status)
def _resolve_amd64_child(index):
"""linux/amd64 child digest from a base index; malformed index
shapes degrade to the not-found RegistryApiError, never a crash."""
manifests = index.get("manifests") if isinstance(index, dict) else None
if not isinstance(manifests, list):
manifests = []
for descriptor in manifests:
if not isinstance(descriptor, dict):
continue
platform = descriptor.get("platform")
if not isinstance(platform, dict):
continue
if platform.get("os") == "linux" and platform.get(
"architecture") == "amd64":
return descriptor.get("digest")
raise RegistryApiError(
"no linux/amd64 child manifest in the base index", 404,
json.dumps(manifests).encode("utf-8", "replace"))
def _fetch_blob(registry, repository, digest, bearer, timeout):
url = "{}/v2/{}/blobs/{}".format(
registry.rstrip("/"), repository, digest)
status, body, _ = _request(
"GET", url, {"Authorization": "Bearer {}".format(bearer)}, None,
timeout)
if status != 200:
raise RegistryApiError("blob fetch {}".format(digest), status, body)
if "sha256:" + hashlib.sha256(body).hexdigest() != digest:
raise RegistryApiError("blob digest mismatch {}".format(digest),
500, b"")
return body
def _child_parts(child):
"""Validated (config descriptor, layer descriptors) of a base child
manifest — missing or malformed keys raise a clear RegistryApiError
(never an uncaught KeyError), keeping the marker contract for this
failure class (CWE-754)."""
def malformed(reason):
return RegistryApiError(
"malformed base child manifest: {}".format(reason), 500,
json.dumps(child).encode("utf-8", "replace"))
if not isinstance(child, dict):
raise malformed("not a JSON object")
config = child.get("config")
if not isinstance(config, dict) or not config.get("digest"):
raise malformed("missing or empty config.digest")
layers = child.get("layers")
if not isinstance(layers, list):
raise malformed("missing or non-list layers")
for descriptor in layers:
if not isinstance(descriptor, dict) or not descriptor.get("digest"):
raise malformed("layer descriptor without a digest")
return config, layers
def _gitea_jwt(token_url, owner, repo, token, timeout):
"""Token->JWT exchange at a token endpoint URL (Basic auth).
``token_url`` is the FULL endpoint: the registry's ``/v2/token``
for the publish flow, or the realm recorded from a ``/v2/``
challenge for the smoke (the standard token-auth flow).
"""
query = urllib.parse.urlencode({
"service": GITEA_AUTH_SERVICE,
"scope": "repository:{}/{}:pull,push".format(owner, repo),
})
basic = base64.b64encode(
"publisher:{}".format(token).encode("utf-8")).decode("ascii")
status, body, _ = _request(
"GET", "{}?{}".format(token_url, query),
{"Authorization": "Basic {}".format(basic)}, None, timeout)
if status != 200:
raise RegistryApiError("gitea registry token request failed",
status, body)
payload = _parse_json(body, "gitea registry token endpoint", status)
jwt = (payload.get("token") or "") if isinstance(payload, dict) else ""
if not jwt:
# The endpoint answers 200 even anonymously with an unusable
# (empty-scope) JWT — a 200 proves nothing about push rights.
raise RegistryApiError(
"gitea registry token endpoint returned an empty token "
"(anonymous-200 trap refused; never treat 200 as authorization)",
status, body)
return jwt
def _bearer_challenge(www_authenticate):
"""Realm recorded from a Bearer WWW-Authenticate challenge.
Returns None when the header is absent, is not a Bearer challenge,
or carries no realm — a challenge without a realm cannot be
followed to the token endpoint, so the smoke treats that as no
challenge at all.
"""
if not www_authenticate:
return None
parts = www_authenticate.strip().split(None, 1)
if len(parts) < 2 or parts[0].lower() != "bearer":
return None
match = re.search(r'\brealm\s*=\s*"([^"]*)"', parts[1])
return match.group(1) if match else None
def _asset_names(release):
"""Asset display names from a gitea release payload — odd shapes
degrade to [] (the read-only smoke never crashes on them)."""
if not isinstance(release, dict):
return []
assets = release.get("assets")
if not isinstance(assets, list):
return []
names = []
for asset in assets:
if isinstance(asset, dict) and isinstance(asset.get("name"), str):
names.append(asset["name"])
return names
def _blob_exists(gitea_registry, name, digest, jwt, timeout):
url = "{}/v2/{}/blobs/{}".format(
gitea_registry.rstrip("/"), name, digest)
status, _, _ = _request(
"HEAD", url, {"Authorization": "Bearer {}".format(jwt)}, None,
timeout)
return status == 200
def _upload_blob(gitea_registry, name, digest, blob, jwt, timeout):
base_url = gitea_registry.rstrip("/")
url = "{}/v2/{}/blobs/uploads/?digest={}".format(base_url, name, digest)
headers = {"Authorization": "Bearer {}".format(jwt),
"Content-Type": "application/octet-stream"}
status, body, resp_headers = _request(
"POST", url, headers, blob, timeout)
if status in (200, 201):
return
if status == 202 and resp_headers.get("Location"):
location = urllib.parse.urljoin(url, resp_headers["Location"])
if "digest=" not in location:
separator = "&" if "?" in location else "?"
location = "{}{}digest={}".format(location, separator, digest)
status, body, _ = _request(
"PUT", location, headers, blob, timeout)
if status in (200, 201):
return
raise RegistryApiError("blob upload {}".format(digest), status, body)
def _push_blob_if_missing(gitea_registry, name, digest, blob, jwt, timeout):
"""HEAD-skip then upload — retries resume cheap."""
if _blob_exists(gitea_registry, name, digest, jwt, timeout):
return False
_upload_blob(gitea_registry, name, digest, blob, jwt, timeout)
return True
def build_broker_layer(artifact_bytes):
"""Deterministic layer carrying /app/coreci-broker.pyz.
Returns (gzip_blob, layer_digest, diff_id): sorted single entry,
fixed TarInfo metadata, gzip with mtime=0 — byte-stable rebuilds.
"""
info = tarfile.TarInfo("app/coreci-broker.pyz")
info.size = len(artifact_bytes)
info.mode = 0o644
info.uid = 0
info.gid = 0
info.uname = ""
info.gname = ""
info.mtime = 0
buffer = io.BytesIO()
with tarfile.open(fileobj=buffer, mode="w",
format=tarfile.GNU_FORMAT) as archive:
archive.addfile(info, io.BytesIO(artifact_bytes))
raw = buffer.getvalue()
compressed = gzip.compress(raw, compresslevel=9, mtime=0)
layer_digest = "sha256:" + hashlib.sha256(compressed).hexdigest()
diff_id = "sha256:" + hashlib.sha256(raw).hexdigest()
return compressed, layer_digest, diff_id
def build_config_blob(base_config, diff_id, tag):
"""Base config + appended diff_id + CLI entrypoint/WorkingDir/label.
Serialized with sort_keys — deterministic config digest.
"""
config = dict(base_config)
rootfs = dict(base_config.get("rootfs") or {})
rootfs["diff_ids"] = list(rootfs.get("diff_ids") or []) + [diff_id]
if "type" not in rootfs:
rootfs["type"] = "layers"
config["rootfs"] = rootfs
inner = dict(base_config.get("config") or {})
inner["Entrypoint"] = ["python3", "/app/coreci-broker.pyz"]
inner["Cmd"] = []
inner["WorkingDir"] = "/state"
inner["Labels"] = {"org.opencontainers.image.version": tag}
config["config"] = inner
return json.dumps(config, sort_keys=True,
separators=(",", ":")).encode("utf-8")
def build_manifest_blob(base_layer_descriptors, broker_layer_descriptor,
config_descriptor):
manifest = {
"schemaVersion": 2,
"mediaType": OCI_MANIFEST,
"config": config_descriptor,
"layers": list(base_layer_descriptors) + [broker_layer_descriptor],
}
return json.dumps(manifest, sort_keys=True,
separators=(",", ":")).encode("utf-8")
def _descriptor(digest, size, media_type):
return {"mediaType": media_type, "digest": digest, "size": size}
def _put_manifest(gitea_registry, name, tag, manifest_blob, jwt, timeout):
url = "{}/v2/{}/manifests/{}".format(
gitea_registry.rstrip("/"), name, tag)
status, body, _ = _request(
"PUT", url, {"Authorization": "Bearer {}".format(jwt),
"Content-Type": OCI_MANIFEST}, manifest_blob, timeout)
if status not in (200, 201):
raise RegistryApiError("manifest PUT {}".format(tag), status, body)
return "sha256:" + hashlib.sha256(manifest_blob).hexdigest()
def _load_artifact(path):
if not os.path.isfile(path):
raise LocalError("artifact not found: {}".format(path))
with open(path, "rb") as handle:
return handle.read()
def smoke(tag, gitea_registry, hub_registry, hub_auth, owner, repo,
timeout, env_root):
"""G-013 read-only live smoke: the four GET-only pre-ship checks.
Never part of the default publish flow; never mutates remote state
(no uploads, no manifest PUTs). The LIVE invocation happens only
at the P3-05 ship step, which gates it by setting PUBLISH_SMOKE=1
in the environment — this script neither reads nor requires that
variable (the SHIP GATE owns the gating). The default unittest
suite exercises this LOGIC against stubs (zero network).
Checks, in order, stopping at the first failure (see the module
docstring for the full G-021 tie-in): [1/4] the gitea /v2/ 401
Bearer challenge (realm recorded); [2/4] the token->JWT exchange at
the recorded realm — the gitea token resolves only from the
sanctioned env files via ``_env_secrets.get_token`` (never the
shell environment), and an empty token is the anonymous-200 trap
refused loudly; [3/4] the gitea API release-by-tag GET — 200
(exists) and 404 (not yet created) are both acceptable pre-ship,
asset names print as a fact, and the hard no-.pyz assertion binds
ONLY the tag v0.2.0 exactly (G-021 forge-side confirmation);
[4/4] the Docker Hub anonymous index resolve of python:3.11-slim —
the linux/amd64 child digest prints (the drift fact P3-05
compares against expectations).
Returns 0 (all PASS), or 1 after one clear FAILED line (token
values are never printed; IMAGE_PENDING is never emitted — nothing
was attempted, so nothing is pending). Local preconditions (a
missing token) raise ``_env_secrets.SecretsError`` BEFORE any
request; main() maps that to exit 2.
"""
gitea_token = _env_secrets.get_token(root=env_root)
metadata_timeout = min(METADATA_TIMEOUT_SECONDS, timeout)
gitea_base = gitea_registry.rstrip("/")
# [1/4] gitea /v2/ auth challenge (anonymous; realm recorded).
try:
status, _, resp_headers = _request(
"GET", "{}/v2/".format(gitea_base), {}, None, metadata_timeout)
except (TransientFailure, RegistryApiError) as error:
print("FAILED [1/4] gitea /v2/ auth challenge: {}".format(error))
return 1
challenge = _header(resp_headers, "WWW-Authenticate")
realm = _bearer_challenge(challenge)
if status != 401 or not realm:
print("FAILED [1/4] gitea /v2/ auth challenge: expected 401 with "
"a Bearer challenge, got HTTP {} (WWW-Authenticate: {!r})"
.format(status, challenge))
return 1
print("PASS [1/4] gitea /v2/ auth challenge: 401 + Bearer challenge "
"(realm {})".format(realm))
# [2/4] token->JWT exchange at the recorded realm (Basic auth).
try:
_gitea_jwt(realm, owner, repo, gitea_token, metadata_timeout)
except (TransientFailure, RegistryApiError) as error:
print("FAILED [2/4] gitea token->JWT exchange at {}: {}".format(
realm, error))
return 1
print("PASS [2/4] gitea token->JWT exchange: 200 + non-empty token "
"(service {}, scope repository:{}/{}:pull,push)".format(
GITEA_AUTH_SERVICE, owner, repo))
# [3/4] release-by-tag GET (gitea API; 200 exists, 404 pre-ship ok).
release_url = "{}/api/v1/repos/{}/{}/releases/tags/{}".format(
gitea_base, owner, repo, urllib.parse.quote(tag, safe=""))
try:
status, body, _ = _request(
"GET", release_url,
{"Authorization": "token {}".format(gitea_token)}, None,
metadata_timeout)
except (TransientFailure, RegistryApiError) as error:
print("FAILED [3/4] release-by-tag {}: {}".format(tag, error))
return 1
if status == 404:
print("PASS [3/4] release-by-tag {}: not yet created (404) — "
"acceptable pre-ship".format(tag))
elif status != 200:
print("FAILED [3/4] release-by-tag {}: expected 200 or 404, got "
"HTTP {}".format(tag, status))
return 1
else:
try:
release = _parse_json(
body, "release-by-tag {}".format(tag), status)
except RegistryApiError as error:
print("FAILED [3/4] release-by-tag {}: {}".format(tag, error))
return 1
asset_names = _asset_names(release)
pyz_assets = [name for name in asset_names
if name.endswith(".pyz")]
if tag == SMOKE_G021_TAG and pyz_assets:
print("FAILED [3/4] release-by-tag {}: G-021 violation — the "
"phase-0 release must carry no .pyz asset, found: {}"
.format(tag, ", ".join(pyz_assets)))
return 1
listing = ", ".join(asset_names) if asset_names else "(none)"
note = (" — G-021 confirmed: no .pyz asset"
if tag == SMOKE_G021_TAG else "")
print("PASS [3/4] release-by-tag {}: exists (assets: {}){}".format(
tag, listing, note))
# [4/4] Docker Hub index resolve (anonymous; amd64 drift print).
try:
hub_bearer = _hub_token(
hub_auth, "repository:{}:pull".format(BASE_REPO))
index = _fetch_manifest(hub_registry, BASE_REPO, BASE_TAG,
hub_bearer)
child_digest = _resolve_amd64_child(index)
if not child_digest:
raise RegistryApiError(
"amd64 child descriptor carries no digest", 500, b"")
except (TransientFailure, RegistryApiError) as error:
print("FAILED [4/4] hub index resolve {}:{}: {}".format(
BASE_REPO, BASE_TAG, error))
return 1
print("PASS [4/4] hub index resolve {}:{} -> linux/amd64 child {}"
.format(BASE_REPO, BASE_TAG, child_digest))
return 0
def publish(tag, artifact_path, hub_registry, hub_auth, gitea_registry,
owner, repo, base_manifest, timeout, env_root):
"""Run the full publish flow; returns the image provenance dict."""
artifact_bytes = _load_artifact(artifact_path)
gitea_token = _env_secrets.get_token(root=env_root)
name = "{}/{}".format(owner, repo)
metadata_timeout = min(METADATA_TIMEOUT_SECONDS, timeout)
hub_bearer = _hub_token(hub_auth, "repository:{}:pull".format(BASE_REPO))
if base_manifest:
child_digest = base_manifest
child = _fetch_manifest(hub_registry, BASE_REPO, base_manifest,
hub_bearer)
index_digest = None
else:
base = _fetch_manifest(hub_registry, BASE_REPO, BASE_TAG, hub_bearer)
index_digest = "sha256:" + hashlib.sha256(
json.dumps(base, sort_keys=True,
separators=(",", ":")).encode("utf-8")).hexdigest()
child_digest = (_resolve_amd64_child(base)
if "manifests" in base else BASE_TAG)
child = _fetch_manifest(hub_registry, BASE_REPO, child_digest,
hub_bearer)
child_config, base_layer_descriptors = _child_parts(child)
base_config_bytes = _fetch_blob(
hub_registry, BASE_REPO, child_config["digest"], hub_bearer,
timeout)
base_config = _parse_json(base_config_bytes, "base config blob", 200)
if not isinstance(base_config, dict):
raise RegistryApiError("base config blob is not a JSON object",
500, base_config_bytes)
jwt = _gitea_jwt("{}/v2/token".format(gitea_registry.rstrip("/")),
owner, repo, gitea_token, metadata_timeout)
pushed = []
pushed.append(_push_blob_if_missing(
gitea_registry, name, child_config["digest"], base_config_bytes,
jwt, timeout))
for descriptor in base_layer_descriptors:
if _blob_exists(gitea_registry, name, descriptor["digest"], jwt,
timeout):
continue
blob = _fetch_blob(hub_registry, BASE_REPO, descriptor["digest"],
hub_bearer, timeout)
_upload_blob(gitea_registry, name, descriptor["digest"], blob, jwt,
timeout)
pushed.append(True)
layer_blob, layer_digest, diff_id = build_broker_layer(artifact_bytes)
config_blob = build_config_blob(base_config, diff_id, tag)
config_digest = "sha256:" + hashlib.sha256(config_blob).hexdigest()
_push_blob_if_missing(gitea_registry, name, layer_digest, layer_blob,
jwt, timeout)
_push_blob_if_missing(gitea_registry, name, config_digest, config_blob,
jwt, timeout)
manifest_blob = build_manifest_blob(
base_layer_descriptors,
_descriptor(layer_digest, len(layer_blob), OCI_LAYER),
_descriptor(config_digest, len(config_blob), OCI_CONFIG))
manifest_digest = _put_manifest(
gitea_registry, name, tag, manifest_blob, jwt, timeout)
_put_manifest(gitea_registry, name, "latest", manifest_blob, jwt,
timeout)
return {
"base_index_digest": index_digest,
"base_child_digest": child_digest if child_digest != BASE_TAG
else "tag-resolved",
"base_config_digest": child_config["digest"],
"base_layers": [descriptor["digest"]
for descriptor in base_layer_descriptors],
"broker_layer_digest": layer_digest,
"broker_layer_diff_id": diff_id,
"config_digest": config_digest,
"manifest_digest": manifest_digest,
"blobs_uploaded": sum(1 for flag in pushed if flag),
}
def main(argv=None):
parser = argparse.ArgumentParser(
description="Daemonless OCI image publish (REQ-23, D-019)."
)
parser.add_argument("tag", help="image tag, e.g. v0.2.3")
parser.add_argument("--artifact",
help="path to the built .pyz artifact "
"(default: dist/coreci-broker-v<tag>.pyz)")
parser.add_argument("--hub-registry",
default="https://registry-1.docker.io",
help="public base-image registry")
parser.add_argument("--hub-auth", default="https://auth.docker.io/token",
help="public registry token service")
parser.add_argument("--gitea-registry", default="https://git.coreci.dev",
help="gitea registry base URL")
parser.add_argument("--owner", help="gitea owner (default: release.owner)")
parser.add_argument("--repo", help="gitea repo (default: release.repo)")
parser.add_argument("--base-manifest",
help="base child-manifest digest for exact "
"re-publishes (skips tag resolution)")
parser.add_argument("--smoke", action="store_true",
help="G-013 read-only live smoke: four GET-only "
"pre-ship checks, NO publish (see the "
"module docstring)")
parser.add_argument("--timeout", type=float, default=BLOB_TIMEOUT_SECONDS,
help="per-request timeout for blob operations "
"(seconds; metadata uses the smaller of this "
"and 30)")
parser.add_argument("--env-root",
help="root dir for the sanctioned .env* files "
"(default: repo root)")
args = parser.parse_args(argv)
defaults = load_release_defaults()
owner = args.owner or defaults.get("owner")
repo = args.repo or defaults.get("repo")
if not owner or not repo:
print("ERROR: missing gitea target (owner/repo); pass the flags or"
" extend the .ciagent/config.json release block",
file=sys.stderr)
return 2
blob_timeout = max(1.0, float(args.timeout))
if args.smoke:
# G-013 read-only live smoke: dispatched BEFORE any
# publish-side work and returning directly — the default
# publish flow below can never trigger it.
try:
return smoke(
tag=args.tag, gitea_registry=args.gitea_registry,
hub_registry=args.hub_registry, hub_auth=args.hub_auth,
owner=owner, repo=repo, timeout=blob_timeout,
env_root=args.env_root)
except _env_secrets.SecretsError as error:
print("ERROR: {}".format(error), file=sys.stderr)
return 2
artifact_path = args.artifact or os.path.join(
"dist", "coreci-broker-{}.pyz".format(args.tag))
# Reuse the smaller metadata timeout for the registry auth exchange so
# the timeout-abort path stays bounded for every request class.
try:
provenance = publish(
tag=args.tag, artifact_path=artifact_path,
hub_registry=args.hub_registry, hub_auth=args.hub_auth,
gitea_registry=args.gitea_registry, owner=owner, repo=repo,
base_manifest=args.base_manifest, timeout=blob_timeout,
env_root=args.env_root)
except _env_secrets.SecretsError as error:
print("ERROR: {}".format(error), file=sys.stderr)
return 2
except LocalError as error:
print("ERROR: {}".format(error), file=sys.stderr)
return 2
except (TransientFailure, RegistryApiError) as error:
print("ERROR: {}".format(error), file=sys.stderr)
registry_host = urllib.parse.urlsplit(
args.gitea_registry).netloc or args.gitea_registry
print("IMAGE_PENDING {}/{}:{}".format(registry_host, owner, repo,
args.tag))
return 1
print("image: {}/{}:{} (+ latest)".format(
args.gitea_registry.rstrip("/"), owner, repo, args.tag))
print("base index: {}".format(
provenance["base_index_digest"] or "direct (--base-manifest)"))
print("base amd64 manifest: {}".format(provenance["base_child_digest"]))
print("base config: {}".format(provenance["base_config_digest"]))
print("base layers: {}".format(", ".join(provenance["base_layers"])))
print("broker layer: {} (diff_id {})".format(
provenance["broker_layer_digest"],
provenance["broker_layer_diff_id"]))
print("config: {}".format(provenance["config_digest"]))
print("manifest: {}".format(provenance["manifest_digest"]))
print("blobs uploaded this run: {}".format(provenance["blobs_uploaded"]))
return 0
if __name__ == "__main__":
sys.exit(main())
+380
View File
@@ -0,0 +1,380 @@
#!/usr/bin/env python3
"""Attach the built CLI artifact to a gitea release (stdlib only).
Usage:
python3 scripts/release.py <tag> <artifact>
[--base-url URL] [--owner O] [--repo R] [--env-root DIR]
Flow (REQ-22, RESEARCH §2): resolve the forge token from the sanctioned
env files via ``scripts/_env_secrets.py`` — never the process environment,
asserted non-empty BEFORE the first API call (v1.8 lesson: an empty token
makes gitea answer 404, masking the auth failure) -> compute the artifact
sha256 from the FILE itself (G-017: there is no sha argument, ever) ->
build the release notes (Summary / Requirements / Verification / CLI
install + usage with the digest inline / Docker usage) -> create-or-get
the release by tag -> idempotent attach (same name AND size -> skip) ->
multipart attach (field ``attachment``, ``?name=``, ``Authorization:
token <T>``) under bounded retry (3 attempts, 2s/4s backoff on connection
errors and 5xx).
Exit codes: 0 only on a confirmed or idempotent attach; 1 after a remote
failure — bounded retries exhausted OR a permanent 4xx — both print the
machine-readable ``RELEASE_PENDING <tag> <artifact>`` marker on stdout for
the ship harness's release_pending escalation (tag+merge is never
blocked); 2 for local precondition errors (missing artifact, token, or
config — nothing was attempted remotely, so no marker). Gitea JSON error
bodies are included in failure diagnostics. Token values are never
printed.
"""
import argparse
import hashlib
import http.client
import json
import os
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
_SCRIPTS_DIR = os.path.dirname(os.path.abspath(__file__))
if _SCRIPTS_DIR not in sys.path:
sys.path.insert(0, _SCRIPTS_DIR)
import _env_secrets # sibling import (scripts/ dir on sys.path)
TIMEOUT_SECONDS = 30
MAX_ATTEMPTS = 3
BACKOFF_SECONDS = (2, 4)
ATTACHMENT_FIELD = "attachment"
# Fixed, distinctive boundary: a hand-rolled multipart body needs our own
# delimiter; collision with artifact bytes is negligible at this length.
MULTIPART_BOUNDARY = "coreci-broker-release-boundary-9a7f3c2e6b41"
class ReleaseError(Exception):
"""Release-attacher failure base."""
class TransientFailure(ReleaseError):
"""Connection error or 5xx persisting past the bounded retries."""
class GiteaApiError(ReleaseError):
"""Permanent API answer (4xx or malformed) carrying gitea's detail."""
def __init__(self, context, status, body):
self.status = status
self.body = body
super().__init__("{} -> HTTP {}: {}".format(
context, status, _error_detail(body)))
def repo_root():
return os.path.dirname(_SCRIPTS_DIR)
def load_release_defaults(root=None):
"""Read the release block from .ciagent/config.json ({} if absent)."""
path = os.path.join(root or repo_root(), ".ciagent", "config.json")
try:
with open(path, "r", encoding="utf-8") as handle:
config = json.load(handle)
except (OSError, ValueError):
return {}
block = config.get("release")
return block if isinstance(block, dict) else {}
def _error_detail(body):
"""Summarize a gitea JSON error body ({"message": ...} shape)."""
try:
payload = json.loads(body.decode("utf-8", "replace"))
except ValueError:
text = body.decode("utf-8", "replace").strip()
return text[:200] if text else "<empty body>"
if isinstance(payload, dict) and payload.get("message"):
return str(payload["message"])
return json.dumps(payload)[:200] if not isinstance(payload, str) else payload[:200]
def _api_base(base_url, owner, repo):
return "{}/api/v1/repos/{}/{}".format(base_url.rstrip("/"), owner, repo)
def _image_reference(base_url, owner, repo):
host = urllib.parse.urlsplit(base_url).netloc or base_url
return "{}/{}/{}".format(host.rstrip("/"), owner, repo)
def build_notes(tag, artifact_name, sha256, image_reference):
"""Release-notes body (RESEARCH §2.3 model; REQ-22 CLI + docker docs)."""
lines = [
"## Summary",
"",
"coreci-broker {tag}: the broker packaged as a self-contained CLI"
" zipapp artifact. {artifact} is built by scripts/build_cli.py with"
" a deterministic normalization pass - the same sources rebuild"
" byte-identical on the same environment, so the digest below is"
" meaningful across rebuilds.".format(tag=tag, artifact=artifact_name),
"",
"## Requirements",
"",
"REQ-22: from the first CLI-building release onward, every gitea"
" release attaches the built coreci-broker artifact alongside the"
" existing source archive. The upload resolves the forge token only"
" via the sanctioned .env* files (never the process environment),"
" asserted non-empty before the first API call.",
"",
"## Verification",
"",
"The artifact sha256 in the CLI section below is computed from the"
" attached file itself (G-017 - never an argument, never"
" hand-pasted). Re-run the cumulative suite on the tagged commit:",
"",
" python3 -m unittest discover -s tests",
"",
"## CLI install + usage",
"",
"1. Download {artifact} from this release's assets.".format(
artifact=artifact_name),
"2. Install: chmod +x {artifact} then ./{artifact}, or run it"
" directly with python3 {artifact} (Python 3.11+).".format(
artifact=artifact_name),
"3. Check the build: ./{artifact} --version prints {tag}.".format(
artifact=artifact_name, tag=tag),
"4. The env contract (every BROKER_* variable) is documented in the"
" README; state is anchored by BROKER_ENV_PATH - the .env file's"
" directory is the state dir and relative paths stay CWD-relative.",
"5. First boot prints BROKER_ADMIN_PASS=... exactly once and creates"
" the .env, broker.db, and broker.audit.log state files with mode"
" 0600; the printed admin pass is not shown again - store it.",
"",
"Artifact sha256 (computed from the attached file):",
"",
" sha256:{sha} {artifact}".format(sha=sha256, artifact=artifact_name),
"",
"## Docker usage",
"",
"docker pull {image}:{tag}".format(image=image_reference, tag=tag),
"",
"docker run -p 8765:8765 -v broker-state:/state {image}:{tag}".format(
image=image_reference, tag=tag),
"",
"State lives on the /state volume (WORKDIR /state; the first boot"
" generates the .env, broker.db, and broker.audit.log there) -"
" mount a persistent volume or the admin pass and any registered"
" apps are lost with the container.",
"",
]
return "\n".join(lines)
def _request_once(method, url, token, data=None, headers=None):
"""One HTTP exchange; returns (status, body). HTTPError becomes a
completed answer (status + gitea error body); connection-level errors
propagate for the caller's retry policy."""
request = urllib.request.Request(url, data=data, method=method)
request.add_header("Authorization", "token {}".format(token))
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=TIMEOUT_SECONDS) as response:
return response.status, response.read()
except urllib.error.HTTPError as error:
return error.code, error.read()
def _request(method, url, token, data=None, headers=None,
max_attempts=None, backoff=None):
"""Bounded-retry HTTP: retries connection errors and 5xx answers.
Returns (status, body) for any answer below 500. A 4xx returns
immediately (permanent — surfaced with the gitea error body). Raises
TransientFailure once the retry budget is exhausted.
"""
attempts = MAX_ATTEMPTS if max_attempts is None else int(max_attempts)
delays = BACKOFF_SECONDS if backoff is None else tuple(backoff)
failure = "unknown failure"
for attempt in range(1, attempts + 1):
try:
status, body = _request_once(method, url, token, data, headers)
except (OSError, http.client.HTTPException) as error:
status = None
failure = "connection error: {}: {}".format(
type(error).__name__, error)
else:
if status < 500:
return status, body
failure = "HTTP {}: {}".format(status, _error_detail(body))
if attempt < attempts and delays:
time.sleep(delays[min(attempt - 1, len(delays) - 1)])
raise TransientFailure(
"{} {} failed after {} attempt(s): {}".format(
method, url, attempts, failure))
def _release_id_from(payload_body, context, status):
try:
return int(json.loads(payload_body)["id"])
except (ValueError, KeyError, TypeError):
raise GiteaApiError(context + " returned malformed JSON", status, payload_body)
def _get_or_create_release(base, tag, token, notes):
by_tag = base + "/releases/tags/" + urllib.parse.quote(tag, safe="")
status, body = _request("GET", by_tag, token)
if status == 200:
return _release_id_from(body, "GET release by tag", status)
if status != 404:
# A 401/403 here is a genuine auth failure — the non-empty token
# assert already ran, so this 404-vs-auth ambiguity is resolved.
raise GiteaApiError("GET release by tag failed", status, body)
payload = json.dumps({
"tag_name": tag,
"name": "{}: coreci-broker release".format(tag),
"body": notes,
"draft": False,
"prerelease": False,
}).encode("utf-8")
status, body = _request(
"POST", base + "/releases", token, data=payload,
headers={"Content-Type": "application/json"})
if status not in (200, 201):
raise GiteaApiError("create release failed", status, body)
return _release_id_from(body, "create release", status)
def _assets_url(base, release_id):
return "{}/releases/{}/assets".format(base, release_id)
def _list_assets(base, release_id, token):
status, body = _request("GET", _assets_url(base, release_id), token)
if status != 200:
raise GiteaApiError("list release assets failed", status, body)
try:
assets = json.loads(body)
except ValueError:
raise GiteaApiError(
"list release assets returned malformed JSON", status, body)
if not isinstance(assets, list):
raise GiteaApiError(
"list release assets returned non-list JSON", status, body)
return assets
def _asset_already_attached(assets, name, size):
"""Idempotency: same name AND same size means the upload succeeded
on an earlier (possibly retried) run."""
for asset in assets:
if not isinstance(asset, dict) or asset.get("name") != name:
continue
try:
if int(asset.get("size")) == size:
return True
except (TypeError, ValueError):
continue
return False
def _multipart_body(field, filename, content):
head = (
"--{boundary}\r\n"
"Content-Disposition: form-data; name=\"{field}\";"
" filename=\"{filename}\"\r\n"
"Content-Type: application/octet-stream\r\n"
"\r\n"
).format(boundary=MULTIPART_BOUNDARY, field=field,
filename=filename).encode("utf-8")
tail = "\r\n--{}--\r\n".format(MULTIPART_BOUNDARY).encode("utf-8")
return head + content + tail
def _attach_asset(base, release_id, token, name, content):
url = "{}?{}".format(
_assets_url(base, release_id), urllib.parse.urlencode({"name": name}))
status, body = _request(
"POST", url, token, data=_multipart_body(ATTACHMENT_FIELD, name, content),
headers={
"Content-Type":
"multipart/form-data; boundary={}".format(MULTIPART_BOUNDARY),
})
if status not in (200, 201):
raise GiteaApiError("attach asset failed", status, body)
try:
return json.loads(body)
except ValueError:
return {}
def main(argv=None):
parser = argparse.ArgumentParser(
description="Attach the built CLI artifact to a gitea release (REQ-22)."
)
parser.add_argument("tag", help="release tag, e.g. v0.2.1")
parser.add_argument("artifact", help="path to the built .pyz artifact")
parser.add_argument("--base-url",
help="gitea base URL (default: .ciagent/config.json release.base_url)")
parser.add_argument("--owner",
help="gitea owner (default: release.owner)")
parser.add_argument("--repo",
help="gitea repo (default: release.repo)")
parser.add_argument("--env-root",
help="root dir for the sanctioned .env* files (default: repo root)")
args = parser.parse_args(argv)
defaults = load_release_defaults()
base_url = args.base_url or defaults.get("base_url")
owner = args.owner or defaults.get("owner")
repo = args.repo or defaults.get("repo")
missing = [flag for flag, value in (
("--base-url", base_url), ("--owner", owner), ("--repo", repo))
if not value]
if missing:
print("ERROR: missing gitea target {}; pass the flag(s) or extend"
" the .ciagent/config.json release block".format(
", ".join(missing)), file=sys.stderr)
return 2
if not os.path.isfile(args.artifact):
print("ERROR: artifact not found: {}".format(args.artifact), file=sys.stderr)
return 2
# Token FIRST, from the sanctioned env files only, non-empty enforced —
# strictly before any HTTP call (v1.8: an empty token makes gitea answer
# 404 and masks the auth failure).
try:
token = _env_secrets.get_token(root=args.env_root)
except _env_secrets.SecretsError as error:
print("ERROR: {}".format(error), file=sys.stderr)
return 2
with open(args.artifact, "rb") as handle:
content = handle.read()
# G-017: the digest is computed from the artifact file itself — there is
# no sha argument, ever.
sha256 = hashlib.sha256(content).hexdigest()
name = os.path.basename(args.artifact)
size = len(content)
notes = build_notes(args.tag, name, sha256,
_image_reference(base_url, owner, repo))
base = _api_base(base_url, owner, repo)
try:
release_id = _get_or_create_release(base, args.tag, token, notes)
if _asset_already_attached(_list_assets(base, release_id, token), name, size):
print("attach skipped (idempotent): {} already on {}/{} release"
" {} (sha256 {})".format(name, owner, repo, args.tag, sha256))
return 0
attachment = _attach_asset(base, release_id, token, name, content)
asset_id = attachment.get("id", "?") if isinstance(attachment, dict) else "?"
print("attached: {} -> {}/{} release {} (asset id {}, sha256 {})".format(
name, owner, repo, args.tag, asset_id, sha256))
return 0
except (TransientFailure, GiteaApiError) as error:
print("ERROR: {}".format(error), file=sys.stderr)
print("RELEASE_PENDING {} {}".format(args.tag, args.artifact))
return 1
if __name__ == "__main__":
sys.exit(main())
+345
View File
@@ -0,0 +1,345 @@
import ast
import base64
import contextlib
import datetime
import importlib.util
import io
import json
import os
import sys
import tempfile
import time
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
BROKER_SOURCE_PATH = os.path.join(REPO_ROOT, "coreci_broker.py")
CLIENT_SOURCE_PATH = os.path.join(REPO_ROOT, "coreci_broker_client.py")
CLIENT_STATIC_PATH = "/static/coreci_broker_client.py"
LOGIN_PATH = "/api/broker/auth/login"
LOGOUT_PATH = "/api/broker/auth/logout"
REGISTER_PATH = "/api/broker/agents/register"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
AUDIT_PATH = "/api/broker/audit"
SECRETS_PREFIX = "/api/broker/secrets/"
PROXMOX_PATH = "coreci/proxmox"
GITEA_PATH = "coreci/gitea"
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
GITEA_SECRET = {"url": "https://git.coreci.dev", "token": "git-tok"}
AC_PIN_IDS = {
"AC1": "test_acceptance_p2.AcceptanceP2Tests.test_ac1_first_boot_printed_pass_login_via_ui_login_page",
"AC2": "test_acceptance_p2.AcceptanceP2Tests.test_ac2_app_registration_lifecycle_via_session_cookie",
"AC3": "test_acceptance_p2.AcceptanceP2Tests.test_ac3_user_allowlist_lifecycle_via_session_cookie",
"AC4": "test_acceptance_p3.AcceptanceP3Tests.test_ac4_derivation_identical_both_sides_stored_as_slow_hash",
"AC5": "test_acceptance_p4.AcceptanceP4Tests.test_ac5_valid_token_reads_secret_tampered_token_is_401",
"AC6": "test_acceptance_p4.AcceptanceP4Tests.test_ac6_cross_path_read_is_403_with_denied_audit",
"AC7": "test_acceptance_p3.AcceptanceP3Tests.test_ac7_revoked_or_unknown_uuid_denied_existing_token_honored",
"AC8": "test_acceptance_p3.AcceptanceP3Tests.test_ac8_unknown_or_revoked_user_denied",
"AC9": "test_acceptance_p2.AcceptanceP2Tests.test_ac9_login_and_logout_audit_lines_with_null_identity",
"AC10": "test_acceptance_p3.AcceptanceP3Tests.test_ac10_expiry_401_reregister_refreshes_expiry_and_audits",
"AC11": "test_acceptance_full.AcceptanceFullTests.test_ac11_fresh_boot_state_files_0600_umask_independent",
"AC12": "test_acceptance_p3.AcceptanceP3Tests.test_ac12_state_roundtrip_with_identity_enforcement",
"AC13": "test_acceptance_full.AcceptanceFullTests.test_ac13_client_module_fetch_byte_identical_importable_standalone",
"AC14": "test_acceptance_full.AcceptanceFullTests.test_ac14_coreci_broker_line_count_under_1400",
}
def _basic_header(password):
raw = "%s:%s" % (ADMIN_USER, password)
return {"Authorization": "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")}
class AcceptanceFullTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.creds_path = os.path.join(self.tmp, "creds.json")
with open(self.creds_path, "w", encoding="utf-8") as handle:
handle.write(json.dumps({PROXMOX_PATH: PROXMOX_SECRET, GITEA_PATH: GITEA_SECRET}))
def _env(self, root=None, extra=None):
root = self.tmp if root is None else root
env = {
"BROKER_ENV_PATH": os.path.join(root, ".env"),
"BROKER_DB_PATH": os.path.join(root, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(root, "broker.audit.log"),
"BROKER_CREDS_SOURCE": "file",
"BROKER_CREDS_FILE": self.creds_path,
"BROKER_UI_ROOT": os.path.join(REPO_ROOT, "coreci_broker_ui"),
"BROKER_STATIC_ROOT": REPO_ROOT,
}
env.update(extra or {})
return env
def _boot(self, env):
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer):
broker = coreci_broker.boot(dict(env))
self.addCleanup(broker.audit.close)
self.addCleanup(broker.store.close)
return broker, buffer.getvalue()
def _serve(self, broker):
server = coreci_broker.serve("127.0.0.1", 0, broker)
self.addCleanup(server.server_close)
self.addCleanup(server.shutdown)
return "http://127.0.0.1:%d" % server.server_address[1]
def _printed_pass(self, output):
printed = [
line.split("=", 1)[1]
for line in output.splitlines()
if line.startswith("BROKER_ADMIN_PASS=")
]
self.assertEqual(1, len(printed))
return printed[0]
def _request(self, base, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
return response.status, response.read(), response.headers
except urllib.error.HTTPError as error:
return error.code, error.read(), error.headers
def _request_json(self, base, method, path, payload=None, headers=None):
status, body, response_headers = self._request(base, method, path, payload, headers)
return status, json.loads(body.decode("utf-8")), response_headers
def _assert_null_identity(self, entry):
self.assertIsNone(entry["user_id"])
self.assertIsNone(entry["app_registration_uuid"])
self.assertIsNone(entry["agent_id"])
def _assert_tool_identity(self, entry, app_uuid):
self.assertEqual("ceo", entry["user_id"])
self.assertEqual(app_uuid, entry["app_registration_uuid"])
self.assertEqual("proxmox", entry["agent_id"])
def _assert_state_files_0600(self, env):
for key in ("BROKER_DB_PATH", "BROKER_AUDIT_LOG_PATH", "BROKER_ENV_PATH"):
path = env[key]
self.assertTrue(os.path.isfile(path), path)
self.assertEqual(0o600, os.stat(path).st_mode & 0o777, path)
def test_ac11_fresh_boot_state_files_0600_umask_independent(self):
env = self._env()
self._boot(env)
self._assert_state_files_0600(env)
other = tempfile.TemporaryDirectory()
self.addCleanup(other.cleanup)
other_env = self._env(root=other.name)
previous_umask = os.umask(0o777)
try:
probe = os.path.join(other.name, "umask-probe")
os.close(os.open(probe, os.O_CREAT | os.O_WRONLY, 0o666))
self.assertEqual(0o000, os.stat(probe).st_mode & 0o777)
self._boot(other_env)
finally:
os.umask(previous_umask)
self._assert_state_files_0600(other_env)
def test_ac13_client_module_fetch_byte_identical_importable_standalone(self):
broker, _ = self._boot(self._env())
base = self._serve(broker)
with open(CLIENT_SOURCE_PATH, "rb") as handle:
expected = handle.read()
with urllib.request.urlopen(base + CLIENT_STATIC_PATH, timeout=10) as response:
self.assertEqual(200, response.status)
fetched = response.read()
self.assertEqual(expected, fetched)
imports = set()
for node in ast.walk(ast.parse(fetched.decode("utf-8"))):
if isinstance(node, ast.Import):
imports.update(alias.name for alias in node.names)
elif isinstance(node, ast.ImportFrom):
imports.add(node.module or "")
self.assertTrue(imports)
for name in imports:
self.assertIn(name.split(".")[0], sys.stdlib_module_names, name)
copy_path = os.path.join(self.tmp, "fetched_coreci_broker_client.py")
with open(copy_path, "wb") as handle:
handle.write(fetched)
spec = importlib.util.spec_from_file_location("full_fetched_client", copy_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
self.assertTrue(callable(module.BrokerClient))
self.assertTrue(callable(module.fetch_client_module))
for method in ("ensure_registered", "get_state", "put_state", "read_secret"):
self.assertTrue(callable(getattr(module.BrokerClient, method)), method)
def test_ac14_coreci_broker_line_count_under_1400(self):
with open(BROKER_SOURCE_PATH, "r", encoding="utf-8") as handle:
line_count = sum(1 for _ in handle)
self.assertLess(line_count, 1400)
def test_milestone_story_full_happy_path_consolidates_ac1_ac10_ac12(self):
env = self._env(extra={"BROKER_TOKEN_TTL": "1"})
first, output = self._boot(env)
printed_pass = self._printed_pass(output)
self.assertGreaterEqual(len(printed_pass), 32)
self.assertEqual(1, output.count(printed_pass))
pepper = None
with open(env["BROKER_ENV_PATH"], "r", encoding="utf-8") as handle:
for line in handle:
if line.startswith("BROKER_INSTANCE_PEPPER="):
pepper = line.split("=", 1)[1].strip()
self.assertIsNotNone(pepper)
self.assertNotIn(pepper, output)
second, silent = self._boot(env)
self.assertEqual("", silent)
second.audit.close()
second.store.close()
base = self._serve(first)
status, body, headers = self._request(base, "GET", "/ui/login")
self.assertEqual(200, status)
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
self.assertIn('name="password"', body.decode("utf-8"))
status, body, response_headers = self._request_json(
base, "POST", LOGIN_PATH, {"username": ADMIN_USER, "password": printed_pass}
)
self.assertEqual(200, status)
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
raw_cookie = response_headers.get("Set-Cookie")
self.assertTrue(raw_cookie)
self.assertTrue(raw_cookie.startswith("broker_session="))
self.assertIn("HttpOnly", raw_cookie)
self.assertIn("SameSite=Lax", raw_cookie)
cookie = {"Cookie": raw_cookie.split(";", 1)[0]}
app_uuid = str(uuidlib.uuid4())
status, body, _ = self._request_json(
base, "POST", APPS_PATH,
{"uuid": app_uuid, "name": "proxmox-tool"}, headers=cookie,
)
self.assertEqual(201, status)
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
status, body, _ = self._request_json(
base, "POST", USERS_PATH,
{"user_id": "ceo", "email": "ceo@coreci.dev"}, headers=cookie,
)
self.assertEqual(201, status)
self.assertEqual({"user_id": "ceo", "status": "active"}, body)
with urllib.request.urlopen(base + CLIENT_STATIC_PATH, timeout=10) as response:
self.assertEqual(200, response.status)
client_source = response.read()
with open(CLIENT_SOURCE_PATH, "rb") as handle:
self.assertEqual(handle.read(), client_source)
copy_path = os.path.join(self.tmp, "story_coreci_broker_client.py")
with open(copy_path, "wb") as handle:
handle.write(client_source)
spec = importlib.util.spec_from_file_location("full_story_client", copy_path)
client_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(client_module)
client = client_module.BrokerClient(base, "proxmox", app_uuid, "ceo", ["coreci"])
state = {}
client.ensure_registered(state)
self.assertEqual(44, len(state["current_token"]))
self.assertTrue(state["current_token"].isascii())
self.assertIn("pepper", state)
self.assertEqual("proxmox", state["agent_id"])
self.assertEqual(app_uuid, state["app_registration_uuid"])
self.assertEqual("ceo", state["user_id"])
self.assertTrue(client.put_state(state))
snapshot = dict(state)
self.assertEqual(snapshot, client.get_state(state))
document = client.read_secret(PROXMOX_PATH, state)
self.assertEqual({"path", "data", "version", "updated_at"}, set(document))
self.assertEqual(PROXMOX_PATH, document["path"])
self.assertEqual(PROXMOX_SECRET, document["data"])
with self.assertRaises(client_module.BrokerError) as caught:
client.read_secret(GITEA_PATH, state)
self.assertEqual(403, caught.exception.status)
expired_at = state["token_expires_at"]
time.sleep(1.2)
document = client.read_secret(PROXMOX_PATH, state)
self.assertEqual(PROXMOX_SECRET, document["data"])
self.assertGreater(
datetime.datetime.fromisoformat(state["token_expires_at"]),
datetime.datetime.fromisoformat(expired_at),
)
status, body, _ = self._request_json(base, "POST", LOGOUT_PATH, headers=cookie)
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
status, body, _ = self._request_json(
base, "GET", AUDIT_PATH + "?limit=1000", headers=_basic_header(printed_pass)
)
self.assertEqual(200, status)
self.assertIsNone(body["next_since"])
entries = body["entries"]
def select(action, result=None):
return [
entry for entry in entries
if entry["action"] == action and (result is None or entry["result"] == result)
]
logins = select("login", "ok")
self.assertEqual(1, len(logins))
self.assertEqual(LOGIN_PATH, logins[0]["path"])
self._assert_null_identity(logins[0])
add_apps = select("add_app", "ok")
self.assertEqual(1, len(add_apps))
self.assertEqual(APPS_PATH + "/" + app_uuid, add_apps[0]["path"])
self._assert_null_identity(add_apps[0])
add_users = select("add_user", "ok")
self.assertEqual(1, len(add_users))
self.assertEqual(USERS_PATH + "/ceo", add_users[0]["path"])
self._assert_null_identity(add_users[0])
registers = select("register", "ok")
self.assertGreaterEqual(len(registers), 2)
for entry in registers:
self.assertEqual(REGISTER_PATH, entry["path"])
self._assert_tool_identity(entry, app_uuid)
ok_reads = select("read", "ok")
self.assertGreaterEqual(len(ok_reads), 2)
for entry in ok_reads:
self.assertEqual(PROXMOX_PATH, entry["path"])
self._assert_tool_identity(entry, app_uuid)
denied_reads = select("read", "denied")
self.assertEqual(1, len(denied_reads))
self.assertEqual(GITEA_PATH, denied_reads[0]["path"])
self._assert_tool_identity(denied_reads[0], app_uuid)
auth_denied = [
entry for entry in select("denied", "denied")
if entry["path"] == SECRETS_PREFIX + PROXMOX_PATH
]
self.assertGreaterEqual(len(auth_denied), 1)
self.assertEqual(app_uuid, auth_denied[0]["app_registration_uuid"])
self.assertIsNone(auth_denied[0]["user_id"])
self.assertIsNone(auth_denied[0]["agent_id"])
logouts = select("logout", "ok")
self.assertEqual(1, len(logouts))
self.assertEqual(LOGOUT_PATH, logouts[0]["path"])
self._assert_null_identity(logouts[0])
def test_coverage_tally_pins_ac1_through_ac14_across_suite(self):
self.assertEqual(14, len(AC_PIN_IDS))
loader = unittest.TestLoader()
suite = loader.discover(os.path.dirname(os.path.abspath(__file__)))
stack, ids = [suite], set()
while stack:
item = stack.pop()
if isinstance(item, unittest.TestSuite):
stack.extend(item)
else:
ids.add(item.id())
for ac in sorted(AC_PIN_IDS):
with self.subTest(ac=ac):
self.assertIn(AC_PIN_IDS[ac], ids)
if __name__ == "__main__":
unittest.main()
+114
View File
@@ -0,0 +1,114 @@
import contextlib
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.request
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
class AcceptanceP1Tests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
}
def _boot(self):
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer):
broker = coreci_broker.boot(dict(self.env))
return broker, buffer.getvalue()
def test_ac14_coreci_broker_line_count_under_1400(self):
source_path = os.path.join(os.path.dirname(__file__), "..", "coreci_broker.py")
with open(source_path, "r", encoding="utf-8") as handle:
line_count = sum(1 for _ in handle)
self.assertLess(line_count, 1400)
def test_first_boot_prints_pass_once_second_boot_silent_with_0600_state(self):
first = io.StringIO()
with contextlib.redirect_stdout(first):
broker = coreci_broker.boot(dict(self.env))
server = coreci_broker.serve("127.0.0.1", 0, broker)
db_path = self.env["BROKER_DB_PATH"]
audit_path = self.env["BROKER_AUDIT_LOG_PATH"]
env_path = self.env["BROKER_ENV_PATH"]
self.assertTrue(os.path.isfile(db_path))
self.assertTrue(os.path.isfile(audit_path))
self.assertTrue(os.path.isfile(env_path))
self.assertEqual(0o600, os.stat(db_path).st_mode & 0o777)
self.assertEqual(0o600, os.stat(audit_path).st_mode & 0o777)
admin_pass = broker.config["BROKER_ADMIN_PASS"]
pepper = broker.config["BROKER_INSTANCE_PEPPER"]
self.assertIn(admin_pass, first.getvalue())
self.assertEqual(1, first.getvalue().count(admin_pass))
self.assertNotIn(pepper, first.getvalue())
server.shutdown()
server.server_close()
broker.audit.close()
broker.store.close()
second = io.StringIO()
with contextlib.redirect_stdout(second):
rebooted = coreci_broker.boot(dict(self.env))
self.assertEqual("", second.getvalue())
rebooted.audit.close()
rebooted.store.close()
def test_restart_preserves_allowlist_row_and_audit_history(self):
broker, _ = self._boot()
server = coreci_broker.serve("127.0.0.1", 0, broker)
user_id = "acceptance-restart-user"
broker.store.conn.execute(
"INSERT INTO user_allowlist (user_id, email, status, created_at) VALUES (?, ?, ?, ?)",
(user_id, "ceo@coreci.dev", "active", coreci_broker._utc_now_iso()),
)
broker.store.conn.commit()
broker.audit.write("add_user", user_id=user_id)
server.shutdown()
server.server_close()
broker.audit.close()
broker.store.close()
rebooted, _ = self._boot()
self.addCleanup(rebooted.audit.close)
self.addCleanup(rebooted.store.close)
row = rebooted.store.conn.execute(
"SELECT email, status FROM user_allowlist WHERE user_id = ?", (user_id,)
).fetchone()
self.assertEqual(("ceo@coreci.dev", "active"), row)
audit_path = self.env["BROKER_AUDIT_LOG_PATH"]
with open(audit_path, "r", encoding="utf-8") as handle:
lines = [line for line in handle if line.strip()]
self.assertEqual(1, len(lines))
entry = json.loads(lines[0])
self.assertEqual("add_user", entry["action"])
self.assertEqual(user_id, entry["user_id"])
self.assertEqual(0o600, os.stat(audit_path).st_mode & 0o777)
def test_healthz_end_to_end(self):
broker, _ = self._boot()
server = coreci_broker.serve("127.0.0.1", 0, broker)
self.addCleanup(broker.audit.close)
self.addCleanup(broker.store.close)
self.addCleanup(server.server_close)
self.addCleanup(server.shutdown)
port = server.server_address[1]
with urllib.request.urlopen(
"http://127.0.0.1:%d/healthz" % port, timeout=5
) as response:
self.assertEqual(200, response.status)
payload = json.loads(response.read().decode("utf-8"))
self.assertEqual({"ok": True}, payload)
if __name__ == "__main__":
unittest.main()
+241
View File
@@ -0,0 +1,241 @@
import base64
import contextlib
import io
import json
import os
import sys
import tempfile
import time
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
LOGIN_PATH = "/api/broker/auth/login"
LOGOUT_PATH = "/api/broker/auth/logout"
WHOAMI_PATH = "/api/broker/auth/whoami"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
AUDIT_PATH = "/api/broker/audit"
class AcceptanceP2Tests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
}
def _boot_served(self, extra=None):
env = dict(self.env)
env.update(extra or {})
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer):
broker = coreci_broker.boot(env)
self.addCleanup(broker.audit.close)
self.addCleanup(broker.store.close)
printed = [
line.split("=", 1)[1]
for line in buffer.getvalue().splitlines()
if line.startswith("BROKER_ADMIN_PASS=")
]
self.assertEqual(1, len(printed))
server = coreci_broker.serve("127.0.0.1", 0, broker)
self.addCleanup(server.server_close)
self.addCleanup(server.shutdown)
return "http://127.0.0.1:%d" % server.server_address[1], printed[0]
def _request(self, base, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=5) as response:
return response.status, response.read(), response.headers
except urllib.error.HTTPError as error:
return error.code, error.read(), error.headers
def _request_json(self, base, method, path, payload=None, headers=None):
status, body, response_headers = self._request(
base, method, path, payload, headers
)
return status, json.loads(body.decode("utf-8")), response_headers
def _get_page(self, base, path):
status, body, headers = self._request(base, "GET", path)
self.assertEqual(200, status)
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
return body.decode("utf-8")
def _basic(self, password):
raw = "%s:%s" % (ADMIN_USER, password)
encoded = base64.b64encode(raw.encode("utf-8")).decode("ascii")
return {"Authorization": "Basic " + encoded}
def _login(self, base, password):
status, body, headers = self._request_json(
base, "POST", LOGIN_PATH, {"username": ADMIN_USER, "password": password}
)
self.assertEqual(200, status)
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
raw = headers.get("Set-Cookie")
self.assertTrue(raw)
self.assertTrue(raw.startswith("broker_session="))
self.assertIn("HttpOnly", raw)
self.assertIn("SameSite=Lax", raw)
return {"Cookie": raw.split(";", 1)[0]}
def _find(self, rows, key, value):
for row in rows:
if row[key] == value:
return row
self.fail("no row with %s=%r" % (key, value))
def _ok_entries(self, entries):
return [entry for entry in entries if entry["result"] == "ok"]
def _assert_null_identity(self, entries):
self.assertTrue(entries)
for entry in entries:
self.assertIsNone(entry["user_id"])
self.assertIsNone(entry["app_registration_uuid"])
self.assertIsNone(entry["agent_id"])
def test_ac1_first_boot_printed_pass_login_via_ui_login_page(self):
base, printed_pass = self._boot_served()
self._get_page(base, "/ui/login")
cookie = self._login(base, printed_pass)
status, body, _ = self._request_json(
base, "GET", WHOAMI_PATH, headers=cookie
)
self.assertEqual(200, status)
self.assertEqual(ADMIN_USER, body["username"])
def test_ac2_app_registration_lifecycle_via_session_cookie(self):
base, printed_pass = self._boot_served()
cookie = self._login(base, printed_pass)
app_uuid = str(uuidlib.uuid4())
status, body, _ = self._request_json(
base, "POST", APPS_PATH,
{"uuid": app_uuid, "name": "proxmox-tool"}, headers=cookie,
)
self.assertEqual(201, status)
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
status, body, _ = self._request_json(base, "GET", APPS_PATH, headers=cookie)
self.assertEqual(200, status)
row = self._find(body["registrations"], "uuid", app_uuid)
self.assertEqual("proxmox-tool", row["name"])
self.assertEqual("active", row["status"])
self._get_page(base, "/ui/app_registrations")
status, body, _ = self._request_json(
base, "DELETE", APPS_PATH + "/" + app_uuid, headers=cookie
)
self.assertEqual(200, status)
self.assertEqual("revoked", body["status"])
status, body, _ = self._request_json(base, "GET", APPS_PATH, headers=cookie)
self.assertEqual(
"revoked", self._find(body["registrations"], "uuid", app_uuid)["status"]
)
status, body, _ = self._request_json(
base, "POST", APPS_PATH + "/" + app_uuid + "/reinstate", headers=cookie
)
self.assertEqual(200, status)
self.assertEqual("active", body["status"])
status, body, _ = self._request_json(base, "GET", APPS_PATH, headers=cookie)
self.assertEqual(
"active", self._find(body["registrations"], "uuid", app_uuid)["status"]
)
def test_ac3_user_allowlist_lifecycle_via_session_cookie(self):
base, printed_pass = self._boot_served()
cookie = self._login(base, printed_pass)
status, body, _ = self._request_json(
base, "POST", USERS_PATH,
{"user_id": "ceo", "email": "ceo@coreci.dev"}, headers=cookie,
)
self.assertEqual(201, status)
self.assertEqual({"user_id": "ceo", "status": "active"}, body)
status, body, _ = self._request_json(base, "GET", USERS_PATH, headers=cookie)
self.assertEqual(200, status)
row = self._find(body["allowlist"], "user_id", "ceo")
self.assertEqual("ceo@coreci.dev", row["email"])
self.assertEqual("active", row["status"])
self._get_page(base, "/ui/user_allowlist")
status, body, _ = self._request_json(
base, "DELETE", USERS_PATH + "/ceo", headers=cookie
)
self.assertEqual(200, status)
self.assertEqual("revoked", body["status"])
status, body, _ = self._request_json(base, "GET", USERS_PATH, headers=cookie)
self.assertEqual(
"revoked", self._find(body["allowlist"], "user_id", "ceo")["status"]
)
status, body, _ = self._request_json(
base, "POST", USERS_PATH + "/ceo/reinstate", headers=cookie
)
self.assertEqual(200, status)
self.assertEqual("active", body["status"])
status, body, _ = self._request_json(base, "GET", USERS_PATH, headers=cookie)
self.assertEqual(
"active", self._find(body["allowlist"], "user_id", "ceo")["status"]
)
def test_ac9_login_and_logout_audit_lines_with_null_identity(self):
base, printed_pass = self._boot_served()
cookie = self._login(base, printed_pass)
status, body, _ = self._request_json(
base, "POST", LOGOUT_PATH, headers=cookie
)
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
admin = self._basic(printed_pass)
status, body, _ = self._request_json(
base, "GET", AUDIT_PATH + "?action=login", headers=admin
)
self.assertEqual(200, status)
logins = self._ok_entries(body["entries"])
self.assertEqual(["login"], [entry["action"] for entry in logins])
self._assert_null_identity(logins)
status, body, _ = self._request_json(
base, "GET", AUDIT_PATH + "?action=logout", headers=admin
)
self.assertEqual(200, status)
logouts = self._ok_entries(body["entries"])
self.assertEqual(["logout"], [entry["action"] for entry in logouts])
self._assert_null_identity(logouts)
def test_ui_pages_served_and_static_hygiene(self):
base, printed_pass = self._boot_served()
dashboard = self._get_page(base, "/ui/dashboard")
self.assertIn("Dashboard", dashboard)
audit_page = self._get_page(base, "/ui/audit")
self.assertNotIn("export", audit_page.lower())
login_page = self._get_page(base, "/ui/login")
self.assertIn('name="username"', login_page)
self.assertIn('name="password"', login_page)
def test_session_ttl_expiry_end_to_end(self):
base, printed_pass = self._boot_served({"BROKER_SESSION_TTL": "1"})
cookie = self._login(base, printed_pass)
status, _, _ = self._request(base, "GET", WHOAMI_PATH, headers=cookie)
self.assertEqual(200, status)
time.sleep(1.2)
status, body, _ = self._request_json(
base, "GET", WHOAMI_PATH, headers=cookie
)
self.assertEqual(401, status)
self.assertEqual(401, body["error"])
if __name__ == "__main__":
unittest.main()
+309
View File
@@ -0,0 +1,309 @@
import base64
import contextlib
import datetime
import hashlib
import io
import json
import os
import sys
import tempfile
import time
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
REGISTER_PATH = "/api/broker/agents/register"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
AUDIT_PATH = "/api/broker/audit"
AGENT_PATH = "/api/broker/agents/%s"
STATE_PATH = "/api/broker/agents/%s/state"
AUTH_401_BODY = {"error": 401, "message": "invalid app credentials"}
REGISTER_403_BODY = {"error": 403, "message": "registration not allowed"}
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
def _derive(pepper, app_uuid, user_id, agent_id, tags):
material = "%s:%s:%s:%s:%s" % (
pepper, app_uuid, user_id, agent_id, ",".join(sorted(tags))
)
digest = hashlib.sha256(material.encode("utf-8")).digest()
return base64.b64encode(digest).decode("ascii")
class AcceptanceP3Tests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
}
self.broker, self.base = self._boot()
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
self.app_uuid = str(uuidlib.uuid4())
self.user_id = "ceo"
self._seed(self.base, self.app_uuid, self.user_id)
def _boot(self, extra=None):
env = dict(self.env)
env.update(extra or {})
with contextlib.redirect_stdout(io.StringIO()):
broker = coreci_broker.boot(env)
self.addCleanup(broker.audit.close)
self.addCleanup(broker.store.close)
server = coreci_broker.serve("127.0.0.1", 0, broker)
self.addCleanup(server.shutdown)
self.addCleanup(server.server_close)
return broker, "http://127.0.0.1:%d" % server.server_address[1]
def _request(self, method, path, payload=None, headers=None, base=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(
(base or self.base) + path, data=data, method=method
)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
raw = response.read().decode("utf-8")
return response.status, json.loads(raw) if raw else None
except urllib.error.HTTPError as error:
raw = error.read().decode("utf-8")
return error.code, json.loads(raw) if raw else None
def _seed(self, base, app_uuid, user_id):
status, body = self._request(
"POST", APPS_PATH, {"uuid": app_uuid}, headers=self.admin, base=base
)
self.assertEqual(201, status)
status, body = self._request(
"POST", USERS_PATH, {"user_id": user_id}, headers=self.admin, base=base
)
self.assertEqual(201, status)
def _register(self, agent_id, tags, base=None, app_uuid=None, user_id=None):
payload = {
"agent_id": agent_id,
"app_registration_uuid": app_uuid or self.app_uuid,
"user_id": user_id or self.user_id,
"tags": list(tags),
}
return self._request("POST", REGISTER_PATH, payload, base=base)
def _app_header(self, token, app_uuid=None):
return {"Authorization": "App %s %s" % (app_uuid or self.app_uuid, token)}
def _audit_entries(self, base, query):
status, body = self._request(
"GET", AUDIT_PATH + query, headers=self.admin, base=base
)
self.assertEqual(200, status)
return body["entries"]
def test_ac4_derivation_identical_both_sides_stored_as_slow_hash(self):
agent_id = "px-1"
tags = ["zeta", "alpha"]
status, body = self._register(agent_id, tags)
self.assertEqual(201, status)
self.assertTrue(body["created"])
token = body["token"]
pepper = self.broker.store.get_config("BROKER_INSTANCE_PEPPER")
self.assertEqual(pepper, body["pepper"])
broker_side = _derive(pepper, self.app_uuid, self.user_id, agent_id, tags)
tool_side = _derive(body["pepper"], self.app_uuid, self.user_id, agent_id, tags)
self.assertEqual(broker_side, token)
self.assertEqual(tool_side, token)
self.assertEqual(44, len(token))
self.assertTrue(token.isascii())
row = self.broker.store.get_agent(agent_id, self.app_uuid)
stored = row[4]
self.assertNotEqual(token, stored)
if coreci_broker.bcrypt is None:
self.assertTrue(stored.startswith("scrypt$"))
else:
self.assertTrue(stored.startswith(coreci_broker.BCRYPT_PREFIXES))
self.assertTrue(coreci_broker.verify_token(token, stored))
self.assertFalse(coreci_broker.verify_token(token, token))
self.assertEqual(self.user_id, row[2])
self.assertEqual(json.dumps(sorted(tags)), row[3])
self.assertEqual(body["token_expires_at"], row[5])
def test_ac7_revoked_or_unknown_uuid_denied_existing_token_honored(self):
status, first = self._register("tool-1", ["coreci"])
self.assertEqual(201, status)
token = first["token"]
status, body = self._request(
"DELETE", APPS_PATH + "/" + self.app_uuid, headers=self.admin
)
self.assertEqual(200, status)
self.assertEqual({"uuid": self.app_uuid, "status": "revoked"}, body)
status, body = self._register("tool-2", ["coreci"])
self.assertEqual(403, status)
self.assertEqual(REGISTER_403_BODY, body)
self.assertIsNone(self.broker.store.get_agent("tool-2", self.app_uuid))
status, body = self._register("tool-3", ["coreci"], app_uuid=str(uuidlib.uuid4()))
self.assertEqual(403, status)
self.assertEqual(REGISTER_403_BODY, body)
status, body = self._request(
"GET", AGENT_PATH % "tool-1", headers=self._app_header(token)
)
self.assertEqual(200, status)
self.assertEqual("tool-1", body["agent_id"])
self.assertEqual(self.user_id, body["user_id"])
self.assertEqual(self.app_uuid, body["app_registration_uuid"])
self.assertEqual(["coreci"], body["tags"])
def test_ac8_unknown_or_revoked_user_denied(self):
status, body = self._register("tool-1", ["coreci"], user_id="intern")
self.assertEqual(403, status)
self.assertEqual(REGISTER_403_BODY, body)
status, body = self._request(
"POST", USERS_PATH, {"user_id": "intern"}, headers=self.admin
)
self.assertEqual(201, status)
status, body = self._register("tool-1", ["coreci"], user_id="intern")
self.assertEqual(201, status)
self.assertTrue(body["created"])
status, body = self._request(
"DELETE", USERS_PATH + "/intern", headers=self.admin
)
self.assertEqual(200, status)
self.assertEqual({"user_id": "intern", "status": "revoked"}, body)
status, body = self._register("tool-1", ["coreci"], user_id="intern")
self.assertEqual(403, status)
self.assertEqual(REGISTER_403_BODY, body)
def test_ac10_expiry_401_reregister_refreshes_expiry_and_audits(self):
broker, base = self._boot({
"BROKER_DB_PATH": os.path.join(self.tmp, "ttl.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "ttl.audit.log"),
"BROKER_TOKEN_TTL": "1",
})
self._seed(base, self.app_uuid, self.user_id)
agent_id = "ttl-tool"
status, first = self._register(agent_id, ["coreci"], base=base)
self.assertEqual(201, status)
delta = (datetime.datetime.fromisoformat(first["token_expires_at"])
- datetime.datetime.now(datetime.timezone.utc)).total_seconds()
self.assertGreater(delta, 0)
self.assertLess(delta, 3)
status, body = self._request(
"GET", AGENT_PATH % agent_id,
headers=self._app_header(first["token"]), base=base,
)
self.assertEqual(200, status)
time.sleep(1.2)
status, body = self._request(
"GET", AGENT_PATH % agent_id,
headers=self._app_header(first["token"]), base=base,
)
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
status, body = self._register(agent_id, ["coreci"], base=base, user_id="ghost-user")
self.assertEqual(403, status)
status, second = self._register(agent_id, ["coreci"], base=base)
self.assertEqual(200, status)
self.assertFalse(second["created"])
self.assertEqual(first["token"], second["token"])
self.assertIn("pepper", second)
self.assertGreater(
datetime.datetime.fromisoformat(second["token_expires_at"]),
datetime.datetime.fromisoformat(first["token_expires_at"]),
)
status, body = self._request(
"GET", AGENT_PATH % agent_id,
headers=self._app_header(second["token"]), base=base,
)
self.assertEqual(200, status)
row = broker.store.get_agent(agent_id, self.app_uuid)
self.assertEqual(second["token_expires_at"], row[5])
self.assertTrue(coreci_broker.verify_token(second["token"], row[4]))
entries = self._audit_entries(base, "?action=register")
self.assertEqual(["ok", "denied", "ok"], [entry["result"] for entry in entries])
for entry in entries:
self.assertEqual(agent_id, entry["agent_id"])
self.assertEqual(self.app_uuid, entry["app_registration_uuid"])
self.assertEqual(REGISTER_PATH, entry["path"])
self.assertEqual(self.user_id, entries[0]["user_id"])
self.assertEqual("ghost-user", entries[1]["user_id"])
self.assertEqual(self.user_id, entries[2]["user_id"])
def test_ac12_state_roundtrip_with_identity_enforcement(self):
status, a = self._register("tool-a", ["coreci", "alpha"])
self.assertEqual(201, status)
status, b = self._register("tool-b", ["coreci"])
self.assertEqual(201, status)
state = {
"agent_id": "tool-a",
"app_registration_uuid": self.app_uuid,
"user_id": self.user_id,
"tags": ["alpha", "coreci"],
"pepper": a["pepper"],
"current_token": a["token"],
"token_expires_at": a["token_expires_at"],
"first_registered_at": "t0",
"last_seen_at": "t1",
"payload": {"counter": 7, "notes": ["x", "y"], "nested": {"k": None}},
}
status, body = self._request(
"PUT", STATE_PATH % "tool-a", state, headers=self._app_header(a["token"])
)
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
status, body = self._request(
"GET", STATE_PATH % "tool-a", headers=self._app_header(a["token"])
)
self.assertEqual(200, status)
self.assertEqual(state, body)
self.assertEqual(list(state), list(body))
status, body = self._request(
"GET", STATE_PATH % "tool-a", headers=self._app_header(b["token"])
)
self.assertEqual(403, status)
self.assertEqual({"error": 403, "message": "agent identity mismatch"}, body)
status, body = self._request(
"PUT", STATE_PATH % "tool-a", {"intruder": True},
headers=self._app_header(b["token"]),
)
self.assertEqual(403, status)
self.assertEqual({"error": 403, "message": "agent identity mismatch"}, body)
status, body = self._request(
"PUT", STATE_PATH % "tool-b", {"mine": True},
headers=self._app_header(b["token"]),
)
self.assertEqual(200, status)
status, body = self._request(
"GET", STATE_PATH % "tool-b", headers=self._app_header(b["token"])
)
self.assertEqual(200, status)
self.assertEqual({"mine": True}, body)
status, body = self._request(
"GET", STATE_PATH % "ghost-agent", headers=self._app_header(a["token"])
)
self.assertEqual(404, status)
self.assertEqual({"error": 404, "message": "not found"}, body)
status, body = self._request(
"GET", STATE_PATH % "tool-a", headers=self._app_header(a["token"])
)
self.assertEqual(200, status)
self.assertEqual(state, body)
if __name__ == "__main__":
unittest.main()
+290
View File
@@ -0,0 +1,290 @@
import ast
import base64
import contextlib
import importlib.util
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.parse
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
REGISTER_PATH = "/api/broker/agents/register"
AUDIT_PATH = "/api/broker/audit"
SECRETS_PREFIX = "/api/broker/secrets/"
CLIENT_STATIC_PATH = "/static/coreci_broker_client.py"
PROXMOX_PATH = "coreci/proxmox"
GITEA_PATH = "coreci/gitea"
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
GITEA_SECRET = {"url": "https://git.coreci.dev", "token": "git-tok"}
AUTH_401_BODY = {"error": 401, "message": "invalid app credentials"}
GITEA_403_BODY = {"error": 403, "message": "forbidden"}
REFERENCE_DIR = os.path.abspath(
os.path.join(os.path.dirname(__file__), os.pardir, "reference_tools")
)
STANDALONE_CLIENT_IMPORTS = {"json", "time", "urllib.parse", "urllib.request"}
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
def _flip(token):
replacement = "x" if token[21] != "x" else "y"
return token[:21] + replacement + token[22:]
class AcceptanceP4Tests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.creds_path = os.path.join(self.tmp, "creds.json")
with open(self.creds_path, "w", encoding="utf-8") as handle:
handle.write(json.dumps(
{PROXMOX_PATH: PROXMOX_SECRET, GITEA_PATH: GITEA_SECRET}
))
env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
"BROKER_CREDS_SOURCE": "file",
"BROKER_CREDS_FILE": self.creds_path,
}
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(env)
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
self.app_uuid = str(uuidlib.uuid4())
self.user_id = "ceo"
self._seed()
def _request(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(self.base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
raw = response.read().decode("utf-8")
return response.status, json.loads(raw) if raw else None
except urllib.error.HTTPError as error:
raw = error.read().decode("utf-8")
return error.code, json.loads(raw) if raw else None
def _seed(self):
status, _ = self._request("POST", APPS_PATH, {"uuid": self.app_uuid}, self.admin)
self.assertEqual(201, status)
status, _ = self._request("POST", USERS_PATH, {"user_id": self.user_id}, self.admin)
self.assertEqual(201, status)
def _register(self, tags):
payload = {"agent_id": "proxmox", "app_registration_uuid": self.app_uuid,
"user_id": self.user_id, "tags": list(tags)}
status, body = self._request("POST", REGISTER_PATH, payload)
self.assertEqual(201, status)
return body
def _app_header(self, token):
return {"Authorization": "App %s %s" % (self.app_uuid, token)}
def _secret(self, suffix, headers):
return self._request("GET", SECRETS_PREFIX + suffix, headers=headers)
def _audit(self, action):
query = urllib.parse.urlencode({"action": action, "limit": "1000"})
status, body = self._request(
"GET", AUDIT_PATH + "?" + query, headers=self.admin
)
self.assertEqual(200, status)
return body["entries"]
def _fetch_client_source(self):
with urllib.request.urlopen(
self.base + CLIENT_STATIC_PATH, timeout=10
) as response:
self.assertEqual(200, response.status)
return response.read().decode("utf-8")
def _import_module(self, name, source):
path = os.path.join(self.tmp, name + ".py")
with open(path, "w", encoding="utf-8") as handle:
handle.write(source)
spec = importlib.util.spec_from_file_location(name, path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def _assert_standalone_imports(self, source):
modules = set()
for node in ast.walk(ast.parse(source)):
if isinstance(node, ast.Import):
modules.update(alias.name for alias in node.names)
elif isinstance(node, ast.ImportFrom):
modules.add(node.module or "")
self.assertTrue(modules)
for module in modules:
self.assertIn(module, STANDALONE_CLIENT_IMPORTS, module)
def _bootstrapped_client(self):
module = self._import_module(
"p4_fetched_coreci_broker_client", self._fetch_client_source()
)
return module.BrokerClient(
self.base, "proxmox", self.app_uuid, self.user_id, ["coreci", "bootstrap"]
)
def _reference_tools(self):
source_path = os.path.join(REFERENCE_DIR, "proxmox_readonly_reference.py")
spec = importlib.util.spec_from_file_location(
"p4_proxmox_readonly_reference", source_path
)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
tools = module.Tools()
tools.valves.base_url = self.base
tools.valves.agent_id = "proxmox"
tools.valves.app_uuid = self.app_uuid
tools.valves.user_id = self.user_id
return tools
def _persisted_state(self):
return self.broker.store.get_agent_state("proxmox", self.app_uuid)
def _assert_tool_identity(self, entry):
self.assertEqual(self.user_id, entry["user_id"])
self.assertIsNotNone(entry["user_id"])
self.assertEqual(self.app_uuid, entry["app_registration_uuid"])
self.assertIsNotNone(entry["app_registration_uuid"])
self.assertEqual("proxmox", entry["agent_id"])
self.assertIsNotNone(entry["agent_id"])
def test_ac5_valid_token_reads_secret_tampered_token_is_401(self):
registered = self._register(["coreci"])
status, body = self._secret(
"coreci/proxmox", self._app_header(registered["token"])
)
self.assertEqual(200, status)
self.assertEqual({"path", "data", "version", "updated_at"}, set(body))
self.assertEqual(PROXMOX_PATH, body["path"])
self.assertEqual(PROXMOX_SECRET, body["data"])
self.assertRegex(body["version"], r"^[0-9a-f]{8}$")
flipped = _flip(registered["token"])
self.assertNotEqual(registered["token"], flipped)
status, body = self._secret("coreci/proxmox", self._app_header(flipped))
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
def test_ac6_cross_path_read_is_403_with_denied_audit(self):
registered = self._register(["coreci"])
status, body = self._secret("coreci/gitea", self._app_header(registered["token"]))
self.assertEqual(403, status)
self.assertEqual(GITEA_403_BODY, body)
denied = [
entry for entry in self._audit("read") if entry["result"] == "denied"
]
self.assertEqual(1, len(denied))
self._assert_tool_identity(denied[0])
self.assertEqual("read", denied[0]["action"])
self.assertEqual(GITEA_PATH, denied[0]["path"])
def test_client_bootstrap_public_fetch_import_register_state_secret(self):
source = self._fetch_client_source()
self.assertIn("class BrokerClient", source)
self._assert_standalone_imports(source)
client = self._bootstrapped_client()
state = {}
client.ensure_registered(state)
self.assertEqual(44, len(state["current_token"]))
self.assertEqual("proxmox", state["agent_id"])
self.assertEqual(self.app_uuid, state["app_registration_uuid"])
self.assertEqual(self.user_id, state["user_id"])
self.assertIn("pepper", state)
self.assertTrue(client.put_state(state))
document = client.read_secret(PROXMOX_PATH, state)
self.assertEqual(PROXMOX_PATH, document["path"])
self.assertEqual(PROXMOX_SECRET, document["data"])
persisted = self._persisted_state()
self.assertEqual(state["current_token"], persisted["current_token"])
def test_reference_tool_full_flow_single_register_line(self):
tools = self._reference_tools()
before = len(self._audit("register"))
response = tools.list_nodes()
self.assertIs(True, response["dry_run"])
self.assertIs(True, response["credential_present"])
registers = self._audit("register")
self.assertEqual(1, len(registers) - before)
self.assertEqual("ok", registers[-1]["result"])
self._assert_tool_identity(registers[-1])
persisted = self._persisted_state()
self.assertEqual(tools._state["current_token"], persisted["current_token"])
def test_phase_audit_completeness_after_full_drive(self):
registered = self._register(["coreci"])
token = registered["token"]
status, _ = self._secret("coreci/proxmox", self._app_header(token))
self.assertEqual(200, status)
status, _ = self._secret("coreci/proxmox", self._app_header(_flip(token)))
self.assertEqual(401, status)
status, _ = self._secret("coreci/gitea", self._app_header(token))
self.assertEqual(403, status)
client = self._bootstrapped_client()
state = {}
client.ensure_registered(state)
self.assertTrue(client.put_state(state))
self.assertEqual(PROXMOX_SECRET, client.read_secret(PROXMOX_PATH, state)["data"])
before = len(self._audit("register"))
tools = self._reference_tools()
response = tools.list_nodes()
self.assertIs(True, response["dry_run"])
self.assertIs(True, response["credential_present"])
registers = self._audit("register")
self.assertEqual(1, len(registers) - before)
self.assertEqual(3, len(registers))
for entry in registers:
self.assertEqual("ok", entry["result"])
self.assertEqual(REGISTER_PATH, entry["path"])
self._assert_tool_identity(entry)
reads = self._audit("read")
ok_reads = [entry for entry in reads if entry["result"] == "ok"]
denied_reads = [entry for entry in reads if entry["result"] == "denied"]
self.assertEqual(3, len(ok_reads))
for entry in ok_reads:
self.assertEqual(PROXMOX_PATH, entry["path"])
self._assert_tool_identity(entry)
self.assertEqual(1, len(denied_reads))
self.assertEqual(GITEA_PATH, denied_reads[0]["path"])
self._assert_tool_identity(denied_reads[0])
denied_actions = self._audit("denied")
self.assertEqual(1, len(denied_actions))
self.assertEqual("denied", denied_actions[0]["action"])
self.assertEqual("denied", denied_actions[0]["result"])
self.assertEqual(SECRETS_PREFIX + "coreci/proxmox", denied_actions[0]["path"])
self.assertEqual(self.app_uuid, denied_actions[0]["app_registration_uuid"])
self.assertIsNone(denied_actions[0]["user_id"])
self.assertIsNone(denied_actions[0]["agent_id"])
if __name__ == "__main__":
unittest.main()
+548
View File
@@ -0,0 +1,548 @@
"""v0.2 acceptance matrix (PLAN P3-01; REQ-21/REQ-22/REQ-23 acceptance).
Consolidates the milestone's acceptance as thin, high-value assertions that
the deliverable holds end-to-end. The deep per-feature logic stays in its own
files — test_build_cli, test_cli_packaging, test_publish_image,
test_release_upload, test_dockerfile_consistency — which run green in the
same cumulative suite (``python3 -m unittest discover -s tests``; suite
greenness is the runner's job, asserted by nobody in here). This file does
NOT re-run their cases; the stub roundtrips are smoke-asserted instead via
module presence + runnable TestSuite sizes (defaultTestLoader).
The matrix (binding order):
1. boot parity through the artifact — the built ``.pyz`` is booted as a real
subprocess child (``[sys.executable, artifact]``, never a bare
"python3") with an ephemeral bind + temp state, and v0.1 acceptance
parity must hold through the packaged path: admin pass printed exactly
once, ``broker.db`` + ``broker.audit.log`` 0600 in the state dir,
``/ui/login`` + ``/ui/dashboard`` 200, ``/static/coreci_broker_client.py``
byte-exact vs the repo file, one admin-login + agent-register +
state-put/get + secret-read flow (v0.1 request patterns), second boot
silent (asserted only after terminate + full drain — G-010);
2. reproducibility — two consecutive same-environment builds produce
byte-identical artifacts (identical sha256, G-009);
3. self-containment — the archive contains ONLY the staging allowlist
(broker + client + UI pages + shim + ``_cli_version.py``): no ``.env``,
no ``dist/``, no state, no tests;
4. stub publish roundtrip + stub release upload green — their test modules
import cleanly and expose runnable TestSuites of the expected size;
5. broker zero-delta — ``coreci_broker.py`` unchanged at its v0.1.5 count
of 1390 lines (AC14 under the D-016..D-018 cap of 1400) and carrying no
v0.2 packaging references (``_cli_version`` / the shim / the shim's
materialized-asset dir — frozen means untouched); plus the AC14
no-touch clause machine-checked against the ``v0.1.5`` tag when the
working tree carries it (skipped otherwise);
6. Dockerfile/.dockerignore consistency — module present with its full
TestCase set (it runs in the same suite).
The build happens ONCE per class (setUpClass) and every artifact/boot is
reused across methods where safe; the boot tests dominate and the whole
file is budgeted to stay well under 30s. Every child is terminated +
drained exactly once (idempotent ``stop`` via ``addCleanup``); every temp
dir is ``addCleanup``-registered. Builds use the ``scripts/build_cli.py``
API (never a shell path) into ``TemporaryDirectory`` dist dirs — no
committed ``dist/``, no state, no live-forge contact.
"""
import hashlib
import importlib.util
import json
import os
import select
import subprocess
import sys
import tempfile
import time
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
import zipfile
REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
TESTS_DIR = os.path.dirname(os.path.abspath(__file__))
BUILD_CLI_PATH = os.path.join(REPO, "scripts", "build_cli.py")
BROKER_SOURCE_PATH = os.path.join(REPO, "coreci_broker.py")
REPO_CLIENT_PATH = os.path.join(REPO, "coreci_broker_client.py")
VERSION = "v0.2.3" # the milestone release this acceptance gates (P3-05 ship)
ADMIN_USER = "admin"
ADMIN_PASS_PREFIX = "BROKER_ADMIN_PASS="
LISTENING_PREFIX = "coreci-broker listening on http://"
STATE_FILE_NAMES = (".env", "broker.db", "broker.audit.log")
LOGIN_PATH = "/api/broker/auth/login"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
REGISTER_PATH = "/api/broker/agents/register"
AGENT_STATE_PATH = "/api/broker/agents/proxmox/state"
SECRETS_PATH = "/api/broker/secrets/coreci/proxmox"
CLIENT_STATIC_PATH = "/static/coreci_broker_client.py"
UI_LOGIN_PATH = "/ui/login"
UI_DASHBOARD_PATH = "/ui/dashboard"
PROXMOX_PATH = "coreci/proxmox"
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
BOOT_DEADLINE_SECONDS = 30.0
POLL_SECONDS = 0.1
DRAIN_TIMEOUT_SECONDS = 20.0
HTTP_TIMEOUT_SECONDS = 10
GIT_TIMEOUT_SECONDS = 30
# Frozen-broker sources (PLAN: no task edits these in v0.2) — used only by
# the AC14 no-touch clause check, never modified.
FROZEN_SOURCES = ("coreci_broker.py", "coreci_broker_client.py",
"coreci_broker_ui", "reference_tools")
# v0.2 packaging artifacts the frozen broker must know nothing about.
FROZEN_MARKER_STRINGS = ("_cli_version", "cli/__main__", "coreci_broker_assets")
# The v0.2 test surface consolidated here — presence/count floors only
# (the roundtrips themselves run green in their own files, same suite).
STUB_MODULE_MIN_CASES = {
"test_publish_image.py": 9, # stub publish roundtrip
"test_release_upload.py": 15, # stub release upload + env-secrets
"test_dockerfile_consistency.py": 7,
}
def _load_build_cli():
spec = importlib.util.spec_from_file_location(
"acceptance_build_cli", BUILD_CLI_PATH)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
build_cli = _load_build_cli()
def _sha256(path):
digest = hashlib.sha256()
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(65536), b""):
digest.update(chunk)
return digest.hexdigest()
class _BrokerChild:
"""A subprocess child booted from the built zipapp artifact.
The merged stdout/stderr pipe is pumped via ``select`` + raw ``os.read``
under an explicit deadline until the broker announces its ephemeral
listen address. ``stop()`` is the G-010 kill/drain protocol:
``terminate()`` then a full ``communicate()`` drain (kill fallback on
drain timeout). It is idempotent and registered with ``addCleanup`` at
construction, so no child leaks even when an assertion trips mid-test.
Silence and pass-once assertions are made ONLY on ``full_output()``
after ``stop()``.
"""
def __init__(self, testcase, artifact, env, cwd):
self.testcase = testcase
self.artifact = artifact
self.proc = subprocess.Popen(
[sys.executable, artifact],
env=env, cwd=cwd,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
)
self.lines = []
self.partial = b""
self.drained = b""
self.base_url = None
self._eof = False
self._stopped = False
testcase.addCleanup(self.stop)
self._await_listening(BOOT_DEADLINE_SECONDS)
def _pump(self, timeout):
if self._eof:
return False
fd = self.proc.stdout.fileno()
ready, _, _ = select.select([fd], [], [], timeout)
if not ready:
return True
try:
chunk = os.read(fd, 4096)
except OSError:
chunk = b""
if not chunk:
self._eof = True
return False
self.partial += chunk
while b"\n" in self.partial:
raw, self.partial = self.partial.split(b"\n", 1)
self.lines.append(raw.decode("utf-8", "replace"))
return True
def _await_listening(self, deadline_seconds):
deadline = time.monotonic() + deadline_seconds
cursor = 0
while time.monotonic() < deadline:
alive = self._pump(POLL_SECONDS)
while cursor < len(self.lines):
line = self.lines[cursor]
cursor += 1
if line.startswith(LISTENING_PREFIX):
self.base_url = line.split(" on ", 1)[1].strip()
return
if not alive:
break
self.stop()
self.testcase.fail(
"packaged broker did not announce %r within %.0fs "
"(artifact=%s exit=%s)\n--- captured child output ---\n%s"
% (LISTENING_PREFIX, deadline_seconds, self.artifact,
self.proc.returncode, self.full_output())
)
def stop(self):
if self._stopped:
return
self._stopped = True
if self.proc.poll() is None:
self.proc.terminate()
try:
remainder, _ = self.proc.communicate(timeout=DRAIN_TIMEOUT_SECONDS)
except subprocess.TimeoutExpired:
self.proc.kill()
remainder, _ = self.proc.communicate()
self.drained = remainder or b""
def full_output(self):
text = "\n".join(self.lines)
tail = self.partial + self.drained
if tail:
text += "\n" + tail.decode("utf-8", "replace")
return text
def printed_admin_pass(self):
values = [
line.split("=", 1)[1]
for line in self.lines
if line.startswith(ADMIN_PASS_PREFIX)
]
self.testcase.assertEqual(
1, len(values),
"expected exactly one %s line, got: %r" % (ADMIN_PASS_PREFIX, values)
)
return values[0]
class AcceptanceV02Tests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.build_root = tempfile.TemporaryDirectory(prefix="coreci-acc-v02-")
cls.addClassCleanup(cls.build_root.cleanup)
cls.artifact = build_cli.build(
VERSION, repo=REPO, dist_dir=cls.build_root.name
)
cls.artifact_sha256 = _sha256(cls.artifact)
# -- harness helpers --------------------------------------------------
def _temp_dir(self):
handle = tempfile.TemporaryDirectory(prefix="coreci-acc-v02-")
self.addCleanup(handle.cleanup)
return handle.name
def _child_env(self, **overrides):
"""A child env with no inherited BROKER_* keys, then overrides."""
env = {
key: value
for key, value in os.environ.items()
if not key.startswith("BROKER_")
}
env.update(overrides)
return env
def _state_env(self, state_dir):
"""Ephemeral bind + temp state + a seeded file creds source."""
creds_path = os.path.join(state_dir, "creds.json")
with open(creds_path, "w", encoding="utf-8") as handle:
json.dump({PROXMOX_PATH: PROXMOX_SECRET}, handle)
return self._child_env(
BROKER_BIND="127.0.0.1:0",
BROKER_ENV_PATH=os.path.join(state_dir, ".env"),
BROKER_DB_PATH=os.path.join(state_dir, "broker.db"),
BROKER_AUDIT_LOG_PATH=os.path.join(state_dir, "broker.audit.log"),
BROKER_CREDS_SOURCE="file",
BROKER_CREDS_FILE=creds_path,
)
def _boot(self, state_dir):
"""Boot the artifact against state_dir; CWD is a separate temp dir."""
return _BrokerChild(
self, self.artifact, self._state_env(state_dir), self._temp_dir()
)
def _request(self, base, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(
request, timeout=HTTP_TIMEOUT_SECONDS
) as response:
return response.status, response.read(), response.headers
except urllib.error.HTTPError as error:
return error.code, error.read(), error.headers
def _request_json(self, base, method, path, payload=None, headers=None):
status, body, response_headers = self._request(
base, method, path, payload, headers
)
return status, json.loads(body.decode("utf-8")), response_headers
def _admin_login(self, base, password):
"""v0.1 pattern: POST login, assert the broker_session cookie."""
status, body, headers = self._request_json(
base, "POST", LOGIN_PATH,
{"username": ADMIN_USER, "password": password},
)
self.assertEqual(200, status)
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
raw_cookie = headers.get("Set-Cookie")
self.assertTrue(raw_cookie)
self.assertTrue(raw_cookie.startswith("broker_session="))
self.assertIn("HttpOnly", raw_cookie)
self.assertIn("SameSite=Lax", raw_cookie)
return {"Cookie": raw_cookie.split(";", 1)[0]}
def _load_test_module(self, file_name):
"""Import a sibling test module cleanly (by path, any invocation)."""
path = os.path.join(TESTS_DIR, file_name)
stem = os.path.splitext(file_name)[0]
spec = importlib.util.spec_from_file_location(
"acceptance_probe_" + stem, path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def _runnable_case_count(self, module):
"""Flatten defaultTestLoader's suite, counting runnable TestCases."""
suite = unittest.defaultTestLoader.loadTestsFromModule(module)
stack, count = [suite], 0
while stack:
item = stack.pop()
if isinstance(item, unittest.TestSuite):
stack.extend(item)
else:
self.assertIsInstance(item, unittest.TestCase)
count += 1
return count
# -- matrix 1: boot parity through the artifact ----------------------
def test_boot_parity_through_artifact_second_boot_silent(self):
state_dir = self._temp_dir()
child = self._boot(state_dir)
base = child.base_url
printed_pass = child.printed_admin_pass()
# liveness + full UI served from the packaged artifact
status, body, _ = self._request_json(base, "GET", "/healthz")
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
for ui_path in (UI_LOGIN_PATH, UI_DASHBOARD_PATH):
status, body, headers = self._request(base, "GET", ui_path)
self.assertEqual(200, status, ui_path)
self.assertTrue(
headers.get("Content-Type", "").startswith("text/html"),
ui_path,
)
# the tool-author surface: client module byte-exact vs the repo file
status, served, _ = self._request(base, "GET", CLIENT_STATIC_PATH)
self.assertEqual(200, status)
with open(REPO_CLIENT_PATH, "rb") as handle:
self.assertEqual(handle.read(), served)
# one consolidated v0.1 request-pattern flow:
# admin login -> app + user allowlist -> agent register -> state
# put/get -> secret read
cookie = self._admin_login(base, printed_pass)
app_uuid = str(uuidlib.uuid4())
status, body, _ = self._request_json(
base, "POST", APPS_PATH,
{"uuid": app_uuid, "name": "acceptance-tool"}, headers=cookie,
)
self.assertEqual(201, status)
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
status, body, _ = self._request_json(
base, "POST", USERS_PATH,
{"user_id": "ceo", "email": "ceo@coreci.dev"}, headers=cookie,
)
self.assertEqual(201, status)
status, body, _ = self._request_json(
base, "POST", REGISTER_PATH,
{"agent_id": "proxmox", "app_registration_uuid": app_uuid,
"user_id": "ceo", "tags": ["coreci"]},
)
self.assertEqual(201, status)
app_header = {"Authorization": "App %s %s" % (app_uuid, body["token"])}
state_document = {"milestone": "v0.2", "matrix": "acceptance"}
status, body, _ = self._request_json(
base, "PUT", AGENT_STATE_PATH, state_document, headers=app_header
)
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
status, body, _ = self._request_json(
base, "GET", AGENT_STATE_PATH, headers=app_header
)
self.assertEqual(200, status)
self.assertEqual(state_document, body)
status, body, _ = self._request_json(
base, "GET", SECRETS_PATH, headers=app_header
)
self.assertEqual(200, status)
self.assertEqual({"path", "data", "version", "updated_at"}, set(body))
self.assertEqual(PROXMOX_PATH, body["path"])
self.assertEqual(PROXMOX_SECRET, body["data"])
# v0.1 state-file parity: 0600 in the state dir
for name in STATE_FILE_NAMES:
path = os.path.join(state_dir, name)
self.assertTrue(os.path.isfile(path), path)
self.assertEqual(0o600, os.stat(path).st_mode & 0o777, path)
# pass printed exactly once + pepper never printed — asserted on
# the FULLY drained output after terminate (G-010)
pepper = None
with open(os.path.join(state_dir, ".env"), "r", encoding="utf-8") as handle:
for line in handle:
if line.startswith("BROKER_INSTANCE_PEPPER="):
pepper = line.split("=", 1)[1].strip()
self.assertIsNotNone(pepper)
child.stop()
first_output = child.full_output()
self.assertEqual(1, first_output.count(ADMIN_PASS_PREFIX), first_output)
self.assertEqual(1, first_output.count(printed_pass))
self.assertNotIn(pepper, first_output)
# SECOND boot on the same state dir: silent after terminate + drain
second = self._boot(state_dir)
# the SAME pass still logs in: state persisted across the reboot
self._admin_login(second.base_url, printed_pass)
second.stop()
second_output = second.full_output()
self.assertIn(LISTENING_PREFIX, second_output)
self.assertNotIn(ADMIN_PASS_PREFIX, second_output)
self.assertNotIn(printed_pass, second_output)
# -- matrix 2: reproducibility ----------------------------------------
def test_reproducibility_two_consecutive_builds_identical_sha256(self):
digests = []
for _ in range(2):
with tempfile.TemporaryDirectory(prefix="coreci-acc-repro-") as dist_dir:
artifact = build_cli.build(
VERSION, repo=REPO, dist_dir=dist_dir
)
digests.append(_sha256(artifact))
self.assertNotEqual(digests[0], "0" * 64)
self.assertEqual(digests[0], digests[1])
# and the shared setUpClass build matches too (same environment)
self.assertEqual(self.artifact_sha256, digests[0])
print("v0.2 acceptance two-build sha256: {}".format(digests[0]))
# -- matrix 3: self-containment ---------------------------------------
def test_self_containment_archive_is_exactly_the_allowlist(self):
ui_root = os.path.join(REPO, "coreci_broker_ui")
expected = {
"__main__.py",
"_cli_version.py",
"coreci_broker.py",
"coreci_broker_client.py",
"coreci_broker_ui/",
}
expected.update(
"coreci_broker_ui/%s" % name
for name in os.listdir(ui_root)
if os.path.isfile(os.path.join(ui_root, name))
)
with zipfile.ZipFile(self.artifact) as archive:
names = set(archive.namelist())
# ONLY the allowlist members — nothing more, nothing less
self.assertEqual(expected, names)
# no .env, no dist/, no state, no tests, no bytecode can ride along
for name in names:
for forbidden in (".env", "dist", "broker.db", "broker.audit.log",
"test", "__pycache__", ".pyc"):
self.assertNotIn(forbidden, name, name)
# per-release artifact name (D-020) for the milestone release
self.assertEqual(
build_cli.artifact_name(VERSION),
os.path.basename(self.artifact),
)
# -- matrix 4: stub publish roundtrip + stub release upload green ------
def test_stub_publish_roundtrip_module_green_in_suite(self):
module = self._load_test_module("test_publish_image.py")
self.assertTrue(hasattr(module, "PublishImageTests"))
count = self._runnable_case_count(module)
self.assertGreaterEqual(
count, STUB_MODULE_MIN_CASES["test_publish_image.py"]
)
def test_stub_release_upload_module_green_in_suite(self):
module = self._load_test_module("test_release_upload.py")
self.assertTrue(hasattr(module, "ReleaseUploadTests"))
self.assertTrue(hasattr(module, "EnvSecretsLoaderTests"))
count = self._runnable_case_count(module)
self.assertGreaterEqual(
count, STUB_MODULE_MIN_CASES["test_release_upload.py"]
)
# -- matrix 6: Dockerfile/.dockerignore consistency ---------------------
def test_dockerfile_consistency_module_green_in_suite(self):
module = self._load_test_module("test_dockerfile_consistency.py")
self.assertTrue(hasattr(module, "DockerfileConsistencyTests"))
count = self._runnable_case_count(module)
self.assertGreaterEqual(
count, STUB_MODULE_MIN_CASES["test_dockerfile_consistency.py"]
)
# -- matrix 5: broker zero-delta (AC14, frozen means untouched) --------
def test_broker_zero_delta_1390_lines_no_packaging_references(self):
with open(BROKER_SOURCE_PATH, "r", encoding="utf-8") as handle:
line_count = sum(1 for _ in handle)
# exact v0.1.5 count, under the D-016..D-018 cap of 1400 (AC14)
self.assertEqual(1390, line_count)
self.assertLess(line_count, 1400)
with open(BROKER_SOURCE_PATH, "r", encoding="utf-8") as handle:
source = handle.read()
# frozen means untouched: no v0.2 packaging reference leaked into it
for marker in FROZEN_MARKER_STRINGS:
self.assertNotIn(marker, source)
def test_broker_untouched_by_any_commit_since_v0_1_5(self):
"""The AC14 no-touch clause, machine-checked when the tag exists."""
try:
verify = subprocess.run(
["git", "rev-parse", "-q", "--verify", "v0.1.5^{commit}"],
cwd=REPO, capture_output=True, text=True,
timeout=GIT_TIMEOUT_SECONDS,
)
except (OSError, subprocess.TimeoutExpired):
self.skipTest("git unavailable in this environment")
if verify.returncode != 0:
self.skipTest("v0.1.5 tag not present in this working tree")
log = subprocess.run(
["git", "log", "--oneline", "v0.1.5..HEAD", "--"] + list(FROZEN_SOURCES),
cwd=REPO, capture_output=True, text=True,
timeout=GIT_TIMEOUT_SECONDS,
)
self.assertEqual(0, log.returncode, log.stderr)
self.assertEqual(
"", log.stdout.strip(),
"v0.2 commits touched frozen broker sources:\n" + log.stdout
)
if __name__ == "__main__":
unittest.main()
+251
View File
@@ -0,0 +1,251 @@
import base64
import contextlib
import datetime
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.request
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
LOGIN_PATH = "/api/broker/auth/login"
LOGOUT_PATH = "/api/broker/auth/logout"
WHOAMI_PATH = "/api/broker/auth/whoami"
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
def _iso_offset(seconds):
now = datetime.datetime.now(datetime.timezone.utc)
return (now + datetime.timedelta(seconds=seconds)).isoformat()
class AdminAuthTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
}
self.broker = self._boot_broker(self.env)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
def _boot_broker(self, env):
with contextlib.redirect_stdout(io.StringIO()):
broker = coreci_broker.boot(dict(env))
self.addCleanup(broker.audit.close)
self.addCleanup(broker.store.close)
return broker
def _request(self, method, path, payload=None, headers=None, base=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(
(base or self.base) + path, data=data, method=method
)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=5) as response:
return response.status, response.read(), response.headers
except urllib.error.HTTPError as error:
return error.code, error.read(), error.headers
def _request_json(self, method, path, payload=None, headers=None, base=None):
status, body, response_headers = self._request(
method, path, payload, headers, base
)
return status, json.loads(body.decode("utf-8")), response_headers
def _login(self, username=ADMIN_USER, password=ADMIN_PASS, base=None):
return self._request_json(
"POST",
LOGIN_PATH,
{"username": username, "password": password},
base=base,
)
def _login_cookie(self, base=None):
status, body, headers = self._login(base=base)
self.assertEqual(200, status)
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
raw = headers.get("Set-Cookie")
self.assertTrue(raw)
return raw
def _cookie_id(self, raw):
return raw.split(";", 1)[0].split("=", 1)[1]
def _audit_entries(self):
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
return [json.loads(raw) for raw in handle if raw.strip()]
def _audit_find(self, action, result):
for entry in self._audit_entries():
if entry["action"] == action and entry["result"] == result:
return entry
return None
def _assert_null_identity(self, entry):
self.assertIsNotNone(entry)
self.assertIsNone(entry["user_id"])
self.assertIsNone(entry["app_registration_uuid"])
self.assertIsNone(entry["agent_id"])
def test_login_logout_whoami_flow_and_cookie_contract(self):
raw = self._login_cookie()
self.assertTrue(raw.startswith("broker_session="))
self.assertIn("Path=/", raw)
self.assertIn("HttpOnly", raw)
self.assertIn("SameSite=Lax", raw)
cookie = {"Cookie": "broker_session=" + self._cookie_id(raw)}
status, body, _ = self._request_json("GET", WHOAMI_PATH, headers=cookie)
self.assertEqual(200, status)
self.assertEqual(ADMIN_USER, body["username"])
created = datetime.datetime.fromisoformat(body["created_at"])
expires = datetime.datetime.fromisoformat(body["expires_at"])
self.assertEqual(datetime.timedelta(seconds=300), expires - created)
status, body, _ = self._request_json("POST", LOGOUT_PATH, headers=cookie)
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
status, body, _ = self._request_json("GET", WHOAMI_PATH, headers=cookie)
self.assertEqual(401, status)
self.assertEqual(401, body["error"])
self.assertIsNone(self.broker.store.get_admin_session(self._cookie_id(raw)))
self._assert_null_identity(self._audit_find("login", "ok"))
self._assert_null_identity(self._audit_find("logout", "ok"))
def test_wrong_password_is_401_with_denied_audit(self):
status, body, headers = self._login(password="not-the-pass")
self.assertEqual(401, status)
self.assertEqual(401, body["error"])
self.assertTrue(body["message"])
self.assertIsNone(headers.get("Set-Cookie"))
self._assert_null_identity(self._audit_find("login", "denied"))
def test_wrong_username_is_401(self):
status, body, headers = self._login(username="mallory")
self.assertEqual(401, status)
self.assertIsNone(headers.get("Set-Cookie"))
self.assertIsNotNone(self._audit_find("login", "denied"))
def test_login_with_missing_credentials_is_401(self):
status, body, _ = self._request_json("POST", LOGIN_PATH, payload={})
self.assertEqual(401, status)
status, body, _ = self._request_json("POST", LOGIN_PATH)
self.assertEqual(401, status)
self.assertIsNotNone(self._audit_find("login", "denied"))
def test_expired_session_is_rejected_lazily(self):
raw = self._login_cookie()
session_id = self._cookie_id(raw)
self.broker.store.conn.execute(
"UPDATE admin_sessions SET expires_at = ? WHERE session_id = ?",
(_iso_offset(-60), session_id),
)
self.broker.store.conn.commit()
status, body, _ = self._request_json(
"GET", WHOAMI_PATH, headers={"Cookie": "broker_session=" + session_id}
)
self.assertEqual(401, status)
self.assertEqual(401, body["error"])
def test_boot_cleanup_deletes_stale_sessions_keeps_live(self):
self.broker.store.conn.executemany(
"INSERT INTO admin_sessions VALUES (?, ?, ?)",
[
("stale-session", _iso_offset(-600), _iso_offset(-60)),
("live-session", _iso_offset(-600), _iso_offset(300)),
],
)
self.broker.store.conn.commit()
rebooted = self._boot_broker(self.env)
self.assertIsNone(rebooted.store.get_admin_session("stale-session"))
self.assertIsNotNone(rebooted.store.get_admin_session("live-session"))
def test_basic_valid_on_whoami(self):
headers = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
status, body, _ = self._request_json("GET", WHOAMI_PATH, headers=headers)
self.assertEqual(200, status)
self.assertEqual(ADMIN_USER, body["username"])
self.assertIsNone(body["created_at"])
self.assertIsNone(body["expires_at"])
def test_basic_invalid_is_401(self):
headers = {"Authorization": _basic_header(ADMIN_USER, "wrong-pass")}
status, body, _ = self._request_json("GET", WHOAMI_PATH, headers=headers)
self.assertEqual(401, status)
self.assertEqual(401, body["error"])
self.assertTrue(body["message"])
def test_whoami_without_credentials_is_401(self):
status, body, _ = self._request_json("GET", WHOAMI_PATH)
self.assertEqual(401, status)
self.assertEqual(401, body["error"])
def test_valid_session_cookie_wins_over_invalid_basic(self):
raw = self._login_cookie()
headers = {
"Cookie": "broker_session=" + self._cookie_id(raw),
"Authorization": _basic_header(ADMIN_USER, "wrong-pass"),
}
status, body, _ = self._request_json("GET", WHOAMI_PATH, headers=headers)
self.assertEqual(200, status)
self.assertEqual(ADMIN_USER, body["username"])
def test_logout_without_credentials_is_401(self):
status, body, _ = self._request_json("POST", LOGOUT_PATH)
self.assertEqual(401, status)
self.assertEqual(401, body["error"])
self.assertIsNone(self._audit_find("logout", "ok"))
def test_logout_via_basic_without_session_is_ok(self):
headers = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
status, body, _ = self._request_json("POST", LOGOUT_PATH, headers=headers)
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
self._assert_null_identity(self._audit_find("logout", "ok"))
def test_session_ttl_is_configurable(self):
env = dict(self.env)
env["BROKER_DB_PATH"] = os.path.join(self.tmp, "ttl.db")
env["BROKER_AUDIT_LOG_PATH"] = os.path.join(self.tmp, "ttl.audit.log")
env["BROKER_SESSION_TTL"] = "60"
broker = self._boot_broker(env)
server = coreci_broker.serve("127.0.0.1", 0, broker)
self.addCleanup(server.shutdown)
self.addCleanup(server.server_close)
base = "http://127.0.0.1:%d" % server.server_address[1]
raw = self._login_cookie(base=base)
status, body, _ = self._request_json(
"GET",
WHOAMI_PATH,
headers={"Cookie": "broker_session=" + self._cookie_id(raw)},
base=base,
)
self.assertEqual(200, status)
created = datetime.datetime.fromisoformat(body["created_at"])
expires = datetime.datetime.fromisoformat(body["expires_at"])
self.assertEqual(datetime.timedelta(seconds=60), expires - created)
if __name__ == "__main__":
unittest.main()
+134
View File
@@ -0,0 +1,134 @@
import base64
import contextlib
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
REGISTER_PATH = "/api/broker/agents/register"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
META_KEYS = {"agent_id", "app_registration_uuid", "user_id", "tags",
"token_expires_at", "first_registered_at", "last_seen_at"}
FORBIDDEN_RAW = ('"token"', '"token_hash"', '"pepper"')
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
class AgentMetaTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
}
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(dict(self.env))
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
self.app_uuid = str(uuidlib.uuid4())
self.user_id = "ceo"
self._seed(self.app_uuid, self.user_id)
self.registered = self._register("agent-1", ["b", "a"])
self.token = self.registered["token"]
self.pepper = self.registered["pepper"]
self.app = {"Authorization": "App %s %s" % (self.app_uuid, self.token)}
def _request(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(self.base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
raw = response.read().decode("utf-8")
return response.status, json.loads(raw), raw
except urllib.error.HTTPError as error:
raw = error.read().decode("utf-8")
return error.code, json.loads(raw), raw
def _seed(self, app_uuid, user_id):
status, _, _ = self._request("POST", APPS_PATH, {"uuid": app_uuid}, self.admin)
self.assertEqual(201, status)
status, _, _ = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
self.assertEqual(201, status)
def _register(self, agent_id, tags):
payload = {"agent_id": agent_id, "app_registration_uuid": self.app_uuid,
"user_id": self.user_id, "tags": tags}
status, body, _ = self._request("POST", REGISTER_PATH, payload)
self.assertEqual(201, status)
return body
def _meta(self, agent_id, headers=None):
return self._request(
"GET", "/api/broker/agents/" + agent_id, headers=headers if headers is not None else self.app
)
def test_identity_match_returns_metadata_only(self):
status, body, raw = self._meta("agent-1")
self.assertEqual(200, status)
self.assertEqual(META_KEYS, set(body))
self.assertEqual("agent-1", body["agent_id"])
self.assertEqual(self.app_uuid, body["app_registration_uuid"])
self.assertEqual(self.user_id, body["user_id"])
self.assertEqual(self.registered["token_expires_at"], body["token_expires_at"])
row = self.broker.store.get_agent("agent-1", self.app_uuid)
self.assertEqual(row[6], body["first_registered_at"])
self.assertEqual(row[7], body["last_seen_at"])
for forbidden in FORBIDDEN_RAW:
self.assertNotIn(forbidden, raw)
self.assertNotIn(self.token, raw)
self.assertNotIn(self.pepper, raw)
def test_tags_roundtrip_is_sorted_canonical_list(self):
status, body, _ = self._meta("agent-1")
self.assertEqual(200, status)
self.assertEqual(["a", "b"], body["tags"])
def test_url_agent_id_mismatch_is_403(self):
self._register("agent-2", ["coreci"])
status, body, raw = self._meta("agent-2")
self.assertEqual(403, status)
self.assertEqual(403, body["error"])
self.assertTrue(body["message"])
for forbidden in FORBIDDEN_RAW:
self.assertNotIn(forbidden, raw)
def test_unknown_agent_id_is_404(self):
status, body, _ = self._meta("ghost-agent")
self.assertEqual(404, status)
self.assertEqual({"error": 404, "message": "not found"}, body)
def test_missing_app_header_is_401(self):
status, body, _ = self._meta("agent-1", headers={})
self.assertEqual(401, status)
self.assertEqual({"error": 401, "message": "invalid app credentials"}, body)
if __name__ == "__main__":
unittest.main()
+177
View File
@@ -0,0 +1,177 @@
import base64
import contextlib
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.request
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
BASE_PATH = "/api/broker/user_allowlist"
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
class UserAllowlistTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
}
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(dict(self.env))
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.server_close)
self.addCleanup(self.server.shutdown)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
def _request(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(self.base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=5) as response:
return response.status, json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
return error.code, json.loads(error.read().decode("utf-8"))
def _add(self, user_id, email=None, headers=None):
payload = {"user_id": user_id}
if email is not None:
payload["email"] = email
return self._request("POST", BASE_PATH, payload, headers or self.admin)
def _row(self, user_id):
status, body = self._request("GET", BASE_PATH, headers=self.admin)
self.assertEqual(200, status)
for row in body["allowlist"]:
if row["user_id"] == user_id:
return row
return None
def _audit(self):
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
return [json.loads(raw) for raw in handle if raw.strip()]
def test_list_empty_then_add_then_list(self):
status, body = self._request("GET", BASE_PATH, headers=self.admin)
self.assertEqual(200, status)
self.assertEqual({"allowlist": []}, body)
status, body = self._add("alice", "alice@coreci.dev")
self.assertEqual(201, status)
self.assertEqual({"user_id": "alice", "status": "active"}, body)
row = self._row("alice")
self.assertEqual("alice@coreci.dev", row["email"])
self.assertEqual("active", row["status"])
self.assertTrue(row["created_at"])
self.assertIsNone(row["revoked_at"])
def test_add_without_email_stores_null_email(self):
status, _ = self._add("bob")
self.assertEqual(201, status)
self.assertIsNone(self._row("bob")["email"])
status, _ = self._request("POST", BASE_PATH, {"user_id": "bob2", "email": 42}, self.admin)
self.assertEqual(201, status)
self.assertIsNone(self._row("bob2")["email"])
def test_add_missing_or_empty_user_id_is_400(self):
for payload in ({"email": "x@coreci.dev"}, {}, {"user_id": ""},
{"user_id": None}, {"user_id": 12345},
{"user_id": ["alice"]}, [1, 2]):
status, body = self._request("POST", BASE_PATH, payload, self.admin)
self.assertEqual(400, status, repr(payload))
self.assertEqual(400, body["error"])
def test_add_duplicate_active_or_revoked_is_409(self):
self._add("carol", "carol@coreci.dev")
status, body = self._add("carol")
self.assertEqual(409, status)
self.assertEqual(409, body["error"])
self.assertTrue(body["message"])
status, _ = self._request("DELETE", BASE_PATH + "/carol", headers=self.admin)
self.assertEqual(200, status)
status, body = self._add("carol")
self.assertEqual(409, status)
def test_revoke_revoke_again_reinstate_lifecycle(self):
self._add("dave", "dave@coreci.dev")
status, body = self._request("DELETE", BASE_PATH + "/dave", headers=self.admin)
self.assertEqual(200, status)
self.assertEqual("revoked", body["status"])
revoked_at = self._row("dave")["revoked_at"]
self.assertIsNotNone(revoked_at)
self.assertEqual("revoked", self._row("dave")["status"])
status, body = self._request("DELETE", BASE_PATH + "/dave", headers=self.admin)
self.assertEqual(200, status)
self.assertEqual(revoked_at, self._row("dave")["revoked_at"])
status, body = self._request(
"POST", BASE_PATH + "/dave/reinstate", headers=self.admin
)
self.assertEqual(200, status)
self.assertEqual("active", body["status"])
row = self._row("dave")
self.assertEqual("active", row["status"])
self.assertIsNone(row["revoked_at"])
def test_revoke_missing_is_404(self):
status, body = self._request("DELETE", BASE_PATH + "/ghost", headers=self.admin)
self.assertEqual(404, status)
self.assertEqual(404, body["error"])
def test_reinstate_missing_is_404(self):
status, body = self._request(
"POST", BASE_PATH + "/ghost/reinstate", headers=self.admin
)
self.assertEqual(404, status)
self.assertEqual(404, body["error"])
def test_admin_actions_audited_with_null_identity(self):
self._add("erin", "erin@coreci.dev")
self._request("DELETE", BASE_PATH + "/erin", headers=self.admin)
self._request("POST", BASE_PATH + "/erin/reinstate", headers=self.admin)
entries = self._audit()
self.assertEqual(["add_user", "remove_user", "add_user"],
[entry["action"] for entry in entries])
for entry in entries:
self.assertEqual("ok", entry["result"])
self.assertIsNone(entry["user_id"])
self.assertIsNone(entry["app_registration_uuid"])
self.assertIsNone(entry["agent_id"])
self.assertEqual("/api/broker/user_allowlist/erin", entry["path"])
def test_unauthenticated_requests_are_401(self):
checks = (
("GET", BASE_PATH, None),
("POST", BASE_PATH, {"user_id": "zoe"}),
("DELETE", BASE_PATH + "/zoe", None),
("POST", BASE_PATH + "/zoe/reinstate", None),
)
for method, path, payload in checks:
status, body = self._request(method, path, payload)
self.assertEqual(401, status, method + " " + path)
self.assertEqual(401, body["error"])
if __name__ == "__main__":
unittest.main()
+263
View File
@@ -0,0 +1,263 @@
import base64
import contextlib
import datetime
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
REGISTER_PATH = "/api/broker/agents/register"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
PROTECTED_PATH = "/test/app-auth"
AUTH_401_BODY = {"error": 401, "message": "invalid app credentials"}
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
def _protected_echo(request):
request.send_json(200, dict(request.app_auth))
class AppAuthTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
}
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(dict(self.env))
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
self.app_uuid = str(uuidlib.uuid4())
self.user_id = "ceo"
self._seed(self.app_uuid, self.user_id)
self.broker.router.add(
"GET", PROTECTED_PATH, coreci_broker._require_app(_protected_echo)
)
self.token = self._register("agent-1")["token"]
def _request(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(self.base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
return response.status, json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
return error.code, json.loads(error.read().decode("utf-8"))
def _seed(self, app_uuid, user_id):
status, _ = self._request("POST", APPS_PATH, {"uuid": app_uuid}, self.admin)
self.assertEqual(201, status)
status, _ = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
self.assertEqual(201, status)
def _register(self, agent_id):
payload = {
"agent_id": agent_id,
"app_registration_uuid": self.app_uuid,
"user_id": self.user_id,
"tags": ["coreci"],
}
status, body = self._request("POST", REGISTER_PATH, payload)
self.assertEqual(201, status)
return body
def _auth(self, authorization):
headers = {} if authorization is None else {"Authorization": authorization}
return self._request("GET", PROTECTED_PATH, headers=headers)
def _app_header(self, token=None, app_uuid=None):
return "App %s %s" % (app_uuid or self.app_uuid, token or self.token)
def _revoke_app(self):
status, _ = self._request(
"DELETE", APPS_PATH + "/" + self.app_uuid, headers=self.admin
)
self.assertEqual(200, status)
def _patch_column(self, agent_id, column, value):
self.broker.store.conn.execute(
"UPDATE agent_registrations SET %s = ? WHERE agent_id = ?" % column,
(value, agent_id),
)
self.broker.store.conn.commit()
def _count_verify_calls(self):
counter = {"calls": 0}
original = coreci_broker.verify_token
def counted(token, stored):
counter["calls"] += 1
return original(token, stored)
coreci_broker.verify_token = counted
self.addCleanup(setattr, coreci_broker, "verify_token", original)
return counter
def _past(self):
return (datetime.datetime.now(datetime.timezone.utc)
- datetime.timedelta(seconds=60)).isoformat()
def _future(self):
return (datetime.datetime.now(datetime.timezone.utc)
+ datetime.timedelta(seconds=3600)).isoformat()
def test_valid_header_returns_context_and_updates_last_seen(self):
past = self._past()
self._patch_column("agent-1", "last_seen_at", past)
status, body = self._auth(self._app_header())
self.assertEqual(200, status)
self.assertEqual("agent-1", body["agent_id"])
self.assertEqual(self.user_id, body["user_id"])
self.assertEqual(self.app_uuid, body["app_registration_uuid"])
row = self.broker.store.get_agent("agent-1", self.app_uuid)
self.assertEqual(row[5], body["token_expires_at"])
self.assertGreater(row[7], past)
def test_401_matrix_uniform_contract(self):
cases = (
(None),
(_basic_header(ADMIN_USER, ADMIN_PASS)),
("App"),
("App %s" % self.app_uuid),
("App %s %s extra" % (self.app_uuid, self.token)),
(self._app_header(app_uuid=str(uuidlib.uuid4()))),
(self._app_header(token="A" * 44)),
("App not-a-uuid %s" % self.token),
("App %s %s" % (self.app_uuid.upper(), self.token)),
)
for header in cases:
with self.subTest(header=header):
status, body = self._auth(header)
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
def test_cache_first_success_scans_then_identical_call_hits_cache(self):
counter = self._count_verify_calls()
status, body = self._auth(self._app_header())
self.assertEqual(200, status)
self.assertEqual(1, counter["calls"])
status, body = self._auth(self._app_header())
self.assertEqual(200, status)
self.assertEqual("agent-1", body["agent_id"])
self.assertEqual(1, counter["calls"])
self.assertIn((self.app_uuid, self.token), self.broker.token_cache)
def test_clear_token_cache_forces_reverify(self):
counter = self._count_verify_calls()
self._auth(self._app_header())
self.assertEqual(1, counter["calls"])
self.broker.clear_token_cache()
self.assertEqual({}, self.broker.token_cache)
status, _ = self._auth(self._app_header())
self.assertEqual(200, status)
self.assertEqual(2, counter["calls"])
def test_cache_hit_revoked_app_still_200_until_ttl(self):
counter = self._count_verify_calls()
self._auth(self._app_header())
self._revoke_app()
status, body = self._auth(self._app_header())
self.assertEqual(200, status)
self.assertEqual("agent-1", body["agent_id"])
self.assertEqual(1, counter["calls"])
row = self.broker.store.get_row("app_registrations", "uuid", self.app_uuid)
self.assertEqual("revoked", row[2])
def test_revoked_app_first_auth_still_200(self):
self._revoke_app()
counter = self._count_verify_calls()
status, body = self._auth(self._app_header())
self.assertEqual(200, status)
self.assertEqual("agent-1", body["agent_id"])
self.assertEqual(1, counter["calls"])
def test_revoked_app_expired_row_is_401(self):
self._revoke_app()
self._patch_column("agent-1", "token_expires_at", self._past())
status, body = self._auth(self._app_header())
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
def test_cache_hit_expired_row_401_then_evicted_and_reverifies(self):
counter = self._count_verify_calls()
self._auth(self._app_header())
self.assertEqual(1, counter["calls"])
self._patch_column("agent-1", "token_expires_at", self._past())
status, body = self._auth(self._app_header())
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
self.assertEqual(1, counter["calls"])
self.assertEqual({}, self.broker.token_cache)
self._patch_column("agent-1", "token_expires_at", self._future())
status, body = self._auth(self._app_header())
self.assertEqual(200, status)
self.assertEqual(2, counter["calls"])
self.assertIn((self.app_uuid, self.token), self.broker.token_cache)
def test_cache_hit_row_absent_is_401(self):
self._auth(self._app_header())
self.broker.store.conn.execute(
"DELETE FROM agent_registrations WHERE agent_id = ?", ("agent-1",)
)
self.broker.store.conn.commit()
status, body = self._auth(self._app_header())
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
self.assertEqual({}, self.broker.token_cache)
def test_scrypt_hash_format_authenticates(self):
token2 = self._register("agent-2")["token"]
original_bcrypt = coreci_broker.bcrypt
coreci_broker.bcrypt = None
try:
scrypt_hash = coreci_broker.hash_token(token2)
finally:
coreci_broker.bcrypt = original_bcrypt
self.assertTrue(scrypt_hash.startswith("scrypt$"))
self._patch_column("agent-2", "token_hash", scrypt_hash)
status, body = self._auth(self._app_header(token=token2))
self.assertEqual(200, status)
self.assertEqual("agent-2", body["agent_id"])
def test_bcrypt_hash_format_authenticates(self):
if coreci_broker.bcrypt is None:
self.skipTest("bcrypt not importable in this environment")
token3 = self._register("agent-3")["token"]
stored = self.broker.store.get_agent("agent-3", self.app_uuid)[4]
self.assertTrue(stored.startswith("$2"))
status, body = self._auth(self._app_header(token=token3))
self.assertEqual(200, status)
self.assertEqual("agent-3", body["agent_id"])
if __name__ == "__main__":
unittest.main()
+187
View File
@@ -0,0 +1,187 @@
import base64
import contextlib
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
BASE_PATH = "/api/broker/app_registrations"
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
class AppRegistrationsTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
}
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(dict(self.env))
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.server_close)
self.addCleanup(self.server.shutdown)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
def _request(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(self.base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=5) as response:
return response.status, json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
return error.code, json.loads(error.read().decode("utf-8"))
def _add(self, app_uuid, name=None, headers=None):
payload = {"uuid": app_uuid}
if name is not None:
payload["name"] = name
return self._request("POST", BASE_PATH, payload, headers or self.admin)
def _row(self, app_uuid):
status, body = self._request("GET", BASE_PATH, headers=self.admin)
self.assertEqual(200, status)
for row in body["registrations"]:
if row["uuid"] == app_uuid:
return row
return None
def _audit(self):
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
return [json.loads(raw) for raw in handle if raw.strip()]
def test_list_empty_then_add_then_list(self):
status, body = self._request("GET", BASE_PATH, headers=self.admin)
self.assertEqual(200, status)
self.assertEqual({"registrations": []}, body)
app_uuid = str(uuidlib.uuid4())
status, body = self._add(app_uuid, "proxmox-tool")
self.assertEqual(201, status)
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
row = self._row(app_uuid)
self.assertEqual("proxmox-tool", row["name"])
self.assertEqual("active", row["status"])
self.assertTrue(row["created_at"])
self.assertIsNone(row["revoked_at"])
def test_add_without_name_stores_null_name(self):
app_uuid = str(uuidlib.uuid4())
status, _ = self._add(app_uuid)
self.assertEqual(201, status)
self.assertIsNone(self._row(app_uuid)["name"])
def test_add_duplicate_active_or_revoked_is_409(self):
app_uuid = str(uuidlib.uuid4())
self._add(app_uuid)
status, body = self._add(app_uuid)
self.assertEqual(409, status)
self.assertEqual(409, body["error"])
self.assertTrue(body["message"])
status, _ = self._request("DELETE", BASE_PATH + "/" + app_uuid, headers=self.admin)
self.assertEqual(200, status)
status, body = self._add(app_uuid)
self.assertEqual(409, status)
def test_add_invalid_or_noncanonical_uuid_is_400(self):
canonical = str(uuidlib.uuid4())
for bad in ("not-a-uuid", "", "{" + canonical.upper() + "}", canonical.upper(),
canonical.replace("-", ""), "urn:uuid:" + canonical):
status, body = self._add(bad)
self.assertEqual(400, status, bad)
self.assertEqual(400, body["error"])
status, _ = self._request("POST", BASE_PATH, {}, self.admin)
self.assertEqual(400, status)
status, _ = self._request("POST", BASE_PATH, {"uuid": 12345}, self.admin)
self.assertEqual(400, status)
status, _ = self._request("POST", BASE_PATH, {"name": "no-uuid"}, self.admin)
self.assertEqual(400, status)
def test_revoke_revoke_again_reinstate_lifecycle(self):
app_uuid = str(uuidlib.uuid4())
self._add(app_uuid)
status, body = self._request("DELETE", BASE_PATH + "/" + app_uuid, headers=self.admin)
self.assertEqual(200, status)
self.assertEqual("revoked", body["status"])
revoked_at = self._row(app_uuid)["revoked_at"]
self.assertIsNotNone(revoked_at)
self.assertEqual("revoked", self._row(app_uuid)["status"])
status, body = self._request("DELETE", BASE_PATH + "/" + app_uuid, headers=self.admin)
self.assertEqual(200, status)
self.assertEqual(revoked_at, self._row(app_uuid)["revoked_at"])
status, body = self._request(
"POST", BASE_PATH + "/" + app_uuid + "/reinstate", headers=self.admin
)
self.assertEqual(200, status)
self.assertEqual("active", body["status"])
row = self._row(app_uuid)
self.assertEqual("active", row["status"])
self.assertIsNone(row["revoked_at"])
def test_revoke_missing_is_404(self):
status, body = self._request(
"DELETE", BASE_PATH + "/" + str(uuidlib.uuid4()), headers=self.admin
)
self.assertEqual(404, status)
self.assertEqual(404, body["error"])
def test_reinstate_missing_is_404(self):
status, body = self._request(
"POST", BASE_PATH + "/" + str(uuidlib.uuid4()) + "/reinstate", headers=self.admin
)
self.assertEqual(404, status)
self.assertEqual(404, body["error"])
def test_admin_actions_audited_with_null_identity(self):
app_uuid = str(uuidlib.uuid4())
self._add(app_uuid)
self._request("DELETE", BASE_PATH + "/" + app_uuid, headers=self.admin)
self._request("POST", BASE_PATH + "/" + app_uuid + "/reinstate", headers=self.admin)
entries = self._audit()
self.assertEqual(["add_app", "remove_app", "add_app"], [e["action"] for e in entries])
for entry in entries:
self.assertEqual("ok", entry["result"])
self.assertIsNone(entry["user_id"])
self.assertIsNone(entry["app_registration_uuid"])
self.assertIsNone(entry["agent_id"])
self.assertEqual("/api/broker/app_registrations/" + app_uuid, entry["path"])
def test_unauthenticated_requests_are_401(self):
checks = (
("GET", BASE_PATH, None),
("POST", BASE_PATH, {"uuid": str(uuidlib.uuid4())}),
("DELETE", BASE_PATH + "/" + str(uuidlib.uuid4()), None),
("POST", BASE_PATH + "/" + str(uuidlib.uuid4()) + "/reinstate", None),
)
for method, path, payload in checks:
status, body = self._request(method, path, payload)
self.assertEqual(401, status, method + " " + path)
self.assertEqual(401, body["error"])
if __name__ == "__main__":
unittest.main()
+213
View File
@@ -0,0 +1,213 @@
import contextlib
import datetime
import io
import json
import os
import sys
import tempfile
import threading
import unittest
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
AUDIT_FIELDS = [
"ts",
"user_id",
"app_registration_uuid",
"agent_id",
"action",
"path",
"result",
]
ALL_ACTIONS = [
"register",
"read",
"denied",
"login",
"logout",
"add_app",
"remove_app",
"add_user",
"remove_user",
]
ADMIN_ACTIONS = [
"login",
"logout",
"add_app",
"remove_app",
"add_user",
"remove_user",
]
class AuditTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.env_path = os.path.join(tmp.name, ".env")
self.db_path = os.path.join(tmp.name, "broker.db")
self.audit_path = os.path.join(tmp.name, "broker.audit.log")
def _boot(self, extra_env=None):
env = {
"BROKER_ENV_PATH": self.env_path,
"BROKER_DB_PATH": self.db_path,
"BROKER_AUDIT_LOG_PATH": self.audit_path,
}
if extra_env:
env.update(extra_env)
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer):
broker = coreci_broker.boot(env)
self.addCleanup(broker.store.close)
self.addCleanup(broker.audit.close)
return broker
def _raw_lines(self):
with open(self.audit_path, "r", encoding="utf-8") as handle:
return handle.read().splitlines()
def _entries(self):
return [json.loads(raw) for raw in self._raw_lines()]
def test_boot_wires_ready_audit_at_mode_0600(self):
broker = self._boot()
self.assertIsInstance(broker.audit, coreci_broker.Audit)
self.assertEqual(self.audit_path, broker.audit.path)
self.assertTrue(os.path.isfile(self.audit_path))
self.assertEqual(0o600, os.stat(self.audit_path).st_mode & 0o777)
def test_audit_mode_0600_independent_of_umask(self):
old_umask = os.umask(0o777)
try:
broker = self._boot()
finally:
os.umask(old_umask)
broker.audit.write("login", path="/ui/login")
self.assertEqual(0o600, os.stat(self.audit_path).st_mode & 0o777)
def test_each_write_appends_exactly_one_valid_json_line(self):
broker = self._boot()
self.assertEqual(0, len(self._raw_lines()))
broker.audit.write("login", path="/api/broker/auth/login")
broker.audit.write(
"read",
path="coreci/proxmox",
user_id="staff-7",
app_registration_uuid="0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a",
agent_id="proxmox",
)
broker.audit.write("denied", path="coreci/gitea", result="denied")
lines = self._raw_lines()
self.assertEqual(3, len(lines))
for line in lines:
self.assertTrue(line)
entries = self._entries()
for entry in entries:
self.assertEqual(AUDIT_FIELDS, list(entry))
self.assertEqual(["login", "read", "denied"], [e["action"] for e in entries])
def test_all_nine_action_verbs_accepted_and_written(self):
broker = self._boot()
for action in ALL_ACTIONS:
broker.audit.write(action, path="/x")
entries = self._entries()
self.assertEqual(9, len(entries))
self.assertEqual(sorted(ALL_ACTIONS), sorted(e["action"] for e in entries))
def test_unknown_action_verb_raises_and_appends_nothing(self):
broker = self._boot()
for bad in ("bogus", "", "LOGIN", None, "rotate"):
with self.assertRaises(ValueError):
broker.audit.write(bad, path="/x")
with self.assertRaises(ValueError):
broker.audit.write("read", path="/x", result="bogus")
self.assertEqual(0, len(self._raw_lines()))
def test_admin_actions_carry_null_identity_fields(self):
broker = self._boot()
for action in ADMIN_ACTIONS:
broker.audit.write(action, path="/api/broker/app_registrations")
entries = self._entries()
self.assertEqual(len(ADMIN_ACTIONS), len(entries))
for entry in entries:
self.assertIsNone(entry["user_id"])
self.assertIsNone(entry["app_registration_uuid"])
self.assertIsNone(entry["agent_id"])
broker.audit.write(
"register",
path="/api/broker/agents/register",
user_id="staff-7",
app_registration_uuid="0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a",
agent_id="proxmox",
)
entry = self._entries()[-1]
self.assertEqual("staff-7", entry["user_id"])
self.assertEqual("0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a", entry["app_registration_uuid"])
self.assertEqual("proxmox", entry["agent_id"])
def test_append_only_preserves_every_line_across_many_writes(self):
broker = self._boot()
for i in range(50):
broker.audit.write("read", path=f"coreci/agent-{i}")
entries = self._entries()
self.assertEqual(50, len(entries))
self.assertEqual("coreci/agent-0", entries[0]["path"])
self.assertEqual("coreci/agent-49", entries[-1]["path"])
self.assertEqual(
[f"coreci/agent-{i}" for i in range(50)],
[e["path"] for e in entries],
)
def test_reboot_appends_without_truncating_history(self):
first = self._boot()
first.audit.write("login", path="/ui/login")
first.store.close()
first.audit.close()
second = self._boot()
second.audit.write("logout", path="/ui/logout")
entries = self._entries()
self.assertEqual(["login", "logout"], [e["action"] for e in entries])
self.assertEqual(0o600, os.stat(self.audit_path).st_mode & 0o777)
def test_ts_parses_as_iso8601_utc(self):
broker = self._boot()
broker.audit.write("read", path="coreci/proxmox", user_id="staff-7")
for entry in self._entries():
parsed = datetime.datetime.fromisoformat(entry["ts"])
self.assertEqual(datetime.timedelta(0), parsed.utcoffset())
def test_result_values_written(self):
broker = self._boot()
broker.audit.write("read", path="coreci/proxmox", result="ok")
broker.audit.write("read", path="coreci/gitea", result="denied")
broker.audit.write("read", path="coreci/boom", result="error")
self.assertEqual(["ok", "denied", "error"], [e["result"] for e in self._entries()])
def test_concurrent_writes_are_serialized(self):
broker = self._boot()
threads = []
for t in range(4):
def worker(t=t):
for i in range(10):
broker.audit.write("read", path=f"coreci/t{t}-i{i}")
thread = threading.Thread(target=worker)
threads.append(thread)
for thread in threads:
thread.start()
for thread in threads:
thread.join()
entries = self._entries()
self.assertEqual(40, len(entries))
self.assertEqual(
sorted(f"coreci/t{t}-i{i}" for t in range(4) for i in range(10)),
sorted(e["path"] for e in entries),
)
if __name__ == "__main__":
unittest.main()
+305
View File
@@ -0,0 +1,305 @@
import base64
import contextlib
import io
import json
import os
import sys
import tempfile
import time
import unittest
import urllib.error
import urllib.parse
import urllib.request
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
AUDIT_PATH = "/api/broker/audit"
UUID_A = "0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a"
UUID_B = "3f2d8a04-9c17-4b1e-8d2f-6a5b4c3d2e1f"
SEED = [
{"action": "read", "path": "coreci/proxmox", "user_id": "staff-1",
"app_registration_uuid": UUID_A, "agent_id": "proxmox"},
{"action": "read", "path": "coreci/gitea", "user_id": "staff-2",
"app_registration_uuid": UUID_A, "agent_id": "gitea"},
{"action": "denied", "path": "coreci/gitea", "result": "denied",
"user_id": "staff-2", "app_registration_uuid": UUID_B, "agent_id": "gitea"},
{"action": "add_app", "path": "/api/broker/app_registrations"},
{"action": "read", "path": "coreci/proxmox", "user_id": "staff-1",
"app_registration_uuid": UUID_B, "agent_id": "proxmox"},
{"action": "login", "path": "/api/broker/auth/login"},
{"action": "remove_user", "path": "/api/broker/user_allowlist/staff-2"},
]
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
class AuditQueryTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.audit_path = os.path.join(tmp.name, "broker.audit.log")
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot({
"BROKER_ENV_PATH": os.path.join(tmp.name, ".env"),
"BROKER_DB_PATH": os.path.join(tmp.name, "broker.db"),
"BROKER_AUDIT_LOG_PATH": self.audit_path,
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
})
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.auth = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
def _request_json(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(self.base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=5) as response:
return response.status, json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
return error.code, json.loads(error.read().decode("utf-8"))
def _get(self, params=None, headers=None):
path = AUDIT_PATH
if params:
path += "?" + urllib.parse.urlencode(params)
return self._request_json(
"GET", path, headers=self.auth if headers is None else headers
)
def _seed(self, specs):
for spec in specs:
self.broker.audit.write(**spec)
time.sleep(0.005)
def _ts(self):
values = []
with open(self.audit_path, "r", encoding="utf-8") as handle:
for raw in handle:
try:
values.append(json.loads(raw)["ts"])
except (ValueError, TypeError, KeyError):
continue
return values
def _paths(self, body):
return [entry["path"] for entry in body["entries"]]
def test_no_filters_returns_all_entries_with_null_cursor(self):
self._seed(SEED)
status, body = self._get()
self.assertEqual(200, status)
with open(self.audit_path, "r", encoding="utf-8") as handle:
expected = [json.loads(raw) for raw in handle]
self.assertEqual(expected, body["entries"])
self.assertEqual(len(SEED), len(body["entries"]))
self.assertIsNone(body["next_since"])
def test_user_id_filter_exact_match(self):
self._seed(SEED)
status, body = self._get({"user_id": "staff-1"})
self.assertEqual(200, status)
self.assertEqual(["coreci/proxmox", "coreci/proxmox"], self._paths(body))
self.assertEqual([UUID_A, UUID_B],
[e["app_registration_uuid"] for e in body["entries"]])
self.assertIsNone(body["next_since"])
def test_app_registration_uuid_filter_exact_match(self):
self._seed(SEED)
status, body = self._get({"app_registration_uuid": UUID_B})
self.assertEqual(200, status)
self.assertEqual(["denied", "read"], [e["action"] for e in body["entries"]])
self.assertEqual(["coreci/gitea", "coreci/proxmox"], self._paths(body))
status, body = self._get({"app_registration_uuid": "no-such-uuid"})
self.assertEqual(200, status)
self.assertEqual([], body["entries"])
self.assertIsNone(body["next_since"])
def test_action_filter_exact_match(self):
self._seed(SEED)
status, body = self._get({"action": "denied"})
self.assertEqual(200, status)
self.assertEqual(["coreci/gitea"], self._paths(body))
status, body = self._get({"action": "read"})
self.assertEqual(["coreci/proxmox", "coreci/gitea", "coreci/proxmox"],
self._paths(body))
status, body = self._get({"action": "rea"})
self.assertEqual(200, status)
self.assertEqual([], body["entries"])
def test_since_filter_is_exclusive(self):
self._seed(SEED)
ts = self._ts()
status, body = self._get({"since": ts[2]})
self.assertEqual(200, status)
self.assertEqual(["add_app", "read", "login", "remove_user"],
[e["action"] for e in body["entries"]])
self.assertEqual(4, len(body["entries"]))
self.assertIsNone(body["next_since"])
status, body = self._get({"since": ts[-1]})
self.assertEqual(200, status)
self.assertEqual([], body["entries"])
self.assertIsNone(body["next_since"])
def test_until_filter_is_inclusive(self):
self._seed(SEED)
ts = self._ts()
status, body = self._get({"until": ts[2]})
self.assertEqual(200, status)
self.assertEqual(["read", "read", "denied"],
[e["action"] for e in body["entries"]])
self.assertEqual(3, len(body["entries"]))
self.assertIsNone(body["next_since"])
def test_since_and_until_range_together(self):
self._seed(SEED)
ts = self._ts()
status, body = self._get({"since": ts[1], "until": ts[4]})
self.assertEqual(200, status)
self.assertEqual(["denied", "add_app", "read"], [e["action"] for e in body["entries"]])
status, body = self._get({"since": ts[3], "until": ts[3]})
self.assertEqual(200, status)
self.assertEqual([], body["entries"])
def test_combined_filters(self):
self._seed(SEED)
ts = self._ts()
status, body = self._get({"user_id": "staff-1", "action": "read"})
self.assertEqual(200, status)
self.assertEqual(["coreci/proxmox", "coreci/proxmox"], self._paths(body))
status, body = self._get({"user_id": "staff-2", "action": "read"})
self.assertEqual(["coreci/gitea"], self._paths(body))
status, body = self._get(
{"app_registration_uuid": UUID_B, "since": ts[2], "until": ts[6]}
)
self.assertEqual(["coreci/proxmox"], self._paths(body))
status, body = self._get(
{"user_id": "staff-1", "since": ts[0], "until": ts[4]}
)
self.assertEqual(["coreci/proxmox"], self._paths(body))
status, body = self._get(
{"user_id": "staff-1", "action": "denied",
"app_registration_uuid": UUID_A, "since": ts[0], "until": ts[6]}
)
self.assertEqual([], body["entries"])
def test_limit_pagination_with_next_since_cursor(self):
specs = [{"action": "read", "path": "coreci/agent-%d" % i} for i in range(5)]
self._seed(specs)
ts = self._ts()
status, body = self._get({"limit": 2})
self.assertEqual(200, status)
self.assertEqual(["coreci/agent-0", "coreci/agent-1"], self._paths(body))
self.assertEqual(ts[1], body["next_since"])
status, body = self._get({"since": body["next_since"], "limit": 2})
self.assertEqual(200, status)
self.assertEqual(["coreci/agent-2", "coreci/agent-3"], self._paths(body))
self.assertEqual(ts[3], body["next_since"])
status, body = self._get({"since": ts[1]})
self.assertEqual(200, status)
self.assertEqual(["coreci/agent-2", "coreci/agent-3", "coreci/agent-4"],
self._paths(body))
self.assertIsNone(body["next_since"])
status, body = self._get({"since": ts[4]})
self.assertEqual(200, status)
self.assertEqual([], body["entries"])
self.assertIsNone(body["next_since"])
def test_limit_zero_and_negative_clamp_to_one(self):
self._seed([{"action": "read", "path": "coreci/a"},
{"action": "read", "path": "coreci/b"},
{"action": "read", "path": "coreci/c"}])
ts = self._ts()
for bad in (0, -3):
status, body = self._get({"limit": bad})
self.assertEqual(200, status)
self.assertEqual(["coreci/a"], self._paths(body))
self.assertEqual(ts[0], body["next_since"])
def test_limit_above_max_and_non_numeric_are_clamped(self):
self._seed([{"action": "read", "path": "coreci/a"},
{"action": "read", "path": "coreci/b"},
{"action": "read", "path": "coreci/c"}])
for value in (5000, "abc"):
status, body = self._get({"limit": value})
self.assertEqual(200, status)
self.assertEqual(3, len(body["entries"]))
self.assertIsNone(body["next_since"])
def test_default_limit_is_exactly_100(self):
specs = [{"action": "read", "path": "coreci/agent-%d" % i} for i in range(100)]
self._seed(specs)
ts = self._ts()
status, body = self._get()
self.assertEqual(200, status)
self.assertEqual(100, len(body["entries"]))
self.assertEqual("coreci/agent-99", body["entries"][-1]["path"])
self.assertEqual(ts[99], body["next_since"])
status, body = self._get({"since": ts[99]})
self.assertEqual(200, status)
self.assertEqual([], body["entries"])
self.assertIsNone(body["next_since"])
def test_invalid_since_or_until_is_400(self):
self._seed(SEED)
status, body = self._get({"since": "not-a-timestamp"})
self.assertEqual(400, status)
self.assertEqual(400, body["error"])
self.assertTrue(body["message"])
status, body = self._get({"until": "2026-02-30T00:00:00Z"})
self.assertEqual(400, status)
self.assertEqual(400, body["error"])
self.assertTrue(body["message"])
status, body = self._get({"since": ""})
self.assertEqual(200, status)
self.assertEqual(len(SEED), len(body["entries"]))
def test_malformed_lines_skipped(self):
self._seed([{"action": "read", "path": "coreci/a"},
{"action": "read", "path": "coreci/b"}])
with open(self.audit_path, "a", encoding="utf-8") as handle:
handle.write("{not json\n")
handle.write("[1, 2, 3]\n")
self._seed([{"action": "read", "path": "coreci/c"}])
ts = self._ts()
self.assertEqual(3, len(ts))
status, body = self._get()
self.assertEqual(200, status)
self.assertEqual(["coreci/a", "coreci/b", "coreci/c"], self._paths(body))
self.assertIsNone(body["next_since"])
status, body = self._get({"limit": 2})
self.assertEqual(200, status)
self.assertEqual(["coreci/a", "coreci/b"], self._paths(body))
self.assertEqual(ts[1], body["next_since"])
status, body = self._get({"since": ts[1], "action": "read"})
self.assertEqual(200, status)
self.assertEqual(["coreci/c"], self._paths(body))
def test_unauthenticated_is_401(self):
self._seed(SEED)
status, body = self._get(headers={})
self.assertEqual(401, status)
self.assertEqual(401, body["error"])
status, body = self._get(
headers={"Authorization": _basic_header(ADMIN_USER, "wrong-pass")}
)
self.assertEqual(401, status)
self.assertEqual(401, body["error"])
if __name__ == "__main__":
unittest.main()
+188
View File
@@ -0,0 +1,188 @@
import base64
import contextlib
import datetime
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.parse
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
REGISTER_PATH = "/api/broker/agents/register"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
AUDIT_PATH = "/api/broker/audit"
STATE_PATH = "/api/broker/agents/%s/state"
AUTH_401_BODY = {"error": 401, "message": "invalid app credentials"}
BAD_JSON_BODY = {"error": 400, "message": "invalid JSON body"}
NEST_DEPTH = 50000
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
class AuthAuditTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
}
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(dict(self.env))
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
self.app_uuid = str(uuidlib.uuid4())
self.user_id = "ceo"
self.agent_id = "agent-1"
self._seed(self.app_uuid, self.user_id)
self.token = self._register()["token"]
self.app = {"Authorization": "App %s %s" % (self.app_uuid, self.token)}
def _raw(self, method, path, data, headers):
request = urllib.request.Request(self.base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
return response.status, json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
return error.code, json.loads(error.read().decode("utf-8"))
def _request(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
return self._raw(method, path, data, headers)
def _seed(self, app_uuid, user_id):
status, _ = self._request("POST", APPS_PATH, {"uuid": app_uuid}, self.admin)
self.assertEqual(201, status)
status, _ = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
self.assertEqual(201, status)
def _register(self):
payload = {"agent_id": self.agent_id, "app_registration_uuid": self.app_uuid,
"user_id": self.user_id, "tags": ["coreci"]}
status, body = self._request("POST", REGISTER_PATH, payload)
self.assertEqual(201, status)
return body
def _denied(self):
status, body = self._request("GET", AUDIT_PATH + "?action=denied",
headers=self.admin)
self.assertEqual(200, status)
return body["entries"]
def _assert_denied_line(self, entry, app_uuid, path):
self.assertEqual("denied", entry["action"])
self.assertEqual("denied", entry["result"])
self.assertEqual(app_uuid, entry["app_registration_uuid"])
self.assertEqual(path, entry["path"])
self.assertIsNone(entry["user_id"])
self.assertIsNone(entry["agent_id"])
def test_deep_nested_json_body_is_400_not_500(self):
data = ("[" * NEST_DEPTH).encode("utf-8")
self.assertLess(len(data), 1024 * 1024)
status, body = self._raw("PUT", STATE_PATH % self.agent_id, data, self.app)
self.assertEqual(400, status)
self.assertEqual(BAD_JSON_BODY, body)
status, body = self._request("GET", "/healthz")
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
def test_wrong_token_denied_audit_carries_uuid_and_path(self):
path = STATE_PATH % self.agent_id
status, body = self._request(
"GET", path, headers={"Authorization": "App %s %s" % (self.app_uuid, "A" * 44)}
)
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
entries = self._denied()
self.assertEqual(1, len(entries))
self._assert_denied_line(entries[0], self.app_uuid, path)
def test_expired_or_absent_token_same_denied_audit(self):
path = STATE_PATH % self.agent_id
past = (datetime.datetime.now(datetime.timezone.utc)
- datetime.timedelta(seconds=60)).isoformat()
self.broker.store.conn.execute(
"UPDATE agent_registrations SET token_expires_at = ? WHERE agent_id = ?",
(past, self.agent_id),
)
self.broker.store.conn.commit()
status, body = self._request("GET", path, headers=self.app)
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
status, body = self._request(
"GET", path, headers={"Authorization": "App %s" % self.app_uuid}
)
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
entries = self._denied()
self.assertEqual(2, len(entries))
for entry in entries:
self._assert_denied_line(entry, self.app_uuid, path)
def test_missing_app_header_401_writes_no_denied_audit(self):
self.assertEqual(0, len(self._denied()))
path = STATE_PATH % self.agent_id
status, body = self._request("GET", path, headers={})
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
status, body = self._request("GET", path, headers=self.admin)
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
self.assertEqual(0, len(self._denied()))
def test_malformed_uuid_denied_audit_carries_null_uuid(self):
path = STATE_PATH % self.agent_id
status, body = self._request(
"GET", path, headers={"Authorization": "App not-a-uuid %s" % self.token}
)
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
entries = self._denied()
self.assertEqual(1, len(entries))
self._assert_denied_line(entries[0], None, path)
def test_admin_audit_action_filter_returns_denied_lines(self):
path = STATE_PATH % self.agent_id
for header in ("App %s wrong" % self.app_uuid,
"App not-a-uuid %s" % self.token):
status, _ = self._request("GET", path, headers={"Authorization": header})
self.assertEqual(401, status)
query = urllib.parse.urlencode({"action": "denied"})
status, body = self._request("GET", AUDIT_PATH + "?" + query,
headers=self.admin)
self.assertEqual(200, status)
self.assertEqual(2, len(body["entries"]))
self.assertEqual([self.app_uuid, None],
[entry["app_registration_uuid"] for entry in body["entries"]])
for entry in body["entries"]:
self.assertEqual(path, entry["path"])
if __name__ == "__main__":
unittest.main()
+163
View File
@@ -0,0 +1,163 @@
"""Reproducibility and packaging tests for the coreci-broker CLI build.
Covers scripts/build_cli.py (REQ-21): two consecutive same-environment
builds are byte-identical, `_cli_version.VERSION` is embedded, the archive
contains exactly the staging allowlist, entry metadata is normalized to
constants (G-009), both new files py_compile green, and the built artifact
answers `--version` with exit code 0 (shebang + exec bit verified).
"""
import contextlib
import hashlib
import importlib.util
import io
import os
import pathlib
import py_compile
import subprocess
import sys
import tempfile
import unittest
import zipfile
REPO = pathlib.Path(__file__).resolve().parent.parent
BUILD_CLI_PATH = REPO / "scripts" / "build_cli.py"
SHIM_PATH = REPO / "cli" / "__main__.py"
VERSION = "v0.2.0"
UI_PAGES = (
"app_registrations.html",
"audit.html",
"dashboard.html",
"index.html",
"login.html",
"nav.js",
"shared.css",
"user_allowlist.html",
)
FILE_EXTERNAL_ATTR = 0o100644 << 16
DIR_EXTERNAL_ATTR = 0o40755 << 16
SHEBANG = b"#!/usr/bin/env python3\n"
FIXED_DATE_TIME = (1980, 1, 1, 0, 0, 0)
def _load_build_cli():
spec = importlib.util.spec_from_file_location("build_cli", BUILD_CLI_PATH)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
build_cli = _load_build_cli()
def _sha256(path):
digest = hashlib.sha256()
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(65536), b""):
digest.update(chunk)
return digest.hexdigest()
class BuildCliTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.build_root = tempfile.TemporaryDirectory(prefix="coreci-broker-pyz-")
cls.addClassCleanup(cls.build_root.cleanup)
cls.artifact = build_cli.build(
VERSION, repo=str(REPO), dist_dir=cls.build_root.name
)
cls.artifact_sha256 = _sha256(cls.artifact)
def test_two_builds_byte_identical(self):
digests = []
for _ in range(2):
with tempfile.TemporaryDirectory(prefix="coreci-broker-pyz-") as dist_dir:
artifact = build_cli.build(
VERSION, repo=str(REPO), dist_dir=dist_dir
)
digests.append(_sha256(artifact))
self.assertEqual(digests[0], digests[1])
self.assertEqual(self.artifact_sha256, digests[0])
print("two-build sha256: {}".format(digests[0]))
def test_cli_version_module_embedded(self):
with zipfile.ZipFile(self.artifact) as archive:
source = archive.read("_cli_version.py").decode("utf-8")
self.assertEqual('VERSION = "{}"\n'.format(VERSION), source)
namespace = {}
exec(compile(source, "_cli_version.py", "exec"), namespace)
self.assertEqual(VERSION, namespace["VERSION"])
def test_archive_allowlist_exact(self):
ui_dir = REPO / "coreci_broker_ui"
on_disk = {entry.name for entry in ui_dir.iterdir() if entry.is_file()}
self.assertEqual(set(UI_PAGES), on_disk)
expected = {
"__main__.py",
"_cli_version.py",
"coreci_broker.py",
"coreci_broker_client.py",
"coreci_broker_ui/",
}
expected.update("coreci_broker_ui/{}".format(name) for name in UI_PAGES)
with zipfile.ZipFile(self.artifact) as archive:
self.assertEqual(expected, set(archive.namelist()))
def test_entry_metadata_constants(self):
with zipfile.ZipFile(self.artifact) as archive:
infos = archive.infolist()
names = [info.filename for info in infos]
self.assertEqual(sorted(names), names)
self.assertTrue(any(info.is_dir() for info in infos))
self.assertTrue(any(not info.is_dir() for info in infos))
for info in infos:
self.assertEqual(FIXED_DATE_TIME, info.date_time)
if info.is_dir():
self.assertEqual(DIR_EXTERNAL_ATTR, info.external_attr)
else:
self.assertEqual(FILE_EXTERNAL_ATTR, info.external_attr)
self.assertEqual(zipfile.ZIP_DEFLATED, info.compress_type)
def test_py_compile_green(self):
for path in (SHIM_PATH, BUILD_CLI_PATH):
py_compile.compile(str(path), doraise=True)
def test_version_flag(self):
completed = subprocess.run(
[sys.executable, self.artifact, "--version"],
capture_output=True,
text=True,
timeout=60,
)
self.assertEqual(0, completed.returncode)
self.assertEqual(VERSION, completed.stdout.strip())
def test_shebang_and_exec_bit(self):
with open(self.artifact, "rb") as handle:
self.assertEqual(SHEBANG, handle.readline())
self.assertTrue(os.stat(self.artifact).st_mode & 0o111)
completed = subprocess.run(
[self.artifact, "--version"],
capture_output=True,
text=True,
timeout=60,
)
self.assertEqual(0, completed.returncode)
self.assertEqual(VERSION, completed.stdout.strip())
def test_artifact_name_no_double_v(self):
self.assertEqual("coreci-broker-v0.2.0.pyz", build_cli.artifact_name(VERSION))
self.assertEqual("coreci-broker-v0.2.0.pyz", build_cli.artifact_name("0.2.0"))
self.assertEqual("coreci-broker-v0.2.0.pyz", os.path.basename(self.artifact))
def test_cli_requires_version_argument(self):
with contextlib.redirect_stderr(io.StringIO()):
with self.assertRaises(SystemExit) as raised:
build_cli.main([])
self.assertEqual(2, raised.exception.code)
if __name__ == "__main__":
unittest.main()
+528
View File
@@ -0,0 +1,528 @@
"""End-to-end packaged-boot acceptance for the coreci-broker CLI zipapp (REQ-21).
Builds the P1-01 artifact once per class via the scripts/build_cli.py build
API into a ``TemporaryDirectory``, then boots it as a real subprocess child —
always ``[sys.executable, "<pyz>"]``, never a bare "python3" — and drives the
live server with stdlib urllib, reusing the v0.1 acceptance request patterns
(tests/test_acceptance_full.py, tests/test_acceptance_p4.py).
G-010 harness hardening, all bound here:
- explicit deadline loop (30s) polling the merged stdout/stderr pipe for the
``coreci-broker listening on http://…`` line; loud failure with the captured
child output when it never arrives;
- pass-once and silent-second-boot assertions are made only after
``terminate()`` plus a full ``communicate()`` drain of the child;
- child env pins ``BROKER_CREDS_SOURCE=file`` + a seeded ``BROKER_CREDS_FILE``
(v0.1 precedent: tests/test_acceptance_full.py);
- one un-overridden default-anchoring boot — state lands in the child's CWD;
- every child is terminated + drained exactly once (idempotent ``stop``
registered with ``addCleanup``): no leaked processes, and every wait is
deadline-bounded (boot wait, drain timeout with kill fallback, HTTP and
subprocess timeouts).
"""
import importlib.util
import json
import os
import select
import subprocess
import sys
import tempfile
import time
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
BUILD_CLI_PATH = os.path.join(REPO, "scripts", "build_cli.py")
REPO_CLIENT_PATH = os.path.join(REPO, "coreci_broker_client.py")
REPO_LOGIN_PAGE_PATH = os.path.join(REPO, "coreci_broker_ui", "login.html")
REPO_DASHBOARD_PAGE_PATH = os.path.join(REPO, "coreci_broker_ui", "dashboard.html")
VERSION = "v0.2.0"
ADMIN_USER = "admin"
ADMIN_PASS_PREFIX = "BROKER_ADMIN_PASS="
LISTENING_PREFIX = "coreci-broker listening on http://"
ASSET_DIR_NAME = ".coreci_broker_assets"
STATE_FILE_NAMES = (".env", "broker.db", "broker.audit.log")
LOGIN_PATH = "/api/broker/auth/login"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
REGISTER_PATH = "/api/broker/agents/register"
AGENT_STATE_PATH = "/api/broker/agents/proxmox/state"
SECRETS_PATH = "/api/broker/secrets/coreci/proxmox"
AUDIT_PATH = "/api/broker/audit"
CLIENT_STATIC_PATH = "/static/coreci_broker_client.py"
UI_LOGIN_PATH = "/ui/login"
UI_DASHBOARD_PATH = "/ui/dashboard"
PROXMOX_PATH = "coreci/proxmox"
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
GITEA_PATH = "coreci/gitea"
GITEA_SECRET = {"url": "https://git.coreci.dev", "token": "git-tok"}
BOOT_DEADLINE_SECONDS = 30.0
POLL_SECONDS = 0.1
DRAIN_TIMEOUT_SECONDS = 20.0
HTTP_TIMEOUT_SECONDS = 10
SUBPROCESS_TIMEOUT_SECONDS = 60
def _load_build_cli():
spec = importlib.util.spec_from_file_location("build_cli", BUILD_CLI_PATH)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
build_cli = _load_build_cli()
class PackagedBrokerChild:
"""A subprocess child booted from the built zipapp artifact.
stdout and stderr are merged into one pipe read via ``select`` + ``os.read``
(raw fd reads — no buffered-reader/select skew) under an explicit deadline
until the broker announces its ephemeral listen address. ``stop()`` is the
G-010 kill/drain protocol: ``terminate()`` then a full ``communicate()``
drain; it is idempotent and registered with ``addCleanup`` at construction,
so no child leaks even when a boot fails or an assertion trips mid-test.
"""
def __init__(self, testcase, artifact, env, cwd):
self.testcase = testcase
self.artifact = artifact
self.proc = subprocess.Popen(
[sys.executable, artifact],
env=env, cwd=cwd,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
)
self.lines = []
self.partial = b""
self.drained = b""
self.base_url = None
self._eof = False
self._stopped = False
testcase.addCleanup(self.stop)
self._await_listening(BOOT_DEADLINE_SECONDS)
def _pump(self, timeout):
"""Move whatever is on the merged pipe into line storage.
Returns False once the child's stream reached EOF; True otherwise
(including when the poll window simply elapsed with no data).
"""
if self._eof:
return False
fd = self.proc.stdout.fileno()
ready, _, _ = select.select([fd], [], [], timeout)
if not ready:
return True
try:
chunk = os.read(fd, 4096)
except OSError:
chunk = b""
if not chunk:
self._eof = True
return False
self.partial += chunk
while b"\n" in self.partial:
raw, self.partial = self.partial.split(b"\n", 1)
self.lines.append(raw.decode("utf-8", "replace"))
return True
def _await_listening(self, deadline_seconds):
deadline = time.monotonic() + deadline_seconds
cursor = len(self.lines)
while time.monotonic() < deadline:
alive = self._pump(POLL_SECONDS)
while cursor < len(self.lines):
line = self.lines[cursor]
cursor += 1
if line.startswith(LISTENING_PREFIX):
self.base_url = line.split(" on ", 1)[1].strip()
return
if not alive:
break
self.stop()
self.testcase.fail(
"packaged broker did not announce %r within %.0fs "
"(artifact=%s exit=%s)\n--- captured child output ---\n%s"
% (LISTENING_PREFIX, deadline_seconds, self.artifact,
self.proc.returncode, self.full_output())
)
def stop(self):
"""Kill/drain protocol (G-010): terminate, then fully drain."""
if self._stopped:
return
self._stopped = True
if self.proc.poll() is None:
self.proc.terminate()
try:
remainder, _ = self.proc.communicate(timeout=DRAIN_TIMEOUT_SECONDS)
except subprocess.TimeoutExpired:
self.proc.kill()
remainder, _ = self.proc.communicate()
self.drained = remainder or b""
def full_output(self):
"""Everything the child printed: complete lines + unterminated tail."""
text = "\n".join(self.lines)
tail = self.partial + self.drained
if tail:
text += "\n" + tail.decode("utf-8", "replace")
return text
def printed_admin_pass(self):
values = [
line.split("=", 1)[1]
for line in self.lines
if line.startswith(ADMIN_PASS_PREFIX)
]
self.testcase.assertEqual(
1, len(values),
"expected exactly one %s line, got: %r" % (ADMIN_PASS_PREFIX, values)
)
return values[0]
class CliPackagingAcceptanceTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.build_root = tempfile.TemporaryDirectory(prefix="coreci-broker-pkg-")
cls.addClassCleanup(cls.build_root.cleanup)
cls.artifact = build_cli.build(
VERSION, repo=REPO, dist_dir=cls.build_root.name
)
# -- harness helpers -------------------------------------------------
def _temp_dir(self):
handle = tempfile.TemporaryDirectory(prefix="coreci-broker-pkg-")
self.addCleanup(handle.cleanup)
return handle.name
def _child_env(self, **overrides):
"""A child env with no inherited BROKER_* keys, then test overrides."""
env = {
key: value
for key, value in os.environ.items()
if not key.startswith("BROKER_")
}
env.update(overrides)
return env
def _seed_creds(self, state_dir):
creds_path = os.path.join(state_dir, "creds.json")
with open(creds_path, "w", encoding="utf-8") as handle:
json.dump({PROXMOX_PATH: PROXMOX_SECRET, GITEA_PATH: GITEA_SECRET},
handle)
return creds_path
def _state_env(self, state_dir):
"""G-010: pin bind + state paths + a seeded file creds source."""
return self._child_env(
BROKER_BIND="127.0.0.1:0",
BROKER_ENV_PATH=os.path.join(state_dir, ".env"),
BROKER_DB_PATH=os.path.join(state_dir, "broker.db"),
BROKER_AUDIT_LOG_PATH=os.path.join(state_dir, "broker.audit.log"),
BROKER_CREDS_SOURCE="file",
BROKER_CREDS_FILE=self._seed_creds(state_dir),
)
def _boot(self, env, cwd):
return PackagedBrokerChild(self, self.artifact, env, cwd)
def _boot_in_state(self, state_dir, cwd=None):
"""Boot with state pinned into state_dir; CWD is a separate temp dir."""
return self._boot(self._state_env(state_dir),
cwd if cwd is not None else self._temp_dir())
def _assert_state_files_0600_in(self, directory):
for name in STATE_FILE_NAMES:
path = os.path.join(directory, name)
self.assertTrue(os.path.isfile(path), path)
self.assertEqual(0o600, os.stat(path).st_mode & 0o777, path)
def _request(self, base, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(
request, timeout=HTTP_TIMEOUT_SECONDS
) as response:
return response.status, response.read(), response.headers
except urllib.error.HTTPError as error:
return error.code, error.read(), error.headers
def _request_json(self, base, method, path, payload=None, headers=None):
status, body, response_headers = self._request(
base, method, path, payload, headers
)
return status, json.loads(body.decode("utf-8")), response_headers
def _admin_login(self, base, password):
"""v0.1 pattern: POST login, assert the broker_session cookie is set."""
status, body, headers = self._request_json(
base, "POST", LOGIN_PATH,
{"username": ADMIN_USER, "password": password},
)
self.assertEqual(200, status)
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
raw_cookie = headers.get("Set-Cookie")
self.assertTrue(raw_cookie)
self.assertTrue(raw_cookie.startswith("broker_session="))
self.assertIn("HttpOnly", raw_cookie)
self.assertIn("SameSite=Lax", raw_cookie)
return {"Cookie": raw_cookie.split(";", 1)[0]}
def _assert_tool_identity(self, entry, app_uuid):
self.assertEqual("ceo", entry["user_id"])
self.assertEqual(app_uuid, entry["app_registration_uuid"])
self.assertEqual("proxmox", entry["agent_id"])
def _read_repo_bytes(self, path):
with open(path, "rb") as handle:
return handle.read()
# -- tests ------------------------------------------------------------
def test_version_flag_subprocess_exits_zero_printing_version(self):
completed = subprocess.run(
[sys.executable, self.artifact, "--version"],
capture_output=True, text=True, timeout=SUBPROCESS_TIMEOUT_SECONDS,
)
self.assertEqual(0, completed.returncode)
self.assertEqual(VERSION, completed.stdout.strip())
def test_first_boot_state_0600_cwd_independent_and_http_acceptance(self):
state_dir = self._temp_dir()
cwd_dir = self._temp_dir() # deliberately NOT the state dir
child = self._boot_in_state(state_dir, cwd=cwd_dir)
base = child.base_url
status, body, _ = self._request_json(base, "GET", "/healthz")
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
# UI pages served from the materialized assets (byte-exact vs repo)
status, body, headers = self._request(base, "GET", UI_LOGIN_PATH)
self.assertEqual(200, status)
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
self.assertEqual(self._read_repo_bytes(REPO_LOGIN_PAGE_PATH), body)
status, body, headers = self._request(base, "GET", UI_DASHBOARD_PATH)
self.assertEqual(200, status)
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
self.assertEqual(self._read_repo_bytes(REPO_DASHBOARD_PAGE_PATH), body)
# packaged client module served byte-exact vs the repo file
status, served, _ = self._request(base, "GET", CLIENT_STATIC_PATH)
self.assertEqual(200, status)
self.assertEqual(self._read_repo_bytes(REPO_CLIENT_PATH), served)
# materialized assets under <state>/.coreci_broker_assets/<VERSION>/
asset_base = os.path.join(state_dir, ASSET_DIR_NAME, VERSION)
self.assertTrue(os.path.isdir(os.path.join(asset_base, "ui")))
self.assertTrue(os.path.isdir(os.path.join(asset_base, "static")))
for name in ("login.html", "dashboard.html"):
self.assertTrue(
os.path.isfile(os.path.join(asset_base, "ui", name)), name
)
self.assertTrue(os.path.isfile(
os.path.join(asset_base, "static", "coreci_broker_client.py")
))
# state resolved from BROKER_ENV_PATH: 0600 files in the state dir,
# nothing whatsoever in the child's CWD
self._assert_state_files_0600_in(state_dir)
self.assertEqual([], os.listdir(cwd_dir))
# pass printed exactly once — asserted on the FULLY drained output
printed_pass = child.printed_admin_pass()
child.stop()
output = child.full_output()
self.assertEqual(1, output.count(ADMIN_PASS_PREFIX), output)
self.assertEqual(1, output.count(printed_pass))
pepper = None
with open(os.path.join(state_dir, ".env"), "r", encoding="utf-8") as handle:
for line in handle:
if line.startswith("BROKER_INSTANCE_PEPPER="):
pepper = line.split("=", 1)[1].strip()
self.assertIsNotNone(pepper)
self.assertNotIn(pepper, output)
def test_v01_parity_flows_through_packaged_artifact(self):
state_dir = self._temp_dir()
child = self._boot_in_state(state_dir)
base = child.base_url
printed_pass = child.printed_admin_pass()
cookie = self._admin_login(base, printed_pass)
# seed allowlists first via the admin API (v0.1 pattern)
app_uuid = str(uuidlib.uuid4())
status, body, _ = self._request_json(
base, "POST", APPS_PATH,
{"uuid": app_uuid, "name": "proxmox-tool"}, headers=cookie,
)
self.assertEqual(201, status)
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
status, body, _ = self._request_json(
base, "POST", USERS_PATH,
{"user_id": "ceo", "email": "ceo@coreci.dev"}, headers=cookie,
)
self.assertEqual(201, status)
# agent register flow
status, body, _ = self._request_json(
base, "POST", REGISTER_PATH,
{"agent_id": "proxmox", "app_registration_uuid": app_uuid,
"user_id": "ceo", "tags": ["coreci"]},
)
self.assertEqual(201, status)
token = body["token"]
app_header = {"Authorization": "App %s %s" % (app_uuid, token)}
# state put/get roundtrip
state_document = {"phase": "v0.2", "boot": "packaged", "items": ["parity"]}
status, body, _ = self._request_json(
base, "PUT", AGENT_STATE_PATH, state_document, headers=app_header
)
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
status, body, _ = self._request_json(
base, "GET", AGENT_STATE_PATH, headers=app_header
)
self.assertEqual(200, status)
self.assertEqual(state_document, body)
# one secret read via the seeded FileCredsSource
status, body, _ = self._request_json(
base, "GET", SECRETS_PATH, headers=app_header
)
self.assertEqual(200, status)
self.assertEqual({"path", "data", "version", "updated_at"}, set(body))
self.assertEqual(PROXMOX_PATH, body["path"])
self.assertEqual(PROXMOX_SECRET, body["data"])
self.assertRegex(body["version"], r"^[0-9a-f]{8}$")
# audit lines exist for the driven flow
status, body, _ = self._request_json(
base, "GET", AUDIT_PATH + "?limit=1000", headers=cookie
)
self.assertEqual(200, status)
entries = body["entries"]
def select(action, result="ok"):
return [entry for entry in entries
if entry["action"] == action and entry["result"] == result]
logins = select("login")
self.assertEqual(1, len(logins))
self.assertEqual(LOGIN_PATH, logins[0]["path"])
add_apps = select("add_app")
self.assertEqual(1, len(add_apps))
self.assertEqual(APPS_PATH + "/" + app_uuid, add_apps[0]["path"])
add_users = select("add_user")
self.assertEqual(1, len(add_users))
self.assertEqual(USERS_PATH + "/ceo", add_users[0]["path"])
registers = select("register")
self.assertEqual(1, len(registers))
self.assertEqual(REGISTER_PATH, registers[0]["path"])
self._assert_tool_identity(registers[0], app_uuid)
ok_reads = select("read")
self.assertEqual(1, len(ok_reads))
self.assertEqual(PROXMOX_PATH, ok_reads[0]["path"])
self._assert_tool_identity(ok_reads[0], app_uuid)
child.stop()
self.assertEqual(1, child.full_output().count(ADMIN_PASS_PREFIX))
def test_second_boot_silent_and_state_persists(self):
state_dir = self._temp_dir()
first = self._boot_in_state(state_dir)
printed_pass = first.printed_admin_pass()
cookie = self._admin_login(first.base_url, printed_pass)
app_uuid = str(uuidlib.uuid4())
status, _, _ = self._request_json(
first.base_url, "POST", APPS_PATH,
{"uuid": app_uuid, "name": "persist-check"}, headers=cookie,
)
self.assertEqual(201, status)
# kill/drain protocol BEFORE any silence assertion (G-010)
first.stop()
first_output = first.full_output()
self.assertEqual(1, first_output.count(ADMIN_PASS_PREFIX))
self.assertIn(printed_pass, first_output)
second = self._boot_in_state(state_dir)
# the same pass still logs in on the second boot: .env persisted
cookie = self._admin_login(second.base_url, printed_pass)
status, body, _ = self._request_json(
second.base_url, "GET", APPS_PATH, headers=cookie
)
self.assertEqual(200, status)
listed = [row["uuid"] for row in body["registrations"]]
self.assertIn(app_uuid, listed)
second.stop()
second_output = second.full_output()
self.assertIn(LISTENING_PREFIX, second_output)
self.assertNotIn(ADMIN_PASS_PREFIX, second_output)
def test_user_env_ui_root_override_wins_over_materialized_default(self):
state_dir = self._temp_dir()
custom_root = os.path.join(state_dir, "custom_ui")
os.makedirs(custom_root)
marker = b"<h1>packaged-override-marker</h1>\n"
with open(os.path.join(custom_root, "login.html"), "wb") as handle:
handle.write(marker)
# a user .env (at the child's BROKER_ENV_PATH) wins over the shim default
with open(os.path.join(state_dir, ".env"), "w", encoding="utf-8") as handle:
handle.write("BROKER_UI_ROOT=%s\n" % custom_root)
child = self._boot_in_state(state_dir)
status, body, headers = self._request(child.base_url, "GET", UI_LOGIN_PATH)
self.assertEqual(200, status)
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
self.assertEqual(marker, body)
# the custom root is the whole UI root: no dashboard.html there
status, _, _ = self._request(child.base_url, "GET", UI_DASHBOARD_PATH)
self.assertEqual(404, status)
# the materialized static default still applies (not overridden)
status, served, _ = self._request(child.base_url, "GET", CLIENT_STATIC_PATH)
self.assertEqual(200, status)
self.assertEqual(self._read_repo_bytes(REPO_CLIENT_PATH), served)
# materialized defaults still exist on disk; they were simply not forced
self.assertTrue(os.path.isfile(os.path.join(
state_dir, ASSET_DIR_NAME, VERSION, "ui", "login.html"
)))
child.stop()
def test_default_anchoring_boot_writes_state_to_cwd(self):
# the owner's literal happy path: no state overrides except the bind
cwd_dir = self._temp_dir()
env = self._child_env(BROKER_BIND="127.0.0.1:0")
child = self._boot(env, cwd_dir)
status, body, _ = self._request_json(child.base_url, "GET", "/healthz")
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
status, body, headers = self._request(child.base_url, "GET", UI_LOGIN_PATH)
self.assertEqual(200, status)
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
# state lands in the child's CWD, at 0600
self._assert_state_files_0600_in(cwd_dir)
child.stop()
self.assertEqual(1, child.full_output().count(ADMIN_PASS_PREFIX))
if __name__ == "__main__":
unittest.main()
+308
View File
@@ -0,0 +1,308 @@
import ast
import base64
import contextlib
import datetime
import io
import json
import os
import sys
import tempfile
import time
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
import coreci_broker_client
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
GITEA_SECRET = {"url": "https://git.coreci.dev", "token": "git-tok"}
CLIENT_SOURCE_PATH = os.path.abspath(
os.path.join(os.path.dirname(__file__), os.pardir, "coreci_broker_client.py")
)
ALLOWED_CLIENT_IMPORTS = {"urllib.request", "urllib.parse", "json", "time"}
SECRETS_PROXMOX_PATH = "/api/broker/secrets/coreci/proxmox"
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
class ClientHygieneTests(unittest.TestCase):
def setUp(self):
with open(CLIENT_SOURCE_PATH, "r", encoding="utf-8") as handle:
self.source = handle.read()
self.tree = ast.parse(self.source)
def test_imports_are_stdlib_only(self):
modules = set()
for node in ast.walk(self.tree):
if isinstance(node, ast.Import):
modules.update(alias.name for alias in node.names)
elif isinstance(node, ast.ImportFrom):
modules.add(node.module or "")
self.assertIn("urllib.request", modules)
for module in modules:
self.assertIn(module, ALLOWED_CLIENT_IMPORTS, module)
def test_single_module_docstring_and_no_comments(self):
docstring = ast.get_docstring(self.tree)
self.assertIsNotNone(docstring)
self.assertNotIn("\n", docstring)
for node in ast.walk(self.tree):
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
self.assertIsNone(ast.get_docstring(node))
for number, line in enumerate(self.source.splitlines(), 1):
self.assertFalse(line.lstrip().startswith("#"), "comment on line %d" % number)
def test_broker_error_carries_status_and_message(self):
error = coreci_broker_client.BrokerError(403, "forbidden")
self.assertEqual(403, error.status)
self.assertEqual("forbidden", error.message)
self.assertEqual("forbidden", str(error))
class _ClientCase(unittest.TestCase):
extra_env = {}
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.creds_path = os.path.join(self.tmp, "creds.json")
with open(self.creds_path, "w", encoding="utf-8") as handle:
handle.write(json.dumps(
{"coreci/proxmox": PROXMOX_SECRET, "coreci/gitea": GITEA_SECRET}
))
env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
"BROKER_CREDS_SOURCE": "file",
"BROKER_CREDS_FILE": self.creds_path,
}
env.update(self.extra_env)
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(env)
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
self.app_uuid = str(uuidlib.uuid4())
self.user_id = "ceo"
self._seed()
def _raw_request(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(self.base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
raw = response.read().decode("utf-8")
return response.status, json.loads(raw) if raw else None
except urllib.error.HTTPError as error:
raw = error.read().decode("utf-8")
return error.code, json.loads(raw) if raw else None
def _seed(self):
status, _ = self._raw_request("POST", APPS_PATH, {"uuid": self.app_uuid}, self.admin)
self.assertEqual(201, status)
status, _ = self._raw_request("POST", USERS_PATH, {"user_id": self.user_id}, self.admin)
self.assertEqual(201, status)
def _client(self, agent_id="proxmox", tags=("coreci",)):
return coreci_broker_client.BrokerClient(
self.base, agent_id, self.app_uuid, self.user_id, list(tags)
)
def _count_requests(self, client):
calls = []
original = client._request
def counting(method, path, payload=None, headers=None):
calls.append((method, path))
return original(method, path, payload, headers)
def restore():
client._request = original
client._request = counting
return calls, restore
class ClientFlowTests(_ClientCase):
def test_ensure_registered_from_empty_state_populates_token_fields(self):
client = self._client()
state = {}
returned = client.ensure_registered(state)
self.assertIs(state, returned)
token = state["current_token"]
self.assertEqual(44, len(token))
self.assertTrue(token.isascii())
expires = datetime.datetime.fromisoformat(state["token_expires_at"])
self.assertGreater(expires, datetime.datetime.now(datetime.timezone.utc))
self.assertEqual("proxmox", state["agent_id"])
self.assertEqual(self.app_uuid, state["app_registration_uuid"])
self.assertEqual(self.user_id, state["user_id"])
self.assertEqual(["coreci"], state["tags"])
status, body = self._raw_request(
"GET", SECRETS_PROXMOX_PATH,
headers={"Authorization": "App %s %s" % (self.app_uuid, token)},
)
self.assertEqual(200, status)
self.assertEqual(PROXMOX_SECRET, body["data"])
def test_first_register_merges_pepper_into_state(self):
client = self._client()
state = client.ensure_registered({})
self.assertIn("pepper", state)
self.assertEqual(
self.broker.store.get_config("BROKER_INSTANCE_PEPPER"), state["pepper"]
)
def test_ensure_registered_with_live_token_makes_no_network_call(self):
client = self._client()
state = client.ensure_registered({})
snapshot = dict(state)
calls, restore = self._count_requests(client)
try:
returned = client.ensure_registered(state)
finally:
restore()
self.assertIs(state, returned)
self.assertEqual([], calls)
self.assertEqual(snapshot, state)
def test_reregister_with_valid_token_pepperless_response_does_not_crash(self):
client = self._client()
state = client.ensure_registered({})
pepper, token, expires = state["pepper"], state["current_token"], state["token_expires_at"]
status, body = self._raw_request(
"POST", coreci_broker_client.REGISTER_PATH,
{"agent_id": "proxmox", "app_registration_uuid": self.app_uuid,
"user_id": self.user_id, "tags": ["coreci"]},
{"Authorization": "App %s %s" % (self.app_uuid, token)},
)
self.assertEqual(200, status)
self.assertNotIn("pepper", body)
returned = client.register(state)
self.assertIs(state, returned)
self.assertEqual(pepper, state["pepper"])
self.assertEqual(token, state["current_token"])
self.assertEqual(expires, state["token_expires_at"])
def test_put_then_get_state_roundtrip_through_broker(self):
client = self._client()
state = client.ensure_registered({})
state["scratchpad"] = {"calls": 3, "notes": ["x"]}
self.assertTrue(client.put_state(state))
document = client.get_state(state)
self.assertEqual(state, document)
def test_get_state_returns_none_before_first_put(self):
client = self._client()
state = client.ensure_registered({})
self.assertIsNone(client.get_state(state))
def test_read_secret_returns_secret_document(self):
client = self._client()
state = client.ensure_registered({})
document = client.read_secret("coreci/proxmox", state)
self.assertEqual("coreci/proxmox", document["path"])
self.assertEqual(PROXMOX_SECRET, document["data"])
self.assertRegex(document["version"], r"^[0-9a-f]{8}$")
self.assertIn("updated_at", document)
def test_cross_path_read_secret_raises_403_broker_error(self):
client = self._client()
state = client.ensure_registered({})
with self.assertRaises(coreci_broker_client.BrokerError) as caught:
client.read_secret("coreci/gitea", state)
self.assertEqual(403, caught.exception.status)
self.assertTrue(caught.exception.message)
def test_missing_secret_read_raises_404_broker_error(self):
client = self._client("nonexistent")
state = client.ensure_registered({})
with self.assertRaises(coreci_broker_client.BrokerError) as caught:
client.read_secret("coreci/nonexistent", state)
self.assertEqual(404, caught.exception.status)
def test_read_secret_from_cold_state_registers_then_retries_once(self):
client = self._client()
state = {}
calls, restore = self._count_requests(client)
try:
document = client.read_secret("coreci/proxmox", state)
finally:
restore()
self.assertEqual(PROXMOX_SECRET, document["data"])
self.assertEqual(
[("GET", SECRETS_PROXMOX_PATH),
("POST", coreci_broker_client.REGISTER_PATH),
("GET", SECRETS_PROXMOX_PATH)],
calls,
)
self.assertIn("current_token", state)
def test_get_state_with_foreign_agent_token_is_403_identity_enforced(self):
intruder = self._client("proxmox")
other = self._client("gitea-agent")
other_state = other.ensure_registered({})
intruder.ensure_registered({})
with self.assertRaises(coreci_broker_client.BrokerError) as caught:
intruder.get_state({"current_token": other_state["current_token"]})
self.assertEqual(403, caught.exception.status)
class TtlAutoReregisterTests(_ClientCase):
extra_env = {"BROKER_TOKEN_TTL": "1"}
def test_expired_token_secret_read_auto_reregisters_and_retries(self):
client = self._client()
state = {}
first = client.read_secret("coreci/proxmox", state)
self.assertEqual(PROXMOX_SECRET, first["data"])
first_expires = datetime.datetime.fromisoformat(state["token_expires_at"])
time.sleep(1.2)
second = client.read_secret("coreci/proxmox", state)
self.assertEqual(PROXMOX_SECRET, second["data"])
second_expires = datetime.datetime.fromisoformat(state["token_expires_at"])
self.assertGreater(second_expires, first_expires)
self.assertIn("pepper", state)
class StaticDeliveryTests(_ClientCase):
def test_fetch_client_module_returns_own_source(self):
with open(CLIENT_SOURCE_PATH, "r", encoding="utf-8") as handle:
source = handle.read()
fetched = coreci_broker_client.fetch_client_module(self.base)
self.assertEqual(source, fetched)
self.assertIn("class BrokerClient", fetched)
def test_client_module_served_unauthenticated(self):
with open(CLIENT_SOURCE_PATH, "r", encoding="utf-8") as handle:
source = handle.read()
with urllib.request.urlopen(
self.base + "/static/coreci_broker_client.py", timeout=10
) as response:
self.assertEqual(200, response.status)
self.assertEqual(source, response.read().decode("utf-8"))
if __name__ == "__main__":
unittest.main()
+105
View File
@@ -0,0 +1,105 @@
import base64
import contextlib
import io
import os
import sys
import tempfile
import unittest
from unittest import mock
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
class LoadConfigTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.env_path = os.path.join(tmp.name, ".env")
def _boot(self, env):
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer):
config = coreci_broker.load_config(env)
return config, buffer.getvalue()
def _read_env_file(self):
values = {}
with open(self.env_path, "r", encoding="utf-8") as handle:
for raw in handle:
line = raw.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, value = line.partition("=")
values[key.strip()] = value.strip()
return values
def test_first_boot_writes_env_and_prints_pass_once(self):
config, stdout = self._boot({"BROKER_ENV_PATH": self.env_path})
self.assertTrue(os.path.isfile(self.env_path))
stored = self._read_env_file()
admin_pass = stored["BROKER_ADMIN_PASS"]
pepper = stored["BROKER_INSTANCE_PEPPER"]
self.assertGreaterEqual(len(admin_pass), 32)
self.assertEqual(32, len(base64.b64decode(pepper)))
self.assertEqual(admin_pass, config["BROKER_ADMIN_PASS"])
self.assertEqual(pepper, config["BROKER_INSTANCE_PEPPER"])
self.assertEqual(1, stdout.count(admin_pass))
self.assertNotIn(pepper, stdout)
self.assertEqual(0o600, os.stat(self.env_path).st_mode & 0o777)
def test_second_boot_prints_nothing(self):
first, first_stdout = self._boot({"BROKER_ENV_PATH": self.env_path})
second, second_stdout = self._boot({"BROKER_ENV_PATH": self.env_path})
self.assertEqual(1, first_stdout.count(first["BROKER_ADMIN_PASS"]))
self.assertEqual("", second_stdout)
self.assertEqual(first["BROKER_ADMIN_PASS"], second["BROKER_ADMIN_PASS"])
self.assertEqual(first["BROKER_INSTANCE_PEPPER"], second["BROKER_INSTANCE_PEPPER"])
def test_env_overrides_dotenv(self):
with open(self.env_path, "w", encoding="utf-8") as handle:
handle.write("BROKER_BIND=10.1.2.3:9999\n")
handle.write('BROKER_ADMIN_USER="ceo"\n')
handle.write("BROKER_ADMIN_PASS=oldpass\n")
handle.write("BROKER_TOKEN_TTL=100\n")
env = {
"BROKER_ENV_PATH": self.env_path,
"BROKER_ADMIN_PASS": "envpass",
"BROKER_TOKEN_TTL": "55",
"BROKER_DB_PATH": "/tmp/elsewhere/broker.db",
}
config, stdout = self._boot(env)
self.assertEqual("envpass", config["BROKER_ADMIN_PASS"])
self.assertEqual("55", config["BROKER_TOKEN_TTL"])
self.assertEqual("/tmp/elsewhere/broker.db", config["BROKER_DB_PATH"])
self.assertEqual("10.1.2.3:9999", config["BROKER_BIND"])
self.assertEqual("ceo", config["BROKER_ADMIN_USER"])
self.assertEqual("300", config["BROKER_SESSION_TTL"])
self.assertEqual("", stdout)
stored = self._read_env_file()
self.assertEqual("oldpass", stored["BROKER_ADMIN_PASS"])
self.assertIn("BROKER_INSTANCE_PEPPER", stored)
def test_defaults_applied(self):
config, _ = self._boot({"BROKER_ENV_PATH": self.env_path})
self.assertEqual(self.env_path, config["BROKER_ENV_PATH"])
self.assertEqual("127.0.0.1:8765", config["BROKER_BIND"])
self.assertEqual("admin", config["BROKER_ADMIN_USER"])
self.assertEqual("86400", config["BROKER_TOKEN_TTL"])
self.assertEqual("300", config["BROKER_SESSION_TTL"])
self.assertEqual("broker.db", config["BROKER_DB_PATH"])
self.assertEqual("broker.audit.log", config["BROKER_AUDIT_LOG_PATH"])
def test_reads_process_environ_by_default(self):
env = {"BROKER_ENV_PATH": self.env_path, "BROKER_BIND": "127.0.0.1:9000"}
with mock.patch.dict(os.environ, env, clear=True):
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer):
config = coreci_broker.load_config()
self.assertEqual("127.0.0.1:9000", config["BROKER_BIND"])
self.assertEqual(self.env_path, config["BROKER_ENV_PATH"])
if __name__ == "__main__":
unittest.main()
+312
View File
@@ -0,0 +1,312 @@
import contextlib
import hashlib
import io
import json
import os
import sys
import tempfile
import threading
import unittest
import urllib.parse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
API_KEY = "sk-test-key"
KB_NAME = "creds-coreci-coo"
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
PROXMOX_UPDATED = "2026-10-01T12:00:00Z"
def _standard_fixtures():
kbs = [{"id": "kb-1", "name": KB_NAME}]
files = {"kb-1": [{"name": "coreci/proxmox", "id": "f1", "updated_at": PROXMOX_UPDATED}]}
contents = {"f1": json.dumps(PROXMOX_SECRET)}
return kbs, files, contents
class _StubOwuiHandler(BaseHTTPRequestHandler):
kbs = None
files = None
contents = None
failures = None
hits = None
def log_message(self, format, *args):
pass
def _send(self, status, payload):
data = json.dumps(payload).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def do_GET(self):
parsed = urllib.parse.urlsplit(self.path)
path = parsed.path
self.hits.append((path, self.headers.get("Authorization")))
if path in self.failures:
self._send(self.failures[path], {"error": self.failures[path]})
elif path == "/api/v1/knowledge/":
page = max(1, int((urllib.parse.parse_qs(parsed.query).get("page") or ["1"])[0]))
self._send(200, {"items": self.kbs[(page - 1) * 30:page * 30], "total": len(self.kbs)})
elif path.startswith("/api/v1/knowledge/"):
kb_id = path[len("/api/v1/knowledge/"):]
kb = next((entry for entry in self.kbs if entry["id"] == kb_id), None)
if kb is None:
self._send(404, {"error": 404})
else:
self._send(200, dict(kb, files=self.files.get(kb_id, [])))
elif path.startswith("/api/v1/files/") and path.endswith("/data/content"):
file_id = path[len("/api/v1/files/"):-len("/data/content")]
if file_id in self.contents:
self._send(200, {"content": self.contents[file_id]})
else:
self._send(404, {"error": 404})
else:
self._send(404, {"error": 404})
class StubOwuiServer:
def __init__(self, kbs, files, contents):
self.kbs, self.files, self.contents, self.failures = kbs, files, contents, {}
self.hits = []
handler = type("BoundStubHandler", (_StubOwuiHandler,), {
"kbs": kbs, "files": files, "contents": contents,
"failures": self.failures, "hits": self.hits,
})
self.server = ThreadingHTTPServer(("127.0.0.1", 0), handler)
threading.Thread(target=self.server.serve_forever, daemon=True).start()
@property
def base_url(self):
return "http://127.0.0.1:%d" % self.server.server_address[1]
def hits_for(self, path):
return [hit for hit in self.hits if hit[0] == path]
def close(self):
self.server.shutdown()
self.server.server_close()
class FileCredsSourceTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.creds_path = os.path.join(tmp.name, "creds.json")
self.document = {
"coreci/proxmox": {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"},
"coreci/gitea": {"url": "https://git.coreci.dev", "token": "git-tok"},
}
self._write(self.document)
self.source = coreci_broker.FileCredsSource(self.creds_path)
def _write(self, document):
with open(self.creds_path, "w", encoding="utf-8") as handle:
json.dump(document, handle)
def test_fetches_both_paths_with_response_shape(self):
for path, secret in self.document.items():
result = self.source.get_secret(path)
self.assertEqual(path, result["path"])
self.assertEqual(secret, result["data"])
self.assertIsNone(result["updated_at"])
self.assertRegex(result["version"], r"^[0-9a-f]{8}$")
def test_version_is_short_sha256_of_serialized_data(self):
result = self.source.get_secret("coreci/proxmox")
serialized = json.dumps(result["data"], sort_keys=True, separators=(",", ":"))
expected = hashlib.sha256(serialized.encode("utf-8")).hexdigest()[:8]
self.assertEqual(expected, result["version"])
def test_equal_data_yields_equal_versions(self):
first = self.source.get_secret("coreci/proxmox")
second = self.source.get_secret("coreci/proxmox")
self.assertEqual(first["version"], second["version"])
def test_missing_path_is_not_found(self):
with self.assertRaises(coreci_broker._SecretNotFound):
self.source.get_secret("coreci/ghost")
def test_missing_file_means_all_paths_not_found(self):
os.remove(self.creds_path)
for path in self.document:
with self.assertRaises(coreci_broker._SecretNotFound):
self.source.get_secret(path)
def test_invalid_json_is_source_error(self):
with open(self.creds_path, "w", encoding="utf-8") as handle:
handle.write("{not-json")
with self.assertRaises(coreci_broker._CredsSourceError):
self.source.get_secret("coreci/proxmox")
def test_non_object_document_is_source_error(self):
self._write(["not", "a", "mapping"])
with self.assertRaises(coreci_broker._CredsSourceError):
self.source.get_secret("coreci/proxmox")
def test_fresh_read_sees_out_of_band_edits(self):
first = self.source.get_secret("coreci/proxmox")
self.document["coreci/proxmox"]["token"] = "rotated"
self._write(self.document)
second = self.source.get_secret("coreci/proxmox")
self.assertEqual("pve-tok", first["data"]["token"])
self.assertEqual("rotated", second["data"]["token"])
self.assertNotEqual(first["version"], second["version"])
class OwuiApiCredsSourceTests(unittest.TestCase):
def setUp(self):
kbs, files, contents = _standard_fixtures()
self.stub = StubOwuiServer(kbs, files, contents)
self.addCleanup(self.stub.close)
self.source = coreci_broker.OwuiApiCredsSource(self.stub.base_url, API_KEY)
def test_roundtrip_returns_response_shape(self):
result = self.source.get_secret("coreci/proxmox")
self.assertEqual("coreci/proxmox", result["path"])
self.assertEqual(PROXMOX_SECRET, result["data"])
self.assertEqual(PROXMOX_UPDATED, result["updated_at"])
self.assertRegex(result["version"], r"^[0-9a-f]{8}$")
def test_kb_and_file_map_cached_across_calls(self):
self.source.get_secret("coreci/proxmox")
self.source.get_secret("coreci/proxmox")
self.assertEqual(1, len(self.stub.hits_for("/api/v1/knowledge/")))
self.assertEqual(1, len(self.stub.hits_for("/api/v1/knowledge/kb-1")))
self.assertEqual(2, len(self.stub.hits_for("/api/v1/files/f1/data/content")))
def test_bearer_header_present_on_every_upstream_call(self):
self.source.get_secret("coreci/proxmox")
self.assertTrue(self.stub.hits)
for _, auth in self.stub.hits:
self.assertEqual("Bearer " + API_KEY, auth)
def test_refresh_on_miss_picks_up_newly_added_files(self):
self.source.get_secret("coreci/proxmox")
self.stub.files["kb-1"].append(
{"name": "coreci/gitea", "id": "f2", "updated_at": "2026-10-02T00:00:00Z"})
self.stub.contents["f2"] = json.dumps({"user": "ceo"})
result = self.source.get_secret("coreci/gitea")
self.assertEqual({"user": "ceo"}, result["data"])
self.assertEqual("2026-10-02T00:00:00Z", result["updated_at"])
self.assertEqual(2, len(self.stub.hits_for("/api/v1/knowledge/")))
def test_unknown_path_is_not_found(self):
with self.assertRaises(coreci_broker._SecretNotFound):
self.source.get_secret("coreci/ghost")
def test_missing_kb_is_not_found(self):
self.stub.kbs.clear()
with self.assertRaises(coreci_broker._SecretNotFound):
self.source.get_secret("coreci/proxmox")
def test_kb_list_404_is_not_found(self):
self.stub.failures["/api/v1/knowledge/"] = 404
with self.assertRaises(coreci_broker._SecretNotFound):
self.source.get_secret("coreci/proxmox")
def test_kb_detail_404_is_not_found(self):
self.stub.failures["/api/v1/knowledge/kb-1"] = 404
with self.assertRaises(coreci_broker._SecretNotFound):
self.source.get_secret("coreci/proxmox")
def test_file_content_404_is_not_found(self):
del self.stub.contents["f1"]
with self.assertRaises(coreci_broker._SecretNotFound):
self.source.get_secret("coreci/proxmox")
def test_upstream_500_is_source_error_not_crash(self):
self.stub.failures["/api/v1/files/f1/data/content"] = 500
with self.assertRaises(coreci_broker._CredsSourceError):
self.source.get_secret("coreci/proxmox")
def test_kb_list_500_is_source_error(self):
self.stub.failures["/api/v1/knowledge/"] = 500
with self.assertRaises(coreci_broker._CredsSourceError):
self.source.get_secret("coreci/proxmox")
def test_unavailable_upstream_maps_to_distinct_source_error(self):
source = coreci_broker.OwuiApiCredsSource("http://127.0.0.1:9", API_KEY)
with self.assertRaises(coreci_broker._CredsSourceError) as caught:
source.get_secret("coreci/proxmox")
self.assertIn("unavailable", str(caught.exception))
def test_invalid_content_json_is_source_error(self):
self.stub.contents["f1"] = "not-json"
with self.assertRaises(coreci_broker._CredsSourceError):
self.source.get_secret("coreci/proxmox")
def test_pagination_finds_kb_beyond_first_page(self):
self.stub.kbs[:] = [{"id": "kb-%d" % i, "name": "kb-%d" % i} for i in range(30)]
self.stub.kbs.append({"id": "kb-creds", "name": KB_NAME})
self.stub.files["kb-creds"] = [
{"name": "coreci/proxmox", "id": "f1", "updated_at": PROXMOX_UPDATED}]
result = self.source.get_secret("coreci/proxmox")
self.assertEqual(PROXMOX_SECRET, result["data"])
self.assertEqual(2, len(self.stub.hits_for("/api/v1/knowledge/")))
class CredsSourceFactoryTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
def _boot(self, extra):
env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
}
env.update(extra)
with contextlib.redirect_stdout(io.StringIO()):
broker = coreci_broker.boot(env)
self.addCleanup(broker.audit.close)
self.addCleanup(broker.store.close)
return broker
def test_default_source_is_file_with_default_path(self):
broker = self._boot({})
self.assertIsInstance(broker.creds_source, coreci_broker.FileCredsSource)
self.assertEqual("data/creds.json", broker.creds_source.path)
def test_explicit_file_source_with_custom_path(self):
creds_path = os.path.join(self.tmp, "custom-creds.json")
with open(creds_path, "w", encoding="utf-8") as handle:
json.dump({"coreci/proxmox": PROXMOX_SECRET}, handle)
broker = self._boot({
"BROKER_CREDS_SOURCE": "file",
"BROKER_CREDS_FILE": creds_path,
})
self.assertIsInstance(broker.creds_source, coreci_broker.FileCredsSource)
self.assertEqual(creds_path, broker.creds_source.path)
self.assertEqual(PROXMOX_SECRET, broker.creds_source.get_secret("coreci/proxmox")["data"])
def test_owui_source_selected_by_config(self):
kbs, files, contents = _standard_fixtures()
stub = StubOwuiServer(kbs, files, contents)
self.addCleanup(stub.close)
broker = self._boot({
"BROKER_CREDS_SOURCE": "owui",
"BROKER_OWUI_BASE_URL": stub.base_url,
"BROKER_OWUI_API_KEY": API_KEY,
})
source = broker.creds_source
self.assertIsInstance(source, coreci_broker.OwuiApiCredsSource)
self.assertEqual(stub.base_url, source.base_url)
self.assertEqual(API_KEY, source.api_key)
self.assertEqual(PROXMOX_SECRET, source.get_secret("coreci/proxmox")["data"])
def test_unknown_source_value_raises_at_boot(self):
with self.assertRaises(ValueError):
self._boot({"BROKER_CREDS_SOURCE": "banana"})
if __name__ == "__main__":
unittest.main()
+264
View File
@@ -0,0 +1,264 @@
"""Dockerfile consistency tests (G-016 — equivalence made testable).
The Dockerfile documents the docker-user path; the daemonless client
(``scripts/publish_image.py``) is the pipeline's source of truth for
the published image. Byte-identity is NOT required — these tests pin the
equivalence contract: same staged sources (member-set parity — the
docker-staged layout must equal build_cli's staging EXACTLY, or the
packaged shim materializes an empty UI dir and /ui/* 404s in
docker-built images; P3-03/CWE-636), same zipapp invocation, same
entrypoint/working-dir, same pinned base, and a hygienic build context.
"""
import json
import os
import shutil
import subprocess
import sys
import tempfile
import unittest
import zipfile
_SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.dirname(
os.path.abspath(__file__))), "scripts")
if _SCRIPTS_DIR not in sys.path:
sys.path.insert(0, _SCRIPTS_DIR)
import build_cli # noqa: E402 (scripts/ dir on sys.path)
import publish_image # noqa: E402 (scripts/ dir on sys.path)
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
APP_SRC = "/app/src/"
DOCKERFILE_IGNORE_PATTERNS = (
".git", ".env", ".env.*", "data/", "*.db", "*.log", "dist/",
"__pycache__/", "tests/", ".ciagent/",
"**/__pycache__/", "**/*.pyc",
)
def read(name):
with open(os.path.join(REPO, name), "r", encoding="utf-8") as handle:
return handle.read()
def _tree_members(prefix, root):
"""Zipapp member names for the tree at ``root`` staged under
``prefix`` (``""`` for the archive root; the prefix itself is
included — zipapp writes directory entries)."""
members = {prefix} if prefix else set()
for current, dirs, files in os.walk(root):
relative = os.path.relpath(current, root)
base = prefix + ("" if relative == "." else relative + "/")
for name in dirs:
members.add(base + name + "/")
for name in files:
members.add(base + name)
return members
def _build_cli_staged_members():
"""The member set build_cli stages, derived via its constants:
ROOT_SOURCES at the root, UI under the coreci_broker_ui/ prefix,
cli/__main__.py staged as __main__.py, _cli_version.py generated."""
members = set(build_cli.ROOT_SOURCES)
members.add(build_cli.STAGED_SHIM)
members.add(build_cli.VERSION_MODULE)
members |= _tree_members(
build_cli.UI_SOURCE_DIR + "/",
os.path.join(REPO, build_cli.UI_SOURCE_DIR))
return members
def _run_zipapp(staged_dir, artifact):
"""The Dockerfile's packing invocation (zipapp writes dir entries;
the member set is interpreter-stable across 3.x)."""
completed = subprocess.run(
[sys.executable, "-m", "zipapp", staged_dir,
"-o", artifact,
"--python", "/usr/bin/env python3", "--compress"],
capture_output=True, text=True)
if completed.returncode != 0:
raise AssertionError(
"zipapp failed ({}): {}".format(
completed.returncode,
(completed.stderr or completed.stdout).strip()))
class DockerfileConsistencyTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.dockerfile = read("Dockerfile")
cls.dockerignore = read(".dockerignore")
# Join backslash continuations into logical stanzas; skip comments.
stanzas = []
for line in cls.dockerfile.splitlines():
stripped = line.strip()
if not stripped or stripped.startswith("#"):
continue
if stanzas and stanzas[-1].endswith("\\"):
stanzas[-1] = stanzas[-1].rstrip("\\").rstrip() + " " + stripped
continue
stanzas.append(stripped)
cls.stanzas = stanzas
cls.copy_lines = [s for s in stanzas if s.startswith("COPY ")]
cls.zipapp_lines = [s for s in stanzas
if "zipapp" in s and s.startswith("RUN ")]
cls.version_lines = [s for s in stanzas
if "_cli_version.py" in s
and s.startswith("RUN ")]
def _copy_specs(self):
"""[(sources, dest)] per COPY stanza, dest verified to stage
under /app/src/."""
specs = []
for line in self.copy_lines:
tokens = line.split()[1:]
sources, dest = tokens[:-1], tokens[-1]
self.assertTrue(dest.startswith(APP_SRC),
"COPY destination {} must stage under /app/src/"
.format(dest))
specs.append((sources, dest))
return specs
def _docker_staged_members(self):
"""The member set the COPY stanzas stage under /app/src/ (Docker
semantics: directory CONTENTS land flattened INSIDE the
destination dir; a dest without trailing slash is a file rename),
plus the RUN-generated _cli_version.py."""
members = set()
for sources, dest in self._copy_specs():
if dest.endswith("/"):
prefix = dest[len(APP_SRC):]
for source in sources:
path = os.path.join(REPO, source.rstrip("/"))
self.assertTrue(os.path.exists(path),
"COPY source missing in the repo: {}"
.format(source))
if os.path.isdir(path):
members |= _tree_members(prefix, path)
else:
members.add(prefix + os.path.basename(source))
else:
# file-to-file rename (mirrors build_cli's STAGED_SHIM
# staging — zipapp requires __main__.py at the root)
self.assertEqual(len(sources), 1,
"file-rename COPY stages exactly one "
"source: {}".format(dest))
members.add(dest[len(APP_SRC):])
members.add(build_cli.VERSION_MODULE) # the RUN printf stanza
return members
def test_from_pins_base(self):
from_lines = [line for line in self.dockerfile.splitlines()
if line.startswith("FROM ")]
self.assertEqual(len(from_lines), 1)
self.assertEqual(from_lines[0].split()[1], "python:3.11-slim")
def test_copy_staged_member_set_matches_build_cli_staging(self):
"""The COPY stanzas stage EXACTLY build_cli's member set — set
parity, no extras, no missing (a flattened UI copy or a missing
root __main__.py breaks the packaged boot; P3-03/CWE-636)."""
self.assertEqual(self._docker_staged_members(),
_build_cli_staged_members(),
"docker-staged member set == build_cli-staged "
"member set")
for line in self.copy_lines:
self.assertNotIn("dist/", line,
"no pre-built artifact in the build context")
def test_docker_layout_simulation_member_set_parity(self):
"""Docker-layout simulation gate (P3-03/CWE-636): apply the
Dockerfile COPY semantics to the real repo sources, add the
generated _cli_version.py, zipapp-pack, and the archive's member
set must EQUAL the build_cli-built artifact's member set."""
with tempfile.TemporaryDirectory(prefix="docker-layout-") as tmp:
staged = os.path.join(tmp, "src")
os.makedirs(staged)
for sources, dest in self._copy_specs():
if dest.endswith("/"):
target = os.path.join(staged, dest[len(APP_SRC):])
os.makedirs(target, exist_ok=True)
for source in sources:
path = os.path.join(REPO, source.rstrip("/"))
if os.path.isdir(path):
# directory CONTENTS land inside the dest dir
shutil.copytree(path, target,
dirs_exist_ok=True)
else:
shutil.copy2(
path,
os.path.join(target,
os.path.basename(source)))
else:
self.assertEqual(len(sources), 1)
shutil.copy2(os.path.join(REPO, sources[0]),
os.path.join(staged, dest[len(APP_SRC):]))
with open(os.path.join(staged, build_cli.VERSION_MODULE), "w",
encoding="utf-8") as handle:
handle.write('VERSION = "{}"\n'.format("v0.2-layout-sim"))
docker_artifact = os.path.join(tmp, "docker-layout.pyz")
_run_zipapp(staged, docker_artifact)
with zipfile.ZipFile(docker_artifact) as archive:
docker_members = set(archive.namelist())
with tempfile.TemporaryDirectory(prefix="build-cli-ref-") as dist:
reference = build_cli.build("v0.2-layout-sim", repo=REPO,
dist_dir=dist)
with zipfile.ZipFile(reference) as archive:
build_members = set(archive.namelist())
self.assertEqual(docker_members, build_members,
"docker-layout archive members == build_cli "
"artifact members")
def test_zipapp_invocation_flags_match_build_script(self):
"""Same --python interpreter and --compress flag as build_cli."""
self.assertEqual(len(self.zipapp_lines), 1)
stanza = self.zipapp_lines[0]
self.assertIn('--python "/usr/bin/env python3"', stanza)
self.assertIn("--compress", stanza)
self.assertIn("-m zipapp", stanza)
self.assertIn("/app/src", stanza)
self.assertIn("/app/coreci-broker.pyz", stanza)
def test_version_module_format_parity(self):
"""ARG VERSION feeds the same VERSION = "<value>" format."""
self.assertEqual(len(self.version_lines), 1)
self.assertIn('VERSION = "%s"', self.version_lines[0])
self.assertEqual(build_cli.VERSION_MODULE, "_cli_version.py")
self.assertTrue(any(s.startswith("ARG VERSION=") for s in self.stanzas),
"ARG VERSION provides the version")
def test_entrypoint_and_workdir_match_published_config(self):
"""ENTRYPOINT/WORKDIR equal the daemonless client's pushed config."""
base_config = json.loads('{"config": {}, "rootfs": {}}')
blob = json.loads(publish_image.build_config_blob(
base_config, "sha256:" + "0" * 64, "v-parity"))
entrypoint = blob["config"]["Entrypoint"]
working_dir = blob["config"]["WorkingDir"]
docker_entrypoint = [line for line in self.dockerfile.splitlines()
if line.startswith("ENTRYPOINT ")]
docker_workdir = [line for line in self.dockerfile.splitlines()
if line.startswith("WORKDIR ")]
self.assertEqual(len(docker_entrypoint), 1)
self.assertEqual(json.loads(
docker_entrypoint[0].split(" ", 1)[1]), entrypoint)
self.assertEqual(len(docker_workdir), 1)
self.assertEqual(docker_workdir[0].split(" ", 1)[1], working_dir)
self.assertEqual(working_dir, "/state")
def test_dockerignore_hygiene(self):
entries = {line.strip() for line in self.dockerignore.splitlines()
if line.strip() and not line.strip().startswith("#")}
for pattern in DOCKERFILE_IGNORE_PATTERNS:
self.assertIn(pattern, entries,
".dockerignore must exclude {}".format(pattern))
def test_no_absolute_source_copies(self):
for line in self.copy_lines:
for token in line.split()[1:-1]:
self.assertFalse(token.startswith("/"),
"build context sources are repo-relative")
if __name__ == "__main__":
unittest.main()
+162
View File
@@ -0,0 +1,162 @@
import contextlib
import io
import json
import os
import sqlite3
import sys
import tempfile
import threading
import unittest
import urllib.error
import urllib.request
from unittest import mock
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
THREAD_COUNT = 8
WRITES_PER_THREAD = 16
MAX_BODY_BYTES = 1024 * 1024
class HardeningTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
}
def _boot_and_serve(self):
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer):
broker = coreci_broker.boot(dict(self.env))
server = coreci_broker.serve("127.0.0.1", 0, broker)
self.addCleanup(broker.audit.close)
self.addCleanup(broker.store.close)
self.addCleanup(server.server_close)
self.addCleanup(server.shutdown)
return broker, server
def test_oversized_declared_content_length_post_rejected(self):
broker, server = self._boot_and_serve()
port = server.server_address[1]
request = urllib.request.Request(
"http://127.0.0.1:%d/healthz" % port,
data=b"",
headers={"Content-Length": str(MAX_BODY_BYTES + 1)},
method="POST",
)
with self.assertRaises(urllib.error.HTTPError) as raised:
urllib.request.urlopen(request, timeout=5)
error = raised.exception
self.assertEqual(400, error.code)
payload = json.loads(error.read().decode("utf-8"))
self.assertEqual({"error": 400, "message": "body too large"}, payload)
def test_handler_timeout_attribute_is_30(self):
self.assertEqual(30, coreci_broker.BrokerRequestHandler.timeout)
broker, server = self._boot_and_serve()
self.assertEqual(30, server.RequestHandlerClass.timeout)
def test_concurrent_store_writes_from_eight_threads(self):
db_path = os.path.join(self.tmp, "concurrent.db")
store = coreci_broker.Store(db_path)
self.addCleanup(store.close)
errors = []
barrier = threading.Barrier(THREAD_COUNT)
def worker(index):
try:
barrier.wait(timeout=5)
for slot in range(WRITES_PER_THREAD):
store.set_config(
"key-%d-%d" % (index, slot), "value-%d-%d" % (index, slot)
)
store.get_config("key-%d-0" % index)
except BaseException as error:
errors.append(error)
threads = [
threading.Thread(target=worker, args=(index,))
for index in range(THREAD_COUNT)
]
for thread in threads:
thread.start()
for thread in threads:
thread.join(timeout=30)
self.assertEqual([], errors)
for index in range(THREAD_COUNT):
for slot in range(WRITES_PER_THREAD):
key = "key-%d-%d" % (index, slot)
expected = "value-%d-%d" % (index, slot)
self.assertEqual(expected, store.get_config(key))
conn = sqlite3.connect(db_path)
self.addCleanup(conn.close)
count = conn.execute("SELECT COUNT(*) FROM broker_config").fetchone()[0]
self.assertEqual(THREAD_COUNT * WRITES_PER_THREAD, count)
def test_first_boot_env_write_path_still_works(self):
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer):
config = coreci_broker.load_config(dict(self.env))
env_path = self.env["BROKER_ENV_PATH"]
self.assertTrue(os.path.isfile(env_path))
self.assertEqual(0o600, os.stat(env_path).st_mode & 0o777)
stored = {}
with open(env_path, "r", encoding="utf-8") as handle:
for raw in handle:
line = raw.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, value = line.partition("=")
stored[key.strip()] = value.strip()
self.assertEqual(config["BROKER_ADMIN_PASS"], stored["BROKER_ADMIN_PASS"])
self.assertEqual(
config["BROKER_INSTANCE_PEPPER"], stored["BROKER_INSTANCE_PEPPER"]
)
self.assertEqual(1, buffer.getvalue().count(config["BROKER_ADMIN_PASS"]))
second = io.StringIO()
with contextlib.redirect_stdout(second):
coreci_broker.load_config(dict(self.env))
self.assertEqual("", second.getvalue())
def test_append_env_entries_write_error_propagates_with_single_close(self):
env_path = os.path.join(self.tmp, ".env")
with open(env_path, "w", encoding="utf-8") as handle:
handle.write("seed=1\n")
closes = []
class FailingHandle:
def __init__(self, fd):
self.fd = fd
def write(self, data):
raise OSError("simulated write failure")
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
os.close(self.fd)
closes.append(self.fd)
return False
def fake_fdopen(fd, *args, **kwargs):
return FailingHandle(fd)
with mock.patch.object(os, "fdopen", fake_fdopen):
with self.assertRaises(OSError) as raised:
coreci_broker._append_env_entries(env_path, [("NEW", "value")])
self.assertEqual(1, len(closes))
self.assertIn("simulated write failure", str(raised.exception))
with open(env_path, "r", encoding="utf-8") as handle:
self.assertEqual("seed=1\n", handle.read())
if __name__ == "__main__":
unittest.main()
+207
View File
@@ -0,0 +1,207 @@
import base64
import hashlib
import os
import sys
import unittest
from unittest import mock
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
SCRYPT_N = 2 ** 14
SCRYPT_R = 8
SCRYPT_P = 1
BCRYPT_SHAPED = "$2b$12$K7L9O1P3QRSTUVWXYZabcdefghijk012345678901234567890123456789"
def _scrypt_stored(token, salt):
digest = hashlib.scrypt(
token.encode("utf-8"), salt=salt, n=SCRYPT_N, r=SCRYPT_R, p=SCRYPT_P
)
return "{}${}${}".format(
"scrypt",
base64.urlsafe_b64encode(salt).decode("ascii"),
base64.urlsafe_b64encode(digest).decode("ascii"),
)
class _FakeBcryptModule:
def __init__(self):
self.gensalt_calls = []
self.hashpw_calls = []
self.checkpw_calls = []
def gensalt(self):
self.gensalt_calls.append(True)
return b"$2b$12$fakesaltfakesalt"
def hashpw(self, password, salt):
self.hashpw_calls.append((password, salt))
return b"$2b$12$storedhashvalue"
def checkpw(self, password, hashed):
self.checkpw_calls.append((password, hashed))
return password == b"mock-token"
class HashFormatTests(unittest.TestCase):
def test_bcrypt_is_not_importable_on_this_host(self):
self.assertIsNone(coreci_broker.bcrypt)
with self.assertRaises(ImportError):
import bcrypt
def test_hash_output_starts_with_scrypt_prefix(self):
stored = coreci_broker.hash_token("format-token")
self.assertTrue(stored.startswith("scrypt$"))
def test_hash_output_carries_salt_and_digest_b64url_fields(self):
stored = coreci_broker.hash_token("format-token")
fields = stored.split("$")
self.assertEqual(3, len(fields))
self.assertEqual("scrypt", fields[0])
salt = base64.urlsafe_b64decode(fields[1].encode("ascii"))
digest = base64.urlsafe_b64decode(fields[2].encode("ascii"))
self.assertEqual(16, len(salt))
self.assertEqual(64, len(digest))
class RoundtripTests(unittest.TestCase):
def test_hash_then_verify_roundtrip(self):
stored = coreci_broker.hash_token("roundtrip-token")
self.assertTrue(coreci_broker.verify_token("roundtrip-token", stored))
def test_wrong_token_fails(self):
stored = coreci_broker.hash_token("roundtrip-token")
self.assertFalse(coreci_broker.verify_token("wrong-token", stored))
def test_two_hashes_of_same_token_differ(self):
first = coreci_broker.hash_token("same-token")
second = coreci_broker.hash_token("same-token")
self.assertNotEqual(first, second)
self.assertTrue(coreci_broker.verify_token("same-token", first))
self.assertTrue(coreci_broker.verify_token("same-token", second))
class ScryptParseTests(unittest.TestCase):
def test_hand_built_stored_value_verifies(self):
stored = _scrypt_stored("parse-token", bytes(range(16)))
self.assertTrue(stored.startswith("scrypt$"))
self.assertTrue(coreci_broker.verify_token("parse-token", stored))
def test_hand_built_stored_value_rejects_other_token(self):
stored = _scrypt_stored("parse-token", bytes(range(16)))
self.assertFalse(coreci_broker.verify_token("parse-token-x", stored))
def test_tampered_digest_field_fails(self):
salt = bytes(range(16))
digest = hashlib.scrypt(
b"parse-token", salt=salt, n=SCRYPT_N, r=SCRYPT_R, p=SCRYPT_P
)
tampered = digest[:-1] + bytes([digest[-1] ^ 0x01])
stored = "{}${}${}".format(
"scrypt",
base64.urlsafe_b64encode(salt).decode("ascii"),
base64.urlsafe_b64encode(tampered).decode("ascii"),
)
self.assertFalse(coreci_broker.verify_token("parse-token", stored))
def test_malformed_scrypt_values_fail_cleanly(self):
for stored in (
"scrypt$",
"scrypt$abc",
"scrypt$abc$def$ghi",
"scrypt$$",
"scrypt$!@#$%",
):
with self.subTest(stored=stored):
self.assertFalse(coreci_broker.verify_token("x", stored))
class UnknownPrefixTests(unittest.TestCase):
def test_unknown_prefixes_return_false_without_raising(self):
for stored in (
"",
"plaintext",
"$2c$12$abc",
"$2x$12$abc",
"argon2$abc$def",
"SCRYPT$abc$def",
"scrypt2$abc$def",
"0123456789",
):
with self.subTest(stored=stored):
self.assertFalse(coreci_broker.verify_token("x", stored))
def test_non_string_stored_returns_false_cleanly(self):
self.assertFalse(coreci_broker.verify_token("x", None))
self.assertFalse(coreci_broker.verify_token("x", 5))
self.assertFalse(coreci_broker.verify_token("x", ["scrypt$abc$def"]))
class BcryptDispatchContractTests(unittest.TestCase):
def test_bcrypt_prefix_returns_false_cleanly_with_bcrypt_absent(self):
self.assertIsNone(coreci_broker.bcrypt)
with self.assertRaises(ImportError):
import bcrypt
self.assertFalse(coreci_broker.verify_token("x", BCRYPT_SHAPED))
def test_all_bcrypt_prefixes_return_false_cleanly_with_bcrypt_absent(self):
self.assertIsNone(coreci_broker.bcrypt)
for prefix in ("$2a$", "$2b$", "$2y$"):
with self.subTest(prefix=prefix):
self.assertFalse(coreci_broker.verify_token("x", prefix + "12$storedhashvalue"))
class BcryptBranchTests(unittest.TestCase):
def test_hash_token_uses_bcrypt_branch_when_importable(self):
fake = _FakeBcryptModule()
with mock.patch.object(coreci_broker, "bcrypt", fake):
stored = coreci_broker.hash_token("mock-token")
self.assertEqual("$2b$12$storedhashvalue", stored)
self.assertEqual([True], fake.gensalt_calls)
self.assertEqual([(b"mock-token", b"$2b$12$fakesaltfakesalt")], fake.hashpw_calls)
def test_verify_dispatches_bcrypt_prefixes_to_bcrypt_branch(self):
fake = _FakeBcryptModule()
expected = [
(b"mock-token", (prefix + "12$storedhashvalue").encode("ascii"))
for prefix in ("$2a$", "$2b$", "$2y$")
]
with mock.patch.object(coreci_broker, "bcrypt", fake):
for password, hashed in expected:
with self.subTest(hashed=hashed):
self.assertTrue(coreci_broker.verify_token(password.decode("utf-8"), hashed.decode("ascii")))
self.assertEqual(expected, fake.checkpw_calls)
def test_bcrypt_checkpw_rejection_returns_false(self):
fake = _FakeBcryptModule()
with mock.patch.object(coreci_broker, "bcrypt", fake):
self.assertFalse(coreci_broker.verify_token("other-token", "$2b$12$storedhashvalue"))
self.assertEqual([(b"other-token", b"$2b$12$storedhashvalue")], fake.checkpw_calls)
def test_bcrypt_checkpw_error_returns_false_cleanly(self):
fake = _FakeBcryptModule()
def broken(password, hashed):
raise ValueError("invalid salt")
fake.checkpw = broken
with mock.patch.object(coreci_broker, "bcrypt", fake):
self.assertFalse(coreci_broker.verify_token("mock-token", "$2b$garbage"))
def test_scrypt_and_unknown_stored_never_dispatch_to_bcrypt(self):
fake = _FakeBcryptModule()
stored = coreci_broker.hash_token("mock-token")
self.assertTrue(stored.startswith("scrypt$"))
with mock.patch.object(coreci_broker, "bcrypt", fake):
self.assertTrue(coreci_broker.verify_token("mock-token", stored))
self.assertFalse(coreci_broker.verify_token("other", stored))
self.assertFalse(coreci_broker.verify_token("mock-token", "unknown$prefix"))
self.assertEqual([], fake.checkpw_calls)
self.assertEqual([], fake.hashpw_calls)
if __name__ == "__main__":
unittest.main()
+148
View File
@@ -0,0 +1,148 @@
import contextlib
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.request
from http.server import ThreadingHTTPServer
from unittest import mock
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
class MountsTests(unittest.TestCase):
def setUp(self):
coreci_broker._reset_server_guard()
tmp = tempfile.TemporaryDirectory()
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_BIND": "127.0.0.1:0",
}
self.addCleanup(tmp.cleanup)
self.addCleanup(coreci_broker._reset_server_guard)
def _patched_environ(self):
return mock.patch.dict(os.environ, self.env, clear=True)
def _read_env_values(self):
values = {}
with open(self.env["BROKER_ENV_PATH"], "r", encoding="utf-8") as handle:
for raw in handle:
line = raw.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, value = line.partition("=")
values[key.strip()] = value.strip()
return values
def _healthz(self, port):
url = "http://127.0.0.1:%d/healthz" % port
with urllib.request.urlopen(url, timeout=5) as response:
return response.status, json.loads(response.read().decode("utf-8"))
def test_owui_docstring_frontmatter(self):
doc = coreci_broker.__doc__ or ""
self.assertTrue(doc.startswith("title: CoreCI Broker"))
for line in (
"author: coreci",
"description: ",
"version: 0.1.0",
"requirements: []",
"licence: MIT",
):
self.assertIn(line, doc)
def test_source_never_references_dunder_file(self):
with open(coreci_broker.__file__, "r", encoding="utf-8") as handle:
source = handle.read()
self.assertNotIn("__file__", source)
def test_tools_double_instantiation_starts_exactly_one_server(self):
serve_calls = []
real_serve = coreci_broker.serve
def counting_serve(host, port, broker):
serve_calls.append((host, port))
return real_serve(host, port, broker)
with self._patched_environ():
with mock.patch.object(coreci_broker, "serve", counting_serve):
with contextlib.redirect_stdout(io.StringIO()):
first = coreci_broker.Tools()
port_after_first = coreci_broker._started_server.server_address[1]
second = coreci_broker.Tools()
guard = coreci_broker._started_server
self.assertEqual(1, len(serve_calls))
self.assertIsInstance(guard, ThreadingHTTPServer)
self.assertIs(coreci_broker._started_server, first._server)
self.assertIs(first._server, second._server)
self.assertEqual(port_after_first, guard.server_address[1])
self.assertNotEqual(0, guard.server_address[1])
self.assertIsInstance(first.valves, coreci_broker.Tools.Valves)
self.assertIsInstance(second.valves, coreci_broker.Tools.Valves)
status, payload = self._healthz(port_after_first)
self.assertEqual(200, status)
self.assertEqual({"ok": True}, payload)
def test_tools_status_returns_non_empty_string(self):
with self._patched_environ():
with contextlib.redirect_stdout(io.StringIO()):
tools = coreci_broker.Tools()
status = tools.status()
port = tools._server.server_address[1]
self.assertIsInstance(status, str)
self.assertTrue(status.strip())
self.assertIn(str(port), status)
def test_first_boot_print_discipline_through_tools_boot(self):
with self._patched_environ():
first = io.StringIO()
with contextlib.redirect_stdout(first):
coreci_broker.Tools()
second = io.StringIO()
with contextlib.redirect_stdout(second):
coreci_broker.Tools()
stored = self._read_env_values()
admin_pass = stored["BROKER_ADMIN_PASS"]
pepper = stored["BROKER_INSTANCE_PEPPER"]
self.assertEqual(1, first.getvalue().count(admin_pass))
self.assertNotIn(pepper, first.getvalue())
self.assertEqual("", second.getvalue())
def test_run_standalone_boots_serves_and_shuts_down_cleanly(self):
captured = {}
real_serve = coreci_broker.serve
def capturing_serve(host, port, broker):
server = real_serve(host, port, broker)
captured["server"] = server
return server
def fake_sleep(seconds):
server = captured["server"]
status, payload = self._healthz(server.server_address[1])
captured["healthz"] = (status, payload)
raise KeyboardInterrupt
stdout = io.StringIO()
with mock.patch.object(coreci_broker, "serve", capturing_serve):
with mock.patch.object(coreci_broker.time, "sleep", fake_sleep):
with contextlib.redirect_stdout(stdout):
coreci_broker.run_standalone(dict(self.env))
out = stdout.getvalue()
self.assertEqual((200, {"ok": True}), captured["healthz"])
self.assertEqual(1, out.count("BROKER_ADMIN_PASS="))
self.assertIn("listening on http://127.0.0.1:", out)
self.assertIn("shut down", out)
self.assertEqual(-1, captured["server"].socket.fileno())
if __name__ == "__main__":
unittest.main()
+249
View File
@@ -0,0 +1,249 @@
import base64
import contextlib
import datetime
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
REGISTER_PATH = "/api/broker/agents/register"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
PROTECTED_PATH = "/test/p3-remediation"
AUTH_401_BODY = {"error": 401, "message": "invalid app credentials"}
CAP_BODY = {"error": 400, "message": "too many agents for this app"}
AGENT_ID_BODY = {"error": 400, "message": "invalid agent_id"}
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
def _protected_echo(request):
request.send_json(200, dict(request.app_auth))
class P3RemediationTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
}
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(dict(self.env))
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
self.app_uuid = str(uuidlib.uuid4())
self.user_id = "ceo"
self._seed(self.app_uuid, self.user_id)
self.broker.router.add(
"GET", PROTECTED_PATH, coreci_broker._require_app(_protected_echo)
)
def _request(self, method, path, payload=None, headers=None, base=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(
(base or self.base) + path, data=data, method=method
)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
return response.status, json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
return error.code, json.loads(error.read().decode("utf-8"))
def _seed(self, app_uuid, user_id, base=None):
status, _ = self._request("POST", APPS_PATH, {"uuid": app_uuid}, self.admin, base)
self.assertEqual(201, status)
status, _ = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin, base)
self.assertEqual(201, status)
def _register(self, agent_id="agent-1", tags=("coreci",), app_uuid=None,
user_id=None, base=None):
payload = {
"agent_id": agent_id,
"app_registration_uuid": app_uuid or self.app_uuid,
"user_id": user_id or self.user_id,
"tags": list(tags),
}
return self._request("POST", REGISTER_PATH, payload, base=base)
def _auth(self, token, app_uuid=None):
header = "App %s %s" % (app_uuid or self.app_uuid, token)
return self._request("GET", PROTECTED_PATH, headers={"Authorization": header})
def _audit(self, action=None, result=None):
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
entries = [json.loads(raw) for raw in handle if raw.strip()]
return [
entry for entry in entries
if (action is None or entry["action"] == action)
and (result is None or entry["result"] == result)
]
def _future(self, seconds=3600):
return (datetime.datetime.now(datetime.timezone.utc)
+ datetime.timedelta(seconds=seconds)).isoformat()
def test_agent_id_colon_slash_backslash_rejected(self):
for bad in ("a:b", "a/b", "a\\b"):
status, body = self._register(agent_id=bad)
self.assertEqual(400, status, repr(bad))
self.assertEqual(AGENT_ID_BODY, body)
status, body = self._register(agent_id="a:b", app_uuid=str(uuidlib.uuid4()))
self.assertEqual(400, status)
self.assertEqual(AGENT_ID_BODY, body)
self.assertEqual([], self._audit("register"))
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
def test_tag_containing_colon_rejected(self):
status, body = self._register(tags=("ok", "ba:d"))
self.assertEqual(400, status)
self.assertEqual({"error": 400, "message": "invalid tags"}, body)
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
status, body = self._register(tags=("ok",))
self.assertEqual(201, status)
self.assertTrue(body["created"])
def test_user_id_colon_rejected_at_allowlist_add(self):
status, body = self._request("POST", USERS_PATH, {"user_id": "co:o"}, self.admin)
self.assertEqual(400, status)
self.assertEqual({"error": 400, "message": "invalid user_id"}, body)
status, body = self._register(user_id="co:o")
self.assertEqual(403, status)
self.assertEqual({"error": 403, "message": "registration not allowed"}, body)
self.assertEqual(1, len(self._audit("register", "denied")))
status, body = self._request("POST", USERS_PATH, {"user_id": "coo"}, self.admin)
self.assertEqual(201, status)
status, body = self._register(agent_id="coo-agent", user_id="coo")
self.assertEqual(201, status)
def test_default_fifty_agent_cap_blocks_register(self):
self.assertEqual("50", self.broker.config["BROKER_MAX_AGENTS_PER_APP"])
now = datetime.datetime.now(datetime.timezone.utc).isoformat()
seed_hash = coreci_broker.hash_token("seed")
for index in range(50):
self.broker.store.save_agent(
"seeded-%d" % index, self.app_uuid, self.user_id, "[]",
seed_hash, self._future(), now,
)
status, body = self._register(agent_id="overflow")
self.assertEqual(400, status)
self.assertEqual(CAP_BODY, body)
status, body = self._register(agent_id="seeded-0")
self.assertEqual(400, status)
self.assertEqual(CAP_BODY, body)
self.assertIsNone(self.broker.store.get_agent("overflow", self.app_uuid))
self.assertEqual(50, len(self.broker.store.agents_by_uuid(self.app_uuid)))
self.assertEqual([], self._audit("register"))
def test_max_agents_per_app_env_override_honored(self):
env = dict(self.env)
env["BROKER_DB_PATH"] = os.path.join(self.tmp, "cap.db")
env["BROKER_AUDIT_LOG_PATH"] = os.path.join(self.tmp, "cap.audit.log")
env["BROKER_MAX_AGENTS_PER_APP"] = "2"
with contextlib.redirect_stdout(io.StringIO()):
broker = coreci_broker.boot(dict(env))
self.addCleanup(broker.audit.close)
self.addCleanup(broker.store.close)
self.assertEqual("2", broker.config["BROKER_MAX_AGENTS_PER_APP"])
server = coreci_broker.serve("127.0.0.1", 0, broker)
self.addCleanup(server.shutdown)
self.addCleanup(server.server_close)
base = "http://127.0.0.1:%d" % server.server_address[1]
self._seed(self.app_uuid, self.user_id, base=base)
for agent_id in ("agent-a", "agent-b"):
status, body = self._register(agent_id=agent_id, base=base)
self.assertEqual(201, status, agent_id)
self.assertTrue(body["created"])
status, body = self._register(agent_id="agent-c", base=base)
self.assertEqual(400, status)
self.assertEqual(CAP_BODY, body)
self.assertIsNone(broker.store.get_agent("agent-c", self.app_uuid))
self.assertEqual(2, len(broker.store.agents_by_uuid(self.app_uuid)))
def test_overwrite_evicts_uuid_cache_entries(self):
_, first = self._register(tags=("red",))
old_token = first["token"]
status, _ = self._auth(old_token)
self.assertEqual(200, status)
self.assertIn((self.app_uuid, old_token), self.broker.token_cache)
status, second = self._register(tags=("blue",))
self.assertEqual(200, status)
self.assertFalse(second["created"])
new_token = second["token"]
self.assertNotEqual(old_token, new_token)
self.assertEqual([], [k for k in self.broker.token_cache if k[0] == self.app_uuid])
status, body = self._auth(old_token)
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
self.assertNotIn((self.app_uuid, old_token), self.broker.token_cache)
status, body = self._auth(new_token)
self.assertEqual(200, status)
self.assertEqual("agent-1", body["agent_id"])
self.assertIn((self.app_uuid, new_token), self.broker.token_cache)
row = self.broker.store.get_agent("agent-1", self.app_uuid)
self.assertFalse(coreci_broker.verify_token(old_token, row[4]))
self.assertTrue(coreci_broker.verify_token(new_token, row[4]))
def test_integrity_error_race_takes_idempotent_path(self):
agent_id = "race-agent"
tags = ["coreci"]
pepper = self.broker.store.get_config("BROKER_INSTANCE_PEPPER")
token = coreci_broker.derive_token(pepper, self.app_uuid, self.user_id, agent_id, tags)
expires = self._future()
now = datetime.datetime.now(datetime.timezone.utc).isoformat()
self.broker.store.save_agent(
agent_id, self.app_uuid, self.user_id, json.dumps(sorted(tags)),
coreci_broker.hash_token(token), expires, now,
)
real_get_agent = self.broker.store.get_agent
calls = []
def racing_get_agent(agent_id, app_uuid):
calls.append(agent_id)
if len(calls) == 1:
return None
return real_get_agent(agent_id, app_uuid)
self.broker.store.get_agent = racing_get_agent
self.addCleanup(setattr, self.broker.store, "get_agent", real_get_agent)
status, body = self._register(agent_id=agent_id, tags=tags)
self.assertEqual(200, status)
self.assertFalse(body["created"])
self.assertEqual(token, body["token"])
self.assertEqual(expires, body["token_expires_at"])
self.assertIn("pepper", body)
self.assertEqual(2, len(calls))
row = real_get_agent(agent_id, self.app_uuid)
self.assertTrue(coreci_broker.verify_token(token, row[4]))
self.assertGreaterEqual(row[7], now)
self.assertEqual(1, len(self._audit("register", "ok")))
if __name__ == "__main__":
unittest.main()
+207
View File
@@ -0,0 +1,207 @@
import base64
import contextlib
import errno
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
from unittest import mock
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
REGISTER_PATH = "/api/broker/agents/register"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
AGENT_ID_BODY = {"error": 400, "message": "invalid agent_id"}
TAGS_BODY = {"error": 400, "message": "invalid tags"}
USER_ID_BODY = {"error": 400, "message": "invalid user_id"}
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
class P5RegisterValidationTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
}
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(dict(self.env))
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
self.app_uuid = str(uuidlib.uuid4())
self.user_id = "ceo"
self._seed()
def _request(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(self.base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
return response.status, json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
return error.code, json.loads(error.read().decode("utf-8"))
def _seed(self):
status, _ = self._request("POST", APPS_PATH, {"uuid": self.app_uuid}, self.admin)
self.assertEqual(201, status)
status, _ = self._request("POST", USERS_PATH, {"user_id": self.user_id}, self.admin)
self.assertEqual(201, status)
def _register(self, agent_id="agent-1", tags=("coreci",)):
return self._request("POST", REGISTER_PATH, {
"agent_id": agent_id,
"app_registration_uuid": self.app_uuid,
"user_id": self.user_id,
"tags": list(tags),
})
def _audit(self, action=None, result=None):
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
entries = [json.loads(raw) for raw in handle if raw.strip()]
return [
entry for entry in entries
if (action is None or entry["action"] == action)
and (result is None or entry["result"] == result)
]
def test_agent_id_bare_dot_and_dotdot_rejected(self):
for agent_id in (".", ".."):
status, body = self._register(agent_id=agent_id)
self.assertEqual(400, status, repr(agent_id))
self.assertEqual(AGENT_ID_BODY, body)
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
self.assertEqual([], self._audit("register"))
def test_agent_id_control_chars_rejected(self):
for agent_id in ("a\x00b", "a\nb", "a\x1fb", "a\x7fb"):
status, body = self._register(agent_id=agent_id)
self.assertEqual(400, status, repr(agent_id))
self.assertEqual(AGENT_ID_BODY, body)
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
self.assertEqual([], self._audit("register"))
def test_agent_id_embedded_dots_still_accepted(self):
status, body = self._register(agent_id="a.b")
self.assertEqual(201, status)
self.assertTrue(body["created"])
status, body = self._register(agent_id="tool.v2")
self.assertEqual(201, status)
self.assertTrue(body["created"])
self.assertIsNotNone(self.broker.store.get_agent("a.b", self.app_uuid))
self.assertIsNotNone(self.broker.store.get_agent("tool.v2", self.app_uuid))
self.assertEqual(2, len(self._audit("register", "ok")))
def test_tag_control_chars_rejected(self):
for tags in (("ok", "ba\x00d"), ("ok", "ba\x7fd"), ("ok", "ba:d")):
status, body = self._register(tags=tags)
self.assertEqual(400, status, repr(tags))
self.assertEqual(TAGS_BODY, body)
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
status, body = self._register(tags=("a.b", ".."))
self.assertEqual(201, status)
self.assertTrue(body["created"])
row = self.broker.store.get_agent("agent-1", self.app_uuid)
self.assertEqual(["..", "a.b"], json.loads(row[3]))
def test_agent_id_path_separators_still_rejected(self):
for agent_id in ("a:b", "a/b", "a\\b"):
status, body = self._register(agent_id=agent_id)
self.assertEqual(400, status, repr(agent_id))
self.assertEqual(AGENT_ID_BODY, body)
self.assertEqual([], self._audit("register"))
status, body = self._register(agent_id="plain-agent")
self.assertEqual(201, status)
self.assertTrue(body["created"])
def test_user_allowlist_add_shares_id_rules(self):
for user_id in (".", "..", "a\x00b", "a:b", "a/b"):
status, body = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
self.assertEqual(400, status, repr(user_id))
self.assertEqual(USER_ID_BODY, body)
status, body = self._request("POST", USERS_PATH, {"user_id": "ceo.two"}, self.admin)
self.assertEqual(201, status)
class P5ReexecGuardTests(unittest.TestCase):
def setUp(self):
coreci_broker._reset_server_guard()
self.addCleanup(coreci_broker._reset_server_guard)
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
def _env(self, bind):
return {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_BIND": bind,
}
def _exec_broker(self):
with open(coreci_broker.__file__, "r", encoding="utf-8") as handle:
source = handle.read()
namespace = {"__name__": "coreci_broker_reexec"}
exec(compile(source, "coreci_broker.py", "exec"), namespace)
return namespace
def _healthz(self, port):
url = "http://127.0.0.1:%d/healthz" % port
with urllib.request.urlopen(url, timeout=5) as response:
return response.status, json.loads(response.read().decode("utf-8"))
def test_fresh_namespace_tools_fails_loud_when_old_module_serves(self):
first = self._exec_broker()
with mock.patch.dict(os.environ, self._env("127.0.0.1:0"), clear=True):
with contextlib.redirect_stdout(io.StringIO()):
first_tools = first["Tools"]()
first_again = first["Tools"]()
server = first["_started_server"]
self.addCleanup(first["_reset_server_guard"])
self.assertIs(server, first_tools._server)
self.assertIs(first_tools._server, first_again._server)
port = server.server_address[1]
self.assertNotEqual(0, port)
second = self._exec_broker()
self.assertIsNone(second["_started_server"])
with mock.patch.dict(os.environ, self._env("127.0.0.1:%d" % port), clear=True):
with contextlib.redirect_stdout(io.StringIO()):
with self.assertRaises(RuntimeError) as caught:
second["Tools"]()
message = str(caught.exception).lower()
self.assertIn("restart", message)
self.assertIn("open webui", message)
self.assertIsInstance(caught.exception.__cause__, OSError)
self.assertEqual(errno.EADDRINUSE, caught.exception.__cause__.errno)
self.assertIsNone(second["_started_server"])
self.assertEqual((200, {"ok": True}), self._healthz(port))
if __name__ == "__main__":
unittest.main()
+799
View File
@@ -0,0 +1,799 @@
"""Stub-registry tests for scripts/publish_image.py (REQ-23, G-019).
Two local stub servers stand in for the public image registry (anonymous
pull) and the gitea container registry (JWT auth + blob uploads +
manifest PUTs). No network access; tiny in-memory fixtures; every
spawned server is cleaned via addCleanup.
"""
import base64
import hashlib
import io
import json
import os
import sys
import tempfile
import threading
import time
import unittest
import urllib.parse
from contextlib import redirect_stderr, redirect_stdout
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
_SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.dirname(
os.path.abspath(__file__))), "scripts")
if _SCRIPTS_DIR not in sys.path:
sys.path.insert(0, _SCRIPTS_DIR)
import publish_image # noqa: E402 (scripts/ dir on sys.path)
OCI_MANIFEST = "application/vnd.oci.image.manifest.v1+json"
OCI_INDEX = "application/vnd.oci.image.index.v1+json"
OCI_CONFIG = "application/vnd.oci.image.config.v1+json"
OCI_LAYER = "application/vnd.oci.image.layer.v1.tar+gzip"
OWNER = "testowner"
REPO = "testrepo"
NAME = "{}/{}".format(OWNER, REPO)
def _sha(data):
return "sha256:" + hashlib.sha256(data).hexdigest()
def _dump(payload):
return json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
class HubFixtures:
"""A tiny synthetic base image: index -> amd64/arm64 children."""
def __init__(self):
self.layer0 = b"base layer zero payload\x00" + b"\xa5" * 64
self.layer1 = b"base layer one payload" * 7
self.config = _dump({
"architecture": "amd64",
"os": "linux",
"config": {"Cmd": ["python3"], "Env": ["PATH=/usr/local/bin"]},
"rootfs": {"type": "layers",
"diff_ids": [_sha(self.layer0), _sha(self.layer1)]},
})
self.child = {
"schemaVersion": 2,
"mediaType": OCI_MANIFEST,
"config": {"mediaType": OCI_CONFIG,
"digest": _sha(self.config),
"size": len(self.config)},
"layers": [
{"mediaType": OCI_LAYER, "digest": _sha(self.layer0),
"size": len(self.layer0)},
{"mediaType": OCI_LAYER, "digest": _sha(self.layer1),
"size": len(self.layer1)},
],
}
self.child_digest = _sha(_dump(self.child))
arm64_child = dict(self.child)
arm64_child["annotations"] = {"test-fixture": "arm64"}
self.arm64_child_digest = _sha(_dump(arm64_child))
self.index = {
"schemaVersion": 2,
"mediaType": OCI_INDEX,
"manifests": [
{"mediaType": OCI_MANIFEST, "digest": self.child_digest,
"size": len(_dump(self.child)),
"platform": {"os": "linux", "architecture": "amd64"}},
{"mediaType": OCI_MANIFEST,
"digest": self.arm64_child_digest,
"size": len(_dump(arm64_child)),
"platform": {"os": "linux", "architecture": "arm64"}},
],
}
self.blobs = {
_sha(self.config): self.config,
_sha(self.layer0): self.layer0,
_sha(self.layer1): self.layer1,
}
class _StubHubHandler(BaseHTTPRequestHandler):
fixtures = None # set per test class run
requests = None
token_mode = "ok" # ok | malformed (200 with an INVALID JSON body)
def _record(self):
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length else b""
self.requests.append({
"method": self.command,
"path": self.path,
"authorization": self.headers.get("Authorization", ""),
"accept": self.headers.get("Accept", ""),
"body": body,
})
def _send(self, status, payload, content_type):
data = payload if isinstance(payload, bytes) else _dump(payload)
self.send_response(status)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(data)))
self.end_headers()
if self.command != "HEAD":
self.wfile.write(data)
def do_GET(self):
self._record()
path = urllib.parse.urlsplit(self.path).path
query = dict(urllib.parse.parse_qsl(
urllib.parse.urlsplit(self.path).query))
if path == "/token":
if self.token_mode == "malformed":
# 200 + invalid JSON: the CWE-754 probe — must surface as
# a remote failure WITH the IMAGE_PENDING marker, never
# a bare traceback.
self._send(200, b"this { is not valid json", "application/json")
else:
self._send(200, {"token": "hub-anon-token"}, "application/json")
elif path == "/v2/library/python/manifests/3.11-slim":
self._send(200, self.fixtures.index, OCI_INDEX)
elif path == "/v2/library/python/manifests/" + self.fixtures.child_digest:
self._send(200, self.fixtures.child, OCI_MANIFEST)
else:
digest = path.rsplit("/", 1)[-1]
if digest in self.fixtures.blobs:
self._send(200, self.fixtures.blobs[digest],
"application/octet-stream")
else:
self._send(404, {"errors": [{"digest": digest}]},
"application/json")
def log_message(self, format, *args):
pass
class _StubGiteaHandler(BaseHTTPRequestHandler):
"""Scriptable gitea registry stub.
state dict keys: requests, uploaded, token_mode (ok|empty),
upload_mode (monolithic|redirect307|redirect307_cross|stall),
manifest_mode (ok|fail500), redirect_seen, manifest_puts,
cross_origin_base (absolute URL of the second-origin stub used by
redirect307_cross).
"""
state = None
def _record(self):
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length else b""
self.state["requests"].append({
"method": self.command,
"path": self.path,
"authorization": self.headers.get("Authorization", ""),
"content_type": self.headers.get("Content-Type", ""),
"body": body,
})
def _send(self, status, payload=b"", content_type="application/json",
headers=None):
data = payload if isinstance(payload, bytes) else _dump(payload)
self.send_response(status)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(data)))
for key, value in (headers or {}).items():
self.send_header(key, value)
self.end_headers()
if self.command != "HEAD":
self.wfile.write(data)
def do_GET(self):
self._record()
parsed = urllib.parse.urlsplit(self.path)
if parsed.path == "/v2/":
# G-013 smoke check 1: the registry auth challenge (401 +
# Bearer; the realm defaults to this stub's own token
# endpoint so pre-existing state dicts work unchanged).
realm = self.state.get("challenge_realm") or \
"http://{}:{}/v2/token".format(
self.server.server_address[0],
self.server.server_address[1])
self._send(401, {"errors": [{"code": "UNAUTHORIZED"}]},
headers={
"WWW-Authenticate":
'Bearer realm="{}",service="{}"'.format(
realm, publish_image.GITEA_AUTH_SERVICE)})
return
if parsed.path == "/v2/token":
if self.state["token_mode"] == "empty":
self._send(200, {"token": ""})
else:
self._send(200, {"token": "stub-jwt"})
return
if parsed.path.startswith("/api/v1/repos/") and \
"/releases/tags/" in parsed.path:
# G-013 smoke check 3: the gitea API release-by-tag GET
# (state["releases"]: tag -> release payload; absent -> 404).
tag = parsed.path.rsplit("/", 1)[-1]
release = self.state.get("releases", {}).get(tag)
if release is None:
self._send(404, {"errors": [{"message": "release not found"}]})
else:
self._send(200, release)
return
def do_HEAD(self):
self._record()
parsed = urllib.parse.urlsplit(self.path)
digest = parsed.path.rsplit("/", 1)[-1]
if digest in self.state["uploaded"]:
self._send(200, b"")
else:
self._send(404, b"")
def do_POST(self):
self._record()
parsed = urllib.parse.urlsplit(self.path)
query = dict(urllib.parse.parse_qsl(parsed.query))
digest = query.get("digest", "")
if self.state["upload_mode"] == "stall":
time.sleep(15)
return
if self.state["upload_mode"] == "redirect307_cross" and \
digest not in self.state["redirect_seen"]:
# CWE-200 probe: ABSOLUTE Location pointing at a DIFFERENT
# origin (the second stub server) — the client must follow
# WITHOUT the Authorization header.
self.state["redirect_seen"].add(digest)
location = "{}/v2/{}/blobs/uploads/?digest={}".format(
self.state["cross_origin_base"], NAME, digest)
self._send(307, b"", headers={"Location": location})
return
if self.state["upload_mode"] == "redirect307" and \
digest not in self.state["redirect_seen"]:
self.state["redirect_seen"].add(digest)
location = "/v2/{}/blobs/uploads/?digest={}".format(NAME, digest)
self._send(307, b"", headers={"Location": location})
return
if _sha(self.state["requests"][-1]["body"]) == digest:
self.state["uploaded"][digest] = self.state["requests"][-1]["body"]
self._send(201, {"status": "created"})
else:
self._send(400, {"errors": [{"digest": digest}]})
def do_PUT(self):
self._record()
parsed = urllib.parse.urlsplit(self.path)
query = dict(urllib.parse.parse_qsl(parsed.query))
body = self.state["requests"][-1]["body"]
if "/manifests/" in parsed.path:
if self.state["manifest_mode"] == "fail500":
self.state.setdefault("manifest_fail_count", 0)
self.state["manifest_fail_count"] += 1
self._send(500, {"errors": [{"code": "UNKNOWN"}]})
return
manifest = json.loads(body)
referenced = [manifest["config"]["digest"]] + [
layer["digest"] for layer in manifest["layers"]]
if any(digest not in self.state["uploaded"]
for digest in referenced):
self._send(400, {"errors": [{"code": "BLOB_UNKNOWN"}]})
return
if self.headers.get("Content-Type") != OCI_MANIFEST:
self._send(400, {"errors": [{"code": "MANIFEST_INVALID"}]})
return
self.state.setdefault("manifest_puts", []).append(
{"tag": parsed.path.rsplit("/", 1)[-1],
"content_type": self.headers.get("Content-Type", ""),
"manifest": manifest})
self._send(201, {})
return
digest = query.get("digest", "")
if _sha(body) == digest:
self.state["uploaded"][digest] = body
self._send(201, {})
else:
self._send(400, {"errors": [{"code": "DIGEST_INVALID"}]})
class _StubCrossOriginHandler(BaseHTTPRequestHandler):
"""Second-origin stub (another port) for the CWE-200 redirect probe.
state dict keys: requests, uploaded — ``uploaded`` is SHARED with the
gitea stub (the redirect target models the registry's presigned
external storage; a landed blob is visible to the registry), while
``requests`` stays separate so the test can assert on exactly what
this origin received — above all that NO Authorization header
crossed the origin boundary.
"""
state = None
def _record(self):
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length else b""
self.state["requests"].append({
"method": self.command,
"path": self.path,
"authorization": self.headers.get("Authorization", ""),
"content_type": self.headers.get("Content-Type", ""),
"body": body,
})
return body
def _send(self, status, payload=b"", content_type="application/json"):
data = payload if isinstance(payload, bytes) else _dump(payload)
self.send_response(status)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(data)))
self.end_headers()
if self.command != "HEAD":
self.wfile.write(data)
def do_POST(self):
body = self._record()
query = dict(urllib.parse.parse_qsl(
urllib.parse.urlsplit(self.path).query))
digest = query.get("digest", "")
if _sha(body) == digest:
self.state["uploaded"][digest] = body
self._send(201, {"status": "created"})
else:
self._send(400, {"errors": [{"code": "DIGEST_INVALID"}]})
def log_message(self, format, *args):
pass
class PublishImageTests(unittest.TestCase):
"""End-to-end publish flows against local stub registries."""
TAG = "v0.2.2-test"
@classmethod
def setUpClass(cls):
cls.fixtures = HubFixtures()
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.artifact = os.path.join(self.temp.name, "coreci-broker.pyz")
with open(self.artifact, "wb") as handle:
handle.write(b"fake pyz payload for tests" * 3)
self.env_root = os.path.join(self.temp.name, "envroot")
os.makedirs(self.env_root)
with open(os.path.join(self.env_root, ".env.secrets"), "w") as handle:
handle.write("GITEA_TOKEN=testtoken123\n")
self.hub_requests = []
_StubHubHandler.fixtures = self.fixtures
_StubHubHandler.requests = self.hub_requests
_StubHubHandler.token_mode = "ok"
self.hub = ThreadingHTTPServer(("127.0.0.1", 0), _StubHubHandler)
threading.Thread(target=self.hub.serve_forever, daemon=True).start()
self.addCleanup(self.hub.shutdown)
self.addCleanup(self.hub.server_close)
self.gitea_state = {
"requests": [], "uploaded": {}, "token_mode": "ok",
"upload_mode": "monolithic", "manifest_mode": "ok",
"redirect_seen": set(),
}
_StubGiteaHandler.state = self.gitea_state
self.gitea = ThreadingHTTPServer(("127.0.0.1", 0), _StubGiteaHandler)
threading.Thread(target=self.gitea.serve_forever, daemon=True).start()
self.addCleanup(self.gitea.shutdown)
self.addCleanup(self.gitea.server_close)
def run_publish(self, tag=None, timeout=60, artifact=None, env_root=None):
argv = [
tag or self.TAG,
"--artifact", artifact or self.artifact,
"--hub-registry", "http://127.0.0.1:{}".format(self.hub.server_address[1]),
"--hub-auth", "http://127.0.0.1:{}/token".format(self.hub.server_address[1]),
"--gitea-registry", "http://127.0.0.1:{}".format(self.gitea.server_address[1]),
"--owner", OWNER, "--repo", REPO,
"--env-root", env_root or self.env_root,
"--timeout", str(timeout),
]
out, err = io.StringIO(), io.StringIO()
with redirect_stdout(out), redirect_stderr(err):
code = publish_image.main(argv)
return code, out.getvalue(), err.getvalue()
# 1 -----------------------------------------------------------------
def test_happy_path_end_to_end(self):
code, out, err = self.run_publish()
self.assertEqual(code, 0, err or out)
puts = self.gitea_state.get("manifest_puts", [])
self.assertEqual([entry["tag"] for entry in puts],
[self.TAG, "latest"],
"version tag FIRST, then latest (D-021)")
manifest = puts[0]["manifest"]
self.assertEqual(puts[0]["content_type"], OCI_MANIFEST)
self.assertEqual(manifest["config"]["mediaType"], OCI_CONFIG)
broker_layer = manifest["layers"][-1]
self.assertEqual(broker_layer["mediaType"], OCI_LAYER)
base_descriptors = self.fixtures.child["layers"]
self.assertEqual(manifest["layers"][:-1], base_descriptors,
"base layer descriptors reused verbatim")
# deterministic broker layer digest matches an independent build
with open(self.artifact, "rb") as handle:
artifact_bytes = handle.read()
layer_blob, layer_digest, diff_id = publish_image.build_broker_layer(
artifact_bytes)
self.assertEqual(broker_layer["digest"], layer_digest)
self.assertEqual(broker_layer["size"], len(layer_blob))
# pushed config content
pushed_config = json.loads(
self.gitea_state["uploaded"][manifest["config"]["digest"]])
self.assertEqual(pushed_config["config"]["Entrypoint"],
["python3", "/app/coreci-broker.pyz"])
self.assertEqual(pushed_config["config"]["Cmd"], [])
self.assertEqual(pushed_config["config"]["WorkingDir"], "/state")
self.assertEqual(pushed_config["config"]["Labels"],
{"org.opencontainers.image.version": self.TAG})
self.assertEqual(pushed_config["rootfs"]["diff_ids"][-1], diff_id)
self.assertEqual(len(pushed_config["rootfs"]["diff_ids"]),
len(self.fixtures.child["layers"]) + 1)
# base config blob uploaded verbatim (digest invariant)
self.assertEqual(
self.gitea_state["uploaded"][self.fixtures.child["config"]["digest"]],
self.fixtures.config)
# all blobs present
for descriptor in manifest["layers"]:
self.assertIn(descriptor["digest"], self.gitea_state["uploaded"])
self.assertNotIn("IMAGE_PENDING", out)
# 2 -----------------------------------------------------------------
def test_head_skip_blob_not_reuploaded(self):
preseed = self.fixtures.child["layers"][0]["digest"]
self.gitea_state["uploaded"][preseed] = self.fixtures.layer0
code, out, err = self.run_publish()
self.assertEqual(code, 0, err or out)
posts = [r for r in self.gitea_state["requests"]
if r["method"] == "POST" and preseed in r["path"]]
self.assertEqual(posts, [], "pre-seeded blob must not be re-uploaded")
hub_blob_gets = [r for r in self.hub_requests
if r["method"] == "GET" and "/blobs/" in r["path"]
and preseed in r["path"]]
self.assertEqual(hub_blob_gets, [],
"skipped blob is never fetched from the hub")
# 3 -----------------------------------------------------------------
def test_redirect_307_relative_location_followed(self):
self.gitea_state["upload_mode"] = "redirect307"
code, out, err = self.run_publish()
self.assertEqual(code, 0, err or out)
blob_posts = [r for r in self.gitea_state["requests"]
if r["method"] == "POST" and "/blobs/uploads/" in r["path"]
and "digest=" in r["path"]]
self.assertGreater(len(blob_posts), 0)
digests = [dict(urllib.parse.parse_qsl(
urllib.parse.urlsplit(r["path"]).query))["digest"]
for r in blob_posts]
self.assertTrue(all(d in self.gitea_state["uploaded"]
for d in digests),
"every redirected upload must land")
# each redirected digest POSTed twice (307 + follow)
for digest in set(digests):
self.assertEqual(digests.count(digest), 2,
"307 then explicit re-POST per digest")
# same-origin (relative Location) redirects KEEP the
# Authorization header — the CWE-200 strip is cross-origin ONLY
for request in blob_posts:
self.assertEqual(
request["authorization"], "Bearer stub-jwt",
"same-origin redirect must keep Authorization")
# 4 -----------------------------------------------------------------
def test_anonymous_token_trap_refused_before_push(self):
self.gitea_state["token_mode"] = "empty"
code, out, err = self.run_publish()
self.assertEqual(code, 1, "remote auth failure -> non-zero")
self.assertIn("anonymous-200 trap refused", err)
self.assertIn("IMAGE_PENDING", out)
post_put = [r for r in self.gitea_state["requests"]
if r["method"] in ("POST", "PUT")]
self.assertEqual(post_put, [],
"no blob upload or manifest PUT may happen")
# 5 -----------------------------------------------------------------
def test_layer_and_config_determinism(self):
with open(self.artifact, "rb") as handle:
artifact = handle.read()
first = publish_image.build_broker_layer(artifact)
second = publish_image.build_broker_layer(artifact)
self.assertEqual(first[1], second[1], "layer digest stable")
self.assertEqual(first[2], second[2], "diff_id stable")
base_config = json.loads(self.fixtures.config)
c1 = publish_image.build_config_blob(base_config, first[2], "t1")
c2 = publish_image.build_config_blob(base_config, first[2], "t1")
self.assertEqual(_sha(c1), _sha(c2), "config digest stable")
# 6 -----------------------------------------------------------------
def test_amd64_child_selected(self):
code, out, err = self.run_publish()
self.assertEqual(code, 0, err or out)
child_gets = [r for r in self.hub_requests
if "/manifests/sha256:" in r["path"]]
self.assertTrue(child_gets, "child manifest must be fetched by digest")
self.assertTrue(all(self.fixtures.child_digest in r["path"]
for r in child_gets),
"amd64 child only")
self.assertFalse(any(self.fixtures.arm64_child_digest in r["path"]
for r in self.hub_requests),
"arm64 child never touched")
# 7 -----------------------------------------------------------------
def test_timeout_aborts_rather_than_hangs(self):
self.gitea_state["upload_mode"] = "stall"
started = time.monotonic()
code, out, err = self.run_publish(timeout=1)
elapsed = time.monotonic() - started
self.assertEqual(code, 1)
self.assertIn("IMAGE_PENDING", out)
self.assertLess(elapsed, 30, "bounded retries, no hang")
stalls = [r for r in self.gitea_state["requests"]
if r["method"] == "POST" and "/blobs/uploads/" in r["path"]]
self.assertLessEqual(len(stalls), 3, "bounded retry attempts")
# 8 -----------------------------------------------------------------
def test_manifest_retry_exhaustion_marker(self):
self.gitea_state["manifest_mode"] = "fail500"
code, out, err = self.run_publish()
self.assertEqual(code, 1)
self.assertIn("IMAGE_PENDING", out)
self.assertEqual(self.gitea_state.get("manifest_fail_count"), 3,
"exactly three bounded attempts on the tag PUT")
self.assertNotIn("latest",
[r["path"].rsplit("/", 1)[-1]
for r in self.gitea_state["requests"]
if r["method"] == "PUT"],
"latest must not be attempted after exhaustion")
# 9 -----------------------------------------------------------------
def test_token_discipline(self):
code, out, err = self.run_publish()
self.assertEqual(code, 0, err or out)
token_calls = [r for r in self.gitea_state["requests"]
if r["path"].startswith("/v2/token")]
self.assertTrue(token_calls)
expected = base64.b64encode(
b"publisher:testtoken123").decode("ascii")
for call in token_calls:
self.assertEqual(call["authorization"],
"Basic {}".format(expected))
# scope requested with pull,push (registry v2 scope form)
query = urllib.parse.unquote(
urllib.parse.urlsplit(token_calls[0]["path"]).query)
self.assertIn("repository:{}/{}:pull,push".format(OWNER, REPO),
query)
# missing token: local precondition, zero remote requests
empty_root = os.path.join(self.temp.name, "emptyroot")
os.makedirs(empty_root)
before = len(self.gitea_state["requests"]) + len(self.hub_requests)
code, out, err = self.run_publish(env_root=empty_root)
self.assertEqual(code, 2, "local precondition -> exit 2")
self.assertNotIn("IMAGE_PENDING", out, "no marker for local errors")
after = len(self.gitea_state["requests"]) + len(self.hub_requests)
self.assertEqual(before, after, "no request of any kind was made")
# 10 ----------------------------------------------------------------
def test_malformed_hub_token_json_keeps_pending_marker(self):
"""CWE-754 marker discipline: a 200 with an INVALID JSON body from
the hub token endpoint fails as a remote answer — exit 1 WITH the
IMAGE_PENDING marker — never an uncaught-ValueError traceback."""
_StubHubHandler.token_mode = "malformed"
code, out, err = self.run_publish()
self.assertEqual(code, 1)
self.assertIn("malformed JSON", err,
"the failure class must be named in the error")
self.assertIn("IMAGE_PENDING", out,
"marker discipline holds for malformed remote JSON")
self.assertNotIn("Traceback", out + err)
self.assertEqual(self.gitea_state["requests"], [],
"nothing is attempted against gitea")
# 11 ----------------------------------------------------------------
def test_redirect_307_cross_origin_strips_authorization(self):
"""CWE-200: a 307 whose Location resolves to a DIFFERENT origin
is followed WITHOUT the Authorization header — the registry JWT
never crosses origins — and the blob still lands."""
cross_state = {"requests": [], "uploaded": self.gitea_state["uploaded"]}
_StubCrossOriginHandler.state = cross_state
cross = ThreadingHTTPServer(("127.0.0.1", 0), _StubCrossOriginHandler)
threading.Thread(target=cross.serve_forever, daemon=True).start()
self.addCleanup(cross.shutdown)
self.addCleanup(cross.server_close)
self.gitea_state["upload_mode"] = "redirect307_cross"
self.gitea_state["cross_origin_base"] = "http://127.0.0.1:{}".format(
cross.server_address[1])
code, out, err = self.run_publish()
self.assertEqual(code, 0, err or out)
self.assertTrue(cross_state["requests"],
"the redirect was followed to the other origin")
for request in cross_state["requests"]:
self.assertEqual(
request["authorization"], "",
"Authorization must be stripped on cross-origin redirects")
# the blob landed (visible through the shared registry blob store)
self.assertEqual(len(self.gitea_state["uploaded"]), 5,
"base config + 2 base layers + broker layer + "
"image config all landed")
# and the flow still completes: both manifest PUTs back on the
# origin, where Authorization remains in play
self.assertEqual(
[entry["tag"] for entry in self.gitea_state.get(
"manifest_puts", [])],
[self.TAG, "latest"])
class SmokeModeTests(unittest.TestCase):
"""G-013 --smoke logic, stub-tested.
The LIVE smoke never runs in this suite — it executes only at the
P3-05 ship, where the ship gate sets PUBLISH_SMOKE=1 in the
environment (the script itself never reads that variable). Here the
four GET-only checks run against fresh stub instances: zero network,
and the stub records double as the read-only proof (a POST/PUT
landing anywhere in a smoke run is a hard failure).
"""
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.env_root = os.path.join(self.temp.name, "envroot")
os.makedirs(self.env_root)
with open(os.path.join(self.env_root, ".env.secrets"), "w") as handle:
handle.write("GITEA_TOKEN=testtoken123\n")
self.fixtures = HubFixtures()
self.hub_requests = []
_StubHubHandler.fixtures = self.fixtures
_StubHubHandler.requests = self.hub_requests
_StubHubHandler.token_mode = "ok"
self.hub = ThreadingHTTPServer(("127.0.0.1", 0), _StubHubHandler)
threading.Thread(target=self.hub.serve_forever, daemon=True).start()
self.addCleanup(self.hub.shutdown)
self.addCleanup(self.hub.server_close)
self.gitea_state = {
"requests": [], "uploaded": {}, "token_mode": "ok",
"upload_mode": "monolithic", "manifest_mode": "ok",
"redirect_seen": set(), "releases": {},
}
_StubGiteaHandler.state = self.gitea_state
self.gitea = ThreadingHTTPServer(("127.0.0.1", 0), _StubGiteaHandler)
threading.Thread(target=self.gitea.serve_forever, daemon=True).start()
self.addCleanup(self.gitea.shutdown)
self.addCleanup(self.gitea.server_close)
# The /v2/ challenge realm must point back at THIS stub's token
# endpoint so the recorded-realm exchange stays in-stub.
self.gitea_state["challenge_realm"] = \
"http://127.0.0.1:{}/v2/token".format(
self.gitea.server_address[1])
def run_smoke(self, tag, env_root=None):
argv = [
tag, "--smoke",
"--hub-registry",
"http://127.0.0.1:{}".format(self.hub.server_address[1]),
"--hub-auth",
"http://127.0.0.1:{}/token".format(
self.hub.server_address[1]),
"--gitea-registry",
"http://127.0.0.1:{}".format(self.gitea.server_address[1]),
"--owner", OWNER, "--repo", REPO,
"--env-root", env_root or self.env_root,
]
out, err = io.StringIO(), io.StringIO()
with redirect_stdout(out), redirect_stderr(err):
code = publish_image.main(argv)
return code, out.getvalue(), err.getvalue()
def _post_put(self):
"""(gitea POST/PUT, hub POST/PUT) across the stub records."""
return ([r for r in self.gitea_state["requests"]
if r["method"] in ("POST", "PUT")],
[r for r in self.hub_requests
if r["method"] in ("POST", "PUT")])
def _release_gets(self):
return [r for r in self.gitea_state["requests"]
if "/releases/tags/" in r["path"]]
# 12 ----------------------------------------------------------------
def test_smoke_all_checks_pass(self):
self.gitea_state["releases"]["v0.2.0"] = {
"tag_name": "v0.2.0",
"assets": [{"name": "phase0-notes.txt"}],
}
code, out, err = self.run_smoke("v0.2.0")
self.assertEqual(code, 0, err or out)
for marker in ("PASS [1/4]", "PASS [2/4]", "PASS [3/4]",
"PASS [4/4]"):
self.assertIn(marker, out)
self.assertIn("G-021 confirmed: no .pyz asset", out,
"the v0.2.0 run carries the G-021 forge-side "
"confirmation")
self.assertIn("phase0-notes.txt", out, "asset names print as a fact")
self.assertIn(self.fixtures.child_digest, out,
"the amd64 child digest prints (drift check)")
# the manifest GET carries the full Accept set
manifest_gets = [r for r in self.hub_requests
if "/manifests/3.11-slim" in r["path"]]
self.assertEqual(len(manifest_gets), 1)
self.assertEqual(manifest_gets[0]["accept"],
publish_image.ACCEPT_MANIFESTS)
# the release GET uses the gitea API token auth form
release_gets = self._release_gets()
self.assertEqual(len(release_gets), 1)
self.assertEqual(release_gets[0]["authorization"],
"token testtoken123")
# READ-ONLY PROOF: zero POST/PUT recorded at either stub
gitea_writes, hub_writes = self._post_put()
self.assertEqual(gitea_writes, [], "no gitea write of any kind")
self.assertEqual(hub_writes, [], "no hub write of any kind")
self.assertNotIn("IMAGE_PENDING", out)
# 13 ----------------------------------------------------------------
def test_smoke_v0_2_0_no_pyz_assertion(self):
pyz_release = {"tag_name": "v0.2.0",
"assets": [{"name": "coreci-broker-v0.2.0.pyz"}]}
self.gitea_state["releases"]["v0.2.0"] = pyz_release
self.gitea_state["releases"]["v0.2.2"] = pyz_release
# v0.2.0 + a .pyz asset: the G-021 hard assertion fails the smoke
code, out, err = self.run_smoke("v0.2.0")
self.assertEqual(code, 1)
self.assertIn("FAILED [3/4]", out)
self.assertIn("G-021", out, "the failure line names G-021")
self.assertIn(".pyz", out)
self.assertEqual(self.hub_requests, [],
"a FAILED check stops the smoke: the hub "
"resolve (step 4) never runs")
# the SAME release under a different tag: the assertion binds
# v0.2.0 only -> all four checks pass
code, out, err = self.run_smoke("v0.2.2")
self.assertEqual(code, 0, err or out)
for marker in ("PASS [3/4]", "PASS [4/4]"):
self.assertIn(marker, out)
self.assertNotIn("G-021 violation", out)
gitea_writes, hub_writes = self._post_put()
self.assertEqual(gitea_writes + hub_writes, [],
"still read-only across both runs")
# 14 ----------------------------------------------------------------
def test_smoke_trap_refused(self):
self.gitea_state["token_mode"] = "empty"
self.gitea_state["releases"]["v0.2.3"] = {
"tag_name": "v0.2.3", "assets": [],
}
code, out, err = self.run_smoke("v0.2.3")
self.assertEqual(code, 1, "the trap is a loud remote failure")
self.assertIn("FAILED [2/4]", out)
self.assertIn("anonymous-200 trap refused", out)
self.assertIn("PASS [1/4]", out, "check 1 ran (order 1 -> 4)")
self.assertNotIn("PASS [3/4]", out, "checks after the trap never run")
self.assertNotIn("PASS [4/4]", out)
self.assertEqual(self._release_gets(), [],
"the release GET (step 3) is skipped")
self.assertEqual(self.hub_requests, [],
"zero hub requests: the index resolve (step 4) "
"is skipped")
# 15 ----------------------------------------------------------------
def test_smoke_missing_token_local(self):
empty_root = os.path.join(self.temp.name, "emptyroot")
os.makedirs(empty_root)
code, out, err = self.run_smoke("v0.2.3", env_root=empty_root)
self.assertEqual(code, 2, "local precondition -> exit 2")
self.assertIn("ERROR", err)
self.assertIn("GITEA_TOKEN", err)
self.assertEqual(self.gitea_state["requests"], [],
"zero gitea requests — the precondition fires "
"before anything remote")
self.assertEqual(self.hub_requests, [], "zero hub requests")
self.assertNotIn("IMAGE_PENDING", out, "no marker for local errors")
self.assertNotIn("PASS", out, "no smoke check ever ran")
if __name__ == "__main__":
unittest.main()
+271
View File
@@ -0,0 +1,271 @@
import ast
import base64
import contextlib
import importlib.util
import io
import json
import os
import sys
import tempfile
import time
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
REFERENCE_DIR = os.path.abspath(
os.path.join(os.path.dirname(__file__), os.pardir, "reference_tools")
)
REFERENCE_TITLES = {
"proxmox_readonly_reference": "Proxmox Read-Only Reference [CoreCI]",
"minimal_auth_showcase": "Minimal Auth Showcase [CoreCI]",
}
ALLOWED_REFERENCE_IMPORTS = {"coreci_broker_client", "datetime"}
FORBIDDEN_SOURCE_MARKERS = ("urllib", "httpx", "socket", "proxmoxer")
BROKER_ONLY_AUDIT_ACTIONS = {"add_app", "add_user", "register", "read"}
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
def _load_reference(name):
spec = importlib.util.spec_from_file_location(
name, os.path.join(REFERENCE_DIR, name + ".py")
)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def _reference_source(name):
with open(os.path.join(REFERENCE_DIR, name + ".py"), "r", encoding="utf-8") as handle:
return handle.read()
PROXMOX_REFERENCE = _load_reference("proxmox_readonly_reference")
MINIMAL_SHOWCASE = _load_reference("minimal_auth_showcase")
class _ReferenceCase(unittest.TestCase):
extra_env = {}
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.creds_path = os.path.join(self.tmp, "creds.json")
with open(self.creds_path, "w", encoding="utf-8") as handle:
handle.write(json.dumps({"coreci/proxmox": PROXMOX_SECRET}))
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
"BROKER_CREDS_SOURCE": "file",
"BROKER_CREDS_FILE": self.creds_path,
}
self.env.update(self.extra_env)
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(self.env)
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
self.app_uuid = str(uuidlib.uuid4())
self.user_id = "ceo"
self._seed()
def _seed(self):
status, _ = self._request("POST", APPS_PATH, {"uuid": self.app_uuid}, self.admin)
self.assertEqual(201, status)
status, _ = self._request("POST", USERS_PATH, {"user_id": self.user_id}, self.admin)
self.assertEqual(201, status)
def _request(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(self.base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
raw = response.read().decode("utf-8")
return response.status, json.loads(raw) if raw else None
except urllib.error.HTTPError as error:
raw = error.read().decode("utf-8")
return error.code, json.loads(raw) if raw else None
def _audit(self, action=None, result=None):
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
entries = [json.loads(raw) for raw in handle if raw.strip()]
return [
entry for entry in entries
if (action is None or entry["action"] == action)
and (result is None or entry["result"] == result)
]
def _tools(self, module, agent_id="proxmox"):
tools = module.Tools()
tools.valves.base_url = self.base
tools.valves.agent_id = agent_id
tools.valves.app_uuid = self.app_uuid
tools.valves.user_id = self.user_id
return tools
def _persisted_state(self, agent_id="proxmox"):
return self.broker.store.get_agent_state(agent_id, self.app_uuid)
class ProxmoxReferenceFlowTests(_ReferenceCase):
def test_first_call_registers_persists_state_reads_secret_dry_runs(self):
tools = self._tools(PROXMOX_REFERENCE)
response = tools.list_nodes()
self.assertIs(True, response["dry_run"])
self.assertEqual("list_nodes", response["action"])
self.assertIs(True, response["credential_present"])
self.assertTrue(str(response["note"]).strip())
self.assertNotIn("data", response)
self.assertNotIn("pve-tok", json.dumps(response))
self.assertEqual(1, len(self._audit(action="register")))
document = self._persisted_state()
self.assertIsNotNone(document)
self.assertIn("current_token", document)
self.assertEqual(tools._state["current_token"], document["current_token"])
self.assertEqual([], self._audit(result="denied"))
self.assertTrue(
{entry["action"] for entry in self._audit()} <= BROKER_ONLY_AUDIT_ACTIONS
)
def test_second_call_reuses_persisted_state_without_new_register(self):
tools = self._tools(PROXMOX_REFERENCE)
tools.list_nodes()
token = tools._state["current_token"]
second = tools.list_nodes()
self.assertIs(True, second["dry_run"])
self.assertEqual("list_nodes", second["action"])
self.assertIs(True, second["credential_present"])
self.assertEqual(1, len(self._audit(action="register")))
self.assertEqual(token, tools._state["current_token"])
document = self._persisted_state()
self.assertIn("current_token", document)
self.assertEqual(token, document["current_token"])
def test_vm_status_is_dry_run_with_credential_present(self):
tools = self._tools(PROXMOX_REFERENCE)
response = tools.vm_status("101")
self.assertIs(True, response["dry_run"])
self.assertEqual("vm_status", response["action"])
self.assertIs(True, response["credential_present"])
self.assertNotIn("pve-tok", json.dumps(response))
self.assertEqual(1, len(self._audit(action="register")))
class ProxmoxReferenceTtlTests(_ReferenceCase):
extra_env = {"BROKER_TOKEN_TTL": "1"}
def test_expired_subsequent_call_reregisters_and_persists_new_expiry(self):
tools = self._tools(PROXMOX_REFERENCE)
self.assertTrue(tools.list_nodes()["credential_present"])
first_expires = tools._state["token_expires_at"]
time.sleep(1.2)
response = tools.list_nodes()
self.assertIs(True, response["dry_run"])
self.assertIs(True, response["credential_present"])
self.assertEqual(2, len(self._audit(action="register")))
document = self._persisted_state()
self.assertEqual(tools._state["current_token"], document["current_token"])
self.assertEqual(tools._state["token_expires_at"], document["token_expires_at"])
self.assertNotEqual(first_expires, document["token_expires_at"])
class MinimalShowcaseTests(_ReferenceCase):
def test_demo_authenticates_then_reads_secret(self):
tools = self._tools(MINIMAL_SHOWCASE)
response = tools.demo()
self.assertEqual(
{"registered": True, "secret_read": True,
"path": "coreci/proxmox", "note": "auth first, then use"},
response,
)
self.assertNotIn("pve-tok", json.dumps(response))
self.assertEqual(1, len(self._audit(action="register")))
self.assertEqual([], self._audit(result="denied"))
class ReferenceHygieneTests(unittest.TestCase):
def test_sources_carry_no_secret_literals(self):
for name in REFERENCE_TITLES:
source = _reference_source(name)
self.assertNotRegex(source, r"(?im)^\s*(password|pepper)\s*=\s*[\"']", name)
for value in PROXMOX_SECRET.values():
self.assertNotIn(value, source, name)
def test_sources_use_no_direct_network_or_infra_clients(self):
for name in REFERENCE_TITLES:
source = _reference_source(name)
for marker in FORBIDDEN_SOURCE_MARKERS:
self.assertNotIn(marker, source, "%s: %s" % (name, marker))
def test_imports_limited_to_client_module_and_stdlib(self):
for name in REFERENCE_TITLES:
tree = ast.parse(_reference_source(name))
modules = set()
for node in ast.walk(tree):
if isinstance(node, ast.Import):
modules.update(alias.name for alias in node.names)
elif isinstance(node, ast.ImportFrom):
modules.add(node.module or "")
for module in modules:
self.assertIn(module, ALLOWED_REFERENCE_IMPORTS, "%s: %s" % (name, module))
def test_owui_frontmatter_docstrings(self):
for name, title in REFERENCE_TITLES.items():
doc = ast.get_docstring(ast.parse(_reference_source(name)))
self.assertTrue(doc.startswith("title: " + title), name)
for line in ("author: coreci", "description: ", "version: 0.1.0",
"requirements: []", "licence: MIT"):
self.assertIn(line, doc, "%s: %s" % (name, line))
def test_tools_surface_valves_state_and_docstring_discipline(self):
for name in REFERENCE_TITLES:
module = _load_reference(name)
tools = module.Tools()
self.assertIsInstance(tools.valves, module.Tools.Valves)
for attr in ("base_url", "agent_id", "app_uuid", "user_id"):
self.assertEqual("", getattr(module.Tools.Valves, attr), attr)
self.assertEqual("", getattr(tools.valves, attr), attr)
self.assertEqual({}, tools._state)
public = []
for node in ast.walk(ast.parse(_reference_source(name))):
if not isinstance(node, ast.FunctionDef):
continue
doc = ast.get_docstring(node)
if node.name.startswith("_"):
self.assertIsNone(doc, "%s.%s" % (name, node.name))
else:
self.assertIsNotNone(doc, "%s.%s" % (name, node.name))
public.append(node.name)
self.assertTrue(public, name)
def test_no_comments_in_sources(self):
for name in REFERENCE_TITLES:
for number, line in enumerate(_reference_source(name).splitlines(), 1):
self.assertFalse(line.lstrip().startswith("#"), "%s:%d" % (name, number))
if __name__ == "__main__":
unittest.main()
+347
View File
@@ -0,0 +1,347 @@
import base64
import contextlib
import datetime
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
REGISTER_PATH = "/api/broker/agents/register"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
WHOAMI_PATH = "/api/broker/auth/whoami"
DEFAULT_TTL = 86400
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
def _seed_direct(broker, app_uuid, user_id):
broker.store.add_row("app_registrations", app_uuid, None)
broker.store.add_row("user_allowlist", user_id, None)
class AgentRegisterTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
}
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(dict(self.env))
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
self.app_uuid = str(uuidlib.uuid4())
self.user_id = "ceo"
self._seed(self.app_uuid, self.user_id)
def _request(self, method, path, payload=None, headers=None, base=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(
(base or self.base) + path, data=data, method=method
)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
return response.status, json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
return error.code, json.loads(error.read().decode("utf-8"))
def _seed(self, app_uuid, user_id):
status, _ = self._request("POST", APPS_PATH, {"uuid": app_uuid}, self.admin)
self.assertEqual(201, status)
status, _ = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
self.assertEqual(201, status)
def _register(self, agent_id="agent-1", tags=("coreci",), app_uuid=None,
user_id=None, headers=None, payload=None):
if payload is None:
payload = {
"agent_id": agent_id,
"app_registration_uuid": app_uuid or self.app_uuid,
"user_id": user_id or self.user_id,
"tags": list(tags),
}
return self._request("POST", REGISTER_PATH, payload, headers)
def _agent_row(self, agent_id="agent-1", app_uuid=None):
return self.broker.store.get_agent(agent_id, app_uuid or self.app_uuid)
def _audit(self, action=None, result=None):
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
entries = [json.loads(raw) for raw in handle if raw.strip()]
return [
entry for entry in entries
if (action is None or entry["action"] == action)
and (result is None or entry["result"] == result)
]
def _pepper(self):
return self.broker.store.get_config("BROKER_INSTANCE_PEPPER")
def _app_header(self, token, app_uuid=None):
return {"Authorization": "App %s %s" % (app_uuid or self.app_uuid, token)}
def _ttl_delta_seconds(self, body):
expires = datetime.datetime.fromisoformat(body["token_expires_at"])
now = datetime.datetime.now(datetime.timezone.utc)
return (expires - now).total_seconds()
def test_gate_matrix_unknown_and_revoked_uuid_and_user(self):
unknown_uuid = str(uuidlib.uuid4())
status, body = self._register(app_uuid=unknown_uuid)
self.assertEqual(403, status)
self.assertEqual(403, body["error"])
self.assertTrue(body["message"])
self._request("DELETE", APPS_PATH + "/" + self.app_uuid, headers=self.admin)
status, body = self._register()
self.assertEqual(403, status)
self._request("POST", APPS_PATH + "/" + self.app_uuid + "/reinstate", headers=self.admin)
status, body = self._register(user_id="stranger")
self.assertEqual(403, status)
self._request("DELETE", USERS_PATH + "/" + self.user_id, headers=self.admin)
status, body = self._register()
self.assertEqual(403, status)
denied = self._audit("register", "denied")
self.assertEqual(4, len(denied))
for entry in denied:
self.assertEqual(REGISTER_PATH, entry["path"])
self.assertEqual("agent-1", entry["agent_id"])
self.assertEqual(unknown_uuid, denied[0]["app_registration_uuid"])
self.assertEqual(self.user_id, denied[0]["user_id"])
self.assertEqual(self.app_uuid, denied[1]["app_registration_uuid"])
self.assertIsNone(self._agent_row())
self._request("POST", USERS_PATH + "/" + self.user_id + "/reinstate", headers=self.admin)
status, body = self._register()
self.assertEqual(201, status)
self.assertTrue(body["created"])
self.assertIsNotNone(self._agent_row())
def test_malformed_bodies_are_400(self):
base = {"agent_id": "a", "app_registration_uuid": self.app_uuid,
"user_id": self.user_id, "tags": ["x"]}
variants = (
{}, {"agent_id": "a"}, {"agent_id": "a", "app_registration_uuid": self.app_uuid},
dict(base, agent_id=None), dict(base, agent_id=123), dict(base, agent_id=""),
dict(base, app_registration_uuid=None), dict(base, app_registration_uuid=[]),
dict(base, user_id=7), dict(base, user_id=""), dict(base, tags=None),
dict(base, tags="x"), dict(base, tags=["a", 1]), dict(base, tags=[["nested"]]),
dict(base, tags={"a": 1}), [1, 2, 3], "nope",
)
for payload in variants:
status, body = self._request("POST", REGISTER_PATH, payload)
self.assertEqual(400, status, repr(payload))
self.assertEqual(400, body["error"])
self.assertTrue(body["message"])
self.assertIsNone(self._agent_row())
self.assertEqual([], self._audit("register"))
def test_first_register_creates_row_pepper_and_default_ttl(self):
status, body = self._register(agent_id="px-1", tags=("proxmox", "readonly"))
self.assertEqual(201, status)
self.assertTrue(body["created"])
self.assertEqual(self._pepper(), body["pepper"])
expected = coreci_broker.derive_token(
self._pepper(), self.app_uuid, self.user_id, "px-1", ["proxmox", "readonly"]
)
self.assertEqual(expected, body["token"])
self.assertEqual(44, len(body["token"]))
self.assertTrue(body["token"].isascii())
delta = self._ttl_delta_seconds(body)
self.assertGreater(delta, DEFAULT_TTL - 300)
self.assertLess(delta, DEFAULT_TTL + 300)
row = self._agent_row("px-1")
self.assertNotEqual(body["token"], row[4])
self.assertTrue(coreci_broker.verify_token(body["token"], row[4]))
self.assertEqual(self.user_id, row[2])
self.assertEqual(["proxmox", "readonly"], json.loads(row[3]))
self.assertEqual(body["token_expires_at"], row[5])
self.assertTrue(row[6])
self.assertTrue(row[7])
entries = self._audit("register", "ok")
self.assertEqual(1, len(entries))
entry = entries[0]
self.assertEqual(self.user_id, entry["user_id"])
self.assertEqual(self.app_uuid, entry["app_registration_uuid"])
self.assertEqual("px-1", entry["agent_id"])
self.assertEqual(REGISTER_PATH, entry["path"])
def test_reregister_unchanged_reissues_same_token_without_rehash(self):
_, first = self._register()
stored_hash = self._agent_row()[4]
seen_at = self._agent_row()[7]
status, second = self._register()
self.assertEqual(200, status)
self.assertFalse(second["created"])
self.assertEqual(first["token"], second["token"])
self.assertEqual(first["token_expires_at"], second["token_expires_at"])
row = self._agent_row()
self.assertEqual(stored_hash, row[4])
self.assertGreaterEqual(row[7], seen_at)
self.assertTrue(coreci_broker.verify_token(first["token"], row[4]))
self.assertEqual(2, len(self._audit("register", "ok")))
def test_reregister_pepper_rule_tokenless_vs_valid_app_token(self):
_, first = self._register()
status, tokenless = self._register()
self.assertEqual(200, status)
self.assertEqual(first["token"], tokenless["token"])
self.assertIn("pepper", tokenless)
status, with_token = self._register(headers=self._app_header(first["token"]))
self.assertEqual(200, status)
self.assertNotIn("pepper", with_token)
self.assertEqual(first["token"], with_token["token"])
status, bad_token = self._register(
headers={"Authorization": "App %s wrong-token" % self.app_uuid}
)
self.assertEqual(200, status)
self.assertIn("pepper", bad_token)
def test_changed_tags_overwrite_hash_token_and_expiry(self):
_, first = self._register(tags=("red",))
status, second = self._register(tags=("red", "blue"))
self.assertEqual(200, status)
self.assertFalse(second["created"])
self.assertNotEqual(first["token"], second["token"])
row = self._agent_row()
self.assertFalse(coreci_broker.verify_token(first["token"], row[4]))
self.assertTrue(coreci_broker.verify_token(second["token"], row[4]))
self.assertEqual(["blue", "red"], json.loads(row[3]))
self.assertEqual(second["token_expires_at"], row[5])
def test_changed_user_overwrites_row_identity(self):
self._request("POST", USERS_PATH, {"user_id": "coo"}, self.admin)
_, first = self._register()
status, second = self._register(user_id="coo")
self.assertEqual(200, status)
self.assertNotEqual(first["token"], second["token"])
row = self._agent_row()
self.assertEqual("coo", row[2])
self.assertFalse(coreci_broker.verify_token(first["token"], row[4]))
self.assertTrue(coreci_broker.verify_token(second["token"], row[4]))
def test_expired_row_reregister_rehashes_and_extends(self):
_, first = self._register()
row = self._agent_row()
old_hash, first_registered = row[4], row[6]
past = (datetime.datetime.now(datetime.timezone.utc)
- datetime.timedelta(seconds=60)).isoformat()
self.broker.store.conn.execute(
"UPDATE agent_registrations SET token_expires_at = ? WHERE agent_id = ?",
(past, "agent-1"),
)
self.broker.store.conn.commit()
status, second = self._register()
self.assertEqual(200, status)
self.assertFalse(second["created"])
self.assertEqual(first["token"], second["token"])
delta = self._ttl_delta_seconds(second)
self.assertGreater(delta, DEFAULT_TTL - 300)
row = self._agent_row()
self.assertNotEqual(old_hash, row[4])
self.assertEqual(first_registered, row[6])
self.assertTrue(coreci_broker.verify_token(first["token"], row[4]))
self.assertEqual(second["token_expires_at"], row[5])
self.assertIn("pepper", second)
def test_revoked_app_blocks_new_registrations_rows_untouched(self):
_, first = self._register()
row_before = self._agent_row()
self._request("DELETE", APPS_PATH + "/" + self.app_uuid, headers=self.admin)
status, body = self._register(agent_id="fresh-agent")
self.assertEqual(403, status)
status, body = self._register()
self.assertEqual(403, status)
self.assertEqual(2, len(self._audit("register", "denied")))
row_after = self._agent_row()
self.assertEqual(row_before[4], row_after[4])
self.assertEqual(row_before[5], row_after[5])
self.assertTrue(coreci_broker.verify_token(first["token"], row_after[4]))
def test_token_ttl_override_honored(self):
env = dict(self.env)
env["BROKER_DB_PATH"] = os.path.join(self.tmp, "ttl.db")
env["BROKER_AUDIT_LOG_PATH"] = os.path.join(self.tmp, "ttl.audit.log")
env["BROKER_TOKEN_TTL"] = "60"
with contextlib.redirect_stdout(io.StringIO()):
broker = coreci_broker.boot(dict(env))
self.addCleanup(broker.audit.close)
self.addCleanup(broker.store.close)
_seed_direct(broker, self.app_uuid, self.user_id)
server = coreci_broker.serve("127.0.0.1", 0, broker)
self.addCleanup(server.shutdown)
self.addCleanup(server.server_close)
base = "http://127.0.0.1:%d" % server.server_address[1]
payload = {"agent_id": "ttl-agent", "app_registration_uuid": self.app_uuid,
"user_id": self.user_id, "tags": []}
status, body = self._request("POST", REGISTER_PATH, payload, base=base)
self.assertEqual(201, status)
delta = self._ttl_delta_seconds(body)
self.assertGreater(delta, 30)
self.assertLess(delta, 90)
status, body = self._request("POST", REGISTER_PATH, payload, base=base)
self.assertEqual(200, status)
delta = self._ttl_delta_seconds(body)
self.assertGreater(delta, 30)
self.assertLess(delta, 90)
def test_user_allowlist_rejects_slash_backslash_and_control_chars(self):
for bad in ("a/b", "a\\b", "a\nb", "a\x00b", "a\x1fb", "a\x7fb"):
status, body = self._request("POST", USERS_PATH, {"user_id": bad}, self.admin)
self.assertEqual(400, status, repr(bad))
self.assertEqual({"error": 400, "message": "invalid user_id"}, body)
status, body = self._request("GET", USERS_PATH, headers=self.admin)
self.assertEqual([self.user_id], [row["user_id"] for row in body["allowlist"]])
def test_bearer_scheme_with_valid_creds_is_401(self):
header = "Bearer " + base64.b64encode(
("%s:%s" % (ADMIN_USER, ADMIN_PASS)).encode("utf-8")
).decode("ascii")
status, body = self._request("GET", WHOAMI_PATH, headers={"Authorization": header})
self.assertEqual(401, status)
self.assertEqual(401, body["error"])
def test_basic_without_colon_is_401(self):
header = "Basic " + base64.b64encode(b"just-a-username").decode("ascii")
status, body = self._request("GET", WHOAMI_PATH, headers={"Authorization": header})
self.assertEqual(401, status)
self.assertEqual(401, body["error"])
def test_basic_garbage_rejected_lowercase_scheme_accepted(self):
status, body = self._request(
"GET", WHOAMI_PATH, headers={"Authorization": "Basic !!!not-base64!!!"}
)
self.assertEqual(401, status)
header = _basic_header(ADMIN_USER, ADMIN_PASS).replace("Basic", "basic")
status, body = self._request("GET", WHOAMI_PATH, headers={"Authorization": header})
self.assertEqual(200, status)
self.assertEqual(ADMIN_USER, body["username"])
if __name__ == "__main__":
unittest.main()
+511
View File
@@ -0,0 +1,511 @@
"""Stub-gitea release-attacher tests (REQ-22, RESEARCH §2.4).
Drives scripts/release.py against an in-process stub gitea
(``_StubGiteaHandler`` on ``127.0.0.1:0`` — the v0.1 ``_StubOwuiHandler``
precedent; fake token, no live-forge writes) plus the sanctioned
secrets loader scripts/_env_secrets.py: create-or-get in both orders,
byte-exact multipart shape, sha256-from-file discipline (G-017),
idempotent re-attach, bounded-retry exhaustion -> RELEASE_PENDING marker,
and file-only token resolution asserted before the first HTTP call.
"""
import contextlib
import hashlib
import importlib.util
import io
import json
import os
import pathlib
import re
import subprocess
import sys
import tempfile
import threading
import unittest
import urllib.parse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
REPO = pathlib.Path(__file__).resolve().parent.parent
RELEASE_PATH = REPO / "scripts" / "release.py"
ENV_SECRETS_PATH = REPO / "scripts" / "_env_secrets.py"
sys.path.insert(0, str(REPO))
import coreci_broker # repo root on sys.path (parser-parity assertion)
def _load_module(name, path):
spec = importlib.util.spec_from_file_location(name, str(path))
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
release = _load_module("release", RELEASE_PATH)
env_secrets = _load_module("env_secrets", ENV_SECRETS_PATH)
OWNER = "coreci"
REPO_NAME = "broker"
TAG = "v0.2.1"
TOKEN = "testtoken123"
ARTIFACT_NAME = "coreci-broker-v0.2.1.pyz"
_RELEASE_BY_TAG_RE = re.compile(
r"^/api/v1/repos/([^/]+)/([^/]+)/releases/tags/([^/]+)$")
_RELEASES_RE = re.compile(r"^/api/v1/repos/([^/]+)/([^/]+)/releases$")
_ASSETS_RE = re.compile(r"^/api/v1/repos/([^/]+)/([^/]+)/releases/(\d+)/assets$")
def _header_params(value):
"""'a; b=\"c\"' -> {'b': 'c'} — Content-Type/Disposition params."""
params = {}
for item in value.split(";")[1:]:
key, _, val = item.strip().partition("=")
if key:
params[key.strip()] = val.strip().strip('"')
return params
def _split_multipart(content_type, body):
"""Minimal multipart parser: boundary from the Content-Type header."""
boundary = _header_params(content_type).get("boundary", "")
parts = []
if boundary:
for element in body.split(("--" + boundary).encode("utf-8"))[1:]:
if element.startswith(b"--"):
continue # closing delimiter
if element.startswith(b"\r\n"):
element = element[2:]
header_blob, separator, content = element.partition(b"\r\n\r\n")
if not separator:
continue
if content.endswith(b"\r\n"):
content = content[:-2] # framing CRLF before the next delimiter
part_headers = {}
for line in header_blob.decode("utf-8", "replace").split("\r\n"):
name, _, value = line.partition(":")
if name:
part_headers[name.strip().lower()] = value.strip()
parts.append((part_headers, content))
return boundary, parts
class _StubGiteaHandler(BaseHTTPRequestHandler):
"""Scriptable gitea stub (RESEARCH §2.1 routes); records everything."""
state = None # scriptable state dict (class attr, bound per server)
requests = None # every request: method, path, headers, body (+extras)
def log_message(self, format, *args):
pass
def _record(self):
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length else b""
record = {
"method": self.command,
"path": self.path,
"headers": {key.lower(): value
for key, value in self.headers.items()},
"body": body,
}
self.requests.append(record)
return record
def _send_json(self, status, payload):
data = json.dumps(payload).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def do_GET(self):
self._record()
parsed = urllib.parse.urlsplit(self.path)
if _RELEASE_BY_TAG_RE.match(parsed.path):
if self.state.get("release_exists"):
self._send_json(200, {"id": self.state.get("release_id", 970)})
else:
self._send_json(404, {"message": "Release not found"})
return
if _ASSETS_RE.match(parsed.path):
self._send_json(200, list(self.state.get("assets", [])))
return
self._send_json(404, {"message": "Not Found"})
def do_POST(self):
record = self._record()
parsed = urllib.parse.urlsplit(self.path)
if _RELEASES_RE.match(parsed.path):
self.state.setdefault("create_calls", []).append(record)
status = self.state.get("create_status", 201)
if status >= 400:
self._send_json(
status,
{"message": self.state.get("create_error", "create failed")})
else:
self._send_json(status, {"id": self.state.get("release_id", 970)})
return
if _ASSETS_RE.match(parsed.path):
self.state.setdefault("attach_calls", []).append(record)
status = self.state.get("attach_status", 201)
if status >= 400:
self._send_json(
status,
{"message": self.state.get("attach_error", "attach failed")})
return
self._describe_attachment(record, parsed)
self._send_json(201, {
"id": 2,
"name": record["attach"]["query_name"],
"size": record["attach"]["size"],
})
return
self._send_json(404, {"message": "Not Found"})
def _describe_attachment(self, record, parsed):
boundary, parts = _split_multipart(
record["headers"].get("content-type", ""), record["body"])
field_name = filename = part_type = None
content = b""
if len(parts) == 1:
part_headers, content = parts[0]
disposition = _header_params(
part_headers.get("content-disposition", ""))
field_name = disposition.get("name")
filename = disposition.get("filename")
part_type = part_headers.get("content-type")
expected = self.state.get("expected_asset_bytes")
record["attach"] = {
"boundary": boundary,
"field_name": field_name,
"filename": filename,
"part_content_type": part_type,
"query_name":
(urllib.parse.parse_qs(parsed.query).get("name") or [None])[0],
"content": content,
"size": len(content),
"content_matches_expected":
None if expected is None else content == expected,
}
class StubGiteaServer:
"""Bound stub server (v0.1 StubOwuiServer pattern)."""
def __init__(self, state):
self.state = state
self.requests = []
handler = type("BoundStubHandler", (_StubGiteaHandler,), {
"state": state,
"requests": self.requests,
})
self.server = ThreadingHTTPServer(("127.0.0.1", 0), handler)
threading.Thread(target=self.server.serve_forever, daemon=True).start()
@property
def base_url(self):
return "http://127.0.0.1:%d" % self.server.server_address[1]
def close(self):
self.server.shutdown()
self.server.server_close()
class ReleaseUploadTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory(prefix="coreci-release-test-")
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.artifact_path = os.path.join(self.tmp, ARTIFACT_NAME)
# Includes every byte value (CR/LF inside) — byte-exact multipart
# transport must survive content that looks like framing.
self.artifact_bytes = (
b"#!/usr/bin/env python3\nstub-zipapp-payload:" + bytes(range(256)) * 2)
with open(self.artifact_path, "wb") as handle:
handle.write(self.artifact_bytes)
self.env_root = os.path.join(self.tmp, "env-root")
os.makedirs(self.env_root)
secrets_fd = os.open(
os.path.join(self.env_root, ".env.secrets"),
os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(secrets_fd, "w", encoding="utf-8") as handle:
handle.write("GITEA_TOKEN={}\n".format(TOKEN))
self.state = {
"release_exists": True,
"release_id": 970,
"assets": [],
"create_status": 201,
"attach_status": 201,
"expected_asset_bytes": self.artifact_bytes,
}
self.stub = StubGiteaServer(self.state)
self.addCleanup(self.stub.close)
def _argv(self, **overrides):
argv = [
TAG, self.artifact_path,
"--base-url", self.stub.base_url,
"--owner", OWNER, "--repo", REPO_NAME,
"--env-root", self.env_root,
]
for key, value in overrides.items():
argv.extend(["--" + key.replace("_", "-"), value])
return argv
def _run(self, **overrides):
stdout, stderr = io.StringIO(), io.StringIO()
with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr):
code = release.main(self._argv(**overrides))
return code, stdout.getvalue(), stderr.getvalue()
def _paths(self, method):
return [record["path"] for record in self.stub.requests
if record["method"] == method]
def test_existing_release_attaches_exact_multipart_shape(self):
code, out, err = self._run()
self.assertEqual(0, code)
self.assertEqual(
["/api/v1/repos/{}/{}/releases/tags/{}".format(OWNER, REPO_NAME, TAG),
"/api/v1/repos/{}/{}/releases/970/assets".format(OWNER, REPO_NAME)],
self._paths("GET"))
self.assertEqual(
["/api/v1/repos/{}/{}/releases/970/assets?name={}".format(
OWNER, REPO_NAME, ARTIFACT_NAME)],
self._paths("POST"))
self.assertEqual([], self.state.get("create_calls", []))
attach = self.stub.requests[-1]
shape = attach["attach"]
self.assertEqual("attachment", shape["field_name"])
self.assertEqual(ARTIFACT_NAME, shape["filename"])
self.assertEqual(ARTIFACT_NAME, shape["query_name"])
self.assertEqual("application/octet-stream", shape["part_content_type"])
self.assertEqual(self.artifact_bytes, shape["content"])
self.assertTrue(shape["content_matches_expected"])
self.assertIn("multipart/form-data; boundary=",
attach["headers"]["content-type"])
self.assertTrue(self.stub.requests)
for record in self.stub.requests:
self.assertEqual("token " + TOKEN, record["headers"]["authorization"])
def test_missing_release_creates_with_notes_then_attaches(self):
self.state["release_exists"] = False
code, out, err = self._run()
self.assertEqual(0, code)
create_calls = self.state.get("create_calls", [])
self.assertEqual(1, len(create_calls))
create = create_calls[0]
self.assertEqual(
"/api/v1/repos/{}/{}/releases".format(OWNER, REPO_NAME), create["path"])
self.assertEqual("application/json", create["headers"]["content-type"])
payload = json.loads(create["body"])
self.assertEqual(TAG, payload["tag_name"])
self.assertEqual(TAG + ": coreci-broker release", payload["name"])
self.assertIs(False, payload["draft"])
self.assertIs(False, payload["prerelease"])
for section in ("## Summary", "## Requirements", "## Verification",
"## CLI install + usage", "## Docker usage"):
self.assertIn(section, payload["body"])
self.assertEqual(1, len(self.state.get("attach_calls", [])))
def test_matching_asset_is_idempotent_skip(self):
self.state["assets"] = [{
"id": 1, "name": ARTIFACT_NAME, "size": len(self.artifact_bytes),
}]
code, out, err = self._run()
self.assertEqual(0, code)
self.assertEqual([], self._paths("POST"))
self.assertEqual(
["/api/v1/repos/{}/{}/releases/tags/{}".format(OWNER, REPO_NAME, TAG),
"/api/v1/repos/{}/{}/releases/970/assets".format(OWNER, REPO_NAME)],
self._paths("GET"))
self.assertEqual([], self.state.get("attach_calls", []))
self.assertEqual([], self.state.get("create_calls", []))
def test_same_name_but_wrong_size_reattaches(self):
self.state["assets"] = [{"id": 1, "name": ARTIFACT_NAME, "size": 1}]
code, out, err = self._run()
self.assertEqual(0, code)
self.assertEqual(1, len(self.state.get("attach_calls", [])))
def test_attach_500_exhausts_retries_and_marks_pending(self):
self.state["attach_status"] = 500
completed = subprocess.run(
[sys.executable, str(RELEASE_PATH)] + self._argv(),
capture_output=True, text=True, timeout=120,
)
self.assertEqual(1, completed.returncode)
self.assertIn(
"RELEASE_PENDING {} {}".format(TAG, self.artifact_path),
completed.stdout)
self.assertEqual(3, len(self.state.get("attach_calls", [])))
self.assertEqual([], self.state.get("create_calls", []))
self.assertIn("500", completed.stderr)
self.assertNotIn(TOKEN, completed.stdout)
self.assertNotIn(TOKEN, completed.stderr)
def test_permanent_attach_error_reports_gitea_body(self):
self.state["attach_status"] = 400
self.state["attach_error"] = "attachment too large"
code, out, err = self._run()
self.assertEqual(1, code)
self.assertEqual(1, len(self.state.get("attach_calls", []))) # 4xx: no retry
self.assertIn("attachment too large", err)
self.assertIn(
"RELEASE_PENDING {} {}".format(TAG, self.artifact_path), out)
def test_sha256_comes_from_file_and_cli_has_no_sha_argument(self):
before = len(self.stub.requests)
with contextlib.redirect_stderr(io.StringIO()):
with self.assertRaises(SystemExit) as raised:
release.main(self._argv() + ["--sha", "deadbeef"])
self.assertEqual(2, raised.exception.code)
with contextlib.redirect_stderr(io.StringIO()):
with self.assertRaises(SystemExit) as raised:
release.main(self._argv() + ["0deadbeef"])
self.assertEqual(2, raised.exception.code)
self.assertEqual(before, len(self.stub.requests)) # refused pre-HTTP
self.state["release_exists"] = False # force create so notes hit the wire
code, out, err = self._run()
self.assertEqual(0, code)
notes = json.loads(self.state["create_calls"][0]["body"])["body"]
expected = hashlib.sha256(self.artifact_bytes).hexdigest()
self.assertIn("sha256:" + expected, notes)
self.assertNotIn("deadbeef", notes)
def test_token_resolves_from_env_file_on_every_request(self):
code, out, err = self._run()
self.assertEqual(0, code)
self.assertTrue(self.stub.requests)
for record in self.stub.requests:
self.assertEqual("token " + TOKEN, record["headers"]["authorization"])
# Structural discipline: the loader + attacher never touch the
# process environment, and the token value is never printed.
for path in (ENV_SECRETS_PATH, RELEASE_PATH):
source = path.read_text(encoding="utf-8")
self.assertNotIn("os.environ", source)
self.assertNotIn("os.getenv", source)
self.assertNotIn(TOKEN, out)
self.assertNotIn(TOKEN, err)
def test_missing_or_empty_token_fails_before_any_http(self):
empty_root = os.path.join(self.tmp, "empty-env-root")
os.makedirs(empty_root)
code, out, err = self._run(env_root=empty_root)
self.assertEqual(2, code)
self.assertEqual([], self.stub.requests) # no HTTP attempted
self.assertIn("GITEA_TOKEN", err)
self.assertNotIn("RELEASE_PENDING", out)
with open(os.path.join(empty_root, ".env.secrets"), "w",
encoding="utf-8") as handle:
handle.write("GITEA_TOKEN=\n")
code, out, err = self._run(env_root=empty_root)
self.assertEqual(2, code)
self.assertEqual([], self.stub.requests)
self.assertIn("GITEA_TOKEN", err)
def test_notes_sections_sha_line_and_docker_usage(self):
self.state["release_exists"] = False
code, out, err = self._run()
self.assertEqual(0, code)
notes = json.loads(self.state["create_calls"][0]["body"])["body"]
for section in ("## Summary", "## Requirements", "## Verification",
"## CLI install + usage", "## Docker usage"):
self.assertIn(section, notes)
expected = hashlib.sha256(self.artifact_bytes).hexdigest()
self.assertIn("sha256:" + expected, notes)
self.assertIn("chmod +x " + ARTIFACT_NAME, notes)
self.assertIn("--version", notes)
self.assertIn("README", notes)
self.assertIn("BROKER_ENV_PATH", notes)
self.assertIn("BROKER_ADMIN_PASS", notes)
self.assertIn("docker pull ", notes)
self.assertIn("/{}/{}:{}".format(OWNER, REPO_NAME, TAG), notes)
self.assertIn("-v broker-state:/state", notes)
class EnvSecretsLoaderTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory(prefix="coreci-envsecrets-")
self.addCleanup(tmp.cleanup)
self.root = tmp.name
def _write(self, relative, content):
path = os.path.join(self.root, relative)
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w", encoding="utf-8") as handle:
handle.write(content)
return path
def test_parse_semantics_mirror_broker_parser(self):
fixture = (
"# comment\n"
"\n"
"GITEA_TOKEN='quoted-single'\n"
"OTHER_TOKEN=\"quoted double\"\n"
"SPACED = spaced value \n"
"PLAIN=unquoted\n"
"EMPTY=\n"
"ONE_QUOTE=\"\n"
"NO_EQUALS_SIGN\n"
"=\n"
)
path = self._write(".env.secrets", fixture)
self.assertEqual(
coreci_broker._parse_env_file(path),
env_secrets._parse_env_file(path))
values = env_secrets.load(self.root)
self.assertEqual("quoted-single", values["GITEA_TOKEN"])
self.assertEqual("quoted double", values["OTHER_TOKEN"])
self.assertEqual("spaced value", values["SPACED"])
self.assertEqual("unquoted", values["PLAIN"])
self.assertEqual("", values["EMPTY"])
self.assertEqual('"', values["ONE_QUOTE"])
self.assertNotIn("NO_EQUALS_SIGN", values)
def test_later_files_win_across_locations_and_patterns(self):
ladder = (
(".env", "GITEA_TOKEN=step1\n"),
(".env.secrets", "GITEA_TOKEN=step2\n"),
(".env.local", "GITEA_TOKEN=step3\n"), # root .env.* glob
(os.path.join(".ciagent", ".env"), "GITEA_TOKEN=step4\n"),
(os.path.join(".ciagent", ".env.secrets"), "GITEA_TOKEN=step5\n"),
(os.path.join(".ciagent", ".env.staging"), "GITEA_TOKEN=step6\n"),
)
for relative, content in ladder:
self._write(relative, content)
self.assertEqual(
content.partition("=")[2].strip(),
env_secrets.load(self.root)["GITEA_TOKEN"])
def test_missing_or_empty_token_raises_clear_error(self):
with self.assertRaises(env_secrets.SecretsError) as caught:
env_secrets.get_token(root=self.root)
self.assertIn("GITEA_TOKEN", str(caught.exception))
self._write(os.path.join(".ciagent", ".env.secrets"), "GITEA_TOKEN=\n")
with self.assertRaises(env_secrets.SecretsError):
env_secrets.get_token(root=self.root)
self._write(os.path.join(".ciagent", ".env.secrets"),
'GITEA_TOKEN=" "\n') # whitespace-only is empty too
with self.assertRaises(env_secrets.SecretsError):
env_secrets.get_token(root=self.root)
self._write(os.path.join(".ciagent", ".env.secrets"),
"GITEA_TOKEN='tok'\n")
self.assertEqual("tok", env_secrets.get_token(root=self.root))
def test_custom_variable_name(self):
self._write(".env", "OTHER_TOKEN=tok\n")
self.assertEqual("tok", env_secrets.get_token("OTHER_TOKEN", root=self.root))
with self.assertRaises(env_secrets.SecretsError) as caught:
env_secrets.get_token("GITEA_TOKEN", root=self.root)
self.assertIn("GITEA_TOKEN", str(caught.exception))
def test_default_root_is_repo_root(self):
self.assertEqual(
os.path.realpath(str(REPO)), os.path.realpath(env_secrets.default_root()))
if __name__ == "__main__":
unittest.main()
+242
View File
@@ -0,0 +1,242 @@
import contextlib
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.request
from http.server import ThreadingHTTPServer
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
def _echo(request):
request.send_json(
200,
{"body": request.body, "query": request.query, "params": request.path_params},
)
def _unauthorized(request):
request.send_error_json(401, "unauthorized")
def _forbidden(request):
request.send_error_json(403, "forbidden")
def _conflict(request):
request.send_error_json(409, "conflict")
def _explode(request):
raise RuntimeError("boom")
class RouterTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.ui_root = os.path.join(self.tmp, "ui")
self.static_root = os.path.join(self.tmp, "static")
os.makedirs(self.ui_root)
os.makedirs(self.static_root)
env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_UI_ROOT": self.ui_root,
"BROKER_STATIC_ROOT": self.static_root,
}
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(env)
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.broker.router.add("POST", "/test/echo/<name>", _echo)
self.broker.router.add("GET", "/test/401", _unauthorized)
self.broker.router.add("GET", "/test/403", _forbidden)
self.broker.router.add("GET", "/test/409", _conflict)
self.broker.router.add("GET", "/test/500", _explode)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.server_close)
self.addCleanup(self.server.shutdown)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
def _request(self, method, path, data=None):
request = urllib.request.Request(self.base + path, data=data, method=method)
try:
with urllib.request.urlopen(request, timeout=5) as response:
return response.status, response.read(), response.headers
except urllib.error.HTTPError as error:
return error.code, error.read(), error.headers
def _request_json(self, method, path, payload=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
status, body, headers = self._request(method, path, data=data)
return status, json.loads(body.decode("utf-8")), headers
def _write(self, directory, name, content):
with open(os.path.join(directory, name), "w", encoding="utf-8") as handle:
handle.write(content)
def test_serve_returns_started_threading_daemon_server(self):
self.assertIsInstance(self.server, ThreadingHTTPServer)
self.assertTrue(self.server.daemon_threads)
self.assertEqual(self.broker, self.server.broker)
def test_healthz(self):
status, payload, headers = self._request_json("GET", "/healthz")
self.assertEqual(200, status)
self.assertEqual({"ok": True}, payload)
self.assertEqual("application/json; charset=utf-8", headers.get("Content-Type"))
def test_error_contract_for_every_code(self):
cases = [
("GET", "/test/401", None, 401),
("GET", "/test/403", None, 403),
("GET", "/test/409", None, 409),
("GET", "/test/500", None, 500),
("GET", "/no/such/route", None, 404),
("POST", "/test/echo/alice", b"{not json", 400),
]
for method, path, data, code in cases:
with self.subTest(code=code):
status, body, headers = self._request(method, path, data=data)
self.assertEqual(code, status)
self.assertEqual(
"application/json; charset=utf-8", headers.get("Content-Type")
)
payload = json.loads(body.decode("utf-8"))
self.assertEqual({"error", "message"}, set(payload))
self.assertEqual(code, payload["error"])
self.assertIsInstance(payload["message"], str)
self.assertTrue(payload["message"])
def test_json_body_query_and_params_roundtrip(self):
payload = {"k": [1, 2, {"z": True}], "s": "værdi", "n": None}
status, body, _ = self._request_json(
"POST", "/test/echo/alice?x=1&tag=a&tag=b", payload
)
self.assertEqual(200, status)
self.assertEqual(
{
"body": payload,
"query": {"x": "1", "tag": "a"},
"params": {"name": "alice"},
},
body,
)
def test_post_with_empty_body_yields_null_body(self):
status, body, _ = self._request_json("POST", "/test/echo/alice")
self.assertEqual(200, status)
self.assertIsNone(body["body"])
def test_unknown_route_is_404_even_with_valid_json_body(self):
status, body, _ = self._request_json("POST", "/api/broker/nope", {"a": 1})
self.assertEqual(404, status)
self.assertEqual({"error", "message"}, set(body))
self.assertEqual(404, body["error"])
def test_method_mismatch_is_404(self):
status, body, _ = self._request_json("DELETE", "/healthz")
self.assertEqual(404, status)
self.assertEqual(404, body["error"])
def test_static_content_types(self):
fixtures = {
"index.html": ("<html><body>ok</body></html>", "text/html; charset=utf-8"),
"app.css": ("body{margin:0}", "text/css; charset=utf-8"),
"app.js": ("console.log(1)", "application/javascript; charset=utf-8"),
"mod.py": ("import os\n", "text/x-python; charset=utf-8"),
"notes.txt": ("hello world", "text/plain; charset=utf-8"),
"data.json": ("{\"k\": 1}", "application/json; charset=utf-8"),
}
for name, (content, _) in fixtures.items():
self._write(self.ui_root, name, content)
for name, (content, content_type) in fixtures.items():
with self.subTest(name=name):
status, body, headers = self._request("GET", "/ui/" + name)
self.assertEqual(200, status)
self.assertEqual(content.encode("utf-8"), body)
self.assertEqual(content_type, headers.get("Content-Type"))
def test_static_serves_client_module(self):
self._write(
self.static_root, "coreci_broker_client.py", "import urllib.request\n"
)
status, body, headers = self._request("GET", "/static/coreci_broker_client.py")
self.assertEqual(200, status)
self.assertEqual(b"import urllib.request\n", body)
self.assertEqual("text/x-python; charset=utf-8", headers.get("Content-Type"))
def test_ui_index_and_extensionless_page(self):
self._write(self.ui_root, "index.html", "<html>index</html>")
self._write(self.ui_root, "login.html", "<html>login</html>")
status, body, _ = self._request("GET", "/ui/")
self.assertEqual(200, status)
self.assertEqual(b"<html>index</html>", body)
status, body, _ = self._request("GET", "/ui/login")
self.assertEqual(200, status)
self.assertEqual(b"<html>login</html>", body)
def test_static_missing_file_is_404_contract(self):
status, body, headers = self._request("GET", "/ui/absent.html")
self.assertEqual(404, status)
self.assertEqual("application/json; charset=utf-8", headers.get("Content-Type"))
self.assertEqual(
{"error": 404, "message": "not found"}, json.loads(body.decode("utf-8"))
)
def test_static_prefix_rejects_non_get(self):
status, body, _ = self._request("POST", "/ui/index.html", data=b"{}")
self.assertEqual(404, status)
self.assertEqual(404, json.loads(body.decode("utf-8"))["error"])
def test_ui_root_attribute_is_configurable(self):
alt = os.path.join(self.tmp, "ui_alt")
os.makedirs(alt)
self._write(alt, "x.html", "alt root")
self.broker.ui_root = alt
status, body, _ = self._request("GET", "/ui/x.html")
self.assertEqual(200, status)
self.assertEqual(b"alt root", body)
def test_traversal_attempts_rejected(self):
attempts = [
"/ui/../etc/passwd",
"/static/%2e%2e/etc/passwd",
"/static/..%2fetc%2fpasswd",
"/static/..%5cetc%5cpasswd",
"/ui/..%5c..%5cetc%5cpasswd",
"/static/%00etc/passwd",
"/static//etc/passwd",
"/ui/./.././../etc/passwd",
]
for path in attempts:
with self.subTest(path=path):
status, body, _ = self._request("GET", path)
self.assertIn(status, (400, 404))
payload = json.loads(body.decode("utf-8"))
self.assertEqual({"error", "message"}, set(payload))
self.assertEqual(status, payload["error"])
def test_symlink_escape_rejected(self):
outside = os.path.join(self.tmp, "outside")
os.makedirs(outside)
self._write(outside, "secret.txt", "top secret")
os.symlink(outside, os.path.join(self.ui_root, "alias"))
status, body, _ = self._request("GET", "/ui/alias/secret.txt")
self.assertEqual(404, status)
self.assertEqual(
{"error": 404, "message": "not found"}, json.loads(body.decode("utf-8"))
)
if __name__ == "__main__":
unittest.main()
+233
View File
@@ -0,0 +1,233 @@
import base64
import contextlib
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.parse
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
REGISTER_PATH = "/api/broker/agents/register"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
AUDIT_PATH = "/api/broker/audit"
SECRETS_PREFIX = "/api/broker/secrets/"
AUTH_401_BODY = {"error": 401, "message": "invalid app credentials"}
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
GITEA_SECRET = {"url": "https://git.coreci.dev", "token": "git-tok"}
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
class _SecretsCase(unittest.TestCase):
creds_json = json.dumps(
{"coreci/proxmox": PROXMOX_SECRET, "coreci/gitea": GITEA_SECRET}
)
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.creds_path = os.path.join(self.tmp, "creds.json")
with open(self.creds_path, "w", encoding="utf-8") as handle:
handle.write(self.creds_json)
env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
"BROKER_CREDS_SOURCE": "file",
"BROKER_CREDS_FILE": self.creds_path,
}
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(env)
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
self.app_uuid = str(uuidlib.uuid4())
self.user_id = "ceo"
self._seed()
self.registered = self._register("proxmox", ["coreci"])
self.app = {"Authorization": "App %s %s" % (self.app_uuid, self.registered["token"])}
def _request(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(self.base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
raw = response.read().decode("utf-8")
return response.status, json.loads(raw) if raw else None
except urllib.error.HTTPError as error:
raw = error.read().decode("utf-8")
return error.code, json.loads(raw) if raw else None
def _seed(self):
status, _ = self._request("POST", APPS_PATH, {"uuid": self.app_uuid}, self.admin)
self.assertEqual(201, status)
status, _ = self._request("POST", USERS_PATH, {"user_id": self.user_id}, self.admin)
self.assertEqual(201, status)
def _register(self, agent_id, tags):
payload = {"agent_id": agent_id, "app_registration_uuid": self.app_uuid,
"user_id": self.user_id, "tags": list(tags)}
status, body = self._request("POST", REGISTER_PATH, payload)
self.assertEqual(201, status)
return body
def _secret(self, suffix, headers=None):
return self._request("GET", SECRETS_PREFIX + suffix,
headers=self.app if headers is None else headers)
def _audit(self, action):
query = urllib.parse.urlencode({"action": action, "limit": "1000"})
status, body = self._request("GET", AUDIT_PATH + "?" + query, headers=self.admin)
self.assertEqual(200, status)
return body["entries"]
def _read_entries(self, result):
return [entry for entry in self._audit("read") if entry["result"] == result]
def _assert_identity(self, entry, path, agent_id="proxmox"):
self.assertEqual("read", entry["action"])
self.assertEqual(path, entry["path"])
self.assertEqual(self.user_id, entry["user_id"])
self.assertEqual(self.app_uuid, entry["app_registration_uuid"])
self.assertEqual(agent_id, entry["agent_id"])
def _ghost_headers(self, agent_id):
ghost = self._register(agent_id, ["coreci"])
return {"Authorization": "App %s %s" % (self.app_uuid, ghost["token"])}
class SecretsReadTests(_SecretsCase):
def test_valid_read_returns_document_and_ok_audit(self):
status, body = self._secret("coreci/proxmox")
self.assertEqual(200, status)
self.assertEqual("coreci/proxmox", body["path"])
self.assertEqual(PROXMOX_SECRET, body["data"])
self.assertRegex(body["version"], r"^[0-9a-f]{8}$")
self.assertIn("updated_at", body)
entries = self._read_entries("ok")
self.assertEqual(1, len(entries))
self._assert_identity(entries[0], "coreci/proxmox")
def test_encoded_slash_path_resolves_to_same_secret(self):
status, body = self._secret("coreci%2Fproxmox")
self.assertEqual(200, status)
self.assertEqual("coreci/proxmox", body["path"])
self.assertEqual(PROXMOX_SECRET, body["data"])
self.assertEqual(1, len(self._read_entries("ok")))
def test_missing_app_header_is_401(self):
status, body = self._secret("coreci/proxmox", headers={})
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
self.assertEqual([], self._read_entries("ok"))
self.assertEqual([], self._read_entries("denied"))
def test_tampered_token_is_401_with_denied_audit(self):
token = self.registered["token"]
twisted = ("x" if token[0] != "x" else "y") + token[1:]
status, body = self._secret("coreci/proxmox", headers={
"Authorization": "App %s %s" % (self.app_uuid, twisted)})
self.assertEqual(401, status)
self.assertEqual(AUTH_401_BODY, body)
entries = self._audit("denied")
self.assertEqual(1, len(entries))
self.assertEqual("denied", entries[0]["action"])
self.assertEqual("denied", entries[0]["result"])
self.assertEqual(self.app_uuid, entries[0]["app_registration_uuid"])
self.assertIsNone(entries[0]["user_id"])
self.assertIsNone(entries[0]["agent_id"])
def test_cross_path_read_is_403_with_denied_audit(self):
status, body = self._secret("coreci/gitea")
self.assertEqual(403, status)
self.assertEqual(403, body["error"])
self.assertTrue(body["message"])
entries = self._read_entries("denied")
self.assertEqual(1, len(entries))
self._assert_identity(entries[0], "coreci/gitea")
def test_out_of_scope_paths_are_403(self):
suffixes = ("coreci/proxmox/extra", "other/proxmox", "coreci", "CORECI/proxmox")
for suffix in suffixes:
with self.subTest(suffix=suffix):
status, body = self._secret(suffix)
self.assertEqual(403, status)
self.assertEqual(403, body["error"])
self.assertEqual(len(suffixes), len(self._read_entries("denied")))
def test_absent_secret_is_404_with_error_audit(self):
status, body = self._secret("coreci/nonexistent",
headers=self._ghost_headers("nonexistent"))
self.assertEqual(404, status)
self.assertEqual({"error": 404, "message": "not found"}, body)
entries = self._read_entries("error")
self.assertEqual(1, len(entries))
self._assert_identity(entries[0], "coreci/nonexistent", "nonexistent")
def test_traversal_payloads_never_reach_source(self):
original = self.broker.creds_source.get_secret
self.addCleanup(setattr, self.broker.creds_source, "get_secret", original)
def detonate(path):
raise AssertionError("creds source reached for %r" % path)
self.broker.creds_source.get_secret = detonate
attempts = ["%2e%2e/", "..%2f", "coreci%5cproxmox", "coreci%00proxmox",
"//coreci/proxmox", "%2e%2e/coreci/proxmox", "coreci/..%2fproxmox"]
for suffix in attempts:
with self.subTest(suffix=suffix):
status, body = self._secret(suffix)
self.assertEqual(403, status)
self.assertEqual(403, body["error"])
self.assertEqual(len(attempts), len(self._read_entries("denied")))
def test_audit_completeness_action_read_covers_all_results(self):
self._secret("coreci/proxmox")
self._secret("coreci/gitea")
self._secret("coreci/nonexistent", headers=self._ghost_headers("nonexistent"))
entries = self._audit("read")
self.assertEqual(3, len(entries))
by_result = {entry["result"]: entry for entry in entries}
self.assertEqual({"ok", "denied", "error"}, set(by_result))
self._assert_identity(by_result["ok"], "coreci/proxmox")
self._assert_identity(by_result["denied"], "coreci/gitea")
self._assert_identity(by_result["error"], "coreci/nonexistent", "nonexistent")
class SecretsSourceErrorTests(_SecretsCase):
creds_json = "{not-json"
def test_source_failure_is_500_contract_with_error_audit(self):
status, body = self._secret("coreci/proxmox")
self.assertEqual(500, status)
self.assertEqual({"error": 500, "message": "creds source unavailable"}, body)
entries = self._read_entries("error")
self.assertEqual(1, len(entries))
self._assert_identity(entries[0], "coreci/proxmox")
if __name__ == "__main__":
unittest.main()
+318
View File
@@ -0,0 +1,318 @@
import base64
import contextlib
import io
import json
import os
import sys
import tempfile
import threading
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
REGISTER_PATH = "/api/broker/agents/register"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
STATE_PATH = "/api/broker/agents/%s/state"
STUB_PREFIX = "/api/v1/broker-agent-state/"
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
class _StubOwuiHandler(BaseHTTPRequestHandler):
documents = None
records = None
def log_message(self, format, *args):
pass
def do_GET(self):
agent_id = self.path[len(STUB_PREFIX):]
self.records.append(("GET", self.path, self.headers.get("Authorization"), None))
if agent_id not in self.documents:
self._send(404, {"error": 404})
return
self._send(200, self.documents[agent_id])
def do_PUT(self):
agent_id = self.path[len(STUB_PREFIX):]
length = int(self.headers.get("Content-Length") or "0")
raw = self.rfile.read(length)
self.records.append(("PUT", self.path, self.headers.get("Authorization"), raw))
self.documents[agent_id] = json.loads(raw.decode("utf-8"))
self._send(200, {"ok": True})
def _send(self, status, payload):
data = json.dumps(payload).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
class StubOwuiServer:
def __init__(self):
self.documents, self.records = {}, []
handler = type("BoundStubHandler", (_StubOwuiHandler,),
{"documents": self.documents, "records": self.records})
self.server = ThreadingHTTPServer(("127.0.0.1", 0), handler)
threading.Thread(target=self.server.serve_forever, daemon=True).start()
@property
def base_url(self):
return "http://127.0.0.1:%d" % self.server.server_address[1]
def close(self):
self.server.shutdown()
self.server.server_close()
class _StateCase(unittest.TestCase):
backend = "internal"
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
"BROKER_AGENT_STATE_BACKEND": self.backend,
}
self.env.update(getattr(self, "extra_env", {}))
self._start_broker()
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
self.app_uuid = str(uuidlib.uuid4())
self.user_id = "ceo"
self._seed(self.app_uuid, self.user_id)
self.registered = self._register("agent-1", ["b", "a"])
self.app = {"Authorization": "App %s %s" % (self.app_uuid, self.registered["token"])}
def _start_broker(self):
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(dict(self.env))
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
def _request(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(self.base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
raw = response.read().decode("utf-8")
return response.status, json.loads(raw) if raw else None
except urllib.error.HTTPError as error:
raw = error.read().decode("utf-8")
return error.code, json.loads(raw) if raw else None
def _state(self, method, agent_id, payload=None, headers=None):
return self._request(method, STATE_PATH % agent_id, payload,
self.app if headers is None else headers)
def _seed(self, app_uuid, user_id):
status, _ = self._request("POST", APPS_PATH, {"uuid": app_uuid}, self.admin)
self.assertEqual(201, status)
status, _ = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
self.assertEqual(201, status)
def _register(self, agent_id, tags):
payload = {"agent_id": agent_id, "app_registration_uuid": self.app_uuid,
"user_id": self.user_id, "tags": list(tags)}
status, body = self._request("POST", REGISTER_PATH, payload)
self.assertEqual(201, status)
return body
class InternalStateTests(_StateCase):
def test_put_then_get_roundtrip_is_document_faithful(self):
state = {"agent_id": "agent-1", "app_registration_uuid": self.app_uuid,
"user_id": self.user_id, "tags": ["b", "a"],
"pepper": self.registered["pepper"],
"current_token": self.registered["token"],
"token_expires_at": self.registered["token_expires_at"],
"first_registered_at": "t0", "last_seen_at": "t1",
"nested": {"flags": [1, 2.5, None, True], "deep": {"k": None}}, "z": 1}
status, body = self._state("PUT", "agent-1", state)
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
status, body = self._state("GET", "agent-1")
self.assertEqual(200, status)
self.assertEqual(state, body)
self.assertEqual(list(state), list(body))
self.assertEqual(state, self.broker.store.get_agent_state("agent-1", self.app_uuid))
def test_state_survives_broker_restart(self):
state = {"counter": 7, "notes": ["x"], "pepper": "keep-me"}
status, _ = self._state("PUT", "agent-1", state)
self.assertEqual(200, status)
self.server.shutdown()
self.server.server_close()
self.broker.store.close()
with contextlib.redirect_stdout(io.StringIO()):
second = coreci_broker.boot(dict(self.env))
self.addCleanup(second.audit.close)
self.addCleanup(second.store.close)
server = coreci_broker.serve("127.0.0.1", 0, second)
self.addCleanup(server.shutdown)
self.addCleanup(server.server_close)
self.base = "http://127.0.0.1:%d" % server.server_address[1]
status, body = self._state("GET", "agent-1")
self.assertEqual(200, status)
self.assertEqual(state, body)
row = second.store.get_agent("agent-1", self.app_uuid)
self.assertTrue(coreci_broker.verify_token(self.registered["token"], row[4]))
def test_identity_mismatch_is_403_on_get_and_put(self):
second = self._register("agent-2", ["coreci"])
for method, payload in (("GET", None), ("PUT", {"who": "agent-2"})):
status, body = self._state(method, "agent-2", payload)
self.assertEqual(403, status, method)
self.assertEqual(403, body["error"])
self.assertTrue(body["message"])
status, body = self._state("GET", "agent-2", headers={
"Authorization": "App %s %s" % (self.app_uuid, second["token"])})
self.assertEqual(404, status)
def test_unregistered_agent_id_is_404_on_get(self):
status, body = self._state("GET", "ghost-agent")
self.assertEqual(404, status)
self.assertEqual({"error": 404, "message": "not found"}, body)
def test_get_without_state_is_404(self):
status, body = self._state("GET", "agent-1")
self.assertEqual(404, status)
self.assertEqual({"error": 404, "message": "not found"}, body)
def test_put_non_dict_is_400(self):
for payload in ([1, 2], "nope", 5, None):
status, body = self._state("PUT", "agent-1", payload)
self.assertEqual(400, status, repr(payload))
self.assertEqual(400, body["error"])
status, _ = self._state("GET", "agent-1")
self.assertEqual(404, status)
def test_put_above_256kib_is_400(self):
status, body = self._state("PUT", "agent-1", {"pad": "x" * (300 * 1024)})
self.assertEqual(400, status)
self.assertEqual(400, body["error"])
status, _ = self._state("GET", "agent-1")
self.assertEqual(404, status)
def test_put_at_exactly_256kib_is_stored(self):
pad = "x" * (256 * 1024 - len(json.dumps({"pad": ""})))
status, body = self._state("PUT", "agent-1", {"pad": pad})
self.assertEqual(200, status)
status, body = self._state("GET", "agent-1")
self.assertEqual(200, status)
self.assertEqual({"pad": pad}, body)
def test_missing_app_header_is_401(self):
status, body = self._state("GET", "agent-1", headers={})
self.assertEqual(401, status)
self.assertEqual({"error": 401, "message": "invalid app credentials"}, body)
class OwuiStateTests(_StateCase):
backend = "owui"
def setUp(self):
self.stub = StubOwuiServer()
self.addCleanup(self.stub.close)
self.extra_env = {
"BROKER_OWUI_BASE_URL": self.stub.base_url,
"BROKER_OWUI_API_KEY": "test-key",
}
super().setUp()
def test_put_then_get_roundtrip_flows_through_stub(self):
state = {"agent_id": "agent-1", "current_token": "tok", "n": 3}
status, body = self._state("PUT", "agent-1", state)
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
self.assertEqual(state, self.stub.documents["agent-1"])
status, body = self._state("GET", "agent-1")
self.assertEqual(200, status)
self.assertEqual(state, body)
puts = [record for record in self.stub.records if record[0] == "PUT"]
gets = [record for record in self.stub.records if record[0] == "GET"]
self.assertEqual(1, len(puts))
self.assertEqual(1, len(gets))
for _, path, auth, _ in self.stub.records:
self.assertEqual("Bearer test-key", auth)
self.assertEqual(STUB_PREFIX + "agent-1", puts[0][1])
self.assertEqual(STUB_PREFIX + "agent-1", gets[0][1])
self.assertEqual(state, json.loads(puts[0][3].decode("utf-8")))
def test_stub_miss_maps_to_broker_404(self):
status, body = self._state("GET", "agent-1")
self.assertEqual(404, status)
self.assertEqual({"error": 404, "message": "not found"}, body)
def test_identity_enforced_before_backend_call(self):
self._register("agent-2", ["coreci"])
status, body = self._state("GET", "agent-2")
self.assertEqual(403, status)
status, body = self._state("PUT", "agent-2", {"x": 1})
self.assertEqual(403, status)
self.assertEqual([], self.stub.records)
class BackendToggleTests(unittest.TestCase):
def _boot(self, extra):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
env = {
"BROKER_ENV_PATH": os.path.join(tmp.name, ".env"),
"BROKER_DB_PATH": os.path.join(tmp.name, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(tmp.name, "broker.audit.log"),
}
env.update(extra)
with contextlib.redirect_stdout(io.StringIO()):
broker = coreci_broker.boot(env)
self.addCleanup(broker.audit.close)
self.addCleanup(broker.store.close)
return broker
def test_default_backend_is_internal(self):
broker = self._boot({})
self.assertIsInstance(broker.agent_state_backend, coreci_broker.InternalAgentStateBackend)
def test_owui_backend_selected_by_config(self):
broker = self._boot({
"BROKER_AGENT_STATE_BACKEND": "owui",
"BROKER_OWUI_BASE_URL": "http://127.0.0.1:9999/",
"BROKER_OWUI_API_KEY": "k",
})
backend = broker.agent_state_backend
self.assertIsInstance(backend, coreci_broker.OwuiAgentStateBackend)
self.assertEqual("http://127.0.0.1:9999", backend.base_url)
self.assertEqual("k", backend.api_key)
def test_unknown_backend_value_falls_back_to_internal(self):
broker = self._boot({"BROKER_AGENT_STATE_BACKEND": "banana"})
self.assertIsInstance(broker.agent_state_backend, coreci_broker.InternalAgentStateBackend)
if __name__ == "__main__":
unittest.main()
+181
View File
@@ -0,0 +1,181 @@
import contextlib
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.request
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
class StaticDisclosureGuardTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_UI_ROOT": self.tmp,
"BROKER_STATIC_ROOT": self.tmp,
}
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(dict(self.env))
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
with open(
os.path.join(self.tmp, "coreci_broker_client.py"), "w", encoding="utf-8"
) as handle:
handle.write("import urllib.request\n")
with open(
os.path.join(self.tmp, ".hidden.css"), "w", encoding="utf-8"
) as handle:
handle.write("body{display:none}")
os.symlink(self.env["BROKER_ENV_PATH"], os.path.join(self.tmp, "env_alias"))
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.server_close)
self.addCleanup(self.server.shutdown)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
def _get(self, path):
try:
with urllib.request.urlopen(self.base + path, timeout=5) as response:
return response.status, response.read()
except urllib.error.HTTPError as error:
return error.code, error.read()
def test_state_files_never_served_when_roots_overlap_state_dir(self):
for path in (
"/static/.env",
"/static/broker.db",
"/static/broker.db-journal",
"/static/broker.db-wal",
"/static/broker.audit.log",
"/ui/.env",
"/ui/broker.db",
"/ui/broker.audit.log",
):
with self.subTest(path=path):
status, body = self._get(path)
self.assertEqual(404, status)
self.assertEqual(
{"error": 404, "message": "not found"}, json.loads(body.decode("utf-8"))
)
def test_symlink_alias_to_env_rejected(self):
status, _ = self._get("/static/env_alias")
self.assertEqual(404, status)
def test_dotfiles_rejected_even_with_known_extension(self):
for path in ("/static/.hidden.css", "/ui/.hidden.css"):
with self.subTest(path=path):
status, _ = self._get(path)
self.assertEqual(404, status)
def test_public_files_still_served(self):
status, body = self._get("/static/coreci_broker_client.py")
self.assertEqual(200, status)
self.assertEqual(b"import urllib.request\n", body)
def test_default_config_relative_paths_do_not_disclose_state(self):
cwd = os.getcwd()
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
os.chdir(tmp.name)
try:
env = {"BROKER_ENV_PATH": os.path.join(tmp.name, ".env")}
with contextlib.redirect_stdout(io.StringIO()):
broker = coreci_broker.boot(dict(env))
server = coreci_broker.serve("127.0.0.1", 0, broker)
base = "http://127.0.0.1:%d" % server.server_address[1]
for path in ("/static/.env", "/static/broker.db", "/static/broker.audit.log"):
with self.subTest(path=path):
try:
with urllib.request.urlopen(base + path, timeout=5) as response:
status = response.status
except urllib.error.HTTPError as error:
status = error.code
self.assertEqual(404, status)
server.shutdown()
server.server_close()
broker.audit.close()
broker.store.close()
finally:
os.chdir(cwd)
class CredsFileDisclosureGuardTests(unittest.TestCase):
SECRET_JSON = json.dumps({"coreci/proxmox": {"token": "pve-tok"}})
def _get(self, base, path):
try:
with urllib.request.urlopen(base + path, timeout=5) as response:
return response.status, response.read()
except urllib.error.HTTPError as error:
return error.code, error.read()
def _write_creds(self, path):
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w", encoding="utf-8") as handle:
handle.write(self.SECRET_JSON)
def test_default_layout_never_serves_creds_file(self):
cwd = os.getcwd()
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
creds_path = os.path.join(tmp.name, "data", "creds.json")
self._write_creds(creds_path)
os.chdir(tmp.name)
try:
with contextlib.redirect_stdout(io.StringIO()):
broker = coreci_broker.boot({"BROKER_ENV_PATH": os.path.join(tmp.name, ".env")})
server = coreci_broker.serve("127.0.0.1", 0, broker)
base = "http://127.0.0.1:%d" % server.server_address[1]
for path in ("/static/data/creds.json", "/static/data%2Fcreds.json"):
with self.subTest(path=path):
status, body = self._get(base, path)
self.assertEqual(404, status)
self.assertNotIn(b"pve-tok", body)
server.shutdown()
server.server_close()
broker.audit.close()
broker.store.close()
finally:
os.chdir(cwd)
def test_creds_file_blocked_when_roots_overlap_creds_dir(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
creds_path = os.path.join(tmp.name, "creds.json")
self._write_creds(creds_path)
env = {
"BROKER_ENV_PATH": os.path.join(tmp.name, ".env"),
"BROKER_DB_PATH": os.path.join(tmp.name, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(tmp.name, "broker.audit.log"),
"BROKER_UI_ROOT": tmp.name,
"BROKER_STATIC_ROOT": tmp.name,
"BROKER_CREDS_FILE": creds_path,
}
with contextlib.redirect_stdout(io.StringIO()):
broker = coreci_broker.boot(dict(env))
self.addCleanup(broker.audit.close)
self.addCleanup(broker.store.close)
server = coreci_broker.serve("127.0.0.1", 0, broker)
self.addCleanup(server.server_close)
self.addCleanup(server.shutdown)
base = "http://127.0.0.1:%d" % server.server_address[1]
for path in ("/static/creds.json", "/ui/creds.json"):
with self.subTest(path=path):
status, body = self._get(base, path)
self.assertEqual(404, status)
self.assertNotIn(b"pve-tok", body)
if __name__ == "__main__":
unittest.main()
+155
View File
@@ -0,0 +1,155 @@
import base64
import contextlib
import io
import os
import sqlite3
import sys
import tempfile
import unittest
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
TABLE_COLUMNS = {
"broker_config": ["key", "value"],
"app_registrations": ["uuid", "name", "status", "created_at", "revoked_at"],
"user_allowlist": ["user_id", "email", "status", "created_at", "revoked_at"],
"agent_registrations": [
"agent_id",
"app_registration_uuid",
"user_id",
"tags",
"token_hash",
"token_expires_at",
"first_registered_at",
"last_seen_at",
],
"admin_sessions": ["session_id", "created_at", "expires_at"],
}
class StoreTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.env_path = os.path.join(tmp.name, ".env")
self.db_path = os.path.join(tmp.name, "broker.db")
def _boot(self, extra_env=None):
env = {"BROKER_ENV_PATH": self.env_path, "BROKER_DB_PATH": self.db_path}
if extra_env:
env.update(extra_env)
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer):
broker = coreci_broker.boot(env)
self.addCleanup(broker.store.close)
return broker, buffer.getvalue()
def _connect(self):
conn = sqlite3.connect(self.db_path)
self.addCleanup(conn.close)
return conn
def test_boot_creates_db_with_mode_0600(self):
broker, stdout = self._boot()
self.assertTrue(os.path.isfile(self.db_path))
self.assertEqual(0o600, os.stat(self.db_path).st_mode & 0o777)
self.assertEqual(1, stdout.count(broker.config["BROKER_ADMIN_PASS"]))
def test_db_mode_0600_independent_of_umask(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
env = {
"BROKER_ENV_PATH": os.path.join(tmp.name, ".env"),
"BROKER_DB_PATH": os.path.join(tmp.name, "broker.db"),
}
old_umask = os.umask(0o777)
try:
with contextlib.redirect_stdout(io.StringIO()):
broker = coreci_broker.boot(env)
self.addCleanup(broker.store.close)
self.assertEqual(0o600, os.stat(env["BROKER_DB_PATH"]).st_mode & 0o777)
self.assertEqual(0o600, os.stat(env["BROKER_ENV_PATH"]).st_mode & 0o777)
finally:
os.umask(old_umask)
def test_all_five_tables_present_with_columns(self):
self._boot()
conn = self._connect()
tables = {
row[0]
for row in conn.execute("SELECT name FROM sqlite_master WHERE type = 'table'")
}
self.assertTrue(set(TABLE_COLUMNS).issubset(tables))
for table, expected in TABLE_COLUMNS.items():
actual = [row[1] for row in conn.execute("PRAGMA table_info(%s)" % table)]
self.assertEqual(expected, actual)
def test_index_on_agent_registrations_app_registration_uuid(self):
self._boot()
conn = self._connect()
names = [
row[0]
for row in conn.execute(
"SELECT name FROM sqlite_master WHERE type = 'index' AND tbl_name = 'agent_registrations'"
)
]
matching = []
for name in names:
columns = [row[2] for row in conn.execute("PRAGMA index_info(%s)" % name)]
if columns and columns[0] == "app_registration_uuid":
matching.append(name)
self.assertTrue(matching)
def test_pepper_synced_into_broker_config(self):
broker, _ = self._boot()
conn = self._connect()
row = conn.execute(
"SELECT value FROM broker_config WHERE key = 'BROKER_INSTANCE_PEPPER'"
).fetchone()
self.assertIsNotNone(row)
self.assertEqual(broker.config["BROKER_INSTANCE_PEPPER"], row[0])
self.assertEqual(32, len(base64.b64decode(row[0])))
def test_pepper_env_override_synced(self):
self._boot({"BROKER_INSTANCE_PEPPER": "Zm9vYmFy"})
conn = self._connect()
row = conn.execute(
"SELECT value FROM broker_config WHERE key = 'BROKER_INSTANCE_PEPPER'"
).fetchone()
self.assertEqual("Zm9vYmFy", row[0])
def test_second_boot_silent_and_idempotent(self):
first, first_stdout = self._boot()
second, second_stdout = self._boot()
self.assertEqual(1, first_stdout.count(first.config["BROKER_ADMIN_PASS"]))
self.assertEqual("", second_stdout)
self.assertEqual(
first.config["BROKER_INSTANCE_PEPPER"],
second.config["BROKER_INSTANCE_PEPPER"],
)
self.assertEqual(
first.config["BROKER_INSTANCE_PEPPER"],
second.store.get_config("BROKER_INSTANCE_PEPPER"),
)
self.assertEqual(0o600, os.stat(self.db_path).st_mode & 0o777)
def test_restart_preserves_rows(self):
broker, _ = self._boot()
broker.store.conn.execute(
"INSERT INTO app_registrations (uuid, name) VALUES (?, ?)",
("0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a", "proxmox-tool"),
)
broker.store.conn.commit()
broker.store.close()
conn = self._connect()
row = conn.execute(
"SELECT name FROM app_registrations WHERE uuid = ?",
("0b6c3370-6d38-4b92-b9a2-bc2f1c9c1b0a",),
).fetchone()
self.assertEqual("proxmox-tool", row[0])
if __name__ == "__main__":
unittest.main()
+182
View File
@@ -0,0 +1,182 @@
import base64
import hashlib
import os
import sys
import tempfile
import unittest
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
PEPPER = "ZpQ1hP3rLmX7wYc2vB4nT6kS9dF5gR8j"
APP_UUID = "3f2a9c1e-77d0-4f0a-9b2e-6c1d5a8e4b3c"
USER_ID = "user-42"
AGENT_ID = "proxmox"
B64_ALPHABET = set(
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/="
)
def _independent_token(pepper, app_uuid, user_id, agent_id, tags):
payload = "{}:{}:{}:{}:{}".format(
pepper,
app_uuid,
user_id,
agent_id,
",".join(sorted(tags)),
)
digest = hashlib.sha256(payload.encode("utf-8")).digest()
return base64.b64encode(digest).decode("ascii")
class FormulaTests(unittest.TestCase):
def test_matches_independent_computation(self):
token = coreci_broker.derive_token(
PEPPER, APP_UUID, USER_ID, AGENT_ID, ["infra", "proxmox"]
)
expected = _independent_token(
PEPPER, APP_UUID, USER_ID, AGENT_ID, ["infra", "proxmox"]
)
self.assertEqual(expected, token)
def test_exact_spec_formula_byte_for_byte(self):
expected = base64.b64encode(
hashlib.sha256(
"{}:{}:{}:{}:{}".format(
PEPPER, APP_UUID, USER_ID, AGENT_ID, "a,b"
).encode("utf-8")
).digest()
).decode("ascii")
actual = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a", "b"])
self.assertEqual(expected, actual)
def test_empty_tags_match_independent_computation(self):
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, [])
self.assertEqual(
_independent_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, []),
token,
)
def test_deterministic_for_same_inputs(self):
first = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
second = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
self.assertEqual(first, second)
class ShapeTests(unittest.TestCase):
def test_token_is_exactly_44_chars(self):
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
self.assertEqual(44, len(token))
def test_token_is_pure_ascii(self):
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
self.assertTrue(token.isascii())
token.encode("ascii")
def test_token_carries_base64_padding_char(self):
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
self.assertTrue(token.endswith("="))
self.assertEqual(1, token.count("="))
def test_token_uses_only_base64_alphabet(self):
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
self.assertTrue(set(token) <= B64_ALPHABET)
def test_token_decodes_to_32_byte_sha256_digest(self):
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
self.assertEqual(32, len(base64.b64decode(token)))
def test_token_has_no_prefix(self):
token = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
for prefix in ("App", "Bearer", "scrypt$", "$2", "token", "tk_", "coreci"):
with self.subTest(prefix=prefix):
self.assertFalse(token.startswith(prefix))
self.assertNotIn(" ", token)
self.assertNotIn(":", token)
self.assertNotIn("$", token)
class TagNormalizationTests(unittest.TestCase):
def test_tag_order_does_not_change_token(self):
first = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["b", "a"])
second = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a", "b"])
self.assertEqual(first, second)
def test_accepts_tuple_and_set_iterables(self):
expected = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a", "b"])
self.assertEqual(
expected,
coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ("a", "b")),
)
self.assertEqual(
expected,
coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, {"a", "b"}),
)
class InputSensitivityTests(unittest.TestCase):
def test_pepper_changes_token(self):
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
other = coreci_broker.derive_token("other-pepper", APP_UUID, USER_ID, AGENT_ID, ["a"])
self.assertNotEqual(base, other)
def test_app_uuid_changes_token(self):
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
other = coreci_broker.derive_token(
PEPPER, "00000000-0000-0000-0000-000000000000", USER_ID, AGENT_ID, ["a"]
)
self.assertNotEqual(base, other)
def test_user_id_changes_token(self):
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
other = coreci_broker.derive_token(PEPPER, APP_UUID, "user-99", AGENT_ID, ["a"])
self.assertNotEqual(base, other)
def test_agent_id_changes_token(self):
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
other = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, "gitea", ["a"])
self.assertNotEqual(base, other)
def test_tags_change_token(self):
base = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
other = coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a", "c"])
self.assertNotEqual(base, other)
class PurityTests(unittest.TestCase):
def test_derivation_writes_nothing_to_disk(self):
cwd = os.getcwd()
with tempfile.TemporaryDirectory() as tmp:
os.chdir(tmp)
try:
coreci_broker.derive_token(PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"])
self.assertEqual([], os.listdir(tmp))
finally:
os.chdir(cwd)
def test_derivation_isolated_from_store_and_audit(self):
with tempfile.TemporaryDirectory() as tmp:
db_path = os.path.join(tmp, "broker.db")
audit_path = os.path.join(tmp, "broker.audit.log")
store = coreci_broker.Store(db_path)
audit = coreci_broker.Audit(audit_path)
try:
token = coreci_broker.derive_token(
PEPPER, APP_UUID, USER_ID, AGENT_ID, ["a"]
)
rows = store.conn.execute(
"SELECT * FROM agent_registrations"
).fetchall()
self.assertEqual([], rows)
finally:
store.close()
audit.close()
with open(audit_path, "r", encoding="utf-8") as handle:
self.assertEqual([], handle.readlines())
self.assertNotIn(token, open(db_path, "rb").read().decode("latin-1"))
if __name__ == "__main__":
unittest.main()