9 Commits

Author SHA1 Message Date
coreci cd6892a5bd docs(P04): complete secrets-client-ecosystem phase
---ci---
phase: 4
milestone: v0.1
status: complete
requirements:
  covered: [REQ-07, REQ-08, REQ-19, REQ-20]
  partial: []
---/ci---
2026-10-10 14:41:45 +00:00
coreci 68d6ba4652 docs(P03): ship checkpoint — v0.1.3 complete
---ci---
phase: 3
milestone: v0.1
status: complete
---/ci---
2026-10-10 13:28:51 +00:00
coreci 2de11e9724 docs(P03): complete tool-auth-plane phase
---ci---
phase: 3
milestone: v0.1
status: complete
requirements:
  covered: [REQ-02, REQ-04, REQ-09, REQ-10]
  partial: []
---/ci---
2026-10-10 13:28:39 +00:00
coreci 4e326c309c docs(P02): ship checkpoint — v0.1.2 complete
---ci---
phase: 2
milestone: v0.1
status: complete
---/ci---
2026-10-10 01:16:35 +00:00
coreci 0b23fbc952 docs(P02): complete admin-plane phase
---ci---
phase: 2
milestone: v0.1
status: complete
requirements:
  covered: [REQ-01, REQ-05, REQ-06, REQ-12, REQ-13]
  partial: []
---/ci---
2026-10-10 01:16:23 +00:00
coreci 76e871dded docs(P01): ship checkpoint — v0.1.1 complete
---ci---
phase: 1
milestone: v0.1
status: complete
---/ci---
2026-10-10 00:00:35 +00:00
coreci 8b92b23a3f docs(P01): complete core-skeleton phase
---ci---
phase: 1
milestone: v0.1
status: complete
requirements:
  covered: [REQ-03, REQ-11, REQ-14, REQ-15, REQ-16, REQ-17, REQ-18]
  partial: []
---/ci---
2026-10-10 00:00:26 +00:00
coreci 357bd22ff2 docs(P00): ship checkpoint — v0.1.0 complete
---ci---
phase: 0
milestone: v0.1
status: complete
---/ci---
2026-10-09 23:12:22 +00:00
coreci 547a7b812d docs(P00): complete pre-execution phase
---ci---
phase: 0
milestone: v0.1
status: complete
requirements:
  covered: []
  partial: []
---/ci---
2026-10-09 23:10:51 +00:00
28 changed files with 383 additions and 5880 deletions
+1 -32
View File
@@ -45,35 +45,4 @@ Two runtime modes share the same file: (a) Open WebUI workspace tool — a `Tool
- The in-memory verified-token cache maps `(uuid, token) -> registration row`; it is populated on first successful slow-hash verification and cleared on restart. SQLite remains authoritative; cache entries are re-validated against row status/expiry on every hit.
- Creds adapter contract: lookup by path string (`coreci/<agent_id>`) → JSON object. FileCredsSource reads a single JSON document mapping paths to secret objects (default for dev/tests, `BROKER_CREDS_FILE`). OwuiApiCredsSource calls the Open WebUI HTTP API with `BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY` and is exercised in production; its exact KB-file layout is resolved in P4 research.
- This file is the init-time snapshot; it is refined at phase boundaries per ci-files discipline.
## Distribution (v0.2 — cli-packaging)
v0.2 adds a distribution layer around the unchanged v0.1 broker: no broker-line changes (`run_standalone`, `boot`, `load_config`, `_parse_env_file` are consumed as-is). Three new components sit outside `coreci_broker.py` and are exercised by the same stdlib-unittest harness (in-process stub servers on ephemeral ports; subprocess boot tests against the built artifact).
### Components (v0.2)
| Name | Description | Boundaries | Depends on |
|------|-------------|-----------|------------|
| CLI artifact build (`scripts/build_cli.py` + `cli/__main__.py`) | Stages `coreci_broker.py` + `coreci_broker_client.py` + `coreci_broker_ui/` + the boot shim + a generated `_cli_version.py`, builds `coreci-broker.pyz` with stdlib `zipapp` (`--compress`, shebang `#!/usr/bin/env python3`), then a deterministic normalization pass (sorted entries, fixed timestamps) for byte-reproducible output; emits `dist/coreci-broker-vX.Y.Z.pyz` + sha256 | Produces the artifact only; never publishes; owns no state; the shim materializes bundled UI/static assets into `<state>/.coreci_broker_assets/<version>/{ui,static}` at boot (temp+`os.replace`, overwrite-per-boot) and sets `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` only when neither process env nor the user `.env` defines them; `--version` is answered from the generated `_cli_version.py` before any broker import | Broker module surface (`run_standalone`, `_parse_env_file`) — import-only, zero edits |
| Release-asset pipeline (`scripts/release.py` + `scripts/_env_secrets.py`) | Builds release notes (milestone summary, phases, verification, CLI install/usage, docker usage, artifact sha256), create-or-get the gitea release by tag, attaches `coreci-broker-vX.Y.Z.pyz` via `POST /api/v1/repos/{owner}/{repo}/releases/{id}/assets` (multipart field `attachment`, `Authorization: token <T>`); idempotent re-attach; bounded retry (3) → `release_pending` escalation on exhaustion — never blocks tag+merge | Owns the forge API conversation for release assets only; token resolved exclusively from `.env*` files via the file-reading loader (never shell env; assert non-empty before any call — empty token → gitea 404 masking, v1.8); does not build the artifact, does not push images | CLI artifact build (attaches its output); gitea API (`/api/v1/...`) |
| Daemonless registry publish client (`scripts/publish_image.py`) | Once per milestone: pull the pinned `python:3.11-slim` base from Docker Hub (anonymous token exchange → OCI index → resolve linux/amd64 child manifest → config + layer blobs), append one deterministic broker layer carrying `/app/coreci-broker.pyz` (tar+gzip, fixed metadata, `mtime=0`), build a new config blob (diff_id appended; Entrypoint `["python3","/app/coreci-broker.pyz"]`, Cmd `[]`, WorkingDir `/state`), and push `git.coreci.dev/coreci/broker:<milestone-release-tag>` + `latest` to the gitea container registry (Basic-auth token exchange at `/v2/token`, `Bearer` JWT on API calls, HEAD-skip + monolithic blob uploads, OCI image manifest PUT; explicit `Location`-following — stdlib urllib does not follow 307/308 on PUT) | No container runtime (D-019); base layer digests reused verbatim, base blobs re-uploaded (no cross-registry mount); records resolved base digests as provenance; failure follows bounded-retry → `release_pending` escalation — never blocks the milestone merge | CLI artifact build (ships its output in the layer); Docker Hub + gitea registry HTTP APIs (OCI dist-spec v2) |
| Dockerfile (documentation) | Self-contained docker-build path for users WITH docker: same base, in-image `zipapp` build of the same sources + shim, same entrypoint/`/state` anchoring | Documentation + alternative build path only; the daemonless client is the executable source of truth for what the pipeline publishes; equivalence with the published image is required, byte-identity is not | CLI artifact build (same staging content, same shim) |
### DataFlow (v0.2 distribution)
1. **CLI boot (packaged)**: `./coreci-broker.pyz` → shim resolves the state dir (`BROKER_ENV_PATH`-anchored; CWD-relative defaults preserved) → materializes UI/static assets into the versioned assets dir → sets `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` (user env/`.env` overrides win) → `coreci_broker.run_standalone()` → first-boot `.env` generation, admin pass printed once, `broker.db`/`broker.audit.log` 0600 in the state dir, UI served at `/ui/`, client at `/static/coreci_broker_client.py` — behavior byte-identical to the unpackaged standalone mode.
2. **Release attachment (per phase ship from the CLI-building phase)**: `scripts/build_cli.py` → reproducible `dist/coreci-broker-vX.Y.Z.pyz` (+sha256) → `scripts/release.py` → gitea create-or-get release by tag → multipart attach → notes body (CLI install/usage + docker usage + digest). Bounded retry; `release_pending` escalation on exhaustion; tag+merge is the completion gate.
3. **Image publish (once per milestone, at the final ship)**: `scripts/publish_image.py` → anonymous Docker Hub pull (`python:3.11-slim` OCI index → amd64 child + blobs) → deterministic broker layer (contains the pyz) + new config → gitea registry auth (`/v2/token`, Basic `<user>:<GITEA_TOKEN>`) → blob upload (HEAD-skip, monolithic POST, explicit Location-following) → manifest PUT `<milestone-release-tag>` then `latest` → provenance digests recorded. Bounded retry; never blocks the milestone merge.
### BuildOrder (v0.2 — appended to the v0.1 order above)
8. CLI build pipeline + boot shim + packaging/boot-parity tests (P1 — REQ-21)
9. Release-asset uploader + daemonless publish client + Dockerfile + stub tests (P2 — REQ-22/REQ-23)
10. Cumulative acceptance (v0.1 suite through the packaged artifact), security review, milestone ship: attach pyz to `v0.2.3`, publish image, merge (P3)
### Notes (v0.2)
- Distribution components hold no secrets: the archive bundles only broker + UI + client code; the image ships no state; forge tokens live in `.env*` files loaded only by `scripts/_env_secrets.py` (blocked from shell env by `config.security.bash_allowlist`).
- All three pipeline scripts take injectable base URLs (stub-testable); live-forge interactions are read-only in tests and attempted-only at ship per the bounded-retry contract.
- Verified environment facts backing this section live in RESEARCH.md v0.2 (gitea 1.26.1 API shapes, OCI index resolution for `python:3.11-slim`, gitea `/v2/token` Basic-auth exchange, zipapp normalization, urllib 307/308 behavior).
- This file is the init-time snapshot; it is refined at phase boundaries per ci-files discipline.
+8
View File
@@ -0,0 +1,8 @@
{
"phase": 4,
"stage": "verify",
"milestone": "v0.1",
"phase_role": "execution",
"attempts": 0,
"updated_at": "2026-10-10T03:45:00Z"
}
+113 -238
View File
@@ -1,254 +1,129 @@
# CIAgent Grill Report
## Run: 2026-10-10 (mode: interactive, focus: v0.2 plan red-team — cli-packaging)
## Run: 2026-10-09 23:10 (mode: interactive, focus: all)
**Supersedes the v0.1 grill in this file** (v0.1 run of 2026-10-09; its full text and
verdicts G-001..G-008 live in git history at `7f7868f` and remain binding where the plan
depends on them: G-005's owner-run post-ship OWUI smoke checklist and G-008's 90-day
adoption criterion carry forward in ROADMAP.md §Post-Milestone Notes; G-007's
forge-repo-creation test was executed at v0.1's P0 ship and is closed). New verdicts
number G-009 onward.
### Verdict: Proceed (confidence: 0.85)
### Verdict (summary): GO-WITH-CONDITIONS (confidence: 0.82)
7 of 9 axes resolved on evidence; the four escalated items (Axes 7, 9) were resolved interactively by the owner on 2026-10-09 — decisions G-005..G-008, zero open escalations.
The plan rests on genuinely strong research — live-verified forge/registry wire shapes, a
working zipapp prototype booted against the real broker sources, zero-broker-lines
verified against the actual file (1390/1400). Red-team findings are concrete and
fixable by task/ship-step edits: one **unexecutable mechanism** (the v0.2.1 attachment
bridge cites a harness path RESEARCH itself proved absent from this environment),
one **ship-sequence inversion** (P3-05 attaches the release before the tag exists —
gitea would mint `v0.2.3` at the wrong commit), one **false-green determinism gap**
(the normalize pass "preserves" umask-derived zip attributes, so must-have 4 passes
while the "clean checkout" exit criterion silently fails cross-machine), and a set of
test-realism holes (subprocess harness with no timeout discipline, missing creds env
in the packaged secret-read, no live read-only validation before the one live publish).
Three forks were resolved interactively with the owner this session (G-011, G-013,
G-014); everything else is bound on evidence below.
### Axis 1 — Business Case
- **Q1**: Is the per-tool credential problem real and top-priority?
- Evidence: PROJECT.md:9,47 (four live creds via KB/env/0600 JSON today; proxmox tool exists now — no revocation, no audit, no allowlist); ROADMAP.md:5 (owner-directed 2026-10-09)
- Answer: real and current — the estate's path has no revocation or audit; owner directed same-day. (0.85)
- **Q2**: Named sponsor who has decided under pressure?
- Evidence: D-007 owner-confirmed §13 defaults @0.95 (ffdc539); D-012..D-015 forced under clarify pressure (53d99e8)
- Answer: CEO-owner is sole sponsor and decision-maker; pressure decisions on the record. (0.85)
- **Q3**: What breaks if cancelled — and is ROI evidence-based?
- Evidence: PROJECT.md:9 (counterfactual explicit); no financial deck exists; cost is tokens only
- Answer: cancellation returns the estate to unaudited per-tool reads (capability loss, not continuity); ROI is operational, measured against the documented counterfactual. (0.80)
---
### Axis 2 — Scope
- **Q1**: Stable, frozen, owned?
- Evidence: REQ-01..REQ-20 frozen from spec v0.5, full traceability (REQUIREMENTS.md:5,78-99); D-008; "No requirement is deferred" (PROJECT.md:27); owner coreci-coo (REQUIREMENTS.md:3)
- Answer: frozen and traced; one acknowledged in-scope unknown (KB file layout → P4, ARCHITECTURE.md:47) with candidate flows pre-verified (RESEARCH.md:55). (0.85)
- **Q2**: Explicitly out of scope?
- Evidence: 12-row table, each row with reason + spec source (REQUIREMENTS.md:57-74)
- Answer: exhaustive; covers the usual silent creep targets (rotation, HA, MFA, federation). (0.90)
- **Q3**: Hidden requirements (security, ops, accessibility)?
- Evidence: security reviews P3-06/P5-02 (PLAN.md:113,145); traversal research (RESEARCH.md:62); D-013 cookie hardening; REQ-15 0600; ops README P5-04
- Answer: security/ops surfaced pre-code; accessibility unstated — immaterial for a single-admin internal UI in v0.1. (0.75)
## Surface 1 — Determinism claims (must-have 4; P1-01)
### Axis 3 — Architecture
- **Q1**: Validated by the builders, not the sellers?
- Evidence: RESEARCH.md §4 source-verified against OWUI source at main (no activate() hook, Valves rule, `__file__` temp-delete; KB endpoints from knowledge.py/files.py); plan-checker PASS_WITH_NOTES (b8122be)
- Answer: yes — the executing pipeline authored and checked the plan against source. (0.80)
- **Q2**: Integration surface bounded?
- Evidence: exactly two external surfaces — OWUI Tools mount (RESEARCH.md:30-38) + OWUI KB read API (RESEARCH.md:40-55); gitea push per sibling convention (RESEARCH.md §3)
- Answer: bounded and source-verified; no third integration exists. (0.80)
- **Q3**: Replaced-system edge cases; inherited debt?
- Evidence: four creds, single host, no volume concern (PROJECT.md:47); spec defects 1-4 found and resolved pre-code (RESEARCH.md §2; D-004/D-005); greenfield repo (acf5557)
- Answer: edge cases (pepper chicken-and-egg, KB-loss, re-reg KeyError) resolved by decision; external debt = OWUI lifecycle quirks, mitigated by the once-per-process guard (ARCHITECTURE.md:7); 800-line budget fails fast via AC14 at P1-08. (0.75)
**Forcing question**: Is two-build sha256 equality actually achievable and testable, and
what does the normalize pass fix vs. miss?
### Axis 4 — People
- **Q1**: Key-person dependency — acceptable?
- Evidence: D-007 CEO sole approver, binding (PROJECT.md:61); tokens re-register on TTL; allowlist changes are the only CEO-gated act
- Answer: accepted — CEO absence stalls new onboarding only, not broker continuity; by design for a v0.1 credential broker on a trusted single host. (0.75)
- **Q2**: Resources allocated; PO with authority?
- Evidence: agent pipeline, sequential, parallelization off (config.json:33-37); guided autonomy, escalation_hooks ["*"] (config.json:5-9)
- Answer: no BAU contention applies; owner-CEO is PO with absolute authority. (0.80)
- **Q3**: Building capability they don't have?
- Evidence: sibling precedent on same pipeline + forge (RESEARCH.md §3); the one novel surface (OWUI mount) is source-researched
- Answer: capability precedented; unproven only in that zero code exists — P1 is the cheap falsifier. (0.70)
- **Evidence** (stdlib source, read this session): `/usr/lib/python3.11/zipapp.py`
builds via `for child in source.rglob('*'): z.write(child, arcname.as_posix())` —
entry order = scandir order, entry mtime = source mtime (`ZipInfo.from_file`), and
`external_attr` = the staging file's stat mode `<< 16`. PLAN P1-01's normalize pass
fixes order (sorted), `date_time=(1980,1,1,0,0,0)`, `ZIP_DEFLATED` — but specifies
"**preserved** `external_attr`". Preserved from what zipapp wrote, i.e. derived from
staging file modes — which are `0666 & ~umask` if staged via plain copy, or
checkout-mode if via `copy2`. Umask 0022 vs 0002 across machines/clones → different
`external_attr` → different sha256, **while must-have 4 (two consecutive builds, same
machine) still passes**. A silent false green against the phase exit criterion
("builds reproducibly from a clean checkout").
- **gzip is not in the pyz path** (zipfile DEFLATE only). Image-layer gzip is
deterministic on-platform: `gzip.compress(..., mtime=0)` routes to
`zlib.compress(wbits=31)` on 3.11 (verified in gzip.py source) — fixed header, fixed
OS byte, stable per zlib build. Scope claim: byte-identity is **same-environment**
(same machine, same zlib); cross-zlib-version identity is neither claimed nor
required by must-have 4.
- **Answer**: achievable and testable as scoped — provided the normalize pass also
fixes what it currently preserves. → **G-009**.
### Axis 5 — Timeline
- **Q1**: Deadline set before or after scope?
- Evidence: spec v0.5 (ffdc539, 22:37) precedes plan (b8122be, 23:02); no external deadline exists anywhere in the corpus
- Answer: spec-first; no reverse-engineered date to fail against. (0.85)
- **Q2**: Critical path; single 3-month pusher?
- Evidence: P1→P5 sequential (ROADMAP.md:24-57); only live-environment unknown is OwuiApiCredsSource KB layout (P4), bounded by FileCredsSource default + CEO-controlled KB structure
- Answer: no 3-month pusher; worst case is bounded P4 adapter rework. (0.75)
- **Q3**: Evidence-based estimates; definition of done?
- Evidence: 30 one-commit tasks, tests ship with task (PLAN.md:12); must_haves frontmatter (PLAN.md:2-7); AC1-AC14 + per-phase exits (ROADMAP)
- Answer: slice-sized with checker sign-off; done is test-enforced, not demo-defined. (0.80)
## Surface 2 — Boot parity via subprocess (P1-02)
### Axis 6 — Budget
- **Q1**: Cost profile and burn?
- Evidence: token/compute only; phase 0 complete in ~30 min wall-clock (5 commits); no cash budget exists
- Answer: no runway question; pipeline runs on owned infra. (0.75)
- **Q2**: Predictable drivers outside the budget?
- Evidence: stdlib-only eliminates licensing (D-009); forge org exists (RESEARCH.md:9); OWUI self-hosted; no pip, no npm
- Answer: none uncovered — the design removes the usual cost drivers by construction. (0.80)
- **Q3**: Contingent on something unhappened?
- Evidence: forge repo creation untested (assumption d), but release_blocking false (config.json:24) and manual forge-UI fallback is trivial
- Answer: no financial contingency; the sole contingency is operational (→ Axis 7). (0.70)
**Forcing question**: What makes the subprocess acceptance flaky, and what's missing?
### Axis 7 — Risks
- **Q1**: Top assumptions, evidence per item?
- (a) bcrypt ships in OWUI runtime + daemon-thread tools allowed: RESEARCH.md §4 source+docs — but validated at `main`, deployed version unpinned; absorbed by dual-mode scrypt fallback (0.70)
- (b) stdlib dev parity with the OWUI runtime: D-001 dual-mode covers both environments (RESEARCH.md:8) (0.80)
- (c) FileCredsSource makes tests hermetic: tempdir + env-var state (RESEARCH.md §6; PLAN P1-01/P4-01) — plan-level, proven at P1 (0.70)
- (d) forge repo creatable with current token: token valid, org exists, repo 404 — creation UNTESTED (RESEARCH.md:9). **→ ESCALATION**
- **Q2**: Dependencies outside the team?
- Evidence: OWUI (host + API + process perms), gitea — all in daily estate use (PROJECT.md:47; RESEARCH.md §3)
- Answer: bounded and precedented. (0.75)
- **Q3**: Single killer risk; pre-mortem?
- Evidence: worst case = mount or adapter fails on the live host → standalone mode is functionally equivalent (ARCHITECTURE.md:7); defect table (RESEARCH.md §2) is the pre-mortem substance
- Answer: no project-killing risk found on evidence — but NO plan task validates against the live OWUI host before ship. **→ ESCALATION** (0.65)
- **Evidence**: zero v0.1 tests use `subprocess`/`Popen` (grep over `tests/`: no hits) —
P1-02 introduces the repo's first subprocess-boot harness. The plan's env enumeration
(`BROKER_BIND`, `BROKER_ENV_PATH/DB/AUDIT`) omits `BROKER_CREDS_FILE`/
`BROKER_CREDS_SOURCE` while requiring "secret-read reuse the v0.1 acceptance request
patterns" — the v0.1 precedent (`tests/test_acceptance_full.py:76-77`) sets both;
`FileCredsSource` on a missing file raises `_SecretNotFound` → 404
(`coreci_broker.py:440-441`), so the packaged secret-read would fail. `Popen(["python3",
...], env={minimal})` leaves the child without `PATH` (works today via execvp's
confstr fallback — by accident). No timeout discipline on "read stdout lines until
`coreci-broker listening on`" (RESEARCH §4.3) — a failed boot never prints the line and
hangs the suite. "Pass printed exactly once" is asserted against a *still-running*
server; shutdown-path output is unread unless drained after terminate. And every
planned boot pins absolute state paths — the owner's literal happy path (download →
`python3 *.pyz` in some directory → `.env`/db/audit materialize in the CWD) is never
tested.
- **Answer**: harden the harness — the plan's claims are all testable once the harness
is honest about them. → **G-010**.
### Axis 8 — Governance
- **Q1**: Decision-maker under disagreement?
- Evidence: single owner-CEO; guided autonomy escalates every gate (ROADMAP.md:9; config.json:6)
- Answer: no committee exists to deadlock; owner is arbiter. (0.85)
- **Q2**: Cadence, escalation, omissions from status?
- Evidence: ---ci--- blocks with confidence on all four commits (ffdc539..b8122be); 15 decisions recorded; checker notes published, not buried
- Answer: decision log is exemplary; the mild contradiction is RESEARCH.md:82 "no genuine blockers" sitting beside untested assumption (d). (0.70)
- **Q3**: Stop-the-project trigger?
- Evidence: MISSING as a formal trigger — but sponsor = approver = decision-maker is one person
- Answer: implicit owner prerogative suffices where one person holds all three roles. (0.70)
## Surface 3 — One-shot lead upload bridge (P1-03)
### Axis 9 — Adoption
- **Q1**: Who uses it; what's in it for them?
- Evidence: CEO gets allowlist + audit UI replacing blind KB edits; tool authors get a zero-dep client replacing hand-rolled 0600 JSON (REQ-19; PROJECT.md:9)
- Answer: both user classes shed concrete work — not a "train them and pray" case. (0.75)
- **Q2**: Ops handoff; rollback plan?
- Evidence: ops = owner; README + backup guidance P5-04 (PLAN.md:150); full state = broker.db + broker.audit.log + .env (PROJECT.md:37-39); unmigrated tools never depend on the broker
- Answer: rollback = stop process, restore three files; migrated proxmox tool reverts to direct reads. (0.75)
- **Q3**: Success criteria validated with the judges?
- Evidence: judge = owner via AC1-AC14 (spec §12, owner-owned); but every OWUI-facing test is stub-driven (PLAN P3-04, P4-01), no task touches the live host, and proxmox migration is "post-broker", dateless (PROJECT.md:47)
- Answer: acceptance is owner-validated; live readiness and the adoption criterion were escalated and are now resolved as G-005/G-008. (0.85)
**Forcing question**: Is the v0.2.1 ad-hoc attachment a discipline violation, and is it
even executable as specified?
### Meta
- **Auditor's flag**: no live-host validation before ship; forge repo creation untested; deployed OWUI version unpinned; adoption criterion undefined. All cheap to fix; none fatal.
- **Not being done**: live smoke test on the OWUI host; OWUI version pin; a risk register; post-ship adoption criteria.
- **Simplest 80% version**: drop the admin UI + reference tools (CLI allowlists, file creds only) saves ~one phase — but the UI is the CEO's core value and OwuiApiCredsSource is the production path; at 30 tasks / 800 lines, the over-scope charge does not stick.
- **90-day must-be-true**: broker live on the OWUI host; proxmox tool on the client module; allowlist + audit in CEO daily use. Not true today (no code) — but every gating factor is internal.
- **Evidence**: PLAN P1-03 attaches via "the same harness mechanism v0.1 used for its
`broker-vX.Y.Z.tar.gz` assets" — but RESEARCH §2.2 (the plan's own research base)
found that mechanism is the ship harness's `coreci build/package/release upload` CLI,
**"not installed in this dev environment"**; v0.1's archives were produced
container-side, ad-hoc, with no repo-owned script. The bridge as written is therefore
unexecutable via the cited mechanism, and its realistic substitute is an untested
hand-rolled live-forge write (multipart encoding, token handling, idempotency) at
first live use — performed by lead-developer, the one persona barred from code
writes, using token discipline that `scripts/_env_secrets.py` (which does not exist
until P2) is supposed to enforce. This is precisely the practice REQ-22 formalizes
away.
- **Owner decision (this session)**: restructure — release automation moves into P1;
the bridge is deleted. → **G-011**. The hard gate question dissolves: v0.2.1 ships
script-driven; there is no bridge to become permanent.
### Binding Decisions
| ID | Axis | Decision | Confidence |
|----|------|----------|------------|
| G-001 | 4 | Key-person dependency accepted for v0.1: sole-approver CEO absence stalls new onboarding only, not continuity (D-007 by design) | 0.75 |
| G-002 | 7 | Assumptions (a)-(c) accepted as mitigated-by-design (dual-mode hashing, hermetic test plan); (d) + live validation escalate | 0.70 |
| G-003 | 9 | Rollback accepted: full state = broker.db + broker.audit.log + .env; stop-and-restore; tools revert to direct reads | 0.75 |
| G-004 | 3 | 800-line budget risk accepted: AC14 enforced from P1-08 (fails fast); spec §12.14 amendment is the owner's pressure valve | 0.70 |
| G-005 | 9 | Post-ship live-OWUI smoke checklist (owner-run: mount, register, read) is the plan of record; v0.1 ships on standalone + stub tests; no real tool migrates before the checklist passes; rollback = disable the tool | 0.85 |
| G-006 | 7 | Deployed OWUI pinned at v0.11.3 (owner-stated 2026-10-09) — same-era as the researched `main` surface, no re-research required; smoke checklist item 1 re-confirms the two KB read endpoints on v0.11.3 | 0.85 |
| G-007 | 7 | Forge repo creation is tested at the P0 ship (POST /orgs/coreci/repos with the existing token), not deferred to P5-06; on scope failure: local-only fallback + auto-escalation, and the owner's 30-second manual forge-UI creation is the recorded fallback | 0.9 |
| G-008 | 9 | 90-day adoption criterion: proxmox_readonly_tool.py runs on coreci_broker_client.py with direct KB reads removed by 2027-01-09; owner owns the date; recorded in ROADMAP as a post-milestone note (out of v0.1 scope) | 0.8 |
## Surface 4 — Daemonless OCI publish (P2-02, P3-05)
**Forcing question**: Does the milestone ship sequence hold up when publish fails after
the tag exists — and does it even run in the right order?
- **Evidence — sequence inversion**: P3-05 as written runs `release.py` attach →
publish image → squash-merge → `main` → tag `v0.2.3` → push. But `POST /api/v1/repos/
{o}/{r}/releases` with a missing tag creates the tag at `target_commitish`
(default-branch head) — gitea's documented CreateReleaseOption semantics (schema in
RESEARCH §2.1). A create-first flow pre-merge would mint `v0.2.3` pinned to
**pre-merge main**, then collide with (or silently diverge from) the harness's own tag
push. The plan's own P2-04 sequences correctly (tag → merge → push → release) — P3-05
alone inverts it.
- **Evidence — wire discipline**: bounded retry(3) is specified but no per-request
timeout anywhere; a hung 29.8 MB monolithic POST through a throttled gitea turns
"bounded retry" into an unbounded hang. The 307/308 stub test proves only
Location-following, and gitea's `Location` may be **relative** (RESEARCH §3.3) — the
harder case is untested. Upstream `3.11-slim` drift is handled (resolve-at-publish +
`--base-manifest` + provenance digests; RESEARCH §3.7) and the anonymous-200 token
trap fails loudly per spec.
- **Evidence — failure story**: "publish never blocks merge" is honored, but when
publish fails after `v0.2.3` exists, the plan records only `release_pending` — no
named post-milestone re-run/closure step. The ship contract makes the escalation
auto-resolution; somebody must own the retry.
- **Answer**: reorder P3-05, add timeouts + relative-Location coverage, and make the
escalation carry its own closure runbook. → **G-012**, **G-019**, **G-020**.
## Surface 5 — Scope discipline (P2-03, REQ-22/23)
**Forcing question**: Is anything decorative, and is anything REQ-22/23 implies missing?
- **Evidence — decorative risk**: P2-03's verify is presence-only ("`Dockerfile` +
README sections present") and the dev environment cannot build the Dockerfile (no
runtime) — a syntactically or semantically broken Dockerfile passes the gate. The
"equivalence mapping" header is documentation with no enforcement.
- **Evidence — missing**: no `.dockerignore` anywhere in the plan — `docker build .`
ships the whole repo (`.git/`, any local `.env*`, `data/`, `*.db`, `dist/`,
`__pycache__/`, `.ciagent/`) as build context, adjacent to a security review that
asserts "no secrets in any image layer". REQ-22's notes model is covered (§2.3, stub
asserts sections present); the sha256's *source* is unspecified (hand-pasted vs.
computed); `latest` semantics are coherent (once per milestone at final ship —
v0.2.3 is the first image ever, D-021); the state-dir volume contract is documented
(`WORKDIR /state`, README docker run).
- **Answer**: make the Dockerfile contract testable, add `.dockerignore`, pin digest
provenance. → **G-015**, **G-016**, **G-017**.
## Surface 6 — Frozen broker / shim dependency risk (P1-01)
**Forcing question**: Does the shim need broker cooperation, and what is the coupling
risk?
- **Evidence**: RESEARCH §1.2's prototype booted from the real sources with zero broker
cooperation — re-verified against the file this session: `run_standalone`
(`coreci_broker.py:1369`), `_parse_env_file` (:154), `boot` (:598), `load_config`
(:240) are module-level; `load_config` precedence (process env → `.env` → defaults,
:246-257) makes the shim's pre-set `os.environ` approach contract-identical. The
broker's `.env` resolution is CWD-relative by default (`ENV_PATH_DEFAULT=".env"`,
`ENV_DEFAULTS` :53-66) — packaged boots with unset `BROKER_ENV_PATH` anchor state to
the CWD, which is exactly the documented happy path and is currently untested (folded
into G-010). Residual risk: the shim calls `_parse_env_file` — an underscore-private
symbol. Within v0.2 the broker is frozen so the risk is nil; across future milestones
a rename breaks the shim with a loud ImportError (acceptable), but only if somebody
knows to look.
- **Answer**: no cooperation needed (verified twice); the private-API dependency must
be documented and review-tracked, and the default-anchoring boot must be tested. →
**G-018** (anchor case bound under G-010).
## Surface 7 — Test realism: stub fidelity (P2-01/P2-02 tests)
**Forcing question**: What would a stub MISS that the real forge does — and is the
as-built client ever validated against the real forge before its one live write?
- **Evidence**: the stub coverage is unusually good (anonymous-200 trap, index→amd64
resolution with Accept check, HEAD-skip, monolithic+two-step uploads, 307-following,
digest and diff_id assertions, both tags; multipart field/`?name=` byte assertions,
idempotent re-attach, 500×3 marker, token discipline). Gaps: fixtures are frozen at
research-time shapes (2026-10-10) while upstream index and forge auth can move
mid-milestone; the 307 test doesn't pin the relative-`Location` case (bound in
G-019); gitea's 4 MB attachment cap is out-of-band (pyz ≈30 KB — immaterial, noted);
and **no step anywhere validates the as-built scripts against the real
`git.coreci.dev` in read-only mode** — the first live run of `publish_image.py` is
the P3-05 milestone publish itself, the one shot per milestone.
- **Owner decision (this session)**: add an env-gated, read-only live smoke as a
P3-05 pre-ship verify step. → **G-013**.
## Surface 8 — Wave ordering & effort (P2 packing)
**Forcing question**: Is P2 overloaded, and are there hidden dependencies that break
the wave order?
- **Evidence**: original P2 packs `_env_secrets` + `release.py` + `publish_image.py` +
`Dockerfile` + README + 2 test files into 3 waves — the overload is real, and the
bridge finding (Surface 3) is the reason it dissolves: release automation belongs in
P1 (G-011), leaving P2 = publish client → Dockerfile/docs → ship. Hidden
dependencies check out: `publish_image.py` imports `_env_secrets` (lands earlier in
P1 now — order preserved transitively); the Dockerfile task consumes P1-01's staging
allowlist (G-016 formalizes the contract as a test). `publish_image.py` remains the
plan's biggest single commit even after the restructure.
- **Owner decision (this session)**: keep it one vertical task — the pull/push seam is
artificial, the stub fixtures are shared end-to-end, RESEARCH §3.4 is near-pseudocode,
and a split would not isolate the actual risk (first live write — covered by G-013).
→ **G-014**.
---
## Binding Decisions
| ID | Surface | Decision | Binds | Confidence |
|----|---------|----------|-------|------------|
| G-009 | 1 | Normalize pass must SET constant `external_attr` (files `0o100644<<16`, dir entries `0o40755<<16`), not "preserve" staging-derived modes; staging is an explicit allowlist (`coreci_broker.py`, `coreci_broker_client.py`, `coreci_broker_ui/**`, shim staged as root `__main__.py`, generated `_cli_version.py`) — never a repo rglob, `__pycache__`/`dist/` excluded by construction. Byte-identity scope = same environment (same zlib/machine); cross-zlib identity is not claimed. | P1-01 task edit | 0.85 |
| G-010 | 2 | P1-02 harness hardening: subprocess boots use `sys.executable` (absolute); child env adds `BROKER_CREDS_SOURCE=file` + `BROKER_CREDS_FILE=<tmp fixture>` (v0.1 precedent test_acceptance_full.py:76-77); deadline-bounded wait for the listening line (~30s → fail with captured stdout+stderr, terminate child); pass-once/silent-second-boot assertions completed by draining stdout after terminate; ADD a default-anchoring boot case (`cwd=<tmp>`, no state overrides beyond `BROKER_BIND`) asserting `.env`/`broker.db`/`broker.audit.log` in the CWD — the owner's literal happy path. | P1-02 task edit | 0.85 |
| G-011 | 3 | **Restructure (owner, this session)**: move `scripts/_env_secrets.py` + `scripts/release.py` + `tests/test_release_upload.py` into Phase 1 as new wave 3 (P1-03 · release-asset automation); old P1-03 ship renumbers P1-04 and ships v0.2.1 script-driven (`build_cli.py v0.2.1` → `release.py v0.2.1 <pyz>`); the one-shot lead-upload note is DELETED (its cited mechanism is absent from this environment — RESEARCH §2.2). P2 renumbers: publish_image.py → P2-01 (wave 1), Dockerfile+docs → P2-02 (wave 2), ship → P2-03 (wave 3). Four-test-file count unchanged. | PLAN edit (before P1-01 executes) | 0.90 |
| G-012 | 4 | P3-05 sequence corrected: squash-merge → `main` → tag `v0.2.3` → push → THEN `release.py` attach (GET-by-tag finds the real tag; no pre-merge release create) → publish image → closeout. Mirrors P2's own correct ship order; eliminates the gitea create-tag-at-target_commitish hazard. | P3-05 ship-step edit | 0.80 |
| G-013 | 7 | **Env-gated live read-only smoke (owner, this session)**: pre-publish P3-05 verify step — gitea `GET /v2/` 401 challenge shape, `GET .../releases/tags/{tag}`, `GET /v2/token` Basic → non-empty JWT, Docker Hub anonymous token + `3.11-slim` index→amd64 resolve. Implemented as a `--check`/verify mode on the scripts or `scripts/verify_endpoints.py`; GET-only, never in the default unittest run; tokens only via `_env_secrets`. | P3-05 verify + task edit | 0.85 |
| G-014 | 8 | **publish_image.py stays one vertical task (owner, this session)**: shared stub fixtures, RESEARCH §3.4 near-pseudocode; a split would not isolate the real risk (first live write — G-013 covers it). | PLAN shape (no edit) | 0.75 |
| G-015 | 5 | `.dockerignore` added to the Dockerfile task: excludes `.git`, `.env*`, `data/`, `*.db`, `*.log`, `dist/`, `__pycache__`, `tests/`, `.ciagent/`. Build-context hygiene; P3-02 scope gains "no secrets in the build context" alongside "none in any layer". | P2-02 task edit | 0.85 |
| G-016 | 5 | Dockerfile equivalence made testable: the Dockerfile task adds a consistency test (into `tests/test_build_cli.py`) asserting COPY source set == `build_cli.py` staging allowlist, identical zipapp flags (`--python "/usr/bin/env python3" --compress`), identical `_cli_version.py` generation format, `ENTRYPOINT`/`WORKDIR` == publish_image's `config.Entrypoint`/`config.WorkingDir`. Presence-only verification no longer gates a possibly-broken Dockerfile. | P2-02 task edit | 0.80 |
| G-017 | 5 | Digest discipline: the sha256 in release notes is computed by `release.py` from the artifact file itself — never a CLI argument, never hand-pasted. Digest/asset mismatch becomes structurally impossible. | P1-03 task edit | 0.85 |
| G-018 | 6 | Shim private-API dependency tracked: `cli/__main__.py` header documents the coupling to `coreci_broker._parse_env_file` + `run_standalone` ("private-API dependency — any future broker refactor must re-run the v0.2 packaged acceptance"); P3-02 review scope lists it. Loud-ImportError breakage accepted. | P1-01 task edit + P3-02 scope | 0.75 |
| G-019 | 4 | Publish-client wire hardening: explicit per-request timeouts on every registry call (generous on blob uploads, ~120s); blob-level bounded retry with HEAD-skip between attempts (a timed-out 29.8 MB POST resumes cheap); the 307 stub test uses a RELATIVE `Location` at least once (gitea may answer relative — RESEARCH §3.3; absolute is the easy case). | P2-01 task edit | 0.80 |
| G-020 | 4 | Publish-failure closure is a named step, not an open escalation: the `release_pending` entry records the exact re-run command (`python3 scripts/publish_image.py` on `main` @ `v0.2.3`; `--base-manifest` optional pin) and its close condition (both tags visible in gitea packages). "Never blocks merge" unchanged. | P3-05 ship-step edit | 0.80 |
| G-021 | 4 | The `v0.2.0` (phase-0) release carries NO `.pyz` and its notes must not promise one — D-020 starts at `v0.2.1` (build tooling first exists in P1). Prevents a D-020 misread at the phase-0 ship immediately following this grill. | Phase-0 ship note | 0.85 |
## Escalations
- (none open — the three material forks were resolved interactively with the owner this
session: G-011 restructure, G-013 live smoke, G-014 task packing. G-001..G-008 from
v0.1 remain binding per git history; G-005/G-008 carry forward in ROADMAP post-milestone
notes unchanged.)
## Promotion note (explicit, not performed by the grill)
G-011 shifts REQ-22's attach automation into Phase 1; ROADMAP.md's Phase 2 description
("release-asset automation" wording) and Phase 1's requirements line need a one-line
touch-up when the PLAN restructure is promoted — by the orchestrator/owner via the
normal plan-revision path, not by this file. Nothing else in PROJECT/REQUIREMENTS/ROADMAP
is affected by these verdicts.
## Overall Verdict
**GO-WITH-CONDITIONS** — confidence 0.82.
The plan is research-grounded to an unusual degree (live wire shapes, working prototype,
zero-broker-lines verified against the file), and every finding is actionable by task or
ship-step edit. It is GO only with all of the following applied:
1. **G-011 restructure applied to PLAN.md before P1-01 executes** — the bridge is
unexecutable as written; v0.2.1 must ship script-driven.
2. **G-009 + G-010 applied to P1-01/P1-02** — determinism must fix (not preserve)
entry attributes; the subprocess harness needs interpreter-path, creds-env, timeout,
drain, and default-anchoring discipline.
3. **G-012 + G-020 applied before P3-05** — attach-after-tag ordering and the
publish-failure closure runbook are ship-contract corrections, not optional polish.
4. **G-013 live smoke wired as the P3-05 pre-ship verify step** — the one live publish
must not also be the first live wire contact of the as-built client.
5. **G-015/016/017/018/019 applied with their tasks** — context hygiene, testable
Dockerfile contract, digest provenance, tracked private-API coupling, wire timeouts.
6. **G-021 noted at the phase-0 ship** — v0.2.0 carries no `.pyz`.
### Escalations
- (none — all four escalated challenges resolved interactively by the owner on 2026-10-09; decisions G-005..G-008 above)
+27 -42
View File
@@ -1,28 +1,39 @@
# PERSONAS — CoreCI Broker
Execution-stage roster (2026-10-10, milestone v0.2 cli-packaging). Pure stdlib Python 3.11 project: no package.json, no JS framework, no pip, no npm, no container runtime in the dev environment — frameworks are stdlib modules; tests are stdlib unittest against in-process stub HTTP servers and subprocess boots. The v0.1 broker is FROZEN this milestone: `coreci_broker.py` stands at 1390/1400 lines (REQ-17/AC14) and v0.2 adds zero broker lines (verified, RESEARCH §4.2), so no persona holds `coreci_broker.py` — or the v0.1 UI/client/reference sources — as writable territory; a P0 defect there is a lead-developer-dispatched exception only. All v0.2 work is new files: `cli/` boot shim, `scripts/` build+release+publish tooling, `Dockerfile`, four new test files, additive README sections, `.gitignore` additions. The custom release-engineer persona owns the whole distribution pipeline; backend-engineer — v0.1's primary persona — is deactivated with the P0-dispatch exception documented below. Execution is sequential (parallelization off), so phase-specific personas activate only at their phase boundaries per this roster.
RESEARCH-stage roster (2026-10-09, milestone v0.1). Pure stdlib Python 3.11 project: single-file broker, no package.json, no JS framework, no src/ tree; the only UI is server-rendered static HTML/CSS/JS from `coreci_broker_ui/` (P2). Execution is sequential (parallelization off), so phase-specific personas activate only at their phase boundaries per this roster.
---
name: release-engineer
domain: distribution
name: backend-engineer
domain: backend
active: true
phase_specific: false
reason: Primary execution persona — v0.2 is distribution engineering on a frozen broker, a domain distinct from v0.1's broker application work; hence a custom persona rather than a fold into backend-engineer, whose identity, constraints, and territory are the broker's runtime contract — a poor fit for build/release/publish tooling and a structural guarantee that the frozen file is owned by nobody. Owns the full pipeline — P1 zipapp build + materialization boot shim (REQ-21), P2 gitea release-asset automation + daemonless OCI registry publish + Dockerfile (REQ-22/23), P3 ship — and every security-critical surface this milestone: secrets discipline (file-only token loader), artifact integrity (no secrets in the .pyz or image layers, reproducible digests), registry auth flows.
frameworks: [python3.11, zipapp, zipfile, tarfile, gzip, hashlib.sha256, urllib.request, json, subprocess, http.server, unittest]
constraints: ["coreci_broker.py is FROZEN — zero lines added (1390/1400, REQ-17/AC14); broker sources (coreci_broker.py, coreci_broker_client.py, coreci_broker_ui/**, reference_tools/**) are read-only staging inputs to the build, never edited; a P0 defect there routes through lead-developer dispatch", "stdlib-only build path — python3 -m zipapp --compress; no pip, no npm, no wheel/PyInstaller, no container runtime (D-009/D-019)", "secrets discipline — GITEA_TOKEN resolves ONLY via scripts/_env_secrets.py reading .env/.env.secrets/.env.* files, never os.environ (blocked_env_vars: GITEA_TOKEN); assert non-empty before the first API call (v1.8 lesson: empty token → gitea 404 masks auth failure); no secrets ever enter the .pyz or an image layer", "digest determinism — zip entries sorted with date_time (1980,1,1,0,0,0) via the normalize post-pass; tar entries sorted with fixed TarInfo metadata; gzip.compress(mtime=0); json.dumps(sort_keys=True, separators=(',',':')); two-build sha256 equality gates the artifact (REQ-21)", "OCI end-to-end — manifest application/vnd.oci.image.manifest.v1+json, config application/vnd.oci.image.config.v1+json, layers application/vnd.oci.image.layer.v1.tar+gzip; base layer descriptors reused verbatim; never push an index; urllib does NOT follow 307/308 on PUT/POST — implement explicit Location-following; gitea's anonymous-200 /v2/token answer must fail loudly; HEAD-before-upload makes re-runs idempotent", "bounded retry (3 attempts) on release attach and publish; on exhaustion exit non-zero with a machine-readable marker → ship harness records release_pending escalation; never blocks tag+merge", "stub-testability — scripts take base URLs as parameters (no hardcoded hosts); stubs bind ('127.0.0.1', 0) per the v0.1 _StubOwuiHandler precedent; no live-forge writes in tests; v0.1 test files are frozen reference — v0.2 adds test_cli_packaging.py, test_build_cli.py, test_release_upload.py, test_publish_image.py only, and the cumulative suite stays green", "dist/ is build output — gitignored (dist/, *.pyz), never committed; README.md edits are additive sections (CLI install/usage, docker usage) with v0.1 content intact"]
territory: [cli/**, scripts/**, tests/**, Dockerfile, README.md, .gitignore]
reason: Primary execution persona — the entire v0.1 product is Python on one file: P1 core skeleton, P2 admin-plane API, P3 tool auth plane, P4 secrets adapter + public client module + reference tools. Owns every security-critical code path (token derivation, slow-hash verification, allowlist gates, audit trail, path-scoped reads) and every test that gates a phase.
frameworks: [python3.11, http.server, sqlite3, hashlib.scrypt, unittest]
constraints: ["coreci_broker.py stays one file under 800 lines (REQ-17, AC14)", "stdlib-only runtime; bcrypt is the sole optional import; no pip, no npm (D-001/D-009)", "all errors use the {error, message} JSON contract on 400/401/403/404/409/500", "broker.db and broker.audit.log are created mode 0600 (REQ-15)", "the broker never writes to creds-coreci-coo (REQ-08)"]
territory: [coreci_broker.py, tests/**, coreci_broker_client.py, reference_tools/**]
subagent_type: backend-engineer
---
---
name: frontend-engineer
domain: frontend
active: true
phase_specific: true
reason: P2 only (REQ-13 admin UI). Inactive outside P2 — no UI work exists in P1/P3/P4/P5; deactivate at the P2 exit boundary and do not dispatch outside P2.
frameworks: [html, css, vanilla-js]
constraints: ["server-rendered static pages from coreci_broker_ui/ on the same port as the API; no build step, no client framework, no npm (REQ-13)", "pages: /ui/login, /ui/app_registrations, /ui/user_allowlist, /ui/audit plus /ui/dashboard and a persistent nav header with Logout", "pages refresh via the corresponding admin API after each action; the API contract is owned by backend-engineer — frontend adapts", "session cookie broker_session is HttpOnly + SameSite=Lax (D-013); no direct DB or file access — admin API only"]
territory: [coreci_broker_ui/**]
subagent_type: frontend-engineer
---
---
name: security-engineer
domain: security
active: true
phase_specific: true
reason: Review-only pass activated for P3 (milestone security review — artifact integrity, no secrets in the archive/image, token discipline in upload paths, registry auth flows, boot parity of materialized assets); inactive in P1/P2 unless the lead escalates a finding — same posture as v0.1.
frameworks: [python3.11, unittest, STRIDE, CWE, oci-dist-spec]
constraints: ["review-only — every v0.2 file stays owned by release-engineer; findings route through lead-developer, never direct edits; coreci_broker.py is in review scope for zero-delta verification (AC14) only and is writable by nobody this milestone", "findings are STRIDE-classified and CWE-first (cite CWE ids)", "focus — secrets discipline (file-only token path via scripts/_env_secrets.py, never shell env, non-empty assert before first use), artifact integrity (no secrets in the .pyz or any image layer; sha256 provenance recorded in release notes), registry auth flows (gitea anonymous-200 /v2/token quirk, Basic username-ignored shape, 307/308 handling), attachment idempotency and bounded-retry escalation paths"]
territory: [coreci_broker.py, cli/**, scripts/**, Dockerfile, tests/**]
reason: Custom persona activated for P3 (tool auth plane: register gates, App-token verification, identity enforcement) and P5 (security review pass before ship); inactive in P1/P2/P4 unless the lead escalates a finding.
frameworks: [python3.11, unittest, STRIDE, CWE]
constraints: ["review-only: coreci_broker.py and tests stay owned by backend-engineer; findings route through lead-developer, never direct edits", "findings are STRIDE-classified and CWE-first (cite CWE ids)", "focus: path traversal on /secrets/<path> (URL-decode before validation), authz (allowlist gates, path scoping, identity match), audit completeness, pepper never re-printed, 0600 modes, token scoping"]
territory: [coreci_broker.py, tests/**]
subagent_type: ci-security-auditor
---
@@ -30,46 +41,20 @@ subagent_type: ci-security-auditor
name: lead-developer
domain: coordination
active: true
phase_specific: false
reason: Active across all execution phases (P1–P3): decomposes plans into vertical slices along the build → attach → publish pipeline, arbitrates the frozen-broker rule (no persona writes coreci_broker.py — P0 fixes are lead-dispatched exceptions), routes security findings, and sequences tasks across phase boundaries. No direct code writes.
reason: Active across all execution phases (P1–P5): decomposes plans into vertical slices per persona domain, arbitrates territory conflicts, revises the plan at phase boundaries, and sequences tasks across persona dependencies. No direct code writes.
frameworks: [none]
constraints: ["no direct modifications to code files — coordination and conflict resolution only", "sole gate for frozen-broker exceptions — any P0 fix to coreci_broker.py (or the frozen UI/client/reference sources) is a lead-dispatched, defect-scoped task, never a persona territory claim", "arbitration — release-engineer owns every v0.2 file in cross-persona conflicts and defines the artifact/registry/release contracts; security findings flow lead → release-engineer; battle-tested defaults over novel approaches", "enforce — zero broker-line delta (AC14), secrets only via the .env* file loader, no secrets in artifacts, bounded-retry release_pending escalations never block tag+merge"]
constraints: ["no direct modifications to code files — coordination and conflict resolution only", "arbitration: backend-engineer owns files in cross-persona conflicts and defines the API contract; frontend adapts; security findings flow through backend", "enforce boundaries: no direct database access from the UI, no UI logic inside coreci_broker.py; battle-tested defaults over novel approaches"]
territory: [.ciagent/PLAN.md, .ciagent/*.md]
subagent_type: lead-developer
---
---
name: backend-engineer
domain: backend
active: false
phase_specific: false
reason: Deactivated — the broker is frozen in v0.2; RESEARCH §4.2 verified run_standalone/boot/load_config/_parse_env_file are importable as-is, 1390/1400 lines leaves 10 lines of headroom, and zero-change is a hard constraint. All v0.2 engineering is distribution tooling owned by release-engineer. Reactivates only as a lead-developer-dispatched, defect-scoped P0 fix to coreci_broker.py — never via its own territory.
frameworks: [python3.11, http.server, sqlite3, hashlib.scrypt, unittest]
constraints: ["stays inactive for all planned v0.2 work; any broker change is a lead-developer-dispatched exception scoped to the P0 defect", "if dispatched — the 1400-line AC14 cap and every v0.1 runtime constraint (REQ-14..REQ-18, {error, message} JSON contract, 0600 modes, never write to creds-coreci-coo) remain binding"]
territory: []
subagent_type: backend-engineer
---
---
name: frontend-engineer
domain: frontend
active: false
phase_specific: false
reason: Deactivated — no UI work in v0.2; REQ-21..23 touch no page. The v0.1 UI (coreci_broker_ui/) is complete, frozen, and consumed as a read-only staging input by the zipapp build, then materialized verbatim at boot; any UI defect surfaced through the packaged path is a lead-dispatched exception.
frameworks: [html, css, vanilla-js]
constraints: ["stays inactive for the whole milestone; coreci_broker_ui/** is a read-only build input owned by no persona"]
territory: []
subagent_type: frontend-engineer
---
---
name: data-engineer
domain: data
active: false
phase_specific: false
reason: Deactivated (carried from v0.1) — no database tier beyond sqlite3 embedded in the frozen broker; no schema, pipeline, ETL, or migration work in v0.2. Image layers and release assets are build outputs owned by release-engineer, not data surfaces.
reason: Deactivated — no database tier exists beyond sqlite3 embedded in the single file; the spec §5.2 schema and all SQL live inside coreci_broker.py and are owned by backend-engineer; no pipelines, ETL, or migrations in v0.1 scope.
frameworks: [sqlite3]
constraints: ["stays inactive for the whole milestone; any residual data work folds into release-engineer or a lead-dispatched broker fix"]
constraints: ["stays inactive for the whole milestone; any residual data work folds into backend-engineer"]
territory: []
subagent_type: backend-engineer
---
+112 -55
View File
@@ -1,101 +1,158 @@
---
must_haves:
- "python3 -m unittest discover -s tests passes (cumulative — the v0.1 suite plus the v0.2 test files: test_build_cli, test_cli_packaging, test_release_upload, test_publish_image, test_dockerfile_consistency, test_acceptance_v02)"
- "python3 -m py_compile cli/__main__.py scripts/build_cli.py scripts/release.py scripts/publish_image.py scripts/_env_secrets.py"
- "broker line count unchanged: coreci_broker.py stays at its v0.1.5 count of 1390 lines (AC14, D-016..D-018 cap 1400) — zero broker edits"
- "two consecutive builds via scripts/build_cli.py produce byte-identical artifacts (sha256 equal) — same-environment scope (G-009)"
- "the built .pyz boots with v0.1 acceptance parity (admin pass printed exactly once, broker.db + broker.audit.log 0600, UI + client served, silent second boot)"
- "stub-registry publish roundtrip green (scripts/publish_image.py vs a local stub, including a relative-Location 307 case, G-019) and stub release upload green (scripts/release.py vs a local stub)"
- "python3 -m unittest discover -s tests passes (all phases, cumulative)"
- "coreci_broker.py single file, line count < 800 (AC14)"
- "python3 -m py_compile coreci_broker.py coreci_broker_client.py"
- "broker boots standalone on an ephemeral port; DB + audit log created mode 0600; admin pass printed exactly once"
- "all AC1-AC14 acceptance criteria green by end of P5"
---
# PLAN — CoreCI Broker (Milestone v0.2 — cli-packaging)
# PLAN — CoreCI Broker (Milestone v0.1)
Execution conventions: sequential waves (within a wave, tasks run in listed order); each task = one subagent commit `feat(P##): …` carrying a `---ci---` block (phase, milestone: v0.2, status: execute, persona, task, requirements.covered). Tests ship with their task — never deferred. Personas per PERSONAS.md: release-engineer (ALL v0.2 code — `cli/**`, `scripts/**`, the new test files, `Dockerfile`, `.dockerignore`, additive README, `.gitignore`), security-engineer (P3 only, review-only — findings route via lead-developer, never direct edits), lead-developer (coordination + ship steps, no code writes; sole gate for frozen-broker exceptions). The broker is FROZEN this milestone (1390/1400, REQ-17/AC14): no task edits `coreci_broker.py`, `coreci_broker_ui/`, `coreci_broker_client.py`, or `reference_tools/` — a P0 defect there is a lead-developer-dispatched exception only. Verify commands: `python3 -m py_compile <files>`; `python3 -m unittest discover -s tests`; broker zero-delta via the AC14 line-count test. Test harness (v0.1 §6 conventions extended, RESEARCH §4.3): `ThreadingHTTPServer` stubs on `("127.0.0.1", 0)` + `tempfile.TemporaryDirectory()` state + subprocess boots of the built artifact driven with stdlib `urllib`; scripts take base URLs as parameters (stub-testable); NEVER live-forge writes in tests (the only live-forge contact is the G-013 smoke gate + the ships themselves). Secrets resolve only via `scripts/_env_secrets.py` reading `.env`/`.env.secrets`/`.env.*` files — never shell env (`GITEA_TOKEN` is in `blocked_env_vars`); non-empty asserted before the first API call. `dist/` + `*.pyz` are gitignored — build artifacts are never committed. Release attach + image publish failures follow bounded retry (3) → `release_pending` escalation — never blocking tag+merge. Ships: phase 0 → `v0.2.0`, P1 → `v0.2.1`, P2 → `v0.2.2`, P3 → `v0.2.3` (final phase patch = milestone release, D-008 convention). **G-021: the `v0.2.0` phase-0 release carries NO `.pyz` and its notes must not promise one — D-020 starts at `v0.2.1`.** Release-tooling split (G-011): the sanctioned secrets loader + release attacher land in P1 (Wave 3) so every release from `v0.2.1` is script-attached; P2 owns the image publish client + Dockerfile.
Execution conventions: sequential waves (within a wave, tasks run in listed order); each task = one subagent commit `feat(P##): …` carrying a `---ci---` block (phase, milestone: v0.1, status: execute, persona, task, requirements.covered). Tests ship with their task — never deferred. Personas per PERSONAS.md: backend-engineer (all phases), frontend-engineer (P2 only), security-engineer (P3+P5, review-only — findings route via lead-developer, never direct edits), lead-developer (coordination, no code writes). Verify commands: `python3 -m py_compile <files>`; `python3 -m unittest discover -s tests`; AC14 line-count via test. Test harness: in-process `ThreadingHTTPServer` on an ephemeral port + `tempfile.TemporaryDirectory()` state, driven with stdlib `urllib` (RESEARCH §6).
## User-Facing Surface
- **Downloadable CLI artifact** — `coreci-broker-vX.Y.Z.pyz` on the gitea release page (every release from `v0.2.1`, D-020), alongside the existing source archive; its sha256 recorded in the release notes (computed by `scripts/release.py` from the artifact file — G-017).
- **CLI boot** — `python3 coreci-broker-vX.Y.Z.pyz` or `./coreci-broker-vX.Y.Z.pyz` after `chmod +x` (shebang `#!/usr/bin/env python3`); boots the standalone broker — full admin UI at `/ui/`, client module at `/static/coreci_broker_client.py`, bound address per `BROKER_BIND` (default `127.0.0.1:8765`).
- **`--version`** — prints the release version, exits 0.
- **Docker image** — `docker pull git.coreci.dev/coreci/broker:<tag>` (+ `latest`); `docker run` with a volume for state.
- **Release notes** — on the release page: milestone summary, phases, verification, CLI install + usage, docker usage.
- The v0.1 surface (four-page UI + dashboard, client module, audit) unchanged — served through the packaged path.
- **CEO admin UI** — server-rendered from `coreci_broker_ui/`, same port as the API, persistent nav with Logout: `/ui/login` (the CEO's bookmark), `/ui/app_registrations` (table + add-modal + revoke/reinstate), `/ui/user_allowlist` (same, optional email), `/ui/audit` (filterable, read-only, no export), `/ui/dashboard`.
- **Tool-side client module** — `coreci_broker_client.py`: `ensure_registered()`, `get_state()`, `put_state()`, `read_secret()`; stdlib-urllib-only, importable with zero dependencies in any OWUI tool env; fetched unauthenticated at `/static/coreci_broker_client.py`.
- **Standalone CLI** — `python3 coreci_broker.py`: binds `127.0.0.1:8765`; first boot prints the generated admin pass exactly once.
- **OWUI Tools mount** — `Tools` class (same file): daemon-thread server behind a once-per-process guard, `self.valves`, `_`-prefixed helpers.
## Happy Path
1. Owner downloads `coreci-broker-v0.2.3.pyz` from the `v0.2.3` release → `python3 coreci-broker-v0.2.3.pyz` (or `chmod +x` + `./…`) → first boot generates `.env` and prints the admin pass exactly once; `broker.db` + `broker.audit.log` created mode 0600 in the state dir → UI at the bound address (`/ui/login` bookmark works).
2. Second boot: silent — pass never re-printed; state persists.
3. Tool authors fetch `coreci_broker_client.py` from `/static/coreci_broker_client.py` exactly as in v0.1 — the packaged artifact is the only input.
4. `--version` reports the release version without booting.
5. Separately: `docker pull git.coreci.dev/coreci/broker:v0.2.3` → `docker run -p 8765:8765 -v broker-state:/state …` → identical first-boot/0600/UI behavior inside the container, state on the volume.
1. First boot in a fresh directory: broker writes `.env` (32-byte base64 pepper + 32+-char admin pass), prints the pass to the startup log exactly once (AC1); a second boot prints nothing.
2. CEO opens the `/ui/login` bookmark, logs in with the printed pass (session cookie `broker_session`), lands on the dashboard.
3. CEO adds the tool's app UUID on `/ui/app_registrations` and the user_id on `/ui/user_allowlist`; each page reflects the persisted row right after the action (AC2/AC3); audit gets `add_app`/`add_user`.
4. Tool author fetches `/static/coreci_broker_client.py` (unauthenticated) into the OWUI workspace and imports it — no third-party deps.
5. Tool calls `ensure_registered()`: broker gates on the allowlists, derives the token server-side, returns `{token, token_expires_at, pepper}` (row-create); the client persists via `PUT /agents/<id>/state`.
6. Tool reads `coreci/proxmox` with `Authorization: App <UUID> <token>` → 200 secret payload (AC5); audit line `read`/`ok`.
7. Same tool requests `coreci/gitea` → scope mismatch → 403 (AC6); audit `read`/`denied`.
8. Token expires (`BROKER_TOKEN_TTL`): next call → 401 (AC10) → client re-registers → new token → retry succeeds; audit `register`.
9. CEO reviews `/ui/audit`: login/logout (AC9), allowlist adds, registers, reads ok/denied — every step audited.
## UX Acceptance Criteria
- **Self-contained artifact** — the `.pyz` bundles broker + UI pages + client module; only runtime state lives outside it (`.env`, `broker.db`, `broker.audit.log`, materialized assets under `<state>/.coreci_broker_assets/`); one file serves the full admin UI at `/ui/` and the client at `/static/`.
- **`--version`** — prints the release version (from the build-time `_cli_version.py`), exits 0, no broker import on that path.
- **First-boot parity** — behavior through the packaged artifact is byte-exact with v0.1 standalone: admin pass printed exactly once, `broker.db` + `broker.audit.log` mode 0600, UI + client served, second boot silent.
- **Per-release artifact** — every release from `v0.2.1` (D-020) carries `coreci-broker-vX.Y.Z.pyz` + release notes (CLI install/usage + sha256, docker usage); attachment failure escalates `release_pending`, never blocks tag+merge. `v0.2.0` carries none (G-021).
- **Image published once per milestone** — at the final ship: `git.coreci.dev/coreci/broker:v0.2.3` + `latest` (D-021); publish failure escalates per the same contract with the G-020 closure runbook, never blocking the milestone merge.
- **AC1** — first boot prints the admin pass exactly once; the CEO logs in at `/ui/login` with it; second boot prints nothing.
- **AC2/AC3** — app registrations and user-allowlist entries are added and removed via the UI; the page reflects persisted state after every action (no stale views).
- **AC9** — every login and logout writes an audit line (`login`/`logout`).
- **UI** — server-rendered static HTML/CSS/JS from `coreci_broker_ui/` on the same port: no build step, no framework, no npm.
- **Audit page** — read-only with filters; NO export button or download control of any kind.
- **Client module** — `coreci_broker_client.py` imports cleanly with zero deps (stdlib urllib only) in any OWUI tool env; contains no secrets.
## Phase 1 — CLI Packaging + Release Tooling
## Phase 1 — Core Skeleton
Branch `phase/01-cli-packaging` → tag `v0.2.1`. Requirements: REQ-21, REQ-22.
Exit: artifact builds reproducibly from a clean checkout (two consecutive same-environment builds → byte-identical sha256); the built `.pyz` boots with v0.1 acceptance parity (pass printed once, 0600 state files, UI + client served from materialized assets, silent second boot); `--version` wired; release-asset automation stub-verified; REQ-21 + REQ-22 consolidated green; the `v0.2.1` release carries the `.pyz` attached by `scripts/release.py`.
Branch `phase/01-core-skeleton` → tag `v0.1.1`. Requirements: REQ-03, REQ-11, REQ-14, REQ-15, REQ-17, REQ-18.
Exit: broker boots standalone on an ephemeral port; `broker.db` + `broker.audit.log` created mode 0600; admin pass printed exactly once (silent second boot); §6 token derivation byte-exact; source < 800 lines.
### Wave 1 — build pipeline + boot shim
### Wave 1 — persistence layer (independent)
- **P1-01** · release-engineer · `scripts/build_cli.py`, `cli/__main__.py`, `.gitignore`, `tests/test_build_cli.py` · build pipeline + boot shim (RESEARCH §1.2/§1.3; G-009/G-018): `build_cli.py <version>` stages an explicit allowlist tree — `coreci_broker.py`, `coreci_broker_client.py`, `coreci_broker_ui/**`, `cli/__main__.py`, generated `_cli_version.py` (`VERSION = "<tag>"`) — then `python3 -m zipapp <staging> -o dist/coreci-broker-vX.Y.Z.pyz --python "/usr/bin/env python3" --compress` (zipapp defaults to ZIP_STORED — `--compress` mandatory), then a normalize post-pass making same-environment rebuilds byte-identical (G-009: zip entries sorted by name, `date_time=(1980,1,1,0,0,0)`, `ZIP_DEFLATED`, **`external_attr` SET to constants** — files `0o100644 << 16`, dirs `0o40755 << 16` — never preserved/umask-derived; shebang re-prepended + exec bit), and prints the artifact sha256 (digest-in-notes default; no sidecar asset); `cli/__main__.py` = the five-step shim: (1) `--version` first — print `_cli_version.VERSION`, exit 0, no broker import; (2) state-dir resolution identical to the broker (`BROKER_ENV_PATH`-anchored, relative paths stay CWD-relative — zero path changes); (3) materialize assets from `zipfile.ZipFile(sys.path[0])` → `<state>/.coreci_broker_assets/<VERSION>/{ui,static}` (temp + `os.replace`, versioned dirs, same-version overwrite self-repair, no cleanup); (4) override precedence user-wins — `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` set only when absent from process env AND the user `.env` (parsed via `coreci_broker._parse_env_file`); (5) handoff `coreci_broker.run_standalone()`; an `os.path.isdir(sys.path[0])` guard falls back to repo-relative roots when run unzipped (zipapp-only is the documented mode); the shim header documents its private-API coupling to the broker (`_parse_env_file`, `run_standalone` — G-018: loud `ImportError` breakage accepted and tracked in P3-02); `.gitignore` += `dist/`, `*.pyz`; broker sources are read-only staging inputs — zero broker lines · REQ-21 · test: `test_build_cli.py` — two consecutive same-environment builds → byte-identical sha256; `_cli_version.VERSION` embedded; archive enumerates exactly the staging allowlist (broker + client + UI pages + shim); constant `external_attr` asserted (no umask bleed); `py_compile` of the shim green; built artifact `--version` prints the version, exits 0.
- **P1-01** · backend-engineer · `coreci_broker.py`, `tests/test_config.py` · `.env` reader/writer at `BROKER_ENV_PATH` (spec §5.1 contract + documented additions); first-boot generation of `BROKER_INSTANCE_PEPPER` (32 random bytes, base64, synced to SQLite when the store boots — sync lands with P1-02's schema) and `BROKER_ADMIN_PASS` (32+ random chars), written to `.env`, printed to the startup log exactly once — never again · REQ-14 · test: first boot writes `.env` + prints pass once; second boot silent; `BROKER_*` overrides honored.
- **P1-02** · backend-engineer · `coreci_broker.py`, `tests/test_store.py` · SQLite schema per spec §5.2 (`broker_config`, `app_registrations`, `user_allowlist`, `agent_registrations`, `admin_sessions` + indexes); DB created mode 0600 via `os.open(..., 0o600)` (umask-independent); sole writer of `broker.db`, all SQL in-file · REQ-15 · test: boot creates `broker.db` at 0600 with all tables + indexes.
- **P1-03** · backend-engineer · `coreci_broker.py`, `tests/test_audit.py` · append-only JSONL audit writer at `BROKER_AUDIT_LOG_PATH`: one JSON object per line, fields `ts, user_id, app_registration_uuid, agent_id, action, path, result` (admin actions carry null user_id/uuid); mode 0600; flush + fsync per write; no rotation (D-007) · REQ-11 · test: each write appends one valid JSON line at mode 0600; all action verbs accepted.
### Wave 2 — packaged boot acceptance
### Wave 2 — crypto primitives
- **P1-02** · release-engineer · `tests/test_cli_packaging.py` · end-to-end boot acceptance through the built artifact (RESEARCH §4.3 pattern; G-010 harness hardening; consolidates REQ-21): build into a `TemporaryDirectory` → `Popen([sys.executable, "<pyz>"], env={"BROKER_BIND": "127.0.0.1:0", BROKER_ENV_PATH/DB/AUDIT/CREDS_FILE/CREDS_SOURCE: tmp paths})` → parse the `coreci-broker listening on http://…` line for the ephemeral base URL under an explicit deadline loop; first boot prints `BROKER_ADMIN_PASS=` exactly once (stdout fully drained after `terminate()` before the second-boot assertion); `.env` + `broker.db` + `broker.audit.log` created mode 0600 in the state dir (independent of CWD); `/healthz` 200, `/ui/login` + `/ui/dashboard` 200 from materialized assets, `/static/coreci_broker_client.py` byte-exact; admin login + agent register + state + secret-read reuse the v0.1 acceptance request patterns (v0.1 parity through the packaged path); second boot silent (pass never re-printed); `--version` exits 0 with the version string; materialized assets verified under `<state>/.coreci_broker_assets/<VERSION>/`; user override honored (`.env`-set `BROKER_UI_ROOT` wins over the shim default); plus one un-overridden default-anchoring boot (state in CWD of a temp dir — the owner's literal happy path, G-010) · REQ-21 · test: is the acceptance file.
- **P1-04** · backend-engineer · `coreci_broker.py`, `tests/test_hasher.py` · dual-mode hasher (D-001): bcrypt when importable, else stdlib `hashlib.scrypt`; stored hashes format-prefixed (`$2b$…` / `scrypt$…`); verification format-aware · REQ-03 · test: scrypt hash/verify roundtrip + prefix dispatch on both formats (bcrypt path via format contract — no bcrypt locally).
- **P1-05** · backend-engineer · `coreci_broker.py`, `tests/test_token.py` · token engine: `base64(sha256(pepper:UUID:user_id:agent_id:','.join(sorted(tags))))` — 44-char ASCII, no prefix; pure function of (pepper, inputs); never stores plaintext tokens · REQ-03 · test: derivation matches the §6 formula byte-for-byte; 44-char assertion; tag order normalized.
### Wave 3 — release-asset automation (G-011: lands in P1)
### Wave 3 — transport + mounts (listed in dependency order)
- **P1-03** · release-engineer · `scripts/_env_secrets.py`, `scripts/release.py`, `tests/test_release_upload.py` · sanctioned secrets loader + release attacher (REQ-22; RESEARCH §2): `_env_secrets.py` parses `.env`/`.env.secrets`/`.env.*` from the repo root + `.ciagent/` (mirroring the broker's `_parse_env_file` semantics, quote-stripping) — `GITEA_TOKEN` resolves ONLY from these files, never `os.environ` (structurally blocked by `blocked_env_vars`), asserted non-empty before the first API call (v1.8 lesson: empty token → gitea 404, masking auth failure); `release.py <tag> <artifact>` (gitea base URL + owner/repo parameterized): **computes the artifact sha256 from the file itself — never an argument, never hand-pasted (G-017)** → builds the notes body per the RESEARCH §2.3 model (milestone summary, phases, verification, CLI install + usage with the artifact sha256, docker usage) → create-or-get by tag (`GET /api/v1/repos/{o}/{r}/releases/tags/{tag}`; 404 → `POST .../releases` with the notes body) → idempotent attach (`GET .../releases/{id}/assets` — same name + size → success, no duplicate on bounded-retry re-runs) → `POST .../releases/{id}/assets` multipart (field name `attachment`, `?name=coreci-broker-vX.Y.Z.pyz`, `Authorization: token <T>`) → bounded retry (3 attempts, backoff) → exhaustion exits non-zero with a machine-readable marker for the `release_pending` escalation; never blocks tag+merge · REQ-22 · test: `test_release_upload.py` — `_StubGiteaHandler` on `("127.0.0.1", 0)` (v0.1 `_StubOwuiHandler` precedent, fake token, no live-forge writes): create-or-get in both orders (release exists / 404 → create), multipart parsed (field name + `?name=` + body bytes asserted), sha256-from-file asserted (passing a wrong digest argument is ignored/refused), idempotent re-attach uploads nothing twice, 500×3 → the `release_pending` marker, token discipline (file-only resolution + non-empty assert before the first call), notes sections present.
- **P1-06** · backend-engineer · `coreci_broker.py`, `tests/test_router.py` · stdlib `ThreadingHTTPServer` + JSON router: body/param parsing, `{"error, message}` contract on 400/401/403/404/409/500, static-file serving hook for `/ui/` + `/static/` (fixed root, traversal-rejecting); no business logic in the router · REQ-18 · test: ephemeral-port boot; every error code returns the contract shape; static hook serves a fixture.
- **P1-07** · backend-engineer · `coreci_broker.py`, `tests/test_mounts.py` · OWUI `Tools` mount: `__init__` starts the daemon-thread server behind a once-per-process guard (no activate() hook exists — RESEARCH §4), `self.valves = self.Valves()`, `_`-prefixed helpers, paths resolved from env never `__file__`; standalone `__main__` entry · REQ-18 · test: `Tools()` double-instantiation starts exactly one server; standalone boot serves a 200.
### Wave 4 — phase ship
### Wave 4 — phase acceptance
- **P1-04** · lead-developer · no code writes (the release-engineer's built artifact + scripts are consumed read-only) · phase ship: tag `v0.2.1` → squash-merge `phase/01-cli-packaging` → `milestone/v0.2-cli-packaging` → push tag + branch → gitea release via `scripts/release.py` (D-020 starts here: `coreci-broker-v0.2.1.pyz` + its sha256 in the notes, attached by the repo script — no ad-hoc bridge, G-011) · REQ-21, REQ-22 (ship) · verify: `v0.2.1` tag + release on the forge with the `.pyz` attached; attachment failure → bounded retry (3) → `release_pending` escalation, never blocking tag+merge.
- **P1-08** · backend-engineer · `tests/test_acceptance_p1.py` · AC14 line-count test (`coreci_broker.py` < 800 lines, wc-style, no comment stripping) + boot acceptance: 0600 on DB + audit log, admin pass printed exactly once (silent second boot), restart persistence of DB rows + audit history · REQ-17, REQ-16 (per REQUIREMENTS traceability), REQ-15 · test: is the acceptance file.
## Phase 2 — Docker + Image Publish
## Phase 2 — Admin Plane
Branch `phase/02-docker-release-integration` → tag `v0.2.2`. Requirements: REQ-23 (+ REQ-21/REQ-22 docs refinement).
Exit: daemonless publish client stub-verified end-to-end (no live-forge writes, no container runtime); Dockerfile + .dockerignore + README document the docker-user path and stay consistency-tested against the daemonless image; the `v0.2.2` release carries the `.pyz` attached by `scripts/release.py` with CLI + docker notes.
Branch `phase/02-admin-plane` → tag `v0.1.2`. Requirements: REQ-01, REQ-05, REQ-06, REQ-12, REQ-13.
Exit: AC1/AC2/AC3/AC9 green; admin-usable (login → manage allowlists → query audit, all via UI).
### Wave 1 — daemonless OCI publish client
### Wave 1 — admin auth
- **P2-01** · release-engineer · `scripts/publish_image.py`, `tests/test_publish_image.py` · daemonless OCI publish client (D-019, G-014: one vertical task; all endpoints parameterized; G-019 wire hardening): pull — anonymous Docker Hub token (`auth.docker.io/token`, scope `repository:library/python:pull`) → `python:3.11-slim` manifest (D-022) with the full Accept set → branch on `"manifests"` vs `"layers"` (index → resolve the `linux/amd64` child manifest by digest) → fetch child + config + 4 layer blobs; record resolved digests (`--base-manifest` override for exact re-publishes); auth — gitea `GET /v2/token?service=container_registry&scope=repository:coreci/broker:pull,push` with `Authorization: Basic <user>:<GITEA_TOKEN>` (token via `_env_secrets`; username ignored by the forge) — insist on Basic credentials present + issued JWT non-empty (the endpoint answers 200 anonymously with an empty-scope JWT — fail loudly, never treat 200 as authorization); push — **per-request timeouts (~120s on blob POSTs, G-019)** + **blob-level retry with HEAD-skip** (HEAD each blob first: 200 → skip; base layers ~45.7 MB re-upload once per milestone, retries resume cheap) → monolithic `POST /v2/coreci/broker/blobs/uploads/?digest=sha256:<hex>` (fallback: two-step POST → 202 `Location` → PUT) with explicit Location-following on 307/308 (stdlib urllib does NOT follow method-preserving redirects on PUT/POST — verified; **the stub test covers a relative `Location` at least once, G-019**) → deterministic broker layer (tar entries sorted, fixed TarInfo metadata uid/gid 0, `gzip.compress(..., mtime=0, compresslevel=9)`) carrying `/app/coreci-broker.pyz`; config copied from the base + diff_id appended + `Entrypoint ["python3", "/app/coreci-broker.pyz"]`, `Cmd []`, `WorkingDir /state`, optional version Label, serialized `json.dumps(sort_keys=True, separators=(",",":"))`; manifest PUT with OCI media types end-to-end (manifest `application/vnd.oci.image.manifest.v1+json`, config `…image.config.v1+json`, layers `…image.layer.v1.tar+gzip`; base layer descriptors reused verbatim; never push an index) under `<version>` then `latest` (D-021); provenance digests recorded for the release notes · REQ-23 · test: `test_publish_image.py` — `_StubRegistryHandler` on `("127.0.0.1", 0)` (fake token via temp `.env` files): anonymous-token trap (200 with empty-scope JWT → loud failure), HEAD-skip on retry, monolithic upload with 307 → relative `Location` followed explicitly, manifest PUT with correct media types + tag sequence (version then `latest`), deterministic layer/config digests stable across two runs (gzip mtime=0), base manifest/layer handling against a fixture index, timeout wiring (a stalled endpoint → client aborts, does not hang).
- **P2-01** · backend-engineer · `coreci_broker.py`, `tests/test_admin_auth.py` · `POST /api/broker/auth/login|logout|whoami`: verifies `BROKER_ADMIN_USER`/`BROKER_ADMIN_PASS`, creates `admin_sessions` row (TTL `BROKER_SESSION_TTL`, default 300s); cookie `broker_session` HttpOnly + SameSite=Lax (D-013); HTTP Basic accepted on all admin API endpoints; lazy expiry + boot cleanup of stale sessions · REQ-01, REQ-11 · test: login sets cookie; wrong pass → 401; expired session → 401; logout invalidates + audits (AC9 `login`/`logout` lines).
### Wave 2 — Dockerfile + .dockerignore + docs
### Wave 2 — admin API
- **P2-02** · release-engineer · `Dockerfile`, `.dockerignore`, `README.md` (additive section only — no v0.1 content rewrites), `tests/test_dockerfile_consistency.py` · docker-user path + usage docs (G-015/G-016): `Dockerfile` self-contained (no pre-built artifact in the build context): `FROM python:3.11-slim` → `COPY coreci_broker.py coreci_broker_client.py coreci_broker_ui/ cli/ /app/src/` → `ARG VERSION` → generated `_cli_version.py` from the build arg → `RUN python3 -m zipapp /app/src -o /app/coreci-broker.pyz --python "/usr/bin/env python3" --compress` → `WORKDIR /state` → `ENTRYPOINT ["python3", "/app/coreci-broker.pyz"]` → `EXPOSE 8765`; header documents the equivalence mapping to the daemonless-published image (same base family, entrypoint, pyz content, `/state` anchoring; differences: docker-build layering vs verbatim base layers + one broker layer, build-arg vs build-time version provenance; the daemonless client is the source of truth, byte-identity not required — RESEARCH §3.5); `.dockerignore` (G-015): `.git`, `.env`, `.env.*`, `data/`, `*.db`, `dist/`, `__pycache__/`, `tests/`, `.ciagent/` — no secrets, state, or build artifacts can enter an image build context; README additive section: CLI install + usage (release-page download, `chmod +x` / `python3`, `--version`, env-contract pointer, first-boot pass discipline, `BROKER_ENV_PATH` state anchoring, Python 3.11+ note) + docker usage (`docker pull git.coreci.dev/coreci/broker:<tag>` + `latest`; `docker run -p 8765:8765 -v broker-state:/state …`; state-on-volume contract) · REQ-23, REQ-21/REQ-22 (docs) · test: `test_dockerfile_consistency.py` (G-016 — equivalence made testable): Dockerfile COPY set == `build_cli.py` staging allowlist; zipapp flags identical (`--python`/`--compress`); `ARG VERSION` → `_cli_version` format parity; `ENTRYPOINT`/`WORKDIR` == the daemonless-published config values; `.dockerignore` excludes every forbidden context entry.
- **P2-02** · backend-engineer · `coreci_broker.py`, `tests/test_apps.py` · app_registrations CRUD: `GET/POST /api/broker/app_registrations`, `DELETE .../<uuid>` (marks revoked — new registrations rejected, existing tokens honored until TTL), `POST .../reinstate`; UUID format validation → 400; duplicate → 409; audited `add_app`/`remove_app` · REQ-05 · test: add/list/revoke/reinstate matrix + 400/409 + revoke-keeps-tokens + audit lines.
- **P2-03** · backend-engineer · `coreci_broker.py`, `tests/test_allowlist.py` · user_allowlist CRUD: same shape (`GET/POST`, `DELETE .../<user_id>` revoked semantics, `POST .../reinstate`), optional `email`, duplicate → 409; audited `add_user`/`remove_user` · REQ-06 · test: add/remove/reinstate + 409 + audit lines.
- **P2-04** · backend-engineer · `coreci_broker.py`, `tests/test_audit_query.py` · admin-only `GET /api/broker/audit`: filters `since`, `until`, `user_id`, `app_registration_uuid`, `action`; `limit` default 100 / max 1000; `next_since` cursor; streams the file line-by-line · REQ-12 · test: seeded audit file → filter combos, limit clamp, cursor pagination.
### Wave 3 — phase ship
### Wave 3 — admin UI (frontend-engineer; pages refresh via the admin API after each action; no direct DB/file access)
- **P2-03** · lead-developer · no code writes · phase ship: tag `v0.2.2` → squash-merge `phase/02-docker-release-integration` → `milestone/v0.2-cli-packaging` → push tag + branch → release via `scripts/release.py` (notes per the §2.3 model now including docker usage, `coreci-broker-v0.2.2.pyz` + sha256 attached alongside the source archive) · REQ-23 (phase completion) · verify: `v0.2.2` tag + release on the forge with the `.pyz` + notes; attachment failure → bounded retry → `release_pending`, never blocks tag+merge.
- **P2-05** · frontend-engineer · `coreci_broker_ui/` (login page, dashboard, shared nav/CSS) · `/ui/login` (the CEO's bookmark) + `/ui/dashboard`; persistent nav header with Logout on every page · REQ-13, REQ-01 · verify: pages served 200 on the API port, post-login redirect to dashboard; full AC drive lands in P2-08.
- **P2-06** · frontend-engineer · `coreci_broker_ui/` (app_registrations + user_allowlist pages) · table + add-modal + revoke/reinstate per row on both pages; after each action the page re-fetches via the corresponding API and reflects persisted state · REQ-13, REQ-05, REQ-06 · verify: AC2/AC3 flows (tests in P2-08); interim smoke — pages boot + served.
- **P2-07** · frontend-engineer · `coreci_broker_ui/` (audit page) · filterable audit view (since/until/user_id/uuid/action) wired to `GET /api/broker/audit`; strictly read-only — no export button or control of any kind · REQ-13, REQ-12 · verify: page renders query results; no export element in the markup.
## Phase 3 — Final Review + Ship
### Wave 4 — phase acceptance
Branch `phase/03-final-review-ship` → tag `v0.2.3` (final phase patch = milestone release, D-008 convention). Requirements: REQ-21, REQ-22, REQ-23 (acceptance).
Exit: cumulative v0.1 + v0.2 acceptance green; broker zero-delta verified (1390/1400, AC14); security review PASS (or findings remediated); `v0.2.3` on `main` with the `.pyz` attached + full notes; image published as a gitea package (or escalated per the bounded-retry contract + G-020 closure runbook — never blocking merge); REQUIREMENTS/ROADMAP closed out; checkpoint cleared.
- **P2-08** · backend-engineer · `tests/test_acceptance_p2.py` · AC1 (first-boot login at `/ui/login` with the printed pass), AC2/AC3 (add + remove app/user through the UI; `app_registrations`/`user_allowlist` persist; page reflects state after each action), AC9 (login/logout audit lines) · REQ-01, REQ-05, REQ-06, REQ-12, REQ-13 · test: is the acceptance file.
### Wave 1 — acceptance consolidation
## Phase 3 — Tool Auth Plane
- **P3-01** · release-engineer · `tests/test_acceptance_v02.py` · v0.2 acceptance matrix: cumulative suite green via `python3 -m unittest discover -s tests` (all v0.1 files + the v0.2 test files); packaging acceptance consolidated — boot parity through the artifact (first-boot pass printed once, `broker.db` + `broker.audit.log` 0600, UI + client served, silent second boot, register/read flows), reproducibility (two consecutive same-environment builds → identical sha256), self-containment (only runtime state external to the `.pyz`: env/db/audit + materialized assets), stub publish roundtrip + stub release upload green; broker zero-delta — `coreci_broker.py` unchanged at 1390 lines vs `v0.1.5` (AC14 under the D-016..D-018 cap of 1400) · REQ-21, REQ-22, REQ-23 (acceptance) · test: is the acceptance file.
Branch `phase/03-tool-auth-plane` → tag `v0.1.3`. Requirements: REQ-02, REQ-04, REQ-09, REQ-10.
Exit: AC4/AC7/AC8/AC10/AC12 green; tool-auth usable (register → App-auth calls → state persistence).
### Wave 1 — registration
- **P3-01** · backend-engineer · `coreci_broker.py`, `tests/test_register.py` · `POST /api/broker/agents/register` with `{agent_id, app_registration_uuid, user_id, tags}`: allowlist gates → 403 unknown/revoked UUID or user; malformed → 400; idempotent re-derive (D-005) — overwrite stored hash on input change, same derived token when unexpired + unchanged; TTL from `BROKER_TOKEN_TTL` (default 86400, D-007); pepper returned on row-create or tokenless register (KB-loss recovery); audited `register` · REQ-02, REQ-03 · test: gate matrix (AC7/AC8), idempotency/overwrite/same-token, pepper rule, 400s.
### Wave 2 — App-auth + agent surface (listed in dependency order)
- **P3-02** · backend-engineer · `coreci_broker.py`, `tests/test_app_auth.py` · App-auth verification (D-004): parse `Authorization: App <UUID> <token>`; UUID-scoped slow-hash scan over that tool's rows; in-memory verified-token cache populated on first success, never authoritative — every hit re-checks row status + expiry; 401 missing/malformed/failed · REQ-04 · test: cache-miss scan + cache-hit re-check (revoke + expiry), both hash formats, 401 matrix; revoked-app token honored until TTL.
- **P3-03** · backend-engineer · `coreci_broker.py`, `tests/test_agent_meta.py` · `GET /api/broker/agents/<agent_id>` (D-015): App-auth + identity match; metadata-only response (no token, no hash) · REQ-04, REQ-02 · test: correct-identity 200 with metadata-only payload; identity mismatch → 403; no token material in response.
- **P3-04** · backend-engineer · `coreci_broker.py`, `tests/test_state.py` · `GET/PUT /api/broker/agents/<agent_id>/state` (App-auth): broker enforces calling UUID + token-derived user_id match the stored identity; 404 when no state exists; state schema per spec §8 (agent_id, app_registration_uuid, user_id, tags, pepper, current_token, token_expires_at, first_registered_at, last_seen_at); backend toggle `BROKER_AGENT_STATE_BACKEND=internal` (default, rows in `broker.db`) `| owui` (thin OWUI-API adapter, stub-verified) (D-003) · REQ-09, REQ-10 · test: state roundtrip + identity enforcement + 404; owui backend against a stub server.
### Wave 3 — phase acceptance
- **P3-05** · backend-engineer · `tests/test_acceptance_p3.py` · AC4 (broker- and tool-side derivation byte-identical; stored as slow-hash), AC7 (revoked/unknown UUID → 403), AC8 (unknown user → 403), AC10 (TTL expiry → 401 → re-register → new token, audited), AC12 (state read/write with identity enforcement) · REQ-02, REQ-04, REQ-09, REQ-10 · test: is the acceptance file.
### Wave 4 — security review (review-only)
- **P3-06** · security-engineer · review of `coreci_broker.py`, `tests/**` (no edits — findings STRIDE-classified, CWE-cited, routed via lead-developer) · tool-auth security pass: register gates, UUID-scoped scan + cache authority, identity enforcement, TTL/expiry handling, audit completeness · REQ-02, REQ-04, REQ-09 · verify: findings recorded in the task commit's `---ci---` block; remediation (if any) = follow-up backend commit dispatched by lead-developer.
## Phase 4 — Secrets + Client Ecosystem
Branch `phase/04-secrets-client-ecosystem` → tag `v0.1.4`. Requirements: REQ-07, REQ-08, REQ-19, REQ-20.
Exit: AC5/AC6 green; full product — client module fetched from `/static/` drives a reference tool end-to-end against the test broker.
### Wave 1 — secrets plane (listed in dependency order)
- **P4-01** · backend-engineer · `coreci_broker.py`, `tests/test_creds_source.py` · creds-source adapter (D-002): `FileCredsSource` default (JSON at `BROKER_CREDS_FILE`, default `data/creds.json` — D-014) + `OwuiApiCredsSource` (`BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY`; RESEARCH §4 flow: KB lookup → file map → `/api/v1/files/{id}/data/content`); response `{path, data, version, updated_at}` (version synthesized broker-side if absent); broker never writes the KB · REQ-08 · test: file-source hit/miss; owui source against a stub HTTP server.
- **P4-02** · backend-engineer · `coreci_broker.py`, `tests/test_secrets.py` · `GET /api/broker/secrets/<path>` (App-auth): scope check `path == "coreci/" + agent_id` → 403 mismatch, 404 absent, 401 auth failure; every attempt audited (`read` ok/denied/error); traversal defense — URL-decode BEFORE validation, reject `..`, backslash, NUL, absolute paths (decoded form feeds the scope check) · REQ-07 · test: ok/401/403/404 matrix + URL-encoded traversal attempts rejected + audit per attempt.
### Wave 2 — client ecosystem (listed in dependency order)
- **P4-03** · backend-engineer · `coreci_broker_client.py`, `tests/test_client.py`, `coreci_broker.py` (static route) · public client module: stdlib `urllib` only (no httpx, no bcrypt — importable in any OWUI tool env); `ensure_registered()` idempotent; `get_state()`/`put_state()`; `read_secret()` with 401 → re-register → retry-once; `data.get("pepper")` (never assumes the key — RESEARCH §2 defect 3); served unauthenticated at `/static/coreci_broker_client.py` via the P1-06 hook; contains no secrets · REQ-19 · test: zero third-party imports asserted; full flow against the in-process broker; 401→re-register→retry path.
- **P4-04** · backend-engineer · `reference_tools/`, `tests/test_reference_tools.py` · reference OWUI tools (D-006): (a) proxmox-style §11 flow — first call registers + persists state, subsequent calls read the token from state and fetch `coreci/proxmox`, downstream call defaults to a safe dry-run; (b) minimal auth-first-then-use-a-second-tool showcase · REQ-20 · test: both tools drive the test broker (FileCredsSource); dry-run default asserted; no live infra.
### Wave 3 — phase acceptance
- **P4-05** · backend-engineer · `tests/test_acceptance_p4.py` · AC5 (valid token reads the secret; tampered token → 401), AC6 (proxmox-registered tool reading `coreci/gitea` → 403); end-to-end: fetch client from `/static/` unauthenticated → import → register → state → secret read · REQ-07, REQ-19, REQ-20 · test: is the acceptance file.
## Phase 5 — Final Review + Ship
Branch `phase/05-final-review-ship` → tag `v0.1.5` (final phase patch = milestone release, D-008). Requirements: REQ-01..REQ-20 (acceptance).
Exit: AC1–AC14 all green; security review PASS; README complete with deviations; owner approval recorded; milestone merged to main, tagged `v0.1.5`, released on gitea.
### Wave 1 — acceptance matrix
- **P5-01** · backend-engineer · `tests/test_acceptance_full.py` · complete §12 matrix: AC1–AC13 as behavioral tests (consolidating the per-phase AC files and completing any remaining §12 criteria — 0600 modes, restart persistence, client-module fetch) + AC14 line-count capstone; cumulative suite green via `python3 -m unittest discover -s tests` · REQ-01..REQ-20 · test: is the acceptance suite.
### Wave 2 — security review (review-only)
- **P3-02** · security-engineer · review of `coreci_broker.py` (zero-delta verification only — writable by nobody this milestone), `cli/**`, `scripts/**`, `Dockerfile`, `.dockerignore`, `tests/**` (no edits — findings STRIDE-classified, CWE-cited, routed via lead-developer) · v0.2 security pass: secrets in artifacts (no secrets in the `.pyz` or any image layer; `.env` never inside the archive; pepper/admin pass generated at boot; no state baked into the image; build-context hygiene per `.dockerignore`, G-015), digest determinism (zip normalize pass with constant `external_attr` — G-009; sorted tar + fixed TarInfo; `gzip mtime=0`; sorted-key JSON), token discipline (file-only `_env_secrets` path, non-empty assert before first use, never shell env; gitea's anonymous-200 `/v2/token` quirk fails loudly), 307/308 Location handling incl. relative `Location`, traversal + disclosure over materialized asset roots (dot-dir roots vs request-path dot filtering; protected-paths realpaths wherever the state dir sits), shim private-API coupling tracked (G-018), attachment idempotency + bounded-retry escalation paths · REQ-21, REQ-22, REQ-23 (+ all) · verify: findings recorded in the task commit's `---ci---` block; remediation (if any) = the follow-up release-engineer commit dispatched by lead-developer (P3-03).
- **P5-02** · security-engineer · review of `coreci_broker.py`, `coreci_broker_client.py`, `reference_tools/`, `tests/**` (no edits — STRIDE/CWE findings via lead-developer) · pre-ship security pass: path traversal incl. URL-encoded forms, pepper never re-printed (second boot, re-register), 0600 modes, token scoping + identity enforcement, audit completeness, static-serving roots · REQ-07, REQ-11, REQ-14, REQ-15 (+ all) · verify: PASS or CWE-cited findings; remediation follows in P5-03.
### Wave 3 — remediation
### Wave 3 — remediation + docs
- **P3-03** · release-engineer · `cli/**`, `scripts/**`, `Dockerfile`, `.dockerignore`, `tests/**` (never broker sources — a P0 broker defect routes through lead-developer dispatch only) · remediate P3-02 findings (no-op skip if PASS) · affected REQs per finding · test: full cumulative suite re-green.
- **P5-03** · backend-engineer · `coreci_broker.py`, `tests/**` · remediate P5-02 findings (no-op skip if PASS) · affected REQs per finding · test: full suite re-green.
- **P5-04** · backend-engineer · `README.md` · run/deploy notes: standalone + OWUI mount, env/config contract (spec §5.1 + documented additions), first-boot pass discipline, backup guidance (SQLite is the source of truth), client-module bootstrap; documented deviations distilled from D-001..D-015 incl. the spec defects + resolutions (PROJECT.md §Context) · REQ-14, REQ-18 (+ traceability) · test: README covers run/deploy/deviations; suite still green.
### Wave 4 — review + reconstruction
- **P3-04** · lead-developer · `.ciagent/*.md` review notes (no code writes) · multi-persona review of the milestone diff (release-engineer artifacts across P1/P2, security sign-off from P3-02/P3-03); audit/reconstruction: every REQ-21..REQ-23 maps to green tests + traceable commits; broker zero-delta verified (1390/1400, AC14); escalation ledger clean or `release_pending` entries documented · REQ-21, REQ-22, REQ-23 · verify: sign-offs + reconstruction recorded in `---ci---` blocks.
- **P5-05** · lead-developer · `.ciagent/*.md` review notes (no code writes) · multi-persona review of the milestone diff (backend, frontend P2 artifacts, security sign-offs); audit/reconstruction: every REQ-01..REQ-20 maps to green tests + traceable commits · REQ-01..REQ-20 · verify: sign-offs + reconstruction recorded in `---ci---` blocks.
### Wave 5 — milestone ship
### Wave 5 — ship
- **P3-05** · lead-developer · no code writes (release-engineer scripts invoked, never edited) · milestone ship, in this exact order (G-012: merge→tag→push BEFORE release attach — a pre-merge release create would mint `v0.2.3` at pre-merge main via gitea's target_commitish semantics): build the final `.pyz` (`python3 scripts/build_cli.py v0.2.3`) → squash-merge `milestone/v0.2-cli-packaging` → `main` → tag `v0.2.3` → push main + tag → **G-013 live smoke (env-gated `PUBLISH_SMOKE=1`, read-only, never in the default suite): real gitea `/v2/` auth challenge, token→JWT exchange, `GET` release-by-tag, Docker Hub index resolve — all must pass before the one live publish** → attach to the `v0.2.3` release with full notes (milestone summary, phases, verification, CLI install/usage + sha256, docker usage) via `scripts/release.py` → daemonless image publish ONCE (`python3 scripts/publish_image.py` → `git.coreci.dev/coreci/broker:v0.2.3` + `latest`, D-019/D-021/D-022) → branch cleanup → REQUIREMENTS/ROADMAP closeout → checkpoint cleared. Publish failure: bounded retry (3) → `release_pending` escalation **with the G-020 closure runbook committed alongside** (exact re-run command on `main` @ `v0.2.3`: build + smoke + `publish_image.py`; close condition = both tags visible as a gitea package) — never blocks the merge, never left open · REQ-21, REQ-22, REQ-23 · verify: `v0.2.3` on `main`; the release carries the `.pyz` + full notes; the image is visible as a gitea package (or escalated per contract + runbook, never blocking); closeout + checkpoint-clear commits recorded.
- **P5-06** · lead-developer · no code writes · owner approval gate recorded; squash-merge `milestone/v0.1-coreci-broker` → `main`; tag `v0.1.5`; first push to gitea `coreci/broker` (public, default branch main — forge repo exists before push, D-010/D-012) + release · REQ-01..REQ-20 · verify: `v0.1.5` on `main`; release visible on the forge.
+4 -19
View File
@@ -12,7 +12,7 @@ A single canonical credential flow for autonomous agent tools. The broker mediat
### Validated
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09; supersedes v0.4) — distilled into REQ-01..REQ-20 (shipped in v0.1). v0.2 additions validated from owner directives (2026-10-10) — distilled into REQ-21..REQ-23: CLI zipapp packaging, per-release artifact attachment with notes on gitea, once-per-milestone docker publication to the gitea container/package registry. The v0.5 spec's ambiguities were resolved by D-001..D-015 (init + clarify commits, v0.1); v0.2's packaging/distribution ambiguities are resolved as D-019..D-022 (this milestone's clarify stage):
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09; supersedes v0.4). The full v0.5 spec is distilled into REQ-01..REQ-20 in REQUIREMENTS.md, with acceptance criteria drawn from the spec's §12 list. Clarification decisions D-001..D-011 (recorded in the init commit) plus D-012..D-015 (recorded in the clarify stage) resolve the spec's ambiguities and residual defaults:
- Dual-mode token hashing (bcrypt when importable, stdlib scrypt fallback; format-prefixed hashes)
- Pluggable creds-source adapter (file-based default; Open WebUI HTTP API for production)
@@ -24,7 +24,7 @@ Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09; supersed
### Active
Milestone v0.2 (`milestone/v0.2-cli-packaging`) — **CLI + distribution only**, no functional changes to the broker: REQ-21 (zipapp CLI packaging), REQ-22 (release assets attached + notes on gitea), REQ-23 (docker container built once per milestone, published as a gitea package). Owner-locked inputs: zipapp `.pyz` form factor; docker publication starts with v0.2 (no retroactive v0.1 image); scope excludes deferred v2 candidates.
Milestone v0.1 (`milestone/v0.1-coreci-broker`) implements the entire v0.5 spec: REQ-01..REQ-20. No requirement is deferred.
### Out of Scope
@@ -32,7 +32,7 @@ See REQUIREMENTS.md §Out of Scope — federation, secret versioning, high avail
## Constraints
- **Single-file broker**: `coreci_broker.py` under 1400 lines (amended from 800 via owner decisions D-016..D-018; excluding tests, UI HTML, client module, reference tools) — spec §12.14 as amended
- **Single-file broker**: `coreci_broker.py` under 800 lines (excluding tests, UI HTML, client module, reference tools) — spec §12.14
- **Stdlib-only runtime**: no pip in the target dev environment; bcrypt is the sole optional import (dual-mode hashing); tests use unittest
- **SQLite is the source of truth**: single instance, no HA; back it up
- **Bind default** `127.0.0.1:8765`; the bind address controls reachability
@@ -42,21 +42,10 @@ See REQUIREMENTS.md §Out of Scope — federation, secret versioning, high avail
- **Token form**: `base64(sha256(pepper:UUID:user_id:agent_id:sorted_tags))` — 44-char ASCII, no prefix; `BROKER_TOKEN_TTL` default 86400s
- **Open WebUI runtime**: bcrypt and httpx exist there; the client module must not depend on either (stdlib urllib) so any tool can import it
v0.2 (packaging/distribution) constraints:
- **No pip / stdlib-only build path**: the CLI artifact is a stdlib `zipapp` (`.pyz`); no wheel, no PyInstaller, no pip at build or runtime (D-009)
- **No container runtime in the dev environment**: no docker/podman/buildah available — the once-per-milestone publish runs daemonless via a stdlib registry client (D-019)
- **Self-contained artifact**: one `.pyz` bundles broker + UI assets + client module; asset roots resolved from env, never `__file__`
- **Broker behavior frozen**: first-boot semantics, API surface, and all v0.1 acceptance criteria remain byte-identical when driven through the packaged CLI
- **Release secrets discipline**: forge tokens resolved only from `.env*` files via the secrets-loader — never shell environment (v1.8 lesson)
- **`broker.db`/`broker.audit.log` remain state-directory-local** — the CLI boots in a runtime state dir (docker: a volume), never writes inside its own archive
## Context
CoreCI estate: sibling projects researchx and nextcraft live on gitea (`git.coreci.dev/coreci/*`) and use this same ciagent pipeline. The broker runs on the Open WebUI host as one workspace tool (embedded HTTP server in a daemon thread; also runnable standalone). The `creds-coreci-coo` KB currently mediates four credentials (`coreci/proxmox`, `coreci/gitea`, `coreci/executor` TBD, plus the COO's own login — independent of the broker's `BROKER_ADMIN_PASS`). The existing `proxmox_readonly_tool.py` is updated post-broker to use the public client module; it lives outside this repo, so this repo ships the reference implementation of the tool-side bootstrap instead.
v0.2 context: milestone v0.1 shipped complete (all REQ-01..REQ-20, `v0.1.5` on main, six gitea releases each carrying a `broker-vX.Y.Z.tar.gz` source archive, zero escalations). Owner directives for v0.2 (2026-10-10): package the broker as a CLI; attach the CLI to every release alongside notes on gitea; build a docker container once per milestone and publish it as a package in the gitea registry. The v0.1 release-attachment mechanism (sanctioned `.env*` token path, per-release asset upload) is the proven base REQ-22 extends.
Known spec defects resolved by clarification (recorded for traceability): §7.5's in-memory `token_hash` index is impossible with bcrypt salted hashes; §6 claims the tool derives a token every call while §9's flow uses the broker-returned token from the agent KB (the broker derives authoritatively; the tool stores); §9's snippet raises `KeyError` on `data["pepper"]` for re-registrations; the pepper chicken-and-egg (a token is needed to read agent state, but the pepper is needed to derive a token) is fixed by returning the pepper when the caller presents no valid token.
## Key Decisions
@@ -80,8 +69,4 @@ Known spec defects resolved by clarification (recorded for traceability): §7.5'
| D-015 | `GET /api/broker/agents/<agent_id>`: App-auth + identity match, metadata-only response (no token, no hash) | §7.4 auth/response unspecified; mirrors state-endpoint identity enforcement; token material never leaves the broker | Binding for v0.1 |
| D-016 | AC14 line cap amended: coreci_broker.py < 1000 lines (was < 800) | Owner-approved §12.14 amendment at P2: 11 lines of headroom vs ~275 lines of remaining handler surface; preserves the readable single-file intent without dense-cramming | Binding for v0.1 |
| D-017 | D-016 extended: AC14 cap < 1200 lines | Second §12.14 amendment at P3 (owner's D-016 pattern applied with new data): P3-01 lands at 990/1000 with ~150-180 lines of remaining broker surface (App-auth cache, agent state, creds adapter, secrets endpoint); mild compaction reclaims only ~50 — same readability-over-cramming tradeoff the owner chose at D-016; subject to review at the P5 owner approval gate | Binding for v0.1 |
| D-018 | D-017 extended: AC14 cap < 1400 lines — final projected amendment | Third §12.14 amendment, pre-P4 per the P3 verify planning signal: 1178/1200 after P3 with 21 lines headroom vs P4's ~105-140 line need (creds adapter + secrets endpoint, the LAST broker-surface additions); P5 adds only excluded artifacts (tests, README). Same owner pattern (D-016/D-017), flagged for the P5 owner approval gate | Binding for v0.1 |
| D-019 | Daemonless container publish: stdlib registry client (pull pinned base layers, append broker layer, push to gitea registry) | No docker/podman/buildah in the dev environment; gitea's package registry speaks OCI registry HTTP API v2 — fully automated once-per-milestone publish without a container runtime; stub-testable | Binding for v0.2 |
| D-020 | Attach `coreci-broker-vX.Y.Z.pyz` to every release from `v0.2.1` onward | Consistent with v0.1's per-release `broker-vX.Y.Z.tar.gz` asset convention; every tagged release stays independently deployable | Binding for v0.2 |
| D-021 | Image tags `git.coreci.dev/coreci/broker:<milestone-release-tag>` + `latest`, pushed once per milestone | Version tag pins the release; `latest` gives docker users a sane default; once-per-milestone honors the owner directive | Binding for v0.2 |
| D-022 | Base image `python:3.11-slim` (pinned minor line) | Dev environment runs Python 3.11.2 — pinned minor matches tested stdlib behavior and stays reproducible | Binding for v0.2 |
| D-018 | D-017 extended: AC14 cap < 1400 lines — final projected amendment | Third §12.14 amendment, pre-P4 per the P3 verify planning signal: 1178/1200 after P3 with 21 lines headroom vs P4's ~105-140 line need (creds adapter + secrets endpoint, the LAST broker-surface additions); P5 adds only excluded artifacts (tests, README). Same owner pattern (D-016/D-017), flagged for the P5 owner approval gate | Binding for v0.1 |
+4 -18
View File
@@ -2,9 +2,9 @@
## Overview
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09). "v0.5" is the spec's revision label. REQ-01..REQ-20 (milestone v0.1) shipped complete as `v0.1.5`. **Milestone v0.2 (cli-packaging)** adds REQ-21..REQ-23 from owner directives (2026-10-10): package the broker as a CLI, attach it to every gitea release with notes, and publish a docker container once per milestone as a gitea package. Scope is CLI + distribution only — no functional changes to the broker. Clarification decisions D-001..D-015 (v0.1) remain binding; D-016+ resolve v0.2's packaging/distribution ambiguities.
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09; supersedes the v0.4 in-memory draft). "v0.5" is the spec's revision label; the product milestone is v0.1. REQ-01..REQ-20 constitute the full v0.1 scope — nothing is deferred. Acceptance criteria reference the spec's §12 list (AC1–AC14). Clarification decisions D-001..D-011 live in the init commit's `---ci---` block and are distilled in PROJECT.md.
## v1 (Milestone v0.1 — complete, shipped as v0.1.5)
## v1 (Milestone v0.1)
### Auth & Identity
@@ -50,18 +50,7 @@ Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09). "v0.5"
- **REQ-19 Public client module.** `coreci_broker_client.py` — dependency-free (stdlib urllib; no httpx, no bcrypt) — implements the tool-side bootstrap of spec §9: ensure-registered (idempotent), state get/put, secret read with 401 → re-register → retry-once, and never assumes the pepper key is present in register responses (`data.get("pepper")`). Served unauthenticated at `/static/coreci_broker_client.py` so any OWUI tool author can fetch it into their workspace and import it. It contains no secrets.
- **REQ-20 Reference tools.** `reference_tools/` ships at least: (a) a proxmox-style reference implementing §11's flow — first call registers and persists state, subsequent calls read the token from state and fetch `coreci/proxmox`, downstream call defaults to a safe dry-run; (b) a minimal showcase of auth-first-then-use-a-second-tool. These are templates for tool authors, wired to the test broker in tests.
## v2 (Milestone v0.2)
### Packaging
- **REQ-21 CLI packaging (zipapp).** The broker ships as a single-file executable CLI `coreci-broker.pyz` built with stdlib `python3 -m zipapp` from a clean checkout — no pip, no third-party build tooling (D-009 preserved). The archive is self-contained: it bundles `coreci_broker.py`, the `coreci_broker_ui/` pages, and `coreci_broker_client.py`, materializing the UI/static roots into the runtime state directory on boot (paths resolved from env, never `__file__`), so one file serves the full admin UI at `/ui/` and the client module at `/static/coreci_broker_client.py`. Running it boots the standalone server with first-boot semantics identical to REQ-14/REQ-18 (`.env` generation, admin pass printed exactly once, `broker.db` + `broker.audit.log` at mode 0600); `--version` reports the release version. A build script regenerates the artifact reproducibly, and tests boot the built artifact end-to-end.
### Distribution
- **REQ-22 Release assets.** Every gitea release from the phase that first builds the CLI (`v0.2.1`, D-020) onward attaches the built artifact `coreci-broker-vX.Y.Z.pyz` alongside the existing source archive. Uploads resolve the forge token only via the sanctioned secrets path (`.env*` files — never shell environment). Release notes document the CLI: artifact download + install (`chmod +x` / `python3`), the CLI surface (boot, `--version`, env contract), and docker usage per REQ-23. Attachment failure follows the ship workflow's bounded-retry then `release_pending` escalation — it never blocks tag+merge.
- **REQ-23 Docker package.** The repo ships a `Dockerfile` and a **daemonless stdlib registry client** (D-019): once per milestone, at the final ship, the client pulls the pinned `python:3.11-slim` base image manifest + layers from the public registry (HTTP API v2), appends a layer carrying the CLI artifact, and pushes the image `git.coreci.dev/coreci/broker:<milestone-release-tag>` plus the `latest` alias (D-021/D-022) to the gitea container/package registry — no container runtime required. The container runs the CLI standalone with state on a volume. Registry auth, manifest, layer, and digest handling are stub-tested; publish failure follows bounded-retry escalation — it never blocks the milestone merge.
## Deferred Beyond v0.2
## v2 (Deferred)
None defined. Future candidates held as out-of-scope decisions, not requirements: audit rotation with retention, multiple broker instances, approver-agent delegation, OWUI-KB credential sync.
@@ -107,7 +96,4 @@ None defined. Future candidates held as out-of-scope decisions, not requirements
| REQ-17 | 1 | complete |
| REQ-18 | 1 | complete |
| REQ-19 | 4 | complete |
| REQ-20 | 4 | complete |
| REQ-21 | 1 | complete |
| REQ-22 | 1 | complete |
| REQ-23 | 2 | complete |
| REQ-20 | 4 | complete |
+65 -174
View File
@@ -1,191 +1,82 @@
# RESEARCH — CoreCI Broker
Date: 2026-10-10 · Milestone: v0.2 (cli-packaging) · Phase: 0 (pre-execution) · Status: complete
Sources: `.ciagent/` PROJECT/REQUIREMENTS/ARCHITECTURE/ROADMAP/PLAN, `coreci_broker.py` (1390 lines, read in full), git history/tags v0.1.0..v0.1.5, live gitea 1.26.1 API (`git.coreci.dev` swagger.v1.json + releases — read-only, token never printed), live Docker Hub registry + auth.docker.io (anonymous pulls, verified end-to-end), live gitea container-registry auth probes (read-only, no upload sessions created), stdlib source (`/usr/lib/python3.11/zipapp.py`, `urllib/request.py`), and an **end-to-end zipapp prototype booted from the real broker sources** (built + booted as subprocess + driven with urllib; `/tmp` scratch, repo untouched).
Date: 2026-10-09 · Milestone: v0.1 · Phase: 0 (pre-execution) · Status: complete
Sources: `.ciagent/` PROJECT/REQUIREMENTS/ARCHITECTURE/ROADMAP, git log (init ffdc5392, clarify 53d99e80), docs.openwebui.com + open-webui source at `main` (fetched 2026-10-09). No scope changes; D-001..D-015 reconfirmed; no secrets read or printed.
## 0. Carried-Forward v0.1 Findings (still binding)
## 1. Environment Constraints (verified facts — do not re-verify)
- **Environment (v0.1 §1, re-confirmed)**: Python 3.11.2, NO pip anywhere, NO docker/podman/buildah runtime; stdlib-only runtime + build path (D-009 preserved by REQ-21/D-019); tests = stdlib unittest; forge gitea `git.coreci.dev`, owner `coreci`, repo `broker`, public, default branch `main`; `GITEA_TOKEN` resolves from `.ciagent/.env.secrets` only (mode 0600, never printed, `config.security.bash_allowlist.blocked_env_vars` includes GITEA_TOKEN → shell-env tokens are structurally blocked).
- **Test harness (v0.1 §6)**: in-process `ThreadingHTTPServer(("127.0.0.1", 0), Handler)` + daemon thread + `tempfile.TemporaryDirectory()` state + stdlib `urllib.request` client + `addCleanup`; 0600 assertions via `st_mode & 0o777 == 0o600`; stub-HTTP precedent: `_StubOwuiHandler` in `tests/test_creds_source.py` (BaseHTTPRequestHandler with class-level fixtures + hit recording) — this is the template for v0.2 stub registry/gitea servers.
- **Ship contract (`ci/workflows/ship.md` + config.json)**: every phase ships; release creation best-effort with bounded retries (`ship.max_release_retries: 3`); on exhaustion commit escalation `resolution: "auto"`, `type: "release_pending"`, phase still complete — tag+merge is the gate, `release_blocking: false`; **all secrets resolved ONLY from `.env`/`.env.secrets`/`.env.*` files via the secrets-loader, never shell env** (v1.8 root cause: empty token → gitea answers 404, not 401, masking auth failure).
- **OWUI integration surface (v0.1 §4)**: unchanged by v0.2 — the CLI does not touch the `Tools` mount; path resolution from env (never `__file__`) is the v0.1 property that makes zipapp packaging possible at all.
- **Static-serving rules that the packaged CLI must preserve**: `_resolve_static_file` rejects `\x00`, `\`, leading `/`, `..` segments, and any *request* path part starting with `.`; roots are real-pathed (`os.path.realpath`) so a dot-named *root* directory is fine (materialized assets live under a dot-dir; only request paths are dot-filtered). `_broker_protected_paths` protects `.env`/db/audit/creds (+ `-journal/-wal/-shm` suffixes) from static reads.
- Python 3.11.2 on the dev host; NO pip, NO bcrypt locally → D-001 dual-mode hashing active: stdlib `hashlib.scrypt` in dev/tests, bcrypt in the OWUI runtime; hashes format-prefixed (`$2b$…` / `scrypt$…`). Tests = stdlib unittest (D-009). Pure-Python repo, no package.json.
- Forge: gitea `git.coreci.dev`, org `coreci`; release target `coreci/broker` — repo created at first push/ship, public, default branch `main` (D-010/D-012). `GITEA_TOKEN` resolves from `.ciagent/.env.secrets` only (gitignored, mode 0600, never printed).
- Stdlib components carrying the runtime: `http.server.ThreadingHTTPServer`, `sqlite3`, `hashlib` (scrypt/sha256), `secrets`, `json`, `base64`, `urllib` (client module), `threading`.
- Repo state verified at research time: branches `main`, `milestone/v0.1-coreci-broker`, `phase/00-pre-execution`; `.gitignore` at root; `.ciagent/` holds PROJECT/REQUIREMENTS/ARCHITECTURE/ROADMAP/config.json/CHECKPOINT.json + `.env.secrets` (unread, 0600).
## 1. zipapp CLI Packaging (REQ-21)
## 2. Spec Defects & Resolutions (PROJECT.md §Context; decisions in init/clarify commits)
### 1.1 stdlib mechanics (all VERIFIED empirically on Python 3.11.2)
| # | Defect | Resolution | Test implication |
|---|--------|------------|------------------|
| 1 | §7.5 in-memory `token_hash` index impossible with bcrypt (salted hashes neither comparable nor indexable) | D-004: UUID-scoped slow-hash scan + in-memory verified-token cache; SQLite authoritative, cache never authoritative | Cover both paths: cache-miss scan and cache-hit re-check of row status/expiry; both hash formats |
| 2 | §6 (tool derives every call) vs §9 (tool stores broker-returned token) contradiction | Broker derives authoritatively; tool stores via state endpoints (ARCHITECTURE.md DataFlow 2–4) | AC4: broker- and tool-side derivation byte-identical (§6 formula); state GET/PUT roundtrip |
| 3 | §9 snippet `KeyError` on `data["pepper"]` for re-registrations | D-005: pepper returned only on row-create or tokenless register; REQ-19 mandates `data.get("pepper")` in the client | Re-registration without a pepper key must not crash the client; register still audited |
| 4 | Pepper chicken-and-egg (token needed to read state; pepper needed to derive) | Tokenless register returns the pepper — the only KB-loss recovery path (D-005) | Recovery scenario: no valid token → register → pepper returned → tool can re-derive |
- Command: `python3 -m zipapp <srcdir> -o coreci-broker.pyz --python "/usr/bin/env python3" --compress`.
- **`__main__.py` is mandatory** — `zipapp` refuses with `ZipAppError("Archive has no entry point")` without it (the `-m module:fn` form is the only alternative; the shipped `cli/__main__.py` source is cleaner and testable). Files staged: `coreci_broker.py`, `coreci_broker_client.py`, `coreci_broker_ui/**`, `cli/__main__.py`, generated `_cli_version.py`.
- **Compression: `zipapp` defaults to ZIP_STORED** (stdlib source line 136: `zipfile.ZIP_DEFLATED if compressed else zipfile.ZIP_STORED`). `--compress` is required for DEFLATED. Measured: ~95 KB content → 27.6 KB pyz compressed vs ~95 KB stored. Use `--compress`.
- **Shebang + exec bit**: with `--python`, zipapp writes `#!/usr/bin/env python3\n` as a literal first line, then the zip (zip readers locate EOCD from the file end — leading bytes tolerated), and `chmod +x`es the target (`target.chmod(... | S_IEXEC)`). Both `./coreci-broker.pyz` and `python3 coreci-broker.pyz` verified working; `sys.argv[0]` = the archive path as invoked; `python3 -m zipapp --info` reads the interpreter back.
- **Data files are NOT `open()`-able inside the archive** (verified `FileNotFoundError` for `assets/page.html`): zipimport only serves `.py` imports; the broker serves UI/static via **filesystem paths + `open()`** → asset materialization to disk at boot is *mandatory*, not optional. `importlib.resources` cannot provide a directory for `BROKER_UI_ROOT` — zipfile extraction is the only design that fits the broker's config contract. (Not a close call.)
- **`__main__` module collision — none**: the archive's `__main__.py` IS the `__main__` module; `import coreci_broker` inside it executes the broker's module-level code once and its `if __name__ == "__main__"` block does NOT fire (verified: side-effect counter runs once, standalone print block silent). Call `coreci_broker.run_standalone()` — no re-execution of module code, no subprocess re-invocation needed.
- **zipimport limits**: none bite here — the archive contains only pure-Python repo files; stdlib (`sqlite3`, `http.server`, C extensions) resolves from the host interpreter, and the broker's sole optional import (bcrypt) stays optional.
## 3. Sibling Conventions (data provided — not fetched)
### 1.2 Boot shim design (prototype built, booted, verified — recommended as-is)
- researchx: root = `.ciagent/`, `.gitignore`, `README.md`, `backend/`, `frontend/` — public repo, default branch main.
- nextcraft: root = `.ciagent/`, `apps/`, `packages/`, `deploy/`, `scripts/`, pnpm workspace, `Dockerfile`, `.gitea/` — public repo, default branch main.
- Implications: flat single-project `.ciagent/` (matches D-010); `README.md` at repo root expected at ship (P5); per-phase tags on the v0.1.x line (D-008); `---ci---` block in every commit; ships via gitea releases once the forge repo exists.
`cli/__main__.py` (staged into the pyz), five steps:
## 4. Open WebUI Integration Surface (researched 2026-10-09; docs live, post-v0.10 era)
1. **`--version` first, before importing the broker**: parse `sys.argv[1:]`; if `--version`, `import _cli_version; print(_cli_version.VERSION); sys.exit(0)`. Version comes from a build-time-generated `_cli_version.py` (`VERSION = "vX.Y.Z"` written by `scripts/build_cli.py` from the tag) — chosen over a zip-comment (opaque, fragile, still needs a parser). No broker import on the `--version` path.
2. **Resolve the state dir exactly like the broker**: `env_path = os.environ.get("BROKER_ENV_PATH") or ".env"`; `state_dir = dirname(abspath(env_path)) or cwd`. Broker path resolution (read from code, authoritative): `BROKER_ENV_PATH` default `.env`, `BROKER_DB_PATH` `broker.db`, `BROKER_AUDIT_LOG_PATH` `broker.audit.log`, `BROKER_CREDS_FILE` `data/creds.json`, `BROKER_UI_ROOT` `coreci_broker_ui`, `BROKER_STATIC_ROOT` `.` — **all CWD-relative when relative**; process env beats `.env` beats defaults. The CLI therefore needs no path changes at all: relative paths land in the CWD (docker: `WORKDIR /state` + volume).
3. **Materialize assets**: `zipfile.ZipFile(sys.path[0])` (inside a zipapp, `sys.path[0]` is the `.pyz` itself — verified) → extract `coreci_broker_ui/**` → `<state>/.coreci_broker_assets/<VERSION>/ui/` and `coreci_broker_client.py` → `<...>/static/`. Write temp + `os.replace` (atomic vs a concurrently-serving older instance). **Versioned asset dir**: upgrades materialize a new dir; old dirs are left in place (another instance may still serve them; ~30 KB each — cleanup explicitly NOT done in v0.2). Overwrite-same-version on every boot (self-repair, assets are immutable inputs; extraction happens before this instance's server starts).
4. **Override precedence — user wins**: `import coreci_broker`; parse the user `.env` with the broker's own `coreci_broker._parse_env_file(env_path)`; set `os.environ["BROKER_UI_ROOT"]` / `["BROKER_STATIC_ROOT"]` only when the key is in **neither** `os.environ` nor the `.env` values. Net precedence: process env → `.env` → shim defaults → broker defaults — identical to the broker's own contract, achieved with **zero broker changes**.
5. **Hand off**: `coreci_broker.run_standalone()` (foreground loop + KeyboardInterrupt shutdown; prints `coreci-broker listening on http://<host>:<port>` with the *actual bound address* — `BROKER_BIND=127.0.0.1:0` gives an ephemeral port via `_parse_bind` (`"0".isdigit()` → True) and the harness parses the printed URL).
### Workspace Tools — `Tools` class surface (docs: /features/extensibility/plugin/tools/development)
- Single Python file; frontmatter docstring keys: `title`, `author`, `author_url`, `git_url`, `description`, `required_open_webui_version`, `requirements`, `version`, `licence`. Broker ships `requirements:` empty — stdlib-only + bcrypt already present in the OWUI runtime (PROJECT.md Constraints).
- `Tools` class rules: `__init__` MUST set `self.valves = self.Valves()` (stored admin valves load only onto an instance that has a `valves` attribute); optional `Valves`/`UserValves` (pydantic BaseModel); every public callable is exposed to the model → helpers must be `_`-prefixed; method docstrings (reST `:param name:` form) + type hints generate the JSON schema; return str/dict/list; async methods preferred.
- **Lifecycle: there is NO `activate()`/`deactivate()` hook for Tools.** The loader (`backend/open_webui/utils/plugin.py`) execs the source into a `tool_{id}` module in `sys.modules`, instantiates `module.Tools()` at load (create/update) and caches it on `app.state.TOOLS`; re-exec occurs on source change; module-top-level side effects persist, and disabling a tool does NOT undo them. ⇒ ARCHITECTURE.md's "mounts the server at activation" maps to constructor/module-load time: start the daemon-thread `ThreadingHTTPServer` in `__init__` (or module top level) behind a once-per-process idempotency guard.
- Loader sets the module's `__file__` to a temp file deleted after exec ⇒ the broker must resolve DB/audit/env paths from config/env vars, never from `__file__`.
- User context: OWUI provides no `self.user` attribute; the per-call reserved arg `__user__` is a dict (`id`, `email`, `name`, `role`, plus `valves` when UserValves defined) (docs: /features/extensibility/plugin/development/reserved-args). Broker admin auth (`BROKER_ADMIN_PASS`) is independent of OWUI identity; ARCHITECTURE.md's "self.user supplies admin-agnostic context" is the broker's own convention — the broker's Tools class can stash `__user__` or ignore it. Nuance only, no architecture change.
- Background infra: OWUI's documented pattern is `asyncio.create_task` (Under the Hood §Patterns 4); the broker's stdlib analog is `threading.Thread(target=serve_forever, daemon=True)` — same once-per-process guard required (module re-exec on edits).
- Asset serving: a tool may register FastAPI routes via `app.add_api_route` with an idempotency guard (Under the Hood §Patterns 3) — but such routes are unauthenticated unless gated. The broker deliberately serves UI + `/static/` from its OWN `ThreadingHTTPServer` on `127.0.0.1:8765` — no FastAPI coupling; `/static/coreci_broker_client.py` is intentionally unauthenticated (REQ-19).
- Security posture: workspace tools execute unsandboxed Python in the server process (docs Plugin Security Warning) — the broker mount inherits that trust boundary.
**Verified end-to-end prototype results** (real broker sources, subprocess, urllib): first boot printed `BROKER_ADMIN_PASS=…` exactly once then the listening line; `/healthz` 200; `/ui/login` + `/ui/dashboard` 200 (served from materialized assets); `/static/coreci_broker_client.py` 200 byte-exact (6180 B); `.env` + `broker.db` + `broker.audit.log` created mode 0600 in the state dir; second boot silent (no pass re-print); state dir fully independent of CWD (env paths pin it).
### Knowledge Bases API — read path (source-verified: `backend/open_webui/routers/knowledge.py`, `files.py` at main, 2026-10-09)
- Auth: `Authorization: Bearer <API_KEY>` (sk-… key from Settings→Account; JWT also accepted; `x-api-key` alt header via `CUSTOM_API_KEY_HEADER`) — feeds `BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY` (REQ-08/D-002). Exact layout deferred to P4 per ARCHITECTURE.md §Notes; candidates below.
- Router `/api/v1/knowledge` (all endpoints `get_verified_user`):
- `GET /api/v1/knowledge/?page=N` — paginated KB list (30/page) → `{items:[{id,name,description,write_access,…}], total}`
- `GET /api/v1/knowledge/search?query=&page=` — same shape; candidate for finding `creds-coreci-coo` by name
- `GET /api/v1/knowledge/{id}` — KB detail + `files[]` (per-file metadata) + `write_access`
- `GET /api/v1/knowledge/{id}/files?query=&include_content=&page=` — file list; `include_content=true` embeds extracted content (docs flag it as expensive)
- `GET /api/v1/knowledge/{id}/files/pending`, `GET /api/v1/knowledge/{id}/export` (zip) — auxiliary
- Write ops exist (`/create`, `/{id}/update`, `/{id}/file/add|update|remove|move`, `/{id}/files/batch/add`, `/{id}/delete`, `/{id}/reset`, `/{id}/sync/*`, `/{id}/dirs/*`) — the broker never calls them (read-only, REQ-08).
- Router `/api/v1/files`:
- `GET /api/v1/files/{id}` → FileModel; `data.content` holds extracted text
- `GET /api/v1/files/{id}/data/content` → `{"content": "<extracted text>"}` — JSON form, best fit for credential JSON files
- `GET /api/v1/files/{id}/content` → raw bytes (FileResponse)
- `GET /api/v1/files/{id}/process/status` → `{"status": "pending"|"completed"|"failed"}`
- `GET /api/v1/files/?page=` (`content=true` default), `GET /api/v1/files/search?filename=<glob>`
- Candidate P4 flow: search KB by name → map path→file (filename convention TBD in P4) → `GET /api/v1/files/{file_id}/data/content` → parse JSON → `{path, data, version, updated_at}`. Note: OWUI exposes `meta.size`/`updated_at` but no version field → `version` may be synthesized broker-side (P4 detail; REQ-08 response shape unchanged).
### 1.3 Determinism / reproducibility
## 5. Security Prior Art
- **Raw `zipapp` is NOT reliably byte-reproducible**: zip entries carry the source files' mtimes (`z.write` → `ZipInfo.from_file`), no SOURCE_DATE_EPOCH handling exists anywhere in zipapp, and entry order is `rglob('*')` = `os.scandir` order (unsorted, filesystem-dependent). My two identical staging-dir builds hashed equal *only because* mtimes and scandir order happened to match — a fresh `git clone` (checkout-time mtimes) or a different filesystem breaks it.
- **Fix — normalization post-pass (verified byte-identical across rebuilds)**: after zipapp builds the artifact, the build script reads it (shebang line + zip body), rewrites the zip with entries **sorted by name**, `date_time=(1980,1,1,0,0,0)` (zip epoch minimum), `ZIP_DEFLATED`, preserved `external_attr`, and re-prepends the shebang (+`chmod +x`). Verified: two builds → identical sha256; normalized pyz still executes (`--version` OK). Alternative (rejected): `os.utime`-ing the staging tree pre-build — fixes mtimes but not entry order; the post-pass fixes both and keeps `python3 -m zipapp` as the format mechanism (the normalized file is still "shebang + zip", the zipapp format).
- Reproducibility test: `test_build_cli.py` builds twice into separate dirs, asserts identical sha256, and asserts the embedded `_cli_version.VERSION`.
- Peppered derivation + slow-hash storage: the token derives from a 256-bit random pepper (`BROKER_INSTANCE_PEPPER`, 32 bytes), so an attacker holding `token_hash` faces an offline brute force over the pepper — infeasible. Online guessing is rate-limit-free (spec §14: trusted network + `127.0.0.1` bind), but each guess costs one scrypt/bcrypt op, and the UUID-scoped scan bounds verification work (D-004). Storing only the salted slow hash (never the plaintext token) follows OWASP password-storage guidance; format prefixes keep verification format-aware (D-001).
- Append-only JSONL audit: one JSON object per line; flush-per-write (+ `os.fsync` for the process-restart case) is affordable at the broker's write volume (register/read/admin actions) and preserves the no-loss guarantee; file mode 0600; no rotation — manual `mv` (D-007, spec §13 Q1). Exact flush/fsync choice is a P1 plan decision, not research.
- Session cookie `broker_session`: HttpOnly + SameSite=Lax (D-013); no Secure flag (loopback HTTP by default); Path scoped to the broker.
- Path traversal on `/secrets/<path>`: URL-decode (`urllib.parse.unquote`) BEFORE validation; reject `..` segments, absolute paths, backslashes, NUL; the scope check `path == "coreci/" + agent_id` runs on the decoded value (REQ-07). Static serving of `/static/` and UI must likewise resolve against a fixed root and reject traversal.
- OWUI-runtime boundary: broker binds `127.0.0.1:8765` — bind address controls reachability (PROJECT.md Constraints). The OWUI process hosts both the tool (broker mount) and the KB the broker reads; `BROKER_OWUI_API_KEY` is the only OWUI-side credential the broker holds, and it lives in the broker `.env` (0600 discipline, REQ-15).
### 1.4 Defects / risks (zipapp)
## 6. Testing Prior Art
- **RISK** — stale-asset drift if a *user* sets `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` in `.env`: honored (by design, documented in notes) — the archive still carries the assets, they're just unused. Tests cover the override path.
- **RISK** — first-boot extraction torn reads if two instances share a state dir and the same version: mitigated by temp+`os.replace`; multi-instance sharing one SQLite remains unsupported (as v0.1).
- **NOTE** — shebang `env python3` may resolve to a non-3.11 python on exotic hosts; the broker needs 3.7+ features in practice but is tested on 3.11 only — release notes state "Python 3.11+ required" (D-022 alignment).
- **NOTE** — `sys.path[0]` is the archive only when run as a zipapp; the shim is not meant to run from an unzipped checkout (guard: it can `os.path.isdir` check and fall back to repo-relative roots, or simply document zipapp-only — planner's choice, zero-cost either way).
- **NOTE** — assets land under a dot-directory (`.coreci_broker_assets`) — hidden-dir hygiene in the state dir; broker dot-filtering applies to request paths, not roots (verified `_resolve_static_file`).
- **NOTE** — AC14 (line count) counts `coreci_broker.py` ONLY (REQ-17); the shim/build scripts are separate files → v0.2 needs **zero broker lines**; broker is 1390/1400 — only 10 lines of headroom anyway, so zero-change is also a hard constraint, and it is satisfied (verified: `run_standalone`, `_parse_env_file`, `boot` are importable and sufficient as-is).
- In-process server harness (stdlib, matches D-009/REQ-18): `ThreadingHTTPServer(("127.0.0.1", 0), Handler)` for an ephemeral port in `setUp`; `threading.Thread(target=server.serve_forever, daemon=True)`; drive requests with `urllib.request` against `http://127.0.0.1:{port}`; `shutdown()` + `server_close()` via `addCleanup`.
- Temp state: `tempfile.TemporaryDirectory()` per test for `broker.db`, `broker.audit.log`, `.env`, `data/creds.json`; point the broker at them via env vars (`BROKER_ENV_PATH`, `BROKER_CREDS_FILE`, audit path).
- 0600 modes: `os.stat(path).st_mode & 0o777 == 0o600` after first boot; create with `os.open(path, O_CREAT|O_WRONLY, 0o600)` (umask-independent) in the broker, so the test just asserts.
- AC14 line-count test (REQ-17): `coreci_broker.py` is a single file — tests/UI HTML/client module/reference tools are separate files, so a plain line count of that one file is the spec's measure (< 800, spec §12.14); test reads the file and asserts `sum(1 for _ in f) <= 800` (wc -l-style, no comment stripping).
- TTL tests: set `BROKER_TOKEN_TTL` to 1–2 s and `time.sleep`, or inject an expired `token_expires_at` directly in SQLite (faster, deterministic).
- Dual-hash tests: both `scrypt$…` and `$2b$…` verification paths; CI can only execute the scrypt path (no bcrypt locally) — the bcrypt path's format-awareness is tested via the prefix and mocked/verified against the format contract.
## 2. Gitea Release Assets (REQ-22)
## 7. Research Conclusions
### 2.1 Verified API shapes (live gitea 1.26.1, `git.coreci.dev/swagger.v1.json` + working calls)
- **List**: `GET /api/v1/repos/coreci/broker/releases?limit=20` → `[{id, tag_name, name, body, assets: [{id, name, size, download_count, browser_download_url}], draft, prerelease, created_at}]`. **By tag**: `GET /api/v1/repos/coreci/broker/releases/tags/{tag}` (200 or 404). **Latest**: `GET .../releases/latest`.
- **Create**: `POST /api/v1/repos/{owner}/{repo}/releases` — JSON `CreateReleaseOption` `{tag_name, name, body, draft, prerelease, target_commitish, tag_message}` (schema verified from swagger).
- **Attach asset**: `POST /api/v1/repos/{owner}/{repo}/releases/{id}/assets` — **consumes `multipart/form-data`** (or `application/octet-stream`); params verified from swagger: `id` (path, int64), `name` (query), **`attachment` (formData, type file)** — the multipart field name is `attachment`. Response: attachment JSON with `id`, `name`, `size`, `browser_download_url`.
- **Auth**: `Authorization: token <GITEA_TOKEN>` (verified working on live forge; gitea also accepts Basic `user:token` — use the `token` form).
- **Asset size**: gitea `[attachment] MAX_SIZE` default is 4 MB — the pyz is ~30 KB (deflated) → no issue; NOTE if server policy changes, attachment failure escalates through the bounded-retry path rather than blocking.
- **Multipart with stdlib** (no `requests`): hand-rolled body, ~15 lines — boundary string, `Content-Disposition: form-data; name="attachment"; filename="coreci-broker-vX.Y.Z.pyz"`, `Content-Type: application/octet-stream`, `?name=` query param; gitea parses with standard Go multipart — boundary is ours to choose. Test against a stub gitea (§2.4).
### 2.2 v0.1 attachment forensics (how it was actually done)
- All six releases v0.1.0..v0.1.5 carry `broker-vX.Y.Z.tar.gz` (29 KB → 99 KB, growing with the codebase). Downloaded v0.1.5's asset: **57 members = the complete working tree including `.ciagent/`** (53 git-tracked files + 4 dir entries; matches `git ls-tree -r v0.1.5` exactly, plus directory entries), **no top-level prefix**, uniform `mtime 1791646214` (= the release timestamp), `uid 0/uname root`, modes 0664/0775.
- Interpretation: the archive was produced by the **ship harness's packaging infrastructure** (ship.md step "Create distribution packages" via the `coreci build/package/release upload` CLI — **not installed in this dev environment**; the root-owned uniform timestamps indicate a container-side packaging step) — i.e. **ad-hoc from the repo's perspective: no repo-owned script exists**. The only repo-visible traces are ship.md's generic instructions.
- Conclusion for v0.2: REQ-22 formalizes exactly this — a repo-owned `scripts/release.py` (plus `scripts/build_cli.py`) replaces the absent harness path. The v0.1 *conventions* carry over: per-release asset naming (now `coreci-broker-vX.Y.Z.pyz`, D-020), notes model (§2.3), and the bounded-retry contract (ship.md). The existing source-archive `broker-vX.Y.Z.tar.gz` convention continues to be the harness's own product (best-effort); the repo script owns the `.pyz` (+ digest).
### 2.3 Release notes content model (v0.1.5 body verified; v0.2 sections)
v0.1.5 body sections: `Milestone summary` → `Phases` (P0..P5 with tags) → `Verification` (test counts, security review, audit) → `Deployment`. v0.2 releases add the REQ-22-mandated CLI + docker documentation; recommended body structure (build-able by `scripts/release.py` from PLAN/ROADMAP/requirements data):
1. `Milestone summary` — one paragraph (carries the v0.1 style).
2. `Phases` — phase list with tags (auto from `git log vX.Y.Z-1..vX.Y.Z`/ROADMAP).
3. `Verification` — test count, security review verdict, escalations.
4. `CLI install + usage` — download `coreci-broker-vX.Y.Z.pyz` from this release's assets; `chmod +x` then `./coreci-broker-vX.Y.Z.pyz` or `python3 coreci-broker-vX.Y.Z.pyz`; `--version`; env contract table pointer (README); first-boot pass discipline (printed once, 0600 state files); state-dir anchoring (`BROKER_ENV_PATH`); **sha256 of the .pyz embedded** (plus optional `.sha256` sidecar asset — planner's call; lean default: digest in notes only).
5. `Docker usage` (REQ-23/D-021) — `docker pull git.coreci.dev/coreci/broker:vX.Y.Z` (and `latest`); `docker run -p 8765:8765 -v broker-state:/state …`; state-on-volume contract.
### 2.4 `scripts/release.py` design (stdlib, ~200 lines)
- **Secrets**: token via a shared file-reading loader `scripts/_env_secrets.py` (~20 lines: parse `.ciagent/.env.secrets` `KEY=VALUE`, strip quotes — mirror the broker's `_parse_env_file` semantics). NEVER `os.environ` (GITEA_TOKEN is in `blocked_env_vars`). **Assert token non-empty before any API call** (v1.8 lesson: empty token → gitea 404, masking auth failure).
- Flow: (1) resolve tag → `GET /releases/tags/{tag}`; 404 → `POST /releases` (create with notes body, `draft: false`, `prerelease: false`); (2) `GET .../releases/{id}/assets` — if `coreci-broker-vX.Y.Z.pyz` already present with matching size → **idempotent success** (no duplicate asset on bounded-retry re-run); (3) POST multipart attach (field `attachment`, `?name=`); (4) bounded retry: 3 attempts with backoff on URLError/5xx/HTTPError; on exhaustion → exit non-zero with a machine-readable marker → ship harness commits `release_pending` escalation; **never blocks tag+merge**.
- **Stub-test** (`tests/test_release_upload.py`): `_StubGiteaHandler(BaseHTTPRequestHandler)` on `ThreadingHTTPServer(("127.0.0.1", 0))` (v0.1 `_StubOwuiHandler` precedent): routes `GET /api/v1/repos/{o}/{r}/releases/tags/{tag}` (200 fixture / 404), `POST .../releases` (assert body JSON), `POST .../releases/{id}/assets` (parse multipart with a stdlib parser — e.g. `email` or manual boundary split — assert field name `attachment` + `?name=` + body bytes), plus failure-modes: 500×3 → release_pending path; existing asset → no second upload.
### 2.5 Defects / risks (release assets)
- **RISK** — multipart hand-rolling bugs (boundary quoting, CRLF): mitigated by stub-gitea test with byte-exact assertions.
- **RISK** — release may not exist when the attacher runs (ship harness creates it best-effort): create-or-get flow covers both orders; idempotent re-attach covers partial failure.
- **NOTE** — 4 MB attachment cap (default) — pyz far under; failure would surface through bounded-retry, not silently.
- **NOTE** — `GET /releases?limit=50` paging: prefer `/releases/tags/{tag}` (exact, no paging).
## 3. Daemonless OCI Build + Push (REQ-23, D-019/D-021/D-022)
### 3.1 PULL from Docker Hub (verified live, anonymous, end-to-end)
- `GET https://registry-1.docker.io/v2/` → 401 `WWW-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"`; anonymous rate limit header `ratelimit-limit: 100;w=3600` (100 req/h) — ample for once-per-milestone.
- `GET https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/python:pull` → 200 `{token, access_token, expires_in, issued_at}` — anonymous OK (verified).
- `GET /v2/library/python/manifests/3.11-slim` with `Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json` → **200 `Content-Type: application/vnd.oci.image.index.v1+json` — `python:3.11-slim` IS an OCI image index (16 children: 8 platform manifests + 8 attestation manifests `unknown/unknown`)**. The client MUST resolve the `linux/amd64` child manifest by digest and fetch it separately.
- Verified nuance: Docker Hub currently returns the index **even without Accept** (and even Accepting only schema2) — but the client still sends the full Accept set (contract correctness) and branches on the response `"manifests"` key (index) vs `"layers"` (manifest), not on hope.
- amd64 child (verified, digest drifts upstream — resolve at publish time and record it): `application/vnd.oci.image.manifest.v1+json`, config `application/vnd.oci.image.config.v1+json` (`Cmd: ["python3"]`, Env PATH/LANG/GPG_KEY/`PYTHON_VERSION=3.11.17`/PYTHON_SHA256 — base runs 3.11.17 vs dev 3.11.2, same pinned minor line per D-022), **4 layers** `application/vnd.oci.image.layer.v1.tar+gzip` (≈29.8 MB + 1.3 MB + 14.5 MB + 250 B ≈ 45.7 MB total).
- `GET /v2/library/python/blobs/<digest>` → blob bytes (layers + config). All flows urllib-only.
### 3.2 Auth to the gitea registry (verified live; username+token shapes pinned)
- `GET https://git.coreci.dev/v2/` → 401 `WWW-Authenticate: Bearer realm="https://git.coreci.dev/v2/token",service="container_registry",scope="*"`.
- Token endpoint: `GET https://git.coreci.dev/v2/token?service=container_registry&scope=repository:coreci/broker:pull,push` with **`Authorization: Basic base64(<username>:<GITEA_TOKEN>)`** → 200 `{"token": "<JWT>"}` (HS256; `exp-nbf = 86400` s = 24 h).
- **Verified**: the Basic **username is ignored** — `coreci`, `nosuchuser123`, and empty all return the same JWT (UserID 57, scopes `write:activitypub,…,write:package,…` — the existing GITEA_TOKEN has package-write). The token (as password) identifies the user. Docker login convention: username = the gitea user, password = the token; the client sends any non-empty username.
- **Verified quirk (must handle)**: the token endpoint answers **200 even anonymously** with an unusable JWT (`UserID: -1`, empty scope). A 200 from `/v2/token` proves nothing about push rights — the client must (a) insist on Basic credentials being present, (b) check the issued token is non-empty, and let real operations fail loudly.
- Verified: `Authorization: Bearer <JWT>` on `/v2/` → 200; `HEAD /v2/coreci/broker/blobs/sha256:0…0` → 404 (existence checks work); `GET /v2/coreci/broker/tags/list` → 404 `NAME_UNKNOWN` (the container package doesn't exist until first push — expected today).
### 3.3 PUSH flow + media-type choice
- **Manifest media type: OCI image manifest end-to-end** — `application/vnd.oci.image.manifest.v1+json`, config `application/vnd.oci.image.config.v1+json`, layers `application/vnd.oci.image.layer.v1.tar+gzip`. Rationale: (1) the pulled base child manifest already uses exactly these descriptor types — reusing its 4 layer digests **verbatim** keeps descriptor/mediaType/blob consistency; (2) gitea 1.26 accepts docker schema2 and OCI manifests — OCI is the modern default and what upstream itself serves; (3) Docker CLI ≥ 18.06 and podman/buildah pull OCI manifests from any v2 registry. **Do NOT push an index/manifest-list**: one linux/amd64 manifest per tag; docker pulls it directly.
- **Blob uploads (OCI dist-spec, gitea-conformant)**: primary = **monolithic** `POST /v2/coreci/broker/blobs/uploads/?digest=sha256:<hex>` with the blob as body → 201 (single round trip); fallback = two-step `POST /v2/<name>/blobs/uploads/` → 202 + `Location` → `PUT <Location>?digest=…` → 201. Always `HEAD /v2/<name>/blobs/<digest>` first — 200 → skip upload (base layers re-run-proof: a partially-failed publish resumes cheaply). **Base layers must be re-uploaded to gitea** (~45.7 MB once per milestone) — gitea cannot mount blobs from Docker Hub; acceptable per D-019's once-per-milestone cadence.
- **Redirect trap (verified in CPython 3.11 source)**: `HTTPRedirectHandler.redirect_request` only follows GET/HEAD (301/302/303 POST→GET); **307/308 on a PUT/POST raise HTTPError** — stdlib urllib does NOT follow method-preserving redirects. The client must implement explicit Location-following (read `Location`, `urljoin` against the registry base — gitea's `Location` may be relative or absolute — re-issue the request with method+body). Gitea default local storage answers directly, but the client is correct against any conformance-grade registry.
- **Manifest PUT**: `PUT /v2/coreci/broker/manifests/<tag>` with `Content-Type: application/vnd.oci.image.manifest.v1+json` → 201. Tags: `<milestone-release-tag>` (e.g. `v0.2.3`) **and** `latest` — two PUTs of the same manifest (D-021).
### 3.4 Layer + config + manifest construction (exact recipe)
1. **Broker layer**: deterministic tar (`tarfile`, entries sorted, fixed `TarInfo` metadata — uid/gid 0, uname/gname fixed, `mtime=0`/1980-epoch, file mode 0644) containing `/app/coreci-broker.pyz` (the built artifact). Gzip with `gzip.compress(data, compresslevel=9, mtime=0)` — **`mtime=0` is mandatory** (default `None` = wall clock → nondeterministic digests); fix the compresslevel in code (stable per build).
2. Digests: **layer digest** = sha256 of the gzip blob; **diff_id** = sha256 of the *uncompressed* tar.
3. **New config blob**: copy the pulled base config; append the new diff_id to `rootfs.diff_ids`; set `config.Entrypoint = ["python3", "/app/coreci-broker.pyz"]`, `config.Cmd = []` (so `docker run image --version` appends `--version` through the entrypoint), `config.WorkingDir = "/state"`, keep base `Env`; optional `config.Labels = {"org.opencontainers.image.version": "<tag>"}`. Serialize deterministically (`json.dumps(..., sort_keys=True, separators=(",", ":"))`) → reproducible config digest.
4. **Push order**: HEAD all blobs (4 base layers, new layer, new config) → monolithic POST each missing one → PUT manifest referencing the base's 4 layer descriptors **verbatim** (original digests + mediaTypes) + the new layer descriptor + the new config descriptor. Record the resolved base index digest + amd64 child digest + all blob digests in the release notes (provenance; also `--base-manifest` override argument for exact re-publishes, default = tag resolution with documented upstream drift).
5. Image name: `git.coreci.dev/coreci/broker` (gitea registry = `owner/name`; the package links to the repo by owner+name). gitea creates the package at first push (verified `NAME_UNKNOWN` until then).
### 3.5 Dockerfile role + mapping limits (documentation; daemonless client = source of truth)
- The Dockerfile is for users WITH docker; the daemonless client is what the pipeline publishes. **Equivalence required, byte-identity not.** Self-contained Dockerfile (no pre-built artifact in build context): `FROM python:3.11-slim`; `COPY coreci_broker.py coreci_broker_client.py coreci_broker_ui/ cli/ /app/src/`; `ARG VERSION=dev`; generated `_cli_version.py` via build arg; `RUN python3 -m zipapp /app/src -o /app/coreci-broker.pyz --python "/usr/bin/env python3" --compress`; `WORKDIR /state`; `ENTRYPOINT ["python3", "/app/coreci-broker.pyz"]`; `EXPOSE 8765`.
- Mapping: same base family (tag drift upstream ≠ the pinned digests the client resolved at publish time), same entrypoint, same pyz content, same `/state` state anchoring; differences: docker-build layering (COPY+RUN blobs vs verbatim base layers + one broker layer) and version string provenance (build-arg vs build-time injection). Document both in the Dockerfile header.
### 3.6 Stub-test plan (`tests/test_publish_image.py`; v0.1 stub precedent)
- `_StubRegistryHandler` on an ephemeral port: `GET /v2/` → 401 + `WWW-Authenticate` (realm = stub `/token`); `GET /token` → `{"token":"stub"}` (plus the anonymous-empty-token variant → client must fail cleanly); `GET /v2/<name>/manifests/<ref>` → fixture index (Accept-checked) / child manifest; `GET /v2/<name>/blobs/<digest>` → fixture bytes; `HEAD` blob → 200/404; `POST /v2/<name>/blobs/uploads/` (+`?digest=`) → 201 monolithic (assert body sha256 == digest param) or 202+`Location`; `PUT .../uploads/<uuid>?digest=` → 201; `PUT /v2/<name>/manifests/<tag>` → 201 (assert `Content-Type` == OCI manifest, all referenced digests already uploaded, diff_ids count == layers count).
- Real-manifest fixture: tiny synthetic OCI index/manifest/config/1-2 tiny layers built in-test with the same tar+gzip code (NOT the 45 MB real base — suite stays fast); plus one "live-shape" assertion test using the digests/mediaTypes captured in §3.1 as constants.
- End-to-end offline: stub Docker Hub + stub gitea → run the publish flow → assert both tags PUT, base layer bytes passed through, broker layer contains the pyz at `/app/`.
- Redirect edge: stub answers 307 on the two-step upload PUT → client follows Location explicitly (urllib would raise otherwise).
### 3.7 Defects / risks (daemonless)
- **RISK** — upstream tag drift (`3.11-slim` moves): pin at publish time by recording digests + `--base-manifest` override; notes carry provenance.
- **RISK** — 307/308 not followed by urllib (verified): explicit Location-following required; stub-tested.
- **RISK** — ~46 MB upload per milestone (base layers re-uploaded; no cross-registry mount): bounded, once-per-milestone; HEAD-skip makes retries cheap.
- **RISK** — gitea's anonymous-200 token quirk (verified): empty-scope JWT must not be treated as auth success.
- **NOTE** — monolithic upload limits: gitea handles multi-MB bodies; the config/manifest are KB-scale, the largest blob is the 29.8 MB base layer — fine for monolithic POST (dist-spec; gitea conformance-tested upstream).
- **NOTE** — publish failure follows the same bounded-retry → `release_pending` escalation as releases; never blocks the milestone merge (ROADMAP P3).
## 4. Cross-Cutting: Layout, Security, Line Count
### 4.1 Recommended repo layout (new files only)
```
cli/__main__.py # boot shim (staged into the pyz; NOT coreci_broker.py → outside AC14)
scripts/build_cli.py # stage + zipapp + normalize; emits dist/coreci-broker-vX.Y.Z.pyz (+ sha256)
scripts/release.py # notes builder + create-or-get release + asset attach + bounded retry
scripts/publish_image.py # daemonless registry client (pull base → append layer → push, D-019)
scripts/_env_secrets.py # shared .env-file secrets loader (~20 lines, one discipline point)
Dockerfile # documentation + docker-user path (§3.5)
dist/ # build output (gitignored)
tests/test_cli_packaging.py # build in tmp + boot subprocess + boot-parity flows via urllib
tests/test_build_cli.py # reproducibility: two builds → identical sha256; --version wiring
tests/test_release_upload.py # stub-gitea attachment + retry/escalation paths
tests/test_publish_image.py # stub-registry pull/push flows + manifest assembly
```
- `scripts/_env_secrets.py` shared rather than copy-pasted per script: one place asserts the "file-only, non-empty" token discipline; `python3 scripts/release.py` runs with `sys.path[0] == scripts/` so the sibling import works without path setup.
- **.gitignore additions**: `dist/`, `*.pyz` (root-level strays). Existing patterns already cover `__pycache__`, `*.db`, `*.log`, `.env*`, `data/`.
- Phase mapping (per ROADMAP): Phase 1 = `cli/` + `scripts/build_cli.py` + its tests (REQ-21); Phase 2 = `release.py` + `publish_image.py` + `Dockerfile` + stub tests (REQ-22/23).
### 4.2 Broker changes: ZERO (verified)
- `run_standalone()`, `boot()`, `load_config()`, `_parse_env_file()` are module-level and importable; the shim + scripts need nothing added to the broker. `coreci_broker.py` = 1390/1400 lines (D-018 cap) — v0.2 adds no lines; AC14 stays green by construction.
### 4.3 Test-harness conventions (carried v0.1 §6 + new packaging patterns)
- Packaging test pattern (verified in prototype): `TemporaryDirectory` → `build_cli.build(state=tmp)` → subprocess `Popen(["python3", "<pyz>"], env={BROKER_BIND: "127.0.0.1:0", BROKER_ENV_PATH/DB/AUDIT: tmp paths}, stdout=PIPE)` → read stdout lines until `coreci-broker listening on http://…` (parse base URL; the `BROKER_ADMIN_PASS=` line comes first and is captured for login) → drive `urllib` flows (healthz, login, register, secret read — reuse v0.1 acceptance request patterns) → `proc.terminate()` in cleanup. Boot cost ~1–2 s; suite impact acceptable (v0.1 suite ≈ 2 min).
- Stub servers: extend the `_StubOwuiHandler` pattern (class-level fixtures + hit recording + `addCleanup`) to `_StubGiteaHandler` / `_StubRegistryHandler`; all stubs bind `("127.0.0.1", 0)` and the code under test takes a base URL parameter (no hardcoded hosts in scripts — required for stubbing anyway).
### 4.4 Security posture (v0.2-specific)
- **No secrets in the artifact**: the pyz bundles broker + UI + client only; the client module contains no credentials; the broker generates pepper/admin pass at first boot; `.env` is never inside the archive. Same for the image: no state baked in (`WORKDIR /state` empty at build; first boot generates).
- **Token discipline**: `release.py` / `publish_image.py` read `GITEA_TOKEN` ONLY via the `.env*` file loader (never shell env — structurally blocked by `blocked_env_vars`); assert non-empty before first use (v1.8: empty token → gitea 404 masks auth failure). Stubs use fake tokens; no live-forge writes in tests.
- **Artifact integrity**: sha256 of the `.pyz` computed at build, embedded in release notes; reproducible build makes the digest meaningful across rebuilds.
- **Static disclosure**: v0.1's `_broker_protected_paths` behavior extends unchanged to materialized roots (assets contain no secrets; `.env`/db/audit realpaths remain protected wherever the state dir sits).
## 5. Research Conclusions (planner-ready)
- **Phase 1 (REQ-21)**: `cli/__main__.py` five-step shim (§1.2 — prototype verified end-to-end, lift it), `scripts/build_cli.py` (stage → zipapp `--compress` → normalize post-pass §1.3 → sha256), `tests/test_cli_packaging.py` + `tests/test_build_cli.py`. Zero broker lines.
- **Phase 2 (REQ-22/23)**: `scripts/_env_secrets.py` + `release.py` (create-or-get by tag, idempotent attach, bounded retry → release_pending, notes model §2.3); `scripts/publish_image.py` (§3.1–3.4 flows; OCI manifest end-to-end; monolithic uploads + HEAD skip + explicit Location-following); `Dockerfile` (§3.5); stub tests §2.4/§3.6.
- **Phase 3 (acceptance)**: boot-parity matrix through the artifact (first-boot pass once, 0600 modes, silent second boot, UI + client served, register/read flows), reproducibility sha256, stub-releasability of both pipelines, security review (no secrets in artifact/image; token discipline), then milestone ship: build → attach to `v0.2.3` → publish image tags `v0.2.3` + `latest` → merge.
- **No BLOCKERs found.** All environment facts were verified live (registry auth shapes, index resolution, gitea API, zipapp mechanics) and the packaging design is proven by a working prototype against the real broker sources. Remaining open items are NOTE/RISK grade and owned by the plan: sidecar `.sha256` asset vs digest-in-notes; shim fallback when run unzipped (trivial); `latest`-tag manifest PUT ordering (push version tag first, `latest` second — trivial).
- P1 (skeleton): bootstrap `.env` + first-boot generation (pepper/admin pass printed exactly once — log once, never again on second boot); SQLite schema per spec §5.2 at 0600; audit writer (JSONL, 0600); dual-mode hasher with format prefixes; token engine pure-function of (pepper, inputs); ThreadingHTTPServer + `{"error, message}` router; OWUI mount = `Tools.__init__` starts the daemon-thread server behind a once-per-process guard (no activate() hook exists), `self.valves = self.Valves()` required, `_`-prefixed helpers; standalone `__main__`; path resolution from env, never `__file__`; keep the 800-line budget from the first commit (UI/client/reference are separate files).
- P2 (admin plane): cookie `broker_session` HttpOnly+SameSite=Lax; HTTP Basic + session cookie both accepted on all admin API endpoints; four pages + dashboard served from `coreci_broker_ui/` on the same port; audit query streams the file line-by-line with `next_since` cursor (limit default 100/max 1000).
- P3 (tool auth): register semantics per D-005 (idempotent re-derive, overwrite on input change, pepper on row-create-or-tokenless); verification per D-004 (UUID-scoped slow-hash scan; cache hit re-checks row status/expiry); `GET /agents/<id>` per D-015 (metadata-only); state endpoints with identity enforcement; `BROKER_AGENT_STATE_BACKEND=internal` default.
- P4 (secrets + ecosystem): `OwuiApiCredsSource` builds on the KB read endpoints in §4 (Bearer key; `/{id}` + `/{id}/files` for mapping, `/api/v1/files/{id}/data/content` for content); `FileCredsSource` default `data/creds.json` (D-014); client module stdlib-urllib-only, `data.get("pepper")`, served unauthenticated at `/static/`; reference tools with safe dry-run default.
- P5 (review + ship): AC1–AC13 behavioral + AC14 line-count; security checklist (traversal incl. URL-decoded forms, pepper never re-printed, 0600 modes, token scoping, audit completeness); README with documented deviations; ship to gitea (repo exists before first push).
- ARCHITECTURE.md verification: **no assumption changes required.** The adapter-contract note (OWUI KB-file layout resolved in P4, ARCHITECTURE.md §Notes) stands — candidate endpoints captured in §4. One nuance for P1, not a defect: "mounts the server at activation" maps to constructor-time in the actual OWUI loader (no activate()/deactivate() lifecycle for Tools), and OWUI supplies `__user__` per call rather than a `self.user` attribute — the broker's Tools class owns that convention.
- No genuine blockers found. Environment, OWUI surface, and forge conventions all support the v0.1 scope as specified.
+38 -23
View File
@@ -1,45 +1,60 @@
# Roadmap — Milestone v0.2 (cli-packaging)
# Roadmap — Milestone v0.1 (coreci-broker)
## Overview
One milestone, owner-directed 2026-10-10: package the completed v0.1 broker for distribution. **Milestone v0.2 — cli-packaging** (feature type): the broker as a single-file zipapp CLI (`coreci-broker.pyz`, self-contained), the CLI attached to every gitea release alongside release notes, and a docker container built **once per milestone** and published as a package in the gitea container registry. Scope is CLI + distribution only — no functional changes to the broker (REQ-21..REQ-23). Docker publication starts with v0.2; no retroactive v0.1 image.
One milestone, owner-directed 2026-10-09: implement Requirements Specification v0.5 in full. **Milestone v0.1 — coreci-broker** (feature type): the single-file broker, its SQLite store, audit log, four-page admin UI, tool-auth plane, secrets mediation, public client module, and reference tools, with the spec's §12 acceptance matrix (AC1–AC14) green. The spec's "v0.5" is its requirements-doc revision label, not the product version (D-008).
Versioning (greenfield convention, D-008): the milestone branch is `milestone/v0.2-cli-packaging`; phase tags run on the milestone's own line — phase 0 ships `v0.2.0`, execution phases ship `v0.2.1`..`v0.2.2`, and the final phase's patch (`v0.2.3`) IS the milestone release; no separate minor tag. MAJOR remains 0 until an owner-approved v1 release. Ships go to gitea (`git.coreci.dev/coreci/broker`); releases are best-effort with bounded retry, never blocking tag+merge.
Versioning (greenfield deviation, researchx/nextcraft precedent): the milestone branch is `milestone/v0.1-coreci-broker`; phase tags run on the milestone's own line — phase 0 ships `v0.1.0`, execution phases ship `v0.1.1`..`v0.1.4`, and the final phase's patch (`v0.1.5`) IS the milestone release; no separate minor tag. MAJOR remains 0 until an owner-approved v1 release. Ships go to gitea (`git.coreci.dev/coreci/broker`); the forge repo must exist before the first push.
Autonomy: **guided** — escalate on every decision gate; threshold 0.85; 1 revision iteration; clarify budget 20.
## Phase 0 — Pre-Execution
- **Description**: Specification parsing (REQ-21..REQ-23), clarification (packaging/distribution decisions D-019..D-022), research (zipapp mechanics, gitea release-asset API, gitea container registry + OCI registry HTTP API v2, Docker Hub anonymous pulls), plan (task-level breakdown with waves), grill, MVP/UX check. `/ci-run` phase 0 completes REQUIREMENTS.md v2 section, RESEARCH.md, PLAN.md, PERSONAS.md, and the grill verdict before any code.
- **Description**: Specification parsing, clarification (decisions D-001..D-011 recorded in the init commit), `.ciagent/` reference files, and the pre-execution pipeline stages: specify → clarify → research → plan → grill. Init has landed the reference files; `/ci-run` phase 0 completes RESEARCH.md (environment constraints, spec defects, sibling conventions), PLAN.md (task-level breakdown), PERSONAS.md, and the GRILL verdict before any code.
- **Requirements covered**: none (planning artifacts)
- **Success criteria**: REQ-21..REQ-23 defined and traceable; milestone v0.2 established in config.json and PROJECT.md; D-016+ decisions recorded; PLAN.md + PERSONAS.md committed; grill verdict recorded; MVP/UX sections present; phase tagged `v0.2.0`
- **Success criteria**: `.ciagent/` files reflect the 6-phase structure; REQ-01..REQ-20 defined and traceable; milestone v0.1 established in config.json and PROJECT.md; PLAN.md + PERSONAS.md committed; grill verdict recorded; phase tagged `v0.1.0`
- **Depends on**: —
- **Status**: complete (shipped as `v0.2.0`, release id 970)
- **Status**: complete
## Phase 1 — CLI Packaging + Release Tooling
## Phase 1 — Core Skeleton
- **Description**: `coreci-broker.pyz` build pipeline: stdlib `python3 -m zipapp` build script (no pip) with G-009 normalize pass (sorted entries, fixed timestamps, constant external_attr); self-contained archive bundling `coreci_broker.py` + `coreci_broker_ui/` + `coreci_broker_client.py` with asset materialization at boot (env-resolved roots, never `__file__`); standalone boot parity with v0.1 (first-boot `.env` generation, admin pass printed once, 0600 modes, full admin UI + client module served); `--version` flag; release-asset automation (G-011: `scripts/_env_secrets.py` + `scripts/release.py` land here so the v0.2.1 ship attach is script-driven — no ad-hoc bridge); tests boot the built artifact end-to-end and stub-verify the gitea upload.
- **Requirements covered**: REQ-21, REQ-22
- **Success criteria**: artifact builds reproducibly from a clean checkout (two consecutive same-environment builds → byte-identical sha256); `python3 coreci-broker.pyz` boots with behavior byte-identical to `python3 coreci_broker.py` (v0.1 suite green through the packaged path); UI pages + client module served from the archive; first-boot semantics preserved on second boot; the `v0.2.1` release carries the `.pyz` attached by `scripts/release.py` (D-020 starts here; `v0.2.0` carries none — G-021)
- **Description**: `coreci_broker.py` foundation: `.env` read/write + first-boot generation (pepper + admin pass, printed once), SQLite schema + indexes per spec §5.2, mode-0600 on DB and audit log, append-only JSONL audit writer per §5.3, dual-mode bcrypt/scrypt hasher (D-001), server-side token derivation per §6, stdlib `ThreadingHTTPServer` + JSON router + `{"error, message}` contract per §7, Open WebUI `Tools`-class mount (daemon thread) + standalone `__main__` entry.
- **Requirements covered**: REQ-03, REQ-11, REQ-14, REQ-15, REQ-16, REQ-17, REQ-18
- **Success criteria**: broker boots standalone on an ephemeral port, creates DB + audit log at 0600, prints the generated admin pass exactly once (not on second boot); token derivation matches the §6 formula byte-for-byte; source under 800 lines
- **Depends on**: 0
- **Status**: complete (shipped as `v0.2.1` — 325 cumulative tests green, verify four-layer PASS, .pyz attached by script, asset id 396)
- **Status**: complete
## Phase 2 — Docker + Image Publish
## Phase 2 — Admin Plane
- **Description**: Daemonless publish client per D-019 (once-per-milestone pull pinned base layers / append broker layer / push to `git.coreci.dev/coreci/broker` with version + `latest` tags, D-021/D-022; G-019 wire hardening: per-request timeouts, blob retry with HEAD-skip, explicit 307/308 relative-Location handling); `Dockerfile` + `.dockerignore` (G-015) + consistency-tested equivalence mapping (G-016); README additive section (CLI + docker usage); the `v0.2.2` release attaches the `.pyz` via `scripts/release.py` with CLI + docker notes.
- **Requirements covered**: REQ-23 (+ REQ-21/REQ-22 docs)
- **Success criteria**: daemonless publish client stub-verified end-to-end (no live-forge writes, no container runtime); Dockerfile/.dockerignore consistency tests green; this phase's release (`v0.2.2`) carries the `.pyz` attached alongside the source archive with notes; bounded-retry → `release_pending` paths covered by tests
- **Description**: Session auth (cookie `broker_session` + HTTP Basic both accepted), `POST /api/broker/auth/login`, `logout`, `whoami` (§7.1); app_registrations list/add/revoke/reinstate (§7.2, UUID format validation, 409 on duplicate, revoke keeps existing tokens until TTL); user_allowlist list/add/revoke/reinstate (§7.3); audit query with filters + `next_since` cursor (§7.6); the four UI pages + dashboard + persistent nav (§10) served from `coreci_broker_ui/`; session cleanup.
- **Requirements covered**: REQ-01, REQ-05, REQ-06, REQ-12, REQ-13
- **Success criteria**: AC1 (first-boot login with the printed pass), AC2 (app registration add/remove via UI, persisted in `app_registrations`, UI updates after each action), AC3 (user allowlist same), AC9's login/logout audit lines
- **Depends on**: 1
- **Status**: complete (shipped as `v0.2.2` — 341 cumulative tests green, verify four-layer PASS, .pyz attached by script, asset id 397)
- **Status**: complete
## Phase 3 — Final Review + Ship
## Phase 3 — Tool Auth Plane
- **Description**: Full acceptance: cumulative v0.1 suite green plus v0.2 packaging acceptance (boot parity through the artifact, self-containment, daemonless publish flow); security review (artifact integrity, no secrets in the archive/image, token discipline in upload paths, build-context hygiene); multi-persona review of the milestone diff; audit/reconstruction (REQ-21..23 traceable); milestone ship in the G-012-corrected order — build CLI → squash-merge `milestone/v0.2-cli-packaging` → `main` → tag `v0.2.3` → push → G-013 live smoke (read-only, env-gated) → attach `.pyz` + full notes via `scripts/release.py` → daemonless image publish once (D-019..D-022, with the G-020 closure runbook on failure) → branch cleanup → closeout.
- **Requirements covered**: REQ-21, REQ-22, REQ-23 (acceptance)
- **Success criteria**: all v0.1 + v0.2 acceptance green; security review PASS; `v0.2.3` on `main` with the `.pyz` attached + notes on the gitea release; docker image published as a gitea package (or escalated per the bounded-retry contract + closure runbook, never blocking merge); REQUIREMENTS/ROADMAP closed out; checkpoint cleared
- **Description**: `POST /api/broker/agents/register` (§7.4): allowlist gates → 403 on unknown/revoked UUID or user; idempotent re-registration re-derives from request inputs and overwrites on input change (D-005); pepper returned on row-create or when the caller presents no valid token; TTL set from `BROKER_TOKEN_TTL`. App-auth verification: parse `Authorization: App <UUID> <token>`, UUID-scoped slow-hash scan + in-memory verified-token cache (D-004). `GET /api/broker/agents/<agent_id>` (§7.4). State endpoints `GET/PUT /api/broker/agents/<id>/state` (§9.1) with identity enforcement (calling UUID + token-derived user_id must match stored identity) and the backend toggle `BROKER_AGENT_STATE_BACKEND=internal` (default) `| owui` (D-003).
- **Requirements covered**: REQ-02, REQ-04, REQ-09, REQ-10
- **Success criteria**: AC4 (token derived + stored as slow-hash, same value computed both sides), AC7 (revoked/unknown UUID → 403), AC8 (unknown user → 403), AC10 (TTL expiry → 401 → re-register → new token, audited), AC12 (state read/write with identity enforcement)
- **Depends on**: 2
- **Status**: complete (shipped as `v0.2.3` — milestone release; 356 cumulative tests green, security review PASS-WITH-FINDINGS remediated, `.pyz` attached + notes, image published once)
- **Status**: complete
## Post-Milestone Notes (carry-forward from v0.1, GRILL G-008)
## Phase 4 — Secrets + Client Ecosystem
- **90-day adoption criterion**: `proxmox_readonly_tool.py` runs on `coreci_broker_client.py` with direct KB reads removed by **2027-01-09**; the owner owns the date. No real tool migrates to the broker before the owner-run post-ship OWUI smoke checklist (G-005) passes on the deployed v0.11.3 host. These v0.1 post-ship obligations remain open after the v0.1 ship (2026-10-10) and carry forward unchanged.
- **Description**: Creds-source adapter (D-002): `FileCredsSource` (default; JSON document at `BROKER_CREDS_FILE` mapping paths to secret objects) and `OwuiApiCredsSource` (`BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY`). `GET /api/broker/secrets/<path>` (§7.5): App-auth, scope check `path == "coreci/" + agent_id` → 403 on mismatch, 404 when absent, audit every attempt. Public client module `coreci_broker_client.py` (stdlib urllib only): ensure-registered, state get/put, secret read with 401 → re-register → retry-once; served unauthenticated at `/static/coreci_broker_client.py`. Reference OWUI tools (`reference_tools/`): proxmox reference + a minimal auth-then-use showcase, safe dry-run default (§9, §11; D-006).
- **Requirements covered**: REQ-07, REQ-08, REQ-19, REQ-20
- **Success criteria**: AC5 (valid token reads the secret; tampered token → 401), AC6 (proxmox-registered tool reading `coreci/gitea` → 403); client module fetchable unauthenticated and drives a reference tool end-to-end against a test broker
- **Depends on**: 3
- **Status**: complete
## Phase 5 — Final Review + Ship
- **Description**: Full §12 acceptance matrix as tests (AC1–AC13 as behavioral tests, AC14 as a line-count test); security review pass (path traversal on `<path>`, pepper never re-printed, 0600 modes, token scoping, audit completeness); README with run/deploy notes and documented spec deviations, including the owner-run post-ship live-OWUI smoke checklist (mount/register/read on the deployed v0.11.3 host — G-005/G-006); multi-persona review of the milestone diff; owner approval gate; milestone merge to main and tag `v0.1.5` (final phase patch = milestone release); first push to gitea once the forge repo exists.
- **Requirements covered**: REQ-01..REQ-20 (acceptance)
- **Success criteria**: AC1–AC14 all green (`python3 -m unittest discover -s tests`); security review PASS; README complete with deviations; owner approval recorded; milestone squash-merged to main, tagged `v0.1.5`, release created
- **Depends on**: 4
- **Status**: not_started
## Post-Milestone Notes (owner-promoted from GRILL G-008)
- **90-day adoption criterion**: `proxmox_readonly_tool.py` runs on `coreci_broker_client.py` with direct KB reads removed by **2027-01-09**; the owner owns the date. No real tool migrates to the broker before the post-ship OWUI smoke checklist (G-005) passes on the deployed v0.11.3 host.
+3 -3
View File
@@ -46,9 +46,9 @@
"projects": [],
"active_project": null,
"milestone": {
"version": "v0.2",
"name": "cli-packaging",
"version": "v0.1",
"name": "coreci-broker",
"type": "feature",
"branch": "milestone/v0.2-cli-packaging"
"branch": "milestone/v0.1-coreci-broker"
}
}
-17
View File
@@ -1,17 +0,0 @@
# Build-context hygiene (G-015): no secrets, state, or build artifacts
# may enter an image build context.
.git
.env
.env.*
data/
*.db
*.log
dist/
__pycache__/
tests/
.ciagent/
# Nested matches too (P3-03, CWE-693): the root-anchored `__pycache__/`
# above misses cli/__pycache__/*.pyc — build artifacts must stay out of
# the context wherever they sit in the tree.
**/__pycache__/
**/*.pyc
+1 -5
View File
@@ -10,8 +10,4 @@ __pycache__/
# Broker runtime state
data/
*.db
*.log
# v0.2 build artifacts
dist/
*.pyz
*.log
-42
View File
@@ -1,42 +0,0 @@
# coreci-broker docker image (v0.2, REQ-23).
#
# Equivalence mapping to the daemonless-published image
# (scripts/publish_image.py — the pipeline's source of truth):
# same base family: python:3.11-slim (D-022 pinned minor)
# same artifact: coreci_broker.py + coreci_broker_client.py +
# coreci_broker_ui/ + cli/__main__.py + _cli_version.py,
# packed by the same `python3 -m zipapp` invocation
# (--python "/usr/bin/env python3" --compress)
# same entrypoint: ["python3", "/app/coreci-broker.pyz"]
# same state anchor: WORKDIR /state (volume-backed; first boot generates
# .env, broker.db, broker.audit.log there)
# Documented differences (equivalence required, byte-identity NOT):
# - docker-build layering (COPY + RUN) vs the daemonless client's
# verbatim base layers + one deterministic broker layer
# - version provenance: ARG VERSION here vs build-time _cli_version.py
# injection in scripts/build_cli.py
# - no normalize pass here (docker rebuilds are not digest-pinned)
FROM python:3.11-slim
# Staging mirrors scripts/build_cli.py EXACTLY (P3-03, CWE-636): Docker
# copies directory CONTENTS flattened, so the staging layout must come
# from the destination paths — root sources at /app/src/, UI pages under
# /app/src/coreci_broker_ui/ (cli/__main__.py:80-83 materializes only
# coreci_broker_ui/* members; a flattened copy leaves /ui/* 404ing), and
# the shim staged as the archive-root __main__.py (zipapp boots the
# root __main__.py; the file-to-file rename mirrors build_cli's
# STAGED_SHIM staging).
COPY coreci_broker.py coreci_broker_client.py /app/src/
COPY coreci_broker_ui/ /app/src/coreci_broker_ui/
COPY cli/__main__.py /app/src/__main__.py
ARG VERSION=dev
RUN printf 'VERSION = "%s"\n' "$VERSION" > /app/src/_cli_version.py \
&& python3 -m zipapp /app/src -o /app/coreci-broker.pyz \
--python "/usr/bin/env python3" --compress
WORKDIR /state
ENTRYPOINT ["python3", "/app/coreci-broker.pyz"]
EXPOSE 8765
-222
View File
@@ -1,222 +0,0 @@
# CoreCI Broker
**TL;DR** — CoreCI Broker — single-file credential broker for autonomous agent tools (Open WebUI workspace tool / standalone). This file explains how to stand it up and who to contact; the machinery detail stays lower under Details.
A single-file credential broker for autonomous agent tools on the Open WebUI
host. Tools authenticate with derived tokens instead of shared secrets, the
CEO gates access through UI-managed allowlists of app UUIDs and user IDs,
every read lands in an append-only audit log, and secret reads are path-scoped
to `coreci/<agent_id>` — one Python file, one SQLite DB, a four-page admin UI,
zero pip installs, no build step.
## Quickstart (standalone)
```bash
python3 coreci_broker.py
```
- Stdlib-only runtime, Python 3.11+. Zero pip installs: bcrypt when
importable, stdlib scrypt otherwise (format-prefixed hashes).
- First boot generates `.env`: a 32-byte base64 pepper plus a 32+-char admin
password. The password is printed to the startup log as `BROKER_ADMIN_PASS=…`
**exactly once** — save it immediately; it is never printed again. Lost pass:
remove `BROKER_ADMIN_PASS` from `.env` and reboot — a new one prints once.
- The server binds `127.0.0.1:8765`. Open `http://127.0.0.1:8765/ui/login` and
log in with the printed pass.
- Stop with Ctrl+C. Durable state is exactly three files: `broker.db`,
`broker.audit.log`, `.env` — back them up.
## Configuration
Read on boot from the `.env` at `BROKER_ENV_PATH`. Precedence:
**process env → `.env` → defaults**.
| Variable | Default | Purpose |
|----------|---------|---------|
| `BROKER_BIND` | `127.0.0.1:8765` | Listen address; the bind controls reachability |
| `BROKER_ADMIN_USER` | `admin` | Admin login user name |
| `BROKER_ADMIN_PASS` | generated first boot | Admin login password (32+ random chars) |
| `BROKER_INSTANCE_PEPPER` | generated first boot | 32-byte base64 pepper; token derivation input, synced to SQLite |
| `BROKER_ENV_PATH` | `.env` | Path of the `.env` file the broker reads/writes |
| `BROKER_TOKEN_TTL` | `86400` | Tool token lifetime, seconds (24 h) |
| `BROKER_SESSION_TTL` | `300` | Admin UI session cookie lifetime, seconds |
| `BROKER_DB_PATH` | `broker.db` | SQLite database (mode 0600) |
| `BROKER_AUDIT_LOG_PATH` | `broker.audit.log` | Append-only JSONL audit log (mode 0600) |
| `BROKER_CREDS_SOURCE` | `file` | Credential source adapter: `file` (dev/tests) or `owui` (production) |
| `BROKER_CREDS_FILE` | `data/creds.json` | `FileCredsSource` JSON document: paths → secret objects |
| `BROKER_OWUI_BASE_URL` | — | Open WebUI base URL for the OWUI creds source |
| `BROKER_OWUI_API_KEY` | — | Open WebUI API key (Bearer) for KB read access |
| `BROKER_AGENT_STATE_BACKEND` | `internal` | Agent-state storage: `internal` (rows in broker.db) or `owui` |
| `BROKER_MAX_AGENTS_PER_APP` | `50` | Cap on agent registrations per app UUID |
| `BROKER_UI_ROOT` | `coreci_broker_ui` | Directory served for `/ui/*` pages |
| `BROKER_STATIC_ROOT` | `.` | Directory served for `/static/*` (client module) |
Production: `BROKER_CREDS_SOURCE=owui` with `BROKER_OWUI_BASE_URL` +
`BROKER_OWUI_API_KEY` (0600 `.env` discipline). Keep `BROKER_BIND` on
loopback unless you deliberately widen reachability.
## Running as an Open WebUI workspace tool
Paste `coreci_broker.py` into Open WebUI as a workspace Tool (frontmatter
`requirements:` stays empty — the runtime already ships everything). Notes
that matter:
- The server starts in `Tools.__init__` behind a once-per-process guard; the
OWUI loader has no activate/deactivate hook, and disabling a tool does not
stop a previously started server.
- **After editing the broker tool source, restart Open WebUI.** The loader
re-execs the module on source change, but the previously loaded server keeps
its port — the new instance fails loud with a bind-error message telling you
to restart, rather than silently serving old code.
- Paths (DB, audit log, `.env`, UI roots) resolve from environment config,
never from `__file__` — the OWUI loader deletes the temp file it execs.
- Durable state: `broker.db` + `broker.audit.log` + `.env` — **back those
three up**; SQLite is the source of truth, no HA, restore is a file copy.
- `Tools.status()` reports the serving address; call it from a chat to verify
the mount, or hit `/healthz` on the bind.
## Admin UI
Four pages + dashboard, server-rendered on the same port as the API:
- `/ui/login` — the CEO's bookmark. First-boot pass discipline: the printed
pass is shown once; record it in your secret store on first login.
- `/ui/app_registrations` — allowlist of tool app UUIDs: add, revoke, reinstate.
- `/ui/user_allowlist` — allowlist of user IDs (optional email): same actions.
- `/ui/audit` — filterable, read-only view of the audit log (no export in v0.1).
- `/ui/dashboard` — landing page with persistent nav header and Logout.
Allowlist semantics: **revoke keeps existing tokens working until TTL expiry —
by design.** Revocation immediately blocks new registrations; already-issued
tokens live at most one `BROKER_TOKEN_TTL` (24 h by default).
## Tool authors
Fetch the public client module — unauthenticated, from
`http://<broker>/static/coreci_broker_client.py` — into your workspace and
import it. Flow: `ensure_registered(state)` → `put_state(state)` →
`read_secret("coreci/<agent_id>", state)`. Stdlib-urllib-only (no httpx, no
bcrypt), so any tool can import it; on a 401 it re-registers and retries
once. State endpoints are the tool's only persistence channel — keep `state`
there, not in local files. See `reference_tools/` for the proxmox-style §11
flow and a minimal auth-then-use showcase; downstream calls default to a
safe dry-run.
## Security model
- Derived token: `base64(sha256(pepper:UUID:user_id:agent_id:sorted_tags))` —
44 ASCII chars, no prefix, no challenge protocol. Stored only as a slow
hash (`$2b$…` / `scrypt$…`), never in plaintext.
- Allowlists are the gate: unknown or revoked app UUIDs and unknown users get
403 at registration; the token is the consequence, not the authority.
- Every secret-read attempt is audited (ok/denied/error) with user, UUID,
agent, and path.
- Secret reads are path-scoped: a registration reads exactly
`coreci/<agent_id>` — cross-path → 403, absent → 404.
- The broker never writes to `creds-coreci-coo`; the CEO maintains that KB
through the Open WebUI UI.
- `broker.db`, `broker.audit.log`, and `.env` are mode 0600.
- Binds `127.0.0.1` by default; trusted-network posture — no rate limiting by
design (spec §14).
## Documented spec deviations
Honest list; full detail in `.ciagent/PROJECT.md`:
- **Line-cap amendments (D-016/D-017/D-018)**: the single-file constraint grew
800 → 1000 → 1200 → 1400 lines via the owner-approved §12.14 amendment
pattern (readability over dense cramming). Current source: 1390 lines; AC14
enforces the cap.
- **Dual-mode hashing (D-001)**: spec §6 mandates bcrypt; the runtime is
stdlib-only, so scrypt fallback with format-prefixed hashes.
- **Token verification (D-004)**: spec §7.5's in-memory `token_hash` index is
impossible with salted slow hashes; the broker scans rows scoped to the
presented UUID and keeps a non-authoritative verified-token cache.
- **Pepper rule (D-005)**: the pepper is returned only on row creation or
tokenless registration (the KB-loss recovery path); spec §9's snippet would
raise `KeyError` on re-registration.
- **Spec defects resolved**: §7.5 index, §6-vs-§9 (the broker derives
authoritatively; the tool stores), §9's `KeyError` — see PROJECT.md §Context.
- `BROKER_ADMIN_USER` defaults to `admin` (spec silent on a default).
- Register cap `BROKER_MAX_AGENTS_PER_APP` (50) — a documented addition.
- Reinstate actions audit as `add_app`/`add_user` (no separate audit action).
- In-scope but absent secrets audit as `read` with `result: error` (404s are
audited, not silent).
## Post-ship live-OWUI smoke checklist (owner-run)
Before migrating any real tool (G-005; deployed host pinned v0.11.3, G-006):
1. Confirm the deployed Open WebUI version (expected **v0.11.3**) and that the
KB read endpoints match the researched surface: `/api/v1/knowledge/{id}` and
`/api/v1/files/{id}/data/content`.
2. Add the broker as a workspace tool; check `/healthz` on the bind.
3. CEO first-boot login at `/ui/login` with the printed pass.
4. Add one app UUID and one user to the allowlists via the UI.
5. In a scratch tool: import the client module from `/static/` →
`ensure_registered` → read `coreci/proxmox` via the OWUI creds source.
6. Verify every step in `/ui/audit`.
7. Rollback = disable the tool + restore the three state files.
No real tool migrates before this passes; the 90-day adoption criterion
(proxmox tool on the client, direct KB reads removed) is due 2027-01-09.
## Tests
```bash
timeout 300 python3 -m unittest discover -s tests
```
295 tests, stdlib unittest only (~2 min; one skip where bcrypt is not
importable). The AC1–AC14 acceptance matrix is green.
## Repository layout
| Path | What it is |
|------|------------|
| `coreci_broker.py` | The broker: single file — server, store, audit, auth, UI serving |
| `coreci_broker_client.py` | Public stdlib-only client module (served at `/static/`) |
| `coreci_broker_ui/` | Server-rendered HTML for the four pages + dashboard |
| `reference_tools/` | Tool-author templates (proxmox-style §11 flow + minimal showcase) |
| `tests/` | Stdlib-unittest suite incl. the AC1–AC14 acceptance matrix |
| `.ciagent/` | CI pipeline artifacts (project, requirements, roadmap, decisions) |
## CLI and Docker (v0.2)
From v0.2.1 onward every gitea release attaches the broker packaged as a
self-contained CLI zipapp: `coreci-broker-vX.Y.Z.pyz` (its sha256 is in
the release notes).
**CLI install + usage:**
```bash
# from the release page assets:
chmod +x coreci-broker-vX.Y.Z.pyz
./coreci-broker-vX.Y.Z.pyz --version # prints the release version
./coreci-broker-vX.Y.Z.pyz # boots the standalone broker
# or: python3 coreci-broker-vX.Y.Z.pyz (Python 3.11+)
```
The artifact bundles the broker, the admin UI pages, and the client
module — one file serves everything (UI at `/ui/`, client at
`/static/coreci_broker_client.py`). First boot prints
`BROKER_ADMIN_PASS=...` exactly once and creates `.env`, `broker.db`,
and `broker.audit.log` mode 0600 in the state dir; state is anchored by
`BROKER_ENV_PATH` (relative paths stay CWD-relative). The full
`BROKER_*` env contract is in the Configuration section above.
**Docker:**
```bash
docker pull git.coreci.dev/coreci/broker:<tag> # also :latest
docker run -p 8765:8765 -v broker-state:/state \
git.coreci.dev/coreci/broker:<tag>
```
State lives on the `/state` volume (`WORKDIR /state`; first boot
generates `.env`, `broker.db`, and `broker.audit.log there) — mount a
persistent volume or the admin pass and any registrations are lost with
the container. The image is built and published once per milestone at
the milestone release (see `scripts/publish_image.py` — the daemonless
registry client; the `Dockerfile` documents the docker-user path and
its equivalence mapping).
-126
View File
@@ -1,126 +0,0 @@
# coreci-broker CLI boot shim; staged by scripts/build_cli.py as the root
# __main__.py of the coreci-broker-vX.Y.Z.pyz zipapp (the documented mode —
# running this file unzipped from a source tree is a guarded fallback).
#
# Boot sequence: (1) --version prints _cli_version.VERSION and exits 0 before
# any broker import; (2) the state dir is resolved exactly like the broker
# resolves it (BROKER_ENV_PATH-anchored; relative default paths stay
# CWD-relative — the broker's path contract is inherited unchanged);
# (3) bundled assets are materialized from the running archive
# (zipfile.ZipFile(sys.path[0])) into
# <state>/.coreci_broker_assets/<VERSION>/{ui,static} — ui/ receives the
# bundled coreci_broker_ui/ pages, static/ receives the bundled
# coreci_broker_client.py — via a temp dir plus os.replace (versioned dirs;
# same-version overwrite self-repairs; other versions are never cleaned);
# (4) BROKER_UI_ROOT/BROKER_STATIC_ROOT are pointed at the materialized
# dirs only when the key is absent from both the process env and the user
# .env (user wins); (5) coreci_broker.run_standalone() boots the broker.
# Guard: when sys.path[0] is a directory (or empty) this shim is running
# unzipped from a source tree — materialization is skipped and the broker's
# repo-relative defaults apply.
#
# Private-API coupling (grill G-018, tracked in P3-02): this shim imports
# coreci_broker._parse_env_file and coreci_broker.run_standalone —
# module-level but private-surfaced broker APIs (coreci_broker.py is frozen
# at 1390 lines). Any future broker refactor that removes or renames them
# breaks packaged boots loudly (ImportError/AttributeError); that breakage
# is accepted — the fix is to update this shim and re-run the v0.2 packaged
# acceptance (tests/test_cli_packaging.py).
import os
import shutil
import sys
import tempfile
import zipfile
ASSET_DIR_NAME = ".coreci_broker_assets"
def _version_requested():
return "--version" in sys.argv[1:]
def _resolve_state_dir():
env_path = os.environ.get("BROKER_ENV_PATH") or ".env"
return os.path.dirname(os.path.abspath(env_path)) or os.getcwd()
def _safe_relative(name):
if not name or "\x00" in name or "\\" in name or name.startswith("/"):
return None
parts = name.split("/")
if ".." in parts or any(part.startswith(".") for part in parts):
return None
return name
def _write_asset(target_dir, relative, payload):
target = os.path.join(target_dir, relative)
os.makedirs(os.path.dirname(target), exist_ok=True)
with open(target, "wb") as handle:
handle.write(payload)
def _materialize_assets(archive_path, state_dir):
import _cli_version
base_dir = os.path.join(state_dir, ASSET_DIR_NAME, _cli_version.VERSION)
parent_dir = os.path.dirname(base_dir)
os.makedirs(parent_dir, exist_ok=True)
tmp_dir = tempfile.mkdtemp(dir=parent_dir, prefix=".tmp-")
try:
ui_dir = os.path.join(tmp_dir, "ui")
static_dir = os.path.join(tmp_dir, "static")
os.makedirs(ui_dir)
os.makedirs(static_dir)
with zipfile.ZipFile(archive_path) as archive:
for name in archive.namelist():
if name == "coreci_broker_client.py":
_write_asset(static_dir, name, archive.read(name))
elif name.startswith("coreci_broker_ui/") and not name.endswith("/"):
relative = _safe_relative(name[len("coreci_broker_ui/"):])
if relative is not None:
_write_asset(ui_dir, relative, archive.read(name))
if os.path.isdir(base_dir):
stale_dir = base_dir + ".stale"
shutil.rmtree(stale_dir, ignore_errors=True)
os.replace(base_dir, stale_dir)
try:
os.replace(tmp_dir, base_dir)
except OSError:
os.replace(stale_dir, base_dir)
raise
shutil.rmtree(stale_dir, ignore_errors=True)
else:
os.replace(tmp_dir, base_dir)
finally:
shutil.rmtree(tmp_dir, ignore_errors=True)
return {"ui": os.path.join(base_dir, "ui"), "static": os.path.join(base_dir, "static")}
def _apply_asset_roots(roots, env_path):
import coreci_broker
file_values = dict(coreci_broker._parse_env_file(env_path))
for key, name in (("BROKER_UI_ROOT", "ui"), ("BROKER_STATIC_ROOT", "static")):
if key not in os.environ and key not in file_values:
os.environ[key] = roots[name]
def main():
if _version_requested():
import _cli_version
print(_cli_version.VERSION)
return 0
archive_path = sys.path[0]
if archive_path and not os.path.isdir(archive_path):
env_path = os.environ.get("BROKER_ENV_PATH") or ".env"
roots = _materialize_assets(archive_path, _resolve_state_dir())
_apply_asset_roots(roots, env_path)
import coreci_broker
coreci_broker.run_standalone()
return 0
if __name__ == "__main__":
sys.exit(main())
+7 -22
View File
@@ -7,7 +7,6 @@ licence: MIT
"""
import base64
import datetime
import errno
import hashlib
import hmac
import http.cookies
@@ -729,16 +728,11 @@ def _required_string(value):
return value if isinstance(value, str) and value else None
def _has_control_chars(value):
return any(ord(ch) < 32 or ord(ch) == 127 for ch in value)
def _valid_id(value):
def _valid_user_id(value):
if not _required_string(value):
return None
if value in (".", ".."):
return None
if ":" in value or "/" in value or "\\" in value or _has_control_chars(value):
if (":" in value or "/" in value or "\\" in value
or any(ord(ch) < 32 or ord(ch) == 127 for ch in value)):
return None
return value
@@ -766,7 +760,7 @@ def _apps_add(request):
def _users_add(request):
_add_row(request, "user_allowlist", "user_id", "email", "add_user", _valid_id)
_add_row(request, "user_allowlist", "user_id", "email", "add_user", _valid_user_id)
def _set_row_status(request, table, key, status, action):
@@ -881,10 +875,10 @@ def _agents_register(request):
if len(strings) != 3 or not isinstance(tags, list) or not all(isinstance(v, str) for v in tags):
request.send_error_json(400, "invalid registration")
return
if _valid_id(agent_id) is None:
if ":" in agent_id or "/" in agent_id or "\\" in agent_id:
request.send_error_json(400, "invalid agent_id")
return
if any(":" in tag or _has_control_chars(tag) for tag in tags):
if any(":" in tag for tag in tags):
request.send_error_json(400, "invalid tags")
return
path = "/api/broker/agents/register"
@@ -1349,16 +1343,7 @@ class Tools:
def __init__(self):
self.valves = self.Valves()
try:
self._server = _start_guarded_server()
except OSError as error:
if error.errno != errno.EADDRINUSE:
raise
raise RuntimeError(
f"broker bind failed ({error}): a broker from previously loaded "
"module code is still serving the configured bind; restart "
"Open WebUI to load the updated broker"
) from error
self._server = _start_guarded_server()
def status(self) -> str:
"""Report the broker server state as a one-line string."""
-112
View File
@@ -1,112 +0,0 @@
#!/usr/bin/env python3
"""Sanctioned secrets loader for the v0.2 distribution scripts (stdlib only).
Secrets resolve ONLY from env files on disk — never from the process
environment (``GITEA_TOKEN`` sits in ``config.security.bash_allowlist
.blocked_env_vars``, so shell-env tokens are structurally blocked; the
v1.8 lesson: an empty or mis-sourced token makes gitea answer 404 rather
than 401, masking the auth failure).
Parsing mirrors ``coreci_broker._parse_env_file`` semantics exactly:
blank lines and ``#`` comments are skipped, keys/values are whitespace-
stripped, and a value wrapped in matching single or double quotes is
unquoted.
File precedence — later files win. Locations in order (``root``, then the
sanctioned ``.ciagent/`` dir under it); each location in ``config.json``
``secrets.env_files`` pattern order (``.env``, ``.env.secrets``, then the
``.env.*`` glob, sorted)::
<root>/.env, <root>/.env.secrets, <root>/.env.*
<root>/.ciagent/.env, <root>/.ciagent/.env.secrets, <root>/.ciagent/.env.*
``load()`` returns the merged mapping; ``get_token()`` enforces the
non-empty assert that callers MUST let happen before their first API
call. This module never prints secret values.
"""
import glob
import os
ENV_FILE_PATTERNS = (".env", ".env.secrets", ".env.*")
CIAGENT_SUBDIR = ".ciagent"
class SecretsError(RuntimeError):
"""A required secret is missing or empty in the sanctioned env files."""
def default_root():
"""Repo root: the parent directory of this scripts/ folder."""
return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
def _parse_env_file(path):
"""Mirror coreci_broker._parse_env_file (comments, blanks, quotes)."""
entries = []
if not os.path.isfile(path):
return entries
with open(path, "r", encoding="utf-8") as handle:
for raw in handle:
line = raw.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, value = (part.strip() for part in line.partition("="))
if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
value = value[1:-1]
if key:
entries.append((key, value))
return entries
def _candidate_files(root):
"""Ordered, deduplicated env-file paths; later files win."""
seen = set()
files = []
for location in (root, os.path.join(root, CIAGENT_SUBDIR)):
for pattern in ENV_FILE_PATTERNS:
for path in sorted(glob.glob(os.path.join(location, pattern))):
if not os.path.isfile(path):
continue
real = os.path.abspath(path)
if real in seen:
continue
seen.add(real)
files.append(real)
return files
def load(root=None):
"""Merge every sanctioned env file under ``root`` into one dict.
``root`` defaults to the repo root (the parent of scripts/).
"""
base = os.path.abspath(root) if root is not None else default_root()
values = {}
for path in _candidate_files(base):
for key, value in _parse_env_file(path):
values[key] = value
return values
def get_token(var="GITEA_TOKEN", root=None):
"""Return the non-empty value of ``var`` from the env files only.
Raises SecretsError when the value is missing, empty, or whitespace-
only — callers must surface this BEFORE any HTTP call (an empty token
makes gitea answer 404 and masks the auth failure — the v1.8 lesson).
"""
token = load(root).get(var, "")
if not token.strip():
base = os.path.abspath(root) if root is not None else default_root()
searched = ", ".join(
os.path.join(location, pattern)
for location in (base, os.path.join(base, CIAGENT_SUBDIR))
for pattern in ENV_FILE_PATTERNS
)
raise SecretsError(
"{} is missing or empty in the sanctioned env files "
"(searched: {}); it never resolves from the process "
"environment".format(var, searched)
)
return token.strip()
-141
View File
@@ -1,141 +0,0 @@
#!/usr/bin/env python3
"""Build the coreci-broker CLI zipapp artifact (stdlib only).
Usage: python3 scripts/build_cli.py <version>
Stages an explicit allowlist into a temp dir, packs it with
``python3 -m zipapp ... --python "/usr/bin/env python3" --compress``, then
runs a normalize post-pass (G-009) so same-environment rebuilds are
byte-identical: entries sorted by name, ``date_time=(1980,1,1,0,0,0)``,
``ZIP_DEFLATED``, ``external_attr`` set to constants (files
``0o100644 << 16``, dirs ``0o40755 << 16`` — never preserved/umask-derived),
shebang re-prepended, exec bit preserved. Prints the artifact sha256.
"""
import argparse
import hashlib
import io
import os
import shutil
import stat
import subprocess
import sys
import tempfile
import zipfile
ROOT_SOURCES = ("coreci_broker.py", "coreci_broker_client.py")
UI_SOURCE_DIR = "coreci_broker_ui"
SHIM_SOURCE = os.path.join("cli", "__main__.py")
STAGED_SHIM = "__main__.py"
VERSION_MODULE = "_cli_version.py"
FIXED_DATE_TIME = (1980, 1, 1, 0, 0, 0)
FILE_EXTERNAL_ATTR = 0o100644 << 16
DIR_EXTERNAL_ATTR = 0o40755 << 16
def repo_root():
return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
def artifact_name(version):
tag = version[1:] if version.startswith("v") else version
return "coreci-broker-v{}.pyz".format(tag)
def stage_sources(repo, staging):
for name in ROOT_SOURCES:
shutil.copyfile(os.path.join(repo, name), os.path.join(staging, name))
ui_root = os.path.join(repo, UI_SOURCE_DIR)
staged_ui = os.path.join(staging, UI_SOURCE_DIR)
for current, _dirs, files in os.walk(ui_root):
relative = os.path.relpath(current, ui_root)
target_dir = staged_ui if relative == "." else os.path.join(staged_ui, relative)
os.makedirs(target_dir, exist_ok=True)
for name in sorted(files):
shutil.copyfile(os.path.join(current, name), os.path.join(target_dir, name))
shutil.copyfile(os.path.join(repo, SHIM_SOURCE), os.path.join(staging, STAGED_SHIM))
def write_version_module(staging, version):
with open(os.path.join(staging, VERSION_MODULE), "w", encoding="utf-8") as handle:
handle.write('VERSION = "{}"\n'.format(version))
def run_zipapp(staging, artifact):
command = [
sys.executable, "-m", "zipapp", staging,
"-o", artifact,
"--python", "/usr/bin/env python3",
"--compress",
]
completed = subprocess.run(command, capture_output=True, text=True)
if completed.returncode != 0:
detail = (completed.stderr or completed.stdout).strip()
raise RuntimeError(
"zipapp failed with code {}: {}".format(completed.returncode, detail)
)
def normalize_archive(artifact):
with open(artifact, "rb") as handle:
raw = handle.read()
if not raw.startswith(b"#!"):
raise RuntimeError("zipapp artifact is missing its shebang prefix")
shebang = raw.split(b"\n", 1)[0] + b"\n"
entries = []
with zipfile.ZipFile(io.BytesIO(raw)) as archive:
for info in archive.infolist():
payload = b"" if info.is_dir() else archive.read(info)
entries.append((info.filename, payload))
entries.sort(key=lambda entry: entry[0])
body = io.BytesIO()
with zipfile.ZipFile(body, "w", compression=zipfile.ZIP_DEFLATED) as archive:
for name, payload in entries:
info = zipfile.ZipInfo(name, date_time=FIXED_DATE_TIME)
info.compress_type = zipfile.ZIP_DEFLATED
info.external_attr = DIR_EXTERNAL_ATTR if name.endswith("/") else FILE_EXTERNAL_ATTR
archive.writestr(info, payload)
mode = os.stat(artifact).st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH
temp_path = artifact + ".tmp"
with open(temp_path, "wb") as handle:
handle.write(shebang)
handle.write(body.getvalue())
os.replace(temp_path, artifact)
os.chmod(artifact, mode)
def sha256_file(path):
digest = hashlib.sha256()
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(65536), b""):
digest.update(chunk)
return digest.hexdigest()
def build(version, repo=None, dist_dir=None):
repo = os.path.abspath(repo or repo_root())
dist_dir = dist_dir or os.path.join(repo, "dist")
os.makedirs(dist_dir, exist_ok=True)
artifact = os.path.join(dist_dir, artifact_name(version))
with tempfile.TemporaryDirectory(prefix="coreci-broker-cli-") as staging:
stage_sources(repo, staging)
write_version_module(staging, version)
run_zipapp(staging, artifact)
normalize_archive(artifact)
return artifact
def main(argv=None):
parser = argparse.ArgumentParser(
description="Build the coreci-broker CLI zipapp artifact."
)
parser.add_argument("version", help="release version of the artifact, e.g. v0.2.1")
args = parser.parse_args(argv)
artifact = build(args.version)
print(artifact)
print("sha256: {}".format(sha256_file(artifact)))
return 0
if __name__ == "__main__":
sys.exit(main())
-844
View File
@@ -1,844 +0,0 @@
#!/usr/bin/env python3
"""Daemonless OCI image publish client (D-019; stdlib only).
Usage:
python3 scripts/publish_image.py <tag> [--smoke]
[--artifact PATH] [--hub-registry URL] [--hub-auth URL]
[--gitea-registry URL] [--env-root DIR] [--owner O] [--repo R]
[--base-manifest DIGEST] [--timeout SECS]
Flow (REQ-23, RESEARCH §3): pull the pinned ``python:3.11-slim`` base
from the public registry anonymously (token -> OCI index -> resolve the
linux/amd64 child manifest by digest -> config + layer blobs; digests
recorded for provenance; ``--base-manifest`` overrides the tag
resolution for exact re-publishes) -> authenticate to the gitea
container registry (``GET /v2/token`` with Basic credentials — the
gitea token comes only from the sanctioned env files via
``scripts/_env_secrets.py``; the endpoint answers 200 even anonymously
with an empty-scope JWT, so the issued token must be non-empty and
failures must be loud — never treat a 200 alone as authorization) ->
push: HEAD each blob first (200 -> skip — retries resume cheap; the
~45.7 MB base layers are re-uploaded once per milestone because the
registries cannot cross-mount) -> monolithic blob upload
``POST /v2/<name>/blobs/uploads/?digest=`` (202 + ``Location`` ->
two-step PUT) with explicit Location-following on 307/308 (stdlib
urllib does NOT method-preserve on PUT/POST redirects) -> deterministic
broker layer (sorted tar entries, fixed metadata, member at the stable
path ``/app/coreci-broker.pyz``, ``gzip.compress(..., mtime=0)``) ->
new config blob (base config + appended diff_id; Entrypoint
``["python3", "/app/coreci-broker.pyz"]``, ``Cmd []``, ``WorkingDir
"/state"``, version label; serialized with sort_keys) -> OCI manifest
PUT with the base layer descriptors reused verbatim under ``<tag>``
FIRST, then ``latest`` (D-021).
Exit codes: 0 only after both manifest PUTs confirm 201; 1 after a
remote failure — bounded retries (3 attempts, 2s/4s backoff) exhausted
OR a permanent registry answer — printing the machine-readable
``IMAGE_PENDING <image>:<tag>`` marker for the ship escalation (never
blocks tag+merge); 2 for local precondition errors (missing artifact,
token, or config — nothing attempted remotely, no marker). Token values
are never printed.
Smoke mode (``--smoke``; G-013) — read-only pre-ship verify. NEVER part
of the default publish flow (the flag dispatches an early return; the
publish path above never consults it): four GET-only checks against
the live endpoints, run in order and stopping at the first failure —
nothing is uploaded, no manifest is PUT, no remote state is mutated.
1. ``GET {gitea}/v2/`` must answer 401 with a Bearer challenge whose
realm is the gitea token endpoint (the realm is recorded for 2).
2. Token->JWT exchange at that recorded realm
(``?service=container_registry&scope=repository:{o}/{r}:pull,push``)
with Basic auth — the gitea token resolves ONLY from the sanctioned
env files via ``scripts/_env_secrets.py`` (never the shell
environment); an empty token in the 200 answer is the anonymous-200
trap and fails loudly.
3. ``GET {gitea}/api/v1/repos/{o}/{r}/releases/tags/<tag>`` — 200
(exists) and 404 (not yet created) are BOTH acceptable pre-ship and
the line says which; the release's asset names print as a fact, and
a hard assertion binds ONLY when the tag is exactly ``v0.2.0``
(G-021 forge-side confirmation: the phase-0 release predates the
CLI and must carry NO ``.pyz`` asset, so any ``.pyz`` in its asset
list fails the smoke — the structural half of the guarantee,
release.py taking the tag as a positional argument with no v0.2.0
code path, is already machine-checked in the suite).
4. Docker Hub anonymous index resolve of ``python:3.11-slim`` with the
full Accept set must yield an index whose linux/amd64 child is
resolved; the amd64 child digest is printed — the base-drift fact
the P3-05 ship compares against expectations.
Smoke exit codes mirror the publish flow: 0 all four PASS; 1 with one
clear FAILED line per failure (checks after the failed one never run);
2 for local preconditions (missing token — nothing attempted
remotely). Smoke never prints token values and never emits
``IMAGE_PENDING`` (nothing was attempted, so nothing is pending).
Live gating (G-013): the LIVE smoke executes only at the P3-05 ship
step, which sets ``PUBLISH_SMOKE=1`` in the environment before invoking
this mode — this script neither reads nor requires that variable (the
SHIP GATE owns the gating). The smoke's LOGIC is what the default
unittest suite exercises (stub-tested, zero network access); the live
smoke itself never runs in the default suite.
"""
import argparse
import base64
import gzip
import hashlib
import http.client
import io
import json
import os
import re
import sys
import tarfile
import time
import urllib.error
import urllib.parse
import urllib.request
_SCRIPTS_DIR = os.path.dirname(os.path.abspath(__file__))
if _SCRIPTS_DIR not in sys.path:
sys.path.insert(0, _SCRIPTS_DIR)
import _env_secrets # sibling import (scripts/ dir on sys.path)
METADATA_TIMEOUT_SECONDS = 30
BLOB_TIMEOUT_SECONDS = 120
MAX_ATTEMPTS = 3
BACKOFF_SECONDS = (2, 4)
MAX_REDIRECTS = 5
BASE_REPO = "library/python"
BASE_TAG = "3.11-slim"
HUB_AUTH_SERVICE = "registry.docker.io"
GITEA_AUTH_SERVICE = "container_registry"
# The smoke-mode G-021 forge-side no-.pyz assertion binds ONLY this
# tag: the phase-0 release predates the CLI (D-020 starts at v0.2.1).
SMOKE_G021_TAG = "v0.2.0"
OCI_INDEX = "application/vnd.oci.image.index.v1+json"
OCI_MANIFEST = "application/vnd.oci.image.manifest.v1+json"
OCI_CONFIG = "application/vnd.oci.image.config.v1+json"
OCI_LAYER = "application/vnd.oci.image.layer.v1.tar+gzip"
DOCKER_LIST = "application/vnd.docker.distribution.manifest.list.v2+json"
DOCKER_MANIFEST = "application/vnd.docker.distribution.manifest.v2+json"
ACCEPT_MANIFESTS = ", ".join(
(OCI_INDEX, DOCKER_LIST, OCI_MANIFEST, DOCKER_MANIFEST))
class PublishError(Exception):
"""Image-publisher failure base."""
class TransientFailure(PublishError):
"""Connection error or 5xx persisting past the bounded retries."""
class RegistryApiError(PublishError):
"""Permanent registry answer carrying the remote detail."""
class LocalError(PublishError):
"""Local precondition failure (nothing was attempted remotely)."""
def repo_root():
return os.path.dirname(_SCRIPTS_DIR)
def load_release_defaults(root=None):
"""Read the release block from .ciagent/config.json ({} if absent)."""
path = os.path.join(root or repo_root(), ".ciagent", "config.json")
try:
with open(path, "r", encoding="utf-8") as handle:
config = json.load(handle)
except (OSError, ValueError):
return {}
block = config.get("release")
return block if isinstance(block, dict) else {}
def _error_detail(body):
try:
payload = json.loads(body.decode("utf-8", "replace"))
except ValueError:
text = body.decode("utf-8", "replace").strip()
return text[:200] if text else "<empty body>"
if isinstance(payload, dict):
for key in ("errors", "message"):
if payload.get(key):
return json.dumps(payload[key])[:200]
return json.dumps(payload)[:200] if not isinstance(payload, str) else payload[:200]
def _header(headers, name):
"""Case-insensitive response-header lookup — ``_attempt_once``
returns a plain dict whose keys keep the wire casing, and registry
header casing is not guaranteed (e.g. WWW-Authenticate)."""
for key, value in headers.items():
if key.lower() == name.lower():
return value
return None
def _parse_json(body, context, status):
"""Parse a remote JSON body (token / manifest / config answers).
Malformed bodies raise RegistryApiError — never an uncaught
ValueError — so main()'s handler keeps the exit-1 + IMAGE_PENDING
marker contract for this failure class (CWE-754).
"""
try:
return json.loads(body.decode("utf-8", "replace"))
except ValueError as error:
raise RegistryApiError(
"{} answered malformed JSON ({})".format(context, error),
status, body)
def _attempt_once(method, url, headers, data, timeout):
"""One HTTP exchange; HTTPError becomes a completed answer."""
request = urllib.request.Request(url, data=data, method=method)
for key, value in headers.items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
body = b"" if method == "HEAD" else response.read()
return response.status, body, dict(response.headers.items())
except urllib.error.HTTPError as error:
body = b"" if method == "HEAD" else error.read()
return error.code, body, dict((error.headers or {}).items())
def _origin(url):
"""(scheme, netloc) of a URL — two URLs share an origin iff both match."""
parts = urllib.parse.urlsplit(url)
return (parts.scheme, parts.netloc)
def _attempt_following(method, url, headers, data, timeout):
"""One attempt, following 307/308 redirects explicitly (urllib does
NOT method-preserve on PUT/POST — the Location may be relative).
Cross-origin discipline (CWE-200): when a followed Location resolves
to a different origin than the ORIGINAL request's, the Authorization
header is stripped from the re-issued request — the registry JWT
must never reach another origin (blob-upload Locations are commonly
presigned external storage). Same-origin redirects keep it; once
stripped it is never re-attached."""
current_url = url
current_headers = dict(headers)
for _ in range(MAX_REDIRECTS + 1):
status, body, resp_headers = _attempt_once(
method, current_url, current_headers, data, timeout)
if status in (307, 308) and resp_headers.get("Location"):
next_url = urllib.parse.urljoin(
current_url, resp_headers["Location"])
if _origin(next_url) != _origin(url):
current_headers.pop("Authorization", None)
current_url = next_url
continue
return status, body, resp_headers
raise RegistryApiError(
"too many redirects following {} {}".format(method, url), 500, b"")
def _request(method, url, headers, data, timeout,
max_attempts=None, backoff=None):
"""Bounded-retry HTTP: retries connection errors and 5xx answers.
Returns (status, body, response_headers) for any answer below 500;
a 4xx returns immediately (permanent). Raises TransientFailure once
the retry budget is exhausted.
"""
attempts = MAX_ATTEMPTS if max_attempts is None else int(max_attempts)
delays = BACKOFF_SECONDS if backoff is None else tuple(backoff)
failure = "unknown failure"
for attempt in range(1, attempts + 1):
try:
status, body, resp_headers = _attempt_following(
method, url, headers, data, timeout)
except (OSError, http.client.HTTPException) as error:
status = None
failure = "connection error: {}: {}".format(
type(error).__name__, error)
else:
if status < 500:
return status, body, resp_headers
failure = "HTTP {}: {}".format(status, _error_detail(body))
if attempt < attempts and delays:
time.sleep(delays[min(attempt - 1, len(delays) - 1)])
raise TransientFailure(
"{} {} failed after {} attempt(s): {}".format(
method, url, attempts, failure))
def _hub_token(hub_auth, scope):
query = urllib.parse.urlencode(
{"service": HUB_AUTH_SERVICE, "scope": scope})
status, body, _ = _request(
"GET", "{}?{}".format(hub_auth, query), {}, None,
METADATA_TIMEOUT_SECONDS)
if status != 200:
raise RegistryApiError("hub token request failed", status, body)
payload = _parse_json(body, "hub token endpoint", status)
token = ""
if isinstance(payload, dict):
token = payload.get("token") or payload.get("access_token") or ""
if not token:
raise RegistryApiError("hub token endpoint returned no token",
status, body)
return token
def _fetch_manifest(registry, repository, ref, bearer, accept=ACCEPT_MANIFESTS):
url = "{}/v2/{}/manifests/{}".format(registry.rstrip("/"), repository, ref)
status, body, _ = _request(
"GET", url, {"Authorization": "Bearer {}".format(bearer),
"Accept": accept}, None, METADATA_TIMEOUT_SECONDS)
if status != 200:
raise RegistryApiError(
"manifest fetch {} {}".format(repository, ref), status, body)
return _parse_json(
body, "manifest fetch {} {}".format(repository, ref), status)
def _resolve_amd64_child(index):
"""linux/amd64 child digest from a base index; malformed index
shapes degrade to the not-found RegistryApiError, never a crash."""
manifests = index.get("manifests") if isinstance(index, dict) else None
if not isinstance(manifests, list):
manifests = []
for descriptor in manifests:
if not isinstance(descriptor, dict):
continue
platform = descriptor.get("platform")
if not isinstance(platform, dict):
continue
if platform.get("os") == "linux" and platform.get(
"architecture") == "amd64":
return descriptor.get("digest")
raise RegistryApiError(
"no linux/amd64 child manifest in the base index", 404,
json.dumps(manifests).encode("utf-8", "replace"))
def _fetch_blob(registry, repository, digest, bearer, timeout):
url = "{}/v2/{}/blobs/{}".format(
registry.rstrip("/"), repository, digest)
status, body, _ = _request(
"GET", url, {"Authorization": "Bearer {}".format(bearer)}, None,
timeout)
if status != 200:
raise RegistryApiError("blob fetch {}".format(digest), status, body)
if "sha256:" + hashlib.sha256(body).hexdigest() != digest:
raise RegistryApiError("blob digest mismatch {}".format(digest),
500, b"")
return body
def _child_parts(child):
"""Validated (config descriptor, layer descriptors) of a base child
manifest — missing or malformed keys raise a clear RegistryApiError
(never an uncaught KeyError), keeping the marker contract for this
failure class (CWE-754)."""
def malformed(reason):
return RegistryApiError(
"malformed base child manifest: {}".format(reason), 500,
json.dumps(child).encode("utf-8", "replace"))
if not isinstance(child, dict):
raise malformed("not a JSON object")
config = child.get("config")
if not isinstance(config, dict) or not config.get("digest"):
raise malformed("missing or empty config.digest")
layers = child.get("layers")
if not isinstance(layers, list):
raise malformed("missing or non-list layers")
for descriptor in layers:
if not isinstance(descriptor, dict) or not descriptor.get("digest"):
raise malformed("layer descriptor without a digest")
return config, layers
def _gitea_jwt(token_url, owner, repo, token, timeout):
"""Token->JWT exchange at a token endpoint URL (Basic auth).
``token_url`` is the FULL endpoint: the registry's ``/v2/token``
for the publish flow, or the realm recorded from a ``/v2/``
challenge for the smoke (the standard token-auth flow).
"""
query = urllib.parse.urlencode({
"service": GITEA_AUTH_SERVICE,
"scope": "repository:{}/{}:pull,push".format(owner, repo),
})
basic = base64.b64encode(
"publisher:{}".format(token).encode("utf-8")).decode("ascii")
status, body, _ = _request(
"GET", "{}?{}".format(token_url, query),
{"Authorization": "Basic {}".format(basic)}, None, timeout)
if status != 200:
raise RegistryApiError("gitea registry token request failed",
status, body)
payload = _parse_json(body, "gitea registry token endpoint", status)
jwt = (payload.get("token") or "") if isinstance(payload, dict) else ""
if not jwt:
# The endpoint answers 200 even anonymously with an unusable
# (empty-scope) JWT — a 200 proves nothing about push rights.
raise RegistryApiError(
"gitea registry token endpoint returned an empty token "
"(anonymous-200 trap refused; never treat 200 as authorization)",
status, body)
return jwt
def _bearer_challenge(www_authenticate):
"""Realm recorded from a Bearer WWW-Authenticate challenge.
Returns None when the header is absent, is not a Bearer challenge,
or carries no realm — a challenge without a realm cannot be
followed to the token endpoint, so the smoke treats that as no
challenge at all.
"""
if not www_authenticate:
return None
parts = www_authenticate.strip().split(None, 1)
if len(parts) < 2 or parts[0].lower() != "bearer":
return None
match = re.search(r'\brealm\s*=\s*"([^"]*)"', parts[1])
return match.group(1) if match else None
def _asset_names(release):
"""Asset display names from a gitea release payload — odd shapes
degrade to [] (the read-only smoke never crashes on them)."""
if not isinstance(release, dict):
return []
assets = release.get("assets")
if not isinstance(assets, list):
return []
names = []
for asset in assets:
if isinstance(asset, dict) and isinstance(asset.get("name"), str):
names.append(asset["name"])
return names
def _blob_exists(gitea_registry, name, digest, jwt, timeout):
url = "{}/v2/{}/blobs/{}".format(
gitea_registry.rstrip("/"), name, digest)
status, _, _ = _request(
"HEAD", url, {"Authorization": "Bearer {}".format(jwt)}, None,
timeout)
return status == 200
def _upload_blob(gitea_registry, name, digest, blob, jwt, timeout):
base_url = gitea_registry.rstrip("/")
url = "{}/v2/{}/blobs/uploads/?digest={}".format(base_url, name, digest)
headers = {"Authorization": "Bearer {}".format(jwt),
"Content-Type": "application/octet-stream"}
status, body, resp_headers = _request(
"POST", url, headers, blob, timeout)
if status in (200, 201):
return
if status == 202 and resp_headers.get("Location"):
location = urllib.parse.urljoin(url, resp_headers["Location"])
if "digest=" not in location:
separator = "&" if "?" in location else "?"
location = "{}{}digest={}".format(location, separator, digest)
status, body, _ = _request(
"PUT", location, headers, blob, timeout)
if status in (200, 201):
return
raise RegistryApiError("blob upload {}".format(digest), status, body)
def _push_blob_if_missing(gitea_registry, name, digest, blob, jwt, timeout):
"""HEAD-skip then upload — retries resume cheap."""
if _blob_exists(gitea_registry, name, digest, jwt, timeout):
return False
_upload_blob(gitea_registry, name, digest, blob, jwt, timeout)
return True
def build_broker_layer(artifact_bytes):
"""Deterministic layer carrying /app/coreci-broker.pyz.
Returns (gzip_blob, layer_digest, diff_id): sorted single entry,
fixed TarInfo metadata, gzip with mtime=0 — byte-stable rebuilds.
"""
info = tarfile.TarInfo("app/coreci-broker.pyz")
info.size = len(artifact_bytes)
info.mode = 0o644
info.uid = 0
info.gid = 0
info.uname = ""
info.gname = ""
info.mtime = 0
buffer = io.BytesIO()
with tarfile.open(fileobj=buffer, mode="w",
format=tarfile.GNU_FORMAT) as archive:
archive.addfile(info, io.BytesIO(artifact_bytes))
raw = buffer.getvalue()
compressed = gzip.compress(raw, compresslevel=9, mtime=0)
layer_digest = "sha256:" + hashlib.sha256(compressed).hexdigest()
diff_id = "sha256:" + hashlib.sha256(raw).hexdigest()
return compressed, layer_digest, diff_id
def build_config_blob(base_config, diff_id, tag):
"""Base config + appended diff_id + CLI entrypoint/WorkingDir/label.
Serialized with sort_keys — deterministic config digest.
"""
config = dict(base_config)
rootfs = dict(base_config.get("rootfs") or {})
rootfs["diff_ids"] = list(rootfs.get("diff_ids") or []) + [diff_id]
if "type" not in rootfs:
rootfs["type"] = "layers"
config["rootfs"] = rootfs
inner = dict(base_config.get("config") or {})
inner["Entrypoint"] = ["python3", "/app/coreci-broker.pyz"]
inner["Cmd"] = []
inner["WorkingDir"] = "/state"
inner["Labels"] = {"org.opencontainers.image.version": tag}
config["config"] = inner
return json.dumps(config, sort_keys=True,
separators=(",", ":")).encode("utf-8")
def build_manifest_blob(base_layer_descriptors, broker_layer_descriptor,
config_descriptor):
manifest = {
"schemaVersion": 2,
"mediaType": OCI_MANIFEST,
"config": config_descriptor,
"layers": list(base_layer_descriptors) + [broker_layer_descriptor],
}
return json.dumps(manifest, sort_keys=True,
separators=(",", ":")).encode("utf-8")
def _descriptor(digest, size, media_type):
return {"mediaType": media_type, "digest": digest, "size": size}
def _put_manifest(gitea_registry, name, tag, manifest_blob, jwt, timeout):
url = "{}/v2/{}/manifests/{}".format(
gitea_registry.rstrip("/"), name, tag)
status, body, _ = _request(
"PUT", url, {"Authorization": "Bearer {}".format(jwt),
"Content-Type": OCI_MANIFEST}, manifest_blob, timeout)
if status not in (200, 201):
raise RegistryApiError("manifest PUT {}".format(tag), status, body)
return "sha256:" + hashlib.sha256(manifest_blob).hexdigest()
def _load_artifact(path):
if not os.path.isfile(path):
raise LocalError("artifact not found: {}".format(path))
with open(path, "rb") as handle:
return handle.read()
def smoke(tag, gitea_registry, hub_registry, hub_auth, owner, repo,
timeout, env_root):
"""G-013 read-only live smoke: the four GET-only pre-ship checks.
Never part of the default publish flow; never mutates remote state
(no uploads, no manifest PUTs). The LIVE invocation happens only
at the P3-05 ship step, which gates it by setting PUBLISH_SMOKE=1
in the environment — this script neither reads nor requires that
variable (the SHIP GATE owns the gating). The default unittest
suite exercises this LOGIC against stubs (zero network).
Checks, in order, stopping at the first failure (see the module
docstring for the full G-021 tie-in): [1/4] the gitea /v2/ 401
Bearer challenge (realm recorded); [2/4] the token->JWT exchange at
the recorded realm — the gitea token resolves only from the
sanctioned env files via ``_env_secrets.get_token`` (never the
shell environment), and an empty token is the anonymous-200 trap
refused loudly; [3/4] the gitea API release-by-tag GET — 200
(exists) and 404 (not yet created) are both acceptable pre-ship,
asset names print as a fact, and the hard no-.pyz assertion binds
ONLY the tag v0.2.0 exactly (G-021 forge-side confirmation);
[4/4] the Docker Hub anonymous index resolve of python:3.11-slim —
the linux/amd64 child digest prints (the drift fact P3-05
compares against expectations).
Returns 0 (all PASS), or 1 after one clear FAILED line (token
values are never printed; IMAGE_PENDING is never emitted — nothing
was attempted, so nothing is pending). Local preconditions (a
missing token) raise ``_env_secrets.SecretsError`` BEFORE any
request; main() maps that to exit 2.
"""
gitea_token = _env_secrets.get_token(root=env_root)
metadata_timeout = min(METADATA_TIMEOUT_SECONDS, timeout)
gitea_base = gitea_registry.rstrip("/")
# [1/4] gitea /v2/ auth challenge (anonymous; realm recorded).
try:
status, _, resp_headers = _request(
"GET", "{}/v2/".format(gitea_base), {}, None, metadata_timeout)
except (TransientFailure, RegistryApiError) as error:
print("FAILED [1/4] gitea /v2/ auth challenge: {}".format(error))
return 1
challenge = _header(resp_headers, "WWW-Authenticate")
realm = _bearer_challenge(challenge)
if status != 401 or not realm:
print("FAILED [1/4] gitea /v2/ auth challenge: expected 401 with "
"a Bearer challenge, got HTTP {} (WWW-Authenticate: {!r})"
.format(status, challenge))
return 1
print("PASS [1/4] gitea /v2/ auth challenge: 401 + Bearer challenge "
"(realm {})".format(realm))
# [2/4] token->JWT exchange at the recorded realm (Basic auth).
try:
_gitea_jwt(realm, owner, repo, gitea_token, metadata_timeout)
except (TransientFailure, RegistryApiError) as error:
print("FAILED [2/4] gitea token->JWT exchange at {}: {}".format(
realm, error))
return 1
print("PASS [2/4] gitea token->JWT exchange: 200 + non-empty token "
"(service {}, scope repository:{}/{}:pull,push)".format(
GITEA_AUTH_SERVICE, owner, repo))
# [3/4] release-by-tag GET (gitea API; 200 exists, 404 pre-ship ok).
release_url = "{}/api/v1/repos/{}/{}/releases/tags/{}".format(
gitea_base, owner, repo, urllib.parse.quote(tag, safe=""))
try:
status, body, _ = _request(
"GET", release_url,
{"Authorization": "token {}".format(gitea_token)}, None,
metadata_timeout)
except (TransientFailure, RegistryApiError) as error:
print("FAILED [3/4] release-by-tag {}: {}".format(tag, error))
return 1
if status == 404:
print("PASS [3/4] release-by-tag {}: not yet created (404) — "
"acceptable pre-ship".format(tag))
elif status != 200:
print("FAILED [3/4] release-by-tag {}: expected 200 or 404, got "
"HTTP {}".format(tag, status))
return 1
else:
try:
release = _parse_json(
body, "release-by-tag {}".format(tag), status)
except RegistryApiError as error:
print("FAILED [3/4] release-by-tag {}: {}".format(tag, error))
return 1
asset_names = _asset_names(release)
pyz_assets = [name for name in asset_names
if name.endswith(".pyz")]
if tag == SMOKE_G021_TAG and pyz_assets:
print("FAILED [3/4] release-by-tag {}: G-021 violation — the "
"phase-0 release must carry no .pyz asset, found: {}"
.format(tag, ", ".join(pyz_assets)))
return 1
listing = ", ".join(asset_names) if asset_names else "(none)"
note = (" — G-021 confirmed: no .pyz asset"
if tag == SMOKE_G021_TAG else "")
print("PASS [3/4] release-by-tag {}: exists (assets: {}){}".format(
tag, listing, note))
# [4/4] Docker Hub index resolve (anonymous; amd64 drift print).
try:
hub_bearer = _hub_token(
hub_auth, "repository:{}:pull".format(BASE_REPO))
index = _fetch_manifest(hub_registry, BASE_REPO, BASE_TAG,
hub_bearer)
child_digest = _resolve_amd64_child(index)
if not child_digest:
raise RegistryApiError(
"amd64 child descriptor carries no digest", 500, b"")
except (TransientFailure, RegistryApiError) as error:
print("FAILED [4/4] hub index resolve {}:{}: {}".format(
BASE_REPO, BASE_TAG, error))
return 1
print("PASS [4/4] hub index resolve {}:{} -> linux/amd64 child {}"
.format(BASE_REPO, BASE_TAG, child_digest))
return 0
def publish(tag, artifact_path, hub_registry, hub_auth, gitea_registry,
owner, repo, base_manifest, timeout, env_root):
"""Run the full publish flow; returns the image provenance dict."""
artifact_bytes = _load_artifact(artifact_path)
gitea_token = _env_secrets.get_token(root=env_root)
name = "{}/{}".format(owner, repo)
metadata_timeout = min(METADATA_TIMEOUT_SECONDS, timeout)
hub_bearer = _hub_token(hub_auth, "repository:{}:pull".format(BASE_REPO))
if base_manifest:
child_digest = base_manifest
child = _fetch_manifest(hub_registry, BASE_REPO, base_manifest,
hub_bearer)
index_digest = None
else:
base = _fetch_manifest(hub_registry, BASE_REPO, BASE_TAG, hub_bearer)
index_digest = "sha256:" + hashlib.sha256(
json.dumps(base, sort_keys=True,
separators=(",", ":")).encode("utf-8")).hexdigest()
child_digest = (_resolve_amd64_child(base)
if "manifests" in base else BASE_TAG)
child = _fetch_manifest(hub_registry, BASE_REPO, child_digest,
hub_bearer)
child_config, base_layer_descriptors = _child_parts(child)
base_config_bytes = _fetch_blob(
hub_registry, BASE_REPO, child_config["digest"], hub_bearer,
timeout)
base_config = _parse_json(base_config_bytes, "base config blob", 200)
if not isinstance(base_config, dict):
raise RegistryApiError("base config blob is not a JSON object",
500, base_config_bytes)
jwt = _gitea_jwt("{}/v2/token".format(gitea_registry.rstrip("/")),
owner, repo, gitea_token, metadata_timeout)
pushed = []
pushed.append(_push_blob_if_missing(
gitea_registry, name, child_config["digest"], base_config_bytes,
jwt, timeout))
for descriptor in base_layer_descriptors:
if _blob_exists(gitea_registry, name, descriptor["digest"], jwt,
timeout):
continue
blob = _fetch_blob(hub_registry, BASE_REPO, descriptor["digest"],
hub_bearer, timeout)
_upload_blob(gitea_registry, name, descriptor["digest"], blob, jwt,
timeout)
pushed.append(True)
layer_blob, layer_digest, diff_id = build_broker_layer(artifact_bytes)
config_blob = build_config_blob(base_config, diff_id, tag)
config_digest = "sha256:" + hashlib.sha256(config_blob).hexdigest()
_push_blob_if_missing(gitea_registry, name, layer_digest, layer_blob,
jwt, timeout)
_push_blob_if_missing(gitea_registry, name, config_digest, config_blob,
jwt, timeout)
manifest_blob = build_manifest_blob(
base_layer_descriptors,
_descriptor(layer_digest, len(layer_blob), OCI_LAYER),
_descriptor(config_digest, len(config_blob), OCI_CONFIG))
manifest_digest = _put_manifest(
gitea_registry, name, tag, manifest_blob, jwt, timeout)
_put_manifest(gitea_registry, name, "latest", manifest_blob, jwt,
timeout)
return {
"base_index_digest": index_digest,
"base_child_digest": child_digest if child_digest != BASE_TAG
else "tag-resolved",
"base_config_digest": child_config["digest"],
"base_layers": [descriptor["digest"]
for descriptor in base_layer_descriptors],
"broker_layer_digest": layer_digest,
"broker_layer_diff_id": diff_id,
"config_digest": config_digest,
"manifest_digest": manifest_digest,
"blobs_uploaded": sum(1 for flag in pushed if flag),
}
def main(argv=None):
parser = argparse.ArgumentParser(
description="Daemonless OCI image publish (REQ-23, D-019)."
)
parser.add_argument("tag", help="image tag, e.g. v0.2.3")
parser.add_argument("--artifact",
help="path to the built .pyz artifact "
"(default: dist/coreci-broker-v<tag>.pyz)")
parser.add_argument("--hub-registry",
default="https://registry-1.docker.io",
help="public base-image registry")
parser.add_argument("--hub-auth", default="https://auth.docker.io/token",
help="public registry token service")
parser.add_argument("--gitea-registry", default="https://git.coreci.dev",
help="gitea registry base URL")
parser.add_argument("--owner", help="gitea owner (default: release.owner)")
parser.add_argument("--repo", help="gitea repo (default: release.repo)")
parser.add_argument("--base-manifest",
help="base child-manifest digest for exact "
"re-publishes (skips tag resolution)")
parser.add_argument("--smoke", action="store_true",
help="G-013 read-only live smoke: four GET-only "
"pre-ship checks, NO publish (see the "
"module docstring)")
parser.add_argument("--timeout", type=float, default=BLOB_TIMEOUT_SECONDS,
help="per-request timeout for blob operations "
"(seconds; metadata uses the smaller of this "
"and 30)")
parser.add_argument("--env-root",
help="root dir for the sanctioned .env* files "
"(default: repo root)")
args = parser.parse_args(argv)
defaults = load_release_defaults()
owner = args.owner or defaults.get("owner")
repo = args.repo or defaults.get("repo")
if not owner or not repo:
print("ERROR: missing gitea target (owner/repo); pass the flags or"
" extend the .ciagent/config.json release block",
file=sys.stderr)
return 2
blob_timeout = max(1.0, float(args.timeout))
if args.smoke:
# G-013 read-only live smoke: dispatched BEFORE any
# publish-side work and returning directly — the default
# publish flow below can never trigger it.
try:
return smoke(
tag=args.tag, gitea_registry=args.gitea_registry,
hub_registry=args.hub_registry, hub_auth=args.hub_auth,
owner=owner, repo=repo, timeout=blob_timeout,
env_root=args.env_root)
except _env_secrets.SecretsError as error:
print("ERROR: {}".format(error), file=sys.stderr)
return 2
artifact_path = args.artifact or os.path.join(
"dist", "coreci-broker-{}.pyz".format(args.tag))
# Reuse the smaller metadata timeout for the registry auth exchange so
# the timeout-abort path stays bounded for every request class.
try:
provenance = publish(
tag=args.tag, artifact_path=artifact_path,
hub_registry=args.hub_registry, hub_auth=args.hub_auth,
gitea_registry=args.gitea_registry, owner=owner, repo=repo,
base_manifest=args.base_manifest, timeout=blob_timeout,
env_root=args.env_root)
except _env_secrets.SecretsError as error:
print("ERROR: {}".format(error), file=sys.stderr)
return 2
except LocalError as error:
print("ERROR: {}".format(error), file=sys.stderr)
return 2
except (TransientFailure, RegistryApiError) as error:
print("ERROR: {}".format(error), file=sys.stderr)
registry_host = urllib.parse.urlsplit(
args.gitea_registry).netloc or args.gitea_registry
print("IMAGE_PENDING {}/{}:{}".format(registry_host, owner, repo,
args.tag))
return 1
print("image: {}/{}:{} (+ latest)".format(
args.gitea_registry.rstrip("/"), owner, repo, args.tag))
print("base index: {}".format(
provenance["base_index_digest"] or "direct (--base-manifest)"))
print("base amd64 manifest: {}".format(provenance["base_child_digest"]))
print("base config: {}".format(provenance["base_config_digest"]))
print("base layers: {}".format(", ".join(provenance["base_layers"])))
print("broker layer: {} (diff_id {})".format(
provenance["broker_layer_digest"],
provenance["broker_layer_diff_id"]))
print("config: {}".format(provenance["config_digest"]))
print("manifest: {}".format(provenance["manifest_digest"]))
print("blobs uploaded this run: {}".format(provenance["blobs_uploaded"]))
return 0
if __name__ == "__main__":
sys.exit(main())
-380
View File
@@ -1,380 +0,0 @@
#!/usr/bin/env python3
"""Attach the built CLI artifact to a gitea release (stdlib only).
Usage:
python3 scripts/release.py <tag> <artifact>
[--base-url URL] [--owner O] [--repo R] [--env-root DIR]
Flow (REQ-22, RESEARCH §2): resolve the forge token from the sanctioned
env files via ``scripts/_env_secrets.py`` — never the process environment,
asserted non-empty BEFORE the first API call (v1.8 lesson: an empty token
makes gitea answer 404, masking the auth failure) -> compute the artifact
sha256 from the FILE itself (G-017: there is no sha argument, ever) ->
build the release notes (Summary / Requirements / Verification / CLI
install + usage with the digest inline / Docker usage) -> create-or-get
the release by tag -> idempotent attach (same name AND size -> skip) ->
multipart attach (field ``attachment``, ``?name=``, ``Authorization:
token <T>``) under bounded retry (3 attempts, 2s/4s backoff on connection
errors and 5xx).
Exit codes: 0 only on a confirmed or idempotent attach; 1 after a remote
failure — bounded retries exhausted OR a permanent 4xx — both print the
machine-readable ``RELEASE_PENDING <tag> <artifact>`` marker on stdout for
the ship harness's release_pending escalation (tag+merge is never
blocked); 2 for local precondition errors (missing artifact, token, or
config — nothing was attempted remotely, so no marker). Gitea JSON error
bodies are included in failure diagnostics. Token values are never
printed.
"""
import argparse
import hashlib
import http.client
import json
import os
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
_SCRIPTS_DIR = os.path.dirname(os.path.abspath(__file__))
if _SCRIPTS_DIR not in sys.path:
sys.path.insert(0, _SCRIPTS_DIR)
import _env_secrets # sibling import (scripts/ dir on sys.path)
TIMEOUT_SECONDS = 30
MAX_ATTEMPTS = 3
BACKOFF_SECONDS = (2, 4)
ATTACHMENT_FIELD = "attachment"
# Fixed, distinctive boundary: a hand-rolled multipart body needs our own
# delimiter; collision with artifact bytes is negligible at this length.
MULTIPART_BOUNDARY = "coreci-broker-release-boundary-9a7f3c2e6b41"
class ReleaseError(Exception):
"""Release-attacher failure base."""
class TransientFailure(ReleaseError):
"""Connection error or 5xx persisting past the bounded retries."""
class GiteaApiError(ReleaseError):
"""Permanent API answer (4xx or malformed) carrying gitea's detail."""
def __init__(self, context, status, body):
self.status = status
self.body = body
super().__init__("{} -> HTTP {}: {}".format(
context, status, _error_detail(body)))
def repo_root():
return os.path.dirname(_SCRIPTS_DIR)
def load_release_defaults(root=None):
"""Read the release block from .ciagent/config.json ({} if absent)."""
path = os.path.join(root or repo_root(), ".ciagent", "config.json")
try:
with open(path, "r", encoding="utf-8") as handle:
config = json.load(handle)
except (OSError, ValueError):
return {}
block = config.get("release")
return block if isinstance(block, dict) else {}
def _error_detail(body):
"""Summarize a gitea JSON error body ({"message": ...} shape)."""
try:
payload = json.loads(body.decode("utf-8", "replace"))
except ValueError:
text = body.decode("utf-8", "replace").strip()
return text[:200] if text else "<empty body>"
if isinstance(payload, dict) and payload.get("message"):
return str(payload["message"])
return json.dumps(payload)[:200] if not isinstance(payload, str) else payload[:200]
def _api_base(base_url, owner, repo):
return "{}/api/v1/repos/{}/{}".format(base_url.rstrip("/"), owner, repo)
def _image_reference(base_url, owner, repo):
host = urllib.parse.urlsplit(base_url).netloc or base_url
return "{}/{}/{}".format(host.rstrip("/"), owner, repo)
def build_notes(tag, artifact_name, sha256, image_reference):
"""Release-notes body (RESEARCH §2.3 model; REQ-22 CLI + docker docs)."""
lines = [
"## Summary",
"",
"coreci-broker {tag}: the broker packaged as a self-contained CLI"
" zipapp artifact. {artifact} is built by scripts/build_cli.py with"
" a deterministic normalization pass - the same sources rebuild"
" byte-identical on the same environment, so the digest below is"
" meaningful across rebuilds.".format(tag=tag, artifact=artifact_name),
"",
"## Requirements",
"",
"REQ-22: from the first CLI-building release onward, every gitea"
" release attaches the built coreci-broker artifact alongside the"
" existing source archive. The upload resolves the forge token only"
" via the sanctioned .env* files (never the process environment),"
" asserted non-empty before the first API call.",
"",
"## Verification",
"",
"The artifact sha256 in the CLI section below is computed from the"
" attached file itself (G-017 - never an argument, never"
" hand-pasted). Re-run the cumulative suite on the tagged commit:",
"",
" python3 -m unittest discover -s tests",
"",
"## CLI install + usage",
"",
"1. Download {artifact} from this release's assets.".format(
artifact=artifact_name),
"2. Install: chmod +x {artifact} then ./{artifact}, or run it"
" directly with python3 {artifact} (Python 3.11+).".format(
artifact=artifact_name),
"3. Check the build: ./{artifact} --version prints {tag}.".format(
artifact=artifact_name, tag=tag),
"4. The env contract (every BROKER_* variable) is documented in the"
" README; state is anchored by BROKER_ENV_PATH - the .env file's"
" directory is the state dir and relative paths stay CWD-relative.",
"5. First boot prints BROKER_ADMIN_PASS=... exactly once and creates"
" the .env, broker.db, and broker.audit.log state files with mode"
" 0600; the printed admin pass is not shown again - store it.",
"",
"Artifact sha256 (computed from the attached file):",
"",
" sha256:{sha} {artifact}".format(sha=sha256, artifact=artifact_name),
"",
"## Docker usage",
"",
"docker pull {image}:{tag}".format(image=image_reference, tag=tag),
"",
"docker run -p 8765:8765 -v broker-state:/state {image}:{tag}".format(
image=image_reference, tag=tag),
"",
"State lives on the /state volume (WORKDIR /state; the first boot"
" generates the .env, broker.db, and broker.audit.log there) -"
" mount a persistent volume or the admin pass and any registered"
" apps are lost with the container.",
"",
]
return "\n".join(lines)
def _request_once(method, url, token, data=None, headers=None):
"""One HTTP exchange; returns (status, body). HTTPError becomes a
completed answer (status + gitea error body); connection-level errors
propagate for the caller's retry policy."""
request = urllib.request.Request(url, data=data, method=method)
request.add_header("Authorization", "token {}".format(token))
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=TIMEOUT_SECONDS) as response:
return response.status, response.read()
except urllib.error.HTTPError as error:
return error.code, error.read()
def _request(method, url, token, data=None, headers=None,
max_attempts=None, backoff=None):
"""Bounded-retry HTTP: retries connection errors and 5xx answers.
Returns (status, body) for any answer below 500. A 4xx returns
immediately (permanent — surfaced with the gitea error body). Raises
TransientFailure once the retry budget is exhausted.
"""
attempts = MAX_ATTEMPTS if max_attempts is None else int(max_attempts)
delays = BACKOFF_SECONDS if backoff is None else tuple(backoff)
failure = "unknown failure"
for attempt in range(1, attempts + 1):
try:
status, body = _request_once(method, url, token, data, headers)
except (OSError, http.client.HTTPException) as error:
status = None
failure = "connection error: {}: {}".format(
type(error).__name__, error)
else:
if status < 500:
return status, body
failure = "HTTP {}: {}".format(status, _error_detail(body))
if attempt < attempts and delays:
time.sleep(delays[min(attempt - 1, len(delays) - 1)])
raise TransientFailure(
"{} {} failed after {} attempt(s): {}".format(
method, url, attempts, failure))
def _release_id_from(payload_body, context, status):
try:
return int(json.loads(payload_body)["id"])
except (ValueError, KeyError, TypeError):
raise GiteaApiError(context + " returned malformed JSON", status, payload_body)
def _get_or_create_release(base, tag, token, notes):
by_tag = base + "/releases/tags/" + urllib.parse.quote(tag, safe="")
status, body = _request("GET", by_tag, token)
if status == 200:
return _release_id_from(body, "GET release by tag", status)
if status != 404:
# A 401/403 here is a genuine auth failure — the non-empty token
# assert already ran, so this 404-vs-auth ambiguity is resolved.
raise GiteaApiError("GET release by tag failed", status, body)
payload = json.dumps({
"tag_name": tag,
"name": "{}: coreci-broker release".format(tag),
"body": notes,
"draft": False,
"prerelease": False,
}).encode("utf-8")
status, body = _request(
"POST", base + "/releases", token, data=payload,
headers={"Content-Type": "application/json"})
if status not in (200, 201):
raise GiteaApiError("create release failed", status, body)
return _release_id_from(body, "create release", status)
def _assets_url(base, release_id):
return "{}/releases/{}/assets".format(base, release_id)
def _list_assets(base, release_id, token):
status, body = _request("GET", _assets_url(base, release_id), token)
if status != 200:
raise GiteaApiError("list release assets failed", status, body)
try:
assets = json.loads(body)
except ValueError:
raise GiteaApiError(
"list release assets returned malformed JSON", status, body)
if not isinstance(assets, list):
raise GiteaApiError(
"list release assets returned non-list JSON", status, body)
return assets
def _asset_already_attached(assets, name, size):
"""Idempotency: same name AND same size means the upload succeeded
on an earlier (possibly retried) run."""
for asset in assets:
if not isinstance(asset, dict) or asset.get("name") != name:
continue
try:
if int(asset.get("size")) == size:
return True
except (TypeError, ValueError):
continue
return False
def _multipart_body(field, filename, content):
head = (
"--{boundary}\r\n"
"Content-Disposition: form-data; name=\"{field}\";"
" filename=\"{filename}\"\r\n"
"Content-Type: application/octet-stream\r\n"
"\r\n"
).format(boundary=MULTIPART_BOUNDARY, field=field,
filename=filename).encode("utf-8")
tail = "\r\n--{}--\r\n".format(MULTIPART_BOUNDARY).encode("utf-8")
return head + content + tail
def _attach_asset(base, release_id, token, name, content):
url = "{}?{}".format(
_assets_url(base, release_id), urllib.parse.urlencode({"name": name}))
status, body = _request(
"POST", url, token, data=_multipart_body(ATTACHMENT_FIELD, name, content),
headers={
"Content-Type":
"multipart/form-data; boundary={}".format(MULTIPART_BOUNDARY),
})
if status not in (200, 201):
raise GiteaApiError("attach asset failed", status, body)
try:
return json.loads(body)
except ValueError:
return {}
def main(argv=None):
parser = argparse.ArgumentParser(
description="Attach the built CLI artifact to a gitea release (REQ-22)."
)
parser.add_argument("tag", help="release tag, e.g. v0.2.1")
parser.add_argument("artifact", help="path to the built .pyz artifact")
parser.add_argument("--base-url",
help="gitea base URL (default: .ciagent/config.json release.base_url)")
parser.add_argument("--owner",
help="gitea owner (default: release.owner)")
parser.add_argument("--repo",
help="gitea repo (default: release.repo)")
parser.add_argument("--env-root",
help="root dir for the sanctioned .env* files (default: repo root)")
args = parser.parse_args(argv)
defaults = load_release_defaults()
base_url = args.base_url or defaults.get("base_url")
owner = args.owner or defaults.get("owner")
repo = args.repo or defaults.get("repo")
missing = [flag for flag, value in (
("--base-url", base_url), ("--owner", owner), ("--repo", repo))
if not value]
if missing:
print("ERROR: missing gitea target {}; pass the flag(s) or extend"
" the .ciagent/config.json release block".format(
", ".join(missing)), file=sys.stderr)
return 2
if not os.path.isfile(args.artifact):
print("ERROR: artifact not found: {}".format(args.artifact), file=sys.stderr)
return 2
# Token FIRST, from the sanctioned env files only, non-empty enforced —
# strictly before any HTTP call (v1.8: an empty token makes gitea answer
# 404 and masks the auth failure).
try:
token = _env_secrets.get_token(root=args.env_root)
except _env_secrets.SecretsError as error:
print("ERROR: {}".format(error), file=sys.stderr)
return 2
with open(args.artifact, "rb") as handle:
content = handle.read()
# G-017: the digest is computed from the artifact file itself — there is
# no sha argument, ever.
sha256 = hashlib.sha256(content).hexdigest()
name = os.path.basename(args.artifact)
size = len(content)
notes = build_notes(args.tag, name, sha256,
_image_reference(base_url, owner, repo))
base = _api_base(base_url, owner, repo)
try:
release_id = _get_or_create_release(base, args.tag, token, notes)
if _asset_already_attached(_list_assets(base, release_id, token), name, size):
print("attach skipped (idempotent): {} already on {}/{} release"
" {} (sha256 {})".format(name, owner, repo, args.tag, sha256))
return 0
attachment = _attach_asset(base, release_id, token, name, content)
asset_id = attachment.get("id", "?") if isinstance(attachment, dict) else "?"
print("attached: {} -> {}/{} release {} (asset id {}, sha256 {})".format(
name, owner, repo, args.tag, asset_id, sha256))
return 0
except (TransientFailure, GiteaApiError) as error:
print("ERROR: {}".format(error), file=sys.stderr)
print("RELEASE_PENDING {} {}".format(args.tag, args.artifact))
return 1
if __name__ == "__main__":
sys.exit(main())
-345
View File
@@ -1,345 +0,0 @@
import ast
import base64
import contextlib
import datetime
import importlib.util
import io
import json
import os
import sys
import tempfile
import time
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
BROKER_SOURCE_PATH = os.path.join(REPO_ROOT, "coreci_broker.py")
CLIENT_SOURCE_PATH = os.path.join(REPO_ROOT, "coreci_broker_client.py")
CLIENT_STATIC_PATH = "/static/coreci_broker_client.py"
LOGIN_PATH = "/api/broker/auth/login"
LOGOUT_PATH = "/api/broker/auth/logout"
REGISTER_PATH = "/api/broker/agents/register"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
AUDIT_PATH = "/api/broker/audit"
SECRETS_PREFIX = "/api/broker/secrets/"
PROXMOX_PATH = "coreci/proxmox"
GITEA_PATH = "coreci/gitea"
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
GITEA_SECRET = {"url": "https://git.coreci.dev", "token": "git-tok"}
AC_PIN_IDS = {
"AC1": "test_acceptance_p2.AcceptanceP2Tests.test_ac1_first_boot_printed_pass_login_via_ui_login_page",
"AC2": "test_acceptance_p2.AcceptanceP2Tests.test_ac2_app_registration_lifecycle_via_session_cookie",
"AC3": "test_acceptance_p2.AcceptanceP2Tests.test_ac3_user_allowlist_lifecycle_via_session_cookie",
"AC4": "test_acceptance_p3.AcceptanceP3Tests.test_ac4_derivation_identical_both_sides_stored_as_slow_hash",
"AC5": "test_acceptance_p4.AcceptanceP4Tests.test_ac5_valid_token_reads_secret_tampered_token_is_401",
"AC6": "test_acceptance_p4.AcceptanceP4Tests.test_ac6_cross_path_read_is_403_with_denied_audit",
"AC7": "test_acceptance_p3.AcceptanceP3Tests.test_ac7_revoked_or_unknown_uuid_denied_existing_token_honored",
"AC8": "test_acceptance_p3.AcceptanceP3Tests.test_ac8_unknown_or_revoked_user_denied",
"AC9": "test_acceptance_p2.AcceptanceP2Tests.test_ac9_login_and_logout_audit_lines_with_null_identity",
"AC10": "test_acceptance_p3.AcceptanceP3Tests.test_ac10_expiry_401_reregister_refreshes_expiry_and_audits",
"AC11": "test_acceptance_full.AcceptanceFullTests.test_ac11_fresh_boot_state_files_0600_umask_independent",
"AC12": "test_acceptance_p3.AcceptanceP3Tests.test_ac12_state_roundtrip_with_identity_enforcement",
"AC13": "test_acceptance_full.AcceptanceFullTests.test_ac13_client_module_fetch_byte_identical_importable_standalone",
"AC14": "test_acceptance_full.AcceptanceFullTests.test_ac14_coreci_broker_line_count_under_1400",
}
def _basic_header(password):
raw = "%s:%s" % (ADMIN_USER, password)
return {"Authorization": "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")}
class AcceptanceFullTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.creds_path = os.path.join(self.tmp, "creds.json")
with open(self.creds_path, "w", encoding="utf-8") as handle:
handle.write(json.dumps({PROXMOX_PATH: PROXMOX_SECRET, GITEA_PATH: GITEA_SECRET}))
def _env(self, root=None, extra=None):
root = self.tmp if root is None else root
env = {
"BROKER_ENV_PATH": os.path.join(root, ".env"),
"BROKER_DB_PATH": os.path.join(root, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(root, "broker.audit.log"),
"BROKER_CREDS_SOURCE": "file",
"BROKER_CREDS_FILE": self.creds_path,
"BROKER_UI_ROOT": os.path.join(REPO_ROOT, "coreci_broker_ui"),
"BROKER_STATIC_ROOT": REPO_ROOT,
}
env.update(extra or {})
return env
def _boot(self, env):
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer):
broker = coreci_broker.boot(dict(env))
self.addCleanup(broker.audit.close)
self.addCleanup(broker.store.close)
return broker, buffer.getvalue()
def _serve(self, broker):
server = coreci_broker.serve("127.0.0.1", 0, broker)
self.addCleanup(server.server_close)
self.addCleanup(server.shutdown)
return "http://127.0.0.1:%d" % server.server_address[1]
def _printed_pass(self, output):
printed = [
line.split("=", 1)[1]
for line in output.splitlines()
if line.startswith("BROKER_ADMIN_PASS=")
]
self.assertEqual(1, len(printed))
return printed[0]
def _request(self, base, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
return response.status, response.read(), response.headers
except urllib.error.HTTPError as error:
return error.code, error.read(), error.headers
def _request_json(self, base, method, path, payload=None, headers=None):
status, body, response_headers = self._request(base, method, path, payload, headers)
return status, json.loads(body.decode("utf-8")), response_headers
def _assert_null_identity(self, entry):
self.assertIsNone(entry["user_id"])
self.assertIsNone(entry["app_registration_uuid"])
self.assertIsNone(entry["agent_id"])
def _assert_tool_identity(self, entry, app_uuid):
self.assertEqual("ceo", entry["user_id"])
self.assertEqual(app_uuid, entry["app_registration_uuid"])
self.assertEqual("proxmox", entry["agent_id"])
def _assert_state_files_0600(self, env):
for key in ("BROKER_DB_PATH", "BROKER_AUDIT_LOG_PATH", "BROKER_ENV_PATH"):
path = env[key]
self.assertTrue(os.path.isfile(path), path)
self.assertEqual(0o600, os.stat(path).st_mode & 0o777, path)
def test_ac11_fresh_boot_state_files_0600_umask_independent(self):
env = self._env()
self._boot(env)
self._assert_state_files_0600(env)
other = tempfile.TemporaryDirectory()
self.addCleanup(other.cleanup)
other_env = self._env(root=other.name)
previous_umask = os.umask(0o777)
try:
probe = os.path.join(other.name, "umask-probe")
os.close(os.open(probe, os.O_CREAT | os.O_WRONLY, 0o666))
self.assertEqual(0o000, os.stat(probe).st_mode & 0o777)
self._boot(other_env)
finally:
os.umask(previous_umask)
self._assert_state_files_0600(other_env)
def test_ac13_client_module_fetch_byte_identical_importable_standalone(self):
broker, _ = self._boot(self._env())
base = self._serve(broker)
with open(CLIENT_SOURCE_PATH, "rb") as handle:
expected = handle.read()
with urllib.request.urlopen(base + CLIENT_STATIC_PATH, timeout=10) as response:
self.assertEqual(200, response.status)
fetched = response.read()
self.assertEqual(expected, fetched)
imports = set()
for node in ast.walk(ast.parse(fetched.decode("utf-8"))):
if isinstance(node, ast.Import):
imports.update(alias.name for alias in node.names)
elif isinstance(node, ast.ImportFrom):
imports.add(node.module or "")
self.assertTrue(imports)
for name in imports:
self.assertIn(name.split(".")[0], sys.stdlib_module_names, name)
copy_path = os.path.join(self.tmp, "fetched_coreci_broker_client.py")
with open(copy_path, "wb") as handle:
handle.write(fetched)
spec = importlib.util.spec_from_file_location("full_fetched_client", copy_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
self.assertTrue(callable(module.BrokerClient))
self.assertTrue(callable(module.fetch_client_module))
for method in ("ensure_registered", "get_state", "put_state", "read_secret"):
self.assertTrue(callable(getattr(module.BrokerClient, method)), method)
def test_ac14_coreci_broker_line_count_under_1400(self):
with open(BROKER_SOURCE_PATH, "r", encoding="utf-8") as handle:
line_count = sum(1 for _ in handle)
self.assertLess(line_count, 1400)
def test_milestone_story_full_happy_path_consolidates_ac1_ac10_ac12(self):
env = self._env(extra={"BROKER_TOKEN_TTL": "1"})
first, output = self._boot(env)
printed_pass = self._printed_pass(output)
self.assertGreaterEqual(len(printed_pass), 32)
self.assertEqual(1, output.count(printed_pass))
pepper = None
with open(env["BROKER_ENV_PATH"], "r", encoding="utf-8") as handle:
for line in handle:
if line.startswith("BROKER_INSTANCE_PEPPER="):
pepper = line.split("=", 1)[1].strip()
self.assertIsNotNone(pepper)
self.assertNotIn(pepper, output)
second, silent = self._boot(env)
self.assertEqual("", silent)
second.audit.close()
second.store.close()
base = self._serve(first)
status, body, headers = self._request(base, "GET", "/ui/login")
self.assertEqual(200, status)
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
self.assertIn('name="password"', body.decode("utf-8"))
status, body, response_headers = self._request_json(
base, "POST", LOGIN_PATH, {"username": ADMIN_USER, "password": printed_pass}
)
self.assertEqual(200, status)
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
raw_cookie = response_headers.get("Set-Cookie")
self.assertTrue(raw_cookie)
self.assertTrue(raw_cookie.startswith("broker_session="))
self.assertIn("HttpOnly", raw_cookie)
self.assertIn("SameSite=Lax", raw_cookie)
cookie = {"Cookie": raw_cookie.split(";", 1)[0]}
app_uuid = str(uuidlib.uuid4())
status, body, _ = self._request_json(
base, "POST", APPS_PATH,
{"uuid": app_uuid, "name": "proxmox-tool"}, headers=cookie,
)
self.assertEqual(201, status)
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
status, body, _ = self._request_json(
base, "POST", USERS_PATH,
{"user_id": "ceo", "email": "ceo@coreci.dev"}, headers=cookie,
)
self.assertEqual(201, status)
self.assertEqual({"user_id": "ceo", "status": "active"}, body)
with urllib.request.urlopen(base + CLIENT_STATIC_PATH, timeout=10) as response:
self.assertEqual(200, response.status)
client_source = response.read()
with open(CLIENT_SOURCE_PATH, "rb") as handle:
self.assertEqual(handle.read(), client_source)
copy_path = os.path.join(self.tmp, "story_coreci_broker_client.py")
with open(copy_path, "wb") as handle:
handle.write(client_source)
spec = importlib.util.spec_from_file_location("full_story_client", copy_path)
client_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(client_module)
client = client_module.BrokerClient(base, "proxmox", app_uuid, "ceo", ["coreci"])
state = {}
client.ensure_registered(state)
self.assertEqual(44, len(state["current_token"]))
self.assertTrue(state["current_token"].isascii())
self.assertIn("pepper", state)
self.assertEqual("proxmox", state["agent_id"])
self.assertEqual(app_uuid, state["app_registration_uuid"])
self.assertEqual("ceo", state["user_id"])
self.assertTrue(client.put_state(state))
snapshot = dict(state)
self.assertEqual(snapshot, client.get_state(state))
document = client.read_secret(PROXMOX_PATH, state)
self.assertEqual({"path", "data", "version", "updated_at"}, set(document))
self.assertEqual(PROXMOX_PATH, document["path"])
self.assertEqual(PROXMOX_SECRET, document["data"])
with self.assertRaises(client_module.BrokerError) as caught:
client.read_secret(GITEA_PATH, state)
self.assertEqual(403, caught.exception.status)
expired_at = state["token_expires_at"]
time.sleep(1.2)
document = client.read_secret(PROXMOX_PATH, state)
self.assertEqual(PROXMOX_SECRET, document["data"])
self.assertGreater(
datetime.datetime.fromisoformat(state["token_expires_at"]),
datetime.datetime.fromisoformat(expired_at),
)
status, body, _ = self._request_json(base, "POST", LOGOUT_PATH, headers=cookie)
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
status, body, _ = self._request_json(
base, "GET", AUDIT_PATH + "?limit=1000", headers=_basic_header(printed_pass)
)
self.assertEqual(200, status)
self.assertIsNone(body["next_since"])
entries = body["entries"]
def select(action, result=None):
return [
entry for entry in entries
if entry["action"] == action and (result is None or entry["result"] == result)
]
logins = select("login", "ok")
self.assertEqual(1, len(logins))
self.assertEqual(LOGIN_PATH, logins[0]["path"])
self._assert_null_identity(logins[0])
add_apps = select("add_app", "ok")
self.assertEqual(1, len(add_apps))
self.assertEqual(APPS_PATH + "/" + app_uuid, add_apps[0]["path"])
self._assert_null_identity(add_apps[0])
add_users = select("add_user", "ok")
self.assertEqual(1, len(add_users))
self.assertEqual(USERS_PATH + "/ceo", add_users[0]["path"])
self._assert_null_identity(add_users[0])
registers = select("register", "ok")
self.assertGreaterEqual(len(registers), 2)
for entry in registers:
self.assertEqual(REGISTER_PATH, entry["path"])
self._assert_tool_identity(entry, app_uuid)
ok_reads = select("read", "ok")
self.assertGreaterEqual(len(ok_reads), 2)
for entry in ok_reads:
self.assertEqual(PROXMOX_PATH, entry["path"])
self._assert_tool_identity(entry, app_uuid)
denied_reads = select("read", "denied")
self.assertEqual(1, len(denied_reads))
self.assertEqual(GITEA_PATH, denied_reads[0]["path"])
self._assert_tool_identity(denied_reads[0], app_uuid)
auth_denied = [
entry for entry in select("denied", "denied")
if entry["path"] == SECRETS_PREFIX + PROXMOX_PATH
]
self.assertGreaterEqual(len(auth_denied), 1)
self.assertEqual(app_uuid, auth_denied[0]["app_registration_uuid"])
self.assertIsNone(auth_denied[0]["user_id"])
self.assertIsNone(auth_denied[0]["agent_id"])
logouts = select("logout", "ok")
self.assertEqual(1, len(logouts))
self.assertEqual(LOGOUT_PATH, logouts[0]["path"])
self._assert_null_identity(logouts[0])
def test_coverage_tally_pins_ac1_through_ac14_across_suite(self):
self.assertEqual(14, len(AC_PIN_IDS))
loader = unittest.TestLoader()
suite = loader.discover(os.path.dirname(os.path.abspath(__file__)))
stack, ids = [suite], set()
while stack:
item = stack.pop()
if isinstance(item, unittest.TestSuite):
stack.extend(item)
else:
ids.add(item.id())
for ac in sorted(AC_PIN_IDS):
with self.subTest(ac=ac):
self.assertIn(AC_PIN_IDS[ac], ids)
if __name__ == "__main__":
unittest.main()
-548
View File
@@ -1,548 +0,0 @@
"""v0.2 acceptance matrix (PLAN P3-01; REQ-21/REQ-22/REQ-23 acceptance).
Consolidates the milestone's acceptance as thin, high-value assertions that
the deliverable holds end-to-end. The deep per-feature logic stays in its own
files — test_build_cli, test_cli_packaging, test_publish_image,
test_release_upload, test_dockerfile_consistency — which run green in the
same cumulative suite (``python3 -m unittest discover -s tests``; suite
greenness is the runner's job, asserted by nobody in here). This file does
NOT re-run their cases; the stub roundtrips are smoke-asserted instead via
module presence + runnable TestSuite sizes (defaultTestLoader).
The matrix (binding order):
1. boot parity through the artifact — the built ``.pyz`` is booted as a real
subprocess child (``[sys.executable, artifact]``, never a bare
"python3") with an ephemeral bind + temp state, and v0.1 acceptance
parity must hold through the packaged path: admin pass printed exactly
once, ``broker.db`` + ``broker.audit.log`` 0600 in the state dir,
``/ui/login`` + ``/ui/dashboard`` 200, ``/static/coreci_broker_client.py``
byte-exact vs the repo file, one admin-login + agent-register +
state-put/get + secret-read flow (v0.1 request patterns), second boot
silent (asserted only after terminate + full drain — G-010);
2. reproducibility — two consecutive same-environment builds produce
byte-identical artifacts (identical sha256, G-009);
3. self-containment — the archive contains ONLY the staging allowlist
(broker + client + UI pages + shim + ``_cli_version.py``): no ``.env``,
no ``dist/``, no state, no tests;
4. stub publish roundtrip + stub release upload green — their test modules
import cleanly and expose runnable TestSuites of the expected size;
5. broker zero-delta — ``coreci_broker.py`` unchanged at its v0.1.5 count
of 1390 lines (AC14 under the D-016..D-018 cap of 1400) and carrying no
v0.2 packaging references (``_cli_version`` / the shim / the shim's
materialized-asset dir — frozen means untouched); plus the AC14
no-touch clause machine-checked against the ``v0.1.5`` tag when the
working tree carries it (skipped otherwise);
6. Dockerfile/.dockerignore consistency — module present with its full
TestCase set (it runs in the same suite).
The build happens ONCE per class (setUpClass) and every artifact/boot is
reused across methods where safe; the boot tests dominate and the whole
file is budgeted to stay well under 30s. Every child is terminated +
drained exactly once (idempotent ``stop`` via ``addCleanup``); every temp
dir is ``addCleanup``-registered. Builds use the ``scripts/build_cli.py``
API (never a shell path) into ``TemporaryDirectory`` dist dirs — no
committed ``dist/``, no state, no live-forge contact.
"""
import hashlib
import importlib.util
import json
import os
import select
import subprocess
import sys
import tempfile
import time
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
import zipfile
REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
TESTS_DIR = os.path.dirname(os.path.abspath(__file__))
BUILD_CLI_PATH = os.path.join(REPO, "scripts", "build_cli.py")
BROKER_SOURCE_PATH = os.path.join(REPO, "coreci_broker.py")
REPO_CLIENT_PATH = os.path.join(REPO, "coreci_broker_client.py")
VERSION = "v0.2.3" # the milestone release this acceptance gates (P3-05 ship)
ADMIN_USER = "admin"
ADMIN_PASS_PREFIX = "BROKER_ADMIN_PASS="
LISTENING_PREFIX = "coreci-broker listening on http://"
STATE_FILE_NAMES = (".env", "broker.db", "broker.audit.log")
LOGIN_PATH = "/api/broker/auth/login"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
REGISTER_PATH = "/api/broker/agents/register"
AGENT_STATE_PATH = "/api/broker/agents/proxmox/state"
SECRETS_PATH = "/api/broker/secrets/coreci/proxmox"
CLIENT_STATIC_PATH = "/static/coreci_broker_client.py"
UI_LOGIN_PATH = "/ui/login"
UI_DASHBOARD_PATH = "/ui/dashboard"
PROXMOX_PATH = "coreci/proxmox"
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
BOOT_DEADLINE_SECONDS = 30.0
POLL_SECONDS = 0.1
DRAIN_TIMEOUT_SECONDS = 20.0
HTTP_TIMEOUT_SECONDS = 10
GIT_TIMEOUT_SECONDS = 30
# Frozen-broker sources (PLAN: no task edits these in v0.2) — used only by
# the AC14 no-touch clause check, never modified.
FROZEN_SOURCES = ("coreci_broker.py", "coreci_broker_client.py",
"coreci_broker_ui", "reference_tools")
# v0.2 packaging artifacts the frozen broker must know nothing about.
FROZEN_MARKER_STRINGS = ("_cli_version", "cli/__main__", "coreci_broker_assets")
# The v0.2 test surface consolidated here — presence/count floors only
# (the roundtrips themselves run green in their own files, same suite).
STUB_MODULE_MIN_CASES = {
"test_publish_image.py": 9, # stub publish roundtrip
"test_release_upload.py": 15, # stub release upload + env-secrets
"test_dockerfile_consistency.py": 7,
}
def _load_build_cli():
spec = importlib.util.spec_from_file_location(
"acceptance_build_cli", BUILD_CLI_PATH)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
build_cli = _load_build_cli()
def _sha256(path):
digest = hashlib.sha256()
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(65536), b""):
digest.update(chunk)
return digest.hexdigest()
class _BrokerChild:
"""A subprocess child booted from the built zipapp artifact.
The merged stdout/stderr pipe is pumped via ``select`` + raw ``os.read``
under an explicit deadline until the broker announces its ephemeral
listen address. ``stop()`` is the G-010 kill/drain protocol:
``terminate()`` then a full ``communicate()`` drain (kill fallback on
drain timeout). It is idempotent and registered with ``addCleanup`` at
construction, so no child leaks even when an assertion trips mid-test.
Silence and pass-once assertions are made ONLY on ``full_output()``
after ``stop()``.
"""
def __init__(self, testcase, artifact, env, cwd):
self.testcase = testcase
self.artifact = artifact
self.proc = subprocess.Popen(
[sys.executable, artifact],
env=env, cwd=cwd,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
)
self.lines = []
self.partial = b""
self.drained = b""
self.base_url = None
self._eof = False
self._stopped = False
testcase.addCleanup(self.stop)
self._await_listening(BOOT_DEADLINE_SECONDS)
def _pump(self, timeout):
if self._eof:
return False
fd = self.proc.stdout.fileno()
ready, _, _ = select.select([fd], [], [], timeout)
if not ready:
return True
try:
chunk = os.read(fd, 4096)
except OSError:
chunk = b""
if not chunk:
self._eof = True
return False
self.partial += chunk
while b"\n" in self.partial:
raw, self.partial = self.partial.split(b"\n", 1)
self.lines.append(raw.decode("utf-8", "replace"))
return True
def _await_listening(self, deadline_seconds):
deadline = time.monotonic() + deadline_seconds
cursor = 0
while time.monotonic() < deadline:
alive = self._pump(POLL_SECONDS)
while cursor < len(self.lines):
line = self.lines[cursor]
cursor += 1
if line.startswith(LISTENING_PREFIX):
self.base_url = line.split(" on ", 1)[1].strip()
return
if not alive:
break
self.stop()
self.testcase.fail(
"packaged broker did not announce %r within %.0fs "
"(artifact=%s exit=%s)\n--- captured child output ---\n%s"
% (LISTENING_PREFIX, deadline_seconds, self.artifact,
self.proc.returncode, self.full_output())
)
def stop(self):
if self._stopped:
return
self._stopped = True
if self.proc.poll() is None:
self.proc.terminate()
try:
remainder, _ = self.proc.communicate(timeout=DRAIN_TIMEOUT_SECONDS)
except subprocess.TimeoutExpired:
self.proc.kill()
remainder, _ = self.proc.communicate()
self.drained = remainder or b""
def full_output(self):
text = "\n".join(self.lines)
tail = self.partial + self.drained
if tail:
text += "\n" + tail.decode("utf-8", "replace")
return text
def printed_admin_pass(self):
values = [
line.split("=", 1)[1]
for line in self.lines
if line.startswith(ADMIN_PASS_PREFIX)
]
self.testcase.assertEqual(
1, len(values),
"expected exactly one %s line, got: %r" % (ADMIN_PASS_PREFIX, values)
)
return values[0]
class AcceptanceV02Tests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.build_root = tempfile.TemporaryDirectory(prefix="coreci-acc-v02-")
cls.addClassCleanup(cls.build_root.cleanup)
cls.artifact = build_cli.build(
VERSION, repo=REPO, dist_dir=cls.build_root.name
)
cls.artifact_sha256 = _sha256(cls.artifact)
# -- harness helpers --------------------------------------------------
def _temp_dir(self):
handle = tempfile.TemporaryDirectory(prefix="coreci-acc-v02-")
self.addCleanup(handle.cleanup)
return handle.name
def _child_env(self, **overrides):
"""A child env with no inherited BROKER_* keys, then overrides."""
env = {
key: value
for key, value in os.environ.items()
if not key.startswith("BROKER_")
}
env.update(overrides)
return env
def _state_env(self, state_dir):
"""Ephemeral bind + temp state + a seeded file creds source."""
creds_path = os.path.join(state_dir, "creds.json")
with open(creds_path, "w", encoding="utf-8") as handle:
json.dump({PROXMOX_PATH: PROXMOX_SECRET}, handle)
return self._child_env(
BROKER_BIND="127.0.0.1:0",
BROKER_ENV_PATH=os.path.join(state_dir, ".env"),
BROKER_DB_PATH=os.path.join(state_dir, "broker.db"),
BROKER_AUDIT_LOG_PATH=os.path.join(state_dir, "broker.audit.log"),
BROKER_CREDS_SOURCE="file",
BROKER_CREDS_FILE=creds_path,
)
def _boot(self, state_dir):
"""Boot the artifact against state_dir; CWD is a separate temp dir."""
return _BrokerChild(
self, self.artifact, self._state_env(state_dir), self._temp_dir()
)
def _request(self, base, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(
request, timeout=HTTP_TIMEOUT_SECONDS
) as response:
return response.status, response.read(), response.headers
except urllib.error.HTTPError as error:
return error.code, error.read(), error.headers
def _request_json(self, base, method, path, payload=None, headers=None):
status, body, response_headers = self._request(
base, method, path, payload, headers
)
return status, json.loads(body.decode("utf-8")), response_headers
def _admin_login(self, base, password):
"""v0.1 pattern: POST login, assert the broker_session cookie."""
status, body, headers = self._request_json(
base, "POST", LOGIN_PATH,
{"username": ADMIN_USER, "password": password},
)
self.assertEqual(200, status)
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
raw_cookie = headers.get("Set-Cookie")
self.assertTrue(raw_cookie)
self.assertTrue(raw_cookie.startswith("broker_session="))
self.assertIn("HttpOnly", raw_cookie)
self.assertIn("SameSite=Lax", raw_cookie)
return {"Cookie": raw_cookie.split(";", 1)[0]}
def _load_test_module(self, file_name):
"""Import a sibling test module cleanly (by path, any invocation)."""
path = os.path.join(TESTS_DIR, file_name)
stem = os.path.splitext(file_name)[0]
spec = importlib.util.spec_from_file_location(
"acceptance_probe_" + stem, path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def _runnable_case_count(self, module):
"""Flatten defaultTestLoader's suite, counting runnable TestCases."""
suite = unittest.defaultTestLoader.loadTestsFromModule(module)
stack, count = [suite], 0
while stack:
item = stack.pop()
if isinstance(item, unittest.TestSuite):
stack.extend(item)
else:
self.assertIsInstance(item, unittest.TestCase)
count += 1
return count
# -- matrix 1: boot parity through the artifact ----------------------
def test_boot_parity_through_artifact_second_boot_silent(self):
state_dir = self._temp_dir()
child = self._boot(state_dir)
base = child.base_url
printed_pass = child.printed_admin_pass()
# liveness + full UI served from the packaged artifact
status, body, _ = self._request_json(base, "GET", "/healthz")
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
for ui_path in (UI_LOGIN_PATH, UI_DASHBOARD_PATH):
status, body, headers = self._request(base, "GET", ui_path)
self.assertEqual(200, status, ui_path)
self.assertTrue(
headers.get("Content-Type", "").startswith("text/html"),
ui_path,
)
# the tool-author surface: client module byte-exact vs the repo file
status, served, _ = self._request(base, "GET", CLIENT_STATIC_PATH)
self.assertEqual(200, status)
with open(REPO_CLIENT_PATH, "rb") as handle:
self.assertEqual(handle.read(), served)
# one consolidated v0.1 request-pattern flow:
# admin login -> app + user allowlist -> agent register -> state
# put/get -> secret read
cookie = self._admin_login(base, printed_pass)
app_uuid = str(uuidlib.uuid4())
status, body, _ = self._request_json(
base, "POST", APPS_PATH,
{"uuid": app_uuid, "name": "acceptance-tool"}, headers=cookie,
)
self.assertEqual(201, status)
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
status, body, _ = self._request_json(
base, "POST", USERS_PATH,
{"user_id": "ceo", "email": "ceo@coreci.dev"}, headers=cookie,
)
self.assertEqual(201, status)
status, body, _ = self._request_json(
base, "POST", REGISTER_PATH,
{"agent_id": "proxmox", "app_registration_uuid": app_uuid,
"user_id": "ceo", "tags": ["coreci"]},
)
self.assertEqual(201, status)
app_header = {"Authorization": "App %s %s" % (app_uuid, body["token"])}
state_document = {"milestone": "v0.2", "matrix": "acceptance"}
status, body, _ = self._request_json(
base, "PUT", AGENT_STATE_PATH, state_document, headers=app_header
)
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
status, body, _ = self._request_json(
base, "GET", AGENT_STATE_PATH, headers=app_header
)
self.assertEqual(200, status)
self.assertEqual(state_document, body)
status, body, _ = self._request_json(
base, "GET", SECRETS_PATH, headers=app_header
)
self.assertEqual(200, status)
self.assertEqual({"path", "data", "version", "updated_at"}, set(body))
self.assertEqual(PROXMOX_PATH, body["path"])
self.assertEqual(PROXMOX_SECRET, body["data"])
# v0.1 state-file parity: 0600 in the state dir
for name in STATE_FILE_NAMES:
path = os.path.join(state_dir, name)
self.assertTrue(os.path.isfile(path), path)
self.assertEqual(0o600, os.stat(path).st_mode & 0o777, path)
# pass printed exactly once + pepper never printed — asserted on
# the FULLY drained output after terminate (G-010)
pepper = None
with open(os.path.join(state_dir, ".env"), "r", encoding="utf-8") as handle:
for line in handle:
if line.startswith("BROKER_INSTANCE_PEPPER="):
pepper = line.split("=", 1)[1].strip()
self.assertIsNotNone(pepper)
child.stop()
first_output = child.full_output()
self.assertEqual(1, first_output.count(ADMIN_PASS_PREFIX), first_output)
self.assertEqual(1, first_output.count(printed_pass))
self.assertNotIn(pepper, first_output)
# SECOND boot on the same state dir: silent after terminate + drain
second = self._boot(state_dir)
# the SAME pass still logs in: state persisted across the reboot
self._admin_login(second.base_url, printed_pass)
second.stop()
second_output = second.full_output()
self.assertIn(LISTENING_PREFIX, second_output)
self.assertNotIn(ADMIN_PASS_PREFIX, second_output)
self.assertNotIn(printed_pass, second_output)
# -- matrix 2: reproducibility ----------------------------------------
def test_reproducibility_two_consecutive_builds_identical_sha256(self):
digests = []
for _ in range(2):
with tempfile.TemporaryDirectory(prefix="coreci-acc-repro-") as dist_dir:
artifact = build_cli.build(
VERSION, repo=REPO, dist_dir=dist_dir
)
digests.append(_sha256(artifact))
self.assertNotEqual(digests[0], "0" * 64)
self.assertEqual(digests[0], digests[1])
# and the shared setUpClass build matches too (same environment)
self.assertEqual(self.artifact_sha256, digests[0])
print("v0.2 acceptance two-build sha256: {}".format(digests[0]))
# -- matrix 3: self-containment ---------------------------------------
def test_self_containment_archive_is_exactly_the_allowlist(self):
ui_root = os.path.join(REPO, "coreci_broker_ui")
expected = {
"__main__.py",
"_cli_version.py",
"coreci_broker.py",
"coreci_broker_client.py",
"coreci_broker_ui/",
}
expected.update(
"coreci_broker_ui/%s" % name
for name in os.listdir(ui_root)
if os.path.isfile(os.path.join(ui_root, name))
)
with zipfile.ZipFile(self.artifact) as archive:
names = set(archive.namelist())
# ONLY the allowlist members — nothing more, nothing less
self.assertEqual(expected, names)
# no .env, no dist/, no state, no tests, no bytecode can ride along
for name in names:
for forbidden in (".env", "dist", "broker.db", "broker.audit.log",
"test", "__pycache__", ".pyc"):
self.assertNotIn(forbidden, name, name)
# per-release artifact name (D-020) for the milestone release
self.assertEqual(
build_cli.artifact_name(VERSION),
os.path.basename(self.artifact),
)
# -- matrix 4: stub publish roundtrip + stub release upload green ------
def test_stub_publish_roundtrip_module_green_in_suite(self):
module = self._load_test_module("test_publish_image.py")
self.assertTrue(hasattr(module, "PublishImageTests"))
count = self._runnable_case_count(module)
self.assertGreaterEqual(
count, STUB_MODULE_MIN_CASES["test_publish_image.py"]
)
def test_stub_release_upload_module_green_in_suite(self):
module = self._load_test_module("test_release_upload.py")
self.assertTrue(hasattr(module, "ReleaseUploadTests"))
self.assertTrue(hasattr(module, "EnvSecretsLoaderTests"))
count = self._runnable_case_count(module)
self.assertGreaterEqual(
count, STUB_MODULE_MIN_CASES["test_release_upload.py"]
)
# -- matrix 6: Dockerfile/.dockerignore consistency ---------------------
def test_dockerfile_consistency_module_green_in_suite(self):
module = self._load_test_module("test_dockerfile_consistency.py")
self.assertTrue(hasattr(module, "DockerfileConsistencyTests"))
count = self._runnable_case_count(module)
self.assertGreaterEqual(
count, STUB_MODULE_MIN_CASES["test_dockerfile_consistency.py"]
)
# -- matrix 5: broker zero-delta (AC14, frozen means untouched) --------
def test_broker_zero_delta_1390_lines_no_packaging_references(self):
with open(BROKER_SOURCE_PATH, "r", encoding="utf-8") as handle:
line_count = sum(1 for _ in handle)
# exact v0.1.5 count, under the D-016..D-018 cap of 1400 (AC14)
self.assertEqual(1390, line_count)
self.assertLess(line_count, 1400)
with open(BROKER_SOURCE_PATH, "r", encoding="utf-8") as handle:
source = handle.read()
# frozen means untouched: no v0.2 packaging reference leaked into it
for marker in FROZEN_MARKER_STRINGS:
self.assertNotIn(marker, source)
def test_broker_untouched_by_any_commit_since_v0_1_5(self):
"""The AC14 no-touch clause, machine-checked when the tag exists."""
try:
verify = subprocess.run(
["git", "rev-parse", "-q", "--verify", "v0.1.5^{commit}"],
cwd=REPO, capture_output=True, text=True,
timeout=GIT_TIMEOUT_SECONDS,
)
except (OSError, subprocess.TimeoutExpired):
self.skipTest("git unavailable in this environment")
if verify.returncode != 0:
self.skipTest("v0.1.5 tag not present in this working tree")
log = subprocess.run(
["git", "log", "--oneline", "v0.1.5..HEAD", "--"] + list(FROZEN_SOURCES),
cwd=REPO, capture_output=True, text=True,
timeout=GIT_TIMEOUT_SECONDS,
)
self.assertEqual(0, log.returncode, log.stderr)
self.assertEqual(
"", log.stdout.strip(),
"v0.2 commits touched frozen broker sources:\n" + log.stdout
)
if __name__ == "__main__":
unittest.main()
-163
View File
@@ -1,163 +0,0 @@
"""Reproducibility and packaging tests for the coreci-broker CLI build.
Covers scripts/build_cli.py (REQ-21): two consecutive same-environment
builds are byte-identical, `_cli_version.VERSION` is embedded, the archive
contains exactly the staging allowlist, entry metadata is normalized to
constants (G-009), both new files py_compile green, and the built artifact
answers `--version` with exit code 0 (shebang + exec bit verified).
"""
import contextlib
import hashlib
import importlib.util
import io
import os
import pathlib
import py_compile
import subprocess
import sys
import tempfile
import unittest
import zipfile
REPO = pathlib.Path(__file__).resolve().parent.parent
BUILD_CLI_PATH = REPO / "scripts" / "build_cli.py"
SHIM_PATH = REPO / "cli" / "__main__.py"
VERSION = "v0.2.0"
UI_PAGES = (
"app_registrations.html",
"audit.html",
"dashboard.html",
"index.html",
"login.html",
"nav.js",
"shared.css",
"user_allowlist.html",
)
FILE_EXTERNAL_ATTR = 0o100644 << 16
DIR_EXTERNAL_ATTR = 0o40755 << 16
SHEBANG = b"#!/usr/bin/env python3\n"
FIXED_DATE_TIME = (1980, 1, 1, 0, 0, 0)
def _load_build_cli():
spec = importlib.util.spec_from_file_location("build_cli", BUILD_CLI_PATH)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
build_cli = _load_build_cli()
def _sha256(path):
digest = hashlib.sha256()
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(65536), b""):
digest.update(chunk)
return digest.hexdigest()
class BuildCliTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.build_root = tempfile.TemporaryDirectory(prefix="coreci-broker-pyz-")
cls.addClassCleanup(cls.build_root.cleanup)
cls.artifact = build_cli.build(
VERSION, repo=str(REPO), dist_dir=cls.build_root.name
)
cls.artifact_sha256 = _sha256(cls.artifact)
def test_two_builds_byte_identical(self):
digests = []
for _ in range(2):
with tempfile.TemporaryDirectory(prefix="coreci-broker-pyz-") as dist_dir:
artifact = build_cli.build(
VERSION, repo=str(REPO), dist_dir=dist_dir
)
digests.append(_sha256(artifact))
self.assertEqual(digests[0], digests[1])
self.assertEqual(self.artifact_sha256, digests[0])
print("two-build sha256: {}".format(digests[0]))
def test_cli_version_module_embedded(self):
with zipfile.ZipFile(self.artifact) as archive:
source = archive.read("_cli_version.py").decode("utf-8")
self.assertEqual('VERSION = "{}"\n'.format(VERSION), source)
namespace = {}
exec(compile(source, "_cli_version.py", "exec"), namespace)
self.assertEqual(VERSION, namespace["VERSION"])
def test_archive_allowlist_exact(self):
ui_dir = REPO / "coreci_broker_ui"
on_disk = {entry.name for entry in ui_dir.iterdir() if entry.is_file()}
self.assertEqual(set(UI_PAGES), on_disk)
expected = {
"__main__.py",
"_cli_version.py",
"coreci_broker.py",
"coreci_broker_client.py",
"coreci_broker_ui/",
}
expected.update("coreci_broker_ui/{}".format(name) for name in UI_PAGES)
with zipfile.ZipFile(self.artifact) as archive:
self.assertEqual(expected, set(archive.namelist()))
def test_entry_metadata_constants(self):
with zipfile.ZipFile(self.artifact) as archive:
infos = archive.infolist()
names = [info.filename for info in infos]
self.assertEqual(sorted(names), names)
self.assertTrue(any(info.is_dir() for info in infos))
self.assertTrue(any(not info.is_dir() for info in infos))
for info in infos:
self.assertEqual(FIXED_DATE_TIME, info.date_time)
if info.is_dir():
self.assertEqual(DIR_EXTERNAL_ATTR, info.external_attr)
else:
self.assertEqual(FILE_EXTERNAL_ATTR, info.external_attr)
self.assertEqual(zipfile.ZIP_DEFLATED, info.compress_type)
def test_py_compile_green(self):
for path in (SHIM_PATH, BUILD_CLI_PATH):
py_compile.compile(str(path), doraise=True)
def test_version_flag(self):
completed = subprocess.run(
[sys.executable, self.artifact, "--version"],
capture_output=True,
text=True,
timeout=60,
)
self.assertEqual(0, completed.returncode)
self.assertEqual(VERSION, completed.stdout.strip())
def test_shebang_and_exec_bit(self):
with open(self.artifact, "rb") as handle:
self.assertEqual(SHEBANG, handle.readline())
self.assertTrue(os.stat(self.artifact).st_mode & 0o111)
completed = subprocess.run(
[self.artifact, "--version"],
capture_output=True,
text=True,
timeout=60,
)
self.assertEqual(0, completed.returncode)
self.assertEqual(VERSION, completed.stdout.strip())
def test_artifact_name_no_double_v(self):
self.assertEqual("coreci-broker-v0.2.0.pyz", build_cli.artifact_name(VERSION))
self.assertEqual("coreci-broker-v0.2.0.pyz", build_cli.artifact_name("0.2.0"))
self.assertEqual("coreci-broker-v0.2.0.pyz", os.path.basename(self.artifact))
def test_cli_requires_version_argument(self):
with contextlib.redirect_stderr(io.StringIO()):
with self.assertRaises(SystemExit) as raised:
build_cli.main([])
self.assertEqual(2, raised.exception.code)
if __name__ == "__main__":
unittest.main()
-528
View File
@@ -1,528 +0,0 @@
"""End-to-end packaged-boot acceptance for the coreci-broker CLI zipapp (REQ-21).
Builds the P1-01 artifact once per class via the scripts/build_cli.py build
API into a ``TemporaryDirectory``, then boots it as a real subprocess child —
always ``[sys.executable, "<pyz>"]``, never a bare "python3" — and drives the
live server with stdlib urllib, reusing the v0.1 acceptance request patterns
(tests/test_acceptance_full.py, tests/test_acceptance_p4.py).
G-010 harness hardening, all bound here:
- explicit deadline loop (30s) polling the merged stdout/stderr pipe for the
``coreci-broker listening on http://…`` line; loud failure with the captured
child output when it never arrives;
- pass-once and silent-second-boot assertions are made only after
``terminate()`` plus a full ``communicate()`` drain of the child;
- child env pins ``BROKER_CREDS_SOURCE=file`` + a seeded ``BROKER_CREDS_FILE``
(v0.1 precedent: tests/test_acceptance_full.py);
- one un-overridden default-anchoring boot — state lands in the child's CWD;
- every child is terminated + drained exactly once (idempotent ``stop``
registered with ``addCleanup``): no leaked processes, and every wait is
deadline-bounded (boot wait, drain timeout with kill fallback, HTTP and
subprocess timeouts).
"""
import importlib.util
import json
import os
import select
import subprocess
import sys
import tempfile
import time
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
BUILD_CLI_PATH = os.path.join(REPO, "scripts", "build_cli.py")
REPO_CLIENT_PATH = os.path.join(REPO, "coreci_broker_client.py")
REPO_LOGIN_PAGE_PATH = os.path.join(REPO, "coreci_broker_ui", "login.html")
REPO_DASHBOARD_PAGE_PATH = os.path.join(REPO, "coreci_broker_ui", "dashboard.html")
VERSION = "v0.2.0"
ADMIN_USER = "admin"
ADMIN_PASS_PREFIX = "BROKER_ADMIN_PASS="
LISTENING_PREFIX = "coreci-broker listening on http://"
ASSET_DIR_NAME = ".coreci_broker_assets"
STATE_FILE_NAMES = (".env", "broker.db", "broker.audit.log")
LOGIN_PATH = "/api/broker/auth/login"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
REGISTER_PATH = "/api/broker/agents/register"
AGENT_STATE_PATH = "/api/broker/agents/proxmox/state"
SECRETS_PATH = "/api/broker/secrets/coreci/proxmox"
AUDIT_PATH = "/api/broker/audit"
CLIENT_STATIC_PATH = "/static/coreci_broker_client.py"
UI_LOGIN_PATH = "/ui/login"
UI_DASHBOARD_PATH = "/ui/dashboard"
PROXMOX_PATH = "coreci/proxmox"
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
GITEA_PATH = "coreci/gitea"
GITEA_SECRET = {"url": "https://git.coreci.dev", "token": "git-tok"}
BOOT_DEADLINE_SECONDS = 30.0
POLL_SECONDS = 0.1
DRAIN_TIMEOUT_SECONDS = 20.0
HTTP_TIMEOUT_SECONDS = 10
SUBPROCESS_TIMEOUT_SECONDS = 60
def _load_build_cli():
spec = importlib.util.spec_from_file_location("build_cli", BUILD_CLI_PATH)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
build_cli = _load_build_cli()
class PackagedBrokerChild:
"""A subprocess child booted from the built zipapp artifact.
stdout and stderr are merged into one pipe read via ``select`` + ``os.read``
(raw fd reads — no buffered-reader/select skew) under an explicit deadline
until the broker announces its ephemeral listen address. ``stop()`` is the
G-010 kill/drain protocol: ``terminate()`` then a full ``communicate()``
drain; it is idempotent and registered with ``addCleanup`` at construction,
so no child leaks even when a boot fails or an assertion trips mid-test.
"""
def __init__(self, testcase, artifact, env, cwd):
self.testcase = testcase
self.artifact = artifact
self.proc = subprocess.Popen(
[sys.executable, artifact],
env=env, cwd=cwd,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
)
self.lines = []
self.partial = b""
self.drained = b""
self.base_url = None
self._eof = False
self._stopped = False
testcase.addCleanup(self.stop)
self._await_listening(BOOT_DEADLINE_SECONDS)
def _pump(self, timeout):
"""Move whatever is on the merged pipe into line storage.
Returns False once the child's stream reached EOF; True otherwise
(including when the poll window simply elapsed with no data).
"""
if self._eof:
return False
fd = self.proc.stdout.fileno()
ready, _, _ = select.select([fd], [], [], timeout)
if not ready:
return True
try:
chunk = os.read(fd, 4096)
except OSError:
chunk = b""
if not chunk:
self._eof = True
return False
self.partial += chunk
while b"\n" in self.partial:
raw, self.partial = self.partial.split(b"\n", 1)
self.lines.append(raw.decode("utf-8", "replace"))
return True
def _await_listening(self, deadline_seconds):
deadline = time.monotonic() + deadline_seconds
cursor = len(self.lines)
while time.monotonic() < deadline:
alive = self._pump(POLL_SECONDS)
while cursor < len(self.lines):
line = self.lines[cursor]
cursor += 1
if line.startswith(LISTENING_PREFIX):
self.base_url = line.split(" on ", 1)[1].strip()
return
if not alive:
break
self.stop()
self.testcase.fail(
"packaged broker did not announce %r within %.0fs "
"(artifact=%s exit=%s)\n--- captured child output ---\n%s"
% (LISTENING_PREFIX, deadline_seconds, self.artifact,
self.proc.returncode, self.full_output())
)
def stop(self):
"""Kill/drain protocol (G-010): terminate, then fully drain."""
if self._stopped:
return
self._stopped = True
if self.proc.poll() is None:
self.proc.terminate()
try:
remainder, _ = self.proc.communicate(timeout=DRAIN_TIMEOUT_SECONDS)
except subprocess.TimeoutExpired:
self.proc.kill()
remainder, _ = self.proc.communicate()
self.drained = remainder or b""
def full_output(self):
"""Everything the child printed: complete lines + unterminated tail."""
text = "\n".join(self.lines)
tail = self.partial + self.drained
if tail:
text += "\n" + tail.decode("utf-8", "replace")
return text
def printed_admin_pass(self):
values = [
line.split("=", 1)[1]
for line in self.lines
if line.startswith(ADMIN_PASS_PREFIX)
]
self.testcase.assertEqual(
1, len(values),
"expected exactly one %s line, got: %r" % (ADMIN_PASS_PREFIX, values)
)
return values[0]
class CliPackagingAcceptanceTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.build_root = tempfile.TemporaryDirectory(prefix="coreci-broker-pkg-")
cls.addClassCleanup(cls.build_root.cleanup)
cls.artifact = build_cli.build(
VERSION, repo=REPO, dist_dir=cls.build_root.name
)
# -- harness helpers -------------------------------------------------
def _temp_dir(self):
handle = tempfile.TemporaryDirectory(prefix="coreci-broker-pkg-")
self.addCleanup(handle.cleanup)
return handle.name
def _child_env(self, **overrides):
"""A child env with no inherited BROKER_* keys, then test overrides."""
env = {
key: value
for key, value in os.environ.items()
if not key.startswith("BROKER_")
}
env.update(overrides)
return env
def _seed_creds(self, state_dir):
creds_path = os.path.join(state_dir, "creds.json")
with open(creds_path, "w", encoding="utf-8") as handle:
json.dump({PROXMOX_PATH: PROXMOX_SECRET, GITEA_PATH: GITEA_SECRET},
handle)
return creds_path
def _state_env(self, state_dir):
"""G-010: pin bind + state paths + a seeded file creds source."""
return self._child_env(
BROKER_BIND="127.0.0.1:0",
BROKER_ENV_PATH=os.path.join(state_dir, ".env"),
BROKER_DB_PATH=os.path.join(state_dir, "broker.db"),
BROKER_AUDIT_LOG_PATH=os.path.join(state_dir, "broker.audit.log"),
BROKER_CREDS_SOURCE="file",
BROKER_CREDS_FILE=self._seed_creds(state_dir),
)
def _boot(self, env, cwd):
return PackagedBrokerChild(self, self.artifact, env, cwd)
def _boot_in_state(self, state_dir, cwd=None):
"""Boot with state pinned into state_dir; CWD is a separate temp dir."""
return self._boot(self._state_env(state_dir),
cwd if cwd is not None else self._temp_dir())
def _assert_state_files_0600_in(self, directory):
for name in STATE_FILE_NAMES:
path = os.path.join(directory, name)
self.assertTrue(os.path.isfile(path), path)
self.assertEqual(0o600, os.stat(path).st_mode & 0o777, path)
def _request(self, base, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(
request, timeout=HTTP_TIMEOUT_SECONDS
) as response:
return response.status, response.read(), response.headers
except urllib.error.HTTPError as error:
return error.code, error.read(), error.headers
def _request_json(self, base, method, path, payload=None, headers=None):
status, body, response_headers = self._request(
base, method, path, payload, headers
)
return status, json.loads(body.decode("utf-8")), response_headers
def _admin_login(self, base, password):
"""v0.1 pattern: POST login, assert the broker_session cookie is set."""
status, body, headers = self._request_json(
base, "POST", LOGIN_PATH,
{"username": ADMIN_USER, "password": password},
)
self.assertEqual(200, status)
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
raw_cookie = headers.get("Set-Cookie")
self.assertTrue(raw_cookie)
self.assertTrue(raw_cookie.startswith("broker_session="))
self.assertIn("HttpOnly", raw_cookie)
self.assertIn("SameSite=Lax", raw_cookie)
return {"Cookie": raw_cookie.split(";", 1)[0]}
def _assert_tool_identity(self, entry, app_uuid):
self.assertEqual("ceo", entry["user_id"])
self.assertEqual(app_uuid, entry["app_registration_uuid"])
self.assertEqual("proxmox", entry["agent_id"])
def _read_repo_bytes(self, path):
with open(path, "rb") as handle:
return handle.read()
# -- tests ------------------------------------------------------------
def test_version_flag_subprocess_exits_zero_printing_version(self):
completed = subprocess.run(
[sys.executable, self.artifact, "--version"],
capture_output=True, text=True, timeout=SUBPROCESS_TIMEOUT_SECONDS,
)
self.assertEqual(0, completed.returncode)
self.assertEqual(VERSION, completed.stdout.strip())
def test_first_boot_state_0600_cwd_independent_and_http_acceptance(self):
state_dir = self._temp_dir()
cwd_dir = self._temp_dir() # deliberately NOT the state dir
child = self._boot_in_state(state_dir, cwd=cwd_dir)
base = child.base_url
status, body, _ = self._request_json(base, "GET", "/healthz")
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
# UI pages served from the materialized assets (byte-exact vs repo)
status, body, headers = self._request(base, "GET", UI_LOGIN_PATH)
self.assertEqual(200, status)
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
self.assertEqual(self._read_repo_bytes(REPO_LOGIN_PAGE_PATH), body)
status, body, headers = self._request(base, "GET", UI_DASHBOARD_PATH)
self.assertEqual(200, status)
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
self.assertEqual(self._read_repo_bytes(REPO_DASHBOARD_PAGE_PATH), body)
# packaged client module served byte-exact vs the repo file
status, served, _ = self._request(base, "GET", CLIENT_STATIC_PATH)
self.assertEqual(200, status)
self.assertEqual(self._read_repo_bytes(REPO_CLIENT_PATH), served)
# materialized assets under <state>/.coreci_broker_assets/<VERSION>/
asset_base = os.path.join(state_dir, ASSET_DIR_NAME, VERSION)
self.assertTrue(os.path.isdir(os.path.join(asset_base, "ui")))
self.assertTrue(os.path.isdir(os.path.join(asset_base, "static")))
for name in ("login.html", "dashboard.html"):
self.assertTrue(
os.path.isfile(os.path.join(asset_base, "ui", name)), name
)
self.assertTrue(os.path.isfile(
os.path.join(asset_base, "static", "coreci_broker_client.py")
))
# state resolved from BROKER_ENV_PATH: 0600 files in the state dir,
# nothing whatsoever in the child's CWD
self._assert_state_files_0600_in(state_dir)
self.assertEqual([], os.listdir(cwd_dir))
# pass printed exactly once — asserted on the FULLY drained output
printed_pass = child.printed_admin_pass()
child.stop()
output = child.full_output()
self.assertEqual(1, output.count(ADMIN_PASS_PREFIX), output)
self.assertEqual(1, output.count(printed_pass))
pepper = None
with open(os.path.join(state_dir, ".env"), "r", encoding="utf-8") as handle:
for line in handle:
if line.startswith("BROKER_INSTANCE_PEPPER="):
pepper = line.split("=", 1)[1].strip()
self.assertIsNotNone(pepper)
self.assertNotIn(pepper, output)
def test_v01_parity_flows_through_packaged_artifact(self):
state_dir = self._temp_dir()
child = self._boot_in_state(state_dir)
base = child.base_url
printed_pass = child.printed_admin_pass()
cookie = self._admin_login(base, printed_pass)
# seed allowlists first via the admin API (v0.1 pattern)
app_uuid = str(uuidlib.uuid4())
status, body, _ = self._request_json(
base, "POST", APPS_PATH,
{"uuid": app_uuid, "name": "proxmox-tool"}, headers=cookie,
)
self.assertEqual(201, status)
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
status, body, _ = self._request_json(
base, "POST", USERS_PATH,
{"user_id": "ceo", "email": "ceo@coreci.dev"}, headers=cookie,
)
self.assertEqual(201, status)
# agent register flow
status, body, _ = self._request_json(
base, "POST", REGISTER_PATH,
{"agent_id": "proxmox", "app_registration_uuid": app_uuid,
"user_id": "ceo", "tags": ["coreci"]},
)
self.assertEqual(201, status)
token = body["token"]
app_header = {"Authorization": "App %s %s" % (app_uuid, token)}
# state put/get roundtrip
state_document = {"phase": "v0.2", "boot": "packaged", "items": ["parity"]}
status, body, _ = self._request_json(
base, "PUT", AGENT_STATE_PATH, state_document, headers=app_header
)
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
status, body, _ = self._request_json(
base, "GET", AGENT_STATE_PATH, headers=app_header
)
self.assertEqual(200, status)
self.assertEqual(state_document, body)
# one secret read via the seeded FileCredsSource
status, body, _ = self._request_json(
base, "GET", SECRETS_PATH, headers=app_header
)
self.assertEqual(200, status)
self.assertEqual({"path", "data", "version", "updated_at"}, set(body))
self.assertEqual(PROXMOX_PATH, body["path"])
self.assertEqual(PROXMOX_SECRET, body["data"])
self.assertRegex(body["version"], r"^[0-9a-f]{8}$")
# audit lines exist for the driven flow
status, body, _ = self._request_json(
base, "GET", AUDIT_PATH + "?limit=1000", headers=cookie
)
self.assertEqual(200, status)
entries = body["entries"]
def select(action, result="ok"):
return [entry for entry in entries
if entry["action"] == action and entry["result"] == result]
logins = select("login")
self.assertEqual(1, len(logins))
self.assertEqual(LOGIN_PATH, logins[0]["path"])
add_apps = select("add_app")
self.assertEqual(1, len(add_apps))
self.assertEqual(APPS_PATH + "/" + app_uuid, add_apps[0]["path"])
add_users = select("add_user")
self.assertEqual(1, len(add_users))
self.assertEqual(USERS_PATH + "/ceo", add_users[0]["path"])
registers = select("register")
self.assertEqual(1, len(registers))
self.assertEqual(REGISTER_PATH, registers[0]["path"])
self._assert_tool_identity(registers[0], app_uuid)
ok_reads = select("read")
self.assertEqual(1, len(ok_reads))
self.assertEqual(PROXMOX_PATH, ok_reads[0]["path"])
self._assert_tool_identity(ok_reads[0], app_uuid)
child.stop()
self.assertEqual(1, child.full_output().count(ADMIN_PASS_PREFIX))
def test_second_boot_silent_and_state_persists(self):
state_dir = self._temp_dir()
first = self._boot_in_state(state_dir)
printed_pass = first.printed_admin_pass()
cookie = self._admin_login(first.base_url, printed_pass)
app_uuid = str(uuidlib.uuid4())
status, _, _ = self._request_json(
first.base_url, "POST", APPS_PATH,
{"uuid": app_uuid, "name": "persist-check"}, headers=cookie,
)
self.assertEqual(201, status)
# kill/drain protocol BEFORE any silence assertion (G-010)
first.stop()
first_output = first.full_output()
self.assertEqual(1, first_output.count(ADMIN_PASS_PREFIX))
self.assertIn(printed_pass, first_output)
second = self._boot_in_state(state_dir)
# the same pass still logs in on the second boot: .env persisted
cookie = self._admin_login(second.base_url, printed_pass)
status, body, _ = self._request_json(
second.base_url, "GET", APPS_PATH, headers=cookie
)
self.assertEqual(200, status)
listed = [row["uuid"] for row in body["registrations"]]
self.assertIn(app_uuid, listed)
second.stop()
second_output = second.full_output()
self.assertIn(LISTENING_PREFIX, second_output)
self.assertNotIn(ADMIN_PASS_PREFIX, second_output)
def test_user_env_ui_root_override_wins_over_materialized_default(self):
state_dir = self._temp_dir()
custom_root = os.path.join(state_dir, "custom_ui")
os.makedirs(custom_root)
marker = b"<h1>packaged-override-marker</h1>\n"
with open(os.path.join(custom_root, "login.html"), "wb") as handle:
handle.write(marker)
# a user .env (at the child's BROKER_ENV_PATH) wins over the shim default
with open(os.path.join(state_dir, ".env"), "w", encoding="utf-8") as handle:
handle.write("BROKER_UI_ROOT=%s\n" % custom_root)
child = self._boot_in_state(state_dir)
status, body, headers = self._request(child.base_url, "GET", UI_LOGIN_PATH)
self.assertEqual(200, status)
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
self.assertEqual(marker, body)
# the custom root is the whole UI root: no dashboard.html there
status, _, _ = self._request(child.base_url, "GET", UI_DASHBOARD_PATH)
self.assertEqual(404, status)
# the materialized static default still applies (not overridden)
status, served, _ = self._request(child.base_url, "GET", CLIENT_STATIC_PATH)
self.assertEqual(200, status)
self.assertEqual(self._read_repo_bytes(REPO_CLIENT_PATH), served)
# materialized defaults still exist on disk; they were simply not forced
self.assertTrue(os.path.isfile(os.path.join(
state_dir, ASSET_DIR_NAME, VERSION, "ui", "login.html"
)))
child.stop()
def test_default_anchoring_boot_writes_state_to_cwd(self):
# the owner's literal happy path: no state overrides except the bind
cwd_dir = self._temp_dir()
env = self._child_env(BROKER_BIND="127.0.0.1:0")
child = self._boot(env, cwd_dir)
status, body, _ = self._request_json(child.base_url, "GET", "/healthz")
self.assertEqual(200, status)
self.assertEqual({"ok": True}, body)
status, body, headers = self._request(child.base_url, "GET", UI_LOGIN_PATH)
self.assertEqual(200, status)
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
# state lands in the child's CWD, at 0600
self._assert_state_files_0600_in(cwd_dir)
child.stop()
self.assertEqual(1, child.full_output().count(ADMIN_PASS_PREFIX))
if __name__ == "__main__":
unittest.main()
-264
View File
@@ -1,264 +0,0 @@
"""Dockerfile consistency tests (G-016 — equivalence made testable).
The Dockerfile documents the docker-user path; the daemonless client
(``scripts/publish_image.py``) is the pipeline's source of truth for
the published image. Byte-identity is NOT required — these tests pin the
equivalence contract: same staged sources (member-set parity — the
docker-staged layout must equal build_cli's staging EXACTLY, or the
packaged shim materializes an empty UI dir and /ui/* 404s in
docker-built images; P3-03/CWE-636), same zipapp invocation, same
entrypoint/working-dir, same pinned base, and a hygienic build context.
"""
import json
import os
import shutil
import subprocess
import sys
import tempfile
import unittest
import zipfile
_SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.dirname(
os.path.abspath(__file__))), "scripts")
if _SCRIPTS_DIR not in sys.path:
sys.path.insert(0, _SCRIPTS_DIR)
import build_cli # noqa: E402 (scripts/ dir on sys.path)
import publish_image # noqa: E402 (scripts/ dir on sys.path)
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
APP_SRC = "/app/src/"
DOCKERFILE_IGNORE_PATTERNS = (
".git", ".env", ".env.*", "data/", "*.db", "*.log", "dist/",
"__pycache__/", "tests/", ".ciagent/",
"**/__pycache__/", "**/*.pyc",
)
def read(name):
with open(os.path.join(REPO, name), "r", encoding="utf-8") as handle:
return handle.read()
def _tree_members(prefix, root):
"""Zipapp member names for the tree at ``root`` staged under
``prefix`` (``""`` for the archive root; the prefix itself is
included — zipapp writes directory entries)."""
members = {prefix} if prefix else set()
for current, dirs, files in os.walk(root):
relative = os.path.relpath(current, root)
base = prefix + ("" if relative == "." else relative + "/")
for name in dirs:
members.add(base + name + "/")
for name in files:
members.add(base + name)
return members
def _build_cli_staged_members():
"""The member set build_cli stages, derived via its constants:
ROOT_SOURCES at the root, UI under the coreci_broker_ui/ prefix,
cli/__main__.py staged as __main__.py, _cli_version.py generated."""
members = set(build_cli.ROOT_SOURCES)
members.add(build_cli.STAGED_SHIM)
members.add(build_cli.VERSION_MODULE)
members |= _tree_members(
build_cli.UI_SOURCE_DIR + "/",
os.path.join(REPO, build_cli.UI_SOURCE_DIR))
return members
def _run_zipapp(staged_dir, artifact):
"""The Dockerfile's packing invocation (zipapp writes dir entries;
the member set is interpreter-stable across 3.x)."""
completed = subprocess.run(
[sys.executable, "-m", "zipapp", staged_dir,
"-o", artifact,
"--python", "/usr/bin/env python3", "--compress"],
capture_output=True, text=True)
if completed.returncode != 0:
raise AssertionError(
"zipapp failed ({}): {}".format(
completed.returncode,
(completed.stderr or completed.stdout).strip()))
class DockerfileConsistencyTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.dockerfile = read("Dockerfile")
cls.dockerignore = read(".dockerignore")
# Join backslash continuations into logical stanzas; skip comments.
stanzas = []
for line in cls.dockerfile.splitlines():
stripped = line.strip()
if not stripped or stripped.startswith("#"):
continue
if stanzas and stanzas[-1].endswith("\\"):
stanzas[-1] = stanzas[-1].rstrip("\\").rstrip() + " " + stripped
continue
stanzas.append(stripped)
cls.stanzas = stanzas
cls.copy_lines = [s for s in stanzas if s.startswith("COPY ")]
cls.zipapp_lines = [s for s in stanzas
if "zipapp" in s and s.startswith("RUN ")]
cls.version_lines = [s for s in stanzas
if "_cli_version.py" in s
and s.startswith("RUN ")]
def _copy_specs(self):
"""[(sources, dest)] per COPY stanza, dest verified to stage
under /app/src/."""
specs = []
for line in self.copy_lines:
tokens = line.split()[1:]
sources, dest = tokens[:-1], tokens[-1]
self.assertTrue(dest.startswith(APP_SRC),
"COPY destination {} must stage under /app/src/"
.format(dest))
specs.append((sources, dest))
return specs
def _docker_staged_members(self):
"""The member set the COPY stanzas stage under /app/src/ (Docker
semantics: directory CONTENTS land flattened INSIDE the
destination dir; a dest without trailing slash is a file rename),
plus the RUN-generated _cli_version.py."""
members = set()
for sources, dest in self._copy_specs():
if dest.endswith("/"):
prefix = dest[len(APP_SRC):]
for source in sources:
path = os.path.join(REPO, source.rstrip("/"))
self.assertTrue(os.path.exists(path),
"COPY source missing in the repo: {}"
.format(source))
if os.path.isdir(path):
members |= _tree_members(prefix, path)
else:
members.add(prefix + os.path.basename(source))
else:
# file-to-file rename (mirrors build_cli's STAGED_SHIM
# staging — zipapp requires __main__.py at the root)
self.assertEqual(len(sources), 1,
"file-rename COPY stages exactly one "
"source: {}".format(dest))
members.add(dest[len(APP_SRC):])
members.add(build_cli.VERSION_MODULE) # the RUN printf stanza
return members
def test_from_pins_base(self):
from_lines = [line for line in self.dockerfile.splitlines()
if line.startswith("FROM ")]
self.assertEqual(len(from_lines), 1)
self.assertEqual(from_lines[0].split()[1], "python:3.11-slim")
def test_copy_staged_member_set_matches_build_cli_staging(self):
"""The COPY stanzas stage EXACTLY build_cli's member set — set
parity, no extras, no missing (a flattened UI copy or a missing
root __main__.py breaks the packaged boot; P3-03/CWE-636)."""
self.assertEqual(self._docker_staged_members(),
_build_cli_staged_members(),
"docker-staged member set == build_cli-staged "
"member set")
for line in self.copy_lines:
self.assertNotIn("dist/", line,
"no pre-built artifact in the build context")
def test_docker_layout_simulation_member_set_parity(self):
"""Docker-layout simulation gate (P3-03/CWE-636): apply the
Dockerfile COPY semantics to the real repo sources, add the
generated _cli_version.py, zipapp-pack, and the archive's member
set must EQUAL the build_cli-built artifact's member set."""
with tempfile.TemporaryDirectory(prefix="docker-layout-") as tmp:
staged = os.path.join(tmp, "src")
os.makedirs(staged)
for sources, dest in self._copy_specs():
if dest.endswith("/"):
target = os.path.join(staged, dest[len(APP_SRC):])
os.makedirs(target, exist_ok=True)
for source in sources:
path = os.path.join(REPO, source.rstrip("/"))
if os.path.isdir(path):
# directory CONTENTS land inside the dest dir
shutil.copytree(path, target,
dirs_exist_ok=True)
else:
shutil.copy2(
path,
os.path.join(target,
os.path.basename(source)))
else:
self.assertEqual(len(sources), 1)
shutil.copy2(os.path.join(REPO, sources[0]),
os.path.join(staged, dest[len(APP_SRC):]))
with open(os.path.join(staged, build_cli.VERSION_MODULE), "w",
encoding="utf-8") as handle:
handle.write('VERSION = "{}"\n'.format("v0.2-layout-sim"))
docker_artifact = os.path.join(tmp, "docker-layout.pyz")
_run_zipapp(staged, docker_artifact)
with zipfile.ZipFile(docker_artifact) as archive:
docker_members = set(archive.namelist())
with tempfile.TemporaryDirectory(prefix="build-cli-ref-") as dist:
reference = build_cli.build("v0.2-layout-sim", repo=REPO,
dist_dir=dist)
with zipfile.ZipFile(reference) as archive:
build_members = set(archive.namelist())
self.assertEqual(docker_members, build_members,
"docker-layout archive members == build_cli "
"artifact members")
def test_zipapp_invocation_flags_match_build_script(self):
"""Same --python interpreter and --compress flag as build_cli."""
self.assertEqual(len(self.zipapp_lines), 1)
stanza = self.zipapp_lines[0]
self.assertIn('--python "/usr/bin/env python3"', stanza)
self.assertIn("--compress", stanza)
self.assertIn("-m zipapp", stanza)
self.assertIn("/app/src", stanza)
self.assertIn("/app/coreci-broker.pyz", stanza)
def test_version_module_format_parity(self):
"""ARG VERSION feeds the same VERSION = "<value>" format."""
self.assertEqual(len(self.version_lines), 1)
self.assertIn('VERSION = "%s"', self.version_lines[0])
self.assertEqual(build_cli.VERSION_MODULE, "_cli_version.py")
self.assertTrue(any(s.startswith("ARG VERSION=") for s in self.stanzas),
"ARG VERSION provides the version")
def test_entrypoint_and_workdir_match_published_config(self):
"""ENTRYPOINT/WORKDIR equal the daemonless client's pushed config."""
base_config = json.loads('{"config": {}, "rootfs": {}}')
blob = json.loads(publish_image.build_config_blob(
base_config, "sha256:" + "0" * 64, "v-parity"))
entrypoint = blob["config"]["Entrypoint"]
working_dir = blob["config"]["WorkingDir"]
docker_entrypoint = [line for line in self.dockerfile.splitlines()
if line.startswith("ENTRYPOINT ")]
docker_workdir = [line for line in self.dockerfile.splitlines()
if line.startswith("WORKDIR ")]
self.assertEqual(len(docker_entrypoint), 1)
self.assertEqual(json.loads(
docker_entrypoint[0].split(" ", 1)[1]), entrypoint)
self.assertEqual(len(docker_workdir), 1)
self.assertEqual(docker_workdir[0].split(" ", 1)[1], working_dir)
self.assertEqual(working_dir, "/state")
def test_dockerignore_hygiene(self):
entries = {line.strip() for line in self.dockerignore.splitlines()
if line.strip() and not line.strip().startswith("#")}
for pattern in DOCKERFILE_IGNORE_PATTERNS:
self.assertIn(pattern, entries,
".dockerignore must exclude {}".format(pattern))
def test_no_absolute_source_copies(self):
for line in self.copy_lines:
for token in line.split()[1:-1]:
self.assertFalse(token.startswith("/"),
"build context sources are repo-relative")
if __name__ == "__main__":
unittest.main()
-207
View File
@@ -1,207 +0,0 @@
import base64
import contextlib
import errno
import io
import json
import os
import sys
import tempfile
import unittest
import urllib.error
import urllib.request
import uuid as uuidlib
from unittest import mock
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
import coreci_broker
ADMIN_USER = "admin"
ADMIN_PASS = "correct-horse-battery-staple"
REGISTER_PATH = "/api/broker/agents/register"
APPS_PATH = "/api/broker/app_registrations"
USERS_PATH = "/api/broker/user_allowlist"
AGENT_ID_BODY = {"error": 400, "message": "invalid agent_id"}
TAGS_BODY = {"error": 400, "message": "invalid tags"}
USER_ID_BODY = {"error": 400, "message": "invalid user_id"}
def _basic_header(username, password):
raw = "%s:%s" % (username, password)
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
class P5RegisterValidationTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.env = {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_ADMIN_USER": ADMIN_USER,
"BROKER_ADMIN_PASS": ADMIN_PASS,
}
with contextlib.redirect_stdout(io.StringIO()):
self.broker = coreci_broker.boot(dict(self.env))
self.addCleanup(self.broker.audit.close)
self.addCleanup(self.broker.store.close)
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
self.addCleanup(self.server.shutdown)
self.addCleanup(self.server.server_close)
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
self.app_uuid = str(uuidlib.uuid4())
self.user_id = "ceo"
self._seed()
def _request(self, method, path, payload=None, headers=None):
data = None if payload is None else json.dumps(payload).encode("utf-8")
request = urllib.request.Request(self.base + path, data=data, method=method)
for key, value in (headers or {}).items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=10) as response:
return response.status, json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
return error.code, json.loads(error.read().decode("utf-8"))
def _seed(self):
status, _ = self._request("POST", APPS_PATH, {"uuid": self.app_uuid}, self.admin)
self.assertEqual(201, status)
status, _ = self._request("POST", USERS_PATH, {"user_id": self.user_id}, self.admin)
self.assertEqual(201, status)
def _register(self, agent_id="agent-1", tags=("coreci",)):
return self._request("POST", REGISTER_PATH, {
"agent_id": agent_id,
"app_registration_uuid": self.app_uuid,
"user_id": self.user_id,
"tags": list(tags),
})
def _audit(self, action=None, result=None):
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
entries = [json.loads(raw) for raw in handle if raw.strip()]
return [
entry for entry in entries
if (action is None or entry["action"] == action)
and (result is None or entry["result"] == result)
]
def test_agent_id_bare_dot_and_dotdot_rejected(self):
for agent_id in (".", ".."):
status, body = self._register(agent_id=agent_id)
self.assertEqual(400, status, repr(agent_id))
self.assertEqual(AGENT_ID_BODY, body)
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
self.assertEqual([], self._audit("register"))
def test_agent_id_control_chars_rejected(self):
for agent_id in ("a\x00b", "a\nb", "a\x1fb", "a\x7fb"):
status, body = self._register(agent_id=agent_id)
self.assertEqual(400, status, repr(agent_id))
self.assertEqual(AGENT_ID_BODY, body)
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
self.assertEqual([], self._audit("register"))
def test_agent_id_embedded_dots_still_accepted(self):
status, body = self._register(agent_id="a.b")
self.assertEqual(201, status)
self.assertTrue(body["created"])
status, body = self._register(agent_id="tool.v2")
self.assertEqual(201, status)
self.assertTrue(body["created"])
self.assertIsNotNone(self.broker.store.get_agent("a.b", self.app_uuid))
self.assertIsNotNone(self.broker.store.get_agent("tool.v2", self.app_uuid))
self.assertEqual(2, len(self._audit("register", "ok")))
def test_tag_control_chars_rejected(self):
for tags in (("ok", "ba\x00d"), ("ok", "ba\x7fd"), ("ok", "ba:d")):
status, body = self._register(tags=tags)
self.assertEqual(400, status, repr(tags))
self.assertEqual(TAGS_BODY, body)
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
status, body = self._register(tags=("a.b", ".."))
self.assertEqual(201, status)
self.assertTrue(body["created"])
row = self.broker.store.get_agent("agent-1", self.app_uuid)
self.assertEqual(["..", "a.b"], json.loads(row[3]))
def test_agent_id_path_separators_still_rejected(self):
for agent_id in ("a:b", "a/b", "a\\b"):
status, body = self._register(agent_id=agent_id)
self.assertEqual(400, status, repr(agent_id))
self.assertEqual(AGENT_ID_BODY, body)
self.assertEqual([], self._audit("register"))
status, body = self._register(agent_id="plain-agent")
self.assertEqual(201, status)
self.assertTrue(body["created"])
def test_user_allowlist_add_shares_id_rules(self):
for user_id in (".", "..", "a\x00b", "a:b", "a/b"):
status, body = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
self.assertEqual(400, status, repr(user_id))
self.assertEqual(USER_ID_BODY, body)
status, body = self._request("POST", USERS_PATH, {"user_id": "ceo.two"}, self.admin)
self.assertEqual(201, status)
class P5ReexecGuardTests(unittest.TestCase):
def setUp(self):
coreci_broker._reset_server_guard()
self.addCleanup(coreci_broker._reset_server_guard)
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
def _env(self, bind):
return {
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
"BROKER_BIND": bind,
}
def _exec_broker(self):
with open(coreci_broker.__file__, "r", encoding="utf-8") as handle:
source = handle.read()
namespace = {"__name__": "coreci_broker_reexec"}
exec(compile(source, "coreci_broker.py", "exec"), namespace)
return namespace
def _healthz(self, port):
url = "http://127.0.0.1:%d/healthz" % port
with urllib.request.urlopen(url, timeout=5) as response:
return response.status, json.loads(response.read().decode("utf-8"))
def test_fresh_namespace_tools_fails_loud_when_old_module_serves(self):
first = self._exec_broker()
with mock.patch.dict(os.environ, self._env("127.0.0.1:0"), clear=True):
with contextlib.redirect_stdout(io.StringIO()):
first_tools = first["Tools"]()
first_again = first["Tools"]()
server = first["_started_server"]
self.addCleanup(first["_reset_server_guard"])
self.assertIs(server, first_tools._server)
self.assertIs(first_tools._server, first_again._server)
port = server.server_address[1]
self.assertNotEqual(0, port)
second = self._exec_broker()
self.assertIsNone(second["_started_server"])
with mock.patch.dict(os.environ, self._env("127.0.0.1:%d" % port), clear=True):
with contextlib.redirect_stdout(io.StringIO()):
with self.assertRaises(RuntimeError) as caught:
second["Tools"]()
message = str(caught.exception).lower()
self.assertIn("restart", message)
self.assertIn("open webui", message)
self.assertIsInstance(caught.exception.__cause__, OSError)
self.assertEqual(errno.EADDRINUSE, caught.exception.__cause__.errno)
self.assertIsNone(second["_started_server"])
self.assertEqual((200, {"ok": True}), self._healthz(port))
if __name__ == "__main__":
unittest.main()
-799
View File
@@ -1,799 +0,0 @@
"""Stub-registry tests for scripts/publish_image.py (REQ-23, G-019).
Two local stub servers stand in for the public image registry (anonymous
pull) and the gitea container registry (JWT auth + blob uploads +
manifest PUTs). No network access; tiny in-memory fixtures; every
spawned server is cleaned via addCleanup.
"""
import base64
import hashlib
import io
import json
import os
import sys
import tempfile
import threading
import time
import unittest
import urllib.parse
from contextlib import redirect_stderr, redirect_stdout
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
_SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.dirname(
os.path.abspath(__file__))), "scripts")
if _SCRIPTS_DIR not in sys.path:
sys.path.insert(0, _SCRIPTS_DIR)
import publish_image # noqa: E402 (scripts/ dir on sys.path)
OCI_MANIFEST = "application/vnd.oci.image.manifest.v1+json"
OCI_INDEX = "application/vnd.oci.image.index.v1+json"
OCI_CONFIG = "application/vnd.oci.image.config.v1+json"
OCI_LAYER = "application/vnd.oci.image.layer.v1.tar+gzip"
OWNER = "testowner"
REPO = "testrepo"
NAME = "{}/{}".format(OWNER, REPO)
def _sha(data):
return "sha256:" + hashlib.sha256(data).hexdigest()
def _dump(payload):
return json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
class HubFixtures:
"""A tiny synthetic base image: index -> amd64/arm64 children."""
def __init__(self):
self.layer0 = b"base layer zero payload\x00" + b"\xa5" * 64
self.layer1 = b"base layer one payload" * 7
self.config = _dump({
"architecture": "amd64",
"os": "linux",
"config": {"Cmd": ["python3"], "Env": ["PATH=/usr/local/bin"]},
"rootfs": {"type": "layers",
"diff_ids": [_sha(self.layer0), _sha(self.layer1)]},
})
self.child = {
"schemaVersion": 2,
"mediaType": OCI_MANIFEST,
"config": {"mediaType": OCI_CONFIG,
"digest": _sha(self.config),
"size": len(self.config)},
"layers": [
{"mediaType": OCI_LAYER, "digest": _sha(self.layer0),
"size": len(self.layer0)},
{"mediaType": OCI_LAYER, "digest": _sha(self.layer1),
"size": len(self.layer1)},
],
}
self.child_digest = _sha(_dump(self.child))
arm64_child = dict(self.child)
arm64_child["annotations"] = {"test-fixture": "arm64"}
self.arm64_child_digest = _sha(_dump(arm64_child))
self.index = {
"schemaVersion": 2,
"mediaType": OCI_INDEX,
"manifests": [
{"mediaType": OCI_MANIFEST, "digest": self.child_digest,
"size": len(_dump(self.child)),
"platform": {"os": "linux", "architecture": "amd64"}},
{"mediaType": OCI_MANIFEST,
"digest": self.arm64_child_digest,
"size": len(_dump(arm64_child)),
"platform": {"os": "linux", "architecture": "arm64"}},
],
}
self.blobs = {
_sha(self.config): self.config,
_sha(self.layer0): self.layer0,
_sha(self.layer1): self.layer1,
}
class _StubHubHandler(BaseHTTPRequestHandler):
fixtures = None # set per test class run
requests = None
token_mode = "ok" # ok | malformed (200 with an INVALID JSON body)
def _record(self):
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length else b""
self.requests.append({
"method": self.command,
"path": self.path,
"authorization": self.headers.get("Authorization", ""),
"accept": self.headers.get("Accept", ""),
"body": body,
})
def _send(self, status, payload, content_type):
data = payload if isinstance(payload, bytes) else _dump(payload)
self.send_response(status)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(data)))
self.end_headers()
if self.command != "HEAD":
self.wfile.write(data)
def do_GET(self):
self._record()
path = urllib.parse.urlsplit(self.path).path
query = dict(urllib.parse.parse_qsl(
urllib.parse.urlsplit(self.path).query))
if path == "/token":
if self.token_mode == "malformed":
# 200 + invalid JSON: the CWE-754 probe — must surface as
# a remote failure WITH the IMAGE_PENDING marker, never
# a bare traceback.
self._send(200, b"this { is not valid json", "application/json")
else:
self._send(200, {"token": "hub-anon-token"}, "application/json")
elif path == "/v2/library/python/manifests/3.11-slim":
self._send(200, self.fixtures.index, OCI_INDEX)
elif path == "/v2/library/python/manifests/" + self.fixtures.child_digest:
self._send(200, self.fixtures.child, OCI_MANIFEST)
else:
digest = path.rsplit("/", 1)[-1]
if digest in self.fixtures.blobs:
self._send(200, self.fixtures.blobs[digest],
"application/octet-stream")
else:
self._send(404, {"errors": [{"digest": digest}]},
"application/json")
def log_message(self, format, *args):
pass
class _StubGiteaHandler(BaseHTTPRequestHandler):
"""Scriptable gitea registry stub.
state dict keys: requests, uploaded, token_mode (ok|empty),
upload_mode (monolithic|redirect307|redirect307_cross|stall),
manifest_mode (ok|fail500), redirect_seen, manifest_puts,
cross_origin_base (absolute URL of the second-origin stub used by
redirect307_cross).
"""
state = None
def _record(self):
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length else b""
self.state["requests"].append({
"method": self.command,
"path": self.path,
"authorization": self.headers.get("Authorization", ""),
"content_type": self.headers.get("Content-Type", ""),
"body": body,
})
def _send(self, status, payload=b"", content_type="application/json",
headers=None):
data = payload if isinstance(payload, bytes) else _dump(payload)
self.send_response(status)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(data)))
for key, value in (headers or {}).items():
self.send_header(key, value)
self.end_headers()
if self.command != "HEAD":
self.wfile.write(data)
def do_GET(self):
self._record()
parsed = urllib.parse.urlsplit(self.path)
if parsed.path == "/v2/":
# G-013 smoke check 1: the registry auth challenge (401 +
# Bearer; the realm defaults to this stub's own token
# endpoint so pre-existing state dicts work unchanged).
realm = self.state.get("challenge_realm") or \
"http://{}:{}/v2/token".format(
self.server.server_address[0],
self.server.server_address[1])
self._send(401, {"errors": [{"code": "UNAUTHORIZED"}]},
headers={
"WWW-Authenticate":
'Bearer realm="{}",service="{}"'.format(
realm, publish_image.GITEA_AUTH_SERVICE)})
return
if parsed.path == "/v2/token":
if self.state["token_mode"] == "empty":
self._send(200, {"token": ""})
else:
self._send(200, {"token": "stub-jwt"})
return
if parsed.path.startswith("/api/v1/repos/") and \
"/releases/tags/" in parsed.path:
# G-013 smoke check 3: the gitea API release-by-tag GET
# (state["releases"]: tag -> release payload; absent -> 404).
tag = parsed.path.rsplit("/", 1)[-1]
release = self.state.get("releases", {}).get(tag)
if release is None:
self._send(404, {"errors": [{"message": "release not found"}]})
else:
self._send(200, release)
return
def do_HEAD(self):
self._record()
parsed = urllib.parse.urlsplit(self.path)
digest = parsed.path.rsplit("/", 1)[-1]
if digest in self.state["uploaded"]:
self._send(200, b"")
else:
self._send(404, b"")
def do_POST(self):
self._record()
parsed = urllib.parse.urlsplit(self.path)
query = dict(urllib.parse.parse_qsl(parsed.query))
digest = query.get("digest", "")
if self.state["upload_mode"] == "stall":
time.sleep(15)
return
if self.state["upload_mode"] == "redirect307_cross" and \
digest not in self.state["redirect_seen"]:
# CWE-200 probe: ABSOLUTE Location pointing at a DIFFERENT
# origin (the second stub server) — the client must follow
# WITHOUT the Authorization header.
self.state["redirect_seen"].add(digest)
location = "{}/v2/{}/blobs/uploads/?digest={}".format(
self.state["cross_origin_base"], NAME, digest)
self._send(307, b"", headers={"Location": location})
return
if self.state["upload_mode"] == "redirect307" and \
digest not in self.state["redirect_seen"]:
self.state["redirect_seen"].add(digest)
location = "/v2/{}/blobs/uploads/?digest={}".format(NAME, digest)
self._send(307, b"", headers={"Location": location})
return
if _sha(self.state["requests"][-1]["body"]) == digest:
self.state["uploaded"][digest] = self.state["requests"][-1]["body"]
self._send(201, {"status": "created"})
else:
self._send(400, {"errors": [{"digest": digest}]})
def do_PUT(self):
self._record()
parsed = urllib.parse.urlsplit(self.path)
query = dict(urllib.parse.parse_qsl(parsed.query))
body = self.state["requests"][-1]["body"]
if "/manifests/" in parsed.path:
if self.state["manifest_mode"] == "fail500":
self.state.setdefault("manifest_fail_count", 0)
self.state["manifest_fail_count"] += 1
self._send(500, {"errors": [{"code": "UNKNOWN"}]})
return
manifest = json.loads(body)
referenced = [manifest["config"]["digest"]] + [
layer["digest"] for layer in manifest["layers"]]
if any(digest not in self.state["uploaded"]
for digest in referenced):
self._send(400, {"errors": [{"code": "BLOB_UNKNOWN"}]})
return
if self.headers.get("Content-Type") != OCI_MANIFEST:
self._send(400, {"errors": [{"code": "MANIFEST_INVALID"}]})
return
self.state.setdefault("manifest_puts", []).append(
{"tag": parsed.path.rsplit("/", 1)[-1],
"content_type": self.headers.get("Content-Type", ""),
"manifest": manifest})
self._send(201, {})
return
digest = query.get("digest", "")
if _sha(body) == digest:
self.state["uploaded"][digest] = body
self._send(201, {})
else:
self._send(400, {"errors": [{"code": "DIGEST_INVALID"}]})
class _StubCrossOriginHandler(BaseHTTPRequestHandler):
"""Second-origin stub (another port) for the CWE-200 redirect probe.
state dict keys: requests, uploaded — ``uploaded`` is SHARED with the
gitea stub (the redirect target models the registry's presigned
external storage; a landed blob is visible to the registry), while
``requests`` stays separate so the test can assert on exactly what
this origin received — above all that NO Authorization header
crossed the origin boundary.
"""
state = None
def _record(self):
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length else b""
self.state["requests"].append({
"method": self.command,
"path": self.path,
"authorization": self.headers.get("Authorization", ""),
"content_type": self.headers.get("Content-Type", ""),
"body": body,
})
return body
def _send(self, status, payload=b"", content_type="application/json"):
data = payload if isinstance(payload, bytes) else _dump(payload)
self.send_response(status)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(data)))
self.end_headers()
if self.command != "HEAD":
self.wfile.write(data)
def do_POST(self):
body = self._record()
query = dict(urllib.parse.parse_qsl(
urllib.parse.urlsplit(self.path).query))
digest = query.get("digest", "")
if _sha(body) == digest:
self.state["uploaded"][digest] = body
self._send(201, {"status": "created"})
else:
self._send(400, {"errors": [{"code": "DIGEST_INVALID"}]})
def log_message(self, format, *args):
pass
class PublishImageTests(unittest.TestCase):
"""End-to-end publish flows against local stub registries."""
TAG = "v0.2.2-test"
@classmethod
def setUpClass(cls):
cls.fixtures = HubFixtures()
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.artifact = os.path.join(self.temp.name, "coreci-broker.pyz")
with open(self.artifact, "wb") as handle:
handle.write(b"fake pyz payload for tests" * 3)
self.env_root = os.path.join(self.temp.name, "envroot")
os.makedirs(self.env_root)
with open(os.path.join(self.env_root, ".env.secrets"), "w") as handle:
handle.write("GITEA_TOKEN=testtoken123\n")
self.hub_requests = []
_StubHubHandler.fixtures = self.fixtures
_StubHubHandler.requests = self.hub_requests
_StubHubHandler.token_mode = "ok"
self.hub = ThreadingHTTPServer(("127.0.0.1", 0), _StubHubHandler)
threading.Thread(target=self.hub.serve_forever, daemon=True).start()
self.addCleanup(self.hub.shutdown)
self.addCleanup(self.hub.server_close)
self.gitea_state = {
"requests": [], "uploaded": {}, "token_mode": "ok",
"upload_mode": "monolithic", "manifest_mode": "ok",
"redirect_seen": set(),
}
_StubGiteaHandler.state = self.gitea_state
self.gitea = ThreadingHTTPServer(("127.0.0.1", 0), _StubGiteaHandler)
threading.Thread(target=self.gitea.serve_forever, daemon=True).start()
self.addCleanup(self.gitea.shutdown)
self.addCleanup(self.gitea.server_close)
def run_publish(self, tag=None, timeout=60, artifact=None, env_root=None):
argv = [
tag or self.TAG,
"--artifact", artifact or self.artifact,
"--hub-registry", "http://127.0.0.1:{}".format(self.hub.server_address[1]),
"--hub-auth", "http://127.0.0.1:{}/token".format(self.hub.server_address[1]),
"--gitea-registry", "http://127.0.0.1:{}".format(self.gitea.server_address[1]),
"--owner", OWNER, "--repo", REPO,
"--env-root", env_root or self.env_root,
"--timeout", str(timeout),
]
out, err = io.StringIO(), io.StringIO()
with redirect_stdout(out), redirect_stderr(err):
code = publish_image.main(argv)
return code, out.getvalue(), err.getvalue()
# 1 -----------------------------------------------------------------
def test_happy_path_end_to_end(self):
code, out, err = self.run_publish()
self.assertEqual(code, 0, err or out)
puts = self.gitea_state.get("manifest_puts", [])
self.assertEqual([entry["tag"] for entry in puts],
[self.TAG, "latest"],
"version tag FIRST, then latest (D-021)")
manifest = puts[0]["manifest"]
self.assertEqual(puts[0]["content_type"], OCI_MANIFEST)
self.assertEqual(manifest["config"]["mediaType"], OCI_CONFIG)
broker_layer = manifest["layers"][-1]
self.assertEqual(broker_layer["mediaType"], OCI_LAYER)
base_descriptors = self.fixtures.child["layers"]
self.assertEqual(manifest["layers"][:-1], base_descriptors,
"base layer descriptors reused verbatim")
# deterministic broker layer digest matches an independent build
with open(self.artifact, "rb") as handle:
artifact_bytes = handle.read()
layer_blob, layer_digest, diff_id = publish_image.build_broker_layer(
artifact_bytes)
self.assertEqual(broker_layer["digest"], layer_digest)
self.assertEqual(broker_layer["size"], len(layer_blob))
# pushed config content
pushed_config = json.loads(
self.gitea_state["uploaded"][manifest["config"]["digest"]])
self.assertEqual(pushed_config["config"]["Entrypoint"],
["python3", "/app/coreci-broker.pyz"])
self.assertEqual(pushed_config["config"]["Cmd"], [])
self.assertEqual(pushed_config["config"]["WorkingDir"], "/state")
self.assertEqual(pushed_config["config"]["Labels"],
{"org.opencontainers.image.version": self.TAG})
self.assertEqual(pushed_config["rootfs"]["diff_ids"][-1], diff_id)
self.assertEqual(len(pushed_config["rootfs"]["diff_ids"]),
len(self.fixtures.child["layers"]) + 1)
# base config blob uploaded verbatim (digest invariant)
self.assertEqual(
self.gitea_state["uploaded"][self.fixtures.child["config"]["digest"]],
self.fixtures.config)
# all blobs present
for descriptor in manifest["layers"]:
self.assertIn(descriptor["digest"], self.gitea_state["uploaded"])
self.assertNotIn("IMAGE_PENDING", out)
# 2 -----------------------------------------------------------------
def test_head_skip_blob_not_reuploaded(self):
preseed = self.fixtures.child["layers"][0]["digest"]
self.gitea_state["uploaded"][preseed] = self.fixtures.layer0
code, out, err = self.run_publish()
self.assertEqual(code, 0, err or out)
posts = [r for r in self.gitea_state["requests"]
if r["method"] == "POST" and preseed in r["path"]]
self.assertEqual(posts, [], "pre-seeded blob must not be re-uploaded")
hub_blob_gets = [r for r in self.hub_requests
if r["method"] == "GET" and "/blobs/" in r["path"]
and preseed in r["path"]]
self.assertEqual(hub_blob_gets, [],
"skipped blob is never fetched from the hub")
# 3 -----------------------------------------------------------------
def test_redirect_307_relative_location_followed(self):
self.gitea_state["upload_mode"] = "redirect307"
code, out, err = self.run_publish()
self.assertEqual(code, 0, err or out)
blob_posts = [r for r in self.gitea_state["requests"]
if r["method"] == "POST" and "/blobs/uploads/" in r["path"]
and "digest=" in r["path"]]
self.assertGreater(len(blob_posts), 0)
digests = [dict(urllib.parse.parse_qsl(
urllib.parse.urlsplit(r["path"]).query))["digest"]
for r in blob_posts]
self.assertTrue(all(d in self.gitea_state["uploaded"]
for d in digests),
"every redirected upload must land")
# each redirected digest POSTed twice (307 + follow)
for digest in set(digests):
self.assertEqual(digests.count(digest), 2,
"307 then explicit re-POST per digest")
# same-origin (relative Location) redirects KEEP the
# Authorization header — the CWE-200 strip is cross-origin ONLY
for request in blob_posts:
self.assertEqual(
request["authorization"], "Bearer stub-jwt",
"same-origin redirect must keep Authorization")
# 4 -----------------------------------------------------------------
def test_anonymous_token_trap_refused_before_push(self):
self.gitea_state["token_mode"] = "empty"
code, out, err = self.run_publish()
self.assertEqual(code, 1, "remote auth failure -> non-zero")
self.assertIn("anonymous-200 trap refused", err)
self.assertIn("IMAGE_PENDING", out)
post_put = [r for r in self.gitea_state["requests"]
if r["method"] in ("POST", "PUT")]
self.assertEqual(post_put, [],
"no blob upload or manifest PUT may happen")
# 5 -----------------------------------------------------------------
def test_layer_and_config_determinism(self):
with open(self.artifact, "rb") as handle:
artifact = handle.read()
first = publish_image.build_broker_layer(artifact)
second = publish_image.build_broker_layer(artifact)
self.assertEqual(first[1], second[1], "layer digest stable")
self.assertEqual(first[2], second[2], "diff_id stable")
base_config = json.loads(self.fixtures.config)
c1 = publish_image.build_config_blob(base_config, first[2], "t1")
c2 = publish_image.build_config_blob(base_config, first[2], "t1")
self.assertEqual(_sha(c1), _sha(c2), "config digest stable")
# 6 -----------------------------------------------------------------
def test_amd64_child_selected(self):
code, out, err = self.run_publish()
self.assertEqual(code, 0, err or out)
child_gets = [r for r in self.hub_requests
if "/manifests/sha256:" in r["path"]]
self.assertTrue(child_gets, "child manifest must be fetched by digest")
self.assertTrue(all(self.fixtures.child_digest in r["path"]
for r in child_gets),
"amd64 child only")
self.assertFalse(any(self.fixtures.arm64_child_digest in r["path"]
for r in self.hub_requests),
"arm64 child never touched")
# 7 -----------------------------------------------------------------
def test_timeout_aborts_rather_than_hangs(self):
self.gitea_state["upload_mode"] = "stall"
started = time.monotonic()
code, out, err = self.run_publish(timeout=1)
elapsed = time.monotonic() - started
self.assertEqual(code, 1)
self.assertIn("IMAGE_PENDING", out)
self.assertLess(elapsed, 30, "bounded retries, no hang")
stalls = [r for r in self.gitea_state["requests"]
if r["method"] == "POST" and "/blobs/uploads/" in r["path"]]
self.assertLessEqual(len(stalls), 3, "bounded retry attempts")
# 8 -----------------------------------------------------------------
def test_manifest_retry_exhaustion_marker(self):
self.gitea_state["manifest_mode"] = "fail500"
code, out, err = self.run_publish()
self.assertEqual(code, 1)
self.assertIn("IMAGE_PENDING", out)
self.assertEqual(self.gitea_state.get("manifest_fail_count"), 3,
"exactly three bounded attempts on the tag PUT")
self.assertNotIn("latest",
[r["path"].rsplit("/", 1)[-1]
for r in self.gitea_state["requests"]
if r["method"] == "PUT"],
"latest must not be attempted after exhaustion")
# 9 -----------------------------------------------------------------
def test_token_discipline(self):
code, out, err = self.run_publish()
self.assertEqual(code, 0, err or out)
token_calls = [r for r in self.gitea_state["requests"]
if r["path"].startswith("/v2/token")]
self.assertTrue(token_calls)
expected = base64.b64encode(
b"publisher:testtoken123").decode("ascii")
for call in token_calls:
self.assertEqual(call["authorization"],
"Basic {}".format(expected))
# scope requested with pull,push (registry v2 scope form)
query = urllib.parse.unquote(
urllib.parse.urlsplit(token_calls[0]["path"]).query)
self.assertIn("repository:{}/{}:pull,push".format(OWNER, REPO),
query)
# missing token: local precondition, zero remote requests
empty_root = os.path.join(self.temp.name, "emptyroot")
os.makedirs(empty_root)
before = len(self.gitea_state["requests"]) + len(self.hub_requests)
code, out, err = self.run_publish(env_root=empty_root)
self.assertEqual(code, 2, "local precondition -> exit 2")
self.assertNotIn("IMAGE_PENDING", out, "no marker for local errors")
after = len(self.gitea_state["requests"]) + len(self.hub_requests)
self.assertEqual(before, after, "no request of any kind was made")
# 10 ----------------------------------------------------------------
def test_malformed_hub_token_json_keeps_pending_marker(self):
"""CWE-754 marker discipline: a 200 with an INVALID JSON body from
the hub token endpoint fails as a remote answer — exit 1 WITH the
IMAGE_PENDING marker — never an uncaught-ValueError traceback."""
_StubHubHandler.token_mode = "malformed"
code, out, err = self.run_publish()
self.assertEqual(code, 1)
self.assertIn("malformed JSON", err,
"the failure class must be named in the error")
self.assertIn("IMAGE_PENDING", out,
"marker discipline holds for malformed remote JSON")
self.assertNotIn("Traceback", out + err)
self.assertEqual(self.gitea_state["requests"], [],
"nothing is attempted against gitea")
# 11 ----------------------------------------------------------------
def test_redirect_307_cross_origin_strips_authorization(self):
"""CWE-200: a 307 whose Location resolves to a DIFFERENT origin
is followed WITHOUT the Authorization header — the registry JWT
never crosses origins — and the blob still lands."""
cross_state = {"requests": [], "uploaded": self.gitea_state["uploaded"]}
_StubCrossOriginHandler.state = cross_state
cross = ThreadingHTTPServer(("127.0.0.1", 0), _StubCrossOriginHandler)
threading.Thread(target=cross.serve_forever, daemon=True).start()
self.addCleanup(cross.shutdown)
self.addCleanup(cross.server_close)
self.gitea_state["upload_mode"] = "redirect307_cross"
self.gitea_state["cross_origin_base"] = "http://127.0.0.1:{}".format(
cross.server_address[1])
code, out, err = self.run_publish()
self.assertEqual(code, 0, err or out)
self.assertTrue(cross_state["requests"],
"the redirect was followed to the other origin")
for request in cross_state["requests"]:
self.assertEqual(
request["authorization"], "",
"Authorization must be stripped on cross-origin redirects")
# the blob landed (visible through the shared registry blob store)
self.assertEqual(len(self.gitea_state["uploaded"]), 5,
"base config + 2 base layers + broker layer + "
"image config all landed")
# and the flow still completes: both manifest PUTs back on the
# origin, where Authorization remains in play
self.assertEqual(
[entry["tag"] for entry in self.gitea_state.get(
"manifest_puts", [])],
[self.TAG, "latest"])
class SmokeModeTests(unittest.TestCase):
"""G-013 --smoke logic, stub-tested.
The LIVE smoke never runs in this suite — it executes only at the
P3-05 ship, where the ship gate sets PUBLISH_SMOKE=1 in the
environment (the script itself never reads that variable). Here the
four GET-only checks run against fresh stub instances: zero network,
and the stub records double as the read-only proof (a POST/PUT
landing anywhere in a smoke run is a hard failure).
"""
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.env_root = os.path.join(self.temp.name, "envroot")
os.makedirs(self.env_root)
with open(os.path.join(self.env_root, ".env.secrets"), "w") as handle:
handle.write("GITEA_TOKEN=testtoken123\n")
self.fixtures = HubFixtures()
self.hub_requests = []
_StubHubHandler.fixtures = self.fixtures
_StubHubHandler.requests = self.hub_requests
_StubHubHandler.token_mode = "ok"
self.hub = ThreadingHTTPServer(("127.0.0.1", 0), _StubHubHandler)
threading.Thread(target=self.hub.serve_forever, daemon=True).start()
self.addCleanup(self.hub.shutdown)
self.addCleanup(self.hub.server_close)
self.gitea_state = {
"requests": [], "uploaded": {}, "token_mode": "ok",
"upload_mode": "monolithic", "manifest_mode": "ok",
"redirect_seen": set(), "releases": {},
}
_StubGiteaHandler.state = self.gitea_state
self.gitea = ThreadingHTTPServer(("127.0.0.1", 0), _StubGiteaHandler)
threading.Thread(target=self.gitea.serve_forever, daemon=True).start()
self.addCleanup(self.gitea.shutdown)
self.addCleanup(self.gitea.server_close)
# The /v2/ challenge realm must point back at THIS stub's token
# endpoint so the recorded-realm exchange stays in-stub.
self.gitea_state["challenge_realm"] = \
"http://127.0.0.1:{}/v2/token".format(
self.gitea.server_address[1])
def run_smoke(self, tag, env_root=None):
argv = [
tag, "--smoke",
"--hub-registry",
"http://127.0.0.1:{}".format(self.hub.server_address[1]),
"--hub-auth",
"http://127.0.0.1:{}/token".format(
self.hub.server_address[1]),
"--gitea-registry",
"http://127.0.0.1:{}".format(self.gitea.server_address[1]),
"--owner", OWNER, "--repo", REPO,
"--env-root", env_root or self.env_root,
]
out, err = io.StringIO(), io.StringIO()
with redirect_stdout(out), redirect_stderr(err):
code = publish_image.main(argv)
return code, out.getvalue(), err.getvalue()
def _post_put(self):
"""(gitea POST/PUT, hub POST/PUT) across the stub records."""
return ([r for r in self.gitea_state["requests"]
if r["method"] in ("POST", "PUT")],
[r for r in self.hub_requests
if r["method"] in ("POST", "PUT")])
def _release_gets(self):
return [r for r in self.gitea_state["requests"]
if "/releases/tags/" in r["path"]]
# 12 ----------------------------------------------------------------
def test_smoke_all_checks_pass(self):
self.gitea_state["releases"]["v0.2.0"] = {
"tag_name": "v0.2.0",
"assets": [{"name": "phase0-notes.txt"}],
}
code, out, err = self.run_smoke("v0.2.0")
self.assertEqual(code, 0, err or out)
for marker in ("PASS [1/4]", "PASS [2/4]", "PASS [3/4]",
"PASS [4/4]"):
self.assertIn(marker, out)
self.assertIn("G-021 confirmed: no .pyz asset", out,
"the v0.2.0 run carries the G-021 forge-side "
"confirmation")
self.assertIn("phase0-notes.txt", out, "asset names print as a fact")
self.assertIn(self.fixtures.child_digest, out,
"the amd64 child digest prints (drift check)")
# the manifest GET carries the full Accept set
manifest_gets = [r for r in self.hub_requests
if "/manifests/3.11-slim" in r["path"]]
self.assertEqual(len(manifest_gets), 1)
self.assertEqual(manifest_gets[0]["accept"],
publish_image.ACCEPT_MANIFESTS)
# the release GET uses the gitea API token auth form
release_gets = self._release_gets()
self.assertEqual(len(release_gets), 1)
self.assertEqual(release_gets[0]["authorization"],
"token testtoken123")
# READ-ONLY PROOF: zero POST/PUT recorded at either stub
gitea_writes, hub_writes = self._post_put()
self.assertEqual(gitea_writes, [], "no gitea write of any kind")
self.assertEqual(hub_writes, [], "no hub write of any kind")
self.assertNotIn("IMAGE_PENDING", out)
# 13 ----------------------------------------------------------------
def test_smoke_v0_2_0_no_pyz_assertion(self):
pyz_release = {"tag_name": "v0.2.0",
"assets": [{"name": "coreci-broker-v0.2.0.pyz"}]}
self.gitea_state["releases"]["v0.2.0"] = pyz_release
self.gitea_state["releases"]["v0.2.2"] = pyz_release
# v0.2.0 + a .pyz asset: the G-021 hard assertion fails the smoke
code, out, err = self.run_smoke("v0.2.0")
self.assertEqual(code, 1)
self.assertIn("FAILED [3/4]", out)
self.assertIn("G-021", out, "the failure line names G-021")
self.assertIn(".pyz", out)
self.assertEqual(self.hub_requests, [],
"a FAILED check stops the smoke: the hub "
"resolve (step 4) never runs")
# the SAME release under a different tag: the assertion binds
# v0.2.0 only -> all four checks pass
code, out, err = self.run_smoke("v0.2.2")
self.assertEqual(code, 0, err or out)
for marker in ("PASS [3/4]", "PASS [4/4]"):
self.assertIn(marker, out)
self.assertNotIn("G-021 violation", out)
gitea_writes, hub_writes = self._post_put()
self.assertEqual(gitea_writes + hub_writes, [],
"still read-only across both runs")
# 14 ----------------------------------------------------------------
def test_smoke_trap_refused(self):
self.gitea_state["token_mode"] = "empty"
self.gitea_state["releases"]["v0.2.3"] = {
"tag_name": "v0.2.3", "assets": [],
}
code, out, err = self.run_smoke("v0.2.3")
self.assertEqual(code, 1, "the trap is a loud remote failure")
self.assertIn("FAILED [2/4]", out)
self.assertIn("anonymous-200 trap refused", out)
self.assertIn("PASS [1/4]", out, "check 1 ran (order 1 -> 4)")
self.assertNotIn("PASS [3/4]", out, "checks after the trap never run")
self.assertNotIn("PASS [4/4]", out)
self.assertEqual(self._release_gets(), [],
"the release GET (step 3) is skipped")
self.assertEqual(self.hub_requests, [],
"zero hub requests: the index resolve (step 4) "
"is skipped")
# 15 ----------------------------------------------------------------
def test_smoke_missing_token_local(self):
empty_root = os.path.join(self.temp.name, "emptyroot")
os.makedirs(empty_root)
code, out, err = self.run_smoke("v0.2.3", env_root=empty_root)
self.assertEqual(code, 2, "local precondition -> exit 2")
self.assertIn("ERROR", err)
self.assertIn("GITEA_TOKEN", err)
self.assertEqual(self.gitea_state["requests"], [],
"zero gitea requests — the precondition fires "
"before anything remote")
self.assertEqual(self.hub_requests, [], "zero hub requests")
self.assertNotIn("IMAGE_PENDING", out, "no marker for local errors")
self.assertNotIn("PASS", out, "no smoke check ever ran")
if __name__ == "__main__":
unittest.main()
-511
View File
@@ -1,511 +0,0 @@
"""Stub-gitea release-attacher tests (REQ-22, RESEARCH §2.4).
Drives scripts/release.py against an in-process stub gitea
(``_StubGiteaHandler`` on ``127.0.0.1:0`` — the v0.1 ``_StubOwuiHandler``
precedent; fake token, no live-forge writes) plus the sanctioned
secrets loader scripts/_env_secrets.py: create-or-get in both orders,
byte-exact multipart shape, sha256-from-file discipline (G-017),
idempotent re-attach, bounded-retry exhaustion -> RELEASE_PENDING marker,
and file-only token resolution asserted before the first HTTP call.
"""
import contextlib
import hashlib
import importlib.util
import io
import json
import os
import pathlib
import re
import subprocess
import sys
import tempfile
import threading
import unittest
import urllib.parse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
REPO = pathlib.Path(__file__).resolve().parent.parent
RELEASE_PATH = REPO / "scripts" / "release.py"
ENV_SECRETS_PATH = REPO / "scripts" / "_env_secrets.py"
sys.path.insert(0, str(REPO))
import coreci_broker # repo root on sys.path (parser-parity assertion)
def _load_module(name, path):
spec = importlib.util.spec_from_file_location(name, str(path))
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
release = _load_module("release", RELEASE_PATH)
env_secrets = _load_module("env_secrets", ENV_SECRETS_PATH)
OWNER = "coreci"
REPO_NAME = "broker"
TAG = "v0.2.1"
TOKEN = "testtoken123"
ARTIFACT_NAME = "coreci-broker-v0.2.1.pyz"
_RELEASE_BY_TAG_RE = re.compile(
r"^/api/v1/repos/([^/]+)/([^/]+)/releases/tags/([^/]+)$")
_RELEASES_RE = re.compile(r"^/api/v1/repos/([^/]+)/([^/]+)/releases$")
_ASSETS_RE = re.compile(r"^/api/v1/repos/([^/]+)/([^/]+)/releases/(\d+)/assets$")
def _header_params(value):
"""'a; b=\"c\"' -> {'b': 'c'} — Content-Type/Disposition params."""
params = {}
for item in value.split(";")[1:]:
key, _, val = item.strip().partition("=")
if key:
params[key.strip()] = val.strip().strip('"')
return params
def _split_multipart(content_type, body):
"""Minimal multipart parser: boundary from the Content-Type header."""
boundary = _header_params(content_type).get("boundary", "")
parts = []
if boundary:
for element in body.split(("--" + boundary).encode("utf-8"))[1:]:
if element.startswith(b"--"):
continue # closing delimiter
if element.startswith(b"\r\n"):
element = element[2:]
header_blob, separator, content = element.partition(b"\r\n\r\n")
if not separator:
continue
if content.endswith(b"\r\n"):
content = content[:-2] # framing CRLF before the next delimiter
part_headers = {}
for line in header_blob.decode("utf-8", "replace").split("\r\n"):
name, _, value = line.partition(":")
if name:
part_headers[name.strip().lower()] = value.strip()
parts.append((part_headers, content))
return boundary, parts
class _StubGiteaHandler(BaseHTTPRequestHandler):
"""Scriptable gitea stub (RESEARCH §2.1 routes); records everything."""
state = None # scriptable state dict (class attr, bound per server)
requests = None # every request: method, path, headers, body (+extras)
def log_message(self, format, *args):
pass
def _record(self):
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length else b""
record = {
"method": self.command,
"path": self.path,
"headers": {key.lower(): value
for key, value in self.headers.items()},
"body": body,
}
self.requests.append(record)
return record
def _send_json(self, status, payload):
data = json.dumps(payload).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def do_GET(self):
self._record()
parsed = urllib.parse.urlsplit(self.path)
if _RELEASE_BY_TAG_RE.match(parsed.path):
if self.state.get("release_exists"):
self._send_json(200, {"id": self.state.get("release_id", 970)})
else:
self._send_json(404, {"message": "Release not found"})
return
if _ASSETS_RE.match(parsed.path):
self._send_json(200, list(self.state.get("assets", [])))
return
self._send_json(404, {"message": "Not Found"})
def do_POST(self):
record = self._record()
parsed = urllib.parse.urlsplit(self.path)
if _RELEASES_RE.match(parsed.path):
self.state.setdefault("create_calls", []).append(record)
status = self.state.get("create_status", 201)
if status >= 400:
self._send_json(
status,
{"message": self.state.get("create_error", "create failed")})
else:
self._send_json(status, {"id": self.state.get("release_id", 970)})
return
if _ASSETS_RE.match(parsed.path):
self.state.setdefault("attach_calls", []).append(record)
status = self.state.get("attach_status", 201)
if status >= 400:
self._send_json(
status,
{"message": self.state.get("attach_error", "attach failed")})
return
self._describe_attachment(record, parsed)
self._send_json(201, {
"id": 2,
"name": record["attach"]["query_name"],
"size": record["attach"]["size"],
})
return
self._send_json(404, {"message": "Not Found"})
def _describe_attachment(self, record, parsed):
boundary, parts = _split_multipart(
record["headers"].get("content-type", ""), record["body"])
field_name = filename = part_type = None
content = b""
if len(parts) == 1:
part_headers, content = parts[0]
disposition = _header_params(
part_headers.get("content-disposition", ""))
field_name = disposition.get("name")
filename = disposition.get("filename")
part_type = part_headers.get("content-type")
expected = self.state.get("expected_asset_bytes")
record["attach"] = {
"boundary": boundary,
"field_name": field_name,
"filename": filename,
"part_content_type": part_type,
"query_name":
(urllib.parse.parse_qs(parsed.query).get("name") or [None])[0],
"content": content,
"size": len(content),
"content_matches_expected":
None if expected is None else content == expected,
}
class StubGiteaServer:
"""Bound stub server (v0.1 StubOwuiServer pattern)."""
def __init__(self, state):
self.state = state
self.requests = []
handler = type("BoundStubHandler", (_StubGiteaHandler,), {
"state": state,
"requests": self.requests,
})
self.server = ThreadingHTTPServer(("127.0.0.1", 0), handler)
threading.Thread(target=self.server.serve_forever, daemon=True).start()
@property
def base_url(self):
return "http://127.0.0.1:%d" % self.server.server_address[1]
def close(self):
self.server.shutdown()
self.server.server_close()
class ReleaseUploadTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory(prefix="coreci-release-test-")
self.addCleanup(tmp.cleanup)
self.tmp = tmp.name
self.artifact_path = os.path.join(self.tmp, ARTIFACT_NAME)
# Includes every byte value (CR/LF inside) — byte-exact multipart
# transport must survive content that looks like framing.
self.artifact_bytes = (
b"#!/usr/bin/env python3\nstub-zipapp-payload:" + bytes(range(256)) * 2)
with open(self.artifact_path, "wb") as handle:
handle.write(self.artifact_bytes)
self.env_root = os.path.join(self.tmp, "env-root")
os.makedirs(self.env_root)
secrets_fd = os.open(
os.path.join(self.env_root, ".env.secrets"),
os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(secrets_fd, "w", encoding="utf-8") as handle:
handle.write("GITEA_TOKEN={}\n".format(TOKEN))
self.state = {
"release_exists": True,
"release_id": 970,
"assets": [],
"create_status": 201,
"attach_status": 201,
"expected_asset_bytes": self.artifact_bytes,
}
self.stub = StubGiteaServer(self.state)
self.addCleanup(self.stub.close)
def _argv(self, **overrides):
argv = [
TAG, self.artifact_path,
"--base-url", self.stub.base_url,
"--owner", OWNER, "--repo", REPO_NAME,
"--env-root", self.env_root,
]
for key, value in overrides.items():
argv.extend(["--" + key.replace("_", "-"), value])
return argv
def _run(self, **overrides):
stdout, stderr = io.StringIO(), io.StringIO()
with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr):
code = release.main(self._argv(**overrides))
return code, stdout.getvalue(), stderr.getvalue()
def _paths(self, method):
return [record["path"] for record in self.stub.requests
if record["method"] == method]
def test_existing_release_attaches_exact_multipart_shape(self):
code, out, err = self._run()
self.assertEqual(0, code)
self.assertEqual(
["/api/v1/repos/{}/{}/releases/tags/{}".format(OWNER, REPO_NAME, TAG),
"/api/v1/repos/{}/{}/releases/970/assets".format(OWNER, REPO_NAME)],
self._paths("GET"))
self.assertEqual(
["/api/v1/repos/{}/{}/releases/970/assets?name={}".format(
OWNER, REPO_NAME, ARTIFACT_NAME)],
self._paths("POST"))
self.assertEqual([], self.state.get("create_calls", []))
attach = self.stub.requests[-1]
shape = attach["attach"]
self.assertEqual("attachment", shape["field_name"])
self.assertEqual(ARTIFACT_NAME, shape["filename"])
self.assertEqual(ARTIFACT_NAME, shape["query_name"])
self.assertEqual("application/octet-stream", shape["part_content_type"])
self.assertEqual(self.artifact_bytes, shape["content"])
self.assertTrue(shape["content_matches_expected"])
self.assertIn("multipart/form-data; boundary=",
attach["headers"]["content-type"])
self.assertTrue(self.stub.requests)
for record in self.stub.requests:
self.assertEqual("token " + TOKEN, record["headers"]["authorization"])
def test_missing_release_creates_with_notes_then_attaches(self):
self.state["release_exists"] = False
code, out, err = self._run()
self.assertEqual(0, code)
create_calls = self.state.get("create_calls", [])
self.assertEqual(1, len(create_calls))
create = create_calls[0]
self.assertEqual(
"/api/v1/repos/{}/{}/releases".format(OWNER, REPO_NAME), create["path"])
self.assertEqual("application/json", create["headers"]["content-type"])
payload = json.loads(create["body"])
self.assertEqual(TAG, payload["tag_name"])
self.assertEqual(TAG + ": coreci-broker release", payload["name"])
self.assertIs(False, payload["draft"])
self.assertIs(False, payload["prerelease"])
for section in ("## Summary", "## Requirements", "## Verification",
"## CLI install + usage", "## Docker usage"):
self.assertIn(section, payload["body"])
self.assertEqual(1, len(self.state.get("attach_calls", [])))
def test_matching_asset_is_idempotent_skip(self):
self.state["assets"] = [{
"id": 1, "name": ARTIFACT_NAME, "size": len(self.artifact_bytes),
}]
code, out, err = self._run()
self.assertEqual(0, code)
self.assertEqual([], self._paths("POST"))
self.assertEqual(
["/api/v1/repos/{}/{}/releases/tags/{}".format(OWNER, REPO_NAME, TAG),
"/api/v1/repos/{}/{}/releases/970/assets".format(OWNER, REPO_NAME)],
self._paths("GET"))
self.assertEqual([], self.state.get("attach_calls", []))
self.assertEqual([], self.state.get("create_calls", []))
def test_same_name_but_wrong_size_reattaches(self):
self.state["assets"] = [{"id": 1, "name": ARTIFACT_NAME, "size": 1}]
code, out, err = self._run()
self.assertEqual(0, code)
self.assertEqual(1, len(self.state.get("attach_calls", [])))
def test_attach_500_exhausts_retries_and_marks_pending(self):
self.state["attach_status"] = 500
completed = subprocess.run(
[sys.executable, str(RELEASE_PATH)] + self._argv(),
capture_output=True, text=True, timeout=120,
)
self.assertEqual(1, completed.returncode)
self.assertIn(
"RELEASE_PENDING {} {}".format(TAG, self.artifact_path),
completed.stdout)
self.assertEqual(3, len(self.state.get("attach_calls", [])))
self.assertEqual([], self.state.get("create_calls", []))
self.assertIn("500", completed.stderr)
self.assertNotIn(TOKEN, completed.stdout)
self.assertNotIn(TOKEN, completed.stderr)
def test_permanent_attach_error_reports_gitea_body(self):
self.state["attach_status"] = 400
self.state["attach_error"] = "attachment too large"
code, out, err = self._run()
self.assertEqual(1, code)
self.assertEqual(1, len(self.state.get("attach_calls", []))) # 4xx: no retry
self.assertIn("attachment too large", err)
self.assertIn(
"RELEASE_PENDING {} {}".format(TAG, self.artifact_path), out)
def test_sha256_comes_from_file_and_cli_has_no_sha_argument(self):
before = len(self.stub.requests)
with contextlib.redirect_stderr(io.StringIO()):
with self.assertRaises(SystemExit) as raised:
release.main(self._argv() + ["--sha", "deadbeef"])
self.assertEqual(2, raised.exception.code)
with contextlib.redirect_stderr(io.StringIO()):
with self.assertRaises(SystemExit) as raised:
release.main(self._argv() + ["0deadbeef"])
self.assertEqual(2, raised.exception.code)
self.assertEqual(before, len(self.stub.requests)) # refused pre-HTTP
self.state["release_exists"] = False # force create so notes hit the wire
code, out, err = self._run()
self.assertEqual(0, code)
notes = json.loads(self.state["create_calls"][0]["body"])["body"]
expected = hashlib.sha256(self.artifact_bytes).hexdigest()
self.assertIn("sha256:" + expected, notes)
self.assertNotIn("deadbeef", notes)
def test_token_resolves_from_env_file_on_every_request(self):
code, out, err = self._run()
self.assertEqual(0, code)
self.assertTrue(self.stub.requests)
for record in self.stub.requests:
self.assertEqual("token " + TOKEN, record["headers"]["authorization"])
# Structural discipline: the loader + attacher never touch the
# process environment, and the token value is never printed.
for path in (ENV_SECRETS_PATH, RELEASE_PATH):
source = path.read_text(encoding="utf-8")
self.assertNotIn("os.environ", source)
self.assertNotIn("os.getenv", source)
self.assertNotIn(TOKEN, out)
self.assertNotIn(TOKEN, err)
def test_missing_or_empty_token_fails_before_any_http(self):
empty_root = os.path.join(self.tmp, "empty-env-root")
os.makedirs(empty_root)
code, out, err = self._run(env_root=empty_root)
self.assertEqual(2, code)
self.assertEqual([], self.stub.requests) # no HTTP attempted
self.assertIn("GITEA_TOKEN", err)
self.assertNotIn("RELEASE_PENDING", out)
with open(os.path.join(empty_root, ".env.secrets"), "w",
encoding="utf-8") as handle:
handle.write("GITEA_TOKEN=\n")
code, out, err = self._run(env_root=empty_root)
self.assertEqual(2, code)
self.assertEqual([], self.stub.requests)
self.assertIn("GITEA_TOKEN", err)
def test_notes_sections_sha_line_and_docker_usage(self):
self.state["release_exists"] = False
code, out, err = self._run()
self.assertEqual(0, code)
notes = json.loads(self.state["create_calls"][0]["body"])["body"]
for section in ("## Summary", "## Requirements", "## Verification",
"## CLI install + usage", "## Docker usage"):
self.assertIn(section, notes)
expected = hashlib.sha256(self.artifact_bytes).hexdigest()
self.assertIn("sha256:" + expected, notes)
self.assertIn("chmod +x " + ARTIFACT_NAME, notes)
self.assertIn("--version", notes)
self.assertIn("README", notes)
self.assertIn("BROKER_ENV_PATH", notes)
self.assertIn("BROKER_ADMIN_PASS", notes)
self.assertIn("docker pull ", notes)
self.assertIn("/{}/{}:{}".format(OWNER, REPO_NAME, TAG), notes)
self.assertIn("-v broker-state:/state", notes)
class EnvSecretsLoaderTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory(prefix="coreci-envsecrets-")
self.addCleanup(tmp.cleanup)
self.root = tmp.name
def _write(self, relative, content):
path = os.path.join(self.root, relative)
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w", encoding="utf-8") as handle:
handle.write(content)
return path
def test_parse_semantics_mirror_broker_parser(self):
fixture = (
"# comment\n"
"\n"
"GITEA_TOKEN='quoted-single'\n"
"OTHER_TOKEN=\"quoted double\"\n"
"SPACED = spaced value \n"
"PLAIN=unquoted\n"
"EMPTY=\n"
"ONE_QUOTE=\"\n"
"NO_EQUALS_SIGN\n"
"=\n"
)
path = self._write(".env.secrets", fixture)
self.assertEqual(
coreci_broker._parse_env_file(path),
env_secrets._parse_env_file(path))
values = env_secrets.load(self.root)
self.assertEqual("quoted-single", values["GITEA_TOKEN"])
self.assertEqual("quoted double", values["OTHER_TOKEN"])
self.assertEqual("spaced value", values["SPACED"])
self.assertEqual("unquoted", values["PLAIN"])
self.assertEqual("", values["EMPTY"])
self.assertEqual('"', values["ONE_QUOTE"])
self.assertNotIn("NO_EQUALS_SIGN", values)
def test_later_files_win_across_locations_and_patterns(self):
ladder = (
(".env", "GITEA_TOKEN=step1\n"),
(".env.secrets", "GITEA_TOKEN=step2\n"),
(".env.local", "GITEA_TOKEN=step3\n"), # root .env.* glob
(os.path.join(".ciagent", ".env"), "GITEA_TOKEN=step4\n"),
(os.path.join(".ciagent", ".env.secrets"), "GITEA_TOKEN=step5\n"),
(os.path.join(".ciagent", ".env.staging"), "GITEA_TOKEN=step6\n"),
)
for relative, content in ladder:
self._write(relative, content)
self.assertEqual(
content.partition("=")[2].strip(),
env_secrets.load(self.root)["GITEA_TOKEN"])
def test_missing_or_empty_token_raises_clear_error(self):
with self.assertRaises(env_secrets.SecretsError) as caught:
env_secrets.get_token(root=self.root)
self.assertIn("GITEA_TOKEN", str(caught.exception))
self._write(os.path.join(".ciagent", ".env.secrets"), "GITEA_TOKEN=\n")
with self.assertRaises(env_secrets.SecretsError):
env_secrets.get_token(root=self.root)
self._write(os.path.join(".ciagent", ".env.secrets"),
'GITEA_TOKEN=" "\n') # whitespace-only is empty too
with self.assertRaises(env_secrets.SecretsError):
env_secrets.get_token(root=self.root)
self._write(os.path.join(".ciagent", ".env.secrets"),
"GITEA_TOKEN='tok'\n")
self.assertEqual("tok", env_secrets.get_token(root=self.root))
def test_custom_variable_name(self):
self._write(".env", "OTHER_TOKEN=tok\n")
self.assertEqual("tok", env_secrets.get_token("OTHER_TOKEN", root=self.root))
with self.assertRaises(env_secrets.SecretsError) as caught:
env_secrets.get_token("GITEA_TOKEN", root=self.root)
self.assertIn("GITEA_TOKEN", str(caught.exception))
def test_default_root_is_repo_root(self):
self.assertEqual(
os.path.realpath(str(REPO)), os.path.realpath(env_secrets.default_root()))
if __name__ == "__main__":
unittest.main()