Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 6ecd7129f5 | |||
| 8cbc76125c | |||
| d3251edc07 | |||
| 9b012b7606 | |||
| da15bda6d2 | |||
| e650681755 | |||
| 7f7868f7cf |
@@ -45,4 +45,35 @@ Two runtime modes share the same file: (a) Open WebUI workspace tool — a `Tool
|
||||
|
||||
- The in-memory verified-token cache maps `(uuid, token) -> registration row`; it is populated on first successful slow-hash verification and cleared on restart. SQLite remains authoritative; cache entries are re-validated against row status/expiry on every hit.
|
||||
- Creds adapter contract: lookup by path string (`coreci/<agent_id>`) → JSON object. FileCredsSource reads a single JSON document mapping paths to secret objects (default for dev/tests, `BROKER_CREDS_FILE`). OwuiApiCredsSource calls the Open WebUI HTTP API with `BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY` and is exercised in production; its exact KB-file layout is resolved in P4 research.
|
||||
- This file is the init-time snapshot; it is refined at phase boundaries per ci-files discipline.
|
||||
- This file is the init-time snapshot; it is refined at phase boundaries per ci-files discipline.
|
||||
|
||||
## Distribution (v0.2 — cli-packaging)
|
||||
|
||||
v0.2 adds a distribution layer around the unchanged v0.1 broker: no broker-line changes (`run_standalone`, `boot`, `load_config`, `_parse_env_file` are consumed as-is). Three new components sit outside `coreci_broker.py` and are exercised by the same stdlib-unittest harness (in-process stub servers on ephemeral ports; subprocess boot tests against the built artifact).
|
||||
|
||||
### Components (v0.2)
|
||||
|
||||
| Name | Description | Boundaries | Depends on |
|
||||
|------|-------------|-----------|------------|
|
||||
| CLI artifact build (`scripts/build_cli.py` + `cli/__main__.py`) | Stages `coreci_broker.py` + `coreci_broker_client.py` + `coreci_broker_ui/` + the boot shim + a generated `_cli_version.py`, builds `coreci-broker.pyz` with stdlib `zipapp` (`--compress`, shebang `#!/usr/bin/env python3`), then a deterministic normalization pass (sorted entries, fixed timestamps) for byte-reproducible output; emits `dist/coreci-broker-vX.Y.Z.pyz` + sha256 | Produces the artifact only; never publishes; owns no state; the shim materializes bundled UI/static assets into `<state>/.coreci_broker_assets/<version>/{ui,static}` at boot (temp+`os.replace`, overwrite-per-boot) and sets `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` only when neither process env nor the user `.env` defines them; `--version` is answered from the generated `_cli_version.py` before any broker import | Broker module surface (`run_standalone`, `_parse_env_file`) — import-only, zero edits |
|
||||
| Release-asset pipeline (`scripts/release.py` + `scripts/_env_secrets.py`) | Builds release notes (milestone summary, phases, verification, CLI install/usage, docker usage, artifact sha256), create-or-get the gitea release by tag, attaches `coreci-broker-vX.Y.Z.pyz` via `POST /api/v1/repos/{owner}/{repo}/releases/{id}/assets` (multipart field `attachment`, `Authorization: token <T>`); idempotent re-attach; bounded retry (3) → `release_pending` escalation on exhaustion — never blocks tag+merge | Owns the forge API conversation for release assets only; token resolved exclusively from `.env*` files via the file-reading loader (never shell env; assert non-empty before any call — empty token → gitea 404 masking, v1.8); does not build the artifact, does not push images | CLI artifact build (attaches its output); gitea API (`/api/v1/...`) |
|
||||
| Daemonless registry publish client (`scripts/publish_image.py`) | Once per milestone: pull the pinned `python:3.11-slim` base from Docker Hub (anonymous token exchange → OCI index → resolve linux/amd64 child manifest → config + layer blobs), append one deterministic broker layer carrying `/app/coreci-broker.pyz` (tar+gzip, fixed metadata, `mtime=0`), build a new config blob (diff_id appended; Entrypoint `["python3","/app/coreci-broker.pyz"]`, Cmd `[]`, WorkingDir `/state`), and push `git.coreci.dev/coreci/broker:<milestone-release-tag>` + `latest` to the gitea container registry (Basic-auth token exchange at `/v2/token`, `Bearer` JWT on API calls, HEAD-skip + monolithic blob uploads, OCI image manifest PUT; explicit `Location`-following — stdlib urllib does not follow 307/308 on PUT) | No container runtime (D-019); base layer digests reused verbatim, base blobs re-uploaded (no cross-registry mount); records resolved base digests as provenance; failure follows bounded-retry → `release_pending` escalation — never blocks the milestone merge | CLI artifact build (ships its output in the layer); Docker Hub + gitea registry HTTP APIs (OCI dist-spec v2) |
|
||||
| Dockerfile (documentation) | Self-contained docker-build path for users WITH docker: same base, in-image `zipapp` build of the same sources + shim, same entrypoint/`/state` anchoring | Documentation + alternative build path only; the daemonless client is the executable source of truth for what the pipeline publishes; equivalence with the published image is required, byte-identity is not | CLI artifact build (same staging content, same shim) |
|
||||
|
||||
### DataFlow (v0.2 distribution)
|
||||
|
||||
1. **CLI boot (packaged)**: `./coreci-broker.pyz` → shim resolves the state dir (`BROKER_ENV_PATH`-anchored; CWD-relative defaults preserved) → materializes UI/static assets into the versioned assets dir → sets `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` (user env/`.env` overrides win) → `coreci_broker.run_standalone()` → first-boot `.env` generation, admin pass printed once, `broker.db`/`broker.audit.log` 0600 in the state dir, UI served at `/ui/`, client at `/static/coreci_broker_client.py` — behavior byte-identical to the unpackaged standalone mode.
|
||||
2. **Release attachment (per phase ship from the CLI-building phase)**: `scripts/build_cli.py` → reproducible `dist/coreci-broker-vX.Y.Z.pyz` (+sha256) → `scripts/release.py` → gitea create-or-get release by tag → multipart attach → notes body (CLI install/usage + docker usage + digest). Bounded retry; `release_pending` escalation on exhaustion; tag+merge is the completion gate.
|
||||
3. **Image publish (once per milestone, at the final ship)**: `scripts/publish_image.py` → anonymous Docker Hub pull (`python:3.11-slim` OCI index → amd64 child + blobs) → deterministic broker layer (contains the pyz) + new config → gitea registry auth (`/v2/token`, Basic `<user>:<GITEA_TOKEN>`) → blob upload (HEAD-skip, monolithic POST, explicit Location-following) → manifest PUT `<milestone-release-tag>` then `latest` → provenance digests recorded. Bounded retry; never blocks the milestone merge.
|
||||
|
||||
### BuildOrder (v0.2 — appended to the v0.1 order above)
|
||||
|
||||
8. CLI build pipeline + boot shim + packaging/boot-parity tests (P1 — REQ-21)
|
||||
9. Release-asset uploader + daemonless publish client + Dockerfile + stub tests (P2 — REQ-22/REQ-23)
|
||||
10. Cumulative acceptance (v0.1 suite through the packaged artifact), security review, milestone ship: attach pyz to `v0.2.3`, publish image, merge (P3)
|
||||
|
||||
### Notes (v0.2)
|
||||
|
||||
- Distribution components hold no secrets: the archive bundles only broker + UI + client code; the image ships no state; forge tokens live in `.env*` files loaded only by `scripts/_env_secrets.py` (blocked from shell env by `config.security.bash_allowlist`).
|
||||
- All three pipeline scripts take injectable base URLs (stub-testable); live-forge interactions are read-only in tests and attempted-only at ship per the bounded-retry contract.
|
||||
- Verified environment facts backing this section live in RESEARCH.md v0.2 (gitea 1.26.1 API shapes, OCI index resolution for `python:3.11-slim`, gitea `/v2/token` Basic-auth exchange, zipapp normalization, urllib 307/308 behavior).
|
||||
@@ -1,8 +0,0 @@
|
||||
{
|
||||
"phase": 4,
|
||||
"stage": "verify",
|
||||
"milestone": "v0.1",
|
||||
"phase_role": "execution",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-10-10T03:45:00Z"
|
||||
}
|
||||
+238
-113
@@ -1,129 +1,254 @@
|
||||
# CIAgent Grill Report
|
||||
|
||||
## Run: 2026-10-09 23:10 (mode: interactive, focus: all)
|
||||
## Run: 2026-10-10 (mode: interactive, focus: v0.2 plan red-team — cli-packaging)
|
||||
|
||||
### Verdict: Proceed (confidence: 0.85)
|
||||
**Supersedes the v0.1 grill in this file** (v0.1 run of 2026-10-09; its full text and
|
||||
verdicts G-001..G-008 live in git history at `7f7868f` and remain binding where the plan
|
||||
depends on them: G-005's owner-run post-ship OWUI smoke checklist and G-008's 90-day
|
||||
adoption criterion carry forward in ROADMAP.md §Post-Milestone Notes; G-007's
|
||||
forge-repo-creation test was executed at v0.1's P0 ship and is closed). New verdicts
|
||||
number G-009 onward.
|
||||
|
||||
7 of 9 axes resolved on evidence; the four escalated items (Axes 7, 9) were resolved interactively by the owner on 2026-10-09 — decisions G-005..G-008, zero open escalations.
|
||||
### Verdict (summary): GO-WITH-CONDITIONS (confidence: 0.82)
|
||||
|
||||
### Axis 1 — Business Case
|
||||
- **Q1**: Is the per-tool credential problem real and top-priority?
|
||||
- Evidence: PROJECT.md:9,47 (four live creds via KB/env/0600 JSON today; proxmox tool exists now — no revocation, no audit, no allowlist); ROADMAP.md:5 (owner-directed 2026-10-09)
|
||||
- Answer: real and current — the estate's path has no revocation or audit; owner directed same-day. (0.85)
|
||||
- **Q2**: Named sponsor who has decided under pressure?
|
||||
- Evidence: D-007 owner-confirmed §13 defaults @0.95 (ffdc539); D-012..D-015 forced under clarify pressure (53d99e8)
|
||||
- Answer: CEO-owner is sole sponsor and decision-maker; pressure decisions on the record. (0.85)
|
||||
- **Q3**: What breaks if cancelled — and is ROI evidence-based?
|
||||
- Evidence: PROJECT.md:9 (counterfactual explicit); no financial deck exists; cost is tokens only
|
||||
- Answer: cancellation returns the estate to unaudited per-tool reads (capability loss, not continuity); ROI is operational, measured against the documented counterfactual. (0.80)
|
||||
The plan rests on genuinely strong research — live-verified forge/registry wire shapes, a
|
||||
working zipapp prototype booted against the real broker sources, zero-broker-lines
|
||||
verified against the actual file (1390/1400). Red-team findings are concrete and
|
||||
fixable by task/ship-step edits: one **unexecutable mechanism** (the v0.2.1 attachment
|
||||
bridge cites a harness path RESEARCH itself proved absent from this environment),
|
||||
one **ship-sequence inversion** (P3-05 attaches the release before the tag exists —
|
||||
gitea would mint `v0.2.3` at the wrong commit), one **false-green determinism gap**
|
||||
(the normalize pass "preserves" umask-derived zip attributes, so must-have 4 passes
|
||||
while the "clean checkout" exit criterion silently fails cross-machine), and a set of
|
||||
test-realism holes (subprocess harness with no timeout discipline, missing creds env
|
||||
in the packaged secret-read, no live read-only validation before the one live publish).
|
||||
Three forks were resolved interactively with the owner this session (G-011, G-013,
|
||||
G-014); everything else is bound on evidence below.
|
||||
|
||||
### Axis 2 — Scope
|
||||
- **Q1**: Stable, frozen, owned?
|
||||
- Evidence: REQ-01..REQ-20 frozen from spec v0.5, full traceability (REQUIREMENTS.md:5,78-99); D-008; "No requirement is deferred" (PROJECT.md:27); owner coreci-coo (REQUIREMENTS.md:3)
|
||||
- Answer: frozen and traced; one acknowledged in-scope unknown (KB file layout → P4, ARCHITECTURE.md:47) with candidate flows pre-verified (RESEARCH.md:55). (0.85)
|
||||
- **Q2**: Explicitly out of scope?
|
||||
- Evidence: 12-row table, each row with reason + spec source (REQUIREMENTS.md:57-74)
|
||||
- Answer: exhaustive; covers the usual silent creep targets (rotation, HA, MFA, federation). (0.90)
|
||||
- **Q3**: Hidden requirements (security, ops, accessibility)?
|
||||
- Evidence: security reviews P3-06/P5-02 (PLAN.md:113,145); traversal research (RESEARCH.md:62); D-013 cookie hardening; REQ-15 0600; ops README P5-04
|
||||
- Answer: security/ops surfaced pre-code; accessibility unstated — immaterial for a single-admin internal UI in v0.1. (0.75)
|
||||
---
|
||||
|
||||
### Axis 3 — Architecture
|
||||
- **Q1**: Validated by the builders, not the sellers?
|
||||
- Evidence: RESEARCH.md §4 source-verified against OWUI source at main (no activate() hook, Valves rule, `__file__` temp-delete; KB endpoints from knowledge.py/files.py); plan-checker PASS_WITH_NOTES (b8122be)
|
||||
- Answer: yes — the executing pipeline authored and checked the plan against source. (0.80)
|
||||
- **Q2**: Integration surface bounded?
|
||||
- Evidence: exactly two external surfaces — OWUI Tools mount (RESEARCH.md:30-38) + OWUI KB read API (RESEARCH.md:40-55); gitea push per sibling convention (RESEARCH.md §3)
|
||||
- Answer: bounded and source-verified; no third integration exists. (0.80)
|
||||
- **Q3**: Replaced-system edge cases; inherited debt?
|
||||
- Evidence: four creds, single host, no volume concern (PROJECT.md:47); spec defects 1-4 found and resolved pre-code (RESEARCH.md §2; D-004/D-005); greenfield repo (acf5557)
|
||||
- Answer: edge cases (pepper chicken-and-egg, KB-loss, re-reg KeyError) resolved by decision; external debt = OWUI lifecycle quirks, mitigated by the once-per-process guard (ARCHITECTURE.md:7); 800-line budget fails fast via AC14 at P1-08. (0.75)
|
||||
## Surface 1 — Determinism claims (must-have 4; P1-01)
|
||||
|
||||
### Axis 4 — People
|
||||
- **Q1**: Key-person dependency — acceptable?
|
||||
- Evidence: D-007 CEO sole approver, binding (PROJECT.md:61); tokens re-register on TTL; allowlist changes are the only CEO-gated act
|
||||
- Answer: accepted — CEO absence stalls new onboarding only, not broker continuity; by design for a v0.1 credential broker on a trusted single host. (0.75)
|
||||
- **Q2**: Resources allocated; PO with authority?
|
||||
- Evidence: agent pipeline, sequential, parallelization off (config.json:33-37); guided autonomy, escalation_hooks ["*"] (config.json:5-9)
|
||||
- Answer: no BAU contention applies; owner-CEO is PO with absolute authority. (0.80)
|
||||
- **Q3**: Building capability they don't have?
|
||||
- Evidence: sibling precedent on same pipeline + forge (RESEARCH.md §3); the one novel surface (OWUI mount) is source-researched
|
||||
- Answer: capability precedented; unproven only in that zero code exists — P1 is the cheap falsifier. (0.70)
|
||||
**Forcing question**: Is two-build sha256 equality actually achievable and testable, and
|
||||
what does the normalize pass fix vs. miss?
|
||||
|
||||
### Axis 5 — Timeline
|
||||
- **Q1**: Deadline set before or after scope?
|
||||
- Evidence: spec v0.5 (ffdc539, 22:37) precedes plan (b8122be, 23:02); no external deadline exists anywhere in the corpus
|
||||
- Answer: spec-first; no reverse-engineered date to fail against. (0.85)
|
||||
- **Q2**: Critical path; single 3-month pusher?
|
||||
- Evidence: P1→P5 sequential (ROADMAP.md:24-57); only live-environment unknown is OwuiApiCredsSource KB layout (P4), bounded by FileCredsSource default + CEO-controlled KB structure
|
||||
- Answer: no 3-month pusher; worst case is bounded P4 adapter rework. (0.75)
|
||||
- **Q3**: Evidence-based estimates; definition of done?
|
||||
- Evidence: 30 one-commit tasks, tests ship with task (PLAN.md:12); must_haves frontmatter (PLAN.md:2-7); AC1-AC14 + per-phase exits (ROADMAP)
|
||||
- Answer: slice-sized with checker sign-off; done is test-enforced, not demo-defined. (0.80)
|
||||
- **Evidence** (stdlib source, read this session): `/usr/lib/python3.11/zipapp.py`
|
||||
builds via `for child in source.rglob('*'): z.write(child, arcname.as_posix())` —
|
||||
entry order = scandir order, entry mtime = source mtime (`ZipInfo.from_file`), and
|
||||
`external_attr` = the staging file's stat mode `<< 16`. PLAN P1-01's normalize pass
|
||||
fixes order (sorted), `date_time=(1980,1,1,0,0,0)`, `ZIP_DEFLATED` — but specifies
|
||||
"**preserved** `external_attr`". Preserved from what zipapp wrote, i.e. derived from
|
||||
staging file modes — which are `0666 & ~umask` if staged via plain copy, or
|
||||
checkout-mode if via `copy2`. Umask 0022 vs 0002 across machines/clones → different
|
||||
`external_attr` → different sha256, **while must-have 4 (two consecutive builds, same
|
||||
machine) still passes**. A silent false green against the phase exit criterion
|
||||
("builds reproducibly from a clean checkout").
|
||||
- **gzip is not in the pyz path** (zipfile DEFLATE only). Image-layer gzip is
|
||||
deterministic on-platform: `gzip.compress(..., mtime=0)` routes to
|
||||
`zlib.compress(wbits=31)` on 3.11 (verified in gzip.py source) — fixed header, fixed
|
||||
OS byte, stable per zlib build. Scope claim: byte-identity is **same-environment**
|
||||
(same machine, same zlib); cross-zlib-version identity is neither claimed nor
|
||||
required by must-have 4.
|
||||
- **Answer**: achievable and testable as scoped — provided the normalize pass also
|
||||
fixes what it currently preserves. → **G-009**.
|
||||
|
||||
### Axis 6 — Budget
|
||||
- **Q1**: Cost profile and burn?
|
||||
- Evidence: token/compute only; phase 0 complete in ~30 min wall-clock (5 commits); no cash budget exists
|
||||
- Answer: no runway question; pipeline runs on owned infra. (0.75)
|
||||
- **Q2**: Predictable drivers outside the budget?
|
||||
- Evidence: stdlib-only eliminates licensing (D-009); forge org exists (RESEARCH.md:9); OWUI self-hosted; no pip, no npm
|
||||
- Answer: none uncovered — the design removes the usual cost drivers by construction. (0.80)
|
||||
- **Q3**: Contingent on something unhappened?
|
||||
- Evidence: forge repo creation untested (assumption d), but release_blocking false (config.json:24) and manual forge-UI fallback is trivial
|
||||
- Answer: no financial contingency; the sole contingency is operational (→ Axis 7). (0.70)
|
||||
## Surface 2 — Boot parity via subprocess (P1-02)
|
||||
|
||||
### Axis 7 — Risks
|
||||
- **Q1**: Top assumptions, evidence per item?
|
||||
- (a) bcrypt ships in OWUI runtime + daemon-thread tools allowed: RESEARCH.md §4 source+docs — but validated at `main`, deployed version unpinned; absorbed by dual-mode scrypt fallback (0.70)
|
||||
- (b) stdlib dev parity with the OWUI runtime: D-001 dual-mode covers both environments (RESEARCH.md:8) (0.80)
|
||||
- (c) FileCredsSource makes tests hermetic: tempdir + env-var state (RESEARCH.md §6; PLAN P1-01/P4-01) — plan-level, proven at P1 (0.70)
|
||||
- (d) forge repo creatable with current token: token valid, org exists, repo 404 — creation UNTESTED (RESEARCH.md:9). **→ ESCALATION**
|
||||
- **Q2**: Dependencies outside the team?
|
||||
- Evidence: OWUI (host + API + process perms), gitea — all in daily estate use (PROJECT.md:47; RESEARCH.md §3)
|
||||
- Answer: bounded and precedented. (0.75)
|
||||
- **Q3**: Single killer risk; pre-mortem?
|
||||
- Evidence: worst case = mount or adapter fails on the live host → standalone mode is functionally equivalent (ARCHITECTURE.md:7); defect table (RESEARCH.md §2) is the pre-mortem substance
|
||||
- Answer: no project-killing risk found on evidence — but NO plan task validates against the live OWUI host before ship. **→ ESCALATION** (0.65)
|
||||
**Forcing question**: What makes the subprocess acceptance flaky, and what's missing?
|
||||
|
||||
### Axis 8 — Governance
|
||||
- **Q1**: Decision-maker under disagreement?
|
||||
- Evidence: single owner-CEO; guided autonomy escalates every gate (ROADMAP.md:9; config.json:6)
|
||||
- Answer: no committee exists to deadlock; owner is arbiter. (0.85)
|
||||
- **Q2**: Cadence, escalation, omissions from status?
|
||||
- Evidence: ---ci--- blocks with confidence on all four commits (ffdc539..b8122be); 15 decisions recorded; checker notes published, not buried
|
||||
- Answer: decision log is exemplary; the mild contradiction is RESEARCH.md:82 "no genuine blockers" sitting beside untested assumption (d). (0.70)
|
||||
- **Q3**: Stop-the-project trigger?
|
||||
- Evidence: MISSING as a formal trigger — but sponsor = approver = decision-maker is one person
|
||||
- Answer: implicit owner prerogative suffices where one person holds all three roles. (0.70)
|
||||
- **Evidence**: zero v0.1 tests use `subprocess`/`Popen` (grep over `tests/`: no hits) —
|
||||
P1-02 introduces the repo's first subprocess-boot harness. The plan's env enumeration
|
||||
(`BROKER_BIND`, `BROKER_ENV_PATH/DB/AUDIT`) omits `BROKER_CREDS_FILE`/
|
||||
`BROKER_CREDS_SOURCE` while requiring "secret-read reuse the v0.1 acceptance request
|
||||
patterns" — the v0.1 precedent (`tests/test_acceptance_full.py:76-77`) sets both;
|
||||
`FileCredsSource` on a missing file raises `_SecretNotFound` → 404
|
||||
(`coreci_broker.py:440-441`), so the packaged secret-read would fail. `Popen(["python3",
|
||||
...], env={minimal})` leaves the child without `PATH` (works today via execvp's
|
||||
confstr fallback — by accident). No timeout discipline on "read stdout lines until
|
||||
`coreci-broker listening on`" (RESEARCH §4.3) — a failed boot never prints the line and
|
||||
hangs the suite. "Pass printed exactly once" is asserted against a *still-running*
|
||||
server; shutdown-path output is unread unless drained after terminate. And every
|
||||
planned boot pins absolute state paths — the owner's literal happy path (download →
|
||||
`python3 *.pyz` in some directory → `.env`/db/audit materialize in the CWD) is never
|
||||
tested.
|
||||
- **Answer**: harden the harness — the plan's claims are all testable once the harness
|
||||
is honest about them. → **G-010**.
|
||||
|
||||
### Axis 9 — Adoption
|
||||
- **Q1**: Who uses it; what's in it for them?
|
||||
- Evidence: CEO gets allowlist + audit UI replacing blind KB edits; tool authors get a zero-dep client replacing hand-rolled 0600 JSON (REQ-19; PROJECT.md:9)
|
||||
- Answer: both user classes shed concrete work — not a "train them and pray" case. (0.75)
|
||||
- **Q2**: Ops handoff; rollback plan?
|
||||
- Evidence: ops = owner; README + backup guidance P5-04 (PLAN.md:150); full state = broker.db + broker.audit.log + .env (PROJECT.md:37-39); unmigrated tools never depend on the broker
|
||||
- Answer: rollback = stop process, restore three files; migrated proxmox tool reverts to direct reads. (0.75)
|
||||
- **Q3**: Success criteria validated with the judges?
|
||||
- Evidence: judge = owner via AC1-AC14 (spec §12, owner-owned); but every OWUI-facing test is stub-driven (PLAN P3-04, P4-01), no task touches the live host, and proxmox migration is "post-broker", dateless (PROJECT.md:47)
|
||||
- Answer: acceptance is owner-validated; live readiness and the adoption criterion were escalated and are now resolved as G-005/G-008. (0.85)
|
||||
## Surface 3 — One-shot lead upload bridge (P1-03)
|
||||
|
||||
### Meta
|
||||
- **Auditor's flag**: no live-host validation before ship; forge repo creation untested; deployed OWUI version unpinned; adoption criterion undefined. All cheap to fix; none fatal.
|
||||
- **Not being done**: live smoke test on the OWUI host; OWUI version pin; a risk register; post-ship adoption criteria.
|
||||
- **Simplest 80% version**: drop the admin UI + reference tools (CLI allowlists, file creds only) saves ~one phase — but the UI is the CEO's core value and OwuiApiCredsSource is the production path; at 30 tasks / 800 lines, the over-scope charge does not stick.
|
||||
- **90-day must-be-true**: broker live on the OWUI host; proxmox tool on the client module; allowlist + audit in CEO daily use. Not true today (no code) — but every gating factor is internal.
|
||||
**Forcing question**: Is the v0.2.1 ad-hoc attachment a discipline violation, and is it
|
||||
even executable as specified?
|
||||
|
||||
### Binding Decisions
|
||||
| ID | Axis | Decision | Confidence |
|
||||
|----|------|----------|------------|
|
||||
| G-001 | 4 | Key-person dependency accepted for v0.1: sole-approver CEO absence stalls new onboarding only, not continuity (D-007 by design) | 0.75 |
|
||||
| G-002 | 7 | Assumptions (a)-(c) accepted as mitigated-by-design (dual-mode hashing, hermetic test plan); (d) + live validation escalate | 0.70 |
|
||||
| G-003 | 9 | Rollback accepted: full state = broker.db + broker.audit.log + .env; stop-and-restore; tools revert to direct reads | 0.75 |
|
||||
| G-004 | 3 | 800-line budget risk accepted: AC14 enforced from P1-08 (fails fast); spec §12.14 amendment is the owner's pressure valve | 0.70 |
|
||||
| G-005 | 9 | Post-ship live-OWUI smoke checklist (owner-run: mount, register, read) is the plan of record; v0.1 ships on standalone + stub tests; no real tool migrates before the checklist passes; rollback = disable the tool | 0.85 |
|
||||
| G-006 | 7 | Deployed OWUI pinned at v0.11.3 (owner-stated 2026-10-09) — same-era as the researched `main` surface, no re-research required; smoke checklist item 1 re-confirms the two KB read endpoints on v0.11.3 | 0.85 |
|
||||
| G-007 | 7 | Forge repo creation is tested at the P0 ship (POST /orgs/coreci/repos with the existing token), not deferred to P5-06; on scope failure: local-only fallback + auto-escalation, and the owner's 30-second manual forge-UI creation is the recorded fallback | 0.9 |
|
||||
| G-008 | 9 | 90-day adoption criterion: proxmox_readonly_tool.py runs on coreci_broker_client.py with direct KB reads removed by 2027-01-09; owner owns the date; recorded in ROADMAP as a post-milestone note (out of v0.1 scope) | 0.8 |
|
||||
- **Evidence**: PLAN P1-03 attaches via "the same harness mechanism v0.1 used for its
|
||||
`broker-vX.Y.Z.tar.gz` assets" — but RESEARCH §2.2 (the plan's own research base)
|
||||
found that mechanism is the ship harness's `coreci build/package/release upload` CLI,
|
||||
**"not installed in this dev environment"**; v0.1's archives were produced
|
||||
container-side, ad-hoc, with no repo-owned script. The bridge as written is therefore
|
||||
unexecutable via the cited mechanism, and its realistic substitute is an untested
|
||||
hand-rolled live-forge write (multipart encoding, token handling, idempotency) at
|
||||
first live use — performed by lead-developer, the one persona barred from code
|
||||
writes, using token discipline that `scripts/_env_secrets.py` (which does not exist
|
||||
until P2) is supposed to enforce. This is precisely the practice REQ-22 formalizes
|
||||
away.
|
||||
- **Owner decision (this session)**: restructure — release automation moves into P1;
|
||||
the bridge is deleted. → **G-011**. The hard gate question dissolves: v0.2.1 ships
|
||||
script-driven; there is no bridge to become permanent.
|
||||
|
||||
### Escalations
|
||||
- (none — all four escalated challenges resolved interactively by the owner on 2026-10-09; decisions G-005..G-008 above)
|
||||
## Surface 4 — Daemonless OCI publish (P2-02, P3-05)
|
||||
|
||||
**Forcing question**: Does the milestone ship sequence hold up when publish fails after
|
||||
the tag exists — and does it even run in the right order?
|
||||
|
||||
- **Evidence — sequence inversion**: P3-05 as written runs `release.py` attach →
|
||||
publish image → squash-merge → `main` → tag `v0.2.3` → push. But `POST /api/v1/repos/
|
||||
{o}/{r}/releases` with a missing tag creates the tag at `target_commitish`
|
||||
(default-branch head) — gitea's documented CreateReleaseOption semantics (schema in
|
||||
RESEARCH §2.1). A create-first flow pre-merge would mint `v0.2.3` pinned to
|
||||
**pre-merge main**, then collide with (or silently diverge from) the harness's own tag
|
||||
push. The plan's own P2-04 sequences correctly (tag → merge → push → release) — P3-05
|
||||
alone inverts it.
|
||||
- **Evidence — wire discipline**: bounded retry(3) is specified but no per-request
|
||||
timeout anywhere; a hung 29.8 MB monolithic POST through a throttled gitea turns
|
||||
"bounded retry" into an unbounded hang. The 307/308 stub test proves only
|
||||
Location-following, and gitea's `Location` may be **relative** (RESEARCH §3.3) — the
|
||||
harder case is untested. Upstream `3.11-slim` drift is handled (resolve-at-publish +
|
||||
`--base-manifest` + provenance digests; RESEARCH §3.7) and the anonymous-200 token
|
||||
trap fails loudly per spec.
|
||||
- **Evidence — failure story**: "publish never blocks merge" is honored, but when
|
||||
publish fails after `v0.2.3` exists, the plan records only `release_pending` — no
|
||||
named post-milestone re-run/closure step. The ship contract makes the escalation
|
||||
auto-resolution; somebody must own the retry.
|
||||
- **Answer**: reorder P3-05, add timeouts + relative-Location coverage, and make the
|
||||
escalation carry its own closure runbook. → **G-012**, **G-019**, **G-020**.
|
||||
|
||||
## Surface 5 — Scope discipline (P2-03, REQ-22/23)
|
||||
|
||||
**Forcing question**: Is anything decorative, and is anything REQ-22/23 implies missing?
|
||||
|
||||
- **Evidence — decorative risk**: P2-03's verify is presence-only ("`Dockerfile` +
|
||||
README sections present") and the dev environment cannot build the Dockerfile (no
|
||||
runtime) — a syntactically or semantically broken Dockerfile passes the gate. The
|
||||
"equivalence mapping" header is documentation with no enforcement.
|
||||
- **Evidence — missing**: no `.dockerignore` anywhere in the plan — `docker build .`
|
||||
ships the whole repo (`.git/`, any local `.env*`, `data/`, `*.db`, `dist/`,
|
||||
`__pycache__/`, `.ciagent/`) as build context, adjacent to a security review that
|
||||
asserts "no secrets in any image layer". REQ-22's notes model is covered (§2.3, stub
|
||||
asserts sections present); the sha256's *source* is unspecified (hand-pasted vs.
|
||||
computed); `latest` semantics are coherent (once per milestone at final ship —
|
||||
v0.2.3 is the first image ever, D-021); the state-dir volume contract is documented
|
||||
(`WORKDIR /state`, README docker run).
|
||||
- **Answer**: make the Dockerfile contract testable, add `.dockerignore`, pin digest
|
||||
provenance. → **G-015**, **G-016**, **G-017**.
|
||||
|
||||
## Surface 6 — Frozen broker / shim dependency risk (P1-01)
|
||||
|
||||
**Forcing question**: Does the shim need broker cooperation, and what is the coupling
|
||||
risk?
|
||||
|
||||
- **Evidence**: RESEARCH §1.2's prototype booted from the real sources with zero broker
|
||||
cooperation — re-verified against the file this session: `run_standalone`
|
||||
(`coreci_broker.py:1369`), `_parse_env_file` (:154), `boot` (:598), `load_config`
|
||||
(:240) are module-level; `load_config` precedence (process env → `.env` → defaults,
|
||||
:246-257) makes the shim's pre-set `os.environ` approach contract-identical. The
|
||||
broker's `.env` resolution is CWD-relative by default (`ENV_PATH_DEFAULT=".env"`,
|
||||
`ENV_DEFAULTS` :53-66) — packaged boots with unset `BROKER_ENV_PATH` anchor state to
|
||||
the CWD, which is exactly the documented happy path and is currently untested (folded
|
||||
into G-010). Residual risk: the shim calls `_parse_env_file` — an underscore-private
|
||||
symbol. Within v0.2 the broker is frozen so the risk is nil; across future milestones
|
||||
a rename breaks the shim with a loud ImportError (acceptable), but only if somebody
|
||||
knows to look.
|
||||
- **Answer**: no cooperation needed (verified twice); the private-API dependency must
|
||||
be documented and review-tracked, and the default-anchoring boot must be tested. →
|
||||
**G-018** (anchor case bound under G-010).
|
||||
|
||||
## Surface 7 — Test realism: stub fidelity (P2-01/P2-02 tests)
|
||||
|
||||
**Forcing question**: What would a stub MISS that the real forge does — and is the
|
||||
as-built client ever validated against the real forge before its one live write?
|
||||
|
||||
- **Evidence**: the stub coverage is unusually good (anonymous-200 trap, index→amd64
|
||||
resolution with Accept check, HEAD-skip, monolithic+two-step uploads, 307-following,
|
||||
digest and diff_id assertions, both tags; multipart field/`?name=` byte assertions,
|
||||
idempotent re-attach, 500×3 marker, token discipline). Gaps: fixtures are frozen at
|
||||
research-time shapes (2026-10-10) while upstream index and forge auth can move
|
||||
mid-milestone; the 307 test doesn't pin the relative-`Location` case (bound in
|
||||
G-019); gitea's 4 MB attachment cap is out-of-band (pyz ≈30 KB — immaterial, noted);
|
||||
and **no step anywhere validates the as-built scripts against the real
|
||||
`git.coreci.dev` in read-only mode** — the first live run of `publish_image.py` is
|
||||
the P3-05 milestone publish itself, the one shot per milestone.
|
||||
- **Owner decision (this session)**: add an env-gated, read-only live smoke as a
|
||||
P3-05 pre-ship verify step. → **G-013**.
|
||||
|
||||
## Surface 8 — Wave ordering & effort (P2 packing)
|
||||
|
||||
**Forcing question**: Is P2 overloaded, and are there hidden dependencies that break
|
||||
the wave order?
|
||||
|
||||
- **Evidence**: original P2 packs `_env_secrets` + `release.py` + `publish_image.py` +
|
||||
`Dockerfile` + README + 2 test files into 3 waves — the overload is real, and the
|
||||
bridge finding (Surface 3) is the reason it dissolves: release automation belongs in
|
||||
P1 (G-011), leaving P2 = publish client → Dockerfile/docs → ship. Hidden
|
||||
dependencies check out: `publish_image.py` imports `_env_secrets` (lands earlier in
|
||||
P1 now — order preserved transitively); the Dockerfile task consumes P1-01's staging
|
||||
allowlist (G-016 formalizes the contract as a test). `publish_image.py` remains the
|
||||
plan's biggest single commit even after the restructure.
|
||||
- **Owner decision (this session)**: keep it one vertical task — the pull/push seam is
|
||||
artificial, the stub fixtures are shared end-to-end, RESEARCH §3.4 is near-pseudocode,
|
||||
and a split would not isolate the actual risk (first live write — covered by G-013).
|
||||
→ **G-014**.
|
||||
|
||||
---
|
||||
|
||||
## Binding Decisions
|
||||
|
||||
| ID | Surface | Decision | Binds | Confidence |
|
||||
|----|---------|----------|-------|------------|
|
||||
| G-009 | 1 | Normalize pass must SET constant `external_attr` (files `0o100644<<16`, dir entries `0o40755<<16`), not "preserve" staging-derived modes; staging is an explicit allowlist (`coreci_broker.py`, `coreci_broker_client.py`, `coreci_broker_ui/**`, shim staged as root `__main__.py`, generated `_cli_version.py`) — never a repo rglob, `__pycache__`/`dist/` excluded by construction. Byte-identity scope = same environment (same zlib/machine); cross-zlib identity is not claimed. | P1-01 task edit | 0.85 |
|
||||
| G-010 | 2 | P1-02 harness hardening: subprocess boots use `sys.executable` (absolute); child env adds `BROKER_CREDS_SOURCE=file` + `BROKER_CREDS_FILE=<tmp fixture>` (v0.1 precedent test_acceptance_full.py:76-77); deadline-bounded wait for the listening line (~30s → fail with captured stdout+stderr, terminate child); pass-once/silent-second-boot assertions completed by draining stdout after terminate; ADD a default-anchoring boot case (`cwd=<tmp>`, no state overrides beyond `BROKER_BIND`) asserting `.env`/`broker.db`/`broker.audit.log` in the CWD — the owner's literal happy path. | P1-02 task edit | 0.85 |
|
||||
| G-011 | 3 | **Restructure (owner, this session)**: move `scripts/_env_secrets.py` + `scripts/release.py` + `tests/test_release_upload.py` into Phase 1 as new wave 3 (P1-03 · release-asset automation); old P1-03 ship renumbers P1-04 and ships v0.2.1 script-driven (`build_cli.py v0.2.1` → `release.py v0.2.1 <pyz>`); the one-shot lead-upload note is DELETED (its cited mechanism is absent from this environment — RESEARCH §2.2). P2 renumbers: publish_image.py → P2-01 (wave 1), Dockerfile+docs → P2-02 (wave 2), ship → P2-03 (wave 3). Four-test-file count unchanged. | PLAN edit (before P1-01 executes) | 0.90 |
|
||||
| G-012 | 4 | P3-05 sequence corrected: squash-merge → `main` → tag `v0.2.3` → push → THEN `release.py` attach (GET-by-tag finds the real tag; no pre-merge release create) → publish image → closeout. Mirrors P2's own correct ship order; eliminates the gitea create-tag-at-target_commitish hazard. | P3-05 ship-step edit | 0.80 |
|
||||
| G-013 | 7 | **Env-gated live read-only smoke (owner, this session)**: pre-publish P3-05 verify step — gitea `GET /v2/` 401 challenge shape, `GET .../releases/tags/{tag}`, `GET /v2/token` Basic → non-empty JWT, Docker Hub anonymous token + `3.11-slim` index→amd64 resolve. Implemented as a `--check`/verify mode on the scripts or `scripts/verify_endpoints.py`; GET-only, never in the default unittest run; tokens only via `_env_secrets`. | P3-05 verify + task edit | 0.85 |
|
||||
| G-014 | 8 | **publish_image.py stays one vertical task (owner, this session)**: shared stub fixtures, RESEARCH §3.4 near-pseudocode; a split would not isolate the real risk (first live write — G-013 covers it). | PLAN shape (no edit) | 0.75 |
|
||||
| G-015 | 5 | `.dockerignore` added to the Dockerfile task: excludes `.git`, `.env*`, `data/`, `*.db`, `*.log`, `dist/`, `__pycache__`, `tests/`, `.ciagent/`. Build-context hygiene; P3-02 scope gains "no secrets in the build context" alongside "none in any layer". | P2-02 task edit | 0.85 |
|
||||
| G-016 | 5 | Dockerfile equivalence made testable: the Dockerfile task adds a consistency test (into `tests/test_build_cli.py`) asserting COPY source set == `build_cli.py` staging allowlist, identical zipapp flags (`--python "/usr/bin/env python3" --compress`), identical `_cli_version.py` generation format, `ENTRYPOINT`/`WORKDIR` == publish_image's `config.Entrypoint`/`config.WorkingDir`. Presence-only verification no longer gates a possibly-broken Dockerfile. | P2-02 task edit | 0.80 |
|
||||
| G-017 | 5 | Digest discipline: the sha256 in release notes is computed by `release.py` from the artifact file itself — never a CLI argument, never hand-pasted. Digest/asset mismatch becomes structurally impossible. | P1-03 task edit | 0.85 |
|
||||
| G-018 | 6 | Shim private-API dependency tracked: `cli/__main__.py` header documents the coupling to `coreci_broker._parse_env_file` + `run_standalone` ("private-API dependency — any future broker refactor must re-run the v0.2 packaged acceptance"); P3-02 review scope lists it. Loud-ImportError breakage accepted. | P1-01 task edit + P3-02 scope | 0.75 |
|
||||
| G-019 | 4 | Publish-client wire hardening: explicit per-request timeouts on every registry call (generous on blob uploads, ~120s); blob-level bounded retry with HEAD-skip between attempts (a timed-out 29.8 MB POST resumes cheap); the 307 stub test uses a RELATIVE `Location` at least once (gitea may answer relative — RESEARCH §3.3; absolute is the easy case). | P2-01 task edit | 0.80 |
|
||||
| G-020 | 4 | Publish-failure closure is a named step, not an open escalation: the `release_pending` entry records the exact re-run command (`python3 scripts/publish_image.py` on `main` @ `v0.2.3`; `--base-manifest` optional pin) and its close condition (both tags visible in gitea packages). "Never blocks merge" unchanged. | P3-05 ship-step edit | 0.80 |
|
||||
| G-021 | 4 | The `v0.2.0` (phase-0) release carries NO `.pyz` and its notes must not promise one — D-020 starts at `v0.2.1` (build tooling first exists in P1). Prevents a D-020 misread at the phase-0 ship immediately following this grill. | Phase-0 ship note | 0.85 |
|
||||
|
||||
## Escalations
|
||||
|
||||
- (none open — the three material forks were resolved interactively with the owner this
|
||||
session: G-011 restructure, G-013 live smoke, G-014 task packing. G-001..G-008 from
|
||||
v0.1 remain binding per git history; G-005/G-008 carry forward in ROADMAP post-milestone
|
||||
notes unchanged.)
|
||||
|
||||
## Promotion note (explicit, not performed by the grill)
|
||||
|
||||
G-011 shifts REQ-22's attach automation into Phase 1; ROADMAP.md's Phase 2 description
|
||||
("release-asset automation" wording) and Phase 1's requirements line need a one-line
|
||||
touch-up when the PLAN restructure is promoted — by the orchestrator/owner via the
|
||||
normal plan-revision path, not by this file. Nothing else in PROJECT/REQUIREMENTS/ROADMAP
|
||||
is affected by these verdicts.
|
||||
|
||||
## Overall Verdict
|
||||
|
||||
**GO-WITH-CONDITIONS** — confidence 0.82.
|
||||
|
||||
The plan is research-grounded to an unusual degree (live wire shapes, working prototype,
|
||||
zero-broker-lines verified against the file), and every finding is actionable by task or
|
||||
ship-step edit. It is GO only with all of the following applied:
|
||||
|
||||
1. **G-011 restructure applied to PLAN.md before P1-01 executes** — the bridge is
|
||||
unexecutable as written; v0.2.1 must ship script-driven.
|
||||
2. **G-009 + G-010 applied to P1-01/P1-02** — determinism must fix (not preserve)
|
||||
entry attributes; the subprocess harness needs interpreter-path, creds-env, timeout,
|
||||
drain, and default-anchoring discipline.
|
||||
3. **G-012 + G-020 applied before P3-05** — attach-after-tag ordering and the
|
||||
publish-failure closure runbook are ship-contract corrections, not optional polish.
|
||||
4. **G-013 live smoke wired as the P3-05 pre-ship verify step** — the one live publish
|
||||
must not also be the first live wire contact of the as-built client.
|
||||
5. **G-015/016/017/018/019 applied with their tasks** — context hygiene, testable
|
||||
Dockerfile contract, digest provenance, tracked private-API coupling, wire timeouts.
|
||||
6. **G-021 noted at the phase-0 ship** — v0.2.0 carries no `.pyz`.
|
||||
+42
-27
@@ -1,39 +1,28 @@
|
||||
# PERSONAS — CoreCI Broker
|
||||
|
||||
RESEARCH-stage roster (2026-10-09, milestone v0.1). Pure stdlib Python 3.11 project: single-file broker, no package.json, no JS framework, no src/ tree; the only UI is server-rendered static HTML/CSS/JS from `coreci_broker_ui/` (P2). Execution is sequential (parallelization off), so phase-specific personas activate only at their phase boundaries per this roster.
|
||||
Execution-stage roster (2026-10-10, milestone v0.2 cli-packaging). Pure stdlib Python 3.11 project: no package.json, no JS framework, no pip, no npm, no container runtime in the dev environment — frameworks are stdlib modules; tests are stdlib unittest against in-process stub HTTP servers and subprocess boots. The v0.1 broker is FROZEN this milestone: `coreci_broker.py` stands at 1390/1400 lines (REQ-17/AC14) and v0.2 adds zero broker lines (verified, RESEARCH §4.2), so no persona holds `coreci_broker.py` — or the v0.1 UI/client/reference sources — as writable territory; a P0 defect there is a lead-developer-dispatched exception only. All v0.2 work is new files: `cli/` boot shim, `scripts/` build+release+publish tooling, `Dockerfile`, four new test files, additive README sections, `.gitignore` additions. The custom release-engineer persona owns the whole distribution pipeline; backend-engineer — v0.1's primary persona — is deactivated with the P0-dispatch exception documented below. Execution is sequential (parallelization off), so phase-specific personas activate only at their phase boundaries per this roster.
|
||||
|
||||
---
|
||||
name: backend-engineer
|
||||
domain: backend
|
||||
name: release-engineer
|
||||
domain: distribution
|
||||
active: true
|
||||
reason: Primary execution persona — the entire v0.1 product is Python on one file: P1 core skeleton, P2 admin-plane API, P3 tool auth plane, P4 secrets adapter + public client module + reference tools. Owns every security-critical code path (token derivation, slow-hash verification, allowlist gates, audit trail, path-scoped reads) and every test that gates a phase.
|
||||
frameworks: [python3.11, http.server, sqlite3, hashlib.scrypt, unittest]
|
||||
constraints: ["coreci_broker.py stays one file under 800 lines (REQ-17, AC14)", "stdlib-only runtime; bcrypt is the sole optional import; no pip, no npm (D-001/D-009)", "all errors use the {error, message} JSON contract on 400/401/403/404/409/500", "broker.db and broker.audit.log are created mode 0600 (REQ-15)", "the broker never writes to creds-coreci-coo (REQ-08)"]
|
||||
territory: [coreci_broker.py, tests/**, coreci_broker_client.py, reference_tools/**]
|
||||
phase_specific: false
|
||||
reason: Primary execution persona — v0.2 is distribution engineering on a frozen broker, a domain distinct from v0.1's broker application work; hence a custom persona rather than a fold into backend-engineer, whose identity, constraints, and territory are the broker's runtime contract — a poor fit for build/release/publish tooling and a structural guarantee that the frozen file is owned by nobody. Owns the full pipeline — P1 zipapp build + materialization boot shim (REQ-21), P2 gitea release-asset automation + daemonless OCI registry publish + Dockerfile (REQ-22/23), P3 ship — and every security-critical surface this milestone: secrets discipline (file-only token loader), artifact integrity (no secrets in the .pyz or image layers, reproducible digests), registry auth flows.
|
||||
frameworks: [python3.11, zipapp, zipfile, tarfile, gzip, hashlib.sha256, urllib.request, json, subprocess, http.server, unittest]
|
||||
constraints: ["coreci_broker.py is FROZEN — zero lines added (1390/1400, REQ-17/AC14); broker sources (coreci_broker.py, coreci_broker_client.py, coreci_broker_ui/**, reference_tools/**) are read-only staging inputs to the build, never edited; a P0 defect there routes through lead-developer dispatch", "stdlib-only build path — python3 -m zipapp --compress; no pip, no npm, no wheel/PyInstaller, no container runtime (D-009/D-019)", "secrets discipline — GITEA_TOKEN resolves ONLY via scripts/_env_secrets.py reading .env/.env.secrets/.env.* files, never os.environ (blocked_env_vars: GITEA_TOKEN); assert non-empty before the first API call (v1.8 lesson: empty token → gitea 404 masks auth failure); no secrets ever enter the .pyz or an image layer", "digest determinism — zip entries sorted with date_time (1980,1,1,0,0,0) via the normalize post-pass; tar entries sorted with fixed TarInfo metadata; gzip.compress(mtime=0); json.dumps(sort_keys=True, separators=(',',':')); two-build sha256 equality gates the artifact (REQ-21)", "OCI end-to-end — manifest application/vnd.oci.image.manifest.v1+json, config application/vnd.oci.image.config.v1+json, layers application/vnd.oci.image.layer.v1.tar+gzip; base layer descriptors reused verbatim; never push an index; urllib does NOT follow 307/308 on PUT/POST — implement explicit Location-following; gitea's anonymous-200 /v2/token answer must fail loudly; HEAD-before-upload makes re-runs idempotent", "bounded retry (3 attempts) on release attach and publish; on exhaustion exit non-zero with a machine-readable marker → ship harness records release_pending escalation; never blocks tag+merge", "stub-testability — scripts take base URLs as parameters (no hardcoded hosts); stubs bind ('127.0.0.1', 0) per the v0.1 _StubOwuiHandler precedent; no live-forge writes in tests; v0.1 test files are frozen reference — v0.2 adds test_cli_packaging.py, test_build_cli.py, test_release_upload.py, test_publish_image.py only, and the cumulative suite stays green", "dist/ is build output — gitignored (dist/, *.pyz), never committed; README.md edits are additive sections (CLI install/usage, docker usage) with v0.1 content intact"]
|
||||
territory: [cli/**, scripts/**, tests/**, Dockerfile, README.md, .gitignore]
|
||||
subagent_type: backend-engineer
|
||||
---
|
||||
|
||||
---
|
||||
name: frontend-engineer
|
||||
domain: frontend
|
||||
active: true
|
||||
phase_specific: true
|
||||
reason: P2 only (REQ-13 admin UI). Inactive outside P2 — no UI work exists in P1/P3/P4/P5; deactivate at the P2 exit boundary and do not dispatch outside P2.
|
||||
frameworks: [html, css, vanilla-js]
|
||||
constraints: ["server-rendered static pages from coreci_broker_ui/ on the same port as the API; no build step, no client framework, no npm (REQ-13)", "pages: /ui/login, /ui/app_registrations, /ui/user_allowlist, /ui/audit plus /ui/dashboard and a persistent nav header with Logout", "pages refresh via the corresponding admin API after each action; the API contract is owned by backend-engineer — frontend adapts", "session cookie broker_session is HttpOnly + SameSite=Lax (D-013); no direct DB or file access — admin API only"]
|
||||
territory: [coreci_broker_ui/**]
|
||||
subagent_type: frontend-engineer
|
||||
---
|
||||
|
||||
---
|
||||
name: security-engineer
|
||||
domain: security
|
||||
active: true
|
||||
phase_specific: true
|
||||
reason: Custom persona activated for P3 (tool auth plane: register gates, App-token verification, identity enforcement) and P5 (security review pass before ship); inactive in P1/P2/P4 unless the lead escalates a finding.
|
||||
frameworks: [python3.11, unittest, STRIDE, CWE]
|
||||
constraints: ["review-only: coreci_broker.py and tests stay owned by backend-engineer; findings route through lead-developer, never direct edits", "findings are STRIDE-classified and CWE-first (cite CWE ids)", "focus: path traversal on /secrets/<path> (URL-decode before validation), authz (allowlist gates, path scoping, identity match), audit completeness, pepper never re-printed, 0600 modes, token scoping"]
|
||||
territory: [coreci_broker.py, tests/**]
|
||||
reason: Review-only pass activated for P3 (milestone security review — artifact integrity, no secrets in the archive/image, token discipline in upload paths, registry auth flows, boot parity of materialized assets); inactive in P1/P2 unless the lead escalates a finding — same posture as v0.1.
|
||||
frameworks: [python3.11, unittest, STRIDE, CWE, oci-dist-spec]
|
||||
constraints: ["review-only — every v0.2 file stays owned by release-engineer; findings route through lead-developer, never direct edits; coreci_broker.py is in review scope for zero-delta verification (AC14) only and is writable by nobody this milestone", "findings are STRIDE-classified and CWE-first (cite CWE ids)", "focus — secrets discipline (file-only token path via scripts/_env_secrets.py, never shell env, non-empty assert before first use), artifact integrity (no secrets in the .pyz or any image layer; sha256 provenance recorded in release notes), registry auth flows (gitea anonymous-200 /v2/token quirk, Basic username-ignored shape, 307/308 handling), attachment idempotency and bounded-retry escalation paths"]
|
||||
territory: [coreci_broker.py, cli/**, scripts/**, Dockerfile, tests/**]
|
||||
subagent_type: ci-security-auditor
|
||||
---
|
||||
|
||||
@@ -41,20 +30,46 @@ subagent_type: ci-security-auditor
|
||||
name: lead-developer
|
||||
domain: coordination
|
||||
active: true
|
||||
reason: Active across all execution phases (P1–P5): decomposes plans into vertical slices per persona domain, arbitrates territory conflicts, revises the plan at phase boundaries, and sequences tasks across persona dependencies. No direct code writes.
|
||||
phase_specific: false
|
||||
reason: Active across all execution phases (P1–P3): decomposes plans into vertical slices along the build → attach → publish pipeline, arbitrates the frozen-broker rule (no persona writes coreci_broker.py — P0 fixes are lead-dispatched exceptions), routes security findings, and sequences tasks across phase boundaries. No direct code writes.
|
||||
frameworks: [none]
|
||||
constraints: ["no direct modifications to code files — coordination and conflict resolution only", "arbitration: backend-engineer owns files in cross-persona conflicts and defines the API contract; frontend adapts; security findings flow through backend", "enforce boundaries: no direct database access from the UI, no UI logic inside coreci_broker.py; battle-tested defaults over novel approaches"]
|
||||
constraints: ["no direct modifications to code files — coordination and conflict resolution only", "sole gate for frozen-broker exceptions — any P0 fix to coreci_broker.py (or the frozen UI/client/reference sources) is a lead-dispatched, defect-scoped task, never a persona territory claim", "arbitration — release-engineer owns every v0.2 file in cross-persona conflicts and defines the artifact/registry/release contracts; security findings flow lead → release-engineer; battle-tested defaults over novel approaches", "enforce — zero broker-line delta (AC14), secrets only via the .env* file loader, no secrets in artifacts, bounded-retry release_pending escalations never block tag+merge"]
|
||||
territory: [.ciagent/PLAN.md, .ciagent/*.md]
|
||||
subagent_type: lead-developer
|
||||
---
|
||||
|
||||
---
|
||||
name: backend-engineer
|
||||
domain: backend
|
||||
active: false
|
||||
phase_specific: false
|
||||
reason: Deactivated — the broker is frozen in v0.2; RESEARCH §4.2 verified run_standalone/boot/load_config/_parse_env_file are importable as-is, 1390/1400 lines leaves 10 lines of headroom, and zero-change is a hard constraint. All v0.2 engineering is distribution tooling owned by release-engineer. Reactivates only as a lead-developer-dispatched, defect-scoped P0 fix to coreci_broker.py — never via its own territory.
|
||||
frameworks: [python3.11, http.server, sqlite3, hashlib.scrypt, unittest]
|
||||
constraints: ["stays inactive for all planned v0.2 work; any broker change is a lead-developer-dispatched exception scoped to the P0 defect", "if dispatched — the 1400-line AC14 cap and every v0.1 runtime constraint (REQ-14..REQ-18, {error, message} JSON contract, 0600 modes, never write to creds-coreci-coo) remain binding"]
|
||||
territory: []
|
||||
subagent_type: backend-engineer
|
||||
---
|
||||
|
||||
---
|
||||
name: frontend-engineer
|
||||
domain: frontend
|
||||
active: false
|
||||
phase_specific: false
|
||||
reason: Deactivated — no UI work in v0.2; REQ-21..23 touch no page. The v0.1 UI (coreci_broker_ui/) is complete, frozen, and consumed as a read-only staging input by the zipapp build, then materialized verbatim at boot; any UI defect surfaced through the packaged path is a lead-dispatched exception.
|
||||
frameworks: [html, css, vanilla-js]
|
||||
constraints: ["stays inactive for the whole milestone; coreci_broker_ui/** is a read-only build input owned by no persona"]
|
||||
territory: []
|
||||
subagent_type: frontend-engineer
|
||||
---
|
||||
|
||||
---
|
||||
name: data-engineer
|
||||
domain: data
|
||||
active: false
|
||||
reason: Deactivated — no database tier exists beyond sqlite3 embedded in the single file; the spec §5.2 schema and all SQL live inside coreci_broker.py and are owned by backend-engineer; no pipelines, ETL, or migrations in v0.1 scope.
|
||||
phase_specific: false
|
||||
reason: Deactivated (carried from v0.1) — no database tier beyond sqlite3 embedded in the frozen broker; no schema, pipeline, ETL, or migration work in v0.2. Image layers and release assets are build outputs owned by release-engineer, not data surfaces.
|
||||
frameworks: [sqlite3]
|
||||
constraints: ["stays inactive for the whole milestone; any residual data work folds into backend-engineer"]
|
||||
constraints: ["stays inactive for the whole milestone; any residual data work folds into release-engineer or a lead-dispatched broker fix"]
|
||||
territory: []
|
||||
subagent_type: backend-engineer
|
||||
---
|
||||
+55
-112
@@ -1,158 +1,101 @@
|
||||
---
|
||||
must_haves:
|
||||
- "python3 -m unittest discover -s tests passes (all phases, cumulative)"
|
||||
- "coreci_broker.py single file, line count < 800 (AC14)"
|
||||
- "python3 -m py_compile coreci_broker.py coreci_broker_client.py"
|
||||
- "broker boots standalone on an ephemeral port; DB + audit log created mode 0600; admin pass printed exactly once"
|
||||
- "all AC1-AC14 acceptance criteria green by end of P5"
|
||||
- "python3 -m unittest discover -s tests passes (cumulative — the v0.1 suite plus the v0.2 test files: test_build_cli, test_cli_packaging, test_release_upload, test_publish_image, test_dockerfile_consistency, test_acceptance_v02)"
|
||||
- "python3 -m py_compile cli/__main__.py scripts/build_cli.py scripts/release.py scripts/publish_image.py scripts/_env_secrets.py"
|
||||
- "broker line count unchanged: coreci_broker.py stays at its v0.1.5 count of 1390 lines (AC14, D-016..D-018 cap 1400) — zero broker edits"
|
||||
- "two consecutive builds via scripts/build_cli.py produce byte-identical artifacts (sha256 equal) — same-environment scope (G-009)"
|
||||
- "the built .pyz boots with v0.1 acceptance parity (admin pass printed exactly once, broker.db + broker.audit.log 0600, UI + client served, silent second boot)"
|
||||
- "stub-registry publish roundtrip green (scripts/publish_image.py vs a local stub, including a relative-Location 307 case, G-019) and stub release upload green (scripts/release.py vs a local stub)"
|
||||
---
|
||||
|
||||
# PLAN — CoreCI Broker (Milestone v0.1)
|
||||
# PLAN — CoreCI Broker (Milestone v0.2 — cli-packaging)
|
||||
|
||||
Execution conventions: sequential waves (within a wave, tasks run in listed order); each task = one subagent commit `feat(P##): …` carrying a `---ci---` block (phase, milestone: v0.1, status: execute, persona, task, requirements.covered). Tests ship with their task — never deferred. Personas per PERSONAS.md: backend-engineer (all phases), frontend-engineer (P2 only), security-engineer (P3+P5, review-only — findings route via lead-developer, never direct edits), lead-developer (coordination, no code writes). Verify commands: `python3 -m py_compile <files>`; `python3 -m unittest discover -s tests`; AC14 line-count via test. Test harness: in-process `ThreadingHTTPServer` on an ephemeral port + `tempfile.TemporaryDirectory()` state, driven with stdlib `urllib` (RESEARCH §6).
|
||||
Execution conventions: sequential waves (within a wave, tasks run in listed order); each task = one subagent commit `feat(P##): …` carrying a `---ci---` block (phase, milestone: v0.2, status: execute, persona, task, requirements.covered). Tests ship with their task — never deferred. Personas per PERSONAS.md: release-engineer (ALL v0.2 code — `cli/**`, `scripts/**`, the new test files, `Dockerfile`, `.dockerignore`, additive README, `.gitignore`), security-engineer (P3 only, review-only — findings route via lead-developer, never direct edits), lead-developer (coordination + ship steps, no code writes; sole gate for frozen-broker exceptions). The broker is FROZEN this milestone (1390/1400, REQ-17/AC14): no task edits `coreci_broker.py`, `coreci_broker_ui/`, `coreci_broker_client.py`, or `reference_tools/` — a P0 defect there is a lead-developer-dispatched exception only. Verify commands: `python3 -m py_compile <files>`; `python3 -m unittest discover -s tests`; broker zero-delta via the AC14 line-count test. Test harness (v0.1 §6 conventions extended, RESEARCH §4.3): `ThreadingHTTPServer` stubs on `("127.0.0.1", 0)` + `tempfile.TemporaryDirectory()` state + subprocess boots of the built artifact driven with stdlib `urllib`; scripts take base URLs as parameters (stub-testable); NEVER live-forge writes in tests (the only live-forge contact is the G-013 smoke gate + the ships themselves). Secrets resolve only via `scripts/_env_secrets.py` reading `.env`/`.env.secrets`/`.env.*` files — never shell env (`GITEA_TOKEN` is in `blocked_env_vars`); non-empty asserted before the first API call. `dist/` + `*.pyz` are gitignored — build artifacts are never committed. Release attach + image publish failures follow bounded retry (3) → `release_pending` escalation — never blocking tag+merge. Ships: phase 0 → `v0.2.0`, P1 → `v0.2.1`, P2 → `v0.2.2`, P3 → `v0.2.3` (final phase patch = milestone release, D-008 convention). **G-021: the `v0.2.0` phase-0 release carries NO `.pyz` and its notes must not promise one — D-020 starts at `v0.2.1`.** Release-tooling split (G-011): the sanctioned secrets loader + release attacher land in P1 (Wave 3) so every release from `v0.2.1` is script-attached; P2 owns the image publish client + Dockerfile.
|
||||
|
||||
## User-Facing Surface
|
||||
|
||||
- **CEO admin UI** — server-rendered from `coreci_broker_ui/`, same port as the API, persistent nav with Logout: `/ui/login` (the CEO's bookmark), `/ui/app_registrations` (table + add-modal + revoke/reinstate), `/ui/user_allowlist` (same, optional email), `/ui/audit` (filterable, read-only, no export), `/ui/dashboard`.
|
||||
- **Tool-side client module** — `coreci_broker_client.py`: `ensure_registered()`, `get_state()`, `put_state()`, `read_secret()`; stdlib-urllib-only, importable with zero dependencies in any OWUI tool env; fetched unauthenticated at `/static/coreci_broker_client.py`.
|
||||
- **Standalone CLI** — `python3 coreci_broker.py`: binds `127.0.0.1:8765`; first boot prints the generated admin pass exactly once.
|
||||
- **OWUI Tools mount** — `Tools` class (same file): daemon-thread server behind a once-per-process guard, `self.valves`, `_`-prefixed helpers.
|
||||
- **Downloadable CLI artifact** — `coreci-broker-vX.Y.Z.pyz` on the gitea release page (every release from `v0.2.1`, D-020), alongside the existing source archive; its sha256 recorded in the release notes (computed by `scripts/release.py` from the artifact file — G-017).
|
||||
- **CLI boot** — `python3 coreci-broker-vX.Y.Z.pyz` or `./coreci-broker-vX.Y.Z.pyz` after `chmod +x` (shebang `#!/usr/bin/env python3`); boots the standalone broker — full admin UI at `/ui/`, client module at `/static/coreci_broker_client.py`, bound address per `BROKER_BIND` (default `127.0.0.1:8765`).
|
||||
- **`--version`** — prints the release version, exits 0.
|
||||
- **Docker image** — `docker pull git.coreci.dev/coreci/broker:<tag>` (+ `latest`); `docker run` with a volume for state.
|
||||
- **Release notes** — on the release page: milestone summary, phases, verification, CLI install + usage, docker usage.
|
||||
- The v0.1 surface (four-page UI + dashboard, client module, audit) unchanged — served through the packaged path.
|
||||
|
||||
## Happy Path
|
||||
|
||||
1. First boot in a fresh directory: broker writes `.env` (32-byte base64 pepper + 32+-char admin pass), prints the pass to the startup log exactly once (AC1); a second boot prints nothing.
|
||||
2. CEO opens the `/ui/login` bookmark, logs in with the printed pass (session cookie `broker_session`), lands on the dashboard.
|
||||
3. CEO adds the tool's app UUID on `/ui/app_registrations` and the user_id on `/ui/user_allowlist`; each page reflects the persisted row right after the action (AC2/AC3); audit gets `add_app`/`add_user`.
|
||||
4. Tool author fetches `/static/coreci_broker_client.py` (unauthenticated) into the OWUI workspace and imports it — no third-party deps.
|
||||
5. Tool calls `ensure_registered()`: broker gates on the allowlists, derives the token server-side, returns `{token, token_expires_at, pepper}` (row-create); the client persists via `PUT /agents/<id>/state`.
|
||||
6. Tool reads `coreci/proxmox` with `Authorization: App <UUID> <token>` → 200 secret payload (AC5); audit line `read`/`ok`.
|
||||
7. Same tool requests `coreci/gitea` → scope mismatch → 403 (AC6); audit `read`/`denied`.
|
||||
8. Token expires (`BROKER_TOKEN_TTL`): next call → 401 (AC10) → client re-registers → new token → retry succeeds; audit `register`.
|
||||
9. CEO reviews `/ui/audit`: login/logout (AC9), allowlist adds, registers, reads ok/denied — every step audited.
|
||||
1. Owner downloads `coreci-broker-v0.2.3.pyz` from the `v0.2.3` release → `python3 coreci-broker-v0.2.3.pyz` (or `chmod +x` + `./…`) → first boot generates `.env` and prints the admin pass exactly once; `broker.db` + `broker.audit.log` created mode 0600 in the state dir → UI at the bound address (`/ui/login` bookmark works).
|
||||
2. Second boot: silent — pass never re-printed; state persists.
|
||||
3. Tool authors fetch `coreci_broker_client.py` from `/static/coreci_broker_client.py` exactly as in v0.1 — the packaged artifact is the only input.
|
||||
4. `--version` reports the release version without booting.
|
||||
5. Separately: `docker pull git.coreci.dev/coreci/broker:v0.2.3` → `docker run -p 8765:8765 -v broker-state:/state …` → identical first-boot/0600/UI behavior inside the container, state on the volume.
|
||||
|
||||
## UX Acceptance Criteria
|
||||
|
||||
- **AC1** — first boot prints the admin pass exactly once; the CEO logs in at `/ui/login` with it; second boot prints nothing.
|
||||
- **AC2/AC3** — app registrations and user-allowlist entries are added and removed via the UI; the page reflects persisted state after every action (no stale views).
|
||||
- **AC9** — every login and logout writes an audit line (`login`/`logout`).
|
||||
- **UI** — server-rendered static HTML/CSS/JS from `coreci_broker_ui/` on the same port: no build step, no framework, no npm.
|
||||
- **Audit page** — read-only with filters; NO export button or download control of any kind.
|
||||
- **Client module** — `coreci_broker_client.py` imports cleanly with zero deps (stdlib urllib only) in any OWUI tool env; contains no secrets.
|
||||
- **Self-contained artifact** — the `.pyz` bundles broker + UI pages + client module; only runtime state lives outside it (`.env`, `broker.db`, `broker.audit.log`, materialized assets under `<state>/.coreci_broker_assets/`); one file serves the full admin UI at `/ui/` and the client at `/static/`.
|
||||
- **`--version`** — prints the release version (from the build-time `_cli_version.py`), exits 0, no broker import on that path.
|
||||
- **First-boot parity** — behavior through the packaged artifact is byte-exact with v0.1 standalone: admin pass printed exactly once, `broker.db` + `broker.audit.log` mode 0600, UI + client served, second boot silent.
|
||||
- **Per-release artifact** — every release from `v0.2.1` (D-020) carries `coreci-broker-vX.Y.Z.pyz` + release notes (CLI install/usage + sha256, docker usage); attachment failure escalates `release_pending`, never blocks tag+merge. `v0.2.0` carries none (G-021).
|
||||
- **Image published once per milestone** — at the final ship: `git.coreci.dev/coreci/broker:v0.2.3` + `latest` (D-021); publish failure escalates per the same contract with the G-020 closure runbook, never blocking the milestone merge.
|
||||
|
||||
## Phase 1 — Core Skeleton
|
||||
## Phase 1 — CLI Packaging + Release Tooling
|
||||
|
||||
Branch `phase/01-core-skeleton` → tag `v0.1.1`. Requirements: REQ-03, REQ-11, REQ-14, REQ-15, REQ-17, REQ-18.
|
||||
Exit: broker boots standalone on an ephemeral port; `broker.db` + `broker.audit.log` created mode 0600; admin pass printed exactly once (silent second boot); §6 token derivation byte-exact; source < 800 lines.
|
||||
Branch `phase/01-cli-packaging` → tag `v0.2.1`. Requirements: REQ-21, REQ-22.
|
||||
Exit: artifact builds reproducibly from a clean checkout (two consecutive same-environment builds → byte-identical sha256); the built `.pyz` boots with v0.1 acceptance parity (pass printed once, 0600 state files, UI + client served from materialized assets, silent second boot); `--version` wired; release-asset automation stub-verified; REQ-21 + REQ-22 consolidated green; the `v0.2.1` release carries the `.pyz` attached by `scripts/release.py`.
|
||||
|
||||
### Wave 1 — persistence layer (independent)
|
||||
### Wave 1 — build pipeline + boot shim
|
||||
|
||||
- **P1-01** · backend-engineer · `coreci_broker.py`, `tests/test_config.py` · `.env` reader/writer at `BROKER_ENV_PATH` (spec §5.1 contract + documented additions); first-boot generation of `BROKER_INSTANCE_PEPPER` (32 random bytes, base64, synced to SQLite when the store boots — sync lands with P1-02's schema) and `BROKER_ADMIN_PASS` (32+ random chars), written to `.env`, printed to the startup log exactly once — never again · REQ-14 · test: first boot writes `.env` + prints pass once; second boot silent; `BROKER_*` overrides honored.
|
||||
- **P1-02** · backend-engineer · `coreci_broker.py`, `tests/test_store.py` · SQLite schema per spec §5.2 (`broker_config`, `app_registrations`, `user_allowlist`, `agent_registrations`, `admin_sessions` + indexes); DB created mode 0600 via `os.open(..., 0o600)` (umask-independent); sole writer of `broker.db`, all SQL in-file · REQ-15 · test: boot creates `broker.db` at 0600 with all tables + indexes.
|
||||
- **P1-03** · backend-engineer · `coreci_broker.py`, `tests/test_audit.py` · append-only JSONL audit writer at `BROKER_AUDIT_LOG_PATH`: one JSON object per line, fields `ts, user_id, app_registration_uuid, agent_id, action, path, result` (admin actions carry null user_id/uuid); mode 0600; flush + fsync per write; no rotation (D-007) · REQ-11 · test: each write appends one valid JSON line at mode 0600; all action verbs accepted.
|
||||
- **P1-01** · release-engineer · `scripts/build_cli.py`, `cli/__main__.py`, `.gitignore`, `tests/test_build_cli.py` · build pipeline + boot shim (RESEARCH §1.2/§1.3; G-009/G-018): `build_cli.py <version>` stages an explicit allowlist tree — `coreci_broker.py`, `coreci_broker_client.py`, `coreci_broker_ui/**`, `cli/__main__.py`, generated `_cli_version.py` (`VERSION = "<tag>"`) — then `python3 -m zipapp <staging> -o dist/coreci-broker-vX.Y.Z.pyz --python "/usr/bin/env python3" --compress` (zipapp defaults to ZIP_STORED — `--compress` mandatory), then a normalize post-pass making same-environment rebuilds byte-identical (G-009: zip entries sorted by name, `date_time=(1980,1,1,0,0,0)`, `ZIP_DEFLATED`, **`external_attr` SET to constants** — files `0o100644 << 16`, dirs `0o40755 << 16` — never preserved/umask-derived; shebang re-prepended + exec bit), and prints the artifact sha256 (digest-in-notes default; no sidecar asset); `cli/__main__.py` = the five-step shim: (1) `--version` first — print `_cli_version.VERSION`, exit 0, no broker import; (2) state-dir resolution identical to the broker (`BROKER_ENV_PATH`-anchored, relative paths stay CWD-relative — zero path changes); (3) materialize assets from `zipfile.ZipFile(sys.path[0])` → `<state>/.coreci_broker_assets/<VERSION>/{ui,static}` (temp + `os.replace`, versioned dirs, same-version overwrite self-repair, no cleanup); (4) override precedence user-wins — `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` set only when absent from process env AND the user `.env` (parsed via `coreci_broker._parse_env_file`); (5) handoff `coreci_broker.run_standalone()`; an `os.path.isdir(sys.path[0])` guard falls back to repo-relative roots when run unzipped (zipapp-only is the documented mode); the shim header documents its private-API coupling to the broker (`_parse_env_file`, `run_standalone` — G-018: loud `ImportError` breakage accepted and tracked in P3-02); `.gitignore` += `dist/`, `*.pyz`; broker sources are read-only staging inputs — zero broker lines · REQ-21 · test: `test_build_cli.py` — two consecutive same-environment builds → byte-identical sha256; `_cli_version.VERSION` embedded; archive enumerates exactly the staging allowlist (broker + client + UI pages + shim); constant `external_attr` asserted (no umask bleed); `py_compile` of the shim green; built artifact `--version` prints the version, exits 0.
|
||||
|
||||
### Wave 2 — crypto primitives
|
||||
### Wave 2 — packaged boot acceptance
|
||||
|
||||
- **P1-04** · backend-engineer · `coreci_broker.py`, `tests/test_hasher.py` · dual-mode hasher (D-001): bcrypt when importable, else stdlib `hashlib.scrypt`; stored hashes format-prefixed (`$2b$…` / `scrypt$…`); verification format-aware · REQ-03 · test: scrypt hash/verify roundtrip + prefix dispatch on both formats (bcrypt path via format contract — no bcrypt locally).
|
||||
- **P1-05** · backend-engineer · `coreci_broker.py`, `tests/test_token.py` · token engine: `base64(sha256(pepper:UUID:user_id:agent_id:','.join(sorted(tags))))` — 44-char ASCII, no prefix; pure function of (pepper, inputs); never stores plaintext tokens · REQ-03 · test: derivation matches the §6 formula byte-for-byte; 44-char assertion; tag order normalized.
|
||||
- **P1-02** · release-engineer · `tests/test_cli_packaging.py` · end-to-end boot acceptance through the built artifact (RESEARCH §4.3 pattern; G-010 harness hardening; consolidates REQ-21): build into a `TemporaryDirectory` → `Popen([sys.executable, "<pyz>"], env={"BROKER_BIND": "127.0.0.1:0", BROKER_ENV_PATH/DB/AUDIT/CREDS_FILE/CREDS_SOURCE: tmp paths})` → parse the `coreci-broker listening on http://…` line for the ephemeral base URL under an explicit deadline loop; first boot prints `BROKER_ADMIN_PASS=` exactly once (stdout fully drained after `terminate()` before the second-boot assertion); `.env` + `broker.db` + `broker.audit.log` created mode 0600 in the state dir (independent of CWD); `/healthz` 200, `/ui/login` + `/ui/dashboard` 200 from materialized assets, `/static/coreci_broker_client.py` byte-exact; admin login + agent register + state + secret-read reuse the v0.1 acceptance request patterns (v0.1 parity through the packaged path); second boot silent (pass never re-printed); `--version` exits 0 with the version string; materialized assets verified under `<state>/.coreci_broker_assets/<VERSION>/`; user override honored (`.env`-set `BROKER_UI_ROOT` wins over the shim default); plus one un-overridden default-anchoring boot (state in CWD of a temp dir — the owner's literal happy path, G-010) · REQ-21 · test: is the acceptance file.
|
||||
|
||||
### Wave 3 — transport + mounts (listed in dependency order)
|
||||
### Wave 3 — release-asset automation (G-011: lands in P1)
|
||||
|
||||
- **P1-06** · backend-engineer · `coreci_broker.py`, `tests/test_router.py` · stdlib `ThreadingHTTPServer` + JSON router: body/param parsing, `{"error, message}` contract on 400/401/403/404/409/500, static-file serving hook for `/ui/` + `/static/` (fixed root, traversal-rejecting); no business logic in the router · REQ-18 · test: ephemeral-port boot; every error code returns the contract shape; static hook serves a fixture.
|
||||
- **P1-07** · backend-engineer · `coreci_broker.py`, `tests/test_mounts.py` · OWUI `Tools` mount: `__init__` starts the daemon-thread server behind a once-per-process guard (no activate() hook exists — RESEARCH §4), `self.valves = self.Valves()`, `_`-prefixed helpers, paths resolved from env never `__file__`; standalone `__main__` entry · REQ-18 · test: `Tools()` double-instantiation starts exactly one server; standalone boot serves a 200.
|
||||
- **P1-03** · release-engineer · `scripts/_env_secrets.py`, `scripts/release.py`, `tests/test_release_upload.py` · sanctioned secrets loader + release attacher (REQ-22; RESEARCH §2): `_env_secrets.py` parses `.env`/`.env.secrets`/`.env.*` from the repo root + `.ciagent/` (mirroring the broker's `_parse_env_file` semantics, quote-stripping) — `GITEA_TOKEN` resolves ONLY from these files, never `os.environ` (structurally blocked by `blocked_env_vars`), asserted non-empty before the first API call (v1.8 lesson: empty token → gitea 404, masking auth failure); `release.py <tag> <artifact>` (gitea base URL + owner/repo parameterized): **computes the artifact sha256 from the file itself — never an argument, never hand-pasted (G-017)** → builds the notes body per the RESEARCH §2.3 model (milestone summary, phases, verification, CLI install + usage with the artifact sha256, docker usage) → create-or-get by tag (`GET /api/v1/repos/{o}/{r}/releases/tags/{tag}`; 404 → `POST .../releases` with the notes body) → idempotent attach (`GET .../releases/{id}/assets` — same name + size → success, no duplicate on bounded-retry re-runs) → `POST .../releases/{id}/assets` multipart (field name `attachment`, `?name=coreci-broker-vX.Y.Z.pyz`, `Authorization: token <T>`) → bounded retry (3 attempts, backoff) → exhaustion exits non-zero with a machine-readable marker for the `release_pending` escalation; never blocks tag+merge · REQ-22 · test: `test_release_upload.py` — `_StubGiteaHandler` on `("127.0.0.1", 0)` (v0.1 `_StubOwuiHandler` precedent, fake token, no live-forge writes): create-or-get in both orders (release exists / 404 → create), multipart parsed (field name + `?name=` + body bytes asserted), sha256-from-file asserted (passing a wrong digest argument is ignored/refused), idempotent re-attach uploads nothing twice, 500×3 → the `release_pending` marker, token discipline (file-only resolution + non-empty assert before the first call), notes sections present.
|
||||
|
||||
### Wave 4 — phase acceptance
|
||||
### Wave 4 — phase ship
|
||||
|
||||
- **P1-08** · backend-engineer · `tests/test_acceptance_p1.py` · AC14 line-count test (`coreci_broker.py` < 800 lines, wc-style, no comment stripping) + boot acceptance: 0600 on DB + audit log, admin pass printed exactly once (silent second boot), restart persistence of DB rows + audit history · REQ-17, REQ-16 (per REQUIREMENTS traceability), REQ-15 · test: is the acceptance file.
|
||||
- **P1-04** · lead-developer · no code writes (the release-engineer's built artifact + scripts are consumed read-only) · phase ship: tag `v0.2.1` → squash-merge `phase/01-cli-packaging` → `milestone/v0.2-cli-packaging` → push tag + branch → gitea release via `scripts/release.py` (D-020 starts here: `coreci-broker-v0.2.1.pyz` + its sha256 in the notes, attached by the repo script — no ad-hoc bridge, G-011) · REQ-21, REQ-22 (ship) · verify: `v0.2.1` tag + release on the forge with the `.pyz` attached; attachment failure → bounded retry (3) → `release_pending` escalation, never blocking tag+merge.
|
||||
|
||||
## Phase 2 — Admin Plane
|
||||
## Phase 2 — Docker + Image Publish
|
||||
|
||||
Branch `phase/02-admin-plane` → tag `v0.1.2`. Requirements: REQ-01, REQ-05, REQ-06, REQ-12, REQ-13.
|
||||
Exit: AC1/AC2/AC3/AC9 green; admin-usable (login → manage allowlists → query audit, all via UI).
|
||||
Branch `phase/02-docker-release-integration` → tag `v0.2.2`. Requirements: REQ-23 (+ REQ-21/REQ-22 docs refinement).
|
||||
Exit: daemonless publish client stub-verified end-to-end (no live-forge writes, no container runtime); Dockerfile + .dockerignore + README document the docker-user path and stay consistency-tested against the daemonless image; the `v0.2.2` release carries the `.pyz` attached by `scripts/release.py` with CLI + docker notes.
|
||||
|
||||
### Wave 1 — admin auth
|
||||
### Wave 1 — daemonless OCI publish client
|
||||
|
||||
- **P2-01** · backend-engineer · `coreci_broker.py`, `tests/test_admin_auth.py` · `POST /api/broker/auth/login|logout|whoami`: verifies `BROKER_ADMIN_USER`/`BROKER_ADMIN_PASS`, creates `admin_sessions` row (TTL `BROKER_SESSION_TTL`, default 300s); cookie `broker_session` HttpOnly + SameSite=Lax (D-013); HTTP Basic accepted on all admin API endpoints; lazy expiry + boot cleanup of stale sessions · REQ-01, REQ-11 · test: login sets cookie; wrong pass → 401; expired session → 401; logout invalidates + audits (AC9 `login`/`logout` lines).
|
||||
- **P2-01** · release-engineer · `scripts/publish_image.py`, `tests/test_publish_image.py` · daemonless OCI publish client (D-019, G-014: one vertical task; all endpoints parameterized; G-019 wire hardening): pull — anonymous Docker Hub token (`auth.docker.io/token`, scope `repository:library/python:pull`) → `python:3.11-slim` manifest (D-022) with the full Accept set → branch on `"manifests"` vs `"layers"` (index → resolve the `linux/amd64` child manifest by digest) → fetch child + config + 4 layer blobs; record resolved digests (`--base-manifest` override for exact re-publishes); auth — gitea `GET /v2/token?service=container_registry&scope=repository:coreci/broker:pull,push` with `Authorization: Basic <user>:<GITEA_TOKEN>` (token via `_env_secrets`; username ignored by the forge) — insist on Basic credentials present + issued JWT non-empty (the endpoint answers 200 anonymously with an empty-scope JWT — fail loudly, never treat 200 as authorization); push — **per-request timeouts (~120s on blob POSTs, G-019)** + **blob-level retry with HEAD-skip** (HEAD each blob first: 200 → skip; base layers ~45.7 MB re-upload once per milestone, retries resume cheap) → monolithic `POST /v2/coreci/broker/blobs/uploads/?digest=sha256:<hex>` (fallback: two-step POST → 202 `Location` → PUT) with explicit Location-following on 307/308 (stdlib urllib does NOT follow method-preserving redirects on PUT/POST — verified; **the stub test covers a relative `Location` at least once, G-019**) → deterministic broker layer (tar entries sorted, fixed TarInfo metadata uid/gid 0, `gzip.compress(..., mtime=0, compresslevel=9)`) carrying `/app/coreci-broker.pyz`; config copied from the base + diff_id appended + `Entrypoint ["python3", "/app/coreci-broker.pyz"]`, `Cmd []`, `WorkingDir /state`, optional version Label, serialized `json.dumps(sort_keys=True, separators=(",",":"))`; manifest PUT with OCI media types end-to-end (manifest `application/vnd.oci.image.manifest.v1+json`, config `…image.config.v1+json`, layers `…image.layer.v1.tar+gzip`; base layer descriptors reused verbatim; never push an index) under `<version>` then `latest` (D-021); provenance digests recorded for the release notes · REQ-23 · test: `test_publish_image.py` — `_StubRegistryHandler` on `("127.0.0.1", 0)` (fake token via temp `.env` files): anonymous-token trap (200 with empty-scope JWT → loud failure), HEAD-skip on retry, monolithic upload with 307 → relative `Location` followed explicitly, manifest PUT with correct media types + tag sequence (version then `latest`), deterministic layer/config digests stable across two runs (gzip mtime=0), base manifest/layer handling against a fixture index, timeout wiring (a stalled endpoint → client aborts, does not hang).
|
||||
|
||||
### Wave 2 — admin API
|
||||
### Wave 2 — Dockerfile + .dockerignore + docs
|
||||
|
||||
- **P2-02** · backend-engineer · `coreci_broker.py`, `tests/test_apps.py` · app_registrations CRUD: `GET/POST /api/broker/app_registrations`, `DELETE .../<uuid>` (marks revoked — new registrations rejected, existing tokens honored until TTL), `POST .../reinstate`; UUID format validation → 400; duplicate → 409; audited `add_app`/`remove_app` · REQ-05 · test: add/list/revoke/reinstate matrix + 400/409 + revoke-keeps-tokens + audit lines.
|
||||
- **P2-03** · backend-engineer · `coreci_broker.py`, `tests/test_allowlist.py` · user_allowlist CRUD: same shape (`GET/POST`, `DELETE .../<user_id>` revoked semantics, `POST .../reinstate`), optional `email`, duplicate → 409; audited `add_user`/`remove_user` · REQ-06 · test: add/remove/reinstate + 409 + audit lines.
|
||||
- **P2-04** · backend-engineer · `coreci_broker.py`, `tests/test_audit_query.py` · admin-only `GET /api/broker/audit`: filters `since`, `until`, `user_id`, `app_registration_uuid`, `action`; `limit` default 100 / max 1000; `next_since` cursor; streams the file line-by-line · REQ-12 · test: seeded audit file → filter combos, limit clamp, cursor pagination.
|
||||
- **P2-02** · release-engineer · `Dockerfile`, `.dockerignore`, `README.md` (additive section only — no v0.1 content rewrites), `tests/test_dockerfile_consistency.py` · docker-user path + usage docs (G-015/G-016): `Dockerfile` self-contained (no pre-built artifact in the build context): `FROM python:3.11-slim` → `COPY coreci_broker.py coreci_broker_client.py coreci_broker_ui/ cli/ /app/src/` → `ARG VERSION` → generated `_cli_version.py` from the build arg → `RUN python3 -m zipapp /app/src -o /app/coreci-broker.pyz --python "/usr/bin/env python3" --compress` → `WORKDIR /state` → `ENTRYPOINT ["python3", "/app/coreci-broker.pyz"]` → `EXPOSE 8765`; header documents the equivalence mapping to the daemonless-published image (same base family, entrypoint, pyz content, `/state` anchoring; differences: docker-build layering vs verbatim base layers + one broker layer, build-arg vs build-time version provenance; the daemonless client is the source of truth, byte-identity not required — RESEARCH §3.5); `.dockerignore` (G-015): `.git`, `.env`, `.env.*`, `data/`, `*.db`, `dist/`, `__pycache__/`, `tests/`, `.ciagent/` — no secrets, state, or build artifacts can enter an image build context; README additive section: CLI install + usage (release-page download, `chmod +x` / `python3`, `--version`, env-contract pointer, first-boot pass discipline, `BROKER_ENV_PATH` state anchoring, Python 3.11+ note) + docker usage (`docker pull git.coreci.dev/coreci/broker:<tag>` + `latest`; `docker run -p 8765:8765 -v broker-state:/state …`; state-on-volume contract) · REQ-23, REQ-21/REQ-22 (docs) · test: `test_dockerfile_consistency.py` (G-016 — equivalence made testable): Dockerfile COPY set == `build_cli.py` staging allowlist; zipapp flags identical (`--python`/`--compress`); `ARG VERSION` → `_cli_version` format parity; `ENTRYPOINT`/`WORKDIR` == the daemonless-published config values; `.dockerignore` excludes every forbidden context entry.
|
||||
|
||||
### Wave 3 — admin UI (frontend-engineer; pages refresh via the admin API after each action; no direct DB/file access)
|
||||
### Wave 3 — phase ship
|
||||
|
||||
- **P2-05** · frontend-engineer · `coreci_broker_ui/` (login page, dashboard, shared nav/CSS) · `/ui/login` (the CEO's bookmark) + `/ui/dashboard`; persistent nav header with Logout on every page · REQ-13, REQ-01 · verify: pages served 200 on the API port, post-login redirect to dashboard; full AC drive lands in P2-08.
|
||||
- **P2-06** · frontend-engineer · `coreci_broker_ui/` (app_registrations + user_allowlist pages) · table + add-modal + revoke/reinstate per row on both pages; after each action the page re-fetches via the corresponding API and reflects persisted state · REQ-13, REQ-05, REQ-06 · verify: AC2/AC3 flows (tests in P2-08); interim smoke — pages boot + served.
|
||||
- **P2-07** · frontend-engineer · `coreci_broker_ui/` (audit page) · filterable audit view (since/until/user_id/uuid/action) wired to `GET /api/broker/audit`; strictly read-only — no export button or control of any kind · REQ-13, REQ-12 · verify: page renders query results; no export element in the markup.
|
||||
- **P2-03** · lead-developer · no code writes · phase ship: tag `v0.2.2` → squash-merge `phase/02-docker-release-integration` → `milestone/v0.2-cli-packaging` → push tag + branch → release via `scripts/release.py` (notes per the §2.3 model now including docker usage, `coreci-broker-v0.2.2.pyz` + sha256 attached alongside the source archive) · REQ-23 (phase completion) · verify: `v0.2.2` tag + release on the forge with the `.pyz` + notes; attachment failure → bounded retry → `release_pending`, never blocks tag+merge.
|
||||
|
||||
### Wave 4 — phase acceptance
|
||||
## Phase 3 — Final Review + Ship
|
||||
|
||||
- **P2-08** · backend-engineer · `tests/test_acceptance_p2.py` · AC1 (first-boot login at `/ui/login` with the printed pass), AC2/AC3 (add + remove app/user through the UI; `app_registrations`/`user_allowlist` persist; page reflects state after each action), AC9 (login/logout audit lines) · REQ-01, REQ-05, REQ-06, REQ-12, REQ-13 · test: is the acceptance file.
|
||||
Branch `phase/03-final-review-ship` → tag `v0.2.3` (final phase patch = milestone release, D-008 convention). Requirements: REQ-21, REQ-22, REQ-23 (acceptance).
|
||||
Exit: cumulative v0.1 + v0.2 acceptance green; broker zero-delta verified (1390/1400, AC14); security review PASS (or findings remediated); `v0.2.3` on `main` with the `.pyz` attached + full notes; image published as a gitea package (or escalated per the bounded-retry contract + G-020 closure runbook — never blocking merge); REQUIREMENTS/ROADMAP closed out; checkpoint cleared.
|
||||
|
||||
## Phase 3 — Tool Auth Plane
|
||||
### Wave 1 — acceptance consolidation
|
||||
|
||||
Branch `phase/03-tool-auth-plane` → tag `v0.1.3`. Requirements: REQ-02, REQ-04, REQ-09, REQ-10.
|
||||
Exit: AC4/AC7/AC8/AC10/AC12 green; tool-auth usable (register → App-auth calls → state persistence).
|
||||
|
||||
### Wave 1 — registration
|
||||
|
||||
- **P3-01** · backend-engineer · `coreci_broker.py`, `tests/test_register.py` · `POST /api/broker/agents/register` with `{agent_id, app_registration_uuid, user_id, tags}`: allowlist gates → 403 unknown/revoked UUID or user; malformed → 400; idempotent re-derive (D-005) — overwrite stored hash on input change, same derived token when unexpired + unchanged; TTL from `BROKER_TOKEN_TTL` (default 86400, D-007); pepper returned on row-create or tokenless register (KB-loss recovery); audited `register` · REQ-02, REQ-03 · test: gate matrix (AC7/AC8), idempotency/overwrite/same-token, pepper rule, 400s.
|
||||
|
||||
### Wave 2 — App-auth + agent surface (listed in dependency order)
|
||||
|
||||
- **P3-02** · backend-engineer · `coreci_broker.py`, `tests/test_app_auth.py` · App-auth verification (D-004): parse `Authorization: App <UUID> <token>`; UUID-scoped slow-hash scan over that tool's rows; in-memory verified-token cache populated on first success, never authoritative — every hit re-checks row status + expiry; 401 missing/malformed/failed · REQ-04 · test: cache-miss scan + cache-hit re-check (revoke + expiry), both hash formats, 401 matrix; revoked-app token honored until TTL.
|
||||
- **P3-03** · backend-engineer · `coreci_broker.py`, `tests/test_agent_meta.py` · `GET /api/broker/agents/<agent_id>` (D-015): App-auth + identity match; metadata-only response (no token, no hash) · REQ-04, REQ-02 · test: correct-identity 200 with metadata-only payload; identity mismatch → 403; no token material in response.
|
||||
- **P3-04** · backend-engineer · `coreci_broker.py`, `tests/test_state.py` · `GET/PUT /api/broker/agents/<agent_id>/state` (App-auth): broker enforces calling UUID + token-derived user_id match the stored identity; 404 when no state exists; state schema per spec §8 (agent_id, app_registration_uuid, user_id, tags, pepper, current_token, token_expires_at, first_registered_at, last_seen_at); backend toggle `BROKER_AGENT_STATE_BACKEND=internal` (default, rows in `broker.db`) `| owui` (thin OWUI-API adapter, stub-verified) (D-003) · REQ-09, REQ-10 · test: state roundtrip + identity enforcement + 404; owui backend against a stub server.
|
||||
|
||||
### Wave 3 — phase acceptance
|
||||
|
||||
- **P3-05** · backend-engineer · `tests/test_acceptance_p3.py` · AC4 (broker- and tool-side derivation byte-identical; stored as slow-hash), AC7 (revoked/unknown UUID → 403), AC8 (unknown user → 403), AC10 (TTL expiry → 401 → re-register → new token, audited), AC12 (state read/write with identity enforcement) · REQ-02, REQ-04, REQ-09, REQ-10 · test: is the acceptance file.
|
||||
|
||||
### Wave 4 — security review (review-only)
|
||||
|
||||
- **P3-06** · security-engineer · review of `coreci_broker.py`, `tests/**` (no edits — findings STRIDE-classified, CWE-cited, routed via lead-developer) · tool-auth security pass: register gates, UUID-scoped scan + cache authority, identity enforcement, TTL/expiry handling, audit completeness · REQ-02, REQ-04, REQ-09 · verify: findings recorded in the task commit's `---ci---` block; remediation (if any) = follow-up backend commit dispatched by lead-developer.
|
||||
|
||||
## Phase 4 — Secrets + Client Ecosystem
|
||||
|
||||
Branch `phase/04-secrets-client-ecosystem` → tag `v0.1.4`. Requirements: REQ-07, REQ-08, REQ-19, REQ-20.
|
||||
Exit: AC5/AC6 green; full product — client module fetched from `/static/` drives a reference tool end-to-end against the test broker.
|
||||
|
||||
### Wave 1 — secrets plane (listed in dependency order)
|
||||
|
||||
- **P4-01** · backend-engineer · `coreci_broker.py`, `tests/test_creds_source.py` · creds-source adapter (D-002): `FileCredsSource` default (JSON at `BROKER_CREDS_FILE`, default `data/creds.json` — D-014) + `OwuiApiCredsSource` (`BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY`; RESEARCH §4 flow: KB lookup → file map → `/api/v1/files/{id}/data/content`); response `{path, data, version, updated_at}` (version synthesized broker-side if absent); broker never writes the KB · REQ-08 · test: file-source hit/miss; owui source against a stub HTTP server.
|
||||
- **P4-02** · backend-engineer · `coreci_broker.py`, `tests/test_secrets.py` · `GET /api/broker/secrets/<path>` (App-auth): scope check `path == "coreci/" + agent_id` → 403 mismatch, 404 absent, 401 auth failure; every attempt audited (`read` ok/denied/error); traversal defense — URL-decode BEFORE validation, reject `..`, backslash, NUL, absolute paths (decoded form feeds the scope check) · REQ-07 · test: ok/401/403/404 matrix + URL-encoded traversal attempts rejected + audit per attempt.
|
||||
|
||||
### Wave 2 — client ecosystem (listed in dependency order)
|
||||
|
||||
- **P4-03** · backend-engineer · `coreci_broker_client.py`, `tests/test_client.py`, `coreci_broker.py` (static route) · public client module: stdlib `urllib` only (no httpx, no bcrypt — importable in any OWUI tool env); `ensure_registered()` idempotent; `get_state()`/`put_state()`; `read_secret()` with 401 → re-register → retry-once; `data.get("pepper")` (never assumes the key — RESEARCH §2 defect 3); served unauthenticated at `/static/coreci_broker_client.py` via the P1-06 hook; contains no secrets · REQ-19 · test: zero third-party imports asserted; full flow against the in-process broker; 401→re-register→retry path.
|
||||
- **P4-04** · backend-engineer · `reference_tools/`, `tests/test_reference_tools.py` · reference OWUI tools (D-006): (a) proxmox-style §11 flow — first call registers + persists state, subsequent calls read the token from state and fetch `coreci/proxmox`, downstream call defaults to a safe dry-run; (b) minimal auth-first-then-use-a-second-tool showcase · REQ-20 · test: both tools drive the test broker (FileCredsSource); dry-run default asserted; no live infra.
|
||||
|
||||
### Wave 3 — phase acceptance
|
||||
|
||||
- **P4-05** · backend-engineer · `tests/test_acceptance_p4.py` · AC5 (valid token reads the secret; tampered token → 401), AC6 (proxmox-registered tool reading `coreci/gitea` → 403); end-to-end: fetch client from `/static/` unauthenticated → import → register → state → secret read · REQ-07, REQ-19, REQ-20 · test: is the acceptance file.
|
||||
|
||||
## Phase 5 — Final Review + Ship
|
||||
|
||||
Branch `phase/05-final-review-ship` → tag `v0.1.5` (final phase patch = milestone release, D-008). Requirements: REQ-01..REQ-20 (acceptance).
|
||||
Exit: AC1–AC14 all green; security review PASS; README complete with deviations; owner approval recorded; milestone merged to main, tagged `v0.1.5`, released on gitea.
|
||||
|
||||
### Wave 1 — acceptance matrix
|
||||
|
||||
- **P5-01** · backend-engineer · `tests/test_acceptance_full.py` · complete §12 matrix: AC1–AC13 as behavioral tests (consolidating the per-phase AC files and completing any remaining §12 criteria — 0600 modes, restart persistence, client-module fetch) + AC14 line-count capstone; cumulative suite green via `python3 -m unittest discover -s tests` · REQ-01..REQ-20 · test: is the acceptance suite.
|
||||
- **P3-01** · release-engineer · `tests/test_acceptance_v02.py` · v0.2 acceptance matrix: cumulative suite green via `python3 -m unittest discover -s tests` (all v0.1 files + the v0.2 test files); packaging acceptance consolidated — boot parity through the artifact (first-boot pass printed once, `broker.db` + `broker.audit.log` 0600, UI + client served, silent second boot, register/read flows), reproducibility (two consecutive same-environment builds → identical sha256), self-containment (only runtime state external to the `.pyz`: env/db/audit + materialized assets), stub publish roundtrip + stub release upload green; broker zero-delta — `coreci_broker.py` unchanged at 1390 lines vs `v0.1.5` (AC14 under the D-016..D-018 cap of 1400) · REQ-21, REQ-22, REQ-23 (acceptance) · test: is the acceptance file.
|
||||
|
||||
### Wave 2 — security review (review-only)
|
||||
|
||||
- **P5-02** · security-engineer · review of `coreci_broker.py`, `coreci_broker_client.py`, `reference_tools/`, `tests/**` (no edits — STRIDE/CWE findings via lead-developer) · pre-ship security pass: path traversal incl. URL-encoded forms, pepper never re-printed (second boot, re-register), 0600 modes, token scoping + identity enforcement, audit completeness, static-serving roots · REQ-07, REQ-11, REQ-14, REQ-15 (+ all) · verify: PASS or CWE-cited findings; remediation follows in P5-03.
|
||||
- **P3-02** · security-engineer · review of `coreci_broker.py` (zero-delta verification only — writable by nobody this milestone), `cli/**`, `scripts/**`, `Dockerfile`, `.dockerignore`, `tests/**` (no edits — findings STRIDE-classified, CWE-cited, routed via lead-developer) · v0.2 security pass: secrets in artifacts (no secrets in the `.pyz` or any image layer; `.env` never inside the archive; pepper/admin pass generated at boot; no state baked into the image; build-context hygiene per `.dockerignore`, G-015), digest determinism (zip normalize pass with constant `external_attr` — G-009; sorted tar + fixed TarInfo; `gzip mtime=0`; sorted-key JSON), token discipline (file-only `_env_secrets` path, non-empty assert before first use, never shell env; gitea's anonymous-200 `/v2/token` quirk fails loudly), 307/308 Location handling incl. relative `Location`, traversal + disclosure over materialized asset roots (dot-dir roots vs request-path dot filtering; protected-paths realpaths wherever the state dir sits), shim private-API coupling tracked (G-018), attachment idempotency + bounded-retry escalation paths · REQ-21, REQ-22, REQ-23 (+ all) · verify: findings recorded in the task commit's `---ci---` block; remediation (if any) = the follow-up release-engineer commit dispatched by lead-developer (P3-03).
|
||||
|
||||
### Wave 3 — remediation + docs
|
||||
### Wave 3 — remediation
|
||||
|
||||
- **P5-03** · backend-engineer · `coreci_broker.py`, `tests/**` · remediate P5-02 findings (no-op skip if PASS) · affected REQs per finding · test: full suite re-green.
|
||||
- **P5-04** · backend-engineer · `README.md` · run/deploy notes: standalone + OWUI mount, env/config contract (spec §5.1 + documented additions), first-boot pass discipline, backup guidance (SQLite is the source of truth), client-module bootstrap; documented deviations distilled from D-001..D-015 incl. the spec defects + resolutions (PROJECT.md §Context) · REQ-14, REQ-18 (+ traceability) · test: README covers run/deploy/deviations; suite still green.
|
||||
- **P3-03** · release-engineer · `cli/**`, `scripts/**`, `Dockerfile`, `.dockerignore`, `tests/**` (never broker sources — a P0 broker defect routes through lead-developer dispatch only) · remediate P3-02 findings (no-op skip if PASS) · affected REQs per finding · test: full cumulative suite re-green.
|
||||
|
||||
### Wave 4 — review + reconstruction
|
||||
|
||||
- **P5-05** · lead-developer · `.ciagent/*.md` review notes (no code writes) · multi-persona review of the milestone diff (backend, frontend P2 artifacts, security sign-offs); audit/reconstruction: every REQ-01..REQ-20 maps to green tests + traceable commits · REQ-01..REQ-20 · verify: sign-offs + reconstruction recorded in `---ci---` blocks.
|
||||
- **P3-04** · lead-developer · `.ciagent/*.md` review notes (no code writes) · multi-persona review of the milestone diff (release-engineer artifacts across P1/P2, security sign-off from P3-02/P3-03); audit/reconstruction: every REQ-21..REQ-23 maps to green tests + traceable commits; broker zero-delta verified (1390/1400, AC14); escalation ledger clean or `release_pending` entries documented · REQ-21, REQ-22, REQ-23 · verify: sign-offs + reconstruction recorded in `---ci---` blocks.
|
||||
|
||||
### Wave 5 — ship
|
||||
### Wave 5 — milestone ship
|
||||
|
||||
- **P5-06** · lead-developer · no code writes · owner approval gate recorded; squash-merge `milestone/v0.1-coreci-broker` → `main`; tag `v0.1.5`; first push to gitea `coreci/broker` (public, default branch main — forge repo exists before push, D-010/D-012) + release · REQ-01..REQ-20 · verify: `v0.1.5` on `main`; release visible on the forge.
|
||||
- **P3-05** · lead-developer · no code writes (release-engineer scripts invoked, never edited) · milestone ship, in this exact order (G-012: merge→tag→push BEFORE release attach — a pre-merge release create would mint `v0.2.3` at pre-merge main via gitea's target_commitish semantics): build the final `.pyz` (`python3 scripts/build_cli.py v0.2.3`) → squash-merge `milestone/v0.2-cli-packaging` → `main` → tag `v0.2.3` → push main + tag → **G-013 live smoke (env-gated `PUBLISH_SMOKE=1`, read-only, never in the default suite): real gitea `/v2/` auth challenge, token→JWT exchange, `GET` release-by-tag, Docker Hub index resolve — all must pass before the one live publish** → attach to the `v0.2.3` release with full notes (milestone summary, phases, verification, CLI install/usage + sha256, docker usage) via `scripts/release.py` → daemonless image publish ONCE (`python3 scripts/publish_image.py` → `git.coreci.dev/coreci/broker:v0.2.3` + `latest`, D-019/D-021/D-022) → branch cleanup → REQUIREMENTS/ROADMAP closeout → checkpoint cleared. Publish failure: bounded retry (3) → `release_pending` escalation **with the G-020 closure runbook committed alongside** (exact re-run command on `main` @ `v0.2.3`: build + smoke + `publish_image.py`; close condition = both tags visible as a gitea package) — never blocks the merge, never left open · REQ-21, REQ-22, REQ-23 · verify: `v0.2.3` on `main`; the release carries the `.pyz` + full notes; the image is visible as a gitea package (or escalated per contract + runbook, never blocking); closeout + checkpoint-clear commits recorded.
|
||||
+19
-4
@@ -12,7 +12,7 @@ A single canonical credential flow for autonomous agent tools. The broker mediat
|
||||
|
||||
### Validated
|
||||
|
||||
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09; supersedes v0.4). The full v0.5 spec is distilled into REQ-01..REQ-20 in REQUIREMENTS.md, with acceptance criteria drawn from the spec's §12 list. Clarification decisions D-001..D-011 (recorded in the init commit) plus D-012..D-015 (recorded in the clarify stage) resolve the spec's ambiguities and residual defaults:
|
||||
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09; supersedes v0.4) — distilled into REQ-01..REQ-20 (shipped in v0.1). v0.2 additions validated from owner directives (2026-10-10) — distilled into REQ-21..REQ-23: CLI zipapp packaging, per-release artifact attachment with notes on gitea, once-per-milestone docker publication to the gitea container/package registry. The v0.5 spec's ambiguities were resolved by D-001..D-015 (init + clarify commits, v0.1); v0.2's packaging/distribution ambiguities are resolved as D-019..D-022 (this milestone's clarify stage):
|
||||
|
||||
- Dual-mode token hashing (bcrypt when importable, stdlib scrypt fallback; format-prefixed hashes)
|
||||
- Pluggable creds-source adapter (file-based default; Open WebUI HTTP API for production)
|
||||
@@ -24,7 +24,7 @@ Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09; supersed
|
||||
|
||||
### Active
|
||||
|
||||
Milestone v0.1 (`milestone/v0.1-coreci-broker`) implements the entire v0.5 spec: REQ-01..REQ-20. No requirement is deferred.
|
||||
Milestone v0.2 (`milestone/v0.2-cli-packaging`) — **CLI + distribution only**, no functional changes to the broker: REQ-21 (zipapp CLI packaging), REQ-22 (release assets attached + notes on gitea), REQ-23 (docker container built once per milestone, published as a gitea package). Owner-locked inputs: zipapp `.pyz` form factor; docker publication starts with v0.2 (no retroactive v0.1 image); scope excludes deferred v2 candidates.
|
||||
|
||||
### Out of Scope
|
||||
|
||||
@@ -32,7 +32,7 @@ See REQUIREMENTS.md §Out of Scope — federation, secret versioning, high avail
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Single-file broker**: `coreci_broker.py` under 800 lines (excluding tests, UI HTML, client module, reference tools) — spec §12.14
|
||||
- **Single-file broker**: `coreci_broker.py` under 1400 lines (amended from 800 via owner decisions D-016..D-018; excluding tests, UI HTML, client module, reference tools) — spec §12.14 as amended
|
||||
- **Stdlib-only runtime**: no pip in the target dev environment; bcrypt is the sole optional import (dual-mode hashing); tests use unittest
|
||||
- **SQLite is the source of truth**: single instance, no HA; back it up
|
||||
- **Bind default** `127.0.0.1:8765`; the bind address controls reachability
|
||||
@@ -42,10 +42,21 @@ See REQUIREMENTS.md §Out of Scope — federation, secret versioning, high avail
|
||||
- **Token form**: `base64(sha256(pepper:UUID:user_id:agent_id:sorted_tags))` — 44-char ASCII, no prefix; `BROKER_TOKEN_TTL` default 86400s
|
||||
- **Open WebUI runtime**: bcrypt and httpx exist there; the client module must not depend on either (stdlib urllib) so any tool can import it
|
||||
|
||||
v0.2 (packaging/distribution) constraints:
|
||||
|
||||
- **No pip / stdlib-only build path**: the CLI artifact is a stdlib `zipapp` (`.pyz`); no wheel, no PyInstaller, no pip at build or runtime (D-009)
|
||||
- **No container runtime in the dev environment**: no docker/podman/buildah available — the once-per-milestone publish runs daemonless via a stdlib registry client (D-019)
|
||||
- **Self-contained artifact**: one `.pyz` bundles broker + UI assets + client module; asset roots resolved from env, never `__file__`
|
||||
- **Broker behavior frozen**: first-boot semantics, API surface, and all v0.1 acceptance criteria remain byte-identical when driven through the packaged CLI
|
||||
- **Release secrets discipline**: forge tokens resolved only from `.env*` files via the secrets-loader — never shell environment (v1.8 lesson)
|
||||
- **`broker.db`/`broker.audit.log` remain state-directory-local** — the CLI boots in a runtime state dir (docker: a volume), never writes inside its own archive
|
||||
|
||||
## Context
|
||||
|
||||
CoreCI estate: sibling projects researchx and nextcraft live on gitea (`git.coreci.dev/coreci/*`) and use this same ciagent pipeline. The broker runs on the Open WebUI host as one workspace tool (embedded HTTP server in a daemon thread; also runnable standalone). The `creds-coreci-coo` KB currently mediates four credentials (`coreci/proxmox`, `coreci/gitea`, `coreci/executor` TBD, plus the COO's own login — independent of the broker's `BROKER_ADMIN_PASS`). The existing `proxmox_readonly_tool.py` is updated post-broker to use the public client module; it lives outside this repo, so this repo ships the reference implementation of the tool-side bootstrap instead.
|
||||
|
||||
v0.2 context: milestone v0.1 shipped complete (all REQ-01..REQ-20, `v0.1.5` on main, six gitea releases each carrying a `broker-vX.Y.Z.tar.gz` source archive, zero escalations). Owner directives for v0.2 (2026-10-10): package the broker as a CLI; attach the CLI to every release alongside notes on gitea; build a docker container once per milestone and publish it as a package in the gitea registry. The v0.1 release-attachment mechanism (sanctioned `.env*` token path, per-release asset upload) is the proven base REQ-22 extends.
|
||||
|
||||
Known spec defects resolved by clarification (recorded for traceability): §7.5's in-memory `token_hash` index is impossible with bcrypt salted hashes; §6 claims the tool derives a token every call while §9's flow uses the broker-returned token from the agent KB (the broker derives authoritatively; the tool stores); §9's snippet raises `KeyError` on `data["pepper"]` for re-registrations; the pepper chicken-and-egg (a token is needed to read agent state, but the pepper is needed to derive a token) is fixed by returning the pepper when the caller presents no valid token.
|
||||
|
||||
## Key Decisions
|
||||
@@ -69,4 +80,8 @@ Known spec defects resolved by clarification (recorded for traceability): §7.5'
|
||||
| D-015 | `GET /api/broker/agents/<agent_id>`: App-auth + identity match, metadata-only response (no token, no hash) | §7.4 auth/response unspecified; mirrors state-endpoint identity enforcement; token material never leaves the broker | Binding for v0.1 |
|
||||
| D-016 | AC14 line cap amended: coreci_broker.py < 1000 lines (was < 800) | Owner-approved §12.14 amendment at P2: 11 lines of headroom vs ~275 lines of remaining handler surface; preserves the readable single-file intent without dense-cramming | Binding for v0.1 |
|
||||
| D-017 | D-016 extended: AC14 cap < 1200 lines | Second §12.14 amendment at P3 (owner's D-016 pattern applied with new data): P3-01 lands at 990/1000 with ~150-180 lines of remaining broker surface (App-auth cache, agent state, creds adapter, secrets endpoint); mild compaction reclaims only ~50 — same readability-over-cramming tradeoff the owner chose at D-016; subject to review at the P5 owner approval gate | Binding for v0.1 |
|
||||
| D-018 | D-017 extended: AC14 cap < 1400 lines — final projected amendment | Third §12.14 amendment, pre-P4 per the P3 verify planning signal: 1178/1200 after P3 with 21 lines headroom vs P4's ~105-140 line need (creds adapter + secrets endpoint, the LAST broker-surface additions); P5 adds only excluded artifacts (tests, README). Same owner pattern (D-016/D-017), flagged for the P5 owner approval gate | Binding for v0.1 |
|
||||
| D-018 | D-017 extended: AC14 cap < 1400 lines — final projected amendment | Third §12.14 amendment, pre-P4 per the P3 verify planning signal: 1178/1200 after P3 with 21 lines headroom vs P4's ~105-140 line need (creds adapter + secrets endpoint, the LAST broker-surface additions); P5 adds only excluded artifacts (tests, README). Same owner pattern (D-016/D-017), flagged for the P5 owner approval gate | Binding for v0.1 |
|
||||
| D-019 | Daemonless container publish: stdlib registry client (pull pinned base layers, append broker layer, push to gitea registry) | No docker/podman/buildah in the dev environment; gitea's package registry speaks OCI registry HTTP API v2 — fully automated once-per-milestone publish without a container runtime; stub-testable | Binding for v0.2 |
|
||||
| D-020 | Attach `coreci-broker-vX.Y.Z.pyz` to every release from `v0.2.1` onward | Consistent with v0.1's per-release `broker-vX.Y.Z.tar.gz` asset convention; every tagged release stays independently deployable | Binding for v0.2 |
|
||||
| D-021 | Image tags `git.coreci.dev/coreci/broker:<milestone-release-tag>` + `latest`, pushed once per milestone | Version tag pins the release; `latest` gives docker users a sane default; once-per-milestone honors the owner directive | Binding for v0.2 |
|
||||
| D-022 | Base image `python:3.11-slim` (pinned minor line) | Dev environment runs Python 3.11.2 — pinned minor matches tested stdlib behavior and stays reproducible | Binding for v0.2 |
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
## Overview
|
||||
|
||||
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09; supersedes the v0.4 in-memory draft). "v0.5" is the spec's revision label; the product milestone is v0.1. REQ-01..REQ-20 constitute the full v0.1 scope — nothing is deferred. Acceptance criteria reference the spec's §12 list (AC1–AC14). Clarification decisions D-001..D-011 live in the init commit's `---ci---` block and are distilled in PROJECT.md.
|
||||
Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09). "v0.5" is the spec's revision label. REQ-01..REQ-20 (milestone v0.1) shipped complete as `v0.1.5`. **Milestone v0.2 (cli-packaging)** adds REQ-21..REQ-23 from owner directives (2026-10-10): package the broker as a CLI, attach it to every gitea release with notes, and publish a docker container once per milestone as a gitea package. Scope is CLI + distribution only — no functional changes to the broker. Clarification decisions D-001..D-015 (v0.1) remain binding; D-016+ resolve v0.2's packaging/distribution ambiguities.
|
||||
|
||||
## v1 (Milestone v0.1)
|
||||
## v1 (Milestone v0.1 — complete, shipped as v0.1.5)
|
||||
|
||||
### Auth & Identity
|
||||
|
||||
@@ -50,7 +50,18 @@ Source: Requirements Specification v0.5 (owner: coreci-coo, 2026-10-09; supersed
|
||||
- **REQ-19 Public client module.** `coreci_broker_client.py` — dependency-free (stdlib urllib; no httpx, no bcrypt) — implements the tool-side bootstrap of spec §9: ensure-registered (idempotent), state get/put, secret read with 401 → re-register → retry-once, and never assumes the pepper key is present in register responses (`data.get("pepper")`). Served unauthenticated at `/static/coreci_broker_client.py` so any OWUI tool author can fetch it into their workspace and import it. It contains no secrets.
|
||||
- **REQ-20 Reference tools.** `reference_tools/` ships at least: (a) a proxmox-style reference implementing §11's flow — first call registers and persists state, subsequent calls read the token from state and fetch `coreci/proxmox`, downstream call defaults to a safe dry-run; (b) a minimal showcase of auth-first-then-use-a-second-tool. These are templates for tool authors, wired to the test broker in tests.
|
||||
|
||||
## v2 (Deferred)
|
||||
## v2 (Milestone v0.2)
|
||||
|
||||
### Packaging
|
||||
|
||||
- **REQ-21 CLI packaging (zipapp).** The broker ships as a single-file executable CLI `coreci-broker.pyz` built with stdlib `python3 -m zipapp` from a clean checkout — no pip, no third-party build tooling (D-009 preserved). The archive is self-contained: it bundles `coreci_broker.py`, the `coreci_broker_ui/` pages, and `coreci_broker_client.py`, materializing the UI/static roots into the runtime state directory on boot (paths resolved from env, never `__file__`), so one file serves the full admin UI at `/ui/` and the client module at `/static/coreci_broker_client.py`. Running it boots the standalone server with first-boot semantics identical to REQ-14/REQ-18 (`.env` generation, admin pass printed exactly once, `broker.db` + `broker.audit.log` at mode 0600); `--version` reports the release version. A build script regenerates the artifact reproducibly, and tests boot the built artifact end-to-end.
|
||||
|
||||
### Distribution
|
||||
|
||||
- **REQ-22 Release assets.** Every gitea release from the phase that first builds the CLI (`v0.2.1`, D-020) onward attaches the built artifact `coreci-broker-vX.Y.Z.pyz` alongside the existing source archive. Uploads resolve the forge token only via the sanctioned secrets path (`.env*` files — never shell environment). Release notes document the CLI: artifact download + install (`chmod +x` / `python3`), the CLI surface (boot, `--version`, env contract), and docker usage per REQ-23. Attachment failure follows the ship workflow's bounded-retry then `release_pending` escalation — it never blocks tag+merge.
|
||||
- **REQ-23 Docker package.** The repo ships a `Dockerfile` and a **daemonless stdlib registry client** (D-019): once per milestone, at the final ship, the client pulls the pinned `python:3.11-slim` base image manifest + layers from the public registry (HTTP API v2), appends a layer carrying the CLI artifact, and pushes the image `git.coreci.dev/coreci/broker:<milestone-release-tag>` plus the `latest` alias (D-021/D-022) to the gitea container/package registry — no container runtime required. The container runs the CLI standalone with state on a volume. Registry auth, manifest, layer, and digest handling are stub-tested; publish failure follows bounded-retry escalation — it never blocks the milestone merge.
|
||||
|
||||
## Deferred Beyond v0.2
|
||||
|
||||
None defined. Future candidates held as out-of-scope decisions, not requirements: audit rotation with retention, multiple broker instances, approver-agent delegation, OWUI-KB credential sync.
|
||||
|
||||
@@ -96,4 +107,7 @@ None defined. Future candidates held as out-of-scope decisions, not requirements
|
||||
| REQ-17 | 1 | complete |
|
||||
| REQ-18 | 1 | complete |
|
||||
| REQ-19 | 4 | complete |
|
||||
| REQ-20 | 4 | complete |
|
||||
| REQ-20 | 4 | complete |
|
||||
| REQ-21 | 1 | complete |
|
||||
| REQ-22 | 1 | complete |
|
||||
| REQ-23 | 2 | complete |
|
||||
+174
-65
@@ -1,82 +1,191 @@
|
||||
# RESEARCH — CoreCI Broker
|
||||
|
||||
Date: 2026-10-09 · Milestone: v0.1 · Phase: 0 (pre-execution) · Status: complete
|
||||
Sources: `.ciagent/` PROJECT/REQUIREMENTS/ARCHITECTURE/ROADMAP, git log (init ffdc5392, clarify 53d99e80), docs.openwebui.com + open-webui source at `main` (fetched 2026-10-09). No scope changes; D-001..D-015 reconfirmed; no secrets read or printed.
|
||||
Date: 2026-10-10 · Milestone: v0.2 (cli-packaging) · Phase: 0 (pre-execution) · Status: complete
|
||||
Sources: `.ciagent/` PROJECT/REQUIREMENTS/ARCHITECTURE/ROADMAP/PLAN, `coreci_broker.py` (1390 lines, read in full), git history/tags v0.1.0..v0.1.5, live gitea 1.26.1 API (`git.coreci.dev` swagger.v1.json + releases — read-only, token never printed), live Docker Hub registry + auth.docker.io (anonymous pulls, verified end-to-end), live gitea container-registry auth probes (read-only, no upload sessions created), stdlib source (`/usr/lib/python3.11/zipapp.py`, `urllib/request.py`), and an **end-to-end zipapp prototype booted from the real broker sources** (built + booted as subprocess + driven with urllib; `/tmp` scratch, repo untouched).
|
||||
|
||||
## 1. Environment Constraints (verified facts — do not re-verify)
|
||||
## 0. Carried-Forward v0.1 Findings (still binding)
|
||||
|
||||
- Python 3.11.2 on the dev host; NO pip, NO bcrypt locally → D-001 dual-mode hashing active: stdlib `hashlib.scrypt` in dev/tests, bcrypt in the OWUI runtime; hashes format-prefixed (`$2b$…` / `scrypt$…`). Tests = stdlib unittest (D-009). Pure-Python repo, no package.json.
|
||||
- Forge: gitea `git.coreci.dev`, org `coreci`; release target `coreci/broker` — repo created at first push/ship, public, default branch `main` (D-010/D-012). `GITEA_TOKEN` resolves from `.ciagent/.env.secrets` only (gitignored, mode 0600, never printed).
|
||||
- Stdlib components carrying the runtime: `http.server.ThreadingHTTPServer`, `sqlite3`, `hashlib` (scrypt/sha256), `secrets`, `json`, `base64`, `urllib` (client module), `threading`.
|
||||
- Repo state verified at research time: branches `main`, `milestone/v0.1-coreci-broker`, `phase/00-pre-execution`; `.gitignore` at root; `.ciagent/` holds PROJECT/REQUIREMENTS/ARCHITECTURE/ROADMAP/config.json/CHECKPOINT.json + `.env.secrets` (unread, 0600).
|
||||
- **Environment (v0.1 §1, re-confirmed)**: Python 3.11.2, NO pip anywhere, NO docker/podman/buildah runtime; stdlib-only runtime + build path (D-009 preserved by REQ-21/D-019); tests = stdlib unittest; forge gitea `git.coreci.dev`, owner `coreci`, repo `broker`, public, default branch `main`; `GITEA_TOKEN` resolves from `.ciagent/.env.secrets` only (mode 0600, never printed, `config.security.bash_allowlist.blocked_env_vars` includes GITEA_TOKEN → shell-env tokens are structurally blocked).
|
||||
- **Test harness (v0.1 §6)**: in-process `ThreadingHTTPServer(("127.0.0.1", 0), Handler)` + daemon thread + `tempfile.TemporaryDirectory()` state + stdlib `urllib.request` client + `addCleanup`; 0600 assertions via `st_mode & 0o777 == 0o600`; stub-HTTP precedent: `_StubOwuiHandler` in `tests/test_creds_source.py` (BaseHTTPRequestHandler with class-level fixtures + hit recording) — this is the template for v0.2 stub registry/gitea servers.
|
||||
- **Ship contract (`ci/workflows/ship.md` + config.json)**: every phase ships; release creation best-effort with bounded retries (`ship.max_release_retries: 3`); on exhaustion commit escalation `resolution: "auto"`, `type: "release_pending"`, phase still complete — tag+merge is the gate, `release_blocking: false`; **all secrets resolved ONLY from `.env`/`.env.secrets`/`.env.*` files via the secrets-loader, never shell env** (v1.8 root cause: empty token → gitea answers 404, not 401, masking auth failure).
|
||||
- **OWUI integration surface (v0.1 §4)**: unchanged by v0.2 — the CLI does not touch the `Tools` mount; path resolution from env (never `__file__`) is the v0.1 property that makes zipapp packaging possible at all.
|
||||
- **Static-serving rules that the packaged CLI must preserve**: `_resolve_static_file` rejects `\x00`, `\`, leading `/`, `..` segments, and any *request* path part starting with `.`; roots are real-pathed (`os.path.realpath`) so a dot-named *root* directory is fine (materialized assets live under a dot-dir; only request paths are dot-filtered). `_broker_protected_paths` protects `.env`/db/audit/creds (+ `-journal/-wal/-shm` suffixes) from static reads.
|
||||
|
||||
## 2. Spec Defects & Resolutions (PROJECT.md §Context; decisions in init/clarify commits)
|
||||
## 1. zipapp CLI Packaging (REQ-21)
|
||||
|
||||
| # | Defect | Resolution | Test implication |
|
||||
|---|--------|------------|------------------|
|
||||
| 1 | §7.5 in-memory `token_hash` index impossible with bcrypt (salted hashes neither comparable nor indexable) | D-004: UUID-scoped slow-hash scan + in-memory verified-token cache; SQLite authoritative, cache never authoritative | Cover both paths: cache-miss scan and cache-hit re-check of row status/expiry; both hash formats |
|
||||
| 2 | §6 (tool derives every call) vs §9 (tool stores broker-returned token) contradiction | Broker derives authoritatively; tool stores via state endpoints (ARCHITECTURE.md DataFlow 2–4) | AC4: broker- and tool-side derivation byte-identical (§6 formula); state GET/PUT roundtrip |
|
||||
| 3 | §9 snippet `KeyError` on `data["pepper"]` for re-registrations | D-005: pepper returned only on row-create or tokenless register; REQ-19 mandates `data.get("pepper")` in the client | Re-registration without a pepper key must not crash the client; register still audited |
|
||||
| 4 | Pepper chicken-and-egg (token needed to read state; pepper needed to derive) | Tokenless register returns the pepper — the only KB-loss recovery path (D-005) | Recovery scenario: no valid token → register → pepper returned → tool can re-derive |
|
||||
### 1.1 stdlib mechanics (all VERIFIED empirically on Python 3.11.2)
|
||||
|
||||
## 3. Sibling Conventions (data provided — not fetched)
|
||||
- Command: `python3 -m zipapp <srcdir> -o coreci-broker.pyz --python "/usr/bin/env python3" --compress`.
|
||||
- **`__main__.py` is mandatory** — `zipapp` refuses with `ZipAppError("Archive has no entry point")` without it (the `-m module:fn` form is the only alternative; the shipped `cli/__main__.py` source is cleaner and testable). Files staged: `coreci_broker.py`, `coreci_broker_client.py`, `coreci_broker_ui/**`, `cli/__main__.py`, generated `_cli_version.py`.
|
||||
- **Compression: `zipapp` defaults to ZIP_STORED** (stdlib source line 136: `zipfile.ZIP_DEFLATED if compressed else zipfile.ZIP_STORED`). `--compress` is required for DEFLATED. Measured: ~95 KB content → 27.6 KB pyz compressed vs ~95 KB stored. Use `--compress`.
|
||||
- **Shebang + exec bit**: with `--python`, zipapp writes `#!/usr/bin/env python3\n` as a literal first line, then the zip (zip readers locate EOCD from the file end — leading bytes tolerated), and `chmod +x`es the target (`target.chmod(... | S_IEXEC)`). Both `./coreci-broker.pyz` and `python3 coreci-broker.pyz` verified working; `sys.argv[0]` = the archive path as invoked; `python3 -m zipapp --info` reads the interpreter back.
|
||||
- **Data files are NOT `open()`-able inside the archive** (verified `FileNotFoundError` for `assets/page.html`): zipimport only serves `.py` imports; the broker serves UI/static via **filesystem paths + `open()`** → asset materialization to disk at boot is *mandatory*, not optional. `importlib.resources` cannot provide a directory for `BROKER_UI_ROOT` — zipfile extraction is the only design that fits the broker's config contract. (Not a close call.)
|
||||
- **`__main__` module collision — none**: the archive's `__main__.py` IS the `__main__` module; `import coreci_broker` inside it executes the broker's module-level code once and its `if __name__ == "__main__"` block does NOT fire (verified: side-effect counter runs once, standalone print block silent). Call `coreci_broker.run_standalone()` — no re-execution of module code, no subprocess re-invocation needed.
|
||||
- **zipimport limits**: none bite here — the archive contains only pure-Python repo files; stdlib (`sqlite3`, `http.server`, C extensions) resolves from the host interpreter, and the broker's sole optional import (bcrypt) stays optional.
|
||||
|
||||
- researchx: root = `.ciagent/`, `.gitignore`, `README.md`, `backend/`, `frontend/` — public repo, default branch main.
|
||||
- nextcraft: root = `.ciagent/`, `apps/`, `packages/`, `deploy/`, `scripts/`, pnpm workspace, `Dockerfile`, `.gitea/` — public repo, default branch main.
|
||||
- Implications: flat single-project `.ciagent/` (matches D-010); `README.md` at repo root expected at ship (P5); per-phase tags on the v0.1.x line (D-008); `---ci---` block in every commit; ships via gitea releases once the forge repo exists.
|
||||
### 1.2 Boot shim design (prototype built, booted, verified — recommended as-is)
|
||||
|
||||
## 4. Open WebUI Integration Surface (researched 2026-10-09; docs live, post-v0.10 era)
|
||||
`cli/__main__.py` (staged into the pyz), five steps:
|
||||
|
||||
### Workspace Tools — `Tools` class surface (docs: /features/extensibility/plugin/tools/development)
|
||||
- Single Python file; frontmatter docstring keys: `title`, `author`, `author_url`, `git_url`, `description`, `required_open_webui_version`, `requirements`, `version`, `licence`. Broker ships `requirements:` empty — stdlib-only + bcrypt already present in the OWUI runtime (PROJECT.md Constraints).
|
||||
- `Tools` class rules: `__init__` MUST set `self.valves = self.Valves()` (stored admin valves load only onto an instance that has a `valves` attribute); optional `Valves`/`UserValves` (pydantic BaseModel); every public callable is exposed to the model → helpers must be `_`-prefixed; method docstrings (reST `:param name:` form) + type hints generate the JSON schema; return str/dict/list; async methods preferred.
|
||||
- **Lifecycle: there is NO `activate()`/`deactivate()` hook for Tools.** The loader (`backend/open_webui/utils/plugin.py`) execs the source into a `tool_{id}` module in `sys.modules`, instantiates `module.Tools()` at load (create/update) and caches it on `app.state.TOOLS`; re-exec occurs on source change; module-top-level side effects persist, and disabling a tool does NOT undo them. ⇒ ARCHITECTURE.md's "mounts the server at activation" maps to constructor/module-load time: start the daemon-thread `ThreadingHTTPServer` in `__init__` (or module top level) behind a once-per-process idempotency guard.
|
||||
- Loader sets the module's `__file__` to a temp file deleted after exec ⇒ the broker must resolve DB/audit/env paths from config/env vars, never from `__file__`.
|
||||
- User context: OWUI provides no `self.user` attribute; the per-call reserved arg `__user__` is a dict (`id`, `email`, `name`, `role`, plus `valves` when UserValves defined) (docs: /features/extensibility/plugin/development/reserved-args). Broker admin auth (`BROKER_ADMIN_PASS`) is independent of OWUI identity; ARCHITECTURE.md's "self.user supplies admin-agnostic context" is the broker's own convention — the broker's Tools class can stash `__user__` or ignore it. Nuance only, no architecture change.
|
||||
- Background infra: OWUI's documented pattern is `asyncio.create_task` (Under the Hood §Patterns 4); the broker's stdlib analog is `threading.Thread(target=serve_forever, daemon=True)` — same once-per-process guard required (module re-exec on edits).
|
||||
- Asset serving: a tool may register FastAPI routes via `app.add_api_route` with an idempotency guard (Under the Hood §Patterns 3) — but such routes are unauthenticated unless gated. The broker deliberately serves UI + `/static/` from its OWN `ThreadingHTTPServer` on `127.0.0.1:8765` — no FastAPI coupling; `/static/coreci_broker_client.py` is intentionally unauthenticated (REQ-19).
|
||||
- Security posture: workspace tools execute unsandboxed Python in the server process (docs Plugin Security Warning) — the broker mount inherits that trust boundary.
|
||||
1. **`--version` first, before importing the broker**: parse `sys.argv[1:]`; if `--version`, `import _cli_version; print(_cli_version.VERSION); sys.exit(0)`. Version comes from a build-time-generated `_cli_version.py` (`VERSION = "vX.Y.Z"` written by `scripts/build_cli.py` from the tag) — chosen over a zip-comment (opaque, fragile, still needs a parser). No broker import on the `--version` path.
|
||||
2. **Resolve the state dir exactly like the broker**: `env_path = os.environ.get("BROKER_ENV_PATH") or ".env"`; `state_dir = dirname(abspath(env_path)) or cwd`. Broker path resolution (read from code, authoritative): `BROKER_ENV_PATH` default `.env`, `BROKER_DB_PATH` `broker.db`, `BROKER_AUDIT_LOG_PATH` `broker.audit.log`, `BROKER_CREDS_FILE` `data/creds.json`, `BROKER_UI_ROOT` `coreci_broker_ui`, `BROKER_STATIC_ROOT` `.` — **all CWD-relative when relative**; process env beats `.env` beats defaults. The CLI therefore needs no path changes at all: relative paths land in the CWD (docker: `WORKDIR /state` + volume).
|
||||
3. **Materialize assets**: `zipfile.ZipFile(sys.path[0])` (inside a zipapp, `sys.path[0]` is the `.pyz` itself — verified) → extract `coreci_broker_ui/**` → `<state>/.coreci_broker_assets/<VERSION>/ui/` and `coreci_broker_client.py` → `<...>/static/`. Write temp + `os.replace` (atomic vs a concurrently-serving older instance). **Versioned asset dir**: upgrades materialize a new dir; old dirs are left in place (another instance may still serve them; ~30 KB each — cleanup explicitly NOT done in v0.2). Overwrite-same-version on every boot (self-repair, assets are immutable inputs; extraction happens before this instance's server starts).
|
||||
4. **Override precedence — user wins**: `import coreci_broker`; parse the user `.env` with the broker's own `coreci_broker._parse_env_file(env_path)`; set `os.environ["BROKER_UI_ROOT"]` / `["BROKER_STATIC_ROOT"]` only when the key is in **neither** `os.environ` nor the `.env` values. Net precedence: process env → `.env` → shim defaults → broker defaults — identical to the broker's own contract, achieved with **zero broker changes**.
|
||||
5. **Hand off**: `coreci_broker.run_standalone()` (foreground loop + KeyboardInterrupt shutdown; prints `coreci-broker listening on http://<host>:<port>` with the *actual bound address* — `BROKER_BIND=127.0.0.1:0` gives an ephemeral port via `_parse_bind` (`"0".isdigit()` → True) and the harness parses the printed URL).
|
||||
|
||||
### Knowledge Bases API — read path (source-verified: `backend/open_webui/routers/knowledge.py`, `files.py` at main, 2026-10-09)
|
||||
- Auth: `Authorization: Bearer <API_KEY>` (sk-… key from Settings→Account; JWT also accepted; `x-api-key` alt header via `CUSTOM_API_KEY_HEADER`) — feeds `BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY` (REQ-08/D-002). Exact layout deferred to P4 per ARCHITECTURE.md §Notes; candidates below.
|
||||
- Router `/api/v1/knowledge` (all endpoints `get_verified_user`):
|
||||
- `GET /api/v1/knowledge/?page=N` — paginated KB list (30/page) → `{items:[{id,name,description,write_access,…}], total}`
|
||||
- `GET /api/v1/knowledge/search?query=&page=` — same shape; candidate for finding `creds-coreci-coo` by name
|
||||
- `GET /api/v1/knowledge/{id}` — KB detail + `files[]` (per-file metadata) + `write_access`
|
||||
- `GET /api/v1/knowledge/{id}/files?query=&include_content=&page=` — file list; `include_content=true` embeds extracted content (docs flag it as expensive)
|
||||
- `GET /api/v1/knowledge/{id}/files/pending`, `GET /api/v1/knowledge/{id}/export` (zip) — auxiliary
|
||||
- Write ops exist (`/create`, `/{id}/update`, `/{id}/file/add|update|remove|move`, `/{id}/files/batch/add`, `/{id}/delete`, `/{id}/reset`, `/{id}/sync/*`, `/{id}/dirs/*`) — the broker never calls them (read-only, REQ-08).
|
||||
- Router `/api/v1/files`:
|
||||
- `GET /api/v1/files/{id}` → FileModel; `data.content` holds extracted text
|
||||
- `GET /api/v1/files/{id}/data/content` → `{"content": "<extracted text>"}` — JSON form, best fit for credential JSON files
|
||||
- `GET /api/v1/files/{id}/content` → raw bytes (FileResponse)
|
||||
- `GET /api/v1/files/{id}/process/status` → `{"status": "pending"|"completed"|"failed"}`
|
||||
- `GET /api/v1/files/?page=` (`content=true` default), `GET /api/v1/files/search?filename=<glob>`
|
||||
- Candidate P4 flow: search KB by name → map path→file (filename convention TBD in P4) → `GET /api/v1/files/{file_id}/data/content` → parse JSON → `{path, data, version, updated_at}`. Note: OWUI exposes `meta.size`/`updated_at` but no version field → `version` may be synthesized broker-side (P4 detail; REQ-08 response shape unchanged).
|
||||
**Verified end-to-end prototype results** (real broker sources, subprocess, urllib): first boot printed `BROKER_ADMIN_PASS=…` exactly once then the listening line; `/healthz` 200; `/ui/login` + `/ui/dashboard` 200 (served from materialized assets); `/static/coreci_broker_client.py` 200 byte-exact (6180 B); `.env` + `broker.db` + `broker.audit.log` created mode 0600 in the state dir; second boot silent (no pass re-print); state dir fully independent of CWD (env paths pin it).
|
||||
|
||||
## 5. Security Prior Art
|
||||
### 1.3 Determinism / reproducibility
|
||||
|
||||
- Peppered derivation + slow-hash storage: the token derives from a 256-bit random pepper (`BROKER_INSTANCE_PEPPER`, 32 bytes), so an attacker holding `token_hash` faces an offline brute force over the pepper — infeasible. Online guessing is rate-limit-free (spec §14: trusted network + `127.0.0.1` bind), but each guess costs one scrypt/bcrypt op, and the UUID-scoped scan bounds verification work (D-004). Storing only the salted slow hash (never the plaintext token) follows OWASP password-storage guidance; format prefixes keep verification format-aware (D-001).
|
||||
- Append-only JSONL audit: one JSON object per line; flush-per-write (+ `os.fsync` for the process-restart case) is affordable at the broker's write volume (register/read/admin actions) and preserves the no-loss guarantee; file mode 0600; no rotation — manual `mv` (D-007, spec §13 Q1). Exact flush/fsync choice is a P1 plan decision, not research.
|
||||
- Session cookie `broker_session`: HttpOnly + SameSite=Lax (D-013); no Secure flag (loopback HTTP by default); Path scoped to the broker.
|
||||
- Path traversal on `/secrets/<path>`: URL-decode (`urllib.parse.unquote`) BEFORE validation; reject `..` segments, absolute paths, backslashes, NUL; the scope check `path == "coreci/" + agent_id` runs on the decoded value (REQ-07). Static serving of `/static/` and UI must likewise resolve against a fixed root and reject traversal.
|
||||
- OWUI-runtime boundary: broker binds `127.0.0.1:8765` — bind address controls reachability (PROJECT.md Constraints). The OWUI process hosts both the tool (broker mount) and the KB the broker reads; `BROKER_OWUI_API_KEY` is the only OWUI-side credential the broker holds, and it lives in the broker `.env` (0600 discipline, REQ-15).
|
||||
- **Raw `zipapp` is NOT reliably byte-reproducible**: zip entries carry the source files' mtimes (`z.write` → `ZipInfo.from_file`), no SOURCE_DATE_EPOCH handling exists anywhere in zipapp, and entry order is `rglob('*')` = `os.scandir` order (unsorted, filesystem-dependent). My two identical staging-dir builds hashed equal *only because* mtimes and scandir order happened to match — a fresh `git clone` (checkout-time mtimes) or a different filesystem breaks it.
|
||||
- **Fix — normalization post-pass (verified byte-identical across rebuilds)**: after zipapp builds the artifact, the build script reads it (shebang line + zip body), rewrites the zip with entries **sorted by name**, `date_time=(1980,1,1,0,0,0)` (zip epoch minimum), `ZIP_DEFLATED`, preserved `external_attr`, and re-prepends the shebang (+`chmod +x`). Verified: two builds → identical sha256; normalized pyz still executes (`--version` OK). Alternative (rejected): `os.utime`-ing the staging tree pre-build — fixes mtimes but not entry order; the post-pass fixes both and keeps `python3 -m zipapp` as the format mechanism (the normalized file is still "shebang + zip", the zipapp format).
|
||||
- Reproducibility test: `test_build_cli.py` builds twice into separate dirs, asserts identical sha256, and asserts the embedded `_cli_version.VERSION`.
|
||||
|
||||
## 6. Testing Prior Art
|
||||
### 1.4 Defects / risks (zipapp)
|
||||
|
||||
- In-process server harness (stdlib, matches D-009/REQ-18): `ThreadingHTTPServer(("127.0.0.1", 0), Handler)` for an ephemeral port in `setUp`; `threading.Thread(target=server.serve_forever, daemon=True)`; drive requests with `urllib.request` against `http://127.0.0.1:{port}`; `shutdown()` + `server_close()` via `addCleanup`.
|
||||
- Temp state: `tempfile.TemporaryDirectory()` per test for `broker.db`, `broker.audit.log`, `.env`, `data/creds.json`; point the broker at them via env vars (`BROKER_ENV_PATH`, `BROKER_CREDS_FILE`, audit path).
|
||||
- 0600 modes: `os.stat(path).st_mode & 0o777 == 0o600` after first boot; create with `os.open(path, O_CREAT|O_WRONLY, 0o600)` (umask-independent) in the broker, so the test just asserts.
|
||||
- AC14 line-count test (REQ-17): `coreci_broker.py` is a single file — tests/UI HTML/client module/reference tools are separate files, so a plain line count of that one file is the spec's measure (< 800, spec §12.14); test reads the file and asserts `sum(1 for _ in f) <= 800` (wc -l-style, no comment stripping).
|
||||
- TTL tests: set `BROKER_TOKEN_TTL` to 1–2 s and `time.sleep`, or inject an expired `token_expires_at` directly in SQLite (faster, deterministic).
|
||||
- Dual-hash tests: both `scrypt$…` and `$2b$…` verification paths; CI can only execute the scrypt path (no bcrypt locally) — the bcrypt path's format-awareness is tested via the prefix and mocked/verified against the format contract.
|
||||
- **RISK** — stale-asset drift if a *user* sets `BROKER_UI_ROOT`/`BROKER_STATIC_ROOT` in `.env`: honored (by design, documented in notes) — the archive still carries the assets, they're just unused. Tests cover the override path.
|
||||
- **RISK** — first-boot extraction torn reads if two instances share a state dir and the same version: mitigated by temp+`os.replace`; multi-instance sharing one SQLite remains unsupported (as v0.1).
|
||||
- **NOTE** — shebang `env python3` may resolve to a non-3.11 python on exotic hosts; the broker needs 3.7+ features in practice but is tested on 3.11 only — release notes state "Python 3.11+ required" (D-022 alignment).
|
||||
- **NOTE** — `sys.path[0]` is the archive only when run as a zipapp; the shim is not meant to run from an unzipped checkout (guard: it can `os.path.isdir` check and fall back to repo-relative roots, or simply document zipapp-only — planner's choice, zero-cost either way).
|
||||
- **NOTE** — assets land under a dot-directory (`.coreci_broker_assets`) — hidden-dir hygiene in the state dir; broker dot-filtering applies to request paths, not roots (verified `_resolve_static_file`).
|
||||
- **NOTE** — AC14 (line count) counts `coreci_broker.py` ONLY (REQ-17); the shim/build scripts are separate files → v0.2 needs **zero broker lines**; broker is 1390/1400 — only 10 lines of headroom anyway, so zero-change is also a hard constraint, and it is satisfied (verified: `run_standalone`, `_parse_env_file`, `boot` are importable and sufficient as-is).
|
||||
|
||||
## 7. Research Conclusions
|
||||
## 2. Gitea Release Assets (REQ-22)
|
||||
|
||||
- P1 (skeleton): bootstrap `.env` + first-boot generation (pepper/admin pass printed exactly once — log once, never again on second boot); SQLite schema per spec §5.2 at 0600; audit writer (JSONL, 0600); dual-mode hasher with format prefixes; token engine pure-function of (pepper, inputs); ThreadingHTTPServer + `{"error, message}` router; OWUI mount = `Tools.__init__` starts the daemon-thread server behind a once-per-process guard (no activate() hook exists), `self.valves = self.Valves()` required, `_`-prefixed helpers; standalone `__main__`; path resolution from env, never `__file__`; keep the 800-line budget from the first commit (UI/client/reference are separate files).
|
||||
- P2 (admin plane): cookie `broker_session` HttpOnly+SameSite=Lax; HTTP Basic + session cookie both accepted on all admin API endpoints; four pages + dashboard served from `coreci_broker_ui/` on the same port; audit query streams the file line-by-line with `next_since` cursor (limit default 100/max 1000).
|
||||
- P3 (tool auth): register semantics per D-005 (idempotent re-derive, overwrite on input change, pepper on row-create-or-tokenless); verification per D-004 (UUID-scoped slow-hash scan; cache hit re-checks row status/expiry); `GET /agents/<id>` per D-015 (metadata-only); state endpoints with identity enforcement; `BROKER_AGENT_STATE_BACKEND=internal` default.
|
||||
- P4 (secrets + ecosystem): `OwuiApiCredsSource` builds on the KB read endpoints in §4 (Bearer key; `/{id}` + `/{id}/files` for mapping, `/api/v1/files/{id}/data/content` for content); `FileCredsSource` default `data/creds.json` (D-014); client module stdlib-urllib-only, `data.get("pepper")`, served unauthenticated at `/static/`; reference tools with safe dry-run default.
|
||||
- P5 (review + ship): AC1–AC13 behavioral + AC14 line-count; security checklist (traversal incl. URL-decoded forms, pepper never re-printed, 0600 modes, token scoping, audit completeness); README with documented deviations; ship to gitea (repo exists before first push).
|
||||
- ARCHITECTURE.md verification: **no assumption changes required.** The adapter-contract note (OWUI KB-file layout resolved in P4, ARCHITECTURE.md §Notes) stands — candidate endpoints captured in §4. One nuance for P1, not a defect: "mounts the server at activation" maps to constructor-time in the actual OWUI loader (no activate()/deactivate() lifecycle for Tools), and OWUI supplies `__user__` per call rather than a `self.user` attribute — the broker's Tools class owns that convention.
|
||||
- No genuine blockers found. Environment, OWUI surface, and forge conventions all support the v0.1 scope as specified.
|
||||
### 2.1 Verified API shapes (live gitea 1.26.1, `git.coreci.dev/swagger.v1.json` + working calls)
|
||||
|
||||
- **List**: `GET /api/v1/repos/coreci/broker/releases?limit=20` → `[{id, tag_name, name, body, assets: [{id, name, size, download_count, browser_download_url}], draft, prerelease, created_at}]`. **By tag**: `GET /api/v1/repos/coreci/broker/releases/tags/{tag}` (200 or 404). **Latest**: `GET .../releases/latest`.
|
||||
- **Create**: `POST /api/v1/repos/{owner}/{repo}/releases` — JSON `CreateReleaseOption` `{tag_name, name, body, draft, prerelease, target_commitish, tag_message}` (schema verified from swagger).
|
||||
- **Attach asset**: `POST /api/v1/repos/{owner}/{repo}/releases/{id}/assets` — **consumes `multipart/form-data`** (or `application/octet-stream`); params verified from swagger: `id` (path, int64), `name` (query), **`attachment` (formData, type file)** — the multipart field name is `attachment`. Response: attachment JSON with `id`, `name`, `size`, `browser_download_url`.
|
||||
- **Auth**: `Authorization: token <GITEA_TOKEN>` (verified working on live forge; gitea also accepts Basic `user:token` — use the `token` form).
|
||||
- **Asset size**: gitea `[attachment] MAX_SIZE` default is 4 MB — the pyz is ~30 KB (deflated) → no issue; NOTE if server policy changes, attachment failure escalates through the bounded-retry path rather than blocking.
|
||||
- **Multipart with stdlib** (no `requests`): hand-rolled body, ~15 lines — boundary string, `Content-Disposition: form-data; name="attachment"; filename="coreci-broker-vX.Y.Z.pyz"`, `Content-Type: application/octet-stream`, `?name=` query param; gitea parses with standard Go multipart — boundary is ours to choose. Test against a stub gitea (§2.4).
|
||||
|
||||
### 2.2 v0.1 attachment forensics (how it was actually done)
|
||||
|
||||
- All six releases v0.1.0..v0.1.5 carry `broker-vX.Y.Z.tar.gz` (29 KB → 99 KB, growing with the codebase). Downloaded v0.1.5's asset: **57 members = the complete working tree including `.ciagent/`** (53 git-tracked files + 4 dir entries; matches `git ls-tree -r v0.1.5` exactly, plus directory entries), **no top-level prefix**, uniform `mtime 1791646214` (= the release timestamp), `uid 0/uname root`, modes 0664/0775.
|
||||
- Interpretation: the archive was produced by the **ship harness's packaging infrastructure** (ship.md step "Create distribution packages" via the `coreci build/package/release upload` CLI — **not installed in this dev environment**; the root-owned uniform timestamps indicate a container-side packaging step) — i.e. **ad-hoc from the repo's perspective: no repo-owned script exists**. The only repo-visible traces are ship.md's generic instructions.
|
||||
- Conclusion for v0.2: REQ-22 formalizes exactly this — a repo-owned `scripts/release.py` (plus `scripts/build_cli.py`) replaces the absent harness path. The v0.1 *conventions* carry over: per-release asset naming (now `coreci-broker-vX.Y.Z.pyz`, D-020), notes model (§2.3), and the bounded-retry contract (ship.md). The existing source-archive `broker-vX.Y.Z.tar.gz` convention continues to be the harness's own product (best-effort); the repo script owns the `.pyz` (+ digest).
|
||||
|
||||
### 2.3 Release notes content model (v0.1.5 body verified; v0.2 sections)
|
||||
|
||||
v0.1.5 body sections: `Milestone summary` → `Phases` (P0..P5 with tags) → `Verification` (test counts, security review, audit) → `Deployment`. v0.2 releases add the REQ-22-mandated CLI + docker documentation; recommended body structure (build-able by `scripts/release.py` from PLAN/ROADMAP/requirements data):
|
||||
|
||||
1. `Milestone summary` — one paragraph (carries the v0.1 style).
|
||||
2. `Phases` — phase list with tags (auto from `git log vX.Y.Z-1..vX.Y.Z`/ROADMAP).
|
||||
3. `Verification` — test count, security review verdict, escalations.
|
||||
4. `CLI install + usage` — download `coreci-broker-vX.Y.Z.pyz` from this release's assets; `chmod +x` then `./coreci-broker-vX.Y.Z.pyz` or `python3 coreci-broker-vX.Y.Z.pyz`; `--version`; env contract table pointer (README); first-boot pass discipline (printed once, 0600 state files); state-dir anchoring (`BROKER_ENV_PATH`); **sha256 of the .pyz embedded** (plus optional `.sha256` sidecar asset — planner's call; lean default: digest in notes only).
|
||||
5. `Docker usage` (REQ-23/D-021) — `docker pull git.coreci.dev/coreci/broker:vX.Y.Z` (and `latest`); `docker run -p 8765:8765 -v broker-state:/state …`; state-on-volume contract.
|
||||
|
||||
### 2.4 `scripts/release.py` design (stdlib, ~200 lines)
|
||||
|
||||
- **Secrets**: token via a shared file-reading loader `scripts/_env_secrets.py` (~20 lines: parse `.ciagent/.env.secrets` `KEY=VALUE`, strip quotes — mirror the broker's `_parse_env_file` semantics). NEVER `os.environ` (GITEA_TOKEN is in `blocked_env_vars`). **Assert token non-empty before any API call** (v1.8 lesson: empty token → gitea 404, masking auth failure).
|
||||
- Flow: (1) resolve tag → `GET /releases/tags/{tag}`; 404 → `POST /releases` (create with notes body, `draft: false`, `prerelease: false`); (2) `GET .../releases/{id}/assets` — if `coreci-broker-vX.Y.Z.pyz` already present with matching size → **idempotent success** (no duplicate asset on bounded-retry re-run); (3) POST multipart attach (field `attachment`, `?name=`); (4) bounded retry: 3 attempts with backoff on URLError/5xx/HTTPError; on exhaustion → exit non-zero with a machine-readable marker → ship harness commits `release_pending` escalation; **never blocks tag+merge**.
|
||||
- **Stub-test** (`tests/test_release_upload.py`): `_StubGiteaHandler(BaseHTTPRequestHandler)` on `ThreadingHTTPServer(("127.0.0.1", 0))` (v0.1 `_StubOwuiHandler` precedent): routes `GET /api/v1/repos/{o}/{r}/releases/tags/{tag}` (200 fixture / 404), `POST .../releases` (assert body JSON), `POST .../releases/{id}/assets` (parse multipart with a stdlib parser — e.g. `email` or manual boundary split — assert field name `attachment` + `?name=` + body bytes), plus failure-modes: 500×3 → release_pending path; existing asset → no second upload.
|
||||
|
||||
### 2.5 Defects / risks (release assets)
|
||||
|
||||
- **RISK** — multipart hand-rolling bugs (boundary quoting, CRLF): mitigated by stub-gitea test with byte-exact assertions.
|
||||
- **RISK** — release may not exist when the attacher runs (ship harness creates it best-effort): create-or-get flow covers both orders; idempotent re-attach covers partial failure.
|
||||
- **NOTE** — 4 MB attachment cap (default) — pyz far under; failure would surface through bounded-retry, not silently.
|
||||
- **NOTE** — `GET /releases?limit=50` paging: prefer `/releases/tags/{tag}` (exact, no paging).
|
||||
|
||||
## 3. Daemonless OCI Build + Push (REQ-23, D-019/D-021/D-022)
|
||||
|
||||
### 3.1 PULL from Docker Hub (verified live, anonymous, end-to-end)
|
||||
|
||||
- `GET https://registry-1.docker.io/v2/` → 401 `WWW-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"`; anonymous rate limit header `ratelimit-limit: 100;w=3600` (100 req/h) — ample for once-per-milestone.
|
||||
- `GET https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/python:pull` → 200 `{token, access_token, expires_in, issued_at}` — anonymous OK (verified).
|
||||
- `GET /v2/library/python/manifests/3.11-slim` with `Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json` → **200 `Content-Type: application/vnd.oci.image.index.v1+json` — `python:3.11-slim` IS an OCI image index (16 children: 8 platform manifests + 8 attestation manifests `unknown/unknown`)**. The client MUST resolve the `linux/amd64` child manifest by digest and fetch it separately.
|
||||
- Verified nuance: Docker Hub currently returns the index **even without Accept** (and even Accepting only schema2) — but the client still sends the full Accept set (contract correctness) and branches on the response `"manifests"` key (index) vs `"layers"` (manifest), not on hope.
|
||||
- amd64 child (verified, digest drifts upstream — resolve at publish time and record it): `application/vnd.oci.image.manifest.v1+json`, config `application/vnd.oci.image.config.v1+json` (`Cmd: ["python3"]`, Env PATH/LANG/GPG_KEY/`PYTHON_VERSION=3.11.17`/PYTHON_SHA256 — base runs 3.11.17 vs dev 3.11.2, same pinned minor line per D-022), **4 layers** `application/vnd.oci.image.layer.v1.tar+gzip` (≈29.8 MB + 1.3 MB + 14.5 MB + 250 B ≈ 45.7 MB total).
|
||||
- `GET /v2/library/python/blobs/<digest>` → blob bytes (layers + config). All flows urllib-only.
|
||||
|
||||
### 3.2 Auth to the gitea registry (verified live; username+token shapes pinned)
|
||||
|
||||
- `GET https://git.coreci.dev/v2/` → 401 `WWW-Authenticate: Bearer realm="https://git.coreci.dev/v2/token",service="container_registry",scope="*"`.
|
||||
- Token endpoint: `GET https://git.coreci.dev/v2/token?service=container_registry&scope=repository:coreci/broker:pull,push` with **`Authorization: Basic base64(<username>:<GITEA_TOKEN>)`** → 200 `{"token": "<JWT>"}` (HS256; `exp-nbf = 86400` s = 24 h).
|
||||
- **Verified**: the Basic **username is ignored** — `coreci`, `nosuchuser123`, and empty all return the same JWT (UserID 57, scopes `write:activitypub,…,write:package,…` — the existing GITEA_TOKEN has package-write). The token (as password) identifies the user. Docker login convention: username = the gitea user, password = the token; the client sends any non-empty username.
|
||||
- **Verified quirk (must handle)**: the token endpoint answers **200 even anonymously** with an unusable JWT (`UserID: -1`, empty scope). A 200 from `/v2/token` proves nothing about push rights — the client must (a) insist on Basic credentials being present, (b) check the issued token is non-empty, and let real operations fail loudly.
|
||||
- Verified: `Authorization: Bearer <JWT>` on `/v2/` → 200; `HEAD /v2/coreci/broker/blobs/sha256:0…0` → 404 (existence checks work); `GET /v2/coreci/broker/tags/list` → 404 `NAME_UNKNOWN` (the container package doesn't exist until first push — expected today).
|
||||
|
||||
### 3.3 PUSH flow + media-type choice
|
||||
|
||||
- **Manifest media type: OCI image manifest end-to-end** — `application/vnd.oci.image.manifest.v1+json`, config `application/vnd.oci.image.config.v1+json`, layers `application/vnd.oci.image.layer.v1.tar+gzip`. Rationale: (1) the pulled base child manifest already uses exactly these descriptor types — reusing its 4 layer digests **verbatim** keeps descriptor/mediaType/blob consistency; (2) gitea 1.26 accepts docker schema2 and OCI manifests — OCI is the modern default and what upstream itself serves; (3) Docker CLI ≥ 18.06 and podman/buildah pull OCI manifests from any v2 registry. **Do NOT push an index/manifest-list**: one linux/amd64 manifest per tag; docker pulls it directly.
|
||||
- **Blob uploads (OCI dist-spec, gitea-conformant)**: primary = **monolithic** `POST /v2/coreci/broker/blobs/uploads/?digest=sha256:<hex>` with the blob as body → 201 (single round trip); fallback = two-step `POST /v2/<name>/blobs/uploads/` → 202 + `Location` → `PUT <Location>?digest=…` → 201. Always `HEAD /v2/<name>/blobs/<digest>` first — 200 → skip upload (base layers re-run-proof: a partially-failed publish resumes cheaply). **Base layers must be re-uploaded to gitea** (~45.7 MB once per milestone) — gitea cannot mount blobs from Docker Hub; acceptable per D-019's once-per-milestone cadence.
|
||||
- **Redirect trap (verified in CPython 3.11 source)**: `HTTPRedirectHandler.redirect_request` only follows GET/HEAD (301/302/303 POST→GET); **307/308 on a PUT/POST raise HTTPError** — stdlib urllib does NOT follow method-preserving redirects. The client must implement explicit Location-following (read `Location`, `urljoin` against the registry base — gitea's `Location` may be relative or absolute — re-issue the request with method+body). Gitea default local storage answers directly, but the client is correct against any conformance-grade registry.
|
||||
- **Manifest PUT**: `PUT /v2/coreci/broker/manifests/<tag>` with `Content-Type: application/vnd.oci.image.manifest.v1+json` → 201. Tags: `<milestone-release-tag>` (e.g. `v0.2.3`) **and** `latest` — two PUTs of the same manifest (D-021).
|
||||
|
||||
### 3.4 Layer + config + manifest construction (exact recipe)
|
||||
|
||||
1. **Broker layer**: deterministic tar (`tarfile`, entries sorted, fixed `TarInfo` metadata — uid/gid 0, uname/gname fixed, `mtime=0`/1980-epoch, file mode 0644) containing `/app/coreci-broker.pyz` (the built artifact). Gzip with `gzip.compress(data, compresslevel=9, mtime=0)` — **`mtime=0` is mandatory** (default `None` = wall clock → nondeterministic digests); fix the compresslevel in code (stable per build).
|
||||
2. Digests: **layer digest** = sha256 of the gzip blob; **diff_id** = sha256 of the *uncompressed* tar.
|
||||
3. **New config blob**: copy the pulled base config; append the new diff_id to `rootfs.diff_ids`; set `config.Entrypoint = ["python3", "/app/coreci-broker.pyz"]`, `config.Cmd = []` (so `docker run image --version` appends `--version` through the entrypoint), `config.WorkingDir = "/state"`, keep base `Env`; optional `config.Labels = {"org.opencontainers.image.version": "<tag>"}`. Serialize deterministically (`json.dumps(..., sort_keys=True, separators=(",", ":"))`) → reproducible config digest.
|
||||
4. **Push order**: HEAD all blobs (4 base layers, new layer, new config) → monolithic POST each missing one → PUT manifest referencing the base's 4 layer descriptors **verbatim** (original digests + mediaTypes) + the new layer descriptor + the new config descriptor. Record the resolved base index digest + amd64 child digest + all blob digests in the release notes (provenance; also `--base-manifest` override argument for exact re-publishes, default = tag resolution with documented upstream drift).
|
||||
5. Image name: `git.coreci.dev/coreci/broker` (gitea registry = `owner/name`; the package links to the repo by owner+name). gitea creates the package at first push (verified `NAME_UNKNOWN` until then).
|
||||
|
||||
### 3.5 Dockerfile role + mapping limits (documentation; daemonless client = source of truth)
|
||||
|
||||
- The Dockerfile is for users WITH docker; the daemonless client is what the pipeline publishes. **Equivalence required, byte-identity not.** Self-contained Dockerfile (no pre-built artifact in build context): `FROM python:3.11-slim`; `COPY coreci_broker.py coreci_broker_client.py coreci_broker_ui/ cli/ /app/src/`; `ARG VERSION=dev`; generated `_cli_version.py` via build arg; `RUN python3 -m zipapp /app/src -o /app/coreci-broker.pyz --python "/usr/bin/env python3" --compress`; `WORKDIR /state`; `ENTRYPOINT ["python3", "/app/coreci-broker.pyz"]`; `EXPOSE 8765`.
|
||||
- Mapping: same base family (tag drift upstream ≠ the pinned digests the client resolved at publish time), same entrypoint, same pyz content, same `/state` state anchoring; differences: docker-build layering (COPY+RUN blobs vs verbatim base layers + one broker layer) and version string provenance (build-arg vs build-time injection). Document both in the Dockerfile header.
|
||||
|
||||
### 3.6 Stub-test plan (`tests/test_publish_image.py`; v0.1 stub precedent)
|
||||
|
||||
- `_StubRegistryHandler` on an ephemeral port: `GET /v2/` → 401 + `WWW-Authenticate` (realm = stub `/token`); `GET /token` → `{"token":"stub"}` (plus the anonymous-empty-token variant → client must fail cleanly); `GET /v2/<name>/manifests/<ref>` → fixture index (Accept-checked) / child manifest; `GET /v2/<name>/blobs/<digest>` → fixture bytes; `HEAD` blob → 200/404; `POST /v2/<name>/blobs/uploads/` (+`?digest=`) → 201 monolithic (assert body sha256 == digest param) or 202+`Location`; `PUT .../uploads/<uuid>?digest=` → 201; `PUT /v2/<name>/manifests/<tag>` → 201 (assert `Content-Type` == OCI manifest, all referenced digests already uploaded, diff_ids count == layers count).
|
||||
- Real-manifest fixture: tiny synthetic OCI index/manifest/config/1-2 tiny layers built in-test with the same tar+gzip code (NOT the 45 MB real base — suite stays fast); plus one "live-shape" assertion test using the digests/mediaTypes captured in §3.1 as constants.
|
||||
- End-to-end offline: stub Docker Hub + stub gitea → run the publish flow → assert both tags PUT, base layer bytes passed through, broker layer contains the pyz at `/app/`.
|
||||
- Redirect edge: stub answers 307 on the two-step upload PUT → client follows Location explicitly (urllib would raise otherwise).
|
||||
|
||||
### 3.7 Defects / risks (daemonless)
|
||||
|
||||
- **RISK** — upstream tag drift (`3.11-slim` moves): pin at publish time by recording digests + `--base-manifest` override; notes carry provenance.
|
||||
- **RISK** — 307/308 not followed by urllib (verified): explicit Location-following required; stub-tested.
|
||||
- **RISK** — ~46 MB upload per milestone (base layers re-uploaded; no cross-registry mount): bounded, once-per-milestone; HEAD-skip makes retries cheap.
|
||||
- **RISK** — gitea's anonymous-200 token quirk (verified): empty-scope JWT must not be treated as auth success.
|
||||
- **NOTE** — monolithic upload limits: gitea handles multi-MB bodies; the config/manifest are KB-scale, the largest blob is the 29.8 MB base layer — fine for monolithic POST (dist-spec; gitea conformance-tested upstream).
|
||||
- **NOTE** — publish failure follows the same bounded-retry → `release_pending` escalation as releases; never blocks the milestone merge (ROADMAP P3).
|
||||
|
||||
## 4. Cross-Cutting: Layout, Security, Line Count
|
||||
|
||||
### 4.1 Recommended repo layout (new files only)
|
||||
|
||||
```
|
||||
cli/__main__.py # boot shim (staged into the pyz; NOT coreci_broker.py → outside AC14)
|
||||
scripts/build_cli.py # stage + zipapp + normalize; emits dist/coreci-broker-vX.Y.Z.pyz (+ sha256)
|
||||
scripts/release.py # notes builder + create-or-get release + asset attach + bounded retry
|
||||
scripts/publish_image.py # daemonless registry client (pull base → append layer → push, D-019)
|
||||
scripts/_env_secrets.py # shared .env-file secrets loader (~20 lines, one discipline point)
|
||||
Dockerfile # documentation + docker-user path (§3.5)
|
||||
dist/ # build output (gitignored)
|
||||
tests/test_cli_packaging.py # build in tmp + boot subprocess + boot-parity flows via urllib
|
||||
tests/test_build_cli.py # reproducibility: two builds → identical sha256; --version wiring
|
||||
tests/test_release_upload.py # stub-gitea attachment + retry/escalation paths
|
||||
tests/test_publish_image.py # stub-registry pull/push flows + manifest assembly
|
||||
```
|
||||
|
||||
- `scripts/_env_secrets.py` shared rather than copy-pasted per script: one place asserts the "file-only, non-empty" token discipline; `python3 scripts/release.py` runs with `sys.path[0] == scripts/` so the sibling import works without path setup.
|
||||
- **.gitignore additions**: `dist/`, `*.pyz` (root-level strays). Existing patterns already cover `__pycache__`, `*.db`, `*.log`, `.env*`, `data/`.
|
||||
- Phase mapping (per ROADMAP): Phase 1 = `cli/` + `scripts/build_cli.py` + its tests (REQ-21); Phase 2 = `release.py` + `publish_image.py` + `Dockerfile` + stub tests (REQ-22/23).
|
||||
|
||||
### 4.2 Broker changes: ZERO (verified)
|
||||
|
||||
- `run_standalone()`, `boot()`, `load_config()`, `_parse_env_file()` are module-level and importable; the shim + scripts need nothing added to the broker. `coreci_broker.py` = 1390/1400 lines (D-018 cap) — v0.2 adds no lines; AC14 stays green by construction.
|
||||
|
||||
### 4.3 Test-harness conventions (carried v0.1 §6 + new packaging patterns)
|
||||
|
||||
- Packaging test pattern (verified in prototype): `TemporaryDirectory` → `build_cli.build(state=tmp)` → subprocess `Popen(["python3", "<pyz>"], env={BROKER_BIND: "127.0.0.1:0", BROKER_ENV_PATH/DB/AUDIT: tmp paths}, stdout=PIPE)` → read stdout lines until `coreci-broker listening on http://…` (parse base URL; the `BROKER_ADMIN_PASS=` line comes first and is captured for login) → drive `urllib` flows (healthz, login, register, secret read — reuse v0.1 acceptance request patterns) → `proc.terminate()` in cleanup. Boot cost ~1–2 s; suite impact acceptable (v0.1 suite ≈ 2 min).
|
||||
- Stub servers: extend the `_StubOwuiHandler` pattern (class-level fixtures + hit recording + `addCleanup`) to `_StubGiteaHandler` / `_StubRegistryHandler`; all stubs bind `("127.0.0.1", 0)` and the code under test takes a base URL parameter (no hardcoded hosts in scripts — required for stubbing anyway).
|
||||
|
||||
### 4.4 Security posture (v0.2-specific)
|
||||
|
||||
- **No secrets in the artifact**: the pyz bundles broker + UI + client only; the client module contains no credentials; the broker generates pepper/admin pass at first boot; `.env` is never inside the archive. Same for the image: no state baked in (`WORKDIR /state` empty at build; first boot generates).
|
||||
- **Token discipline**: `release.py` / `publish_image.py` read `GITEA_TOKEN` ONLY via the `.env*` file loader (never shell env — structurally blocked by `blocked_env_vars`); assert non-empty before first use (v1.8: empty token → gitea 404 masks auth failure). Stubs use fake tokens; no live-forge writes in tests.
|
||||
- **Artifact integrity**: sha256 of the `.pyz` computed at build, embedded in release notes; reproducible build makes the digest meaningful across rebuilds.
|
||||
- **Static disclosure**: v0.1's `_broker_protected_paths` behavior extends unchanged to materialized roots (assets contain no secrets; `.env`/db/audit realpaths remain protected wherever the state dir sits).
|
||||
|
||||
## 5. Research Conclusions (planner-ready)
|
||||
|
||||
- **Phase 1 (REQ-21)**: `cli/__main__.py` five-step shim (§1.2 — prototype verified end-to-end, lift it), `scripts/build_cli.py` (stage → zipapp `--compress` → normalize post-pass §1.3 → sha256), `tests/test_cli_packaging.py` + `tests/test_build_cli.py`. Zero broker lines.
|
||||
- **Phase 2 (REQ-22/23)**: `scripts/_env_secrets.py` + `release.py` (create-or-get by tag, idempotent attach, bounded retry → release_pending, notes model §2.3); `scripts/publish_image.py` (§3.1–3.4 flows; OCI manifest end-to-end; monolithic uploads + HEAD skip + explicit Location-following); `Dockerfile` (§3.5); stub tests §2.4/§3.6.
|
||||
- **Phase 3 (acceptance)**: boot-parity matrix through the artifact (first-boot pass once, 0600 modes, silent second boot, UI + client served, register/read flows), reproducibility sha256, stub-releasability of both pipelines, security review (no secrets in artifact/image; token discipline), then milestone ship: build → attach to `v0.2.3` → publish image tags `v0.2.3` + `latest` → merge.
|
||||
- **No BLOCKERs found.** All environment facts were verified live (registry auth shapes, index resolution, gitea API, zipapp mechanics) and the packaging design is proven by a working prototype against the real broker sources. Remaining open items are NOTE/RISK grade and owned by the plan: sidecar `.sha256` asset vs digest-in-notes; shim fallback when run unzipped (trivial); `latest`-tag manifest PUT ordering (push version tag first, `latest` second — trivial).
|
||||
+23
-38
@@ -1,60 +1,45 @@
|
||||
# Roadmap — Milestone v0.1 (coreci-broker)
|
||||
# Roadmap — Milestone v0.2 (cli-packaging)
|
||||
|
||||
## Overview
|
||||
|
||||
One milestone, owner-directed 2026-10-09: implement Requirements Specification v0.5 in full. **Milestone v0.1 — coreci-broker** (feature type): the single-file broker, its SQLite store, audit log, four-page admin UI, tool-auth plane, secrets mediation, public client module, and reference tools, with the spec's §12 acceptance matrix (AC1–AC14) green. The spec's "v0.5" is its requirements-doc revision label, not the product version (D-008).
|
||||
One milestone, owner-directed 2026-10-10: package the completed v0.1 broker for distribution. **Milestone v0.2 — cli-packaging** (feature type): the broker as a single-file zipapp CLI (`coreci-broker.pyz`, self-contained), the CLI attached to every gitea release alongside release notes, and a docker container built **once per milestone** and published as a package in the gitea container registry. Scope is CLI + distribution only — no functional changes to the broker (REQ-21..REQ-23). Docker publication starts with v0.2; no retroactive v0.1 image.
|
||||
|
||||
Versioning (greenfield deviation, researchx/nextcraft precedent): the milestone branch is `milestone/v0.1-coreci-broker`; phase tags run on the milestone's own line — phase 0 ships `v0.1.0`, execution phases ship `v0.1.1`..`v0.1.4`, and the final phase's patch (`v0.1.5`) IS the milestone release; no separate minor tag. MAJOR remains 0 until an owner-approved v1 release. Ships go to gitea (`git.coreci.dev/coreci/broker`); the forge repo must exist before the first push.
|
||||
Versioning (greenfield convention, D-008): the milestone branch is `milestone/v0.2-cli-packaging`; phase tags run on the milestone's own line — phase 0 ships `v0.2.0`, execution phases ship `v0.2.1`..`v0.2.2`, and the final phase's patch (`v0.2.3`) IS the milestone release; no separate minor tag. MAJOR remains 0 until an owner-approved v1 release. Ships go to gitea (`git.coreci.dev/coreci/broker`); releases are best-effort with bounded retry, never blocking tag+merge.
|
||||
|
||||
Autonomy: **guided** — escalate on every decision gate; threshold 0.85; 1 revision iteration; clarify budget 20.
|
||||
|
||||
## Phase 0 — Pre-Execution
|
||||
|
||||
- **Description**: Specification parsing, clarification (decisions D-001..D-011 recorded in the init commit), `.ciagent/` reference files, and the pre-execution pipeline stages: specify → clarify → research → plan → grill. Init has landed the reference files; `/ci-run` phase 0 completes RESEARCH.md (environment constraints, spec defects, sibling conventions), PLAN.md (task-level breakdown), PERSONAS.md, and the GRILL verdict before any code.
|
||||
- **Description**: Specification parsing (REQ-21..REQ-23), clarification (packaging/distribution decisions D-019..D-022), research (zipapp mechanics, gitea release-asset API, gitea container registry + OCI registry HTTP API v2, Docker Hub anonymous pulls), plan (task-level breakdown with waves), grill, MVP/UX check. `/ci-run` phase 0 completes REQUIREMENTS.md v2 section, RESEARCH.md, PLAN.md, PERSONAS.md, and the grill verdict before any code.
|
||||
- **Requirements covered**: none (planning artifacts)
|
||||
- **Success criteria**: `.ciagent/` files reflect the 6-phase structure; REQ-01..REQ-20 defined and traceable; milestone v0.1 established in config.json and PROJECT.md; PLAN.md + PERSONAS.md committed; grill verdict recorded; phase tagged `v0.1.0`
|
||||
- **Success criteria**: REQ-21..REQ-23 defined and traceable; milestone v0.2 established in config.json and PROJECT.md; D-016+ decisions recorded; PLAN.md + PERSONAS.md committed; grill verdict recorded; MVP/UX sections present; phase tagged `v0.2.0`
|
||||
- **Depends on**: —
|
||||
- **Status**: complete
|
||||
- **Status**: complete (shipped as `v0.2.0`, release id 970)
|
||||
|
||||
## Phase 1 — Core Skeleton
|
||||
## Phase 1 — CLI Packaging + Release Tooling
|
||||
|
||||
- **Description**: `coreci_broker.py` foundation: `.env` read/write + first-boot generation (pepper + admin pass, printed once), SQLite schema + indexes per spec §5.2, mode-0600 on DB and audit log, append-only JSONL audit writer per §5.3, dual-mode bcrypt/scrypt hasher (D-001), server-side token derivation per §6, stdlib `ThreadingHTTPServer` + JSON router + `{"error, message}` contract per §7, Open WebUI `Tools`-class mount (daemon thread) + standalone `__main__` entry.
|
||||
- **Requirements covered**: REQ-03, REQ-11, REQ-14, REQ-15, REQ-16, REQ-17, REQ-18
|
||||
- **Success criteria**: broker boots standalone on an ephemeral port, creates DB + audit log at 0600, prints the generated admin pass exactly once (not on second boot); token derivation matches the §6 formula byte-for-byte; source under 800 lines
|
||||
- **Description**: `coreci-broker.pyz` build pipeline: stdlib `python3 -m zipapp` build script (no pip) with G-009 normalize pass (sorted entries, fixed timestamps, constant external_attr); self-contained archive bundling `coreci_broker.py` + `coreci_broker_ui/` + `coreci_broker_client.py` with asset materialization at boot (env-resolved roots, never `__file__`); standalone boot parity with v0.1 (first-boot `.env` generation, admin pass printed once, 0600 modes, full admin UI + client module served); `--version` flag; release-asset automation (G-011: `scripts/_env_secrets.py` + `scripts/release.py` land here so the v0.2.1 ship attach is script-driven — no ad-hoc bridge); tests boot the built artifact end-to-end and stub-verify the gitea upload.
|
||||
- **Requirements covered**: REQ-21, REQ-22
|
||||
- **Success criteria**: artifact builds reproducibly from a clean checkout (two consecutive same-environment builds → byte-identical sha256); `python3 coreci-broker.pyz` boots with behavior byte-identical to `python3 coreci_broker.py` (v0.1 suite green through the packaged path); UI pages + client module served from the archive; first-boot semantics preserved on second boot; the `v0.2.1` release carries the `.pyz` attached by `scripts/release.py` (D-020 starts here; `v0.2.0` carries none — G-021)
|
||||
- **Depends on**: 0
|
||||
- **Status**: complete
|
||||
- **Status**: complete (shipped as `v0.2.1` — 325 cumulative tests green, verify four-layer PASS, .pyz attached by script, asset id 396)
|
||||
|
||||
## Phase 2 — Admin Plane
|
||||
## Phase 2 — Docker + Image Publish
|
||||
|
||||
- **Description**: Session auth (cookie `broker_session` + HTTP Basic both accepted), `POST /api/broker/auth/login`, `logout`, `whoami` (§7.1); app_registrations list/add/revoke/reinstate (§7.2, UUID format validation, 409 on duplicate, revoke keeps existing tokens until TTL); user_allowlist list/add/revoke/reinstate (§7.3); audit query with filters + `next_since` cursor (§7.6); the four UI pages + dashboard + persistent nav (§10) served from `coreci_broker_ui/`; session cleanup.
|
||||
- **Requirements covered**: REQ-01, REQ-05, REQ-06, REQ-12, REQ-13
|
||||
- **Success criteria**: AC1 (first-boot login with the printed pass), AC2 (app registration add/remove via UI, persisted in `app_registrations`, UI updates after each action), AC3 (user allowlist same), AC9's login/logout audit lines
|
||||
- **Description**: Daemonless publish client per D-019 (once-per-milestone pull pinned base layers / append broker layer / push to `git.coreci.dev/coreci/broker` with version + `latest` tags, D-021/D-022; G-019 wire hardening: per-request timeouts, blob retry with HEAD-skip, explicit 307/308 relative-Location handling); `Dockerfile` + `.dockerignore` (G-015) + consistency-tested equivalence mapping (G-016); README additive section (CLI + docker usage); the `v0.2.2` release attaches the `.pyz` via `scripts/release.py` with CLI + docker notes.
|
||||
- **Requirements covered**: REQ-23 (+ REQ-21/REQ-22 docs)
|
||||
- **Success criteria**: daemonless publish client stub-verified end-to-end (no live-forge writes, no container runtime); Dockerfile/.dockerignore consistency tests green; this phase's release (`v0.2.2`) carries the `.pyz` attached alongside the source archive with notes; bounded-retry → `release_pending` paths covered by tests
|
||||
- **Depends on**: 1
|
||||
- **Status**: complete
|
||||
- **Status**: complete (shipped as `v0.2.2` — 341 cumulative tests green, verify four-layer PASS, .pyz attached by script, asset id 397)
|
||||
|
||||
## Phase 3 — Tool Auth Plane
|
||||
## Phase 3 — Final Review + Ship
|
||||
|
||||
- **Description**: `POST /api/broker/agents/register` (§7.4): allowlist gates → 403 on unknown/revoked UUID or user; idempotent re-registration re-derives from request inputs and overwrites on input change (D-005); pepper returned on row-create or when the caller presents no valid token; TTL set from `BROKER_TOKEN_TTL`. App-auth verification: parse `Authorization: App <UUID> <token>`, UUID-scoped slow-hash scan + in-memory verified-token cache (D-004). `GET /api/broker/agents/<agent_id>` (§7.4). State endpoints `GET/PUT /api/broker/agents/<id>/state` (§9.1) with identity enforcement (calling UUID + token-derived user_id must match stored identity) and the backend toggle `BROKER_AGENT_STATE_BACKEND=internal` (default) `| owui` (D-003).
|
||||
- **Requirements covered**: REQ-02, REQ-04, REQ-09, REQ-10
|
||||
- **Success criteria**: AC4 (token derived + stored as slow-hash, same value computed both sides), AC7 (revoked/unknown UUID → 403), AC8 (unknown user → 403), AC10 (TTL expiry → 401 → re-register → new token, audited), AC12 (state read/write with identity enforcement)
|
||||
- **Description**: Full acceptance: cumulative v0.1 suite green plus v0.2 packaging acceptance (boot parity through the artifact, self-containment, daemonless publish flow); security review (artifact integrity, no secrets in the archive/image, token discipline in upload paths, build-context hygiene); multi-persona review of the milestone diff; audit/reconstruction (REQ-21..23 traceable); milestone ship in the G-012-corrected order — build CLI → squash-merge `milestone/v0.2-cli-packaging` → `main` → tag `v0.2.3` → push → G-013 live smoke (read-only, env-gated) → attach `.pyz` + full notes via `scripts/release.py` → daemonless image publish once (D-019..D-022, with the G-020 closure runbook on failure) → branch cleanup → closeout.
|
||||
- **Requirements covered**: REQ-21, REQ-22, REQ-23 (acceptance)
|
||||
- **Success criteria**: all v0.1 + v0.2 acceptance green; security review PASS; `v0.2.3` on `main` with the `.pyz` attached + notes on the gitea release; docker image published as a gitea package (or escalated per the bounded-retry contract + closure runbook, never blocking merge); REQUIREMENTS/ROADMAP closed out; checkpoint cleared
|
||||
- **Depends on**: 2
|
||||
- **Status**: complete
|
||||
- **Status**: complete (shipped as `v0.2.3` — milestone release; 356 cumulative tests green, security review PASS-WITH-FINDINGS remediated, `.pyz` attached + notes, image published once)
|
||||
|
||||
## Phase 4 — Secrets + Client Ecosystem
|
||||
## Post-Milestone Notes (carry-forward from v0.1, GRILL G-008)
|
||||
|
||||
- **Description**: Creds-source adapter (D-002): `FileCredsSource` (default; JSON document at `BROKER_CREDS_FILE` mapping paths to secret objects) and `OwuiApiCredsSource` (`BROKER_OWUI_BASE_URL` + `BROKER_OWUI_API_KEY`). `GET /api/broker/secrets/<path>` (§7.5): App-auth, scope check `path == "coreci/" + agent_id` → 403 on mismatch, 404 when absent, audit every attempt. Public client module `coreci_broker_client.py` (stdlib urllib only): ensure-registered, state get/put, secret read with 401 → re-register → retry-once; served unauthenticated at `/static/coreci_broker_client.py`. Reference OWUI tools (`reference_tools/`): proxmox reference + a minimal auth-then-use showcase, safe dry-run default (§9, §11; D-006).
|
||||
- **Requirements covered**: REQ-07, REQ-08, REQ-19, REQ-20
|
||||
- **Success criteria**: AC5 (valid token reads the secret; tampered token → 401), AC6 (proxmox-registered tool reading `coreci/gitea` → 403); client module fetchable unauthenticated and drives a reference tool end-to-end against a test broker
|
||||
- **Depends on**: 3
|
||||
- **Status**: complete
|
||||
|
||||
## Phase 5 — Final Review + Ship
|
||||
|
||||
- **Description**: Full §12 acceptance matrix as tests (AC1–AC13 as behavioral tests, AC14 as a line-count test); security review pass (path traversal on `<path>`, pepper never re-printed, 0600 modes, token scoping, audit completeness); README with run/deploy notes and documented spec deviations, including the owner-run post-ship live-OWUI smoke checklist (mount/register/read on the deployed v0.11.3 host — G-005/G-006); multi-persona review of the milestone diff; owner approval gate; milestone merge to main and tag `v0.1.5` (final phase patch = milestone release); first push to gitea once the forge repo exists.
|
||||
- **Requirements covered**: REQ-01..REQ-20 (acceptance)
|
||||
- **Success criteria**: AC1–AC14 all green (`python3 -m unittest discover -s tests`); security review PASS; README complete with deviations; owner approval recorded; milestone squash-merged to main, tagged `v0.1.5`, release created
|
||||
- **Depends on**: 4
|
||||
- **Status**: not_started
|
||||
## Post-Milestone Notes (owner-promoted from GRILL G-008)
|
||||
|
||||
- **90-day adoption criterion**: `proxmox_readonly_tool.py` runs on `coreci_broker_client.py` with direct KB reads removed by **2027-01-09**; the owner owns the date. No real tool migrates to the broker before the post-ship OWUI smoke checklist (G-005) passes on the deployed v0.11.3 host.
|
||||
- **90-day adoption criterion**: `proxmox_readonly_tool.py` runs on `coreci_broker_client.py` with direct KB reads removed by **2027-01-09**; the owner owns the date. No real tool migrates to the broker before the owner-run post-ship OWUI smoke checklist (G-005) passes on the deployed v0.11.3 host. These v0.1 post-ship obligations remain open after the v0.1 ship (2026-10-10) and carry forward unchanged.
|
||||
@@ -46,9 +46,9 @@
|
||||
"projects": [],
|
||||
"active_project": null,
|
||||
"milestone": {
|
||||
"version": "v0.1",
|
||||
"name": "coreci-broker",
|
||||
"version": "v0.2",
|
||||
"name": "cli-packaging",
|
||||
"type": "feature",
|
||||
"branch": "milestone/v0.1-coreci-broker"
|
||||
"branch": "milestone/v0.2-cli-packaging"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
# Build-context hygiene (G-015): no secrets, state, or build artifacts
|
||||
# may enter an image build context.
|
||||
.git
|
||||
.env
|
||||
.env.*
|
||||
data/
|
||||
*.db
|
||||
*.log
|
||||
dist/
|
||||
__pycache__/
|
||||
tests/
|
||||
.ciagent/
|
||||
# Nested matches too (P3-03, CWE-693): the root-anchored `__pycache__/`
|
||||
# above misses cli/__pycache__/*.pyc — build artifacts must stay out of
|
||||
# the context wherever they sit in the tree.
|
||||
**/__pycache__/
|
||||
**/*.pyc
|
||||
+5
-1
@@ -10,4 +10,8 @@ __pycache__/
|
||||
# Broker runtime state
|
||||
data/
|
||||
*.db
|
||||
*.log
|
||||
*.log
|
||||
|
||||
# v0.2 build artifacts
|
||||
dist/
|
||||
*.pyz
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
# coreci-broker docker image (v0.2, REQ-23).
|
||||
#
|
||||
# Equivalence mapping to the daemonless-published image
|
||||
# (scripts/publish_image.py — the pipeline's source of truth):
|
||||
# same base family: python:3.11-slim (D-022 pinned minor)
|
||||
# same artifact: coreci_broker.py + coreci_broker_client.py +
|
||||
# coreci_broker_ui/ + cli/__main__.py + _cli_version.py,
|
||||
# packed by the same `python3 -m zipapp` invocation
|
||||
# (--python "/usr/bin/env python3" --compress)
|
||||
# same entrypoint: ["python3", "/app/coreci-broker.pyz"]
|
||||
# same state anchor: WORKDIR /state (volume-backed; first boot generates
|
||||
# .env, broker.db, broker.audit.log there)
|
||||
# Documented differences (equivalence required, byte-identity NOT):
|
||||
# - docker-build layering (COPY + RUN) vs the daemonless client's
|
||||
# verbatim base layers + one deterministic broker layer
|
||||
# - version provenance: ARG VERSION here vs build-time _cli_version.py
|
||||
# injection in scripts/build_cli.py
|
||||
# - no normalize pass here (docker rebuilds are not digest-pinned)
|
||||
FROM python:3.11-slim
|
||||
|
||||
# Staging mirrors scripts/build_cli.py EXACTLY (P3-03, CWE-636): Docker
|
||||
# copies directory CONTENTS flattened, so the staging layout must come
|
||||
# from the destination paths — root sources at /app/src/, UI pages under
|
||||
# /app/src/coreci_broker_ui/ (cli/__main__.py:80-83 materializes only
|
||||
# coreci_broker_ui/* members; a flattened copy leaves /ui/* 404ing), and
|
||||
# the shim staged as the archive-root __main__.py (zipapp boots the
|
||||
# root __main__.py; the file-to-file rename mirrors build_cli's
|
||||
# STAGED_SHIM staging).
|
||||
COPY coreci_broker.py coreci_broker_client.py /app/src/
|
||||
COPY coreci_broker_ui/ /app/src/coreci_broker_ui/
|
||||
COPY cli/__main__.py /app/src/__main__.py
|
||||
|
||||
ARG VERSION=dev
|
||||
RUN printf 'VERSION = "%s"\n' "$VERSION" > /app/src/_cli_version.py \
|
||||
&& python3 -m zipapp /app/src -o /app/coreci-broker.pyz \
|
||||
--python "/usr/bin/env python3" --compress
|
||||
|
||||
WORKDIR /state
|
||||
|
||||
ENTRYPOINT ["python3", "/app/coreci-broker.pyz"]
|
||||
|
||||
EXPOSE 8765
|
||||
@@ -0,0 +1,222 @@
|
||||
# CoreCI Broker
|
||||
|
||||
**TL;DR** — CoreCI Broker — single-file credential broker for autonomous agent tools (Open WebUI workspace tool / standalone). This file explains how to stand it up and who to contact; the machinery detail stays lower under Details.
|
||||
|
||||
|
||||
A single-file credential broker for autonomous agent tools on the Open WebUI
|
||||
host. Tools authenticate with derived tokens instead of shared secrets, the
|
||||
CEO gates access through UI-managed allowlists of app UUIDs and user IDs,
|
||||
every read lands in an append-only audit log, and secret reads are path-scoped
|
||||
to `coreci/<agent_id>` — one Python file, one SQLite DB, a four-page admin UI,
|
||||
zero pip installs, no build step.
|
||||
|
||||
## Quickstart (standalone)
|
||||
|
||||
```bash
|
||||
python3 coreci_broker.py
|
||||
```
|
||||
|
||||
- Stdlib-only runtime, Python 3.11+. Zero pip installs: bcrypt when
|
||||
importable, stdlib scrypt otherwise (format-prefixed hashes).
|
||||
- First boot generates `.env`: a 32-byte base64 pepper plus a 32+-char admin
|
||||
password. The password is printed to the startup log as `BROKER_ADMIN_PASS=…`
|
||||
**exactly once** — save it immediately; it is never printed again. Lost pass:
|
||||
remove `BROKER_ADMIN_PASS` from `.env` and reboot — a new one prints once.
|
||||
- The server binds `127.0.0.1:8765`. Open `http://127.0.0.1:8765/ui/login` and
|
||||
log in with the printed pass.
|
||||
- Stop with Ctrl+C. Durable state is exactly three files: `broker.db`,
|
||||
`broker.audit.log`, `.env` — back them up.
|
||||
|
||||
## Configuration
|
||||
|
||||
Read on boot from the `.env` at `BROKER_ENV_PATH`. Precedence:
|
||||
**process env → `.env` → defaults**.
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `BROKER_BIND` | `127.0.0.1:8765` | Listen address; the bind controls reachability |
|
||||
| `BROKER_ADMIN_USER` | `admin` | Admin login user name |
|
||||
| `BROKER_ADMIN_PASS` | generated first boot | Admin login password (32+ random chars) |
|
||||
| `BROKER_INSTANCE_PEPPER` | generated first boot | 32-byte base64 pepper; token derivation input, synced to SQLite |
|
||||
| `BROKER_ENV_PATH` | `.env` | Path of the `.env` file the broker reads/writes |
|
||||
| `BROKER_TOKEN_TTL` | `86400` | Tool token lifetime, seconds (24 h) |
|
||||
| `BROKER_SESSION_TTL` | `300` | Admin UI session cookie lifetime, seconds |
|
||||
| `BROKER_DB_PATH` | `broker.db` | SQLite database (mode 0600) |
|
||||
| `BROKER_AUDIT_LOG_PATH` | `broker.audit.log` | Append-only JSONL audit log (mode 0600) |
|
||||
| `BROKER_CREDS_SOURCE` | `file` | Credential source adapter: `file` (dev/tests) or `owui` (production) |
|
||||
| `BROKER_CREDS_FILE` | `data/creds.json` | `FileCredsSource` JSON document: paths → secret objects |
|
||||
| `BROKER_OWUI_BASE_URL` | — | Open WebUI base URL for the OWUI creds source |
|
||||
| `BROKER_OWUI_API_KEY` | — | Open WebUI API key (Bearer) for KB read access |
|
||||
| `BROKER_AGENT_STATE_BACKEND` | `internal` | Agent-state storage: `internal` (rows in broker.db) or `owui` |
|
||||
| `BROKER_MAX_AGENTS_PER_APP` | `50` | Cap on agent registrations per app UUID |
|
||||
| `BROKER_UI_ROOT` | `coreci_broker_ui` | Directory served for `/ui/*` pages |
|
||||
| `BROKER_STATIC_ROOT` | `.` | Directory served for `/static/*` (client module) |
|
||||
|
||||
Production: `BROKER_CREDS_SOURCE=owui` with `BROKER_OWUI_BASE_URL` +
|
||||
`BROKER_OWUI_API_KEY` (0600 `.env` discipline). Keep `BROKER_BIND` on
|
||||
loopback unless you deliberately widen reachability.
|
||||
|
||||
## Running as an Open WebUI workspace tool
|
||||
|
||||
Paste `coreci_broker.py` into Open WebUI as a workspace Tool (frontmatter
|
||||
`requirements:` stays empty — the runtime already ships everything). Notes
|
||||
that matter:
|
||||
|
||||
- The server starts in `Tools.__init__` behind a once-per-process guard; the
|
||||
OWUI loader has no activate/deactivate hook, and disabling a tool does not
|
||||
stop a previously started server.
|
||||
- **After editing the broker tool source, restart Open WebUI.** The loader
|
||||
re-execs the module on source change, but the previously loaded server keeps
|
||||
its port — the new instance fails loud with a bind-error message telling you
|
||||
to restart, rather than silently serving old code.
|
||||
- Paths (DB, audit log, `.env`, UI roots) resolve from environment config,
|
||||
never from `__file__` — the OWUI loader deletes the temp file it execs.
|
||||
- Durable state: `broker.db` + `broker.audit.log` + `.env` — **back those
|
||||
three up**; SQLite is the source of truth, no HA, restore is a file copy.
|
||||
- `Tools.status()` reports the serving address; call it from a chat to verify
|
||||
the mount, or hit `/healthz` on the bind.
|
||||
|
||||
## Admin UI
|
||||
|
||||
Four pages + dashboard, server-rendered on the same port as the API:
|
||||
|
||||
- `/ui/login` — the CEO's bookmark. First-boot pass discipline: the printed
|
||||
pass is shown once; record it in your secret store on first login.
|
||||
- `/ui/app_registrations` — allowlist of tool app UUIDs: add, revoke, reinstate.
|
||||
- `/ui/user_allowlist` — allowlist of user IDs (optional email): same actions.
|
||||
- `/ui/audit` — filterable, read-only view of the audit log (no export in v0.1).
|
||||
- `/ui/dashboard` — landing page with persistent nav header and Logout.
|
||||
|
||||
Allowlist semantics: **revoke keeps existing tokens working until TTL expiry —
|
||||
by design.** Revocation immediately blocks new registrations; already-issued
|
||||
tokens live at most one `BROKER_TOKEN_TTL` (24 h by default).
|
||||
|
||||
## Tool authors
|
||||
|
||||
Fetch the public client module — unauthenticated, from
|
||||
`http://<broker>/static/coreci_broker_client.py` — into your workspace and
|
||||
import it. Flow: `ensure_registered(state)` → `put_state(state)` →
|
||||
`read_secret("coreci/<agent_id>", state)`. Stdlib-urllib-only (no httpx, no
|
||||
bcrypt), so any tool can import it; on a 401 it re-registers and retries
|
||||
once. State endpoints are the tool's only persistence channel — keep `state`
|
||||
there, not in local files. See `reference_tools/` for the proxmox-style §11
|
||||
flow and a minimal auth-then-use showcase; downstream calls default to a
|
||||
safe dry-run.
|
||||
|
||||
## Security model
|
||||
|
||||
- Derived token: `base64(sha256(pepper:UUID:user_id:agent_id:sorted_tags))` —
|
||||
44 ASCII chars, no prefix, no challenge protocol. Stored only as a slow
|
||||
hash (`$2b$…` / `scrypt$…`), never in plaintext.
|
||||
- Allowlists are the gate: unknown or revoked app UUIDs and unknown users get
|
||||
403 at registration; the token is the consequence, not the authority.
|
||||
- Every secret-read attempt is audited (ok/denied/error) with user, UUID,
|
||||
agent, and path.
|
||||
- Secret reads are path-scoped: a registration reads exactly
|
||||
`coreci/<agent_id>` — cross-path → 403, absent → 404.
|
||||
- The broker never writes to `creds-coreci-coo`; the CEO maintains that KB
|
||||
through the Open WebUI UI.
|
||||
- `broker.db`, `broker.audit.log`, and `.env` are mode 0600.
|
||||
- Binds `127.0.0.1` by default; trusted-network posture — no rate limiting by
|
||||
design (spec §14).
|
||||
|
||||
## Documented spec deviations
|
||||
|
||||
Honest list; full detail in `.ciagent/PROJECT.md`:
|
||||
|
||||
- **Line-cap amendments (D-016/D-017/D-018)**: the single-file constraint grew
|
||||
800 → 1000 → 1200 → 1400 lines via the owner-approved §12.14 amendment
|
||||
pattern (readability over dense cramming). Current source: 1390 lines; AC14
|
||||
enforces the cap.
|
||||
- **Dual-mode hashing (D-001)**: spec §6 mandates bcrypt; the runtime is
|
||||
stdlib-only, so scrypt fallback with format-prefixed hashes.
|
||||
- **Token verification (D-004)**: spec §7.5's in-memory `token_hash` index is
|
||||
impossible with salted slow hashes; the broker scans rows scoped to the
|
||||
presented UUID and keeps a non-authoritative verified-token cache.
|
||||
- **Pepper rule (D-005)**: the pepper is returned only on row creation or
|
||||
tokenless registration (the KB-loss recovery path); spec §9's snippet would
|
||||
raise `KeyError` on re-registration.
|
||||
- **Spec defects resolved**: §7.5 index, §6-vs-§9 (the broker derives
|
||||
authoritatively; the tool stores), §9's `KeyError` — see PROJECT.md §Context.
|
||||
- `BROKER_ADMIN_USER` defaults to `admin` (spec silent on a default).
|
||||
- Register cap `BROKER_MAX_AGENTS_PER_APP` (50) — a documented addition.
|
||||
- Reinstate actions audit as `add_app`/`add_user` (no separate audit action).
|
||||
- In-scope but absent secrets audit as `read` with `result: error` (404s are
|
||||
audited, not silent).
|
||||
|
||||
## Post-ship live-OWUI smoke checklist (owner-run)
|
||||
|
||||
Before migrating any real tool (G-005; deployed host pinned v0.11.3, G-006):
|
||||
|
||||
1. Confirm the deployed Open WebUI version (expected **v0.11.3**) and that the
|
||||
KB read endpoints match the researched surface: `/api/v1/knowledge/{id}` and
|
||||
`/api/v1/files/{id}/data/content`.
|
||||
2. Add the broker as a workspace tool; check `/healthz` on the bind.
|
||||
3. CEO first-boot login at `/ui/login` with the printed pass.
|
||||
4. Add one app UUID and one user to the allowlists via the UI.
|
||||
5. In a scratch tool: import the client module from `/static/` →
|
||||
`ensure_registered` → read `coreci/proxmox` via the OWUI creds source.
|
||||
6. Verify every step in `/ui/audit`.
|
||||
7. Rollback = disable the tool + restore the three state files.
|
||||
|
||||
No real tool migrates before this passes; the 90-day adoption criterion
|
||||
(proxmox tool on the client, direct KB reads removed) is due 2027-01-09.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
timeout 300 python3 -m unittest discover -s tests
|
||||
```
|
||||
|
||||
295 tests, stdlib unittest only (~2 min; one skip where bcrypt is not
|
||||
importable). The AC1–AC14 acceptance matrix is green.
|
||||
|
||||
## Repository layout
|
||||
|
||||
| Path | What it is |
|
||||
|------|------------|
|
||||
| `coreci_broker.py` | The broker: single file — server, store, audit, auth, UI serving |
|
||||
| `coreci_broker_client.py` | Public stdlib-only client module (served at `/static/`) |
|
||||
| `coreci_broker_ui/` | Server-rendered HTML for the four pages + dashboard |
|
||||
| `reference_tools/` | Tool-author templates (proxmox-style §11 flow + minimal showcase) |
|
||||
| `tests/` | Stdlib-unittest suite incl. the AC1–AC14 acceptance matrix |
|
||||
| `.ciagent/` | CI pipeline artifacts (project, requirements, roadmap, decisions) |
|
||||
## CLI and Docker (v0.2)
|
||||
|
||||
From v0.2.1 onward every gitea release attaches the broker packaged as a
|
||||
self-contained CLI zipapp: `coreci-broker-vX.Y.Z.pyz` (its sha256 is in
|
||||
the release notes).
|
||||
|
||||
**CLI install + usage:**
|
||||
|
||||
```bash
|
||||
# from the release page assets:
|
||||
chmod +x coreci-broker-vX.Y.Z.pyz
|
||||
./coreci-broker-vX.Y.Z.pyz --version # prints the release version
|
||||
./coreci-broker-vX.Y.Z.pyz # boots the standalone broker
|
||||
# or: python3 coreci-broker-vX.Y.Z.pyz (Python 3.11+)
|
||||
```
|
||||
|
||||
The artifact bundles the broker, the admin UI pages, and the client
|
||||
module — one file serves everything (UI at `/ui/`, client at
|
||||
`/static/coreci_broker_client.py`). First boot prints
|
||||
`BROKER_ADMIN_PASS=...` exactly once and creates `.env`, `broker.db`,
|
||||
and `broker.audit.log` mode 0600 in the state dir; state is anchored by
|
||||
`BROKER_ENV_PATH` (relative paths stay CWD-relative). The full
|
||||
`BROKER_*` env contract is in the Configuration section above.
|
||||
|
||||
**Docker:**
|
||||
|
||||
```bash
|
||||
docker pull git.coreci.dev/coreci/broker:<tag> # also :latest
|
||||
docker run -p 8765:8765 -v broker-state:/state \
|
||||
git.coreci.dev/coreci/broker:<tag>
|
||||
```
|
||||
|
||||
State lives on the `/state` volume (`WORKDIR /state`; first boot
|
||||
generates `.env`, `broker.db`, and `broker.audit.log there) — mount a
|
||||
persistent volume or the admin pass and any registrations are lost with
|
||||
the container. The image is built and published once per milestone at
|
||||
the milestone release (see `scripts/publish_image.py` — the daemonless
|
||||
registry client; the `Dockerfile` documents the docker-user path and
|
||||
its equivalence mapping).
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
# coreci-broker CLI boot shim; staged by scripts/build_cli.py as the root
|
||||
# __main__.py of the coreci-broker-vX.Y.Z.pyz zipapp (the documented mode —
|
||||
# running this file unzipped from a source tree is a guarded fallback).
|
||||
#
|
||||
# Boot sequence: (1) --version prints _cli_version.VERSION and exits 0 before
|
||||
# any broker import; (2) the state dir is resolved exactly like the broker
|
||||
# resolves it (BROKER_ENV_PATH-anchored; relative default paths stay
|
||||
# CWD-relative — the broker's path contract is inherited unchanged);
|
||||
# (3) bundled assets are materialized from the running archive
|
||||
# (zipfile.ZipFile(sys.path[0])) into
|
||||
# <state>/.coreci_broker_assets/<VERSION>/{ui,static} — ui/ receives the
|
||||
# bundled coreci_broker_ui/ pages, static/ receives the bundled
|
||||
# coreci_broker_client.py — via a temp dir plus os.replace (versioned dirs;
|
||||
# same-version overwrite self-repairs; other versions are never cleaned);
|
||||
# (4) BROKER_UI_ROOT/BROKER_STATIC_ROOT are pointed at the materialized
|
||||
# dirs only when the key is absent from both the process env and the user
|
||||
# .env (user wins); (5) coreci_broker.run_standalone() boots the broker.
|
||||
# Guard: when sys.path[0] is a directory (or empty) this shim is running
|
||||
# unzipped from a source tree — materialization is skipped and the broker's
|
||||
# repo-relative defaults apply.
|
||||
#
|
||||
# Private-API coupling (grill G-018, tracked in P3-02): this shim imports
|
||||
# coreci_broker._parse_env_file and coreci_broker.run_standalone —
|
||||
# module-level but private-surfaced broker APIs (coreci_broker.py is frozen
|
||||
# at 1390 lines). Any future broker refactor that removes or renames them
|
||||
# breaks packaged boots loudly (ImportError/AttributeError); that breakage
|
||||
# is accepted — the fix is to update this shim and re-run the v0.2 packaged
|
||||
# acceptance (tests/test_cli_packaging.py).
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import zipfile
|
||||
|
||||
ASSET_DIR_NAME = ".coreci_broker_assets"
|
||||
|
||||
|
||||
def _version_requested():
|
||||
return "--version" in sys.argv[1:]
|
||||
|
||||
|
||||
def _resolve_state_dir():
|
||||
env_path = os.environ.get("BROKER_ENV_PATH") or ".env"
|
||||
return os.path.dirname(os.path.abspath(env_path)) or os.getcwd()
|
||||
|
||||
|
||||
def _safe_relative(name):
|
||||
if not name or "\x00" in name or "\\" in name or name.startswith("/"):
|
||||
return None
|
||||
parts = name.split("/")
|
||||
if ".." in parts or any(part.startswith(".") for part in parts):
|
||||
return None
|
||||
return name
|
||||
|
||||
|
||||
def _write_asset(target_dir, relative, payload):
|
||||
target = os.path.join(target_dir, relative)
|
||||
os.makedirs(os.path.dirname(target), exist_ok=True)
|
||||
with open(target, "wb") as handle:
|
||||
handle.write(payload)
|
||||
|
||||
|
||||
def _materialize_assets(archive_path, state_dir):
|
||||
import _cli_version
|
||||
|
||||
base_dir = os.path.join(state_dir, ASSET_DIR_NAME, _cli_version.VERSION)
|
||||
parent_dir = os.path.dirname(base_dir)
|
||||
os.makedirs(parent_dir, exist_ok=True)
|
||||
tmp_dir = tempfile.mkdtemp(dir=parent_dir, prefix=".tmp-")
|
||||
try:
|
||||
ui_dir = os.path.join(tmp_dir, "ui")
|
||||
static_dir = os.path.join(tmp_dir, "static")
|
||||
os.makedirs(ui_dir)
|
||||
os.makedirs(static_dir)
|
||||
with zipfile.ZipFile(archive_path) as archive:
|
||||
for name in archive.namelist():
|
||||
if name == "coreci_broker_client.py":
|
||||
_write_asset(static_dir, name, archive.read(name))
|
||||
elif name.startswith("coreci_broker_ui/") and not name.endswith("/"):
|
||||
relative = _safe_relative(name[len("coreci_broker_ui/"):])
|
||||
if relative is not None:
|
||||
_write_asset(ui_dir, relative, archive.read(name))
|
||||
if os.path.isdir(base_dir):
|
||||
stale_dir = base_dir + ".stale"
|
||||
shutil.rmtree(stale_dir, ignore_errors=True)
|
||||
os.replace(base_dir, stale_dir)
|
||||
try:
|
||||
os.replace(tmp_dir, base_dir)
|
||||
except OSError:
|
||||
os.replace(stale_dir, base_dir)
|
||||
raise
|
||||
shutil.rmtree(stale_dir, ignore_errors=True)
|
||||
else:
|
||||
os.replace(tmp_dir, base_dir)
|
||||
finally:
|
||||
shutil.rmtree(tmp_dir, ignore_errors=True)
|
||||
return {"ui": os.path.join(base_dir, "ui"), "static": os.path.join(base_dir, "static")}
|
||||
|
||||
|
||||
def _apply_asset_roots(roots, env_path):
|
||||
import coreci_broker
|
||||
|
||||
file_values = dict(coreci_broker._parse_env_file(env_path))
|
||||
for key, name in (("BROKER_UI_ROOT", "ui"), ("BROKER_STATIC_ROOT", "static")):
|
||||
if key not in os.environ and key not in file_values:
|
||||
os.environ[key] = roots[name]
|
||||
|
||||
|
||||
def main():
|
||||
if _version_requested():
|
||||
import _cli_version
|
||||
print(_cli_version.VERSION)
|
||||
return 0
|
||||
archive_path = sys.path[0]
|
||||
if archive_path and not os.path.isdir(archive_path):
|
||||
env_path = os.environ.get("BROKER_ENV_PATH") or ".env"
|
||||
roots = _materialize_assets(archive_path, _resolve_state_dir())
|
||||
_apply_asset_roots(roots, env_path)
|
||||
import coreci_broker
|
||||
coreci_broker.run_standalone()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+22
-7
@@ -7,6 +7,7 @@ licence: MIT
|
||||
"""
|
||||
import base64
|
||||
import datetime
|
||||
import errno
|
||||
import hashlib
|
||||
import hmac
|
||||
import http.cookies
|
||||
@@ -728,11 +729,16 @@ def _required_string(value):
|
||||
return value if isinstance(value, str) and value else None
|
||||
|
||||
|
||||
def _valid_user_id(value):
|
||||
def _has_control_chars(value):
|
||||
return any(ord(ch) < 32 or ord(ch) == 127 for ch in value)
|
||||
|
||||
|
||||
def _valid_id(value):
|
||||
if not _required_string(value):
|
||||
return None
|
||||
if (":" in value or "/" in value or "\\" in value
|
||||
or any(ord(ch) < 32 or ord(ch) == 127 for ch in value)):
|
||||
if value in (".", ".."):
|
||||
return None
|
||||
if ":" in value or "/" in value or "\\" in value or _has_control_chars(value):
|
||||
return None
|
||||
return value
|
||||
|
||||
@@ -760,7 +766,7 @@ def _apps_add(request):
|
||||
|
||||
|
||||
def _users_add(request):
|
||||
_add_row(request, "user_allowlist", "user_id", "email", "add_user", _valid_user_id)
|
||||
_add_row(request, "user_allowlist", "user_id", "email", "add_user", _valid_id)
|
||||
|
||||
|
||||
def _set_row_status(request, table, key, status, action):
|
||||
@@ -875,10 +881,10 @@ def _agents_register(request):
|
||||
if len(strings) != 3 or not isinstance(tags, list) or not all(isinstance(v, str) for v in tags):
|
||||
request.send_error_json(400, "invalid registration")
|
||||
return
|
||||
if ":" in agent_id or "/" in agent_id or "\\" in agent_id:
|
||||
if _valid_id(agent_id) is None:
|
||||
request.send_error_json(400, "invalid agent_id")
|
||||
return
|
||||
if any(":" in tag for tag in tags):
|
||||
if any(":" in tag or _has_control_chars(tag) for tag in tags):
|
||||
request.send_error_json(400, "invalid tags")
|
||||
return
|
||||
path = "/api/broker/agents/register"
|
||||
@@ -1343,7 +1349,16 @@ class Tools:
|
||||
|
||||
def __init__(self):
|
||||
self.valves = self.Valves()
|
||||
self._server = _start_guarded_server()
|
||||
try:
|
||||
self._server = _start_guarded_server()
|
||||
except OSError as error:
|
||||
if error.errno != errno.EADDRINUSE:
|
||||
raise
|
||||
raise RuntimeError(
|
||||
f"broker bind failed ({error}): a broker from previously loaded "
|
||||
"module code is still serving the configured bind; restart "
|
||||
"Open WebUI to load the updated broker"
|
||||
) from error
|
||||
|
||||
def status(self) -> str:
|
||||
"""Report the broker server state as a one-line string."""
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Sanctioned secrets loader for the v0.2 distribution scripts (stdlib only).
|
||||
|
||||
Secrets resolve ONLY from env files on disk — never from the process
|
||||
environment (``GITEA_TOKEN`` sits in ``config.security.bash_allowlist
|
||||
.blocked_env_vars``, so shell-env tokens are structurally blocked; the
|
||||
v1.8 lesson: an empty or mis-sourced token makes gitea answer 404 rather
|
||||
than 401, masking the auth failure).
|
||||
|
||||
Parsing mirrors ``coreci_broker._parse_env_file`` semantics exactly:
|
||||
blank lines and ``#`` comments are skipped, keys/values are whitespace-
|
||||
stripped, and a value wrapped in matching single or double quotes is
|
||||
unquoted.
|
||||
|
||||
File precedence — later files win. Locations in order (``root``, then the
|
||||
sanctioned ``.ciagent/`` dir under it); each location in ``config.json``
|
||||
``secrets.env_files`` pattern order (``.env``, ``.env.secrets``, then the
|
||||
``.env.*`` glob, sorted)::
|
||||
|
||||
<root>/.env, <root>/.env.secrets, <root>/.env.*
|
||||
<root>/.ciagent/.env, <root>/.ciagent/.env.secrets, <root>/.ciagent/.env.*
|
||||
|
||||
``load()`` returns the merged mapping; ``get_token()`` enforces the
|
||||
non-empty assert that callers MUST let happen before their first API
|
||||
call. This module never prints secret values.
|
||||
"""
|
||||
|
||||
import glob
|
||||
import os
|
||||
|
||||
ENV_FILE_PATTERNS = (".env", ".env.secrets", ".env.*")
|
||||
CIAGENT_SUBDIR = ".ciagent"
|
||||
|
||||
|
||||
class SecretsError(RuntimeError):
|
||||
"""A required secret is missing or empty in the sanctioned env files."""
|
||||
|
||||
|
||||
def default_root():
|
||||
"""Repo root: the parent directory of this scripts/ folder."""
|
||||
return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
|
||||
def _parse_env_file(path):
|
||||
"""Mirror coreci_broker._parse_env_file (comments, blanks, quotes)."""
|
||||
entries = []
|
||||
if not os.path.isfile(path):
|
||||
return entries
|
||||
with open(path, "r", encoding="utf-8") as handle:
|
||||
for raw in handle:
|
||||
line = raw.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, _, value = (part.strip() for part in line.partition("="))
|
||||
if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
|
||||
value = value[1:-1]
|
||||
if key:
|
||||
entries.append((key, value))
|
||||
return entries
|
||||
|
||||
|
||||
def _candidate_files(root):
|
||||
"""Ordered, deduplicated env-file paths; later files win."""
|
||||
seen = set()
|
||||
files = []
|
||||
for location in (root, os.path.join(root, CIAGENT_SUBDIR)):
|
||||
for pattern in ENV_FILE_PATTERNS:
|
||||
for path in sorted(glob.glob(os.path.join(location, pattern))):
|
||||
if not os.path.isfile(path):
|
||||
continue
|
||||
real = os.path.abspath(path)
|
||||
if real in seen:
|
||||
continue
|
||||
seen.add(real)
|
||||
files.append(real)
|
||||
return files
|
||||
|
||||
|
||||
def load(root=None):
|
||||
"""Merge every sanctioned env file under ``root`` into one dict.
|
||||
|
||||
``root`` defaults to the repo root (the parent of scripts/).
|
||||
"""
|
||||
base = os.path.abspath(root) if root is not None else default_root()
|
||||
values = {}
|
||||
for path in _candidate_files(base):
|
||||
for key, value in _parse_env_file(path):
|
||||
values[key] = value
|
||||
return values
|
||||
|
||||
|
||||
def get_token(var="GITEA_TOKEN", root=None):
|
||||
"""Return the non-empty value of ``var`` from the env files only.
|
||||
|
||||
Raises SecretsError when the value is missing, empty, or whitespace-
|
||||
only — callers must surface this BEFORE any HTTP call (an empty token
|
||||
makes gitea answer 404 and masks the auth failure — the v1.8 lesson).
|
||||
"""
|
||||
token = load(root).get(var, "")
|
||||
if not token.strip():
|
||||
base = os.path.abspath(root) if root is not None else default_root()
|
||||
searched = ", ".join(
|
||||
os.path.join(location, pattern)
|
||||
for location in (base, os.path.join(base, CIAGENT_SUBDIR))
|
||||
for pattern in ENV_FILE_PATTERNS
|
||||
)
|
||||
raise SecretsError(
|
||||
"{} is missing or empty in the sanctioned env files "
|
||||
"(searched: {}); it never resolves from the process "
|
||||
"environment".format(var, searched)
|
||||
)
|
||||
return token.strip()
|
||||
@@ -0,0 +1,141 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build the coreci-broker CLI zipapp artifact (stdlib only).
|
||||
|
||||
Usage: python3 scripts/build_cli.py <version>
|
||||
|
||||
Stages an explicit allowlist into a temp dir, packs it with
|
||||
``python3 -m zipapp ... --python "/usr/bin/env python3" --compress``, then
|
||||
runs a normalize post-pass (G-009) so same-environment rebuilds are
|
||||
byte-identical: entries sorted by name, ``date_time=(1980,1,1,0,0,0)``,
|
||||
``ZIP_DEFLATED``, ``external_attr`` set to constants (files
|
||||
``0o100644 << 16``, dirs ``0o40755 << 16`` — never preserved/umask-derived),
|
||||
shebang re-prepended, exec bit preserved. Prints the artifact sha256.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import io
|
||||
import os
|
||||
import shutil
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import zipfile
|
||||
|
||||
ROOT_SOURCES = ("coreci_broker.py", "coreci_broker_client.py")
|
||||
UI_SOURCE_DIR = "coreci_broker_ui"
|
||||
SHIM_SOURCE = os.path.join("cli", "__main__.py")
|
||||
STAGED_SHIM = "__main__.py"
|
||||
VERSION_MODULE = "_cli_version.py"
|
||||
FIXED_DATE_TIME = (1980, 1, 1, 0, 0, 0)
|
||||
FILE_EXTERNAL_ATTR = 0o100644 << 16
|
||||
DIR_EXTERNAL_ATTR = 0o40755 << 16
|
||||
|
||||
|
||||
def repo_root():
|
||||
return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
|
||||
def artifact_name(version):
|
||||
tag = version[1:] if version.startswith("v") else version
|
||||
return "coreci-broker-v{}.pyz".format(tag)
|
||||
|
||||
|
||||
def stage_sources(repo, staging):
|
||||
for name in ROOT_SOURCES:
|
||||
shutil.copyfile(os.path.join(repo, name), os.path.join(staging, name))
|
||||
ui_root = os.path.join(repo, UI_SOURCE_DIR)
|
||||
staged_ui = os.path.join(staging, UI_SOURCE_DIR)
|
||||
for current, _dirs, files in os.walk(ui_root):
|
||||
relative = os.path.relpath(current, ui_root)
|
||||
target_dir = staged_ui if relative == "." else os.path.join(staged_ui, relative)
|
||||
os.makedirs(target_dir, exist_ok=True)
|
||||
for name in sorted(files):
|
||||
shutil.copyfile(os.path.join(current, name), os.path.join(target_dir, name))
|
||||
shutil.copyfile(os.path.join(repo, SHIM_SOURCE), os.path.join(staging, STAGED_SHIM))
|
||||
|
||||
|
||||
def write_version_module(staging, version):
|
||||
with open(os.path.join(staging, VERSION_MODULE), "w", encoding="utf-8") as handle:
|
||||
handle.write('VERSION = "{}"\n'.format(version))
|
||||
|
||||
|
||||
def run_zipapp(staging, artifact):
|
||||
command = [
|
||||
sys.executable, "-m", "zipapp", staging,
|
||||
"-o", artifact,
|
||||
"--python", "/usr/bin/env python3",
|
||||
"--compress",
|
||||
]
|
||||
completed = subprocess.run(command, capture_output=True, text=True)
|
||||
if completed.returncode != 0:
|
||||
detail = (completed.stderr or completed.stdout).strip()
|
||||
raise RuntimeError(
|
||||
"zipapp failed with code {}: {}".format(completed.returncode, detail)
|
||||
)
|
||||
|
||||
|
||||
def normalize_archive(artifact):
|
||||
with open(artifact, "rb") as handle:
|
||||
raw = handle.read()
|
||||
if not raw.startswith(b"#!"):
|
||||
raise RuntimeError("zipapp artifact is missing its shebang prefix")
|
||||
shebang = raw.split(b"\n", 1)[0] + b"\n"
|
||||
entries = []
|
||||
with zipfile.ZipFile(io.BytesIO(raw)) as archive:
|
||||
for info in archive.infolist():
|
||||
payload = b"" if info.is_dir() else archive.read(info)
|
||||
entries.append((info.filename, payload))
|
||||
entries.sort(key=lambda entry: entry[0])
|
||||
body = io.BytesIO()
|
||||
with zipfile.ZipFile(body, "w", compression=zipfile.ZIP_DEFLATED) as archive:
|
||||
for name, payload in entries:
|
||||
info = zipfile.ZipInfo(name, date_time=FIXED_DATE_TIME)
|
||||
info.compress_type = zipfile.ZIP_DEFLATED
|
||||
info.external_attr = DIR_EXTERNAL_ATTR if name.endswith("/") else FILE_EXTERNAL_ATTR
|
||||
archive.writestr(info, payload)
|
||||
mode = os.stat(artifact).st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH
|
||||
temp_path = artifact + ".tmp"
|
||||
with open(temp_path, "wb") as handle:
|
||||
handle.write(shebang)
|
||||
handle.write(body.getvalue())
|
||||
os.replace(temp_path, artifact)
|
||||
os.chmod(artifact, mode)
|
||||
|
||||
|
||||
def sha256_file(path):
|
||||
digest = hashlib.sha256()
|
||||
with open(path, "rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(65536), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def build(version, repo=None, dist_dir=None):
|
||||
repo = os.path.abspath(repo or repo_root())
|
||||
dist_dir = dist_dir or os.path.join(repo, "dist")
|
||||
os.makedirs(dist_dir, exist_ok=True)
|
||||
artifact = os.path.join(dist_dir, artifact_name(version))
|
||||
with tempfile.TemporaryDirectory(prefix="coreci-broker-cli-") as staging:
|
||||
stage_sources(repo, staging)
|
||||
write_version_module(staging, version)
|
||||
run_zipapp(staging, artifact)
|
||||
normalize_archive(artifact)
|
||||
return artifact
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Build the coreci-broker CLI zipapp artifact."
|
||||
)
|
||||
parser.add_argument("version", help="release version of the artifact, e.g. v0.2.1")
|
||||
args = parser.parse_args(argv)
|
||||
artifact = build(args.version)
|
||||
print(artifact)
|
||||
print("sha256: {}".format(sha256_file(artifact)))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,844 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Daemonless OCI image publish client (D-019; stdlib only).
|
||||
|
||||
Usage:
|
||||
python3 scripts/publish_image.py <tag> [--smoke]
|
||||
[--artifact PATH] [--hub-registry URL] [--hub-auth URL]
|
||||
[--gitea-registry URL] [--env-root DIR] [--owner O] [--repo R]
|
||||
[--base-manifest DIGEST] [--timeout SECS]
|
||||
|
||||
Flow (REQ-23, RESEARCH §3): pull the pinned ``python:3.11-slim`` base
|
||||
from the public registry anonymously (token -> OCI index -> resolve the
|
||||
linux/amd64 child manifest by digest -> config + layer blobs; digests
|
||||
recorded for provenance; ``--base-manifest`` overrides the tag
|
||||
resolution for exact re-publishes) -> authenticate to the gitea
|
||||
container registry (``GET /v2/token`` with Basic credentials — the
|
||||
gitea token comes only from the sanctioned env files via
|
||||
``scripts/_env_secrets.py``; the endpoint answers 200 even anonymously
|
||||
with an empty-scope JWT, so the issued token must be non-empty and
|
||||
failures must be loud — never treat a 200 alone as authorization) ->
|
||||
push: HEAD each blob first (200 -> skip — retries resume cheap; the
|
||||
~45.7 MB base layers are re-uploaded once per milestone because the
|
||||
registries cannot cross-mount) -> monolithic blob upload
|
||||
``POST /v2/<name>/blobs/uploads/?digest=`` (202 + ``Location`` ->
|
||||
two-step PUT) with explicit Location-following on 307/308 (stdlib
|
||||
urllib does NOT method-preserve on PUT/POST redirects) -> deterministic
|
||||
broker layer (sorted tar entries, fixed metadata, member at the stable
|
||||
path ``/app/coreci-broker.pyz``, ``gzip.compress(..., mtime=0)``) ->
|
||||
new config blob (base config + appended diff_id; Entrypoint
|
||||
``["python3", "/app/coreci-broker.pyz"]``, ``Cmd []``, ``WorkingDir
|
||||
"/state"``, version label; serialized with sort_keys) -> OCI manifest
|
||||
PUT with the base layer descriptors reused verbatim under ``<tag>``
|
||||
FIRST, then ``latest`` (D-021).
|
||||
|
||||
Exit codes: 0 only after both manifest PUTs confirm 201; 1 after a
|
||||
remote failure — bounded retries (3 attempts, 2s/4s backoff) exhausted
|
||||
OR a permanent registry answer — printing the machine-readable
|
||||
``IMAGE_PENDING <image>:<tag>`` marker for the ship escalation (never
|
||||
blocks tag+merge); 2 for local precondition errors (missing artifact,
|
||||
token, or config — nothing attempted remotely, no marker). Token values
|
||||
are never printed.
|
||||
|
||||
Smoke mode (``--smoke``; G-013) — read-only pre-ship verify. NEVER part
|
||||
of the default publish flow (the flag dispatches an early return; the
|
||||
publish path above never consults it): four GET-only checks against
|
||||
the live endpoints, run in order and stopping at the first failure —
|
||||
nothing is uploaded, no manifest is PUT, no remote state is mutated.
|
||||
|
||||
1. ``GET {gitea}/v2/`` must answer 401 with a Bearer challenge whose
|
||||
realm is the gitea token endpoint (the realm is recorded for 2).
|
||||
2. Token->JWT exchange at that recorded realm
|
||||
(``?service=container_registry&scope=repository:{o}/{r}:pull,push``)
|
||||
with Basic auth — the gitea token resolves ONLY from the sanctioned
|
||||
env files via ``scripts/_env_secrets.py`` (never the shell
|
||||
environment); an empty token in the 200 answer is the anonymous-200
|
||||
trap and fails loudly.
|
||||
3. ``GET {gitea}/api/v1/repos/{o}/{r}/releases/tags/<tag>`` — 200
|
||||
(exists) and 404 (not yet created) are BOTH acceptable pre-ship and
|
||||
the line says which; the release's asset names print as a fact, and
|
||||
a hard assertion binds ONLY when the tag is exactly ``v0.2.0``
|
||||
(G-021 forge-side confirmation: the phase-0 release predates the
|
||||
CLI and must carry NO ``.pyz`` asset, so any ``.pyz`` in its asset
|
||||
list fails the smoke — the structural half of the guarantee,
|
||||
release.py taking the tag as a positional argument with no v0.2.0
|
||||
code path, is already machine-checked in the suite).
|
||||
4. Docker Hub anonymous index resolve of ``python:3.11-slim`` with the
|
||||
full Accept set must yield an index whose linux/amd64 child is
|
||||
resolved; the amd64 child digest is printed — the base-drift fact
|
||||
the P3-05 ship compares against expectations.
|
||||
|
||||
Smoke exit codes mirror the publish flow: 0 all four PASS; 1 with one
|
||||
clear FAILED line per failure (checks after the failed one never run);
|
||||
2 for local preconditions (missing token — nothing attempted
|
||||
remotely). Smoke never prints token values and never emits
|
||||
``IMAGE_PENDING`` (nothing was attempted, so nothing is pending).
|
||||
|
||||
Live gating (G-013): the LIVE smoke executes only at the P3-05 ship
|
||||
step, which sets ``PUBLISH_SMOKE=1`` in the environment before invoking
|
||||
this mode — this script neither reads nor requires that variable (the
|
||||
SHIP GATE owns the gating). The smoke's LOGIC is what the default
|
||||
unittest suite exercises (stub-tested, zero network access); the live
|
||||
smoke itself never runs in the default suite.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import gzip
|
||||
import hashlib
|
||||
import http.client
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import tarfile
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
_SCRIPTS_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
if _SCRIPTS_DIR not in sys.path:
|
||||
sys.path.insert(0, _SCRIPTS_DIR)
|
||||
|
||||
import _env_secrets # sibling import (scripts/ dir on sys.path)
|
||||
|
||||
METADATA_TIMEOUT_SECONDS = 30
|
||||
BLOB_TIMEOUT_SECONDS = 120
|
||||
MAX_ATTEMPTS = 3
|
||||
BACKOFF_SECONDS = (2, 4)
|
||||
MAX_REDIRECTS = 5
|
||||
BASE_REPO = "library/python"
|
||||
BASE_TAG = "3.11-slim"
|
||||
HUB_AUTH_SERVICE = "registry.docker.io"
|
||||
GITEA_AUTH_SERVICE = "container_registry"
|
||||
# The smoke-mode G-021 forge-side no-.pyz assertion binds ONLY this
|
||||
# tag: the phase-0 release predates the CLI (D-020 starts at v0.2.1).
|
||||
SMOKE_G021_TAG = "v0.2.0"
|
||||
|
||||
OCI_INDEX = "application/vnd.oci.image.index.v1+json"
|
||||
OCI_MANIFEST = "application/vnd.oci.image.manifest.v1+json"
|
||||
OCI_CONFIG = "application/vnd.oci.image.config.v1+json"
|
||||
OCI_LAYER = "application/vnd.oci.image.layer.v1.tar+gzip"
|
||||
DOCKER_LIST = "application/vnd.docker.distribution.manifest.list.v2+json"
|
||||
DOCKER_MANIFEST = "application/vnd.docker.distribution.manifest.v2+json"
|
||||
ACCEPT_MANIFESTS = ", ".join(
|
||||
(OCI_INDEX, DOCKER_LIST, OCI_MANIFEST, DOCKER_MANIFEST))
|
||||
|
||||
|
||||
class PublishError(Exception):
|
||||
"""Image-publisher failure base."""
|
||||
|
||||
|
||||
class TransientFailure(PublishError):
|
||||
"""Connection error or 5xx persisting past the bounded retries."""
|
||||
|
||||
|
||||
class RegistryApiError(PublishError):
|
||||
"""Permanent registry answer carrying the remote detail."""
|
||||
|
||||
|
||||
class LocalError(PublishError):
|
||||
"""Local precondition failure (nothing was attempted remotely)."""
|
||||
|
||||
|
||||
def repo_root():
|
||||
return os.path.dirname(_SCRIPTS_DIR)
|
||||
|
||||
|
||||
def load_release_defaults(root=None):
|
||||
"""Read the release block from .ciagent/config.json ({} if absent)."""
|
||||
path = os.path.join(root or repo_root(), ".ciagent", "config.json")
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as handle:
|
||||
config = json.load(handle)
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
block = config.get("release")
|
||||
return block if isinstance(block, dict) else {}
|
||||
|
||||
|
||||
def _error_detail(body):
|
||||
try:
|
||||
payload = json.loads(body.decode("utf-8", "replace"))
|
||||
except ValueError:
|
||||
text = body.decode("utf-8", "replace").strip()
|
||||
return text[:200] if text else "<empty body>"
|
||||
if isinstance(payload, dict):
|
||||
for key in ("errors", "message"):
|
||||
if payload.get(key):
|
||||
return json.dumps(payload[key])[:200]
|
||||
return json.dumps(payload)[:200] if not isinstance(payload, str) else payload[:200]
|
||||
|
||||
|
||||
def _header(headers, name):
|
||||
"""Case-insensitive response-header lookup — ``_attempt_once``
|
||||
returns a plain dict whose keys keep the wire casing, and registry
|
||||
header casing is not guaranteed (e.g. WWW-Authenticate)."""
|
||||
for key, value in headers.items():
|
||||
if key.lower() == name.lower():
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def _parse_json(body, context, status):
|
||||
"""Parse a remote JSON body (token / manifest / config answers).
|
||||
|
||||
Malformed bodies raise RegistryApiError — never an uncaught
|
||||
ValueError — so main()'s handler keeps the exit-1 + IMAGE_PENDING
|
||||
marker contract for this failure class (CWE-754).
|
||||
"""
|
||||
try:
|
||||
return json.loads(body.decode("utf-8", "replace"))
|
||||
except ValueError as error:
|
||||
raise RegistryApiError(
|
||||
"{} answered malformed JSON ({})".format(context, error),
|
||||
status, body)
|
||||
|
||||
|
||||
def _attempt_once(method, url, headers, data, timeout):
|
||||
"""One HTTP exchange; HTTPError becomes a completed answer."""
|
||||
request = urllib.request.Request(url, data=data, method=method)
|
||||
for key, value in headers.items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=timeout) as response:
|
||||
body = b"" if method == "HEAD" else response.read()
|
||||
return response.status, body, dict(response.headers.items())
|
||||
except urllib.error.HTTPError as error:
|
||||
body = b"" if method == "HEAD" else error.read()
|
||||
return error.code, body, dict((error.headers or {}).items())
|
||||
|
||||
|
||||
def _origin(url):
|
||||
"""(scheme, netloc) of a URL — two URLs share an origin iff both match."""
|
||||
parts = urllib.parse.urlsplit(url)
|
||||
return (parts.scheme, parts.netloc)
|
||||
|
||||
|
||||
def _attempt_following(method, url, headers, data, timeout):
|
||||
"""One attempt, following 307/308 redirects explicitly (urllib does
|
||||
NOT method-preserve on PUT/POST — the Location may be relative).
|
||||
|
||||
Cross-origin discipline (CWE-200): when a followed Location resolves
|
||||
to a different origin than the ORIGINAL request's, the Authorization
|
||||
header is stripped from the re-issued request — the registry JWT
|
||||
must never reach another origin (blob-upload Locations are commonly
|
||||
presigned external storage). Same-origin redirects keep it; once
|
||||
stripped it is never re-attached."""
|
||||
current_url = url
|
||||
current_headers = dict(headers)
|
||||
for _ in range(MAX_REDIRECTS + 1):
|
||||
status, body, resp_headers = _attempt_once(
|
||||
method, current_url, current_headers, data, timeout)
|
||||
if status in (307, 308) and resp_headers.get("Location"):
|
||||
next_url = urllib.parse.urljoin(
|
||||
current_url, resp_headers["Location"])
|
||||
if _origin(next_url) != _origin(url):
|
||||
current_headers.pop("Authorization", None)
|
||||
current_url = next_url
|
||||
continue
|
||||
return status, body, resp_headers
|
||||
raise RegistryApiError(
|
||||
"too many redirects following {} {}".format(method, url), 500, b"")
|
||||
|
||||
|
||||
def _request(method, url, headers, data, timeout,
|
||||
max_attempts=None, backoff=None):
|
||||
"""Bounded-retry HTTP: retries connection errors and 5xx answers.
|
||||
|
||||
Returns (status, body, response_headers) for any answer below 500;
|
||||
a 4xx returns immediately (permanent). Raises TransientFailure once
|
||||
the retry budget is exhausted.
|
||||
"""
|
||||
attempts = MAX_ATTEMPTS if max_attempts is None else int(max_attempts)
|
||||
delays = BACKOFF_SECONDS if backoff is None else tuple(backoff)
|
||||
failure = "unknown failure"
|
||||
for attempt in range(1, attempts + 1):
|
||||
try:
|
||||
status, body, resp_headers = _attempt_following(
|
||||
method, url, headers, data, timeout)
|
||||
except (OSError, http.client.HTTPException) as error:
|
||||
status = None
|
||||
failure = "connection error: {}: {}".format(
|
||||
type(error).__name__, error)
|
||||
else:
|
||||
if status < 500:
|
||||
return status, body, resp_headers
|
||||
failure = "HTTP {}: {}".format(status, _error_detail(body))
|
||||
if attempt < attempts and delays:
|
||||
time.sleep(delays[min(attempt - 1, len(delays) - 1)])
|
||||
raise TransientFailure(
|
||||
"{} {} failed after {} attempt(s): {}".format(
|
||||
method, url, attempts, failure))
|
||||
|
||||
|
||||
def _hub_token(hub_auth, scope):
|
||||
query = urllib.parse.urlencode(
|
||||
{"service": HUB_AUTH_SERVICE, "scope": scope})
|
||||
status, body, _ = _request(
|
||||
"GET", "{}?{}".format(hub_auth, query), {}, None,
|
||||
METADATA_TIMEOUT_SECONDS)
|
||||
if status != 200:
|
||||
raise RegistryApiError("hub token request failed", status, body)
|
||||
payload = _parse_json(body, "hub token endpoint", status)
|
||||
token = ""
|
||||
if isinstance(payload, dict):
|
||||
token = payload.get("token") or payload.get("access_token") or ""
|
||||
if not token:
|
||||
raise RegistryApiError("hub token endpoint returned no token",
|
||||
status, body)
|
||||
return token
|
||||
|
||||
|
||||
def _fetch_manifest(registry, repository, ref, bearer, accept=ACCEPT_MANIFESTS):
|
||||
url = "{}/v2/{}/manifests/{}".format(registry.rstrip("/"), repository, ref)
|
||||
status, body, _ = _request(
|
||||
"GET", url, {"Authorization": "Bearer {}".format(bearer),
|
||||
"Accept": accept}, None, METADATA_TIMEOUT_SECONDS)
|
||||
if status != 200:
|
||||
raise RegistryApiError(
|
||||
"manifest fetch {} {}".format(repository, ref), status, body)
|
||||
return _parse_json(
|
||||
body, "manifest fetch {} {}".format(repository, ref), status)
|
||||
|
||||
|
||||
def _resolve_amd64_child(index):
|
||||
"""linux/amd64 child digest from a base index; malformed index
|
||||
shapes degrade to the not-found RegistryApiError, never a crash."""
|
||||
manifests = index.get("manifests") if isinstance(index, dict) else None
|
||||
if not isinstance(manifests, list):
|
||||
manifests = []
|
||||
for descriptor in manifests:
|
||||
if not isinstance(descriptor, dict):
|
||||
continue
|
||||
platform = descriptor.get("platform")
|
||||
if not isinstance(platform, dict):
|
||||
continue
|
||||
if platform.get("os") == "linux" and platform.get(
|
||||
"architecture") == "amd64":
|
||||
return descriptor.get("digest")
|
||||
raise RegistryApiError(
|
||||
"no linux/amd64 child manifest in the base index", 404,
|
||||
json.dumps(manifests).encode("utf-8", "replace"))
|
||||
|
||||
|
||||
def _fetch_blob(registry, repository, digest, bearer, timeout):
|
||||
url = "{}/v2/{}/blobs/{}".format(
|
||||
registry.rstrip("/"), repository, digest)
|
||||
status, body, _ = _request(
|
||||
"GET", url, {"Authorization": "Bearer {}".format(bearer)}, None,
|
||||
timeout)
|
||||
if status != 200:
|
||||
raise RegistryApiError("blob fetch {}".format(digest), status, body)
|
||||
if "sha256:" + hashlib.sha256(body).hexdigest() != digest:
|
||||
raise RegistryApiError("blob digest mismatch {}".format(digest),
|
||||
500, b"")
|
||||
return body
|
||||
|
||||
|
||||
def _child_parts(child):
|
||||
"""Validated (config descriptor, layer descriptors) of a base child
|
||||
manifest — missing or malformed keys raise a clear RegistryApiError
|
||||
(never an uncaught KeyError), keeping the marker contract for this
|
||||
failure class (CWE-754)."""
|
||||
def malformed(reason):
|
||||
return RegistryApiError(
|
||||
"malformed base child manifest: {}".format(reason), 500,
|
||||
json.dumps(child).encode("utf-8", "replace"))
|
||||
|
||||
if not isinstance(child, dict):
|
||||
raise malformed("not a JSON object")
|
||||
config = child.get("config")
|
||||
if not isinstance(config, dict) or not config.get("digest"):
|
||||
raise malformed("missing or empty config.digest")
|
||||
layers = child.get("layers")
|
||||
if not isinstance(layers, list):
|
||||
raise malformed("missing or non-list layers")
|
||||
for descriptor in layers:
|
||||
if not isinstance(descriptor, dict) or not descriptor.get("digest"):
|
||||
raise malformed("layer descriptor without a digest")
|
||||
return config, layers
|
||||
|
||||
|
||||
def _gitea_jwt(token_url, owner, repo, token, timeout):
|
||||
"""Token->JWT exchange at a token endpoint URL (Basic auth).
|
||||
|
||||
``token_url`` is the FULL endpoint: the registry's ``/v2/token``
|
||||
for the publish flow, or the realm recorded from a ``/v2/``
|
||||
challenge for the smoke (the standard token-auth flow).
|
||||
"""
|
||||
query = urllib.parse.urlencode({
|
||||
"service": GITEA_AUTH_SERVICE,
|
||||
"scope": "repository:{}/{}:pull,push".format(owner, repo),
|
||||
})
|
||||
basic = base64.b64encode(
|
||||
"publisher:{}".format(token).encode("utf-8")).decode("ascii")
|
||||
status, body, _ = _request(
|
||||
"GET", "{}?{}".format(token_url, query),
|
||||
{"Authorization": "Basic {}".format(basic)}, None, timeout)
|
||||
if status != 200:
|
||||
raise RegistryApiError("gitea registry token request failed",
|
||||
status, body)
|
||||
payload = _parse_json(body, "gitea registry token endpoint", status)
|
||||
jwt = (payload.get("token") or "") if isinstance(payload, dict) else ""
|
||||
if not jwt:
|
||||
# The endpoint answers 200 even anonymously with an unusable
|
||||
# (empty-scope) JWT — a 200 proves nothing about push rights.
|
||||
raise RegistryApiError(
|
||||
"gitea registry token endpoint returned an empty token "
|
||||
"(anonymous-200 trap refused; never treat 200 as authorization)",
|
||||
status, body)
|
||||
return jwt
|
||||
|
||||
|
||||
def _bearer_challenge(www_authenticate):
|
||||
"""Realm recorded from a Bearer WWW-Authenticate challenge.
|
||||
|
||||
Returns None when the header is absent, is not a Bearer challenge,
|
||||
or carries no realm — a challenge without a realm cannot be
|
||||
followed to the token endpoint, so the smoke treats that as no
|
||||
challenge at all.
|
||||
"""
|
||||
if not www_authenticate:
|
||||
return None
|
||||
parts = www_authenticate.strip().split(None, 1)
|
||||
if len(parts) < 2 or parts[0].lower() != "bearer":
|
||||
return None
|
||||
match = re.search(r'\brealm\s*=\s*"([^"]*)"', parts[1])
|
||||
return match.group(1) if match else None
|
||||
|
||||
|
||||
def _asset_names(release):
|
||||
"""Asset display names from a gitea release payload — odd shapes
|
||||
degrade to [] (the read-only smoke never crashes on them)."""
|
||||
if not isinstance(release, dict):
|
||||
return []
|
||||
assets = release.get("assets")
|
||||
if not isinstance(assets, list):
|
||||
return []
|
||||
names = []
|
||||
for asset in assets:
|
||||
if isinstance(asset, dict) and isinstance(asset.get("name"), str):
|
||||
names.append(asset["name"])
|
||||
return names
|
||||
|
||||
|
||||
def _blob_exists(gitea_registry, name, digest, jwt, timeout):
|
||||
url = "{}/v2/{}/blobs/{}".format(
|
||||
gitea_registry.rstrip("/"), name, digest)
|
||||
status, _, _ = _request(
|
||||
"HEAD", url, {"Authorization": "Bearer {}".format(jwt)}, None,
|
||||
timeout)
|
||||
return status == 200
|
||||
|
||||
|
||||
def _upload_blob(gitea_registry, name, digest, blob, jwt, timeout):
|
||||
base_url = gitea_registry.rstrip("/")
|
||||
url = "{}/v2/{}/blobs/uploads/?digest={}".format(base_url, name, digest)
|
||||
headers = {"Authorization": "Bearer {}".format(jwt),
|
||||
"Content-Type": "application/octet-stream"}
|
||||
status, body, resp_headers = _request(
|
||||
"POST", url, headers, blob, timeout)
|
||||
if status in (200, 201):
|
||||
return
|
||||
if status == 202 and resp_headers.get("Location"):
|
||||
location = urllib.parse.urljoin(url, resp_headers["Location"])
|
||||
if "digest=" not in location:
|
||||
separator = "&" if "?" in location else "?"
|
||||
location = "{}{}digest={}".format(location, separator, digest)
|
||||
status, body, _ = _request(
|
||||
"PUT", location, headers, blob, timeout)
|
||||
if status in (200, 201):
|
||||
return
|
||||
raise RegistryApiError("blob upload {}".format(digest), status, body)
|
||||
|
||||
|
||||
def _push_blob_if_missing(gitea_registry, name, digest, blob, jwt, timeout):
|
||||
"""HEAD-skip then upload — retries resume cheap."""
|
||||
if _blob_exists(gitea_registry, name, digest, jwt, timeout):
|
||||
return False
|
||||
_upload_blob(gitea_registry, name, digest, blob, jwt, timeout)
|
||||
return True
|
||||
|
||||
|
||||
def build_broker_layer(artifact_bytes):
|
||||
"""Deterministic layer carrying /app/coreci-broker.pyz.
|
||||
|
||||
Returns (gzip_blob, layer_digest, diff_id): sorted single entry,
|
||||
fixed TarInfo metadata, gzip with mtime=0 — byte-stable rebuilds.
|
||||
"""
|
||||
info = tarfile.TarInfo("app/coreci-broker.pyz")
|
||||
info.size = len(artifact_bytes)
|
||||
info.mode = 0o644
|
||||
info.uid = 0
|
||||
info.gid = 0
|
||||
info.uname = ""
|
||||
info.gname = ""
|
||||
info.mtime = 0
|
||||
buffer = io.BytesIO()
|
||||
with tarfile.open(fileobj=buffer, mode="w",
|
||||
format=tarfile.GNU_FORMAT) as archive:
|
||||
archive.addfile(info, io.BytesIO(artifact_bytes))
|
||||
raw = buffer.getvalue()
|
||||
compressed = gzip.compress(raw, compresslevel=9, mtime=0)
|
||||
layer_digest = "sha256:" + hashlib.sha256(compressed).hexdigest()
|
||||
diff_id = "sha256:" + hashlib.sha256(raw).hexdigest()
|
||||
return compressed, layer_digest, diff_id
|
||||
|
||||
|
||||
def build_config_blob(base_config, diff_id, tag):
|
||||
"""Base config + appended diff_id + CLI entrypoint/WorkingDir/label.
|
||||
|
||||
Serialized with sort_keys — deterministic config digest.
|
||||
"""
|
||||
config = dict(base_config)
|
||||
rootfs = dict(base_config.get("rootfs") or {})
|
||||
rootfs["diff_ids"] = list(rootfs.get("diff_ids") or []) + [diff_id]
|
||||
if "type" not in rootfs:
|
||||
rootfs["type"] = "layers"
|
||||
config["rootfs"] = rootfs
|
||||
inner = dict(base_config.get("config") or {})
|
||||
inner["Entrypoint"] = ["python3", "/app/coreci-broker.pyz"]
|
||||
inner["Cmd"] = []
|
||||
inner["WorkingDir"] = "/state"
|
||||
inner["Labels"] = {"org.opencontainers.image.version": tag}
|
||||
config["config"] = inner
|
||||
return json.dumps(config, sort_keys=True,
|
||||
separators=(",", ":")).encode("utf-8")
|
||||
|
||||
|
||||
def build_manifest_blob(base_layer_descriptors, broker_layer_descriptor,
|
||||
config_descriptor):
|
||||
manifest = {
|
||||
"schemaVersion": 2,
|
||||
"mediaType": OCI_MANIFEST,
|
||||
"config": config_descriptor,
|
||||
"layers": list(base_layer_descriptors) + [broker_layer_descriptor],
|
||||
}
|
||||
return json.dumps(manifest, sort_keys=True,
|
||||
separators=(",", ":")).encode("utf-8")
|
||||
|
||||
|
||||
def _descriptor(digest, size, media_type):
|
||||
return {"mediaType": media_type, "digest": digest, "size": size}
|
||||
|
||||
|
||||
def _put_manifest(gitea_registry, name, tag, manifest_blob, jwt, timeout):
|
||||
url = "{}/v2/{}/manifests/{}".format(
|
||||
gitea_registry.rstrip("/"), name, tag)
|
||||
status, body, _ = _request(
|
||||
"PUT", url, {"Authorization": "Bearer {}".format(jwt),
|
||||
"Content-Type": OCI_MANIFEST}, manifest_blob, timeout)
|
||||
if status not in (200, 201):
|
||||
raise RegistryApiError("manifest PUT {}".format(tag), status, body)
|
||||
return "sha256:" + hashlib.sha256(manifest_blob).hexdigest()
|
||||
|
||||
|
||||
def _load_artifact(path):
|
||||
if not os.path.isfile(path):
|
||||
raise LocalError("artifact not found: {}".format(path))
|
||||
with open(path, "rb") as handle:
|
||||
return handle.read()
|
||||
|
||||
|
||||
def smoke(tag, gitea_registry, hub_registry, hub_auth, owner, repo,
|
||||
timeout, env_root):
|
||||
"""G-013 read-only live smoke: the four GET-only pre-ship checks.
|
||||
|
||||
Never part of the default publish flow; never mutates remote state
|
||||
(no uploads, no manifest PUTs). The LIVE invocation happens only
|
||||
at the P3-05 ship step, which gates it by setting PUBLISH_SMOKE=1
|
||||
in the environment — this script neither reads nor requires that
|
||||
variable (the SHIP GATE owns the gating). The default unittest
|
||||
suite exercises this LOGIC against stubs (zero network).
|
||||
|
||||
Checks, in order, stopping at the first failure (see the module
|
||||
docstring for the full G-021 tie-in): [1/4] the gitea /v2/ 401
|
||||
Bearer challenge (realm recorded); [2/4] the token->JWT exchange at
|
||||
the recorded realm — the gitea token resolves only from the
|
||||
sanctioned env files via ``_env_secrets.get_token`` (never the
|
||||
shell environment), and an empty token is the anonymous-200 trap
|
||||
refused loudly; [3/4] the gitea API release-by-tag GET — 200
|
||||
(exists) and 404 (not yet created) are both acceptable pre-ship,
|
||||
asset names print as a fact, and the hard no-.pyz assertion binds
|
||||
ONLY the tag v0.2.0 exactly (G-021 forge-side confirmation);
|
||||
[4/4] the Docker Hub anonymous index resolve of python:3.11-slim —
|
||||
the linux/amd64 child digest prints (the drift fact P3-05
|
||||
compares against expectations).
|
||||
|
||||
Returns 0 (all PASS), or 1 after one clear FAILED line (token
|
||||
values are never printed; IMAGE_PENDING is never emitted — nothing
|
||||
was attempted, so nothing is pending). Local preconditions (a
|
||||
missing token) raise ``_env_secrets.SecretsError`` BEFORE any
|
||||
request; main() maps that to exit 2.
|
||||
"""
|
||||
gitea_token = _env_secrets.get_token(root=env_root)
|
||||
metadata_timeout = min(METADATA_TIMEOUT_SECONDS, timeout)
|
||||
gitea_base = gitea_registry.rstrip("/")
|
||||
|
||||
# [1/4] gitea /v2/ auth challenge (anonymous; realm recorded).
|
||||
try:
|
||||
status, _, resp_headers = _request(
|
||||
"GET", "{}/v2/".format(gitea_base), {}, None, metadata_timeout)
|
||||
except (TransientFailure, RegistryApiError) as error:
|
||||
print("FAILED [1/4] gitea /v2/ auth challenge: {}".format(error))
|
||||
return 1
|
||||
challenge = _header(resp_headers, "WWW-Authenticate")
|
||||
realm = _bearer_challenge(challenge)
|
||||
if status != 401 or not realm:
|
||||
print("FAILED [1/4] gitea /v2/ auth challenge: expected 401 with "
|
||||
"a Bearer challenge, got HTTP {} (WWW-Authenticate: {!r})"
|
||||
.format(status, challenge))
|
||||
return 1
|
||||
print("PASS [1/4] gitea /v2/ auth challenge: 401 + Bearer challenge "
|
||||
"(realm {})".format(realm))
|
||||
|
||||
# [2/4] token->JWT exchange at the recorded realm (Basic auth).
|
||||
try:
|
||||
_gitea_jwt(realm, owner, repo, gitea_token, metadata_timeout)
|
||||
except (TransientFailure, RegistryApiError) as error:
|
||||
print("FAILED [2/4] gitea token->JWT exchange at {}: {}".format(
|
||||
realm, error))
|
||||
return 1
|
||||
print("PASS [2/4] gitea token->JWT exchange: 200 + non-empty token "
|
||||
"(service {}, scope repository:{}/{}:pull,push)".format(
|
||||
GITEA_AUTH_SERVICE, owner, repo))
|
||||
|
||||
# [3/4] release-by-tag GET (gitea API; 200 exists, 404 pre-ship ok).
|
||||
release_url = "{}/api/v1/repos/{}/{}/releases/tags/{}".format(
|
||||
gitea_base, owner, repo, urllib.parse.quote(tag, safe=""))
|
||||
try:
|
||||
status, body, _ = _request(
|
||||
"GET", release_url,
|
||||
{"Authorization": "token {}".format(gitea_token)}, None,
|
||||
metadata_timeout)
|
||||
except (TransientFailure, RegistryApiError) as error:
|
||||
print("FAILED [3/4] release-by-tag {}: {}".format(tag, error))
|
||||
return 1
|
||||
if status == 404:
|
||||
print("PASS [3/4] release-by-tag {}: not yet created (404) — "
|
||||
"acceptable pre-ship".format(tag))
|
||||
elif status != 200:
|
||||
print("FAILED [3/4] release-by-tag {}: expected 200 or 404, got "
|
||||
"HTTP {}".format(tag, status))
|
||||
return 1
|
||||
else:
|
||||
try:
|
||||
release = _parse_json(
|
||||
body, "release-by-tag {}".format(tag), status)
|
||||
except RegistryApiError as error:
|
||||
print("FAILED [3/4] release-by-tag {}: {}".format(tag, error))
|
||||
return 1
|
||||
asset_names = _asset_names(release)
|
||||
pyz_assets = [name for name in asset_names
|
||||
if name.endswith(".pyz")]
|
||||
if tag == SMOKE_G021_TAG and pyz_assets:
|
||||
print("FAILED [3/4] release-by-tag {}: G-021 violation — the "
|
||||
"phase-0 release must carry no .pyz asset, found: {}"
|
||||
.format(tag, ", ".join(pyz_assets)))
|
||||
return 1
|
||||
listing = ", ".join(asset_names) if asset_names else "(none)"
|
||||
note = (" — G-021 confirmed: no .pyz asset"
|
||||
if tag == SMOKE_G021_TAG else "")
|
||||
print("PASS [3/4] release-by-tag {}: exists (assets: {}){}".format(
|
||||
tag, listing, note))
|
||||
|
||||
# [4/4] Docker Hub index resolve (anonymous; amd64 drift print).
|
||||
try:
|
||||
hub_bearer = _hub_token(
|
||||
hub_auth, "repository:{}:pull".format(BASE_REPO))
|
||||
index = _fetch_manifest(hub_registry, BASE_REPO, BASE_TAG,
|
||||
hub_bearer)
|
||||
child_digest = _resolve_amd64_child(index)
|
||||
if not child_digest:
|
||||
raise RegistryApiError(
|
||||
"amd64 child descriptor carries no digest", 500, b"")
|
||||
except (TransientFailure, RegistryApiError) as error:
|
||||
print("FAILED [4/4] hub index resolve {}:{}: {}".format(
|
||||
BASE_REPO, BASE_TAG, error))
|
||||
return 1
|
||||
print("PASS [4/4] hub index resolve {}:{} -> linux/amd64 child {}"
|
||||
.format(BASE_REPO, BASE_TAG, child_digest))
|
||||
return 0
|
||||
|
||||
|
||||
def publish(tag, artifact_path, hub_registry, hub_auth, gitea_registry,
|
||||
owner, repo, base_manifest, timeout, env_root):
|
||||
"""Run the full publish flow; returns the image provenance dict."""
|
||||
artifact_bytes = _load_artifact(artifact_path)
|
||||
gitea_token = _env_secrets.get_token(root=env_root)
|
||||
name = "{}/{}".format(owner, repo)
|
||||
metadata_timeout = min(METADATA_TIMEOUT_SECONDS, timeout)
|
||||
|
||||
hub_bearer = _hub_token(hub_auth, "repository:{}:pull".format(BASE_REPO))
|
||||
if base_manifest:
|
||||
child_digest = base_manifest
|
||||
child = _fetch_manifest(hub_registry, BASE_REPO, base_manifest,
|
||||
hub_bearer)
|
||||
index_digest = None
|
||||
else:
|
||||
base = _fetch_manifest(hub_registry, BASE_REPO, BASE_TAG, hub_bearer)
|
||||
index_digest = "sha256:" + hashlib.sha256(
|
||||
json.dumps(base, sort_keys=True,
|
||||
separators=(",", ":")).encode("utf-8")).hexdigest()
|
||||
child_digest = (_resolve_amd64_child(base)
|
||||
if "manifests" in base else BASE_TAG)
|
||||
child = _fetch_manifest(hub_registry, BASE_REPO, child_digest,
|
||||
hub_bearer)
|
||||
child_config, base_layer_descriptors = _child_parts(child)
|
||||
base_config_bytes = _fetch_blob(
|
||||
hub_registry, BASE_REPO, child_config["digest"], hub_bearer,
|
||||
timeout)
|
||||
base_config = _parse_json(base_config_bytes, "base config blob", 200)
|
||||
if not isinstance(base_config, dict):
|
||||
raise RegistryApiError("base config blob is not a JSON object",
|
||||
500, base_config_bytes)
|
||||
|
||||
jwt = _gitea_jwt("{}/v2/token".format(gitea_registry.rstrip("/")),
|
||||
owner, repo, gitea_token, metadata_timeout)
|
||||
|
||||
pushed = []
|
||||
pushed.append(_push_blob_if_missing(
|
||||
gitea_registry, name, child_config["digest"], base_config_bytes,
|
||||
jwt, timeout))
|
||||
for descriptor in base_layer_descriptors:
|
||||
if _blob_exists(gitea_registry, name, descriptor["digest"], jwt,
|
||||
timeout):
|
||||
continue
|
||||
blob = _fetch_blob(hub_registry, BASE_REPO, descriptor["digest"],
|
||||
hub_bearer, timeout)
|
||||
_upload_blob(gitea_registry, name, descriptor["digest"], blob, jwt,
|
||||
timeout)
|
||||
pushed.append(True)
|
||||
|
||||
layer_blob, layer_digest, diff_id = build_broker_layer(artifact_bytes)
|
||||
config_blob = build_config_blob(base_config, diff_id, tag)
|
||||
config_digest = "sha256:" + hashlib.sha256(config_blob).hexdigest()
|
||||
_push_blob_if_missing(gitea_registry, name, layer_digest, layer_blob,
|
||||
jwt, timeout)
|
||||
_push_blob_if_missing(gitea_registry, name, config_digest, config_blob,
|
||||
jwt, timeout)
|
||||
|
||||
manifest_blob = build_manifest_blob(
|
||||
base_layer_descriptors,
|
||||
_descriptor(layer_digest, len(layer_blob), OCI_LAYER),
|
||||
_descriptor(config_digest, len(config_blob), OCI_CONFIG))
|
||||
manifest_digest = _put_manifest(
|
||||
gitea_registry, name, tag, manifest_blob, jwt, timeout)
|
||||
_put_manifest(gitea_registry, name, "latest", manifest_blob, jwt,
|
||||
timeout)
|
||||
|
||||
return {
|
||||
"base_index_digest": index_digest,
|
||||
"base_child_digest": child_digest if child_digest != BASE_TAG
|
||||
else "tag-resolved",
|
||||
"base_config_digest": child_config["digest"],
|
||||
"base_layers": [descriptor["digest"]
|
||||
for descriptor in base_layer_descriptors],
|
||||
"broker_layer_digest": layer_digest,
|
||||
"broker_layer_diff_id": diff_id,
|
||||
"config_digest": config_digest,
|
||||
"manifest_digest": manifest_digest,
|
||||
"blobs_uploaded": sum(1 for flag in pushed if flag),
|
||||
}
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Daemonless OCI image publish (REQ-23, D-019)."
|
||||
)
|
||||
parser.add_argument("tag", help="image tag, e.g. v0.2.3")
|
||||
parser.add_argument("--artifact",
|
||||
help="path to the built .pyz artifact "
|
||||
"(default: dist/coreci-broker-v<tag>.pyz)")
|
||||
parser.add_argument("--hub-registry",
|
||||
default="https://registry-1.docker.io",
|
||||
help="public base-image registry")
|
||||
parser.add_argument("--hub-auth", default="https://auth.docker.io/token",
|
||||
help="public registry token service")
|
||||
parser.add_argument("--gitea-registry", default="https://git.coreci.dev",
|
||||
help="gitea registry base URL")
|
||||
parser.add_argument("--owner", help="gitea owner (default: release.owner)")
|
||||
parser.add_argument("--repo", help="gitea repo (default: release.repo)")
|
||||
parser.add_argument("--base-manifest",
|
||||
help="base child-manifest digest for exact "
|
||||
"re-publishes (skips tag resolution)")
|
||||
parser.add_argument("--smoke", action="store_true",
|
||||
help="G-013 read-only live smoke: four GET-only "
|
||||
"pre-ship checks, NO publish (see the "
|
||||
"module docstring)")
|
||||
parser.add_argument("--timeout", type=float, default=BLOB_TIMEOUT_SECONDS,
|
||||
help="per-request timeout for blob operations "
|
||||
"(seconds; metadata uses the smaller of this "
|
||||
"and 30)")
|
||||
parser.add_argument("--env-root",
|
||||
help="root dir for the sanctioned .env* files "
|
||||
"(default: repo root)")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
defaults = load_release_defaults()
|
||||
owner = args.owner or defaults.get("owner")
|
||||
repo = args.repo or defaults.get("repo")
|
||||
if not owner or not repo:
|
||||
print("ERROR: missing gitea target (owner/repo); pass the flags or"
|
||||
" extend the .ciagent/config.json release block",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
blob_timeout = max(1.0, float(args.timeout))
|
||||
if args.smoke:
|
||||
# G-013 read-only live smoke: dispatched BEFORE any
|
||||
# publish-side work and returning directly — the default
|
||||
# publish flow below can never trigger it.
|
||||
try:
|
||||
return smoke(
|
||||
tag=args.tag, gitea_registry=args.gitea_registry,
|
||||
hub_registry=args.hub_registry, hub_auth=args.hub_auth,
|
||||
owner=owner, repo=repo, timeout=blob_timeout,
|
||||
env_root=args.env_root)
|
||||
except _env_secrets.SecretsError as error:
|
||||
print("ERROR: {}".format(error), file=sys.stderr)
|
||||
return 2
|
||||
artifact_path = args.artifact or os.path.join(
|
||||
"dist", "coreci-broker-{}.pyz".format(args.tag))
|
||||
# Reuse the smaller metadata timeout for the registry auth exchange so
|
||||
# the timeout-abort path stays bounded for every request class.
|
||||
try:
|
||||
provenance = publish(
|
||||
tag=args.tag, artifact_path=artifact_path,
|
||||
hub_registry=args.hub_registry, hub_auth=args.hub_auth,
|
||||
gitea_registry=args.gitea_registry, owner=owner, repo=repo,
|
||||
base_manifest=args.base_manifest, timeout=blob_timeout,
|
||||
env_root=args.env_root)
|
||||
except _env_secrets.SecretsError as error:
|
||||
print("ERROR: {}".format(error), file=sys.stderr)
|
||||
return 2
|
||||
except LocalError as error:
|
||||
print("ERROR: {}".format(error), file=sys.stderr)
|
||||
return 2
|
||||
except (TransientFailure, RegistryApiError) as error:
|
||||
print("ERROR: {}".format(error), file=sys.stderr)
|
||||
registry_host = urllib.parse.urlsplit(
|
||||
args.gitea_registry).netloc or args.gitea_registry
|
||||
print("IMAGE_PENDING {}/{}:{}".format(registry_host, owner, repo,
|
||||
args.tag))
|
||||
return 1
|
||||
|
||||
print("image: {}/{}:{} (+ latest)".format(
|
||||
args.gitea_registry.rstrip("/"), owner, repo, args.tag))
|
||||
print("base index: {}".format(
|
||||
provenance["base_index_digest"] or "direct (--base-manifest)"))
|
||||
print("base amd64 manifest: {}".format(provenance["base_child_digest"]))
|
||||
print("base config: {}".format(provenance["base_config_digest"]))
|
||||
print("base layers: {}".format(", ".join(provenance["base_layers"])))
|
||||
print("broker layer: {} (diff_id {})".format(
|
||||
provenance["broker_layer_digest"],
|
||||
provenance["broker_layer_diff_id"]))
|
||||
print("config: {}".format(provenance["config_digest"]))
|
||||
print("manifest: {}".format(provenance["manifest_digest"]))
|
||||
print("blobs uploaded this run: {}".format(provenance["blobs_uploaded"]))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,380 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Attach the built CLI artifact to a gitea release (stdlib only).
|
||||
|
||||
Usage:
|
||||
python3 scripts/release.py <tag> <artifact>
|
||||
[--base-url URL] [--owner O] [--repo R] [--env-root DIR]
|
||||
|
||||
Flow (REQ-22, RESEARCH §2): resolve the forge token from the sanctioned
|
||||
env files via ``scripts/_env_secrets.py`` — never the process environment,
|
||||
asserted non-empty BEFORE the first API call (v1.8 lesson: an empty token
|
||||
makes gitea answer 404, masking the auth failure) -> compute the artifact
|
||||
sha256 from the FILE itself (G-017: there is no sha argument, ever) ->
|
||||
build the release notes (Summary / Requirements / Verification / CLI
|
||||
install + usage with the digest inline / Docker usage) -> create-or-get
|
||||
the release by tag -> idempotent attach (same name AND size -> skip) ->
|
||||
multipart attach (field ``attachment``, ``?name=``, ``Authorization:
|
||||
token <T>``) under bounded retry (3 attempts, 2s/4s backoff on connection
|
||||
errors and 5xx).
|
||||
|
||||
Exit codes: 0 only on a confirmed or idempotent attach; 1 after a remote
|
||||
failure — bounded retries exhausted OR a permanent 4xx — both print the
|
||||
machine-readable ``RELEASE_PENDING <tag> <artifact>`` marker on stdout for
|
||||
the ship harness's release_pending escalation (tag+merge is never
|
||||
blocked); 2 for local precondition errors (missing artifact, token, or
|
||||
config — nothing was attempted remotely, so no marker). Gitea JSON error
|
||||
bodies are included in failure diagnostics. Token values are never
|
||||
printed.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import http.client
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
_SCRIPTS_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
if _SCRIPTS_DIR not in sys.path:
|
||||
sys.path.insert(0, _SCRIPTS_DIR)
|
||||
|
||||
import _env_secrets # sibling import (scripts/ dir on sys.path)
|
||||
|
||||
TIMEOUT_SECONDS = 30
|
||||
MAX_ATTEMPTS = 3
|
||||
BACKOFF_SECONDS = (2, 4)
|
||||
ATTACHMENT_FIELD = "attachment"
|
||||
# Fixed, distinctive boundary: a hand-rolled multipart body needs our own
|
||||
# delimiter; collision with artifact bytes is negligible at this length.
|
||||
MULTIPART_BOUNDARY = "coreci-broker-release-boundary-9a7f3c2e6b41"
|
||||
|
||||
|
||||
class ReleaseError(Exception):
|
||||
"""Release-attacher failure base."""
|
||||
|
||||
|
||||
class TransientFailure(ReleaseError):
|
||||
"""Connection error or 5xx persisting past the bounded retries."""
|
||||
|
||||
|
||||
class GiteaApiError(ReleaseError):
|
||||
"""Permanent API answer (4xx or malformed) carrying gitea's detail."""
|
||||
|
||||
def __init__(self, context, status, body):
|
||||
self.status = status
|
||||
self.body = body
|
||||
super().__init__("{} -> HTTP {}: {}".format(
|
||||
context, status, _error_detail(body)))
|
||||
|
||||
|
||||
def repo_root():
|
||||
return os.path.dirname(_SCRIPTS_DIR)
|
||||
|
||||
|
||||
def load_release_defaults(root=None):
|
||||
"""Read the release block from .ciagent/config.json ({} if absent)."""
|
||||
path = os.path.join(root or repo_root(), ".ciagent", "config.json")
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as handle:
|
||||
config = json.load(handle)
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
block = config.get("release")
|
||||
return block if isinstance(block, dict) else {}
|
||||
|
||||
|
||||
def _error_detail(body):
|
||||
"""Summarize a gitea JSON error body ({"message": ...} shape)."""
|
||||
try:
|
||||
payload = json.loads(body.decode("utf-8", "replace"))
|
||||
except ValueError:
|
||||
text = body.decode("utf-8", "replace").strip()
|
||||
return text[:200] if text else "<empty body>"
|
||||
if isinstance(payload, dict) and payload.get("message"):
|
||||
return str(payload["message"])
|
||||
return json.dumps(payload)[:200] if not isinstance(payload, str) else payload[:200]
|
||||
|
||||
|
||||
def _api_base(base_url, owner, repo):
|
||||
return "{}/api/v1/repos/{}/{}".format(base_url.rstrip("/"), owner, repo)
|
||||
|
||||
|
||||
def _image_reference(base_url, owner, repo):
|
||||
host = urllib.parse.urlsplit(base_url).netloc or base_url
|
||||
return "{}/{}/{}".format(host.rstrip("/"), owner, repo)
|
||||
|
||||
|
||||
def build_notes(tag, artifact_name, sha256, image_reference):
|
||||
"""Release-notes body (RESEARCH §2.3 model; REQ-22 CLI + docker docs)."""
|
||||
lines = [
|
||||
"## Summary",
|
||||
"",
|
||||
"coreci-broker {tag}: the broker packaged as a self-contained CLI"
|
||||
" zipapp artifact. {artifact} is built by scripts/build_cli.py with"
|
||||
" a deterministic normalization pass - the same sources rebuild"
|
||||
" byte-identical on the same environment, so the digest below is"
|
||||
" meaningful across rebuilds.".format(tag=tag, artifact=artifact_name),
|
||||
"",
|
||||
"## Requirements",
|
||||
"",
|
||||
"REQ-22: from the first CLI-building release onward, every gitea"
|
||||
" release attaches the built coreci-broker artifact alongside the"
|
||||
" existing source archive. The upload resolves the forge token only"
|
||||
" via the sanctioned .env* files (never the process environment),"
|
||||
" asserted non-empty before the first API call.",
|
||||
"",
|
||||
"## Verification",
|
||||
"",
|
||||
"The artifact sha256 in the CLI section below is computed from the"
|
||||
" attached file itself (G-017 - never an argument, never"
|
||||
" hand-pasted). Re-run the cumulative suite on the tagged commit:",
|
||||
"",
|
||||
" python3 -m unittest discover -s tests",
|
||||
"",
|
||||
"## CLI install + usage",
|
||||
"",
|
||||
"1. Download {artifact} from this release's assets.".format(
|
||||
artifact=artifact_name),
|
||||
"2. Install: chmod +x {artifact} then ./{artifact}, or run it"
|
||||
" directly with python3 {artifact} (Python 3.11+).".format(
|
||||
artifact=artifact_name),
|
||||
"3. Check the build: ./{artifact} --version prints {tag}.".format(
|
||||
artifact=artifact_name, tag=tag),
|
||||
"4. The env contract (every BROKER_* variable) is documented in the"
|
||||
" README; state is anchored by BROKER_ENV_PATH - the .env file's"
|
||||
" directory is the state dir and relative paths stay CWD-relative.",
|
||||
"5. First boot prints BROKER_ADMIN_PASS=... exactly once and creates"
|
||||
" the .env, broker.db, and broker.audit.log state files with mode"
|
||||
" 0600; the printed admin pass is not shown again - store it.",
|
||||
"",
|
||||
"Artifact sha256 (computed from the attached file):",
|
||||
"",
|
||||
" sha256:{sha} {artifact}".format(sha=sha256, artifact=artifact_name),
|
||||
"",
|
||||
"## Docker usage",
|
||||
"",
|
||||
"docker pull {image}:{tag}".format(image=image_reference, tag=tag),
|
||||
"",
|
||||
"docker run -p 8765:8765 -v broker-state:/state {image}:{tag}".format(
|
||||
image=image_reference, tag=tag),
|
||||
"",
|
||||
"State lives on the /state volume (WORKDIR /state; the first boot"
|
||||
" generates the .env, broker.db, and broker.audit.log there) -"
|
||||
" mount a persistent volume or the admin pass and any registered"
|
||||
" apps are lost with the container.",
|
||||
"",
|
||||
]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _request_once(method, url, token, data=None, headers=None):
|
||||
"""One HTTP exchange; returns (status, body). HTTPError becomes a
|
||||
completed answer (status + gitea error body); connection-level errors
|
||||
propagate for the caller's retry policy."""
|
||||
request = urllib.request.Request(url, data=data, method=method)
|
||||
request.add_header("Authorization", "token {}".format(token))
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=TIMEOUT_SECONDS) as response:
|
||||
return response.status, response.read()
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read()
|
||||
|
||||
|
||||
def _request(method, url, token, data=None, headers=None,
|
||||
max_attempts=None, backoff=None):
|
||||
"""Bounded-retry HTTP: retries connection errors and 5xx answers.
|
||||
|
||||
Returns (status, body) for any answer below 500. A 4xx returns
|
||||
immediately (permanent — surfaced with the gitea error body). Raises
|
||||
TransientFailure once the retry budget is exhausted.
|
||||
"""
|
||||
attempts = MAX_ATTEMPTS if max_attempts is None else int(max_attempts)
|
||||
delays = BACKOFF_SECONDS if backoff is None else tuple(backoff)
|
||||
failure = "unknown failure"
|
||||
for attempt in range(1, attempts + 1):
|
||||
try:
|
||||
status, body = _request_once(method, url, token, data, headers)
|
||||
except (OSError, http.client.HTTPException) as error:
|
||||
status = None
|
||||
failure = "connection error: {}: {}".format(
|
||||
type(error).__name__, error)
|
||||
else:
|
||||
if status < 500:
|
||||
return status, body
|
||||
failure = "HTTP {}: {}".format(status, _error_detail(body))
|
||||
if attempt < attempts and delays:
|
||||
time.sleep(delays[min(attempt - 1, len(delays) - 1)])
|
||||
raise TransientFailure(
|
||||
"{} {} failed after {} attempt(s): {}".format(
|
||||
method, url, attempts, failure))
|
||||
|
||||
|
||||
def _release_id_from(payload_body, context, status):
|
||||
try:
|
||||
return int(json.loads(payload_body)["id"])
|
||||
except (ValueError, KeyError, TypeError):
|
||||
raise GiteaApiError(context + " returned malformed JSON", status, payload_body)
|
||||
|
||||
|
||||
def _get_or_create_release(base, tag, token, notes):
|
||||
by_tag = base + "/releases/tags/" + urllib.parse.quote(tag, safe="")
|
||||
status, body = _request("GET", by_tag, token)
|
||||
if status == 200:
|
||||
return _release_id_from(body, "GET release by tag", status)
|
||||
if status != 404:
|
||||
# A 401/403 here is a genuine auth failure — the non-empty token
|
||||
# assert already ran, so this 404-vs-auth ambiguity is resolved.
|
||||
raise GiteaApiError("GET release by tag failed", status, body)
|
||||
payload = json.dumps({
|
||||
"tag_name": tag,
|
||||
"name": "{}: coreci-broker release".format(tag),
|
||||
"body": notes,
|
||||
"draft": False,
|
||||
"prerelease": False,
|
||||
}).encode("utf-8")
|
||||
status, body = _request(
|
||||
"POST", base + "/releases", token, data=payload,
|
||||
headers={"Content-Type": "application/json"})
|
||||
if status not in (200, 201):
|
||||
raise GiteaApiError("create release failed", status, body)
|
||||
return _release_id_from(body, "create release", status)
|
||||
|
||||
|
||||
def _assets_url(base, release_id):
|
||||
return "{}/releases/{}/assets".format(base, release_id)
|
||||
|
||||
|
||||
def _list_assets(base, release_id, token):
|
||||
status, body = _request("GET", _assets_url(base, release_id), token)
|
||||
if status != 200:
|
||||
raise GiteaApiError("list release assets failed", status, body)
|
||||
try:
|
||||
assets = json.loads(body)
|
||||
except ValueError:
|
||||
raise GiteaApiError(
|
||||
"list release assets returned malformed JSON", status, body)
|
||||
if not isinstance(assets, list):
|
||||
raise GiteaApiError(
|
||||
"list release assets returned non-list JSON", status, body)
|
||||
return assets
|
||||
|
||||
|
||||
def _asset_already_attached(assets, name, size):
|
||||
"""Idempotency: same name AND same size means the upload succeeded
|
||||
on an earlier (possibly retried) run."""
|
||||
for asset in assets:
|
||||
if not isinstance(asset, dict) or asset.get("name") != name:
|
||||
continue
|
||||
try:
|
||||
if int(asset.get("size")) == size:
|
||||
return True
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
return False
|
||||
|
||||
|
||||
def _multipart_body(field, filename, content):
|
||||
head = (
|
||||
"--{boundary}\r\n"
|
||||
"Content-Disposition: form-data; name=\"{field}\";"
|
||||
" filename=\"{filename}\"\r\n"
|
||||
"Content-Type: application/octet-stream\r\n"
|
||||
"\r\n"
|
||||
).format(boundary=MULTIPART_BOUNDARY, field=field,
|
||||
filename=filename).encode("utf-8")
|
||||
tail = "\r\n--{}--\r\n".format(MULTIPART_BOUNDARY).encode("utf-8")
|
||||
return head + content + tail
|
||||
|
||||
|
||||
def _attach_asset(base, release_id, token, name, content):
|
||||
url = "{}?{}".format(
|
||||
_assets_url(base, release_id), urllib.parse.urlencode({"name": name}))
|
||||
status, body = _request(
|
||||
"POST", url, token, data=_multipart_body(ATTACHMENT_FIELD, name, content),
|
||||
headers={
|
||||
"Content-Type":
|
||||
"multipart/form-data; boundary={}".format(MULTIPART_BOUNDARY),
|
||||
})
|
||||
if status not in (200, 201):
|
||||
raise GiteaApiError("attach asset failed", status, body)
|
||||
try:
|
||||
return json.loads(body)
|
||||
except ValueError:
|
||||
return {}
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Attach the built CLI artifact to a gitea release (REQ-22)."
|
||||
)
|
||||
parser.add_argument("tag", help="release tag, e.g. v0.2.1")
|
||||
parser.add_argument("artifact", help="path to the built .pyz artifact")
|
||||
parser.add_argument("--base-url",
|
||||
help="gitea base URL (default: .ciagent/config.json release.base_url)")
|
||||
parser.add_argument("--owner",
|
||||
help="gitea owner (default: release.owner)")
|
||||
parser.add_argument("--repo",
|
||||
help="gitea repo (default: release.repo)")
|
||||
parser.add_argument("--env-root",
|
||||
help="root dir for the sanctioned .env* files (default: repo root)")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
defaults = load_release_defaults()
|
||||
base_url = args.base_url or defaults.get("base_url")
|
||||
owner = args.owner or defaults.get("owner")
|
||||
repo = args.repo or defaults.get("repo")
|
||||
missing = [flag for flag, value in (
|
||||
("--base-url", base_url), ("--owner", owner), ("--repo", repo))
|
||||
if not value]
|
||||
if missing:
|
||||
print("ERROR: missing gitea target {}; pass the flag(s) or extend"
|
||||
" the .ciagent/config.json release block".format(
|
||||
", ".join(missing)), file=sys.stderr)
|
||||
return 2
|
||||
if not os.path.isfile(args.artifact):
|
||||
print("ERROR: artifact not found: {}".format(args.artifact), file=sys.stderr)
|
||||
return 2
|
||||
# Token FIRST, from the sanctioned env files only, non-empty enforced —
|
||||
# strictly before any HTTP call (v1.8: an empty token makes gitea answer
|
||||
# 404 and masks the auth failure).
|
||||
try:
|
||||
token = _env_secrets.get_token(root=args.env_root)
|
||||
except _env_secrets.SecretsError as error:
|
||||
print("ERROR: {}".format(error), file=sys.stderr)
|
||||
return 2
|
||||
|
||||
with open(args.artifact, "rb") as handle:
|
||||
content = handle.read()
|
||||
# G-017: the digest is computed from the artifact file itself — there is
|
||||
# no sha argument, ever.
|
||||
sha256 = hashlib.sha256(content).hexdigest()
|
||||
name = os.path.basename(args.artifact)
|
||||
size = len(content)
|
||||
notes = build_notes(args.tag, name, sha256,
|
||||
_image_reference(base_url, owner, repo))
|
||||
base = _api_base(base_url, owner, repo)
|
||||
try:
|
||||
release_id = _get_or_create_release(base, args.tag, token, notes)
|
||||
if _asset_already_attached(_list_assets(base, release_id, token), name, size):
|
||||
print("attach skipped (idempotent): {} already on {}/{} release"
|
||||
" {} (sha256 {})".format(name, owner, repo, args.tag, sha256))
|
||||
return 0
|
||||
attachment = _attach_asset(base, release_id, token, name, content)
|
||||
asset_id = attachment.get("id", "?") if isinstance(attachment, dict) else "?"
|
||||
print("attached: {} -> {}/{} release {} (asset id {}, sha256 {})".format(
|
||||
name, owner, repo, args.tag, asset_id, sha256))
|
||||
return 0
|
||||
except (TransientFailure, GiteaApiError) as error:
|
||||
print("ERROR: {}".format(error), file=sys.stderr)
|
||||
print("RELEASE_PENDING {} {}".format(args.tag, args.artifact))
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,345 @@
|
||||
import ast
|
||||
import base64
|
||||
import contextlib
|
||||
import datetime
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
|
||||
BROKER_SOURCE_PATH = os.path.join(REPO_ROOT, "coreci_broker.py")
|
||||
CLIENT_SOURCE_PATH = os.path.join(REPO_ROOT, "coreci_broker_client.py")
|
||||
CLIENT_STATIC_PATH = "/static/coreci_broker_client.py"
|
||||
LOGIN_PATH = "/api/broker/auth/login"
|
||||
LOGOUT_PATH = "/api/broker/auth/logout"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
AUDIT_PATH = "/api/broker/audit"
|
||||
SECRETS_PREFIX = "/api/broker/secrets/"
|
||||
PROXMOX_PATH = "coreci/proxmox"
|
||||
GITEA_PATH = "coreci/gitea"
|
||||
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
|
||||
GITEA_SECRET = {"url": "https://git.coreci.dev", "token": "git-tok"}
|
||||
|
||||
AC_PIN_IDS = {
|
||||
"AC1": "test_acceptance_p2.AcceptanceP2Tests.test_ac1_first_boot_printed_pass_login_via_ui_login_page",
|
||||
"AC2": "test_acceptance_p2.AcceptanceP2Tests.test_ac2_app_registration_lifecycle_via_session_cookie",
|
||||
"AC3": "test_acceptance_p2.AcceptanceP2Tests.test_ac3_user_allowlist_lifecycle_via_session_cookie",
|
||||
"AC4": "test_acceptance_p3.AcceptanceP3Tests.test_ac4_derivation_identical_both_sides_stored_as_slow_hash",
|
||||
"AC5": "test_acceptance_p4.AcceptanceP4Tests.test_ac5_valid_token_reads_secret_tampered_token_is_401",
|
||||
"AC6": "test_acceptance_p4.AcceptanceP4Tests.test_ac6_cross_path_read_is_403_with_denied_audit",
|
||||
"AC7": "test_acceptance_p3.AcceptanceP3Tests.test_ac7_revoked_or_unknown_uuid_denied_existing_token_honored",
|
||||
"AC8": "test_acceptance_p3.AcceptanceP3Tests.test_ac8_unknown_or_revoked_user_denied",
|
||||
"AC9": "test_acceptance_p2.AcceptanceP2Tests.test_ac9_login_and_logout_audit_lines_with_null_identity",
|
||||
"AC10": "test_acceptance_p3.AcceptanceP3Tests.test_ac10_expiry_401_reregister_refreshes_expiry_and_audits",
|
||||
"AC11": "test_acceptance_full.AcceptanceFullTests.test_ac11_fresh_boot_state_files_0600_umask_independent",
|
||||
"AC12": "test_acceptance_p3.AcceptanceP3Tests.test_ac12_state_roundtrip_with_identity_enforcement",
|
||||
"AC13": "test_acceptance_full.AcceptanceFullTests.test_ac13_client_module_fetch_byte_identical_importable_standalone",
|
||||
"AC14": "test_acceptance_full.AcceptanceFullTests.test_ac14_coreci_broker_line_count_under_1400",
|
||||
}
|
||||
|
||||
|
||||
def _basic_header(password):
|
||||
raw = "%s:%s" % (ADMIN_USER, password)
|
||||
return {"Authorization": "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")}
|
||||
|
||||
|
||||
class AcceptanceFullTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.creds_path = os.path.join(self.tmp, "creds.json")
|
||||
with open(self.creds_path, "w", encoding="utf-8") as handle:
|
||||
handle.write(json.dumps({PROXMOX_PATH: PROXMOX_SECRET, GITEA_PATH: GITEA_SECRET}))
|
||||
|
||||
def _env(self, root=None, extra=None):
|
||||
root = self.tmp if root is None else root
|
||||
env = {
|
||||
"BROKER_ENV_PATH": os.path.join(root, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(root, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(root, "broker.audit.log"),
|
||||
"BROKER_CREDS_SOURCE": "file",
|
||||
"BROKER_CREDS_FILE": self.creds_path,
|
||||
"BROKER_UI_ROOT": os.path.join(REPO_ROOT, "coreci_broker_ui"),
|
||||
"BROKER_STATIC_ROOT": REPO_ROOT,
|
||||
}
|
||||
env.update(extra or {})
|
||||
return env
|
||||
|
||||
def _boot(self, env):
|
||||
buffer = io.StringIO()
|
||||
with contextlib.redirect_stdout(buffer):
|
||||
broker = coreci_broker.boot(dict(env))
|
||||
self.addCleanup(broker.audit.close)
|
||||
self.addCleanup(broker.store.close)
|
||||
return broker, buffer.getvalue()
|
||||
|
||||
def _serve(self, broker):
|
||||
server = coreci_broker.serve("127.0.0.1", 0, broker)
|
||||
self.addCleanup(server.server_close)
|
||||
self.addCleanup(server.shutdown)
|
||||
return "http://127.0.0.1:%d" % server.server_address[1]
|
||||
|
||||
def _printed_pass(self, output):
|
||||
printed = [
|
||||
line.split("=", 1)[1]
|
||||
for line in output.splitlines()
|
||||
if line.startswith("BROKER_ADMIN_PASS=")
|
||||
]
|
||||
self.assertEqual(1, len(printed))
|
||||
return printed[0]
|
||||
|
||||
def _request(self, base, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
return response.status, response.read(), response.headers
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read(), error.headers
|
||||
|
||||
def _request_json(self, base, method, path, payload=None, headers=None):
|
||||
status, body, response_headers = self._request(base, method, path, payload, headers)
|
||||
return status, json.loads(body.decode("utf-8")), response_headers
|
||||
|
||||
def _assert_null_identity(self, entry):
|
||||
self.assertIsNone(entry["user_id"])
|
||||
self.assertIsNone(entry["app_registration_uuid"])
|
||||
self.assertIsNone(entry["agent_id"])
|
||||
|
||||
def _assert_tool_identity(self, entry, app_uuid):
|
||||
self.assertEqual("ceo", entry["user_id"])
|
||||
self.assertEqual(app_uuid, entry["app_registration_uuid"])
|
||||
self.assertEqual("proxmox", entry["agent_id"])
|
||||
|
||||
def _assert_state_files_0600(self, env):
|
||||
for key in ("BROKER_DB_PATH", "BROKER_AUDIT_LOG_PATH", "BROKER_ENV_PATH"):
|
||||
path = env[key]
|
||||
self.assertTrue(os.path.isfile(path), path)
|
||||
self.assertEqual(0o600, os.stat(path).st_mode & 0o777, path)
|
||||
|
||||
def test_ac11_fresh_boot_state_files_0600_umask_independent(self):
|
||||
env = self._env()
|
||||
self._boot(env)
|
||||
self._assert_state_files_0600(env)
|
||||
other = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(other.cleanup)
|
||||
other_env = self._env(root=other.name)
|
||||
previous_umask = os.umask(0o777)
|
||||
try:
|
||||
probe = os.path.join(other.name, "umask-probe")
|
||||
os.close(os.open(probe, os.O_CREAT | os.O_WRONLY, 0o666))
|
||||
self.assertEqual(0o000, os.stat(probe).st_mode & 0o777)
|
||||
self._boot(other_env)
|
||||
finally:
|
||||
os.umask(previous_umask)
|
||||
self._assert_state_files_0600(other_env)
|
||||
|
||||
def test_ac13_client_module_fetch_byte_identical_importable_standalone(self):
|
||||
broker, _ = self._boot(self._env())
|
||||
base = self._serve(broker)
|
||||
with open(CLIENT_SOURCE_PATH, "rb") as handle:
|
||||
expected = handle.read()
|
||||
with urllib.request.urlopen(base + CLIENT_STATIC_PATH, timeout=10) as response:
|
||||
self.assertEqual(200, response.status)
|
||||
fetched = response.read()
|
||||
self.assertEqual(expected, fetched)
|
||||
imports = set()
|
||||
for node in ast.walk(ast.parse(fetched.decode("utf-8"))):
|
||||
if isinstance(node, ast.Import):
|
||||
imports.update(alias.name for alias in node.names)
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
imports.add(node.module or "")
|
||||
self.assertTrue(imports)
|
||||
for name in imports:
|
||||
self.assertIn(name.split(".")[0], sys.stdlib_module_names, name)
|
||||
copy_path = os.path.join(self.tmp, "fetched_coreci_broker_client.py")
|
||||
with open(copy_path, "wb") as handle:
|
||||
handle.write(fetched)
|
||||
spec = importlib.util.spec_from_file_location("full_fetched_client", copy_path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
self.assertTrue(callable(module.BrokerClient))
|
||||
self.assertTrue(callable(module.fetch_client_module))
|
||||
for method in ("ensure_registered", "get_state", "put_state", "read_secret"):
|
||||
self.assertTrue(callable(getattr(module.BrokerClient, method)), method)
|
||||
|
||||
def test_ac14_coreci_broker_line_count_under_1400(self):
|
||||
with open(BROKER_SOURCE_PATH, "r", encoding="utf-8") as handle:
|
||||
line_count = sum(1 for _ in handle)
|
||||
self.assertLess(line_count, 1400)
|
||||
|
||||
def test_milestone_story_full_happy_path_consolidates_ac1_ac10_ac12(self):
|
||||
env = self._env(extra={"BROKER_TOKEN_TTL": "1"})
|
||||
first, output = self._boot(env)
|
||||
printed_pass = self._printed_pass(output)
|
||||
self.assertGreaterEqual(len(printed_pass), 32)
|
||||
self.assertEqual(1, output.count(printed_pass))
|
||||
pepper = None
|
||||
with open(env["BROKER_ENV_PATH"], "r", encoding="utf-8") as handle:
|
||||
for line in handle:
|
||||
if line.startswith("BROKER_INSTANCE_PEPPER="):
|
||||
pepper = line.split("=", 1)[1].strip()
|
||||
self.assertIsNotNone(pepper)
|
||||
self.assertNotIn(pepper, output)
|
||||
second, silent = self._boot(env)
|
||||
self.assertEqual("", silent)
|
||||
second.audit.close()
|
||||
second.store.close()
|
||||
base = self._serve(first)
|
||||
status, body, headers = self._request(base, "GET", "/ui/login")
|
||||
self.assertEqual(200, status)
|
||||
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
|
||||
self.assertIn('name="password"', body.decode("utf-8"))
|
||||
status, body, response_headers = self._request_json(
|
||||
base, "POST", LOGIN_PATH, {"username": ADMIN_USER, "password": printed_pass}
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
|
||||
raw_cookie = response_headers.get("Set-Cookie")
|
||||
self.assertTrue(raw_cookie)
|
||||
self.assertTrue(raw_cookie.startswith("broker_session="))
|
||||
self.assertIn("HttpOnly", raw_cookie)
|
||||
self.assertIn("SameSite=Lax", raw_cookie)
|
||||
cookie = {"Cookie": raw_cookie.split(";", 1)[0]}
|
||||
app_uuid = str(uuidlib.uuid4())
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", APPS_PATH,
|
||||
{"uuid": app_uuid, "name": "proxmox-tool"}, headers=cookie,
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", USERS_PATH,
|
||||
{"user_id": "ceo", "email": "ceo@coreci.dev"}, headers=cookie,
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
self.assertEqual({"user_id": "ceo", "status": "active"}, body)
|
||||
with urllib.request.urlopen(base + CLIENT_STATIC_PATH, timeout=10) as response:
|
||||
self.assertEqual(200, response.status)
|
||||
client_source = response.read()
|
||||
with open(CLIENT_SOURCE_PATH, "rb") as handle:
|
||||
self.assertEqual(handle.read(), client_source)
|
||||
copy_path = os.path.join(self.tmp, "story_coreci_broker_client.py")
|
||||
with open(copy_path, "wb") as handle:
|
||||
handle.write(client_source)
|
||||
spec = importlib.util.spec_from_file_location("full_story_client", copy_path)
|
||||
client_module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(client_module)
|
||||
client = client_module.BrokerClient(base, "proxmox", app_uuid, "ceo", ["coreci"])
|
||||
state = {}
|
||||
client.ensure_registered(state)
|
||||
self.assertEqual(44, len(state["current_token"]))
|
||||
self.assertTrue(state["current_token"].isascii())
|
||||
self.assertIn("pepper", state)
|
||||
self.assertEqual("proxmox", state["agent_id"])
|
||||
self.assertEqual(app_uuid, state["app_registration_uuid"])
|
||||
self.assertEqual("ceo", state["user_id"])
|
||||
self.assertTrue(client.put_state(state))
|
||||
snapshot = dict(state)
|
||||
self.assertEqual(snapshot, client.get_state(state))
|
||||
document = client.read_secret(PROXMOX_PATH, state)
|
||||
self.assertEqual({"path", "data", "version", "updated_at"}, set(document))
|
||||
self.assertEqual(PROXMOX_PATH, document["path"])
|
||||
self.assertEqual(PROXMOX_SECRET, document["data"])
|
||||
with self.assertRaises(client_module.BrokerError) as caught:
|
||||
client.read_secret(GITEA_PATH, state)
|
||||
self.assertEqual(403, caught.exception.status)
|
||||
expired_at = state["token_expires_at"]
|
||||
time.sleep(1.2)
|
||||
document = client.read_secret(PROXMOX_PATH, state)
|
||||
self.assertEqual(PROXMOX_SECRET, document["data"])
|
||||
self.assertGreater(
|
||||
datetime.datetime.fromisoformat(state["token_expires_at"]),
|
||||
datetime.datetime.fromisoformat(expired_at),
|
||||
)
|
||||
status, body, _ = self._request_json(base, "POST", LOGOUT_PATH, headers=cookie)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
status, body, _ = self._request_json(
|
||||
base, "GET", AUDIT_PATH + "?limit=1000", headers=_basic_header(printed_pass)
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertIsNone(body["next_since"])
|
||||
entries = body["entries"]
|
||||
|
||||
def select(action, result=None):
|
||||
return [
|
||||
entry for entry in entries
|
||||
if entry["action"] == action and (result is None or entry["result"] == result)
|
||||
]
|
||||
|
||||
logins = select("login", "ok")
|
||||
self.assertEqual(1, len(logins))
|
||||
self.assertEqual(LOGIN_PATH, logins[0]["path"])
|
||||
self._assert_null_identity(logins[0])
|
||||
add_apps = select("add_app", "ok")
|
||||
self.assertEqual(1, len(add_apps))
|
||||
self.assertEqual(APPS_PATH + "/" + app_uuid, add_apps[0]["path"])
|
||||
self._assert_null_identity(add_apps[0])
|
||||
add_users = select("add_user", "ok")
|
||||
self.assertEqual(1, len(add_users))
|
||||
self.assertEqual(USERS_PATH + "/ceo", add_users[0]["path"])
|
||||
self._assert_null_identity(add_users[0])
|
||||
registers = select("register", "ok")
|
||||
self.assertGreaterEqual(len(registers), 2)
|
||||
for entry in registers:
|
||||
self.assertEqual(REGISTER_PATH, entry["path"])
|
||||
self._assert_tool_identity(entry, app_uuid)
|
||||
ok_reads = select("read", "ok")
|
||||
self.assertGreaterEqual(len(ok_reads), 2)
|
||||
for entry in ok_reads:
|
||||
self.assertEqual(PROXMOX_PATH, entry["path"])
|
||||
self._assert_tool_identity(entry, app_uuid)
|
||||
denied_reads = select("read", "denied")
|
||||
self.assertEqual(1, len(denied_reads))
|
||||
self.assertEqual(GITEA_PATH, denied_reads[0]["path"])
|
||||
self._assert_tool_identity(denied_reads[0], app_uuid)
|
||||
auth_denied = [
|
||||
entry for entry in select("denied", "denied")
|
||||
if entry["path"] == SECRETS_PREFIX + PROXMOX_PATH
|
||||
]
|
||||
self.assertGreaterEqual(len(auth_denied), 1)
|
||||
self.assertEqual(app_uuid, auth_denied[0]["app_registration_uuid"])
|
||||
self.assertIsNone(auth_denied[0]["user_id"])
|
||||
self.assertIsNone(auth_denied[0]["agent_id"])
|
||||
logouts = select("logout", "ok")
|
||||
self.assertEqual(1, len(logouts))
|
||||
self.assertEqual(LOGOUT_PATH, logouts[0]["path"])
|
||||
self._assert_null_identity(logouts[0])
|
||||
|
||||
def test_coverage_tally_pins_ac1_through_ac14_across_suite(self):
|
||||
self.assertEqual(14, len(AC_PIN_IDS))
|
||||
loader = unittest.TestLoader()
|
||||
suite = loader.discover(os.path.dirname(os.path.abspath(__file__)))
|
||||
stack, ids = [suite], set()
|
||||
while stack:
|
||||
item = stack.pop()
|
||||
if isinstance(item, unittest.TestSuite):
|
||||
stack.extend(item)
|
||||
else:
|
||||
ids.add(item.id())
|
||||
for ac in sorted(AC_PIN_IDS):
|
||||
with self.subTest(ac=ac):
|
||||
self.assertIn(AC_PIN_IDS[ac], ids)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,548 @@
|
||||
"""v0.2 acceptance matrix (PLAN P3-01; REQ-21/REQ-22/REQ-23 acceptance).
|
||||
|
||||
Consolidates the milestone's acceptance as thin, high-value assertions that
|
||||
the deliverable holds end-to-end. The deep per-feature logic stays in its own
|
||||
files — test_build_cli, test_cli_packaging, test_publish_image,
|
||||
test_release_upload, test_dockerfile_consistency — which run green in the
|
||||
same cumulative suite (``python3 -m unittest discover -s tests``; suite
|
||||
greenness is the runner's job, asserted by nobody in here). This file does
|
||||
NOT re-run their cases; the stub roundtrips are smoke-asserted instead via
|
||||
module presence + runnable TestSuite sizes (defaultTestLoader).
|
||||
|
||||
The matrix (binding order):
|
||||
1. boot parity through the artifact — the built ``.pyz`` is booted as a real
|
||||
subprocess child (``[sys.executable, artifact]``, never a bare
|
||||
"python3") with an ephemeral bind + temp state, and v0.1 acceptance
|
||||
parity must hold through the packaged path: admin pass printed exactly
|
||||
once, ``broker.db`` + ``broker.audit.log`` 0600 in the state dir,
|
||||
``/ui/login`` + ``/ui/dashboard`` 200, ``/static/coreci_broker_client.py``
|
||||
byte-exact vs the repo file, one admin-login + agent-register +
|
||||
state-put/get + secret-read flow (v0.1 request patterns), second boot
|
||||
silent (asserted only after terminate + full drain — G-010);
|
||||
2. reproducibility — two consecutive same-environment builds produce
|
||||
byte-identical artifacts (identical sha256, G-009);
|
||||
3. self-containment — the archive contains ONLY the staging allowlist
|
||||
(broker + client + UI pages + shim + ``_cli_version.py``): no ``.env``,
|
||||
no ``dist/``, no state, no tests;
|
||||
4. stub publish roundtrip + stub release upload green — their test modules
|
||||
import cleanly and expose runnable TestSuites of the expected size;
|
||||
5. broker zero-delta — ``coreci_broker.py`` unchanged at its v0.1.5 count
|
||||
of 1390 lines (AC14 under the D-016..D-018 cap of 1400) and carrying no
|
||||
v0.2 packaging references (``_cli_version`` / the shim / the shim's
|
||||
materialized-asset dir — frozen means untouched); plus the AC14
|
||||
no-touch clause machine-checked against the ``v0.1.5`` tag when the
|
||||
working tree carries it (skipped otherwise);
|
||||
6. Dockerfile/.dockerignore consistency — module present with its full
|
||||
TestCase set (it runs in the same suite).
|
||||
|
||||
The build happens ONCE per class (setUpClass) and every artifact/boot is
|
||||
reused across methods where safe; the boot tests dominate and the whole
|
||||
file is budgeted to stay well under 30s. Every child is terminated +
|
||||
drained exactly once (idempotent ``stop`` via ``addCleanup``); every temp
|
||||
dir is ``addCleanup``-registered. Builds use the ``scripts/build_cli.py``
|
||||
API (never a shell path) into ``TemporaryDirectory`` dist dirs — no
|
||||
committed ``dist/``, no state, no live-forge contact.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import select
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
import zipfile
|
||||
|
||||
REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
|
||||
TESTS_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
BUILD_CLI_PATH = os.path.join(REPO, "scripts", "build_cli.py")
|
||||
BROKER_SOURCE_PATH = os.path.join(REPO, "coreci_broker.py")
|
||||
REPO_CLIENT_PATH = os.path.join(REPO, "coreci_broker_client.py")
|
||||
VERSION = "v0.2.3" # the milestone release this acceptance gates (P3-05 ship)
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS_PREFIX = "BROKER_ADMIN_PASS="
|
||||
LISTENING_PREFIX = "coreci-broker listening on http://"
|
||||
STATE_FILE_NAMES = (".env", "broker.db", "broker.audit.log")
|
||||
|
||||
LOGIN_PATH = "/api/broker/auth/login"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
AGENT_STATE_PATH = "/api/broker/agents/proxmox/state"
|
||||
SECRETS_PATH = "/api/broker/secrets/coreci/proxmox"
|
||||
CLIENT_STATIC_PATH = "/static/coreci_broker_client.py"
|
||||
UI_LOGIN_PATH = "/ui/login"
|
||||
UI_DASHBOARD_PATH = "/ui/dashboard"
|
||||
|
||||
PROXMOX_PATH = "coreci/proxmox"
|
||||
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
|
||||
|
||||
BOOT_DEADLINE_SECONDS = 30.0
|
||||
POLL_SECONDS = 0.1
|
||||
DRAIN_TIMEOUT_SECONDS = 20.0
|
||||
HTTP_TIMEOUT_SECONDS = 10
|
||||
GIT_TIMEOUT_SECONDS = 30
|
||||
|
||||
# Frozen-broker sources (PLAN: no task edits these in v0.2) — used only by
|
||||
# the AC14 no-touch clause check, never modified.
|
||||
FROZEN_SOURCES = ("coreci_broker.py", "coreci_broker_client.py",
|
||||
"coreci_broker_ui", "reference_tools")
|
||||
# v0.2 packaging artifacts the frozen broker must know nothing about.
|
||||
FROZEN_MARKER_STRINGS = ("_cli_version", "cli/__main__", "coreci_broker_assets")
|
||||
|
||||
# The v0.2 test surface consolidated here — presence/count floors only
|
||||
# (the roundtrips themselves run green in their own files, same suite).
|
||||
STUB_MODULE_MIN_CASES = {
|
||||
"test_publish_image.py": 9, # stub publish roundtrip
|
||||
"test_release_upload.py": 15, # stub release upload + env-secrets
|
||||
"test_dockerfile_consistency.py": 7,
|
||||
}
|
||||
|
||||
|
||||
def _load_build_cli():
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"acceptance_build_cli", BUILD_CLI_PATH)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
build_cli = _load_build_cli()
|
||||
|
||||
|
||||
def _sha256(path):
|
||||
digest = hashlib.sha256()
|
||||
with open(path, "rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(65536), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
class _BrokerChild:
|
||||
"""A subprocess child booted from the built zipapp artifact.
|
||||
|
||||
The merged stdout/stderr pipe is pumped via ``select`` + raw ``os.read``
|
||||
under an explicit deadline until the broker announces its ephemeral
|
||||
listen address. ``stop()`` is the G-010 kill/drain protocol:
|
||||
``terminate()`` then a full ``communicate()`` drain (kill fallback on
|
||||
drain timeout). It is idempotent and registered with ``addCleanup`` at
|
||||
construction, so no child leaks even when an assertion trips mid-test.
|
||||
Silence and pass-once assertions are made ONLY on ``full_output()``
|
||||
after ``stop()``.
|
||||
"""
|
||||
|
||||
def __init__(self, testcase, artifact, env, cwd):
|
||||
self.testcase = testcase
|
||||
self.artifact = artifact
|
||||
self.proc = subprocess.Popen(
|
||||
[sys.executable, artifact],
|
||||
env=env, cwd=cwd,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
|
||||
)
|
||||
self.lines = []
|
||||
self.partial = b""
|
||||
self.drained = b""
|
||||
self.base_url = None
|
||||
self._eof = False
|
||||
self._stopped = False
|
||||
testcase.addCleanup(self.stop)
|
||||
self._await_listening(BOOT_DEADLINE_SECONDS)
|
||||
|
||||
def _pump(self, timeout):
|
||||
if self._eof:
|
||||
return False
|
||||
fd = self.proc.stdout.fileno()
|
||||
ready, _, _ = select.select([fd], [], [], timeout)
|
||||
if not ready:
|
||||
return True
|
||||
try:
|
||||
chunk = os.read(fd, 4096)
|
||||
except OSError:
|
||||
chunk = b""
|
||||
if not chunk:
|
||||
self._eof = True
|
||||
return False
|
||||
self.partial += chunk
|
||||
while b"\n" in self.partial:
|
||||
raw, self.partial = self.partial.split(b"\n", 1)
|
||||
self.lines.append(raw.decode("utf-8", "replace"))
|
||||
return True
|
||||
|
||||
def _await_listening(self, deadline_seconds):
|
||||
deadline = time.monotonic() + deadline_seconds
|
||||
cursor = 0
|
||||
while time.monotonic() < deadline:
|
||||
alive = self._pump(POLL_SECONDS)
|
||||
while cursor < len(self.lines):
|
||||
line = self.lines[cursor]
|
||||
cursor += 1
|
||||
if line.startswith(LISTENING_PREFIX):
|
||||
self.base_url = line.split(" on ", 1)[1].strip()
|
||||
return
|
||||
if not alive:
|
||||
break
|
||||
self.stop()
|
||||
self.testcase.fail(
|
||||
"packaged broker did not announce %r within %.0fs "
|
||||
"(artifact=%s exit=%s)\n--- captured child output ---\n%s"
|
||||
% (LISTENING_PREFIX, deadline_seconds, self.artifact,
|
||||
self.proc.returncode, self.full_output())
|
||||
)
|
||||
|
||||
def stop(self):
|
||||
if self._stopped:
|
||||
return
|
||||
self._stopped = True
|
||||
if self.proc.poll() is None:
|
||||
self.proc.terminate()
|
||||
try:
|
||||
remainder, _ = self.proc.communicate(timeout=DRAIN_TIMEOUT_SECONDS)
|
||||
except subprocess.TimeoutExpired:
|
||||
self.proc.kill()
|
||||
remainder, _ = self.proc.communicate()
|
||||
self.drained = remainder or b""
|
||||
|
||||
def full_output(self):
|
||||
text = "\n".join(self.lines)
|
||||
tail = self.partial + self.drained
|
||||
if tail:
|
||||
text += "\n" + tail.decode("utf-8", "replace")
|
||||
return text
|
||||
|
||||
def printed_admin_pass(self):
|
||||
values = [
|
||||
line.split("=", 1)[1]
|
||||
for line in self.lines
|
||||
if line.startswith(ADMIN_PASS_PREFIX)
|
||||
]
|
||||
self.testcase.assertEqual(
|
||||
1, len(values),
|
||||
"expected exactly one %s line, got: %r" % (ADMIN_PASS_PREFIX, values)
|
||||
)
|
||||
return values[0]
|
||||
|
||||
|
||||
class AcceptanceV02Tests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.build_root = tempfile.TemporaryDirectory(prefix="coreci-acc-v02-")
|
||||
cls.addClassCleanup(cls.build_root.cleanup)
|
||||
cls.artifact = build_cli.build(
|
||||
VERSION, repo=REPO, dist_dir=cls.build_root.name
|
||||
)
|
||||
cls.artifact_sha256 = _sha256(cls.artifact)
|
||||
|
||||
# -- harness helpers --------------------------------------------------
|
||||
|
||||
def _temp_dir(self):
|
||||
handle = tempfile.TemporaryDirectory(prefix="coreci-acc-v02-")
|
||||
self.addCleanup(handle.cleanup)
|
||||
return handle.name
|
||||
|
||||
def _child_env(self, **overrides):
|
||||
"""A child env with no inherited BROKER_* keys, then overrides."""
|
||||
env = {
|
||||
key: value
|
||||
for key, value in os.environ.items()
|
||||
if not key.startswith("BROKER_")
|
||||
}
|
||||
env.update(overrides)
|
||||
return env
|
||||
|
||||
def _state_env(self, state_dir):
|
||||
"""Ephemeral bind + temp state + a seeded file creds source."""
|
||||
creds_path = os.path.join(state_dir, "creds.json")
|
||||
with open(creds_path, "w", encoding="utf-8") as handle:
|
||||
json.dump({PROXMOX_PATH: PROXMOX_SECRET}, handle)
|
||||
return self._child_env(
|
||||
BROKER_BIND="127.0.0.1:0",
|
||||
BROKER_ENV_PATH=os.path.join(state_dir, ".env"),
|
||||
BROKER_DB_PATH=os.path.join(state_dir, "broker.db"),
|
||||
BROKER_AUDIT_LOG_PATH=os.path.join(state_dir, "broker.audit.log"),
|
||||
BROKER_CREDS_SOURCE="file",
|
||||
BROKER_CREDS_FILE=creds_path,
|
||||
)
|
||||
|
||||
def _boot(self, state_dir):
|
||||
"""Boot the artifact against state_dir; CWD is a separate temp dir."""
|
||||
return _BrokerChild(
|
||||
self, self.artifact, self._state_env(state_dir), self._temp_dir()
|
||||
)
|
||||
|
||||
def _request(self, base, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(
|
||||
request, timeout=HTTP_TIMEOUT_SECONDS
|
||||
) as response:
|
||||
return response.status, response.read(), response.headers
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read(), error.headers
|
||||
|
||||
def _request_json(self, base, method, path, payload=None, headers=None):
|
||||
status, body, response_headers = self._request(
|
||||
base, method, path, payload, headers
|
||||
)
|
||||
return status, json.loads(body.decode("utf-8")), response_headers
|
||||
|
||||
def _admin_login(self, base, password):
|
||||
"""v0.1 pattern: POST login, assert the broker_session cookie."""
|
||||
status, body, headers = self._request_json(
|
||||
base, "POST", LOGIN_PATH,
|
||||
{"username": ADMIN_USER, "password": password},
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
|
||||
raw_cookie = headers.get("Set-Cookie")
|
||||
self.assertTrue(raw_cookie)
|
||||
self.assertTrue(raw_cookie.startswith("broker_session="))
|
||||
self.assertIn("HttpOnly", raw_cookie)
|
||||
self.assertIn("SameSite=Lax", raw_cookie)
|
||||
return {"Cookie": raw_cookie.split(";", 1)[0]}
|
||||
|
||||
def _load_test_module(self, file_name):
|
||||
"""Import a sibling test module cleanly (by path, any invocation)."""
|
||||
path = os.path.join(TESTS_DIR, file_name)
|
||||
stem = os.path.splitext(file_name)[0]
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"acceptance_probe_" + stem, path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
def _runnable_case_count(self, module):
|
||||
"""Flatten defaultTestLoader's suite, counting runnable TestCases."""
|
||||
suite = unittest.defaultTestLoader.loadTestsFromModule(module)
|
||||
stack, count = [suite], 0
|
||||
while stack:
|
||||
item = stack.pop()
|
||||
if isinstance(item, unittest.TestSuite):
|
||||
stack.extend(item)
|
||||
else:
|
||||
self.assertIsInstance(item, unittest.TestCase)
|
||||
count += 1
|
||||
return count
|
||||
|
||||
# -- matrix 1: boot parity through the artifact ----------------------
|
||||
|
||||
def test_boot_parity_through_artifact_second_boot_silent(self):
|
||||
state_dir = self._temp_dir()
|
||||
child = self._boot(state_dir)
|
||||
base = child.base_url
|
||||
printed_pass = child.printed_admin_pass()
|
||||
|
||||
# liveness + full UI served from the packaged artifact
|
||||
status, body, _ = self._request_json(base, "GET", "/healthz")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
for ui_path in (UI_LOGIN_PATH, UI_DASHBOARD_PATH):
|
||||
status, body, headers = self._request(base, "GET", ui_path)
|
||||
self.assertEqual(200, status, ui_path)
|
||||
self.assertTrue(
|
||||
headers.get("Content-Type", "").startswith("text/html"),
|
||||
ui_path,
|
||||
)
|
||||
# the tool-author surface: client module byte-exact vs the repo file
|
||||
status, served, _ = self._request(base, "GET", CLIENT_STATIC_PATH)
|
||||
self.assertEqual(200, status)
|
||||
with open(REPO_CLIENT_PATH, "rb") as handle:
|
||||
self.assertEqual(handle.read(), served)
|
||||
|
||||
# one consolidated v0.1 request-pattern flow:
|
||||
# admin login -> app + user allowlist -> agent register -> state
|
||||
# put/get -> secret read
|
||||
cookie = self._admin_login(base, printed_pass)
|
||||
app_uuid = str(uuidlib.uuid4())
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", APPS_PATH,
|
||||
{"uuid": app_uuid, "name": "acceptance-tool"}, headers=cookie,
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", USERS_PATH,
|
||||
{"user_id": "ceo", "email": "ceo@coreci.dev"}, headers=cookie,
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", REGISTER_PATH,
|
||||
{"agent_id": "proxmox", "app_registration_uuid": app_uuid,
|
||||
"user_id": "ceo", "tags": ["coreci"]},
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
app_header = {"Authorization": "App %s %s" % (app_uuid, body["token"])}
|
||||
state_document = {"milestone": "v0.2", "matrix": "acceptance"}
|
||||
status, body, _ = self._request_json(
|
||||
base, "PUT", AGENT_STATE_PATH, state_document, headers=app_header
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
status, body, _ = self._request_json(
|
||||
base, "GET", AGENT_STATE_PATH, headers=app_header
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(state_document, body)
|
||||
status, body, _ = self._request_json(
|
||||
base, "GET", SECRETS_PATH, headers=app_header
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"path", "data", "version", "updated_at"}, set(body))
|
||||
self.assertEqual(PROXMOX_PATH, body["path"])
|
||||
self.assertEqual(PROXMOX_SECRET, body["data"])
|
||||
|
||||
# v0.1 state-file parity: 0600 in the state dir
|
||||
for name in STATE_FILE_NAMES:
|
||||
path = os.path.join(state_dir, name)
|
||||
self.assertTrue(os.path.isfile(path), path)
|
||||
self.assertEqual(0o600, os.stat(path).st_mode & 0o777, path)
|
||||
|
||||
# pass printed exactly once + pepper never printed — asserted on
|
||||
# the FULLY drained output after terminate (G-010)
|
||||
pepper = None
|
||||
with open(os.path.join(state_dir, ".env"), "r", encoding="utf-8") as handle:
|
||||
for line in handle:
|
||||
if line.startswith("BROKER_INSTANCE_PEPPER="):
|
||||
pepper = line.split("=", 1)[1].strip()
|
||||
self.assertIsNotNone(pepper)
|
||||
child.stop()
|
||||
first_output = child.full_output()
|
||||
self.assertEqual(1, first_output.count(ADMIN_PASS_PREFIX), first_output)
|
||||
self.assertEqual(1, first_output.count(printed_pass))
|
||||
self.assertNotIn(pepper, first_output)
|
||||
|
||||
# SECOND boot on the same state dir: silent after terminate + drain
|
||||
second = self._boot(state_dir)
|
||||
# the SAME pass still logs in: state persisted across the reboot
|
||||
self._admin_login(second.base_url, printed_pass)
|
||||
second.stop()
|
||||
second_output = second.full_output()
|
||||
self.assertIn(LISTENING_PREFIX, second_output)
|
||||
self.assertNotIn(ADMIN_PASS_PREFIX, second_output)
|
||||
self.assertNotIn(printed_pass, second_output)
|
||||
|
||||
# -- matrix 2: reproducibility ----------------------------------------
|
||||
|
||||
def test_reproducibility_two_consecutive_builds_identical_sha256(self):
|
||||
digests = []
|
||||
for _ in range(2):
|
||||
with tempfile.TemporaryDirectory(prefix="coreci-acc-repro-") as dist_dir:
|
||||
artifact = build_cli.build(
|
||||
VERSION, repo=REPO, dist_dir=dist_dir
|
||||
)
|
||||
digests.append(_sha256(artifact))
|
||||
self.assertNotEqual(digests[0], "0" * 64)
|
||||
self.assertEqual(digests[0], digests[1])
|
||||
# and the shared setUpClass build matches too (same environment)
|
||||
self.assertEqual(self.artifact_sha256, digests[0])
|
||||
print("v0.2 acceptance two-build sha256: {}".format(digests[0]))
|
||||
|
||||
# -- matrix 3: self-containment ---------------------------------------
|
||||
|
||||
def test_self_containment_archive_is_exactly_the_allowlist(self):
|
||||
ui_root = os.path.join(REPO, "coreci_broker_ui")
|
||||
expected = {
|
||||
"__main__.py",
|
||||
"_cli_version.py",
|
||||
"coreci_broker.py",
|
||||
"coreci_broker_client.py",
|
||||
"coreci_broker_ui/",
|
||||
}
|
||||
expected.update(
|
||||
"coreci_broker_ui/%s" % name
|
||||
for name in os.listdir(ui_root)
|
||||
if os.path.isfile(os.path.join(ui_root, name))
|
||||
)
|
||||
with zipfile.ZipFile(self.artifact) as archive:
|
||||
names = set(archive.namelist())
|
||||
# ONLY the allowlist members — nothing more, nothing less
|
||||
self.assertEqual(expected, names)
|
||||
# no .env, no dist/, no state, no tests, no bytecode can ride along
|
||||
for name in names:
|
||||
for forbidden in (".env", "dist", "broker.db", "broker.audit.log",
|
||||
"test", "__pycache__", ".pyc"):
|
||||
self.assertNotIn(forbidden, name, name)
|
||||
# per-release artifact name (D-020) for the milestone release
|
||||
self.assertEqual(
|
||||
build_cli.artifact_name(VERSION),
|
||||
os.path.basename(self.artifact),
|
||||
)
|
||||
|
||||
# -- matrix 4: stub publish roundtrip + stub release upload green ------
|
||||
|
||||
def test_stub_publish_roundtrip_module_green_in_suite(self):
|
||||
module = self._load_test_module("test_publish_image.py")
|
||||
self.assertTrue(hasattr(module, "PublishImageTests"))
|
||||
count = self._runnable_case_count(module)
|
||||
self.assertGreaterEqual(
|
||||
count, STUB_MODULE_MIN_CASES["test_publish_image.py"]
|
||||
)
|
||||
|
||||
def test_stub_release_upload_module_green_in_suite(self):
|
||||
module = self._load_test_module("test_release_upload.py")
|
||||
self.assertTrue(hasattr(module, "ReleaseUploadTests"))
|
||||
self.assertTrue(hasattr(module, "EnvSecretsLoaderTests"))
|
||||
count = self._runnable_case_count(module)
|
||||
self.assertGreaterEqual(
|
||||
count, STUB_MODULE_MIN_CASES["test_release_upload.py"]
|
||||
)
|
||||
|
||||
# -- matrix 6: Dockerfile/.dockerignore consistency ---------------------
|
||||
|
||||
def test_dockerfile_consistency_module_green_in_suite(self):
|
||||
module = self._load_test_module("test_dockerfile_consistency.py")
|
||||
self.assertTrue(hasattr(module, "DockerfileConsistencyTests"))
|
||||
count = self._runnable_case_count(module)
|
||||
self.assertGreaterEqual(
|
||||
count, STUB_MODULE_MIN_CASES["test_dockerfile_consistency.py"]
|
||||
)
|
||||
|
||||
# -- matrix 5: broker zero-delta (AC14, frozen means untouched) --------
|
||||
|
||||
def test_broker_zero_delta_1390_lines_no_packaging_references(self):
|
||||
with open(BROKER_SOURCE_PATH, "r", encoding="utf-8") as handle:
|
||||
line_count = sum(1 for _ in handle)
|
||||
# exact v0.1.5 count, under the D-016..D-018 cap of 1400 (AC14)
|
||||
self.assertEqual(1390, line_count)
|
||||
self.assertLess(line_count, 1400)
|
||||
with open(BROKER_SOURCE_PATH, "r", encoding="utf-8") as handle:
|
||||
source = handle.read()
|
||||
# frozen means untouched: no v0.2 packaging reference leaked into it
|
||||
for marker in FROZEN_MARKER_STRINGS:
|
||||
self.assertNotIn(marker, source)
|
||||
|
||||
def test_broker_untouched_by_any_commit_since_v0_1_5(self):
|
||||
"""The AC14 no-touch clause, machine-checked when the tag exists."""
|
||||
try:
|
||||
verify = subprocess.run(
|
||||
["git", "rev-parse", "-q", "--verify", "v0.1.5^{commit}"],
|
||||
cwd=REPO, capture_output=True, text=True,
|
||||
timeout=GIT_TIMEOUT_SECONDS,
|
||||
)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
self.skipTest("git unavailable in this environment")
|
||||
if verify.returncode != 0:
|
||||
self.skipTest("v0.1.5 tag not present in this working tree")
|
||||
log = subprocess.run(
|
||||
["git", "log", "--oneline", "v0.1.5..HEAD", "--"] + list(FROZEN_SOURCES),
|
||||
cwd=REPO, capture_output=True, text=True,
|
||||
timeout=GIT_TIMEOUT_SECONDS,
|
||||
)
|
||||
self.assertEqual(0, log.returncode, log.stderr)
|
||||
self.assertEqual(
|
||||
"", log.stdout.strip(),
|
||||
"v0.2 commits touched frozen broker sources:\n" + log.stdout
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,163 @@
|
||||
"""Reproducibility and packaging tests for the coreci-broker CLI build.
|
||||
|
||||
Covers scripts/build_cli.py (REQ-21): two consecutive same-environment
|
||||
builds are byte-identical, `_cli_version.VERSION` is embedded, the archive
|
||||
contains exactly the staging allowlist, entry metadata is normalized to
|
||||
constants (G-009), both new files py_compile green, and the built artifact
|
||||
answers `--version` with exit code 0 (shebang + exec bit verified).
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import io
|
||||
import os
|
||||
import pathlib
|
||||
import py_compile
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import zipfile
|
||||
|
||||
REPO = pathlib.Path(__file__).resolve().parent.parent
|
||||
BUILD_CLI_PATH = REPO / "scripts" / "build_cli.py"
|
||||
SHIM_PATH = REPO / "cli" / "__main__.py"
|
||||
VERSION = "v0.2.0"
|
||||
|
||||
UI_PAGES = (
|
||||
"app_registrations.html",
|
||||
"audit.html",
|
||||
"dashboard.html",
|
||||
"index.html",
|
||||
"login.html",
|
||||
"nav.js",
|
||||
"shared.css",
|
||||
"user_allowlist.html",
|
||||
)
|
||||
|
||||
FILE_EXTERNAL_ATTR = 0o100644 << 16
|
||||
DIR_EXTERNAL_ATTR = 0o40755 << 16
|
||||
SHEBANG = b"#!/usr/bin/env python3\n"
|
||||
FIXED_DATE_TIME = (1980, 1, 1, 0, 0, 0)
|
||||
|
||||
|
||||
def _load_build_cli():
|
||||
spec = importlib.util.spec_from_file_location("build_cli", BUILD_CLI_PATH)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
build_cli = _load_build_cli()
|
||||
|
||||
|
||||
def _sha256(path):
|
||||
digest = hashlib.sha256()
|
||||
with open(path, "rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(65536), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
class BuildCliTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.build_root = tempfile.TemporaryDirectory(prefix="coreci-broker-pyz-")
|
||||
cls.addClassCleanup(cls.build_root.cleanup)
|
||||
cls.artifact = build_cli.build(
|
||||
VERSION, repo=str(REPO), dist_dir=cls.build_root.name
|
||||
)
|
||||
cls.artifact_sha256 = _sha256(cls.artifact)
|
||||
|
||||
def test_two_builds_byte_identical(self):
|
||||
digests = []
|
||||
for _ in range(2):
|
||||
with tempfile.TemporaryDirectory(prefix="coreci-broker-pyz-") as dist_dir:
|
||||
artifact = build_cli.build(
|
||||
VERSION, repo=str(REPO), dist_dir=dist_dir
|
||||
)
|
||||
digests.append(_sha256(artifact))
|
||||
self.assertEqual(digests[0], digests[1])
|
||||
self.assertEqual(self.artifact_sha256, digests[0])
|
||||
print("two-build sha256: {}".format(digests[0]))
|
||||
|
||||
def test_cli_version_module_embedded(self):
|
||||
with zipfile.ZipFile(self.artifact) as archive:
|
||||
source = archive.read("_cli_version.py").decode("utf-8")
|
||||
self.assertEqual('VERSION = "{}"\n'.format(VERSION), source)
|
||||
namespace = {}
|
||||
exec(compile(source, "_cli_version.py", "exec"), namespace)
|
||||
self.assertEqual(VERSION, namespace["VERSION"])
|
||||
|
||||
def test_archive_allowlist_exact(self):
|
||||
ui_dir = REPO / "coreci_broker_ui"
|
||||
on_disk = {entry.name for entry in ui_dir.iterdir() if entry.is_file()}
|
||||
self.assertEqual(set(UI_PAGES), on_disk)
|
||||
expected = {
|
||||
"__main__.py",
|
||||
"_cli_version.py",
|
||||
"coreci_broker.py",
|
||||
"coreci_broker_client.py",
|
||||
"coreci_broker_ui/",
|
||||
}
|
||||
expected.update("coreci_broker_ui/{}".format(name) for name in UI_PAGES)
|
||||
with zipfile.ZipFile(self.artifact) as archive:
|
||||
self.assertEqual(expected, set(archive.namelist()))
|
||||
|
||||
def test_entry_metadata_constants(self):
|
||||
with zipfile.ZipFile(self.artifact) as archive:
|
||||
infos = archive.infolist()
|
||||
names = [info.filename for info in infos]
|
||||
self.assertEqual(sorted(names), names)
|
||||
self.assertTrue(any(info.is_dir() for info in infos))
|
||||
self.assertTrue(any(not info.is_dir() for info in infos))
|
||||
for info in infos:
|
||||
self.assertEqual(FIXED_DATE_TIME, info.date_time)
|
||||
if info.is_dir():
|
||||
self.assertEqual(DIR_EXTERNAL_ATTR, info.external_attr)
|
||||
else:
|
||||
self.assertEqual(FILE_EXTERNAL_ATTR, info.external_attr)
|
||||
self.assertEqual(zipfile.ZIP_DEFLATED, info.compress_type)
|
||||
|
||||
def test_py_compile_green(self):
|
||||
for path in (SHIM_PATH, BUILD_CLI_PATH):
|
||||
py_compile.compile(str(path), doraise=True)
|
||||
|
||||
def test_version_flag(self):
|
||||
completed = subprocess.run(
|
||||
[sys.executable, self.artifact, "--version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=60,
|
||||
)
|
||||
self.assertEqual(0, completed.returncode)
|
||||
self.assertEqual(VERSION, completed.stdout.strip())
|
||||
|
||||
def test_shebang_and_exec_bit(self):
|
||||
with open(self.artifact, "rb") as handle:
|
||||
self.assertEqual(SHEBANG, handle.readline())
|
||||
self.assertTrue(os.stat(self.artifact).st_mode & 0o111)
|
||||
completed = subprocess.run(
|
||||
[self.artifact, "--version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=60,
|
||||
)
|
||||
self.assertEqual(0, completed.returncode)
|
||||
self.assertEqual(VERSION, completed.stdout.strip())
|
||||
|
||||
def test_artifact_name_no_double_v(self):
|
||||
self.assertEqual("coreci-broker-v0.2.0.pyz", build_cli.artifact_name(VERSION))
|
||||
self.assertEqual("coreci-broker-v0.2.0.pyz", build_cli.artifact_name("0.2.0"))
|
||||
self.assertEqual("coreci-broker-v0.2.0.pyz", os.path.basename(self.artifact))
|
||||
|
||||
def test_cli_requires_version_argument(self):
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
with self.assertRaises(SystemExit) as raised:
|
||||
build_cli.main([])
|
||||
self.assertEqual(2, raised.exception.code)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,528 @@
|
||||
"""End-to-end packaged-boot acceptance for the coreci-broker CLI zipapp (REQ-21).
|
||||
|
||||
Builds the P1-01 artifact once per class via the scripts/build_cli.py build
|
||||
API into a ``TemporaryDirectory``, then boots it as a real subprocess child —
|
||||
always ``[sys.executable, "<pyz>"]``, never a bare "python3" — and drives the
|
||||
live server with stdlib urllib, reusing the v0.1 acceptance request patterns
|
||||
(tests/test_acceptance_full.py, tests/test_acceptance_p4.py).
|
||||
|
||||
G-010 harness hardening, all bound here:
|
||||
- explicit deadline loop (30s) polling the merged stdout/stderr pipe for the
|
||||
``coreci-broker listening on http://…`` line; loud failure with the captured
|
||||
child output when it never arrives;
|
||||
- pass-once and silent-second-boot assertions are made only after
|
||||
``terminate()`` plus a full ``communicate()`` drain of the child;
|
||||
- child env pins ``BROKER_CREDS_SOURCE=file`` + a seeded ``BROKER_CREDS_FILE``
|
||||
(v0.1 precedent: tests/test_acceptance_full.py);
|
||||
- one un-overridden default-anchoring boot — state lands in the child's CWD;
|
||||
- every child is terminated + drained exactly once (idempotent ``stop``
|
||||
registered with ``addCleanup``): no leaked processes, and every wait is
|
||||
deadline-bounded (boot wait, drain timeout with kill fallback, HTTP and
|
||||
subprocess timeouts).
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import select
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
|
||||
REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
|
||||
BUILD_CLI_PATH = os.path.join(REPO, "scripts", "build_cli.py")
|
||||
REPO_CLIENT_PATH = os.path.join(REPO, "coreci_broker_client.py")
|
||||
REPO_LOGIN_PAGE_PATH = os.path.join(REPO, "coreci_broker_ui", "login.html")
|
||||
REPO_DASHBOARD_PAGE_PATH = os.path.join(REPO, "coreci_broker_ui", "dashboard.html")
|
||||
VERSION = "v0.2.0"
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS_PREFIX = "BROKER_ADMIN_PASS="
|
||||
LISTENING_PREFIX = "coreci-broker listening on http://"
|
||||
ASSET_DIR_NAME = ".coreci_broker_assets"
|
||||
STATE_FILE_NAMES = (".env", "broker.db", "broker.audit.log")
|
||||
|
||||
LOGIN_PATH = "/api/broker/auth/login"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
AGENT_STATE_PATH = "/api/broker/agents/proxmox/state"
|
||||
SECRETS_PATH = "/api/broker/secrets/coreci/proxmox"
|
||||
AUDIT_PATH = "/api/broker/audit"
|
||||
CLIENT_STATIC_PATH = "/static/coreci_broker_client.py"
|
||||
UI_LOGIN_PATH = "/ui/login"
|
||||
UI_DASHBOARD_PATH = "/ui/dashboard"
|
||||
|
||||
PROXMOX_PATH = "coreci/proxmox"
|
||||
PROXMOX_SECRET = {"url": "https://pve.coreci.dev:8006", "token": "pve-tok"}
|
||||
GITEA_PATH = "coreci/gitea"
|
||||
GITEA_SECRET = {"url": "https://git.coreci.dev", "token": "git-tok"}
|
||||
|
||||
BOOT_DEADLINE_SECONDS = 30.0
|
||||
POLL_SECONDS = 0.1
|
||||
DRAIN_TIMEOUT_SECONDS = 20.0
|
||||
HTTP_TIMEOUT_SECONDS = 10
|
||||
SUBPROCESS_TIMEOUT_SECONDS = 60
|
||||
|
||||
|
||||
def _load_build_cli():
|
||||
spec = importlib.util.spec_from_file_location("build_cli", BUILD_CLI_PATH)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
build_cli = _load_build_cli()
|
||||
|
||||
|
||||
class PackagedBrokerChild:
|
||||
"""A subprocess child booted from the built zipapp artifact.
|
||||
|
||||
stdout and stderr are merged into one pipe read via ``select`` + ``os.read``
|
||||
(raw fd reads — no buffered-reader/select skew) under an explicit deadline
|
||||
until the broker announces its ephemeral listen address. ``stop()`` is the
|
||||
G-010 kill/drain protocol: ``terminate()`` then a full ``communicate()``
|
||||
drain; it is idempotent and registered with ``addCleanup`` at construction,
|
||||
so no child leaks even when a boot fails or an assertion trips mid-test.
|
||||
"""
|
||||
|
||||
def __init__(self, testcase, artifact, env, cwd):
|
||||
self.testcase = testcase
|
||||
self.artifact = artifact
|
||||
self.proc = subprocess.Popen(
|
||||
[sys.executable, artifact],
|
||||
env=env, cwd=cwd,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
|
||||
)
|
||||
self.lines = []
|
||||
self.partial = b""
|
||||
self.drained = b""
|
||||
self.base_url = None
|
||||
self._eof = False
|
||||
self._stopped = False
|
||||
testcase.addCleanup(self.stop)
|
||||
self._await_listening(BOOT_DEADLINE_SECONDS)
|
||||
|
||||
def _pump(self, timeout):
|
||||
"""Move whatever is on the merged pipe into line storage.
|
||||
|
||||
Returns False once the child's stream reached EOF; True otherwise
|
||||
(including when the poll window simply elapsed with no data).
|
||||
"""
|
||||
if self._eof:
|
||||
return False
|
||||
fd = self.proc.stdout.fileno()
|
||||
ready, _, _ = select.select([fd], [], [], timeout)
|
||||
if not ready:
|
||||
return True
|
||||
try:
|
||||
chunk = os.read(fd, 4096)
|
||||
except OSError:
|
||||
chunk = b""
|
||||
if not chunk:
|
||||
self._eof = True
|
||||
return False
|
||||
self.partial += chunk
|
||||
while b"\n" in self.partial:
|
||||
raw, self.partial = self.partial.split(b"\n", 1)
|
||||
self.lines.append(raw.decode("utf-8", "replace"))
|
||||
return True
|
||||
|
||||
def _await_listening(self, deadline_seconds):
|
||||
deadline = time.monotonic() + deadline_seconds
|
||||
cursor = len(self.lines)
|
||||
while time.monotonic() < deadline:
|
||||
alive = self._pump(POLL_SECONDS)
|
||||
while cursor < len(self.lines):
|
||||
line = self.lines[cursor]
|
||||
cursor += 1
|
||||
if line.startswith(LISTENING_PREFIX):
|
||||
self.base_url = line.split(" on ", 1)[1].strip()
|
||||
return
|
||||
if not alive:
|
||||
break
|
||||
self.stop()
|
||||
self.testcase.fail(
|
||||
"packaged broker did not announce %r within %.0fs "
|
||||
"(artifact=%s exit=%s)\n--- captured child output ---\n%s"
|
||||
% (LISTENING_PREFIX, deadline_seconds, self.artifact,
|
||||
self.proc.returncode, self.full_output())
|
||||
)
|
||||
|
||||
def stop(self):
|
||||
"""Kill/drain protocol (G-010): terminate, then fully drain."""
|
||||
if self._stopped:
|
||||
return
|
||||
self._stopped = True
|
||||
if self.proc.poll() is None:
|
||||
self.proc.terminate()
|
||||
try:
|
||||
remainder, _ = self.proc.communicate(timeout=DRAIN_TIMEOUT_SECONDS)
|
||||
except subprocess.TimeoutExpired:
|
||||
self.proc.kill()
|
||||
remainder, _ = self.proc.communicate()
|
||||
self.drained = remainder or b""
|
||||
|
||||
def full_output(self):
|
||||
"""Everything the child printed: complete lines + unterminated tail."""
|
||||
text = "\n".join(self.lines)
|
||||
tail = self.partial + self.drained
|
||||
if tail:
|
||||
text += "\n" + tail.decode("utf-8", "replace")
|
||||
return text
|
||||
|
||||
def printed_admin_pass(self):
|
||||
values = [
|
||||
line.split("=", 1)[1]
|
||||
for line in self.lines
|
||||
if line.startswith(ADMIN_PASS_PREFIX)
|
||||
]
|
||||
self.testcase.assertEqual(
|
||||
1, len(values),
|
||||
"expected exactly one %s line, got: %r" % (ADMIN_PASS_PREFIX, values)
|
||||
)
|
||||
return values[0]
|
||||
|
||||
|
||||
class CliPackagingAcceptanceTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.build_root = tempfile.TemporaryDirectory(prefix="coreci-broker-pkg-")
|
||||
cls.addClassCleanup(cls.build_root.cleanup)
|
||||
cls.artifact = build_cli.build(
|
||||
VERSION, repo=REPO, dist_dir=cls.build_root.name
|
||||
)
|
||||
|
||||
# -- harness helpers -------------------------------------------------
|
||||
|
||||
def _temp_dir(self):
|
||||
handle = tempfile.TemporaryDirectory(prefix="coreci-broker-pkg-")
|
||||
self.addCleanup(handle.cleanup)
|
||||
return handle.name
|
||||
|
||||
def _child_env(self, **overrides):
|
||||
"""A child env with no inherited BROKER_* keys, then test overrides."""
|
||||
env = {
|
||||
key: value
|
||||
for key, value in os.environ.items()
|
||||
if not key.startswith("BROKER_")
|
||||
}
|
||||
env.update(overrides)
|
||||
return env
|
||||
|
||||
def _seed_creds(self, state_dir):
|
||||
creds_path = os.path.join(state_dir, "creds.json")
|
||||
with open(creds_path, "w", encoding="utf-8") as handle:
|
||||
json.dump({PROXMOX_PATH: PROXMOX_SECRET, GITEA_PATH: GITEA_SECRET},
|
||||
handle)
|
||||
return creds_path
|
||||
|
||||
def _state_env(self, state_dir):
|
||||
"""G-010: pin bind + state paths + a seeded file creds source."""
|
||||
return self._child_env(
|
||||
BROKER_BIND="127.0.0.1:0",
|
||||
BROKER_ENV_PATH=os.path.join(state_dir, ".env"),
|
||||
BROKER_DB_PATH=os.path.join(state_dir, "broker.db"),
|
||||
BROKER_AUDIT_LOG_PATH=os.path.join(state_dir, "broker.audit.log"),
|
||||
BROKER_CREDS_SOURCE="file",
|
||||
BROKER_CREDS_FILE=self._seed_creds(state_dir),
|
||||
)
|
||||
|
||||
def _boot(self, env, cwd):
|
||||
return PackagedBrokerChild(self, self.artifact, env, cwd)
|
||||
|
||||
def _boot_in_state(self, state_dir, cwd=None):
|
||||
"""Boot with state pinned into state_dir; CWD is a separate temp dir."""
|
||||
return self._boot(self._state_env(state_dir),
|
||||
cwd if cwd is not None else self._temp_dir())
|
||||
|
||||
def _assert_state_files_0600_in(self, directory):
|
||||
for name in STATE_FILE_NAMES:
|
||||
path = os.path.join(directory, name)
|
||||
self.assertTrue(os.path.isfile(path), path)
|
||||
self.assertEqual(0o600, os.stat(path).st_mode & 0o777, path)
|
||||
|
||||
def _request(self, base, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(
|
||||
request, timeout=HTTP_TIMEOUT_SECONDS
|
||||
) as response:
|
||||
return response.status, response.read(), response.headers
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read(), error.headers
|
||||
|
||||
def _request_json(self, base, method, path, payload=None, headers=None):
|
||||
status, body, response_headers = self._request(
|
||||
base, method, path, payload, headers
|
||||
)
|
||||
return status, json.loads(body.decode("utf-8")), response_headers
|
||||
|
||||
def _admin_login(self, base, password):
|
||||
"""v0.1 pattern: POST login, assert the broker_session cookie is set."""
|
||||
status, body, headers = self._request_json(
|
||||
base, "POST", LOGIN_PATH,
|
||||
{"username": ADMIN_USER, "password": password},
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True, "username": ADMIN_USER}, body)
|
||||
raw_cookie = headers.get("Set-Cookie")
|
||||
self.assertTrue(raw_cookie)
|
||||
self.assertTrue(raw_cookie.startswith("broker_session="))
|
||||
self.assertIn("HttpOnly", raw_cookie)
|
||||
self.assertIn("SameSite=Lax", raw_cookie)
|
||||
return {"Cookie": raw_cookie.split(";", 1)[0]}
|
||||
|
||||
def _assert_tool_identity(self, entry, app_uuid):
|
||||
self.assertEqual("ceo", entry["user_id"])
|
||||
self.assertEqual(app_uuid, entry["app_registration_uuid"])
|
||||
self.assertEqual("proxmox", entry["agent_id"])
|
||||
|
||||
def _read_repo_bytes(self, path):
|
||||
with open(path, "rb") as handle:
|
||||
return handle.read()
|
||||
|
||||
# -- tests ------------------------------------------------------------
|
||||
|
||||
def test_version_flag_subprocess_exits_zero_printing_version(self):
|
||||
completed = subprocess.run(
|
||||
[sys.executable, self.artifact, "--version"],
|
||||
capture_output=True, text=True, timeout=SUBPROCESS_TIMEOUT_SECONDS,
|
||||
)
|
||||
self.assertEqual(0, completed.returncode)
|
||||
self.assertEqual(VERSION, completed.stdout.strip())
|
||||
|
||||
def test_first_boot_state_0600_cwd_independent_and_http_acceptance(self):
|
||||
state_dir = self._temp_dir()
|
||||
cwd_dir = self._temp_dir() # deliberately NOT the state dir
|
||||
child = self._boot_in_state(state_dir, cwd=cwd_dir)
|
||||
base = child.base_url
|
||||
|
||||
status, body, _ = self._request_json(base, "GET", "/healthz")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
|
||||
# UI pages served from the materialized assets (byte-exact vs repo)
|
||||
status, body, headers = self._request(base, "GET", UI_LOGIN_PATH)
|
||||
self.assertEqual(200, status)
|
||||
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
|
||||
self.assertEqual(self._read_repo_bytes(REPO_LOGIN_PAGE_PATH), body)
|
||||
status, body, headers = self._request(base, "GET", UI_DASHBOARD_PATH)
|
||||
self.assertEqual(200, status)
|
||||
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
|
||||
self.assertEqual(self._read_repo_bytes(REPO_DASHBOARD_PAGE_PATH), body)
|
||||
|
||||
# packaged client module served byte-exact vs the repo file
|
||||
status, served, _ = self._request(base, "GET", CLIENT_STATIC_PATH)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(self._read_repo_bytes(REPO_CLIENT_PATH), served)
|
||||
|
||||
# materialized assets under <state>/.coreci_broker_assets/<VERSION>/
|
||||
asset_base = os.path.join(state_dir, ASSET_DIR_NAME, VERSION)
|
||||
self.assertTrue(os.path.isdir(os.path.join(asset_base, "ui")))
|
||||
self.assertTrue(os.path.isdir(os.path.join(asset_base, "static")))
|
||||
for name in ("login.html", "dashboard.html"):
|
||||
self.assertTrue(
|
||||
os.path.isfile(os.path.join(asset_base, "ui", name)), name
|
||||
)
|
||||
self.assertTrue(os.path.isfile(
|
||||
os.path.join(asset_base, "static", "coreci_broker_client.py")
|
||||
))
|
||||
|
||||
# state resolved from BROKER_ENV_PATH: 0600 files in the state dir,
|
||||
# nothing whatsoever in the child's CWD
|
||||
self._assert_state_files_0600_in(state_dir)
|
||||
self.assertEqual([], os.listdir(cwd_dir))
|
||||
|
||||
# pass printed exactly once — asserted on the FULLY drained output
|
||||
printed_pass = child.printed_admin_pass()
|
||||
child.stop()
|
||||
output = child.full_output()
|
||||
self.assertEqual(1, output.count(ADMIN_PASS_PREFIX), output)
|
||||
self.assertEqual(1, output.count(printed_pass))
|
||||
pepper = None
|
||||
with open(os.path.join(state_dir, ".env"), "r", encoding="utf-8") as handle:
|
||||
for line in handle:
|
||||
if line.startswith("BROKER_INSTANCE_PEPPER="):
|
||||
pepper = line.split("=", 1)[1].strip()
|
||||
self.assertIsNotNone(pepper)
|
||||
self.assertNotIn(pepper, output)
|
||||
|
||||
def test_v01_parity_flows_through_packaged_artifact(self):
|
||||
state_dir = self._temp_dir()
|
||||
child = self._boot_in_state(state_dir)
|
||||
base = child.base_url
|
||||
printed_pass = child.printed_admin_pass()
|
||||
cookie = self._admin_login(base, printed_pass)
|
||||
|
||||
# seed allowlists first via the admin API (v0.1 pattern)
|
||||
app_uuid = str(uuidlib.uuid4())
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", APPS_PATH,
|
||||
{"uuid": app_uuid, "name": "proxmox-tool"}, headers=cookie,
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
self.assertEqual({"uuid": app_uuid, "status": "active"}, body)
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", USERS_PATH,
|
||||
{"user_id": "ceo", "email": "ceo@coreci.dev"}, headers=cookie,
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
# agent register flow
|
||||
status, body, _ = self._request_json(
|
||||
base, "POST", REGISTER_PATH,
|
||||
{"agent_id": "proxmox", "app_registration_uuid": app_uuid,
|
||||
"user_id": "ceo", "tags": ["coreci"]},
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
token = body["token"]
|
||||
app_header = {"Authorization": "App %s %s" % (app_uuid, token)}
|
||||
|
||||
# state put/get roundtrip
|
||||
state_document = {"phase": "v0.2", "boot": "packaged", "items": ["parity"]}
|
||||
status, body, _ = self._request_json(
|
||||
base, "PUT", AGENT_STATE_PATH, state_document, headers=app_header
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
status, body, _ = self._request_json(
|
||||
base, "GET", AGENT_STATE_PATH, headers=app_header
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(state_document, body)
|
||||
|
||||
# one secret read via the seeded FileCredsSource
|
||||
status, body, _ = self._request_json(
|
||||
base, "GET", SECRETS_PATH, headers=app_header
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"path", "data", "version", "updated_at"}, set(body))
|
||||
self.assertEqual(PROXMOX_PATH, body["path"])
|
||||
self.assertEqual(PROXMOX_SECRET, body["data"])
|
||||
self.assertRegex(body["version"], r"^[0-9a-f]{8}$")
|
||||
|
||||
# audit lines exist for the driven flow
|
||||
status, body, _ = self._request_json(
|
||||
base, "GET", AUDIT_PATH + "?limit=1000", headers=cookie
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
entries = body["entries"]
|
||||
|
||||
def select(action, result="ok"):
|
||||
return [entry for entry in entries
|
||||
if entry["action"] == action and entry["result"] == result]
|
||||
|
||||
logins = select("login")
|
||||
self.assertEqual(1, len(logins))
|
||||
self.assertEqual(LOGIN_PATH, logins[0]["path"])
|
||||
add_apps = select("add_app")
|
||||
self.assertEqual(1, len(add_apps))
|
||||
self.assertEqual(APPS_PATH + "/" + app_uuid, add_apps[0]["path"])
|
||||
add_users = select("add_user")
|
||||
self.assertEqual(1, len(add_users))
|
||||
self.assertEqual(USERS_PATH + "/ceo", add_users[0]["path"])
|
||||
registers = select("register")
|
||||
self.assertEqual(1, len(registers))
|
||||
self.assertEqual(REGISTER_PATH, registers[0]["path"])
|
||||
self._assert_tool_identity(registers[0], app_uuid)
|
||||
ok_reads = select("read")
|
||||
self.assertEqual(1, len(ok_reads))
|
||||
self.assertEqual(PROXMOX_PATH, ok_reads[0]["path"])
|
||||
self._assert_tool_identity(ok_reads[0], app_uuid)
|
||||
|
||||
child.stop()
|
||||
self.assertEqual(1, child.full_output().count(ADMIN_PASS_PREFIX))
|
||||
|
||||
def test_second_boot_silent_and_state_persists(self):
|
||||
state_dir = self._temp_dir()
|
||||
first = self._boot_in_state(state_dir)
|
||||
printed_pass = first.printed_admin_pass()
|
||||
cookie = self._admin_login(first.base_url, printed_pass)
|
||||
app_uuid = str(uuidlib.uuid4())
|
||||
status, _, _ = self._request_json(
|
||||
first.base_url, "POST", APPS_PATH,
|
||||
{"uuid": app_uuid, "name": "persist-check"}, headers=cookie,
|
||||
)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
# kill/drain protocol BEFORE any silence assertion (G-010)
|
||||
first.stop()
|
||||
first_output = first.full_output()
|
||||
self.assertEqual(1, first_output.count(ADMIN_PASS_PREFIX))
|
||||
self.assertIn(printed_pass, first_output)
|
||||
|
||||
second = self._boot_in_state(state_dir)
|
||||
# the same pass still logs in on the second boot: .env persisted
|
||||
cookie = self._admin_login(second.base_url, printed_pass)
|
||||
status, body, _ = self._request_json(
|
||||
second.base_url, "GET", APPS_PATH, headers=cookie
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
listed = [row["uuid"] for row in body["registrations"]]
|
||||
self.assertIn(app_uuid, listed)
|
||||
|
||||
second.stop()
|
||||
second_output = second.full_output()
|
||||
self.assertIn(LISTENING_PREFIX, second_output)
|
||||
self.assertNotIn(ADMIN_PASS_PREFIX, second_output)
|
||||
|
||||
def test_user_env_ui_root_override_wins_over_materialized_default(self):
|
||||
state_dir = self._temp_dir()
|
||||
custom_root = os.path.join(state_dir, "custom_ui")
|
||||
os.makedirs(custom_root)
|
||||
marker = b"<h1>packaged-override-marker</h1>\n"
|
||||
with open(os.path.join(custom_root, "login.html"), "wb") as handle:
|
||||
handle.write(marker)
|
||||
# a user .env (at the child's BROKER_ENV_PATH) wins over the shim default
|
||||
with open(os.path.join(state_dir, ".env"), "w", encoding="utf-8") as handle:
|
||||
handle.write("BROKER_UI_ROOT=%s\n" % custom_root)
|
||||
|
||||
child = self._boot_in_state(state_dir)
|
||||
status, body, headers = self._request(child.base_url, "GET", UI_LOGIN_PATH)
|
||||
self.assertEqual(200, status)
|
||||
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
|
||||
self.assertEqual(marker, body)
|
||||
# the custom root is the whole UI root: no dashboard.html there
|
||||
status, _, _ = self._request(child.base_url, "GET", UI_DASHBOARD_PATH)
|
||||
self.assertEqual(404, status)
|
||||
# the materialized static default still applies (not overridden)
|
||||
status, served, _ = self._request(child.base_url, "GET", CLIENT_STATIC_PATH)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(self._read_repo_bytes(REPO_CLIENT_PATH), served)
|
||||
# materialized defaults still exist on disk; they were simply not forced
|
||||
self.assertTrue(os.path.isfile(os.path.join(
|
||||
state_dir, ASSET_DIR_NAME, VERSION, "ui", "login.html"
|
||||
)))
|
||||
child.stop()
|
||||
|
||||
def test_default_anchoring_boot_writes_state_to_cwd(self):
|
||||
# the owner's literal happy path: no state overrides except the bind
|
||||
cwd_dir = self._temp_dir()
|
||||
env = self._child_env(BROKER_BIND="127.0.0.1:0")
|
||||
child = self._boot(env, cwd_dir)
|
||||
|
||||
status, body, _ = self._request_json(child.base_url, "GET", "/healthz")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual({"ok": True}, body)
|
||||
status, body, headers = self._request(child.base_url, "GET", UI_LOGIN_PATH)
|
||||
self.assertEqual(200, status)
|
||||
self.assertTrue(headers.get("Content-Type", "").startswith("text/html"))
|
||||
|
||||
# state lands in the child's CWD, at 0600
|
||||
self._assert_state_files_0600_in(cwd_dir)
|
||||
|
||||
child.stop()
|
||||
self.assertEqual(1, child.full_output().count(ADMIN_PASS_PREFIX))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,264 @@
|
||||
"""Dockerfile consistency tests (G-016 — equivalence made testable).
|
||||
|
||||
The Dockerfile documents the docker-user path; the daemonless client
|
||||
(``scripts/publish_image.py``) is the pipeline's source of truth for
|
||||
the published image. Byte-identity is NOT required — these tests pin the
|
||||
equivalence contract: same staged sources (member-set parity — the
|
||||
docker-staged layout must equal build_cli's staging EXACTLY, or the
|
||||
packaged shim materializes an empty UI dir and /ui/* 404s in
|
||||
docker-built images; P3-03/CWE-636), same zipapp invocation, same
|
||||
entrypoint/working-dir, same pinned base, and a hygienic build context.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import zipfile
|
||||
|
||||
_SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.dirname(
|
||||
os.path.abspath(__file__))), "scripts")
|
||||
if _SCRIPTS_DIR not in sys.path:
|
||||
sys.path.insert(0, _SCRIPTS_DIR)
|
||||
|
||||
import build_cli # noqa: E402 (scripts/ dir on sys.path)
|
||||
import publish_image # noqa: E402 (scripts/ dir on sys.path)
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
APP_SRC = "/app/src/"
|
||||
DOCKERFILE_IGNORE_PATTERNS = (
|
||||
".git", ".env", ".env.*", "data/", "*.db", "*.log", "dist/",
|
||||
"__pycache__/", "tests/", ".ciagent/",
|
||||
"**/__pycache__/", "**/*.pyc",
|
||||
)
|
||||
|
||||
|
||||
def read(name):
|
||||
with open(os.path.join(REPO, name), "r", encoding="utf-8") as handle:
|
||||
return handle.read()
|
||||
|
||||
|
||||
def _tree_members(prefix, root):
|
||||
"""Zipapp member names for the tree at ``root`` staged under
|
||||
``prefix`` (``""`` for the archive root; the prefix itself is
|
||||
included — zipapp writes directory entries)."""
|
||||
members = {prefix} if prefix else set()
|
||||
for current, dirs, files in os.walk(root):
|
||||
relative = os.path.relpath(current, root)
|
||||
base = prefix + ("" if relative == "." else relative + "/")
|
||||
for name in dirs:
|
||||
members.add(base + name + "/")
|
||||
for name in files:
|
||||
members.add(base + name)
|
||||
return members
|
||||
|
||||
|
||||
def _build_cli_staged_members():
|
||||
"""The member set build_cli stages, derived via its constants:
|
||||
ROOT_SOURCES at the root, UI under the coreci_broker_ui/ prefix,
|
||||
cli/__main__.py staged as __main__.py, _cli_version.py generated."""
|
||||
members = set(build_cli.ROOT_SOURCES)
|
||||
members.add(build_cli.STAGED_SHIM)
|
||||
members.add(build_cli.VERSION_MODULE)
|
||||
members |= _tree_members(
|
||||
build_cli.UI_SOURCE_DIR + "/",
|
||||
os.path.join(REPO, build_cli.UI_SOURCE_DIR))
|
||||
return members
|
||||
|
||||
|
||||
def _run_zipapp(staged_dir, artifact):
|
||||
"""The Dockerfile's packing invocation (zipapp writes dir entries;
|
||||
the member set is interpreter-stable across 3.x)."""
|
||||
completed = subprocess.run(
|
||||
[sys.executable, "-m", "zipapp", staged_dir,
|
||||
"-o", artifact,
|
||||
"--python", "/usr/bin/env python3", "--compress"],
|
||||
capture_output=True, text=True)
|
||||
if completed.returncode != 0:
|
||||
raise AssertionError(
|
||||
"zipapp failed ({}): {}".format(
|
||||
completed.returncode,
|
||||
(completed.stderr or completed.stdout).strip()))
|
||||
|
||||
|
||||
class DockerfileConsistencyTests(unittest.TestCase):
|
||||
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.dockerfile = read("Dockerfile")
|
||||
cls.dockerignore = read(".dockerignore")
|
||||
# Join backslash continuations into logical stanzas; skip comments.
|
||||
stanzas = []
|
||||
for line in cls.dockerfile.splitlines():
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith("#"):
|
||||
continue
|
||||
if stanzas and stanzas[-1].endswith("\\"):
|
||||
stanzas[-1] = stanzas[-1].rstrip("\\").rstrip() + " " + stripped
|
||||
continue
|
||||
stanzas.append(stripped)
|
||||
cls.stanzas = stanzas
|
||||
cls.copy_lines = [s for s in stanzas if s.startswith("COPY ")]
|
||||
cls.zipapp_lines = [s for s in stanzas
|
||||
if "zipapp" in s and s.startswith("RUN ")]
|
||||
cls.version_lines = [s for s in stanzas
|
||||
if "_cli_version.py" in s
|
||||
and s.startswith("RUN ")]
|
||||
|
||||
def _copy_specs(self):
|
||||
"""[(sources, dest)] per COPY stanza, dest verified to stage
|
||||
under /app/src/."""
|
||||
specs = []
|
||||
for line in self.copy_lines:
|
||||
tokens = line.split()[1:]
|
||||
sources, dest = tokens[:-1], tokens[-1]
|
||||
self.assertTrue(dest.startswith(APP_SRC),
|
||||
"COPY destination {} must stage under /app/src/"
|
||||
.format(dest))
|
||||
specs.append((sources, dest))
|
||||
return specs
|
||||
|
||||
def _docker_staged_members(self):
|
||||
"""The member set the COPY stanzas stage under /app/src/ (Docker
|
||||
semantics: directory CONTENTS land flattened INSIDE the
|
||||
destination dir; a dest without trailing slash is a file rename),
|
||||
plus the RUN-generated _cli_version.py."""
|
||||
members = set()
|
||||
for sources, dest in self._copy_specs():
|
||||
if dest.endswith("/"):
|
||||
prefix = dest[len(APP_SRC):]
|
||||
for source in sources:
|
||||
path = os.path.join(REPO, source.rstrip("/"))
|
||||
self.assertTrue(os.path.exists(path),
|
||||
"COPY source missing in the repo: {}"
|
||||
.format(source))
|
||||
if os.path.isdir(path):
|
||||
members |= _tree_members(prefix, path)
|
||||
else:
|
||||
members.add(prefix + os.path.basename(source))
|
||||
else:
|
||||
# file-to-file rename (mirrors build_cli's STAGED_SHIM
|
||||
# staging — zipapp requires __main__.py at the root)
|
||||
self.assertEqual(len(sources), 1,
|
||||
"file-rename COPY stages exactly one "
|
||||
"source: {}".format(dest))
|
||||
members.add(dest[len(APP_SRC):])
|
||||
members.add(build_cli.VERSION_MODULE) # the RUN printf stanza
|
||||
return members
|
||||
|
||||
def test_from_pins_base(self):
|
||||
from_lines = [line for line in self.dockerfile.splitlines()
|
||||
if line.startswith("FROM ")]
|
||||
self.assertEqual(len(from_lines), 1)
|
||||
self.assertEqual(from_lines[0].split()[1], "python:3.11-slim")
|
||||
|
||||
def test_copy_staged_member_set_matches_build_cli_staging(self):
|
||||
"""The COPY stanzas stage EXACTLY build_cli's member set — set
|
||||
parity, no extras, no missing (a flattened UI copy or a missing
|
||||
root __main__.py breaks the packaged boot; P3-03/CWE-636)."""
|
||||
self.assertEqual(self._docker_staged_members(),
|
||||
_build_cli_staged_members(),
|
||||
"docker-staged member set == build_cli-staged "
|
||||
"member set")
|
||||
for line in self.copy_lines:
|
||||
self.assertNotIn("dist/", line,
|
||||
"no pre-built artifact in the build context")
|
||||
|
||||
def test_docker_layout_simulation_member_set_parity(self):
|
||||
"""Docker-layout simulation gate (P3-03/CWE-636): apply the
|
||||
Dockerfile COPY semantics to the real repo sources, add the
|
||||
generated _cli_version.py, zipapp-pack, and the archive's member
|
||||
set must EQUAL the build_cli-built artifact's member set."""
|
||||
with tempfile.TemporaryDirectory(prefix="docker-layout-") as tmp:
|
||||
staged = os.path.join(tmp, "src")
|
||||
os.makedirs(staged)
|
||||
for sources, dest in self._copy_specs():
|
||||
if dest.endswith("/"):
|
||||
target = os.path.join(staged, dest[len(APP_SRC):])
|
||||
os.makedirs(target, exist_ok=True)
|
||||
for source in sources:
|
||||
path = os.path.join(REPO, source.rstrip("/"))
|
||||
if os.path.isdir(path):
|
||||
# directory CONTENTS land inside the dest dir
|
||||
shutil.copytree(path, target,
|
||||
dirs_exist_ok=True)
|
||||
else:
|
||||
shutil.copy2(
|
||||
path,
|
||||
os.path.join(target,
|
||||
os.path.basename(source)))
|
||||
else:
|
||||
self.assertEqual(len(sources), 1)
|
||||
shutil.copy2(os.path.join(REPO, sources[0]),
|
||||
os.path.join(staged, dest[len(APP_SRC):]))
|
||||
with open(os.path.join(staged, build_cli.VERSION_MODULE), "w",
|
||||
encoding="utf-8") as handle:
|
||||
handle.write('VERSION = "{}"\n'.format("v0.2-layout-sim"))
|
||||
docker_artifact = os.path.join(tmp, "docker-layout.pyz")
|
||||
_run_zipapp(staged, docker_artifact)
|
||||
with zipfile.ZipFile(docker_artifact) as archive:
|
||||
docker_members = set(archive.namelist())
|
||||
with tempfile.TemporaryDirectory(prefix="build-cli-ref-") as dist:
|
||||
reference = build_cli.build("v0.2-layout-sim", repo=REPO,
|
||||
dist_dir=dist)
|
||||
with zipfile.ZipFile(reference) as archive:
|
||||
build_members = set(archive.namelist())
|
||||
self.assertEqual(docker_members, build_members,
|
||||
"docker-layout archive members == build_cli "
|
||||
"artifact members")
|
||||
|
||||
def test_zipapp_invocation_flags_match_build_script(self):
|
||||
"""Same --python interpreter and --compress flag as build_cli."""
|
||||
self.assertEqual(len(self.zipapp_lines), 1)
|
||||
stanza = self.zipapp_lines[0]
|
||||
self.assertIn('--python "/usr/bin/env python3"', stanza)
|
||||
self.assertIn("--compress", stanza)
|
||||
self.assertIn("-m zipapp", stanza)
|
||||
self.assertIn("/app/src", stanza)
|
||||
self.assertIn("/app/coreci-broker.pyz", stanza)
|
||||
|
||||
def test_version_module_format_parity(self):
|
||||
"""ARG VERSION feeds the same VERSION = "<value>" format."""
|
||||
self.assertEqual(len(self.version_lines), 1)
|
||||
self.assertIn('VERSION = "%s"', self.version_lines[0])
|
||||
self.assertEqual(build_cli.VERSION_MODULE, "_cli_version.py")
|
||||
self.assertTrue(any(s.startswith("ARG VERSION=") for s in self.stanzas),
|
||||
"ARG VERSION provides the version")
|
||||
|
||||
def test_entrypoint_and_workdir_match_published_config(self):
|
||||
"""ENTRYPOINT/WORKDIR equal the daemonless client's pushed config."""
|
||||
base_config = json.loads('{"config": {}, "rootfs": {}}')
|
||||
blob = json.loads(publish_image.build_config_blob(
|
||||
base_config, "sha256:" + "0" * 64, "v-parity"))
|
||||
entrypoint = blob["config"]["Entrypoint"]
|
||||
working_dir = blob["config"]["WorkingDir"]
|
||||
docker_entrypoint = [line for line in self.dockerfile.splitlines()
|
||||
if line.startswith("ENTRYPOINT ")]
|
||||
docker_workdir = [line for line in self.dockerfile.splitlines()
|
||||
if line.startswith("WORKDIR ")]
|
||||
self.assertEqual(len(docker_entrypoint), 1)
|
||||
self.assertEqual(json.loads(
|
||||
docker_entrypoint[0].split(" ", 1)[1]), entrypoint)
|
||||
self.assertEqual(len(docker_workdir), 1)
|
||||
self.assertEqual(docker_workdir[0].split(" ", 1)[1], working_dir)
|
||||
self.assertEqual(working_dir, "/state")
|
||||
|
||||
def test_dockerignore_hygiene(self):
|
||||
entries = {line.strip() for line in self.dockerignore.splitlines()
|
||||
if line.strip() and not line.strip().startswith("#")}
|
||||
for pattern in DOCKERFILE_IGNORE_PATTERNS:
|
||||
self.assertIn(pattern, entries,
|
||||
".dockerignore must exclude {}".format(pattern))
|
||||
|
||||
def test_no_absolute_source_copies(self):
|
||||
for line in self.copy_lines:
|
||||
for token in line.split()[1:-1]:
|
||||
self.assertFalse(token.startswith("/"),
|
||||
"build context sources are repo-relative")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,207 @@
|
||||
import base64
|
||||
import contextlib
|
||||
import errno
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid as uuidlib
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)))
|
||||
|
||||
import coreci_broker
|
||||
|
||||
ADMIN_USER = "admin"
|
||||
ADMIN_PASS = "correct-horse-battery-staple"
|
||||
REGISTER_PATH = "/api/broker/agents/register"
|
||||
APPS_PATH = "/api/broker/app_registrations"
|
||||
USERS_PATH = "/api/broker/user_allowlist"
|
||||
AGENT_ID_BODY = {"error": 400, "message": "invalid agent_id"}
|
||||
TAGS_BODY = {"error": 400, "message": "invalid tags"}
|
||||
USER_ID_BODY = {"error": 400, "message": "invalid user_id"}
|
||||
|
||||
|
||||
def _basic_header(username, password):
|
||||
raw = "%s:%s" % (username, password)
|
||||
return "Basic " + base64.b64encode(raw.encode("utf-8")).decode("ascii")
|
||||
|
||||
|
||||
class P5RegisterValidationTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.env = {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_ADMIN_USER": ADMIN_USER,
|
||||
"BROKER_ADMIN_PASS": ADMIN_PASS,
|
||||
}
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
self.broker = coreci_broker.boot(dict(self.env))
|
||||
self.addCleanup(self.broker.audit.close)
|
||||
self.addCleanup(self.broker.store.close)
|
||||
self.server = coreci_broker.serve("127.0.0.1", 0, self.broker)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
self.addCleanup(self.server.server_close)
|
||||
self.base = "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
self.admin = {"Authorization": _basic_header(ADMIN_USER, ADMIN_PASS)}
|
||||
self.app_uuid = str(uuidlib.uuid4())
|
||||
self.user_id = "ceo"
|
||||
self._seed()
|
||||
|
||||
def _request(self, method, path, payload=None, headers=None):
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(self.base + path, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
return response.status, json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, json.loads(error.read().decode("utf-8"))
|
||||
|
||||
def _seed(self):
|
||||
status, _ = self._request("POST", APPS_PATH, {"uuid": self.app_uuid}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
status, _ = self._request("POST", USERS_PATH, {"user_id": self.user_id}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
def _register(self, agent_id="agent-1", tags=("coreci",)):
|
||||
return self._request("POST", REGISTER_PATH, {
|
||||
"agent_id": agent_id,
|
||||
"app_registration_uuid": self.app_uuid,
|
||||
"user_id": self.user_id,
|
||||
"tags": list(tags),
|
||||
})
|
||||
|
||||
def _audit(self, action=None, result=None):
|
||||
with open(self.env["BROKER_AUDIT_LOG_PATH"], "r", encoding="utf-8") as handle:
|
||||
entries = [json.loads(raw) for raw in handle if raw.strip()]
|
||||
return [
|
||||
entry for entry in entries
|
||||
if (action is None or entry["action"] == action)
|
||||
and (result is None or entry["result"] == result)
|
||||
]
|
||||
|
||||
def test_agent_id_bare_dot_and_dotdot_rejected(self):
|
||||
for agent_id in (".", ".."):
|
||||
status, body = self._register(agent_id=agent_id)
|
||||
self.assertEqual(400, status, repr(agent_id))
|
||||
self.assertEqual(AGENT_ID_BODY, body)
|
||||
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
|
||||
self.assertEqual([], self._audit("register"))
|
||||
|
||||
def test_agent_id_control_chars_rejected(self):
|
||||
for agent_id in ("a\x00b", "a\nb", "a\x1fb", "a\x7fb"):
|
||||
status, body = self._register(agent_id=agent_id)
|
||||
self.assertEqual(400, status, repr(agent_id))
|
||||
self.assertEqual(AGENT_ID_BODY, body)
|
||||
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
|
||||
self.assertEqual([], self._audit("register"))
|
||||
|
||||
def test_agent_id_embedded_dots_still_accepted(self):
|
||||
status, body = self._register(agent_id="a.b")
|
||||
self.assertEqual(201, status)
|
||||
self.assertTrue(body["created"])
|
||||
status, body = self._register(agent_id="tool.v2")
|
||||
self.assertEqual(201, status)
|
||||
self.assertTrue(body["created"])
|
||||
self.assertIsNotNone(self.broker.store.get_agent("a.b", self.app_uuid))
|
||||
self.assertIsNotNone(self.broker.store.get_agent("tool.v2", self.app_uuid))
|
||||
self.assertEqual(2, len(self._audit("register", "ok")))
|
||||
|
||||
def test_tag_control_chars_rejected(self):
|
||||
for tags in (("ok", "ba\x00d"), ("ok", "ba\x7fd"), ("ok", "ba:d")):
|
||||
status, body = self._register(tags=tags)
|
||||
self.assertEqual(400, status, repr(tags))
|
||||
self.assertEqual(TAGS_BODY, body)
|
||||
self.assertEqual([], self.broker.store.agents_by_uuid(self.app_uuid))
|
||||
status, body = self._register(tags=("a.b", ".."))
|
||||
self.assertEqual(201, status)
|
||||
self.assertTrue(body["created"])
|
||||
row = self.broker.store.get_agent("agent-1", self.app_uuid)
|
||||
self.assertEqual(["..", "a.b"], json.loads(row[3]))
|
||||
|
||||
def test_agent_id_path_separators_still_rejected(self):
|
||||
for agent_id in ("a:b", "a/b", "a\\b"):
|
||||
status, body = self._register(agent_id=agent_id)
|
||||
self.assertEqual(400, status, repr(agent_id))
|
||||
self.assertEqual(AGENT_ID_BODY, body)
|
||||
self.assertEqual([], self._audit("register"))
|
||||
status, body = self._register(agent_id="plain-agent")
|
||||
self.assertEqual(201, status)
|
||||
self.assertTrue(body["created"])
|
||||
|
||||
def test_user_allowlist_add_shares_id_rules(self):
|
||||
for user_id in (".", "..", "a\x00b", "a:b", "a/b"):
|
||||
status, body = self._request("POST", USERS_PATH, {"user_id": user_id}, self.admin)
|
||||
self.assertEqual(400, status, repr(user_id))
|
||||
self.assertEqual(USER_ID_BODY, body)
|
||||
status, body = self._request("POST", USERS_PATH, {"user_id": "ceo.two"}, self.admin)
|
||||
self.assertEqual(201, status)
|
||||
|
||||
|
||||
class P5ReexecGuardTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
coreci_broker._reset_server_guard()
|
||||
self.addCleanup(coreci_broker._reset_server_guard)
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
|
||||
def _env(self, bind):
|
||||
return {
|
||||
"BROKER_ENV_PATH": os.path.join(self.tmp, ".env"),
|
||||
"BROKER_DB_PATH": os.path.join(self.tmp, "broker.db"),
|
||||
"BROKER_AUDIT_LOG_PATH": os.path.join(self.tmp, "broker.audit.log"),
|
||||
"BROKER_BIND": bind,
|
||||
}
|
||||
|
||||
def _exec_broker(self):
|
||||
with open(coreci_broker.__file__, "r", encoding="utf-8") as handle:
|
||||
source = handle.read()
|
||||
namespace = {"__name__": "coreci_broker_reexec"}
|
||||
exec(compile(source, "coreci_broker.py", "exec"), namespace)
|
||||
return namespace
|
||||
|
||||
def _healthz(self, port):
|
||||
url = "http://127.0.0.1:%d/healthz" % port
|
||||
with urllib.request.urlopen(url, timeout=5) as response:
|
||||
return response.status, json.loads(response.read().decode("utf-8"))
|
||||
|
||||
def test_fresh_namespace_tools_fails_loud_when_old_module_serves(self):
|
||||
first = self._exec_broker()
|
||||
with mock.patch.dict(os.environ, self._env("127.0.0.1:0"), clear=True):
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
first_tools = first["Tools"]()
|
||||
first_again = first["Tools"]()
|
||||
server = first["_started_server"]
|
||||
self.addCleanup(first["_reset_server_guard"])
|
||||
self.assertIs(server, first_tools._server)
|
||||
self.assertIs(first_tools._server, first_again._server)
|
||||
port = server.server_address[1]
|
||||
self.assertNotEqual(0, port)
|
||||
second = self._exec_broker()
|
||||
self.assertIsNone(second["_started_server"])
|
||||
with mock.patch.dict(os.environ, self._env("127.0.0.1:%d" % port), clear=True):
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
with self.assertRaises(RuntimeError) as caught:
|
||||
second["Tools"]()
|
||||
message = str(caught.exception).lower()
|
||||
self.assertIn("restart", message)
|
||||
self.assertIn("open webui", message)
|
||||
self.assertIsInstance(caught.exception.__cause__, OSError)
|
||||
self.assertEqual(errno.EADDRINUSE, caught.exception.__cause__.errno)
|
||||
self.assertIsNone(second["_started_server"])
|
||||
self.assertEqual((200, {"ok": True}), self._healthz(port))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,799 @@
|
||||
"""Stub-registry tests for scripts/publish_image.py (REQ-23, G-019).
|
||||
|
||||
Two local stub servers stand in for the public image registry (anonymous
|
||||
pull) and the gitea container registry (JWT auth + blob uploads +
|
||||
manifest PUTs). No network access; tiny in-memory fixtures; every
|
||||
spawned server is cleaned via addCleanup.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
import urllib.parse
|
||||
from contextlib import redirect_stderr, redirect_stdout
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
_SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.dirname(
|
||||
os.path.abspath(__file__))), "scripts")
|
||||
if _SCRIPTS_DIR not in sys.path:
|
||||
sys.path.insert(0, _SCRIPTS_DIR)
|
||||
|
||||
import publish_image # noqa: E402 (scripts/ dir on sys.path)
|
||||
|
||||
OCI_MANIFEST = "application/vnd.oci.image.manifest.v1+json"
|
||||
OCI_INDEX = "application/vnd.oci.image.index.v1+json"
|
||||
OCI_CONFIG = "application/vnd.oci.image.config.v1+json"
|
||||
OCI_LAYER = "application/vnd.oci.image.layer.v1.tar+gzip"
|
||||
OWNER = "testowner"
|
||||
REPO = "testrepo"
|
||||
NAME = "{}/{}".format(OWNER, REPO)
|
||||
|
||||
|
||||
def _sha(data):
|
||||
return "sha256:" + hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def _dump(payload):
|
||||
return json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
|
||||
|
||||
|
||||
class HubFixtures:
|
||||
"""A tiny synthetic base image: index -> amd64/arm64 children."""
|
||||
|
||||
def __init__(self):
|
||||
self.layer0 = b"base layer zero payload\x00" + b"\xa5" * 64
|
||||
self.layer1 = b"base layer one payload" * 7
|
||||
self.config = _dump({
|
||||
"architecture": "amd64",
|
||||
"os": "linux",
|
||||
"config": {"Cmd": ["python3"], "Env": ["PATH=/usr/local/bin"]},
|
||||
"rootfs": {"type": "layers",
|
||||
"diff_ids": [_sha(self.layer0), _sha(self.layer1)]},
|
||||
})
|
||||
self.child = {
|
||||
"schemaVersion": 2,
|
||||
"mediaType": OCI_MANIFEST,
|
||||
"config": {"mediaType": OCI_CONFIG,
|
||||
"digest": _sha(self.config),
|
||||
"size": len(self.config)},
|
||||
"layers": [
|
||||
{"mediaType": OCI_LAYER, "digest": _sha(self.layer0),
|
||||
"size": len(self.layer0)},
|
||||
{"mediaType": OCI_LAYER, "digest": _sha(self.layer1),
|
||||
"size": len(self.layer1)},
|
||||
],
|
||||
}
|
||||
self.child_digest = _sha(_dump(self.child))
|
||||
arm64_child = dict(self.child)
|
||||
arm64_child["annotations"] = {"test-fixture": "arm64"}
|
||||
self.arm64_child_digest = _sha(_dump(arm64_child))
|
||||
self.index = {
|
||||
"schemaVersion": 2,
|
||||
"mediaType": OCI_INDEX,
|
||||
"manifests": [
|
||||
{"mediaType": OCI_MANIFEST, "digest": self.child_digest,
|
||||
"size": len(_dump(self.child)),
|
||||
"platform": {"os": "linux", "architecture": "amd64"}},
|
||||
{"mediaType": OCI_MANIFEST,
|
||||
"digest": self.arm64_child_digest,
|
||||
"size": len(_dump(arm64_child)),
|
||||
"platform": {"os": "linux", "architecture": "arm64"}},
|
||||
],
|
||||
}
|
||||
self.blobs = {
|
||||
_sha(self.config): self.config,
|
||||
_sha(self.layer0): self.layer0,
|
||||
_sha(self.layer1): self.layer1,
|
||||
}
|
||||
|
||||
|
||||
class _StubHubHandler(BaseHTTPRequestHandler):
|
||||
fixtures = None # set per test class run
|
||||
requests = None
|
||||
token_mode = "ok" # ok | malformed (200 with an INVALID JSON body)
|
||||
|
||||
def _record(self):
|
||||
length = int(self.headers.get("Content-Length") or 0)
|
||||
body = self.rfile.read(length) if length else b""
|
||||
self.requests.append({
|
||||
"method": self.command,
|
||||
"path": self.path,
|
||||
"authorization": self.headers.get("Authorization", ""),
|
||||
"accept": self.headers.get("Accept", ""),
|
||||
"body": body,
|
||||
})
|
||||
|
||||
def _send(self, status, payload, content_type):
|
||||
data = payload if isinstance(payload, bytes) else _dump(payload)
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", content_type)
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
if self.command != "HEAD":
|
||||
self.wfile.write(data)
|
||||
|
||||
def do_GET(self):
|
||||
self._record()
|
||||
path = urllib.parse.urlsplit(self.path).path
|
||||
query = dict(urllib.parse.parse_qsl(
|
||||
urllib.parse.urlsplit(self.path).query))
|
||||
if path == "/token":
|
||||
if self.token_mode == "malformed":
|
||||
# 200 + invalid JSON: the CWE-754 probe — must surface as
|
||||
# a remote failure WITH the IMAGE_PENDING marker, never
|
||||
# a bare traceback.
|
||||
self._send(200, b"this { is not valid json", "application/json")
|
||||
else:
|
||||
self._send(200, {"token": "hub-anon-token"}, "application/json")
|
||||
elif path == "/v2/library/python/manifests/3.11-slim":
|
||||
self._send(200, self.fixtures.index, OCI_INDEX)
|
||||
elif path == "/v2/library/python/manifests/" + self.fixtures.child_digest:
|
||||
self._send(200, self.fixtures.child, OCI_MANIFEST)
|
||||
else:
|
||||
digest = path.rsplit("/", 1)[-1]
|
||||
if digest in self.fixtures.blobs:
|
||||
self._send(200, self.fixtures.blobs[digest],
|
||||
"application/octet-stream")
|
||||
else:
|
||||
self._send(404, {"errors": [{"digest": digest}]},
|
||||
"application/json")
|
||||
|
||||
def log_message(self, format, *args):
|
||||
pass
|
||||
|
||||
|
||||
class _StubGiteaHandler(BaseHTTPRequestHandler):
|
||||
"""Scriptable gitea registry stub.
|
||||
|
||||
state dict keys: requests, uploaded, token_mode (ok|empty),
|
||||
upload_mode (monolithic|redirect307|redirect307_cross|stall),
|
||||
manifest_mode (ok|fail500), redirect_seen, manifest_puts,
|
||||
cross_origin_base (absolute URL of the second-origin stub used by
|
||||
redirect307_cross).
|
||||
"""
|
||||
|
||||
state = None
|
||||
|
||||
def _record(self):
|
||||
length = int(self.headers.get("Content-Length") or 0)
|
||||
body = self.rfile.read(length) if length else b""
|
||||
self.state["requests"].append({
|
||||
"method": self.command,
|
||||
"path": self.path,
|
||||
"authorization": self.headers.get("Authorization", ""),
|
||||
"content_type": self.headers.get("Content-Type", ""),
|
||||
"body": body,
|
||||
})
|
||||
|
||||
def _send(self, status, payload=b"", content_type="application/json",
|
||||
headers=None):
|
||||
data = payload if isinstance(payload, bytes) else _dump(payload)
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", content_type)
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
for key, value in (headers or {}).items():
|
||||
self.send_header(key, value)
|
||||
self.end_headers()
|
||||
if self.command != "HEAD":
|
||||
self.wfile.write(data)
|
||||
|
||||
def do_GET(self):
|
||||
self._record()
|
||||
parsed = urllib.parse.urlsplit(self.path)
|
||||
if parsed.path == "/v2/":
|
||||
# G-013 smoke check 1: the registry auth challenge (401 +
|
||||
# Bearer; the realm defaults to this stub's own token
|
||||
# endpoint so pre-existing state dicts work unchanged).
|
||||
realm = self.state.get("challenge_realm") or \
|
||||
"http://{}:{}/v2/token".format(
|
||||
self.server.server_address[0],
|
||||
self.server.server_address[1])
|
||||
self._send(401, {"errors": [{"code": "UNAUTHORIZED"}]},
|
||||
headers={
|
||||
"WWW-Authenticate":
|
||||
'Bearer realm="{}",service="{}"'.format(
|
||||
realm, publish_image.GITEA_AUTH_SERVICE)})
|
||||
return
|
||||
if parsed.path == "/v2/token":
|
||||
if self.state["token_mode"] == "empty":
|
||||
self._send(200, {"token": ""})
|
||||
else:
|
||||
self._send(200, {"token": "stub-jwt"})
|
||||
return
|
||||
if parsed.path.startswith("/api/v1/repos/") and \
|
||||
"/releases/tags/" in parsed.path:
|
||||
# G-013 smoke check 3: the gitea API release-by-tag GET
|
||||
# (state["releases"]: tag -> release payload; absent -> 404).
|
||||
tag = parsed.path.rsplit("/", 1)[-1]
|
||||
release = self.state.get("releases", {}).get(tag)
|
||||
if release is None:
|
||||
self._send(404, {"errors": [{"message": "release not found"}]})
|
||||
else:
|
||||
self._send(200, release)
|
||||
return
|
||||
|
||||
def do_HEAD(self):
|
||||
self._record()
|
||||
parsed = urllib.parse.urlsplit(self.path)
|
||||
digest = parsed.path.rsplit("/", 1)[-1]
|
||||
if digest in self.state["uploaded"]:
|
||||
self._send(200, b"")
|
||||
else:
|
||||
self._send(404, b"")
|
||||
|
||||
def do_POST(self):
|
||||
self._record()
|
||||
parsed = urllib.parse.urlsplit(self.path)
|
||||
query = dict(urllib.parse.parse_qsl(parsed.query))
|
||||
digest = query.get("digest", "")
|
||||
if self.state["upload_mode"] == "stall":
|
||||
time.sleep(15)
|
||||
return
|
||||
if self.state["upload_mode"] == "redirect307_cross" and \
|
||||
digest not in self.state["redirect_seen"]:
|
||||
# CWE-200 probe: ABSOLUTE Location pointing at a DIFFERENT
|
||||
# origin (the second stub server) — the client must follow
|
||||
# WITHOUT the Authorization header.
|
||||
self.state["redirect_seen"].add(digest)
|
||||
location = "{}/v2/{}/blobs/uploads/?digest={}".format(
|
||||
self.state["cross_origin_base"], NAME, digest)
|
||||
self._send(307, b"", headers={"Location": location})
|
||||
return
|
||||
if self.state["upload_mode"] == "redirect307" and \
|
||||
digest not in self.state["redirect_seen"]:
|
||||
self.state["redirect_seen"].add(digest)
|
||||
location = "/v2/{}/blobs/uploads/?digest={}".format(NAME, digest)
|
||||
self._send(307, b"", headers={"Location": location})
|
||||
return
|
||||
if _sha(self.state["requests"][-1]["body"]) == digest:
|
||||
self.state["uploaded"][digest] = self.state["requests"][-1]["body"]
|
||||
self._send(201, {"status": "created"})
|
||||
else:
|
||||
self._send(400, {"errors": [{"digest": digest}]})
|
||||
|
||||
def do_PUT(self):
|
||||
self._record()
|
||||
parsed = urllib.parse.urlsplit(self.path)
|
||||
query = dict(urllib.parse.parse_qsl(parsed.query))
|
||||
body = self.state["requests"][-1]["body"]
|
||||
if "/manifests/" in parsed.path:
|
||||
if self.state["manifest_mode"] == "fail500":
|
||||
self.state.setdefault("manifest_fail_count", 0)
|
||||
self.state["manifest_fail_count"] += 1
|
||||
self._send(500, {"errors": [{"code": "UNKNOWN"}]})
|
||||
return
|
||||
manifest = json.loads(body)
|
||||
referenced = [manifest["config"]["digest"]] + [
|
||||
layer["digest"] for layer in manifest["layers"]]
|
||||
if any(digest not in self.state["uploaded"]
|
||||
for digest in referenced):
|
||||
self._send(400, {"errors": [{"code": "BLOB_UNKNOWN"}]})
|
||||
return
|
||||
if self.headers.get("Content-Type") != OCI_MANIFEST:
|
||||
self._send(400, {"errors": [{"code": "MANIFEST_INVALID"}]})
|
||||
return
|
||||
self.state.setdefault("manifest_puts", []).append(
|
||||
{"tag": parsed.path.rsplit("/", 1)[-1],
|
||||
"content_type": self.headers.get("Content-Type", ""),
|
||||
"manifest": manifest})
|
||||
self._send(201, {})
|
||||
return
|
||||
digest = query.get("digest", "")
|
||||
if _sha(body) == digest:
|
||||
self.state["uploaded"][digest] = body
|
||||
self._send(201, {})
|
||||
else:
|
||||
self._send(400, {"errors": [{"code": "DIGEST_INVALID"}]})
|
||||
|
||||
|
||||
class _StubCrossOriginHandler(BaseHTTPRequestHandler):
|
||||
"""Second-origin stub (another port) for the CWE-200 redirect probe.
|
||||
|
||||
state dict keys: requests, uploaded — ``uploaded`` is SHARED with the
|
||||
gitea stub (the redirect target models the registry's presigned
|
||||
external storage; a landed blob is visible to the registry), while
|
||||
``requests`` stays separate so the test can assert on exactly what
|
||||
this origin received — above all that NO Authorization header
|
||||
crossed the origin boundary.
|
||||
"""
|
||||
|
||||
state = None
|
||||
|
||||
def _record(self):
|
||||
length = int(self.headers.get("Content-Length") or 0)
|
||||
body = self.rfile.read(length) if length else b""
|
||||
self.state["requests"].append({
|
||||
"method": self.command,
|
||||
"path": self.path,
|
||||
"authorization": self.headers.get("Authorization", ""),
|
||||
"content_type": self.headers.get("Content-Type", ""),
|
||||
"body": body,
|
||||
})
|
||||
return body
|
||||
|
||||
def _send(self, status, payload=b"", content_type="application/json"):
|
||||
data = payload if isinstance(payload, bytes) else _dump(payload)
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", content_type)
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
if self.command != "HEAD":
|
||||
self.wfile.write(data)
|
||||
|
||||
def do_POST(self):
|
||||
body = self._record()
|
||||
query = dict(urllib.parse.parse_qsl(
|
||||
urllib.parse.urlsplit(self.path).query))
|
||||
digest = query.get("digest", "")
|
||||
if _sha(body) == digest:
|
||||
self.state["uploaded"][digest] = body
|
||||
self._send(201, {"status": "created"})
|
||||
else:
|
||||
self._send(400, {"errors": [{"code": "DIGEST_INVALID"}]})
|
||||
|
||||
def log_message(self, format, *args):
|
||||
pass
|
||||
|
||||
|
||||
class PublishImageTests(unittest.TestCase):
|
||||
"""End-to-end publish flows against local stub registries."""
|
||||
|
||||
TAG = "v0.2.2-test"
|
||||
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.fixtures = HubFixtures()
|
||||
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.artifact = os.path.join(self.temp.name, "coreci-broker.pyz")
|
||||
with open(self.artifact, "wb") as handle:
|
||||
handle.write(b"fake pyz payload for tests" * 3)
|
||||
self.env_root = os.path.join(self.temp.name, "envroot")
|
||||
os.makedirs(self.env_root)
|
||||
with open(os.path.join(self.env_root, ".env.secrets"), "w") as handle:
|
||||
handle.write("GITEA_TOKEN=testtoken123\n")
|
||||
|
||||
self.hub_requests = []
|
||||
_StubHubHandler.fixtures = self.fixtures
|
||||
_StubHubHandler.requests = self.hub_requests
|
||||
_StubHubHandler.token_mode = "ok"
|
||||
self.hub = ThreadingHTTPServer(("127.0.0.1", 0), _StubHubHandler)
|
||||
threading.Thread(target=self.hub.serve_forever, daemon=True).start()
|
||||
self.addCleanup(self.hub.shutdown)
|
||||
self.addCleanup(self.hub.server_close)
|
||||
|
||||
self.gitea_state = {
|
||||
"requests": [], "uploaded": {}, "token_mode": "ok",
|
||||
"upload_mode": "monolithic", "manifest_mode": "ok",
|
||||
"redirect_seen": set(),
|
||||
}
|
||||
_StubGiteaHandler.state = self.gitea_state
|
||||
self.gitea = ThreadingHTTPServer(("127.0.0.1", 0), _StubGiteaHandler)
|
||||
threading.Thread(target=self.gitea.serve_forever, daemon=True).start()
|
||||
self.addCleanup(self.gitea.shutdown)
|
||||
self.addCleanup(self.gitea.server_close)
|
||||
|
||||
def run_publish(self, tag=None, timeout=60, artifact=None, env_root=None):
|
||||
argv = [
|
||||
tag or self.TAG,
|
||||
"--artifact", artifact or self.artifact,
|
||||
"--hub-registry", "http://127.0.0.1:{}".format(self.hub.server_address[1]),
|
||||
"--hub-auth", "http://127.0.0.1:{}/token".format(self.hub.server_address[1]),
|
||||
"--gitea-registry", "http://127.0.0.1:{}".format(self.gitea.server_address[1]),
|
||||
"--owner", OWNER, "--repo", REPO,
|
||||
"--env-root", env_root or self.env_root,
|
||||
"--timeout", str(timeout),
|
||||
]
|
||||
out, err = io.StringIO(), io.StringIO()
|
||||
with redirect_stdout(out), redirect_stderr(err):
|
||||
code = publish_image.main(argv)
|
||||
return code, out.getvalue(), err.getvalue()
|
||||
|
||||
# 1 -----------------------------------------------------------------
|
||||
def test_happy_path_end_to_end(self):
|
||||
code, out, err = self.run_publish()
|
||||
self.assertEqual(code, 0, err or out)
|
||||
puts = self.gitea_state.get("manifest_puts", [])
|
||||
self.assertEqual([entry["tag"] for entry in puts],
|
||||
[self.TAG, "latest"],
|
||||
"version tag FIRST, then latest (D-021)")
|
||||
manifest = puts[0]["manifest"]
|
||||
self.assertEqual(puts[0]["content_type"], OCI_MANIFEST)
|
||||
self.assertEqual(manifest["config"]["mediaType"], OCI_CONFIG)
|
||||
broker_layer = manifest["layers"][-1]
|
||||
self.assertEqual(broker_layer["mediaType"], OCI_LAYER)
|
||||
base_descriptors = self.fixtures.child["layers"]
|
||||
self.assertEqual(manifest["layers"][:-1], base_descriptors,
|
||||
"base layer descriptors reused verbatim")
|
||||
# deterministic broker layer digest matches an independent build
|
||||
with open(self.artifact, "rb") as handle:
|
||||
artifact_bytes = handle.read()
|
||||
layer_blob, layer_digest, diff_id = publish_image.build_broker_layer(
|
||||
artifact_bytes)
|
||||
self.assertEqual(broker_layer["digest"], layer_digest)
|
||||
self.assertEqual(broker_layer["size"], len(layer_blob))
|
||||
# pushed config content
|
||||
pushed_config = json.loads(
|
||||
self.gitea_state["uploaded"][manifest["config"]["digest"]])
|
||||
self.assertEqual(pushed_config["config"]["Entrypoint"],
|
||||
["python3", "/app/coreci-broker.pyz"])
|
||||
self.assertEqual(pushed_config["config"]["Cmd"], [])
|
||||
self.assertEqual(pushed_config["config"]["WorkingDir"], "/state")
|
||||
self.assertEqual(pushed_config["config"]["Labels"],
|
||||
{"org.opencontainers.image.version": self.TAG})
|
||||
self.assertEqual(pushed_config["rootfs"]["diff_ids"][-1], diff_id)
|
||||
self.assertEqual(len(pushed_config["rootfs"]["diff_ids"]),
|
||||
len(self.fixtures.child["layers"]) + 1)
|
||||
# base config blob uploaded verbatim (digest invariant)
|
||||
self.assertEqual(
|
||||
self.gitea_state["uploaded"][self.fixtures.child["config"]["digest"]],
|
||||
self.fixtures.config)
|
||||
# all blobs present
|
||||
for descriptor in manifest["layers"]:
|
||||
self.assertIn(descriptor["digest"], self.gitea_state["uploaded"])
|
||||
self.assertNotIn("IMAGE_PENDING", out)
|
||||
|
||||
# 2 -----------------------------------------------------------------
|
||||
def test_head_skip_blob_not_reuploaded(self):
|
||||
preseed = self.fixtures.child["layers"][0]["digest"]
|
||||
self.gitea_state["uploaded"][preseed] = self.fixtures.layer0
|
||||
code, out, err = self.run_publish()
|
||||
self.assertEqual(code, 0, err or out)
|
||||
posts = [r for r in self.gitea_state["requests"]
|
||||
if r["method"] == "POST" and preseed in r["path"]]
|
||||
self.assertEqual(posts, [], "pre-seeded blob must not be re-uploaded")
|
||||
hub_blob_gets = [r for r in self.hub_requests
|
||||
if r["method"] == "GET" and "/blobs/" in r["path"]
|
||||
and preseed in r["path"]]
|
||||
self.assertEqual(hub_blob_gets, [],
|
||||
"skipped blob is never fetched from the hub")
|
||||
|
||||
# 3 -----------------------------------------------------------------
|
||||
def test_redirect_307_relative_location_followed(self):
|
||||
self.gitea_state["upload_mode"] = "redirect307"
|
||||
code, out, err = self.run_publish()
|
||||
self.assertEqual(code, 0, err or out)
|
||||
blob_posts = [r for r in self.gitea_state["requests"]
|
||||
if r["method"] == "POST" and "/blobs/uploads/" in r["path"]
|
||||
and "digest=" in r["path"]]
|
||||
self.assertGreater(len(blob_posts), 0)
|
||||
digests = [dict(urllib.parse.parse_qsl(
|
||||
urllib.parse.urlsplit(r["path"]).query))["digest"]
|
||||
for r in blob_posts]
|
||||
self.assertTrue(all(d in self.gitea_state["uploaded"]
|
||||
for d in digests),
|
||||
"every redirected upload must land")
|
||||
# each redirected digest POSTed twice (307 + follow)
|
||||
for digest in set(digests):
|
||||
self.assertEqual(digests.count(digest), 2,
|
||||
"307 then explicit re-POST per digest")
|
||||
# same-origin (relative Location) redirects KEEP the
|
||||
# Authorization header — the CWE-200 strip is cross-origin ONLY
|
||||
for request in blob_posts:
|
||||
self.assertEqual(
|
||||
request["authorization"], "Bearer stub-jwt",
|
||||
"same-origin redirect must keep Authorization")
|
||||
|
||||
# 4 -----------------------------------------------------------------
|
||||
def test_anonymous_token_trap_refused_before_push(self):
|
||||
self.gitea_state["token_mode"] = "empty"
|
||||
code, out, err = self.run_publish()
|
||||
self.assertEqual(code, 1, "remote auth failure -> non-zero")
|
||||
self.assertIn("anonymous-200 trap refused", err)
|
||||
self.assertIn("IMAGE_PENDING", out)
|
||||
post_put = [r for r in self.gitea_state["requests"]
|
||||
if r["method"] in ("POST", "PUT")]
|
||||
self.assertEqual(post_put, [],
|
||||
"no blob upload or manifest PUT may happen")
|
||||
|
||||
# 5 -----------------------------------------------------------------
|
||||
def test_layer_and_config_determinism(self):
|
||||
with open(self.artifact, "rb") as handle:
|
||||
artifact = handle.read()
|
||||
first = publish_image.build_broker_layer(artifact)
|
||||
second = publish_image.build_broker_layer(artifact)
|
||||
self.assertEqual(first[1], second[1], "layer digest stable")
|
||||
self.assertEqual(first[2], second[2], "diff_id stable")
|
||||
base_config = json.loads(self.fixtures.config)
|
||||
c1 = publish_image.build_config_blob(base_config, first[2], "t1")
|
||||
c2 = publish_image.build_config_blob(base_config, first[2], "t1")
|
||||
self.assertEqual(_sha(c1), _sha(c2), "config digest stable")
|
||||
|
||||
# 6 -----------------------------------------------------------------
|
||||
def test_amd64_child_selected(self):
|
||||
code, out, err = self.run_publish()
|
||||
self.assertEqual(code, 0, err or out)
|
||||
child_gets = [r for r in self.hub_requests
|
||||
if "/manifests/sha256:" in r["path"]]
|
||||
self.assertTrue(child_gets, "child manifest must be fetched by digest")
|
||||
self.assertTrue(all(self.fixtures.child_digest in r["path"]
|
||||
for r in child_gets),
|
||||
"amd64 child only")
|
||||
self.assertFalse(any(self.fixtures.arm64_child_digest in r["path"]
|
||||
for r in self.hub_requests),
|
||||
"arm64 child never touched")
|
||||
|
||||
# 7 -----------------------------------------------------------------
|
||||
def test_timeout_aborts_rather_than_hangs(self):
|
||||
self.gitea_state["upload_mode"] = "stall"
|
||||
started = time.monotonic()
|
||||
code, out, err = self.run_publish(timeout=1)
|
||||
elapsed = time.monotonic() - started
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("IMAGE_PENDING", out)
|
||||
self.assertLess(elapsed, 30, "bounded retries, no hang")
|
||||
stalls = [r for r in self.gitea_state["requests"]
|
||||
if r["method"] == "POST" and "/blobs/uploads/" in r["path"]]
|
||||
self.assertLessEqual(len(stalls), 3, "bounded retry attempts")
|
||||
|
||||
# 8 -----------------------------------------------------------------
|
||||
def test_manifest_retry_exhaustion_marker(self):
|
||||
self.gitea_state["manifest_mode"] = "fail500"
|
||||
code, out, err = self.run_publish()
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("IMAGE_PENDING", out)
|
||||
self.assertEqual(self.gitea_state.get("manifest_fail_count"), 3,
|
||||
"exactly three bounded attempts on the tag PUT")
|
||||
self.assertNotIn("latest",
|
||||
[r["path"].rsplit("/", 1)[-1]
|
||||
for r in self.gitea_state["requests"]
|
||||
if r["method"] == "PUT"],
|
||||
"latest must not be attempted after exhaustion")
|
||||
|
||||
# 9 -----------------------------------------------------------------
|
||||
def test_token_discipline(self):
|
||||
code, out, err = self.run_publish()
|
||||
self.assertEqual(code, 0, err or out)
|
||||
token_calls = [r for r in self.gitea_state["requests"]
|
||||
if r["path"].startswith("/v2/token")]
|
||||
self.assertTrue(token_calls)
|
||||
expected = base64.b64encode(
|
||||
b"publisher:testtoken123").decode("ascii")
|
||||
for call in token_calls:
|
||||
self.assertEqual(call["authorization"],
|
||||
"Basic {}".format(expected))
|
||||
# scope requested with pull,push (registry v2 scope form)
|
||||
query = urllib.parse.unquote(
|
||||
urllib.parse.urlsplit(token_calls[0]["path"]).query)
|
||||
self.assertIn("repository:{}/{}:pull,push".format(OWNER, REPO),
|
||||
query)
|
||||
# missing token: local precondition, zero remote requests
|
||||
empty_root = os.path.join(self.temp.name, "emptyroot")
|
||||
os.makedirs(empty_root)
|
||||
before = len(self.gitea_state["requests"]) + len(self.hub_requests)
|
||||
code, out, err = self.run_publish(env_root=empty_root)
|
||||
self.assertEqual(code, 2, "local precondition -> exit 2")
|
||||
self.assertNotIn("IMAGE_PENDING", out, "no marker for local errors")
|
||||
after = len(self.gitea_state["requests"]) + len(self.hub_requests)
|
||||
self.assertEqual(before, after, "no request of any kind was made")
|
||||
|
||||
# 10 ----------------------------------------------------------------
|
||||
def test_malformed_hub_token_json_keeps_pending_marker(self):
|
||||
"""CWE-754 marker discipline: a 200 with an INVALID JSON body from
|
||||
the hub token endpoint fails as a remote answer — exit 1 WITH the
|
||||
IMAGE_PENDING marker — never an uncaught-ValueError traceback."""
|
||||
_StubHubHandler.token_mode = "malformed"
|
||||
code, out, err = self.run_publish()
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("malformed JSON", err,
|
||||
"the failure class must be named in the error")
|
||||
self.assertIn("IMAGE_PENDING", out,
|
||||
"marker discipline holds for malformed remote JSON")
|
||||
self.assertNotIn("Traceback", out + err)
|
||||
self.assertEqual(self.gitea_state["requests"], [],
|
||||
"nothing is attempted against gitea")
|
||||
|
||||
# 11 ----------------------------------------------------------------
|
||||
def test_redirect_307_cross_origin_strips_authorization(self):
|
||||
"""CWE-200: a 307 whose Location resolves to a DIFFERENT origin
|
||||
is followed WITHOUT the Authorization header — the registry JWT
|
||||
never crosses origins — and the blob still lands."""
|
||||
cross_state = {"requests": [], "uploaded": self.gitea_state["uploaded"]}
|
||||
_StubCrossOriginHandler.state = cross_state
|
||||
cross = ThreadingHTTPServer(("127.0.0.1", 0), _StubCrossOriginHandler)
|
||||
threading.Thread(target=cross.serve_forever, daemon=True).start()
|
||||
self.addCleanup(cross.shutdown)
|
||||
self.addCleanup(cross.server_close)
|
||||
self.gitea_state["upload_mode"] = "redirect307_cross"
|
||||
self.gitea_state["cross_origin_base"] = "http://127.0.0.1:{}".format(
|
||||
cross.server_address[1])
|
||||
code, out, err = self.run_publish()
|
||||
self.assertEqual(code, 0, err or out)
|
||||
self.assertTrue(cross_state["requests"],
|
||||
"the redirect was followed to the other origin")
|
||||
for request in cross_state["requests"]:
|
||||
self.assertEqual(
|
||||
request["authorization"], "",
|
||||
"Authorization must be stripped on cross-origin redirects")
|
||||
# the blob landed (visible through the shared registry blob store)
|
||||
self.assertEqual(len(self.gitea_state["uploaded"]), 5,
|
||||
"base config + 2 base layers + broker layer + "
|
||||
"image config all landed")
|
||||
# and the flow still completes: both manifest PUTs back on the
|
||||
# origin, where Authorization remains in play
|
||||
self.assertEqual(
|
||||
[entry["tag"] for entry in self.gitea_state.get(
|
||||
"manifest_puts", [])],
|
||||
[self.TAG, "latest"])
|
||||
|
||||
|
||||
class SmokeModeTests(unittest.TestCase):
|
||||
"""G-013 --smoke logic, stub-tested.
|
||||
|
||||
The LIVE smoke never runs in this suite — it executes only at the
|
||||
P3-05 ship, where the ship gate sets PUBLISH_SMOKE=1 in the
|
||||
environment (the script itself never reads that variable). Here the
|
||||
four GET-only checks run against fresh stub instances: zero network,
|
||||
and the stub records double as the read-only proof (a POST/PUT
|
||||
landing anywhere in a smoke run is a hard failure).
|
||||
"""
|
||||
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.env_root = os.path.join(self.temp.name, "envroot")
|
||||
os.makedirs(self.env_root)
|
||||
with open(os.path.join(self.env_root, ".env.secrets"), "w") as handle:
|
||||
handle.write("GITEA_TOKEN=testtoken123\n")
|
||||
|
||||
self.fixtures = HubFixtures()
|
||||
self.hub_requests = []
|
||||
_StubHubHandler.fixtures = self.fixtures
|
||||
_StubHubHandler.requests = self.hub_requests
|
||||
_StubHubHandler.token_mode = "ok"
|
||||
self.hub = ThreadingHTTPServer(("127.0.0.1", 0), _StubHubHandler)
|
||||
threading.Thread(target=self.hub.serve_forever, daemon=True).start()
|
||||
self.addCleanup(self.hub.shutdown)
|
||||
self.addCleanup(self.hub.server_close)
|
||||
|
||||
self.gitea_state = {
|
||||
"requests": [], "uploaded": {}, "token_mode": "ok",
|
||||
"upload_mode": "monolithic", "manifest_mode": "ok",
|
||||
"redirect_seen": set(), "releases": {},
|
||||
}
|
||||
_StubGiteaHandler.state = self.gitea_state
|
||||
self.gitea = ThreadingHTTPServer(("127.0.0.1", 0), _StubGiteaHandler)
|
||||
threading.Thread(target=self.gitea.serve_forever, daemon=True).start()
|
||||
self.addCleanup(self.gitea.shutdown)
|
||||
self.addCleanup(self.gitea.server_close)
|
||||
# The /v2/ challenge realm must point back at THIS stub's token
|
||||
# endpoint so the recorded-realm exchange stays in-stub.
|
||||
self.gitea_state["challenge_realm"] = \
|
||||
"http://127.0.0.1:{}/v2/token".format(
|
||||
self.gitea.server_address[1])
|
||||
|
||||
def run_smoke(self, tag, env_root=None):
|
||||
argv = [
|
||||
tag, "--smoke",
|
||||
"--hub-registry",
|
||||
"http://127.0.0.1:{}".format(self.hub.server_address[1]),
|
||||
"--hub-auth",
|
||||
"http://127.0.0.1:{}/token".format(
|
||||
self.hub.server_address[1]),
|
||||
"--gitea-registry",
|
||||
"http://127.0.0.1:{}".format(self.gitea.server_address[1]),
|
||||
"--owner", OWNER, "--repo", REPO,
|
||||
"--env-root", env_root or self.env_root,
|
||||
]
|
||||
out, err = io.StringIO(), io.StringIO()
|
||||
with redirect_stdout(out), redirect_stderr(err):
|
||||
code = publish_image.main(argv)
|
||||
return code, out.getvalue(), err.getvalue()
|
||||
|
||||
def _post_put(self):
|
||||
"""(gitea POST/PUT, hub POST/PUT) across the stub records."""
|
||||
return ([r for r in self.gitea_state["requests"]
|
||||
if r["method"] in ("POST", "PUT")],
|
||||
[r for r in self.hub_requests
|
||||
if r["method"] in ("POST", "PUT")])
|
||||
|
||||
def _release_gets(self):
|
||||
return [r for r in self.gitea_state["requests"]
|
||||
if "/releases/tags/" in r["path"]]
|
||||
|
||||
# 12 ----------------------------------------------------------------
|
||||
def test_smoke_all_checks_pass(self):
|
||||
self.gitea_state["releases"]["v0.2.0"] = {
|
||||
"tag_name": "v0.2.0",
|
||||
"assets": [{"name": "phase0-notes.txt"}],
|
||||
}
|
||||
code, out, err = self.run_smoke("v0.2.0")
|
||||
self.assertEqual(code, 0, err or out)
|
||||
for marker in ("PASS [1/4]", "PASS [2/4]", "PASS [3/4]",
|
||||
"PASS [4/4]"):
|
||||
self.assertIn(marker, out)
|
||||
self.assertIn("G-021 confirmed: no .pyz asset", out,
|
||||
"the v0.2.0 run carries the G-021 forge-side "
|
||||
"confirmation")
|
||||
self.assertIn("phase0-notes.txt", out, "asset names print as a fact")
|
||||
self.assertIn(self.fixtures.child_digest, out,
|
||||
"the amd64 child digest prints (drift check)")
|
||||
# the manifest GET carries the full Accept set
|
||||
manifest_gets = [r for r in self.hub_requests
|
||||
if "/manifests/3.11-slim" in r["path"]]
|
||||
self.assertEqual(len(manifest_gets), 1)
|
||||
self.assertEqual(manifest_gets[0]["accept"],
|
||||
publish_image.ACCEPT_MANIFESTS)
|
||||
# the release GET uses the gitea API token auth form
|
||||
release_gets = self._release_gets()
|
||||
self.assertEqual(len(release_gets), 1)
|
||||
self.assertEqual(release_gets[0]["authorization"],
|
||||
"token testtoken123")
|
||||
# READ-ONLY PROOF: zero POST/PUT recorded at either stub
|
||||
gitea_writes, hub_writes = self._post_put()
|
||||
self.assertEqual(gitea_writes, [], "no gitea write of any kind")
|
||||
self.assertEqual(hub_writes, [], "no hub write of any kind")
|
||||
self.assertNotIn("IMAGE_PENDING", out)
|
||||
|
||||
# 13 ----------------------------------------------------------------
|
||||
def test_smoke_v0_2_0_no_pyz_assertion(self):
|
||||
pyz_release = {"tag_name": "v0.2.0",
|
||||
"assets": [{"name": "coreci-broker-v0.2.0.pyz"}]}
|
||||
self.gitea_state["releases"]["v0.2.0"] = pyz_release
|
||||
self.gitea_state["releases"]["v0.2.2"] = pyz_release
|
||||
# v0.2.0 + a .pyz asset: the G-021 hard assertion fails the smoke
|
||||
code, out, err = self.run_smoke("v0.2.0")
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("FAILED [3/4]", out)
|
||||
self.assertIn("G-021", out, "the failure line names G-021")
|
||||
self.assertIn(".pyz", out)
|
||||
self.assertEqual(self.hub_requests, [],
|
||||
"a FAILED check stops the smoke: the hub "
|
||||
"resolve (step 4) never runs")
|
||||
# the SAME release under a different tag: the assertion binds
|
||||
# v0.2.0 only -> all four checks pass
|
||||
code, out, err = self.run_smoke("v0.2.2")
|
||||
self.assertEqual(code, 0, err or out)
|
||||
for marker in ("PASS [3/4]", "PASS [4/4]"):
|
||||
self.assertIn(marker, out)
|
||||
self.assertNotIn("G-021 violation", out)
|
||||
gitea_writes, hub_writes = self._post_put()
|
||||
self.assertEqual(gitea_writes + hub_writes, [],
|
||||
"still read-only across both runs")
|
||||
|
||||
# 14 ----------------------------------------------------------------
|
||||
def test_smoke_trap_refused(self):
|
||||
self.gitea_state["token_mode"] = "empty"
|
||||
self.gitea_state["releases"]["v0.2.3"] = {
|
||||
"tag_name": "v0.2.3", "assets": [],
|
||||
}
|
||||
code, out, err = self.run_smoke("v0.2.3")
|
||||
self.assertEqual(code, 1, "the trap is a loud remote failure")
|
||||
self.assertIn("FAILED [2/4]", out)
|
||||
self.assertIn("anonymous-200 trap refused", out)
|
||||
self.assertIn("PASS [1/4]", out, "check 1 ran (order 1 -> 4)")
|
||||
self.assertNotIn("PASS [3/4]", out, "checks after the trap never run")
|
||||
self.assertNotIn("PASS [4/4]", out)
|
||||
self.assertEqual(self._release_gets(), [],
|
||||
"the release GET (step 3) is skipped")
|
||||
self.assertEqual(self.hub_requests, [],
|
||||
"zero hub requests: the index resolve (step 4) "
|
||||
"is skipped")
|
||||
|
||||
# 15 ----------------------------------------------------------------
|
||||
def test_smoke_missing_token_local(self):
|
||||
empty_root = os.path.join(self.temp.name, "emptyroot")
|
||||
os.makedirs(empty_root)
|
||||
code, out, err = self.run_smoke("v0.2.3", env_root=empty_root)
|
||||
self.assertEqual(code, 2, "local precondition -> exit 2")
|
||||
self.assertIn("ERROR", err)
|
||||
self.assertIn("GITEA_TOKEN", err)
|
||||
self.assertEqual(self.gitea_state["requests"], [],
|
||||
"zero gitea requests — the precondition fires "
|
||||
"before anything remote")
|
||||
self.assertEqual(self.hub_requests, [], "zero hub requests")
|
||||
self.assertNotIn("IMAGE_PENDING", out, "no marker for local errors")
|
||||
self.assertNotIn("PASS", out, "no smoke check ever ran")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,511 @@
|
||||
"""Stub-gitea release-attacher tests (REQ-22, RESEARCH §2.4).
|
||||
|
||||
Drives scripts/release.py against an in-process stub gitea
|
||||
(``_StubGiteaHandler`` on ``127.0.0.1:0`` — the v0.1 ``_StubOwuiHandler``
|
||||
precedent; fake token, no live-forge writes) plus the sanctioned
|
||||
secrets loader scripts/_env_secrets.py: create-or-get in both orders,
|
||||
byte-exact multipart shape, sha256-from-file discipline (G-017),
|
||||
idempotent re-attach, bounded-retry exhaustion -> RELEASE_PENDING marker,
|
||||
and file-only token resolution asserted before the first HTTP call.
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
import urllib.parse
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
REPO = pathlib.Path(__file__).resolve().parent.parent
|
||||
RELEASE_PATH = REPO / "scripts" / "release.py"
|
||||
ENV_SECRETS_PATH = REPO / "scripts" / "_env_secrets.py"
|
||||
|
||||
sys.path.insert(0, str(REPO))
|
||||
|
||||
import coreci_broker # repo root on sys.path (parser-parity assertion)
|
||||
|
||||
|
||||
def _load_module(name, path):
|
||||
spec = importlib.util.spec_from_file_location(name, str(path))
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
release = _load_module("release", RELEASE_PATH)
|
||||
env_secrets = _load_module("env_secrets", ENV_SECRETS_PATH)
|
||||
|
||||
OWNER = "coreci"
|
||||
REPO_NAME = "broker"
|
||||
TAG = "v0.2.1"
|
||||
TOKEN = "testtoken123"
|
||||
ARTIFACT_NAME = "coreci-broker-v0.2.1.pyz"
|
||||
|
||||
_RELEASE_BY_TAG_RE = re.compile(
|
||||
r"^/api/v1/repos/([^/]+)/([^/]+)/releases/tags/([^/]+)$")
|
||||
_RELEASES_RE = re.compile(r"^/api/v1/repos/([^/]+)/([^/]+)/releases$")
|
||||
_ASSETS_RE = re.compile(r"^/api/v1/repos/([^/]+)/([^/]+)/releases/(\d+)/assets$")
|
||||
|
||||
|
||||
def _header_params(value):
|
||||
"""'a; b=\"c\"' -> {'b': 'c'} — Content-Type/Disposition params."""
|
||||
params = {}
|
||||
for item in value.split(";")[1:]:
|
||||
key, _, val = item.strip().partition("=")
|
||||
if key:
|
||||
params[key.strip()] = val.strip().strip('"')
|
||||
return params
|
||||
|
||||
|
||||
def _split_multipart(content_type, body):
|
||||
"""Minimal multipart parser: boundary from the Content-Type header."""
|
||||
boundary = _header_params(content_type).get("boundary", "")
|
||||
parts = []
|
||||
if boundary:
|
||||
for element in body.split(("--" + boundary).encode("utf-8"))[1:]:
|
||||
if element.startswith(b"--"):
|
||||
continue # closing delimiter
|
||||
if element.startswith(b"\r\n"):
|
||||
element = element[2:]
|
||||
header_blob, separator, content = element.partition(b"\r\n\r\n")
|
||||
if not separator:
|
||||
continue
|
||||
if content.endswith(b"\r\n"):
|
||||
content = content[:-2] # framing CRLF before the next delimiter
|
||||
part_headers = {}
|
||||
for line in header_blob.decode("utf-8", "replace").split("\r\n"):
|
||||
name, _, value = line.partition(":")
|
||||
if name:
|
||||
part_headers[name.strip().lower()] = value.strip()
|
||||
parts.append((part_headers, content))
|
||||
return boundary, parts
|
||||
|
||||
|
||||
class _StubGiteaHandler(BaseHTTPRequestHandler):
|
||||
"""Scriptable gitea stub (RESEARCH §2.1 routes); records everything."""
|
||||
|
||||
state = None # scriptable state dict (class attr, bound per server)
|
||||
requests = None # every request: method, path, headers, body (+extras)
|
||||
|
||||
def log_message(self, format, *args):
|
||||
pass
|
||||
|
||||
def _record(self):
|
||||
length = int(self.headers.get("Content-Length") or 0)
|
||||
body = self.rfile.read(length) if length else b""
|
||||
record = {
|
||||
"method": self.command,
|
||||
"path": self.path,
|
||||
"headers": {key.lower(): value
|
||||
for key, value in self.headers.items()},
|
||||
"body": body,
|
||||
}
|
||||
self.requests.append(record)
|
||||
return record
|
||||
|
||||
def _send_json(self, status, payload):
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def do_GET(self):
|
||||
self._record()
|
||||
parsed = urllib.parse.urlsplit(self.path)
|
||||
if _RELEASE_BY_TAG_RE.match(parsed.path):
|
||||
if self.state.get("release_exists"):
|
||||
self._send_json(200, {"id": self.state.get("release_id", 970)})
|
||||
else:
|
||||
self._send_json(404, {"message": "Release not found"})
|
||||
return
|
||||
if _ASSETS_RE.match(parsed.path):
|
||||
self._send_json(200, list(self.state.get("assets", [])))
|
||||
return
|
||||
self._send_json(404, {"message": "Not Found"})
|
||||
|
||||
def do_POST(self):
|
||||
record = self._record()
|
||||
parsed = urllib.parse.urlsplit(self.path)
|
||||
if _RELEASES_RE.match(parsed.path):
|
||||
self.state.setdefault("create_calls", []).append(record)
|
||||
status = self.state.get("create_status", 201)
|
||||
if status >= 400:
|
||||
self._send_json(
|
||||
status,
|
||||
{"message": self.state.get("create_error", "create failed")})
|
||||
else:
|
||||
self._send_json(status, {"id": self.state.get("release_id", 970)})
|
||||
return
|
||||
if _ASSETS_RE.match(parsed.path):
|
||||
self.state.setdefault("attach_calls", []).append(record)
|
||||
status = self.state.get("attach_status", 201)
|
||||
if status >= 400:
|
||||
self._send_json(
|
||||
status,
|
||||
{"message": self.state.get("attach_error", "attach failed")})
|
||||
return
|
||||
self._describe_attachment(record, parsed)
|
||||
self._send_json(201, {
|
||||
"id": 2,
|
||||
"name": record["attach"]["query_name"],
|
||||
"size": record["attach"]["size"],
|
||||
})
|
||||
return
|
||||
self._send_json(404, {"message": "Not Found"})
|
||||
|
||||
def _describe_attachment(self, record, parsed):
|
||||
boundary, parts = _split_multipart(
|
||||
record["headers"].get("content-type", ""), record["body"])
|
||||
field_name = filename = part_type = None
|
||||
content = b""
|
||||
if len(parts) == 1:
|
||||
part_headers, content = parts[0]
|
||||
disposition = _header_params(
|
||||
part_headers.get("content-disposition", ""))
|
||||
field_name = disposition.get("name")
|
||||
filename = disposition.get("filename")
|
||||
part_type = part_headers.get("content-type")
|
||||
expected = self.state.get("expected_asset_bytes")
|
||||
record["attach"] = {
|
||||
"boundary": boundary,
|
||||
"field_name": field_name,
|
||||
"filename": filename,
|
||||
"part_content_type": part_type,
|
||||
"query_name":
|
||||
(urllib.parse.parse_qs(parsed.query).get("name") or [None])[0],
|
||||
"content": content,
|
||||
"size": len(content),
|
||||
"content_matches_expected":
|
||||
None if expected is None else content == expected,
|
||||
}
|
||||
|
||||
|
||||
class StubGiteaServer:
|
||||
"""Bound stub server (v0.1 StubOwuiServer pattern)."""
|
||||
|
||||
def __init__(self, state):
|
||||
self.state = state
|
||||
self.requests = []
|
||||
handler = type("BoundStubHandler", (_StubGiteaHandler,), {
|
||||
"state": state,
|
||||
"requests": self.requests,
|
||||
})
|
||||
self.server = ThreadingHTTPServer(("127.0.0.1", 0), handler)
|
||||
threading.Thread(target=self.server.serve_forever, daemon=True).start()
|
||||
|
||||
@property
|
||||
def base_url(self):
|
||||
return "http://127.0.0.1:%d" % self.server.server_address[1]
|
||||
|
||||
def close(self):
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
|
||||
|
||||
class ReleaseUploadTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory(prefix="coreci-release-test-")
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = tmp.name
|
||||
self.artifact_path = os.path.join(self.tmp, ARTIFACT_NAME)
|
||||
# Includes every byte value (CR/LF inside) — byte-exact multipart
|
||||
# transport must survive content that looks like framing.
|
||||
self.artifact_bytes = (
|
||||
b"#!/usr/bin/env python3\nstub-zipapp-payload:" + bytes(range(256)) * 2)
|
||||
with open(self.artifact_path, "wb") as handle:
|
||||
handle.write(self.artifact_bytes)
|
||||
self.env_root = os.path.join(self.tmp, "env-root")
|
||||
os.makedirs(self.env_root)
|
||||
secrets_fd = os.open(
|
||||
os.path.join(self.env_root, ".env.secrets"),
|
||||
os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(secrets_fd, "w", encoding="utf-8") as handle:
|
||||
handle.write("GITEA_TOKEN={}\n".format(TOKEN))
|
||||
self.state = {
|
||||
"release_exists": True,
|
||||
"release_id": 970,
|
||||
"assets": [],
|
||||
"create_status": 201,
|
||||
"attach_status": 201,
|
||||
"expected_asset_bytes": self.artifact_bytes,
|
||||
}
|
||||
self.stub = StubGiteaServer(self.state)
|
||||
self.addCleanup(self.stub.close)
|
||||
|
||||
def _argv(self, **overrides):
|
||||
argv = [
|
||||
TAG, self.artifact_path,
|
||||
"--base-url", self.stub.base_url,
|
||||
"--owner", OWNER, "--repo", REPO_NAME,
|
||||
"--env-root", self.env_root,
|
||||
]
|
||||
for key, value in overrides.items():
|
||||
argv.extend(["--" + key.replace("_", "-"), value])
|
||||
return argv
|
||||
|
||||
def _run(self, **overrides):
|
||||
stdout, stderr = io.StringIO(), io.StringIO()
|
||||
with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr):
|
||||
code = release.main(self._argv(**overrides))
|
||||
return code, stdout.getvalue(), stderr.getvalue()
|
||||
|
||||
def _paths(self, method):
|
||||
return [record["path"] for record in self.stub.requests
|
||||
if record["method"] == method]
|
||||
|
||||
def test_existing_release_attaches_exact_multipart_shape(self):
|
||||
code, out, err = self._run()
|
||||
self.assertEqual(0, code)
|
||||
self.assertEqual(
|
||||
["/api/v1/repos/{}/{}/releases/tags/{}".format(OWNER, REPO_NAME, TAG),
|
||||
"/api/v1/repos/{}/{}/releases/970/assets".format(OWNER, REPO_NAME)],
|
||||
self._paths("GET"))
|
||||
self.assertEqual(
|
||||
["/api/v1/repos/{}/{}/releases/970/assets?name={}".format(
|
||||
OWNER, REPO_NAME, ARTIFACT_NAME)],
|
||||
self._paths("POST"))
|
||||
self.assertEqual([], self.state.get("create_calls", []))
|
||||
attach = self.stub.requests[-1]
|
||||
shape = attach["attach"]
|
||||
self.assertEqual("attachment", shape["field_name"])
|
||||
self.assertEqual(ARTIFACT_NAME, shape["filename"])
|
||||
self.assertEqual(ARTIFACT_NAME, shape["query_name"])
|
||||
self.assertEqual("application/octet-stream", shape["part_content_type"])
|
||||
self.assertEqual(self.artifact_bytes, shape["content"])
|
||||
self.assertTrue(shape["content_matches_expected"])
|
||||
self.assertIn("multipart/form-data; boundary=",
|
||||
attach["headers"]["content-type"])
|
||||
self.assertTrue(self.stub.requests)
|
||||
for record in self.stub.requests:
|
||||
self.assertEqual("token " + TOKEN, record["headers"]["authorization"])
|
||||
|
||||
def test_missing_release_creates_with_notes_then_attaches(self):
|
||||
self.state["release_exists"] = False
|
||||
code, out, err = self._run()
|
||||
self.assertEqual(0, code)
|
||||
create_calls = self.state.get("create_calls", [])
|
||||
self.assertEqual(1, len(create_calls))
|
||||
create = create_calls[0]
|
||||
self.assertEqual(
|
||||
"/api/v1/repos/{}/{}/releases".format(OWNER, REPO_NAME), create["path"])
|
||||
self.assertEqual("application/json", create["headers"]["content-type"])
|
||||
payload = json.loads(create["body"])
|
||||
self.assertEqual(TAG, payload["tag_name"])
|
||||
self.assertEqual(TAG + ": coreci-broker release", payload["name"])
|
||||
self.assertIs(False, payload["draft"])
|
||||
self.assertIs(False, payload["prerelease"])
|
||||
for section in ("## Summary", "## Requirements", "## Verification",
|
||||
"## CLI install + usage", "## Docker usage"):
|
||||
self.assertIn(section, payload["body"])
|
||||
self.assertEqual(1, len(self.state.get("attach_calls", [])))
|
||||
|
||||
def test_matching_asset_is_idempotent_skip(self):
|
||||
self.state["assets"] = [{
|
||||
"id": 1, "name": ARTIFACT_NAME, "size": len(self.artifact_bytes),
|
||||
}]
|
||||
code, out, err = self._run()
|
||||
self.assertEqual(0, code)
|
||||
self.assertEqual([], self._paths("POST"))
|
||||
self.assertEqual(
|
||||
["/api/v1/repos/{}/{}/releases/tags/{}".format(OWNER, REPO_NAME, TAG),
|
||||
"/api/v1/repos/{}/{}/releases/970/assets".format(OWNER, REPO_NAME)],
|
||||
self._paths("GET"))
|
||||
self.assertEqual([], self.state.get("attach_calls", []))
|
||||
self.assertEqual([], self.state.get("create_calls", []))
|
||||
|
||||
def test_same_name_but_wrong_size_reattaches(self):
|
||||
self.state["assets"] = [{"id": 1, "name": ARTIFACT_NAME, "size": 1}]
|
||||
code, out, err = self._run()
|
||||
self.assertEqual(0, code)
|
||||
self.assertEqual(1, len(self.state.get("attach_calls", [])))
|
||||
|
||||
def test_attach_500_exhausts_retries_and_marks_pending(self):
|
||||
self.state["attach_status"] = 500
|
||||
completed = subprocess.run(
|
||||
[sys.executable, str(RELEASE_PATH)] + self._argv(),
|
||||
capture_output=True, text=True, timeout=120,
|
||||
)
|
||||
self.assertEqual(1, completed.returncode)
|
||||
self.assertIn(
|
||||
"RELEASE_PENDING {} {}".format(TAG, self.artifact_path),
|
||||
completed.stdout)
|
||||
self.assertEqual(3, len(self.state.get("attach_calls", [])))
|
||||
self.assertEqual([], self.state.get("create_calls", []))
|
||||
self.assertIn("500", completed.stderr)
|
||||
self.assertNotIn(TOKEN, completed.stdout)
|
||||
self.assertNotIn(TOKEN, completed.stderr)
|
||||
|
||||
def test_permanent_attach_error_reports_gitea_body(self):
|
||||
self.state["attach_status"] = 400
|
||||
self.state["attach_error"] = "attachment too large"
|
||||
code, out, err = self._run()
|
||||
self.assertEqual(1, code)
|
||||
self.assertEqual(1, len(self.state.get("attach_calls", []))) # 4xx: no retry
|
||||
self.assertIn("attachment too large", err)
|
||||
self.assertIn(
|
||||
"RELEASE_PENDING {} {}".format(TAG, self.artifact_path), out)
|
||||
|
||||
def test_sha256_comes_from_file_and_cli_has_no_sha_argument(self):
|
||||
before = len(self.stub.requests)
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
with self.assertRaises(SystemExit) as raised:
|
||||
release.main(self._argv() + ["--sha", "deadbeef"])
|
||||
self.assertEqual(2, raised.exception.code)
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
with self.assertRaises(SystemExit) as raised:
|
||||
release.main(self._argv() + ["0deadbeef"])
|
||||
self.assertEqual(2, raised.exception.code)
|
||||
self.assertEqual(before, len(self.stub.requests)) # refused pre-HTTP
|
||||
self.state["release_exists"] = False # force create so notes hit the wire
|
||||
code, out, err = self._run()
|
||||
self.assertEqual(0, code)
|
||||
notes = json.loads(self.state["create_calls"][0]["body"])["body"]
|
||||
expected = hashlib.sha256(self.artifact_bytes).hexdigest()
|
||||
self.assertIn("sha256:" + expected, notes)
|
||||
self.assertNotIn("deadbeef", notes)
|
||||
|
||||
def test_token_resolves_from_env_file_on_every_request(self):
|
||||
code, out, err = self._run()
|
||||
self.assertEqual(0, code)
|
||||
self.assertTrue(self.stub.requests)
|
||||
for record in self.stub.requests:
|
||||
self.assertEqual("token " + TOKEN, record["headers"]["authorization"])
|
||||
# Structural discipline: the loader + attacher never touch the
|
||||
# process environment, and the token value is never printed.
|
||||
for path in (ENV_SECRETS_PATH, RELEASE_PATH):
|
||||
source = path.read_text(encoding="utf-8")
|
||||
self.assertNotIn("os.environ", source)
|
||||
self.assertNotIn("os.getenv", source)
|
||||
self.assertNotIn(TOKEN, out)
|
||||
self.assertNotIn(TOKEN, err)
|
||||
|
||||
def test_missing_or_empty_token_fails_before_any_http(self):
|
||||
empty_root = os.path.join(self.tmp, "empty-env-root")
|
||||
os.makedirs(empty_root)
|
||||
code, out, err = self._run(env_root=empty_root)
|
||||
self.assertEqual(2, code)
|
||||
self.assertEqual([], self.stub.requests) # no HTTP attempted
|
||||
self.assertIn("GITEA_TOKEN", err)
|
||||
self.assertNotIn("RELEASE_PENDING", out)
|
||||
with open(os.path.join(empty_root, ".env.secrets"), "w",
|
||||
encoding="utf-8") as handle:
|
||||
handle.write("GITEA_TOKEN=\n")
|
||||
code, out, err = self._run(env_root=empty_root)
|
||||
self.assertEqual(2, code)
|
||||
self.assertEqual([], self.stub.requests)
|
||||
self.assertIn("GITEA_TOKEN", err)
|
||||
|
||||
def test_notes_sections_sha_line_and_docker_usage(self):
|
||||
self.state["release_exists"] = False
|
||||
code, out, err = self._run()
|
||||
self.assertEqual(0, code)
|
||||
notes = json.loads(self.state["create_calls"][0]["body"])["body"]
|
||||
for section in ("## Summary", "## Requirements", "## Verification",
|
||||
"## CLI install + usage", "## Docker usage"):
|
||||
self.assertIn(section, notes)
|
||||
expected = hashlib.sha256(self.artifact_bytes).hexdigest()
|
||||
self.assertIn("sha256:" + expected, notes)
|
||||
self.assertIn("chmod +x " + ARTIFACT_NAME, notes)
|
||||
self.assertIn("--version", notes)
|
||||
self.assertIn("README", notes)
|
||||
self.assertIn("BROKER_ENV_PATH", notes)
|
||||
self.assertIn("BROKER_ADMIN_PASS", notes)
|
||||
self.assertIn("docker pull ", notes)
|
||||
self.assertIn("/{}/{}:{}".format(OWNER, REPO_NAME, TAG), notes)
|
||||
self.assertIn("-v broker-state:/state", notes)
|
||||
|
||||
|
||||
class EnvSecretsLoaderTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory(prefix="coreci-envsecrets-")
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.root = tmp.name
|
||||
|
||||
def _write(self, relative, content):
|
||||
path = os.path.join(self.root, relative)
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "w", encoding="utf-8") as handle:
|
||||
handle.write(content)
|
||||
return path
|
||||
|
||||
def test_parse_semantics_mirror_broker_parser(self):
|
||||
fixture = (
|
||||
"# comment\n"
|
||||
"\n"
|
||||
"GITEA_TOKEN='quoted-single'\n"
|
||||
"OTHER_TOKEN=\"quoted double\"\n"
|
||||
"SPACED = spaced value \n"
|
||||
"PLAIN=unquoted\n"
|
||||
"EMPTY=\n"
|
||||
"ONE_QUOTE=\"\n"
|
||||
"NO_EQUALS_SIGN\n"
|
||||
"=\n"
|
||||
)
|
||||
path = self._write(".env.secrets", fixture)
|
||||
self.assertEqual(
|
||||
coreci_broker._parse_env_file(path),
|
||||
env_secrets._parse_env_file(path))
|
||||
values = env_secrets.load(self.root)
|
||||
self.assertEqual("quoted-single", values["GITEA_TOKEN"])
|
||||
self.assertEqual("quoted double", values["OTHER_TOKEN"])
|
||||
self.assertEqual("spaced value", values["SPACED"])
|
||||
self.assertEqual("unquoted", values["PLAIN"])
|
||||
self.assertEqual("", values["EMPTY"])
|
||||
self.assertEqual('"', values["ONE_QUOTE"])
|
||||
self.assertNotIn("NO_EQUALS_SIGN", values)
|
||||
|
||||
def test_later_files_win_across_locations_and_patterns(self):
|
||||
ladder = (
|
||||
(".env", "GITEA_TOKEN=step1\n"),
|
||||
(".env.secrets", "GITEA_TOKEN=step2\n"),
|
||||
(".env.local", "GITEA_TOKEN=step3\n"), # root .env.* glob
|
||||
(os.path.join(".ciagent", ".env"), "GITEA_TOKEN=step4\n"),
|
||||
(os.path.join(".ciagent", ".env.secrets"), "GITEA_TOKEN=step5\n"),
|
||||
(os.path.join(".ciagent", ".env.staging"), "GITEA_TOKEN=step6\n"),
|
||||
)
|
||||
for relative, content in ladder:
|
||||
self._write(relative, content)
|
||||
self.assertEqual(
|
||||
content.partition("=")[2].strip(),
|
||||
env_secrets.load(self.root)["GITEA_TOKEN"])
|
||||
|
||||
def test_missing_or_empty_token_raises_clear_error(self):
|
||||
with self.assertRaises(env_secrets.SecretsError) as caught:
|
||||
env_secrets.get_token(root=self.root)
|
||||
self.assertIn("GITEA_TOKEN", str(caught.exception))
|
||||
self._write(os.path.join(".ciagent", ".env.secrets"), "GITEA_TOKEN=\n")
|
||||
with self.assertRaises(env_secrets.SecretsError):
|
||||
env_secrets.get_token(root=self.root)
|
||||
self._write(os.path.join(".ciagent", ".env.secrets"),
|
||||
'GITEA_TOKEN=" "\n') # whitespace-only is empty too
|
||||
with self.assertRaises(env_secrets.SecretsError):
|
||||
env_secrets.get_token(root=self.root)
|
||||
self._write(os.path.join(".ciagent", ".env.secrets"),
|
||||
"GITEA_TOKEN='tok'\n")
|
||||
self.assertEqual("tok", env_secrets.get_token(root=self.root))
|
||||
|
||||
def test_custom_variable_name(self):
|
||||
self._write(".env", "OTHER_TOKEN=tok\n")
|
||||
self.assertEqual("tok", env_secrets.get_token("OTHER_TOKEN", root=self.root))
|
||||
with self.assertRaises(env_secrets.SecretsError) as caught:
|
||||
env_secrets.get_token("GITEA_TOKEN", root=self.root)
|
||||
self.assertIn("GITEA_TOKEN", str(caught.exception))
|
||||
|
||||
def test_default_root_is_repo_root(self):
|
||||
self.assertEqual(
|
||||
os.path.realpath(str(REPO)), os.path.realpath(env_secrets.default_root()))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user