approval-queue runbook — executor rail as approval-request queue (REQ-27, P04-04)

The executor rail (LXC 116 coreci-executor + LXC 117 coreci-broker) stays RUNNING at all times; its job is to park, route, decide, mirror and report approval requests. It never executes anything (README "The NEVER-EXECUTES guarantee"). This repo is frozen: the change that repurposes it IS the PR holding this file + allowlist.yaml v5.0 + routing.json.

Queue model

Request: a credential-free envelope from an acting employee's pipeline (POST /submit_request, EXECUTOR_API_KEY auth). Fields: id, requestor (Gitea username = seat), system_id (optional — a systems-ownership row id), title, class (L1|L2|L3), justification (+ legacy-compat kind/targets/content for the UI display).
Routing: resolved at intake from /config/routing.json (GENERATED from coreci/company org records — never hand-edited here): lane chairman-direct (C-09: acting C-suite + CEO + coreci-sre → Chairman decides), lane manager:owner (requestor's manager one hop + the touched system's owner; both when both, deduped, manager listed first), lane skip-path (unresolvable chain → approver CEO, queue Chairman — sanctioned skip-path 2). No ownership-table row for a submitted system_id → request REFUSED at intake (fail-closed). Full trail (rule-by-rule, with timestamps) is stored on the request and mirrored to the Gitea issue.
Approval: the resolved approver decides at the executor UI (/ui/request/{rid}/approve|deny, passcode session). Approve/deny is a DECISION-RECORD only — the legacy approve-triggers-run path is disabled in approval-queue mode (0.4.0 decision-only server build; see "Apply").
Result: GET /request_status?request_id=… returns {status: open|routed|decided, decision, decided_by, decided_at, …} — the requesting pipeline polls this, never the queue's internals.
Audit: every event is appended to logs/audit-YYYY-MM-DD.jsonl (append-only, git-durable) with the vocabulary request.created, route.resolved, decision.posted, mirror.updated. /ui/audit renders recent rows for humans.

Issue mirror (coreci/meetings)

The queue posts one issue per request into coreci/meetings — the same lane the meeting machinery's approval-requests already live in — labels approval-request + approval:L2|L3 + dept:<dept>; body carries the machine-notes block (requestor: / system: / owner: / lane: / status:). Decisions are posted as comments by the EXECUTOR APP ACCOUNT (coreci-executor, its own PAT) and the body flips to status: decided (…). Idempotency: the mirror dedups on the request id token in the issue body (ar-…); a mirror-retry can never double-post a request or a decision. Full contract: coreci/sre scripts/p04-executor/mirror-contract.md.

Apply the change (operator-gated, on-host — NO pipeline does this)

The rail hot-reads this repo's config ONLY through its bind mount — there is no redeploy pipeline watching this repo; nothing applies until the operator runs the steps below (read-only from SRE's side: the sre repo only AUTHORS the patch and opens the PR).

PR merged into executor-config:main (merge = the operator's confirmation; the merge itself changes the frozen repo's files).
On LXC 116, in the executor-config working copy mounted at /config (and the container's own clone):
```
git fetch --all && git pull --ff-only
```
- allowlist.yaml v5.0 takes effect on the NEXT classification pass (the legacy classifier hot-reads per call — current_allowlist() opens the file every time: NO restart needed), so the last auto-run grants die the moment main updates.
Server build to 0.4.0 (decision-only / never-executes). IMPORTANT recon fact: the LIVE server (0.3.0) is NOT built from this repo's stale app/main.py (repo app = v0.1.1 classifier module only). The 0.4.0 build must be produced from the on-host source of truth, wire it: read ALLOWLIST_PATH=/config/allowlist.yaml (repo-mounted) and EXECUTOR_ROUTING_TABLE=/config/routing.json, honor mode: approval_queue_only (refuse to boot without it or without never_execute: true — fail-closed), run the routing + mirror half from this runbook + the mirror contract. Then:
```
podman compose up -d --build    # or docker compose
curl -s http://127.0.0.1:8765/health
-> expect {"status":"ok","version":"0.4.0","mode":"approval_queue_only",...}
```
LXC 117 (coreci/broker) is UNTOUCHED by this repurpose (broker repo frozen; its change channel is a separate one-way PR when needed).
Post-apply asserts (all must be GREEN): GET /health version 0.4.0; GET /request_status?request_id=probe → 404-with-id (queue reachable, no phantom request); executor identity can create/label issues in coreci/meetings; executor identity CANNOT write any other repo (negative assert — try a write via the API and expect 403); audit JSONL gains the 4-event vocabulary on a test request; mirror issue appears with the machine-notes block; decision on the test request mirrors correctly and body flips status.

Rollback

Revert the merged PR (one-way channel: a revert PR), keep the 0.4.0 server: it boots fail-closed WITHOUT mode: approval_queue_only — with the v4.1 allowlist restored the legacy classification semantics return, and nothing auto-runs (all grants were revoked in this repurpose; that revocation stands until org records say otherwise via new org-record PRs).

Escalation

Executor unreachable → notify-ladder (bao KV) as usual; queue outages NEVER auto-approve: pipelines wait on request_status, and an outage shows as routed/open with a heartbeat gap in the audit trail.

5.7 KiB Raw Permalink Blame History