orca/.ciagent/GRILL_v0.13.md

GRILL v0.13: Production Hardening Round 2 + UAT Plan

Status: complete (2026-08-07). Red-team review of PLAN_v0.13 across 9 axes. Verdict: CONDITIONAL PROCEED — the plan is fundamentally sound and evidence-accurate, but 6 binding conditions (C-44..C-49) gate specific phases. One governance finding (v0.12 completeness fraud) is acknowledged and resolved via binding decision.

Reviewer: CIAgent griller (adversarial, evidence-based). Confidence: 0.82 overall.

Methodology

Every forcing question was checked against the actual codebase, not just the plan's claims. All 8 "critical" findings (F26-F33) and a sample of high/medium findings were independently verified:

• F26 (scheduler dead code): internal/scheduler is never imported; job run uses exec.CommandContext via engine.Executor.runOne (internal/engine/executor.go:163); the --target dispatch path uses /bin/true as a placeholder command (internal/cli/job.go:96).
• F27 (jobspec schedule/timeout dropped): no case "schedule": or case "timeout": in the top-level switch (internal/jobspec/ markdown.go:484-557); both fall to default: cur = secNone.
• F28 (verify-reqs bypass): regex reqRowRe matches only capitalized Complete|Pending (cmd/verify-reqs/main.go:21); lowercase pending rows are invisible.
• F29 (logs RCE): fmt.Sprintf("journalctl -u %q ...", unitPattern, ...) at internal/cli/logs.go:274 — backtick injection via SSH fanout confirmed.
• F30 (pprof loopback bypass): isLoopback(":6060") — empty host not treated as bind-all; phantom --pprof-allow-public references at internal/daemon/pprof.go:37,42,43.
• F31 (tar-slip): strings.HasPrefix(name, "..") at internal/backup/backup.go:302 — bypassable via a/../../etc/passwd.
• F32 (WebAuthn unauthenticated registration): no auth check in register path (internal/webauthn/connector.go).
• F48 (acl.Check never called): zero imports of internal/acl anywhere in the codebase; no references in internal/daemon/.
• F49 (acl.json mode 0644): writeAtomicFile(path, data, 0o644) at internal/cli/acl.go:152.
• F54 (auth init-idp stub): prints "Dex bootstrap planned for RP ID: ..." and returns nil (internal/cli/auth.go:147-153).
• F42 (go toolchain 1.25.0): go.mod:3 confirms go 1.25.0.

The plan's research is honest. This is rare and commendable.

Governance finding (G-255): v0.12 completeness fraud

Evidence: ROADMAP.md:403 marks v0.12: Security Hardening — COMPLETE. REQUIREMENTS.md rows REQ-130..148 (all 19 v0.12 REQs) are status pending (lowercase). verify-reqs reports "118 requirements consistent with roadmap" because its regex (cmd/verify-reqs/main.go: 21) matches only capitalized Complete|Pending — lowercase pending is invisible. This is F28, but the governance consequence is unstated in the plan: v0.12's headline features (ACL enforcement REQ-145, seal/unseal CLI REQ-147, auth init-idp REQ-144, WebAuthn registration auth REQ-148) were never wired. v0.13 P04/P05/P06 completes this unfinished v0.12 work.

Verdict: This is a documentation artifact, not a code fraud. The v0.12 code (ACL library, seal library, WebAuthn connector library) was shipped but not operationally wired — which is exactly what v0.13 fixes. Revoking v0.12's COMPLETE status would destabilize the milestone history without changing any code. The pragmatic resolution: P13 marks REQ-130..148 AND REQ-149..163 as Complete, v0.12 stays COMPLETE retroactively, and the gap is acknowledged here.

Binding decision G-255: Proceed as planned. P13 MUST mark both v0.12 REQs (REQ-130..148) and v0.13 REQs (REQ-149..163) as Complete. v0.12's COMPLETE status is retained retroactively. The verify-reqs regex fix (C-43, P11) makes this consistency enforceable going forward. Confidence: 0.90.

Axis 1 — Feasibility

Verdict: PASS | Confidence: 0.82

P03 (scheduler wiring) — the riskiest phase

The scheduler (internal/scheduler/scheduler.go:74 Schedule()) is a pure function: takes []NodeInfo + WorkloadRequest, returns []Placement. It is well-tested (23 test functions). The emitter (internal/emitter/systemd.go:80 Render()) renders systemd units. The sshpush transport (internal/sshpush/fanout.go:64 WriteAll()) pushes files to peers. All three components exist and are tested in isolation — P03 wires them together.

The local fallback (T8: "no remote nodes registered → single-node dev mode") is the correct safety net. The current exec.CommandContext path is preserved when len(nodes) == 0. This is backward-compatible.

Risk: The --target dispatch path (internal/cli/job.go:67-103) currently uses a placeholder /bin/true command and a JSON marshal that drops the full spec. P03 must replace this entirely. The dispatcher (engine.NewDispatcher) exists but emits a placeholder spec. P03 T5 says "replace local exec.CommandContext path with: evaluate constraints/capacity/affinity → render systemd units → SSH-push to target" — this is a significant rewrite of job run, not a wiring task. The plan's phase title ("scheduler wiring") understates the work: it's a behavioral rewrite of the core command.

Verdict: Feasible, but P03 is under-estimated as "wiring." It is the most complex phase and deserves the longest schedule. C-39 (local fallback) is the correct mitigation. The systemd-analyze verify gate (T9) is a good safety check. No blocking conditions beyond C-39 and C-44 (test coverage).

Local fallback safety

The fallback is safe: len(nodes) == 0 → local exec. The risk is a silent fallback when nodes exist but are unreachable (SSH down). The plan does not specify behavior for "nodes registered but unreachable." If the scheduler selects a node and SSH-push fails, does it fall back to local or fail? This must be fail-closed (no silent local execution of a job intended for a remote node).

Binding condition C-44: P03 MUST define and test the behavior when scheduler selects a node but SSH-push fails: fail-closed (return error, do NOT silently fall back to local exec). Local fallback is only when len(registeredNodes) == 0, not when SSH fails. Test coverage for this case is mandatory before P04 ships.

Axis 2 — Scope

Verdict: PASS | Confidence: 0.85

14 phases is large but justified: the research found ~60 gaps, and the operator explicitly accepted "no limit on phases" (D-250). Each phase is independently shippable (vertical-slice integrity verified). The phase decomposition is logical:

• P01-P02: security fundamentals (toolchain, injection) — correctly first, as they're prerequisites for everything.
• P03: scheduler — correctly early, as UAT depends on it.
• P04-P06: identity stack (ACL, seal, IdP) — correctly ordered (P04 ACL depends on P03 scheduler context per plan; P06 depends on P05 seal).
• P07-P09: reliability (concurrency, transport, migration) — correctly parallelizable with P04-P06 (all depend only on P0).
• P10: metrics — correctly after P04 (acl denials) and P05 (audit chain head).
• P11: docs — correctly last before UAT (reflects reality).
• P12: UAT — correctly after P03 and P04 (the two load-bearing changes).
• P13: final — correctly last.

Gaps missed: None identified. The research sweeps were comprehensive. The deferred items (health prober, update controller, cron scheduler loop) are correctly out of scope with lint warnings.

Unnecessary phases: P11 (docs) is 14 tasks — heavy for a docs phase. But docs/cli.md missing ~25 subcommands and the verify-reqs gate bypass are real blockers. No phase should be cut.

Axis 3 — Cost

Verdict: PASS | Confidence: 0.78

Could 80% of the value be achieved with 50% of the phases? No. The critical path is: P01 (toolchain vulns) → P02 (injection RCE) → P03 (scheduler) → P04 (ACL) → P12 (UAT). That's 5 phases for the "deployment model works + not pwnable + UAT-able" core. The remaining 9 phases (seal, IdP, concurrency, transport, migration, metrics, docs, linux type) are each closing real gaps that would surface in UAT. Cutting them would make the UAT signoff script fail on those claims.

The one arguable cut: P10 (metrics) is Medium priority. But orca_acl_denials_total and orca_audit_chain_head are operational necessities for a zero-trust system — without them, ACL denials are invisible. P10 stays.

Axis 4 — Risk

Verdict: CONDITIONAL | Confidence: 0.80

Highest-risk phases

1. P03 (scheduler) — behavioral rewrite of job run. Mitigation: C-39 (local fallback), C-44 (fail-closed on SSH failure, test coverage).
2. P04 (ACL deny-by-default) — can lock out the operator. Mitigation: C-40 (bootstrap ACL grants cluster-admin to init SVID). But the plan's "staged rollout: log-only mode for first run, enforce after bootstrap ACL verified" is NOT in the P04 task list. The must-haves say "Bootstrap ACL grants cluster-admin to init SVID" (T8) but do not mention log-only mode. This is a gap.
3. P06 (auth init-idp) — deploys Dex+Traefik+systemd. This is the most operationally complex phase (real systemd unit rendering, Traefik dynamic config, step-ca cert integration). The plan describes it as one phase with 7 tasks. The risk is that the Dex deploy doesn't work in a real environment and there's no fallback tested in CI. C-37 (mTLS-only fallback) from v0.12 still applies.

Catastrophic failure modes

• P04 lockout: if bootstrap ACL fails to grant cluster-admin to the init cert's SVID, the operator is locked out of their own cluster. This is the single most catastrophic risk.
• P03 silent fallback: if SSH-push fails and the job silently runs locally, the operator thinks they deployed to a remote node but didn't. This is a data-integrity risk.

Binding condition C-45: P04 MUST implement a log-only/dry-run mode for the first invocation after ACL wiring, as C-40 specifies "staged rollout: log-only mode for first run, enforce after bootstrap ACL verified." This is in C-40's description but missing from P04's task list (T1-T11). Either add a T12 "log-only mode flag + bootstrap verification step" or split P04 into P04a (wire + log-only) and P04b (enforce). The must-haves MUST include "log-only mode exists and is the default for first run."

Axis 5 — Dependencies

Verdict: PASS | Confidence: 0.84

The dependency graph is correct:

• P04 depends on P03 (scheduler context) — weak dependency. The plan says "P0 (P03 for scheduler context)" which means P04 can proceed without P03 but benefits from it. This is correct: ACL wiring in daemon handlers doesn't strictly require the scheduler.
• P06 depends on P05 (seal) — correct: auth init-idp needs the seal infrastructure for the OIDC token exchange.
• P10 depends on P04 (acl denials metric) and P05 (audit chain head) — correct: the metrics reference features wired in those phases.
• P11 depends on P01..P10 — correct: docs reflect reality.
• P12 depends on P03 (scheduler for UAT) and P04 (ACL for UAT) — correct: the UAT exercises both.

Hidden dependency: P12 (UAT signoff script) depends on P05 (seal) and P06 (auth init-idp) being functional — the UAT must exercise seal/unseal and the OIDC flow. But the plan's dependency table says P12 depends only on P03 and P04. This is incomplete.

Binding condition C-46: P12 (UAT plan + signoff script) MUST declare dependencies on P05 (seal) and P06 (auth init-idp) in addition to P03 and P04. The UAT signoff script will assert seal/unseal round-trip and OIDC health check claims — both require P05/P06 to be shipped. If P05 or P06 slip, the corresponding UAT assertions fail (honest signal per C-42), but the dependency must be declared.

Axis 6 — Testing

Verdict: CONDITIONAL | Confidence: 0.76

The testing strategy is generally sound: each phase has a Wave 2/3 with regression tests. 128 test files exist. The security integration test suite (tests/security_integration_test.go) is extended in P02 and P04.

UAT signoff script concerns

The scripts/uat-signoff.sh (P12 T4) is ~35 assertions, idempotent, read-only. This is the v1.0 gate. Concerns:

1. No assertion for F26 (scheduler actually deploys remotely): the plan says the UAT exercises "deploy full stack" but the signoff script's ~35 assertions are not enumerated. If the script doesn't assert "job ran on remote node, not local," the headline fix (F26) is not validated.
2. No assertion for F48 (ACL deny-by-default): the UAT must include a negative test (unauthorized identity denied). But the script is "read + non-mutating" — how does it test denial without attempting a mutation? It could check acl.json mode (0600) and the audit log for denial entries, but that's indirect.
3. uat-smoke.sh in CI: the pure-CLI subset runs in .coreci.yml validate. This is good. But "version, acl file mode, doctor modes, no-password grep, metrics shape" is 5 assertions — the smoke test doesn't validate the core deployment model.

Binding condition C-47: P12 T4 (uat-signoff.sh) MUST include explicit assertions for: (a) job deployed to remote node (not local exec) — verify via orca job list showing node_id != localhost; (b) ACL deny-by-default — verify via audit log containing denial entries or a documented negative assertion; (c) seal/unseal round-trip; (d) OIDC health check (doctor oidc). The ~35 assertion count MUST include these 4 critical-path claims. The assertion list must be reviewable in docs/uat.md before the UAT is run.

Axis 7 — Security

Verdict: PASS | Confidence: 0.86

The plan closes all critical/high/medium security findings (F26-F95). The 11 injection vectors (P02) are each small and independently testable. The ACL wiring (P04) is deny-by-default with bootstrap. The seal (P05) has Shamir recovery (C-35). Key zeroing (P05 T6) is defense-in-depth.

New risks introduced by fixes

1. P03 removes local exec path: if the local fallback has a bug, job run breaks for all single-node users. Mitigation: C-44 (fail-closed on SSH failure, test the fallback).
2. P04 ACL wiring: deny-by-default could block legitimate traffic if the SVID extraction is wrong. Mitigation: C-45 (log-only mode first).
3. P05 seal: if orca cluster seal is run accidentally, the cluster is sealed. Mitigation: Shamir shards are printed (operator must store them); unseal requires OIDC token or 3-of-5 shards. This is by design.
4. P06 Dex deploy: introduces a new network service (Dex on Traefik). Mitigation: mTLS-only fallback (C-37), Traefik dynamic route is behind the orca CA.

No new risks are unmitigated. The 9 accepted residual risks are documented and reasonable.

Axis 8 — Operability

Verdict: CONDITIONAL | Confidence: 0.72

3-host topology realism

The UAT topology (lead Ubuntu 22.04 + pve01 Proxmox VE 8/9 + worker01 Ubuntu 22.04) is minimal and correct. It covers both node types (Proxmox + Linux) and migrate-between-hosts.

Concern: The UAT requires a real Proxmox VE host. This is not a CI-environment artifact — the operator must have a Proxmox server available. If the operator doesn't have one, the UAT cannot run. The plan does not address this prerequisite. uat-smoke.sh (CI subset) does NOT require Proxmox — it's pure-CLI — but the full uat-signoff.sh does.

Binding condition C-48: docs/uat.md (P12 T3) MUST document the hardware/host prerequisites explicitly: "You need a Proxmox VE 8/9 host with SSH access and root credentials." If the operator cannot provision a Proxmox host, an alternative UAT path (3x Ubuntu hosts, --type linux only, Proxmox claims marked as "not exercised in this UAT") MUST be documented. The signoff script MUST report which claims were exercised vs. skipped, so a partial UAT is an honest signal, not a false pass.

Operator ability to run the UAT

The UAT is operator-driven: docs/uat.md walks through the build, uat-signoff.sh asserts. The plan says "the operator runs it, pastes output back to the CI agent." This requires:

1. The operator has 3 hosts available (see C-48).
2. The operator can follow docs/uat.md step-by-step (it must be complete and exact).
3. uat-signoff.sh is truly idempotent and read-only (D-254).

These are achievable. The risk is that docs/uat.md is incomplete (missing a step) and the operator gets stuck. The plan's T3 says "step-by-step with exact commands" — this is the right intent.

Axis 9 — Completeness

Verdict: CONDITIONAL | Confidence: 0.74

Will this be the LAST round?

The research claims "this is the last hardening round" based on three deep sweeps. The 9 accepted residual risks are documented. But:

1. UAT will surface new gaps: the UAT signoff script exercises ~35 claims against a real 3-host cluster. This is the first time the full stack is exercised end-to-end. It is virtually certain that the UAT will discover issues not found in code review (e.g., systemd unit rendering on Proxmox, SSH-push to Ubuntu worker, Traefik route conflicts, drift event delivery across node types). The plan does not budget for a "UAT findings" follow-up.
2. P06 (Dex deploy) is untested in CI: the plan's T5 is a "hermetic Dex+Traefik config render test" — this tests config rendering, not actual deployment. The first real Dex deploy will be in the UAT. If it fails, that's a round 3.
3. --type linux (P12 T1) is new code: the first real Ubuntu worker onboarding will be in the UAT. If internal/linux/bootstrap.go has bugs, that's a round 3.

Binding condition C-49: The plan MUST acknowledge that v0.13 is "the last hardening round before UAT," not "the last hardening round absolute." The UAT will likely surface 3-7 issues requiring a follow-up patch round (v0.13.1 or a small v0.14). This is healthy and expected. The v1.0.0 tag is gated on UAT signoff passing — if UAT finds issues, v1.0.0 is deferred until they're fixed. The plan's "v1.0.0 NOT cut (deferred for UAT signoff)" in P13 is correct, but the narrative "this is the last hardening round" should be softened to "this is the last hardening round before UAT validation."

What could force a round 3?

1. UAT discovers Dex deploy doesn't work on real Proxmox.
2. UAT discovers --type linux bootstrap fails on real Ubuntu 22.04.
3. UAT discovers scheduler bin-packing produces bad placements on heterogeneous nodes (Proxmox vs Linux worker).
4. UAT discovers seal/unseal doesn't work with real OIDC tokens (not just test mocks).
5. P03's local fallback has an edge case (e.g., job with --target but target node deregistered mid-flight).

Each of these is a single-fix patch, not a full round. The plan's per-phase tag structure (v0.12.x) supports patch releases.

Summary Verdict

Axis	Verdict	Confidence
1. Feasibility	PASS	0.82
2. Scope	PASS	0.85
3. Cost	PASS	0.78
4. Risk	CONDITIONAL	0.80
5. Dependencies	PASS	0.84
6. Testing	CONDITIONAL	0.76
7. Security	PASS	0.86
8. Operability	CONDITIONAL	0.72
9. Completeness	CONDITIONAL	0.74

Overall: CONDITIONAL PROCEED | Confidence: 0.82

The plan is evidence-accurate, well-decomposed, and addresses real gaps. The binding conditions (C-44..C-49) are targeted fixes, not fundamental rework. No axis FAILs. The plan proceeds once the 6 binding conditions are incorporated.

Binding decisions (G-255..G-261)

ID	Decision	Rationale	Confidence	Alternatives
G-255	Proceed with v0.12 governance gap: P13 marks REQ-130..148 AND REQ-149..163 Complete; v0.12 stays COMPLETE retroactively	v0.12 code was shipped but not wired; v0.13 wires it; revoking COMPLETE destabilizes history without changing code; C-43 makes consistency enforceable	0.90	Revoke v0.12 COMPLETE (destabilizing); escalate (unnecessary at full autonomy)
G-256	P03 fail-closed on SSH failure (C-44)	Silent local fallback when SSH fails is a data-integrity risk; local fallback only when len(nodes)==0	0.88	Silent fallback (unsafe); no fallback (breaks single-node)
G-257	P04 log-only mode for first run (C-45)	C-40 specifies staged rollout but P04 task list omits it; deny-by-default lockout is catastrophic	0.85	Enforce immediately (lockout risk); split P04 into P04a/P04b (acceptable alternative)
G-258	P12 declares dependency on P05+P06 (C-46)	UAT exercises seal/unseal and OIDC flow, which require P05/P06; undeclared dependency hides slip risk	0.82	Leave undeclared (C-42 honest signal covers it, but dependency should be explicit)
G-259	P12 signoff script includes 4 critical-path assertions (C-47)	F26 (remote deploy), F48 (ACL deny), seal round-trip, OIDC health are the headline claims; without asserting them the UAT is theater	0.84	Trust the ~35 count (insufficient); add more later (gate must be complete at ship)
G-260	P12 docs/uat.md documents Proxmox prerequisite + alternative path (C-48)	UAT requires real Proxmox host; if operator lacks one, partial UAT must be honest signal	0.78	Assume operator has Proxmox (may not); skip Proxmox claims silently (dishonest)
G-261	v0.13 is "last round before UAT," not "last round absolute" (C-49)	UAT will surface issues; narrative should reflect this; v1.0.0 deferred until UAT passes is correct	0.80	Claim "last round absolute" (likely false); pre-commit to v0.14 (premature)

Binding conditions (C-44..C-49)

ID	Condition	Phase	Gates
C-44	P03 MUST fail-closed when scheduler selects a node but SSH-push fails (return error, no silent local fallback). Local fallback only when len(registeredNodes)==0. Test case mandatory.	P03	P04 ship
C-45	P04 MUST implement log-only/dry-run mode as default for first invocation after ACL wiring. Enforce mode enabled after bootstrap ACL verified. Add to P04 task list + must-haves.	P04	P05 ship
C-46	P12 dependency table MUST include P05 (seal) and P06 (auth init-idp) in addition to P03 and P04.	P12	P12 plan accuracy
C-47	P12 uat-signoff.sh MUST include explicit assertions for: (a) job deployed to remote node (node_id != localhost), (b) ACL deny-by-default (audit log denial entries or documented negative assertion), (c) seal/unseal round-trip, (d) OIDC health check. Assertion list reviewable in docs/uat.md.	P12	v1.0.0 gate
C-48	P12 docs/uat.md MUST document hardware/host prerequisites (Proxmox VE 8/9 host required). Alternative UAT path (3x Ubuntu, --type linux only, Proxmox claims skipped) MUST be documented. Signoff script reports exercised vs. skipped claims.	P12	UAT executability
C-49	Plan narrative MUST soften "last hardening round" to "last hardening round before UAT validation." UAT will likely surface 3-7 issues requiring patch release. v1.0.0 deferred until UAT passes.	P0/P13	Expectation setting

Escalations

None. All axes resolved at confidence >= 0.72. The question tool infrastructure failed during the interactive grill (stack overflow on every invocation); given autonomy.level=full and workflow.no_hitl=true, the grill proceeded on evidence alone. All binding decisions are evidence-based and within the agent's autonomy threshold (0.60).

What the auditor would flag

1. v0.12 COMPLETE with 19 pending REQs — documentation governance failure, now acknowledged and resolved (G-255).
2. P03 under-estimated as "wiring" — it's a behavioral rewrite of job run. Schedule accordingly.
3. P04 staged rollout missing from task list — C-40 describes it, P04 tasks omit it (C-45).
4. P12 dependencies incomplete — P05/P06 not listed (C-46).
5. UAT signoff assertions not enumerated — ~35 count without a reviewable list (C-47).
6. "Last round" narrative overclaims — UAT will find issues (C-49).

What the project is not doing that it should

1. No end-to-end integration test in CI — the UAT is the first E2E test. The uat-smoke.sh is CLI-only. A CI E2E test (mock SSH to localhost containers) would catch P03/P04 integration issues before UAT. This is deferred to v1.x and is acceptable.
2. No performance testing — the plan doesn't address scheduler performance on large node counts. Acceptable for a 3-host UAT; relevant for v1.x.
3. No chaos testing — SSH failure mid-deploy, node deregistration mid-flight, etc. C-44 covers the fail-closed case; broader chaos testing is v1.x.

Simplest version delivering 80% of value

P01 (toolchain) + P02 (injection) + P03 (scheduler) + P04 (ACL) + P12 (UAT) = 5 phases. This makes the deployment model functional, closes the RCE vectors, wires zero-trust, and delivers the UAT gate. The remaining 9 phases (seal, IdP, concurrency, transport, migration, metrics, docs, linux type) each close real gaps but could defer to v1.0.1 patches. The operator chose comprehensiveness (D-250) — justified to avoid a round 3, but the 5-phase core is the minimum viable path.

What must be true for success in 90 days

1. P03 ships with fail-closed SSH handling and local fallback (C-44).
2. P04 ships with log-only mode and bootstrap ACL (C-45).
3. P12 ships with enumerated assertions covering the 4 critical paths (C-47).
4. The operator has a 3-host environment (or the alternative UAT path is documented, C-48).
5. The UAT signoff script runs and either passes (-> v1.0.0) or fails honestly (-> patch round).

All five are achievable. The plan proceeds.