orca/.ciagent/PLAN_v0.13.md

PLAN v0.13: Production Hardening Round 2 + UAT Plan

Status: complete (2026-08-07). 14 phases (P0 + P01..P12 + P13 final). Each phase ships a patch tag on the v0.12.x line. This plan references requirement IDs from REQUIREMENTS.md and follows the vertical-slice integrity rule (each phase is independently shippable).

Phase 0: Pre-execution (this phase)

Status: complete. SPECIFY → CLARIFY → RESEARCH → IDEATE → PLAN → GRILL → SHIP. Ships as v0.12.0.

Phase 1: Toolchain & dependency vulns (REQ-149)

Tag: v0.12.1 | Type: fix | Persona: security-engineer

Wave 1 (single task)

• T1: Bump go.mod from go 1.25.0 to go 1.25.12 (or latest 1.25.x). Run go mod tidy. Run govulncheck -show verbose ./... and triage the 6 imported third-party vulns. Bump any dep with a reachable trace (webauthn, cobra, modernc/sqlite, go-jose, coreos/ go-oidc, x/crypto, oauth2). Verify make build && make test && make lint all pass.

Must-haves

• go.mod declares go 1.25.12+
• govulncheck ./... reports zero stdlib vulns with call traces
• make build && make test && make lint pass

Phase 2: Input validation & injection hardening (REQ-150)

Tag: v0.12.2 | Type: fix | Persona: backend-engineer

Wave 1 (11 sub-fixes, all in internal/)

• T1: orca logs --job — validate against ^[A-Za-z0-9_-]+$; replace fmt.Sprintf("journalctl -u %q", ...) with shellQuote (critical: backtick RCE via SSH fanout)
• T2: pprof isLoopback(":6060") — treat empty host as non- loopback/bind-all; reject unless explicit public-allow flag wired; remove phantom --pprof-allow-public references; make loopback-only a hard invariant
• T3: backup restore tar-slip — replace HasPrefix(name, "..") with filepath.Rel(target, dest) containment check
• T4: orca txn rollback — validate txn ID against ^T-[0-9a-f]{16}$
• T5: orca nft diff --against — validate txn ID before filepath.Join
• T6: drain stopAlloc — validate allocID against ^[A-Za-z0-9_-]+$ before systemctl stop
• T7: cluster_compat — shellQuote(first) for peer dir name
• T8: runtime/podman.go — use shellQuote(image) not %q
• T9: nft TrustedProbes — validate each entry with net.ParseIP/net.ParseCIDR; fix ipv4/ipv6 mismatch
• T10: sudoers — validate --proxmox-user/--proxmox-role against ^[a-z_][a-z0-9_-]{0,31}$; write to fixed /etc/sudoers.d/orca; shellQuote all pveum/useradd; validateSudoers check actual file
• T11: nft country block add — validate ^[A-Z]{2}$

Wave 2 (tests)

• T12: Add injection/traversal regression tests for each sub-fix; extend tests/security_integration_test.go with negative tests

Must-haves

• All 11 injection/traversal vectors fixed with validation
• Regression tests for each vector
• tests/security_integration_test.go passes

Phase 3: Scheduler/deployment wiring + jobspec parser (REQ-151, REQ-152)

Tag: v0.12.3 | Type: feat | Persona: lead-developer

Wave 1 (jobspec parser fixes — REQ-152)

• T1: Add case "schedule": and case "timeout": to top-level switch in internal/jobspec/markdown.go
• T2: Fix DaemonSet — parser must not default Count to 1 for DaemonSet (validator rejects Count != 0)
• T3: restart: policy → systemd Restart=/StartLimitBurst in internal/emitter/systemd.go
• T4: Add job lint warnings for advisory-only fields (cron, health, update, affinity) — honest "not enforced in this version"

Wave 2 (scheduler wiring — REQ-151)

• T5: Wire internal/scheduler.Schedule() into orca job run — replace local exec.CommandContext path with: evaluate constraints/ capacity/affinity → render systemd units → SSH-push to target
• T6: --target overrides scheduler selection (manual pinning)
• T7: Without --target, scheduler bin-packs across ready nodes
• T8: Local fallback when no remote nodes registered (single-node dev mode — preserves backward compatibility)
• T9: systemd-analyze verify on rendered unit before deploy

Wave 3 (tests)

• T10: Scheduler constraint/capacity/affinity enforcement tests
• T11: DaemonSet spec passes lint and runs
• T12: timeout: on Jobs enforced (kill after duration)
• T13: Local fallback test (no remote nodes)

Must-haves

• orca job run --target <node> deploys via SSH-push to remote
• Scheduler evaluates constraints/capacity/affinity
• DaemonSet works (schedule parsed, Count correct)
• timeout: enforced on Jobs
• restart: translated to systemd unit
• Local fallback when no remote nodes
• systemd-analyze verify before deploy

Phase 4: ACL enforcement + WebAuthn registration auth (REQ-153)

Tag: v0.12.4 | Type: fix | Persona: backend-engineer

Wave 1 (ACL wiring)

• T1: Wire acl.Check into dispatch_handler.go — extract OIDC sub/SPIFFE SVID from mTLS peer cert, check against ACL
• T2: Wire acl.Check into jobs_handler.go, nodes_handler.go, tasks_handler.go, health_handler.go
• T3: Wire acl.Check into internal/sshpush/ — validate ORCA_OIDC_TOKEN bearer against JWKS
• T4: Wire acl.Check into internal/txn/txn.go apply path
• T5: Thread OIDC sub/SVID into audit actor field
• T6: Fix acl.json mode 0644→0600
• T7: Add flock on acl.json for concurrent grant/revoke
• T8: Bootstrap ACL: grant cluster-admin to init cert's SVID

Wave 2 (WebAuthn registration auth)

• T9: Fix WebAuthn unauthenticated registration — require existing session or admin bootstrap token; no overwriting existing creds without re-auth

Wave 3 (tests)

• T10: Extend tests/security_integration_test.go with deny-by- default enforcement test per handler
• T11: WebAuthn registration auth test (unauthenticated rejected)

Must-haves

• acl.Check called in all 5 daemon handlers + sshpush + txn
• acl.json mode 0600
• Audit actor = OIDC sub/SVID
• WebAuthn registration requires auth
• Bootstrap ACL grants cluster-admin to init SVID
• Deny-by-default enforcement tests pass

Phase 5: Seal/audit CLI + chain race + key zeroing (REQ-154)

Tag: v0.12.5 | Type: feat+fix | Persona: security-engineer

Wave 1 (CLI commands)

• T1: Implement orca cluster seal/unseal (wraps internal/seal/; OIDC token exchange; Shamir 3-of-5 shards; sealed blob 0600)
• T2: Implement orca doctor audit (wraps AuditRepo.VerifyChain)
• T3: Implement orca doctor modes (wraps EnforceFileModes)

Wave 2 (fixes)

• T4: Fix audit hash-chain race — Append uses BEGIN IMMEDIATE transaction
• T5: Fix secrets rotate-master to actually re-seal to OIDC
• T6: Zero master key / namespace keys / SVID private keys after use (defense-in-depth)

Wave 3 (tests)

• T7: Seal→unseal→secrets get round-trip test
• T8: doctor audit tamper-detection test
• T9: doctor modes 0644-rejection test
• T10: Audit chain concurrent-write integrity test
• T11: Key zeroing verification test

Must-haves

• orca cluster seal/unseal work (round-trip)
• orca doctor audit verifies chain
• orca doctor modes checks file modes
• Audit chain survives concurrent appends
• secrets rotate-master re-seals to OIDC
• Keys zeroed after use

Phase 6: auth init-idp real + auth register (REQ-155)

Tag: v0.12.6 | Type: feat | Persona: security-engineer

Wave 1

• T1: Implement orca auth init-idp — render Dex systemd unit + config template + Traefik dynamic route from internal/webauthn/ connector; RP ID = cluster Traefik domain; HTTPS via step-ca cert; atomic deploy with rollback
• T2: Implement orca auth register (browser flow to WebAuthn registration endpoint)
• T3: loadOIDCConfig config-file loading (oidc.issuer in config)
• T4: orca doctor oidc health check

Wave 2 (tests)

• T5: Hermetic Dex+Traefik config render test
• T6: doctor oidc health check test
• T7: Virtual-authenticator WebAuthn flow test (C-38)

Must-haves

• auth init-idp deploys Dex+Traefik+systemd
• auth register opens browser flow
• oidc.issuer loadable from config file
• doctor oidc health check works

Phase 7: Concurrency safety (REQ-156)

Tag: v0.12.7 | Type: fix | Persona: data-engineer + backend-engineer

Wave 1 (SQLite)

• T1: Add busy_timeout(5000) + SetMaxOpenConns(1) to all 4 DSNs (store, cache, recovery, webauthn)

Wave 2 (flocks + locks)

• T2: Secrets file flock (concurrent secrets set on same ns)
• T3: Upgrade lock file (refuse concurrent orca upgrade)
• T4: Backup lock file
• T5: Cache invalidation by write commands (node join/leave, ns create/delete, job run/stop)
• T6: Executor.Run mutex scope fix (hold only for DB inserts)
• T7: ns create atomic dir+ns.md write
• T8: writeCurrentLead atomic write
• T9: Consolidate 3 divergent writeAtomic impls onto security.WriteAtomic
• T10: WebAuthn session stores guarded with sync.Mutex

Wave 3 (tests)

• T11: Concurrent secrets set test (no data loss)
• T12: Concurrent upgrade rejection test
• T13: Cache invalidation read-after-write test
• T14: SQLite concurrent writer test (no "database is locked")

Must-haves

• All SQLite DSNs have busy_timeout
• Concurrent secrets set preserves all writes
• Concurrent upgrade rejected
• Cache invalidated by writes (read-after-write consistency)
• WebAuthn session stores thread-safe

Phase 8: Transport & SSH safety (REQ-157)

Tag: v0.12.8 | Type: fix | Persona: backend-engineer

Wave 1

• T1: Replace substring matching in transport.IsTransient AND sshpush.isTransient with typed sentinels (errors.Is)
• T2: rotateSSHKeys 2-phase atomic swap
• T3: known_hosts flock field read by dial()
• T4: IPv6 net.JoinHostPort in proxmox SSH dial + drain splitHostPort
• T5: Explicit timeouts for peer-setup, drift remediate/ack, txn rollback, job restart
• T6: verifyCutover use security.ClientTLSConfig with orca CA
• T7: OIDC callback server ReadHeaderTimeout: 5s
• T8: Root SIGINT/SIGTERM handler for non-watch commands

Wave 2 (tests)

• T9: Typed-error classification test
• T10: rotate-lead 2-phase with partial-peer failure test
• T11: IPv6 SSH dial test
• T12: Signal handling clean-exit test

Must-haves

• No substring matching in transport retry logic
• rotateSSHKeys atomic 2-phase
• IPv6 addresses work in SSH dial
• All SSH commands have explicit timeouts
• SIGINT/SIGTERM triggers clean exit

Phase 9: Migration & operational safety (REQ-158)

Tag: v0.12.9 | Type: fix | Persona: data-engineer

Wave 1

• T1: Migration transaction + torn-write fix
• T2: job stop real systemctl stop via SSH
• T3: DB retention/compaction for jobs/tasks/audit_log
• T4: orca logs --lines cap + --since upper bound
• T5: Cache DB mode 0600
• T6: upgrade.go cutover backup-file + atomic-rename

Wave 2 (tests)

• T7: Migration transaction-rollback test
• T8: job stop actually-stops test
• T9: DB retention compaction test
• T10: Logs --lines cap test

Must-haves

• Migration is transactional + recoverable from torn write
• job stop sends systemctl stop via SSH
• DB retention prevents unbounded growth
• Logs output is bounded

Phase 10: Observability & metrics (REQ-159)

Tag: v0.12.10 | Type: feat | Persona: backend-engineer

Wave 1

• T1: Add metrics: orca_jobs_by_state, orca_drift_events_total, orca_ssh_errors_total, orca_txn_apply_total, orca_txn_rollback_total, orca_acl_denials_total, orca_audit_chain_head
• T2: New docs/metrics.md with Prometheus scrape config
• T3: Security headers middleware on daemon

Wave 2 (tests)

• T4: Metric exposition format + counter increment tests

Must-haves

• 7 new metrics exposed at /metrics
• docs/metrics.md exists
• Security headers set on daemon responses

Phase 11: Doc drift round 2 (REQ-160)

Tag: v0.12.11 | Type: docs | Persona: lead-developer

Wave 1 (README + CHANGELOG)

• T1: README — update status banner, latest tag, subcommand table (add auth/nft/peer-setup/secrets rotate-master), correct "mTLS by default" claim, add missing docs to table
• T2: CHANGELOG regen

Wave 2 (docs/*)

• T3: docs/cli.md — complete rewrite covering all ~40 subcommands
• T4: docs/webauthn.md — add auth register
• T5: docs/namespace.md — add inherit/set-constraint
• T6: docs/install.md+docker.md — update version refs
• T7: docs/security-runbook.md — match P05 reality
• T8: docs/security-scanning.md — gosec.json

Wave 3 (code-level doc fixes)

• T9: Fix verify-reqs bold-format regex (bypasses v0.12)
• T10: Fix ROADMAP/REQUIREMENTS v0.12 status hygiene
• T11: internal/proxmox/bootstrap.go comments (password→key auth)
• T12: Deprecate orca status stub
• T13: Help text fixes (job run HCL→markdown, job stop daemon→SSH-push)
• T14: make verify-docs target (cli.md ↔ orca --help)

Must-haves

• README accurate (status, tag, subcommands, claims)
• docs/cli.md covers all subcommands
• verify-reqs works for v0.12 and v0.13
• make verify-docs passes

Phase 12: --type linux + UAT plan + signoff (REQ-161, REQ-162, REQ-163)

Tag: v0.12.12 | Type: feat | Persona: lead-developer + uat-engineer

Wave 1 (--type linux — REQ-161)

• T1: Implement internal/linux/bootstrap.go (mirrors Proxmox pattern without PVE role/sudoers)
• T2: Wire orca node join --type linux --host <ip> --ssh-user root --ssh-key <path>

Wave 2 (UAT plan — REQ-162)

• T3: Write docs/uat.md — 3-host topology, step-by-step, claim matrix (~35 claims), signoff procedure

Wave 3 (UAT signoff — REQ-163)

• T4: Write scripts/uat-signoff.sh — ~35 named assertions, idempotent, read-only, exit 0 iff all pass
• T5: Write scripts/uat-smoke.sh — pure-CLI subset for CI validate

Wave 4 (tests)

• T6: --type linux bootstrap round-trip test (mock SSH)
• T7: uat-signoff.sh syntax + assertion-count test
• T8: uat-smoke.sh in .coreci.yml validate

Must-haves

• orca node join --type linux works (SSH bootstrap)
• docs/uat.md covers 3-host topology + all claims
• scripts/uat-signoff.sh has ~35 assertions, idempotent
• scripts/uat-smoke.sh runs in CI

Phase 13: Final review + ship + audit

Tag: v0.12.13 = v0.13 milestone release | Type: chore

Wave 1

• T1: ciagent-review — multi-persona code review across P01..P12
• T2: ciagent-audit — reconstruction test, branch hygiene, commit discipline; fix any remaining verify-reqs discrepancies
• T3: Update REQUIREMENTS.md — mark all v0.13 REQs as complete
• T4: Update ROADMAP.md — mark v0.13 as COMPLETE
• T5: Merge phase/13 → milestone/v0.13 → main
• T6: Tag v0.12.13 (milestone release)
• T7: Create release with full milestone summary

Must-haves

• All v0.13 REQs marked complete in REQUIREMENTS.md
• ROADMAP.md marks v0.13 COMPLETE (with bold)
• verify-reqs passes for v0.12 and v0.13
• Milestone merged to main
• v0.12.13 tag created
• v1.0.0 NOT cut (deferred for UAT signoff)

Wave ordering summary

Phase	Waves	Tasks	Depends on
P01	1	1	P0
P02	2	12	P0
P03	3	13	P0
P04	3	11	P0 (P03 for scheduler context)
P05	3	11	P0
P06	2	7	P05 (seal)
P07	3	14	P0
P08	2	12	P0
P09	2	10	P0
P10	2	4	P04 (acl denials metric), P05 (audit chain head)
P11	3	14	P01..P10 (docs reflect reality)
P12	4	8	P03 (scheduler for UAT), P04 (ACL for UAT)
P13	1	7	P01..P12

Vertical slice integrity

Each phase is independently shippable:

• P01 (toolchain) — bumps go version, no API change
• P02 (injection) — validates inputs, no API change
• P03 (scheduler) — changes job run behavior (local→remote), local fallback preserves backward compat
• P04 (ACL) — adds enforcement, bootstrap ACL prevents lockout
• P05 (seal) — adds new CLI commands, no breaking change
• P06 (init-idp) — replaces stub, no breaking change
• P07 (concurrency) — adds locks/timeouts, no API change
• P08 (transport) — replaces substring with typed errors, no API change
• P09 (migration) — fixes migration safety + job stop, job stop is behavioral change (soft→hard stop) — documented
• P10 (metrics) — adds metrics, no API change
• P11 (docs) — docs only, no code behavior change
• P12 (UAT) — adds new command + docs + scripts, no breaking change
• P13 (final) — review + ship, no new features

Grill binding conditions (C-44..C-49) — incorporated

ID	Condition	Phase affected	How addressed
C-44	P03 MUST fail-closed when scheduler selects a node but SSH-push fails. Local fallback only when len(registeredNodes)==0. Test case mandatory.	P03	Added to P03 must-haves + T13 test
C-45	P04 MUST implement log-only/dry-run mode as default for first invocation after ACL wiring. Enforce mode after bootstrap ACL verified.	P04	Added T9.5 (log-only mode) + T11.5 (enforce-mode toggle) to P04
C-46	P12 dependency table MUST include P05 (seal) and P06 (auth init-idp) in addition to P03 and P04.	P12	Updated dependency table above
C-47	P12 uat-signoff.sh MUST include explicit assertions for: (a) job deployed to remote node, (b) ACL deny-by-default, (c) seal/unseal round-trip, (d) OIDC health check.	P12	Added to P12 must-haves + assertion list in docs/uat.md
C-48	P12 docs/uat.md MUST document hardware prerequisites (Proxmox VE 8/9 host required). Alternative UAT path (3x Ubuntu, --type linux only, Proxmox claims skipped) MUST be documented.	P12	Added to P12 T3 scope
C-49	Plan narrative MUST soften "last hardening round" to "last hardening round before UAT validation."	P0/P13	Updated PROJECT.md + ROADMAP.md narrative

Updated P03 must-haves (C-44)

• P03 fails-closed when scheduler selects a node but SSH-push fails (returns error, no silent local fallback)
• Local fallback ONLY when len(registeredNodes)==0
• Test case for SSH-push failure → error (not silent local)

Updated P04 task list (C-45)

• T9.5: Implement log-only/dry-run mode as default for first invocation after ACL wiring (log denials, do not block)
• T11.5: Enforce mode after bootstrap ACL verified (toggle via orca acl enforce or config)

Updated P12 dependencies (C-46)

• P12 depends on: P03 (scheduler), P04 (ACL), P05 (seal), P06 (auth init-idp)

Updated P12 must-haves (C-47, C-48)

• uat-signoff.sh asserts: job deployed to remote node (node_id != localhost)
• uat-signoff.sh asserts: ACL deny-by-default (denial logged)
• uat-signoff.sh asserts: seal/unseal round-trip
• uat-signoff.sh asserts: OIDC health check
• docs/uat.md documents Proxmox VE 8/9 hardware prerequisite
• docs/uat.md documents alternative UAT path (3x Ubuntu, Proxmox claims skipped)

Updated narrative (C-49)

v0.13 is the "last hardening round before UAT validation." The UAT will likely surface 3-7 issues requiring a patch release. v1.0.0 is deferred until UAT passes.