orca/.ciagent/RESEARCH_v0.13.md

RESEARCH v0.13: Production Hardening Round 2 — Threat Model & Gap Analysis

Status: complete (2026-08-07). Three deep codebase sweeps (security, reliability, feature/doc claims) performed via parallel sub-agents. ~60 gaps surfaced beyond v0.12. Findings drive the 15 new requirements (REQ-149..REQ-163) and 14-phase plan.

Methodology

Three parallel explore agents investigated the codebase:

1. Security sweep — input validation, injection, SSH, crypto, TLS, race conditions, SQL, secrets, backup, pprof, rate limiting, memory, dependencies, toolchain vulns.
2. Reliability sweep — idempotency, concurrency, SQLite, partial failure, SSH fanout, timeouts, systemd, journald, cache, watch streams, scheduler, capacity, namespace isolation, DB growth, time, signals, temp files, flock.
3. Feature/doc sweep — README claims, docs/, examples/, Makefile, .coreci.yml, CHANGELOG, REQUIREMENTS/ROADMAP consistency, help text, deprecation warnings, WASM claim.

Each agent produced a structured report with file:line evidence. This document synthesizes the findings into the v0.13 plan.

Threat Model Round 3 — Findings

Critical (must fix in v0.13)

ID	Finding	file:line	REQ
F26	orca job run runs locally via exec.CommandContext — scheduler/emitter/SSH-push are dead code; documented deployment model non-functional	internal/cli/job.go:352-372, internal/engine/executor.go:150-180	REQ-151
F27	jobspec schedule: and timeout: silently dropped by markdown parser — DaemonSet fundamentally broken	internal/jobspec/markdown.go:480-573	REQ-152
F28	verify-reqs gate bypassed for v0.12 (bold-format regex mismatch)	cmd/verify-reqs/main.go:29	REQ-160
F29	Command injection in orca logs --job via %q+backtick (RCE via SSH fanout)	internal/cli/logs.go:283,289	REQ-150
F30	pprof loopback bypass via :6060 (empty host = bind-all)	internal/daemon/pprof.go:21-29	REQ-150
F31	Tar-slip in backup restore (a/../../etc/passwd bypasses HasPrefix(name,".."))	internal/backup/backup.go:302-304	REQ-150
F32	Unauthenticated WebAuthn registration (account takeover)	internal/webauthn/connector.go:85,120	REQ-153
F33	ROADMAP marks v0.12 COMPLETE but seal/unseal/init-idp/auth-register don't exist	.ciagent/ROADMAP.md:403	REQ-154,155

High (must fix in v0.13)

ID	Finding	file:line	REQ
F34	nft ruleset injection via unvalidated TrustedProbes IPs	internal/emitter/nft.go:101-107	REQ-150
F35	sudoers/shell injection via --proxmox-user/--proxmox-role	internal/proxmox/bootstrap.go:445-452	REQ-150
F36	validateSudoers checks wrong filename when ProxmoxUser != "orca"	internal/proxmox/bootstrap.go:474	REQ-150
F37	orca txn rollback shell injection via unvalidated txn ID	internal/cli/txn.go:240-241	REQ-150
F38	orca nft diff --against path traversal	internal/cli/nft.go:225	REQ-150
F39	drain stopAlloc stored injection from compromised peer	internal/cli/drain.go:132	REQ-150
F40	cluster_compat stored injection from peer	internal/cli/cluster_compat.go:399	REQ-150
F41	podman image %q backtick injection	internal/runtime/podman.go:67	REQ-150
F42	Go toolchain 1.25.0 — 24 stdlib vulns (tar, tls, x509, http, pem...)	go.mod:3	REQ-149
F43	No SQLite busy_timeout — "database is locked" under concurrency	internal/store/store.go:21	REQ-156
F44	Audit hash-chain race — concurrent appends corrupt tamper-evidence	internal/store/audit_repo.go:908-919	REQ-154
F45	Concurrent secrets set silently loses data (no flock)	internal/cli/secrets.go:135-148	REQ-156
F46	Concurrent orca upgrade races on Traefik cutover + binary install	internal/cli/upgrade.go:111	REQ-156
F47	Cache never invalidated by writes — stale reads after join/create/run	internal/cli/cache.go:763-770	REQ-156
F48	acl.Check called zero times — v0.12 zero-trust not wired	internal/daemon/, internal/sshpush/	REQ-153
F49	acl.json mode 0644 (should be 0600 per REQ-145)	internal/cli/acl.go:152	REQ-153
F50	README "mTLS by default" is false — SSH-push is canonical, mTLS deprecated	README.md, internal/cli/node.go:93-98	REQ-160
F51	docs/cli.md missing ~25 subcommands; CHANGELOG stale at v0.1	docs/cli.md:4, CHANGELOG.md:9-32	REQ-160
F52	docs/security-runbook.md documents seal/unseal/doctor audit that don't exist	docs/security-runbook.md:5-11,23	REQ-160
F53	docs/webauthn.md documents orca auth register that doesn't exist	docs/webauthn.md:13	REQ-155,160
F54	auth init-idp is a stub — v0.12 R-021 load-bearing change has no working IdP	internal/cli/auth.go:151-155	REQ-155
F55	secrets rotate-master writes raw key, doesn't re-seal to OIDC	internal/cli/secrets.go:358	REQ-154
F56	orca cluster seal/unseal documented but not implemented	docs/security-runbook.md:3-9	REQ-154
F57	orca doctor audit documented but not implemented	docs/security-runbook.md:18	REQ-154
F58	orca doctor modes not implemented (REQ-130)	internal/security/ca.go:236	REQ-154
F59	Audit actor field is "cli"/"daemon" not OIDC sub/SVID	internal/cli/drain.go, internal/daemon/server.go	REQ-153
F60	Executor.Run holds mutex for whole job duration	internal/engine/executor.go:101-103	REQ-156
F61	splitHostPort in drain.go breaks IPv6 addresses	internal/cli/drain.go:68-74	REQ-157
F62	transport.IsTransient + sshpush.isTransient both use substring matching	internal/transport/retry.go:44, internal/sshpush/transport.go:395-414	REQ-157
F63	rotateSSHKeys partial-result window (old key overwritten before all peers updated)	internal/cli/rotate_lead.go:132	REQ-157
F64	known_hosts flock field stored but not read by dial()	internal/sshpush/transport.go:60-63	REQ-157
F65	verifyCutover uses default http.Client against orca CA (will fail TLS verification)	internal/cli/upgrade.go:313-314	REQ-157
F66	v0.8→v0.11 migration torn-write window (crash after rename, before schema fixup)	internal/migration/migrate.go:135-140	REQ-158
F67	job stop is soft-stop only (doesn't signal process)	internal/cli/job.go:266	REQ-158
F68	upgrade.go cutover uses direct sed -i (no backup file)	internal/cli/upgrade.go:performCutover	REQ-158
F69	nft country block add validates length but not content; uses %q	internal/cli/nft.go:136,259	REQ-150
F70	--type linux reserved but unimplemented	internal/model/node.go:29	REQ-161
F71	No UAT/E2E test doc exists	repo-wide	REQ-162,163

Medium (fix in v0.13)

ID	Finding	file:line	REQ
F72	Master/SVID keys never zeroed from memory after use	throughout internal/secrets/, internal/seal/	REQ-154
F73	Cache DB mode 0644 (not 0600)	internal/cache/cache.go:61-64	REQ-158
F74	writeAtomic0600/collector: predictable tmp, no cleanup, leaks	internal/identity/oidc.go:134, internal/cli/collector.go:179	REQ-156
F75	cli/acl.go writeAtomicFile no fsync (durability gap)	internal/cli/acl.go:161-181	REQ-156
F76	WebAuthn session stores unsynchronized global maps (data race)	internal/webauthn/connector.go:67,171	REQ-156
F77	loadOIDCConfig TODO for config-file loading	internal/cli/auth.go:168	REQ-155
F78	No retention/compaction for jobs/tasks/audit_log tables	internal/store/	REQ-158
F79	orca logs no --lines cap, --since unbounded (OOM risk)	internal/cli/logs.go:173-185	REQ-158
F80	ns create non-atomic (partial dir creation on mid-failure)	internal/cli/ns.go:906-918	REQ-156
F81	writeCurrentLead non-atomic os.WriteFile	internal/cli/rotate_lead.go:315-322	REQ-156
F82	secrets set doesn't validate namespace exists (creates phantom ns)	internal/cli/secrets.go:130	REQ-156
F83	backup has no lock; concurrent backups may clobber	internal/cli/backup.go:42-68	REQ-156
F84	Root command has no SIGINT/SIGTERM handler for non-watch commands	cmd/orca/main.go:17-22	REQ-157
F85	SSH commands without explicit timeouts (peer-setup, drift, txn rollback, job restart)	various	REQ-157
F86	Rendered systemd units never validated (systemd-analyze verify) before deploy	internal/emitter/systemd.go:80-98	REQ-151
F87	OIDC callback HTTP server has no timeouts (slowloris)	internal/identity/oidc.go:244	REQ-157
F88	No security headers on daemon TLS surface	internal/daemon/health.go:93	REQ-159
F89	orca status returns hardcoded v0.1 stub, not deprecated	internal/cli/status.go:22	REQ-160
F90	job run help text says "HCL spec file" but HCL is deprecated	internal/cli/job.go:47-48	REQ-160
F91	README subcommand table omits auth, nft, peer-setup	README.md	REQ-160
F92	docs/namespace.md omits inherit/set-constraint	docs/namespace.md:114-134	REQ-160
F93	README "latest tag: v0.10.19" is stale (actual: v0.11.29)	README.md:30,39	REQ-160
F94	docs/install.md+docker.md reference stale v0.4.x and deprecated daemon	docs/install.md:42,62, docs/docker.md:21,43	REQ-160
F95	IPv6 host not bracketed in proxmox SSH dial	internal/proxmox/bootstrap.go:140	REQ-157

Low (fix in v0.13 where cheap, document otherwise)

ID	Finding	file:line	REQ
F96	--pprof-allow-public documented but never implemented	internal/daemon/pprof.go:37,42,43	REQ-150
F97	nft country block add weak code validation	internal/cli/nft.go:136	REQ-150
F98	cert show/fingerprint don't emit deprecation warnings	internal/cli/cert.go	REQ-160
F99	docs/namespace.md references orca doctor --legacy-paths that doesn't exist	docs/namespace.md:165	REQ-160
F100	release.sh only builds linux-amd64; install.sh advertises arm64	scripts/release.sh:94-102	accepted (D-193)
F101	docs/cli.md version example shows "v0.9.1" but default is "0.1.0-dev"	docs/cli.md:253	REQ-160

CLEAN categories (verified, no new findings)

• SQL injection in internal/store/ — all queries use ? placeholders
• TLS version/cipher policy — TLS 1.3 only, AEAD cipher allowlist
• SSH key generation — Ed25519, crypto/rand, PKCS8, 0600
• TOFU host-key pinning — fail-closed on mismatch, constant-time comparison
• Self-signed cert generation — RSA 3072, 128-bit serial, correct KeyUsage
• Nonce reuse in secrets — fresh 12-byte nonce per line from crypto/rand
• Gitleaks / secrets in git history — only test fixtures
• Secrets logged in errors — only keys/namespaces logged, never values
• CSRF on HTTP surfaces — daemon is GET-only, no state-changing GETs
• Watch streams (iter.Seq) — pull-based, defer cleanup, no goroutine leak
• DNS resolution — bounded by net.Dialer{Timeout: 15s}
• Multi-namespace DB isolation — per-ns file layout

Accepted residual risks (documented, not fixed)

1. OIDC tokens plaintext at rest (0600) — sealing on every CLI invocation conflicts with "no orca binary on servers" model
2. HSTS on daemon — mTLS-only API, no browser-facing surface
3. DNS resolution timeout — bounded by net.Dialer{Timeout: 15s}
4. Temp file cleanup on SIGKILL — orphaned temp files, operator-visible
5. Flock timeout on NFS — stuck holder is rare; tryFlockEx exists
6. "WASM-first" pillar aspirational — document as "WASM runtime available, process is default"
7. arm64/armv7 release — D-193 deferred; install.sh detection is forward-looking
8. OIDC callback slowloris — loopback, short-lived, single CLI invocation
9. --pprof-allow-public flag — remove references, make loopback-only a hard invariant

Architecture updates (for ARCHITECTURE.md)

• R-022: orca job run deploys via scheduler → emitter → SSH-push (local exec path removed)
• R-023: Zero-trust enforcement wired (acl.Check on every request path)
• New component: internal/linux/bootstrap.go (Ubuntu/Debian SSH-join, mirrors Proxmox pattern)
• New artifact: docs/uat.md + scripts/uat-signoff.sh (v1.0 gate)
• New artifact: docs/metrics.md (expanded Prometheus metric set)

Conclusion

Three deep sweeps found ~60 gaps. v0.13 closes all critical/high/medium (REQ-149..REQ-163, 14 phases). 9 low-severity residual risks are documented and accepted. This is the last hardening round. v1.0.0 is gated on the UAT signoff script delivered by P12.