orca/.ciagent/RESEARCH_v0.11.md

# Research: v0.11 Production Hardening

## Source material

Five research documents were ingested 2026-08-07 as directional input
(not verbatim) for v0.11 Phase 0. The current ciagent files
(R-001…R-016, D-001…D-206) are authoritative and take precedence; where
research conflicted, the ciagent files won. The research drove the
adoption of R-017…R-020 and D-215…D-237 (see PROJECT.md, PRD_v0.11.md).

| Doc | Theme | Adopted as |
|-----|-------|------------|
| 1 | Ingress hybrid (nft DNAT → Traefik on 127.0.0.1:8443) | R-017, D-215..D-226, REQ-099..REQ-102 |
| 2 | Platform-engineer playbook (8 differentiators, TCO, honest trade-offs) | README positioning (Q5=A), CLI surface gap analysis |
| 3 | Strategic positioning ("be Proxmox-for-bare-metal, not K8s-without-K8s") | README framing (Q5=A: Nomad-inspired, honest trade-offs table from doc 3, not Proxmox-first lead) |
| 4 | Drift detection cadence (R-018/R-019/R-020, tiered cadence, hard gate) | R-018, R-019, R-020, D-227..D-237, REQ-103..REQ-113 |
| 5 | Drift detection concrete impl (systemd Path units, orca-drift-notify.sh, orca-remediate.sh) | D-227..D-237 detail, REQ-103..REQ-113 |

## Thread A — Ingress hardening (doc 1)

### What changes vs v0.9/v0.10

Traefik static config gains `entryPoints.websecure.address: 127.0.0.1:8443`
(default) instead of `:443`. A new nftables emitter renders
`/etc/nftables.d/orca.nft` with DNAT rules. Certs, mTLS, dynamic config,
and the workload SPIFFE validation path are **completely unchanged**.
Only the `address` line shifts + one new emitter + `orca doctor nft` +
`orca nft ...` CLI.

### Defense in depth

Two layers: kernel (nftables: SYN flood, rate limit, GeoIP, conntrack)
and application (Traefik: mTLS, SNI, ACL, dynamic routing, health
checks). Neither can replace the other; they catch different attack
classes.

### Codebase reality (verified 2026-08-07)

- `internal/emitter/traefik.go` + `traefik_atomic.go` exist (v0.9 P02).
  The static-config emitter is where the `address:` line change lands.
- `internal/emitter/systemd.go` exists. New `.path`/`.service` unit
  types extend this emitter pattern (shared with drift detection, doc 5).
- `internal/emitter/nft.go` does **not** exist — greenfield, ~200 LoC.
- `scripts/` has `orca-verify-render.sh` but **not** `orca-aggregate.sh`,
  `orca-pull.sh`, `orca-apply-render.sh`, `orca-remediate.sh` — all are
  v0.11 P09/P10 scope.

## Thread B — Drift detection + transactional plane (docs 4 + 5)

### Architecture

systemd Path units (R-001-clean; systemd is OS, not Orca) watch critical
paths via inotify. On change, a oneshot service computes sha256 and
writes an event JSON to `/etc/orca/state/drift-events/`. The lead's
aggregator timer (10s, C-11) rsyncs these events, validates against the
applied txn manifest, and triggers remediation for auto-remediable
paths.

### Tiered cadence

| Tier | Detection | Auto-remediate | Latency |
|------|-----------|----------------|--------|
| Critical | Path unit + 5s polling backstop | yes (config files only) | ~10s |
| Standard | 30s polling | optional (systemd units: require approval) | 30s |
| Default | 60s polling | no | 60s |

### R-020 hard gate

Applier refuses new txns if pre-flight consistency check fails. Override:
`--force` flag + per-namespace scoping (Q4=A) — a drifted peer in ns-A
does not block ns-B.

### Codebase reality (verified 2026-08-07)

- `internal/store/node_repo.go:80` and `internal/store/job_task_repo.go:82`
  already use `iter.Seq[T]`. Doc 5's `iter.Seq2[Event, error]` is the
  natural extension per D-017 (settled, shipped v0.3).
- `internal/paths/paths.go:86` has `TxnDir()` — the txn staging dir the
  drift detector hooks into.
- `internal/emit/contract.go` has the Go↔bash render-contract anti-drift
  (C-16). The *runtime* drift detector (doc 5) is net-new.
- `internal/drift/` package does **not** exist — greenfield, ~500 LoC.
- No `orca` system user creation in code — net-new operational
  requirement (REQ-111).
- No NFS detection at peer setup — net-new (REQ-112, D-233).
- `doctor.go` has an OS-drift *check* (one-shot, on-demand) but **not**
  a 60s runtime drift-polling loop. Doc 5's design is net-new scope.

### Alignment with existing gates

- **C-09** (`orca-pull.sh` failure contract) — R-020 refines
  "deterministic state" into an explicit refusal contract.
- **C-11** (lead-side watchdog meta-timer) — doc 5's aggregator
  extension is the input C-11 monitors.
- **REQ-075** (lead applier execution model) — doc 5's `orca-remediate.sh`
  is literally the same code path as a normal txn-apply, triggered by
  drift instead of a new submission.

## Thread C — Positioning/messaging (docs 2 + 3)

### Consistent with locked vision

The vision is *"A minimalist, offline-first, CLI-first orchestration
engine inspired by HashiCorp Nomad"* — explicitly Nomad-inspired, not
K8s. Doc 3's recommendation ("be Proxmox-for-bare-metal, not
K8s-without-the-complexity") is consistent with the locked vision.

### Where doc 3 diverges (resolved per Q5=A)

Doc 3 recommends "leading with Proxmox positioning." But R-003 says
"Proxmox can never be lead." Leading the *project identity* with a node
type that can't be the lead is subtly contradictory. **Q5=A decision**:
README uses the Nomad-inspired, OS-as-cluster framing (locked vision),
mentions Proxmox as one node type, and incorporates doc 3's "honest
trade-offs" table but not its Proxmox-first lead-positioning advice.

### CLI surface gap analysis (doc 2)

Doc 2's playbook cites ~10 CLI commands. Verified against the live
codebase (`internal/cli/*.go`):

**Exist today**: `orca init`, `orca node {join,leave,list,key-reset,
capacity}`, `orca job {run,list,stop,logs}`, `orca ns {list,create,
delete,inspect,validate}`, `orca cert {ca-init,gen,show,renew,
fingerprint}`, `orca doctor {cert,network,db,os,proxmox}`, `orca audit
list`, `orca status`, `orca version`, `orca daemon` (deprecated).

**Not in v0.11 ROADMAP, added per Q2=C**: `orca cluster rotate-lead`
(REQ-114, P14b), `orca upgrade` (REQ-115, P14a), `orca job migrate`
(REQ-116, P05), `orca logs --all-nodes --since` (REQ-117, P06),
`orca doctor mTLS` (REQ-118, P15.5).

**Already in v0.11 ROADMAP**: `orca node drain` (P05), `orca job lint`
(P11), `orca job verify` (P12), `orca restore` (P07), `orca backup`
(P04).

### Unverified performance claims in doc 3

Doc 3's "10s applier timer = 10,000x slower than K8s informers" and
"60s drift polling" are **forward-looking design constraints**, not
current-state limitations — no applier timer or drift-polling loop
exists in the codebase. These are answered by R-018/R-019/R-020
(doc 4 + doc 5): the drift detector is a backstop, not the primary
detector, and critical paths get ~10s latency via systemd Path units.

## Persona assessment

v0.11 touches these territories:

| Territory | Persona | Phases |
|-----------|---------|--------|
| `internal/cli/**`, `internal/drift/**`, `internal/nft/**` | backend-engineer | P10, P15.5, P05, P06, P14a, P14b |
| `internal/emitter/**`, `internal/sshpush/**` | backend-engineer + lead-developer | P09, P10, P15.5 |
| `internal/store/**`, migrations | data-engineer | P14a (data migration) |
| `scripts/orca-*.sh` | backend-engineer (bash tooling) | P09, P10 |
| `docs/**`, `README.md`, `examples/**` | lead-developer + docs-engineer (phase-specific) | P15, P08 |
| Threat model, security review, mTLS, secrets | security-engineer | P03, P15.5 |
| nftables, Traefik binding, cluster mesh | network-engineer | P15.5, P09 |
| Test coverage, integration harness | devops-engineer (phase-specific) | P08 |

No frontend-engineer work (no UI). The data-engineer persona is
reactivated for P14a (v0.8→v1.0 data migration). A docs-engineer custom
persona is created for P15 (README) and P08 (integration test docs).
See `PERSONAS.md` for the updated roster.