docs(P00): research findings + persona assessment for v0.11

RESEARCH stage: consolidate 5 research docs (ingress hybrid, drift detection, platform-engineer playbook, strategic positioning, systemd Path unit impl) + codebase verification into RESEARCH_v0.11.md. Update PERSONAS.md: data-engineer reactivated for P14a, docs-engineer phase-specific for P15, devops-engineer owns drift-detection bash scripts + integration tests. ---ci--- project: orca phase: 0 milestone: v0.11 status: research ---/ci---
2026-08-07 03:38:17 +00:00
parent 6f7a5122cc
commit a70eb0d83d
3 changed files with 212 additions and 5 deletions
@@ -1,14 +1,13 @@
 {
  "phase": 0,
-  "stage": "clarify",
+  "stage": "research",
  "milestone": "v0.11",
  "milestone_slug": "production-hardening",
  "phase_role": "pre_execution",
  "attempts": 0,
-  "updated_at": "2026-08-07T00:01:00Z",
+  "updated_at": "2026-08-07T00:02:00Z",
  "milestone_complete": false,
  "previous_milestone": "v0.10",
  "research_docs_ingested": 5,
-  "locked_decisions": {"Q1": "A", "Q2": "C", "Q3": "A", "Q4": "A", "Q5": "A"},
-  "clarifications": {"C1": "hybrid-default-fresh-migrate-existing", "C2": "thin-wrapper", "C3": "drain-reschedule", "C4": "cooldown-on-success-only", "C5": "chain-plus-live-probe"}
+  "locked_decisions": {"Q1": "A", "Q2": "C", "Q3": "A", "Q4": "A", "Q5": "A"}
 }
@@ -210,4 +210,48 @@ reason: |
 | `security-engineer` reactivated | Owns SSH keygen, TOFU host-key, sudoers, PVE role — first-class security surface. |
 | `devops-engineer` deactivated | v0.6 has no packaging/distribution surface. |
 | `network-engineer` remains deactivated | No transport/mTLS surface. |
-| `frontend-engineer` remains deactivated | No web UI. |
+| `frontend-engineer` remains deactivated | No web UI. |
+## v0.11 Update (Production Hardening)
+
+The v0.9 persona roster carries forward to v0.11 with these additions:
+
+### Roster changes
+
+- **lead-developer**: RETAINED — owns `orca cluster rotate-lead` (P14b),
+  `orca upgrade` (P14a), README framing (P15, Q5=A Nomad-inspired),
+  milestone coordination.
+- **backend-engineer**: RETAINED — owns `internal/drift/` (P10, ~500 LoC
+  greenfield), `internal/emitter/nft.go` (P15.5, ~200 LoC greenfield),
+  `orca drift` CLI tree (P10), `orca nft` CLI (P15.5), `orca job migrate`
+  (P05), `orca logs --all-nodes` (P06), `orca doctor mTLS`/`orca doctor nft`
+  (P15.5), `scripts/orca-drift-notify.sh` + `orca-remediate.sh` (P10).
+  Frameworks: cobra, `iter.Seq2` (D-017 extension), `signal.NotifyContext`
+  (D-023), golang.org/x/crypto/ssh (existing).
+- **data-engineer**: REACTIVATED for P14a — owns v0.8→v1.0 data migration
+  (REQ-066), schema migration for `orca upgrade` binding cutover. Was
+  deactivated in v0.10 (docs-only milestone); reactivated for the
+  migration phase.
+- **security-engineer**: RETAINED — owns threat model (P15.5, C-19),
+  secrets subsystem (P03), `orca doctor mTLS` (P15.5), ingress-hybrid
+  trust-boundary review (R-017), drift-detection threat model (R-020
+  deadlock, secret exclusion D-234).
+- **network-engineer**: RETAINED — owns nftables emitter (P15.5, R-017),
+  Traefik binding cutover (P14a/P15.5), cross-node cluster mesh (D-219,
+  unchanged private IP), drift-detection network paths (NFS detection
+  D-233, SSH fanout for aggregator).
+- **devops-engineer**: RETAINED — owns `scripts/orca-aggregate.sh`
+  extension (P09, D-237), `scripts/orca-drift-notify.sh` (P10),
+  `scripts/orca-remediate.sh` (P10), systemd Path unit emitter (P10),
+  drift-detection integration tests (P08: auto-remediation, NFS fallback,
+  cooldown, secret exclusion), `orca` system user setup (P10, REQ-111).
+- **docs-engineer**: PHASE-SPECIFIC (P15) — owns README refresh (Q5=A
+  Nomad-inspired framing, honest-trade-offs table from research doc 3).
+  Created for P15; removed after phase completes.
+- **cli-engineer**: remains DEACTIVATED — CLI surface growth is owned by
+  lead-developer + backend-engineer.
+- **frontend-engineer**: remains DEACTIVATED — no web UI.
+
+### Phase-specific personas
+
+- `docs-engineer`: active for P15 only (README refresh). Removed after
+  phase completes.
@@ -0,0 +1,164 @@
+# Research: v0.11 Production Hardening
+
+## Source material
+
+Five research documents were ingested 2026-08-07 as directional input
+(not verbatim) for v0.11 Phase 0. The current ciagent files
+(R-001…R-016, D-001…D-206) are authoritative and take precedence; where
+research conflicted, the ciagent files won. The research drove the
+adoption of R-017…R-020 and D-215…D-237 (see PROJECT.md, PRD_v0.11.md).
+
+| Doc | Theme | Adopted as |
+|-----|-------|------------|
+| 1 | Ingress hybrid (nft DNAT → Traefik on 127.0.0.1:8443) | R-017, D-215..D-226, REQ-099..REQ-102 |
+| 2 | Platform-engineer playbook (8 differentiators, TCO, honest trade-offs) | README positioning (Q5=A), CLI surface gap analysis |
+| 3 | Strategic positioning ("be Proxmox-for-bare-metal, not K8s-without-K8s") | README framing (Q5=A: Nomad-inspired, honest trade-offs table from doc 3, not Proxmox-first lead) |
+| 4 | Drift detection cadence (R-018/R-019/R-020, tiered cadence, hard gate) | R-018, R-019, R-020, D-227..D-237, REQ-103..REQ-113 |
+| 5 | Drift detection concrete impl (systemd Path units, orca-drift-notify.sh, orca-remediate.sh) | D-227..D-237 detail, REQ-103..REQ-113 |
+
+## Thread A — Ingress hardening (doc 1)
+
+### What changes vs v0.9/v0.10
+
+Traefik static config gains `entryPoints.websecure.address: 127.0.0.1:8443`
+(default) instead of `:443`. A new nftables emitter renders
+`/etc/nftables.d/orca.nft` with DNAT rules. Certs, mTLS, dynamic config,
+and the workload SPIFFE validation path are **completely unchanged**.
+Only the `address` line shifts + one new emitter + `orca doctor nft` +
+`orca nft ...` CLI.
+
+### Defense in depth
+
+Two layers: kernel (nftables: SYN flood, rate limit, GeoIP, conntrack)
+and application (Traefik: mTLS, SNI, ACL, dynamic routing, health
+checks). Neither can replace the other; they catch different attack
+classes.
+
+### Codebase reality (verified 2026-08-07)
+
+- `internal/emitter/traefik.go` + `traefik_atomic.go` exist (v0.9 P02).
+  The static-config emitter is where the `address:` line change lands.
+- `internal/emitter/systemd.go` exists. New `.path`/`.service` unit
+  types extend this emitter pattern (shared with drift detection, doc 5).
+- `internal/emitter/nft.go` does **not** exist — greenfield, ~200 LoC.
+- `scripts/` has `orca-verify-render.sh` but **not** `orca-aggregate.sh`,
+  `orca-pull.sh`, `orca-apply-render.sh`, `orca-remediate.sh` — all are
+  v0.11 P09/P10 scope.
+
+## Thread B — Drift detection + transactional plane (docs 4 + 5)
+
+### Architecture
+
+systemd Path units (R-001-clean; systemd is OS, not Orca) watch critical
+paths via inotify. On change, a oneshot service computes sha256 and
+writes an event JSON to `/etc/orca/state/drift-events/`. The lead's
+aggregator timer (10s, C-11) rsyncs these events, validates against the
+applied txn manifest, and triggers remediation for auto-remediable
+paths.
+
+### Tiered cadence
+
+| Tier | Detection | Auto-remediate | Latency |
+|------|-----------|----------------|--------|
+| Critical | Path unit + 5s polling backstop | yes (config files only) | ~10s |
+| Standard | 30s polling | optional (systemd units: require approval) | 30s |
+| Default | 60s polling | no | 60s |
+
+### R-020 hard gate
+
+Applier refuses new txns if pre-flight consistency check fails. Override:
+`--force` flag + per-namespace scoping (Q4=A) — a drifted peer in ns-A
+does not block ns-B.
+
+### Codebase reality (verified 2026-08-07)
+
+- `internal/store/node_repo.go:80` and `internal/store/job_task_repo.go:82`
+  already use `iter.Seq[T]`. Doc 5's `iter.Seq2[Event, error]` is the
+  natural extension per D-017 (settled, shipped v0.3).
+- `internal/paths/paths.go:86` has `TxnDir()` — the txn staging dir the
+  drift detector hooks into.
+- `internal/emit/contract.go` has the Go↔bash render-contract anti-drift
+  (C-16). The *runtime* drift detector (doc 5) is net-new.
+- `internal/drift/` package does **not** exist — greenfield, ~500 LoC.
+- No `orca` system user creation in code — net-new operational
+  requirement (REQ-111).
+- No NFS detection at peer setup — net-new (REQ-112, D-233).
+- `doctor.go` has an OS-drift *check* (one-shot, on-demand) but **not**
+  a 60s runtime drift-polling loop. Doc 5's design is net-new scope.
+
+### Alignment with existing gates
+
+- **C-09** (`orca-pull.sh` failure contract) — R-020 refines
+  "deterministic state" into an explicit refusal contract.
+- **C-11** (lead-side watchdog meta-timer) — doc 5's aggregator
+  extension is the input C-11 monitors.
+- **REQ-075** (lead applier execution model) — doc 5's `orca-remediate.sh`
+  is literally the same code path as a normal txn-apply, triggered by
+  drift instead of a new submission.
+
+## Thread C — Positioning/messaging (docs 2 + 3)
+
+### Consistent with locked vision
+
+The vision is *"A minimalist, offline-first, CLI-first orchestration
+engine inspired by HashiCorp Nomad"* — explicitly Nomad-inspired, not
+K8s. Doc 3's recommendation ("be Proxmox-for-bare-metal, not
+K8s-without-the-complexity") is consistent with the locked vision.
+
+### Where doc 3 diverges (resolved per Q5=A)
+
+Doc 3 recommends "leading with Proxmox positioning." But R-003 says
+"Proxmox can never be lead." Leading the *project identity* with a node
+type that can't be the lead is subtly contradictory. **Q5=A decision**:
+README uses the Nomad-inspired, OS-as-cluster framing (locked vision),
+mentions Proxmox as one node type, and incorporates doc 3's "honest
+trade-offs" table but not its Proxmox-first lead-positioning advice.
+
+### CLI surface gap analysis (doc 2)
+
+Doc 2's playbook cites ~10 CLI commands. Verified against the live
+codebase (`internal/cli/*.go`):
+
+**Exist today**: `orca init`, `orca node {join,leave,list,key-reset,
+capacity}`, `orca job {run,list,stop,logs}`, `orca ns {list,create,
+delete,inspect,validate}`, `orca cert {ca-init,gen,show,renew,
+fingerprint}`, `orca doctor {cert,network,db,os,proxmox}`, `orca audit
+list`, `orca status`, `orca version`, `orca daemon` (deprecated).
+
+**Not in v0.11 ROADMAP, added per Q2=C**: `orca cluster rotate-lead`
+(REQ-114, P14b), `orca upgrade` (REQ-115, P14a), `orca job migrate`
+(REQ-116, P05), `orca logs --all-nodes --since` (REQ-117, P06),
+`orca doctor mTLS` (REQ-118, P15.5).
+
+**Already in v0.11 ROADMAP**: `orca node drain` (P05), `orca job lint`
+(P11), `orca job verify` (P12), `orca restore` (P07), `orca backup`
+(P04).
+
+### Unverified performance claims in doc 3
+
+Doc 3's "10s applier timer = 10,000x slower than K8s informers" and
+"60s drift polling" are **forward-looking design constraints**, not
+current-state limitations — no applier timer or drift-polling loop
+exists in the codebase. These are answered by R-018/R-019/R-020
+(doc 4 + doc 5): the drift detector is a backstop, not the primary
+detector, and critical paths get ~10s latency via systemd Path units.
+
+## Persona assessment
+
+v0.11 touches these territories:
+
+| Territory | Persona | Phases |
+|-----------|---------|--------|
+| `internal/cli/**`, `internal/drift/**`, `internal/nft/**` | backend-engineer | P10, P15.5, P05, P06, P14a, P14b |
+| `internal/emitter/**`, `internal/sshpush/**` | backend-engineer + lead-developer | P09, P10, P15.5 |
+| `internal/store/**`, migrations | data-engineer | P14a (data migration) |
+| `scripts/orca-*.sh` | backend-engineer (bash tooling) | P09, P10 |
+| `docs/**`, `README.md`, `examples/**` | lead-developer + docs-engineer (phase-specific) | P15, P08 |
+| Threat model, security review, mTLS, secrets | security-engineer | P03, P15.5 |
+| nftables, Traefik binding, cluster mesh | network-engineer | P15.5, P09 |
+| Test coverage, integration harness | devops-engineer (phase-specific) | P08 |
+
+No frontend-engineer work (no UI). The data-engineer persona is
+reactivated for P14a (v0.8→v1.0 data migration). A docs-engineer custom
+persona is created for P15 (README) and P08 (integration test docs).
+See `PERSONAS.md` for the updated roster.