Jon Chery
e45611b416
GRILL stage: adversarial review across 6 forcing questions. Verdict: PROCEED-WITH-CONDITIONS. - C-23: orca-pull.sh distinguishes cluster-wide vs ns-scoped txns (gate P10a) - C-24: split P10 into P10a (txn plane) + P10b (drift detection); phase count 23->24; tags shift by 1 - C-25: orca upgrade post-cutover verification + rollback (gate P14a) - C-26: per-phase LoC soft ceiling ~800 (no gate, recorded) - C-27: orca upgrade creates orca system user on existing peers (gate P14a) - C-28: P15.5 two sub-waves (ingress+doctor nft, then threat model+doctor mTLS) (gate P15.5) ROADMAP updated: 24 phases, tags v0.10.0..v0.10.22. ---ci--- project: orca phase: 0 milestone: v0.11 status: grill ---/ci---
172 lines
9.1 KiB
Markdown
172 lines
9.1 KiB
Markdown
# Grill: v0.11 Production Hardening — Phase 0 Adversarial Review
|
|
|
|
**Status**: PROCEED-WITH-CONDITIONS. The v0.11 plan is sound; 6 binding
|
|
conditions (C-23…C-28) gate specific phases. The plan adopts R-017…R-020
|
|
and D-215…D-237 from 5 research docs with operator decisions Q1=A, Q2=C,
|
|
Q3=A, Q4=A, Q5=A. The grill reviewed the plan adversarially across the
|
|
same 9 axes as GRILL_v0.9 (vision, feasibility, scope, risk, security,
|
|
operational, cost, competitive, exit).
|
|
|
|
## Forcing questions + verdicts
|
|
|
|
### FQ1 — R-020 deadlock with `--force` + per-ns scoping
|
|
|
|
**Question**: With `--force` + per-namespace scoping (Q4=A), can a single
|
|
drifted peer still block a *cluster-wide* txn (e.g., namespace creation)?
|
|
If yes, is the `--force` escape hatch documented in C-09's failure
|
|
contract?
|
|
|
|
**Verdict**: PARTIAL-BLOCK remains for cluster-wide txns. A namespace
|
|
*creation* txn touches all peers (the new namespace dir is created on
|
|
every peer). If one peer is drifted, the pre-flight gate refuses the
|
|
txn cluster-wide. `--force` overrides this, but `--force` on a
|
|
namespace-creation txn is risky (it forces the new namespace onto a
|
|
drifted peer without reconciling the drift first).
|
|
|
|
**Binding condition C-23**: `orca-pull.sh` (C-09) must distinguish
|
|
*cluster-wide* txns from *namespace-scoped* txns. Cluster-wide txns
|
|
require `--force` with an explicit `--i-understand-the-risk` confirmation
|
|
(or `--yes` for non-interactive). Namespace-scoped txns use per-ns
|
|
scoping (drifted peer in ns-A doesn't block ns-B). **Gate**: P10.
|
|
|
|
**Confidence**: 0.88
|
|
|
|
### FQ2 — P10 sizing (txn plane + drift detection in one phase)
|
|
|
|
**Question**: P10 now absorbs drift detection (~500 LoC Go + 150 LoC
|
|
bash + systemd units), the largest single phase. Is this a vertical
|
|
slice that can ship atomically, or does it need splitting (P10a txn
|
|
plane, P10b drift)?
|
|
|
|
**Verdict**: SPLIT RECOMMENDED. P10 has 13 tasks spanning two distinct
|
|
subsystems: (1) the transactional plane (T1-T2: txn bundle render, SCP,
|
|
apply, C-09 failure contract) and (2) drift detection (T3-T13: `internal/drift/`,
|
|
Path unit emitter, notify/remediate scripts, cadence config, pre-flight
|
|
gate, `orca` user, NFS detection, job restart). The txn plane is a
|
|
prerequisite for drift detection (T3's `Aggregate` reads applied txn
|
|
manifests), so the split is clean: P10a (txn plane, T1-T2) ships first,
|
|
P10b (drift detection, T3-T13) ships after P10a.
|
|
|
|
**Binding condition C-24**: Split P10 into P10a (transactional plane,
|
|
REQ-075/079, C-09) and P10b (drift detection, R-018/R-019/R-020,
|
|
REQ-103..113). P10a ships first; P10b depends on P10a. Tags: P10a
|
|
`v0.10.12`, P10b `v0.10.13`. All subsequent phase tags shift by 1
|
|
(P11→`v0.10.14`, …, P16→`v0.10.22`). **Phase count: 23 → 24.**
|
|
|
|
**Confidence**: 0.92
|
|
|
|
### FQ3 — Ingress default migration path (C1)
|
|
|
|
**Question**: Existing v0.9/v0.10 clusters run Traefik on `:443`. R-017
|
|
makes `127.0.0.1:8443` + nft the default. What's the upgrade path? Does
|
|
`orca upgrade` (Q2=C) handle the binding cutover, or is it a manual
|
|
operator step?
|
|
|
|
**Verdict**: UPGRADE HANDLES IT, but with a safety check. `orca upgrade`
|
|
(REQ-115, P14a) is the thin wrapper (C2=a) that handles the Traefik
|
|
binding cutover. The cutover is: (1) emit new Traefik static config with
|
|
`127.0.0.1:8443`, (2) emit `/etc/nftables.d/orca.nft` with DNAT, (3)
|
|
`systemctl reload traefik` + `nft -f`, (4) verify `curl :443` still
|
|
routes. If step 4 fails, rollback to `:443` + remove nft rules.
|
|
|
|
**Binding condition C-25**: `orca upgrade` (REQ-115) must include a
|
|
post-cutover verification step (`curl -k https://localhost:443/` returns
|
|
200 from Traefik) with automatic rollback on failure. Document the
|
|
rollback procedure in `docs/ingress.md`. **Gate**: P14a.
|
|
|
|
**Confidence**: 0.90
|
|
|
|
### FQ4 — Scope ceiling (LoC vs phase count)
|
|
|
|
**Question**: v0.11 stays at 23 phases (now 24 with C-24), but P09/P10
|
|
(now P10a/P10b)/P15.5 grow substantially. Is the *phase count* the right
|
|
ceiling, or should there be a *LoC/effort* ceiling per phase?
|
|
|
|
**Verdict**: LOOSE LoC CEILING. Phase count is a proxy for effort, but
|
|
P10b (drift detection) is ~650 LoC across Go + bash + systemd — at the
|
|
upper end of what a single-phase vertical slice can handle. The grill
|
|
recommends a soft LoC ceiling of ~800 LoC per phase (Go + bash + config),
|
|
with splitting required above ~1200 LoC.
|
|
|
|
**Binding condition C-26**: Per-phase LoC soft ceiling: ~800 LoC (Go +
|
|
bash + config). Split required above ~1200 LoC. P10b (~650 LoC) is within
|
|
the soft ceiling; P15.5 (~400 LoC: nft emitter 200 + doctor mTLS 100 +
|
|
threat model doc) is within. No action required for v0.11; recorded for
|
|
future milestones. **No gate.**
|
|
|
|
**Confidence**: 0.85
|
|
|
|
### FQ5 — `orca` system user on peers (operational impact)
|
|
|
|
**Question**: Creating a system user on every peer is a new operational
|
|
requirement. Does this break any existing v0.9/v0.10 deployment that
|
|
runs as root or as an existing service account?
|
|
|
|
**Verdict**: NO BREAK for existing deployments; NEW requirement for drift
|
|
detection. The `orca` system user (REQ-111) is created at peer setup
|
|
(`orca node join` / peer-setup script). Existing v0.9/v0.10 peers don't
|
|
have the `orca` user, so drift detection's systemd Path units (which run
|
|
as `User=orca`) won't start until the user is created. `orca upgrade`
|
|
(REQ-115) must create the `orca` user on existing peers as part of the
|
|
v0.11 migration.
|
|
|
|
**Binding condition C-27**: `orca upgrade` (REQ-115, P14a) must create
|
|
the `orca` system user on existing peers (`useradd -r orca` idempotent)
|
|
before P10b's drift detection can function. Document this as a
|
|
migration prerequisite. **Gate**: P14a.
|
|
|
|
**Confidence**: 0.91
|
|
|
|
### FQ6 — P15.5 is now a mega-phase (threat model + ingress + doctor mTLS)
|
|
|
|
**Question**: P15.5 was originally "threat model + security review" (C-19).
|
|
It now absorbs ingress hybrid (R-017; REQ-099..102, ~400 LoC) + `orca
|
|
doctor mTLS` (REQ-118). Is this too much for one phase?
|
|
|
|
**Verdict**: MANAGEABLE but at the ceiling. P15.5 is now ~500 LoC (nft
|
|
emitter 200 + doctor mTLS 100 + threat model doc + tests). The ingress
|
|
hybrid and threat model are related (both are security-hardening), so
|
|
keeping them together is defensible. The `orca doctor mTLS` (REQ-118)
|
|
is small and reuses P01/P01.5 infrastructure. The grill recommends
|
|
keeping P15.5 as one phase but splitting the *work* into two sub-waves
|
|
within the phase: (1) ingress hybrid + doctor nft, (2) threat model +
|
|
doctor mTLS.
|
|
|
|
**Binding condition C-28**: P15.5 commits in two sub-waves: (1) ingress
|
|
hybrid (REQ-099..102) + `orca doctor nft` (REQ-101), (2) threat model
|
|
(C-19) + `orca doctor mTLS` (REQ-118). Both ship under the same phase
|
|
tag (`v0.10.20`). **No new phase; internal ordering only.**
|
|
|
|
**Confidence**: 0.89
|
|
|
|
## Binding conditions summary
|
|
|
|
| ID | Condition | Gate | Verification |
|
|
|----|-----------|------|--------------|
|
|
| C-23 | `orca-pull.sh` distinguishes cluster-wide vs namespace-scoped txns; cluster-wide requires `--force` + `--i-understand-the-risk` (or `--yes`) | P10a | Test: cluster-wide txn refused without `--force`; ns-scoped txn blocks only the drifted ns |
|
|
| C-24 | Split P10 into P10a (txn plane, REQ-075/079, C-09) + P10b (drift detection, R-018/R-019/R-020, REQ-103..113); P10b depends on P10a; tags shift by 1 | P10a→P10b | Plan shows P10a + P10b as separate phases; P10b tasks reference P10a txn manifests |
|
|
| C-25 | `orca upgrade` (REQ-115) includes post-cutover verification (`curl -k https://localhost:443/` returns 200) with automatic rollback on failure; rollback documented in `docs/ingress.md` | P14a | Test: cutover succeeds → 200; cutover fails → rollback to `:443` |
|
|
| C-26 | Per-phase LoC soft ceiling: ~800 LoC (Go + bash + config); split required above ~1200 LoC | (no gate) | Recorded for future milestones |
|
|
| C-27 | `orca upgrade` (REQ-115) creates `orca` system user on existing peers before P10b drift detection can function | P14a | Test: existing peer without `orca` user → `orca upgrade` creates it → drift detection starts |
|
|
| C-28 | P15.5 commits in two sub-waves: (1) ingress hybrid + doctor nft, (2) threat model + doctor mTLS; same phase tag | P15.5 | Commits show two sub-waves; both under `v0.10.20` |
|
|
|
|
## Phase challenge summary
|
|
|
|
| PC | Phase | Challenge | Resolution |
|
|
|----|-------|-----------|------------|
|
|
| PC-11 | P10a/P10b | Txn plane + drift detection too large for one phase | Split per C-24; P10a ships first, P10b depends on it |
|
|
| PC-12 | P15.5 | Mega-phase (threat model + ingress + doctor mTLS) | Keep as one phase; two sub-waves per C-28 |
|
|
| PC-13 | P14a | `orca upgrade` handles 3 migrations (data + binding + orca user) | All three land in P14a per C-25, C-27; thin wrapper (C2=a) |
|
|
| PC-14 | P09 | Aggregator extension depends on P10b drift detection | P09 in Wave 6 (after Wave 5 P10b); aggregator extension (REQ-107) only works once drift events exist |
|
|
|
|
## Overall verdict
|
|
|
|
**PROCEED-WITH-CONDITIONS**. The v0.11 plan is sound. 6 binding conditions
|
|
(C-23…C-28) gate specific phases. The plan grows from 23 → 24 phases
|
|
(C-24 splits P10 into P10a/P10b). All other phases are unchanged in
|
|
count; their scope expands per the research folding (Q2=C, Q3=A).
|
|
|
|
The grill's confidence in the v0.11 plan is high (avg 0.89 across FQs).
|
|
The primary risks (P10 sizing, R-020 deadlock, ingress migration) are
|
|
all gated with verifiable conditions.
|