docs(P00): grill + plan revision — v0.14 binding conditions

Grill verdict: RETHINK (0.45) → revised plan addresses all 12
binding conditions (C-50..C-61):
- C-50: install podman if absent (linux/lead)
- C-51: DNATTarget validation (nft injection guard)
- C-53: apt-get idempotency (command -v podman check)
- C-54: offline-first tension documented (podman pull exception)
- C-55: native-mode nft single-apply (discover LXC IP first)
- C-56: MAC collision check against registry
- C-57: v0.13→v0.14 upgrade path (remove legacy systemd+binary)
- C-58: mount static config from host (preserve REQ-100 opt-out)
- C-59: migration 0009 (not 0007)
- C-60: certpaths.CACertPath() (not CAPath())
- C-61: --restart=unless-stopped, omit :Z
- C-62/G-003: mTLS deferred to v0.15 (confidence 0.55 < 0.60)

---ci---
project: orca
phase: 0
milestone: v0.14
status: grill
---/ci---
This commit is contained in:
Jon Chery
2026-08-10 18:26:33 +00:00
parent 65e50e465b
commit 3551b37ac0
4 changed files with 49 additions and 25 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
{
"phase": 0,
"stage": "plan",
"stage": "grill",
"milestone": "v0.14",
"milestone_slug": "ingress-bootstrap",
"phase_role": "pre_execution",
+41 -19
View File
@@ -5,17 +5,27 @@ patch tag on the v0.13.x line. This plan references requirement IDs from
REQUIREMENTS.md and follows the vertical-slice integrity rule (each phase
is independently shippable).
**Research-validated decisions** (from RESEARCH_v0.14.md):
**Research-validated decisions** (from RESEARCH_v0.14.md + GRILL_v0.14.md):
- nft postrouting: `ip saddr 127.0.0.0/8 oifname != "lo" masquerade`
- nft first-apply: pre-create table (`nft add table inet orca-ingress 2>/dev/null || true`) before `nft -f`
- pve-firewall: shift orca input/forward chains to `priority -10` (before pve-firewall's 0)
- LXC features: `nesting=1,keyctl=1,fuse=1` (fuse=1 for fuse-overlayfs)
- traefik TLS: **drop `certResolver: orca`** — does not exist in v3.3; use dynamic `tls.certificates` + `tls.options.default.clientAuth.caFiles` instead
- traefik TLS: **drop `certResolver: orca`** — does not exist in v3.3; emit `tls: {}` for v0.14 (real mTLS via dynamic `tls.certificates` + `clientAuth.caFiles` deferred to v0.15 — grill G-003 confidence 0.55 < 0.60 threshold, auto-resolved to defer)
- podman restart: `--restart=unless-stopped` + enable `podman-restart.service`
- volumes: omit `:Z` flag, use `:ro` on both mounts
- traefik image: `FROM traefik:v3.3.0`, `ENTRYPOINT ["/traefik"]` inherited, `CMD ["--configFile=/etc/traefik/traefik.yml"]`
- `NftClusterConfig.DNATTarget`: default `127.0.0.1:8443`/`:8080`; proxmox-native = `<lxc-ip>:8443`/`:8080`
- Migration: `0009_ingress_mode.sql` (NOT 0007 — already taken by certs_serial_unique)
- CA path: `certpaths.CACertPath()` (NOT `certpaths.CAPath()` — does not exist)
- Upgrade path: P2 must detect+remove legacy `orca-traefik.service` + `/usr/local/bin/traefik` before starting podman container (C-57)
- Podman install: `BootstrapLocalIngress` and `BootstrapRemoteIngress` must install podman if absent (C-50)
- Offline-first: `podman pull` requires registry reachability — documented exception to R-001 for ingress bootstrap (C-54)
- Static config: mount from host (not baked) to preserve `traefik-on-public-ip` opt-out (C-58)
- DNATTarget validation: `net.ParseIP` or `ip:port` parse before render (C-51)
- apt-get idempotency: `command -v podman` check before install (C-53)
- MAC collision: check against existing nodes' MACs (C-56)
- Native-mode nft: first apply uses LXC IP (not default 127.0.0.1) — discover LXC IP before first nft apply (C-55)
## Phase 0: Pre-execution (this phase)
@@ -27,7 +37,7 @@ Ships as `v0.13.0`.
**Tag**: `v0.13.1` | **Type**: feat | **Persona**: release-engineer (phase-specific) + backend-engineer
### Wave 1 (image)
- **T1**: Create `docker/orca-traefik/traefik.yml` — the baked static config:
- **T1**: Create `docker/orca-traefik/traefik.yml` — the **default** static config baked into the image (used when no host-side override is mounted):
```yaml
entryPoints:
websecure:
@@ -47,6 +57,7 @@ Ships as `v0.13.0`.
format: json
```
No `certificatesResolvers` (research finding: does not exist for CA-based; TLS is via dynamic config).
**C-58**: The baked config is a default. The podman run command also mounts a host-side `/etc/traefik/traefik.yml` if it exists (overriding the baked one), preserving the `traefik-on-public-ip` opt-out (REQ-100). The reconciler renders the static config via `emitter.RenderTraefikStaticConfig` to `/etc/traefik/traefik.yml` on the host, then mounts it `-v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro`. This way `PublicBinding` opt-out still works.
- **T2**: Create `Dockerfile.traefik` at repo root:
```dockerfile
FROM traefik:v3.3.0
@@ -96,12 +107,20 @@ Ships as `v0.13.0`.
- Image/tag resolution: `git.cloudinit.dev/coreci/orca-traefik:<version>` where version = `internal/cli.version` (or `latest` if dev)
- **Remove** systemd unit generation + `systemctl enable`
- **T2**: Add `podman-restart.service` enable step: `systemctl enable --now podman-restart.service` (research finding: needed for reboot persistence)
- **T2a**: **C-50**: `EnsureTraefikContainerLocal`/`Remote` must check `command -v podman` first. If absent: on localhost, attempt `apt-get install -y podman` (or fail with clear install instructions if no apt). On remote, `apt-get install -y podman conmon crun fuse-overlayfs` via SSH. Non-fatal warn if podman unavailable (offline host) — traefik won't start but `orca init` succeeds (same tolerance as v0.13).
### Wave 2 (callsite updates)
### Wave 2 (callsite updates + v0.13 upgrade path)
- **T3**: `internal/cli/init.go:254-266` — replace `installTraefikLocal()` with `EnsureTraefikContainerLocal`
- **T4**: `internal/linux/bootstrap.go:160-172` — replace `traefik.InstallRemote` with `EnsureTraefikContainerRemote`
- **T5**: `internal/proxmox/bootstrap.go:250-255` — replace `traefik.InstallRemote` with `EnsureTraefikContainerRemote` (for native mode; floating-IP calls it inside the LXC in P6)
- **T6**: `internal/cli/upgrade.go` — update legacy cutover to pull new image + recreate container instead of sed-ing traefik.yml
- **T6**: **C-57 (v0.13→v0.14 upgrade path)**: `internal/cli/upgrade.go` — rewrite the Traefik cutover to:
1. Detect legacy `orca-traefik.service`: `systemctl is-active orca-traefik.service`
2. If active: `systemctl stop orca-traefik.service && systemctl disable orca-traefik.service`
3. Remove `/etc/systemd/system/orca-traefik.service` + `/usr/local/bin/traefik` (if exists)
4. `systemctl daemon-reload`
5. Render static config via `emitter.RenderTraefikStaticConfig` to `/etc/traefik/traefik.yml`
6. `EnsureTraefikContainerLocal` (pull + run podman container)
7. Idempotent: if no legacy unit, skip steps 1-4
- **T7**: `internal/cli/traefik_install.go` — update CLI wrapper
### Wave 3 (TLS model fix — research finding)
@@ -109,7 +128,8 @@ Ships as `v0.13.0`.
- **T9**: Update `internal/emitter/traefik_test.go` — remove assertion for `certResolver: orca`, add assertion for `tls: {}` presence.
### Wave 4 (tests)
- **T10**: `internal/traefik/install_test.go` — rewrite tests to assert `podman run` is invoked (not `curl|tar`), `--restart=unless-stopped --network host` present, volume mounts present, `podman-restart.service` enabled.
- **T10**: Create `internal/traefik/install_test.go` (new file — F1.3: does not exist today) — assert `podman run` is invoked (not `curl|tar`), `--restart=unless-stopped --network host` present, volume mounts present, `podman-restart.service` enabled.
- **T10a**: **C-57/F6.2**: Add v0.13→v0.14 upgrade test: simulate a host with `orca-traefik.service` present (fake), run upgrade, assert unit stopped+disabled+removed, podman container running.
### Must-haves
- [ ] `orca init` → `podman inspect orca-traefik` shows running
@@ -123,7 +143,7 @@ Ships as `v0.13.0`.
**Tag**: `v0.13.3` | **Type**: feat | **Persona**: lead-developer + security-engineer
### Wave 1 (nft emitter extension)
- **T1**: `internal/emitter/nft.go` — add `DNATTarget` field to `NftClusterConfig` (default `127.0.0.1`). Render DNAT rules as `dnat to <DNATTarget>:8443` / `dnat to <DNATTarget>:8080`.
- **T1**: `internal/emitter/nft.go` — add `DNATTarget` field to `NftClusterConfig` (default `127.0.0.1`). Render DNAT rules as `dnat to <DNATTarget>:8443` / `dnat to <DNATTarget>:8080`. **C-51**: Validate `DNATTarget` with `net.ParseIP` before rendering. Reject invalid values with error (same F9 injection guard pattern as `partitionTrustedProbes`).
- **T2**: `internal/emitter/nft.go` — add `EnableSNAT bool` (default true) + `postrouting` chain:
```nft
chain postrouting {
@@ -139,11 +159,13 @@ Ships as `v0.13.0`.
- **T5**: New `internal/ingress/bootstrap.go`:
- `BootstrapLocalIngress(ctx)`:
1. `mkdir -p /etc/traefik/dynamic /etc/orca`
2. Push cluster root CA to `/etc/orca/step-ca-root.crt` (from `certpaths.CAPath()` if exists, else empty placeholder)
3. Render `orca.nft` via `NftEmitter.RenderNftConfig` + write to `/etc/nftables.d/orca.nft`
4. Pre-create table: `nft add table inet orca-ingress 2>/dev/null || true`
5. Apply: `nft -f /etc/nftables.d/orca.nft`
6. `EnsureTraefikContainerLocal` (from P2)
2. **C-60**: Push cluster root CA to `/etc/orca/step-ca-root.crt` from `certpaths.CACertPath()` (if exists, else empty placeholder)
3. Render static config via `emitter.RenderTraefikStaticConfig` to `/etc/traefik/traefik.yml` (preserves `traefik-on-public-ip` opt-out — C-58)
4. Render `orca.nft` via `NftEmitter.RenderNftConfig` + write to `/etc/nftables.d/orca.nft`
5. Pre-create table: `nft add table inet orca-ingress 2>/dev/null || true`
6. Apply: `nft -f /etc/nftables.d/orca.nft`
7. **C-50**: Ensure podman installed (check `command -v podman`, install if absent)
8. `EnsureTraefikContainerLocal` (from P2) — mounts `/etc/traefik/traefik.yml:ro` + `/etc/traefik/dynamic:ro` + `/etc/orca/step-ca-root.crt:ro`
- Each step non-fatal warn (offline host tolerance)
- **T6**: Wire into `internal/cli/init.go` after `EnsureTraefikContainerLocal` (Step 4e, replacing the old traefik install step).
@@ -168,7 +190,7 @@ Ships as `v0.13.0`.
### Wave 1 (remote ingress)
- **T1**: `internal/ingress/bootstrap.go` — add `BootstrapRemoteIngress(ctx, execFn)`:
1. `mkdir -p /etc/traefik/dynamic /etc/orca` (remote)
2. Push step-ca root CA to remote `/etc/orca/step-ca-root.crt` via `WriteFile`
2. **C-60**: Push step-ca root CA to remote `/etc/orca/step-ca-root.crt` from `certpaths.CACertPath()` via `WriteFile`
3. Render `orca.nft` + write to remote `/etc/nftables.d/orca.nft` via `WriteFile`
4. `nft add table inet orca-ingress 2>/dev/null || true` (remote)
5. `nft -f /etc/nftables.d/orca.nft` (remote)
@@ -191,20 +213,20 @@ Ships as `v0.13.0`.
### Wave 1 (flags + schema)
- **T1**: Add flags to `node join`: `--ingress-mode` (values: `native` default, `floating-ip`), `--floating-ip`, `--gateway`, `--mac`, `--net-prefix` (default `24`).
- **T2**: Add `IngressMode` field to `model.Node` (string: `""`, `"native"`, `"floating-ip"`).
- **T3**: Schema migration 0007: `ALTER TABLE nodes ADD COLUMN ingress_mode TEXT DEFAULT ''`.
- **T3**: **C-59**: Schema migration `0009_ingress_mode.sql` (NOT 0007 — already taken): `ALTER TABLE nodes ADD COLUMN ingress_mode TEXT DEFAULT '';`
### Wave 2 (native mode bootstrap)
- **T4**: In `proxmox.BootstrapProxmox`, when `IngressMode == "native"`:
1. On the PVE host: render+apply nft with `DNATTarget = <lxc-bridge-ip>` (the traefik LXC's IP, discovered after `pct start`)
2. Create unprivileged LXC with `--features nesting=1,keyctl=1,fuse=1` (research finding: fuse=1 for fuse-overlayfs). `pct create <vmid> local:vztmpl/<template> --hostname orca-traefik --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --onboot 1 --memory 2048 --swap 0 --rootfs local:8`
3. `pct start <vmid>`
4. Inside the LXC: `apt-get update && apt-get install -y podman conmon crun fuse-overlayfs nftables`
3a. **C-55**: Discover LXC IP via `pct config <vmid>` (parse `net0` line for `ip=`) or `pct exec <vmid> -- hostname -I` BEFORE the first nft apply. The nft DNAT target is set to the LXC IP from the start — no two-phase apply, no downtime window.
4. Inside the LXC: **C-53**: `command -v podman >/dev/null 2>&1 || (apt-get update && apt-get install -y podman conmon crun fuse-overlayfs nftables)` — idempotent, skip if podman already installed
5. Configure podman storage (`/etc/containers/storage.conf`): `mount_program = "/usr/bin/fuse-overlayfs"` (fallback: `driver = "vfs"`)
6. `systemctl enable --now podman-restart.service` (inside LXC)
7. Push step-ca root CA into LXC
8. `EnsureTraefikContainerRemote` (podman pull + run inside LXC with `--network host`)
9. Discover LXC IP via `pct config <vmid>` (parse `net0` line) or `pct exec <vmid> -- hostname -I`
10. Update PVE-host nft DNAT target to `<lxc-ip>:8443`/`:8080` and re-apply
9. **C-55**: Render+apply nft on PVE host with `DNATTarget = <lxc-ip>` (discovered in step 3a) — single apply, no downtime window
### Wave 3 (registration)
- **T5**: Register PVE host as `proxmox` node with `IngressMode: "native"`.
@@ -230,7 +252,7 @@ Ships as `v0.13.0`.
3. `pct create <vmid> local:vztmpl/<template> --hostname ingress --unprivileged 1 --features nesting=1,keyctl=1,fuse=1 --net0 name=eth0,bridge=vmbr0,hwaddr=<mac>,ip=<floating-ip>/<prefix>,gw=<gateway> --onboot 1 --memory 2048 --swap 0 --rootfs local:8`
4. `pct start <vmid>`
5. Wait for LXC network (retry SSH to `<floating-ip>` with backoff, 60s timeout)
6. Inside the LXC: `apt-get update && apt-get install -y podman conmon crun fuse-overlayfs nftables`
6. Inside the LXC: **C-53**: `command -v podman >/dev/null 2>&1 || (apt-get update && apt-get install -y podman conmon crun fuse-overlayfs nftables)` — idempotent
7. Configure podman storage (fuse-overlayfs / vfs fallback)
8. `systemctl enable --now podman-restart.service`
9. Push step-ca root CA into LXC
@@ -245,7 +267,7 @@ Ships as `v0.13.0`.
### Wave 3 (interactive prompting)
- **T4**: Interactive prompting in `joinProxmox` (node.go): when `--ingress-mode` empty and `!jsonOutput`:
- Prompt "Ingress mode [native/floating-ip] (default native): "
- If `floating-ip`: prompt for floating IP (validate `net.ParseIP`), gateway (validate `net.ParseIP`), MAC (validate `net.ParseMAC`; generate `02:XX:XX:XX:XX:XX` random if empty + confirm), net-prefix (default 24, validate 8-32)
- If `floating-ip`: prompt for floating IP (validate `net.ParseIP`), gateway (validate `net.ParseIP`), MAC (validate `net.ParseMAC`; generate `02:XX:XX:XX:XX:XX` random if empty + confirm; **C-56**: check MAC against existing nodes' MACs in cluster registry, regenerate on collision), net-prefix (default 24, validate 8-32)
- In `--json` mode: require `--mac` explicitly if `--ingress-mode floating-ip` (no silent generation)
### Wave 4 (routing)
+1 -1
View File
@@ -745,7 +745,7 @@ release). 9 phases, 9 net-new requirements (REQ-171..REQ-179).
|----|----------|----------|-----------|------------|
| D-255 | Traefik deployment model? | **Podman container from custom `orca-traefik` image** | Operator constraint: traefik always deployed as a container. Replaces v0.13 binary+systemd. Image bakes static config. | 0.95 |
| D-256 | Container network mode? | **`--network host`** | Binds 127.0.0.1:8080/8443 directly on host/LXC loopback; nft DNAT targets that. No port publishing complexity. | 0.92 |
| D-257 | TLS cert resolver in image? | **Mounted step-ca root CA, no ACME** | Offline-first (R-001), no external CA. `certificatesResolvers.orca.tls` references `/etc/orca/step-ca-root.crt` (volume-mounted). Graceful degradation if absent. | 0.90 |
| D-257 | TLS cert resolver in image? | **No certResolver; `tls: {}` for v0.14, real mTLS deferred to v0.15** | Grill G-003 (confidence 0.55 < 0.60) auto-resolved to defer. Traefik v3.3 `certificatesResolvers` only supports acme/tailscale, not CA-file. Drop `certResolver: orca` (broken). Emit `tls: {}` in dynamic config. Real mTLS via dynamic `tls.certificates` + `tls.options.default.clientAuth.caFiles` lands in v0.15 when step-ca mints server certs. | 0.90 |
| D-258 | Dynamic config volume? | **Mount `/etc/traefik/dynamic` from host** | Zero changes to existing `deployRemote` WriteFile path (`job_dispatch.go:243`). File provider watches it. | 0.95 |
| D-259 | Floating-IP mode: register PVE host too? | **Yes — PVE host as `proxmox` + ingress LXC as `linux`** | PVE host needed in registry for `pct`/`qm` workload dispatch. Both register. | 0.92 |
| D-260 | `--ingress-mode` persistence? | **Store `IngressMode` on `model.Node`** | `doctor ingress` needs to know which check path to run. Schema migration. | 0.90 |
+6 -4
View File
@@ -394,11 +394,13 @@ node type.
the `orca-traefik` image published per release. Every orca-managed ingress
surface bootstraps: nft DNAT (`:443→127.0.0.1:8443`,
`:80→127.0.0.1:8080`) + SNAT/MASQUERADE postrouting + `podman run -d
--restart=always --network host -v /etc/traefik/dynamic:/etc/traefik/dynamic:Z
--restart=unless-stopped --network host -v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro
-v /etc/traefik/dynamic:/etc/traefik/dynamic:ro
-v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro
git.cloudinit.dev/coreci/orca-traefik:<tag>`. No node joins without a
functional podman-traefik ingress data plane. The image's baked static
config is canonical; host-side `traefik.yml` is no longer written.
config is a default; host-side `traefik.yml` mounted `:ro` overrides it
(preserves `traefik-on-public-ip` opt-out, REQ-100).
**Three topologies** (per operator constraints):
@@ -409,8 +411,8 @@ node type.
| ID | Requirement | Priority | Phase | Status |
|----|-------------|----------|-------|--------|
| REQ-171 | `Dockerfile.traefik` + release pipeline: build + publish `git.cloudinit.dev/coreci/orca-traefik:<version>` alongside the orca image per release; `.coreci.yml` `container-publish-traefik` step; image bakes static `traefik.yml` (entrypoints `websecure` 127.0.0.1:8443, `web` 127.0.0.1:8080, `traefik` 127.0.0.1:8081 + file provider watching `/etc/traefik/dynamic` + `certificatesResolvers.orca.tls` referencing mounted step-ca root CA at `/etc/orca/step-ca-root.crt` + json log/accessLog); graceful degradation if CA file absent | Critical | **v0.14 P1** | pending |
| REQ-172 | Replace `internal/traefik/install.go` binary+systemd install with a podman-container reconciler: `podman pull orca-traefik:<tag>` + `podman run -d --restart=always --network host --name orca-traefik -v /etc/traefik/dynamic:/etc/traefik/dynamic:Z -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro <image>`; idempotent (pull+run if absent, start if stopped); works locally + over SSH-push; update all callsites (`init.go`, `linux/bootstrap.go`, `proxmox/bootstrap.go`, `upgrade.go`); remove systemd unit generation | Critical | **v0.14 P2** | pending |
| REQ-171 | `Dockerfile.traefik` + release pipeline: build + publish `git.cloudinit.dev/coreci/orca-traefik:<version>` alongside the orca image per release; `.coreci.yml` `container-publish-traefik` step; image bakes default `traefik.yml` (entrypoints `websecure` 127.0.0.1:8443, `web` 127.0.0.1:8080, `traefik` 127.0.0.1:8081 + file provider watching `/etc/traefik/dynamic` + json log/accessLog); host-side `/etc/traefik/traefik.yml` mounted `:ro` overrides baked config (preserves `traefik-on-public-ip` opt-out REQ-100); no `certificatesResolvers` (traefik v3.3 only supports acme/tailscale); `tls: {}` in dynamic config for v0.14 (real mTLS deferred to v0.15) | Critical | **v0.14 P1** | pending |
| REQ-172 | Replace `internal/traefik/install.go` binary+systemd install with a podman-container reconciler: `podman pull orca-traefik:<tag>` + `podman run -d --restart=unless-stopped --network host --name orca-traefik -v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro -v /etc/traefik/dynamic:/etc/traefik/dynamic:ro -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro <image>`; idempotent (pull+run if absent, start if stopped); install podman if absent (C-50); v0.13→v0.14 upgrade: detect+stop+disable+remove legacy `orca-traefik.service` + `/usr/local/bin/traefik` (C-57); works locally + over SSH-push; enable `podman-restart.service`; remove systemd unit generation | Critical | **v0.14 P2** | pending |
| REQ-173 | nft SNAT+DNAT ruleset render+apply: extend `internal/emitter/nft.go` with postrouting masquerade chain; new `internal/ingress/bootstrap.go` renders `orca.nft` + applies `nft -f` + ensures `/etc/traefik/dynamic` dir + pushes step-ca root CA + invokes podman traefik reconciler; wired into `orca init` (localhost lead) | Critical | **v0.14 P3** | pending |
| REQ-174 | Remote ingress bootstrap via SSH-push for `orca node join --type linux`: push step-ca root CA, render+apply nft remotely, invoke podman traefik reconciler remotely; register node as `linux` | Critical | **v0.14 P4** | pending |
| REQ-175 | Proxmox native ingress mode (`--ingress-mode native`, default): on PVE host, render+apply nft (vmbr-compatible, separate `orca-ingress` table avoids pve-firewall conflict); create unprivileged LXC with `--features nesting=1,keyctl=1` running podman+orca-traefik; push step-ca root CA into LXC; nft DNAT target = LXC bridge IP; register PVE host as `proxmox` node; add `IngressMode` field to `model.Node` + schema migration | Critical | **v0.14 P5** | pending |