# PLAN v0.14: Ingress Bootstrap Completeness **Status**: active. 9 phases (P0 + P1..P7 + P8 final). Each phase ships a patch tag on the v0.13.x line. This plan references requirement IDs from REQUIREMENTS.md and follows the vertical-slice integrity rule (each phase is independently shippable). **Research-validated decisions** (from RESEARCH_v0.14.md + GRILL_v0.14.md): - nft postrouting: `ip saddr 127.0.0.0/8 oifname != "lo" masquerade` - nft first-apply: pre-create table (`nft add table inet orca-ingress 2>/dev/null || true`) before `nft -f` - pve-firewall: shift orca input/forward chains to `priority -10` (before pve-firewall's 0) - LXC features: `nesting=1,keyctl=1,fuse=1` (fuse=1 for fuse-overlayfs) - traefik TLS: **drop `certResolver: orca`** — does not exist in v3.3; emit `tls: {}` for v0.14 (real mTLS via dynamic `tls.certificates` + `clientAuth.caFiles` deferred to v0.15 — grill G-003 confidence 0.55 < 0.60 threshold, auto-resolved to defer) - podman restart: `--restart=unless-stopped` + enable `podman-restart.service` - volumes: omit `:Z` flag, use `:ro` on both mounts - traefik image: `FROM traefik:v3.3.0`, `ENTRYPOINT ["/traefik"]` inherited, `CMD ["--configFile=/etc/traefik/traefik.yml"]` - `NftClusterConfig.DNATTarget`: default `127.0.0.1:8443`/`:8080`; proxmox-native = `:8443`/`:8080` - Migration: `0009_ingress_mode.sql` (NOT 0007 — already taken by certs_serial_unique) - CA path: `certpaths.CACertPath()` (NOT `certpaths.CAPath()` — does not exist) - Upgrade path: P2 must detect+remove legacy `orca-traefik.service` + `/usr/local/bin/traefik` before starting podman container (C-57) - Podman install: `BootstrapLocalIngress` and `BootstrapRemoteIngress` must install podman if absent (C-50) - Offline-first: `podman pull` requires registry reachability — documented exception to R-001 for ingress bootstrap (C-54) - Static config: mount from host (not baked) to preserve `traefik-on-public-ip` opt-out (C-58) - DNATTarget validation: `net.ParseIP` or `ip:port` parse before render (C-51) - apt-get idempotency: `command -v podman` check before install (C-53) - MAC collision: check against existing nodes' MACs (C-56) - Native-mode nft: first apply uses LXC IP (not default 127.0.0.1) — discover LXC IP before first nft apply (C-55) ## Phase 0: Pre-execution (this phase) **Status**: complete. SPECIFY → CLARIFY → RESEARCH → PLAN → GRILL → SHIP. Ships as `v0.13.0`. ## Phase 1: `orca-traefik` container image + release pipeline (REQ-171) **Tag**: `v0.13.1` | **Type**: feat | **Persona**: release-engineer (phase-specific) + backend-engineer ### Wave 1 (image) - **T1**: Create `docker/orca-traefik/traefik.yml` — the **default** static config baked into the image (used when no host-side override is mounted): ```yaml entryPoints: websecure: address: "127.0.0.1:8443" web: address: "127.0.0.1:8080" traefik: address: "127.0.0.1:8081" providers: file: directory: "/etc/traefik/dynamic" watch: true log: level: INFO format: json accessLog: format: json ``` No `certificatesResolvers` (research finding: does not exist for CA-based; TLS is via dynamic config). **C-58**: The baked config is a default. The podman run command also mounts a host-side `/etc/traefik/traefik.yml` if it exists (overriding the baked one), preserving the `traefik-on-public-ip` opt-out (REQ-100). The reconciler renders the static config via `emitter.RenderTraefikStaticConfig` to `/etc/traefik/traefik.yml` on the host, then mounts it `-v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro`. This way `PublicBinding` opt-out still works. - **T2**: Create `Dockerfile.traefik` at repo root: ```dockerfile FROM traefik:v3.3.0 LABEL org.opencontainers.image.title="orca-traefik" LABEL org.opencontainers.image.source="https://git.cloudinit.dev/coreci/orca" COPY docker/orca-traefik/traefik.yml /etc/traefik/traefik.yml CMD ["--configFile=/etc/traefik/traefik.yml"] ``` (ENTRYPOINT inherited as `["/traefik"]` from base image.) - **T3**: Create placeholder `docker/orca-traefik/step-ca-root.crt` (empty file) — real CA is volume-mounted at runtime. If absent, traefik starts without TLS termination (graceful). ### Wave 2 (release pipeline) - **T4**: `scripts/release.sh` — add a second docker block after the existing one (~line 212): ```bash # Build + push orca-traefik image TRAEFIK_IMAGE="${CONTAINER_REGISTRY}/${CONTAINER_OWNER}/orca-traefik" if command -v docker >/dev/null 2>&1; then docker build -f Dockerfile.traefik -t "${TRAEFIK_IMAGE}:${VERSION}" -t "${TRAEFIK_IMAGE}:latest" . docker push "${TRAEFIK_IMAGE}:${VERSION}" docker push "${TRAEFIK_IMAGE}:latest" fi ``` - **T5**: `.coreci.yml` — add `container-publish-traefik` step mirroring `container-publish` with `CONTAINER_IMAGE=orca-traefik` + `DOCKERFILE=Dockerfile.traefik`. ### Wave 3 (tests) - **T6**: Verify `docker build -f Dockerfile.traefik .` succeeds and the resulting image starts traefik with `--configFile=/etc/traefik/traefik.yml` (can test with `docker run --rm orca-traefik --version`). ### Must-haves - [ ] `Dockerfile.traefik` builds successfully - [ ] Image starts traefik with the baked static config - [ ] `release.sh` publishes `orca-traefik:` + `:latest` - [ ] `.coreci.yml` has `container-publish-traefik` step ## Phase 2: Podman traefik reconciler (REQ-172) **Tag**: `v0.13.2` | **Type**: feat | **Persona**: lead-developer ### Wave 1 (reconciler) - **T1**: Rewrite `internal/traefik/install.go` — replace binary+systemd install with podman container reconciler: - `EnsureTraefikContainer(ctx, execFn, image, tag)` — idempotent: 1. `podman inspect orca-traefik` → if running, no-op; if stopped, `podman start orca-traefik`; if absent, go to step 2 2. `mkdir -p /etc/traefik/dynamic /etc/orca` 3. `podman pull :` 4. `podman run -d --name orca-traefik --restart=unless-stopped --network host -v /etc/traefik/dynamic:/etc/traefik/dynamic:ro -v /etc/orca/step-ca-root.crt:/etc/orca/step-ca-root.crt:ro :` - `EnsureTraefikContainerLocal(ctx, image, tag)` — uses `exec.CommandContext("podman", ...)` locally - `EnsureTraefikContainerRemote(ctx, execFn, image, tag)` — uses SSH exec function - Image/tag resolution: `git.cloudinit.dev/coreci/orca-traefik:` where version = `internal/cli.version` (or `latest` if dev) - **Remove** systemd unit generation + `systemctl enable` - **T2**: Add `podman-restart.service` enable step: `systemctl enable --now podman-restart.service` (research finding: needed for reboot persistence) - **T2a**: **C-50**: `EnsureTraefikContainerLocal`/`Remote` must check `command -v podman` first. If absent: on localhost, attempt `apt-get install -y podman` (or fail with clear install instructions if no apt). On remote, `apt-get install -y podman conmon crun fuse-overlayfs` via SSH. Non-fatal warn if podman unavailable (offline host) — traefik won't start but `orca init` succeeds (same tolerance as v0.13). ### Wave 2 (callsite updates + v0.13 upgrade path) - **T3**: `internal/cli/init.go:254-266` — replace `installTraefikLocal()` with `EnsureTraefikContainerLocal` - **T4**: `internal/linux/bootstrap.go:160-172` — replace `traefik.InstallRemote` with `EnsureTraefikContainerRemote` - **T5**: `internal/proxmox/bootstrap.go:250-255` — replace `traefik.InstallRemote` with `EnsureTraefikContainerRemote` (for native mode; floating-IP calls it inside the LXC in P6) - **T6**: **C-57 (v0.13→v0.14 upgrade path)**: `internal/cli/upgrade.go` — rewrite the Traefik cutover to: 1. Detect legacy `orca-traefik.service`: `systemctl is-active orca-traefik.service` 2. If active: `systemctl stop orca-traefik.service && systemctl disable orca-traefik.service` 3. Remove `/etc/systemd/system/orca-traefik.service` + `/usr/local/bin/traefik` (if exists) 4. `systemctl daemon-reload` 5. Render static config via `emitter.RenderTraefikStaticConfig` to `/etc/traefik/traefik.yml` 6. `EnsureTraefikContainerLocal` (pull + run podman container) 7. Idempotent: if no legacy unit, skip steps 1-4 - **T7**: `internal/cli/traefik_install.go` — update CLI wrapper ### Wave 3 (TLS model fix — research finding) - **T8**: `internal/emitter/traefik.go` — drop `certResolver: orca` from the dynamic config router TLS stanza (line ~185-188). Replace with `tls: {}` (empty TLS stanza — traefik uses its default cert). Document that real mTLS via `tls.certificates` + `tls.options.default.clientAuth.caFiles` will be wired when step-ca mints certs into the dynamic dir (post-v0.14 or v1.x). - **T9**: Update `internal/emitter/traefik_test.go` — remove assertion for `certResolver: orca`, add assertion for `tls: {}` presence. ### Wave 4 (tests) - **T10**: Create `internal/traefik/install_test.go` (new file — F1.3: does not exist today) — assert `podman run` is invoked (not `curl|tar`), `--restart=unless-stopped --network host` present, volume mounts present, `podman-restart.service` enabled. - **T10a**: **C-57/F6.2**: Add v0.13→v0.14 upgrade test: simulate a host with `orca-traefik.service` present (fake), run upgrade, assert unit stopped+disabled+removed, podman container running. ### Must-haves - [ ] `orca init` → `podman inspect orca-traefik` shows running - [ ] `podman logs orca-traefik` shows traefik started with baked config - [ ] No systemd `orca-traefik.service` generated - [ ] `--restart=unless-stopped` + `podman-restart.service` enabled - [ ] `certResolver: orca` removed from dynamic config ## Phase 3: nft SNAT+DNAT + `orca init` ingress bootstrap (REQ-173) **Tag**: `v0.13.3` | **Type**: feat | **Persona**: lead-developer + security-engineer ### Wave 1 (nft emitter extension) - **T1**: `internal/emitter/nft.go` — add `DNATTarget` field to `NftClusterConfig` (default `127.0.0.1`). Render DNAT rules as `dnat to :8443` / `dnat to :8080`. **C-51**: Validate `DNATTarget` with `net.ParseIP` before rendering. Reject invalid values with error (same F9 injection guard pattern as `partitionTrustedProbes`). - **T2**: `internal/emitter/nft.go` — add `EnableSNAT bool` (default true) + `postrouting` chain: ```nft chain postrouting { type nat hook postrouting priority 100; policy accept; ip saddr 127.0.0.0/8 oifname != "lo" masquerade } ``` Only when `EnableSNAT == true`. - **T3**: `internal/emitter/nft.go` — shift `input` and `forward` chain priorities from `filter` (=0) to `-10` (research finding: avoids pve-firewall same-priority undefined order). - **T4**: `internal/emitter/nft.go` — fix first-apply flush-table bug: change `flush table inet orca-ingress` to `delete table inet orca-ingress` (nft ≥1.0 treats delete-of-missing as warning in `-f` mode). If that's version-unsafe, the apply step (T7) pre-creates the table. ### Wave 2 (ingress bootstrap) - **T5**: New `internal/ingress/bootstrap.go`: - `BootstrapLocalIngress(ctx)`: 1. `mkdir -p /etc/traefik/dynamic /etc/orca` 2. **C-60**: Push cluster root CA to `/etc/orca/step-ca-root.crt` from `certpaths.CACertPath()` (if exists, else empty placeholder) 3. Render static config via `emitter.RenderTraefikStaticConfig` to `/etc/traefik/traefik.yml` (preserves `traefik-on-public-ip` opt-out — C-58) 4. Render `orca.nft` via `NftEmitter.RenderNftConfig` + write to `/etc/nftables.d/orca.nft` 5. Pre-create table: `nft add table inet orca-ingress 2>/dev/null || true` 6. Apply: `nft -f /etc/nftables.d/orca.nft` 7. **C-50**: Ensure podman installed (check `command -v podman`, install if absent) 8. `EnsureTraefikContainerLocal` (from P2) — mounts `/etc/traefik/traefik.yml:ro` + `/etc/traefik/dynamic:ro` + `/etc/orca/step-ca-root.crt:ro` - Each step non-fatal warn (offline host tolerance) - **T6**: Wire into `internal/cli/init.go` after `EnsureTraefikContainerLocal` (Step 4e, replacing the old traefik install step). ### Wave 3 (doctor nft update) - **T7**: `internal/cli/doctor_nft.go` — extend assertions: postrouting masquerade present, DNAT target matches `NftClusterConfig.DNATTarget`. ### Wave 4 (tests) - **T8**: `internal/emitter/nft_test.go` — assert postrouting chain present when `EnableSNAT=true`, absent when false. Assert `DNATTarget` substitution. Assert priority `-10` on input/forward. - **T9**: Integration test: `orca init` → `nft list table inet orca-ingress` shows DNAT + postrouting; `podman inspect orca-traefik` running. ### Must-haves - [ ] `orca init` → nft table has DNAT + postrouting masquerade - [ ] nft input/forward chains at priority -10 - [ ] First-apply doesn't error (table pre-created or delete-table idiom) - [ ] `/etc/orca/step-ca-root.crt` exists (real CA or placeholder) - [ ] `podman inspect orca-traefik` running ## Phase 4: `orca node join --type linux` remote ingress bootstrap (REQ-174) **Tag**: `v0.13.4` | **Type**: feat | **Persona**: lead-developer ### Wave 1 (remote ingress) - **T1**: `internal/ingress/bootstrap.go` — add `BootstrapRemoteIngress(ctx, execFn)`: 1. `mkdir -p /etc/traefik/dynamic /etc/orca` (remote) 2. **C-60**: Push step-ca root CA to remote `/etc/orca/step-ca-root.crt` from `certpaths.CACertPath()` via `WriteFile` 3. Render `orca.nft` + write to remote `/etc/nftables.d/orca.nft` via `WriteFile` 4. `nft add table inet orca-ingress 2>/dev/null || true` (remote) 5. `nft -f /etc/nftables.d/orca.nft` (remote) 6. `systemctl enable --now podman-restart.service` (remote) 7. `EnsureTraefikContainerRemote` (from P2) - **T2**: Wire into `internal/linux/bootstrap.go` after the traefik container reconciler step. - **T3**: Extend `linux.Result` with `IngressOK bool` for reporting. ### Wave 2 (tests) - **T4**: Fake-SSH test: assert remote `nft -f` + `podman run` + `WriteFile` for step-ca CA invoked. ### Must-haves - [ ] `orca node join --type linux --host ` → remote has podman traefik running + nft applied + step-ca CA mounted - [ ] `doctor ingress --peer ` passes ## Phase 5: Proxmox native ingress mode (REQ-175) **Tag**: `v0.13.5` | **Type**: feat | **Persona**: backend-engineer + data-engineer ### Wave 1 (flags + schema) - **T1**: Add flags to `node join`: `--ingress-mode` (values: `native` default, `floating-ip`), `--floating-ip`, `--gateway`, `--mac`, `--net-prefix` (default `24`). - **T2**: Add `IngressMode` field to `model.Node` (string: `""`, `"native"`, `"floating-ip"`). - **T3**: **C-59**: Schema migration `0009_ingress_mode.sql` (NOT 0007 — already taken): `ALTER TABLE nodes ADD COLUMN ingress_mode TEXT DEFAULT '';` ### Wave 2 (native mode bootstrap) - **T4**: In `proxmox.BootstrapProxmox`, when `IngressMode == "native"`: 1. On the PVE host: render+apply nft with `DNATTarget = ` (the traefik LXC's IP, discovered after `pct start`) 2. Create unprivileged LXC with `--features nesting=1,keyctl=1,fuse=1` (research finding: fuse=1 for fuse-overlayfs). `pct create local:vztmpl/