# Research: v0.14 Ingress Bootstrap Completeness **Milestone**: v0.14 — Ingress Bootstrap Completeness **Role**: ci-researcher **Date**: 2026-08-10 **Status**: Complete This document records findings, recommended approaches, confidence levels, and pitfalls for the seven technical topics blocking the v0.14 ingress bootstrap milestone. The milestone replaces the traefik binary+systemd install (`internal/traefik/install.go`) with a podman container running a custom `orca-traefik` image, and completes the nft SNAT+DNAT ingress stack in `internal/emitter/nft.go`. Each topic is structured as: (a) findings, (b) recommended approach, (c) confidence level, (d) pitfalls. --- ## Topic 1: nft SNAT/MASQUERADE postrouting syntax ### (a) Findings The existing emitter (`internal/emitter/nft.go:180-184`) defines a `prerouting` nat chain with DNAT rules but has NO `postrouting` chain. For the R-017 hybrid model (Traefik on 127.0.0.1:8443, nft DNATs public :443 → 127.0.0.1:8443), the return path needs masquerade so replies from 127.0.0.1:8443 appear to come from the public IP. **Chain type and hook**: nft nat chains support `prerouting` and `postrouting` hooks in the `inet` family (nftables wiki "Configuring chains": nat chain type is "supported by the ip, ip6 and inet table families"). The postrouting hook sees packets "after routing, just before they leave the local system." **`masquerade` vs `snat to`**: Per Gentoo wiki nftables/Examples "Basic NAT": "If we have a static IP, it would be slightly faster to use source nat (SNAT) instead of masquerade. This way the router would replace the source with a predefined IP, instead of looking up the outgoing IP for every packet." `masquerade` is correct when the outgoing IP is dynamic or when there are multiple egress interfaces — which is the orca case (the public IP may be on any interface). `masquerade` is available since kernel 3.18. **Match expression — `oifname != "lo"` vs `ip saddr 127.0.0.0/8`**: The DNAT rewrites destination to 127.0.0.1:8443, so the reply source is 127.0.0.1. We must masquerade only traffic whose source was rewritten to loopback by the DNAT — i.e. traffic leaving a non-loopback interface with source 127.0.0.0/8. The correct expression is: ``` ip saddr 127.0.0.0/8 oifname != "lo" masquerade ``` A bare `oifname != "lo" masquerade` would masquerade ALL non-loopback egress, which is over-broad (it would NAT legitimate traffic that wasn't DNAT'd). Matching `ip saddr 127.0.0.0/8` scopes masquerade to exactly the DNAT'd return path. This is the canonical pattern for "hairpin NAT" / "loopback DNAT return path." **`inet` family and IPv6**: `masquerade` in an `inet` table works for both IPv4 and IPv6, BUT: (1) the `ip saddr 127.0.0.0/8` match is IPv4-only, so the rule only applies to IPv4 packets; (2) IPv6 has no NAT-masquerade analogue in common use (RFC 6296 is rarely deployed) and orca's DNAT is IPv4-only (`dnat to 127.0.0.1:8443` is IPv4). For v0.14 the postrouting chain should be IPv4-scoped. If IPv6 ingress is added later, a separate `ip6 saddr ::1/128 oifname != "lo" masquerade` rule would be needed, but that is out of scope for v0.14. **`flush table inet orca-ingress` + re-apply**: The existing emitter does `flush table inet orca-ingress` then recreates the table (`nft.go:144-145`). This is the documented idempotent pattern (nftables wiki "Flushing chains": `flush table` deletes all rules in the table). Adding a postrouting chain to the flushed+recreated table is safe — `flush table` removes all chains/sets/rules in the table but keeps the table itself; the declarative re-create reinstates everything. The one caveat: `flush table` on a non-existent table errors. The emitter currently relies on the table existing or on `nft -f` tolerating the flush-then-create. Since the file is `#!/usr/sbin/nft -f` and is applied as a single transaction, `flush table` failing on first apply (non-existent table) would abort the whole file. **This is a latent bug**: on first-ever apply, `flush table inet orca-ingress` errors with "No such file or directory" and the table is never created. The fix is `delete table inet orca-ingress` (which tolerates absence? — no, it also errors) or `add table inet orca-ingress` first, or use `flush ruleset inet orca-ingress` — actually the robust idiom is: ``` table inet orca-ingress delete table inet orca-ingress table inet orca-ingress { ... } ``` But `delete table` on a missing table also errors. The truly idempotent pattern is to wrap in `add table inet orca-ingress` (no-op if exists in `nft -f`? — `add table` errors if exists). The cleanest solution: use `flush ruleset` (wipes EVERYTHING — dangerous, conflicts with pve-firewall, see Topic 2) OR omit the flush and rely on `nft -f` replacing the table atomically. **Actually `nft -f` with a `table ...{ }` block does NOT atomically replace an existing table of the same name — it errors "File exists"** unless preceded by `delete table` or `flush table`. The working idiom that survives first-apply is: ``` #!/usr/sbin/nft -f add table inet orca-ingress 2>/dev/null flush table inet orca-ingress table inet orca-ingress { ... } ``` But `nft -f` doesn't support shell redirect semantics inside the file. The real fix: emit `delete table inet orca-ingress` and accept that `nft -f` treats a missing-table delete as a warning (nft 1.0+ tolerates this in `-f` mode? — needs verification). The safest cross-version approach is to apply via `nft -f ` where the file begins with `flush table inet orca-ingress` and the operator pre-creates the table with `nft add table inet orca-ingress` if absent. For orca, the install step should run `nft add table inet orca-ingress 2>/dev/null || true` before the first `nft -f` apply. **This is a pitfall the v0.14 emitter must address** — see pitfalls. ### (b) Recommended approach Add a `postrouting` nat chain to `renderNftRuleset` in `internal/emitter/nft.go`, after the `prerouting` chain: ```nft chain postrouting { type nat hook postrouting priority 100; policy accept; ip saddr 127.0.0.0/8 oifname != "lo" masquerade } ``` Priority `100` is the standard `srcnat` priority (nftables wiki "Netfilter hooks": `NF_IP_PRI_SRCNAT = 100`). The existing `prerouting` uses priority `-100` (`dstnat`), which is correct and consistent. For the first-apply / flush-table pitfall: change the emitter to emit `add table inet orca-ingress` is not valid inside a `table {}` block. Instead, the apply command in the SSH-push transport should run: ```bash nft list table inet orca-ingress >/dev/null 2>&1 || nft add table inet orca-ingress nft -f /etc/nftables.d/orca.nft ``` OR change the rendered file to use the `delete table` idiom (nft ≥ 1.0 treats delete-of-missing as warning, not error, in `-f` mode — but this is version-dependent). The pre-create approach is robust across all nft versions. ### (c) Confidence level **High**. The chain syntax, hook, priority, and `masquerade` keyword are all documented and widely deployed. The `ip saddr 127.0.0.0/8 oifname != "lo"` match is the standard loopback-DNAT-return pattern. The only medium-confidence item is the first-apply flush-table behavior, which depends on nft version and should be verified on the target Proxmox kernel. ### (d) Pitfalls 1. **First-apply `flush table` on non-existent table errors** and aborts the `nft -f` transaction, leaving no table created. The installer must pre-create the table or the emitter must use a tolerate-absence idiom. 2. **Over-broad masquerade** (`oifname != "lo" masquerade` without the `ip saddr 127.0.0.0/8` match) would NAT all egress traffic and break non-ingress routing. Always scope to the loopback source. 3. **IPv6**: the `ip saddr` match is IPv4-only; do not assume the rule covers IPv6. The DNAT itself is IPv4-only (`127.0.0.1`), so this is consistent, but document it. 4. **`masquerade` vs `snat to `**: if the public IP is static and known, `snat to ` is slightly faster (no per-packet interface lookup). For orca's generic case (IP may be dynamic, multiple interfaces), `masquerade` is safer. If the cluster config carries an explicit public IP, a future optimization can emit `snat to`. 5. **Connection tracking**: `masquerade` only applies to the first packet of a flow (nftables wiki: "Only the first packet of a given flow hits this chain; subsequent packets bypass it"). This is correct behavior — conntrack handles the rest — but means the chain must not be used for filtering. --- ## Topic 2: Proxmox pve-firewall vs nft conflict ### (a) Findings **pve-firewall uses iptables-nft, not native nft tables.** Per the Proxmox VE Firewall docs (pve-docs chapter-pve-firewall, version 9.2.4): "The firewall runs two service daemons on each node: pvefw-logger (NFLOG daemon) and pve-firewall (updates iptables rules)." The docs explicitly say `iptables-save` to inspect generated rules. So the stock `pve-firewall` manages **iptables** rules (which on modern Proxmox translate to the nft `inet` backend via `iptables-nft`), in the `security` chains, NOT a custom nft table. **Proxmox also offers `proxmox-firewall` (nftables-based, tech preview).** The docs note: "As an alternative to pve-firewall we offer proxmox-firewall, which is an implementation of the Proxmox VE firewall based on the newer nftables rather than iptables." This is gated behind the `nftables: ` option in `/etc/pve/nodes//host.fw` (default `0`). When enabled, proxmox-firewall uses its own nft tables. **Does pve-firewall flush ALL nft tables?** The stock pve-firewall manages iptables rules via `iptables-restore`-style operations on its own chains. It does NOT flush unrelated nft tables — `iptables-nft` operates on the `xt` compat chains in nft, not on arbitrary user tables. A separate `table inet orca-ingress` is invisible to pve-firewall and will NOT be flushed by `pve-firewall restart` or `pve-firewall update`. **However**, if `proxmox-firewall` (the nft tech-preview) is enabled, its behavior is less certain — it may use `flush ruleset` or operate on a specific table. The docs don't specify its flush scope, and it's a tech preview, so the risk is low for v0.14 (most Proxmox 8/9 deployments use stock pve-firewall). **Priority conflicts**: pve-firewall's iptables chains run at standard iptables priorities. orca's `orca-ingress` table uses `priority -100` (prerouting nat) and `priority 100` (postrouting nat), and `priority filter` (input/forward). These are standard priorities and nft executes all base chains at a hook in priority order — pve-firewall's iptables chains and orca's nft chains coexist (a packet traverses ALL base chains at a hook in priority order, per nftables wiki "Base chain priority": "packets will traverse all of the chains within the scope of a given hook until they are either dropped or no more base chains exist"). So there is no priority collision that would skip orca's rules — but there IS a semantic interaction: if pve-firewall DROPs a packet in its input chain (priority 0, after orca's input at priority `filter` which is also 0 — same priority means undefined order!), orca's accept verdict is not final. **The orca `input` chain uses `priority filter` which is the symbolic name for 0, the same as pve-firewall's INPUT chain**. This is a real concern: two base chains at the same hook + same priority have undefined evaluation order. The fix: give orca's chains a distinct priority (e.g. `priority -10` for input, ahead of pve-firewall's 0) so orca's SYN-flood filter runs deterministically before pve-firewall. **Recommended approach for third-party nft rules on Proxmox**: The Proxmox community consensus (forums, docs) is that a separate nft table with a distinct name (`orca-ingress`) coexists fine with pve-firewall's iptables-managed chains, as long as you don't touch pve-firewall's chains. `pve-firewall restart` regenerates only its own iptables rules. ### (b) Recommended approach 1. Keep `table inet orca-ingress` as a separate, named table — do NOT use `flush ruleset` anywhere in the orca emitter (that would wipe pve-firewall's rules). 2. Shift orca's `input` and `forward` chains to a priority ahead of pve-firewall's iptables chains to guarantee deterministic order. Change `priority filter` → `priority -10` (or `priority mangle -10`) for the `input` chain, and similarly for `forward`. Keep the nat chains at `priority -100`/`100` (nat priorities don't collide with pve-firewall's filter chains). 3. Document that orca's nft table is independent of pve-firewall and survives `pve-firewall restart`/`update`. 4. If `proxmox-firewall` (nft tech-preview) is enabled, add a doctor check warning that interaction is untested; recommend staying on stock pve-firewall for v0.14. ### (c) Confidence level **Medium-High** for stock pve-firewall coexistence (well-documented iptables-based, separate table is safe). **Low** for the nft tech-preview `proxmox-firewall` (docs don't specify flush scope; it's a tech preview and uncommon). **Medium** for the priority-collision fix (the nftables wiki confirms same-priority undefined order; shifting priority is the textbook fix, but the exact pve-firewall iptables priority values aren't in the docs — they're standard iptables priorities which map to 0). ### (d) Pitfalls 1. **Same-priority base chains have undefined evaluation order** — orca's `input` (`priority filter` = 0) and pve-firewall's INPUT (iptables priority 0) may run in either order. An `accept` from orca does NOT prevent pve-firewall from later dropping the packet. Shift orca's priority to be deterministic. 2. **`flush ruleset` would destroy pve-firewall** — never emit it. The current emitter uses `flush table inet orca-ingress` (scoped), which is safe. 3. **`proxmox-firewall` (nft tech-preview)**: untested interaction. Add a doctor check. 4. **pve-firewall's `ipfilter-net*` IP sets** enforce source-IP spoofing protection on VM/CT interfaces — if orca's DNAT'd traffic egresses a bridge that has ipfilter enabled, the 127.0.0.1 source may be dropped as spoofed. This is only relevant if orca runs inside a VM/CT with firewall+ipfilter enabled on its net interface. For the host-level ingress case (nft on the Proxmox host), this doesn't apply. 5. **`pve-firewall stop` does NOT remove orca's table** (good) but `pve-firewall start` re-adds its own rules (also fine). --- ## Topic 3: Podman inside Proxmox LXC (nesting/keyctl) ### (a) Findings **`nesting=1,keyctl=1` is the documented requirement for running containers (docker/podman) inside an unprivileged LXC.** Per Proxmox VE Linux Container docs (`features` key, line 2174-2200 of the LXC doc): - `keyctl=`: "For unprivileged containers only: Allow the use of the keyctl() system call. This is required to use docker inside a container. By default unprivileged containers will see this system call as non-existent." - `nesting=`: "Allow nesting. Best used with unprivileged containers with additional id mapping. Note that this will expose procfs and sysfs contents of the host to the guest. This is also required by systemd to isolate services." So `--features nesting=1,keyctl=1` on an **unprivileged** LXC is the correct and sufficient configuration for podman. `keyctl` is explicitly "required to use docker inside a container" and applies to podman equally (podman uses the kernel keyring for storage creds). **Is `--privileged` needed for rootful podman?** No. Rootful podman inside an unprivileged LXC with `nesting=1,keyctl=1` works because the LXC's "unprivileged" refers to the user-namespace mapping of the LXC itself; rootful podman inside it runs as the LXC's root (mapped UID). The Proxmox docs note privileged containers are "unsafe" and should be avoided. **Recommendation: unprivileged LXC + `nesting=1,keyctl=1` + rootful podman inside** (root inside the LXC is fine; the LXC is still unprivileged from the host's view). **Ubuntu 24.04 LXC template packages**: The Ubuntu 24.04 LXC template is minimal. Beyond `podman`, you need podman's runtime dependencies which are NOT all pulled in by the `podman` metapackage on Ubuntu 24.04: - `conmon` (container monitor — sometimes a separate package) - `crun` or `runc` (OCI runtime — `crun` preferred, `runc` ≥ 1.1.11) - `fuse-overlayfs` (for rootless overlay; see next finding) - `uidmap` (for rootless subuid/subgid — only needed if running rootless podman; rootful podman inside the LXC doesn't need it) - `netavark` or `containernetworking-plugins` (CNI networking — only if using bridged container networking, NOT needed for `--network host`) - `passt` (rootless networking — not needed for rootful/host network) For orca's case (rootful podman, `--network host`), the minimal set is: `podman`, `conmon`, `crun`, `fuse-overlayfs`. Install via `apt-get install -y podman conmon crun fuse-overlayfs`. **fuse-overlayfs inside LXC**: There IS a known issue. Podman's default storage driver is `overlay`, which requires a backing filesystem that supports overlay (the LXC's rootfs, if on ZFS subvolume or ext4 image, may or may not support native overlay). Inside an LXC, the overlay driver often fails because the kernel's overlay mount requires `CONFIG_OVERLAY_FS` and the underlying fs must not be on a copy-on-write filesystem that confuses overlay (ZFS subvolumes as LXC rootfs are a known problem). The standard workaround is `fuse-overlayfs` as a `mount_program` in `storage.conf`, OR fall back to the `vfs` storage driver (correct but slow — full copy per layer). The Proxmox LXC `features` key also has a `fuse=` option (default 0): "Allow using fuse file systems in a container. Note that interactions between fuse and the freezer cgroup can potentially cause I/O deadlocks." **So for fuse-overlayfs inside LXC, you must also set `--features fuse=1`** in addition to nesting/keyctl. This is a critical finding — without `fuse=1`, fuse-overlayfs cannot mount inside the LXC. **`--network host` inside an LXC**: `--network host` makes the podman container share the LXC's network namespace. Since the LXC already has its own network namespace (that's what an LXC IS), `--network host` binds the container to the LXC's netns — which is exactly what orca wants (traefik binds 127.0.0.1:8080/8443 on the LXC's loopback, and nft on the Proxmox host DNATs to the LXC's IP). This works correctly and is the simplest networking model. No `netavark`/CNI needed. **Performance**: Podman inside LXC vs on the host has negligible overhead for CPU/memory (LXC uses the host kernel directly, no emulation). The main overheads are: (1) double namespace overhead (LXC netns + podman container netns — avoided with `--network host`); (2) storage I/O — overlay-in-LXC may be slower if using `vfs` or fuse-overlayfs vs native overlay on the host; (3) no device passthrough limitations beyond normal LXC. For a traefik data plane (network I/O bound, low disk I/O), LXC+podman performance is effectively native. ### (b) Recommended approach Create the LXC unprivileged with: ``` pct create local:vztmpl/ubuntu-24.04-standard__amd64.tar.zst \ --hostname ingress \ --features nesting=1,keyctl=1,fuse=1 \ --unprivileged 1 \ --net0 bridge=vmbr0,hwaddr=,ip=/,gw= \ --onboot 1 \ --memory 2048 --swap 0 ``` Inside the LXC, install rootful podman: ```bash apt-get update && apt-get install -y podman conmon crun fuse-overlayfs ``` Run traefik rootful with `--network host`: ```bash podman run -d --name orca-traefik --restart=always --network host \ -v /etc/traefik/dynamic:/etc/traefik/dynamic \ -v /etc/traefik/traefik.yml:/etc/traefik/traefik.yml:ro \ orca-traefik:v3.3.0 ``` Configure podman storage to use fuse-overlayfs as the overlay `mount_program` (in `/etc/containers/storage.conf`): ```ini [storage] driver = "overlay" runroot = "/run/containers/storage" graphroot = "/var/lib/containers/storage" [storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs" ``` If fuse-overlayfs still fails (some LXC kernel configs block it), fall back to `driver = "vfs"` in `storage.conf` (documented as a fallback, slow but correct). ### (c) Confidence level **High** for `nesting=1,keyctl=1` (explicitly documented as required for docker/podman in LXC). **High** for `--network host` binding the LXC's netns. **Medium** for the `fuse=1` requirement (the `features.fuse` docs say it's needed for fuse filesystems in LXC, and fuse-overlayfs is a fuse filesystem — logical, but not explicitly tested in the docs with podman). **Medium** for the package list (Ubuntu 24.04's podman metapackage may already pull conmon/crun — needs verification on the actual template version). ### (d) Pitfalls 1. **Missing `fuse=1` feature** → fuse-overlayfs cannot mount → podman storage init fails. Must include `fuse=1` in `--features`, or fall back to `vfs` storage driver. 2. **`vfs` storage driver is slow** (full copy per layer, no CoW) — only use as fallback. For traefik (small image, read-mostly), the performance hit is acceptable but not ideal. 3. **ZFS-backed LXC rootfs + overlay** is a known-bad combination. If the Proxmox storage is ZFS, the LXC rootfs is a ZFS subvolume, and native overlay may fail. fuse-overlayfs (with `fuse=1`) is the workaround. 4. **`keyctl=1` breaks systemd-networkd** — the Proxmox docs warn: "Essentially, you can choose between running systemd-networkd or docker [keyctl]." If the LXC uses systemd-networkd for networking, enabling `keyctl=1` can cause systemd-networkd to fail. orca's ingress LXC uses static IP config (not systemd-networkd), so this is not an issue — but document it. 5. **`--privileged` LXC is unnecessary and unsafe** — don't use it. Unprivileged + nesting/keyctl is the supported path. 6. **AppArmor in LXC** may restrict podman. Proxmox LXC uses AppArmor; podman generally works but if conmon is blocked, check `dmesg`/audit for AppArmor denials. 7. **`--onboot 1`** starts the LXC on Proxmox boot, but does NOT auto-start the podman container — see Topic 6. --- ## Topic 4: Traefik v3.3 container image customization ### (a) Findings **Base image entrypoint**: The official `traefik:v3.3.0` Dockerfile (GitHub `traefik/traefik` v3.3 branch, `Dockerfile`): ```dockerfile FROM alpine:3.21 RUN apk add --no-cache --no-progress ca-certificates tzdata COPY ./dist/$TARGETPLATFORM/traefik / EXPOSE 80 VOLUME ["/tmp"] ENTRYPOINT ["/traefik"] ``` So the binary is at `/traefik` (not `/usr/local/bin/traefik`), and `ENTRYPOINT ["/traefik"]` is correct. There is no default `CMD` — traefik expects CLI args or `--configFile`. **Bake static config + CMD**: Yes. You can bake `/etc/traefik/traefik.yml` into the image and set `CMD ["--configFile=/etc/traefik/traefik.yml"]`. Traefik reads static config from the file specified by `--configFile` (Traefik docs "Static Configuration: File"). The static config can also be at the default search paths (`/traefik.yml`, `/etc/traefik/traefik.yml`, etc.) without `--configFile`, but being explicit is safer. Example `Dockerfile.traefik`: ```dockerfile FROM traefik:v3.3.0 COPY traefik.yml /etc/traefik/traefik.yml CMD ["--configFile=/etc/traefik/traefik.yml"] ``` The existing `internal/traefik/install.go:72` already uses `--configFile=/etc/traefik/traefik.yml`, so the container CMD matches. **CA-based TLS resolver — CRITICAL FINDING**: Traefik v3.3 `certificatesResolvers` supports ONLY `acme` and `tailscale` (confirmed in the static-config reference: `certificatesResolvers..acme.*` and `certificatesResolvers..tailscale.*` are the only sub-keys). **There is NO `certificatesResolvers.orca.tls` pointing at a CA file.** The existing `internal/emitter/traefik.go:53` defines `traefikRouterTLSCertResolver = "orca"` and the dynamic config emits `tls: certResolver: orca` — but this only works if a `certificatesResolvers.orca` exists in the STATIC config, and that resolver must be `acme` or `tailscale`. A custom CA is NOT a "certificate resolver" in Traefik's terminology. **How custom CA TLS actually works in Traefik**: Custom CA certificates are provided via the DYNAMIC config, not a cert resolver: - **Server-side TLS cert**: `tls.certificates: [{certFile: ..., keyFile: ...}]` in dynamic config (Traefik "TLS" docs, "User defined" section). - **Client-auth CA (mTLS)**: `tls.options..clientAuth.caFiles: [...]` + `clientAuth.clientAuthType: RequireAndVerifyClientCert` in dynamic config. - **Default cert**: `tls.stores.default.defaultCertificate: {certFile, keyFile}` in dynamic config. So the orca "step-ca root CA as TLS resolver" model is architecturally mismatched with Traefik. The correct approach for v0.14: - Drop `certResolver: orca` from the dynamic-config router TLS stanza (it references a non-existent resolver). - Instead, emit `tls.certificates` with the step-ca-issued cert+key (server identity) and `tls.options.default.clientAuth.caFiles` with the step-ca root CA (for mTLS client verification). - OR, if mTLS is not required for v0.14 and only server TLS is needed, emit `tls.stores.default.defaultCertificate` pointing at a step-ca-issued server cert+key. **Graceful degradation when CA file is absent**: Traefik logs an error and holds the last-good dynamic config if a cert file is missing or unparseable (documented behavior: "If the new config is malformed, Traefik logs an error and holds last-good config"). It does NOT crash on a missing dynamic-config cert file — it skips that cert and logs. For the STATIC config, a missing `--configFile` IS fatal (traefik won't start). So: bake a minimal valid static config in the image (always present), and mount dynamic config (certs) from the host — if the cert file is absent, traefik starts but that TLS config doesn't load. **CAP_NET_BIND_SERVICE for `--network host`**: With `--network host`, the container shares the host (LXC) netns. Binding ports < 1024 requires `CAP_NET_BIND_SERVICE` OR root. Since orca runs rootful podman (root inside the LXC), the traefik process runs as root and can bind privileged ports without `CAP_NET_BIND_SERVICE`. **But** orca binds 127.0.0.1:8080 and 127.0.0.1:8443 (both ≥ 1024), so `CAP_NET_BIND_SERVICE` is NOT needed at all. No `--cap-add` required. (If the opt-out `traefik-on-public-ip` mode binds :80/:443 directly, then root handles it — still no cap needed for rootful.) **Base image is Alpine** — note that the orca traefik install currently downloads a Linux binary (`internal/traefik/install.go:24`) which is the same binary. The container image uses the official Alpine-based image with `ca-certificates` and `tzdata` pre-installed, which is an advantage (mTLS CA verification needs `ca-certificates`). ### (b) Recommended approach `Dockerfile.traefik`: ```dockerfile FROM traefik:v3.3.0 LABEL org.opencontainers.image.title="orca-traefik" LABEL org.opencontainers.image.source="https://git.cloudinit.dev/coreci/orca" # Bake the static config (entrypoints, file provider, logging). # The dynamic config (routers, certs) is mounted at runtime. COPY traefik.yml /etc/traefik/traefik.yml ENTRYPOINT ["/traefik"] CMD ["--configFile=/etc/traefik/traefik.yml"] ``` Static config (`traefik.yml`, rendered by `internal/emitter/traefik.go:282-300` — already correct, just bake it): ```yaml entryPoints: websecure: address: "127.0.0.1:8443" web: address: "127.0.0.1:8080" traefik: address: "127.0.0.1:8081" providers: file: directory: "/etc/traefik/dynamic" watch: true log: level: INFO format: json accessLog: format: json ``` Run command (no `--cap-add`, no SELinux flag — see Topic 7): ```bash podman run -d --name orca-traefik --restart=always --network host \ -v /etc/traefik/dynamic:/etc/traefik/dynamic \ orca-traefik:v3.3.0 ``` **TLS model change for v0.14**: The `TraefikEmitter` dynamic config must drop `tls.certResolver: orca` and instead reference a TLS cert from the dynamic `tls.certificates` store. This is a code change in `internal/emitter/traefik.go:185-188` (the `tls:` stanza). Until the step-ca integration mints server certs, emit a `tls: {}` stanza (Traefik will use its generated default cert) OR omit `tls:` entirely (plain HTTP). Document this as a v0.14 limitation: real mTLS lands when step-ca mints certs into the dynamic dir. ### (c) Confidence level **High** for the Dockerfile structure (verified against the official v3.3 Dockerfile). **High** that `certificatesResolvers` only supports acme/tailscale (confirmed in the static-config reference — no other sub-keys exist). **High** that CAP_NET_BIND_SERVICE is not needed (ports ≥ 1024 + rootful). **Medium** for graceful degradation behavior on missing dynamic cert (documented but should be tested). ### (d) Pitfalls 1. **`certificatesResolvers.orca.tls` does not exist** — the existing emitter emits `certResolver: orca` which Traefik will reject or ignore (router TLS with a non-existent resolver → Traefik logs a warning and may not serve TLS). This is the biggest v0.14 finding: the TLS model must change from "cert resolver" to "dynamic tls.certificates". 2. **Binary path is `/traefik`** in the official image, not `/usr/local/bin/traefik`. The `Dockerfile.traefik` extends the official image so `ENTRYPOINT ["/traefik"]` is inherited — don't override it unless you copy the binary elsewhere. 3. **`CMD` vs `ENTRYPOINT`**: `ENTRYPOINT ["/traefik"]` + `CMD ["--configFile=..."]` means `podman run orca-traefik` runs `/traefik --configFile=...`. If the operator passes extra args (`podman run orca-traefik --log.level=DEBUG`), they APPEND to CMD (not replace) — actually they REPLACE CMD. To append, use `podman run orca-traefik --configFile=/etc/traefik/traefik.yml --log.level=DEBUG`. Document this. 4. **Static config baked, dynamic mounted**: the static config (`traefik.yml`) is baked into the image (immutable, versioned). The dynamic config (routers/services/certs) is bind-mounted from `/etc/traefik/dynamic` on the host so orca can update it atomically (C-10 protocol). Do NOT bake the dynamic config into the image. 5. **Alpine base + CA certs**: the official image has `ca-certificates` installed. If you build a custom image from `scratch` or `distroless`, you must install CA certs or TLS verification to upstreams (step-ca, ACME) will fail. 6. **`--network host` + Alpine**: Alpine's `/etc/resolv.conf` handling under `--network host` is fine (shares host netns). No issue. --- ## Topic 5: `pct create` with floating IP + MAC ### (a) Findings **Correct `pct create` syntax**: Per the Proxmox VE Linux Container docs ("Managing Containers with pct", line 1792 and "CLI Usage Examples", line 1812): ``` pct set 100 -net0 name=eth0,bridge=vmbr0,ip=192.168.15.147/24,gw=192.168.15.1 ``` The `net[n]` parameter format (docs line 959): ``` net[n]: name= [,bridge=] [,firewall=<1|0>] [,gw=] [,gw6=] [,hwaddr=] [,ip=<(IPv4/CIDR|dhcp|manual)>] [,ip6=<...>] [,type=] ``` So the correct `pct create` with static IP, custom MAC, and gateway is: ```bash pct create local:vztmpl/ubuntu-24.04-standard__amd64.tar.zst \ --hostname ingress \ --net0 name=eth0,bridge=vmbr0,hwaddr=,ip=/,gw= ``` **`ip=/` with public IP**: Yes, the `ip` value takes `IPv4/CIDR` format (docs: `ip=<(IPv4/CIDR|dhcp|manual)>`). A public IP with prefix works: `ip=203.0.113.10/24`. The prefix is REQUIRED (not `ip=` bare) — without the prefix, Proxmox may not configure the interface correctly (it needs the netmask). Use CIDR always. **`gw=`**: Correct parameter name (docs: `gw=`, "Default gateway for IPv4 traffic"). For IPv6, use `gw6`. **Getting the LXC's IP after `pct start`**: The IP is configured in the LXC config (`/etc/pve/lxc/.conf`), so it's known before start. But to verify the LXC actually came up with it: - `pct config ` — prints the config including `net0: ...ip=...` (the configured IP). - `pct list` — lists VMIDs and status (running/stopped), NOT IPs. - `pct inspect ` — not a real command. Use `pct config`. - `lxc-info -n ` — works if `lxc-tools` installed; shows IP if running. - **Most reliable**: `pct exec -- ip -j addr show eth0 | jq -r '.[0].addr_info[0].local'` (runs inside the LXC). Or `pct status ` for status. For orca, the IP is known at create time (it's in the `pct create` command), so discovery is only needed for verification. Use `pct config ` and parse the `net0` line, or `pct exec -- hostname -I`. **Proxmox native mode (separate LXC for traefik) — discovering the LXC's bridge IP for the nft DNAT target**: In native mode, the nft DNAT rule on the Proxmox host must target the LXC's bridge IP (not 127.0.0.1, since traefik is in a separate LXC, not on the host's loopback). The LXC's IP is the `ip=/` from `pct create`. The DNAT rule becomes: ```nft tcp dport 443 dnat to :8443 tcp dport 80 dnat to :8080 ``` The `` is the floating IP assigned to the LXC (or a dedicated bridge IP if the LXC is on a private bridge). The emitter needs the LXC's IP as input — this is a cluster-config field (e.g. `ingress.traefik_ip` or derived from the LXC VMID via `pct config `). For v0.14, the `NftClusterConfig` should carry a `TraefikDNATTarget` field (default `127.0.0.1` for the hybrid/host mode, set to the LXC IP for native mode). ### (b) Recommended approach ```bash # Create the ingress LXC (unprivileged, nesting for podman) pct create 201 local:vztmpl/ubuntu-24.04-standard_24.04-1_amd64.tar.zst \ --hostname ingress \ --unprivileged 1 \ --features nesting=1,keyctl=1,fuse=1 \ --net0 name=eth0,bridge=vmbr0,hwaddr=02:ca:fe:00:00:01,ip=10.99.0.10/24,gw=10.99.0.1 \ --onboot 1 \ --memory 2048 --swap 0 \ --rootfs local-zfs:8 pct start 201 # Verify IP pct config 201 | grep '^net0' # or pct exec 201 -- hostname -I ``` For the nft emitter, add a `TraefikDNATTarget` field to `NftClusterConfig` (default `127.0.0.1`): ```go type NftClusterConfig struct { TrustedProbes []string RateLimit int RateBurst int TraefikDNATTarget string // "127.0.0.1" (hybrid/host) or LXC IP (native) } ``` And render: ```nft tcp dport 443 dnat to :8443 tcp dport 80 dnat to :8080 ``` ### (c) Confidence level **High** — the `pct create` syntax is directly from the Proxmox docs examples. The `ip=/` CIDR format and `gw=` parameter are documented. IP discovery via `pct config`/`pct exec` is standard. ### (d) Pitfalls 1. **`ip=` without prefix** may not configure the interface — always use CIDR (`ip=/`). 2. **`hwaddr` must be unique** on the bridge — Proxmox auto-generates one if omitted; if specifying a custom MAC, ensure no collision. 3. **`pct create` requires the template to be downloaded first** via `pveam update && pveam download local