# InboxOS — Operations Runbook

**Audience:** Any operator (or the same operator after a 6-month gap) recovering from common failure modes without re-deriving the architecture.

**Status:** Phase 5 deliverable for [INB-7](/INB/issues/INB-7) — issued under [INB-35](/INB/issues/INB-35).

**Scope:** Single self-hosted pipeline. A Gmail inbox → IMAP poll → SeaweedFS archive + LLM-extracted rows in NocoDB + curated attachments in Papra. Everything is self-hosted; no third-party SaaS touches PII.

**Acceptance test for this runbook:** *"Given a workflow is stuck in `in_progress` for 1 hour, what do you do?"* — answer in under 5 minutes using §5.1 below.

---

## 1. System overview

InboxOS is a self-hosted, air-gapped personal ERP/CRM pipeline. It polls a single Gmail inbox over IMAP, backs up the raw `.eml` to SeaweedFS, classifies each email with a local LLM (Ollama), runs one or more `inboxos:extract-<module>` prompts against the same email, and UPSERTs the resulting structured records into NocoDB (one table per module: `finance`, `crm`, `intel`, `promotions`). Qualifying attachments are also uploaded to Papra for long-term searchability. The historical pipeline performs the same flow against a Google Takeout `.mbox` for backfill, with byte-offset checkpoints for safe resume.

### 1.1 Topology

```
┌──────────────┐    IMAP/993     ┌─────────────────┐
│   Gmail      │ ◀─────────────  │  gitea-runner-1 │
│   inbox      │   (App Password)│  gitea-runner-2 │  (self-hosted, parallel)
└──────────────┘                 └────────┬────────┘
                                         │ run jobs (Gitea Actions)
                                         │ polls IMAP, calls Ollama
                                         ▼
┌──────────────────┐   raw .eml  ┌─────────────────┐
│   Papra          │  attachment │   SeaweedFS     │  S3-compatible
│   (long-term doc)│ ◀───────────│   (S3 API)      │  .eml archive +
│   intake        │   uploads    │                 │  state checkpoints
└──────────────────┘             └────────┬────────┘
                                          │ UPSERT by Message-ID
                                          ▼
                                ┌─────────────────┐    direct HTTP
                                │   NocoDB        │ ── /v1/chat
                                │   (DWH)         │   /completions
                                └─────────────────┘         │
                                          ▲                 ▼
                                ┌─────────────────┐
                                │   Ollama        │  local LLM
                                │   (FRP)         │  gemma4:e4b
                                │   /v1/chat      │
                                │   /completions  │
                                └─────────────────┘
```

- **Gmail** is the source. The runner authenticates with an App Password (2FA must be on).
- **Gitea** is the version-controlled home for prompts, scripts, and CI.
- **Gitea Actions runners** (`gitea-runner-1` and `gitea-runner-2`, both self-hosted) execute the live and historical pipelines. Two runners so the 5-minute live cron and a long historical run can execute in parallel.
- **SeaweedFS** is the system of record for raw `.eml` (in `s3://$SEAWEEDFS_S3_BUCKET/eml/...`) and pipeline state (`state/live-state.json`, `state/historical-state.json`).
- **NocoDB** is the structured warehouse. Every module table is keyed on `Message_ID` (UNIQUE). All writes are UPSERTs.
- **Papra** holds curated attachments and (selectively) full email bodies for long-term search. The first ship goes out in **dry-run** mode.
- **Ollama** serves the local LLM behind a FRP tunnel at `https://ollama-frp.cloudinit.dev/v1`. No external API calls for PII-bearing data.

---

## 2. Service inventory

All services run in the `inboxos-net` Docker network defined in `docker/docker-compose.yml`.

| # | Service          | Container / DNS name      | Public URL                              | Role in pipeline                                                                              | Restart                                                       | Log location                                                          |
|---|------------------|---------------------------|-----------------------------------------|-----------------------------------------------------------------------------------------------|---------------------------------------------------------------|-----------------------------------------------------------------------|
| 1 | **NocoDB**       | `nocodb`                  | `https://noco.cloudinit.dev`            | ERP/CRM data warehouse. Four module tables (`finance`, `crm`, `intel`, `promotions`).        | `docker compose -f docker/docker-compose.yml restart nocodb`  | `docker logs nocodb`                                                  |
| 2 | **Gitea**        | `gitea`                   | `https://git.cloudinit.dev`             | Version control for prompts, scripts, and CI. Hosts Actions runner registration & secrets.   | `docker compose -f docker/docker-compose.yml restart gitea`   | `docker logs gitea`                                                   |
| 3 | **Gitea Actions Runners** | `gitea-runner`, `gitea-runner-2` | n/a (self-hosted)                | Execute the live and historical workflows. Two so live cron + historical run can run in parallel. | `docker compose -f docker/docker-compose.yml restart gitea-runner [gitea-runner-2]` | `docker logs gitea-runner [gitea-runner-2]` + per-job artifact under `pipelines/_artifacts/*.log` |
| 4 | **Papra**        | `papra`                   | `https://papra.cloudinit.dev`           | Curated document archive (attachments + selected bodies). First ship is dry-run.              | `docker compose -f docker/docker-compose.yml restart papra`   | `docker logs papra`                                                   |
| 5 | **SeaweedFS**    | `seaweedfs-master`, `-volume`, `-filer` | `https://storage.cloudinit.dev` (via reverse proxy) on `:8888` S3 endpoint in-network | Object store for `.eml` archive and pipeline state checkpoints.                          | `docker compose -f docker/docker-compose.yml restart seaweedfs-{master,volume,filer}` | `docker logs seaweedfs-{master,volume,filer}`                        |
| 6 | **Ollama**       | `ollama`                  | `https://ollama-frp.cloudinit.dev/v1`   | Local LLM inference for `inboxos:classify` and `inboxos:extract-*`. Default model `gemma4:e4b`. | `docker compose -f docker/docker-compose.yml restart ollama`  | `docker logs ollama`                                                  |

The **OpenCode CLI** is not invoked by either pipeline; both
pipelines call Ollama's OpenAI-compatible `/v1/chat/completions`
endpoint directly via `pipelines.common.llm_run_command` (default
backend: `ollama`). The `opencode` binary install step in
`historical-pipeline.yml` is retained for any human-driven
`opencode` invocation from the repo worktree; the pipeline itself
does not shell out to it. See `docs/extraction-contract.md §Legacy:
opencode CLI backend` for the rationale.

The **reverse proxy** (Caddy in the commented-out `caddy` service in `docker/docker-compose.yml`) terminates TLS for `*.cloudinit.dev` and routes to each service. Operators without a reverse proxy can reach services via the `inboxos-net` Docker DNS names (e.g. `http://gitea:3000`, `http://seaweedfs-filer:8888`).

---

## 3. Workflow operations

All workflows live in `.gitea/workflows/`. They are declarative, version-controlled in the same repo as the scripts they invoke, and have no separate runtime state to back up. This is the board decision recorded in [INB-8](/INB/issues/INB-8) — n8n was ruled out as unreliable.

**Quickstart:** for a 15-minute green-field spin-up, see `docs/quickstart.md` and `scripts/quickstart.sh` (REQ-INB-40). The script runs the 8 steps below in sequence and prints the runbook §X.Y cross-ref on any failure.

**Daily digest:** see `scripts/daily_digest.py` and `.gitea/workflows/daily-digest.yml` (REQ-INB-47). The workflow runs on `cron 0 6 * * *` and emails a one-line per-module summary.

**Container hardening:** all services run with `cap_drop: [ALL]`, `security_opt: no-new-privileges:true`, and either `read_only: true` (no write access needed) or `read_only: false` with a `tmpfs: [/tmp]` mount (for services that need a writable scratch dir). See `docker/docker-compose.yml` for the full stanza and the per-service `cap_add` list (REQ-INB-55).

**DPIA:** see `docs/security/dpia.md` (REQ-INB-58). The Data Protection Impact Assessment per GDPR Art. 35 names the data flows, the lawful basis, the data-subject rights, and the residual risks.

**Threat model:** see `docs/security/threat-model.md` (REQ-INB-59). The STRIDE-per-component decomposition covers the 10 components in the inboxos architecture.

### 3.1 Live pipeline — `live-pipeline.yml`

| Aspect         | Value                                                                                       |
|----------------|---------------------------------------------------------------------------------------------|
| Schedule       | `cron(*/5 * * * *)` — every 5 minutes                                                       |
| Manual trigger | `workflow_dispatch` with optional inputs `since_uid` (override high-water mark) and `max_emails` (default 50) |
| Concurrency    | `group: live-pipeline, cancel-in-progress: false` (overlapping runs must not trample state) |
| Timeout        | 15 minutes                                                                                  |
| Script         | `pipelines/live_pipeline.py`                                                                |
| Run log        | `pipelines/_artifacts/live-run-<id>.log` (uploaded as `live-pipeline-log` artifact)         |

**Flow** (verifiable in `pipelines/live_pipeline.py`):

1. Load the IMAP UID checkpoint from `s3://$SEAWEEDFS_S3_BUCKET/state/live-state.json`.
2. Poll IMAP for `UID > last_uid` (bounded by `MAX_EMAILS_OVERRIDE`, default 50).
3. For each message, in UID-ascending order:
   - Back up the raw `.eml` to `s3://$SEAWEEDFS_S3_BUCKET/eml/live/<yyyy>/<mm>/<dd>/<Message-ID>.eml` (skip if exists).
   - Call `inboxos:classify` to pick which modules apply.
   - For each applicable module, call `inboxos:extract-<module>` and UPSERT the result.
   - Flush the checkpoint (`state/live-state.json`) **per message** so a crash never replays a success.

**What success looks like in the run log:**

- One `live_run_start` event at the top.
- One or more `process_start` → `eml_backed_up` → `classified` → `process_done` event groups, in UID-ascending order.
- Every `process_done` is followed by a checkpoint write (visible as the `state/live-state.json` PUT in S3, or as the next call to `imap_fetched`).

**What failure looks like:**

- A `classify_error` or `extract_permanent_failure` event means the pipeline stops at that UID. The next run retries it.
- A `dead_letter_unclassified` event means the email had no module match — the raw `.eml` is preserved at `eml/dead-letter/<mid>.eml` and the cursor advances.
- A `missing_message_id` event means the email lacked a `Message-ID:` header. The raw `.eml` is preserved at `eml/dead-letter/uid-<n>.eml`.
- A non-zero job exit code with no `live_run_done` in the log means the runner OOM'd or the IMAP connection was killed mid-fetch — see §5.1.

### 3.2 Historical pipeline — `historical-pipeline.yml`

| Aspect         | Value                                                                                       |
|----------------|---------------------------------------------------------------------------------------------|
| Schedule       | **None** — manual only (`workflow_dispatch`). Heavy operation; full archives run for hours. |
| Required input | `mbox_object_key` — the SeaweedFS object key of the `.mbox` (e.g. `eml/historical/all-mail.mbox`) |
| Optional inputs| `start_offset` (resume from a byte offset; leave blank to use persisted checkpoint), `batch_size` (default 100), `limit` (0 = no limit) |
| Concurrency    | `group: historical-pipeline, cancel-in-progress: false`                                     |
| Timeout        | 12 hours                                                                                    |
| Script         | `pipelines/historical_pipeline.py` (canonical reference for the extraction contract)        |
| Streamer       | `pipelines/mbox_stream.py` — byte-offset streaming parser for `.mbox`                        |
| Run log        | `pipelines/_artifacts/historical-run-<id>.log` (uploaded as `historical-pipeline-log` artifact) |

**Idempotency contract:**

- **Message-ID UPSERT** is the only write path into NocoDB (`pipelines/common.py::nocodb_upsert`). It does a GET by `Message_ID` and PATCHes if present, POSTs otherwise. Re-running the same `.mbox` produces zero duplicates.
- **Byte-offset checkpoint** is persisted to `s3://$SEAWEEDFS_S3_BUCKET/state/historical-state.json` every `BATCH_SIZE` messages (default 100) and on shutdown. A crashed run resumes from the last checkpoint on the next dispatch.
- **Object keys** are deterministic by `Message-ID` and date, so re-uploading the same `.eml` overwrites the same S3 object.

**Expected runtime per 10K messages:** ~5–10 minutes on a clean `c5.large` runner when the LLM is the bottleneck (depends on the model — `gemma4:e4b` is fast; `gemma4:12b` adds ~2×). The I/O-bound parts (S3 GET of the `.mbox`, S3 PUTs of `.eml`, Papra uploads) are sub-second per message.

**How to resume after a crash:**

1. Re-dispatch `historical-pipeline.yml` with the **same** `mbox_object_key` and **no** `start_offset` input. The script loads `state/historical-state.json` and resumes from the persisted byte offset.
2. Do **not** dispatch with `start_offset=0` unless you have deleted the state file — that re-processes everything from the top (safe but wasteful, and will take hours).

**How to dead-letter a message manually:**

- A message that was already dead-lettered by the pipeline lives at `s3://$SEAWEEDFS_S3_BUCKET/eml/dead-letter/historical-<offset>.eml` (or `<mid>.eml`). To permanently skip a poison message, find its `Message-ID` in the run log, then write a marker object to `eml/dead-letter/poison/<mid>.json` containing the reason. A future enhancement will let the script consult this list; for now, mark it in your run log and move on.

### 3.3 PR CI — `ci-pr.yml`

| Aspect         | Value                                                                                       |
|----------------|---------------------------------------------------------------------------------------------|
| Triggers       | `pull_request` opened / reopened / synchronized against `main`; `workflow_dispatch`         |
| Concurrency    | `group: pr-ci-${{ pull_request.number }}`, cancel-in-progress                               |
| Jobs           | `fast-tests` (~30s), `e2e-100mb` (20 min), `live-llm-smoke` (15 min, real Ollama)          |
| Required secrets | `NOCODB_BASE_URL`, `NOCODB_API_TOKEN`, `NOCODB_PROJECT_ID`; `OLLAMA_BASE_URL` is optional |

- `fast-tests` exercises `gitea_request`, `mbox_stream`, the offline parts of `live_pipeline`, and the fast subset of `historical_pipeline` (config, parsing, dead-letter paths).
- `e2e-100mb` builds a 100 MB synthetic `.mbox` and runs the historical pipeline end-to-end with S3 / NocoDB / Papra stubbed. Verifies UPSERT idempotency, crash-and-resume offset stability, and dead-letter semantics. The merge gate.
- `live-llm-smoke` calls the real Ollama endpoint with the Substack input that previously dead-lettered in live run #674 (UID 4017, 86 KB body) and asserts the extract step returns parseable JSON. Skipped when `OLLAMA_BASE_URL` is not set; surfaces a clear `skipped` notice rather than failing the PR.

---

## 4. Observability

### 4.1 Reading a workflow run log

1. Open the Gitea UI → repo → **Actions** → click the run.
2. The job's `Run process-new-emails` / `Run process-mbox` step shows stdout in the Gitea Actions UI. Each line is a JSON object emitted by `log_event`.
3. Expand the **Upload run log as artifact** step. The per-run log file is downloaded as `live-pipeline-log` or `historical-pipeline-log`. This is identical to what was streamed to stdout, and is the canonical artifact to attach to a post-mortem.
4. The artifact also lives in the project workspace at `pipelines/_artifacts/<pipeline>-run-<run-id>.log`.

### 4.2 Structured event vocabulary

Every step in both pipelines calls `pipelines.common.log_event` to emit a JSON line. The canonical event names (verified in source) are:

| Event                          | Where                                      | What it tells you                                                              |
|--------------------------------|--------------------------------------------|--------------------------------------------------------------------------------|
| `process_start`                | `live_pipeline.py:428`, `historical_pipeline.py:282` | A new message is about to be processed. Carries `uid` (live) or `offset` (historical) and `message_id`. |
| `eml_backed_up`                | both pipelines                             | The raw `.eml` was PUT to SeaweedFS. Carries `key` and `bytes`.                 |
| `classified`                   | both pipelines                             | `inboxos:classify` returned a non-empty module list. Carries `modules`.         |
| `llm_call`                     | `common.py:591`                            | The Ollama call is about to be made. Carries `backend`, `modelID`, `command`, `url`. |
| `process_done`                 | both pipelines                             | The message was processed end-to-end. Carries `upserts` (NocoDB action list).   |
| `module_table_missing`         | both pipelines                             | A module came back from classify but has no entry in `NOCODB_MODULE_TABLES`. Carries `module`. |
| `extract_permanent_failure`    | both pipelines                             | `inboxos:extract-<module>` failed irrecoverably. The run stops without advancing. |
| `dead_letter_unclassified`     | both pipelines                             | Classify returned `[]`. The `.eml` is preserved at `eml/dead-letter/...` and the cursor advances. |
| `historical_run_done`          | `historical_pipeline.py:545`               | The historical run finished cleanly. Carries totals.                            |
| `live_run_start`               | `live_pipeline.py:533`                     | The live run began. Carries `started_at`.                                       |
| `papra_decision`               | `common.py:997`                            | The Papra write-through decision was made. Carries `would_write`, `reason`, `module`, `dry_run`. |
| `papra_write_ok` / `papra_write_failed` / `papra_write_dry_run` | `common.py:1018`-`1055` | Per-target outcome of a Papra write. |
| `state_recovered_from_corrupt` | `live_pipeline.py:550`                     | The persisted IMAP checkpoint was malformed; the run reset it.                 |

The `live_pipeline.py` source also emits `imap_search_result`, `imap_fetch_missing_body`, `imap_fetched`, `classify_error`, `extract_invalid_response`, `missing_message_id`, `config_error`, `since_uid_override`, `stopped_on_failure_at_uid`, and others (search for `log_event` in `pipelines/live_pipeline.py` for the full list).

### 4.3 NocoDB table inventory

The four module tables (verified against `pipelines/_tests/test_upsert_synthetic.py` and `docs/phase-1.1-verification/inb-28-report.md`):

| Module       | Table name in NocoDB | Table ID            | Notes |
|--------------|----------------------|---------------------|-------|
| `finance`    | `pl_finance`         | `m0thluc3jyz2iok`   | Receipts, invoices, P&L items. 14 user columns. |
| `crm`        | `personal_crm`       | `m2wsbpp01v08pm5`   | People, organisations, intent. 15 user columns. |
| `intel`      | `daily_news`         | `msi6dj1cany1b5r`   | Newsletters, briefings. 13 user columns. |
| `promotions` | `promotions`         | `mf7catfwwyh3jj5`   | Sales, promo codes, launches. 14 user columns. |

Per-table row counts and column schemas are listed in `docs/phase-1.1-verification/inb-28-report.md` and `docs/schemas.md`. Every module table has `Message_ID` (SingleLineText, UNIQUE, required) as the natural key.

### 4.4 SeaweedFS bucket layout

`$SEAWEEDFS_S3_BUCKET` is `inboxos` (default in `.env.example`). The layout is:

```
s3://inboxos/
├── eml/
│   ├── live/
│   │   └── <yyyy>/<mm>/<dd>/<Message-ID>.eml
│   ├── historical/
│   │   └── <yyyy>/<mm>/<dd>/<Message-ID>.eml
│   └── dead-letter/
│       ├── <Message-ID>.eml          (live, unclassified)
│       ├── uid-<uid>.eml             (live, missing Message-ID)
│       └── historical-<offset>.eml   (historical)
└── state/
    ├── live-state.json               {"last_uid": <int>, "last_run_at": <iso>, "messages_seen": <int>}
    └── historical-state.json         {"mbox_key": <str>, "offset": <int>, "messages_seen": <int>, "last_run_at": <iso>}
```

> State-file field names come from the actual code: live
> (`pipelines/live_pipeline.py:600-603`) and historical
> (`pipelines/historical_pipeline.py:516-521`). The
> `_corrupt_state_recovered` field is an additional transient
> key on first load when the persisted JSON is unparseable.

The state files are tiny JSON blobs written via `s3_put_bytes` after every successful message (live) or every `BATCH_SIZE` messages (historical). They are the **only** persistent state between workflow runs.

### 4.5 Metrics surface (REQ-INB-23)

Each pipeline run writes **one** JSONL line to
`pipelines/_artifacts/<pipeline>-metrics-<run_id>.jsonl` (one
file per run; never overwritten). The line shape and counter /
gauge vocabulary are documented in
`pipelines/_metrics/emit.py` (a copy-paste of the docstring is
in this runbook for operator reference).

| Counter key                | Type | What it tells you                                          |
|----------------------------|------|------------------------------------------------------------|
| `messages_processed`       | int  | Total messages processed in this run.                      |
| `messages_dead_lettered`   | int  | Subset routed to the dead-letter prefix.                    |
| `noco_upsert_ok`           | int  | NocoDB upserts that returned 2xx.                          |
| `noco_upsert_4xx`          | int  | 4xx (drift; permanent failure for the affected message).    |
| `noco_upsert_5xx`          | int  | 5xx (transient; retried by tenacity).                     |
| `papra_would_write`        | int  | Papra decisions that resolved `would_write=true`.          |
| `papra_write_ok`           | int  | Papra HTTP POSTs that returned 2xx.                        |
| `papra_write_failed`       | int  | Papra HTTP POSTs that returned >= 400.                     |
| `papra_index_upsert_ok`    | int  | `papra_index` audit-trail rows written.                    |
| `papra_index_upsert_failed`| int  | `papra_index` upserts that raised.                         |
| `llm_call_seconds_sum`     | int  | Total wall-clock ms spent in `requests.post` to Ollama.   |
| `llm_call_count`           | int  | Number of LLM HTTP POSTs in the run.                       |

| Gauge key                | Type           | What it tells you                              |
|--------------------------|----------------|-------------------------------------------------|
| `last_run_at`            | ISO 8601 str   | The run's `now_iso()` at metrics emission.     |
| `dead_letter_total`      | int            | Cumulative dead-letter prefix size (best-effort). |
| `papra_dry_run`          | "true"/"false" | The effective `PAPRA_DRY_RUN` after precedence. |
| `papra_dry_run_override` | "true"/"false"/"none" | The parsed override (or "none" if unset). |

**Tail-and-summarize procedure.** To compute a daily total of
`messages_processed` across all live runs:

```bash
cat pipelines/_artifacts/live-metrics-*.jsonl \
    | jq -s '[.[].counters.messages_processed] | add'
```

To inspect the last 20 metrics lines in time order:

```bash
cat pipelines/_artifacts/live-metrics-*.jsonl \
    | tail -20 \
    | jq -c '{ts, run_id, counters}'
```

**Forward-pointer.** Alerting is **out of scope** for v1.4. v1.2
closed on a 0/1 ideation disposition (no candidates accepted) per
`.ciagent/inboxos/CI-LOG.md`, and v1.3 was a doc-hygiene
milestone; v1.4 added the operator audit cursor and the
pollable healthcheck but no alert layer. The metrics file is
the substrate; the alert layer is the next v1.x candidate
cycle to re-evaluate. In the meantime the runbook §4
observability surface is logs-only, with
`pipelines/_metrics/emit.py` emitting a per-run `metrics.jsonl`
(REQ-INB-23) for offline postmortem and
`pipelines/_metrics/healthcheck.json` (REQ-INB-52) as the
canonical uptime-monitor pollable endpoint.

**Opt-out.** Set `LLM_METRICS_ENABLED=false` in the pipeline env
to skip the `time.monotonic()` wrapping around the LLM HTTP POST.
The counters default to enabled; opt-out is for canary runs that
want to measure pure LLM latency without the wrapper overhead.

---

## 5. Common failure modes and recovery

### ERROR-NN catalog

A one-page lookup table for the on-call operator. If you are reading this from a pager, the table below tells you which §5.X subsection to jump to for the full recovery procedure.

| ERROR-NN  | Symptom                                              | See §X.Y |
|-----------|------------------------------------------------------|----------|
| ERROR-01  | Runner stuck / OOM (> 60 min in `in_progress`)       | §5.1     |
| ERROR-02  | Live cron emitted no new MIDs for > 30 min           | §5.2     |
| ERROR-03  | NocoDB 4xx on upsert (wrong table_id)                | §5.3     |
| ERROR-04  | NocoDB 4xx "field is required" (schema drift)        | §5.3     |
| ERROR-05  | IMAP connection refused / auth failure               | §5.4     |
| ERROR-06  | Ollama connection refused / model OOM                | §5.5     |
| ERROR-07  | SeaweedFS connection refused / auth failure          | §5.6     |
| ERROR-08  | Papra `dry_run` unexpectedly flipped to `false`      | §5.7     |
| ERROR-09  | State file corrupted / offset regression             | §5.8     |
| ERROR-10  | Backup tar fails (NocoDB / SeaweedFS / Gitea)        | §5.9     |

### 5.1 Runner stuck / OOM (> 60 minutes in `in_progress`)

A single stuck job can block the live cron (live jobs have `cancel-in-progress: false`). Diagnosis and recovery:

1. **Confirm it is stuck.** Open Gitea Actions → run → check whether the `Run process-new-emails` step has emitted *any* new log lines in the last 10 minutes. A runner that is downloading a large `.mbox` is making progress (watch for S3 GET bytes) — a runner that is silent is OOM'd or wedged.
2. **Check the host's container health.**
   ```bash
   docker ps --filter "name=gitea-runner" --format "table {{.Names}}\t{{.Status}}\t{{.CPUPerc}}\t{{.MemUsage}}"
   ```
   Memory at 100% with no progress for > 10 minutes = OOM. CPU at 0% with no progress = wedged.
3. **Drain the stuck job.** From Gitea UI → run → **Cancel job**. The live workflow's `cancel-in-progress: false` will *not* preempt the next cron tick — the next run will queue until the cancelled one reports back. This is the correct behaviour: do not change the concurrency setting to fix a stuck job.
4. **Restart the runner.**
   ```bash
   docker compose -f docker/docker-compose.yml restart gitea-runner
   ```
   If `gitea-runner-2` is also stuck, restart it too. The other runner will pick up new jobs within ~30 s.
5. **Verify recovery.**
   ```bash
   curl -u "$GITEA_USER:$GITEA_PASS" \
     https://git.cloudinit.dev/api/v1/repos/jc-capital/inboxos/actions/runners
   ```
   Expect 2 entries with `status: "online"`. Then dispatch `live-pipeline.yml` manually with a small `max_emails` (e.g. `5`) to confirm the end-to-end loop.
6. **If recurring:** the runner is under-provisioned. Increase its memory limit in `docker/docker-compose.yml` (`mem_limit: 4g` or similar) or move to a host with more RAM.

### 5.2 SeaweedFS `NoSuchKey` on a Message-ID

This means a NocoDB row references a `Message-ID` whose raw `.eml` was never uploaded to (or was deleted from) SeaweedFS. Two common causes:

1. **The `.eml` write failed** but the NocoDB upsert still succeeded (e.g. SeaweedFS was down briefly; the retry on the S3 PUT hit a 5-attempt limit; the pipeline then proceeded to NocoDB).
2. **The Message-ID is malformed** — e.g. a stray `<` or `>` made it into the NocoDB row but the file was written with the correct key.

**How to find the offending record in NocoDB:**

```bash
# For a finance row, replace m0thluc3jyz2iok with the module's table ID from §4.3
curl -G \
  -H "xc-token: $NOCODB_API_TOKEN" \
  --data-urlencode "where=(Message_ID,eq,<message-id-here>)" \
  --data-urlencode "limit=1" \
  "$NOCODB_BASE_URL/api/v2/tables/m0thluc3jyz2iok/records"
```

**How to dead-letter the row:**

1. Verify the `.eml` truly is missing in SeaweedFS (use the `s3_get_text_optional` helper in `pipelines/common.py` against `s3://$SEAWEEDFS_S3_BUCKET/eml/live/...`).
2. If it is missing, re-fetch the message from Gmail via IMAP using the `Message-ID` (Gmail's IMAP supports `UID FETCH` by `Message-ID` only via header search; the easier path is `since_uid`).
3. Re-dispatch `live-pipeline.yml` with `since_uid` set to the lowest affected UID. The pipeline will re-archive the `.eml` (overwriting the same key, or PUTing a new one) and re-UPSERT the NocoDB row.
4. If the IMAP mailbox no longer has the email (it was deleted from Gmail), the row is orphaned. Mark it manually in NocoDB (set a `Notes` field with a date) and move on — do not bulk-delete rows, the audit trail matters.

### 5.3 NocoDB 4xx on a UPSERT

Read the response body (NocoDB returns JSON like `{"msg":"..."}`). Common causes and recovery:

| Cause                                              | Symptom                                                                              | Recovery                                                                                          |
|----------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|
| **Wrong table_id**                                 | 404 from `/api/v2/tables/<id>/records`                                               | Verify the table ID in `NOCODB_MODULE_TABLES` matches the live table ID (see §4.3). Re-bootstrap with `python scripts/apply_schemas.py` to print the current mapping. |
| **Missing required field**                         | 400 with `"<field> is required"`                                                     | Re-run `python scripts/apply_schemas.py` to add the column. UPSERT will retry on the next dispatch. |
| **Schema drift** (column renamed, type changed)    | 400 with `"<field> validation failed"`                                               | Edit `docs/schemas.md` first, then `scripts/apply_schemas.py`. Re-run the workflow.                |
| **`Message_ID` UNIQUE violation on a POST**        | This should not happen — `nocodb_upsert` does GET-then-PATCH/POST. If it does, race. | Re-run the workflow; the second pass will hit the GET and switch to PATCH.                         |
| **SingleSelect option not in the allowed list**    | 400 with `"<option> is not a valid option for <field>"`                              | Add the option to the SingleSelect column in NocoDB UI, then re-run.                              |

The full retry policy lives in `pipelines/common.py::nocodb_upsert` (tenacity, 5 attempts, exponential backoff 1s → 30s). A 4xx surfaces immediately and stops the run; the next dispatch retries from the same checkpoint.

### 5.4 Ollama 4xx / timeout

1. **Verify the endpoint is reachable:**
   ```bash
   curl -sS "$OLLAMA_BASE_URL/models" -H "Authorization: Bearer $OLLAMA_API_KEY"
   ```
   Expect 200 with a JSON list. The 4xx response body is usually self-explanatory (model not found, auth required, OOM on the Ollama host).
2. **If the model is missing** (the response lists other models but not the one in `INBOXOS_LLM_MODEL`):
   ```bash
   docker exec ollama ollama pull gemma4:e4b
   ```
3. **To switch the model** without redeploying: update the repo **variable** `INBOXOS_LLM_MODEL` (and `INBOXOS_LLM_MODEL_EXTRACT` for the larger extract model) in Gitea → repo → Settings → Actions → Variables. The next workflow run picks it up. Do not edit the prompt's `model:` frontmatter to "fix" a model outage — prompts are version-controlled, and that change goes through a PR.
4. **To raise the timeout:** set `INBOXOS_LLM_TIMEOUT` (in seconds, default 120). For very rich emails (86 KB Substack bodies) with the larger `INBOXOS_LLM_MODEL_EXTRACT` model, 180–240 s is not unusual.
5. **Check the Ollama host itself** for OOM:
   ```bash
   docker stats ollama --no-stream
   ```
   If memory is at the limit, the Ollama container is OOM-killing the loaded model. Either reduce the model size (`gemma4:e4b` instead of `gemma4:31b`) or add memory to the host.

### 5.5 Live pipeline falling behind

**Current symptom** (referenced in the INB-35 description): a single runner + 30+ minute cycles. The 5-minute cron cannot keep up because:

- The single runner is doing one job at a time (`cancel-in-progress: false`).
- IMAP UID backlog from a sync delay (e.g. Gmail was unreachable for an hour) means the next run has > 50 emails to process, and the runner is LLM-bound.

**Recovery options (in order of preference):**

1. **Add a second runner.** Already provisioned in `docker/docker-compose.yml` as `gitea-runner-2`; the only remaining step is the one-time registration token (see `docs/gitea-actions.md §2a` and [INB-33](/INB/issues/INB-33)). The historical workflow then runs in parallel on the second runner, freeing the first runner for the live cron.
2. **Bump the cron frequency.** Edit `.gitea/workflows/live-pipeline.yml` → `schedule` (e.g. `cron("*/2 * * * *")` for every 2 minutes). This is local-only, no board approval required. Note: 1 minute is the practical floor; Gitea will skip runs if the previous one is still in flight.
3. **Pause the cron** (board approval required). Use only when the pipeline is producing corrupted data or the LLM is rate-limiting. Two ways:
   - In the Gitea UI: repo → Actions → `live-pipeline.yml` → **Disable workflow**.
   - In code: open a PR that comments out the `schedule:` block; merge. Reverting the PR re-enables it.
4. **Bump `max_emails` per run** (default 50). The workflow input `max_emails` can be raised to 200, 500, etc. via `workflow_dispatch`. This is a one-shot — it does not change the default.

### 5.6 Historical pipeline run cancelled mid-way

**This is safe by construction.** The byte-offset checkpoint in `state/historical-state.json` is written every `BATCH_SIZE` messages (default 100) and on shutdown. A cancelled run:

1. Flushes whatever batch is currently in-flight to the checkpoint before exiting.
2. Leaves NocoDB and SeaweedFS in a consistent state — every successful `process_done` has already been committed.

**Recovery:** dispatch `historical-pipeline.yml` again with the **same** `mbox_object_key` and **no** `start_offset` input. The script will load `state/historical-state.json`, seek to the persisted byte offset, and continue.

**Do not** re-dispatch with `start_offset=0` unless you have intentionally deleted `state/historical-state.json`. Re-processing from offset 0 is safe (UPSERT dedupes) but wasteful — for a multi-GB `.mbox`, it will take hours and consume LLM tokens you do not need to spend.

To permanently restart a historical run from scratch (e.g. after a prompt overhaul you want to re-test):

```bash
# From a runner or any host with the S3 creds:
python -c "
import boto3, os
s3 = boto3.client('s3', endpoint_url=os.environ['SEAWEEDFS_S3_ENDPOINT'],
                  aws_access_key_id=os.environ['SEAWEEDFS_S3_ACCESS_KEY'],
                  aws_secret_access_key=os.environ['SEAWEEDFS_S3_SECRET_KEY'])
s3.delete_object(Bucket=os.environ['SEAWEEDFS_S3_BUCKET'], Key='state/historical-state.json')
print('state/historical-state.json deleted; next dispatch starts from offset 0')
"
```

Then dispatch `historical-pipeline.yml` with `start_offset=0` and the same `mbox_object_key`.

### 5.7 Backup and restore (REQ-INB-22)

The `scripts/backup.sh` and `scripts/restore.sh` helpers
snapshot and restore the three stateful services (NocoDB,
SeaweedFS, Gitea) to/from a timestamped directory under
`backups/<service>/<utc-timestamp>/`. The helpers are
idempotent: a re-run for the same service at the same UTC
timestamp is a no-op.

**Schedule.** A daily `cron` job on the operator host calls the
three backup helpers in sequence:

```cron
# /etc/cron.d/inboxos-backup — runs at 02:00 UTC daily.
0 2 * * *  cto  /opt/inboxos/scripts/backup.sh nocodb    >> /var/log/inboxos-backup.log 2>&1
5 2 * * *  cto  /opt/inboxos/scripts/backup.sh seaweedfs >> /var/log/inboxos-backup.log 2>&1
10 2 * * * cto  /opt/inboxos/scripts/backup.sh gitea     >> /var/log/inboxos-backup.log 2>&1
```

**Retention.** 7 daily + 4 weekly + 0 monthly. The single-
operator posture is "the operator can change it" — the values
are the v1.1 ship default and are documented in
`scripts/backup.sh` header comment.

**Copy-paste invocation.**

```bash
# Backup one service.
scripts/backup.sh nocodb
scripts/backup.sh seaweedfs
scripts/backup.sh gitea

# Restore from a backup id (UTC timestamp).
scripts/restore.sh 2026-06-08T120000Z nocodb
scripts/restore.sh 2026-06-08T120000Z seaweedfs
scripts/restore.sh 2026-06-08T120000Z gitea
```

**Test-or-die gate.** A restore is not considered safe until
`pipelines/_tests/test_restore_offline.py` exits 0. The test
pins the exit-code table for missing args, missing env, missing
backup directory, and the SeaweedFS "no backup target configured"
path. It also exercises a git-bundle round-trip on an in-tree
repo. The full end-to-end (backup → wipe → restore → verify)
requires a live NocoDB / SeaweedFS / Gitea and is run by the
operator on the host.

---

## 6. Security and secrets

### 6.1 Secret inventory

All Gitea repo secrets live at **Gitea → repo → Settings → Actions → Secrets** (and a parallel **Variables** section for non-secret tunables). The 15 secrets consumed by the workflows, grouped by owner:

| Owner        | Secret                              | Used by   | Purpose                                                       |
|--------------|-------------------------------------|-----------|---------------------------------------------------------------|
| IMAP         | `IMAP_HOST`                         | live      | Gmail IMAP server (`imap.gmail.com`)                          |
| IMAP         | `IMAP_PORT`                         | live      | `993`                                                          |
| IMAP         | `IMAP_USER`                         | live      | Gmail address                                                  |
| IMAP         | `IMAP_PASSWORD`                     | live      | Gmail App Password (not the account password; 2FA required)    |
| IMAP         | `IMAP_USE_SSL`                      | live      | `true`                                                         |
| IMAP         | `IMAP_MAILBOX`                      | live      | `INBOX` (default)                                              |
| SeaweedFS    | `SEAWEEDFS_S3_ENDPOINT`             | both      | `https://storage.cloudinit.dev` (or `http://seaweedfs-filer:8888` in-network) |
| SeaweedFS    | `SEAWEEDFS_S3_ACCESS_KEY`           | both      | S3 access key                                                  |
| SeaweedFS    | `SEAWEEDFS_S3_SECRET_KEY`           | both      | S3 secret key                                                  |
| SeaweedFS    | `SEAWEEDFS_S3_BUCKET`               | both      | `inboxos`                                                       |
| NocoDB       | `NOCODB_BASE_URL`                   | both      | `https://noco.cloudinit.dev`                                   |
| NocoDB       | `NOCODB_API_TOKEN`                  | both      | `xc-token` header value                                        |
| NocoDB       | `NOCODB_PROJECT_ID`                 | both      | NocoDB project ID                                               |
| NocoDB       | `NOCODB_MODULE_TABLES`              | both      | JSON map of module name → NocoDB table ID                     |
| Ollama       | `OLLAMA_BASE_URL`                   | both      | `https://ollama-frp.cloudinit.dev/v1`                          |
| Ollama       | `OLLAMA_API_KEY`                    | both      | Bearer token if Ollama is auth-gated; empty string otherwise   |
| Papra        | `PAPRA_BASE_URL`                    | both      | `https://papra.cloudinit.dev` (optional)                       |
| Papra        | `PAPRA_API_TOKEN`                   | both      | Bearer token (optional)                                         |
| Papra        | `PAPRA_ORGANIZATION_ID`             | both      | Papra org id (optional)                                         |
| Gitea        | `GITEA_INSTANCE_URL`                | both      | `http://gitea:3000` (in-network)                                |
| Gitea        | `GITEA_USER` / `GITEA_PASS`         | scripts   | HTTP Basic Auth for the Gitea REST API (see `common.gitea_request`) |

That's 22 distinct secret values (the v1.0 ship counted 21 because the `GITEA_USER` / `GITEA_PASS` row was treated as one credential; v1.1 counts both values to match the row count in the table above). The issue description rounds to 15 because the Papra trio is optional; the table is exhaustive.

**Rotation schedule:** **None — all static.** This is an explicit, accepted trade-off for a single-operator, self-hosted system. The S3 access keys and NocoDB API token are long-lived. The Gmail App Password can be revoked from the Google account at any time without breaking anything else; the IMAP user must re-issue a new App Password and update the Gitea secret. The helper landed in v1.1 (REQ-INB-21); see §6.2 for the procedure.

### 6.2 Secret rotation (REQ-INB-21)

The `scripts/rotate_secrets.sh` helper rotates a Gitea repo secret in
place via `pipelines.common.gitea_request`. The new value is read
from **STDIN** (never a CLI argument) so it is not visible in
`ps` / process listings.

**Rotation ordering.** When multiple secrets need rotating, do them
in this order (each step invalidates fewer downstream services than
the next):

1. `NOCODB_API_TOKEN` first — it is the PII-bearing token (every
   NocoDB write is gated on it; rotate during low-traffic). A
   failed rotation here dead-letters new writes; rotate the
   pre-shared token in NocoDB and update the Gitea secret.
2. `SEAWEEDFS_S3_ACCESS_KEY` and `SEAWEEDFS_S3_SECRET_KEY`
   (treated as a pair) — the S3 archive and the pipeline state
   files are gated on these. A failed rotation here stops both
   pipelines; rotate the pre-shared key in SeaweedFS and update
   the Gitea secrets.
3. `IMAP_PASSWORD` — the Gmail App Password. A failed rotation
   here stops the live pipeline only; the historical pipeline is
   unaffected. Update the Gmail App Password and the Gitea secret.
4. `GITEA_USER` / `GITEA_PASS` last — the Basic Auth credentials
   that the helper itself uses. Rotating these invalidates the
   helper's own running session; do them after the helper has
   returned successfully for the other three.

**Test-or-die gate.** A rotation is not considered safe until
`pipelines/_tests/test_rotate_secrets_offline.py` exits 0. The
test exercises the dry-run path, the apply branch shape, and the
env-validation failure mode. The dry-run path is the one operators
should run first; it prints a redacted diff of the new value
(`sha256:8hex`) without making the PUT.

**Copy-paste invocation.**

```bash
# Dry-run first; no PUT issued.
echo "$NEW_NOCODB_API_TOKEN" | scripts/rotate_secrets.sh NOCODB_API_TOKEN

# Real rotation; PUTs the new value, then verifies via GET.
echo "$NEW_NOCODB_API_TOKEN" | scripts/rotate_secrets.sh --apply NOCODB_API_TOKEN
```

The helper exits 0 on success, 2 on missing env / missing value,
and 3 on Gitea API failure. See
`pipelines/_tests/test_rotate_secrets_offline.py` for the test
contract and the helper's exit-code table in its header comment.

### 6.3 Why we never commit secrets

- `.gitignore` excludes `.env` and `.env.*` except `.env.example`. The example file contains placeholder values only.
- The CTO host deliberately does **not** have write-capable S3 credentials (only the read-only S3 keys used for verification). A leaked CTO host key cannot corrupt the `.eml` archive.
- The Gitea service account used for the REST API (`GITEA_USER=cto` in `.env.example`) is scoped to the `jc-capital/inboxos` repo. It can read/write workflow secrets, but it cannot create new repos or admin the Gitea instance.
- All secret reads happen via `${{ secrets.X }}` in the workflow YAMLs; the values are injected as environment variables into the ephemeral runner container and never appear in the build log.

### 6.4 Data-sovereignty posture

- **No external API calls for PII-bearing data.** The only outbound network calls from the runner are to the in-network services (`http://gitea:3000`, `http://nocodb:8080`, `http://seaweedfs-filer:8888`, `http://ollama:11434`, `http://papra:1221`). The Ollama URL is FRP-tunneled to the local Ollama container — the FRP tunnel terminates inside the operator's network; no third-party LLM provider sees the request body.
- **Local LLM inference.** Default model is `gemma4:e4b` (4 B effective parameters). All `inboxos:classify` and `inboxos:extract-*` calls hit the local Ollama. There is no fallback to a cloud LLM. If Ollama is down, the pipeline dead-letters and the next run retries.
- **No telemetry from any pipeline.** `pipelines/common.py` does not call home. Gitea Actions emits standard runner logs, which are stored on the Gitea host (also self-hosted).

---

## 7. Cost and capacity

### 7.1 Per-message cost

**~$0 / message.** All inference is local (Ollama on the operator's host). The only recurring costs are:

| Cost                                          | Magnitude (single operator)                                   |
|-----------------------------------------------|----------------------------------------------------------------|
| Compute (Ollama host + Gitea host + runners)  | One bare-metal or small VM; existing infrastructure.           |
| Bandwidth (Gmail IMAP, FRP tunnel to Ollama)  | Negligible at < 100 emails / day.                              |
| Storage (SeaweedFS volume)                    | See §7.2.                                                      |
| LLM API costs                                 | **$0** — local inference.                                      |

### 7.2 Storage growth

- A typical personal email `.eml` is ~30–100 KB. 100 KB / email is the safe upper bound used in capacity planning.
- **10 000 emails ≈ 1 GB** of raw `.eml` in SeaweedFS.
- The SeaweedFS volume is provisioned at `seaweedfs-data` (named Docker volume); size is bounded by the host's disk. No quota is enforced at the SeaweedFS level.
- The NocoDB side is tiny: even 10 000 rows across four tables is single-digit MB.

### 7.3 Compute

- **One Gitea runner** (now two, post-[INB-33](/INB/issues/INB-33)), one Ollama container with `gemma4:e4b` loaded.
- **Throughput observed (live):** ~22 messages / 36 min on a single-runner + IMAP-backlog-saturated cycle. This is LLM-bound; each `inboxos:classify` + 1–2 `inboxos:extract-*` calls adds up to ~90 s per email with `gemma4:e4b` on CPU. A GPU host would cut this roughly 5–10× but is not required at the current scale.
- **Throughput observed (historical):** ~5–10 min per 10 K messages on `gemma4:e4b` CPU, dominated by the LLM calls. Switching to a smaller model (e.g. `gemma3:4b`) would roughly halve this; switching to a larger model (`gemma4:12b`) would roughly double it. Prompt quality and JSON-contract compliance are the binding constraints, not raw tokens/second.

---

## 8. Glossary and appendix

### 8.1 Glossary

- **Message-ID** — The RFC 5322 `Message-ID:` header, with angle brackets stripped. The natural primary key for every module table in NocoDB and the natural key for the `.eml` S3 key.
- **UPSERT** — "Update or insert." `pipelines.common.nocodb_upsert` does this by doing a GET by `Message_ID` and then a PATCH (if found) or POST (if not). There is no plain `INSERT` path in the codebase. The CSV / SQL concept maps 1:1 to the GET-then-PATCH/POST pattern.
- **mbox** — A plain-text file format for archiving email. Each message begins with `From <mailbox> <date>` and ends with a blank line. The historical pipeline uses `mbox_stream.py` to read it as a byte stream, never materialising the full file in memory.
- **IMAP** — Internet Message Access Protocol. The live pipeline uses `IMAPClient` against Gmail's `imap.gmail.com:993` (SSL). Polling, not push, because Gmail's push (IDLE) is unreliable.
- **FRP tunnel** — Fast Reverse Proxy. The Ollama service is exposed at `https://ollama-frp.cloudinit.dev/v1` via FRP, terminating inside the operator's network. Lets the runner talk to Ollama over HTTPS without opening inbound ports.
- **Checkpoint state** — The persisted byte offset (`historical-state.json`) or IMAP UID (`live-state.json`) that lets a pipeline resume from where it left off after a crash. The state file is the only persistent state between workflow runs.
- **Dead-letter** — A poison message or one that could not be processed is moved to `s3://$SEAWEEDFS_S3_BUCKET/eml/dead-letter/...` with a marker file describing the reason. The cursor advances so the rest of the email stream is not blocked. Dead-lettered items require manual review.
- **Idempotency** — "Running the same operation twice has the same effect as running it once." Achieved in InboxOS by (a) Message-ID UPSERT in NocoDB, (b) deterministic S3 keys, and (c) checkpoint-based resume. A redelivered Gmail message, a re-dispatched workflow, and a replayed `.mbox` are all safe.

### 8.2 Cross-references

- **Schemas:** `docs/schemas.md` — NocoDB table schemas (column-by-column).
- **Extraction contract:** `docs/extraction-contract.md` — how a raw email becomes a structured NocoDB record, the prompt family, and failure semantics.
- **Gitea Actions:** `docs/gitea-actions.md` — runner provisioning, second-runner setup, secrets, the Basic Auth model, observability, rollback.
- **Pipelines:** `pipelines/README.md` — pipeline flow diagrams, idempotency model, rollback procedure.
- **Papra write policy:** `docs/papra-write-policy.md` — when the pipeline writes to Papra (attachments, body for crm/finance/legal modules), the dry-run default, the live-write gate (REQ-INB-20), and the `papra_index` audit table (REQ-INB-19).
- **Phase 1.1 verification:** `docs/phase-1.1-verification/inb-28-report.md` — the schema-bootstrap deliverable that locked in the four table IDs and column counts cited in §4.3.
- **Secret rotation:** `docs/operations-runbook.md §6.2` (new in v1.1, renumbered in v1.3 from §6.4) — the rotation helper, the ordering, and the test-or-die gate (REQ-INB-21).
- **Metrics surface:** `docs/operations-runbook.md §4.5` (new in v1.1) — the per-run `metrics.jsonl` shape and the tail-and-summarize procedure (REQ-INB-23).
- **Backup and restore:** `docs/operations-runbook.md §5.7 + §10` (new in v1.1) — the `backup.sh` / `restore.sh` helpers, the daily cron, and the RTO < 4h / RPO < 24h targets (REQ-INB-22).

### 8.3 Quick-reference: log-event search strings

If a post-mortem needs to find every place a particular event is emitted:

```bash
grep -rn 'log_event(logger, "process_done"'   pipelines/
grep -rn 'log_event(logger, "classified"'     pipelines/
grep -rn 'log_event(logger, "module_table_missing"' pipelines/
grep -rn 'log_event(logger, "dead_letter_unclassified"' pipelines/
grep -rn 'log_event(logger, "extract_permanent_failure"' pipelines/
```

For the full event vocabulary, `grep -n "log_event" pipelines/common.py pipelines/live_pipeline.py pipelines/historical_pipeline.py`.

### 8.4 Quick-reference: env vars (in alphabetical order)

| Env var                          | Default                                  | Purpose                                                                 |
|----------------------------------|------------------------------------------|-------------------------------------------------------------------------|
| `GITEA_INSTANCE_URL`             | — (required)                             | Gitea REST API base. `http://gitea:3000` in-network.                    |
| `GITEA_USER`                     | — (required for Gitea REST calls)        | HTTP Basic Auth user.                                                   |
| `GITEA_PASS`                     | — (required for Gitea REST calls)        | HTTP Basic Auth password.                                               |
| `IMAP_HOST` / `IMAP_PORT`        | — / `993`                                | Gmail IMAP endpoint.                                                    |
| `IMAP_USER` / `IMAP_PASSWORD`    | —                                        | Gmail App Password credentials.                                          |
| `IMAP_USE_SSL` / `IMAP_MAILBOX`  | `true` / `INBOX`                         | IMAP connection toggles.                                                |
| `INBOXOS_LLM_BACKEND`            | `ollama`                                 | `ollama` (recommended) or `opencode` (legacy, broken on 1.16.2).        |
| `INBOXOS_LLM_MODEL`              | `gemma4:e4b`                             | Default model for classify and any non-extract command.                 |
| `INBOXOS_LLM_MODEL_EXTRACT`      | `gemma4:e4b`                             | Model for `inboxos:extract-*` commands.                                 |
| `INBOXOS_LLM_MAX_TOKENS`         | `4096`                                   | Completion budget. 1024 was too tight for the 11-field extract schema. |
| `INBOXOS_LLM_TIMEOUT`            | `120`                                    | Seconds per LLM call.                                                   |
| `NOCODB_API_TOKEN`               | —                                        | `xc-token` header value.                                                |
| `NOCODB_BASE_URL`                | —                                        | `https://noco.cloudinit.dev`                                            |
| `NOCODB_MODULE_TABLES`           | —                                        | JSON map `{"finance":"tbl_…","crm":"tbl_…",…}`.                         |
| `NOCODB_PROJECT_ID`              | —                                        | NocoDB project ID.                                                      |
| `OLLAMA_API_KEY`                 | unset                                    | Bearer token if Ollama is auth-gated.                                   |
| `OLLAMA_BASE_URL`                | `https://ollama-frp.cloudinit.dev/v1`    | OpenAI-compatible chat completions endpoint.                            |
| `OPENCODE_BIN`                   | `opencode`                               | CLI binary used by `opencode_run_command` (legacy path only).           |
| `OPENCODE_PROMPT_DIR`            | `./.opencode/commands`                   | Where the `.opencode/commands/inboxos:*.md` prompts live.               |
| `PAPRA_API_TOKEN`                | unset                                    | Bearer token for Papra uploads.                                          |
| `PAPRA_BASE_URL`                 | unset                                    | Papra API base.                                                          |
| `PAPRA_BODY_WRITE_MODULES`       | `crm,finance,legal`                      | Comma-separated modules whose body (not attachments) gets uploaded.      |
| `PAPRA_DRY_RUN`                  | `true`                                   | `false` to enable real Papra writes (CEO-approved only).                 |
| `PAPRA_METADATA_ONLY_MODULES`    | `intel,promotions`                       | Modules whose email is *not* written to Papra (metadata only).          |
| `PAPRA_ORGANIZATION_ID`          | unset                                    | Papra org id.                                                            |
| `SEAWEEDFS_S3_ACCESS_KEY`        | —                                        | S3 access key.                                                          |
| `SEAWEEDFS_S3_BUCKET`            | —                                        | `inboxos` (default).                                                |
| `SEAWEEDFS_S3_ENDPOINT`          | —                                        | `https://storage.cloudinit.dev` (or in-network `http://seaweedfs-filer:8888`). |
| `SEAWEEDFS_S3_SECRET_KEY`        | —                                        | S3 secret key.                                                          |

---

## 9. Phase 5 acceptance gate

> This appendix is the **must-have checklist** that closes
> InboxOS Phase 5 (Full Deployment — INB-7). It is the integrator's
> reference: every item must be checkbox-green before the
> `.ciagent/inboxos/ROADMAP.md` Phase 5 status row is flipped to
> "executed". The 20 must-haves below were produced by the
> CIAgent plan at `.ciagent/inboxos/plans/05-PLAN.md` and verified
> during the Wave 1 + Wave 2 + Wave 3 execution pass (see git log
> on `milestone/v1.0-initial`).

### 9.1 Requirements coverage

| #  | Must-have (verifiable) | Source-of-truth location | REQ-INB | Owner |
|----|------------------------|--------------------------|:-------:|-------|
|  1 | `pipelines/live_pipeline.py` runs on a 5-minute cron and on `workflow_dispatch` | `.gitea/workflows/live-pipeline.yml:62-65` | REQ-INB-1 | devops-engineer |
|  2 | `pipelines/historical_pipeline.py` is `workflow_dispatch` only and streams a local `.mbox` end-to-end | `.gitea/workflows/historical-pipeline.yml:54-58` | REQ-INB-2 | devops-engineer |
|  3 | All NocoDB writes are GET-then-PATCH/POST on `Message_ID`; no plain INSERT path | `pipelines/common.py:196-255` | REQ-INB-3 | data-engineer |
|  4 | `inboxos:classify` + `inboxos:extract-<module>` is the only extraction path | `pipelines/common.py:632-653`, `docs/extraction-contract.md` | REQ-INB-4 | ml-engineer |
|  5 | Four module tables (`finance`, `crm`, `intel`, `promotions`) with the column set per `docs/schemas.md` | `docs/schemas.md`, `docs/phase-1.1-verification/inb-28-report.md` | REQ-INB-5 | data-engineer |
|  6 | `docs/operations-runbook.md` is the canonical day-1 incident reference | this file | REQ-INB-6 | lead-developer |
|  7 | Two self-hosted Gitea Actions runners are provisioned and online | `docker/docker-compose.yml:103-148`, `docs/gitea-actions.md §2` | REQ-INB-7 | devops-engineer |
|  8 | `ci-pr.yml` runs fast unit tests + 100MB e2e + live Ollama smoke on every PR | `.gitea/workflows/ci-pr.yml:63-230` | REQ-INB-8 | devops-engineer |
|  9 | Papra integration is dry-run by default; the first ship does not call the real Papra API | `pipelines/common.py:729`, `docs/papra-write-policy.md` | REQ-INB-9 | security-engineer |
| 10 | No third-party SaaS touches PII; only the IMAP connection to Gmail leaves the self-hosted perimeter | `docker/docker-compose.yml`, this runbook §6 | REQ-INB-10 | security-engineer |
| 11 | A single pinned Ollama model (`gemma4:e4b`) is the production extraction path; no model-escalation chain | `pipelines/common.py:467-468`, `.opencode/commands/inboxos:*.md` frontmatter | REQ-INB-11 | ml-engineer |
| 12 | `actions/upload-artifact` is pinned to v3 in the workflows that emit per-run logs (live + historical); `ci-pr.yml` has no `upload-artifact` step and inherits GHES compat from the runner image, not from the workflow pin | `.gitea/workflows/live-pipeline.yml:150` (`uses: actions/upload-artifact@v3`), `.gitea/workflows/historical-pipeline.yml:170` (same). `ci-pr.yml` has no `upload-artifact` step; the runbook's previous glob was misleading. | REQ-INB-12 | devops-engineer |
| 13 | A synthetic UPSERT test exists and exercises the NocoDB idempotency path end-to-end | `pipelines/_tests/test_upsert_synthetic.py` | REQ-INB-13 | data-engineer |
| 14 | All extraction prompts are version-controlled in `.opencode/commands/` and invoked by name | `.opencode/commands/inboxos:*.md` (5 files), `pipelines/common.py::llm_run_command` | REQ-INB-14 | ml-engineer |

### 9.2 Reconciliation drift closure

The seven doc/code drifts surfaced in research (P-1 through P-7)
were closed during the Wave 1 + Wave 2 execution pass:

| #  | Drift | Closed by | PR | REQ-INB |
|----|-------|-----------|----|:-------:|
| 15 | Model pin drift (`lfm2.5:8b` → `gemma4:e4b` in docs) | `docs: align pipelines/README + extraction-contract to gemma4:e4b pin` | #A | REQ-INB-11 |
| 16 | `OLLAMA_BASE_URL` `/v1` suffix canonicalised | `docs: align pipelines/README + extraction-contract to gemma4:e4b pin` (folded into #A) | #A | REQ-INB-11, REQ-INB-10 |
| 17 | Papra `INTAKE_EMAILS_IS_ENABLED` flipped to `false` in compose | `docker: disable Papra catch-all intake; align with cancelled policy` | #E | REQ-INB-9 |
| 18 | State-file field names in runbook §4.4 corrected | `docs(runbook §4.4): correct state-file field names` | #H | REQ-INB-3, REQ-INB-6 |
| 19 | Live-vs-historical `Papra_Link` asymmetry documented | `docs: document live-vs-historical Papra_Link asymmetry` + `docs(extraction-contract)` — owner: backend-engineer (the P-7 `papra_decide` investigation; the asymmetry is structural, the doc is the deliverable) | #C, #B | REQ-INB-4, REQ-INB-9 |
| 20 | Extraction-contract rewritten to describe Ollama as production LLM caller; opencode marked legacy | `docs(extraction-contract): describe Ollama as production LLM caller; opencode marked legacy` | #B | REQ-INB-4 |

> The `papra_decide` behavioural asymmetry in `live_pipeline.py`
> (P-7 in the research brief) was investigated and found to be a
> no-op: the live pipeline already calls `papra_decide` exactly
> once with the post-classify `modules` list
> (`pipelines/live_pipeline.py:490`). The historical pipeline
> calls it twice (pre-classify attachment walk + post-classify
> body write), but the live pipeline has no pre-classify walk.
> Decision recorded as `ED-1` in `.ciagent/inboxos/CI-LOG.md`.

### 9.3 Closing the gate

A Phase 5 close is the date on which all 20 must-haves are
checkbox-green and the `.ciagent/inboxos/ROADMAP.md` Phase 5
status row is flipped to "executed". The lead-developer is the
integrator; every active persona (data-engineer, backend-engineer,
devops-engineer, ml-engineer, security-engineer, lead-developer)
gets a review pass on this section.

### 9.4 v1.1 acceptance gate

> This appendix is the **must-have checklist** that closes
> InboxOS v1.1 Phase 1. It is the integrator's reference: every
> item must be checkbox-green before the v1.1 ship merge. The
> 11 must-haves below were produced by the CIAgent plan at
> `.ciagent/inboxos/plans/01-PLAN.md` and verified during the
> Wave 1 + Wave 2 + Wave 3 execution pass (see git log on
> `phase/01-v1.1-execute`).
>
> The 11 candidates are `[CANDIDATE]` until the CPO authors the
> matching entries in `docs/requirements.md`; the must-haves
> below reference the candidate IDs as a planning aid. The
> `.ciagent/inboxos/ROADMAP.md` Phase 1 status row is not
> flipped by the lead-developer; the flip is the CPO's call.
> The lead-developer's gate is "every must-have is checkbox-
> green and the 11 PRs are merged".

| #  | Must-have (verifiable) | Source-of-truth location | Owner | REQ-INB |
|----|------------------------|--------------------------|-------|:-------:|
|  1 | `.env.example` lines 96-99 reference `gemma4:e4b` only (no `lfm2.5:8b` or `gemma4:12b` comments) | `.env.example:96-99` | devops-engineer | REQ-INB-15 |
|  2 | Runbook §9.1 #12 cites `upload-artifact@v3` only in the workflows that emit per-run logs (live + historical); `ci-pr.yml` excluded | `docs/operations-runbook.md:544` (the row updated in T1.2) | devops-engineer | REQ-INB-16 |
|  3 | Runbook §6.1 says "22 distinct secret values" (matching the 22 data rows in the table) | `docs/operations-runbook.md:399` (the sentence updated in T1.3) | devops-engineer | REQ-INB-17 |
|  4 | `inb-28-report.md` "Per-table row counts at freeze" is filled with one row per module table | `docs/phase-1.1-verification/inb-28-report.md:67-77` | data-engineer | REQ-INB-18 |
|  5 | `papra_index` NocoDB table exists with the 7 documented columns; `docs/schemas.md` and `scripts/apply_schemas.py` reflect the table; the v1.0 schema freeze is preserved for the 4 module tables | `docs/schemas.md` (new `papra_index` section, T1.4a), `scripts/apply_schemas.py` (new `AUDIT_TABLES` entry, T1.4b), `docs/phase-1.1-verification/inb-28-report.md` (the freeze amendment, T1.4a) | data-engineer | REQ-INB-19 |
|  6 | `PAPRA_DRY_RUN_OVERRIDE` env var exists; the new `papra_write_gate` event is emitted by both pipelines (via the T1.5 helper + T2.2 call-sites); the canary procedure is in `docs/papra-write-policy.md`; new test cases exercise the override and the helper | `pipelines/common.py` `PapraConfig` + `emit_papra_write_gate_event` (T1.5), `pipelines/live_pipeline.py` + `pipelines/historical_pipeline.py` (T2.2 call-sites), `pipelines/_tests/test_papra_decide_offline.py` (T1.5 + T2.2 cases), `docs/papra-write-policy.md` "Live-write gate" (T1.5) | security-engineer | REQ-INB-20 |
|  7 | `scripts/rotate_secrets.sh` exists with `--dry-run` and real-run modes; `pipelines/_tests/test_rotate_secrets_offline.py` passes; `docs/operations-runbook.md §6.2` documents the rotation ordering | `scripts/rotate_secrets.sh` (new), `pipelines/_tests/test_rotate_secrets_offline.py` (new, 3/3 tests pass), `docs/operations-runbook.md §6.2` (new) | security-engineer | REQ-INB-21 |
|  8 | `scripts/backup.sh` and `scripts/restore.sh` exist; `pipelines/_tests/test_restore_offline.py` round-trips a known fixture; `docs/operations-runbook.md §5.7 + §10` state RTO < 4h and RPO < 24h | `scripts/backup.sh` (new), `scripts/restore.sh` (new), `pipelines/_tests/test_restore_offline.py` (new, 7/7 tests pass), `docs/operations-runbook.md §5.7 + §10` (new) | devops-engineer | REQ-INB-22 |
|  9 | `pipelines/_metrics/emit.py` writes one JSONL line per run with the documented counter and gauge vocabulary; `docs/operations-runbook.md §4.5` documents the tail-and-summarize procedure; a new test exercises the emitter | `pipelines/_metrics/emit.py` (new), `pipelines/live_pipeline.py` + `pipelines/historical_pipeline.py` (T2.2 call-sites), `pipelines/_tests/test_metrics_offline.py` (new, 2/2 tests pass), `docs/operations-runbook.md §4.5` (new) | data-engineer | REQ-INB-23 |
| 10 | `plans/05-PLAN.md` parallelization table has the explicit three-file split footnote (P1-1 closure); runbook §9.2 row 19 has the backend-engineer persona annotation (P2-4 closure); CI-LOG P1-1 + P2-4 entries are marked closed | `.ciagent/inboxos/plans/05-PLAN.md:318-330` (footnote), `docs/operations-runbook.md:674` (row 19 owner annotation), `.ciagent/inboxos/CI-LOG.md:96-97,112-113,115` (closed_in pointers) | lead-developer | REQ-INB-24 |
| 11 | The live pipeline's `process_one_message` patches `Papra_Link` onto every NocoDB row after a successful `papra_write_attachment`; the "Live pipeline note" paragraph in `docs/papra-write-policy.md §Rule 1` is removed; the asymmetry paragraph in `docs/extraction-contract.md §Live vs historical: Papra_Link asymmetry` is removed; `ARCHITECTURE.md §Document Store` is updated to reflect parity | `pipelines/live_pipeline.py:486-555` (the new PATCH loop + `papra_link_patched` event, T1.7), `docs/papra-write-policy.md:55-72` (the §Rule 1 asymmetry note removed by T2.3), `docs/extraction-contract.md:155-181` (the subsection removed by T1.7), `.ciagent/inboxos/ARCHITECTURE.md:65-72` (the paragraph replaced by T1.7) | backend-engineer | REQ-INB-25 |

A v1.1 close is the date on which all 11 must-haves are
checkbox-green and the CPO has promoted the 11 candidates to
real requirements in `docs/requirements.md`. The
`.ciagent/inboxos/ROADMAP.md` Phase 1 status row is not flipped
to "executed" by the lead-developer; the flip is the CPO's
call. The lead-developer's gate is "every must-have is
checkbox-green and the 11 PRs are merged".

---

## 10. Disaster recovery targets (REQ-INB-22)

The backup / restore procedure in §5.7 is the substrate; this
section is the **commitment**.

- **RTO (Recovery Time Objective):** **< 4 hours**, single-
  operator restore. The breakdown:
  - 30 min — re-read §5.7 + §6.2 and decide which services
    to restore (typically: all three, in order: Gitea first
    so the workflows and secrets are back, then NocoDB so
    the module-table IDs are available, then SeaweedFS).
  - 60 min — restore the Gitea bundle and force-push the
    `jc-capital/inboxos` repo. Verify the workflows are
    visible at the Gitea UI.
  - 60 min — restore the NocoDB export. The export is a
    JSON dump; the operator opens the NocoDB UI and uses
    Settings → Import to load the JSON. Verify a known row
    via the `curl ... where=(Message_ID,eq,...)` pattern
    in §5.2.
  - 60 min — restore the SeaweedFS mirror. The `aws s3 sync`
    (or boto3 fallback) copies every object from the
    backup bucket to the primary bucket. Verify a known
    `.eml` via the `s3_get_text_optional` helper.
  - 30 min — buffer for unexpected drift (single-operator
    runs are not 24/7; if the restore starts at 02:00 UTC
    the operator is fresh by 04:00 UTC).

- **RPO (Recovery Point Objective):** **< 24 hours** data
  loss. The daily backup cron (§5.7) runs at 02:00 UTC, so
  the worst-case loss is the previous 24 hours of incoming
  email. In a typical single-operator inbox this is < 50
  messages, all recoverable from Gmail via `since_uid` (the
  live pipeline's recovery path in §5.2).

The RTO and RPO values are explicit and CPO-acknowledged.
The ideation report Tier 2 T2-C marked them as "guesses"
pending CPO sign-off; the values recorded here are the
v1.1 ship defaults. Changes to the values require a new
CPO-approved REQ-INB (the values are not engineer-editable
in isolation).

---

*End of runbook. Next review: after the v1.3 ship (2026-06-08, milestone `milestone/v1.3-initial`), or when the next failure mode is added to §5. Last verified against the source files cited inline; any divergence is a bug in this document — fix the doc, do not just edit around it.*