coreci/inboxos-v1
18
0
Fork 0
Code Issues Pull Requests Actions Packages Projects 1 Releases 3 Wiki Activity
Files
main
inboxos-v1/docs/operations-runbook.md
T
cloudinit-bot dc435f8d71 docs(security): threat-model.md — STRIDE per component (REQ-INB-59) 
---ci---
project: inboxos
phase: 1
milestone: v1.4
status: execute
wave: 3
task: T2
---/ci---
2026-06-09 14:23:32 +00:00

858 lines
70 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# InboxOS — Operations Runbook
**Audience:** Any operator (or the same operator after a 6-month gap) recovering from common failure modes without re-deriving the architecture.
**Status:** Phase 5 deliverable for [INB-7](/INB/issues/INB-7) — issued under [INB-35](/INB/issues/INB-35).
**Scope:** Single self-hosted pipeline. A Gmail inbox → IMAP poll → SeaweedFS archive + LLM-extracted rows in NocoDB + curated attachments in Papra. Everything is self-hosted; no third-party SaaS touches PII.
**Acceptance test for this runbook:** *"Given a workflow is stuck in `in_progress` for 1 hour, what do you do?"* — answer in under 5 minutes using §5.1 below.
---
## 1. System overview
InboxOS is a self-hosted, air-gapped personal ERP/CRM pipeline. It polls a single Gmail inbox over IMAP, backs up the raw `.eml` to SeaweedFS, classifies each email with a local LLM (Ollama), runs one or more `inboxos:extract-<module>` prompts against the same email, and UPSERTs the resulting structured records into NocoDB (one table per module: `finance`, `crm`, `intel`, `promotions`). Qualifying attachments are also uploaded to Papra for long-term searchability. The historical pipeline performs the same flow against a Google Takeout `.mbox` for backfill, with byte-offset checkpoints for safe resume.
### 1.1 Topology
```
┌──────────────┐ IMAP/993 ┌─────────────────┐
│ Gmail │ ◀───────────── │ gitea-runner-1 │
│ inbox │ (App Password)│ gitea-runner-2 │ (self-hosted, parallel)
└──────────────┘ └────────┬────────┘
│ run jobs (Gitea Actions)
│ polls IMAP, calls Ollama
▼
┌──────────────────┐ raw .eml ┌─────────────────┐
│ Papra │ attachment │ SeaweedFS │ S3-compatible
│ (long-term doc)│ ◀───────────│ (S3 API) │ .eml archive +
│ intake │ uploads │ │ state checkpoints
└──────────────────┘ └────────┬────────┘
│ UPSERT by Message-ID
▼
┌─────────────────┐ direct HTTP
│ NocoDB │ ── /v1/chat
│ (DWH) │ /completions
└─────────────────┘ │
▲ ▼
┌─────────────────┐
│ Ollama │ local LLM
│ (FRP) │ gemma4:e4b
│ /v1/chat │
│ /completions │
└─────────────────┘
```
- **Gmail** is the source. The runner authenticates with an App Password (2FA must be on).
- **Gitea** is the version-controlled home for prompts, scripts, and CI.
- **Gitea Actions runners** (`gitea-runner-1` and `gitea-runner-2`, both self-hosted) execute the live and historical pipelines. Two runners so the 5-minute live cron and a long historical run can execute in parallel.
- **SeaweedFS** is the system of record for raw `.eml` (in `s3://$SEAWEEDFS_S3_BUCKET/eml/...`) and pipeline state (`state/live-state.json`, `state/historical-state.json`).
- **NocoDB** is the structured warehouse. Every module table is keyed on `Message_ID` (UNIQUE). All writes are UPSERTs.
- **Papra** holds curated attachments and (selectively) full email bodies for long-term search. The first ship goes out in **dry-run** mode.
- **Ollama** serves the local LLM behind a FRP tunnel at `https://ollama-frp.cloudinit.dev/v1`. No external API calls for PII-bearing data.
---
## 2. Service inventory
All services run in the `inboxos-net` Docker network defined in `docker/docker-compose.yml`.
| # | Service | Container / DNS name | Public URL | Role in pipeline | Restart | Log location |
|---|------------------|---------------------------|-----------------------------------------|-----------------------------------------------------------------------------------------------|---------------------------------------------------------------|-----------------------------------------------------------------------|
| 1 | **NocoDB** | `nocodb` | `https://noco.cloudinit.dev` | ERP/CRM data warehouse. Four module tables (`finance`, `crm`, `intel`, `promotions`). | `docker compose -f docker/docker-compose.yml restart nocodb` | `docker logs nocodb` |
| 2 | **Gitea** | `gitea` | `https://git.cloudinit.dev` | Version control for prompts, scripts, and CI. Hosts Actions runner registration & secrets. | `docker compose -f docker/docker-compose.yml restart gitea` | `docker logs gitea` |
| 3 | **Gitea Actions Runners** | `gitea-runner`, `gitea-runner-2` | n/a (self-hosted) | Execute the live and historical workflows. Two so live cron + historical run can run in parallel. | `docker compose -f docker/docker-compose.yml restart gitea-runner [gitea-runner-2]` | `docker logs gitea-runner [gitea-runner-2]` + per-job artifact under `pipelines/_artifacts/*.log` |
| 4 | **Papra** | `papra` | `https://papra.cloudinit.dev` | Curated document archive (attachments + selected bodies). First ship is dry-run. | `docker compose -f docker/docker-compose.yml restart papra` | `docker logs papra` |
| 5 | **SeaweedFS** | `seaweedfs-master`, `-volume`, `-filer` | `https://storage.cloudinit.dev` (via reverse proxy) on `:8888` S3 endpoint in-network | Object store for `.eml` archive and pipeline state checkpoints. | `docker compose -f docker/docker-compose.yml restart seaweedfs-{master,volume,filer}` | `docker logs seaweedfs-{master,volume,filer}` |
| 6 | **Ollama** | `ollama` | `https://ollama-frp.cloudinit.dev/v1` | Local LLM inference for `inboxos:classify` and `inboxos:extract-*`. Default model `gemma4:e4b`. | `docker compose -f docker/docker-compose.yml restart ollama` | `docker logs ollama` |
The **OpenCode CLI** is not invoked by either pipeline; both
pipelines call Ollama's OpenAI-compatible `/v1/chat/completions`
endpoint directly via `pipelines.common.llm_run_command` (default
backend: `ollama`). The `opencode` binary install step in
`historical-pipeline.yml` is retained for any human-driven
`opencode` invocation from the repo worktree; the pipeline itself
does not shell out to it. See `docs/extraction-contract.md §Legacy:
opencode CLI backend` for the rationale.
The **reverse proxy** (Caddy in the commented-out `caddy` service in `docker/docker-compose.yml`) terminates TLS for `*.cloudinit.dev` and routes to each service. Operators without a reverse proxy can reach services via the `inboxos-net` Docker DNS names (e.g. `http://gitea:3000`, `http://seaweedfs-filer:8888`).
---
## 3. Workflow operations
All workflows live in `.gitea/workflows/`. They are declarative, version-controlled in the same repo as the scripts they invoke, and have no separate runtime state to back up. This is the board decision recorded in [INB-8](/INB/issues/INB-8) — n8n was ruled out as unreliable.
**Quickstart:** for a 15-minute green-field spin-up, see `docs/quickstart.md` and `scripts/quickstart.sh` (REQ-INB-40). The script runs the 8 steps below in sequence and prints the runbook §X.Y cross-ref on any failure.
**Daily digest:** see `scripts/daily_digest.py` and `.gitea/workflows/daily-digest.yml` (REQ-INB-47). The workflow runs on `cron 0 6 * * *` and emails a one-line per-module summary.
**Container hardening:** all services run with `cap_drop: [ALL]`, `security_opt: no-new-privileges:true`, and either `read_only: true` (no write access needed) or `read_only: false` with a `tmpfs: [/tmp]` mount (for services that need a writable scratch dir). See `docker/docker-compose.yml` for the full stanza and the per-service `cap_add` list (REQ-INB-55).
**DPIA:** see `docs/security/dpia.md` (REQ-INB-58). The Data Protection Impact Assessment per GDPR Art. 35 names the data flows, the lawful basis, the data-subject rights, and the residual risks.
**Threat model:** see `docs/security/threat-model.md` (REQ-INB-59). The STRIDE-per-component decomposition covers the 10 components in the inboxos architecture.
### 3.1 Live pipeline — `live-pipeline.yml`
| Aspect | Value |
|----------------|---------------------------------------------------------------------------------------------|
| Schedule | `cron(*/5 * * * *)` — every 5 minutes |
| Manual trigger | `workflow_dispatch` with optional inputs `since_uid` (override high-water mark) and `max_emails` (default 50) |
| Concurrency | `group: live-pipeline, cancel-in-progress: false` (overlapping runs must not trample state) |
| Timeout | 15 minutes |
| Script | `pipelines/live_pipeline.py` |
| Run log | `pipelines/_artifacts/live-run-<id>.log` (uploaded as `live-pipeline-log` artifact) |
**Flow** (verifiable in `pipelines/live_pipeline.py`):
1. Load the IMAP UID checkpoint from `s3://$SEAWEEDFS_S3_BUCKET/state/live-state.json`.
2. Poll IMAP for `UID > last_uid` (bounded by `MAX_EMAILS_OVERRIDE`, default 50).
3. For each message, in UID-ascending order:
- Back up the raw `.eml` to `s3://$SEAWEEDFS_S3_BUCKET/eml/live/<yyyy>/<mm>/<dd>/<Message-ID>.eml` (skip if exists).
- Call `inboxos:classify` to pick which modules apply.
- For each applicable module, call `inboxos:extract-<module>` and UPSERT the result.
- Flush the checkpoint (`state/live-state.json`) **per message** so a crash never replays a success.
**What success looks like in the run log:**
- One `live_run_start` event at the top.
- One or more `process_start` → `eml_backed_up` → `classified` → `process_done` event groups, in UID-ascending order.
- Every `process_done` is followed by a checkpoint write (visible as the `state/live-state.json` PUT in S3, or as the next call to `imap_fetched`).
**What failure looks like:**
- A `classify_error` or `extract_permanent_failure` event means the pipeline stops at that UID. The next run retries it.
- A `dead_letter_unclassified` event means the email had no module match — the raw `.eml` is preserved at `eml/dead-letter/<mid>.eml` and the cursor advances.
- A `missing_message_id` event means the email lacked a `Message-ID:` header. The raw `.eml` is preserved at `eml/dead-letter/uid-<n>.eml`.
- A non-zero job exit code with no `live_run_done` in the log means the runner OOM'd or the IMAP connection was killed mid-fetch — see §5.1.
### 3.2 Historical pipeline — `historical-pipeline.yml`
| Aspect | Value |
|----------------|---------------------------------------------------------------------------------------------|
| Schedule | **None** — manual only (`workflow_dispatch`). Heavy operation; full archives run for hours. |
| Required input | `mbox_object_key` — the SeaweedFS object key of the `.mbox` (e.g. `eml/historical/all-mail.mbox`) |
| Optional inputs| `start_offset` (resume from a byte offset; leave blank to use persisted checkpoint), `batch_size` (default 100), `limit` (0 = no limit) |
| Concurrency | `group: historical-pipeline, cancel-in-progress: false` |
| Timeout | 12 hours |
| Script | `pipelines/historical_pipeline.py` (canonical reference for the extraction contract) |
| Streamer | `pipelines/mbox_stream.py` — byte-offset streaming parser for `.mbox` |
| Run log | `pipelines/_artifacts/historical-run-<id>.log` (uploaded as `historical-pipeline-log` artifact) |
**Idempotency contract:**
- **Message-ID UPSERT** is the only write path into NocoDB (`pipelines/common.py::nocodb_upsert`). It does a GET by `Message_ID` and PATCHes if present, POSTs otherwise. Re-running the same `.mbox` produces zero duplicates.
- **Byte-offset checkpoint** is persisted to `s3://$SEAWEEDFS_S3_BUCKET/state/historical-state.json` every `BATCH_SIZE` messages (default 100) and on shutdown. A crashed run resumes from the last checkpoint on the next dispatch.
- **Object keys** are deterministic by `Message-ID` and date, so re-uploading the same `.eml` overwrites the same S3 object.
**Expected runtime per 10K messages:** ~5–10 minutes on a clean `c5.large` runner when the LLM is the bottleneck (depends on the model — `gemma4:e4b` is fast; `gemma4:12b` adds ~2×). The I/O-bound parts (S3 GET of the `.mbox`, S3 PUTs of `.eml`, Papra uploads) are sub-second per message.
**How to resume after a crash:**
1. Re-dispatch `historical-pipeline.yml` with the **same** `mbox_object_key` and **no** `start_offset` input. The script loads `state/historical-state.json` and resumes from the persisted byte offset.
2. Do **not** dispatch with `start_offset=0` unless you have deleted the state file — that re-processes everything from the top (safe but wasteful, and will take hours).
**How to dead-letter a message manually:**
- A message that was already dead-lettered by the pipeline lives at `s3://$SEAWEEDFS_S3_BUCKET/eml/dead-letter/historical-<offset>.eml` (or `<mid>.eml`). To permanently skip a poison message, find its `Message-ID` in the run log, then write a marker object to `eml/dead-letter/poison/<mid>.json` containing the reason. A future enhancement will let the script consult this list; for now, mark it in your run log and move on.
### 3.3 PR CI — `ci-pr.yml`
| Aspect | Value |
|----------------|---------------------------------------------------------------------------------------------|
| Triggers | `pull_request` opened / reopened / synchronized against `main`; `workflow_dispatch` |
| Concurrency | `group: pr-ci-${{ pull_request.number }}`, cancel-in-progress |
| Jobs | `fast-tests` (~30s), `e2e-100mb` (20 min), `live-llm-smoke` (15 min, real Ollama) |
| Required secrets | `NOCODB_BASE_URL`, `NOCODB_API_TOKEN`, `NOCODB_PROJECT_ID`; `OLLAMA_BASE_URL` is optional |
- `fast-tests` exercises `gitea_request`, `mbox_stream`, the offline parts of `live_pipeline`, and the fast subset of `historical_pipeline` (config, parsing, dead-letter paths).
- `e2e-100mb` builds a 100 MB synthetic `.mbox` and runs the historical pipeline end-to-end with S3 / NocoDB / Papra stubbed. Verifies UPSERT idempotency, crash-and-resume offset stability, and dead-letter semantics. The merge gate.
- `live-llm-smoke` calls the real Ollama endpoint with the Substack input that previously dead-lettered in live run #674 (UID 4017, 86 KB body) and asserts the extract step returns parseable JSON. Skipped when `OLLAMA_BASE_URL` is not set; surfaces a clear `skipped` notice rather than failing the PR.
---
## 4. Observability
### 4.1 Reading a workflow run log
1. Open the Gitea UI → repo → **Actions** → click the run.
2. The job's `Run process-new-emails` / `Run process-mbox` step shows stdout in the Gitea Actions UI. Each line is a JSON object emitted by `log_event`.
3. Expand the **Upload run log as artifact** step. The per-run log file is downloaded as `live-pipeline-log` or `historical-pipeline-log`. This is identical to what was streamed to stdout, and is the canonical artifact to attach to a post-mortem.
4. The artifact also lives in the project workspace at `pipelines/_artifacts/<pipeline>-run-<run-id>.log`.
### 4.2 Structured event vocabulary
Every step in both pipelines calls `pipelines.common.log_event` to emit a JSON line. The canonical event names (verified in source) are:
| Event | Where | What it tells you |
|--------------------------------|--------------------------------------------|--------------------------------------------------------------------------------|
| `process_start` | `live_pipeline.py:428`, `historical_pipeline.py:282` | A new message is about to be processed. Carries `uid` (live) or `offset` (historical) and `message_id`. |
| `eml_backed_up` | both pipelines | The raw `.eml` was PUT to SeaweedFS. Carries `key` and `bytes`. |
| `classified` | both pipelines | `inboxos:classify` returned a non-empty module list. Carries `modules`. |
| `llm_call` | `common.py:591` | The Ollama call is about to be made. Carries `backend`, `modelID`, `command`, `url`. |
| `process_done` | both pipelines | The message was processed end-to-end. Carries `upserts` (NocoDB action list). |
| `module_table_missing` | both pipelines | A module came back from classify but has no entry in `NOCODB_MODULE_TABLES`. Carries `module`. |
| `extract_permanent_failure` | both pipelines | `inboxos:extract-<module>` failed irrecoverably. The run stops without advancing. |
| `dead_letter_unclassified` | both pipelines | Classify returned `[]`. The `.eml` is preserved at `eml/dead-letter/...` and the cursor advances. |
| `historical_run_done` | `historical_pipeline.py:545` | The historical run finished cleanly. Carries totals. |
| `live_run_start` | `live_pipeline.py:533` | The live run began. Carries `started_at`. |
| `papra_decision` | `common.py:997` | The Papra write-through decision was made. Carries `would_write`, `reason`, `module`, `dry_run`. |
| `papra_write_ok` / `papra_write_failed` / `papra_write_dry_run` | `common.py:1018`-`1055` | Per-target outcome of a Papra write. |
| `state_recovered_from_corrupt` | `live_pipeline.py:550` | The persisted IMAP checkpoint was malformed; the run reset it. |
The `live_pipeline.py` source also emits `imap_search_result`, `imap_fetch_missing_body`, `imap_fetched`, `classify_error`, `extract_invalid_response`, `missing_message_id`, `config_error`, `since_uid_override`, `stopped_on_failure_at_uid`, and others (search for `log_event` in `pipelines/live_pipeline.py` for the full list).
### 4.3 NocoDB table inventory
The four module tables (verified against `pipelines/_tests/test_upsert_synthetic.py` and `docs/phase-1.1-verification/inb-28-report.md`):
| Module | Table name in NocoDB | Table ID | Notes |
|--------------|----------------------|---------------------|-------|
| `finance` | `pl_finance` | `m0thluc3jyz2iok` | Receipts, invoices, P&L items. 14 user columns. |
| `crm` | `personal_crm` | `m2wsbpp01v08pm5` | People, organisations, intent. 15 user columns. |
| `intel` | `daily_news` | `msi6dj1cany1b5r` | Newsletters, briefings. 13 user columns. |
| `promotions` | `promotions` | `mf7catfwwyh3jj5` | Sales, promo codes, launches. 14 user columns. |
Per-table row counts and column schemas are listed in `docs/phase-1.1-verification/inb-28-report.md` and `docs/schemas.md`. Every module table has `Message_ID` (SingleLineText, UNIQUE, required) as the natural key.
### 4.4 SeaweedFS bucket layout
`$SEAWEEDFS_S3_BUCKET` is `inboxos` (default in `.env.example`). The layout is:
```
s3://inboxos/
├── eml/
│ ├── live/
│ │ └── <yyyy>/<mm>/<dd>/<Message-ID>.eml
│ ├── historical/
│ │ └── <yyyy>/<mm>/<dd>/<Message-ID>.eml
│ └── dead-letter/
│ ├── <Message-ID>.eml (live, unclassified)
│ ├── uid-<uid>.eml (live, missing Message-ID)
│ └── historical-<offset>.eml (historical)
└── state/
├── live-state.json {"last_uid": <int>, "last_run_at": <iso>, "messages_seen": <int>}
└── historical-state.json {"mbox_key": <str>, "offset": <int>, "messages_seen": <int>, "last_run_at": <iso>}
```
> State-file field names come from the actual code: live
> (`pipelines/live_pipeline.py:600-603`) and historical
> (`pipelines/historical_pipeline.py:516-521`). The
> `_corrupt_state_recovered` field is an additional transient
> key on first load when the persisted JSON is unparseable.
The state files are tiny JSON blobs written via `s3_put_bytes` after every successful message (live) or every `BATCH_SIZE` messages (historical). They are the **only** persistent state between workflow runs.
### 4.5 Metrics surface (REQ-INB-23)
Each pipeline run writes **one** JSONL line to
`pipelines/_artifacts/<pipeline>-metrics-<run_id>.jsonl` (one
file per run; never overwritten). The line shape and counter /
gauge vocabulary are documented in
`pipelines/_metrics/emit.py` (a copy-paste of the docstring is
in this runbook for operator reference).
| Counter key | Type | What it tells you |
|----------------------------|------|------------------------------------------------------------|
| `messages_processed` | int | Total messages processed in this run. |
| `messages_dead_lettered` | int | Subset routed to the dead-letter prefix. |
| `noco_upsert_ok` | int | NocoDB upserts that returned 2xx. |
| `noco_upsert_4xx` | int | 4xx (drift; permanent failure for the affected message). |
| `noco_upsert_5xx` | int | 5xx (transient; retried by tenacity). |
| `papra_would_write` | int | Papra decisions that resolved `would_write=true`. |
| `papra_write_ok` | int | Papra HTTP POSTs that returned 2xx. |
| `papra_write_failed` | int | Papra HTTP POSTs that returned >= 400. |
| `papra_index_upsert_ok` | int | `papra_index` audit-trail rows written. |
| `papra_index_upsert_failed`| int | `papra_index` upserts that raised. |
| `llm_call_seconds_sum` | int | Total wall-clock ms spent in `requests.post` to Ollama. |
| `llm_call_count` | int | Number of LLM HTTP POSTs in the run. |
| Gauge key | Type | What it tells you |
|--------------------------|----------------|-------------------------------------------------|
| `last_run_at` | ISO 8601 str | The run's `now_iso()` at metrics emission. |
| `dead_letter_total` | int | Cumulative dead-letter prefix size (best-effort). |
| `papra_dry_run` | "true"/"false" | The effective `PAPRA_DRY_RUN` after precedence. |
| `papra_dry_run_override` | "true"/"false"/"none" | The parsed override (or "none" if unset). |
**Tail-and-summarize procedure.** To compute a daily total of
`messages_processed` across all live runs:
```bash
cat pipelines/_artifacts/live-metrics-*.jsonl \
| jq -s '[.[].counters.messages_processed] | add'
```
To inspect the last 20 metrics lines in time order:
```bash
cat pipelines/_artifacts/live-metrics-*.jsonl \
| tail -20 \
| jq -c '{ts, run_id, counters}'
```
**Forward-pointer.** Alerting is **out of scope** for v1.4. v1.2
closed on a 0/1 ideation disposition (no candidates accepted) per
`.ciagent/inboxos/CI-LOG.md`, and v1.3 was a doc-hygiene
milestone; v1.4 added the operator audit cursor and the
pollable healthcheck but no alert layer. The metrics file is
the substrate; the alert layer is the next v1.x candidate
cycle to re-evaluate. In the meantime the runbook §4
observability surface is logs-only, with
`pipelines/_metrics/emit.py` emitting a per-run `metrics.jsonl`
(REQ-INB-23) for offline postmortem and
`pipelines/_metrics/healthcheck.json` (REQ-INB-52) as the
canonical uptime-monitor pollable endpoint.
**Opt-out.** Set `LLM_METRICS_ENABLED=false` in the pipeline env
to skip the `time.monotonic()` wrapping around the LLM HTTP POST.
The counters default to enabled; opt-out is for canary runs that
want to measure pure LLM latency without the wrapper overhead.
---
## 5. Common failure modes and recovery
### ERROR-NN catalog
A one-page lookup table for the on-call operator. If you are reading this from a pager, the table below tells you which §5.X subsection to jump to for the full recovery procedure.
| ERROR-NN | Symptom | See §X.Y |
|-----------|------------------------------------------------------|----------|
| ERROR-01 | Runner stuck / OOM (> 60 min in `in_progress`) | §5.1 |
| ERROR-02 | Live cron emitted no new MIDs for > 30 min | §5.2 |
| ERROR-03 | NocoDB 4xx on upsert (wrong table_id) | §5.3 |
| ERROR-04 | NocoDB 4xx "field is required" (schema drift) | §5.3 |
| ERROR-05 | IMAP connection refused / auth failure | §5.4 |
| ERROR-06 | Ollama connection refused / model OOM | §5.5 |
| ERROR-07 | SeaweedFS connection refused / auth failure | §5.6 |
| ERROR-08 | Papra `dry_run` unexpectedly flipped to `false` | §5.7 |
| ERROR-09 | State file corrupted / offset regression | §5.8 |
| ERROR-10 | Backup tar fails (NocoDB / SeaweedFS / Gitea) | §5.9 |
### 5.1 Runner stuck / OOM (> 60 minutes in `in_progress`)
A single stuck job can block the live cron (live jobs have `cancel-in-progress: false`). Diagnosis and recovery:
1. **Confirm it is stuck.** Open Gitea Actions → run → check whether the `Run process-new-emails` step has emitted *any* new log lines in the last 10 minutes. A runner that is downloading a large `.mbox` is making progress (watch for S3 GET bytes) — a runner that is silent is OOM'd or wedged.
2. **Check the host's container health.**
```bash
docker ps --filter "name=gitea-runner" --format "table {{.Names}}\t{{.Status}}\t{{.CPUPerc}}\t{{.MemUsage}}"
```
Memory at 100% with no progress for > 10 minutes = OOM. CPU at 0% with no progress = wedged.
3. **Drain the stuck job.** From Gitea UI → run → **Cancel job**. The live workflow's `cancel-in-progress: false` will *not* preempt the next cron tick — the next run will queue until the cancelled one reports back. This is the correct behaviour: do not change the concurrency setting to fix a stuck job.
4. **Restart the runner.**
```bash
docker compose -f docker/docker-compose.yml restart gitea-runner
```
If `gitea-runner-2` is also stuck, restart it too. The other runner will pick up new jobs within ~30 s.
5. **Verify recovery.**
```bash
curl -u "$GITEA_USER:$GITEA_PASS" \
https://git.cloudinit.dev/api/v1/repos/jc-capital/inboxos/actions/runners
```
Expect 2 entries with `status: "online"`. Then dispatch `live-pipeline.yml` manually with a small `max_emails` (e.g. `5`) to confirm the end-to-end loop.
6. **If recurring:** the runner is under-provisioned. Increase its memory limit in `docker/docker-compose.yml` (`mem_limit: 4g` or similar) or move to a host with more RAM.
### 5.2 SeaweedFS `NoSuchKey` on a Message-ID
This means a NocoDB row references a `Message-ID` whose raw `.eml` was never uploaded to (or was deleted from) SeaweedFS. Two common causes:
1. **The `.eml` write failed** but the NocoDB upsert still succeeded (e.g. SeaweedFS was down briefly; the retry on the S3 PUT hit a 5-attempt limit; the pipeline then proceeded to NocoDB).
2. **The Message-ID is malformed** — e.g. a stray `<` or `>` made it into the NocoDB row but the file was written with the correct key.
**How to find the offending record in NocoDB:**
```bash
# For a finance row, replace m0thluc3jyz2iok with the module's table ID from §4.3
curl -G \
-H "xc-token: $NOCODB_API_TOKEN" \
--data-urlencode "where=(Message_ID,eq,<message-id-here>)" \
--data-urlencode "limit=1" \
"$NOCODB_BASE_URL/api/v2/tables/m0thluc3jyz2iok/records"
```
**How to dead-letter the row:**
1. Verify the `.eml` truly is missing in SeaweedFS (use the `s3_get_text_optional` helper in `pipelines/common.py` against `s3://$SEAWEEDFS_S3_BUCKET/eml/live/...`).
2. If it is missing, re-fetch the message from Gmail via IMAP using the `Message-ID` (Gmail's IMAP supports `UID FETCH` by `Message-ID` only via header search; the easier path is `since_uid`).
3. Re-dispatch `live-pipeline.yml` with `since_uid` set to the lowest affected UID. The pipeline will re-archive the `.eml` (overwriting the same key, or PUTing a new one) and re-UPSERT the NocoDB row.
4. If the IMAP mailbox no longer has the email (it was deleted from Gmail), the row is orphaned. Mark it manually in NocoDB (set a `Notes` field with a date) and move on — do not bulk-delete rows, the audit trail matters.
### 5.3 NocoDB 4xx on a UPSERT
Read the response body (NocoDB returns JSON like `{"msg":"..."}`). Common causes and recovery:
| Cause | Symptom | Recovery |
|----------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|
| **Wrong table_id** | 404 from `/api/v2/tables/<id>/records` | Verify the table ID in `NOCODB_MODULE_TABLES` matches the live table ID (see §4.3). Re-bootstrap with `python scripts/apply_schemas.py` to print the current mapping. |
| **Missing required field** | 400 with `"<field> is required"` | Re-run `python scripts/apply_schemas.py` to add the column. UPSERT will retry on the next dispatch. |
| **Schema drift** (column renamed, type changed) | 400 with `"<field> validation failed"` | Edit `docs/schemas.md` first, then `scripts/apply_schemas.py`. Re-run the workflow. |
| **`Message_ID` UNIQUE violation on a POST** | This should not happen — `nocodb_upsert` does GET-then-PATCH/POST. If it does, race. | Re-run the workflow; the second pass will hit the GET and switch to PATCH. |
| **SingleSelect option not in the allowed list** | 400 with `"<option> is not a valid option for <field>"` | Add the option to the SingleSelect column in NocoDB UI, then re-run. |
The full retry policy lives in `pipelines/common.py::nocodb_upsert` (tenacity, 5 attempts, exponential backoff 1s → 30s). A 4xx surfaces immediately and stops the run; the next dispatch retries from the same checkpoint.
### 5.4 Ollama 4xx / timeout
1. **Verify the endpoint is reachable:**
```bash
curl -sS "$OLLAMA_BASE_URL/models" -H "Authorization: Bearer $OLLAMA_API_KEY"
```
Expect 200 with a JSON list. The 4xx response body is usually self-explanatory (model not found, auth required, OOM on the Ollama host).
2. **If the model is missing** (the response lists other models but not the one in `INBOXOS_LLM_MODEL`):
```bash
docker exec ollama ollama pull gemma4:e4b
```
3. **To switch the model** without redeploying: update the repo **variable** `INBOXOS_LLM_MODEL` (and `INBOXOS_LLM_MODEL_EXTRACT` for the larger extract model) in Gitea → repo → Settings → Actions → Variables. The next workflow run picks it up. Do not edit the prompt's `model:` frontmatter to "fix" a model outage — prompts are version-controlled, and that change goes through a PR.
4. **To raise the timeout:** set `INBOXOS_LLM_TIMEOUT` (in seconds, default 120). For very rich emails (86 KB Substack bodies) with the larger `INBOXOS_LLM_MODEL_EXTRACT` model, 180–240 s is not unusual.
5. **Check the Ollama host itself** for OOM:
```bash
docker stats ollama --no-stream
```
If memory is at the limit, the Ollama container is OOM-killing the loaded model. Either reduce the model size (`gemma4:e4b` instead of `gemma4:31b`) or add memory to the host.
### 5.5 Live pipeline falling behind
**Current symptom** (referenced in the INB-35 description): a single runner + 30+ minute cycles. The 5-minute cron cannot keep up because:
- The single runner is doing one job at a time (`cancel-in-progress: false`).
- IMAP UID backlog from a sync delay (e.g. Gmail was unreachable for an hour) means the next run has > 50 emails to process, and the runner is LLM-bound.
**Recovery options (in order of preference):**
1. **Add a second runner.** Already provisioned in `docker/docker-compose.yml` as `gitea-runner-2`; the only remaining step is the one-time registration token (see `docs/gitea-actions.md §2a` and [INB-33](/INB/issues/INB-33)). The historical workflow then runs in parallel on the second runner, freeing the first runner for the live cron.
2. **Bump the cron frequency.** Edit `.gitea/workflows/live-pipeline.yml` → `schedule` (e.g. `cron("*/2 * * * *")` for every 2 minutes). This is local-only, no board approval required. Note: 1 minute is the practical floor; Gitea will skip runs if the previous one is still in flight.
3. **Pause the cron** (board approval required). Use only when the pipeline is producing corrupted data or the LLM is rate-limiting. Two ways:
- In the Gitea UI: repo → Actions → `live-pipeline.yml` → **Disable workflow**.
- In code: open a PR that comments out the `schedule:` block; merge. Reverting the PR re-enables it.
4. **Bump `max_emails` per run** (default 50). The workflow input `max_emails` can be raised to 200, 500, etc. via `workflow_dispatch`. This is a one-shot — it does not change the default.
### 5.6 Historical pipeline run cancelled mid-way
**This is safe by construction.** The byte-offset checkpoint in `state/historical-state.json` is written every `BATCH_SIZE` messages (default 100) and on shutdown. A cancelled run:
1. Flushes whatever batch is currently in-flight to the checkpoint before exiting.
2. Leaves NocoDB and SeaweedFS in a consistent state — every successful `process_done` has already been committed.
**Recovery:** dispatch `historical-pipeline.yml` again with the **same** `mbox_object_key` and **no** `start_offset` input. The script will load `state/historical-state.json`, seek to the persisted byte offset, and continue.
**Do not** re-dispatch with `start_offset=0` unless you have intentionally deleted `state/historical-state.json`. Re-processing from offset 0 is safe (UPSERT dedupes) but wasteful — for a multi-GB `.mbox`, it will take hours and consume LLM tokens you do not need to spend.
To permanently restart a historical run from scratch (e.g. after a prompt overhaul you want to re-test):
```bash
# From a runner or any host with the S3 creds:
python -c "
import boto3, os
s3 = boto3.client('s3', endpoint_url=os.environ['SEAWEEDFS_S3_ENDPOINT'],
aws_access_key_id=os.environ['SEAWEEDFS_S3_ACCESS_KEY'],
aws_secret_access_key=os.environ['SEAWEEDFS_S3_SECRET_KEY'])
s3.delete_object(Bucket=os.environ['SEAWEEDFS_S3_BUCKET'], Key='state/historical-state.json')
print('state/historical-state.json deleted; next dispatch starts from offset 0')
"
```
Then dispatch `historical-pipeline.yml` with `start_offset=0` and the same `mbox_object_key`.
### 5.7 Backup and restore (REQ-INB-22)
The `scripts/backup.sh` and `scripts/restore.sh` helpers
snapshot and restore the three stateful services (NocoDB,
SeaweedFS, Gitea) to/from a timestamped directory under
`backups/<service>/<utc-timestamp>/`. The helpers are
idempotent: a re-run for the same service at the same UTC
timestamp is a no-op.
**Schedule.** A daily `cron` job on the operator host calls the
three backup helpers in sequence:
```cron
# /etc/cron.d/inboxos-backup — runs at 02:00 UTC daily.
0 2 * * * cto /opt/inboxos/scripts/backup.sh nocodb >> /var/log/inboxos-backup.log 2>&1
5 2 * * * cto /opt/inboxos/scripts/backup.sh seaweedfs >> /var/log/inboxos-backup.log 2>&1
10 2 * * * cto /opt/inboxos/scripts/backup.sh gitea >> /var/log/inboxos-backup.log 2>&1
```
**Retention.** 7 daily + 4 weekly + 0 monthly. The single-
operator posture is "the operator can change it" — the values
are the v1.1 ship default and are documented in
`scripts/backup.sh` header comment.
**Copy-paste invocation.**
```bash
# Backup one service.
scripts/backup.sh nocodb
scripts/backup.sh seaweedfs
scripts/backup.sh gitea
# Restore from a backup id (UTC timestamp).
scripts/restore.sh 2026-06-08T120000Z nocodb
scripts/restore.sh 2026-06-08T120000Z seaweedfs
scripts/restore.sh 2026-06-08T120000Z gitea
```
**Test-or-die gate.** A restore is not considered safe until
`pipelines/_tests/test_restore_offline.py` exits 0. The test
pins the exit-code table for missing args, missing env, missing
backup directory, and the SeaweedFS "no backup target configured"
path. It also exercises a git-bundle round-trip on an in-tree
repo. The full end-to-end (backup → wipe → restore → verify)
requires a live NocoDB / SeaweedFS / Gitea and is run by the
operator on the host.
---
## 6. Security and secrets
### 6.1 Secret inventory
All Gitea repo secrets live at **Gitea → repo → Settings → Actions → Secrets** (and a parallel **Variables** section for non-secret tunables). The 15 secrets consumed by the workflows, grouped by owner:
| Owner | Secret | Used by | Purpose |
|--------------|-------------------------------------|-----------|---------------------------------------------------------------|
| IMAP | `IMAP_HOST` | live | Gmail IMAP server (`imap.gmail.com`) |
| IMAP | `IMAP_PORT` | live | `993` |
| IMAP | `IMAP_USER` | live | Gmail address |
| IMAP | `IMAP_PASSWORD` | live | Gmail App Password (not the account password; 2FA required) |
| IMAP | `IMAP_USE_SSL` | live | `true` |
| IMAP | `IMAP_MAILBOX` | live | `INBOX` (default) |
| SeaweedFS | `SEAWEEDFS_S3_ENDPOINT` | both | `https://storage.cloudinit.dev` (or `http://seaweedfs-filer:8888` in-network) |
| SeaweedFS | `SEAWEEDFS_S3_ACCESS_KEY` | both | S3 access key |
| SeaweedFS | `SEAWEEDFS_S3_SECRET_KEY` | both | S3 secret key |
| SeaweedFS | `SEAWEEDFS_S3_BUCKET` | both | `inboxos` |
| NocoDB | `NOCODB_BASE_URL` | both | `https://noco.cloudinit.dev` |
| NocoDB | `NOCODB_API_TOKEN` | both | `xc-token` header value |
| NocoDB | `NOCODB_PROJECT_ID` | both | NocoDB project ID |
| NocoDB | `NOCODB_MODULE_TABLES` | both | JSON map of module name → NocoDB table ID |
| Ollama | `OLLAMA_BASE_URL` | both | `https://ollama-frp.cloudinit.dev/v1` |
| Ollama | `OLLAMA_API_KEY` | both | Bearer token if Ollama is auth-gated; empty string otherwise |
| Papra | `PAPRA_BASE_URL` | both | `https://papra.cloudinit.dev` (optional) |
| Papra | `PAPRA_API_TOKEN` | both | Bearer token (optional) |
| Papra | `PAPRA_ORGANIZATION_ID` | both | Papra org id (optional) |
| Gitea | `GITEA_INSTANCE_URL` | both | `http://gitea:3000` (in-network) |
| Gitea | `GITEA_USER` / `GITEA_PASS` | scripts | HTTP Basic Auth for the Gitea REST API (see `common.gitea_request`) |
That's 22 distinct secret values (the v1.0 ship counted 21 because the `GITEA_USER` / `GITEA_PASS` row was treated as one credential; v1.1 counts both values to match the row count in the table above). The issue description rounds to 15 because the Papra trio is optional; the table is exhaustive.
**Rotation schedule:** **None — all static.** This is an explicit, accepted trade-off for a single-operator, self-hosted system. The S3 access keys and NocoDB API token are long-lived. The Gmail App Password can be revoked from the Google account at any time without breaking anything else; the IMAP user must re-issue a new App Password and update the Gitea secret. The helper landed in v1.1 (REQ-INB-21); see §6.2 for the procedure.
### 6.2 Secret rotation (REQ-INB-21)
The `scripts/rotate_secrets.sh` helper rotates a Gitea repo secret in
place via `pipelines.common.gitea_request`. The new value is read
from **STDIN** (never a CLI argument) so it is not visible in
`ps` / process listings.
**Rotation ordering.** When multiple secrets need rotating, do them
in this order (each step invalidates fewer downstream services than
the next):
1. `NOCODB_API_TOKEN` first — it is the PII-bearing token (every
NocoDB write is gated on it; rotate during low-traffic). A
failed rotation here dead-letters new writes; rotate the
pre-shared token in NocoDB and update the Gitea secret.
2. `SEAWEEDFS_S3_ACCESS_KEY` and `SEAWEEDFS_S3_SECRET_KEY`
(treated as a pair) — the S3 archive and the pipeline state
files are gated on these. A failed rotation here stops both
pipelines; rotate the pre-shared key in SeaweedFS and update
the Gitea secrets.
3. `IMAP_PASSWORD` — the Gmail App Password. A failed rotation
here stops the live pipeline only; the historical pipeline is
unaffected. Update the Gmail App Password and the Gitea secret.
4. `GITEA_USER` / `GITEA_PASS` last — the Basic Auth credentials
that the helper itself uses. Rotating these invalidates the
helper's own running session; do them after the helper has
returned successfully for the other three.
**Test-or-die gate.** A rotation is not considered safe until
`pipelines/_tests/test_rotate_secrets_offline.py` exits 0. The
test exercises the dry-run path, the apply branch shape, and the
env-validation failure mode. The dry-run path is the one operators
should run first; it prints a redacted diff of the new value
(`sha256:8hex`) without making the PUT.
**Copy-paste invocation.**
```bash
# Dry-run first; no PUT issued.
echo "$NEW_NOCODB_API_TOKEN" | scripts/rotate_secrets.sh NOCODB_API_TOKEN
# Real rotation; PUTs the new value, then verifies via GET.
echo "$NEW_NOCODB_API_TOKEN" | scripts/rotate_secrets.sh --apply NOCODB_API_TOKEN
```
The helper exits 0 on success, 2 on missing env / missing value,
and 3 on Gitea API failure. See
`pipelines/_tests/test_rotate_secrets_offline.py` for the test
contract and the helper's exit-code table in its header comment.
### 6.3 Why we never commit secrets
- `.gitignore` excludes `.env` and `.env.*` except `.env.example`. The example file contains placeholder values only.
- The CTO host deliberately does **not** have write-capable S3 credentials (only the read-only S3 keys used for verification). A leaked CTO host key cannot corrupt the `.eml` archive.
- The Gitea service account used for the REST API (`GITEA_USER=cto` in `.env.example`) is scoped to the `jc-capital/inboxos` repo. It can read/write workflow secrets, but it cannot create new repos or admin the Gitea instance.
- All secret reads happen via `${{ secrets.X }}` in the workflow YAMLs; the values are injected as environment variables into the ephemeral runner container and never appear in the build log.
### 6.4 Data-sovereignty posture
- **No external API calls for PII-bearing data.** The only outbound network calls from the runner are to the in-network services (`http://gitea:3000`, `http://nocodb:8080`, `http://seaweedfs-filer:8888`, `http://ollama:11434`, `http://papra:1221`). The Ollama URL is FRP-tunneled to the local Ollama container — the FRP tunnel terminates inside the operator's network; no third-party LLM provider sees the request body.
- **Local LLM inference.** Default model is `gemma4:e4b` (4 B effective parameters). All `inboxos:classify` and `inboxos:extract-*` calls hit the local Ollama. There is no fallback to a cloud LLM. If Ollama is down, the pipeline dead-letters and the next run retries.
- **No telemetry from any pipeline.** `pipelines/common.py` does not call home. Gitea Actions emits standard runner logs, which are stored on the Gitea host (also self-hosted).
---
## 7. Cost and capacity
### 7.1 Per-message cost
**~$0 / message.** All inference is local (Ollama on the operator's host). The only recurring costs are:
| Cost | Magnitude (single operator) |
|-----------------------------------------------|----------------------------------------------------------------|
| Compute (Ollama host + Gitea host + runners) | One bare-metal or small VM; existing infrastructure. |
| Bandwidth (Gmail IMAP, FRP tunnel to Ollama) | Negligible at < 100 emails / day. |
| Storage (SeaweedFS volume) | See §7.2. |
| LLM API costs | **$0** — local inference. |
### 7.2 Storage growth
- A typical personal email `.eml` is ~30–100 KB. 100 KB / email is the safe upper bound used in capacity planning.
- **10 000 emails ≈ 1 GB** of raw `.eml` in SeaweedFS.
- The SeaweedFS volume is provisioned at `seaweedfs-data` (named Docker volume); size is bounded by the host's disk. No quota is enforced at the SeaweedFS level.
- The NocoDB side is tiny: even 10 000 rows across four tables is single-digit MB.
### 7.3 Compute
- **One Gitea runner** (now two, post-[INB-33](/INB/issues/INB-33)), one Ollama container with `gemma4:e4b` loaded.
- **Throughput observed (live):** ~22 messages / 36 min on a single-runner + IMAP-backlog-saturated cycle. This is LLM-bound; each `inboxos:classify` + 1–2 `inboxos:extract-*` calls adds up to ~90 s per email with `gemma4:e4b` on CPU. A GPU host would cut this roughly 5–10× but is not required at the current scale.
- **Throughput observed (historical):** ~5–10 min per 10 K messages on `gemma4:e4b` CPU, dominated by the LLM calls. Switching to a smaller model (e.g. `gemma3:4b`) would roughly halve this; switching to a larger model (`gemma4:12b`) would roughly double it. Prompt quality and JSON-contract compliance are the binding constraints, not raw tokens/second.
---
## 8. Glossary and appendix
### 8.1 Glossary
- **Message-ID** — The RFC 5322 `Message-ID:` header, with angle brackets stripped. The natural primary key for every module table in NocoDB and the natural key for the `.eml` S3 key.
- **UPSERT** — "Update or insert." `pipelines.common.nocodb_upsert` does this by doing a GET by `Message_ID` and then a PATCH (if found) or POST (if not). There is no plain `INSERT` path in the codebase. The CSV / SQL concept maps 1:1 to the GET-then-PATCH/POST pattern.
- **mbox** — A plain-text file format for archiving email. Each message begins with `From <mailbox> <date>` and ends with a blank line. The historical pipeline uses `mbox_stream.py` to read it as a byte stream, never materialising the full file in memory.
- **IMAP** — Internet Message Access Protocol. The live pipeline uses `IMAPClient` against Gmail's `imap.gmail.com:993` (SSL). Polling, not push, because Gmail's push (IDLE) is unreliable.
- **FRP tunnel** — Fast Reverse Proxy. The Ollama service is exposed at `https://ollama-frp.cloudinit.dev/v1` via FRP, terminating inside the operator's network. Lets the runner talk to Ollama over HTTPS without opening inbound ports.
- **Checkpoint state** — The persisted byte offset (`historical-state.json`) or IMAP UID (`live-state.json`) that lets a pipeline resume from where it left off after a crash. The state file is the only persistent state between workflow runs.
- **Dead-letter** — A poison message or one that could not be processed is moved to `s3://$SEAWEEDFS_S3_BUCKET/eml/dead-letter/...` with a marker file describing the reason. The cursor advances so the rest of the email stream is not blocked. Dead-lettered items require manual review.
- **Idempotency** — "Running the same operation twice has the same effect as running it once." Achieved in InboxOS by (a) Message-ID UPSERT in NocoDB, (b) deterministic S3 keys, and (c) checkpoint-based resume. A redelivered Gmail message, a re-dispatched workflow, and a replayed `.mbox` are all safe.
### 8.2 Cross-references
- **Schemas:** `docs/schemas.md` — NocoDB table schemas (column-by-column).
- **Extraction contract:** `docs/extraction-contract.md` — how a raw email becomes a structured NocoDB record, the prompt family, and failure semantics.
- **Gitea Actions:** `docs/gitea-actions.md` — runner provisioning, second-runner setup, secrets, the Basic Auth model, observability, rollback.
- **Pipelines:** `pipelines/README.md` — pipeline flow diagrams, idempotency model, rollback procedure.
- **Papra write policy:** `docs/papra-write-policy.md` — when the pipeline writes to Papra (attachments, body for crm/finance/legal modules), the dry-run default, the live-write gate (REQ-INB-20), and the `papra_index` audit table (REQ-INB-19).
- **Phase 1.1 verification:** `docs/phase-1.1-verification/inb-28-report.md` — the schema-bootstrap deliverable that locked in the four table IDs and column counts cited in §4.3.
- **Secret rotation:** `docs/operations-runbook.md §6.2` (new in v1.1, renumbered in v1.3 from §6.4) — the rotation helper, the ordering, and the test-or-die gate (REQ-INB-21).
- **Metrics surface:** `docs/operations-runbook.md §4.5` (new in v1.1) — the per-run `metrics.jsonl` shape and the tail-and-summarize procedure (REQ-INB-23).
- **Backup and restore:** `docs/operations-runbook.md §5.7 + §10` (new in v1.1) — the `backup.sh` / `restore.sh` helpers, the daily cron, and the RTO < 4h / RPO < 24h targets (REQ-INB-22).
### 8.3 Quick-reference: log-event search strings
If a post-mortem needs to find every place a particular event is emitted:
```bash
grep -rn 'log_event(logger, "process_done"' pipelines/
grep -rn 'log_event(logger, "classified"' pipelines/
grep -rn 'log_event(logger, "module_table_missing"' pipelines/
grep -rn 'log_event(logger, "dead_letter_unclassified"' pipelines/
grep -rn 'log_event(logger, "extract_permanent_failure"' pipelines/
```
For the full event vocabulary, `grep -n "log_event" pipelines/common.py pipelines/live_pipeline.py pipelines/historical_pipeline.py`.
### 8.4 Quick-reference: env vars (in alphabetical order)
| Env var | Default | Purpose |
|----------------------------------|------------------------------------------|-------------------------------------------------------------------------|
| `GITEA_INSTANCE_URL` | — (required) | Gitea REST API base. `http://gitea:3000` in-network. |
| `GITEA_USER` | — (required for Gitea REST calls) | HTTP Basic Auth user. |
| `GITEA_PASS` | — (required for Gitea REST calls) | HTTP Basic Auth password. |
| `IMAP_HOST` / `IMAP_PORT` | — / `993` | Gmail IMAP endpoint. |
| `IMAP_USER` / `IMAP_PASSWORD` | — | Gmail App Password credentials. |
| `IMAP_USE_SSL` / `IMAP_MAILBOX` | `true` / `INBOX` | IMAP connection toggles. |
| `INBOXOS_LLM_BACKEND` | `ollama` | `ollama` (recommended) or `opencode` (legacy, broken on 1.16.2). |
| `INBOXOS_LLM_MODEL` | `gemma4:e4b` | Default model for classify and any non-extract command. |
| `INBOXOS_LLM_MODEL_EXTRACT` | `gemma4:e4b` | Model for `inboxos:extract-*` commands. |
| `INBOXOS_LLM_MAX_TOKENS` | `4096` | Completion budget. 1024 was too tight for the 11-field extract schema. |
| `INBOXOS_LLM_TIMEOUT` | `120` | Seconds per LLM call. |
| `NOCODB_API_TOKEN` | — | `xc-token` header value. |
| `NOCODB_BASE_URL` | — | `https://noco.cloudinit.dev` |
| `NOCODB_MODULE_TABLES` | — | JSON map `{"finance":"tbl_…","crm":"tbl_…",…}`. |
| `NOCODB_PROJECT_ID` | — | NocoDB project ID. |
| `OLLAMA_API_KEY` | unset | Bearer token if Ollama is auth-gated. |
| `OLLAMA_BASE_URL` | `https://ollama-frp.cloudinit.dev/v1` | OpenAI-compatible chat completions endpoint. |
| `OPENCODE_BIN` | `opencode` | CLI binary used by `opencode_run_command` (legacy path only). |
| `OPENCODE_PROMPT_DIR` | `./.opencode/commands` | Where the `.opencode/commands/inboxos:*.md` prompts live. |
| `PAPRA_API_TOKEN` | unset | Bearer token for Papra uploads. |
| `PAPRA_BASE_URL` | unset | Papra API base. |
| `PAPRA_BODY_WRITE_MODULES` | `crm,finance,legal` | Comma-separated modules whose body (not attachments) gets uploaded. |
| `PAPRA_DRY_RUN` | `true` | `false` to enable real Papra writes (CEO-approved only). |
| `PAPRA_METADATA_ONLY_MODULES` | `intel,promotions` | Modules whose email is *not* written to Papra (metadata only). |
| `PAPRA_ORGANIZATION_ID` | unset | Papra org id. |
| `SEAWEEDFS_S3_ACCESS_KEY` | — | S3 access key. |
| `SEAWEEDFS_S3_BUCKET` | — | `inboxos` (default). |
| `SEAWEEDFS_S3_ENDPOINT` | — | `https://storage.cloudinit.dev` (or in-network `http://seaweedfs-filer:8888`). |
| `SEAWEEDFS_S3_SECRET_KEY` | — | S3 secret key. |
---
## 9. Phase 5 acceptance gate
> This appendix is the **must-have checklist** that closes
> InboxOS Phase 5 (Full Deployment — INB-7). It is the integrator's
> reference: every item must be checkbox-green before the
> `.ciagent/inboxos/ROADMAP.md` Phase 5 status row is flipped to
> "executed". The 20 must-haves below were produced by the
> CIAgent plan at `.ciagent/inboxos/plans/05-PLAN.md` and verified
> during the Wave 1 + Wave 2 + Wave 3 execution pass (see git log
> on `milestone/v1.0-initial`).
### 9.1 Requirements coverage
| # | Must-have (verifiable) | Source-of-truth location | REQ-INB | Owner |
|----|------------------------|--------------------------|:-------:|-------|
| 1 | `pipelines/live_pipeline.py` runs on a 5-minute cron and on `workflow_dispatch` | `.gitea/workflows/live-pipeline.yml:62-65` | REQ-INB-1 | devops-engineer |
| 2 | `pipelines/historical_pipeline.py` is `workflow_dispatch` only and streams a local `.mbox` end-to-end | `.gitea/workflows/historical-pipeline.yml:54-58` | REQ-INB-2 | devops-engineer |
| 3 | All NocoDB writes are GET-then-PATCH/POST on `Message_ID`; no plain INSERT path | `pipelines/common.py:196-255` | REQ-INB-3 | data-engineer |
| 4 | `inboxos:classify` + `inboxos:extract-<module>` is the only extraction path | `pipelines/common.py:632-653`, `docs/extraction-contract.md` | REQ-INB-4 | ml-engineer |
| 5 | Four module tables (`finance`, `crm`, `intel`, `promotions`) with the column set per `docs/schemas.md` | `docs/schemas.md`, `docs/phase-1.1-verification/inb-28-report.md` | REQ-INB-5 | data-engineer |
| 6 | `docs/operations-runbook.md` is the canonical day-1 incident reference | this file | REQ-INB-6 | lead-developer |
| 7 | Two self-hosted Gitea Actions runners are provisioned and online | `docker/docker-compose.yml:103-148`, `docs/gitea-actions.md §2` | REQ-INB-7 | devops-engineer |
| 8 | `ci-pr.yml` runs fast unit tests + 100MB e2e + live Ollama smoke on every PR | `.gitea/workflows/ci-pr.yml:63-230` | REQ-INB-8 | devops-engineer |
| 9 | Papra integration is dry-run by default; the first ship does not call the real Papra API | `pipelines/common.py:729`, `docs/papra-write-policy.md` | REQ-INB-9 | security-engineer |
| 10 | No third-party SaaS touches PII; only the IMAP connection to Gmail leaves the self-hosted perimeter | `docker/docker-compose.yml`, this runbook §6 | REQ-INB-10 | security-engineer |
| 11 | A single pinned Ollama model (`gemma4:e4b`) is the production extraction path; no model-escalation chain | `pipelines/common.py:467-468`, `.opencode/commands/inboxos:*.md` frontmatter | REQ-INB-11 | ml-engineer |
| 12 | `actions/upload-artifact` is pinned to v3 in the workflows that emit per-run logs (live + historical); `ci-pr.yml` has no `upload-artifact` step and inherits GHES compat from the runner image, not from the workflow pin | `.gitea/workflows/live-pipeline.yml:150` (`uses: actions/upload-artifact@v3`), `.gitea/workflows/historical-pipeline.yml:170` (same). `ci-pr.yml` has no `upload-artifact` step; the runbook's previous glob was misleading. | REQ-INB-12 | devops-engineer |
| 13 | A synthetic UPSERT test exists and exercises the NocoDB idempotency path end-to-end | `pipelines/_tests/test_upsert_synthetic.py` | REQ-INB-13 | data-engineer |
| 14 | All extraction prompts are version-controlled in `.opencode/commands/` and invoked by name | `.opencode/commands/inboxos:*.md` (5 files), `pipelines/common.py::llm_run_command` | REQ-INB-14 | ml-engineer |
### 9.2 Reconciliation drift closure
The seven doc/code drifts surfaced in research (P-1 through P-7)
were closed during the Wave 1 + Wave 2 execution pass:
| # | Drift | Closed by | PR | REQ-INB |
|----|-------|-----------|----|:-------:|
| 15 | Model pin drift (`lfm2.5:8b` → `gemma4:e4b` in docs) | `docs: align pipelines/README + extraction-contract to gemma4:e4b pin` | #A | REQ-INB-11 |
| 16 | `OLLAMA_BASE_URL` `/v1` suffix canonicalised | `docs: align pipelines/README + extraction-contract to gemma4:e4b pin` (folded into #A) | #A | REQ-INB-11, REQ-INB-10 |
| 17 | Papra `INTAKE_EMAILS_IS_ENABLED` flipped to `false` in compose | `docker: disable Papra catch-all intake; align with cancelled policy` | #E | REQ-INB-9 |
| 18 | State-file field names in runbook §4.4 corrected | `docs(runbook §4.4): correct state-file field names` | #H | REQ-INB-3, REQ-INB-6 |
| 19 | Live-vs-historical `Papra_Link` asymmetry documented | `docs: document live-vs-historical Papra_Link asymmetry` + `docs(extraction-contract)` — owner: backend-engineer (the P-7 `papra_decide` investigation; the asymmetry is structural, the doc is the deliverable) | #C, #B | REQ-INB-4, REQ-INB-9 |
| 20 | Extraction-contract rewritten to describe Ollama as production LLM caller; opencode marked legacy | `docs(extraction-contract): describe Ollama as production LLM caller; opencode marked legacy` | #B | REQ-INB-4 |
> The `papra_decide` behavioural asymmetry in `live_pipeline.py`
> (P-7 in the research brief) was investigated and found to be a
> no-op: the live pipeline already calls `papra_decide` exactly
> once with the post-classify `modules` list
> (`pipelines/live_pipeline.py:490`). The historical pipeline
> calls it twice (pre-classify attachment walk + post-classify
> body write), but the live pipeline has no pre-classify walk.
> Decision recorded as `ED-1` in `.ciagent/inboxos/CI-LOG.md`.
### 9.3 Closing the gate
A Phase 5 close is the date on which all 20 must-haves are
checkbox-green and the `.ciagent/inboxos/ROADMAP.md` Phase 5
status row is flipped to "executed". The lead-developer is the
integrator; every active persona (data-engineer, backend-engineer,
devops-engineer, ml-engineer, security-engineer, lead-developer)
gets a review pass on this section.
### 9.4 v1.1 acceptance gate
> This appendix is the **must-have checklist** that closes
> InboxOS v1.1 Phase 1. It is the integrator's reference: every
> item must be checkbox-green before the v1.1 ship merge. The
> 11 must-haves below were produced by the CIAgent plan at
> `.ciagent/inboxos/plans/01-PLAN.md` and verified during the
> Wave 1 + Wave 2 + Wave 3 execution pass (see git log on
> `phase/01-v1.1-execute`).
>
> The 11 candidates are `[CANDIDATE]` until the CPO authors the
> matching entries in `docs/requirements.md`; the must-haves
> below reference the candidate IDs as a planning aid. The
> `.ciagent/inboxos/ROADMAP.md` Phase 1 status row is not
> flipped by the lead-developer; the flip is the CPO's call.
> The lead-developer's gate is "every must-have is checkbox-
> green and the 11 PRs are merged".
| # | Must-have (verifiable) | Source-of-truth location | Owner | REQ-INB |
|----|------------------------|--------------------------|-------|:-------:|
| 1 | `.env.example` lines 96-99 reference `gemma4:e4b` only (no `lfm2.5:8b` or `gemma4:12b` comments) | `.env.example:96-99` | devops-engineer | REQ-INB-15 |
| 2 | Runbook §9.1 #12 cites `upload-artifact@v3` only in the workflows that emit per-run logs (live + historical); `ci-pr.yml` excluded | `docs/operations-runbook.md:544` (the row updated in T1.2) | devops-engineer | REQ-INB-16 |
| 3 | Runbook §6.1 says "22 distinct secret values" (matching the 22 data rows in the table) | `docs/operations-runbook.md:399` (the sentence updated in T1.3) | devops-engineer | REQ-INB-17 |
| 4 | `inb-28-report.md` "Per-table row counts at freeze" is filled with one row per module table | `docs/phase-1.1-verification/inb-28-report.md:67-77` | data-engineer | REQ-INB-18 |
| 5 | `papra_index` NocoDB table exists with the 7 documented columns; `docs/schemas.md` and `scripts/apply_schemas.py` reflect the table; the v1.0 schema freeze is preserved for the 4 module tables | `docs/schemas.md` (new `papra_index` section, T1.4a), `scripts/apply_schemas.py` (new `AUDIT_TABLES` entry, T1.4b), `docs/phase-1.1-verification/inb-28-report.md` (the freeze amendment, T1.4a) | data-engineer | REQ-INB-19 |
| 6 | `PAPRA_DRY_RUN_OVERRIDE` env var exists; the new `papra_write_gate` event is emitted by both pipelines (via the T1.5 helper + T2.2 call-sites); the canary procedure is in `docs/papra-write-policy.md`; new test cases exercise the override and the helper | `pipelines/common.py` `PapraConfig` + `emit_papra_write_gate_event` (T1.5), `pipelines/live_pipeline.py` + `pipelines/historical_pipeline.py` (T2.2 call-sites), `pipelines/_tests/test_papra_decide_offline.py` (T1.5 + T2.2 cases), `docs/papra-write-policy.md` "Live-write gate" (T1.5) | security-engineer | REQ-INB-20 |
| 7 | `scripts/rotate_secrets.sh` exists with `--dry-run` and real-run modes; `pipelines/_tests/test_rotate_secrets_offline.py` passes; `docs/operations-runbook.md §6.2` documents the rotation ordering | `scripts/rotate_secrets.sh` (new), `pipelines/_tests/test_rotate_secrets_offline.py` (new, 3/3 tests pass), `docs/operations-runbook.md §6.2` (new) | security-engineer | REQ-INB-21 |
| 8 | `scripts/backup.sh` and `scripts/restore.sh` exist; `pipelines/_tests/test_restore_offline.py` round-trips a known fixture; `docs/operations-runbook.md §5.7 + §10` state RTO < 4h and RPO < 24h | `scripts/backup.sh` (new), `scripts/restore.sh` (new), `pipelines/_tests/test_restore_offline.py` (new, 7/7 tests pass), `docs/operations-runbook.md §5.7 + §10` (new) | devops-engineer | REQ-INB-22 |
| 9 | `pipelines/_metrics/emit.py` writes one JSONL line per run with the documented counter and gauge vocabulary; `docs/operations-runbook.md §4.5` documents the tail-and-summarize procedure; a new test exercises the emitter | `pipelines/_metrics/emit.py` (new), `pipelines/live_pipeline.py` + `pipelines/historical_pipeline.py` (T2.2 call-sites), `pipelines/_tests/test_metrics_offline.py` (new, 2/2 tests pass), `docs/operations-runbook.md §4.5` (new) | data-engineer | REQ-INB-23 |
| 10 | `plans/05-PLAN.md` parallelization table has the explicit three-file split footnote (P1-1 closure); runbook §9.2 row 19 has the backend-engineer persona annotation (P2-4 closure); CI-LOG P1-1 + P2-4 entries are marked closed | `.ciagent/inboxos/plans/05-PLAN.md:318-330` (footnote), `docs/operations-runbook.md:674` (row 19 owner annotation), `.ciagent/inboxos/CI-LOG.md:96-97,112-113,115` (closed_in pointers) | lead-developer | REQ-INB-24 |
| 11 | The live pipeline's `process_one_message` patches `Papra_Link` onto every NocoDB row after a successful `papra_write_attachment`; the "Live pipeline note" paragraph in `docs/papra-write-policy.md §Rule 1` is removed; the asymmetry paragraph in `docs/extraction-contract.md §Live vs historical: Papra_Link asymmetry` is removed; `ARCHITECTURE.md §Document Store` is updated to reflect parity | `pipelines/live_pipeline.py:486-555` (the new PATCH loop + `papra_link_patched` event, T1.7), `docs/papra-write-policy.md:55-72` (the §Rule 1 asymmetry note removed by T2.3), `docs/extraction-contract.md:155-181` (the subsection removed by T1.7), `.ciagent/inboxos/ARCHITECTURE.md:65-72` (the paragraph replaced by T1.7) | backend-engineer | REQ-INB-25 |
A v1.1 close is the date on which all 11 must-haves are
checkbox-green and the CPO has promoted the 11 candidates to
real requirements in `docs/requirements.md`. The
`.ciagent/inboxos/ROADMAP.md` Phase 1 status row is not flipped
to "executed" by the lead-developer; the flip is the CPO's
call. The lead-developer's gate is "every must-have is
checkbox-green and the 11 PRs are merged".
---
## 10. Disaster recovery targets (REQ-INB-22)
The backup / restore procedure in §5.7 is the substrate; this
section is the **commitment**.
- **RTO (Recovery Time Objective):** **< 4 hours**, single-
operator restore. The breakdown:
- 30 min — re-read §5.7 + §6.2 and decide which services
to restore (typically: all three, in order: Gitea first
so the workflows and secrets are back, then NocoDB so
the module-table IDs are available, then SeaweedFS).
- 60 min — restore the Gitea bundle and force-push the
`jc-capital/inboxos` repo. Verify the workflows are
visible at the Gitea UI.
- 60 min — restore the NocoDB export. The export is a
JSON dump; the operator opens the NocoDB UI and uses
Settings → Import to load the JSON. Verify a known row
via the `curl ... where=(Message_ID,eq,...)` pattern
in §5.2.
- 60 min — restore the SeaweedFS mirror. The `aws s3 sync`
(or boto3 fallback) copies every object from the
backup bucket to the primary bucket. Verify a known
`.eml` via the `s3_get_text_optional` helper.
- 30 min — buffer for unexpected drift (single-operator
runs are not 24/7; if the restore starts at 02:00 UTC
the operator is fresh by 04:00 UTC).
- **RPO (Recovery Point Objective):** **< 24 hours** data
loss. The daily backup cron (§5.7) runs at 02:00 UTC, so
the worst-case loss is the previous 24 hours of incoming
email. In a typical single-operator inbox this is < 50
messages, all recoverable from Gmail via `since_uid` (the
live pipeline's recovery path in §5.2).
The RTO and RPO values are explicit and CPO-acknowledged.
The ideation report Tier 2 T2-C marked them as "guesses"
pending CPO sign-off; the values recorded here are the
v1.1 ship defaults. Changes to the values require a new
CPO-approved REQ-INB (the values are not engineer-editable
in isolation).
---
*End of runbook. Next review: after the v1.3 ship (2026-06-08, milestone `milestone/v1.3-initial`), or when the next failure mode is added to §5. Last verified against the source files cited inline; any divergence is a bug in this document — fix the doc, do not just edit around it.*
Reference in New Issue View Git Blame Copy Permalink