inboxos-v1/docs/operations-runbook.md

InboxOS — Operations Runbook

Audience: Any operator (or the same operator after a 6-month gap) recovering from common failure modes without re-deriving the architecture.

Status: Phase 5 deliverable for INB-7 — issued under INB-35.

Scope: Single self-hosted pipeline. A Gmail inbox → IMAP poll → SeaweedFS archive + LLM-extracted rows in NocoDB + curated attachments in Papra. Everything is self-hosted; no third-party SaaS touches PII.

Acceptance test for this runbook: "Given a workflow is stuck in in_progress for 1 hour, what do you do?" — answer in under 5 minutes using §5.1 below.

---

1. System overview

InboxOS is a self-hosted, air-gapped personal ERP/CRM pipeline. It polls a single Gmail inbox over IMAP, backs up the raw .eml to SeaweedFS, classifies each email with a local LLM (Ollama), runs one or more inboxos:extract-<module> prompts against the same email, and UPSERTs the resulting structured records into NocoDB (one table per module: finance, crm, intel, promotions). Qualifying attachments are also uploaded to Papra for long-term searchability. The historical pipeline performs the same flow against a Google Takeout .mbox for backfill, with byte-offset checkpoints for safe resume.

1.1 Topology

┌──────────────┐    IMAP/993     ┌─────────────────┐
│   Gmail      │ ◀─────────────  │  gitea-runner-1 │
│   inbox      │   (App Password)│  gitea-runner-2 │  (self-hosted, parallel)
└──────────────┘                 └────────┬────────┘
                                         │ run jobs (Gitea Actions)
                                         │ polls IMAP, calls Ollama
                                         ▼
┌──────────────────┐   raw .eml  ┌─────────────────┐
│   Papra          │  attachment │   SeaweedFS     │  S3-compatible
│   (long-term doc)│ ◀───────────│   (S3 API)      │  .eml archive +
│   intake        │   uploads    │                 │  state checkpoints
└──────────────────┘             └────────┬────────┘
                                          │ UPSERT by Message-ID
                                          ▼
                                ┌─────────────────┐    direct HTTP
                                │   NocoDB        │ ── /v1/chat
                                │   (DWH)         │   /completions
                                └─────────────────┘         │
                                          ▲                 ▼
                                ┌─────────────────┐
                                │   Ollama        │  local LLM
                                │   (FRP)         │  gemma4:e4b
                                │   /v1/chat      │
                                │   /completions  │
                                └─────────────────┘

• Gmail is the source. The runner authenticates with an App Password (2FA must be on).
• Gitea is the version-controlled home for prompts, scripts, and CI.
• Gitea Actions runners (gitea-runner-1 and gitea-runner-2, both self-hosted) execute the live and historical pipelines. Two runners so the 5-minute live cron and a long historical run can execute in parallel.
• SeaweedFS is the system of record for raw .eml (in s3://$SEAWEEDFS_S3_BUCKET/eml/...) and pipeline state (state/live-state.json, state/historical-state.json).
• NocoDB is the structured warehouse. Every module table is keyed on Message_ID (UNIQUE). All writes are UPSERTs.
• Papra holds curated attachments and (selectively) full email bodies for long-term search. The first ship goes out in dry-run mode.
• Ollama serves the local LLM behind a FRP tunnel at https://ollama-frp.cloudinit.dev/v1. No external API calls for PII-bearing data.

---

2. Service inventory

All services run in the inboxos-net Docker network defined in docker/docker-compose.yml.

#	Service	Container / DNS name	Public URL	Role in pipeline	Restart	Log location
1	NocoDB	nocodb	https://noco.cloudinit.dev	ERP/CRM data warehouse. Four module tables (finance, crm, intel, promotions).	docker compose -f docker/docker-compose.yml restart nocodb	docker logs nocodb
2	Gitea	gitea	https://git.cloudinit.dev	Version control for prompts, scripts, and CI. Hosts Actions runner registration & secrets.	docker compose -f docker/docker-compose.yml restart gitea	docker logs gitea
3	Gitea Actions Runners	gitea-runner, gitea-runner-2	n/a (self-hosted)	Execute the live and historical workflows. Two so live cron + historical run can run in parallel.	docker compose -f docker/docker-compose.yml restart gitea-runner [gitea-runner-2]	docker logs gitea-runner [gitea-runner-2] + per-job artifact under pipelines/_artifacts/*.log
4	Papra	papra	https://papra.cloudinit.dev	Curated document archive (attachments + selected bodies). First ship is dry-run.	docker compose -f docker/docker-compose.yml restart papra	docker logs papra
5	SeaweedFS	seaweedfs-master, -volume, -filer	https://storage.cloudinit.dev (via reverse proxy) on :8888 S3 endpoint in-network	Object store for .eml archive and pipeline state checkpoints.	docker compose -f docker/docker-compose.yml restart seaweedfs-{master,volume,filer}	docker logs seaweedfs-{master,volume,filer}
6	Ollama	ollama	https://ollama-frp.cloudinit.dev/v1	Local LLM inference for inboxos:classify and inboxos:extract-*. Default model gemma4:e4b.	docker compose -f docker/docker-compose.yml restart ollama	docker logs ollama

The OpenCode CLI is not invoked by either pipeline; both pipelines call Ollama's OpenAI-compatible /v1/chat/completions endpoint directly via pipelines.common.llm_run_command (default backend: ollama). The opencode binary install step in historical-pipeline.yml is retained for any human-driven opencode invocation from the repo worktree; the pipeline itself does not shell out to it. See docs/extraction-contract.md §Legacy: opencode CLI backend for the rationale.

The reverse proxy (Caddy in the commented-out caddy service in docker/docker-compose.yml) terminates TLS for *.cloudinit.dev and routes to each service. Operators without a reverse proxy can reach services via the inboxos-net Docker DNS names (e.g. http://gitea:3000, http://seaweedfs-filer:8888).

---

3. Workflow operations

All workflows live in .gitea/workflows/. They are declarative, version-controlled in the same repo as the scripts they invoke, and have no separate runtime state to back up. This is the board decision recorded in INB-8 — n8n was ruled out as unreliable.

Quickstart: for a 15-minute green-field spin-up, see docs/quickstart.md and scripts/quickstart.sh (REQ-INB-40). The script runs the 8 steps below in sequence and prints the runbook §X.Y cross-ref on any failure.

Daily digest: see scripts/daily_digest.py and .gitea/workflows/daily-digest.yml (REQ-INB-47). The workflow runs on cron 0 6 * * * and emails a one-line per-module summary.

Container hardening: all services run with cap_drop: [ALL], security_opt: no-new-privileges:true, and either read_only: true (no write access needed) or read_only: false with a tmpfs: [/tmp] mount (for services that need a writable scratch dir). See docker/docker-compose.yml for the full stanza and the per-service cap_add list (REQ-INB-55).

DPIA: see docs/security/dpia.md (REQ-INB-58). The Data Protection Impact Assessment per GDPR Art. 35 names the data flows, the lawful basis, the data-subject rights, and the residual risks.

Threat model: see docs/security/threat-model.md (REQ-INB-59). The STRIDE-per-component decomposition covers the 10 components in the inboxos architecture.

3.1 Live pipeline — live-pipeline.yml

Aspect	Value
Schedule	cron(*/5 * * * *) — every 5 minutes
Manual trigger	workflow_dispatch with optional inputs since_uid (override high-water mark) and max_emails (default 50)
Concurrency	group: live-pipeline, cancel-in-progress: false (overlapping runs must not trample state)
Timeout	15 minutes
Script	pipelines/live_pipeline.py
Run log	pipelines/_artifacts/live-run-<id>.log (uploaded as live-pipeline-log artifact)

Flow (verifiable in pipelines/live_pipeline.py):

1. Load the IMAP UID checkpoint from s3://$SEAWEEDFS_S3_BUCKET/state/live-state.json.
2. Poll IMAP for UID > last_uid (bounded by MAX_EMAILS_OVERRIDE, default 50).
3. For each message, in UID-ascending order:
   • Back up the raw .eml to s3://$SEAWEEDFS_S3_BUCKET/eml/live/<yyyy>/<mm>/<dd>/<Message-ID>.eml (skip if exists).
   • Call inboxos:classify to pick which modules apply.
   • For each applicable module, call inboxos:extract-<module> and UPSERT the result.
   • Flush the checkpoint (state/live-state.json) per message so a crash never replays a success.

What success looks like in the run log:

• One live_run_start event at the top.
• One or more process_start → eml_backed_up → classified → process_done event groups, in UID-ascending order.
• Every process_done is followed by a checkpoint write (visible as the state/live-state.json PUT in S3, or as the next call to imap_fetched).

What failure looks like:

• A classify_error or extract_permanent_failure event means the pipeline stops at that UID. The next run retries it.
• A dead_letter_unclassified event means the email had no module match — the raw .eml is preserved at eml/dead-letter/<mid>.eml and the cursor advances.
• A missing_message_id event means the email lacked a Message-ID: header. The raw .eml is preserved at eml/dead-letter/uid-<n>.eml.
• A non-zero job exit code with no live_run_done in the log means the runner OOM'd or the IMAP connection was killed mid-fetch — see §5.1.

3.2 Historical pipeline — historical-pipeline.yml

Aspect	Value
Schedule	None — manual only (workflow_dispatch). Heavy operation; full archives run for hours.
Required input	mbox_object_key — the SeaweedFS object key of the .mbox (e.g. eml/historical/all-mail.mbox)
Optional inputs	start_offset (resume from a byte offset; leave blank to use persisted checkpoint), batch_size (default 100), limit (0 = no limit)
Concurrency	group: historical-pipeline, cancel-in-progress: false
Timeout	12 hours
Script	pipelines/historical_pipeline.py (canonical reference for the extraction contract)
Streamer	pipelines/mbox_stream.py — byte-offset streaming parser for .mbox
Run log	pipelines/_artifacts/historical-run-<id>.log (uploaded as historical-pipeline-log artifact)

Idempotency contract:

• Message-ID UPSERT is the only write path into NocoDB (pipelines/common.py::nocodb_upsert). It does a GET by Message_ID and PATCHes if present, POSTs otherwise. Re-running the same .mbox produces zero duplicates.
• Byte-offset checkpoint is persisted to s3://$SEAWEEDFS_S3_BUCKET/state/historical-state.json every BATCH_SIZE messages (default 100) and on shutdown. A crashed run resumes from the last checkpoint on the next dispatch.
• Object keys are deterministic by Message-ID and date, so re-uploading the same .eml overwrites the same S3 object.

Expected runtime per 10K messages: ~5–10 minutes on a clean c5.large runner when the LLM is the bottleneck (depends on the model — gemma4:e4b is fast; gemma4:12b adds ~2×). The I/O-bound parts (S3 GET of the .mbox, S3 PUTs of .eml, Papra uploads) are sub-second per message.

How to resume after a crash:

1. Re-dispatch historical-pipeline.yml with the same mbox_object_key and no start_offset input. The script loads state/historical-state.json and resumes from the persisted byte offset.
2. Do not dispatch with start_offset=0 unless you have deleted the state file — that re-processes everything from the top (safe but wasteful, and will take hours).

How to dead-letter a message manually:

• A message that was already dead-lettered by the pipeline lives at s3://$SEAWEEDFS_S3_BUCKET/eml/dead-letter/historical-<offset>.eml (or <mid>.eml). To permanently skip a poison message, find its Message-ID in the run log, then write a marker object to eml/dead-letter/poison/<mid>.json containing the reason. A future enhancement will let the script consult this list; for now, mark it in your run log and move on.

3.3 PR CI — ci-pr.yml

Aspect	Value
Triggers	pull_request opened / reopened / synchronized against main; workflow_dispatch
Concurrency	group: pr-ci-${{ pull_request.number }}, cancel-in-progress
Jobs	fast-tests (~30s), e2e-100mb (20 min), live-llm-smoke (15 min, real Ollama)
Required secrets	NOCODB_BASE_URL, NOCODB_API_TOKEN, NOCODB_PROJECT_ID; OLLAMA_BASE_URL is optional

• fast-tests exercises gitea_request, mbox_stream, the offline parts of live_pipeline, and the fast subset of historical_pipeline (config, parsing, dead-letter paths).
• e2e-100mb builds a 100 MB synthetic .mbox and runs the historical pipeline end-to-end with S3 / NocoDB / Papra stubbed. Verifies UPSERT idempotency, crash-and-resume offset stability, and dead-letter semantics. The merge gate.
• live-llm-smoke calls the real Ollama endpoint with the Substack input that previously dead-lettered in live run #674 (UID 4017, 86 KB body) and asserts the extract step returns parseable JSON. Skipped when OLLAMA_BASE_URL is not set; surfaces a clear skipped notice rather than failing the PR.

---

4. Observability

4.1 Reading a workflow run log

1. Open the Gitea UI → repo → Actions → click the run.
2. The job's Run process-new-emails / Run process-mbox step shows stdout in the Gitea Actions UI. Each line is a JSON object emitted by log_event.
3. Expand the Upload run log as artifact step. The per-run log file is downloaded as live-pipeline-log or historical-pipeline-log. This is identical to what was streamed to stdout, and is the canonical artifact to attach to a post-mortem.
4. The artifact also lives in the project workspace at pipelines/_artifacts/<pipeline>-run-<run-id>.log.

4.2 Structured event vocabulary

Every step in both pipelines calls pipelines.common.log_event to emit a JSON line. The canonical event names (verified in source) are:

Event	Where	What it tells you
process_start	live_pipeline.py:428, historical_pipeline.py:282	A new message is about to be processed. Carries uid (live) or offset (historical) and message_id.
eml_backed_up	both pipelines	The raw .eml was PUT to SeaweedFS. Carries key and bytes.
classified	both pipelines	inboxos:classify returned a non-empty module list. Carries modules.
llm_call	common.py:591	The Ollama call is about to be made. Carries backend, modelID, command, url.
process_done	both pipelines	The message was processed end-to-end. Carries upserts (NocoDB action list).
module_table_missing	both pipelines	A module came back from classify but has no entry in NOCODB_MODULE_TABLES. Carries module.
extract_permanent_failure	both pipelines	inboxos:extract-<module> failed irrecoverably. The run stops without advancing.
dead_letter_unclassified	both pipelines	Classify returned []. The .eml is preserved at eml/dead-letter/... and the cursor advances.
historical_run_done	historical_pipeline.py:545	The historical run finished cleanly. Carries totals.
live_run_start	live_pipeline.py:533	The live run began. Carries started_at.
papra_decision	common.py:997	The Papra write-through decision was made. Carries would_write, reason, module, dry_run.
papra_write_ok / papra_write_failed / papra_write_dry_run	common.py:1018-1055	Per-target outcome of a Papra write.
state_recovered_from_corrupt	live_pipeline.py:550	The persisted IMAP checkpoint was malformed; the run reset it.

The live_pipeline.py source also emits imap_search_result, imap_fetch_missing_body, imap_fetched, classify_error, extract_invalid_response, missing_message_id, config_error, since_uid_override, stopped_on_failure_at_uid, and others (search for log_event in pipelines/live_pipeline.py for the full list).

4.3 NocoDB table inventory

The four module tables (verified against pipelines/_tests/test_upsert_synthetic.py and docs/phase-1.1-verification/inb-28-report.md):

Module	Table name in NocoDB	Table ID	Notes
finance	pl_finance	m0thluc3jyz2iok	Receipts, invoices, P&L items. 14 user columns.
crm	personal_crm	m2wsbpp01v08pm5	People, organisations, intent. 15 user columns.
intel	daily_news	msi6dj1cany1b5r	Newsletters, briefings. 13 user columns.
promotions	promotions	mf7catfwwyh3jj5	Sales, promo codes, launches. 14 user columns.

Per-table row counts and column schemas are listed in docs/phase-1.1-verification/inb-28-report.md and docs/schemas.md. Every module table has Message_ID (SingleLineText, UNIQUE, required) as the natural key.

4.4 SeaweedFS bucket layout

$SEAWEEDFS_S3_BUCKET is inboxos (default in .env.example). The layout is:

s3://inboxos/
├── eml/
│   ├── live/
│   │   └── <yyyy>/<mm>/<dd>/<Message-ID>.eml
│   ├── historical/
│   │   └── <yyyy>/<mm>/<dd>/<Message-ID>.eml
│   └── dead-letter/
│       ├── <Message-ID>.eml          (live, unclassified)
│       ├── uid-<uid>.eml             (live, missing Message-ID)
│       └── historical-<offset>.eml   (historical)
└── state/
    ├── live-state.json               {"last_uid": <int>, "last_run_at": <iso>, "messages_seen": <int>}
    └── historical-state.json         {"mbox_key": <str>, "offset": <int>, "messages_seen": <int>, "last_run_at": <iso>}

State-file field names come from the actual code: live (pipelines/live_pipeline.py:600-603) and historical (pipelines/historical_pipeline.py:516-521). The _corrupt_state_recovered field is an additional transient key on first load when the persisted JSON is unparseable.

The state files are tiny JSON blobs written via s3_put_bytes after every successful message (live) or every BATCH_SIZE messages (historical). They are the only persistent state between workflow runs.

4.5 Metrics surface (REQ-INB-23)

Each pipeline run writes one JSONL line to pipelines/_artifacts/<pipeline>-metrics-<run_id>.jsonl (one file per run; never overwritten). The line shape and counter / gauge vocabulary are documented in pipelines/_metrics/emit.py (a copy-paste of the docstring is in this runbook for operator reference).

Counter key	Type	What it tells you
messages_processed	int	Total messages processed in this run.
messages_dead_lettered	int	Subset routed to the dead-letter prefix.
noco_upsert_ok	int	NocoDB upserts that returned 2xx.
noco_upsert_4xx	int	4xx (drift; permanent failure for the affected message).
noco_upsert_5xx	int	5xx (transient; retried by tenacity).
papra_would_write	int	Papra decisions that resolved would_write=true.
papra_write_ok	int	Papra HTTP POSTs that returned 2xx.
papra_write_failed	int	Papra HTTP POSTs that returned >= 400.
papra_index_upsert_ok	int	papra_index audit-trail rows written.
papra_index_upsert_failed	int	papra_index upserts that raised.
llm_call_seconds_sum	int	Total wall-clock ms spent in requests.post to Ollama.
llm_call_count	int	Number of LLM HTTP POSTs in the run.

Gauge key	Type	What it tells you
last_run_at	ISO 8601 str	The run's now_iso() at metrics emission.
dead_letter_total	int	Cumulative dead-letter prefix size (best-effort).
papra_dry_run	"true"/"false"	The effective PAPRA_DRY_RUN after precedence.
papra_dry_run_override	"true"/"false"/"none"	The parsed override (or "none" if unset).

Tail-and-summarize procedure. To compute a daily total of messages_processed across all live runs:

cat pipelines/_artifacts/live-metrics-*.jsonl \
    | jq -s '[.[].counters.messages_processed] | add'

To inspect the last 20 metrics lines in time order:

cat pipelines/_artifacts/live-metrics-*.jsonl \
    | tail -20 \
    | jq -c '{ts, run_id, counters}'

Forward-pointer. Alerting is out of scope for v1.4. v1.2 closed on a 0/1 ideation disposition (no candidates accepted) per .ciagent/inboxos/CI-LOG.md, and v1.3 was a doc-hygiene milestone; v1.4 added the operator audit cursor and the pollable healthcheck but no alert layer. The metrics file is the substrate; the alert layer is the next v1.x candidate cycle to re-evaluate. In the meantime the runbook §4 observability surface is logs-only, with pipelines/_metrics/emit.py emitting a per-run metrics.jsonl (REQ-INB-23) for offline postmortem and pipelines/_metrics/healthcheck.json (REQ-INB-52) as the canonical uptime-monitor pollable endpoint.

Opt-out. Set LLM_METRICS_ENABLED=false in the pipeline env to skip the time.monotonic() wrapping around the LLM HTTP POST. The counters default to enabled; opt-out is for canary runs that want to measure pure LLM latency without the wrapper overhead.

---

5. Common failure modes and recovery

ERROR-NN catalog

A one-page lookup table for the on-call operator. If you are reading this from a pager, the table below tells you which §5.X subsection to jump to for the full recovery procedure.

ERROR-NN	Symptom	See §X.Y
ERROR-01	Runner stuck / OOM (> 60 min in in_progress)	§5.1
ERROR-02	Live cron emitted no new MIDs for > 30 min	§5.2
ERROR-03	NocoDB 4xx on upsert (wrong table_id)	§5.3
ERROR-04	NocoDB 4xx "field is required" (schema drift)	§5.3
ERROR-05	IMAP connection refused / auth failure	§5.4
ERROR-06	Ollama connection refused / model OOM	§5.5
ERROR-07	SeaweedFS connection refused / auth failure	§5.6
ERROR-08	Papra dry_run unexpectedly flipped to false	§5.7
ERROR-09	State file corrupted / offset regression	§5.8
ERROR-10	Backup tar fails (NocoDB / SeaweedFS / Gitea)	§5.9

5.1 Runner stuck / OOM (> 60 minutes in in_progress)

A single stuck job can block the live cron (live jobs have cancel-in-progress: false). Diagnosis and recovery:

1. Confirm it is stuck. Open Gitea Actions → run → check whether the Run process-new-emails step has emitted any new log lines in the last 10 minutes. A runner that is downloading a large .mbox is making progress (watch for S3 GET bytes) — a runner that is silent is OOM'd or wedged.
2. Check the host's container health.

   docker ps --filter "name=gitea-runner" --format "table {{.Names}}\t{{.Status}}\t{{.CPUPerc}}\t{{.MemUsage}}"
   

   Memory at 100% with no progress for > 10 minutes = OOM. CPU at 0% with no progress = wedged.
3. Drain the stuck job. From Gitea UI → run → Cancel job. The live workflow's cancel-in-progress: false will not preempt the next cron tick — the next run will queue until the cancelled one reports back. This is the correct behaviour: do not change the concurrency setting to fix a stuck job.
4. Restart the runner.

   docker compose -f docker/docker-compose.yml restart gitea-runner
   

   If gitea-runner-2 is also stuck, restart it too. The other runner will pick up new jobs within ~30 s.
5. Verify recovery.

   curl -u "$GITEA_USER:$GITEA_PASS" \
     https://git.cloudinit.dev/api/v1/repos/jc-capital/inboxos/actions/runners
   

   Expect 2 entries with status: "online". Then dispatch live-pipeline.yml manually with a small max_emails (e.g. 5) to confirm the end-to-end loop.
6. If recurring: the runner is under-provisioned. Increase its memory limit in docker/docker-compose.yml (mem_limit: 4g or similar) or move to a host with more RAM.

5.2 SeaweedFS NoSuchKey on a Message-ID

This means a NocoDB row references a Message-ID whose raw .eml was never uploaded to (or was deleted from) SeaweedFS. Two common causes:

1. The .eml write failed but the NocoDB upsert still succeeded (e.g. SeaweedFS was down briefly; the retry on the S3 PUT hit a 5-attempt limit; the pipeline then proceeded to NocoDB).
2. The Message-ID is malformed — e.g. a stray < or > made it into the NocoDB row but the file was written with the correct key.

How to find the offending record in NocoDB:

# For a finance row, replace m0thluc3jyz2iok with the module's table ID from §4.3
curl -G \
  -H "xc-token: $NOCODB_API_TOKEN" \
  --data-urlencode "where=(Message_ID,eq,<message-id-here>)" \
  --data-urlencode "limit=1" \
  "$NOCODB_BASE_URL/api/v2/tables/m0thluc3jyz2iok/records"

How to dead-letter the row:

1. Verify the .eml truly is missing in SeaweedFS (use the s3_get_text_optional helper in pipelines/common.py against s3://$SEAWEEDFS_S3_BUCKET/eml/live/...).
2. If it is missing, re-fetch the message from Gmail via IMAP using the Message-ID (Gmail's IMAP supports UID FETCH by Message-ID only via header search; the easier path is since_uid).
3. Re-dispatch live-pipeline.yml with since_uid set to the lowest affected UID. The pipeline will re-archive the .eml (overwriting the same key, or PUTing a new one) and re-UPSERT the NocoDB row.
4. If the IMAP mailbox no longer has the email (it was deleted from Gmail), the row is orphaned. Mark it manually in NocoDB (set a Notes field with a date) and move on — do not bulk-delete rows, the audit trail matters.

5.3 NocoDB 4xx on a UPSERT

Read the response body (NocoDB returns JSON like {"msg":"..."}). Common causes and recovery:

Cause	Symptom	Recovery
Wrong table_id	404 from /api/v2/tables/<id>/records	Verify the table ID in NOCODB_MODULE_TABLES matches the live table ID (see §4.3). Re-bootstrap with python scripts/apply_schemas.py to print the current mapping.
Missing required field	400 with "<field> is required"	Re-run python scripts/apply_schemas.py to add the column. UPSERT will retry on the next dispatch.
Schema drift (column renamed, type changed)	400 with "<field> validation failed"	Edit docs/schemas.md first, then scripts/apply_schemas.py. Re-run the workflow.
Message_ID UNIQUE violation on a POST	This should not happen — nocodb_upsert does GET-then-PATCH/POST. If it does, race.	Re-run the workflow; the second pass will hit the GET and switch to PATCH.
SingleSelect option not in the allowed list	400 with "<option> is not a valid option for <field>"	Add the option to the SingleSelect column in NocoDB UI, then re-run.

The full retry policy lives in pipelines/common.py::nocodb_upsert (tenacity, 5 attempts, exponential backoff 1s → 30s). A 4xx surfaces immediately and stops the run; the next dispatch retries from the same checkpoint.

5.4 Ollama 4xx / timeout

1. Verify the endpoint is reachable:

   curl -sS "$OLLAMA_BASE_URL/models" -H "Authorization: Bearer $OLLAMA_API_KEY"
   

   Expect 200 with a JSON list. The 4xx response body is usually self-explanatory (model not found, auth required, OOM on the Ollama host).
2. If the model is missing (the response lists other models but not the one in INBOXOS_LLM_MODEL):

   docker exec ollama ollama pull gemma4:e4b
   

3. To switch the model without redeploying: update the repo variable INBOXOS_LLM_MODEL (and INBOXOS_LLM_MODEL_EXTRACT for the larger extract model) in Gitea → repo → Settings → Actions → Variables. The next workflow run picks it up. Do not edit the prompt's model: frontmatter to "fix" a model outage — prompts are version-controlled, and that change goes through a PR.
4. To raise the timeout: set INBOXOS_LLM_TIMEOUT (in seconds, default 120). For very rich emails (86 KB Substack bodies) with the larger INBOXOS_LLM_MODEL_EXTRACT model, 180–240 s is not unusual.
5. Check the Ollama host itself for OOM:

   docker stats ollama --no-stream
   

   If memory is at the limit, the Ollama container is OOM-killing the loaded model. Either reduce the model size (gemma4:e4b instead of gemma4:31b) or add memory to the host.

5.5 Live pipeline falling behind

Current symptom (referenced in the INB-35 description): a single runner + 30+ minute cycles. The 5-minute cron cannot keep up because:

• The single runner is doing one job at a time (cancel-in-progress: false).
• IMAP UID backlog from a sync delay (e.g. Gmail was unreachable for an hour) means the next run has > 50 emails to process, and the runner is LLM-bound.

Recovery options (in order of preference):

1. Add a second runner. Already provisioned in docker/docker-compose.yml as gitea-runner-2; the only remaining step is the one-time registration token (see docs/gitea-actions.md §2a and INB-33). The historical workflow then runs in parallel on the second runner, freeing the first runner for the live cron.
2. Bump the cron frequency. Edit .gitea/workflows/live-pipeline.yml → schedule (e.g. cron("*/2 * * * *") for every 2 minutes). This is local-only, no board approval required. Note: 1 minute is the practical floor; Gitea will skip runs if the previous one is still in flight.
3. Pause the cron (board approval required). Use only when the pipeline is producing corrupted data or the LLM is rate-limiting. Two ways:
   • In the Gitea UI: repo → Actions → live-pipeline.yml → Disable workflow.
   • In code: open a PR that comments out the schedule: block; merge. Reverting the PR re-enables it.
4. Bump max_emails per run (default 50). The workflow input max_emails can be raised to 200, 500, etc. via workflow_dispatch. This is a one-shot — it does not change the default.

5.6 Historical pipeline run cancelled mid-way

This is safe by construction. The byte-offset checkpoint in state/historical-state.json is written every BATCH_SIZE messages (default 100) and on shutdown. A cancelled run:

1. Flushes whatever batch is currently in-flight to the checkpoint before exiting.
2. Leaves NocoDB and SeaweedFS in a consistent state — every successful process_done has already been committed.

Recovery: dispatch historical-pipeline.yml again with the same mbox_object_key and no start_offset input. The script will load state/historical-state.json, seek to the persisted byte offset, and continue.

Do not re-dispatch with start_offset=0 unless you have intentionally deleted state/historical-state.json. Re-processing from offset 0 is safe (UPSERT dedupes) but wasteful — for a multi-GB .mbox, it will take hours and consume LLM tokens you do not need to spend.

To permanently restart a historical run from scratch (e.g. after a prompt overhaul you want to re-test):

# From a runner or any host with the S3 creds:
python -c "
import boto3, os
s3 = boto3.client('s3', endpoint_url=os.environ['SEAWEEDFS_S3_ENDPOINT'],
                  aws_access_key_id=os.environ['SEAWEEDFS_S3_ACCESS_KEY'],
                  aws_secret_access_key=os.environ['SEAWEEDFS_S3_SECRET_KEY'])
s3.delete_object(Bucket=os.environ['SEAWEEDFS_S3_BUCKET'], Key='state/historical-state.json')
print('state/historical-state.json deleted; next dispatch starts from offset 0')
"

Then dispatch historical-pipeline.yml with start_offset=0 and the same mbox_object_key.

5.7 Backup and restore (REQ-INB-22)

The scripts/backup.sh and scripts/restore.sh helpers snapshot and restore the three stateful services (NocoDB, SeaweedFS, Gitea) to/from a timestamped directory under backups/<service>/<utc-timestamp>/. The helpers are idempotent: a re-run for the same service at the same UTC timestamp is a no-op.

Schedule. A daily cron job on the operator host calls the three backup helpers in sequence:

# /etc/cron.d/inboxos-backup — runs at 02:00 UTC daily.
0 2 * * *  cto  /opt/inboxos/scripts/backup.sh nocodb    >> /var/log/inboxos-backup.log 2>&1
5 2 * * *  cto  /opt/inboxos/scripts/backup.sh seaweedfs >> /var/log/inboxos-backup.log 2>&1
10 2 * * * cto  /opt/inboxos/scripts/backup.sh gitea     >> /var/log/inboxos-backup.log 2>&1

Retention. 7 daily + 4 weekly + 0 monthly. The single- operator posture is "the operator can change it" — the values are the v1.1 ship default and are documented in scripts/backup.sh header comment.

Copy-paste invocation.

# Backup one service.
scripts/backup.sh nocodb
scripts/backup.sh seaweedfs
scripts/backup.sh gitea

# Restore from a backup id (UTC timestamp).
scripts/restore.sh 2026-06-08T120000Z nocodb
scripts/restore.sh 2026-06-08T120000Z seaweedfs
scripts/restore.sh 2026-06-08T120000Z gitea

Test-or-die gate. A restore is not considered safe until pipelines/_tests/test_restore_offline.py exits 0. The test pins the exit-code table for missing args, missing env, missing backup directory, and the SeaweedFS "no backup target configured" path. It also exercises a git-bundle round-trip on an in-tree repo. The full end-to-end (backup → wipe → restore → verify) requires a live NocoDB / SeaweedFS / Gitea and is run by the operator on the host.

---

6. Security and secrets

6.1 Secret inventory

All Gitea repo secrets live at Gitea → repo → Settings → Actions → Secrets (and a parallel Variables section for non-secret tunables). The 15 secrets consumed by the workflows, grouped by owner:

Owner	Secret	Used by	Purpose
IMAP	IMAP_HOST	live	Gmail IMAP server (imap.gmail.com)
IMAP	IMAP_PORT	live	993
IMAP	IMAP_USER	live	Gmail address
IMAP	IMAP_PASSWORD	live	Gmail App Password (not the account password; 2FA required)
IMAP	IMAP_USE_SSL	live	true
IMAP	IMAP_MAILBOX	live	INBOX (default)
SeaweedFS	SEAWEEDFS_S3_ENDPOINT	both	https://storage.cloudinit.dev (or http://seaweedfs-filer:8888 in-network)
SeaweedFS	SEAWEEDFS_S3_ACCESS_KEY	both	S3 access key
SeaweedFS	SEAWEEDFS_S3_SECRET_KEY	both	S3 secret key
SeaweedFS	SEAWEEDFS_S3_BUCKET	both	inboxos
NocoDB	NOCODB_BASE_URL	both	https://noco.cloudinit.dev
NocoDB	NOCODB_API_TOKEN	both	xc-token header value
NocoDB	NOCODB_PROJECT_ID	both	NocoDB project ID
NocoDB	NOCODB_MODULE_TABLES	both	JSON map of module name → NocoDB table ID
Ollama	OLLAMA_BASE_URL	both	https://ollama-frp.cloudinit.dev/v1
Ollama	OLLAMA_API_KEY	both	Bearer token if Ollama is auth-gated; empty string otherwise
Papra	PAPRA_BASE_URL	both	https://papra.cloudinit.dev (optional)
Papra	PAPRA_API_TOKEN	both	Bearer token (optional)
Papra	PAPRA_ORGANIZATION_ID	both	Papra org id (optional)
Gitea	GITEA_INSTANCE_URL	both	http://gitea:3000 (in-network)
Gitea	GITEA_USER / GITEA_PASS	scripts	HTTP Basic Auth for the Gitea REST API (see common.gitea_request)

That's 22 distinct secret values (the v1.0 ship counted 21 because the GITEA_USER / GITEA_PASS row was treated as one credential; v1.1 counts both values to match the row count in the table above). The issue description rounds to 15 because the Papra trio is optional; the table is exhaustive.

Rotation schedule: None — all static. This is an explicit, accepted trade-off for a single-operator, self-hosted system. The S3 access keys and NocoDB API token are long-lived. The Gmail App Password can be revoked from the Google account at any time without breaking anything else; the IMAP user must re-issue a new App Password and update the Gitea secret. The helper landed in v1.1 (REQ-INB-21); see §6.2 for the procedure.

6.2 Secret rotation (REQ-INB-21)

The scripts/rotate_secrets.sh helper rotates a Gitea repo secret in place via pipelines.common.gitea_request. The new value is read from STDIN (never a CLI argument) so it is not visible in ps / process listings.

Rotation ordering. When multiple secrets need rotating, do them in this order (each step invalidates fewer downstream services than the next):

1. NOCODB_API_TOKEN first — it is the PII-bearing token (every NocoDB write is gated on it; rotate during low-traffic). A failed rotation here dead-letters new writes; rotate the pre-shared token in NocoDB and update the Gitea secret.
2. SEAWEEDFS_S3_ACCESS_KEY and SEAWEEDFS_S3_SECRET_KEY (treated as a pair) — the S3 archive and the pipeline state files are gated on these. A failed rotation here stops both pipelines; rotate the pre-shared key in SeaweedFS and update the Gitea secrets.
3. IMAP_PASSWORD — the Gmail App Password. A failed rotation here stops the live pipeline only; the historical pipeline is unaffected. Update the Gmail App Password and the Gitea secret.
4. GITEA_USER / GITEA_PASS last — the Basic Auth credentials that the helper itself uses. Rotating these invalidates the helper's own running session; do them after the helper has returned successfully for the other three.

Test-or-die gate. A rotation is not considered safe until pipelines/_tests/test_rotate_secrets_offline.py exits 0. The test exercises the dry-run path, the apply branch shape, and the env-validation failure mode. The dry-run path is the one operators should run first; it prints a redacted diff of the new value (sha256:8hex) without making the PUT.

Copy-paste invocation.

# Dry-run first; no PUT issued.
echo "$NEW_NOCODB_API_TOKEN" | scripts/rotate_secrets.sh NOCODB_API_TOKEN

# Real rotation; PUTs the new value, then verifies via GET.
echo "$NEW_NOCODB_API_TOKEN" | scripts/rotate_secrets.sh --apply NOCODB_API_TOKEN

The helper exits 0 on success, 2 on missing env / missing value, and 3 on Gitea API failure. See pipelines/_tests/test_rotate_secrets_offline.py for the test contract and the helper's exit-code table in its header comment.

6.3 Why we never commit secrets

• .gitignore excludes .env and .env.* except .env.example. The example file contains placeholder values only.
• The CTO host deliberately does not have write-capable S3 credentials (only the read-only S3 keys used for verification). A leaked CTO host key cannot corrupt the .eml archive.
• The Gitea service account used for the REST API (GITEA_USER=cto in .env.example) is scoped to the jc-capital/inboxos repo. It can read/write workflow secrets, but it cannot create new repos or admin the Gitea instance.
• All secret reads happen via ${{ secrets.X }} in the workflow YAMLs; the values are injected as environment variables into the ephemeral runner container and never appear in the build log.

6.4 Data-sovereignty posture

• No external API calls for PII-bearing data. The only outbound network calls from the runner are to the in-network services (http://gitea:3000, http://nocodb:8080, http://seaweedfs-filer:8888, http://ollama:11434, http://papra:1221). The Ollama URL is FRP-tunneled to the local Ollama container — the FRP tunnel terminates inside the operator's network; no third-party LLM provider sees the request body.
• Local LLM inference. Default model is gemma4:e4b (4 B effective parameters). All inboxos:classify and inboxos:extract-* calls hit the local Ollama. There is no fallback to a cloud LLM. If Ollama is down, the pipeline dead-letters and the next run retries.
• No telemetry from any pipeline. pipelines/common.py does not call home. Gitea Actions emits standard runner logs, which are stored on the Gitea host (also self-hosted).

---

7. Cost and capacity

7.1 Per-message cost

~$0 / message. All inference is local (Ollama on the operator's host). The only recurring costs are:

Cost	Magnitude (single operator)
Compute (Ollama host + Gitea host + runners)	One bare-metal or small VM; existing infrastructure.
Bandwidth (Gmail IMAP, FRP tunnel to Ollama)	Negligible at < 100 emails / day.
Storage (SeaweedFS volume)	See §7.2.
LLM API costs	$0 — local inference.

7.2 Storage growth

• A typical personal email .eml is ~30–100 KB. 100 KB / email is the safe upper bound used in capacity planning.
• 10 000 emails ≈ 1 GB of raw .eml in SeaweedFS.
• The SeaweedFS volume is provisioned at seaweedfs-data (named Docker volume); size is bounded by the host's disk. No quota is enforced at the SeaweedFS level.
• The NocoDB side is tiny: even 10 000 rows across four tables is single-digit MB.

7.3 Compute

• One Gitea runner (now two, post-INB-33), one Ollama container with gemma4:e4b loaded.
• Throughput observed (live): ~22 messages / 36 min on a single-runner + IMAP-backlog-saturated cycle. This is LLM-bound; each inboxos:classify + 1–2 inboxos:extract-* calls adds up to ~90 s per email with gemma4:e4b on CPU. A GPU host would cut this roughly 5–10× but is not required at the current scale.
• Throughput observed (historical): ~5–10 min per 10 K messages on gemma4:e4b CPU, dominated by the LLM calls. Switching to a smaller model (e.g. gemma3:4b) would roughly halve this; switching to a larger model (gemma4:12b) would roughly double it. Prompt quality and JSON-contract compliance are the binding constraints, not raw tokens/second.

---

8. Glossary and appendix

8.1 Glossary

• Message-ID — The RFC 5322 Message-ID: header, with angle brackets stripped. The natural primary key for every module table in NocoDB and the natural key for the .eml S3 key.
• UPSERT — "Update or insert." pipelines.common.nocodb_upsert does this by doing a GET by Message_ID and then a PATCH (if found) or POST (if not). There is no plain INSERT path in the codebase. The CSV / SQL concept maps 1:1 to the GET-then-PATCH/POST pattern.
• mbox — A plain-text file format for archiving email. Each message begins with From <mailbox> <date> and ends with a blank line. The historical pipeline uses mbox_stream.py to read it as a byte stream, never materialising the full file in memory.
• IMAP — Internet Message Access Protocol. The live pipeline uses IMAPClient against Gmail's imap.gmail.com:993 (SSL). Polling, not push, because Gmail's push (IDLE) is unreliable.
• FRP tunnel — Fast Reverse Proxy. The Ollama service is exposed at https://ollama-frp.cloudinit.dev/v1 via FRP, terminating inside the operator's network. Lets the runner talk to Ollama over HTTPS without opening inbound ports.
• Checkpoint state — The persisted byte offset (historical-state.json) or IMAP UID (live-state.json) that lets a pipeline resume from where it left off after a crash. The state file is the only persistent state between workflow runs.
• Dead-letter — A poison message or one that could not be processed is moved to s3://$SEAWEEDFS_S3_BUCKET/eml/dead-letter/... with a marker file describing the reason. The cursor advances so the rest of the email stream is not blocked. Dead-lettered items require manual review.
• Idempotency — "Running the same operation twice has the same effect as running it once." Achieved in InboxOS by (a) Message-ID UPSERT in NocoDB, (b) deterministic S3 keys, and (c) checkpoint-based resume. A redelivered Gmail message, a re-dispatched workflow, and a replayed .mbox are all safe.

8.2 Cross-references

• Schemas: docs/schemas.md — NocoDB table schemas (column-by-column).
• Extraction contract: docs/extraction-contract.md — how a raw email becomes a structured NocoDB record, the prompt family, and failure semantics.
• Gitea Actions: docs/gitea-actions.md — runner provisioning, second-runner setup, secrets, the Basic Auth model, observability, rollback.
• Pipelines: pipelines/README.md — pipeline flow diagrams, idempotency model, rollback procedure.
• Papra write policy: docs/papra-write-policy.md — when the pipeline writes to Papra (attachments, body for crm/finance/legal modules), the dry-run default, the live-write gate (REQ-INB-20), and the papra_index audit table (REQ-INB-19).
• Phase 1.1 verification: docs/phase-1.1-verification/inb-28-report.md — the schema-bootstrap deliverable that locked in the four table IDs and column counts cited in §4.3.
• Secret rotation: docs/operations-runbook.md §6.2 (new in v1.1, renumbered in v1.3 from §6.4) — the rotation helper, the ordering, and the test-or-die gate (REQ-INB-21).
• Metrics surface: docs/operations-runbook.md §4.5 (new in v1.1) — the per-run metrics.jsonl shape and the tail-and-summarize procedure (REQ-INB-23).
• Backup and restore: docs/operations-runbook.md §5.7 + §10 (new in v1.1) — the backup.sh / restore.sh helpers, the daily cron, and the RTO < 4h / RPO < 24h targets (REQ-INB-22).

8.3 Quick-reference: log-event search strings

If a post-mortem needs to find every place a particular event is emitted:

grep -rn 'log_event(logger, "process_done"'   pipelines/
grep -rn 'log_event(logger, "classified"'     pipelines/
grep -rn 'log_event(logger, "module_table_missing"' pipelines/
grep -rn 'log_event(logger, "dead_letter_unclassified"' pipelines/
grep -rn 'log_event(logger, "extract_permanent_failure"' pipelines/

For the full event vocabulary, grep -n "log_event" pipelines/common.py pipelines/live_pipeline.py pipelines/historical_pipeline.py.

8.4 Quick-reference: env vars (in alphabetical order)

Env var	Default	Purpose
GITEA_INSTANCE_URL	— (required)	Gitea REST API base. http://gitea:3000 in-network.
GITEA_USER	— (required for Gitea REST calls)	HTTP Basic Auth user.
GITEA_PASS	— (required for Gitea REST calls)	HTTP Basic Auth password.
IMAP_HOST / IMAP_PORT	— / 993	Gmail IMAP endpoint.
IMAP_USER / IMAP_PASSWORD	—	Gmail App Password credentials.
IMAP_USE_SSL / IMAP_MAILBOX	true / INBOX	IMAP connection toggles.
INBOXOS_LLM_BACKEND	ollama	ollama (recommended) or opencode (legacy, broken on 1.16.2).
INBOXOS_LLM_MODEL	gemma4:e4b	Default model for classify and any non-extract command.
INBOXOS_LLM_MODEL_EXTRACT	gemma4:e4b	Model for inboxos:extract-* commands.
INBOXOS_LLM_MAX_TOKENS	4096	Completion budget. 1024 was too tight for the 11-field extract schema.
INBOXOS_LLM_TIMEOUT	120	Seconds per LLM call.
NOCODB_API_TOKEN	—	xc-token header value.
NOCODB_BASE_URL	—	https://noco.cloudinit.dev
NOCODB_MODULE_TABLES	—	JSON map {"finance":"tbl_…","crm":"tbl_…",…}.
NOCODB_PROJECT_ID	—	NocoDB project ID.
OLLAMA_API_KEY	unset	Bearer token if Ollama is auth-gated.
OLLAMA_BASE_URL	https://ollama-frp.cloudinit.dev/v1	OpenAI-compatible chat completions endpoint.
OPENCODE_BIN	opencode	CLI binary used by opencode_run_command (legacy path only).
OPENCODE_PROMPT_DIR	./.opencode/commands	Where the .opencode/commands/inboxos:*.md prompts live.
PAPRA_API_TOKEN	unset	Bearer token for Papra uploads.
PAPRA_BASE_URL	unset	Papra API base.
PAPRA_BODY_WRITE_MODULES	crm,finance,legal	Comma-separated modules whose body (not attachments) gets uploaded.
PAPRA_DRY_RUN	true	false to enable real Papra writes (CEO-approved only).
PAPRA_METADATA_ONLY_MODULES	intel,promotions	Modules whose email is not written to Papra (metadata only).
PAPRA_ORGANIZATION_ID	unset	Papra org id.
SEAWEEDFS_S3_ACCESS_KEY	—	S3 access key.
SEAWEEDFS_S3_BUCKET	—	inboxos (default).
SEAWEEDFS_S3_ENDPOINT	—	https://storage.cloudinit.dev (or in-network http://seaweedfs-filer:8888).
SEAWEEDFS_S3_SECRET_KEY	—	S3 secret key.

---

9. Phase 5 acceptance gate

This appendix is the must-have checklist that closes InboxOS Phase 5 (Full Deployment — INB-7). It is the integrator's reference: every item must be checkbox-green before the .ciagent/inboxos/ROADMAP.md Phase 5 status row is flipped to "executed". The 20 must-haves below were produced by the CIAgent plan at .ciagent/inboxos/plans/05-PLAN.md and verified during the Wave 1 + Wave 2 + Wave 3 execution pass (see git log on milestone/v1.0-initial).

9.1 Requirements coverage

#	Must-have (verifiable)	Source-of-truth location	REQ-INB	Owner
1	pipelines/live_pipeline.py runs on a 5-minute cron and on workflow_dispatch	.gitea/workflows/live-pipeline.yml:62-65	REQ-INB-1	devops-engineer
2	pipelines/historical_pipeline.py is workflow_dispatch only and streams a local .mbox end-to-end	.gitea/workflows/historical-pipeline.yml:54-58	REQ-INB-2	devops-engineer
3	All NocoDB writes are GET-then-PATCH/POST on Message_ID; no plain INSERT path	pipelines/common.py:196-255	REQ-INB-3	data-engineer
4	inboxos:classify + inboxos:extract-<module> is the only extraction path	pipelines/common.py:632-653, docs/extraction-contract.md	REQ-INB-4	ml-engineer
5	Four module tables (finance, crm, intel, promotions) with the column set per docs/schemas.md	docs/schemas.md, docs/phase-1.1-verification/inb-28-report.md	REQ-INB-5	data-engineer
6	docs/operations-runbook.md is the canonical day-1 incident reference	this file	REQ-INB-6	lead-developer
7	Two self-hosted Gitea Actions runners are provisioned and online	docker/docker-compose.yml:103-148, docs/gitea-actions.md §2	REQ-INB-7	devops-engineer
8	ci-pr.yml runs fast unit tests + 100MB e2e + live Ollama smoke on every PR	.gitea/workflows/ci-pr.yml:63-230	REQ-INB-8	devops-engineer
9	Papra integration is dry-run by default; the first ship does not call the real Papra API	pipelines/common.py:729, docs/papra-write-policy.md	REQ-INB-9	security-engineer
10	No third-party SaaS touches PII; only the IMAP connection to Gmail leaves the self-hosted perimeter	docker/docker-compose.yml, this runbook §6	REQ-INB-10	security-engineer
11	A single pinned Ollama model (gemma4:e4b) is the production extraction path; no model-escalation chain	pipelines/common.py:467-468, .opencode/commands/inboxos:*.md frontmatter	REQ-INB-11	ml-engineer
12	actions/upload-artifact is pinned to v3 in the workflows that emit per-run logs (live + historical); ci-pr.yml has no upload-artifact step and inherits GHES compat from the runner image, not from the workflow pin	.gitea/workflows/live-pipeline.yml:150 (uses: actions/upload-artifact@v3), .gitea/workflows/historical-pipeline.yml:170 (same). ci-pr.yml has no upload-artifact step; the runbook's previous glob was misleading.	REQ-INB-12	devops-engineer
13	A synthetic UPSERT test exists and exercises the NocoDB idempotency path end-to-end	pipelines/_tests/test_upsert_synthetic.py	REQ-INB-13	data-engineer
14	All extraction prompts are version-controlled in .opencode/commands/ and invoked by name	.opencode/commands/inboxos:*.md (5 files), pipelines/common.py::llm_run_command	REQ-INB-14	ml-engineer

9.2 Reconciliation drift closure

The seven doc/code drifts surfaced in research (P-1 through P-7) were closed during the Wave 1 + Wave 2 execution pass:

#	Drift	Closed by	PR	REQ-INB
15	Model pin drift (lfm2.5:8b → gemma4:e4b in docs)	docs: align pipelines/README + extraction-contract to gemma4:e4b pin	#A	REQ-INB-11
16	OLLAMA_BASE_URL /v1 suffix canonicalised	docs: align pipelines/README + extraction-contract to gemma4:e4b pin (folded into #A)	#A	REQ-INB-11, REQ-INB-10
17	Papra INTAKE_EMAILS_IS_ENABLED flipped to false in compose	docker: disable Papra catch-all intake; align with cancelled policy	#E	REQ-INB-9
18	State-file field names in runbook §4.4 corrected	docs(runbook §4.4): correct state-file field names	#H	REQ-INB-3, REQ-INB-6
19	Live-vs-historical Papra_Link asymmetry documented	docs: document live-vs-historical Papra_Link asymmetry + docs(extraction-contract) — owner: backend-engineer (the P-7 papra_decide investigation; the asymmetry is structural, the doc is the deliverable)	#C, #B	REQ-INB-4, REQ-INB-9
20	Extraction-contract rewritten to describe Ollama as production LLM caller; opencode marked legacy	docs(extraction-contract): describe Ollama as production LLM caller; opencode marked legacy	#B	REQ-INB-4

The papra_decide behavioural asymmetry in live_pipeline.py (P-7 in the research brief) was investigated and found to be a no-op: the live pipeline already calls papra_decide exactly once with the post-classify modules list (pipelines/live_pipeline.py:490). The historical pipeline calls it twice (pre-classify attachment walk + post-classify body write), but the live pipeline has no pre-classify walk. Decision recorded as ED-1 in .ciagent/inboxos/CI-LOG.md.

9.3 Closing the gate

A Phase 5 close is the date on which all 20 must-haves are checkbox-green and the .ciagent/inboxos/ROADMAP.md Phase 5 status row is flipped to "executed". The lead-developer is the integrator; every active persona (data-engineer, backend-engineer, devops-engineer, ml-engineer, security-engineer, lead-developer) gets a review pass on this section.

9.4 v1.1 acceptance gate

This appendix is the must-have checklist that closes InboxOS v1.1 Phase 1. It is the integrator's reference: every item must be checkbox-green before the v1.1 ship merge. The 11 must-haves below were produced by the CIAgent plan at .ciagent/inboxos/plans/01-PLAN.md and verified during the Wave 1 + Wave 2 + Wave 3 execution pass (see git log on phase/01-v1.1-execute).

The 11 candidates are [CANDIDATE] until the CPO authors the matching entries in docs/requirements.md; the must-haves below reference the candidate IDs as a planning aid. The .ciagent/inboxos/ROADMAP.md Phase 1 status row is not flipped by the lead-developer; the flip is the CPO's call. The lead-developer's gate is "every must-have is checkbox- green and the 11 PRs are merged".

#	Must-have (verifiable)	Source-of-truth location	Owner	REQ-INB
1	.env.example lines 96-99 reference gemma4:e4b only (no lfm2.5:8b or gemma4:12b comments)	.env.example:96-99	devops-engineer	REQ-INB-15
2	Runbook §9.1 #12 cites upload-artifact@v3 only in the workflows that emit per-run logs (live + historical); ci-pr.yml excluded	docs/operations-runbook.md:544 (the row updated in T1.2)	devops-engineer	REQ-INB-16
3	Runbook §6.1 says "22 distinct secret values" (matching the 22 data rows in the table)	docs/operations-runbook.md:399 (the sentence updated in T1.3)	devops-engineer	REQ-INB-17
4	inb-28-report.md "Per-table row counts at freeze" is filled with one row per module table	docs/phase-1.1-verification/inb-28-report.md:67-77	data-engineer	REQ-INB-18
5	papra_index NocoDB table exists with the 7 documented columns; docs/schemas.md and scripts/apply_schemas.py reflect the table; the v1.0 schema freeze is preserved for the 4 module tables	docs/schemas.md (new papra_index section, T1.4a), scripts/apply_schemas.py (new AUDIT_TABLES entry, T1.4b), docs/phase-1.1-verification/inb-28-report.md (the freeze amendment, T1.4a)	data-engineer	REQ-INB-19
6	PAPRA_DRY_RUN_OVERRIDE env var exists; the new papra_write_gate event is emitted by both pipelines (via the T1.5 helper + T2.2 call-sites); the canary procedure is in docs/papra-write-policy.md; new test cases exercise the override and the helper	pipelines/common.py PapraConfig + emit_papra_write_gate_event (T1.5), pipelines/live_pipeline.py + pipelines/historical_pipeline.py (T2.2 call-sites), pipelines/_tests/test_papra_decide_offline.py (T1.5 + T2.2 cases), docs/papra-write-policy.md "Live-write gate" (T1.5)	security-engineer	REQ-INB-20
7	scripts/rotate_secrets.sh exists with --dry-run and real-run modes; pipelines/_tests/test_rotate_secrets_offline.py passes; docs/operations-runbook.md §6.2 documents the rotation ordering	scripts/rotate_secrets.sh (new), pipelines/_tests/test_rotate_secrets_offline.py (new, 3/3 tests pass), docs/operations-runbook.md §6.2 (new)	security-engineer	REQ-INB-21
8	scripts/backup.sh and scripts/restore.sh exist; pipelines/_tests/test_restore_offline.py round-trips a known fixture; docs/operations-runbook.md §5.7 + §10 state RTO < 4h and RPO < 24h	scripts/backup.sh (new), scripts/restore.sh (new), pipelines/_tests/test_restore_offline.py (new, 7/7 tests pass), docs/operations-runbook.md §5.7 + §10 (new)	devops-engineer	REQ-INB-22
9	pipelines/_metrics/emit.py writes one JSONL line per run with the documented counter and gauge vocabulary; docs/operations-runbook.md §4.5 documents the tail-and-summarize procedure; a new test exercises the emitter	pipelines/_metrics/emit.py (new), pipelines/live_pipeline.py + pipelines/historical_pipeline.py (T2.2 call-sites), pipelines/_tests/test_metrics_offline.py (new, 2/2 tests pass), docs/operations-runbook.md §4.5 (new)	data-engineer	REQ-INB-23
10	plans/05-PLAN.md parallelization table has the explicit three-file split footnote (P1-1 closure); runbook §9.2 row 19 has the backend-engineer persona annotation (P2-4 closure); CI-LOG P1-1 + P2-4 entries are marked closed	.ciagent/inboxos/plans/05-PLAN.md:318-330 (footnote), docs/operations-runbook.md:674 (row 19 owner annotation), .ciagent/inboxos/CI-LOG.md:96-97,112-113,115 (closed_in pointers)	lead-developer	REQ-INB-24
11	The live pipeline's process_one_message patches Papra_Link onto every NocoDB row after a successful papra_write_attachment; the "Live pipeline note" paragraph in docs/papra-write-policy.md §Rule 1 is removed; the asymmetry paragraph in docs/extraction-contract.md §Live vs historical: Papra_Link asymmetry is removed; ARCHITECTURE.md §Document Store is updated to reflect parity	pipelines/live_pipeline.py:486-555 (the new PATCH loop + papra_link_patched event, T1.7), docs/papra-write-policy.md:55-72 (the §Rule 1 asymmetry note removed by T2.3), docs/extraction-contract.md:155-181 (the subsection removed by T1.7), .ciagent/inboxos/ARCHITECTURE.md:65-72 (the paragraph replaced by T1.7)	backend-engineer	REQ-INB-25

A v1.1 close is the date on which all 11 must-haves are checkbox-green and the CPO has promoted the 11 candidates to real requirements in docs/requirements.md. The .ciagent/inboxos/ROADMAP.md Phase 1 status row is not flipped to "executed" by the lead-developer; the flip is the CPO's call. The lead-developer's gate is "every must-have is checkbox-green and the 11 PRs are merged".

---

10. Disaster recovery targets (REQ-INB-22)

The backup / restore procedure in §5.7 is the substrate; this section is the commitment.

• RTO (Recovery Time Objective): < 4 hours, single- operator restore. The breakdown:

  • 30 min — re-read §5.7 + §6.2 and decide which services to restore (typically: all three, in order: Gitea first so the workflows and secrets are back, then NocoDB so the module-table IDs are available, then SeaweedFS).
  • 60 min — restore the Gitea bundle and force-push the jc-capital/inboxos repo. Verify the workflows are visible at the Gitea UI.
  • 60 min — restore the NocoDB export. The export is a JSON dump; the operator opens the NocoDB UI and uses Settings → Import to load the JSON. Verify a known row via the curl ... where=(Message_ID,eq,...) pattern in §5.2.
  • 60 min — restore the SeaweedFS mirror. The aws s3 sync (or boto3 fallback) copies every object from the backup bucket to the primary bucket. Verify a known .eml via the s3_get_text_optional helper.
  • 30 min — buffer for unexpected drift (single-operator runs are not 24/7; if the restore starts at 02:00 UTC the operator is fresh by 04:00 UTC).

• RPO (Recovery Point Objective): < 24 hours data loss. The daily backup cron (§5.7) runs at 02:00 UTC, so the worst-case loss is the previous 24 hours of incoming email. In a typical single-operator inbox this is < 50 messages, all recoverable from Gmail via since_uid (the live pipeline's recovery path in §5.2).

The RTO and RPO values are explicit and CPO-acknowledged. The ideation report Tier 2 T2-C marked them as "guesses" pending CPO sign-off; the values recorded here are the v1.1 ship defaults. Changes to the values require a new CPO-approved REQ-INB (the values are not engineer-editable in isolation).

---

End of runbook. Next review: after the v1.3 ship (2026-06-08, milestone milestone/v1.3-initial), or when the next failure mode is added to §5. Last verified against the source files cited inline; any divergence is a bug in this document — fix the doc, do not just edit around it.