inboxos-v1/docs/papra-write-policy.md

Papra write policy (INB-27)

This document describes how the InboxOS live and historical pipelines decide which emails get written to Papra for long-term storage and searchability. The implementation lives in pipelines/common.py (PapraConfig, papra_decide, papra_write_attachment, papra_write_body) and is exercised by pipelines/_tests/test_papra_decide_offline.py.

Why a policy, not a blanket copy

The earlier INTAKE_EMAILS_DRIVER=catch-all integration tried to copy every email into Papra. That approach was cancelled by the CEO on 2026-06-08 (see INB-27 thread, comment 563d0801):

"Lets cancel this integration, I see little value in it. Instead, as a separate effort, I want to see for certain documents, as they are ingested by the live/historical pipelines, a certain set of relevant document are written directly to papra for long term storage and searcheability. certains emails where the metadate extracted is enough don't need the email replicated in papra."

The new policy makes the selection explicit, deterministic, and operator-tunable. As a corollary, the docker/docker-compose.yml papra service no longer advertises the INTAKE_EMAILS_* env vars; Papra writes are pipeline-driven only.

The decision, in one diagram

                    ┌────────────────────────┐
                    │  Email arrives in      │
                    │  process_one_message   │
                    └─────────────┬──────────┘
                                  │
                                  ▼
                    ┌────────────────────────┐
                    │ papra_decide()          │
                    │  pure function         │
                    └─────────────┬──────────┘
                                  │
       ┌──────────────┬───────────┴────────────┬────────────────┐
       │              │                         │                │
       ▼              ▼                         ▼                ▼
attachment_present  body_module      metadata_only_module  no_matching_module
       │              │                         │                │
   write each     write the body              skip             skip
   attachment     (capped 64 KB)            (NocoDB only)   (NocoDB only)

Every path emits a papra_decision structured log event so the audit trail is complete.

Rule details

Rule 1 — attachments present (always write)

If the RFC-822 message contains one or more MIME parts with Content-Disposition: attachment and a non-empty decoded payload, each qualifying attachment is uploaded to Papra. The email body itself is not written alongside — the attachment is the document of record.

Inline images and signature blobs are skipped (they do not have Content-Disposition: attachment).

Each attachment is uploaded to POST /api/organizations/{org}/documents with a tags field of ["inboxos", "message_id:<hash12>"]. The first attachment's resulting document id is stored on the NocoDB row as Papra_Link. Both pipelines now capture Papra_Link on the NocoDB row whenever the attachment walk returns a papra_link — see ARCHITECTURE.md §Document Store (curated) and the papra_link_patched event in pipelines/live_pipeline.py.

Rule 2 — body is the artifact (module-driven)

If there is no attachment AND the classifier routed the email to a module in PAPRA_BODY_WRITE_MODULES, the email body is uploaded to Papra as a single document:

• HTML bodies are stripped to plain text (the same _strip_html_to_text algorithm used in the live pipeline prompt builder).
• The body is capped at 64 KB to keep the upload bounded.
• The resulting document's filename is <message-id>.txt and its content-type is text/plain; charset=utf-8.

The module used is the first one in classifier-output order that appears in the allow-list. The classifier is the source of truth for which module is primary; the env list is just a membership filter.

Rule 3 — metadata is enough (skip Papra, keep NocoDB)

If there is no attachment AND the classifier routed the email to a module in PAPRA_METADATA_ONLY_MODULES, nothing is written to Papra. The structured fields in NocoDB are sufficient; we don't need the original document body in Papra.

Rule 4 — default (skip)

If none of the above match (no attachment, no module in the body-write list, no module in the metadata-only list), nothing is written to Papra.

Configuration (env vars)

Var	Default	Meaning
PAPRA_BASE_URL	unset	The Papra server root. When unset, the helper silently skips.
PAPRA_API_TOKEN	unset	Bearer token for POST /api/.../documents.
PAPRA_ORGANIZATION_ID	unset	Org id appended to the upload URL.
PAPRA_DRY_RUN	true	When true, no HTTP request is made; the helper logs papra_write_dry_run and returns a synthetic id. The default for the first ship.
PAPRA_BODY_WRITE_MODULES	crm,finance,legal	Comma-separated module list that triggers a body write (Rule 2).
PAPRA_METADATA_ONLY_MODULES	intel,promotions	Comma-separated module list that skips Papra (Rule 3).
PAPRA_HTTP_TIMEOUT	60	Seconds for one Papra POST.

Flipping the dry-run switch

1. Run with PAPRA_DRY_RUN=true (default). Inspect the live-pipeline-log and historical-pipeline-log artifacts; every message has a papra_decision event with would_write, reason, and (for writes) target_count and the synthetic id.
2. Confirm the decisions match expectations (no surprise writes, no missed attachments).
3. Set the repo var PAPRA_DRY_RUN=false (in Gitea → repo → Settings → Actions → Variables, or as a secret). Re-run.
4. Verify the new papra_write_ok events include real papra_document_id values and that the documents are searchable in the Papra web UI.

Live-write gate (REQ-INB-20)

The Papra live-write switch is guarded by an engineering gate so the first real write is observable end-to-end before the repo-wide PAPRA_DRY_RUN flip is finalised.

Override precedence. When the live and historical pipelines boot, they parse the dry-run state in this order (highest precedence first):

1. PAPRA_DRY_RUN_OVERRIDE — a per-message kill switch. Truthy values (true / 1 / yes / on) force dry-run on; falsy values (false / 0 / no / off) force dry-run off. Empty string means "no override".
2. PAPRA_DRY_RUN — the repo var (the existing ship-default true lives here).
3. The hard-coded default true (the first-ship posture).

The effective value is stored on PapraConfig.dry_run; the override is preserved on PapraConfig.dry_run_override for the papra_write_gate log event.

Canary procedure. To audit the gate before the real-write flip:

1. Dispatch live-pipeline.yml with max_emails=1 and PAPRA_DRY_RUN_OVERRIDE=false; inspect the per-run log for a papra_write_gate event followed by papra_write_ok / papra_write_failed.
2. Repeat for max_emails=10 and max_emails=100.
3. Confirm the papra_index audit table (REQ-INB-19) has one row per processed message; the row's outcome is ok for every successful write and failed for every error.
4. Set the repo var PAPRA_DRY_RUN=false (in Gitea → repo → Settings → Actions → Variables) and re-run; the papra_dry_run_env field in papra_write_gate will be false.
5. Inspect the Papra web UI for the new documents; verify the papra_document_id is non-null on the matching papra_index rows.

Test coverage. pipelines/_tests/test_papra_decide_offline.py gains two cases (added in the same commit that landed this subsection):

• test_papra_dry_run_override_wins — asserts the precedence table; the override beats the repo var, and an empty override falls back to the repo var.
• test_emit_papra_write_gate_event_logs_three_fields — asserts the helper emits exactly one papra_write_gate event with the documented dry_run, dry_run_override, and papra_dry_run_env fields.

Failure handling

Papra failures are non-fatal:

• A transient HTTP error is retried by tenacity (3 attempts, exponential backoff).
• A permanent failure logs papra_write_failed and the pipeline continues. The NocoDB row is not updated with Papra_Status=pending in this child; that column arrives in INB-27.b alongside the papra_index NocoDB table.
• A Papra outage does not stall a multi-hour historical run; the .eml is still in SeaweedFS and a re-run after the outage is over will retry the upload.

Tests

pipelines/_tests/test_papra_decide_offline.py covers 15 cases:

• attachment collection (inline images skipped, real PDFs found)
• the four rule outcomes (attachment, body, metadata-only, default)
• HTML body stripping
• classifier-order precedence in the body-write rule
• dry-run mode (no HTTP)
• env parsing for PAPRA_DRY_RUN and the module lists
• is_configured() returns False when any of base_url / token / org is empty

All 15 cases pass on the test environment.