inboxos-v1/docs/extraction-contract.md

# InboxOS — Extraction Contract (canonical)

**Status:** Phase 2 deliverable for [INB-4](/INB/issues/INB-4)
**Scope:** Single source of truth for how a raw email becomes a
structured NocoDB record, end-to-end.

## The contract, in one paragraph

For every email that the InboxOS pipelines process, the pipeline
calls `pipelines.common.llm_run_command` with the `ollama` backend
**twice**:

1. `inboxos:classify` — given the email as JSON on stdin, returns
   `{"modules": ["finance", "crm", ...]}` listing the modules that
   apply (or `[]` if none).
2. For each module in that list, `inboxos:extract-<module>` — given
   the same email as JSON on stdin, returns a flat record matching
   the module's NocoDB schema (see `docs/schemas.md`).

The pipeline then UPSERTs each module record into the corresponding
NocoDB table, keyed on `Message_ID` for idempotency.

This is the only extraction path. Anything that writes to a NocoDB
module table MUST go through this contract.

## Prompt files

| File | Role | Output shape |
| --- | --- | --- |
| `.opencode/commands/inboxos:classify.md` | Route an email to modules | `{"modules": [...]}` |
| `.opencode/commands/inboxos:extract-finance.md` | Extract a `finance` record | `{"Direction": ..., "Counterparty": ..., ...}` |
| `.opencode/commands/inboxos:extract-crm.md` | Extract a `crm` record | `{"Contact_Name": ..., "Contact_Email": ..., ...}` |
| `.opencode/commands/inboxos:extract-intel.md` | Extract an `intel` record | `{"Source": ..., "Topic": ..., "Items": [...], ...}` |
| `.opencode/commands/inboxos:extract-promotions.md` | Extract a `promotions` record | `{"Brand": ..., "Offer_Type": ..., "Worth_Saving": true, ...}` |

All five use model `ollama/gemma4:e4b` (confirmed available at
`https://ollama-frp.cloudinit.dev/api/tags`). The canonical pin per
REQ-INB-11 is `gemma4:e4b`; the prompt frontmatter holds the
*default* model which the runtime can override via the
`INBOXOS_LLM_MODEL` env var (see `pipelines/common.py:545-546`).

The full field contracts for each module are listed in
`docs/schemas.md`. The prompts are the implementation; the schemas
doc is the spec.

## Model spec

The production extraction path uses a single pinned Ollama model:
`gemma4:e4b`. This is the canonical pin per REQ-INB-11. There is no
model-escalation chain in the production path.

Each prompt's frontmatter declares `model: ollama/gemma4:e4b`. The
runtime reads `INBOXOS_LLM_MODEL` (default: `gemma4:e4b`) and uses
that as the authoritative model; the prompt frontmatter is the
fallback. The classifier and per-module extractors share the same
model — no escalation, no per-prompt overrides in production.

This is a deliberate CTO choice (lens: "Cost efficiency — zero-cost
inference on local Ollama") and is the smallest model that, in our
limited testing, has been observed to follow the strict-JSON
contract on the modules we route to.

## How the pipelines invoke the prompts

### `pipelines/historical_pipeline.py` (canonical reference)

```python
CLASSIFY_COMMAND = os.environ.get("INBOXOS_CLASSIFY_COMMAND", "inboxos:classify")
EXTRACT_COMMAND_PREFIX = os.environ.get("INBOXOS_EXTRACT_COMMAND_PREFIX", "inboxos:extract-")

def classify(llm_cfg, summary, logger):
    result = llm_run_command(llm_cfg, CLASSIFY_COMMAND, summary, logger=logger)
    modules = result.get("modules") or []
    return [m.strip().lower() for m in modules if m.strip()]

def extract_for_module(llm_cfg, module, summary, papra_link, logger):
    command_name = f"{EXTRACT_COMMAND_PREFIX}{module}"
    result = llm_run_command(llm_cfg, command_name, summary, logger=logger)
    if not isinstance(result, dict):
        return None
    result["Message_ID"] = summary["message_id"]   # pipeline overwrites
    if papra_link:
        result["Papra_Link"] = papra_link
    return result

# in process_one_message:
modules = classify(llm_cfg, summary, logger)
for module in modules:
    table_id = module_tables.get(module)         # from NOCODB_MODULE_TABLES env
    if not table_id:
        log_event(logger, "module_table_missing", module=module, ...)
        continue
    extracted = extract_for_module(llm_cfg, module, summary, papra_link, logger)
    upsert_result = nocodb_upsert(nocodb_cfg, table_id, extracted, message_id)
```

`LLMConfig` is constructed from environment via
`LLMConfig.from_env()` (see `pipelines/common.py:492-585`). The
`llm_run_command` entry point dispatches to the configured backend —
default `ollama`, with the prompt loaded from `INBOXOS_PROMPT_DIR`
and parsed for frontmatter `model:` (see
`pipelines/common.py::llm_run_command` and `_ollama_call_one_model`,
lines 636-680).

The historical pipeline reads `NOCODB_MODULE_TABLES` from the
environment — a JSON dict like
`{"finance":"tbl_xxx","crm":"tbl_yyy","intel":"tbl_zzz","promotions":"tbl_www"}`.
The `apply_schemas.py` bootstrap script prints this value after
creating the tables.

### `pipelines/live_pipeline.py` (aligned)

`live_pipeline.py` is wired to the canonical contract: it reads
`NOCODB_MODULE_TABLES` (JSON dict), calls `inboxos:classify` once
per message, then invokes `inboxos:extract-<module>` for each
matched module. The two pipelines share the same env var and the
same prompt family — there is no second contract.

## Why "classify first, then extract" (not "one big extract all")

- **Cost.** A receipt email doesn't need to invoke the CRM, intel,
  and promotions extractors. Routing first lets us skip the
  irrelevant calls.
- **Isolation.** A bad prompt for one module doesn't fail the whole
  email. The historical pipeline logs `extract_permanent_failure` for
  the one module and continues.
- **Observability.** Per-module logs make it obvious which extractor
  is producing low-quality data.
- **Prompt size.** Each per-module prompt is small and focused,
  which gives the small local model (4B effective params) a better
  chance of following the strict-JSON contract.

A single "extract all" call would be cheaper to invoke (one round
trip) but more expensive to operate (always pays the full LLM cost)
and harder to debug. The CTO call is "classify first".

## Prompt-as-code (governance)

Per CTO lens ("Prompt as code — extraction prompts are
version-controlled in Gitea, not embedded in n8n workflows"):

- Prompts are `.md` files in `.opencode/commands/`. No prompt is
  embedded in a workflow `.yml` or a Python string.
- Changes go through git: open a PR, get a review, merge. Bad
  changes are reverted by reverting the commit. UPSERT semantics
  mean a bad row is overwritten by the next run with the reverted
  prompt, so no data is permanently corrupted.
- The strict-JSON contract is enforced by the prompt's hard-rules
  section. The Python wrapper `llm_run_command` /
  `_ollama_call_one_model` in `pipelines/common.py` (lines 605-624)
  parses the model's reply with `json.JSONDecoder.raw_decode` and
  fails loudly if the first non-whitespace token is not valid JSON.
  There is no "best effort" path.

## Legacy: opencode CLI backend

`INBOXOS_LLM_BACKEND=opencode` is **kept for offline tests only**.
The `opencode` binary install step in `historical-pipeline.yml` is
retained for any human-driven `opencode` invocation from the repo
worktree; the pipeline itself does not shell out to it. The
`opencode` backend in `pipelines/common.py:440-487` is documented
as broken on opencode 1.16.2 and is not exercised in production or
in CI. Use the `ollama` backend (the default) for production.

See `docs/operations-runbook.md §6.3` for the data-sovereignty
posture (local LLM only).

## Failure semantics

| Failure | What the pipeline does | Operator action |
| --- | --- | --- |
| Ollama classify returns invalid JSON or `modules: []` | `.eml` is dead-lettered to `eml/dead-letter/<mid>.json` with the failure reason; cursor advances. | Inspect the prompt output; fix the prompt. |
| Ollama extract returns invalid JSON for one module | Run logs `extract_permanent_failure` for that module, stops without advancing offset, retries next run. | Inspect the prompt output; fix the prompt. |
| Ollama extract returns a record that does not match the table | NocoDB rejects the row (column error). Run stops; offset does not advance. | Add the column to the schema (`apply_schemas.py`), retry. |
| NocoDB is down / 5xx | Tenacity retries with exponential backoff. Permanent failure stops the run. | Restore NocoDB; re-run. |

## PII masking (v1.4 REQ-INB-57)

Before the email body is sent to Ollama, `pipelines/pii_scrub.py::scrub_pii()` applies 4 regex-based maskers in sequence:

1. **SSN**: `\b\d{3}-\d{2}-\d{4}\b` → `[SSN-MASKED]`
2. **Credit card** (Luhn-validated): `\b(?:\d[ -]*?){13,19}\b` where the digit sequence passes the Luhn check → `[CC-MASKED]`. Non-CC digit sequences (e.g., order numbers) are not masked.
3. **IBAN**: per-country regex (`[A-Z]{2}\d{2}[A-Z0-9]{1,30}`) → `[IBAN-MASKED]`
4. **Phone (E.164)**: `\+[1-9]\d{1,14}\b` → `[PHONE-MASKED]`

The masked body is what Ollama sees. The raw body is never sent to the LLM.

The audit trail is per-message: a `PIIMaskSummary` dataclass records the count of each PII type masked, and the summary is emitted to `_artifacts/pii_scrub.jsonl` (operator-local state, `.gitignore`-d).

The masking is conservative: false positives are preferred over false negatives. The Luhn check on credit cards ensures non-CC digit sequences are not masked. The CPO can amend the masking policy (which PII types to mask, which to preserve for extraction) at any time via a follow-up PR without changing the LLM contract.