inboxos-v1/docs/extraction-contract.md

InboxOS — Extraction Contract (canonical)

Status: Phase 2 deliverable for INB-4 Scope: Single source of truth for how a raw email becomes a structured NocoDB record, end-to-end.

The contract, in one paragraph

For every email that the InboxOS pipelines process, the pipeline calls pipelines.common.llm_run_command with the ollama backend twice:

1. inboxos:classify — given the email as JSON on stdin, returns {"modules": ["finance", "crm", ...]} listing the modules that apply (or [] if none).
2. For each module in that list, inboxos:extract-<module> — given the same email as JSON on stdin, returns a flat record matching the module's NocoDB schema (see docs/schemas.md).

The pipeline then UPSERTs each module record into the corresponding NocoDB table, keyed on Message_ID for idempotency.

This is the only extraction path. Anything that writes to a NocoDB module table MUST go through this contract.

Prompt files

File	Role	Output shape
.opencode/commands/inboxos:classify.md	Route an email to modules	{"modules": [...]}
.opencode/commands/inboxos:extract-finance.md	Extract a finance record	{"Direction": ..., "Counterparty": ..., ...}
.opencode/commands/inboxos:extract-crm.md	Extract a crm record	{"Contact_Name": ..., "Contact_Email": ..., ...}
.opencode/commands/inboxos:extract-intel.md	Extract an intel record	{"Source": ..., "Topic": ..., "Items": [...], ...}
.opencode/commands/inboxos:extract-promotions.md	Extract a promotions record	{"Brand": ..., "Offer_Type": ..., "Worth_Saving": true, ...}

All five use model ollama/gemma4:e4b (confirmed available at https://ollama-frp.cloudinit.dev/api/tags). The canonical pin per REQ-INB-11 is gemma4:e4b; the prompt frontmatter holds the default model which the runtime can override via the INBOXOS_LLM_MODEL env var (see pipelines/common.py:545-546).

The full field contracts for each module are listed in docs/schemas.md. The prompts are the implementation; the schemas doc is the spec.

Model spec

The production extraction path uses a single pinned Ollama model: gemma4:e4b. This is the canonical pin per REQ-INB-11. There is no model-escalation chain in the production path.

Each prompt's frontmatter declares model: ollama/gemma4:e4b. The runtime reads INBOXOS_LLM_MODEL (default: gemma4:e4b) and uses that as the authoritative model; the prompt frontmatter is the fallback. The classifier and per-module extractors share the same model — no escalation, no per-prompt overrides in production.

This is a deliberate CTO choice (lens: "Cost efficiency — zero-cost inference on local Ollama") and is the smallest model that, in our limited testing, has been observed to follow the strict-JSON contract on the modules we route to.

How the pipelines invoke the prompts

pipelines/historical_pipeline.py (canonical reference)

CLASSIFY_COMMAND = os.environ.get("INBOXOS_CLASSIFY_COMMAND", "inboxos:classify")
EXTRACT_COMMAND_PREFIX = os.environ.get("INBOXOS_EXTRACT_COMMAND_PREFIX", "inboxos:extract-")

def classify(llm_cfg, summary, logger):
    result = llm_run_command(llm_cfg, CLASSIFY_COMMAND, summary, logger=logger)
    modules = result.get("modules") or []
    return [m.strip().lower() for m in modules if m.strip()]

def extract_for_module(llm_cfg, module, summary, papra_link, logger):
    command_name = f"{EXTRACT_COMMAND_PREFIX}{module}"
    result = llm_run_command(llm_cfg, command_name, summary, logger=logger)
    if not isinstance(result, dict):
        return None
    result["Message_ID"] = summary["message_id"]   # pipeline overwrites
    if papra_link:
        result["Papra_Link"] = papra_link
    return result

# in process_one_message:
modules = classify(llm_cfg, summary, logger)
for module in modules:
    table_id = module_tables.get(module)         # from NOCODB_MODULE_TABLES env
    if not table_id:
        log_event(logger, "module_table_missing", module=module, ...)
        continue
    extracted = extract_for_module(llm_cfg, module, summary, papra_link, logger)
    upsert_result = nocodb_upsert(nocodb_cfg, table_id, extracted, message_id)

LLMConfig is constructed from environment via LLMConfig.from_env() (see pipelines/common.py:492-585). The llm_run_command entry point dispatches to the configured backend — default ollama, with the prompt loaded from INBOXOS_PROMPT_DIR and parsed for frontmatter model: (see pipelines/common.py::llm_run_command and _ollama_call_one_model, lines 636-680).

The historical pipeline reads NOCODB_MODULE_TABLES from the environment — a JSON dict like {"finance":"tbl_xxx","crm":"tbl_yyy","intel":"tbl_zzz","promotions":"tbl_www"}. The apply_schemas.py bootstrap script prints this value after creating the tables.

pipelines/live_pipeline.py (aligned)

live_pipeline.py is wired to the canonical contract: it reads NOCODB_MODULE_TABLES (JSON dict), calls inboxos:classify once per message, then invokes inboxos:extract-<module> for each matched module. The two pipelines share the same env var and the same prompt family — there is no second contract.

Why "classify first, then extract" (not "one big extract all")

• Cost. A receipt email doesn't need to invoke the CRM, intel, and promotions extractors. Routing first lets us skip the irrelevant calls.
• Isolation. A bad prompt for one module doesn't fail the whole email. The historical pipeline logs extract_permanent_failure for the one module and continues.
• Observability. Per-module logs make it obvious which extractor is producing low-quality data.
• Prompt size. Each per-module prompt is small and focused, which gives the small local model (4B effective params) a better chance of following the strict-JSON contract.

A single "extract all" call would be cheaper to invoke (one round trip) but more expensive to operate (always pays the full LLM cost) and harder to debug. The CTO call is "classify first".

Prompt-as-code (governance)

Per CTO lens ("Prompt as code — extraction prompts are version-controlled in Gitea, not embedded in n8n workflows"):

• Prompts are .md files in .opencode/commands/. No prompt is embedded in a workflow .yml or a Python string.
• Changes go through git: open a PR, get a review, merge. Bad changes are reverted by reverting the commit. UPSERT semantics mean a bad row is overwritten by the next run with the reverted prompt, so no data is permanently corrupted.
• The strict-JSON contract is enforced by the prompt's hard-rules section. The Python wrapper llm_run_command / _ollama_call_one_model in pipelines/common.py (lines 605-624) parses the model's reply with json.JSONDecoder.raw_decode and fails loudly if the first non-whitespace token is not valid JSON. There is no "best effort" path.

Legacy: opencode CLI backend

INBOXOS_LLM_BACKEND=opencode is kept for offline tests only. The opencode binary install step in historical-pipeline.yml is retained for any human-driven opencode invocation from the repo worktree; the pipeline itself does not shell out to it. The opencode backend in pipelines/common.py:440-487 is documented as broken on opencode 1.16.2 and is not exercised in production or in CI. Use the ollama backend (the default) for production.

See docs/operations-runbook.md §6.3 for the data-sovereignty posture (local LLM only).

Failure semantics

Failure	What the pipeline does	Operator action
Ollama classify returns invalid JSON or modules: []	.eml is dead-lettered to eml/dead-letter/<mid>.json with the failure reason; cursor advances.	Inspect the prompt output; fix the prompt.
Ollama extract returns invalid JSON for one module	Run logs extract_permanent_failure for that module, stops without advancing offset, retries next run.	Inspect the prompt output; fix the prompt.
Ollama extract returns a record that does not match the table	NocoDB rejects the row (column error). Run stops; offset does not advance.	Add the column to the schema (apply_schemas.py), retry.
NocoDB is down / 5xx	Tenacity retries with exponential backoff. Permanent failure stops the run.	Restore NocoDB; re-run.

PII masking (v1.4 REQ-INB-57)

Before the email body is sent to Ollama, pipelines/pii_scrub.py::scrub_pii() applies 4 regex-based maskers in sequence:

1. SSN: \b\d{3}-\d{2}-\d{4}\b → [SSN-MASKED]
2. Credit card (Luhn-validated): \b(?:\d[ -]*?){13,19}\b where the digit sequence passes the Luhn check → [CC-MASKED]. Non-CC digit sequences (e.g., order numbers) are not masked.
3. IBAN: per-country regex ([A-Z]{2}\d{2}[A-Z0-9]{1,30}) → [IBAN-MASKED]
4. Phone (E.164): \+[1-9]\d{1,14}\b → [PHONE-MASKED]

The masked body is what Ollama sees. The raw body is never sent to the LLM.

The audit trail is per-message: a PIIMaskSummary dataclass records the count of each PII type masked, and the summary is emitted to _artifacts/pii_scrub.jsonl (operator-local state, .gitignore-d).

The masking is conservative: false positives are preferred over false negatives. The Luhn check on credit cards ensures non-CC digit sequences are not masked. The CPO can amend the masking policy (which PII types to mask, which to preserve for extraction) at any time via a follow-up PR without changing the LLM contract.