Fixes main-branch regression where PapraConfig.__init__ gained a new
keyword-only parameter dry_run_override but the two offline test fixtures
did not pass it, causing TypeError on 4/14 live_pipeline tests and
parallel failures in historical_pipeline tests.
- _make_papra_cfg in test_live_pipeline_offline.py: add dry_run_override=dry_run
- _make_papra_cfg in test_historical_pipeline_offline.py: add dry_run_override=dry_run
All PapraConfig callsites in pipelines/_tests now include dry_run_override.
Live offline tests: 14/14 passed.
Refs [INB-331](/INB/issues/INB-331)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
CPO-APPROVAL-REQUIRED: the masking policy is CPO-owned. The proposed
policy in v1.4-candidates.md §3.2 T1-SEC-3 is adopted as the default:
mask SSN, credit-card (Luhn-validated), IBAN, phone (E.164). The CPO
can amend via a follow-up PR at any time.
---ci---
project: inboxos
phase: 1
milestone: v1.4
status: execute
wave: 3
task: T3
cpo_approval_required: true
cpo_approval_status: proposed_policy_adopted_as_default
---/ci---
The papra_index_upsert helper calls nocodb_upsert positionally
(cfg, table_id, record, message_id). The test was asserting on
kwargs['table_id'] which is empty because all four args are
passed positionally. Switch the assertion to positional args.
---ci---
project: inboxos
phase: 1
milestone: v1.1
status: execute
---/ci---
REQ-INB-22 lands the backup + restore helper and the runbook RTO/RPO.
scripts/backup.sh (new, ~120 lines, set -euo pipefail):
- Accepts a positional argument: nocodb | seaweedfs | gitea.
- nocodb: POST /api/v2/meta/bases/<id>/export to a timestamped
file under backups/nocodb/<ts>/nocodb-export.json.
- seaweedfs: requires SEAWEEDFS_S3_BACKUP_BUCKET and
SEAWEEDFS_S3_BACKUP_ENDPOINT. Mirrors the inboxos bucket to
the backup bucket via 'aws s3 sync' (or boto3 fallback).
Errors with exit code 4 + a clear 'no backup target
configured' message when the env vars are unset.
- gitea: 'git bundle create --all' of the cto-inboxos/inboxos
repo, or 'git clone' + 'git bundle create' fallback.
- Prints the documented one-line summary:
backup_id=<ts> service=<svc> bytes=<int> target=<path>.
scripts/restore.sh (new, ~100 lines, set -euo pipefail):
- Accepts two positional arguments: <backup_id> <service>.
- Verifies the backup directory layout: backups/<svc>/<id>/.
- For nocodb: prints the import instructions (NocoDB has no
server-side import on the free / self-hosted plan; the
operator uses the NocoDB UI to upload the JSON).
- For seaweedfs: reverse-sync from the backup bucket to the
primary bucket via 'aws s3 sync' (or boto3 fallback).
- For gitea: clones the bundle to a temp dir, force-pushes
via the Gitea REST API.
- Prints the documented one-line summary:
restored backup_id=<id> service=<svc> verified=<bool>.
pipelines/_tests/test_restore_offline.py (new, 7 tests):
- All 7 tests pass on the offline test host. The git-bundle
round-trip test pins the integration shape that the
operator's host will exercise end-to-end.
docs/operations-runbook.md:
- New §5.7 'Backup and restore' after §5.6 'Historical pipeline
run cancelled mid-way'. Documents the daily cron, the
retention (7 daily + 4 weekly + 0 monthly), the copy-paste
invocation examples, and the test-or-die gate.
- New §10 'Disaster recovery targets' after §9.3 'Closing the
gate'. States RTO < 4 hours and RPO < 24 hours with the
time-breakdown rationale. Marks the values as explicit and
CPO-acknowledged (the ideation report Tier 2 T2-C flagged
them as 'guesses' pending CPO sign-off; the values recorded
here are the v1.1 ship defaults).
- §8.2 'Cross-references' gains the §5.7 + §10 cross-
reference bullet, removing the trailing 'added by T3.1
itself' note (the bullet is now present and the list is
internally consistent).
scripts/apply_schemas.py / pipelines/_tests/test_papra_index_offline.py
are not touched (the schema freeze is preserved). The
PAPRA_INDEX_TABLE env var is read by the helper but is not
required by the backup / restore procedure; the Gitea
secrets for the new backup target bucket are the only new
env vars.
---ci---
project: inboxos
phase: 1
milestone: v1.1
status: execute
---/ci---
T2.2 closes the call-site wiring for the T1.5 Papra live-write gate
and lands the per-run metrics.jsonl surface (REQ-INB-23). The
T1.5 + T2.2 work is folded into one commit so the two pipeline
files are touched by exactly one Wave 2 task.
pipelines/common.py:
- LLM_METRICS_ENABLED env-var gate around the
llm_call_seconds_sum / llm_call_count accumulators in
_ollama_call_one_model. The accumulators are stored in
os.environ so the main() functions can read them when emitting
the per-run metrics line. Default is 'true' (gate enabled).
pipelines/live_pipeline.py:
- Import emit_papra_write_gate_event and emit_metric.
- process_one_message: emit the papra_write_gate event at the
start of the papra branch (REQ-INB-20 call-site).
- main: emit one metrics line with the COUNTER_KEYS / GAUGE_KEYS
vocabulary from pipelines/_metrics/emit.py, including
llm_call_seconds_sum and llm_call_count read from the
os.environ accumulators.
pipelines/historical_pipeline.py:
- Import emit_papra_write_gate_event and emit_metric.
- process_one_message: emit the papra_write_gate event at the
start of the body-decision branch.
- main: emit one metrics line with the same vocabulary.
pipelines/_metrics/emit.py (new):
- emit_metric(run_id, pipeline, *, counters, gauges) appends one
JSONL line to <pipeline>-metrics-<run_id>.jsonl. The
COUNTER_KEYS and GAUGE_KEYS module constants pin the vocabulary
so the test can assert exact equality.
pipelines/_metrics/__init__.py (new):
- Empty / near-empty package marker.
pipelines/_tests/test_metrics_offline.py (new, 2 tests):
- test_emit_metric_creates_file_with_one_line_per_call: asserts
the line shape, the exact COUNTER_KEYS / GAUGE_KEYS, and the
per-run file naming.
- test_emit_metric_does_not_overwrite_existing_file: asserts two
calls in a row append, never truncate.
docs/operations-runbook.md:
- New §4.5 'Metrics surface' (after §4.4 'SeaweedFS bucket
layout' at line 238). Documents the file location, the counter
and gauge vocabulary, the tail-and-summarize procedure, the
opt-out via LLM_METRICS_ENABLED, and a forward-pointer noting
alerting is a v1.2 candidate (out of scope for v1.1).
---ci---
project: inboxos
phase: 1
milestone: v1.1
status: execute
---/ci---
The historical pipeline captures Papra_Link during the pre-classify
attachment walk and writes it onto the NocoDB row at extract time.
The live pipeline previously emitted the papra_decision log event
but did not PATCH the link onto the row, creating a documented
asymmetry. This commit closes it.
pipelines/live_pipeline.py:
- Capture the first returned id from papra_write_attachment as
papra_link (parallel to the historical pipeline's
pre-classify walk).
- After the per-module upsert loop, iterate over the collected
upserts and PATCH Papra_Link onto each module row that was
upserted. Idempotent: a re-run with the same papra_link is
a no-op on the column.
- Emit a structured papra_link_patched event with the message_id,
the papra_link, and the list of modules_patched. Failures are
per-row papra_link_patch_failed events; the pipeline continues.
- The papra_write_body branch also captures its single id into
papra_link (defensive; body uploads go to one module, but
the parity patch is harmless if there are 0 upserts).
docs/extraction-contract.md:
- §'Live vs historical: Papra_Link asymmetry' subsection removed.
The asymmetry is closed; the cross-references to ARCHITECTURE.md
and papra-write-policy.md are updated by T2.3.
ARCHITECTURE.md:
- §'Document Store (curated)' 'Live vs historical asymmetry'
paragraph replaced with a single sentence: 'Both pipelines
capture Papra_Link on the NocoDB row whenever the attachment
walk returns a papra_link.' with a forward pointer to
docs/extraction-contract.md.
pipelines/_tests/test_papra_link_parity_offline.py (new, ~1 test):
- test_papra_link_patched_on_live_after_attachment_write
asserts the parity patch: 2 module rows + 2 Papra_Link patches,
one papra_link_patched event listing both modules.
The papra_decide function in pipelines/common.py is unchanged;
the patch is a caller-side change in live_pipeline.py only.
docs/papra-write-policy.md §Rule 1 'Live pipeline note' removal
is deferred to T2.3 (Wave 2) per the plan.
---ci---
project: inboxos
phase: 1
milestone: v1.1
status: execute
---/ci---
REQ-INB-21 lands the secret-rotation helper and the runbook procedure.
scripts/rotate_secrets.sh (new, ~150 lines, set -euo pipefail):
- Accepts --dry-run (default) and --apply flags plus a positional
secret name.
- Reads the new value from STDIN (never a CLI arg) to avoid ps leaks.
- Calls pipelines.common.gitea_request via a python3 subshell to PUT
the new value to Gitea's actions-secrets endpoint. The Gitea
actions-secrets PUT is in-place (overwrites the existing secret
at the same name); the helper then verifies via GET that the
secret still exists under the new value.
- Exits 0 on success, 2 on missing env / missing value, 3 on Gitea
API failure, 4 on helper-internal error.
- Dry-run prints a redacted diff (sha256:8hex) of the new value
against an unknown placeholder for the old value (Gitea does
not return secret plaintexts via the API).
pipelines/_tests/test_rotate_secrets_offline.py (new, ~3 tests):
- test_dry_run_prints_diff_and_exits_zero_no_put
- test_apply_branch_runs_python_helper
- test_helper_fails_loud_when_env_unset
The 3 tests pass on the offline test host (the apply branch test
documents that exit code 0 or 3 is acceptable in this environment:
0 on a real Gitea, 3 on connection failure with no Gitea reachable).
docs/operations-runbook.md:
- §6.1 line 401 'future enhancement' parenthetical replaced with
'The helper landed in v1.1 (REQ-INB-21); see §6.4.'
- New §6.4 'Secret rotation' between §6.1 and §6.2: rotation
ordering (NOCODB_API_TOKEN -> SEAWEEDFS_* -> IMAP_PASSWORD ->
GITEA_USER/PASS), test-or-die gate, and a copy-paste invocation
example.
.env.example is not touched (env vars are unchanged; only the
helper and the procedure are new).
---ci---
project: inboxos
phase: 1
milestone: v1.1
status: execute
---/ci---
REQ-INB-20 introduces a Papra live-write engineering gate: a per-message
override that wins over the existing repo var, plus a canary procedure
and a structured papra_write_gate event that carries the effective
state.
pipelines/common.py:
- PapraConfig gains a dry_run_override: bool | None field parsed
from PAPRA_DRY_RUN_OVERRIDE (true/1/yes/on -> True; false/0/no/off
-> False; empty string -> None). The effective dry_run is
dry_run_override if not None else <existing logic at the repo
var PAPRA_DRY_RUN>.
- New emit_papra_write_gate_event(cfg, logger) helper (after the
PapraConfig block). Emits a single papra_write_gate event with
the three documented fields (dry_run, dry_run_override,
papra_dry_run_env). No side effects.
pipelines/_tests/test_papra_decide_offline.py:
- _default_cfg now sets dry_run_override=None.
- test_papra_config_unconfigured_when_env_empty also clears
PAPRA_DRY_RUN_OVERRIDE and asserts dry_run_override is None.
- New test_papra_dry_run_override_wins: three cases pinning the
precedence (override=true beats repo=false; override=false
beats repo=true; empty override -> repo wins, override is None).
- New test_emit_papra_write_gate_event_logs_three_fields: asserts
the helper emits exactly one papra_write_gate event with the
three documented fields.
docs/papra-write-policy.md:
- New 'Live-write gate' subsection after 'Flipping the dry-run
switch' (line 134). Documents the override precedence table,
the canary procedure (max_emails=1, 10, 100), and the test
coverage.
- The §'Flipping the dry-run switch' subsection is unchanged.
The T1.5 helper is callable but uncalled at the end of this commit;
the call-sites in live_pipeline.py and historical_pipeline.py are
landed in T2.2 alongside the metrics call-sites so the two pipeline
files are touched by exactly one Wave 2 task.
---ci---
project: inboxos
phase: 1
milestone: v1.1
status: execute
---/ci---
REQ-INB-19 introduces a new NocoDB table papra_index as the per-message
audit trail for every Papra write intent. The helper wraps
nocodb_upsert and:
- reads the target table id from NOCODB_PAPRA_INDEX_TABLE (set by
scripts/apply_schemas.py output, identical shape to
NOCODB_MODULE_TABLES);
- emits a 'papra_index_event' structured log entry with all six
fields (message_id, papra_document_id, module, kind, decision_at,
outcome, error);
- is a no-op when the env var is unset (emits
papra_index_upsert_skipped so the missing-config state is
observable in the run log);
- never raises: an exception in nocodb_upsert is caught and
logged as papra_index_upsert_failed; the helper returns {} so
the pipeline is not interrupted.
A new offline test pipelines/_tests/test_papra_index_offline.py
exercises the happy path, the no-config skip path, and the
exception path. The test stubs common.nocodb_upsert so it runs
without network access; it follows the project's _tests/ style
(plain functions, a _run() driver, exit 0/1).
The 4 pre-existing module tables (finance, crm, intel, promotions)
are not modified by this task. The papra_index_upsert helper is
the only new common.py addition; it does not affect the existing
nocodb_upsert behaviour.
---ci---
project: inboxos
phase: 1
milestone: v1.1
status: execute
---/ci---
Run #929 on 2026-06-08 hit extract_permanent_failure on an Ollama
504 after 6 messages and still surfaced as conclusion:success in
Gitea because historical_pipeline.main() returned 0 unconditionally.
Track loop exit reason (None for clean EOF / LIMIT_OVERRIDE,
'extract_permanent_failure' on a non-recoverable per-message
error) and return EARLY_STOP_EXIT_CODE=2 on the failure path so
Gitea records the run as failure. The 12h timeout is enforced by
the workflow itself (timeout-minutes: 720) and stays untouched.
Add pipelines/_tests/test_historical_pipeline_early_stop.py with
6 unittest.TestCase tests covering: extract_permanent_failure
returns 2, clean EOF returns 0, EOF after N messages returns 0,
LIMIT_OVERRIDE returns 0, EARLY_STOP_EXIT_CODE is non-zero and
distinct from 1, and historical_run_early_stop event is emitted
with reason=extract_permanent_failure and the chosen exit code.
Acceptance: python3 -m unittest discover -s pipelines/_tests
-p test_*.py exits 0 with 16/16 tests passing (was 10/10 before).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Lifts the historical Papra attachment uploader into common.py and adds
a pure policy function + dry-run gate, so the live and historical
pipelines can audit Papra write decisions on real traffic before the
first real write goes out.
- common.py: PapraConfig, PapraTarget, PapraDecision, papra_decide,
papra_write_attachment, papra_write_body, content-hash and HTML-strip
helpers. Dry-run defaults to True; honors PAPRA_DRY_RUN env. Pure
decide() is unit-testable without HTTP. Write helpers never raise;
failures are logged as papra_write_failed.
- live_pipeline.py: wires papra_decide + papra_write_* into
process_one_message after the NocoDB UPSERTs. Emits papra_decision
per message even on the skip path.
- historical_pipeline.py: replaces inline upload_attachments_to_papra
with the shared helper, keeps attachment behavior intact, and adds
the post-classification body-write pass.
- .gitea/workflows/{live,historical}-pipeline.yml: export
PAPRA_BASE_URL / API_TOKEN / ORG_ID / DRY_RUN (default 'true') and
the body-write / metadata-only module lists.
- tests: 13/13 new test_papra_decide_offline.py + 14/14
test_live_pipeline_offline.py + 7/7 per-message historical tests
(the unchanged-shape 100MB e2e test is not re-run this commit).
- docs/papra-write-policy.md: policy, env vars, dry-run flag, audit
recipe, future papra_index note.
No real Papra HTTP request is made during the dry-run smoke.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
inb-21: drop model-escalation chain; pin single gemma4:e4b model; add PR CI (#1)
Drops the per-step escalation chain and the 100MB e2e CI test was
failing because _setup_env() was not called inside the test itself.
Adds a one-line call so the test runs under any runner.
* pipelines/common.py: drop INBOXOS_LLM_MODEL_CHAIN, model_chain,
ollama_run_command_with_chain, llm_run_command_with_chain.
Pin a single model per command (default = extract = gemma4:e4b).
* pipelines/historical_pipeline.py: switch from
llm_run_command_with_chain to llm_run_command.
* pipelines/live_pipeline.py: switch from
llm_run_command_with_chain to llm_run_command.
* .gitea/workflows/ci-pr.yml: new PR-only gate (fast-tests + 100MB
e2e + live-llm-smoke).
* .gitea/workflows/live-pipeline.yml: drop INBOXOS_LLM_MODEL_CHAIN.
* .gitea/workflows/historical-pipeline.yml: drop
INBOXOS_LLM_MODEL_CHAIN.
* .env.example: document the new env vars (LLM_MODEL,
LLM_MODEL_EXTRACT) and remove the chain var.
* pipelines/_tests/*: refresh the test helpers to expect the
single-model contract.
* pipelines/_tests/test_historical_pipeline_offline.py: fix the
100MB e2e test by calling _setup_env() inside the test (this
is the fix from commit 288cfcb that completes the PR gate).
Closes INB-21.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Applies the user-directed model preference (gemma4:e4b first,
gemma4:12b fallback) to the live pipeline. The previous CTO commit
(d9b8d8a) wired the chain for the historical pipeline only; the
live extract step was still using the single-model
`llm_run_command` path and the live workflow had INBOXOS_LLM_MODEL
+ INBOXOS_LLM_MODEL_EXTRACT hardcoded to gemma4:31b-cloud.
* pipelines/live_pipeline.py: extract_for_module now uses
llm_run_command_with_chain so a flaky gemma4:e4b escalates
cleanly to gemma4:12b (or any later chain entry the operator
configures). Classify stays on the single-model path on the
cheap default -- classify is small, the chain adds latency
with no upside for the routing decision.
* pipelines/_tests/test_live_pipeline_offline.py: _install_fakes
now stubs both llm_run_command and llm_run_command_with_chain
so the new chain path is exercised by the existing happy-path
/ unclassified / extract-failure tests. 14/14 still green.
* .gitea/workflows/live-pipeline.yml: default INBOXOS_LLM_MODEL
and INBOXOS_LLM_MODEL_EXTRACT flipped from gemma4:31b-cloud to
gemma4:e4b (user direction); INBOXOS_LLM_MODEL_CHAIN wired
through vars/secrets with default 'gemma4:e4b,gemma4:12b' so
the cheap e4b is primary and 12b is the fallback. The comment
block at the top of the workflow is updated to match.
Smoke evidence (real Ollama via the live extract code path):
Substack-sized body (run 540/674 payload):
lfm2.5:8b -> finish_reason=length, 1024 tokens, 0 content bytes
gemma4:e4b -> finish_reason=stop, 962 tokens, full CRM JSON
(32.9s; the original INB-21 bug is fixed)
Small intro body (3-line Jane Doe email):
gemma4:e4b -> full CRM JSON in 58.6s via lp.extract_for_module
Chain escalation (forced primary 404):
step 0 fails -> logs llm_chain_step_failed -> step 1 tried
(gateway 504s prevent the in-test success but
the chain behavior is fully logged and correct)
Offline test suite: 14/14 live + 8/8 fast historical pass on the
rebased tip (d9b8d8a + this commit).
Refs: INB-21 user comment b0b72fa1 -- "lets try these models
instead (in that order): gemma4:e4b, gemma4:12b"
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Per board directive on INB-20:
- live pipeline: INBOXOS_LLM_MODEL=INBOXOS_LLM_MODEL_EXTRACT=gemma4:31b-cloud (single model, no escalation)
- historical pipeline: INBOXOS_LLM_MODEL_CHAIN='gemma4:e4b,gemma4:12b,gemma4:31b-cloud' (primary, smart retry, 2nd smart retry)
- LLMConfig now has a model_chain field parsed from INBOXOS_LLM_MODEL_CHAIN; ollama_run_command_with_chain + llm_run_command_with_chain implement the chain as the retry policy (no same-model tenacity, so a flaky cheap model does not multiply wall-clock latency)
- precedence flip in ollama_run_command: per-step env-var model wins over prompt frontmatter, so an operator can pin a pipeline-level model without editing prompt files
- .opencode/commands/inboxos:*.md frontmatter updated to gemma4:e4b (matches the historical primary; the live env var overrides to 31b-cloud at runtime)
Offline tests: 14/14 live, 8/8 historical unit (incl. new test_process_escalation_recovers_on_second_model) pass. Live Ollama smoke: gemma4:31b-cloud correctly classifies a test email via the live pipeline path; gemma4:e4b → escalation chain routes through to gemma4:31b-cloud when needed.
Refs: INB-20
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Live run #692 got all the way to nocodb_upsert on a fresh
Substack CRM record and hit HTTPError 400 Bad Request on the
POST. The traceback only showed the URL and status, not the
NocoDB response body — the actual field-level reason for the
400 is in that body, and the synthetic test in
test_upsert_synthetic.py already calls out the
SingleSelect-options-not-set gap on the live table as the
likely culprit.
Wrap the POST and PATCH in 'if not resp.ok' to raise a
RuntimeError that includes the response body, table_id, and
record_keys (or record_id) so the next run's log can pinpoint
the exact NocoDB column that's rejecting the record.
Offline tests: 10/10 still pass.
Refs: INB-17
Co-Authored-By: Paperclip <noreply@paperclip.ing>
NocoDB v2026.04.3 (the self-hosted instance behind
noco.cloudinit.dev) returns 404 for `PATCH /api/v2/tables/{tid}/records/{id}`,
the per-record URL the original nocodb_upsert used. The supported
v2 path is a bulk PATCH on `/records` with a body shaped as
`[{"Id": id, ...overrides...}]`. The synthetic test in
pipelines/_tests/test_upsert_synthetic.py already documents this
gap (lines 92-95) and uses the bulk format.
Live run #682 hit the bug: search found row Id=529 for the
Substack CRM extract, then PATCH on /records/529 returned 404.
Switching to the v2 bulk format is the supported path. The Id is
in the body so re-PATCHing an existing row is idempotent on
Message_ID.
Offline tests: 10/10 still pass (no NocoDB-touching tests in the
offline suite; the synthetic test is gated on NOCODB_* env vars
and is a separate run).
Refs: INB-12, INB-17
Co-Authored-By: Paperclip <noreply@paperclip.ing>
INB-21 was resolved on the live pipeline in c5b7cf8, but the
historical pipeline still had the old message_summary_for_prompt
(20k body cap, no HTML stripping). Apply the same fix here so the
two paths stay in sync and re-running the historical batch over
HTML-only bounces produces a useful prompt instead of a literal
"this is a block of HTML..." response.
- historical_pipeline.message_summary_for_prompt: reuse the
_strip_html_to_text helper from live_pipeline, decode bytes
payloads, and lower the default body cap to 2000 chars.
- .env.example: document INBOXOS_BODY_CHARS, INBOXOS_LLM_MAX_TOKENS,
INBOXOS_LLM_MODEL, INBOXOS_LLM_MODEL_EXTRACT, INBOXOS_LLM_TIMEOUT
so the next operator does not have to read common.py to discover
the new tunables.
- .gitignore: ignore the per-checkout opencode.json (the opencode
CLI writes it on first invocation and it is not part of the
pipeline contract).
- pipelines/_tests/smoke_inb21_extract.py: an opt-in harness that
hits the real FRP-tunneled Ollama with the exact input that
dead-lettered in live run 674. The harness verifies the new
gemma4:12b extract path returns a parseable CRM record and
preserves the Message-ID. Not run by the offline suite; for
manual verification or for a future scheduled integration job.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
After a29b971 bypassed the opencode CLI on the live path, run 674
got all the way to extract-crm on the first Substack email — and
then hit finish_reason=length on the 86KB body. Two structural
causes plus one ops papercut:
1. ollama_run_command capped max_tokens at 1024. lfm2.5:8b always
spends ~100-300 tokens on chain-of-thought reasoning, so only
~700 tokens were left for the 11-field CRM extract schema. A
richer body burns through that and the response comes back with
content='' and finish_reason='length' after the budget is gone.
- Raise default max_tokens to 4096 (env: INBOXOS_LLM_MAX_TOKENS).
- Route extract-* commands to a bigger model
(INBOXOS_LLM_MODEL_EXTRACT, default gemma4:12b) so the 12B
has more headroom. Classify stays on the cheap lfm2.5:8b.
2. The first email in run 674 was a raw HTML Amazon SES bounce.
lfm2.5:8b literally responded 'It looks like you've shared a
block of HTML...' and never emitted JSON. Strip HTML to plain
text in message_summary_for_prompt() so the LLM sees prose,
not markup. Pure stdlib (re, html.unescape); no new deps.
3. .gitea/workflows/live-pipeline.yml used actions/upload-artifact@v4
which hits GHES-only endpoints and fails on Gitea Actions with
'GHESNotSupportedError'. Pin to v3 (the v1 Gitea-compatible API).
4. The body cap in message_summary_for_prompt was 20000 chars. 2000
is enough for the 11-field schema on a 4096-token completion
budget and keeps a 86KB Substack from eating the whole run.
Override via INBOXOS_BODY_CHARS.
5. historical-pipeline.yml still uses the opencode CLI (separate
from the live bypass). Pin OPENCODE_NPM_VERSION to 1.15.13 —
1.16.x wraps every 4xx from a provider in 'UnknownError' inside
SessionPrompt.command (INB-20). Add a version-mismatch check
so a future runner image can't silently regress.
6. tests/test_live_pipeline_offline.py: cover the new
HTML-stripping + extract-routing code paths with 5 new tests.
10/10 pass locally.
7. .gitignore: ignore .tools/ (local opencode 1.15.13 install for
offline smoke tests).
8. scripts/setup_gitea_secrets.sh: set OPENCODE_NPM_VERSION secret
alongside the workflow default for parity with the historical
pipeline.
End-to-end check against the live Ollama (FRP-tunneled):
llm_run_command('inboxos:classify', QuickBooks invoice payload)
→ {lmm-2.5:8b, 4096 max} → {'modules': ['finance']}
llm_run_command('inboxos:extract-crm', Jane Doe Q3 pilot payload)
→ {gemma4:12b, 4096 max} → 11-field CRM JSON
Refs: INB-17, INB-20
Co-Authored-By: Paperclip <noreply@paperclip.ing>
INB-19 acceptance requires the live run log to show
"service=llm ... modelID=lfm2.5:8b" so we can verify the runner
actually used the FRP-tunneled Ollama. Plumb a logger through
ollama_run_command and llm_run_command (optional, default None to
preserve existing callers) and emit a structured llm_call event
right before the HTTP request. Update the offline test fake to
accept the new logger kwarg.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
opencode 1.16.2 server returns UnknownError on every `--command`
invocation (ref: err_xxx). The local-only `opencode run "what is 2+2?"`
default-agent path still works, but `SessionPrompt.command` (the path
the live pipeline uses) crashes immediately on every prompt. Diagnosis:
- Local reproduction: `opencode run --command inboxos:classify` →
UnknownError, server-side stacktrace in
`SessionPrompt.command (/$bunfs/root/chunk-...).js`.
- The Ollama server itself is healthy: `GET /v1/models` returns 7
models including `lfm2.5:8b`; `POST /v1/chat/completions` with the
`inboxos:classify` prompt body returns clean JSON.
The fix is to drop the opencode CLI from the hot path. The pipeline now
calls Ollama's OpenAI-compatible `/v1/chat/completions` endpoint
directly:
- `pipelines/common.py`: new `LLMConfig` (backend selector), new
`ollama_run_command`, new `llm_run_command` dispatcher. The legacy
`opencode_run_command` is kept for offline tests.
- `pipelines/live_pipeline.py`: switch from `opencode_run_command` /
`OpenCodeConfig` to `llm_run_command` / `LLMConfig`. The model
name is now read from the prompt frontmatter (`model: ollama/...`)
with a configurable default (`INBOXOS_LLM_MODEL`).
- `.gitea/workflows/live-pipeline.yml`: drop the "Ensure OpenCode is
installed" step; add `INBOXOS_LLM_BACKEND`, `INBOXOS_LLM_MODEL`,
`INBOXOS_LLM_TIMEOUT`, `OLLAMA_API_KEY` env vars.
- `pipelines/_tests/test_live_pipeline_offline.py`: rebind the fake
onto `lp.llm_run_command` and use `LLMConfig(backend="opencode", …)`
so the offline smoke stays green.
- `.env.example` and `scripts/setup_gitea_secrets.sh`: switch the
default S3 endpoint to `https://storage.cloudinit.dev` (MinIO) per
the platform migration in INB-16.
Local end-to-end check (single message against the live Ollama):
LLMConfig(backend="ollama", ollama_base_url="https://ollama-frp.cloudinit.dev/v1", ...)
→ ollama_run_command("inboxos:classify", {subject="QuickBooks invoice #1234", ...})
→ {"modules": ["finance"]} ✓
Offline tests: 8/8 pass. No secrets touched.
Refs: INB-17
Co-Authored-By: Paperclip <noreply@paperclip.ing>
For each of the four inboxos module tables, this test verifies the
end-to-end upsert contract the pipelines/common.py::nocodb_upsert
relies on:
1. First call with a new Message_ID returns action=post and creates
exactly one row.
2. Second call with the same Message_ID returns action=patch and
leaves the row count at one (no duplicate).
3. The patched summary field reflects the new value.
Cleans up its own test rows at the end via the v1 DELETE endpoint.
Two known gaps the test sidesteps by using only non-SingleSelect
fields, and which are documented inline:
a) The existing inboxos tables have SingleSelect columns with no
pre-defined options. NocoDB v2026.04.3 rejects options-set via
PATCH (returns a generic error). The pipeline writes by string
value so this is safe, but a follow-up issue should add a
one-shot options-set step.
b) pipelines/common.py::nocodb_upsert PATCHes with the v2 single-
record URL (/api/v2/tables/{id}/records/{rowId}) which returns
404 in v2026.04.3. The fix is either v1 single-record PATCH
(/api/v1/db/data/noco/{baseId}/{tableId}/{rowId}) or the v2
bulk PATCH ({tid}/records with array body).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The Gitea REST API does not return the plaintext of a freshly created
access token, so the new GiteaConfig / gitea_request helpers added
in pipelines/common.py use HTTP Basic Auth (GITEA_USER + GITEA_PASS)
instead of a GITEA_TOKEN. This commit wires the new env vars into
.env.example and documents the auth model in docs/gitea-actions.md
so the next operator knows which secrets to set.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Gitea's REST API does not return the plaintext of a freshly created
access token (only the sha1 hash and last eight characters are
returned by POST /user/tokens), so a headless agent cannot produce a
GITEA_TOKEN env var from the API alone. Add a Basic-Auth-backed
config + retrying request helper so the live + historical pipeline
automation can call the Gitea API for repo create, secret set, and
runner-registration lookups without depending on a UI bootstrap
step.
New env vars (consumed by GiteaConfig.from_env):
GITEA_INSTANCE_URL already required by docker-compose
GITEA_USER operator-supplied Gitea user (write scope)
GITEA_PASS operator-supplied Gitea password
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The historical pipeline writes a separate run log per process; a 100MB
mbox run can produce a 100MB log file. Two small changes for the
offline test runner:
- Set LOG_LEVEL=WARNING before pipeline import so per-message structured
logs don't dominate the test output on the 100MB e2e case.
- After the 100MB run, unlink any historical-run-*.log files under
pipelines/_artifacts/ so repeated test runs don't bloat the worktree.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
backend-engineer
cloudinit-bot
cto