9 Commits

Author SHA1 Message Date
cloudinit-bot ae31098d9a ci(workflows): secret-redaction filter for 9 secret env vars (REQ-INB-56)
---ci---
project: inboxos
phase: 1
milestone: v1.4
status: execute
wave: 1
task: T7
---/ci---
2026-06-09 14:09:29 +00:00
cloudinit-bot b54df85c49 chore(env): add 7 workflow tunables to .env.example (REQ-INB-33)
The .env.example file documented the IMAP, NocoDB, SeaweedFS,
Papra (bare connection vars), Ollama base URL, and Gitea env
blocks, but was missing 7 env vars that the workflows reference:

1. OLLAMA_API_KEY="" — empty by default (Ollama local is
   unauthenticated; the entry is for FRP-tunneled or remote
   Ollama where the operator has set a key)
2. INBOXOS_LLM_BACKEND=ollama — default backend; the v1.0
   production path is Ollama. The "opencode" backend is wired
   in pipelines/common.py::opencode_run_command but documented
   as offline-tests-only (common.py:455-461).
3. PAPRA_DRY_RUN=true — the v1.0 + v1.1 + v1.2 default. Set
   to "false" to enable live Papra writes (the Phase 6 /
   future-v1.x gate per docs/papra-write-policy.md).
4. PAPRA_BODY_WRITE_MODULES=crm,finance,legal — modules whose
   bodies (not just attachments) get written to Papra on the
   post-classify pass.
5. PAPRA_METADATA_ONLY_MODULES=intel,promotions — modules whose
   Papra write is metadata-only (no body upload).
6. MBOX_BUCKET= — historical pipeline's S3 bucket for the
   .mbox source; defaults to $SEAWEEDFS_S3_BUCKET if unset.
7. OPENCODE_NPM_VERSION=latest — historical pipeline's
   opencode CLI version.

This commit adds a new "Workflow tunables" section at the end
of .env.example containing all 7 commented-out entries with
workflow-baked defaults and 1-line explanations. The
INBOXOS_LLM_MODEL / INBOXOS_LLM_MODEL_EXTRACT block (lines
96-99) is unchanged (REQ-INB-15 / v1.1 P1-5 closure).

Verified: no literal token values are introduced (the
v1.0 secret-scan convention is honored: pre-existing
NOCODB_API_TOKEN=change-me-* / GITEA_PASS=change-me-* /
IMAP_PASSWORD=change-me-* are placeholder values, not real
secrets). No cto-inboxos references are introduced
(REQ-INB-28 / REQ-INB-37 / v1.1 P0 sweep closure preserved).

---ci---
project: inboxos
phase: 1
milestone: v1.3
status: execute
tasks_completed: [T3.1]
candidates_closed: [REQ-INB-33]
files_touched:
  - .env.example (1 file, 45 lines added at the end, new "Workflow tunables" section)
---/ci---
2026-06-08 23:29:14 +00:00
cloudinit-bot 6b05e8da2e chore(env): pin INBOXOS_LLM_MODEL comments to gemma4:e4b (REQ-INB-15, P1-5)
The legacy two-model split (cheap classify + larger extract) was retired in v1.0; the .env.example comments still described the old split. Replace the stale lfm2.5:8b / gemma4:12b comments with the canonical gemma4:e4b pin and a one-line rationale citing REQ-INB-11.

Verification:
- pipelines/common.py:467-468 default to gemma4:e4b (no code change).
- docs/extraction-contract.md §Model spec (lines 46-56) already declares gemma4:e4b (no edit).

---ci---
project: inboxos
phase: 1
milestone: v1.1
status: execute
---/ci---
2026-06-08 20:59:07 +00:00
cto 034b08ff3e INB-117: correct documented SeaweedFS bucket and historical .mbox key
InboxOS — PR CI / Fast offline tests (pull_request) Successful in 28s
InboxOS — PR CI / 100MB historical e2e (pull_request) Successful in 11m9s
InboxOS — PR CI / Live Ollama extract smoke (pull_request) Successful in 51s
The actual deployment uses bucket 'inboxos' (not 'inboxos-eml') and the
.mbox lives at key 'eml/historical/all-mail.mbox' (not 'historical/all-mail.mbox').
Full canonical URI: s3://inboxos/eml/historical/all-mail.mbox. Confirmed by
Gitea Actions run #922 (probe dispatch on runner-2, 799 MB read, end-to-end
UPSERT into NocoDB).

Updates:
- README.md: add Historical archive section with canonical URI and mc ls hint.
- .env.example: SEAWEEDFS_S3_BUCKET=inboxos.
- docs/operations-runbook.md: §3.2 key example, §4.4 layout header + s3:// URI,
  §6.1 secret table, §6 env-var inventory.
- .gitea/workflows/historical-pipeline.yml: comment + workflow_dispatch input.
- .gitea/workflows/live-pipeline.yml: comment.
- pipelines/README.md: live + historical secret blocks.
- scripts/setup_gitea_secrets.sh: SEAWEEDFS_S3_BUCKET default.
- pipelines/historical_pipeline.py + live_pipeline.py: bucket_default kwarg.

INB-34 description also updated to match (doc-text correction; INB-34 itself
is closed done in INB-43 with the correct values).

Test fixtures in pipelines/_tests/ retain the 'inboxos-eml' string as a
self-contained test-only value; the issue explicitly allows this.

Verification:
  git grep -n 'inboxos-eml' -- ':!pipelines/_tests/'     -> 0 hits
  git grep -n 'historical/all-mail.mbox' (no eml/ prefix) -> 0 hits

Refs: INB-34, INB-43 (probe run #922), INB-7

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-06-08 13:05:41 +00:00
cto b6878f6fdc INB-33: provision second self-hosted Gitea Actions runner (#3) 2026-06-08 07:26:00 +00:00
cto ab7331d016 historical: mirror live INB-21 fixes; .env docs; smoke harness
INB-21 was resolved on the live pipeline in c5b7cf8, but the
historical pipeline still had the old message_summary_for_prompt
(20k body cap, no HTML stripping). Apply the same fix here so the
two paths stay in sync and re-running the historical batch over
HTML-only bounces produces a useful prompt instead of a literal
"this is a block of HTML..." response.

- historical_pipeline.message_summary_for_prompt: reuse the
  _strip_html_to_text helper from live_pipeline, decode bytes
  payloads, and lower the default body cap to 2000 chars.
- .env.example: document INBOXOS_BODY_CHARS, INBOXOS_LLM_MAX_TOKENS,
  INBOXOS_LLM_MODEL, INBOXOS_LLM_MODEL_EXTRACT, INBOXOS_LLM_TIMEOUT
  so the next operator does not have to read common.py to discover
  the new tunables.
- .gitignore: ignore the per-checkout opencode.json (the opencode
  CLI writes it on first invocation and it is not part of the
  pipeline contract).
- pipelines/_tests/smoke_inb21_extract.py: an opt-in harness that
  hits the real FRP-tunneled Ollama with the exact input that
  dead-lettered in live run 674. The harness verifies the new
  gemma4:12b extract path returns a parseable CRM record and
  preserves the Message-ID. Not run by the offline suite; for
  manual verification or for a future scheduled integration job.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-06-07 18:27:38 +00:00
cto a29b97132b pipelines: bypass opencode CLI; call Ollama directly
opencode 1.16.2 server returns UnknownError on every `--command`
invocation (ref: err_xxx). The local-only `opencode run "what is 2+2?"`
default-agent path still works, but `SessionPrompt.command` (the path
the live pipeline uses) crashes immediately on every prompt. Diagnosis:
  - Local reproduction: `opencode run --command inboxos:classify` →
    UnknownError, server-side stacktrace in
    `SessionPrompt.command (/$bunfs/root/chunk-...).js`.
  - The Ollama server itself is healthy: `GET /v1/models` returns 7
    models including `lfm2.5:8b`; `POST /v1/chat/completions` with the
    `inboxos:classify` prompt body returns clean JSON.

The fix is to drop the opencode CLI from the hot path. The pipeline now
calls Ollama's OpenAI-compatible `/v1/chat/completions` endpoint
directly:

  - `pipelines/common.py`: new `LLMConfig` (backend selector), new
    `ollama_run_command`, new `llm_run_command` dispatcher. The legacy
    `opencode_run_command` is kept for offline tests.
  - `pipelines/live_pipeline.py`: switch from `opencode_run_command` /
    `OpenCodeConfig` to `llm_run_command` / `LLMConfig`. The model
    name is now read from the prompt frontmatter (`model: ollama/...`)
    with a configurable default (`INBOXOS_LLM_MODEL`).
  - `.gitea/workflows/live-pipeline.yml`: drop the "Ensure OpenCode is
    installed" step; add `INBOXOS_LLM_BACKEND`, `INBOXOS_LLM_MODEL`,
    `INBOXOS_LLM_TIMEOUT`, `OLLAMA_API_KEY` env vars.
  - `pipelines/_tests/test_live_pipeline_offline.py`: rebind the fake
    onto `lp.llm_run_command` and use `LLMConfig(backend="opencode", …)`
    so the offline smoke stays green.
  - `.env.example` and `scripts/setup_gitea_secrets.sh`: switch the
    default S3 endpoint to `https://storage.cloudinit.dev` (MinIO) per
    the platform migration in INB-16.

Local end-to-end check (single message against the live Ollama):
  LLMConfig(backend="ollama", ollama_base_url="https://ollama-frp.cloudinit.dev/v1", ...)
  → ollama_run_command("inboxos:classify", {subject="QuickBooks invoice #1234", ...})
  → {"modules": ["finance"]}    ✓

Offline tests: 8/8 pass. No secrets touched.

Refs: INB-17
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-06-07 17:49:00 +00:00
InboxOS CTO dbdd935c58 env,docs: document GITEA_USER/GITEA_PASS Basic Auth for the Gitea REST helper
The Gitea REST API does not return the plaintext of a freshly created
access token, so the new GiteaConfig / gitea_request helpers added
in pipelines/common.py use HTTP Basic Auth (GITEA_USER + GITEA_PASS)
instead of a GITEA_TOKEN. This commit wires the new env vars into
.env.example and documents the auth model in docs/gitea-actions.md
so the next operator knows which secrets to set.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-06-05 20:36:05 +00:00
InboxOS CTO 1d74d851ac InboxOS Phase 2 — initial bootstrap
- Schemas for finance / crm / intel / promotions modules (docs/schemas.md)
- Extraction contract canonicalisation (docs/extraction-contract.md)
- scripts/apply_schemas.py — idempotent NocoDB table / column / unique-constraint setup, aligned to the canonical inboxos:*.md prompts and NOCODB_MODULE_TABLES env var
- scripts/push_to_gitea.sh — this bootstrap script (idempotent, --reset safe)
- .gitignore — secrets, live data, n8n CLI workspace, and runtime artifacts stay out of git
- Existing pipeline + workflow + prompt files from Phase 3 / 4 work

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-06-05 20:23:39 +00:00