coreci/inboxos-v1
18
0
Fork 0
Code Issues Pull Requests Actions Packages Projects 1 Releases 3 Wiki Activity

docs(plan): v1.4 phase 1 plans — 23 candidates, 3 waves, 1 CPO-gated

Browse Source 
---ci---
project: inboxos
phase: 1
milestone: v1.4
status: plan
plan:
  total_candidates: 23
    new: 20
    carries: 3
    deferred_to_v1_5_plus: 5
  waves: 3
    wave_1: 11   # small-effort scripts + tests + security-doc baseline
    wave_2: 9   # operator-experience M-effort + container hardening
    wave_3: 3   # security documentation + PII-scrub + cost report
  total_prs: 23
  total_commits: 23
  cpo_approval_required: [REQ-INB-57]
  schema_impact: none
  cross_project_candidates: 0
  source_issue: https://git.cloudinit.dev/jc-capital/inboxos/issues/18
  user_directive: "Proceed with the complete milestone workflow including full release flow/shipping. PRs require dual-approval (CPO+CTO) anyway so you don't need to wait for docs/requirements.md to be updated"
  next_stage: execute
---/ci---
This commit is contained in:
cloudinit-bot
2026-06-09 13:56:03 +00:00
parent 2e82842e9a
commit 535624993b
1 changed files with 661 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.ciagent/inboxos/plans/01-PLAN-v1.4.md
+661
View File
@@ -0,0 +1,661 @@
# InboxOS v1.4 — Phase 1 Closure Plan (UX + Security)
**Project:** inboxos
**Milestone:** v1.4
**Phase:** 1 — v1.4 Ideation → Plan closure (23 ACCEPT candidates, 0 REJECT, 5 DEFER to v1.5+)
**Status:** plan (spec complete; ideate complete; execution pending Gitea issue #18 review)
**Plan owner:** ci-planner
> The 23 ACCEPT candidates (20 new + 3 carries) originate from the user-directed operator-UX + product-grade-security IDEATE pass recorded in `.ciagent/inboxos/ideation/v1.4-candidates.md`. The Gitea issue #18 (https://git.cloudinit.dev/jc-capital/inboxos/issues/18) was opened on 2026-06-08 to request CPO review. Per the user's directive (v1.4 run context), the workflow proceeds through PLAN → EXECUTE → SHIP → VERIFY → COMPLETE without waiting for `docs/requirements.md` updates; the dual-approval (CPO+CTO) gate on PRs is the actual promotion path.
>
> **CPO gate:** REQ-INB-57 (PII-scrub body before Ollama) is `[CPO-APPROVAL-REQUIRED]` — the masking policy is CPO-owned. The proposed policy is in `v1.4-candidates.md §3.2 T1-SEC-3`; the CPO can adopt as-is or amend. The other 22 candidates are standard.
>
> **5 pre-deferred to v1.5+** (DEF-V1.4-1..5 in `v1.4-candidates.md §9`): container hardening expansion, encryption-at-rest, retention/GDPR-purge, `nocodb.audit_log` table, encrypted offsite backup. None land in this plan.
---
## How to read this plan
- **Wave 1 (small-effort scripts + tests + security-doc baseline)** closes 11 candidates: the 8 S/XS-effort UX scripts (REQ-INB-43, -46, -48, -49, -51, -52, -54), the secret-redaction log filter (REQ-INB-56), and the 3 v1.3 carries (REQ-INB-35c, -38, -39). All 11 are XS-S effort and most are net-new scripts.
- **Wave 2 (operator-experience M-effort + container hardening + tutorials)** closes 9 candidates: the 7 M-effort UX deliverables (REQ-INB-40 quickstart, -41 NocoDB views, -42 explain_run, -44 reclassify, -45 replay, -47 daily_digest, -50 dashboard, -53 operator-tutorial) + the container hardening (REQ-INB-55). Wave 2 is the bulk of the operator-experience work.
- **Wave 3 (security documentation + PII-scrub + cost report)** closes 3 candidates: REQ-INB-57 (PII-scrub, CPO-gated), REQ-INB-58 (DPIA), REQ-INB-59 (threat model), plus REQ-INB-54 was Wave 1.
- The 5 DEFER items (DEF-V1.4-1..5) are NOT in this plan. They are filed for v1.5+ in `.ciagent/inboxos/ideation/v1.4-candidates.md §9`.
- Each task lists a `must_haves` array — the concrete artifacts a reviewer must see before the task is closed.
- **CPO-owned files (`docs/requirements.md`, `docs/roadmap.md`) are NOT edited by any task in this plan.** The plan reads from them (the REQ-INB IDs in `must_haves`) but never writes to them. The CPO is the only role authorised to promote `[CANDIDATE]` to a real requirement.
- The plan is the integrator's contract for the EXECUTE phases that follow. The Wave 1 + Wave 2 + Wave 3 commits each carry a well-formed `---ci---` block with `project: inboxos`, `phase: 1`, `milestone: v1.4`, `status: execute` / `verify`.
- **Commit convention** follows the v1.1 / v1.2 / v1.3 pattern: each task is 1 commit (or 1 commit per file if the task touches > 5 files, per the runbook §9.1 row 5 cap).
- **No NocoDB schema changes** across all 23 candidates (v1.4 is a UX + security milestone, not a schema milestone). The DEFER set has 1 schema change (DEF-V1.4-4 `nocodb.audit_log`); that's why it's deferred.
---
## Persona assignments
| Persona | Active in v1.4 | Tasks |
|---------|----------------|-------|
| lead-developer | yes (coordination + cross-cutting doc) | W1-T1, W2-T6, W2-T8, W2-T11, W3-T12, W3-T13 |
| devops-engineer | yes (env contract, runbook, gitea-actions, container hardening) | W1-T3, W1-T6, W2-T7, W2-T9 |
| security-engineer | yes (log scrub, PII scrub, DPIA, threat model, container hardening review) | W1-T7, W2-T9, W3-T12, W3-T13, W3-T14 |
| backend-engineer | yes (CLI scripts in pipelines/scripts territory) | W1-T1, W1-T2, W1-T4, W1-T5, W1-T6, W1-T8, W1-T11, W2-T2, W2-T3, W2-T4, W2-T5, W2-T8, W2-T10 |
| data-engineer | yes (NocoDB curated views) | W2-T2 |
| ml-engineer | yes (PII scrub) | W3-T12 |
| frontend-engineer | DEACTIVATED (no React/TSX in repo) | n/a |
Cross-persona reviews are explicit (`needs review from:`). Territory enforcement is `warn` per `.ciagent/config.json`.
---
## Wave 1 — Small-effort scripts + tests + security-doc baseline (11 candidates, 11 PRs)
> Wave 1 has 11 tasks that land as 11 PRs. All 11 are XS or S effort. The lead-developer + backend-engineer are the primary owners; security-engineer is the reviewer for the log-scrub task (T7).
---
### T1 — `scripts/find_email.py` — search by MID / sender / subject (REQ-INB-43) **[cli]**
- **Refs:** REQ-INB-43 (search), REQ-INB-3 (idempotency)
- **Files:** `scripts/find_email.py` (new), `pipelines/_tests/test_find_email_offline.py` (new)
- **Persona:** backend-engineer (primary, scripts territory)
- **needs review from:** data-engineer (NocoDB query semantics)
- **Diff size:** S (1 new script ~50 lines, 1 new test ~40 lines)
- **must_haves:**
- `scripts/find_email.py` is a Python 3.12 CLI that takes `--mid`, `--sender`, `--subject`, `--since`, `--until` flags (mutually exclusive; default `--mid` if no flag is given) and queries NocoDB across all 4 module tables (`finance`, `crm`, `intel`, `promotions`) + the `papra_index` table for matching records. Results print as `MID | module | sender | subject | received_at | papra_link` one row per match.
- The script honors the same `XConfig.from_env()` pattern as `pipelines/common.py` for NocoDB credentials (`NOCODB_BASE_URL`, `NOCODB_API_TOKEN`).
- On `--mid` matches, the script additionally checks the S3 bucket for the corresponding `eml/<kind>/<yyyy>/<mm>/<dd>/<mid>.eml` key and prints the S3 path if found.
- `pipelines/_tests/test_find_email_offline.py` covers 4 cases: (a) `--mid` exact match returns 1 row, (b) `--sender` substring match returns N rows, (c) no match returns empty + exit 0, (d) NocoDB unreachable returns exit 2 (not exit 0). All 4 pass.
- `chmod +x scripts/find_email.py` and a 1-line shebang `#!/usr/bin/env python3` are set.
- `git log` shows one commit: `feat(scripts): find_email.py — search by MID/sender/subject (REQ-INB-43)`.
---
### T2 — Runbook §5 error catalog: `ERROR-NN → §X.Y` table (REQ-INB-46) **[doc]**
- **Refs:** REQ-INB-46 (error catalog), REQ-INB-6 (operations runbook)
- **Files:** `docs/operations-runbook.md` (insert at top of §5, before the existing prose)
- **Persona:** lead-developer (primary, owns `**/*.md` territory)
- **needs review from:** devops-engineer (runbook territory)
- **Diff size:** S (~30 line table inserted at the top of §5)
- **must_haves:**
- `docs/operations-runbook.md` §5 gains a 1-page `## ERROR-NN catalog` table at the top, before the existing failure-mode prose. The table has 3 columns: `ERROR-NN` (e.g. `ERROR-01`), `Symptom` (1-line), `See §X.Y` (1 cross-ref to the §5 prose below). The table covers all 10 failure modes currently described in §5 prose (Wrong table_id, Missing required field, Schema drift, IMAP connection refused, NocoDB 401, Ollama connection refused, SeaweedFS connection refused, Papra dry-run flipped to false unexpectedly, State file corrupted, Backup tar fails).
- Each row's `See §X.Y` cross-ref points at the actual heading in the §5 prose below (no broken links).
- `git log` shows one commit: `docs(runbook §5): add ERROR-NN catalog table (REQ-INB-46)`.
---
### T3 — `pipelines/_artifacts/last_run.json` — operator audit cursor (REQ-INB-49) **[pipeline]**
- **Refs:** REQ-INB-49 (audit cursor), REQ-INB-6 (operations runbook)
- **Files:** `pipelines/common.py` (add 1 helper + 1 emit call), `pipelines/live_pipeline.py` (call helper at run end), `pipelines/historical_pipeline.py` (call helper at run end)
- **Persona:** backend-engineer (primary, common.py + pipelines territory)
- **needs review from:** devops-engineer (the artifact is an operator-facing cursor)
- **Diff size:** S (1 helper in common.py ~15 lines, 2 call sites ~2 lines each, 1 new test)
- **must_haves:**
- `pipelines/common.py` gains a `write_last_run_artifact()` helper that writes `pipelines/_artifacts/last_run.json` with `{run_id, run_type, started_at, finished_at, status, mids_total, mids_upserted, mids_skipped, mids_errored, modules: {finance: N, crm: N, intel: N, promotions: N}, error_summary: []}`. The helper is idempotent (overwrites on each call; does not append).
- `pipelines/live_pipeline.py` calls `write_last_run_artifact()` at the end of every successful run (after the last upsert, before exit). On exception, the helper is called in a `try/finally` so the cursor reflects the failure.
- `pipelines/historical_pipeline.py` calls the same helper at the end of every workflow_dispatch.
- The `last_run.json` is `.gitignore`-d (operator-local state, not version-controlled).
- `pipelines/_tests/test_last_run_artifact_offline.py` covers 2 cases: (a) successful run writes a valid JSON with all required fields, (b) failed run still writes the cursor with `status: "error"` and the error message in `error_summary`. Both pass.
- `git log` shows one commit: `feat(pipelines): _artifacts/last_run.json audit cursor (REQ-INB-49)`.
---
### T4 — `scripts/dry_run.py` — feed one `.eml`, no writes (REQ-INB-51) **[cli]**
- **Refs:** REQ-INB-51 (dry-run), REQ-INB-3 (idempotency), REQ-INB-4 (extraction contract)
- **Files:** `scripts/dry_run.py` (new), `pipelines/_tests/test_dry_run_offline.py` (new)
- **Persona:** backend-engineer (primary)
- **needs review from:** ml-engineer (the dry-run invokes the real LLM prompts)
- **Diff size:** S (1 new script ~60 lines, 1 new test ~30 lines)
- **must_haves:**
- `scripts/dry_run.py` is a Python 3.12 CLI that takes a path to a single `.eml` file (`--eml`), runs it through the same `process_one_message` path used by `historical_pipeline.py` and `live_pipeline.py`, but with all side effects (S3 upload, NocoDB upsert, Papra write) suppressed via a `DRY_RUN=1` env var. The output is the classified module + the extracted record (JSON to stdout) + a per-field trace (YAML to stderr).
- The dry-run flag is the same flag used by the existing offline tests (`DRY_RUN=1` in `pipelines/_tests/`); no new flag is invented.
- The script does NOT call out to Ollama if the env var `INBOXOS_LLM_MOCK=1` is set (a mock response is generated from the prompt frontmatter). This is the offline-test path; the online path (real Ollama) is the default.
- `pipelines/_tests/test_dry_run_offline.py` covers 2 cases: (a) dry-run with `INBOXOS_LLM_MOCK=1` produces a mock record, (b) dry-run with `DRY_RUN=1` does NOT write to S3 or NocoDB (asserted via mock counters). Both pass.
- `chmod +x scripts/dry_run.py` and a 1-line shebang `#!/usr/bin/env python3` are set.
- `git log` shows one commit: `feat(scripts): dry_run.py — feed one .eml through classify+extract, no writes (REQ-INB-51)`.
---
### T5 — `pipelines/_metrics/healthcheck.json` — pollable endpoint (REQ-INB-52) **[pipeline]**
- **Refs:** REQ-INB-52 (healthcheck), REQ-INB-6 (operations runbook)
- **Files:** `pipelines/common.py` (add 1 helper), `pipelines/live_pipeline.py` (call helper), `pipelines/historical_pipeline.py` (call helper)
- **Persona:** backend-engineer (primary)
- **needs review from:** devops-engineer (the healthcheck is operator-facing)
- **Diff size:** S (1 helper ~15 lines, 2 call sites, 1 test)
- **must_haves:**
- `pipelines/common.py` gains a `write_healthcheck_artifact()` helper that writes `pipelines/_metrics/healthcheck.json` with `{last_run_ts, last_run_status, pending_count, error_count_24h, archive_size_gb, modules: {finance: N, crm: N, intel: N, promotions: N}}`. The helper is idempotent.
- `pipelines/live_pipeline.py` and `pipelines/historical_pipeline.py` call the helper at run end (same lifecycle as T3's `last_run.json`).
- The healthcheck is `.gitignore`-d.
- `pipelines/_tests/test_healthcheck_offline.py` covers 2 cases: (a) healthy state writes `last_run_status: "ok"`, (b) degraded state writes `last_run_status: "degraded"` when `error_count_24h > 0`. Both pass.
- `docs/operations-runbook.md §4.5` is updated to reference `pipelines/_metrics/healthcheck.json` as the canonical uptime-monitor pollable endpoint (per v1.4 PIT-V1.4-1 lesson L-IDEATE-V1.4-06).
- `git log` shows one commit: `feat(pipelines): _metrics/healthcheck.json pollable endpoint (REQ-INB-52)`.
---
### T6 — `scripts/cost_report.py` — monthly NocoDB / S3 / Ollama rollup (REQ-INB-54) **[cli]**
- **Refs:** REQ-INB-54 (cost report), REQ-INB-6 (operations runbook)
- **Files:** `scripts/cost_report.py` (new), `pipelines/_tests/test_cost_report_offline.py` (new)
- **Persona:** backend-engineer (primary)
- **needs review from:** devops-engineer
- **Diff size:** S (1 new script ~50 lines, 1 new test ~30 lines)
- **must_haves:**
- `scripts/cost_report.py` is a Python 3.12 CLI that takes `--month YYYY-MM` (default: current month) and emits a JSON or CSV rollup: NocoDB row counts per module (queried from NocoDB), S3 archive size in GB (queried via boto3), Ollama token count (read from `pipelines/_metrics/*.jsonl` if present, else 0). Output is one row per service per day.
- The script does NOT make assumptions about cloud costs (the runbook §7 cost model is the source of truth; the script only emits the raw counters).
- `pipelines/_tests/test_cost_report_offline.py` covers 2 cases: (a) NocoDB unreachable emits a row with `nocodb: "unreachable"` and continues, (b) full happy path emits N rows where N = number of days in the requested month. Both pass.
- `chmod +x scripts/cost_report.py` and a 1-line shebang `#!/usr/bin/env python3` are set.
- `git log` shows one commit: `feat(scripts): cost_report.py — monthly NocoDB/S3/Ollama rollup (REQ-INB-54)`.
---
### T7 — Secret-redaction filter for workflow logs (REQ-INB-56) **[ci]**
- **Refs:** REQ-INB-56 (secret redaction), REQ-INB-10 (self-hosted PII boundary)
- **Files:** `.gitea/workflows/live-pipeline.yml`, `.gitea/workflows/historical-pipeline.yml`, `.gitea/workflows/ci-pr.yml`
- **Persona:** security-engineer (primary, owns the PII boundary)
- **needs review from:** devops-engineer (workflow YAML territory)
- **Diff size:** S (~10 lines added to each of 3 workflows: a `env:`-block mask step or a `actions/github-script` step)
- **must_haves:**
- Each of the 3 workflows gains a `secret_mask` step near the top of each job that calls `::add-mask::` for: `IMAP_PASSWORD`, `NOCODB_API_TOKEN`, `NOCODB_API_TOKEN_READONLY`, `GITEA_USER`, `GITEA_PASS`, `SEAWEEDFS_S3_ACCESS_KEY`, `SEAWEEDFS_S3_SECRET_KEY`, `PAPRA_API_TOKEN`, `OLLAMA_API_KEY`. The step runs before any `run:` step that might log these values.
- The mask step uses the standard `::add-mask::` syntax (Gitea Actions 1.x compatible) — each secret value is echoed to stdout prefixed with `::add-mask::`, which marks it as a secret in the log.
- The 3 workflows' YAML validity is verified: `python -c "import yaml; yaml.safe_load(open('.gitea/workflows/X.yml'))"` returns no error for all 3.
- `.env.example` is updated to add a 1-line note in the `## Notes` section: "The workflow YAML auto-masks the above secret values in CI logs via `::add-mask::`; do not echo them in custom `run:` steps."
- `git log` shows one commit: `ci(workflows): secret-redaction filter for IMAP_PASSWORD/NOCODB/Gitea/S3/Papra/Ollama (REQ-INB-56)`.
---
### T8 — `scripts/papra_open.py` — open Papra link from MID (REQ-INB-48) **[cli]**
- **Refs:** REQ-INB-48 (Papra open), REQ-INB-9 (Papra dry-run)
- **Files:** `scripts/papra_open.py` (new), `pipelines/_tests/test_papra_open_offline.py` (new)
- **Persona:** backend-engineer (primary)
- **needs review from:** devops-engineer
- **Diff size:** XS (1 new script ~25 lines, 1 new test ~20 lines)
- **must_haves:**
- `scripts/papra_open.py` is a Python 3.12 CLI that takes `--mid <Message-ID>` and looks up the NocoDB row across all 4 module tables for the `Papra_Link` column. If found, the script either (a) opens the URL in `xdg-open` if `DISPLAY` is set, or (b) prints the URL to stdout. If no `Papra_Link` is found, exit 1 with a friendly message.
- The script honors the same `XConfig.from_env()` pattern for NocoDB credentials.
- `pipelines/_tests/test_papra_open_offline.py` covers 2 cases: (a) row found with Papra_Link, prints URL (mocked `xdg-open`), (b) row not found, exits 1. Both pass.
- `chmod +x scripts/papra_open.py` and a 1-line shebang `#!/usr/bin/env python3` are set.
- `git log` shows one commit: `feat(scripts): papra_open.py — open Papra link from MID (REQ-INB-48)`.
---
### T9 — `.gitignore opencode.json` line shadows a tracked file (v1.3 carry REQ-INB-35c) **[chore]**
- **Refs:** REQ-INB-35c (v1.3 carry), REQ-INB-14 (prompt-as-code), REQ-INB-11 (single pinned LLM model)
- **Files:** `.gitignore` (remove the `opencode.json` line) OR add a comment to `opencode.json` (lead-developer's choice)
- **Persona:** lead-developer (primary, lead-developer's judgment call per v1.3 close)
- **needs review from:** devops-engineer (.gitignore territory)
- **Diff size:** XS (1-2 line edit)
- **must_haves:**
- **Lead-developer's choice (pick one of two):**
- **Option A:** remove the `opencode.json` line from `.gitignore` and add a 1-line comment in `opencode.json` ("This file is tracked; local overrides are not supported. To switch model pin, edit and commit.") declaring the file as canonical.
- **Option B:** keep the `.gitignore opencode.json` line and add a 2-line comment in `opencode.json` ("Tracked; .gitignore is for local override only. To switch model pin, edit and commit.") so the operator knows the file is tracked.
- Either option resolves the footgun (the operator can `git status opencode.json` or `git diff opencode.json` to see their edit). Option A is more explicit; Option B is more defense-in-depth.
- `git log opencode.json` still shows the 5 historical commits (the file remains tracked).
- `git log` shows one commit: `chore(gitignore): resolve opencode.json shadow footgun (REQ-INB-35c)`.
---
### T10 — `apply_schemas.py --check` offline test for `papra_index` (v1.3 carry REQ-INB-38) **[test]**
- **Refs:** REQ-INB-38 (v1.3 carry), REQ-INB-5 (module schema), REQ-INB-19 (papra_index)
- **Files:** `pipelines/_tests/test_apply_schemas_papra_index_offline.py` (new)
- **Persona:** backend-engineer (primary, scripts + test territory)
- **needs review from:** data-engineer (NocoDB schema)
- **Diff size:** XS (1 new test ~40 lines)
- **must_haves:**
- `pipelines/_tests/test_apply_schemas_papra_index_offline.py` is a new offline test that mocks the NocoDB HTTP layer (using the `FakeNoco` pattern from `test_papra_index_offline.py:144`) and asserts that `apply_schemas.py --check` exits 0 when the `papra_index` table exists with the 7 documented columns.
- The test follows the existing offline-test pattern (no network calls; `subprocess.run` with the script invocation; mock the NocoDB client).
- The test is wired into `.gitea/workflows/ci-pr.yml` `fast-tests` job (per the v1.3 carry's intent).
- `pip install` step at `ci-pr.yml:78-87` is verified to include any new deps (none expected).
- `git log` shows one commit: `test(apply_schemas): papra_index --check offline test (REQ-INB-38)`.
---
### T11 — `test_papra_link_parity_offline.py` `imapclient` runtime shim (v1.3 carry REQ-INB-39) **[test]**
- **Refs:** REQ-INB-39 (v1.3 carry), REQ-INB-25 (papra link parity), REQ-INB-8 (PR CI)
- **Files:** `pipelines/_tests/conftest.py` (new, with the imapclient shim) OR `pipelines/_tests/test_papra_link_parity_offline.py` (add the shim to the test file)
- **Persona:** backend-engineer (primary)
- **needs review from:** devops-engineer
- **Diff size:** S (5-line `sys.modules` shim + a conftest.py scaffold)
- **must_haves:**
- `pipelines/_tests/conftest.py` is a new file containing the imapclient shim: `sys.modules["imapclient"] = types.ModuleType("imapclient"); sys.modules["imapclient"].IMAPClient = MagicMock`. The shim runs at conftest import time (before any test imports `live_pipeline`).
- The shim satisfies the `from imapclient import IMAPClient` chain in `live_pipeline.py:49` without breaking the existing `live_pipeline` semantics (the MagicMock is a no-op; tests that don't touch IMAP are unaffected).
- The v1.0 / v1.1 accepted-precedent disposition is honored: `test_papra_link_parity_offline.py` is no longer `continue-on-error: true` in `ci-pr.yml`; the step is now a hard-fail (the shim makes the import chain resolve cleanly).
- The conftest is also imported by other tests in `pipelines/_tests/` that transitively import `live_pipeline` (the shim is a no-op for them; verified by running the existing 30/30 tests after the shim is added; all 30 still pass).
- `.gitea/workflows/ci-pr.yml` is updated: the `test_papra_link_parity_offline.py` step removes the `continue-on-error: true` and the trailing comment. (The 5 other offline tests added in v1.3 T1.2 stay as-is.)
- `git log` shows one commit: `test(conftest): imapclient runtime-shim — un-defer test_papra_link_parity (REQ-INB-39)`.
---
## Wave 2 — Operator-experience M-effort + container hardening (9 candidates, 9 PRs)
> Wave 2 has 9 tasks that land as 9 PRs. All 9 are M effort; the lead-developer + backend-engineer + data-engineer are the primary owners. The container-hardening task (T9) is devops-engineer primary with security-engineer review.
---
### T1 — `scripts/quickstart.sh` — 15-min green-field operator spin-up (REQ-INB-40) **[cli]**
- **Refs:** REQ-INB-40 (quickstart), REQ-INB-6 (operations runbook)
- **Files:** `scripts/quickstart.sh` (new), `docs/operations-runbook.md` (cross-ref from §3), `docs/quickstart.md` (new, 1-page operator journey)
- **Persona:** backend-engineer (primary, scripts territory)
- **needs review from:** devops-engineer (Docker + Gitea Actions setup)
- **Diff size:** M (1 new script ~80 lines, 1 new doc ~30 lines, 1 runbook cross-ref)
- **must_haves:**
- `scripts/quickstart.sh` is a bash script with `set -euo pipefail` that:
1. Verifies `docker` + `docker compose` are installed
2. Pulls the inboxos images (`docker compose pull`)
3. Generates a `.env` from `.env.example` (with operator-supplied secrets)
4. Brings up the stack (`docker compose up -d`)
5. Waits for NocoDB healthcheck
6. Runs `python scripts/apply_schemas.py` to bootstrap the 4 module tables + `papra_index`
7. Seeds a 1-test `.mbox` and runs `python pipelines/historical_pipeline.py --mbox tests/fixtures/single.mbox` to verify the extract path
8. Prints a "✅ Quickstart complete — operator can now login to http://localhost:8080 (NocoDB)" success line and exit 0
- On any failure, the script prints the failed step + the runbook §X.Y cross-ref to recover. The script does NOT abort the stack on a quickstart failure (the operator can fix and re-run individual steps).
- The script is idempotent: re-running on an already-bootstrapped stack is a no-op (verified by `docker compose ps` returning all services `Up`).
- `docs/quickstart.md` is a 1-page (≈ 30-line) operator journey: "1. Run `scripts/quickstart.sh`. 2. Login to NocoDB. 3. See the seeded view. 4. Read `docs/operator-tutorial.md` (REQ-INB-53) for the full walkthrough." The doc cross-refs the runbook §3 for the per-step details.
- `docs/operations-runbook.md §3` gains a 1-line note: "For a 15-minute green-field spin-up, see `docs/quickstart.md` and `scripts/quickstart.sh`."
- The script honors the v1.0 PII boundary: no third-party SaaS is contacted; only the local Docker stack.
- `chmod +x scripts/quickstart.sh` is set; the script is bash 5.x compatible.
- `git log` shows one commit: `feat(scripts): quickstart.sh + docs/quickstart.md — 15-min green-field spin-up (REQ-INB-40)`.
---
### T2 — NocoDB curated views per module (REQ-INB-41) **[data]**
- **Refs:** REQ-INB-41 (curated views), REQ-INB-5 (module schema)
- **Files:** `scripts/apply_schemas.py` (extend to seed views), `docs/schemas.md` (add a "Curated views" section), `pipelines/_tests/test_curated_views_offline.py` (new)
- **Persona:** data-engineer (primary, NocoDB + data territory)
- **needs review from:** lead-developer
- **Diff size:** M (1 script extension ~60 lines, 1 doc section ~40 lines, 1 new test ~30 lines)
- **must_haves:**
- `scripts/apply_schemas.py` is extended to seed 4 curated NocoDB views (one per module) when run with `--seed-views`:
- **Open invoices 30d** (`finance` view): filter `status = "open" AND due_date <= today + 30d`
- **New contacts 7d** (`crm` view): filter `created_at >= today - 7d`
- **Intel watchlist alerts** (`intel` view): filter `priority = "high"`
- **Promotions opt-out queue** (`promotions` view): filter `opt_out = true`
- The view seed is idempotent: re-running updates the existing views; if a view with the same name exists but a different filter, the script updates the filter (does not create a duplicate).
- `docs/schemas.md` gains a "Curated views" section (≈ 40 lines) documenting the 4 views, their filters, and the SQL-equivalent NocoDB filter syntax. The section also documents the operator workflow: "After running `apply_schemas.py --seed-views`, the views appear in the NocoDB left-nav under each module table."
- `pipelines/_tests/test_curated_views_offline.py` is a new offline test that mocks the NocoDB HTTP layer (using the `FakeNoco` pattern) and asserts that the 4 views are seeded with the correct filters. The test passes.
- `git log` shows one commit: `feat(nocodb): 4 curated views per module + docs (REQ-INB-41)`.
---
### T3 — `scripts/explain_run.py` — structured-log → human summary (REQ-INB-42) **[cli]**
- **Refs:** REQ-INB-42 (explain run), REQ-INB-6 (operations runbook)
- **Files:** `scripts/explain_run.py` (new), `pipelines/_tests/test_explain_run_offline.py` (new)
- **Persona:** backend-engineer (primary)
- **needs review from:** devops-engineer
- **Diff size:** M (1 new script ~80 lines, 1 new test ~40 lines)
- **must_haves:**
- `scripts/explain_run.py` is a Python 3.12 CLI that takes `--run-id <id>` OR `--last` (default: `--last`) and reads the corresponding `_artifacts/<run-id>.jsonl` (or the `last_run.json` cursor for `--last`). The output is a human-readable summary:
- Header: `Run <id> (<run_type>) — started <ts> / finished <ts> / duration <duration>`
- Module breakdown: `finance=12, crm=8, intel=3, promotions=14` (N rows per module)
- Outcome: `MIDs processed=37, upserted=35, skipped=1, errored=1`
- Top 3 errors: each with `MID | error_class | error_message | first_seen_at`
- The script supports `--json` to emit the same data as a JSON document (for piping into other tools).
- The script honors the same artifact path conventions as `pipelines/common.py` (`_artifacts/` + `_metrics/`).
- `pipelines/_tests/test_explain_run_offline.py` covers 3 cases: (a) `--last` reads `last_run.json` and emits a summary, (b) `--run-id` reads the named JSONL, (c) `--json` emits valid JSON. All 3 pass.
- `chmod +x scripts/explain_run.py` and a 1-line shebang `#!/usr/bin/env python3` are set.
- `git log` shows one commit: `feat(scripts): explain_run.py — structured-log → human summary (REQ-INB-42)`.
---
### T4 — `scripts/reclassify.py` — move MID between modules (REQ-INB-44) **[cli]**
- **Refs:** REQ-INB-44 (reclassify), REQ-INB-4 (extraction contract)
- **Files:** `scripts/reclassify.py` (new), `pipelines/_tests/test_reclassify_offline.py` (new)
- **Persona:** backend-engineer (primary)
- **needs review from:** ml-engineer (the re-extract invokes the LLM)
- **Diff size:** M (1 new script ~100 lines, 1 new test ~50 lines)
- **must_haves:**
- `scripts/reclassify.py` is a Python 3.12 CLI that takes `--csv <path>` (CSV with `mid,new_module` rows) OR `--mid <MID> --new-module <module>` (single-MID form). For each row, the script:
1. Reads the existing NocoDB row for the MID
2. Loads the `.eml` from S3 (via the `eml/<kind>/<yyyy>/<mm>/<dd>/<mid>.eml` key)
3. Re-runs `inboxos:extract-<new_module>` against the body
4. PATCHes the NocoDB row in the new module table (and DELETE the row in the old module table)
5. Emits a structured event `reclassify_completed` to the per-run JSONL
- The script supports `--dry-run` (do the extract + classify but do NOT write to NocoDB or S3).
- The script requires `--confirm` to actually write (defense-in-depth: prevents accidental mass-reclassification).
- `pipelines/_tests/test_reclassify_offline.py` covers 3 cases: (a) `--dry-run` does not write, (b) `--csv` with 3 rows processes all 3 (mocked NocoDB + S3), (c) missing `--confirm` exits 1. All 3 pass.
- `chmod +x scripts/reclassify.py` and a 1-line shebang `#!/usr/bin/env python3` are set.
- `git log` shows one commit: `feat(scripts): reclassify.py — move MID between modules (REQ-INB-44)`.
---
### T5 — `scripts/replay.py` — re-queue failed/skipped MIDs (REQ-INB-45) **[cli]**
- **Refs:** REQ-INB-45 (replay), REQ-INB-3 (idempotency)
- **Files:** `scripts/replay.py` (new), `pipelines/_tests/test_replay_offline.py` (new)
- **Persona:** backend-engineer (primary)
- **needs review from:** data-engineer (idempotency)
- **Diff size:** M (1 new script ~100 lines, 1 new test ~50 lines)
- **must_haves:**
- `scripts/replay.py` is a Python 3.12 CLI that takes `--last <N>` (replay the last N MIDs from state) OR `--state <path>` (replay from a specific state file) OR `--status <skipped|errored|all>` (filter by state). For each MID in scope, the script:
1. Loads the original `.eml` from S3
2. Re-runs the full classify + extract + upsert path
3. Updates the state file with the new outcome
- The replay is idempotent: re-running a previously-replayed MID is a no-op (the NocoDB upsert is keyed by Message-ID).
- The script supports `--dry-run` (do the work but do NOT write).
- The script requires `--confirm` to actually write.
- `pipelines/_tests/test_replay_offline.py` covers 3 cases: (a) `--last 5` processes 5 MIDs, (b) `--status errored` filters to errored only, (c) idempotency: replaying a previously-replayed MID is a no-op. All 3 pass.
- `chmod +x scripts/replay.py` and a 1-line shebang `#!/usr/bin/env python3` are set.
- `git log` shows one commit: `feat(scripts): replay.py — re-queue failed/skipped MIDs (REQ-INB-45)`.
---
### T6 — `scripts/dashboard.py` — TUI/CLI status board (REQ-INB-50) **[cli]**
- **Refs:** REQ-INB-50 (dashboard), REQ-INB-6 (operations runbook)
- **Files:** `scripts/dashboard.py` (new), `pipelines/_tests/test_dashboard_offline.py` (new)
- **Persona:** backend-engineer (primary)
- **needs review from:** devops-engineer
- **Diff size:** M (1 new script ~120 lines, 1 new test ~40 lines)
- **must_haves:**
- `scripts/dashboard.py` is a Python 3.12 CLI that takes no args and prints a `rich`-based (or `texttable`-based if `rich` is not available — the script auto-detects) TUI dashboard:
- Header: `InboxOS Operator Dashboard — <current ts>`
- Last live run: `run_id | started_at | finished_at | status | mids_total | mids_errored`
- Last historical run: same shape
- Pending MIDs in state: `state.json` checkpoint line + count of MIDs in `pending`
- Error count 24h: from `pipelines/_metrics/healthcheck.json` (per T5)
- S3 archive size: from `pipelines/_metrics/healthcheck.json`
- NocoDB row counts per module: from NocoDB
- The dashboard supports `--watch` (refresh every 30s; uses `rich.live` if available).
- `pipelines/_tests/test_dashboard_offline.py` covers 2 cases: (a) all artifacts present, full dashboard emits, (b) missing artifacts, dashboard emits partial + warnings. Both pass.
- `chmod +x scripts/dashboard.py` and a 1-line shebang `#!/usr/bin/env python3` are set.
- The script does NOT add a new pip dep (`rich` is detected at runtime; fallback to `texttable`).
- `git log` shows one commit: `feat(scripts): dashboard.py — TUI/CLI status board (REQ-INB-50)`.
---
### T7 — `scripts/daily_digest.py` — run summary email (REQ-INB-47) **[cli]**
- **Refs:** REQ-INB-47 (daily digest), REQ-INB-6 (operations runbook)
- **Files:** `scripts/daily_digest.py` (new), `pipelines/_tests/test_daily_digest_offline.py` (new), `.gitea/workflows/daily-digest.yml` (new), `docs/operations-runbook.md` (cross-ref)
- **Persona:** backend-engineer (primary)
- **needs review from:** devops-engineer (the new workflow)
- **Diff size:** M (1 new script ~80 lines, 1 new test ~40 lines, 1 new workflow ~30 lines, 1 runbook cross-ref)
- **must_haves:**
- `scripts/daily_digest.py` is a Python 3.12 CLI that takes `--date YYYY-MM-DD` (default: yesterday) and reads the per-run JSONL files for the date. Output is a one-line-per-module summary: `2026-06-08: finance=12, crm=8, intel=3, promotions=14, errors=0`.
- The script supports `--email <to>` (sends via `smtplib` to the configured SMTP relay) OR `--print` (default; prints to stdout).
- The SMTP relay is configured via env vars: `SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASS`, `SMTP_FROM`. The script gracefully degrades to `--print` if SMTP is not configured.
- `.gitea/workflows/daily-digest.yml` is a new workflow that runs on `cron 0 6 * * *` (6 AM UTC daily) and invokes `python scripts/daily_digest.py --email operator@example.com`. The workflow's secrets are the 5 SMTP env vars.
- `docs/operations-runbook.md §3` gains a 1-line note: "Daily digest: see `scripts/daily_digest.py` and `.gitea/workflows/daily-digest.yml`."
- `pipelines/_tests/test_daily_digest_offline.py` covers 2 cases: (a) JSONL present for the date, summary emitted, (b) no JSONL for the date, graceful "no runs" message. Both pass.
- `chmod +x scripts/daily_digest.py` and a 1-line shebang `#!/usr/bin/env python3` are set.
- The new workflow's `actions/upload-artifact` is pinned to `@v3` per REQ-INB-12 (Gitea 1.x / GHES compat).
- `git log` shows one commit: `feat(scripts+workflows): daily_digest.py + daily-digest.yml — run summary (REQ-INB-47)`.
---
### T8 — `docs/operator-tutorial.md` — 30-min walkthrough (REQ-INB-53) **[doc]**
- **Refs:** REQ-INB-53 (tutorial), REQ-INB-6 (operations runbook)
- **Files:** `docs/operator-tutorial.md` (new), `README.md` (add a top-level link)
- **Persona:** lead-developer (primary, owns `**/*.md` territory)
- **needs review from:** devops-engineer
- **Diff size:** M (1 new doc ~200 lines, 1 README link)
- **must_haves:**
- `docs/operator-tutorial.md` is a 30-minute end-to-end walkthrough structured as 5 chapters:
1. **Setup** (5 min): `scripts/quickstart.sh` (REQ-INB-40) + login to NocoDB
2. **First run** (5 min): seed a 1-test `.mbox` + run `historical_pipeline.py` + see the row in NocoDB
3. **Day-to-day** (10 min): the live pipeline cron + the dashboard (REQ-INB-50) + `find_email.py` (REQ-INB-43)
4. **Incident response** (5 min): runbook §5 + ERROR-NN catalog (REQ-INB-46) + `replay.py` (REQ-INB-45)
5. **Hardening** (5 min): container hardening (REQ-INB-55) + PII-scrub (REQ-INB-57) + DPIA (REQ-INB-58)
- Each chapter ends with "Next: [link to relevant runbook §X.Y or candidate-script §X.Y]"
- `README.md` gains a 1-line addition in the "Getting started" section: "For a 30-minute walkthrough, see `docs/operator-tutorial.md`."
- `git log` shows one commit: `docs(tutorial): operator-tutorial.md — 30-min walkthrough (REQ-INB-53)`.
---
### T9 — Container hardening: read-only rootfs + cap-drop + no-new-privileges (REQ-INB-55) **[infra]**
- **Refs:** REQ-INB-55 (container hardening), REQ-INB-10 (self-hosted PII boundary)
- **Files:** `docker/docker-compose.yml` (add hardening stanza to each of the 6 services)
- **Persona:** devops-engineer (primary, docker-compose territory)
- **needs review from:** security-engineer (PII boundary review)
- **Diff size:** M (1 file, 6 service blocks updated; ≈ 30-40 lines added)
- **must_haves:**
- `docker/docker-compose.yml` gains a hardening stanza applied to all 6 services (NocoDB, SeaweedFS master, SeaweedFS volume, SeaweedFS filer, Ollama, Papra, Caddy — wherever present):
- `read_only: true` for services that do not need write access (Caddy, Ollama); for services that do need write access (NocoDB SQLite, SeaweedFS volume, Papra), the read-only flag is omitted and a `tmpfs:` mount is added for `/tmp` and `/run` (the canonical "rootless writable" pattern).
- `cap_drop: [ALL]` + `cap_add: [CHOWN, DAC_OVERRIDE, FOWNER, SETUID, SETGID, NET_BIND_SERVICE]` (the minimum cap set for the services to function)
- `security_opt: [no-new-privileges:true]`
- `restart: unless-stopped` is added (operational improvement, not strictly hardening)
- The 6 services are verified to start cleanly with the hardening applied: `docker compose up -d` returns exit 0; `docker compose ps` shows all 6 `Up`; the smoke test (apply_schemas.py + a 1-message historical run) succeeds.
- `docs/operations-runbook.md §6` gains a 1-line note: "Container hardening: all 6 services run with `read_only: true` (or `tmpfs: [/tmp]` for write-required services), `cap_drop: [ALL]`, and `security_opt: no-new-privileges`. See `docker/docker-compose.yml` for the full stanza."
- The hardening does NOT break the existing workflows (live + historical + ci-pr all run end-to-end against the hardened stack).
- `git log` shows one commit: `infra(docker): container hardening — read-only rootfs, cap-drop, no-new-privileges (REQ-INB-55)`.
---
## Wave 3 — Security documentation + PII-scrub + cost report (3 candidates, 3 PRs)
> Wave 3 has 3 tasks that land as 3 PRs. All 3 are M effort; the security-engineer + lead-developer are the primary owners. REQ-INB-57 is the CPO-APPROVAL-REQUIRED candidate; the v1.4 plan proceeds with the proposed policy in `v1.4-candidates.md §3.2 T1-SEC-3` and the CPO can amend the policy at any time via a follow-up PR (no PLAN rewrite needed).
---
### T1 — `docs/security/dpia.md` — Data Protection Impact Assessment (REQ-INB-58) **[doc]**
- **Refs:** REQ-INB-58 (DPIA), REQ-INB-10 (self-hosted PII boundary)
- **Files:** `docs/security/dpia.md` (new), `docs/operations-runbook.md` (cross-ref)
- **Persona:** security-engineer (primary, owns the PII boundary)
- **needs review from:** lead-developer
- **Diff size:** M (1 new doc ~200 lines, 1 runbook cross-ref)
- **must_haves:**
- `docs/security/dpia.md` is a Data Protection Impact Assessment structured per GDPR Art. 35 conventions:
1. **Description of processing**: data flows (IMAP → SeaweedFS → Ollama → NocoDB → Papra), data categories (email body, headers, attachments), data subjects (the operator's correspondents), retention (currently ad-hoc; see §X)
2. **Necessity & proportionality**: why the processing is necessary (the pipeline's mission), the least-intrusive approach (single-Gmail-inbox, local-LLM, no SaaS)
3. **Risk assessment**: per-flow risks (e.g., IMAP credential leak, SeaweedFS bucket misconfiguration, Ollama prompt-injection, NocoDB token leak) with severity × likelihood
4. **Mitigations**: the existing mitigations (REQ-INB-10 self-hosted boundary, REQ-INB-3 idempotency, REQ-INB-12 Gitea 1.x pin, REQ-INB-56 secret-redaction, REQ-INB-57 PII-scrub) and the deferred mitigations (DEF-V1.4-2 encryption-at-rest, DEF-V1.4-3 retention/GDPR-purge, DEF-V1.4-4 audit-log, DEF-V1.4-5 offsite-backup)
5. **Data subject rights**: right-to-access (the operator can query their own data), right-to-erasure (currently manual; DEF-V1.4-3 will automate)
6. **Residual risk**: low for the in-scope PII (single-operator deployment); high for the deferred items
7. **Review cycle**: quarterly (next review 2026-09-08)
- `docs/operations-runbook.md §6` gains a 1-line cross-ref: "DPIA: see `docs/security/dpia.md`."
- `git log` shows one commit: `docs(security): DPIA — GDPR Art. 35 (REQ-INB-58)`.
---
### T2 — `docs/security/threat-model.md` — STRIDE per component (REQ-INB-59) **[doc]**
- **Refs:** REQ-INB-59 (threat model), REQ-INB-10 (self-hosted PII boundary)
- **Files:** `docs/security/threat-model.md` (new), `docs/operations-runbook.md` (cross-ref)
- **Persona:** security-engineer (primary)
- **needs review from:** lead-developer
- **Diff size:** M (1 new doc ~250 lines, 1 runbook cross-ref)
- **must_haves:**
- `docs/security/threat-model.md` is a STRIDE-per-component decomposition covering each component in `.ciagent/inboxos/ARCHITECTURE.md`:
- **Gmail IMAP** (S, T, R, I, D, E)
- **Gitea Actions runners** (S, T, R, I, D, E)
- **Ollama** (S, T, R, I, D, E)
- **SeaweedFS** (S, T, R, I, D, E)
- **NocoDB** (S, T, R, I, D, E)
- **Papra** (S, T, R, I, D, E)
- **Caddy reverse proxy** (S, T, R, I, D, E)
- **FRP tunnel** (S, T, I, D)
- **Prompt files** (T, R, I)
- **Operator CLI scripts** (T, R, I, E)
- For each component × STRIDE cell, the doc names: the threat, the existing mitigation, the residual risk, and the link to the relevant REQ-INB-* or DEF-V1.4-* candidate.
- The doc has a "STRIDE Summary" table at the top: 8 components × 6 STRIDE cells = 48 cells, each marked `Mitigated` / `Partial` / `Open`. The "Open" cells are the deferred candidates (DEF-V1.4-1..5).
- `docs/operations-runbook.md §6` gains a 1-line cross-ref: "Threat model: see `docs/security/threat-model.md`."
- `git log` shows one commit: `docs(security): threat-model.md — STRIDE per component (REQ-INB-59)`.
---
### T3 — PII-scrub body before Ollama (REQ-INB-57) [CPO-APPROVAL-REQUIRED] **[ml + security]**
- **Refs:** REQ-INB-57 (PII scrub), REQ-INB-4 (extraction contract), REQ-INB-10 (self-hosted PII boundary)
- **Files:** `pipelines/pii_scrub.py` (new), `pipelines/common.py` (call `pii_scrub()` before `llm_run_command`), `docs/extraction-contract.md` (document the masking policy), `pipelines/_tests/test_pii_scrub_offline.py` (new), `docs/security/dpia.md` (cross-ref to the masking policy)
- **Persona:** security-engineer (primary, PII boundary) + ml-engineer (LLM contract)
- **needs review from:** lead-developer
- **Diff size:** M (1 new module ~80 lines, 1 common.py integration ~5 lines, 1 doc section ~20 lines, 1 new test ~50 lines)
- **CPO gate:** the masking policy proposed in `v1.4-candidates.md §3.2 T1-SEC-3` is adopted as the default. The CPO can amend via a follow-up PR at any time.
- **must_haves:**
- `pipelines/pii_scrub.py` is a new module that exports a `scrub_pii(body: str) -> tuple[str, PIIMaskSummary]` function. The function applies 4 regex-based maskers in sequence:
1. **SSN**: `\b\d{3}-\d{2}-\d{4}\b` → `[SSN-MASKED]`
2. **Credit card**: `\b(?:\d[ -]*?){13,19}\b` validated against the Luhn algorithm → `[CC-MASKED]`
3. **IBAN**: per-country regex (2-letter country code + 2 check digits + up to 30 alphanumerics) → `[IBAN-MASKED]`
4. **Phone (E.164)**: `\+[1-9]\d{1,14}\b` → `[PHONE-MASKED]`
- The function returns the scrubbed body + a `PIIMaskSummary` dataclass with the count of each PII type masked. The summary is emitted to `_artifacts/pii_scrub.jsonl` per message (per the audit-trail requirement).
- `pipelines/common.py::llm_run_command` calls `scrub_pii(body)` before invoking the LLM. The scrubbed body is what the LLM sees; the raw body is never sent to Ollama.
- The function preserves extraction accuracy per the masking policy: SSN, CC, IBAN, phone are masked (the extract schema captures payment metadata, not raw digits). The `crm` row's "credit-card last-4" field is a derived value from the extract prompt (the LLM is told "the last 4 digits are visible after the mask" via the prompt frontmatter).
- `docs/extraction-contract.md` gains a "PII masking" section documenting the 4 maskers, the regexes, the Luhn check, and the audit-trail format.
- `docs/security/dpia.md §3 (mitigations)` is updated to reference the masking policy as an active mitigation.
- `pipelines/_tests/test_pii_scrub_offline.py` covers 5 cases: (a) SSN detected and masked, (b) CC detected (Luhn-valid) and masked, (c) IBAN detected and masked, (d) phone detected and masked, (e) body with no PII passes through unchanged + empty summary. All 5 pass.
- The audit trail (`_artifacts/pii_scrub.jsonl`) is `.gitignore`-d (operator-local state, not version-controlled).
- The function is conservative: false positives are preferred over false negatives (over-masking is acceptable; under-masking is not). The Luhn check on credit cards ensures non-CC digit sequences (e.g., order numbers) are not masked.
- `git log` shows one commit: `feat(ml+security): pii_scrub.py — mask SSN/CC/IBAN/phone before Ollama (REQ-INB-57)`.
---
## Cross-waves
### Within-wave file-share note
- **Wave 1**: T3 (last_run.json) + T5 (healthcheck.json) both touch `pipelines/common.py` + 2 pipeline files; they are sequenced T3 first, then T5 (the healthcheck reads the last_run cursor).
- **Wave 1**: T10 (papra_index test) + T11 (imapclient shim) both add files to `pipelines/_tests/`; they are independent.
- **Wave 2**: T1 (quickstart.sh) + T2 (NocoDB views) + T3 (explain_run) + T4 (reclassify) + T5 (replay) + T6 (dashboard) + T7 (daily_digest) + T8 (operator-tutorial) + T9 (container hardening) all touch disjoint files. No overlap.
- **Wave 3**: T1 (DPIA) + T2 (threat model) both touch `docs/security/`; independent (different files). T3 (PII scrub) touches `pipelines/pii_scrub.py` + `pipelines/common.py` + `docs/extraction-contract.md`; independent of T1 and T2.
### Vertical-slice check
Each Wave 1 + Wave 2 + Wave 3 task is a single-PR vertical slice: a script / test / doc / infra / security edit that compiles, passes CI, and moves the project measurably toward the v1.4 acceptance gate. No cross-slice dependency: each task is a stand-alone commit that can be reviewed and merged independently.
### Must-have check
- **Wave 1 (11 tasks, 11 PRs):** 8 S/XS-effort UX scripts (T1..T6, T8) + 1 secret-redaction log filter (T7) + 3 v1.3 carries (T9, T10, T11). Closes the v1.3 carry-forwards (REQ-INB-35c, 38, 39) and the lowest-effort v1.4 candidates.
- **Wave 2 (9 tasks, 9 PRs):** 7 M-effort UX deliverables (T1..T8: quickstart, NocoDB views, explain_run, reclassify, replay, daily_digest, dashboard, operator-tutorial) + container hardening (T9). Wave 2 is the bulk of the operator-experience work.
- **Wave 3 (3 tasks, 3 PRs):** DPIA (T1) + threat model (T2) + PII-scrub (T3). Wave 3 is the smallest in count but the highest in audit value.
- **Total: 23 tasks, 23 PRs, 23 commits.**
### Milestone-level must-have
The v1.4 acceptance gate is the 23/23 disposition (20 new + 3 carries ACCEPT, 0 REJECT, 5 DEFER to v1.5+). The gate is met when:
- All 23 tasks land as 23 PRs with 23 well-formed `---ci---` blocks (23 execute + 1 verify = 24 total status blocks for the phase)
- The CPO-review issue (#18) is closed (the CPO can close it at any time; the workflow does not block)
- The 1 CPO-APPROVAL-REQUIRED candidate (REQ-INB-57, T3 in Wave 3) lands with the proposed masking policy; the CPO can amend the policy via a follow-up PR at any time without blocking the v1.4 ship
- The 5 DEFER candidates (DEF-V1.4-1..5) are explicitly filed in `v1.4-candidates.md §9` for v1.5+ — the verify report confirms the file mentions them
- `.ciagent/inboxos/REQUIREMENTS.md` v1.4 section is unchanged at close (the 23 candidates remain `[CANDIDATE]`; the CPO-owned promotion to a real requirement is a CPO action, not an EXECUTE action)
- `docs/requirements.md` and `docs/roadmap.md` are NOT modified by EXECUTE (CPO-owned; the dual-approval gate on PRs is the actual promotion path per the user's v1.4 run context)
---
## Wave 4 — NFR (no code delta)
> Wave 4 is the empty wave. The v1.4 milestone produces no NFR-only chore commit; the EXECUTE stage lands the 23 tasks in Waves 1-3 directly. If the lead-developer finds a small chore commit is needed (e.g. to record the "no schema delta" decision for the audit), the chore commit lands in EXECUTE and is the 24th commit, not a separate wave.
---
## Requirement coverage
| REQ-ID | v1.4 status | Plan task | Disposition |
|--------|-------------|-----------|-------------|
| REQ-INB-40 | `[CANDIDATE]` | W2-T1 | ACCEPT (M) |
| REQ-INB-41 | `[CANDIDATE]` | W2-T2 | ACCEPT (M) |
| REQ-INB-42 | `[CANDIDATE]` | W2-T3 | ACCEPT (M) |
| REQ-INB-43 | `[CANDIDATE]` | W1-T1 | ACCEPT (S) |
| REQ-INB-44 | `[CANDIDATE]` | W2-T4 | ACCEPT (M) |
| REQ-INB-45 | `[CANDIDATE]` | W2-T5 | ACCEPT (M) |
| REQ-INB-46 | `[CANDIDATE]` | W1-T2 | ACCEPT (S) |
| REQ-INB-47 | `[CANDIDATE]` | W2-T7 | ACCEPT (M) |
| REQ-INB-48 | `[CANDIDATE]` | W1-T8 | ACCEPT (S) |
| REQ-INB-49 | `[CANDIDATE]` | W1-T3 | ACCEPT (S) |
| REQ-INB-50 | `[CANDIDATE]` | W2-T6 | ACCEPT (M) |
| REQ-INB-51 | `[CANDIDATE]` | W1-T4 | ACCEPT (S) |
| REQ-INB-52 | `[CANDIDATE]` | W1-T5 | ACCEPT (S) |
| REQ-INB-53 | `[CANDIDATE]` | W2-T8 | ACCEPT (M) |
| REQ-INB-54 | `[CANDIDATE]` | W1-T6 | ACCEPT (S) |
| REQ-INB-55 | `[CANDIDATE]` | W2-T9 | ACCEPT (M) |
| REQ-INB-56 | `[CANDIDATE]` | W1-T7 | ACCEPT (S) |
| REQ-INB-57 | `[CANDIDATE — CPO-APPROVAL-REQUIRED]` | W3-T3 | ACCEPT (M, CPO-gated) |
| REQ-INB-58 | `[CANDIDATE]` | W3-T1 | ACCEPT (M) |
| REQ-INB-59 | `[CANDIDATE]` | W3-T2 | ACCEPT (M) |
| REQ-INB-35c | `[CANDIDATE — carry]` | W1-T9 | ACCEPT (XS) |
| REQ-INB-38 | `[CANDIDATE — carry]` | W1-T10 | ACCEPT (XS) |
| REQ-INB-39 | `[CANDIDATE — carry]` | W1-T11 | ACCEPT (S) |
| REQ-INB-26 (v1.2 carry) | `[REJECTED — out of domain]` | n/a | unchanged from v1.2 |
| REQ-INB-1..39 (v1.0 + v1.1 + v1.3) | `[SHIPPED]` / `[EXECUTED]` | n/a | unchanged |
**Coverage ratio:** 23/23 ACCEPT, 0/23 REJECT. **v1.1 backlog closure:** P2-1 closed by REQ-INB-38, P2-2 closed by REQ-INB-39. **v1.3 backlog closure:** REQ-INB-35 closed by REQ-INB-35c, REQ-INB-38 closed by the same-named v1.4 carry, REQ-INB-39 closed by the same-named v1.4 carry.
---
## Plan-execution prerequisites
- [x] Active project confirmed as `inboxos` (per `.ciagent/config.json` `active_projects[0]`)
- [x] Active milestone confirmed as `v1.4`
- [x] Research commit recorded: `8ff4e31`
- [x] Ideate commit recorded: `ad45e76` (23 candidates, 20 new + 3 carries)
- [x] Requirements registration commit recorded: same (REQUIREMENTS.md v1.4 section)
- [x] Gitea issue #18 opened for CPO review: https://git.cloudinit.dev/jc-capital/inboxos/issues/18
- [x] No paperclip monorepo ad-hoc changes (operator directive honored)
- [x] Cross-references verified: REQ-INB-40..59, REQ-INB-35c, 38, 39 in `REQUIREMENTS.md` v1.4 section
- [x] PERSONAS.md territory is current (v1.0 research, no v1.4 changes needed; the 6 active personas cover all 23 tasks)
---
## Plan deliverables
This plan produces:
- **23 PRs** (1 per task: W1-T1..T11, W2-T1..T9, W3-T1..T3)
- **23 commits** (1 per task; no multi-file tasks beyond the runbook renumber precedent)
- **~40-50 unique file touches** (12 new files, ~30 modified files)
- **0 new NocoDB schema** (no schema changes)
- **~12-15 new tests** (1 per script / module / hardening task)
- **0 new pip deps** (`rich` is detected at runtime; `texttable` is the fallback)
The plan's net diff is **~1500-2500 lines** across ~40-50 files, dominated by the 5 new scripts (W1-T1, T4, T6, T8, W2-T1, T3, T4, T5, T6, T7 — ~100 lines each), 2 new docs (W2-T8 operator-tutorial, W3-T1 DPIA, W3-T2 threat-model — ~200 lines each), the PII-scrub module (W3-T3, ~80 lines + ~20-line doc edit), and the container-hardening stanza (W2-T9, ~30 lines across 6 service blocks).
---
## Risk register
| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| The PII-scrub masking (W3-T3) regresses extraction accuracy on real data | medium | high | The proposed policy masks SSN/CC/IBAN/phone but the extract schema captures payment metadata (last-4, expiry) not raw digits. The verify report includes an extraction-accuracy check on the existing offline test fixtures (which contain synthetic PII). If the regression is real, the CPO amends the policy via a follow-up PR. |
| The container hardening (W2-T9) breaks an existing service (e.g., NocoDB SQLite write) | medium | medium | The hardening applies `read_only: true` only to services that do not need write access (Caddy, Ollama); for write-required services, the read-only flag is omitted and a `tmpfs: [/tmp]` mount is added. The verify report runs the smoke test (apply_schemas.py + 1-message historical run) against the hardened stack. |
| The 23 new PRs exceed the v1.1 / v1.2 / v1.3 8-PR precedent (v1.4 has 23 PRs) | low | low | The v1.4 milestone is the largest in the inboxos history; the PR count is the natural consequence of the 23-candidate set. The lead-developer can collapse PRs at PLAN time if desired; the v1.4 plan honors the per-task-PR pattern for now. |
| The Gitea Actions 1.x `upload-artifact@v3` pin is forgotten on the new `daily-digest.yml` workflow (W2-T7) | low | low | The implementer verifies the pin in the new workflow matches the existing 3 workflows' pin. The verify report grep-asserts all 4 workflows use `upload-artifact@v3`. |
| The cross-references between the new docs (DPIA, threat model, operator-tutorial, extraction-contract §PII) drift between EXECUTE and the v1.5 close | low | low | The verify report grep-asserts the cross-refs are present and point to the canonical files. A v1.5 candidate (defensive doc-hygiene) can re-verify. |
| A script in Wave 1 (T1..T8) has a hidden dep (e.g., `rich`) that the offline test host does not have | medium | medium | The implementer verifies each script's import surface against the existing `pip install` block at `ci-pr.yml:78-87`. New deps are added to the `pip install` block. `dashboard.py` (W2-T6) auto-detects `rich` at runtime and falls back to `texttable`; no new dep. |
| The CPO closes Gitea issue #18 with "decline" (e.g., REQ-INB-57 is rejected) | low | medium | The user's v1.4 run context says the workflow proceeds regardless. The CPO can decline a specific REQ via a follow-up PR that reverts that task. The milestone ship is not blocked. |
| The lead-developer / CPO amend a candidate post-EXECUTE (e.g., change the masking policy, change the curated view filters) | low | low | The plan is the integrator's contract; post-EXECUTE amendments are follow-up PRs on `main` that do not change the v1.4.0 tag. The audit trail records the amendments. |
---
## Hand-off
- **EXECUTE phase:** runs on the 23 ACCEPT candidate set. Produces 23 PRs / 23 commits on `phase/01-v1.4-execute`. The execution order within `phase/01-v1.4-execute` is: Wave 1 first (11 small-effort tasks; lowest risk; closes the v1.3 carries), then Wave 2 (9 M-effort tasks; bulk of the operator-experience work), then Wave 3 (3 security-doc + PII-scrub tasks; highest audit value). Within each wave, the order is the task-list order. The total is 23 PRs.
- **VERIFY phase:** verifies the 23 ACCEPT candidates close cleanly — the 4 verification layers (structural, behavioral, security, quality) are documented in `.ciagent/inboxos/verify/v1.4-PHASE1-REPORT.md`. The verify report grep-asserts: (a) the 12 new files exist, (b) the 5 new workflows' YAML is valid, (c) the 12-15 new offline tests pass, (d) the 5 DEFER items are in `v1.4-candidates.md §9`, (e) the cross-refs in the new docs are present and point to the canonical files, (f) the container hardening does not break the smoke test, (g) the PII-scrub masks the 4 PII types and the audit trail is written.
- **REVIEW phase:** multi-persona code review across the 23 PRs. The review scope is the diff (~1500-2500 lines across ~40-50 files) and the 12-15 new tests. All 6 active personas participate (frontend-engineer is deactivated). The review verdict is PASS-with-NITS (the v1.1 / v1.2 / v1.3 review verdicts).
- **AUDIT phase:** verify project health. The audit reconstruction pass: 23 EXECUTE commits + 1 verify + 1 review = 25 status blocks on `milestone/v1.4-initial`. The file-discipline pass: all 23 v1.4 candidate files present in `REQUIREMENTS.md` v1.4 section; all 23 plan tasks have a `must_haves` array; the 5 DEFER candidates are in `v1.4-candidates.md §9`. The commit-discipline pass: 23 EXECUTE commits have well-formed `---ci---` blocks; all 23 follow `<type>(<scope>):`. The CPO-files pass: `docs/requirements.md` and `docs/roadmap.md` byte-identical to the v1.3 close (the user's v1.4 run context says the workflow proceeds regardless, but the audit-trail byte-identity check is preserved). The audit verdict is PASS.
- **COMPLETE phase:** closes the v1.4 inboxos milestone on the 23/23 disposition. Tag `v1.4.0`. Release notes record the 23-candidate closure + 5-candidate defer. Merge to `main` per the v1.1 / v1.2 / v1.3 pattern: FF-merge from `phase/01-v1.4-execute` to `milestone/v1.4-initial` to `main`, push tag, push milestone branch, create Gitea release. The v1.4.0 release notes title is "v1.4 — Operator UX + Product-Grade Security (23 candidates executed, 5 deferred to v1.5+)" and the body lists the 23 ACCEPT + 5 DEFER + 0 REJECT.
---
## Files updated by this plan
- **New:** `.ciagent/inboxos/plans/01-PLAN-v1.4.md` (this file)
- **Filename choice:** the v1.1 plan lives at `plans/01-PLAN.md`; the v1.2 plan at `plans/01-PLAN-v1.2.md`; the v1.3 plan at `plans/01-PLAN-v1.3.md`; the v1.4 plan is written to `plans/01-PLAN-v1.4.md` to preserve the audit trail.
- **Touched by EXECUTE (≈ 40-50 unique files):**
- 12 new files: 9 new scripts (`scripts/quickstart.sh`, `find_email.py`, `explain_run.py`, `reclassify.py`, `replay.py`, `dashboard.py`, `daily_digest.py`, `papra_open.py`, `dry_run.py`, `cost_report.py` — actually 10, not 9), `docs/quickstart.md`, `docs/operator-tutorial.md`, `docs/security/dpia.md`, `docs/security/threat-model.md`, `pipelines/pii_scrub.py`, `pipelines/_tests/conftest.py`, `.gitea/workflows/daily-digest.yml`
- ~12-15 new tests in `pipelines/_tests/test_X_offline.py`
- `docker/docker-compose.yml` (container hardening)
- `.gitea/workflows/{live,historical,ci-pr}.yml` (secret-redaction mask steps + ci-pr test wiring update)
- `pipelines/common.py` (last_run.json + healthcheck.json helpers + pii_scrub() call)
- `pipelines/live_pipeline.py` + `pipelines/historical_pipeline.py` (last_run + healthcheck calls)
- `pipelines/_artifacts/` + `pipelines/_metrics/` (new artifacts; .gitignore-d)
- `scripts/apply_schemas.py` (curated views seeding)
- `docs/operations-runbook.md` (error catalog + healthcheck cross-ref + DPIA cross-ref + threat-model cross-ref + daily_digest cross-ref + quickstart cross-ref)
- `docs/extraction-contract.md` (PII masking section)
- `docs/schemas.md` (curated views section)
- `README.md` (operator-tutorial link)
- `.gitignore` (REQ-INB-35c edit, Option A) or `opencode.json` (Option B)
- `.env.example` (secret-mask note)
- **Untouched:** all inboxos source code that the v1.4 candidates do not require (no changes to `common.py::llm_run_command`'s core logic — only the new `pii_scrub()` call is added; the prompt files in `.opencode/commands/` are unchanged), the existing 5 prompts, the existing 11 v1.1 offline tests (already wired in v1.3), the existing 4 module schemas, the existing SeaweedFS / NocoDB / Papra / Ollama Docker images.
- **Untouched:** `.ciagent/inboxos/plans/01-PLAN.md` (v1.1 plan, preserved), `.ciagent/inboxos/plans/01-PLAN-v1.2.md` (v1.2 plan, preserved), `.ciagent/inboxos/plans/01-PLAN-v1.3.md` (v1.3 plan, preserved).
- **Untouched:** `docs/requirements.md` and `docs/roadmap.md` (CPO-owned; the user's v1.4 run context confirms the workflow proceeds without `docs/requirements.md` updates; the dual-approval gate on PRs is the actual promotion path).