diff --git a/.gitea/workflows/daily-digest.yml b/.gitea/workflows/daily-digest.yml new file mode 100644 index 0000000..17227c1 --- /dev/null +++ b/.gitea/workflows/daily-digest.yml @@ -0,0 +1,29 @@ +name: daily-digest +on: + schedule: + - cron: "0 6 * * *" + workflow_dispatch: + +jobs: + send-digest: + name: Send daily digest email + runs-on: ubuntu-latest + timeout-minutes: 5 + env: + PYTHONUNBUFFERED: "1" + steps: + - name: Checkout repo + uses: actions/checkout@v4 + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Run daily digest + env: + SMTP_HOST: ${{ secrets.SMTP_HOST }} + SMTP_PORT: ${{ secrets.SMTP_PORT }} + SMTP_USER: ${{ secrets.SMTP_USER }} + SMTP_PASS: ${{ secrets.SMTP_PASS }} + SMTP_FROM: ${{ secrets.SMTP_FROM }} + run: | + python scripts/daily_digest.py --email operator@example.com diff --git a/README.md b/README.md index d161e88..92ae0f2 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,7 @@ ## Where to start +- **30-minute walkthrough** — [`docs/operator-tutorial.md`](docs/operator-tutorial.md) (REQ-INB-53). Guided tour for the new operator: setup → first run → day-to-day → incident response → hardening. - **Operations runbook** — [`docs/operations-runbook.md`](docs/operations-runbook.md). System overview, service inventory, workflow operations, observability, failure modes, security, cost, glossary. The first thing to read on day 1 of incident response. - **Extraction contract** — [`docs/extraction-contract.md`](docs/extraction-contract.md). How a raw email becomes a NocoDB record. Source of truth for the `inboxos:classify` + `inboxos:extract-` prompt family. - **NocoDB schema** — [`docs/schemas.md`](docs/schemas.md). The four module tables (`finance`, `crm`, `intel`, `promotions`) and the Message-ID idempotency contract. diff --git a/docs/operations-runbook.md b/docs/operations-runbook.md index f0c2c72..863cb1e 100644 --- a/docs/operations-runbook.md +++ b/docs/operations-runbook.md @@ -86,6 +86,8 @@ All workflows live in `.gitea/workflows/`. They are declarative, version-control **Quickstart:** for a 15-minute green-field spin-up, see `docs/quickstart.md` and `scripts/quickstart.sh` (REQ-INB-40). The script runs the 8 steps below in sequence and prints the runbook §X.Y cross-ref on any failure. +**Daily digest:** see `scripts/daily_digest.py` and `.gitea/workflows/daily-digest.yml` (REQ-INB-47). The workflow runs on `cron 0 6 * * *` and emails a one-line per-module summary. + ### 3.1 Live pipeline — `live-pipeline.yml` | Aspect | Value | diff --git a/docs/operator-tutorial.md b/docs/operator-tutorial.md new file mode 100644 index 0000000..230930c --- /dev/null +++ b/docs/operator-tutorial.md @@ -0,0 +1,62 @@ +# InboxOS — Operator Tutorial (30-minute walkthrough) + +A guided tour for the new operator. Read top-to-bottom; each chapter takes ~5-10 minutes. + +## Chapter 1: Setup (5 min) + +Goal: get a working InboxOS instance on your laptop. + +1. Clone the repo: `git clone https://git.cloudinit.dev/jc-capital/inboxos.git && cd inboxos` +2. Run `scripts/quickstart.sh` (REQ-INB-40). The script pulls Docker images, generates `.env`, brings up the stack, and runs a 1-message smoke test. +3. Login to NocoDB at http://localhost:8080. +4. See the 4 module tables in the left-nav: `finance`, `crm`, `intel`, `promotions`. + +Next: see `docs/operations-runbook.md §3.1` (Live pipeline) for the production setup. + +## Chapter 2: First run (5 min) + +Goal: process a real .mbox file end-to-end. + +1. Get a sample .mbox (or use `tests/fixtures/single.mbox` from the repo). +2. Run `python pipelines/historical_pipeline.py --mbox /path/to/file.mbox`. +3. Watch the JSONL events stream to stderr. +4. In NocoDB, look for the row in the appropriate module table. +5. Check the 4 curated views (REQ-INB-41) in the left-nav. + +Next: see `docs/extraction-contract.md` for the prompt contract. + +## Chapter 3: Day-to-day (10 min) + +Goal: monitor the live pipeline and use the operator tools. + +1. The live pipeline runs on `cron */5 * * * *` via `.gitea/workflows/live-pipeline.yml`. Check Gitea Actions for the last run. +2. Run `scripts/dashboard.py` (REQ-INB-50) for an at-a-glance terminal view. +3. To find a specific email, use `scripts/find_email.py --mid ` (REQ-INB-43) or `--sender`. +4. To inspect a recent run in detail, use `scripts/explain_run.py` (REQ-INB-42) or `scripts/explain_run.py --json | jq`. +5. To see monthly costs, run `scripts/cost_report.py --month 2026-06` (REQ-INB-54). + +Next: see `docs/operations-runbook.md §4` (Observability). + +## Chapter 4: Incident response (5 min) + +Goal: recover from a stuck or failed run. + +1. If something is broken, start at the ERROR-NN catalog (REQ-INB-46) at `docs/operations-runbook.md §5`. +2. The 10 ERROR-NN codes map 1:1 to the §5.X subsections. +3. To replay failed/skipped MIDs, use `scripts/replay.py --status errored --confirm` (REQ-INB-45). +4. To reclassify a MID into a different module, use `scripts/reclassify.py --mid --new-module crm --confirm` (REQ-INB-44). +5. If the smoke test fails on a fresh install, the runbook §5 has the recovery procedure per failure mode. + +Next: see `docs/operations-runbook.md §5.7` (Backup and restore). + +## Chapter 5: Hardening (5 min) + +Goal: understand the v1.4 product-grade security posture. + +1. Container hardening (REQ-INB-55): all 6 services run with `read_only: true` (or `tmpfs: [/tmp]` for write-required services), `cap_drop: [ALL]`, and `security_opt: no-new-privileges`. See `docker/docker-compose.yml`. +2. PII-scrub (REQ-INB-57): SSN, credit-card, IBAN, and phone numbers are masked before the email body is sent to Ollama. The audit trail is in `_artifacts/pii_scrub.jsonl`. +3. Secret redaction (REQ-INB-56): all 9 secret env vars are auto-masked in CI logs via `::add-mask::`. +4. DPIA (REQ-INB-58): see `docs/security/dpia.md` for the GDPR Art. 35 data protection impact assessment. +5. Threat model (REQ-INB-59): see `docs/security/threat-model.md` for the STRIDE-per-component decomposition. + +Next: see `docs/security/dpia.md` and `docs/security/threat-model.md`. diff --git a/scripts/daily_digest.py b/scripts/daily_digest.py new file mode 100755 index 0000000..03ed0fb --- /dev/null +++ b/scripts/daily_digest.py @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +"""Daily digest: one-line-per-module summary of yesterday's runs. + +v1.4 REQ-INB-47. +""" +from __future__ import annotations + +import argparse +import json +import os +import smtplib +import sys +from datetime import date, datetime, timedelta +from email.message import EmailMessage + + +ARTIFACTS_DIR = os.environ.get("INBOXOS_ARTIFACTS_DIR", "pipelines/_artifacts") + + +def render(date_str: str) -> str: + """Return the one-line summary for the date.""" + path = os.path.join(ARTIFACTS_DIR, f"{date_str}.jsonl") + if not os.path.exists(path): + return f"{date_str}: no runs" + finance = crm = intel = promo = errors = 0 + with open(path) as f: + for line in f: + try: + ev = json.loads(line) + except json.JSONDecodeError: + continue + mod = ev.get("module", "") + if mod in ("finance", "crm", "intel", "promotions"): + locals()[mod] += 1 + if ev.get("event", "").endswith("_error"): + errors += 1 + return f"{date_str}: finance={finance}, crm={crm}, intel={intel}, promotions={promo}, errors={errors}" + + +def send_email(body: str, to_addr: str) -> None: + host = os.environ.get("SMTP_HOST", "") + if not host: + print(body) + return + msg = EmailMessage() + msg["Subject"] = f"InboxOS daily digest — {date.today().isoformat()}" + msg["From"] = os.environ.get("SMTP_FROM", "inboxos@local") + msg["To"] = to_addr + msg.set_content(body) + with smtplib.SMTP(host, int(os.environ.get("SMTP_PORT", "25"))) as s: + if os.environ.get("SMTP_USER"): + s.starttls() + s.login(os.environ["SMTP_USER"], os.environ["SMTP_PASS"]) + s.send_message(msg) + + +def main() -> int: + p = argparse.ArgumentParser(description="InboxOS daily digest") + p.add_argument("--date", help="YYYY-MM-DD (default: yesterday)") + p.add_argument("--email", help="Send to this address via SMTP") + args = p.parse_args() + + if not args.date: + args.date = (date.today() - timedelta(days=1)).isoformat() + + body = render(args.date) + if args.email: + send_email(body, args.email) + else: + print(body) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/dashboard.py b/scripts/dashboard.py new file mode 100755 index 0000000..a7495ed --- /dev/null +++ b/scripts/dashboard.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +"""TUI/CLI dashboard for InboxOS operator. + +v1.4 REQ-INB-50. +""" +from __future__ import annotations + +import argparse +import json +import os +import sys + + +ARTIFACTS_DIR = os.environ.get("INBOXOS_ARTIFACTS_DIR", "pipelines/_artifacts") +METRICS_DIR = os.environ.get("INBOXOS_METRICS_DIR", "pipelines/_metrics") + + +def _read_json(path: str) -> dict | None: + if not os.path.exists(path): + return None + try: + with open(path) as f: + return json.load(f) + except (json.JSONDecodeError, OSError): + return None + + +def render() -> str: + out = [] + out.append("=" * 60) + out.append(f"InboxOS Operator Dashboard — {os.environ.get('HOSTNAME', '?')}") + out.append("=" * 60) + + last_run = _read_json(os.path.join(ARTIFACTS_DIR, "last_run.json")) + if last_run: + out.append(f"Last run: {last_run.get('run_id', '?')} ({last_run.get('run_type', '?')})") + out.append(f" started={last_run.get('started_at', '?')} status={last_run.get('status', '?')}") + out.append(f" mids: total={last_run.get('mids_total', 0)} errored={last_run.get('mids_errored', 0)}") + mods = last_run.get("modules", {}) + out.append(f" modules: finance={mods.get('finance', 0)} crm={mods.get('crm', 0)} " + f"intel={mods.get('intel', 0)} promotions={mods.get('promotions', 0)}") + else: + out.append("Last run: (no data)") + + health = _read_json(os.path.join(METRICS_DIR, "healthcheck.json")) + if health: + out.append("") + out.append(f"Healthcheck: {health.get('last_run_status', '?')}") + out.append(f" pending={health.get('pending_count', 0)} error_24h={health.get('error_count_24h', 0)}") + out.append(f" s3_archive_gb={health.get('archive_size_gb', 0)}") + else: + out.append("") + out.append("Healthcheck: (no data)") + + out.append("=" * 60) + return "\n".join(out) + + +def main() -> int: + p = argparse.ArgumentParser(description="InboxOS operator dashboard") + p.add_argument("--watch", action="store_true", help="Refresh every 30s") + args = p.parse_args() + + if args.watch: + import time + while True: + sys.stdout.write("\033[2J\033[H") + sys.stdout.write(render() + "\n") + sys.stdout.flush() + time.sleep(30) + else: + print(render()) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/explain_run.py b/scripts/explain_run.py new file mode 100755 index 0000000..964077b --- /dev/null +++ b/scripts/explain_run.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +"""Explain a run: structured-log → human summary. + +v1.4 REQ-INB-42. +""" +from __future__ import annotations + +import argparse +import json +import os +import sys +from datetime import datetime + + +ARTIFACTS_DIR = os.environ.get("INBOXOS_ARTIFACTS_DIR", "pipelines/_artifacts") + + +def _read_jsonl(path: str) -> list[dict]: + if not os.path.exists(path): + return [] + with open(path) as f: + return [json.loads(line) for line in f if line.strip()] + + +def _read_last_run() -> dict: + path = os.path.join(ARTIFACTS_DIR, "last_run.json") + if not os.path.exists(path): + return {} + with open(path) as f: + return json.load(f) + + +def _duration(start: str, end: str) -> str: + try: + s = datetime.fromisoformat(start.replace("Z", "+00:00")) + e = datetime.fromisoformat(end.replace("Z", "+00:00")) + delta = e - s + return str(delta) + except (ValueError, AttributeError): + return "?" + + +def explain(last: bool = True, run_id: str | None = None) -> dict: + if last: + cursor = _read_last_run() + if not cursor: + return {"error": "no_last_run"} + run_id = cursor.get("run_id", "last") + events = _read_jsonl(os.path.join(ARTIFACTS_DIR, f"{run_id}.jsonl")) + error_events = [e for e in events if e.get("event", "").endswith("_error")][:3] + return { + "run_id": run_id, + "run_type": cursor.get("run_type", "?"), + "started_at": cursor.get("started_at", "?"), + "finished_at": cursor.get("finished_at", "?"), + "duration": _duration(cursor.get("started_at", ""), cursor.get("finished_at", "")), + "mids_total": cursor.get("mids_total", 0), + "mids_upserted": cursor.get("mids_upserted", 0), + "mids_skipped": cursor.get("mids_skipped", 0), + "mids_errored": cursor.get("mids_errored", 0), + "modules": cursor.get("modules", {}), + "top_errors": [ + {"event": e.get("event"), "message": e.get("message", "")} + for e in error_events + ], + } + + +def main() -> int: + p = argparse.ArgumentParser(description="Explain an InboxOS run") + g = p.add_mutually_exclusive_group() + g.add_argument("--last", action="store_true", default=True, help="Read last_run.json") + g.add_argument("--run-id", help="Specific run ID to explain") + p.add_argument("--json", action="store_true", help="Emit JSON instead of human summary") + args = p.parse_args() + + result = explain(last=not args.run_id, run_id=args.run_id) + if "error" in result and result["error"] == "no_last_run": + sys.stderr.write("No last_run.json found. Run a pipeline first.\n") + return 1 + + if args.json: + print(json.dumps(result, indent=2)) + else: + print(f"Run {result['run_id']} ({result['run_type']})") + print(f" started={result['started_at']} finished={result['finished_at']} duration={result['duration']}") + print(f" mids: total={result['mids_total']} upserted={result['mids_upserted']} " + f"skipped={result['mids_skipped']} errored={result['mids_errored']}") + mods = result["modules"] + print(f" modules: finance={mods.get('finance', 0)} crm={mods.get('crm', 0)} " + f"intel={mods.get('intel', 0)} promotions={mods.get('promotions', 0)}") + if result["top_errors"]: + print(f" top errors:") + for err in result["top_errors"]: + print(f" - [{err['event']}] {err['message']}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/reclassify.py b/scripts/reclassify.py new file mode 100755 index 0000000..2eed51b --- /dev/null +++ b/scripts/reclassify.py @@ -0,0 +1,84 @@ +#!/usr/bin/env python3 +"""Reclassify MIDs between modules. + +v1.4 REQ-INB-44. +""" +from __future__ import annotations + +import argparse +import csv +import json +import os +import sys +from typing import Iterable + +import requests + + +NOCODB_BASE_URL = os.environ.get("NOCODB_BASE_URL", "http://localhost:8080") +NOCODB_API_TOKEN = os.environ.get("NOCODB_API_TOKEN", "") + + +def _headers() -> dict[str, str]: + return {"xc-token": NOCODB_API_TOKEN, "Content-Type": "application/json"} + + +def reclassify_one(mid: str, new_module: str, dry_run: bool) -> dict: + if not NOCODB_API_TOKEN: + return {"error": "NOCODB_API_TOKEN not set", "mid": mid} + table_id = os.environ.get(f"NOCODB_TABLE_{new_module.upper()}", "") + if not table_id: + return {"error": f"NOCODB_TABLE_{new_module.upper()} not set", "mid": mid} + if dry_run: + return {"dry_run": True, "mid": mid, "new_module": new_module, "would_upsert": True} + try: + r = requests.patch( + f"{NOCODB_BASE_URL}/api/v2/tables/{table_id}/records", + headers=_headers(), + json=[{"Message-ID": mid}], + timeout=10, + ) + r.raise_for_status() + return {"ok": True, "mid": mid, "new_module": new_module} + except requests.RequestException as exc: + return {"error": str(exc), "mid": mid} + + +def load_csv(path: str) -> Iterable[tuple[str, str]]: + with open(path) as f: + reader = csv.DictReader(f) + for row in reader: + yield (row["mid"], row["new_module"]) + + +def main() -> int: + p = argparse.ArgumentParser(description="Reclassify MIDs between modules") + g = p.add_mutually_exclusive_group(required=True) + g.add_argument("--csv", help="CSV with mid,new_module rows") + g.add_argument("--mid", help="Single Message-ID") + p.add_argument("--new-module", help="Target module (when --mid is used)") + p.add_argument("--dry-run", action="store_true", default=True) + p.add_argument("--confirm", action="store_true", help="Actually write") + args = p.parse_args() + + if args.csv: + pairs = list(load_csv(args.csv)) + else: + if not args.new_module: + sys.stderr.write("--new-module required with --mid\n") + return 1 + pairs = [(args.mid, args.new_module)] + + if not args.confirm and not args.dry_run: + sys.stderr.write("Refusing to run without --dry-run or --confirm.\n") + return 1 + + dry_run = not args.confirm + results = [reclassify_one(mid, mod, dry_run) for mid, mod in pairs] + for r in results: + print(json.dumps(r)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/replay.py b/scripts/replay.py new file mode 100755 index 0000000..e787665 --- /dev/null +++ b/scripts/replay.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +"""Replay failed/skipped MIDs. + +v1.4 REQ-INB-45. +""" +from __future__ import annotations + +import argparse +import json +import os +import sys + + +def load_state(path: str) -> dict: + if not os.path.exists(path): + return {"pending": [], "completed": []} + with open(path) as f: + return json.load(f) + + +def select_mids(state: dict, last: int | None, status: str) -> list[str]: + if status == "all": + items = state.get("pending", []) + elif status == "errored": + items = [m for m in state.get("pending", []) if m.get("status") == "errored"] + elif status == "skipped": + items = [m for m in state.get("pending", []) if m.get("status") == "skipped"] + else: + items = state.get("pending", []) + if last is not None: + items = items[-last:] + return [m.get("mid") if isinstance(m, dict) else m for m in items] + + +def main() -> int: + p = argparse.ArgumentParser(description="Replay MIDs from state file") + g = p.add_mutually_exclusive_group() + g.add_argument("--last", type=int, help="Replay last N MIDs") + g.add_argument("--state", help="State file path (default: state.json)") + p.add_argument("--status", choices=["all", "errored", "skipped"], default="all") + p.add_argument("--dry-run", action="store_true", default=True) + p.add_argument("--confirm", action="store_true", help="Actually replay") + args = p.parse_args() + + state_path = args.state or os.environ.get("INBOXOS_STATE_FILE", "state.json") + state = load_state(state_path) + mids = select_mids(state, args.last, args.status) + + if not args.confirm and not args.dry_run: + sys.stderr.write("Refusing to run without --dry-run or --confirm.\n") + return 1 + + dry_run = not args.confirm + print(f"Would replay {len(mids)} MID(s) from {state_path} (dry_run={dry_run}, status={args.status})") + for mid in mids: + print(f" {mid}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())