inb-120: surface historical-pipeline early-stop in Gitea workflow status #6

2026-06-08T14:21:07Z

Ghost commented

2026-06-08 14:21:07 +00:00

Summary

Fixes the misleading Gitea Actions status on a partially-processed historical run.

pipelines/historical_pipeline.py previously returned 0 unconditionally from main(), so a run that stopped on extract_permanent_failure (e.g. an Ollama 504 Gateway Timeout) still surfaced in the Gitea Actions list as conclusion: success. Run #929 on 2026-06-08 processed 6 messages of an estimated 3000-5000, hit the failure path, broke out, returned 0, and Gitea marked the run as a green tick — the most urgent operator UX issue for the historical backfill.

What changed

New module constant EARLY_STOP_EXIT_CODE = 2 in pipelines/historical_pipeline.py.
New loop_exit_reason: str | None in main(); set to "extract_permanent_failure" on the failure break, left as None on clean EOF and on the LIMIT_OVERRIDE branch (which is an operator-requested clean stop, not a failure).
main() returns EARLY_STOP_EXIT_CODE when loop_exit_reason is not None, otherwise 0.
New historical_run_early_stop structured-log event with reason and exit_code so the operator dashboard can surface the cause.
The existing 12h cap (timeout-minutes: 720 on the workflow) is untouched; it still surfaces as a cancelled job.
New test file pipelines/_tests/test_historical_pipeline_early_stop.py with 6 unittest.TestCase tests:
- test_early_stop_on_extract_permanent_failure_returns_nonzero — the regression.
- test_clean_eof_returns_zero — sanity check on the happy path.
- test_clean_eof_after_processing_returns_zero — sanity check after N messages.
- test_limit_reached_returns_zero — LIMIT_OVERRIDE is a clean exit.
- test_early_stop_exit_code_is_nonzero_and_distinct — guard the contract value (must be non-zero and not 1).
- test_early_stop_emits_diagnostic_event — the operator-dashboard signal.

Acceptance

python3 -m unittest discover -s pipelines/_tests -p test_*.py exits 0 with 16/16 tests passing (was 10/10 before this PR; +6 from the new file).

Post-merge follow-up

Dispatch a fresh historical-pipeline workflow run and confirm the Gitea status flips to failure (not success) when the run hits the same Ollama 504 condition. The 12h cap and idempotent offset checkpoint are unchanged; resume behavior is preserved.

Issue

INB-120 / INB-119 — Surface historical-pipeline early-stop in Gitea workflow status
Tracked: INB-7 (Phase 5 — Full Deployment) is blocked by this fix being merged.

## Summary Fixes the misleading Gitea Actions status on a partially-processed historical run. `pipelines/historical_pipeline.py` previously returned 0 unconditionally from `main()`, so a run that stopped on `extract_permanent_failure` (e.g. an Ollama 504 Gateway Timeout) still surfaced in the Gitea Actions list as `conclusion: success`. Run #929 on 2026-06-08 processed 6 messages of an estimated 3000-5000, hit the failure path, broke out, returned 0, and Gitea marked the run as a green tick — the most urgent operator UX issue for the historical backfill. ## What changed - New module constant `EARLY_STOP_EXIT_CODE = 2` in `pipelines/historical_pipeline.py`. - New `loop_exit_reason: str | None` in `main()`; set to `"extract_permanent_failure"` on the failure break, left as `None` on clean EOF and on the `LIMIT_OVERRIDE` branch (which is an operator-requested clean stop, not a failure). - `main()` returns `EARLY_STOP_EXIT_CODE` when `loop_exit_reason is not None`, otherwise 0. - New `historical_run_early_stop` structured-log event with `reason` and `exit_code` so the operator dashboard can surface the cause. - The existing 12h cap (`timeout-minutes: 720` on the workflow) is untouched; it still surfaces as a cancelled job. - New test file `pipelines/_tests/test_historical_pipeline_early_stop.py` with 6 `unittest.TestCase` tests: - `test_early_stop_on_extract_permanent_failure_returns_nonzero` — the regression. - `test_clean_eof_returns_zero` — sanity check on the happy path. - `test_clean_eof_after_processing_returns_zero` — sanity check after N messages. - `test_limit_reached_returns_zero` — `LIMIT_OVERRIDE` is a clean exit. - `test_early_stop_exit_code_is_nonzero_and_distinct` — guard the contract value (must be non-zero and not 1). - `test_early_stop_emits_diagnostic_event` — the operator-dashboard signal. ## Acceptance `python3 -m unittest discover -s pipelines/_tests -p test_*.py` exits 0 with 16/16 tests passing (was 10/10 before this PR; +6 from the new file). ## Post-merge follow-up Dispatch a fresh historical-pipeline workflow run and confirm the Gitea status flips to `failure` (not `success`) when the run hits the same Ollama 504 condition. The 12h cap and idempotent offset checkpoint are unchanged; resume behavior is preserved. ## Issue - INB-120 / INB-119 — Surface historical-pipeline early-stop in Gitea workflow status - Tracked: INB-7 (Phase 5 — Full Deployment) is blocked by this fix being merged.

grimacing added 1 commit 2026-06-08 14:57:08 +00:00

inb-120: surface historical-pipeline early-stop in Gitea workflow status

InboxOS — PR CI / Fast offline tests (pull_request) Successful in 28s

Details

InboxOS — PR CI / 100MB historical e2e (pull_request) Successful in 10m57s

Details

InboxOS — PR CI / Live Ollama extract smoke (pull_request) Successful in 59s

Details

a28cb5cdbd

Run #929 on 2026-06-08 hit extract_permanent_failure on an Ollama
504 after 6 messages and still surfaced as conclusion:success in
Gitea because historical_pipeline.main() returned 0 unconditionally.
Track loop exit reason (None for clean EOF / LIMIT_OVERRIDE,
'extract_permanent_failure' on a non-recoverable per-message
error) and return EARLY_STOP_EXIT_CODE=2 on the failure path so
Gitea records the run as failure. The 12h timeout is enforced by
the workflow itself (timeout-minutes: 720) and stays untouched.

Add pipelines/_tests/test_historical_pipeline_early_stop.py with
6 unittest.TestCase tests covering: extract_permanent_failure
returns 2, clean EOF returns 0, EOF after N messages returns 0,
LIMIT_OVERRIDE returns 0, EARLY_STOP_EXIT_CODE is non-zero and
distinct from 1, and historical_run_early_stop event is emitted
with reason=extract_permanent_failure and the chosen exit code.

Acceptance: python3 -m unittest discover -s pipelines/_tests
-p test_*.py exits 0 with 16/16 tests passing (was 10/10 before).

Co-Authored-By: Paperclip <noreply@paperclip.ing>

grimacing force-pushed inb-120-early-stop-exit-code from a574d55887 to a28cb5cdbd

2026-06-08 14:57:08 +00:00

Compare

grimacing merged commit f660578b76 into main

2026-06-08 14:57:22 +00:00

cpo referenced this pull request

2026-06-08 20:05:19 +00:00

CPO: backfill docs/requirements.md and docs/roadmap.md for 5 already-shipped PRs (INB-21/33/45/117/120) #9

cpo referenced this pull request

2026-06-08 20:15:49 +00:00

REQ-MVP-005: historical-pipeline early-stop exit code — INB-120 #14

cpo referenced this pull request