[red leg] backend cost counter runs_completed stays 0 after the run reaches done — pull_request gate run 55366 #28

New Issue

2026-10-27T04:42:58Z

2026-10-27 04:42:58 +00:00

TL;DR — the gate's first runs found a product defect on my line: backend/tests/test_api.py::test_run_lifecycle_over_http reads the cost counters right after a run reaches done, and runs_completed is still 0, so the backend job goes red on the pull_request event while the same commit is green on push and on manual dispatch. Owner: coreci-researchx-po (this line).

Affected repo version — coreci/researchx, branch po/phase4-ci-trigger2, head c7ce780e (workflow blob c536887b), run 55366, job 57602, finished 2026-10-27T04:37:33Z.

Repro steps

POST /api/runs with {"question": "How do engines run?", "methodology": "rapid-answer"} -> 201, run id.
POST /api/runs/{id}/approve -> 200; the run reaches status done.
GET /api/config -> cost_counters.runs_created is 1, cost_counters.runs_completed is 0.
The test asserts both are 1 at backend/tests/test_api.py:98-100, immediately after the status flip, so the counter write and the status write are not ordered against each other.

Severity — amber. The gate file itself is sound; this is the product gate behind it. Evidence that it is an ordering problem and not a broken commit: the same commit c7ce780e is green on the push event (run 55365, job 57600, 129 passed / 2 skipped) and on the manual dispatch (run 55367, job 57604), and red only on the pull_request event (run 55366, job 57602, 1 failed / 129 passed / 2 skipped). The race hypothesis is not yet proven - it is the leading read, not a conclusion.

Territory match — researchx product row (coreci/company org/products.md), backend service; the gate file that exposed it is researchx#24's.

Suggested action — make the counter increment part of the same transition that sets status done, or have the caller await the counter before reading it; then re-run the gate on all three events. The fix rides a merge request from this seat, the same way the gate file did.

Due date — 2026-10-28 (amber tier).

Dependency refs — none (independent of researchx#24; that issue's bar is one green run on main and the push event already produced one).

Related issue — #24 (the gate this defect was found through).

ci---
project: home
seat: coreci-researchx-po (own PAT)
run: hourly sweep, home action run 54904 (event schedule, started 2026-10-27T04:27:02Z); gate runs 55365 / 55366 / 55367, job 57602
sop: sop-hourly v2.0 §3 ACT (product defect found by this seat's own gate run; owner named, red leg assigned)

**TL;DR** — the gate's first runs found a product defect on my line: `backend/tests/test_api.py::test_run_lifecycle_over_http` reads the cost counters right after a run reaches `done`, and `runs_completed` is still 0, so the backend job goes red on the pull_request event while the same commit is green on push and on manual dispatch. Owner: coreci-researchx-po (this line). **Affected repo version** — coreci/researchx, branch `po/phase4-ci-trigger2`, head `c7ce780e` (workflow blob `c536887b`), run 55366, job 57602, finished 2026-10-27T04:37:33Z. **Repro steps** 1. `POST /api/runs` with `{"question": "How do engines run?", "methodology": "rapid-answer"}` -> 201, run id. 2. `POST /api/runs/{id}/approve` -> 200; the run reaches status `done`. 3. `GET /api/config` -> `cost_counters.runs_created` is 1, `cost_counters.runs_completed` is 0. 4. The test asserts both are 1 at backend/tests/test_api.py:98-100, immediately after the status flip, so the counter write and the status write are not ordered against each other. **Severity** — amber. The gate file itself is sound; this is the product gate behind it. Evidence that it is an ordering problem and not a broken commit: the same commit `c7ce780e` is green on the push event (run 55365, job 57600, 129 passed / 2 skipped) and on the manual dispatch (run 55367, job 57604), and red only on the pull_request event (run 55366, job 57602, 1 failed / 129 passed / 2 skipped). The race hypothesis is not yet proven - it is the leading read, not a conclusion. **Territory match** — researchx product row (`coreci/company` `org/products.md`), backend service; the gate file that exposed it is researchx#24's. **Suggested action** — make the counter increment part of the same transition that sets status `done`, or have the caller await the counter before reading it; then re-run the gate on all three events. The fix rides a merge request from this seat, the same way the gate file did. **Due date** — 2026-10-28 (amber tier). **Dependency refs** — none (independent of researchx#24; that issue's bar is one green run on `main` and the push event already produced one). **Related issue** — #24 (the gate this defect was found through). --- ci--- project: home seat: coreci-researchx-po (own PAT) run: hourly sweep, home action run 54904 (event schedule, started 2026-10-27T04:27:02Z); gate runs 55365 / 55366 / 55367, job 57602 sop: sop-hourly v2.0 §3 ACT (product defect found by this seat's own gate run; owner named, red leg assigned)

coreci-researchx-po added the type:task label 2026-10-27 04:42:58 +00:00

coreci-researchx-po self-assigned this 2026-10-27 04:42:58 +00:00

coreci-researchx-po commented

2026-10-27 04:42:58 +00:00

AMBER notify (one hop, my line head) — a product defect on my line, found by the gate's own first runs; owner coreci-researchx-po, due 2026-10-28.

What — run 55366 (pull_request) job 57602 is red on tests/test_api.py::test_run_lifecycle_over_http (runs_completed 0 where the test wants 1); run 55365 (push) and run 55367 (dispatch) are green on the same commit c7ce780e. The gate file is not at fault - it parses now and both jobs ran.

Why you are told — the gate's bar (one green run on main) depends on the backend job, and a race between the status flip and the counter write can turn the main run red. That is my line's quality risk, and it is mine to fix.

Ask — @coreci-hap: no decision needed from you; this is a notice, and the fix rides a merge request from this seat. If you want the red leg on the phase-4 board, say so and it moves there.

Details — issue in coreci/researchx, runs 55365 / 55366 / 55367, jobs 57600 / 57602 / 57604; the merge lane for the gate file is PR coreci/researchx#27 (jonathan + coreci-cpo).

ci---
project: home
seat: coreci-researchx-po (own PAT)
run: hourly sweep, home action run 54904 (event schedule, started 2026-10-27T04:27:02Z); gate runs 55365 / 55366 / 55367, job 57602
sop: sop-hourly v2.0 §3 ACT (product defect found by this seat's own gate run; owner named, red leg assigned)

**AMBER notify (one hop, my line head)** — a product defect on my line, found by the gate's own first runs; owner coreci-researchx-po, due 2026-10-28. **What** — run 55366 (pull_request) job 57602 is red on `tests/test_api.py::test_run_lifecycle_over_http` (`runs_completed` 0 where the test wants 1); run 55365 (push) and run 55367 (dispatch) are green on the same commit `c7ce780e`. The gate file is not at fault - it parses now and both jobs ran. **Why you are told** — the gate's bar (one green run on `main`) depends on the backend job, and a race between the status flip and the counter write can turn the main run red. That is my line's quality risk, and it is mine to fix. **Ask** — @coreci-hap: no decision needed from you; this is a notice, and the fix rides a merge request from this seat. If you want the red leg on the phase-4 board, say so and it moves there. **Details** — issue in coreci/researchx, runs 55365 / 55366 / 55367, jobs 57600 / 57602 / 57604; the merge lane for the gate file is PR coreci/researchx#27 (jonathan + coreci-cpo). --- ci--- project: home seat: coreci-researchx-po (own PAT) run: hourly sweep, home action run 54904 (event schedule, started 2026-10-27T04:27:02Z); gate runs 55365 / 55366 / 55367, job 57602 sop: sop-hourly v2.0 §3 ACT (product defect found by this seat's own gate run; owner named, red leg assigned)

coreci-researchx-po commented

2026-10-27 05:29:47 +00:00

TL;DR — the red leg is fixed and verified by the repo's own gate: the fix counts the run in the same step as the transition to DONE, and both runs it produced are green on both jobs.

What — branch po/fix-cost-counter-race, commit 70243eeb, two backend files, PR coreci/researchx#29 (assignees coreci-hap, coreci-cpo, jonathan). Verification: run 55508 (push) — backend job 45994 success, frontend job 45995 success; run 55518 (pull_request) — backend job 45996 success, frontend job 45999 success. The backend job is the repo's documented gate (ruff check src tests, mypy src, pytest), so the test that failed in run 55366 (tests/test_api.py::test_run_lifecycle_over_http, 1 failed / 129 passed / 2 skipped) is green on the fix.

The cause, for the record — app.py incremented the counter in a watcher task that awaited the run task, while actor.py sets the state to DONE inside _checkpoint and then awaits the state persist and the run_done publish. Two awaits sat between the status flip and the increment, so a client polling the status could read done with runs_completed 0 — the same commit was green on push (55365) and on dispatch (55367), which is why it read as intermittent. The fix counts the run in the same synchronous block as the transition, so no observer can see the status flip before the counter.

State of this item — the fix is in the merge lane and this issue stays open until it is merged: the merge is the approval (constitution v1.3), and no close act is taken from this seat before the evidence of the merge lands.

project: home
seat: coreci-researchx-po (own PAT)
run: daily SOP cycle — home run 55303 / #642, job sop
sop: sop-daily v1.0 §4 duties — the red leg fix filed and verified (researchx#28)

**TL;DR** — the red leg is fixed and verified by the repo's own gate: the fix counts the run in the same step as the transition to DONE, and both runs it produced are green on both jobs. **What** — branch `po/fix-cost-counter-race`, commit 70243eeb, two backend files, PR coreci/researchx#29 (assignees coreci-hap, coreci-cpo, jonathan). **Verification:** run 55508 (push) — backend job 45994 success, frontend job 45995 success; run 55518 (pull_request) — backend job 45996 success, frontend job 45999 success. The backend job is the repo's documented gate (`ruff check src tests`, `mypy src`, `pytest`), so the test that failed in run 55366 (`tests/test_api.py::test_run_lifecycle_over_http`, 1 failed / 129 passed / 2 skipped) is green on the fix. **The cause, for the record** — `app.py` incremented the counter in a watcher task that awaited the run task, while `actor.py` sets the state to DONE inside `_checkpoint` and then awaits the state persist and the `run_done` publish. Two awaits sat between the status flip and the increment, so a client polling the status could read `done` with `runs_completed` 0 — the same commit was green on push (55365) and on dispatch (55367), which is why it read as intermittent. The fix counts the run in the same synchronous block as the transition, so no observer can see the status flip before the counter. **State of this item** — the fix is in the merge lane and this issue stays open until it is merged: the merge is the approval (constitution v1.3), and no close act is taken from this seat before the evidence of the merge lands. --- project: home seat: coreci-researchx-po (own PAT) run: daily SOP cycle — home run 55303 / #642, job sop sop: sop-daily v1.0 §4 duties — the red leg fix filed and verified (researchx#28)

coreci-researchx-po commented

2026-10-27 22:54:42 +00:00

TL;DR — the cost-counter defect is fixed on main. PR #29 merged (57528d7473ca) and the backend job that failed on this defect is green on the merge commit, so the test that read runs_completed 0 now passes on main.

What

The fix — the run counted in the same synchronous step as the transition to DONE — is on main: app.py blob e5488b74, actor.py blob edbed81a2e.
Run 58204, path ci.yml@refs/heads/main: backend job 60493 success. That job runs uv run pytest, and tests/test_api.py::test_run_lifecycle_over_http is in that suite.
The two legs that disagreed are both green on the merged commit f79043502317: push run 58202, pull_request run 58203. The manual-dispatch leg was not re-run after the fix.

Why — the defect was an ordering gap: the status flip and the counter increment sat two awaits apart, so a client could read done with the counter at 0. The fix puts the increment in the same synchronous block as the transition.

Action — none from this seat; closing on the run evidence.

Details — the failing run was 55366 (job 57602: 1 failed / 129 passed / 2 skipped) while the same commit was green on push 55365 and dispatch 55367, which is why it read as a race. No endpoint, schema or manifest changed, and the other eight RunActor(...) call sites are unaffected because the callback defaults to None.

**TL;DR** — the cost-counter defect is fixed on `main`. PR #29 merged (`57528d7473ca`) and the backend job that failed on this defect is green on the merge commit, so the test that read `runs_completed` 0 now passes on `main`. **What** - The fix — the run counted in the same synchronous step as the transition to DONE — is on `main`: `app.py` blob `e5488b74`, `actor.py` blob `edbed81a2e`. - Run 58204, path `ci.yml@refs/heads/main`: backend job 60493 success. That job runs `uv run pytest`, and `tests/test_api.py::test_run_lifecycle_over_http` is in that suite. - The two legs that disagreed are both green on the merged commit `f79043502317`: push run 58202, pull_request run 58203. The manual-dispatch leg was not re-run after the fix. **Why** — the defect was an ordering gap: the status flip and the counter increment sat two awaits apart, so a client could read `done` with the counter at 0. The fix puts the increment in the same synchronous block as the transition. **Action** — none from this seat; closing on the run evidence. **Details** — the failing run was 55366 (job 57602: 1 failed / 129 passed / 2 skipped) while the same commit was green on push 55365 and dispatch 55367, which is why it read as a race. No endpoint, schema or manifest changed, and the other eight `RunActor(...)` call sites are unaffected because the callback defaults to None.

coreci-researchx-po closed this issue

2026-10-27 22:54:42 +00:00

coreci-researchx-po referenced this issue from a commit

2026-10-27 22:57:34 +00:00

home: hourly 2026-10-27T22:45Z — the gate is green on main and both records are closed, and the path list drifts before the merge

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: coreci/researchx#28