inboxos-v1/docs/gitea-actions.md

InboxOS — Gitea Actions Setup

Owner: CTO Status: Complete (scaffolding) Source issue: INB-8

This document explains how Gitea Actions is wired up to run the InboxOS live and historical email pipelines. It is intentionally short — the workflows themselves are the source of truth.

1. Topology

┌──────────────┐      ┌─────────────────┐      ┌──────────────┐
│  Gitea       │ <─── │ gitea-runner    │ <─── │  Ollama      │
│  (web + API) │      │ gitea-runner-2  │      │  (LLM)       │
└──────────────┘      │ (act_runner ×2) │      └──────────────┘
       ▲              └─────────────────┘
       │ poll IMAP             │ run jobs
       │ upload .eml           │ (live + historical
       │                       │  in parallel)
       │                       ▼
       │              ┌─────────────────┐
       │              │  SeaweedFS      │  raw .eml archive
       │              │  (S3 API)       │  + state checkpoints
       │              └─────────────────┘
       │                       │
       │ UPSERT                │ UPSERT
       ▼                       ▼
┌──────────────┐      ┌─────────────────┐
│  NocoDB      │      │  Papra          │
│  (DWH)       │      │  (attachments)  │
└──────────────┘      └─────────────────┘

All services run in the inboxos-net Docker network defined in docker/docker-compose.yml. The runner can reach every other service by its compose service name (gitea, nocodb, seaweedfs-master, ollama, papra).

2. Provisioning the runner

1. Bring the stack up:

   cp .env.example .env
   # fill in real secrets (see .env.example for the full list)
   docker compose -f docker/docker-compose.yml up -d
   

2. Generate a runner registration token in Gitea:

   • Log in to https://gitea.cloudinit.dev as an admin.
   • Go to Site Administration → Actions → Runners → Create new runner.
   • Copy the registration token into .env as GITEA_RUNNER_REGISTRATION_TOKEN.

3. Restart the runner so it picks up the new token:

   docker compose -f docker/docker-compose.yml restart gitea-runner
   

4. Verify in the Gitea UI under Actions → Runners. The runner named inboxos-runner-1 should appear with status Idle and the labels self-hosted,linux,x64,gitea-runner:docker.

The runner's long-lived token is stored in the gitea-runner-data named volume, so the registration token is only needed for first-boot registration. Subsequent restarts re-use the stored token automatically.

2a. Adding a second runner (INB-33)

A single runner is a point of contention: with cancel-in-progress: false on the live pipeline, the historical pipeline cannot start while a live run is in flight. The fix is to register a second runner (inboxos-runner-2) with the same label set so both pipelines can execute in parallel on the same host.

1. Generate a fresh registration token from the same Gitea admin page (Site Administration → Actions → Runners → Create new runner). Tokens are single-use — runner #1's token cannot be reused.

2. Add it to .env as GITEA_RUNNER_REGISTRATION_TOKEN_2.

3. Bring up the second runner:

   docker compose -f docker/docker-compose.yml up -d gitea-runner-2
   

4. Verify both runners are registered and online:

    curl -u "$GITEA_USER:$GITEA_PASS" \
      https://git.cloudinit.dev/api/v1/repos/${GITEA_OWNER:-jc-capital}/inboxos/actions/runners
   

   Expected: 2 entries with status: "online" and the self-hosted,linux,x64,gitea-runner:docker label set.

5. Confirm parallelism by dispatching historical-pipeline.yml while a live-pipeline.yml run is in progress. The historical job should start within ~30 s instead of sitting in queued.

To scale further (N>2) we'd templatize this; for now the explicit gitea-runner-2 service is the simplest, safest two-runner topology.

3. Repository secrets

Both workflows consume Gitea Actions secrets on the repository that hosts the pipeline code. In the Gitea web UI, go to the repo → Settings → Actions → Secrets and add the values from .env. The required secrets are listed in comments at the top of each workflow file.

In addition, any workflow step that needs to call the Gitea REST API (e.g. for repo management or secret rotation) must have GITEA_INSTANCE_URL, GITEA_USER, and GITEA_PASS set as secrets. See §5 for the auth model.

Minimum set per workflow:

OPENCODE_BIN and OPENCODE_PROMPT_DIR are still read by pipelines.common.LLMConfig.from_env (for the prompt directory) and by the historical workflow's Ensure OpenCode is installed step (for operator convenience). Neither pipeline shells out to the opencode CLI in production; the INBOXOS_LLM_BACKEND=ollama path is the default and only path used by CI and runtime. See docs/extraction-contract.md §Legacy: opencode CLI backend for the rationale.

Secret	Live	Historical
IMAP_HOST / IMAP_PORT / IMAP_USER / IMAP_PASSWORD / IMAP_USE_SSL / IMAP_MAILBOX	✅	—
SEAWEEDFS_S3_ENDPOINT / SEAWEEDFS_S3_ACCESS_KEY / SEAWEEDFS_S3_SECRET_KEY / SEAWEEDFS_S3_BUCKET	✅	✅
MBOX_BUCKET / MBOX_OBJECT_KEY	—	✅ (object key passed as workflow_dispatch input)
PAPRA_BASE_URL / PAPRA_API_TOKEN / PAPRA_ORGANIZATION_ID	—	✅
NOCODB_BASE_URL / NOCODB_API_TOKEN / NOCODB_PROJECT_ID	✅	✅
OPENCODE_BIN / OPENCODE_PROMPT_DIR	✅	✅
OLLAMA_BASE_URL	✅	✅

4. Workflows

live-pipeline.yml — net-new email processing

• Trigger: cron(*/5 * * * *) plus workflow_dispatch.
• Concurrency: live-pipeline group, no cancel (overlapping runs of the live pipeline must not trample each other's state).
• Steps: checkout → install Python deps → run python pipelines/live_pipeline.py → upload log artifact.
• Inputs (manual): since_uid (override the high-water mark), max_emails (default 50).

The script is idempotent on Message_ID and on the IMAP UID high-water mark. See pipelines/README.md for the contract.

historical-pipeline.yml — backlog / rescan

• Trigger: workflow_dispatch only (heavy, manual).
• Concurrency: historical-pipeline group, no cancel.
• Timeout: 12 hours.
• Inputs (manual): mbox_object_key (required), start_offset, batch_size (default 100), limit (default 0 = unlimited).

The script streams the .mbox from SeaweedFS, uploads attachments to Papra, and UPSERTs to NocoDB. It checkpoints to state/historical-state.json every batch_size messages and on exit, so a failed run can resume mid-file.

5. Authenticating to the Gitea REST API

Any code in this repo that needs to call the Gitea REST API (repo create, secrets, runner registration lookups, etc.) must go through pipelines.common.gitea_request with GiteaConfig.from_env(). The helper attaches HTTP Basic credentials from GITEA_USER / GITEA_PASS to every call.

from common import GiteaConfig, gitea_request

cfg = GiteaConfig.from_env()
resp = gitea_request(cfg, "GET", "/api/v1/user")
resp.raise_for_status()
me = resp.json()

GITEA_INSTANCE_URL should point at the Gitea API base the pipeline can reach:

• Inside the gitea-runner container: http://gitea:3000 (compose service name on inboxos-net).
• From the host / an external agent: https://git.cloudinit.dev (the public Gitea hostname).

Why Basic Auth and not a personal access token? Gitea's REST API does not return the plaintext of a freshly created access token — the response only carries sha1 (a hash) and token_last_eight, and the plaintext is shown exactly once in the web UI's "Generate Token" modal. A headless agent can therefore create the token record but cannot recover its value to put it in a GITEA_TOKEN env var. Basic Auth with a long-lived service account password avoids that gap and matches the air-gapped, self-hosted posture of the project.

5xx responses are retried (tenacity, max 5 attempts, exponential backoff) and then surface as RuntimeError; 4xx responses are returned to the caller untouched so the caller can decide how to react. Offline tests live in pipelines/_tests/test_gitea_request_offline.py and pin the contract (10 tests covering env parsing, Basic Auth header, 4xx passthrough, 5xx retry, and connection-error retry).

6. Prompt version control

Extraction prompts live in .opencode/commands/*.md in the same repository. Both pipelines call pipelines.common.llm_run_command (default backend: ollama) twice per email:

1. inboxos:classify — returns {"modules": ["finance", "crm"]} listing which module extractors should run.
2. For each module in that list, inboxos:extract-<module> — returns a flat NocoDB-shaped record for that table.

llm_run_command is expected to return one parsed JSON object per invocation. The exact field contracts are defined in the prompt files themselves and are reviewed via PR. See docs/extraction-contract.md for the full contract and the §Legacy: opencode CLI backend subsection for the offline-test-only INBOXOS_LLM_BACKEND=opencode path (broken on opencode 1.16.2; not used in production or in CI).

To roll a prompt change:

1. Open a PR with the modified .md.
2. Run the historical pipeline against a small .mbox slice using the limit input to validate.
3. Merge. The next live or historical run picks up the new prompt.

7. Observability

• Each run writes structured JSON logs to pipelines/_artifacts/<pipeline>-run-<run-id>.log. The workflows upload these as build artifacts (live-pipeline-log, historical-pipeline-log).
• Artifact upload is pinned to actions/upload-artifact@v3 in both live-pipeline.yml and historical-pipeline.yml. v4 requires GitHub GHES-only endpoints that Gitea Actions does not implement and fails the run with GHESNotSupportedError (observed on live run #674 on 2026-06-07 and on the INB-43 historical probe run #860 / job 1023 on 2026-06-08). v3 talks the older v1 Gitea-compatible artifact API and works. Do not bump to v4+ on either workflow without verifying against the live Gitea instance first; if you need a v4-only feature, raise an issue and route through the CTO.
• Gitea surfaces job step output in the Actions UI.
• NocoDB writes return the upserted record; the pipeline logs {action: post|patch, id, modules} per email.

8. Rollback

Failure	Recovery
Runner offline	docker compose restart gitea-runner (or gitea-runner-2); the other runner continues serving jobs.
Bad prompt merged	Revert the PR. The next run uses the reverted prompt; UPSERT semantics mean no data is lost, only overwritten.
SeaweedFS state file corrupted	Delete state/live-state.json or state/historical-state.json from the bucket; the next run starts fresh from UID 0 / offset 0 (re-runs are safe — UPSERT dedupes).
NocoDB schema broken	Fix the schema, then re-run a workflow. UPSERT keeps the existing Message_ID rows consistent.
Runner needs to re-register	Delete the runner's data volume (gitea-runner-data or gitea-runner-2-data), regenerate the registration token, set it in .env (GITEA_RUNNER_REGISTRATION_TOKEN / _2), restart the runner.