pipelines: bypass opencode CLI; call Ollama directly

opencode 1.16.2 server returns UnknownError on every `--command`
invocation (ref: err_xxx). The local-only `opencode run "what is 2+2?"`
default-agent path still works, but `SessionPrompt.command` (the path
the live pipeline uses) crashes immediately on every prompt. Diagnosis:
  - Local reproduction: `opencode run --command inboxos:classify` →
    UnknownError, server-side stacktrace in
    `SessionPrompt.command (/$bunfs/root/chunk-...).js`.
  - The Ollama server itself is healthy: `GET /v1/models` returns 7
    models including `lfm2.5:8b`; `POST /v1/chat/completions` with the
    `inboxos:classify` prompt body returns clean JSON.

The fix is to drop the opencode CLI from the hot path. The pipeline now
calls Ollama's OpenAI-compatible `/v1/chat/completions` endpoint
directly:

  - `pipelines/common.py`: new `LLMConfig` (backend selector), new
    `ollama_run_command`, new `llm_run_command` dispatcher. The legacy
    `opencode_run_command` is kept for offline tests.
  - `pipelines/live_pipeline.py`: switch from `opencode_run_command` /
    `OpenCodeConfig` to `llm_run_command` / `LLMConfig`. The model
    name is now read from the prompt frontmatter (`model: ollama/...`)
    with a configurable default (`INBOXOS_LLM_MODEL`).
  - `.gitea/workflows/live-pipeline.yml`: drop the "Ensure OpenCode is
    installed" step; add `INBOXOS_LLM_BACKEND`, `INBOXOS_LLM_MODEL`,
    `INBOXOS_LLM_TIMEOUT`, `OLLAMA_API_KEY` env vars.
  - `pipelines/_tests/test_live_pipeline_offline.py`: rebind the fake
    onto `lp.llm_run_command` and use `LLMConfig(backend="opencode", …)`
    so the offline smoke stays green.
  - `.env.example` and `scripts/setup_gitea_secrets.sh`: switch the
    default S3 endpoint to `https://storage.cloudinit.dev` (MinIO) per
    the platform migration in INB-16.

Local end-to-end check (single message against the live Ollama):
  LLMConfig(backend="ollama", ollama_base_url="https://ollama-frp.cloudinit.dev/v1", ...)
  → ollama_run_command("inboxos:classify", {subject="QuickBooks invoice #1234", ...})
  → {"modules": ["finance"]}    ✓

Offline tests: 8/8 pass. No secrets touched.

Refs: INB-17
Co-Authored-By: Paperclip <noreply@paperclip.ing>

.env.example

@@ -19,7 +19,7 @@ NOCODB_MODULE_TABLES='{"finance":"tbl_finance_placeholder","crm":"tbl_crm_placeh
 
 # ── SeaweedFS (S3-compatible object store) ──────────────
 # Used for raw .eml archival and pipeline state (checkpoints).
-SEAWEEDFS_S3_ENDPOINT=http://fs.cloudinit.dev:8333
+SEAWEEDFS_S3_ENDPOINT=https://storage.cloudinit.dev
 SEAWEEDFS_S3_ACCESS_KEY=change-me-to-a-seaweedfs-access-key
 SEAWEEDFS_S3_SECRET_KEY=change-me-to-a-seaweedfs-secret-key
 SEAWEEDFS_S3_BUCKET=inboxos-eml

.gitea/workflows/live-pipeline.yml

@@ -24,7 +24,19 @@
 #   SEAWEEDFS_S3_BUCKET (e.g. "inboxos-eml")
 #   NOCODB_BASE_URL, NOCODB_API_TOKEN, NOCODB_PROJECT_ID
 #   NOCODB_MODULE_TABLES (JSON map: {"finance":"tbl_xxx","crm":"tbl_yyy",...})
-#   OPENCODE_BIN (default "opencode"), OPENCODE_PROMPT_DIR (default "./.opencode/commands")
+#   OLLAMA_BASE_URL (e.g. https://ollama-frp.cloudinit.dev/v1)  ← FRP-tunneled Ollama
+#   OLLAMA_API_KEY (optional, only if your Ollama is auth-gated)
+#   INBOXOS_LLM_BACKEND (default "ollama"; legacy "opencode" available)
+#   INBOXOS_LLM_MODEL (default "lfm2.5:8b"; must be one of the served models)
+#   INBOXOS_LLM_TIMEOUT (default "120" seconds)
+#
+# Backend note: the live pipeline no longer shells out to the opencode
+# CLI. It uses `pipelines.common.llm_run_command` to call Ollama's
+# OpenAI-compatible /v1/chat/completions endpoint directly, with the
+# prompt body loaded from `OPENCODE_PROMPT_DIR`. The `ollama` backend
+# in the runner is configured via env vars (OLLAMA_BASE_URL,
+# INBOXOS_LLM_BACKEND, INBOXOS_LLM_MODEL) — no per-job
+# `~/.config/opencode/opencode.json` is required.
 
 name: InboxOS — Live Pipeline
 
@@ -73,23 +85,6 @@ jobs:
             requests \
             tenacity
 
-      - name: Ensure OpenCode is installed
-        env:
-          OPENCODE_BIN: ${{ vars.OPENCODE_BIN || secrets.OPENCODE_BIN || 'opencode' }}
-          OPENCODE_NPM_VERSION: ${{ vars.OPENCODE_NPM_VERSION || secrets.OPENCODE_NPM_VERSION || 'latest' }}
-        run: |
-          set -euo pipefail
-          if ! command -v "$OPENCODE_BIN" >/dev/null 2>&1; then
-            echo "::notice::OpenCode CLI not on PATH; installing opencode-ai@${OPENCODE_NPM_VERSION} via npm"
-            npm install -g "opencode-ai@${OPENCODE_NPM_VERSION}"
-            hash -r
-          fi
-          if ! command -v "$OPENCODE_BIN" >/dev/null 2>&1; then
-            echo "::error::OpenCode CLI still not found on runner PATH after install (tried: $OPENCODE_BIN)"
-            exit 1
-          fi
-          "$OPENCODE_BIN" --version
-
       - name: Run live pipeline
         env:
           IMAP_HOST: ${{ secrets.IMAP_HOST }}
@@ -106,9 +101,12 @@ jobs:
           NOCODB_API_TOKEN: ${{ secrets.NOCODB_API_TOKEN }}
           NOCODB_PROJECT_ID: ${{ secrets.NOCODB_PROJECT_ID }}
           NOCODB_MODULE_TABLES: ${{ secrets.NOCODB_MODULE_TABLES }}
-          OPENCODE_BIN: ${{ vars.OPENCODE_BIN || secrets.OPENCODE_BIN || 'opencode' }}
           OPENCODE_PROMPT_DIR: ${{ vars.OPENCODE_PROMPT_DIR || secrets.OPENCODE_PROMPT_DIR || './.opencode/commands' }}
           OLLAMA_BASE_URL: ${{ secrets.OLLAMA_BASE_URL }}
+          OLLAMA_API_KEY: ${{ secrets.OLLAMA_API_KEY }}
+          INBOXOS_LLM_BACKEND: ${{ vars.INBOXOS_LLM_BACKEND || secrets.INBOXOS_LLM_BACKEND || 'ollama' }}
+          INBOXOS_LLM_MODEL: ${{ vars.INBOXOS_LLM_MODEL || secrets.INBOXOS_LLM_MODEL || 'lfm2.5:8b' }}
+          INBOXOS_LLM_TIMEOUT: ${{ vars.INBOXOS_LLM_TIMEOUT || secrets.INBOXOS_LLM_TIMEOUT || '120' }}
           SINCE_UID_OVERRIDE: ${{ inputs.since_uid }}
           MAX_EMAILS_OVERRIDE: ${{ inputs.max_emails }}
         run: |

pipelines/_tests/test_live_pipeline_offline.py

@@ -137,7 +137,7 @@ def _install_fakes(lp, fake_s3, tracker, classify_modules, extract_responses):
         tracker.append({"table": table_id, "mid": message_id, "rec": record})
         return {"action": "post", "record": {"Id": 1, **record}}
 
-    lp.opencode_run_command = fake_oc
+    lp.llm_run_command = fake_oc
     lp.s3_put_bytes = fake_put
     lp.nocodb_upsert = fake_upsert
 
@@ -153,7 +153,7 @@ class FakeS3:
 
 def test_process_happy_path() -> None:
     import live_pipeline as lp
-    from common import NocoConfig, OpenCodeConfig
+    from common import LLMConfig, NocoConfig
 
     fake_s3 = FakeS3()
     tracker: list[dict] = []
@@ -178,7 +178,7 @@ def test_process_happy_path() -> None:
         s3=fake_s3,
         s3_bucket="inboxos-eml",
         nocodb_cfg=NocoConfig(base_url="x", api_token="y", project_id="z"),
-        opencode_cfg=OpenCodeConfig(bin="opencode", prompt_dir="."),
+        llm_cfg=LLMConfig(backend="opencode", prompt_dir=".", opencode_bin="opencode", ollama_base_url="http://localhost", ollama_api_key=None, default_model="lfm2.5:8b", timeout=120),
         module_tables={"finance": "tbl_fin"},
         logger=_silent_logger(),
     )
@@ -191,7 +191,7 @@ def test_process_happy_path() -> None:
 
 def test_process_missing_message_id() -> None:
     import live_pipeline as lp
-    from common import NocoConfig, OpenCodeConfig
+    from common import LLMConfig, NocoConfig
 
     fake_s3 = FakeS3()
     tracker: list[dict] = []
@@ -203,7 +203,7 @@ def test_process_missing_message_id() -> None:
         s3=fake_s3,
         s3_bucket="inboxos-eml",
         nocodb_cfg=NocoConfig(base_url="x", api_token="y", project_id="z"),
-        opencode_cfg=OpenCodeConfig(bin="opencode", prompt_dir="."),
+        llm_cfg=LLMConfig(backend="opencode", prompt_dir=".", opencode_bin="opencode", ollama_base_url="http://localhost", ollama_api_key=None, default_model="lfm2.5:8b", timeout=120),
         module_tables={"finance": "tbl_fin"},
         logger=_silent_logger(),
     )
@@ -214,7 +214,7 @@ def test_process_missing_message_id() -> None:
 
 def test_process_unclassified() -> None:
     import live_pipeline as lp
-    from common import NocoConfig, OpenCodeConfig
+    from common import LLMConfig, NocoConfig
 
     fake_s3 = FakeS3()
     tracker: list[dict] = []
@@ -226,7 +226,7 @@ def test_process_unclassified() -> None:
         s3=fake_s3,
         s3_bucket="inboxos-eml",
         nocodb_cfg=NocoConfig(base_url="x", api_token="y", project_id="z"),
-        opencode_cfg=OpenCodeConfig(bin="opencode", prompt_dir="."),
+        llm_cfg=LLMConfig(backend="opencode", prompt_dir=".", opencode_bin="opencode", ollama_base_url="http://localhost", ollama_api_key=None, default_model="lfm2.5:8b", timeout=120),
         module_tables={"finance": "tbl_fin"},
         logger=_silent_logger(),
     )
@@ -240,7 +240,7 @@ def test_process_unclassified() -> None:
 
 def test_process_extract_permanent_failure() -> None:
     import live_pipeline as lp
-    from common import NocoConfig, OpenCodeConfig
+    from common import LLMConfig, NocoConfig
 
     fake_s3 = FakeS3()
     tracker: list[dict] = []
@@ -258,7 +258,7 @@ def test_process_extract_permanent_failure() -> None:
         s3=fake_s3,
         s3_bucket="inboxos-eml",
         nocodb_cfg=NocoConfig(base_url="x", api_token="y", project_id="z"),
-        opencode_cfg=OpenCodeConfig(bin="opencode", prompt_dir="."),
+        llm_cfg=LLMConfig(backend="opencode", prompt_dir=".", opencode_bin="opencode", ollama_base_url="http://localhost", ollama_api_key=None, default_model="lfm2.5:8b", timeout=120),
         module_tables={"finance": "tbl_fin"},
         logger=_silent_logger(),
     )

pipelines/common.py

@@ -357,6 +357,12 @@ def opencode_run_command(
     The wrapper passes the email as JSON on stdin and parses the
     response. Non-zero exit, empty stdout, or invalid JSON are
     treated as failures and retried.
+
+    NOTE: opencode 1.16.2 server returns `UnknownError` on every
+    `--command` invocation (ref: err_xxx). This path is currently
+    broken end-to-end; use `llm_run_command` with the `ollama`
+    backend instead. This function is kept for offline tests and
+    future opencode versions.
     """
     cmd = [
         cfg.bin,
@@ -383,6 +389,167 @@ def opencode_run_command(
         raise RuntimeError(f"opencode did not emit valid JSON: {exc}; raw={stdout[:500]!r}") from exc
 
 
+# ── LLM invocation (backend selector) ──────────────────
+
+
+@dataclass
+class LLMConfig:
+    """Backend selector for the language-model step.
+
+    Two backends are supported:
+
+    - `ollama` (default, recommended): direct call to an
+      OpenAI-compatible /v1/chat/completions endpoint. Bypasses the
+      opencode CLI entirely. Reads the prompt from
+      `prompt_dir/<command_name>.md` and sends the body as the
+      system message, the JSON payload as the user message.
+    - `opencode`: legacy path that shells out to the opencode CLI.
+      Currently broken on opencode 1.16.2 (`UnknownError` on every
+      `--command` invocation); kept for offline tests only.
+    """
+
+    backend: str
+    prompt_dir: str
+    opencode_bin: str
+    ollama_base_url: str
+    ollama_api_key: str | None
+    default_model: str
+    timeout: int
+
+    @classmethod
+    def from_env(cls) -> "LLMConfig":
+        return cls(
+            backend=os.environ.get("INBOXOS_LLM_BACKEND", "ollama").strip().lower(),
+            prompt_dir=os.environ.get("OPENCODE_PROMPT_DIR", "./.opencode/commands"),
+            opencode_bin=os.environ.get("OPENCODE_BIN", "opencode"),
+            ollama_base_url=os.environ.get(
+                "OLLAMA_BASE_URL", "https://ollama-frp.cloudinit.dev/v1"
+            ).rstrip("/"),
+            ollama_api_key=os.environ.get("OLLAMA_API_KEY") or None,
+            default_model=os.environ.get("INBOXOS_LLM_MODEL", "lfm2.5:8b"),
+            timeout=int(os.environ.get("INBOXOS_LLM_TIMEOUT", "120")),
+        )
+
+
+def _load_prompt(prompt_dir: str, command_name: str) -> tuple[str, str | None]:
+    """Load a prompt .md file, returning (system_body, declared_model_or_none).
+
+    The file is the standard opencode command format: YAML
+    frontmatter delimited by `---` lines, followed by markdown body.
+    The `model:` field, if present, is the model's `provider/name`
+    string; we extract the `name` half as the default Ollama model.
+    """
+    path = Path(prompt_dir) / f"{command_name}.md"
+    raw = path.read_text(encoding="utf-8")
+    declared_model: str | None = None
+    body = raw
+    if raw.startswith("---"):
+        # Split on the second `---` line.
+        parts = raw.split("\n---\n", 2)
+        if len(parts) >= 2:
+            front = parts[0].lstrip("-").strip()
+            body = parts[1] if len(parts) == 2 else "\n---\n".join(parts[1:])
+            for line in front.splitlines():
+                line = line.strip()
+                if line.startswith("model:"):
+                    value = line.split(":", 1)[1].strip()
+                    declared_model = value.split("/", 1)[-1] or None
+                    break
+    return body.strip(), declared_model
+
+
+@retry(
+    reraise=True,
+    stop=stop_after_attempt(3),
+    wait=wait_exponential(multiplier=2, min=2, max=30),
+    retry=retry_if_exception_type((requests.RequestException, ValueError, RuntimeError)),
+)
+def ollama_run_command(
+    cfg: LLMConfig,
+    command_name: str,
+    payload: dict[str, Any],
+) -> dict[str, Any]:
+    """Call an Ollama-served model directly with the named prompt.
+
+    The system message is the prompt's markdown body (frontmatter
+    stripped). The user message is the JSON-serialised payload.
+    The model's reply is parsed as JSON. Retryable on transient
+    HTTP errors, JSON parse errors, and explicit RuntimeErrors
+    raised by the parser.
+    """
+    system_prompt, declared_model = _load_prompt(cfg.prompt_dir, command_name)
+    model = declared_model or cfg.default_model
+
+    url = f"{cfg.ollama_base_url}/chat/completions"
+    headers = {"Content-Type": "application/json"}
+    if cfg.ollama_api_key:
+        headers["Authorization"] = f"Bearer {cfg.ollama_api_key}"
+
+    body = {
+        "model": model,
+        "messages": [
+            {"role": "system", "content": system_prompt},
+            {"role": "user", "content": json.dumps(payload, default=str)},
+        ],
+        "stream": False,
+        "temperature": 0,
+        "max_tokens": 1024,
+    }
+
+    resp = requests.post(url, headers=headers, json=body, timeout=cfg.timeout)
+    resp.raise_for_status()
+    data = resp.json()
+
+    try:
+        content = data["choices"][0]["message"]["content"]
+    except (KeyError, IndexError, TypeError) as exc:
+        raise RuntimeError(f"Ollama response missing message content: {data!r}") from exc
+
+    if not content or not content.strip():
+        raise RuntimeError(f"Ollama returned empty content: {data!r}")
+
+    # Find the first balanced JSON object in the response.
+    text = content.strip()
+    decoder = json.JSONDecoder()
+    for i, ch in enumerate(text):
+        if ch != "{":
+            continue
+        try:
+            obj, _end = decoder.raw_decode(text[i:])
+        except json.JSONDecodeError:
+            continue
+        return obj
+    raise RuntimeError(f"Ollama response did not contain a JSON object: {text[:500]!r}")
+
+
+@retry(
+    reraise=True,
+    stop=stop_after_attempt(3),
+    wait=wait_exponential(multiplier=2, min=2, max=30),
+)
+def llm_run_command(
+    cfg: LLMConfig,
+    command_name: str,
+    payload: dict[str, Any],
+) -> dict[str, Any]:
+    """Backend-dispatching LLM call.
+
+    Default backend is `ollama` (see `LLMConfig`). The legacy
+    `opencode` backend shells out to the opencode CLI and is kept
+    for offline tests.
+    """
+    if cfg.backend == "ollama":
+        return ollama_run_command(cfg, command_name, payload)
+    if cfg.backend == "opencode":
+        return opencode_run_command(
+            OpenCodeConfig(bin=cfg.opencode_bin, prompt_dir=cfg.prompt_dir),
+            command_name,
+            payload,
+            timeout=cfg.timeout,
+        )
+    raise RuntimeError(f"Unknown INBOXOS_LLM_BACKEND: {cfg.backend!r}")
+
+
 # ── Misc helpers ───────────────────────────────────────
 
 def _require(name: str) -> str:

pipelines/live_pipeline.py

@@ -54,16 +54,16 @@ from tenacity import (
 )
 
 from common import (
+    LLMConfig,
     NocoConfig,
-    OpenCodeConfig,
     S3Config,
     ensure_bucket,
     get_logger,
+    llm_run_command,
     log_event,
     make_s3_client,
     nocodb_upsert,
     now_iso,
-    opencode_run_command,
     s3_get_text_optional,
     s3_put_bytes,
     safe_message_id,
@@ -269,19 +269,19 @@ def dead_letter_key(message_id: str | None, uid: int) -> str:
 
 # ── Extraction ─────────────────────────────────────────
 
-def classify(opencode_cfg: OpenCodeConfig, summary: dict[str, Any], logger) -> list[str]:
-    """Ask OpenCode which extraction modules apply to this email.
+def classify(llm_cfg: LLMConfig, summary: dict[str, Any], logger) -> list[str]:
+    """Ask the LLM backend which extraction modules apply to this email.
 
     Expected prompt contract (defined in Gitea-managed prompt file):
 
-        Input (stdin, JSON): the email summary produced above.
-        Output (stdout, JSON): {"modules": ["finance", "crm", ...]}
+        Input (user message, JSON): the email summary produced above.
+        Output (assistant message, JSON): {"modules": ["finance", "crm", ...]}
 
     On parse failure we fall back to the empty list and dead-letter
     the message so the operator notices.
     """
     try:
-        result = opencode_run_command(opencode_cfg, CLASSIFY_COMMAND, summary)
+        result = llm_run_command(llm_cfg, CLASSIFY_COMMAND, summary)
     except Exception as exc:
         log_event(logger, "classify_error", message_id=summary.get("message_id"), error=str(exc))
         return []
@@ -299,14 +299,14 @@ def classify(opencode_cfg: OpenCodeConfig, summary: dict[str, Any], logger) -> l
 
 
 def extract_for_module(
-    opencode_cfg: OpenCodeConfig,
+    llm_cfg: LLMConfig,
     module: str,
     summary: dict[str, Any],
     logger,
 ) -> dict[str, Any] | None:
     command_name = f"{EXTRACT_COMMAND_PREFIX}{module}"
     try:
-        result = opencode_run_command(opencode_cfg, command_name, summary)
+        result = llm_run_command(llm_cfg, command_name, summary)
     except Exception as exc:
         log_event(
             logger,
@@ -334,7 +334,7 @@ def process_one_message(
     s3: Any,
     s3_bucket: str,
     nocodb_cfg: NocoConfig,
-    opencode_cfg: OpenCodeConfig,
+    llm_cfg: LLMConfig,
     module_tables: dict[str, str],
     logger,
 ) -> bool:
@@ -359,7 +359,7 @@ def process_one_message(
     log_event(logger, "eml_backed_up", message_id=message_id, key=key, bytes=len(raw))
 
     # 2. Classify.
-    modules = classify(opencode_cfg, summary, logger)
+    modules = classify(llm_cfg, summary, logger)
     log_event(logger, "classified", message_id=message_id, modules=modules)
 
     if not modules:
@@ -394,7 +394,7 @@ def process_one_message(
             )
             continue
 
-        extracted = extract_for_module(opencode_cfg, module, summary, logger)
+        extracted = extract_for_module(llm_cfg, module, summary, logger)
         if extracted is None:
             # Treat permanent extraction failure as a hard error: we
             # do NOT advance the checkpoint, so the next run will
@@ -419,7 +419,7 @@ def main(argv: list[str] | None = None) -> int:
         live_cfg = LiveConfig.from_env()
         s3_cfg = S3Config.from_env(bucket_default="inboxos-eml")
         nocodb_cfg = NocoConfig.from_env()
-        opencode_cfg = OpenCodeConfig.from_env()
+        llm_cfg = LLMConfig.from_env()
     except Exception as exc:
         log_event(logger, "config_error", error=str(exc))
         return 2
@@ -456,7 +456,7 @@ def main(argv: list[str] | None = None) -> int:
             s3=s3,
             s3_bucket=s3_cfg.bucket,
             nocodb_cfg=nocodb_cfg,
-            opencode_cfg=opencode_cfg,
+            llm_cfg=llm_cfg,
             module_tables=live_cfg.module_tables,
             logger=logger,
         )

scripts/setup_gitea_secrets.sh

@@ -92,7 +92,7 @@ put_secret "IMAP_USE_SSL"    "${IMAP_USE_SSL:-true}"
 put_secret "IMAP_MAILBOX"    "${IMAP_MAILBOX:-INBOX}"
 
 # SeaweedFS
-put_secret "SEAWEEDFS_S3_ENDPOINT"   "${SEAWEEDFS_S3_ENDPOINT:-http://fs.cloudinit.dev:8333}"
+put_secret "SEAWEEDFS_S3_ENDPOINT"   "${SEAWEEDFS_S3_ENDPOINT:-https://storage.cloudinit.dev}"
 put_secret "SEAWEEDFS_S3_BUCKET"     "${SEAWEEDFS_S3_BUCKET:-inboxos-eml}"
 put_secret "SEAWEEDFS_S3_ACCESS_KEY" "${SEAWEEDFS_S3_ACCESS_KEY:-}" "no"
 put_secret "SEAWEEDFS_S3_SECRET_KEY" "${SEAWEEDFS_S3_SECRET_KEY:-}" "no"