coreci-chat/.ciagent/RESEARCH.md

Research Findings (Phase 0)

Spec: CoreCI Chat v0.1 v1.1 (locked 2026-08-24). All architectural decisions locked in CLARIFY.md. This document records implementation patterns, pitfalls, and references for each M1 technology choice so the EXECUTE waves have a grounded baseline. Research is scoped to M1 (REQ-001..014, 038, 039, 040); M2/M3 technologies are noted where they touch M1 foundations.

---

R-001 — Trigger.dev bootstrap & durable execution pattern

Scope: M1 Wave A bootstraps the runtime; M3 adds chat orchestration tasks. M1 must not couple to Trigger.dev in a way that forces an M3 rewrite.

Findings:

• Trigger.dev v3 runs tasks as idempotent functions decorated with task(); long-running workflows use runTask() checkpoints. The runtime connects to a Trigger.dev server (cloud or self-hosted) via TRIGGER_API_KEY + TRIGGER_API_URL.
• Bootstrap pattern: a single packages/runtime (or inside apps/control-plane/lib/runtime) that initializes the Trigger.dev client at process start and exports a registerTask helper. M1 wires the client + a no-op health task; M3 registers the chat orchestration task.
• Pitfall: Trigger.dev cloud requires outbound HTTPS to https://api.trigger.dev. Since the control plane is the only thing that talks to Trigger.dev (not the Relay Agent, not the browser), this is fine for the SaaS deployment. Document the firewall egress.
• Pitfall: TRIGGER_API_KEY is infra-level config — goes through packages/config env loading, NOT through packages/secrets. It is not a tenant secret.
• M1 decision: Bootstrap the client + a runtimeHealthCheck task that runs every 5 min and appends an audit entry. Proves the runtime works end-to-end without coupling to chat logic.
• Reference: Trigger.dev v3 docs — trigger.dev/docs.

R-002 — WorkOS SSO + RBAC role mapping + SCIM

Scope: M1 Wave B (REQ-001..005). WorkOS is the only IdP in v0.1.

Findings:

• WorkOS provides authenticateWithCode() (OAuth/OIDC code flow) and a hosted SSO portal. The control plane exchanges the auth code for a session, then resolves the user to a tenant.
• User ↔ tenant mapping is owned by CoreCI Chat, not WorkOS. WorkOS gives us userId, email, organizationId (optional). We map organizationId → tenant at first signup (REQ-002): if no tenant exists for this org, create one and assign the user Admin.
• RBAC roles (Admin/Operator/Viewer) are CoreCI Chat's, stored in tenant_memberships.role. WorkOS role/ group claims are advisory only — we do not trust them for authorization (REQ-005 enforces at our API gateway, not at the IdP).
• Session: httpOnly cookie + a server-side session row carrying tenantId + role. The API gateway reads the cookie, loads the session, and runs RBAC.
• SCIM (REQ-003 invitations): WorkOS exposes a SCIM endpoint and an invitation API. For M1, use the WorkOS Invitation resource (single-use acceptance link emailed via WorkOS) — simpler than building email ourselves. Edge 15 (bounce) handled by WorkOS webhook → we mark the invite invalid.
• Pitfall: WorkOS SSO provider down (Edge 10) — surface a retry screen, block tenant creation. Do not fall back to local auth.
• Pitfall: Multi-tenant users (same email in two orgs) — resolve the active tenant from the session, allow tenant switching via an explicit endpoint (out of M1 scope to build the switcher UI; the API supports it).
• Reference: WorkOS Node SDK — workos.com/docs.

R-003 — Postgres RLS + audit hash-chain

Scope: M1 Wave A (REQ-038, REQ-039). The pattern propagates to every M2/M3 table.

Findings:

• RLS pattern: every tenant-scoped table has tenant_id UUID NOT NULL and a policy USING (tenant_id = current_setting('app.tenant_id')::uuid). The app connects as a role with app.tenant_id set per-transaction via SET LOCAL app.tenant_id = $1 inside a transaction. packages/db exposes withTenant(tenantId, async fn) that opens a transaction, SET LOCAL, runs fn, commits. No query outside withTenant touches tenant-scoped tables.
• Pitfall: current_setting('app.tenant_id') returns NULL if unset → policy tenant_id = NULL is false → rows invisible (safe default, but throws if a query runs outside withTenant). Enforce in code: a lint rule or a wrapper that rejects queries without a tenant context.
• Pitfall: Superuser bypasses RLS. The app role must NOT be superuser. Migrations run as a separate migrator role (BYPASSRLS) gated by CI, not the app role.
• Audit hash-chain: audit_log (id BIGSERIAL, tenant_id UUID, prev_hash BYTEA, curr_hash BYTEA, payload JSONB, created_at TIMESTAMPTZ, ...). curr_hash = sha256(prev_hash || canonical_jsonb(payload)). The first row's prev_hash is a fixed genesis constant. id is monotonic; the chain is verifiable by walking ORDER BY id.
• Immutability: REVOKE UPDATE, DELETE ON audit_log FROM app_role. Add a trigger that raises if anyone tries INSERT with a forged prev_hash (the app computes curr_hash in-app, but prev_hash must equal the last row's curr_hash for that tenant — a constraint trigger enforces this).
• Edge 7 (write failure halts): the audit write runs inside the same transaction as the business operation. If the INSERT fails, the transaction rolls back and the operation never happened. Admin alert fires from the error handler.
• Pitfall: Per-tenant hash-chain vs global hash-chain. Per-tenant chain is simpler to verify and avoids cross-tenant ordering contention. Use per-tenant chains (partition audit_log by tenant_id or index heavily on (tenant_id, id)).
• Reference: Postgres RLS docs — postgresql.org/docs/16/ddl-rowsecurity.html.

R-004 — AWS Secrets Manager + KMS + local-encrypted dev fallback

Scope: M1 Wave A (REQ-040). SecretProvider interface, two impls.

Findings:

• SecretProvider interface: get(tenantId, name): Promise<SecretValue>, put(tenantId, name, value): Promise<SecretRef>, delete(tenantId, name): Promise<void>. SecretRef is a string like aws-sm:coreci/<tenantId>/<name> or local:<tenantId>/<name>.
• AwsSecretsManagerProvider (prod): uses @aws-sdk/client-secrets-manager. Secret name convention coreci/<tenantId>/<name>. KMS key per tenant (or a shared CMK with encryption context {tenantId}). put creates or updates; get fetches + decrypts. IAM role scoped to the coreci/* prefix.
• LocalEncryptedProvider (dev/test): AES-256-GCM. Master key from SECRET_MASTER_KEY_DEV env var (the ONE allowed env var for secrets — everything else is provider-resolved). Ciphertext stored in .secrets/local-encrypted.json (gitignored). Each entry: {ciphertext, iv, authTag, salt}. Key derived via PBKDF2 from the master key + per-entry salt.
• Pitfall: The DB never stores the secret — only the SecretRef. byom_endpoints.secret_ref TEXT holds aws-sm:coreci/<tenantId>/byom. The app calls secrets.get(tenantId, 'byom') to resolve at use time.
• Pitfall: Logging — never console.log a resolved secret. The SecretValue type should have a custom toString() that returns [REDACTED]. Add a lint rule banning console.log(secret).
• Pitfall: Rotation — out of M1 scope. The interface supports it (put overwrites); a rotation job is M3.
• Reference: AWS Secrets Manager Node SDK — docs.aws.amazon.com/secretsmanager.

R-005 — Go Relay Agent: install script, systemd, WebSocket, SSH whitelist hook

Scope: M1 Wave D (REQ-010..013, REQ-026 whitelist hook). The SSH adapter itself is M2.

Install script (modular):

• Separate functions: detect_os, install_binary, write_systemd_unit, register_target, main.
• detect_os: reads /etc/os-release, parses ID + VERSION_ID. Supported: ubuntu ≥ 24.04, debian ≥ 12. Anything else → exit non-zero with a clear message listing supported OS + versions (Edge 16).
• install_binary: downloads the static Go binary for the detected arch (uname -m → amd64/arm64) from the control plane's release URL. Verifies SHA256 checksum. Installs to /usr/local/bin/coreci-relay-agent. Fallback: apt package from a configured repo (documented; same script path, different binary source).
• write_systemd_unit: writes /etc/systemd/system/coreci-relay-agent.service with ExecStart, Restart=on-failure, RestartSec=5, WantedBy=multi-user.target, Environment=CORECI_CONFIG=/etc/coreci/relay.env. systemctl daemon-reload && systemctl enable --now coreci-relay-agent.
• register_target: writes /etc/coreci/relay.env with CORECI_TENANT_TOKEN=<token> (the tenant registration token issued by the dashboard), CORECI_SAAS_URL=https://.... The token is a secret-manager reference bootstrap — the agent uses it to authenticate the first WebSocket; long-lived credentials are issued by the control plane post-registration.
• Pitfall: curl|bash anti-patterns — always download to a temp file, verify checksum before executing, never pipe to a shell that runs as root without a checksum gate. The install script is curl -fsSL https://.../install.sh | sh but the script itself verifies the binary checksum before install.
• Pitfall: Idempotency — re-running the script must upgrade, not fail. install_binary overwrites; write_systemd_unit overwrites + reloads; register_target preserves an existing token.

Go binary:

• WebSocket client: gorilla/websocket or nhooyr.io/websocket. Outbound wss://<saas>/api/relay/ws. Auth: Authorization: Bearer <tenant_token> on the initial handshake.
• Registration: first message after connect is {type: "register", tenantId, hostname, os, osVersion, ip, agentVersion}. Control plane responds {type: "registered", targetId}.
• Heartbeat: send {type: "ping", ts} every 30s; control plane echoes {type: "pong", ts}. If no pong within 60s, drop + reconnect. last_seen updated on every ping → dashboard green/yellow/red.
• Reconnect: exponential backoff (1s, 2s, 4s, 8s, 16s), max 5 attempts → log alert + keep trying every 60s. systemd Restart=on-failure handles hard crashes.
• Pitfall: Clock skew — use server time for last_seen, not agent time.
• Pitfall: TLS — pin the SaaS cert via the system trust store; never allow self-signed in prod (dev only flag).

SSH whitelist hook (M1 ships format + hook, M2 plugs adapter):

• Whitelist file: /etc/coreci/ssh-whitelist.json, shipped with the binary. Format: {"commands": ["cat", "ls", "systemctl status", "journalctl", "df", "du", "ps", "top", "ss", "netstat", "ip", "uptime", "uname", "free", "who", "w", "last", "dmesg", "lscpu", "lspci", "lsblk", "mount", "findmnt", "hostname", "ip addr", "ip route", "ss -tlnp"], "arguments": {"deny": ["-exec", "-execdir", "--exec", "|", ">", ">>", "&", ";", "&&", "||"]}}.
• Enforcement hook: a Go function CheckCommand(cmd string) error that parses the command, checks the base command against the whitelist, checks arguments against the deny list, and returns an error if rejected. M2's SSH adapter calls CheckCommand before exec.Command. M1 ships the function + a unit test + the whitelist file; no SSH execution path yet.
• Pitfall: find -exec is the classic whitelist escape — deny -exec/-execdir. Argument deny list catches redirections and shell operators.
• Reference: systemd unit docs — systemd.io ; gorilla/websocket — github.com/gorilla/websocket.

R-006 — Vanta evidence collection (M3 — noted, not built in M1)

Scope: M3 only (REQ-042). Recorded here so M1 foundations don't block M3 instrumentation.

Findings:

• Vanta collects evidence via integrations (AWS, GitHub, HR systems) and via custom controls that call Vanta's API. The control plane exposes a /vanta/evidence endpoint that Vanta polls for control evidence (access logs, change management, vendor risk, incident response).
• M1 action: ensure audit_log is queryable by an admin-scoped read role (not the app role) so M3's Vanta exporter can read it without bypassing RLS. The exporter runs as a tenant-scoped admin reader.
• No M1 build. Just the architectural note.

R-007 — Cross-tenant isolation test pattern (REQ-039 pen test)

Scope: M1 acceptance gate requires "cross-tenant isolation pen test result (must show zero leakage)".

Findings:

• Test pattern: create two tenants (T1, T2), each with a target + a BYOM endpoint. Issue a query as T1's user attempting to read T2's data (direct table scan, join, subquery, SET app.tenant_id bypass attempt). Assert every query returns zero T2 rows.
• Test the withTenant wrapper: any query outside withTenant must throw, not return rows.
• Test the audit log: T1's audit entries are invisible to T2's admin export.
• Reference: This becomes an integration test in Wave A + a dedicated pen-test script for the M1 review.