praxis/.ciagent/ARCHITECTURE.md

Praxis — Architecture (Research-Refined)

Status: Research-refined (Phase 0 RESEARCH stage). Informed by .ciagent/RESEARCH.md — web-verified vendor catalogs, GitHub metadata, official docs.

High-Level Topology

Three-tier architecture per PRD §7:

┌──────────────────────────────────────────────────────────┐
│  Client (Android, iOS, Web, WhatsApp, USSD)              │
│  - Voice I/O, cached scenarios, offline scenarios        │
└────────────────┬─────────────────────────────────────────┘
                 │
┌────────────────▼─────────────────────────────────────────┐
│  Edge / Region (per market)                              │
│  - ASR + TTS (low-latency, local accent models)          │
│  - Scenario runtime + role orchestration                 │
│  - Caching layer                                         │
└────────────────┬─────────────────────────────────────────┘
                 │
┌────────────────▼─────────────────────────────────────────┐
│  Core Platform                                           │
│  - LLM tutor (long-context, persona-aware, safety-tuned) │
│  - Scenario Authoring & Tagging                          │
│  - Mastery Rubric Engine                                 │
│  - User state, progress, credentialing                  │
│  - Analytics                                             │
└──────────────────────────────────────────────────────────┘

LLM Foundation (D-003, D-020 — research-verified)

Open-weights models hosted via Ollama Cloud direct API (https://ollama.com/api/chat + OLLAMA_API_KEY) — no local daemon required for v0.1.

Model	Verified status	Role	Context	Mode
gemma4:cloud	✅ Real, current (256K ctx, Text+Image, "Low Usage" tier)	Role-play fast path / persona turns	256K	standard
deepseek-v4-flash:cloud	✅ Real, current (1M ctx, 284B MoE / 13B active, "Medium Usage" tier)	Coaching debrief + scenario-branch decisions	1M	no-think (latency); think/max-think reserved for offline analysis

Post-pilot cost-reduction path: self-host gemma4:e4b (edge, native audio modality, 9.6GB) on partner hardware for the ≤$3/learner/month target. Architecture must keep the model-call layer swappable (D-020).

Notable future option: gemma4:e2b/e4b support Text+Image+Audio input — potential future Ollama-hosted ASR for cost reduction (not v0.1; dedicated Deepgram is lower-latency + more accent-robust).

v0.1 Component Map (research-refined minimal viable voice loop)

Client: React + WebRTC (Pipecat client SDK)
   │  audio in/out (WebRTC, UDP, sub-50ms)
   ▼
Pipecat server (Python)
   ├─ VAD: Silero
   ├─ STT: Deepgram Nova-3 (cloud, streaming, WebSocket)
   ├─ LLM: Ollama Cloud direct API (https://ollama.com/api/chat)
   │     ├─ gemma4:cloud         (role-play fast path)
   │     └─ deepseek-v4-flash:cloud (debrief, no-think mode)
   ├─ TTS: Cartesia Sonic (cloud, ~120ms)  ← behind interface
   │     └─ fallback: Piper (self-hosted, ~80ms)  ← R4 mitigation
   ├─ Scenario runtime: Pipecat Flows + YAML→Pydantic scenarios
   ├─ Guardrail layer: pluggable interface (v0.1: Customer Service ruleset)
   └─ Learner state: SQLite (praxis.db, single-learner, no auth)

v0.1 deliberately excludes: edge-region split, multi-market deployment, caching layer, scenario authoring tools, mastery engine, credentialing, analytics, WhatsApp/USSD surfaces.

Latency Budget (< 600ms end-to-end — research-revised)

Segment	Budget	Source / note
Client capture + WebRTC uplink	~50ms	WebRTC UDP, Canada region
ASR (Deepgram Nova-3 first partial)	~250ms	Vendor claim; R1: measure in Phase 1
LLM first token (gemma4:cloud direct API)	~200ms	R3: measure in Phase 1
TTS first audio (Cartesia Sonic)	~120ms	Vendor/leaderboard; R2: measure in Phase 1
WebRTC downlink + playback	~50ms
Total (all-cloud target)	~670ms	⚠️ Marginally over 600ms
Total (Piper TTS mitigation)	~550ms	R4: pre-stage Piper self-hosted on pilot server

R4 — single biggest v0.1 technical risk: the all-cloud three-hop path likely lands ~670ms. The TTS service MUST sit behind an interface (D-014) and Piper-on-pilot-server MUST be pre-staged as the likely production v0.1 TTS. This is the first Phase 1 spike.

Critical Risks to Engineer Around

1. Accent robustness — even a great LLM fails if ASR mishears the learner. Canadian English/French accents, code-switching.
2. Hallucinated advice in safety-sensitive domains — health, electrical. Domain-specific guardrails, escalation, disclaimers. (v0.1 uses Customer Service path, lower risk, but architecture must support the guardrail layer.)
3. Cost per learner per month must stay ≤ $3 in target markets. v0.1 Canada pilot relaxes this, but architecture must not bake in assumptions that violate it.
4. Ollama model availability / cost — :cloud variants imply hosted inference; verify pricing and rate limits at research phase.

Deployment (v0.1)

• Single-region pilot (Canada)
• LLM via Ollama Cloud direct API (no local daemon)
• ASR via Deepgram cloud (North American endpoint)
• TTS: Cartesia cloud (quality benchmark) + Piper self-hosted on pilot server (R4 latency mitigation, likely production v0.1)
• Pipecat server on single pilot host (Python)
• Client: React web app (Pipecat client SDK, WebRTC transport)
• SQLite local file (praxis.db) on pilot host

Open Architecture Questions (resolved by research)

Question (from initial ARCHITECTURE.md)	Resolution
Client framework	React + WebRTC via Pipecat client SDK (D-015)
Streaming transport	WebRTC (Pipecat); WebSocket dev fallback (D-016)
ASR/TTS provider	Deepgram Nova-3 (ASR, D-013); Cartesia Sonic + Piper fallback (TTS, D-014)
Learner state store	SQLite confirmed (D-007 → 0.90)
Ollama deployment	Ollama Cloud direct API (D-020)
Scenario definition format	YAML DSL → Pydantic → Pipecat Flows (D-018)

Open Architecture Questions (remaining for PLAN stage)

• R1-R4 latency spikes (see Risks below) — first Phase 1 tasks
• Pipecat Flows schema mapping for the one branch point (escalate vs accept) in the refund scenario
• Guardrail ruleset concrete implementation (D-019) — system-prompt template + output filter
• SQLite schema for session log + progress + scenario state
• OLLAMA_API_KEY + DEEPGRAM_API_KEY + CARTESIA_API_KEY secret management (extend config.secrets.scopes)

---

v0.2 Deployment Architecture (Proxmox LXC + Docker-in-LXC)

Status: Research-refined (v0.2 RESEARCH stage). Informed by .ciagent/RESEARCH.md — Proxmox VE wiki, coreci script analysis, Docker/systemd ecosystem. Decisions: D-021 (LXC deploy), D-022 (Docker in LXC, nesting=1), D-023 (FastAPI StaticFiles), D-024 (infra-only keys), D-025/D-029 (build inside CT), D-026 (coreci secrets), D-027 (auto VMID), D-028 (Docker via apt), D-030 (vmbr0 DHCP).

Docker-in-LXC Topology

                    ┌─────────────────────────────────────────────────────────┐
                    │  Proxmox VE Host (PROXMOX_NODE)                          │
                    │  (D-026: secrets sourced from ~/coreci/.ciagent/         │
                    │   .env.secrets + praxis .ciagent/.env.secrets)           │
                    │                                                          │
                    │  Deploy operator runs:                                   │
                    │    scripts/proxmox/lxc-deploy.sh                         │
                    │     ├─ stage-snippet.sh  (upload hookscript to snippets) │
                    │     ├─ lxc-clone.sh      (POST /nodes/{node}/lxc)        │
                    │     ├─ lxc-config.sh     (PUT /config + SSH lxc.env)     │
                    │     ├─ lxc-start.sh      (POST /status/start)            │
                    │     └─ health-check.sh   (poll /health:8789)             │
                    │                                                          │
                    │  ┌────────────────────────────────────────────────────┐  │
                    │  │  LXC Container (VMID: auto via pve_nextid, D-027)  │  │
                    │  │  hostname: praxis                                   │  │
                    │  │  memory: 4096MB   rootfs: 16GB   (bumped from 2/8) │  │
                    │  │  features: nesting=1                                │  │
                    │  │  net0: bridge=vmbr0, ip=dhcp  (D-030)              │  │
                    │  │  hookscript: local:snippets/praxis-firstboot.sh     │  │
                    │  │  lxc.environment: GITEA_TOKEN, DEEPGRAM_API_KEY,   │  │
                    │  │    PRAXIS_PORT=8789, PRAXIS_HOST=0.0.0.0, ...       │  │
                    │  │                                                    │  │
                    │  │  post-start hook (runs on PVE host, pct exec → CT): │  │
                    │  │   1. apt install docker.io docker-compose-v2 git    │  │
                    │  │   2. git clone praxis repo → /opt/praxis            │  │
                    │  │   3. install-service.sh (user + env + systemd unit) │  │
                    │  │   4. systemctl start praxis                         │  │
                    │  │      → ExecStartPre: docker compose build           │  │
                    │  │      → ExecStart: docker compose up (foreground)    │  │
                    │  │                                                    │  │
                    │  │  ┌──────────────────────────────────────────────┐  │  │
                    │  │  │  Docker daemon                                │  │  │
                    │  │  │  ┌────────────────────────────────────────┐  │  │  │
                    │  │  │  │  praxis container                       │  │  │  │
                    │  │  │  │  image: python:3.12-slim + deps + dist │  │  │  │
                    │  │  │  │  ports: 8789:8789                      │  │  │  │
                    │  │  │  │  env_file: /etc/praxis/server.env      │  │  │  │
                    │  │  │  │  volume: praxis-db → /app/data         │  │  │  │
                    │  │  │  │  restart: unless-stopped               │  │  │  │
                    │  │  │  │                                         │  │  │  │
                    │  │  │  │  uvicorn 0.0.0.0:8789                  │  │  │  │
                    │  │  │  │  ├─ GET  /health       (FastAPI)       │  │  │  │
                    │  │  │  │  ├─ POST /pipecat/webrtc (FastAPI)     │  │  │  │
                    │  │  │  │  └─ GET  / ... (StaticFiles client/dist)│  │  │  │
                    │  │  │  └────────────────────────────────────────┘  │  │  │
                    │  │  └──────────────────────────────────────────────┘  │  │
                    │  └────────────────────────────────────────────────────┘  │
                    │           │                                              │
                    │       vmbr0 (bridge) ──── DHCP ──── CT eth0              │
                    └───────────┬──────────────────────────────────────────────┘
                                │  <ct-bridge-ip>:8789
                    ┌───────────▼───────────────────────┐
                    │  Operator / Learner (browser)     │
                    │  http://<ct-ip>:8789               │
                    │  (direct access, no proxy/TLS)    │
                    └───────────────────────────────────┘

Image Build Pipeline (Multi-stage Dockerfile)

Two-stage build, Debian-slim bases, python -m server entrypoint:

Stage 1: client-builder (node:22-slim)
  COPY client/package.json client/package-lock.json
  RUN npm ci                          ← cached unless deps change
  COPY client/
  RUN npm run build                   ← tsc -b && vite build → client/dist/

Stage 2: server (python:3.12-slim)
  RUN apt-get install gcc g++ libasound2-dev   ← only if source compilation
  COPY pyproject.toml
  RUN pip install --no-cache-dir .   ← pipecat-ai[deepgram,cartesia,piper,webrtc] + deps
  COPY server/ scenarios/ db/
  COPY --from=client-builder /app/client/dist ./client/dist
  EXPOSE 8789
  CMD ["python", "-m", "server"]      ← calls uvicorn.run(app, host=HOST, port=PORT)

Why Debian-slim (not Alpine): numpy + pipecat-ai native extensions compile against glibc; musl wheels are less universally available. The ~50MB size saving of Alpine isn't worth the compatibility risk.

Why python -m server (not uvicorn server.__main__:app): Matches the existing entrypoint (server/__main__.py:main()) which reads PRAXIS_HOST/PRAXIS_PORT from env and calls uvicorn.run(...). Single uvicorn process is correct for WebRTC/WebSocket (long-lived connections, not request-per-response).

Secret Injection Chain

~/coreci/.ciagent/.env.secrets          praxis/.ciagent/.env.secrets
  PROXMOX_API_URL                         GITEA_TOKEN
  PROXMOX_API_TOKEN                       DEEPGRAM_API_KEY
  PROXMOX_NODE                            CARTESIA_API_KEY (empty, D-024)
  PROXMOX_STORAGE                         OLLAMA_API_KEY (empty, D-024)
  PROXMOX_TEMPLATE_VOLID
  PROXMOX_TLS_SKIP_VERIFY
           │                    │
           └────────┬───────────┘
                    ▼
          lxc-deploy.sh sources both
                    │
                    ▼
          lxc-config.sh (SSH to PVE host)
          writes /etc/pve/lxc/<vmid>.conf:
            lxc.environment: GITEA_TOKEN=<token>
            lxc.environment: DEEPGRAM_API_KEY=<key>
            lxc.environment: PRAXIS_PORT=8789
            lxc.environment: PRAXIS_HOST=0.0.0.0
            lxc.environment: OLLAMA_BASE_URL=https://ollama.com/v1
            ...
                    │
                    ▼  (CT boots; systemd PID 1 has these env vars)
          firstboot-hook.sh → pct exec install-service.sh
                    │
                    ▼
          /etc/praxis/server.env  (root:praxis, chmod 0640)
            GITEA_TOKEN=<token>
            DEEPGRAM_API_KEY=<key>
            PRAXIS_PORT=8789
            ...
                    │
                    ▼
          praxis.service (EnvironmentFile=/etc/praxis/server.env)
          → ExecStart: docker compose up
                    │
                    ▼
          docker-compose.yml (env_file: /etc/praxis/server.env)
                    │
                    ▼
          Docker container (os.environ)
          → server/__main__.py reads PRAXIS_HOST, PRAXIS_PORT, DEEPGRAM_API_KEY, ...

.gitignore coverage: .env, .env.secrets, .env.* are all gitignored in praxis (verified). No secrets are committed.

CT Resource Sizing

Resource	Coreci default	Praxis v0.2	Rationale
Memory	2048 MB	4096 MB	Docker daemon (~200MB) + build peak (~1.2GB pip) + runtime (~500MB) + headroom
Rootfs	8 GB	16 GB	Docker engine (~400MB) + build layers (~1.6GB) + final image (~1GB) + repo + apt + headroom
CPU cores	(default)	2	Sufficient for build + single-learner runtime
Swap	(default)	0	LXC swap is host swap; not needed for pilot

Configured via lxc-clone.sh (memory=${PROXMOX_MEMORY_MB:-4096}, rootfs=${storage}:16) or env vars in the deploy script.

Health-Check Path

lxc-deploy.sh
  └─ health-check.sh <vmid>
       │
       ├─ PRAXIS_HEALTH_URL set? → use directly
       │
       └─ else: pve_get /nodes/{node}/lxc/{vmid}/interfaces
            │
            ├─ jq: .[] | select(.name != "lo") | (.inet? // .ip? // empty)
            │  (NOT .hwaddr — P18 bug fix from coreci)
            │
            └─ health_url = http://<bridge-ip>:8789/health
                 │
                 └─ poll curl -fsS --connect-timeout 2 $health_url
                    for PRAXIS_HEALTH_TIMEOUT seconds (default 300s)

Timing: CT start → DHCP lease (~5s) → firstboot hook: apt install Docker (~90s) + git clone (~10s) + install-service + systemctl start (~120s: docker compose build + up) → uvicorn binds :8789 → health passes. Total: ~3-5 min. PRAXIS_HEALTH_TIMEOUT=300 (5 min) covers this with margin.

Firstboot Hook Sequence

Proxmox invokes hookscript at post-start phase (runs on PVE HOST):
  $1 = VMID, $2 = phase

  Phase: post-start
    │
    ├─ 1. pct exec <vmid> -- apt-get install docker.io docker-compose-v2 git curl
    │     (D-028: Docker via apt inside CT)
    │
    ├─ 2. pct exec <vmid> -- git clone https://<GITEA_TOKEN>@git.cloudinit.dev/coreci/praxis.git /opt/praxis
    │     (D-029: clone inside CT, self-contained)
    │
    ├─ 3. pct exec <vmid> -- sh /opt/praxis/scripts/install-service.sh
    │     │
    │     ├─ create praxis user (useradd --system, add to docker group)
    │     ├─ mkdir /var/lib/praxis/data /var/log/praxis /etc/praxis
    │     ├─ write /etc/praxis/server.env from lxc.environment vars
    │     ├─ install praxis.service systemd unit
    │     └─ systemctl daemon-reload && enable praxis && restart praxis
    │           │
    │           ├─ ExecStartPre: docker compose build  (TimeoutStartSec=300)
    │           └─ ExecStart: docker compose up  (foreground, Type=simple)
    │
    └─ 4. (hook exits 0; external health-check.sh polls /health:8789)

Idempotency: The hook checks if praxis is already installed + active before re-running (mirrors coreci's pattern at firstboot-hook.sh:82). Re-running lxc-deploy.sh against a healthy CT skips the hook entirely (P16 idempotency via ct_exists + ct_running + health-check).

What's Reused Verbatim from CoreCI vs Adapted

Component	Verdict	Notes
api.sh	Verbatim	REQ-DEPLOY-03. PVE REST helpers are project-agnostic.
lxc-start.sh	Verbatim	POST /status/start is identical.
proxy/ct-exists.sh	Verbatim	Used by lxc-deploy.sh idempotency; no proxy dependency in the helper.
lxc-clone.sh	Adapted	hostname=praxis, memory=4096, rootfs=16, features=nesting=1 (kept).
lxc-config.sh	Adapted	hookscript=praxis-firstboot.sh, lxc.environment vars for praxis.
health-check.sh	Adapted	/health (not /healthz), port 8789, PRAXIS_* env names, timeout 300s.
rollback.sh	Adapted	Remove proxy backend-remove (no proxy in v0.2).
stage-snippet.sh	Adapted	SNIPPET_NAME=praxis-firstboot.sh, praxis repo raw URL.
timing.sh	Adapted	Metric prefix: praxis_deploy_timing_.
lxc-deploy.sh	Adapted	Remove PROXY_VMID/BACKEND_DOMAIN steps; VMID=auto (D-027).
firstboot-hook.sh	Heavy adaptation	Docker install + git clone + compose build/up (not host-fetch binary).
install-service.sh	Heavy adaptation	praxis user (docker group), /etc/praxis/server.env, praxis.service (docker compose up).

v0.2 Deployment Risks (from RESEARCH.md)

ID	Risk	Mitigation
R-DEPLOY-01	Pipecat wheel missing → source compilation OOM	Pre-test docker build locally; bump memory if needed
R-DEPLOY-02	systemd TimeoutStartSec insufficient for build+up	Set 300-600s or split build into separate oneshot service
R-DEPLOY-03	CT can't reach Gitea/apt mirrors	Validate internet access; fallback to host-clone+pct-push (D-025 hybrid)
R-DEPLOY-04	Docker-in-LXC on ZFS rootfs	Check storage type; use local (directory) if ZFS
R-DEPLOY-05	journald log flooding from compose up	Log rotation or StandardOutput=null for pilot
R-DEPLOY-06	First-boot build > 5 min (NFR breach)	Pre-build on host + docker load fallback