nextcraft/.ciagent/ARCHITECTURE.md

# Nextcraft — ARCHITECTURE.md

## Overview

Nextcraft is a TypeScript monorepo (pnpm workspaces + turborepo) with a Next.js application hosting four surfaces (Learner, Marketplace, Employer Dashboard, Admin), a shared component library, typed mock data layer, and shared types package. As of v0.2, a Python FastAPI application (`apps/ai-service`) hosts six AI tutor agents backed by a provider-agnostic LLM layer.

**v0.2 additions:** real AI services (streaming chat), agent framework, mock engine inputs for Lab/Assessor/Proctor. **Still no database, no auth** — in-memory session store; real engines (sandbox fabric, assessment engine, identity) are v0.3+.

### Confirmed Technology Stack (v0.2)

| Technology | Version | Purpose |
|------------|---------|---------|
| Node.js | v24.15.0 | Runtime (web) |
| pnpm | 12.3.4 | Package manager + workspaces |
| turborepo | 2.3.3 | Build orchestration |
| Next.js | 15 (App Router) | Web application framework |
| React | 19+ | UI library |
| TypeScript | 5.x | Type system |
| Tailwind CSS | v4 | Utility-first CSS |
| lucide-react | latest | Icons |
| recharts | latest | Charts |
| @xyflow/react | latest | Competency graph viewer |
| Python | 3.11.2 | Runtime (ai-service) |
| FastAPI | 0.141.x | AI service framework |
| uvicorn | 0.52.x | ASGI server |
| pydantic | 2.13.x | Request/response models, structured outputs |
| pydantic-settings | 2.15.x | Settings + env-file loading (replaces python-dotenv) |
| httpx | 0.28.x | Async LLM HTTP client (ollama-cloud + local providers) |
| sse-starlette | 3.4.x | SSE framing, ping keep-alive |
| pytest | 9.x | Test runner |
| pytest-asyncio | 1.4.x | Async tests (auto mode) |
| ruff | latest | Python lint (check-only, no formatter) — `pnpm ai:lint` |
| ollama-cloud | https://ollama.com/v1 | Default LLM provider (OpenAI-compatible, Bearer auth) |

Deliberately **not** used: openai-python SDK (the `LLMProvider` protocol is the port; raw httpx keeps delta passthrough and ollama-cloud quirk tolerance), python-dotenv (pydantic-settings reads .env natively), respx (httpx MockTransport is built in).

### v0.2 Architecture Decisions (from Research)

1. **D-016 SSE envelope** — `meta` event (agent/session/model, flushed before first token) → raw OpenAI-compatible chunk passthrough → optional `done` → `error` event before `[DONE]` on mid-stream failure. Pre-first-byte failures use proper HTTP status. Headers: `Cache-Control: no-cache`, `X-Accel-Buffering: no`.
2. **D-017 httpx direct client** — lifespan-managed `httpx.AsyncClient` (10s connect / 300s read), shared by ollama-cloud and local providers; no SDK.
3. **D-018 Agent framework** — `BaseAgent` ABC (system_prompt/build_messages/stream_reply/structured_reply) + explicit registry; prompts are versioned code in `prompts/`.
4. **D-019 Session store** — `SessionStore` protocol + `InMemorySessionStore` (asyncio.Lock, 20-message window, 500-cap LRU, agent-scoped sessions). DB-migration-ready.
5. **D-020 Structured outputs** — 4-layer defense: `response_format` (auto-degrade) → prompt-embedded schema → fence-strip/first-balanced-object parse → single bounded retry.
6. **D-021 Mock corpus in Python** — `ai_service/corpus/` pydantic-typed, convention-aligned with TS `packages/mock-data` (shared IDs, cross-referencing headers); no codegen in v0.2.
7. **D-022 Monorepo integration** — zero-dependency shim `package.json` in apps/ai-service + `ai#*` turbo passthrough tasks (`cache:false, outputs:[]`) + root `ai:dev`/`ai:test` scripts + idempotent venv bootstrap.
8. **D-023 Testing** — pytest-asyncio auto mode; TestClient `client.stream()` for SSE; httpx MockTransport for byte-exact provider parser tests; scripted mock provider incl. failure modes. Tests never call the cloud.

---

## Components

### apps/ai-service — AI Tutor Service (v0.2 NEW)

| Component | Description | Boundaries | Depends On |
|-----------|-------------|------------|------------|
| `ai_service/main.py` | FastAPI app factory, lifespan (httpx client pool, provider factory), CORS (localhost only), /health | App entry | config, llm, agents, api |
| `ai_service/config.py` | pydantic-settings Settings (env_prefix="AI_", env_file, SecretStr key) | Configuration only | None |
| `ai_service/api/` | Endpoints: chat.py (POST /v1/chat/stream, SSE), lab.py, assessment.py, proctor.py, mentor.py; deps.py (DI) | Composes agents + sessions; never imported by llm/ or agents/ | agents, llm |
| `ai_service/llm/` | types.py (Message; ChatDelta/ChoiceDelta removed in P3 — no consumers), base.py (LLMProvider protocol), openai_compat.py (ollama-cloud + local), mock.py (deterministic), factory.py | Never imports agents/ or api/ | config |
| `ai_service/agents/` | base.py (BaseAgent ABC), registry.py, session.py (SessionStore), structured.py (JSON defense), coach/tutor/lab/assessor/proctor/mentor.py | Never imports api/ | llm, prompts, corpus |
| `ai_service/prompts/` | Per-agent system prompt constants + render_context functions (str.format_map) | Data only | None |
| `ai_service/corpus/` | Mock engine inputs: learner_context.py, telemetry.py (Lab/Proctor scenarios), artifacts.py (pre-baked artifacts, rubrics, transcripts) | Pydantic-typed; aligned with TS packages/mock-data by convention | None |
| `scripts/` | bootstrap.sh (venv + pip install idempotent), dev.sh (exports keys from .ciagent/.env.secrets → uvicorn), test.sh (pytest), lint.sh (ruff check, G-3) | Dev entry points | pyproject.toml |
| `tests/` | conftest.py (mock provider, settings override, TestClient), health, llm (MockTransport parser), agents (framework + per-agent), api (SSE stream tests) | Mock provider only — no cloud | all |

**Module boundary rules:** `llm/` never imports `agents/` or `api/`; `agents/` never imports `api/`; `api/` composes both via DI. `corpus/` is the only home of mock engine data. Prompts are code — versioned and reviewed in git.

### apps/web — Next.js Application

| Component | Description | Boundaries | Depends On |
|-----------|-------------|------------|------------|
| `app/(learner)/` | Learner surface route group: landing, catalog, competency stack, dashboard, byte viewer, sandbox mockup, assessment mockup | Learner-only routes and layouts | packages/ui, packages/mock-data, packages/types |
| `app/(marketplace)/` | Marketplace surface route group: job board, job detail, employer profile, search/filter, pricing | Marketplace-only routes and layouts | packages/ui, packages/mock-data, packages/types |
| `app/(employer)/` | Employer dashboard route group: overview, talent search, candidate profile, posting management | Employer-only routes and layouts | packages/ui, packages/mock-data, packages/types |
| `app/(admin)/` | Admin surface route group: overview, learner management, competency graph viewer, moderation | Admin-only routes and layouts | packages/ui, packages/mock-data, packages/types |
| `app/layout.tsx` | Root layout: theme provider, navigation shell, responsive container | All routes | packages/ui |
| `components/` | Surface-specific components (learner/, marketplace/, employer/, admin/) plus shared chrome (navigation-shell, header/footer, role-switcher, theme-provider, breadcrumbs, dark-mode-toggle) | App-level components | packages/ui |
| `hooks/` | use-chat-stream.ts — SSE client hook: fetch + ReadableStream, byte buffering + frame reassembly, idempotent AbortController cleanup | Client components only | ai-service SSE |
| `lib/` | sse.ts (shared SSE frame parser — CRLF normalization + `: ping` immunity, G-1), breadcrumbs.ts, format.ts | Pure utilities | None |

### packages/ui — Shared Component Library

| Component | Description | Boundaries | Depends On |
|-----------|-------------|------------|------------|
| `tokens/` | Design tokens as TS constants: colors, spacing, radii, shadows, breakpoints (mirrored as Tailwind v4 `@theme` tokens in apps/web globals.css) | Foundation layer — no dependencies | None |
| `primitives/` | Button, Input, Card, Badge, Avatar — each with a Storybook story | Atomic UI components | tokens, packages/types |

Composite/layout/theme components (navigation shell, tables, chat panels, graph viewer, theme provider) live in `apps/web/components/` as app-level components, not in packages/ui.

### packages/mock-data — Mock Data Layer

| Component | Description | Boundaries | Depends On |
|-----------|-------------|------------|------------|
| `competency-stacks.ts` | 5 competency stacks (AI Orchestration Engineer, AI Safety & Governance, Human-AI Product Designer, AI-Augmented Field Operator, Computational Sciences), each with 12-18 competencies | Typed mock data | packages/types |
| `jobs.ts` | 20+ mock AI-era job listings with skills, seniority, salary, match scores | Typed mock data | packages/types |
| `candidates.ts` | 15+ mock candidate profiles with artifacts, process traces, defense scores, microcredentials | Typed mock data | packages/types |
| `employers.ts` | 10+ mock employer profiles with logos, descriptions, open positions | Typed mock data | packages/types |
| `learner-progress.ts` | Mock learner progress data: active competencies, completion percentages, recent artifacts | Typed mock data | packages/types |
| `admin.ts` | Admin surface mock data: platform metrics, activity feed, system health, learner roster (admin view), moderation queues | Typed mock data | packages/types |
| `ai-scenarios.ts` | AI engine-input scenario IDs + display metadata for the learner agent panels; IDs string-identical to `ai_service/corpus/` (D-021) | Typed mock data | packages/types |

### packages/types — Shared Types

| Component | Description | Boundaries | Depends On |
|-----------|-------------|------------|------------|
| `domain.ts` | Competency, CompetencyStack, Microcredential, Artifact, ProcessTrace, OralDefense, AssessmentRubric | Domain types | None |
| `marketplace.ts` | Job, Employer, Candidate, JobPosting, TalentMatch, SearchFilter | Marketplace types | None |
| `user.ts` | Learner, Admin, EmployerUser, AgeGroup, Role | User types | None |
| `ui.ts` | Component props, theme config, breakpoint definitions | UI types | None |

---

## Data Flow

```
[packages/mock-data + packages/types]      [ai_service/corpus]
        │ (TS, web surfaces)                      │ (Python, agent inputs)
        ▼                                         ▼
[Next.js Route Groups]                    [ai-service agents]
  (learner)/ (marketplace)/                 coach tutor lab assessor proctor mentor
  (employer)/ (admin)/                                │
        │                                             ▼
        │      SSE (fetch + ReadableStream)     [LLMProvider]
        └────── client components ◄───────────────────┤
               http://localhost:8420          ollama-cloud / local / mock
                                               (https://ollama.com/v1)
```

- Web surfaces remain server-component-first; client components (chat, filters, graph viewer, dark mode toggle) fetch directly from ai-service over SSE (A-002: no Next.js API-route proxy in v0.2).
- The LLM provider layer is a dumb pipe — OpenAI-compatible chunks pass through byte-identical; envelope logic (meta/done/error) lives only in the API layer (D-016).
- Lab/Assessor/Proctor read mock scenarios from `ai_service/corpus/` — real engines are v0.3+.
- All automated tests use the deterministic mock provider; the cloud is for manual probes only.

---

## Build Order (v0.2)

1. **AI service scaffolding** — apps/ai-service: FastAPI app, config, provider layer (ollama-cloud/local/mock), SSE chat endpoint, pytest harness, turbo integration
2. **Agent framework** — BaseAgent, registry, session store, structured output, prompts scaffolding, learner-context corpus
3. **Coach + Tutor agents** — full implementations, chat endpoint agent routing
4. **Lab + Assessor agents** — telemetry scenarios + pre-baked artifacts corpus, /v1/lab/feedback + /v1/assessment/evaluate
5. **Proctor + Mentor agents** — proctor scenarios, /v1/proctor/signals + /v1/mentor/narrative
6. **Learner surface integration** — useChatStream hook, agent switcher, streaming/error/loading states, agent output panels across the four learner surfaces

The v0.1 build order (monorepo → types → mock data → tokens → primitives → layout → composites → surfaces → polish) is complete and preserved in git history (tags v0.0.1–v0.1.0).

---

## Future Architecture (Post-v0.2, for reference)

v0.2 delivers the ai-service skeleton that later milestones fill in:

- **Mock corpus → real engines** — Lab consumes real sandbox telemetry (v0.3 sandbox fabric); Assessor grades real process traces (v0.3 assessment engine); Proctor consumes real identity/attention signals (v0.3 identity verification)
- **In-memory sessions → PostgreSQL + Drizzle ORM** — SessionStore protocol swap, no API changes
- **Mock provider → per-agent model routing** — provider factory already selects by config; per-agent `AI_<AGENT>_MODEL` overrides
- **No auth → real KYC + sessions** — A-008 dropped in v0.3 when identity verification lands
- **No search → Semantic vector search (pgvector)** — Filter UI replaced with vector similarity search
- **No payments → Payment processing** — Pricing page replaced with real subscription/payment flows

The monorepo structure (apps/web + apps/ai-service + packages/*) accommodates further apps without restructuring.