docs(milestone): complete v0.2-ai-tutor-architecture
---ci--- phase: 7 milestone: v0.2 status: complete requirements: covered: [REQ-2-001, REQ-2-002, REQ-2-003, REQ-2-004, REQ-2-005, REQ-2-006, REQ-2-007, REQ-2-008, REQ-2-009, REQ-2-010, REQ-2-011, REQ-2-012] partial: [] ---/ci--- Milestone v0.2 (ai-tutor-architecture) merged to main. Escalation record (audit remediation, durable): P1 executor delegation failed twice (empty subagent results, zero files created); auto-resolved at full autonomy to inline execution with identical plan fidelity (commit 3271373, reflog-only after phase branch squash-delete).
This commit is contained in:
CIAgent
+88
-64
@@ -2,42 +2,69 @@
|
|||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
Nextcraft v0.1 is a high-fidelity UI/UX prototype built as a TypeScript monorepo using pnpm workspaces and turborepo. The prototype consists of a single Next.js application with route groups for each surface (Learner, Marketplace, Employer Dashboard, Admin), backed by a shared component library, typed mock data layer, and shared types package.
|
Nextcraft is a TypeScript monorepo (pnpm workspaces + turborepo) with a Next.js application hosting four surfaces (Learner, Marketplace, Employer Dashboard, Admin), a shared component library, typed mock data layer, and shared types package. As of v0.2, a Python FastAPI application (`apps/ai-service`) hosts six AI tutor agents backed by a provider-agnostic LLM layer.
|
||||||
|
|
||||||
**No backend, no database, no authentication logic.** All data is static/mock. The architecture is designed to be replaced piece-by-piece with real backend services in future milestones.
|
**v0.2 additions:** real AI services (streaming chat), agent framework, mock engine inputs for Lab/Assessor/Proctor. **Still no database, no auth** — in-memory session store; real engines (sandbox fabric, assessment engine, identity) are v0.3+.
|
||||||
|
|
||||||
### Confirmed Technology Stack (Research Findings)
|
### Confirmed Technology Stack (v0.2)
|
||||||
|
|
||||||
| Technology | Version | Purpose |
|
| Technology | Version | Purpose |
|
||||||
|------------|---------|---------|
|
|------------|---------|---------|
|
||||||
| Node.js | v24.15.0 | Runtime |
|
| Node.js | v24.15.0 | Runtime (web) |
|
||||||
| pnpm | 12.3.4 | Package manager + workspaces |
|
| pnpm | 12.3.4 | Package manager + workspaces |
|
||||||
| turborepo | latest | Build orchestration |
|
| turborepo | 2.3.3 | Build orchestration |
|
||||||
| Next.js | latest (App Router) | Web application framework |
|
| Next.js | 15 (App Router) | Web application framework |
|
||||||
| React | 19+ | UI library |
|
| React | 19+ | UI library |
|
||||||
| TypeScript | 5.x | Type system |
|
| TypeScript | 5.x | Type system |
|
||||||
| Tailwind CSS | v4 | Utility-first CSS framework |
|
| Tailwind CSS | v4 | Utility-first CSS |
|
||||||
| lucide-react | latest | Icon system |
|
| lucide-react | latest | Icons |
|
||||||
| recharts | latest | Charts for employer/admin dashboards |
|
| recharts | latest | Charts |
|
||||||
| @xyflow/react (react-flow) | latest | Competency graph viewer in admin surface |
|
| @xyflow/react | latest | Competency graph viewer |
|
||||||
| Inter font | via next/font | Typography |
|
| Python | 3.11.2 | Runtime (ai-service) |
|
||||||
| ESLint | via Next.js | Linting |
|
| FastAPI | 0.141.x | AI service framework |
|
||||||
| Prettier | latest | Code formatting |
|
| uvicorn | 0.52.x | ASGI server |
|
||||||
|
| pydantic | 2.13.x | Request/response models, structured outputs |
|
||||||
|
| pydantic-settings | 2.15.x | Settings + env-file loading (replaces python-dotenv) |
|
||||||
|
| httpx | 0.28.x | Async LLM HTTP client (ollama-cloud + local providers) |
|
||||||
|
| sse-starlette | 3.4.x | SSE framing, ping keep-alive |
|
||||||
|
| pytest | 9.x | Test runner |
|
||||||
|
| pytest-asyncio | 1.4.x | Async tests (auto mode) |
|
||||||
|
| ruff | latest | Python lint (check-only, no formatter) — `pnpm ai:lint` |
|
||||||
|
| ollama-cloud | https://ollama.com/v1 | Default LLM provider (OpenAI-compatible, Bearer auth) |
|
||||||
|
|
||||||
### Architecture Decisions from Research
|
Deliberately **not** used: openai-python SDK (the `LLMProvider` protocol is the port; raw httpx keeps delta passthrough and ollama-cloud quirk tolerance), python-dotenv (pydantic-settings reads .env natively), respx (httpx MockTransport is built in).
|
||||||
|
|
||||||
1. **Next.js App Router with route groups** — `(learner)`, `(marketplace)`, `(employer)`, `(admin)` provide clean URL separation without affecting paths
|
### v0.2 Architecture Decisions (from Research)
|
||||||
2. **Tailwind CSS v4** — Configured via `@theme` in CSS, no `tailwind.config.js` needed (v4 paradigm shift). Dark mode via `class` strategy.
|
|
||||||
3. **Server components by default** — All pages are server components. Client components only for interactive elements (filters, search, chat, graph viewer, dark mode toggle)
|
1. **D-016 SSE envelope** — `meta` event (agent/session/model, flushed before first token) → raw OpenAI-compatible chunk passthrough → optional `done` → `error` event before `[DONE]` on mid-stream failure. Pre-first-byte failures use proper HTTP status. Headers: `Cache-Control: no-cache`, `X-Accel-Buffering: no`.
|
||||||
4. **Mock data as ES modules** — Typed TS files exported from `packages/mock-data`. No JSON files — all mock data is programmatically generated for richer structure.
|
2. **D-017 httpx direct client** — lifespan-managed `httpx.AsyncClient` (10s connect / 300s read), shared by ollama-cloud and local providers; no SDK.
|
||||||
5. **react-flow (@xyflow/react)** — Confirmed for competency graph viewer. Provides interactive node/edge rendering with built-in controls.
|
3. **D-018 Agent framework** — `BaseAgent` ABC (system_prompt/build_messages/stream_reply/structured_reply) + explicit registry; prompts are versioned code in `prompts/`.
|
||||||
6. **recharts** — Confirmed for analytics dashboards. Responsive, composable, integrates well with React server components.
|
4. **D-019 Session store** — `SessionStore` protocol + `InMemorySessionStore` (asyncio.Lock, 20-message window, 500-cap LRU, agent-scoped sessions). DB-migration-ready.
|
||||||
7. **pnpm workspaces** — `apps/web` + `packages/ui` + `packages/mock-data` + `packages/types`. Shared deps hoisted.
|
5. **D-020 Structured outputs** — 4-layer defense: `response_format` (auto-degrade) → prompt-embedded schema → fence-strip/first-balanced-object parse → single bounded retry.
|
||||||
|
6. **D-021 Mock corpus in Python** — `ai_service/corpus/` pydantic-typed, convention-aligned with TS `packages/mock-data` (shared IDs, cross-referencing headers); no codegen in v0.2.
|
||||||
|
7. **D-022 Monorepo integration** — zero-dependency shim `package.json` in apps/ai-service + `ai#*` turbo passthrough tasks (`cache:false, outputs:[]`) + root `ai:dev`/`ai:test` scripts + idempotent venv bootstrap.
|
||||||
|
8. **D-023 Testing** — pytest-asyncio auto mode; TestClient `client.stream()` for SSE; httpx MockTransport for byte-exact provider parser tests; scripted mock provider incl. failure modes. Tests never call the cloud.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Components
|
## Components
|
||||||
|
|
||||||
|
### apps/ai-service — AI Tutor Service (v0.2 NEW)
|
||||||
|
|
||||||
|
| Component | Description | Boundaries | Depends On |
|
||||||
|
|-----------|-------------|------------|------------|
|
||||||
|
| `ai_service/main.py` | FastAPI app factory, lifespan (httpx client pool, provider factory), CORS (localhost only), /health | App entry | config, llm, agents, api |
|
||||||
|
| `ai_service/config.py` | pydantic-settings Settings (env_prefix="AI_", env_file, SecretStr key) | Configuration only | None |
|
||||||
|
| `ai_service/api/` | Endpoints: chat.py (POST /v1/chat/stream, SSE), lab.py, assessment.py, proctor.py, mentor.py; deps.py (DI) | Composes agents + sessions; never imported by llm/ or agents/ | agents, llm |
|
||||||
|
| `ai_service/llm/` | types.py (Message; ChatDelta/ChoiceDelta removed in P3 — no consumers), base.py (LLMProvider protocol), openai_compat.py (ollama-cloud + local), mock.py (deterministic), factory.py | Never imports agents/ or api/ | config |
|
||||||
|
| `ai_service/agents/` | base.py (BaseAgent ABC), registry.py, session.py (SessionStore), structured.py (JSON defense), coach/tutor/lab/assessor/proctor/mentor.py | Never imports api/ | llm, prompts, corpus |
|
||||||
|
| `ai_service/prompts/` | Per-agent system prompt constants + render_context functions (str.format_map) | Data only | None |
|
||||||
|
| `ai_service/corpus/` | Mock engine inputs: learner_context.py, telemetry.py (Lab/Proctor scenarios), artifacts.py (pre-baked artifacts, rubrics, transcripts) | Pydantic-typed; aligned with TS packages/mock-data by convention | None |
|
||||||
|
| `scripts/` | bootstrap.sh (venv + pip install idempotent), dev.sh (exports keys from .ciagent/.env.secrets → uvicorn), test.sh (pytest), lint.sh (ruff check, G-3) | Dev entry points | pyproject.toml |
|
||||||
|
| `tests/` | conftest.py (mock provider, settings override, TestClient), health, llm (MockTransport parser), agents (framework + per-agent), api (SSE stream tests) | Mock provider only — no cloud | all |
|
||||||
|
|
||||||
|
**Module boundary rules:** `llm/` never imports `agents/` or `api/`; `agents/` never imports `api/`; `api/` composes both via DI. `corpus/` is the only home of mock engine data. Prompts are code — versioned and reviewed in git.
|
||||||
|
|
||||||
### apps/web — Next.js Application
|
### apps/web — Next.js Application
|
||||||
|
|
||||||
| Component | Description | Boundaries | Depends On |
|
| Component | Description | Boundaries | Depends On |
|
||||||
@@ -47,17 +74,18 @@ Nextcraft v0.1 is a high-fidelity UI/UX prototype built as a TypeScript monorepo
|
|||||||
| `app/(employer)/` | Employer dashboard route group: overview, talent search, candidate profile, posting management | Employer-only routes and layouts | packages/ui, packages/mock-data, packages/types |
|
| `app/(employer)/` | Employer dashboard route group: overview, talent search, candidate profile, posting management | Employer-only routes and layouts | packages/ui, packages/mock-data, packages/types |
|
||||||
| `app/(admin)/` | Admin surface route group: overview, learner management, competency graph viewer, moderation | Admin-only routes and layouts | packages/ui, packages/mock-data, packages/types |
|
| `app/(admin)/` | Admin surface route group: overview, learner management, competency graph viewer, moderation | Admin-only routes and layouts | packages/ui, packages/mock-data, packages/types |
|
||||||
| `app/layout.tsx` | Root layout: theme provider, navigation shell, responsive container | All routes | packages/ui |
|
| `app/layout.tsx` | Root layout: theme provider, navigation shell, responsive container | All routes | packages/ui |
|
||||||
| `components/` | Surface-specific components (not shared across surfaces) | Per-surface only | packages/ui |
|
| `components/` | Surface-specific components (learner/, marketplace/, employer/, admin/) plus shared chrome (navigation-shell, header/footer, role-switcher, theme-provider, breadcrumbs, dark-mode-toggle) | App-level components | packages/ui |
|
||||||
|
| `hooks/` | use-chat-stream.ts — SSE client hook: fetch + ReadableStream, byte buffering + frame reassembly, idempotent AbortController cleanup | Client components only | ai-service SSE |
|
||||||
|
| `lib/` | sse.ts (shared SSE frame parser — CRLF normalization + `: ping` immunity, G-1), breadcrumbs.ts, format.ts | Pure utilities | None |
|
||||||
|
|
||||||
### packages/ui — Shared Component Library
|
### packages/ui — Shared Component Library
|
||||||
|
|
||||||
| Component | Description | Boundaries | Depends On |
|
| Component | Description | Boundaries | Depends On |
|
||||||
|-----------|-------------|------------|------------|
|
|-----------|-------------|------------|------------|
|
||||||
| `design-tokens/` | CSS custom properties: color palette, typography scale, spacing system, breakpoints, shadows, radii | Foundation layer — no dependencies | None |
|
| `tokens/` | Design tokens as TS constants: colors, spacing, radii, shadows, breakpoints (mirrored as Tailwind v4 `@theme` tokens in apps/web globals.css) | Foundation layer — no dependencies | None |
|
||||||
| `primitives/` | Button, Input, Card, Badge, Avatar, Dialog, Tabs, Progress, Tooltip, Skeleton, Toast | Atomic UI components | design-tokens |
|
| `primitives/` | Button, Input, Card, Badge, Avatar — each with a Storybook story | Atomic UI components | tokens, packages/types |
|
||||||
| `composites/` | Navigation, Table, SearchBar, FilterPanel, ChatInterface, GraphViewer, ArtifactCard, CompetencyBadge, JobCard, CandidateCard, MetricCard | Composite components built from primitives | primitives, packages/types |
|
|
||||||
| `layouts/` | Container, Grid, Sidebar, SplitPanel, DashboardLayout | Layout components | primitives, design-tokens |
|
Composite/layout/theme components (navigation shell, tables, chat panels, graph viewer, theme provider) live in `apps/web/components/` as app-level components, not in packages/ui.
|
||||||
| `theme/` | Theme provider, CSS variable overrides per surface (learner, marketplace, employer, admin) | Theme context | design-tokens |
|
|
||||||
|
|
||||||
### packages/mock-data — Mock Data Layer
|
### packages/mock-data — Mock Data Layer
|
||||||
|
|
||||||
@@ -68,7 +96,8 @@ Nextcraft v0.1 is a high-fidelity UI/UX prototype built as a TypeScript monorepo
|
|||||||
| `candidates.ts` | 15+ mock candidate profiles with artifacts, process traces, defense scores, microcredentials | Typed mock data | packages/types |
|
| `candidates.ts` | 15+ mock candidate profiles with artifacts, process traces, defense scores, microcredentials | Typed mock data | packages/types |
|
||||||
| `employers.ts` | 10+ mock employer profiles with logos, descriptions, open positions | Typed mock data | packages/types |
|
| `employers.ts` | 10+ mock employer profiles with logos, descriptions, open positions | Typed mock data | packages/types |
|
||||||
| `learner-progress.ts` | Mock learner progress data: active competencies, completion percentages, recent artifacts | Typed mock data | packages/types |
|
| `learner-progress.ts` | Mock learner progress data: active competencies, completion percentages, recent artifacts | Typed mock data | packages/types |
|
||||||
| `ai-tutor-responses.ts` | Pre-scripted AI tutor chat responses for Coach and Tutor agent mockups | Typed mock data | packages/types |
|
| `admin.ts` | Admin surface mock data: platform metrics, activity feed, system health, learner roster (admin view), moderation queues | Typed mock data | packages/types |
|
||||||
|
| `ai-scenarios.ts` | AI engine-input scenario IDs + display metadata for the learner agent panels; IDs string-identical to `ai_service/corpus/` (D-021) | Typed mock data | packages/types |
|
||||||
|
|
||||||
### packages/types — Shared Types
|
### packages/types — Shared Types
|
||||||
|
|
||||||
@@ -84,53 +113,48 @@ Nextcraft v0.1 is a high-fidelity UI/UX prototype built as a TypeScript monorepo
|
|||||||
## Data Flow
|
## Data Flow
|
||||||
|
|
||||||
```
|
```
|
||||||
[Mock Data Layer] ──typed──> [Shared Types] <──typed──> [UI Components]
|
[packages/mock-data + packages/types] [ai_service/corpus]
|
||||||
│ │
|
│ (TS, web surfaces) │ (Python, agent inputs)
|
||||||
│ │
|
▼ ▼
|
||||||
▼ ▼
|
[Next.js Route Groups] [ai-service agents]
|
||||||
[Next.js Route Groups] [Surface Components]
|
(learner)/ (marketplace)/ coach tutor lab assessor proctor mentor
|
||||||
(learner)/ (marketplace)/ (employer)/ (admin)/
|
(employer)/ (admin)/ │
|
||||||
│ │ │ │
|
│ ▼
|
||||||
└──────────────────┴─────────────────┴───────────────┘
|
│ SSE (fetch + ReadableStream) [LLMProvider]
|
||||||
│
|
└────── client components ◄───────────────────┤
|
||||||
▼
|
http://localhost:8420 ollama-cloud / local / mock
|
||||||
[Root Layout + Theme Provider]
|
(https://ollama.com/v1)
|
||||||
│
|
|
||||||
▼
|
|
||||||
[Responsive Navigation Shell]
|
|
||||||
```
|
```
|
||||||
|
|
||||||
All data flows from the mock data layer through typed imports into Next.js route handlers / server components, which pass data as props to UI components. No client-side data fetching, no API routes, no server actions in v0.1.
|
- Web surfaces remain server-component-first; client components (chat, filters, graph viewer, dark mode toggle) fetch directly from ai-service over SSE (A-002: no Next.js API-route proxy in v0.2).
|
||||||
|
- The LLM provider layer is a dumb pipe — OpenAI-compatible chunks pass through byte-identical; envelope logic (meta/done/error) lives only in the API layer (D-016).
|
||||||
|
- Lab/Assessor/Proctor read mock scenarios from `ai_service/corpus/` — real engines are v0.3+.
|
||||||
|
- All automated tests use the deterministic mock provider; the cloud is for manual probes only.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Build Order
|
## Build Order (v0.2)
|
||||||
|
|
||||||
1. **Monorepo scaffolding** — pnpm-workspace.yaml, turbo.json, tsconfig.json, package.json, Next.js app initialization
|
1. **AI service scaffolding** — apps/ai-service: FastAPI app, config, provider layer (ollama-cloud/local/mock), SSE chat endpoint, pytest harness, turbo integration
|
||||||
2. **Shared types** — packages/types with all domain, marketplace, user, and UI type definitions
|
2. **Agent framework** — BaseAgent, registry, session store, structured output, prompts scaffolding, learner-context corpus
|
||||||
3. **Mock data layer** — packages/mock-data with typed mock data for all surfaces
|
3. **Coach + Tutor agents** — full implementations, chat endpoint agent routing
|
||||||
4. **Design tokens** — packages/ui/design-tokens with CSS custom properties
|
4. **Lab + Assessor agents** — telemetry scenarios + pre-baked artifacts corpus, /v1/lab/feedback + /v1/assessment/evaluate
|
||||||
5. **UI primitives** — Button, Input, Card, Badge, Avatar, Dialog, Tabs, Progress, Tooltip, Skeleton, Toast
|
5. **Proctor + Mentor agents** — proctor scenarios, /v1/proctor/signals + /v1/mentor/narrative
|
||||||
6. **Root layout + navigation shell** — Root layout with theme provider, responsive navigation, role switcher
|
6. **Learner surface integration** — useChatStream hook, agent switcher, streaming/error/loading states, agent output panels across the four learner surfaces
|
||||||
7. **UI composites** — Navigation, Table, SearchBar, FilterPanel, ChatInterface, GraphViewer, ArtifactCard, CompetencyBadge, JobCard, CandidateCard, MetricCard
|
|
||||||
8. **Learner surface routes** — Landing, catalog, competency stack, dashboard, byte viewer, sandbox mockup, assessment mockup
|
The v0.1 build order (monorepo → types → mock data → tokens → primitives → layout → composites → surfaces → polish) is complete and preserved in git history (tags v0.0.1–v0.1.0).
|
||||||
9. **Marketplace surface routes** — Job board, job detail, employer profile, search/filter, pricing
|
|
||||||
10. **Employer dashboard routes** — Overview, talent search, candidate profile, posting management
|
|
||||||
11. **Admin surface routes** — Overview, learner management, competency graph viewer, moderation
|
|
||||||
12. **Polish + integration** — Cross-surface navigation, responsive QA, visual consistency, Storybook
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Future Architecture (Post-v0.1, for reference)
|
## Future Architecture (Post-v0.2, for reference)
|
||||||
|
|
||||||
The v0.1 prototype is designed to be replaced piece-by-piece with real backend services:
|
v0.2 delivers the ai-service skeleton that later milestones fill in:
|
||||||
|
|
||||||
- **Mock data → PostgreSQL + Drizzle ORM** — mock data files replaced with database queries via repository layer
|
- **Mock corpus → real engines** — Lab consumes real sandbox telemetry (v0.3 sandbox fabric); Assessor grades real process traces (v0.3 assessment engine); Proctor consumes real identity/attention signals (v0.3 identity verification)
|
||||||
- **Static routes → Fastify API + Next.js SSR** — API routes replaced with Fastify backend services
|
- **In-memory sessions → PostgreSQL + Drizzle ORM** — SessionStore protocol swap, no API changes
|
||||||
- **Mock AI tutor → Python FastAPI AI services** — Chat interface mockup replaced with real AI agent microservices
|
- **Mock provider → per-agent model routing** — provider factory already selects by config; per-agent `AI_<AGENT>_MODEL` overrides
|
||||||
- **Mock assessment → Assessment engine** — Assessment mockup replaced with process-trace grading + oral defense engine
|
- **No auth → real KYC + sessions** — A-008 dropped in v0.3 when identity verification lands
|
||||||
- **No auth → Identity verification + age-gating** — Registration flow mockup replaced with real KYC and age verification
|
|
||||||
- **No search → Semantic vector search (pgvector)** — Filter UI replaced with vector similarity search
|
- **No search → Semantic vector search (pgvector)** — Filter UI replaced with vector similarity search
|
||||||
- **No payments → Payment processing** — Pricing page replaced with real subscription/payment flows
|
- **No payments → Payment processing** — Pricing page replaced with real subscription/payment flows
|
||||||
|
|
||||||
The monorepo structure (apps/web + packages/*) is designed to accommodate additional apps (e.g., apps/api, apps/ai-service) in future milestones without restructuring.
|
The monorepo structure (apps/web + apps/ai-service + packages/*) accommodates further apps without restructuring.
|
||||||
@@ -1,8 +1,8 @@
|
|||||||
{
|
{
|
||||||
"phase": 7,
|
"phase": 7,
|
||||||
"stage": "complete",
|
"stage": "execute",
|
||||||
"milestone": "v0.1",
|
"milestone": "v0.2",
|
||||||
"phase_role": "final",
|
"phase_role": "final",
|
||||||
"attempts": 0,
|
"attempts": 0,
|
||||||
"updated_at": "2026-09-10T22:48:00Z"
|
"updated_at": "2026-09-11T23:15:00Z"
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# Nextcraft v0.2 — GRILL.md (Adversarial Review Verdict)
|
||||||
|
|
||||||
|
**Stage:** GRILL, Phase 0 pre-execution · **Verdict:** GO-WITH-CHANGES · **Confidence:** 0.82
|
||||||
|
|
||||||
|
## Per-Axis Findings
|
||||||
|
|
||||||
|
| Axis | Verdict | Rationale |
|
||||||
|
|------|---------|-----------|
|
||||||
|
| Feasibility | PASS | Environment claims verified (venv works, all 8 pinned packages on PyPI for py3.11, port 8420 free, secrets gitignored). Stall points pre-mitigated (idempotent bootstrap, 300s read timeout, idempotent abort; reactStrictMode confirmed in next.config). |
|
||||||
|
| Over-scoping | CONCERN (mild) | 4-layer JSON defense justified; 500-cap LRU is over-spec but cheap — keep, extend no further. Real gaps: no Python lint (fixed via G-3), cache dirs in gitignore (fixed), script path resolution (advisory c). |
|
||||||
|
| Architecture risk | CONCERN | sse-starlette `: ping` frames invisible to short TestClient streams — parser gap fixed via G-1. Agent registration had two contradictory patterns — fixed via G-4. Cloud outage blast radius contained by mock-only tests. |
|
||||||
|
| Phase sequencing | PASS | Integration-last correct: P1 freezes the SSE contract before client code exists; A-002 eliminates dev-server buffering trap. P4 heaviest but mechanical. |
|
||||||
|
| Verification honesty | CONCERN | Persona distinctness circular against self-authored mocks (disclosed; cloud probe optional). P6 must-haves manual-only. D-021 ID alignment unmechanized (advisory b). Disclosed honestly. |
|
||||||
|
| Cost/quota | PASS | Cloud burn bounded (~<100K tokens milestone-wide, manual probes only). Mock-only rule structurally enforced; mechanical guard via advisory (a). |
|
||||||
|
| Milestone honesty | PASS (conditional) | Mock-engine caveats present everywhere that matters. Release note content now bound by G-5; dead `aiTutorResponses` disposal bound by G-5. |
|
||||||
|
|
||||||
|
## Binding Decisions (applied to PLAN.md/ROADMAP.md)
|
||||||
|
|
||||||
|
- **G-1 (BINDING):** SSE frame parser in Task 6-1-01 must ignore frames with no `data:` lines (sse-starlette ping keep-alive). Added to Action + Phase 6 must-have.
|
||||||
|
- **G-2 (BINDING):** P6 end-to-end verification may run with `AI_PROVIDER=mock` fallback — real ai-service over HTTP is the requirement; provider choice is service-internal. Prevents cloud outage blocking P6.
|
||||||
|
- **G-3 (BINDING):** ruff (check-only) added: Task 1-1-04, pyproject dev extra, scripts/lint.sh, root `ai:lint`, turbo `ai#lint`, Phase 1 must-have.
|
||||||
|
- **G-4 (BINDING):** Mentor/Proctor registered centrally in `registry.py` (single registration pattern), matching P3/P4.
|
||||||
|
- **G-5 (BINDING):** v0.2.0 release note must state Lab/Assessor/Proctor run on mock engine inputs (v0.3+ for real); dead `aiTutorResponses` export disposed of in P7.
|
||||||
|
|
||||||
|
## Advisory (non-binding; applied where cheap)
|
||||||
|
|
||||||
|
- (a) conftest asserts provider is MockProvider — **applied in P1 implementation**
|
||||||
|
- (b) pytest reads packages/mock-data as text, asserts corpus IDs appear — **applied in P4 implementation**
|
||||||
|
- (c) scripts resolve repo root via script-relative dirname — **folded into Task 1-1-04/1-1-03**
|
||||||
|
- (d) `.pytest_cache/` + `.ruff_cache/` in .gitignore — **applied**
|
||||||
|
- (e) keep LRU as-is; no further session-store sophistication in v0.2
|
||||||
|
- (f) Task 6-2-02 REQ tag fixed to REQ-2-011 (routing) — **applied**
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
GO — all five binding decisions applied to PLAN.md/ROADMAP.md/.gitignore/ARCHITECTURE.md before Phase 1 execution. No axis requires escalation.
|
||||||
+66
-21
@@ -6,12 +6,13 @@
|
|||||||
```yaml
|
```yaml
|
||||||
active: true
|
active: true
|
||||||
phase_specific: false
|
phase_specific: false
|
||||||
reason: Coordinates task decomposition across surfaces, resolves conflicts between frontend and data personas
|
reason: Coordinates task decomposition across web, AI service, and data territories; resolves conflicts between frontend, backend, and AI personas
|
||||||
domain: coordination
|
domain: coordination
|
||||||
frameworks:
|
frameworks:
|
||||||
- next.js
|
- next.js
|
||||||
- turborepo
|
- turborepo
|
||||||
- pnpm
|
- pnpm
|
||||||
|
- fastapi
|
||||||
constraints:
|
constraints:
|
||||||
- pragmatic
|
- pragmatic
|
||||||
- battle-tested defaults
|
- battle-tested defaults
|
||||||
@@ -21,13 +22,14 @@ territory:
|
|||||||
- "**/turbo.json"
|
- "**/turbo.json"
|
||||||
- "**/pnpm-workspace.yaml"
|
- "**/pnpm-workspace.yaml"
|
||||||
- "**/tsconfig.json"
|
- "**/tsconfig.json"
|
||||||
|
- "apps/ai-service/pyproject.toml"
|
||||||
```
|
```
|
||||||
|
|
||||||
### frontend-engineer
|
### frontend-engineer
|
||||||
```yaml
|
```yaml
|
||||||
active: true
|
active: true
|
||||||
phase_specific: false
|
phase_specific: false
|
||||||
reason: Primary persona for v0.1 — all 28 REQ-IDs are UI/frontend work. Owns all page components, layouts, and surface-specific UI.
|
reason: Phase 6 learner-surface integration — useChatStream hook, agent switcher, streaming/error/loading states, Lab/Assessor/Proctor output panels. Owns all page components, layouts, and surface-specific UI.
|
||||||
domain: frontend
|
domain: frontend
|
||||||
frameworks:
|
frameworks:
|
||||||
- react
|
- react
|
||||||
@@ -40,7 +42,8 @@ constraints:
|
|||||||
- component-first
|
- component-first
|
||||||
- server-components-default
|
- server-components-default
|
||||||
- minimal-client-js
|
- minimal-client-js
|
||||||
- mock-data-only
|
- sse-client-buffering (buffer bytes, split frames on \n\n, join data: lines)
|
||||||
|
- abortcontroller-cleanup (idempotent abort in effect cleanup)
|
||||||
- responsive-all-breakpoints
|
- responsive-all-breakpoints
|
||||||
- dark-mode-support
|
- dark-mode-support
|
||||||
territory:
|
territory:
|
||||||
@@ -54,7 +57,7 @@ territory:
|
|||||||
```yaml
|
```yaml
|
||||||
active: true
|
active: true
|
||||||
phase_specific: false
|
phase_specific: false
|
||||||
reason: Owns mock data layer schema and typed definitions. No real database in v0.1, but data structures must be well-typed for future migration.
|
reason: Owns TS mock data layer schema and typed definitions. Does NOT own the Python corpus (that is ai-engineer territory) — the two are aligned by documented convention (D-021).
|
||||||
domain: data
|
domain: data
|
||||||
frameworks:
|
frameworks:
|
||||||
- typescript
|
- typescript
|
||||||
@@ -70,33 +73,62 @@ territory:
|
|||||||
|
|
||||||
### backend-engineer
|
### backend-engineer
|
||||||
```yaml
|
```yaml
|
||||||
active: false
|
active: true
|
||||||
phase_specific: false
|
phase_specific: false
|
||||||
reason: No backend in v0.1. All data is mock/static. Backend persona deactivated until v0.2+ when API services are needed.
|
reason: Reactivated for v0.2 — owns apps/ai-service infrastructure: FastAPI app, settings, SSE plumbing, endpoints, monorepo/turbo integration, test harness.
|
||||||
domain: backend
|
domain: backend
|
||||||
frameworks: []
|
frameworks:
|
||||||
constraints: []
|
- fastapi
|
||||||
territory: []
|
- uvicorn
|
||||||
|
- pydantic
|
||||||
|
- httpx
|
||||||
|
- pytest
|
||||||
|
constraints:
|
||||||
|
- provider-agnostic-boundaries (llm/ imports nothing from agents/ or api/)
|
||||||
|
- streaming-first
|
||||||
|
- no-database-v0.2
|
||||||
|
- secrets-via-env-only
|
||||||
|
- mock-provider-in-tests
|
||||||
|
territory:
|
||||||
|
- "apps/ai-service/ai_service/main.py"
|
||||||
|
- "apps/ai-service/ai_service/config.py"
|
||||||
|
- "apps/ai-service/ai_service/api/**"
|
||||||
|
- "apps/ai-service/scripts/**"
|
||||||
|
- "apps/ai-service/package.json"
|
||||||
|
- "apps/ai-service/tests/api/**"
|
||||||
|
- "turbo.json"
|
||||||
```
|
```
|
||||||
|
|
||||||
### security-auditor
|
### ai-engineer
|
||||||
```yaml
|
```yaml
|
||||||
active: false
|
active: true
|
||||||
phase_specific: false
|
phase_specific: false
|
||||||
reason: No auth, no API, no real data in v0.1. Security review handled by verifier's STRIDE analysis layer. No dedicated security persona needed for UI prototype.
|
reason: Custom persona for v0.2 — owns the LLM provider layer, agent framework, prompt library, structured outputs, and mock corpora for the six tutor agents.
|
||||||
domain: security
|
domain: ai
|
||||||
frameworks: []
|
frameworks:
|
||||||
constraints: []
|
- pydantic
|
||||||
territory: []
|
- httpx
|
||||||
|
- pytest
|
||||||
|
constraints:
|
||||||
|
- provider-agnostic-protocol
|
||||||
|
- prompts-are-code
|
||||||
|
- json-defensive-parsing
|
||||||
|
- never-call-cloud-in-tests
|
||||||
|
- delta-passthrough
|
||||||
|
territory:
|
||||||
|
- "apps/ai-service/ai_service/llm/**"
|
||||||
|
- "apps/ai-service/ai_service/agents/**"
|
||||||
|
- "apps/ai-service/ai_service/prompts/**"
|
||||||
|
- "apps/ai-service/ai_service/corpus/**"
|
||||||
|
- "apps/ai-service/tests/llm/**"
|
||||||
|
- "apps/ai-service/tests/agents/**"
|
||||||
```
|
```
|
||||||
|
|
||||||
## Custom Personas
|
|
||||||
|
|
||||||
### design-system-engineer
|
### design-system-engineer
|
||||||
```yaml
|
```yaml
|
||||||
active: true
|
active: true
|
||||||
phase_specific: false
|
phase_specific: false
|
||||||
reason: Custom persona for v0.1 — owns the shared component library, design tokens, and visual consistency. Combines frontend expertise with design system ownership.
|
reason: Owns the shared component library, design tokens, and visual consistency. Light duty in v0.2: Phase 6 may need new primitives (agent-switcher control, stream-status indicator, error toast variant).
|
||||||
domain: frontend
|
domain: frontend
|
||||||
frameworks:
|
frameworks:
|
||||||
- tailwindcss
|
- tailwindcss
|
||||||
@@ -111,14 +143,27 @@ territory:
|
|||||||
- "packages/ui/**"
|
- "packages/ui/**"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### security-auditor
|
||||||
|
```yaml
|
||||||
|
active: false
|
||||||
|
phase_specific: false
|
||||||
|
reason: No auth in v0.2 (A-008); CORS is localhost-only; no real user data. Security review handled by verifier's STRIDE analysis layer plus a Phase 7 checklist item: secrets hygiene (key absent from code/logs/commits/errors), localhost-only CORS, no PII in prompts.
|
||||||
|
domain: security
|
||||||
|
frameworks: []
|
||||||
|
constraints: []
|
||||||
|
territory: []
|
||||||
|
```
|
||||||
|
|
||||||
## Phase-Specific Personas
|
## Phase-Specific Personas
|
||||||
|
|
||||||
None for v0.1. All active personas span the entire milestone.
|
None for v0.2. All active personas span the entire milestone. data-engineer and design-system-engineer are light-touch after Phase 1.
|
||||||
|
|
||||||
## Territory Conflict Resolution
|
## Territory Conflict Resolution
|
||||||
|
|
||||||
| Conflict | Resolution |
|
| Conflict | Resolution |
|
||||||
|----------|-----------|
|
|----------|------------|
|
||||||
| frontend-engineer vs data-engineer (packages/types, packages/mock-data) | data-engineer owns type definitions and mock data schema; frontend-engineer consumes them. If changes needed, data-engineer updates types first. |
|
| frontend-engineer vs data-engineer (packages/types, packages/mock-data) | data-engineer owns type definitions and mock data schema; frontend-engineer consumes them. If changes needed, data-engineer updates types first. |
|
||||||
| frontend-engineer vs design-system-engineer (packages/ui) | design-system-engineer owns design tokens and primitive components; frontend-engineer owns composite components and page-level UI. |
|
| frontend-engineer vs design-system-engineer (packages/ui) | design-system-engineer owns design tokens and primitive components; frontend-engineer owns composite components and page-level UI. |
|
||||||
|
| ai-engineer vs data-engineer (mock data duplication) | ai-engineer owns `ai_service/corpus/` (Python); data-engineer owns `packages/mock-data` (TS). Shared entity IDs and shapes kept aligned by documented convention (D-021): cross-referencing file headers, identical `comp-*` ID strings. |
|
||||||
|
| backend-engineer vs ai-engineer (apps/ai-service) | backend-engineer owns app shell, config, API endpoints, scripts, and test harness; ai-engineer owns llm/, agents/, prompts/, corpus/. Boundary: `ai_service/api/` (backend) composes `ai_service/agents/` (AI) via DI — agents never import api/. |
|
||||||
| lead-developer vs any | lead-developer coordinates only, does not directly modify code files. |
|
| lead-developer vs any | lead-developer coordinates only, does not directly modify code files. |
|
||||||
+333
-309
@@ -1,398 +1,422 @@
|
|||||||
# Nextcraft v0.1 — PLAN.md
|
# Nextcraft v0.2 — PLAN.md
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
This plan covers execution phases 1-6 of milestone v0.1 (UI/UX Prototype). Each phase is a vertical slice that produces a demoable milestone. Phases are ordered by dependency — later phases build on earlier ones.
|
This plan covers execution phases 1-6 of milestone v0.2 (AI Tutor Architecture): the six AI tutor agents (Coach, Tutor, Lab, Assessor, Proctor, Mentor) as real LLM-backed services in a new `apps/ai-service` Python FastAPI application, wired into the existing v0.1 learner surface with streaming responses. Phases are strictly sequential (P1→P6); within each phase, Wave 1 tasks are parallelizable (no cross-file dependencies) and later waves depend on earlier ones.
|
||||||
|
|
||||||
|
**Environment facts (apply throughout):** Python via `python3 -m venv` (no system pip, no uv); pnpm 12.3.4 via corepack; ai-service port **8420**; default model `gemma4:31b` (config via `AI_TUTOR_MODEL`); ollama-cloud base `https://ollama.com/v1` (OpenAI-compatible, Bearer auth); API keys live only in gitignored `.ciagent/.env.secrets`, exported by `scripts/dev.sh` — never in code, commits, or logs; all automated tests use the deterministic mock provider and **never call the cloud**.
|
||||||
|
|
||||||
|
| Phase | Name | Requirements | Waves | Personas |
|
||||||
|
|-------|------|-------------|-------|----------|
|
||||||
|
| 1 | AI service scaffolding | REQ-2-001, 002, 003 | 3 | backend-engineer, ai-engineer |
|
||||||
|
| 2 | Agent framework | REQ-2-004 | 2 | ai-engineer, backend-engineer |
|
||||||
|
| 3 | Coach + Tutor agents | REQ-2-005, 006 | 3 | ai-engineer, backend-engineer |
|
||||||
|
| 4 | Lab + Assessor agents | REQ-2-007, 008 | 4 | ai-engineer, backend-engineer, data-engineer |
|
||||||
|
| 5 | Proctor + Mentor agents | REQ-2-009, 010 | 3 | ai-engineer, backend-engineer |
|
||||||
|
| 6 | Learner surface integration | REQ-2-011, 012 | 3 | frontend-engineer, design-system-engineer |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 1: Project Scaffolding
|
## Phase 1: AI Service Scaffolding
|
||||||
|
|
||||||
**Requirements:** REQ-001, REQ-002, REQ-003, REQ-004, REQ-005
|
**Requirements:** REQ-2-001, REQ-2-002, REQ-2-003
|
||||||
**Persona:** design-system-engineer (Wave 1), data-engineer (Wave 1), frontend-engineer (Wave 2)
|
**Goal:** apps/ai-service runs under uvicorn, /health responds, provider-agnostic LLM layer with ollama-cloud/local/mock providers, SSE chat streaming verified, pytest suite green with mock provider, turbo integration wired
|
||||||
**Goal:** Monorepo builds, dev server starts, component library has all primitives, mock data typed, routing works
|
|
||||||
|
|
||||||
### Wave 1: Foundation (parallel — no shared file conflicts)
|
### Wave 1: Service shell + LLM core (parallel — no shared files)
|
||||||
|
|
||||||
#### Task 1-1-01: Monorepo scaffolding
|
#### Task 1-1-01: FastAPI app scaffolding
|
||||||
- **Persona:** design-system-engineer
|
- **Persona:** backend-engineer — **REQ:** REQ-2-001
|
||||||
- **Files:** `package.json`, `pnpm-workspace.yaml`, `turbo.json`, `tsconfig.json`, `.npmrc`
|
- **Files:** `apps/ai-service/pyproject.toml`, `apps/ai-service/ai_service/main.py`, `apps/ai-service/ai_service/config.py`, `apps/ai-service/ai_service/__init__.py`, `apps/ai-service/tests/conftest.py`, `apps/ai-service/tests/test_health.py`, `apps/ai-service/.env.example`, `apps/ai-service/README.md`
|
||||||
- **Action:** Create root monorepo config. pnpm workspaces pointing to `apps/*` and `packages/*`. Turborepo with build/dev/lint/typecheck pipelines. Root tsconfig with path aliases.
|
- **Action:** pydantic-settings `Settings` (env_prefix `AI_`, env_file, `SecretStr` key, port 8420, provider select, `AI_TUTOR_MODEL` default `gemma4:31b`). FastAPI app factory in `main.py` with lifespan stub (httpx client pool comes in Wave 2), CORS localhost-only (A-008), `GET /health`. pyproject with pinned deps (fastapi, uvicorn, pydantic, pydantic-settings, httpx, sse-starlette, pytest, pytest-asyncio) and dev extra. conftest: settings override + TestClient fixture. `.env.example` documents all `AI_*` vars; README documents venv setup and dev workflow.
|
||||||
- **Verify:** `pnpm install` succeeds; workspace packages detected
|
- **Verify:** `scripts/bootstrap.sh && scripts/test.sh` — test_health passes; `curl localhost:8420/health` returns 200
|
||||||
- **Done:** `pnpm list -r` shows all workspace packages
|
|
||||||
|
|
||||||
#### Task 1-1-02: Shared types package
|
#### Task 1-1-02: LLM types, protocol, mock provider
|
||||||
- **Persona:** data-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-002
|
||||||
- **Files:** `packages/types/package.json`, `packages/types/tsconfig.json`, `packages/types/domain.ts`, `packages/types/marketplace.ts`, `packages/types/user.ts`, `packages/types/ui.ts`, `packages/types/index.ts`
|
- **Files:** `apps/ai-service/ai_service/llm/types.py`, `apps/ai-service/ai_service/llm/base.py`, `apps/ai-service/ai_service/llm/mock.py`, `apps/ai-service/ai_service/llm/__init__.py`
|
||||||
- **Action:** Create all shared TypeScript type definitions: Competency, CompetencyStack, Microcredential, Artifact, ProcessTrace, OralDefense, AssessmentRubric, Job, Employer, Candidate, JobPosting, TalentMatch, SearchFilter, Learner, Admin, EmployerUser, AgeGroup, Role, plus UI types (ComponentProps, ThemeConfig, Breakpoint).
|
- **Action:** `types.py`: pydantic `Message` (role/content), `ChatDelta` (OpenAI-compatible chunk shape). `base.py`: `LLMProvider` protocol — async `stream_chat(messages, model, response_format=None) -> AsyncIterator[ChatDelta]`; the provider is a dumb pipe, no envelope logic (D-016 keeps envelope in API layer). `mock.py`: deterministic scripted provider (hash-seeded token streams, scripted failure modes: connect error, mid-stream error, malformed JSON) for tests and CI.
|
||||||
- **Verify:** `pnpm typecheck` passes in packages/types
|
- **Verify:** mock provider importable and deterministic; two identical calls yield identical streams
|
||||||
- **Done:** All types exported from index.ts; no type errors
|
|
||||||
|
|
||||||
#### Task 1-1-03: Mock data package
|
#### Task 1-1-03: Monorepo integration (shim + turbo + scripts)
|
||||||
- **Persona:** data-engineer
|
- **Persona:** backend-engineer — **REQ:** REQ-2-001
|
||||||
- **Files:** `packages/mock-data/package.json`, `packages/mock-data/tsconfig.json`, `packages/mock-data/competency-stacks.ts`, `packages/mock-data/jobs.ts`, `packages/mock-data/candidates.ts`, `packages/mock-data/employers.ts`, `packages/mock-data/learner-progress.ts`, `packages/mock-data/ai-tutor-responses.ts`, `packages/mock-data/index.ts`
|
- **Files:** `apps/ai-service/package.json`, `apps/ai-service/scripts/bootstrap.sh`, `apps/ai-service/scripts/dev.sh`, `apps/ai-service/scripts/test.sh`, `turbo.json` (update), `package.json` (root, update)
|
||||||
- **Action:** Create typed mock data: 5 competency stacks (AI Orchestration Engineer with 15 competencies, AI Safety & Governance with 14, Human-AI Product Designer with 13, AI-Augmented Field Operator with 12, Computational Sciences with 16). 20+ mock jobs. 15+ mock candidates. 10+ mock employers. Mock learner progress. Pre-scripted AI tutor responses.
|
- **Action:** zero-dependency shim `package.json` in apps/ai-service with `dev`/`test`/`bootstrap` script entries. Turbo passthrough tasks `ai#dev`, `ai#test`, `ai#bootstrap` (`cache: false`, `outputs: []`). Root scripts `ai:dev`, `ai:test`, `ai:bootstrap`. `bootstrap.sh`: idempotent `python3 -m venv .venv` + pip install. `dev.sh`: exports keys from `.ciagent/.env.secrets` → uvicorn on 8420. `test.sh`: pytest via venv.
|
||||||
- **Verify:** All mock data matches types from packages/types; `pnpm typecheck` passes
|
- **Verify:** `corepack pnpm install && pnpm ai:bootstrap && pnpm ai:test` runs pytest through turbo; re-running bootstrap is a no-op
|
||||||
- **Done:** All mock data exported from index.ts; types match
|
|
||||||
|
|
||||||
### Wave 2: UI Foundation (depends on Wave 1)
|
#### Task 1-1-04: Python lint (ruff, check-only)
|
||||||
|
- **Persona:** backend-engineer — **REQ:** REQ-2-001
|
||||||
|
- **Files:** `apps/ai-service/pyproject.toml` (update: `[tool.ruff]` config + `ruff` in dev extra), `apps/ai-service/scripts/lint.sh`, `package.json` (root, update), `turbo.json` (update)
|
||||||
|
- **Action:** Add `ruff` (check-only, no formatter) to the dev extra; `[tool.ruff]` with line-length 100, target py311. `scripts/lint.sh`: `.venv/bin/ruff check .` with repo-root path resolution via script-relative dirname (not CWD). Root script `ai:lint`, turbo passthrough `ai#lint` (`cache: false`, `outputs: []`). Run over the entire ai-service tree; fix all findings before phase ship (G-3).
|
||||||
|
- **Verify:** `pnpm ai:lint` exits 0 on the Phase 1 codebase
|
||||||
|
|
||||||
#### Task 1-2-01: Next.js app initialization
|
### Wave 2: Real providers + SSE endpoint (depends on Wave 1)
|
||||||
- **Persona:** frontend-engineer
|
|
||||||
- **Files:** `apps/web/package.json`, `apps/web/tsconfig.json`, `apps/web/next.config.ts`, `apps/web/app/layout.tsx`, `apps/web/app/page.tsx`, `apps/web/app/globals.css`
|
|
||||||
- **Action:** Initialize Next.js app with App Router, TypeScript, Tailwind CSS v4. Configure Inter font via next/font. Set up path aliases to packages. Create root layout with theme provider. Create placeholder home page.
|
|
||||||
- **Verify:** `pnpm dev` starts; `pnpm build` succeeds; page renders at localhost:3000
|
|
||||||
- **Done:** Next.js app runs with Tailwind CSS and Inter font
|
|
||||||
|
|
||||||
#### Task 1-2-02: Design tokens and Tailwind config
|
#### Task 1-2-01: OpenAI-compatible provider + factory
|
||||||
- **Persona:** design-system-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-002
|
||||||
- **Files:** `packages/ui/package.json`, `packages/ui/tsconfig.json`, `packages/ui/src/index.ts`, `packages/ui/src/tokens/index.ts`, `apps/web/app/globals.css` (update)
|
- **Files:** `apps/ai-service/ai_service/llm/openai_compat.py`, `apps/ai-service/ai_service/llm/factory.py`
|
||||||
- **Action:** Create packages/ui package. Define design tokens as TypeScript constants and CSS custom properties: color palette (indigo/violet primary, emerald accent, slate neutral, dark mode variants), typography scale (12px-48px), spacing system (4px base), breakpoints (375px, 768px, 1280px), shadows, radii. Configure Tailwind v4 `@theme` with custom colors.
|
- **Action:** `openai_compat.py`: single `OpenAICompatProvider` for ollama-cloud (`https://ollama.com/v1`, Bearer) and local endpoints (base URL from settings); raw httpx against `/v1/chat/completions` with `stream: true`, byte-identical delta passthrough, tolerant of ollama-cloud quirks. Uses the lifespan-managed `httpx.AsyncClient` (10s connect / 300s read, D-017) — no openai SDK. `factory.py`: select provider from settings (`ollama-cloud` | `local` | `mock`).
|
||||||
- **Verify:** Design tokens importable from packages/ui; Tailwind classes work with custom colors
|
- **Verify:** provider constructs from settings for all 3 names; manual probe against ollama-cloud streams tokens (documented in README, not a test)
|
||||||
- **Done:** `import { tokens } from '@nextcraft/ui'` works; `bg-primary-500` class works
|
|
||||||
|
|
||||||
#### Task 1-2-03: UI primitives
|
#### Task 1-2-02: Lifespan wiring + SSE chat endpoint
|
||||||
- **Persona:** design-system-engineer
|
- **Persona:** backend-engineer — **REQ:** REQ-2-003
|
||||||
- **Files:** `packages/ui/src/primitives/button.tsx`, `packages/ui/src/primitives/input.tsx`, `packages/ui/src/primitives/card.tsx`, `packages/ui/src/primitives/badge.tsx`, `packages/ui/src/primitives/avatar.tsx`, `packages/ui/src/primitives/dialog.tsx`, `packages/ui/src/primitives/tabs.tsx`, `packages/ui/src/primitives/progress.tsx`, `packages/ui/src/primitives/tooltip.tsx`, `packages/ui/src/primitives/skeleton.tsx`, `packages/ui/src/primitives/toast.tsx`, `packages/ui/src/primitives/index.ts`
|
- **Files:** `apps/ai-service/ai_service/main.py` (update), `apps/ai-service/ai_service/api/deps.py`, `apps/ai-service/ai_service/api/chat.py`, `apps/ai-service/ai_service/api/__init__.py`
|
||||||
- **Action:** Create all 12 primitive components. Each is a React component with TypeScript props, Tailwind styling, forwardRef, and size variants. Button (primary, secondary, ghost, destructive). Input (text, search, with label). Card (with header, body, footer slots). Badge (default, success, warning, error). Avatar (with image, fallback, sizes). Dialog (modal, with overlay). Tabs (horizontal, with active indicator). Progress (linear, circular). Tooltip (on hover). Skeleton (shimmer). Toast (with variants).
|
- **Action:** Lifespan creates the shared `httpx.AsyncClient` and provider factory; deps.py provides provider via DI. `POST /v1/chat/stream` in chat.py implements the D-016 envelope: `meta` event (agent/session/model) flushed before first token → raw OpenAI chunks passed through as `data: {json}` → `done` event → `error` event before `[DONE]` on mid-stream failure; pre-first-byte failures return proper HTTP status codes. Headers `Cache-Control: no-cache`, `X-Accel-Buffering: no`; sse-starlette ping keep-alive.
|
||||||
- **Verify:** All primitives importable; each renders without error; variants work
|
- **Verify:** `curl -N -X POST localhost:8420/v1/chat/stream` with mock provider shows meta event, token deltas, done, `[DONE]`
|
||||||
- **Done:** 12 primitives exported from packages/ui
|
|
||||||
|
|
||||||
#### Task 1-2-04: Layout components
|
### Wave 3: Provider + endpoint test suites (depends on Wave 2)
|
||||||
- **Persona:** design-system-engineer
|
|
||||||
- **Files:** `packages/ui/src/layouts/container.tsx`, `packages/ui/src/layouts/grid.tsx`, `packages/ui/src/layouts/sidebar.tsx`, `packages/ui/src/layouts/split-panel.tsx`, `packages/ui/src/layouts/dashboard-layout.tsx`, `packages/ui/src/layouts/index.ts`
|
|
||||||
- **Action:** Create layout components. Container (max-width variants: sm, md, lg, xl, full). Grid (responsive cols prop). Sidebar (collapsible, with nav items). SplitPanel (resizable divider). DashboardLayout (sidebar + main content area).
|
|
||||||
- **Verify:** Layout components render; responsive breakpoints work
|
|
||||||
- **Done:** 5 layout components exported
|
|
||||||
|
|
||||||
#### Task 1-2-05: Root layout + navigation shell
|
#### Task 1-3-01: LLM provider tests (byte-exact, no cloud)
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-002
|
||||||
- **Files:** `apps/web/app/layout.tsx` (update), `apps/web/components/navigation-shell.tsx`, `apps/web/components/role-switcher.tsx`, `apps/web/components/header.tsx`, `apps/web/components/footer.tsx`
|
- **Files:** `apps/ai-service/tests/llm/test_openai_compat.py`, `apps/ai-service/tests/llm/test_mock.py`, `apps/ai-service/tests/llm/__init__.py`
|
||||||
- **Action:** Update root layout with theme provider (dark mode toggle), Inter font, responsive navigation shell. Create role switcher (learner/employer/admin toggle in header). Create header with logo, nav links, role switcher, dark mode toggle. Create footer with links.
|
- **Action:** httpx `MockTransport` tests parsing byte-exact fixture streams (happy path, empty delta, `[DONE]`, malformed line, mid-stream disconnect). Mock provider tests: determinism, scripted failure modes, response_format echo.
|
||||||
- **Verify:** Dark mode toggle works; role switcher navigates between surfaces; responsive at 375px/768px/1280px
|
- **Verify:** `pnpm ai:test` — llm suite green; zero network calls in tests
|
||||||
- **Done:** Navigation shell renders on all pages; role switcher functional
|
|
||||||
|
|
||||||
#### Task 1-2-06: Route groups with placeholder pages
|
#### Task 1-3-02: SSE stream endpoint tests
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** backend-engineer — **REQ:** REQ-2-003
|
||||||
- **Files:** `apps/web/app/(learner)/page.tsx`, `apps/web/app/(learner)/layout.tsx`, `apps/web/app/(marketplace)/page.tsx`, `apps/web/app/(marketplace)/layout.tsx`, `apps/web/app/(employer)/page.tsx`, `apps/web/app/(employer)/layout.tsx`, `apps/web/app/(admin)/page.tsx`, `apps/web/app/(admin)/layout.tsx`
|
- **Files:** `apps/ai-service/tests/api/test_chat_stream.py`, `apps/ai-service/tests/api/__init__.py`
|
||||||
- **Action:** Create route groups with layouts for each surface. Each layout has surface-specific navigation. Placeholder pages with surface name and brief description.
|
- **Action:** TestClient `client.stream()` tests: meta-first ordering, delta passthrough, done + `[DONE]` sentinel, mid-stream error event, pre-first-byte failure → HTTP status, required headers. pytest-asyncio auto mode (D-023).
|
||||||
- **Verify:** Navigation to / (learner), /marketplace, /employer, /admin works
|
- **Verify:** `pnpm ai:test` — api suite green
|
||||||
- **Done:** 4 route groups with layouts and placeholder pages
|
|
||||||
|
|
||||||
### Must-Haves (Phase 1)
|
### Must-Haves (Phase 1)
|
||||||
- [ ] `pnpm install` succeeds
|
- [ ] `scripts/bootstrap.sh` is idempotent; creates venv + installs deps without system pip
|
||||||
- [ ] `pnpm dev` starts Next.js dev server
|
- [ ] `pnpm ai:dev` starts uvicorn; `curl localhost:8420/health` returns 200
|
||||||
- [ ] `pnpm build` succeeds
|
- [ ] `pnpm ai:lint` exits 0 (ruff check over the ai-service tree) (G-3)
|
||||||
- [ ] `pnpm typecheck` passes
|
- [ ] `pnpm ai:test` runs the full pytest suite via turbo and passes (mock provider only — no network)
|
||||||
- [ ] 12 UI primitives importable from `@nextcraft/ui`
|
- [ ] SSE stream delivers tokens: meta event, incremental deltas, done, `[DONE]` observed via `curl -N`
|
||||||
- [ ] Mock data typed and importable from `@nextcraft/mock-data`
|
- [ ] Mid-stream failure emits `error` event before `[DONE]`; pre-first-byte failure returns HTTP error status
|
||||||
- [ ] Types importable from `@nextcraft/types`
|
- [ ] Provider factory resolves ollama-cloud / local / mock from settings; manual ollama-cloud probe documented in README
|
||||||
- [ ] 4 route groups with layouts
|
- [ ] `llm/` imports nothing from `agents/` or `api/` (boundary rule holds)
|
||||||
- [ ] Dark mode toggle works
|
|
||||||
- [ ] Role switcher navigates between surfaces
|
|
||||||
- [ ] Responsive at 375px, 768px, 1280px
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 2: Learner Surface UI
|
## Phase 2: Agent Framework
|
||||||
|
|
||||||
**Requirements:** REQ-006, REQ-007, REQ-008, REQ-009, REQ-010, REQ-011, REQ-012
|
**Requirements:** REQ-2-004
|
||||||
**Persona:** frontend-engineer
|
**Goal:** Shared framework all six agents use: BaseAgent contract, session store, prompt library, registry, structured outputs — all tested against the mock provider
|
||||||
**Goal:** All 7 learner pages render with mock data; navigation works; responsive
|
|
||||||
|
|
||||||
### Wave 1: Core learner pages
|
### Wave 1: Framework primitives (parallel — no shared files)
|
||||||
|
|
||||||
#### Task 2-1-01: Landing page
|
#### Task 2-1-01: BaseAgent ABC
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-004
|
||||||
- **Files:** `apps/web/app/(learner)/page.tsx`, `apps/web/components/learner/hero.tsx`, `apps/web/components/learner/how-it-works.tsx`, `apps/web/components/learner/program-highlights.tsx`, `apps/web/components/learner/testimonials.tsx`
|
- **Files:** `apps/ai-service/ai_service/agents/base.py`, `apps/ai-service/tests/agents/test_base.py`, `apps/ai-service/ai_service/agents/__init__.py`, `apps/ai-service/tests/agents/__init__.py`
|
||||||
- **Action:** Build landing page with hero section (headline, subheadline, CTA buttons), how-it-works section (Byte→Build→Demonstrate→Defend visual flow), program highlights (3-4 featured competency stacks), testimonials mockup (3 cards with avatar, quote, name, role), CTA to program catalog.
|
- **Action:** `BaseAgent` ABC (D-018): `name`, `system_prompt`, `build_messages(history, learner_context)`, `stream_reply(...) -> AsyncIterator[ChatDelta]` (delegates to provider), `structured_reply(...)` (delegates to structured module, landed Wave 2). Subclass contract tested with a stub agent + mock provider.
|
||||||
- **Verify:** Landing page renders with all sections; CTA links to /catalog
|
- **Verify:** `pnpm ai:test` — test_base green
|
||||||
- **Done:** Landing page complete
|
|
||||||
|
|
||||||
#### Task 2-1-02: Program catalog
|
#### Task 2-1-02: SessionStore protocol + in-memory implementation
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-004
|
||||||
- **Files:** `apps/web/app/(learner)/catalog/page.tsx`, `apps/web/components/learner/stack-card.tsx`
|
- **Files:** `apps/ai-service/ai_service/agents/session.py`, `apps/ai-service/tests/test_session.py`
|
||||||
- **Action:** Build program catalog page. Grid of 5 competency stack cards. Each card: stack name, role description, competency count, estimated duration, microcredential count, "Explore" button linking to stack detail.
|
- **Action:** `SessionStore` protocol + `InMemorySessionStore` (D-019): asyncio.Lock-guarded dict, agent-scoped session keys, 20-message rolling window, 500-cap LRU eviction. Protocol shape is DB-migration-ready (A-003).
|
||||||
- **Verify:** 5 stack cards render with mock data; clicking "Explore" navigates to stack detail
|
- **Verify:** test_session covers create/append/window-trim/LRU-eviction/agent scoping
|
||||||
- **Done:** Catalog page complete
|
|
||||||
|
|
||||||
#### Task 2-1-03: Competency stack view
|
#### Task 2-1-03: Prompt library scaffolding
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-004
|
||||||
- **Files:** `apps/web/app/(learner)/catalog/[stackId]/page.tsx`, `apps/web/components/learner/competency-list.tsx`, `apps/web/components/learner/competency-item.tsx`, `apps/web/components/learner/microcredential-badge.tsx`
|
- **Files:** `apps/ai-service/ai_service/prompts/coach.py`, `apps/ai-service/ai_service/prompts/tutor.py`, `apps/ai-service/ai_service/prompts/lab.py`, `apps/ai-service/ai_service/prompts/assessor.py`, `apps/ai-service/ai_service/prompts/proctor.py`, `apps/ai-service/ai_service/prompts/mentor.py`, `apps/ai-service/ai_service/prompts/__init__.py`
|
||||||
- **Action:** Build competency stack detail page. Header with stack name, role, description. List of 12-18 competencies. Each competency: name, description, status (locked/available/in-progress/mastered), microcredential badge, progress bar. Clicking a competency navigates to byte tutorial viewer.
|
- **Action:** Per-agent module with a versioned `SYSTEM_PROMPT` constant + `render_context(learner_context) -> dict` using `str.format_map` for learner-context injection (D-018: prompts are code, versioned in git). Initial drafts for all six; final personas land in Phases 3-5.
|
||||||
- **Verify:** Competencies render with correct status indicators; microcredential badges display
|
- **Verify:** all six prompt modules import; render_context fills placeholders without KeyError
|
||||||
- **Done:** Stack detail page complete
|
|
||||||
|
|
||||||
#### Task 2-1-04: Learner dashboard
|
#### Task 2-1-04: Learner context corpus
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-004
|
||||||
- **Files:** `apps/web/app/(learner)/dashboard/page.tsx`, `apps/web/components/learner/dashboard/active-competencies.tsx`, `apps/web/components/learner/dashboard/progress-graph.tsx`, `apps/web/components/learner/dashboard/recent-artifacts.tsx`, `apps/web/components/learner/dashboard/upcoming-defenses.tsx`, `apps/web/components/learner/dashboard/ai-tutor-chat.tsx`, `apps/web/components/learner/dashboard/milestone-tracker.tsx`
|
- **Files:** `apps/ai-service/ai_service/corpus/learner_context.py`, `apps/ai-service/ai_service/corpus/__init__.py`
|
||||||
- **Action:** Build learner dashboard. Active competencies panel (current stack, progress). Progress graph (recharts area chart showing mastery over time). Recent artifacts (cards with artifact name, type, date). Upcoming defenses (list with competency name, date, status). AI tutor chat mockup (chat interface with pre-scripted responses, message input). Milestone tracker (progress through stack).
|
- **Action:** Pydantic-typed learner context (active stack, competencies, progress, recent artifacts) mirroring TS `packages/mock-data` IDs per D-021 convention (cross-referencing header comment, identical `comp-*`/`stack-*` ID strings).
|
||||||
- **Verify:** Dashboard renders all panels; AI tutor chat displays scripted responses on "send"
|
- **Verify:** context renders into prompt placeholders; IDs match packages/mock-data strings
|
||||||
- **Done:** Dashboard complete
|
|
||||||
|
|
||||||
### Wave 2: Detail views
|
### Wave 2: Composition layers (depends on Wave 1)
|
||||||
|
|
||||||
#### Task 2-2-01: Byte tutorial viewer
|
#### Task 2-2-01: Structured output defense
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-004
|
||||||
- **Files:** `apps/web/app/(learner)/learn/[competencyId]/page.tsx`, `apps/web/components/learner/byte-viewer/concept-panel.tsx`, `apps/web/components/learner/byte-viewer/worked-example.tsx`, `apps/web/components/learner/byte-viewer/viewer-tabs.tsx`
|
- **Files:** `apps/ai-service/ai_service/agents/structured.py`, `apps/ai-service/tests/test_structured.py`
|
||||||
- **Action:** Build byte tutorial viewer. Layout: left panel (concept text, 3-7 minute read), right panel (worked example with code/design/simulation viewer mockup). Tabbed viewer (code, design, simulation). Navigation to build sandbox.
|
- **Action:** 4-layer defense (D-020): (1) `response_format` request with auto-degrade on provider 400; (2) prompt-embedded JSON schema; (3) parse: strip code fences → first balanced JSON object; (4) single bounded retry with validation-error feedback. Returns pydantic-validated model or raises `StructuredOutputError`.
|
||||||
- **Verify:** Tutorial viewer renders with concept and example panels; tabs switch
|
- **Verify:** test_structured covers fenced/unfenced/invalid JSON, retry path, degrade path — all against mock provider
|
||||||
- **Done:** Byte tutorial viewer complete
|
|
||||||
|
|
||||||
#### Task 2-2-02: Build sandbox mockup
|
#### Task 2-2-02: Agent registry
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-004
|
||||||
- **Files:** `apps/web/app/(learner)/build/[competencyId]/page.tsx`, `apps/web/components/learner/sandbox/toolbar.tsx`, `apps/web/components/learner/sandbox/file-explorer.tsx`, `apps/web/components/learner/sandbox/editor-area.tsx`, `apps/web/components/learner/sandbox/telemetry-sidebar.tsx`
|
- **Files:** `apps/ai-service/ai_service/agents/registry.py`, `apps/ai-service/tests/test_registry.py`
|
||||||
- **Action:** Build sandbox mockup IDE UI. Layout: top toolbar (run, save, submit buttons), left sidebar (file explorer tree), center (code editor area with syntax-highlighted mock code), right sidebar (telemetry — process capture indicators showing commits, keystrokes, time spent). No real code execution.
|
- **Action:** Explicit registry: name → agent factory map with `register(name, factory)` / `get(name)`; raises on unknown agent. Agents are registered in their own phases (P3-P5).
|
||||||
- **Verify:** Sandbox UI renders with all panels; toolbar buttons are interactive (non-functional)
|
- **Verify:** test_registry: register/get round-trip, unknown-agent error, duplicate registration error
|
||||||
- **Done:** Build sandbox mockup complete
|
|
||||||
|
|
||||||
#### Task 2-2-03: Assessment/defense mockup
|
#### Task 2-2-03: Session + agent DI wiring into API layer
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** backend-engineer — **REQ:** REQ-2-004
|
||||||
- **Files:** `apps/web/app/(learner)/defend/[competencyId]/page.tsx`, `apps/web/components/learner/assessment/rubric-display.tsx`, `apps/web/components/learner/assessment/ai-reviewer-panel.tsx`, `apps/web/components/learner/assessment/oral-defense.tsx`, `apps/web/components/learner/assessment/process-trace.tsx`, `apps/web/components/learner/assessment/artifact-viewer.tsx`
|
- **Files:** `apps/ai-service/ai_service/api/deps.py` (update), `apps/ai-service/ai_service/api/chat.py` (update)
|
||||||
- **Action:** Build assessment/defense mockup. Layout: top (artifact viewer — submitted work preview), left (rubric display — competency criteria with checkmarks), right (AI reviewer panel — assessment results), bottom (oral defense interface — mic button mockup, waveform animation, transcript area). Process trace timeline (showing build steps with timestamps).
|
- **Action:** deps.py exposes SessionStore and provider singletons via DI. chat.py persists turn history through the session store (agent-scoped) and includes session ID in the meta event. API composes agents via DI — agents never import api/.
|
||||||
- **Verify:** Assessment mockup renders all panels; mic button has hover state; timeline displays mock events
|
- **Verify:** chat request appends to and replays windowed history; `pnpm ai:test` green
|
||||||
- **Done:** Assessment/defense mockup complete
|
|
||||||
|
|
||||||
### Must-Haves (Phase 2)
|
### Must-Haves (Phase 2)
|
||||||
- [ ] Landing page renders with hero, how-it-works, highlights, testimonials, CTA
|
- [ ] BaseAgent unit tests pass (stub agent streams via mock provider)
|
||||||
- [ ] Program catalog shows 5 competency stack cards
|
- [ ] Session store tested: create/append, 20-message window trim, 500-cap LRU eviction, agent-scoped keys
|
||||||
- [ ] Competency stack view shows 12-18 competencies with status and badges
|
- [ ] Structured output parsing tested against mock provider: fence-strip, first-balanced-object, invalid JSON, one bounded retry, response_format auto-degrade
|
||||||
- [ ] Learner dashboard shows all 6 panels including AI tutor chat mockup
|
- [ ] Registry tested: register/get/unknown/duplicate
|
||||||
- [ ] Byte tutorial viewer renders with concept and worked example panels
|
- [ ] Six prompt modules render learner context without errors
|
||||||
- [ ] Build sandbox mockup renders with toolbar, file explorer, editor, telemetry
|
- [ ] Module boundaries hold: `agents/` never imports `api/`; `llm/` never imports `agents/` or `api/`
|
||||||
- [ ] Assessment mockup renders with rubric, AI reviewer, oral defense, process trace
|
|
||||||
- [ ] All pages responsive at 375px, 768px, 1280px
|
|
||||||
- [ ] Navigation between all learner pages works
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 3: Marketplace Surface UI
|
## Phase 3: Coach + Tutor Agents
|
||||||
|
|
||||||
**Requirements:** REQ-013, REQ-014, REQ-015, REQ-016, REQ-017
|
**Requirements:** REQ-2-005, REQ-2-006
|
||||||
**Persona:** frontend-engineer
|
**Goal:** Both learner-facing conversational agents fully implemented with distinct personas, registered, routed through the chat streaming endpoint
|
||||||
**Goal:** All 5 marketplace pages render with mock data; search/filter interactive
|
|
||||||
|
|
||||||
### Wave 1: Job board + search
|
### Wave 1: Agent implementations (parallel — no shared files)
|
||||||
|
|
||||||
#### Task 3-1-01: Job board listing
|
#### Task 3-1-01: Coach agent
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-005
|
||||||
- **Files:** `apps/web/app/(marketplace)/page.tsx`, `apps/web/components/marketplace/job-card.tsx`, `apps/web/components/marketplace/job-listing-grid.tsx`
|
- **Files:** `apps/ai-service/ai_service/prompts/coach.py` (finalize), `apps/ai-service/ai_service/agents/coach.py`, `apps/ai-service/tests/test_coach.py`
|
||||||
- **Action:** Build job board listing page. Layout: left sidebar (filters), main area (grid of job cards). Each job card: job title, company name, logo, location, remote badge, salary range, match score (percentage badge), required skills (tag chips), posted date. 20+ mock job listings.
|
- **Action:** Final Coach persona: pacing guidance, motivation, retrieval practice prompts; system prompt injects learner context (active stack, progress). `CoachAgent(BaseAgent)` streams replies. Mock provider scripts a distinct coach-voice response for tests.
|
||||||
- **Verify:** 20+ job cards render; match scores display; skill tags show
|
- **Verify:** test_coach: build_messages includes system prompt + windowed history; stream_reply yields deltas; on-persona content asserted against mock script
|
||||||
- **Done:** Job board listing complete
|
|
||||||
|
|
||||||
#### Task 3-1-02: Search/filter UI
|
#### Task 3-1-02: Tutor agent
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-006
|
||||||
- **Files:** `apps/web/components/marketplace/filter-sidebar.tsx`, `apps/web/components/marketplace/search-bar.tsx`, `apps/web/components/marketplace/saved-searches.tsx`
|
- **Files:** `apps/ai-service/ai_service/prompts/tutor.py` (finalize), `apps/ai-service/ai_service/agents/tutor.py`, `apps/ai-service/tests/test_tutor.py`
|
||||||
- **Action:** Build filter sidebar. Semantic search bar at top (with icon, placeholder text). Filters: skills (multi-select chips), seniority (dropdown), location (text input), remote toggle, salary range (min/max inputs). Saved searches mockup section. Client-side filtering of mock jobs using useState.
|
- **Action:** Final Tutor persona: concept delivery, Socratic questioning, worked examples. `TutorAgent(BaseAgent)` streams replies; mock scripts a distinct tutor-voice response.
|
||||||
- **Verify:** Typing in search filters jobs; selecting skill filters jobs; toggling remote filters jobs
|
- **Verify:** test_tutor mirrors test_coach; Coach and Tutor mock outputs are observably distinct
|
||||||
- **Done:** Search/filter UI functional with client-side filtering
|
|
||||||
|
|
||||||
### Wave 2: Detail pages
|
### Wave 2: Registration + persona verification (depends on Wave 1)
|
||||||
|
|
||||||
#### Task 3-2-01: Job detail page
|
#### Task 3-2-01: Register Coach + Tutor; document cloud probe
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-005, REQ-2-006
|
||||||
- **Files:** `apps/web/app/(marketplace)/jobs/[jobId]/page.tsx`, `apps/web/components/marketplace/job-detail.tsx`, `apps/web/components/marketplace/skills-breakdown.tsx`, `apps/web/components/marketplace/related-jobs.tsx`
|
- **Files:** `apps/ai-service/ai_service/agents/registry.py` (update), `apps/ai-service/tests/test_registry.py` (update), `apps/ai-service/README.md` (update)
|
||||||
- **Action:** Build job detail page. Layout: header (job title, company, location, salary, apply button), body (full description, required competencies list, AI-matched skills breakdown with match percentages), sidebar (employer info card, related jobs list).
|
- **Action:** Register both agents in the explicit registry. Extend test_registry to assert both resolve. Document the manual ollama-cloud persona probe in README (curl commands with `AI_PROVIDER=ollama-cloud`): Coach and Tutor produce distinct on-persona responses; tests remain cloud-free.
|
||||||
- **Verify:** Job detail renders with all sections; apply button is interactive (non-functional)
|
- **Verify:** `pnpm ai:test` green; manual probe against ollama-cloud shows distinct personas (documented, not automated)
|
||||||
- **Done:** Job detail page complete
|
|
||||||
|
|
||||||
#### Task 3-2-02: Employer profile
|
### Wave 3: Chat endpoint agent routing (depends on Wave 2)
|
||||||
- **Persona:** frontend-engineer
|
|
||||||
- **Files:** `apps/web/app/(marketplace)/employers/[employerId]/page.tsx`, `apps/web/components/marketplace/employer-header.tsx`, `apps/web/components/marketplace/employer-openings.tsx`, `apps/web/components/marketplace/employer-culture.tsx`
|
|
||||||
- **Action:** Build employer profile page. Header: logo, company name, tagline, industry, size, location, social links. Body: about section, culture section (mock photos grid), open positions list (job cards).
|
|
||||||
- **Verify:** Employer profile renders with all sections; social links are clickable (mock URLs)
|
|
||||||
- **Done:** Employer profile complete
|
|
||||||
|
|
||||||
#### Task 3-2-03: Pricing page
|
#### Task 3-3-01: Agent routing on /v1/chat/stream
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** backend-engineer — **REQ:** REQ-2-005, REQ-2-006
|
||||||
- **Files:** `apps/web/app/(marketplace)/pricing/page.tsx`, `apps/web/components/marketplace/pricing-card.tsx`, `apps/web/components/marketplace/feature-comparison.tsx`
|
- **Files:** `apps/ai-service/ai_service/api/chat.py` (update), `apps/ai-service/ai_service/api/deps.py` (update), `apps/ai-service/tests/api/test_chat_stream.py` (update)
|
||||||
- **Action:** Build pricing page. Three pricing cards: Job Posting (single — $49, bundle — $399 for 10, enterprise — custom). Talent Access plans (starter, pro, enterprise). Feature comparison table. CTA buttons on each plan.
|
- **Action:** Chat request gains `agent` field (validated against the registry; unknown agent → 422). Endpoint resolves the agent via DI, persists to the agent-scoped session, meta event carries the agent name. No autonomous routing in v0.2 (A-007).
|
||||||
- **Verify:** Pricing page renders with 3 tiers and comparison table; responsive
|
- **Verify:** TestClient tests: `agent=coach` and `agent=tutor` route correctly, session scoped per agent, unknown agent rejected
|
||||||
- **Done:** Pricing page complete
|
|
||||||
|
|
||||||
### Must-Haves (Phase 3)
|
### Must-Haves (Phase 3)
|
||||||
- [ ] Job board shows 20+ mock job listings with match scores
|
- [ ] Both agents produce distinct, on-persona responses (mock-asserted; manual ollama-cloud probe documented in README)
|
||||||
- [ ] Search bar filters jobs client-side
|
- [ ] Agent routing tested: coach/tutor resolve via registry; unknown agent returns 422
|
||||||
- [ ] Filter sidebar filters by skills, seniority, remote, salary
|
- [ ] Both agents exposed end-to-end via `POST /v1/chat/stream` with agent-scoped session history
|
||||||
- [ ] Job detail page shows full description, competencies, skills breakdown
|
- [ ] `pnpm ai:test` green; no cloud calls in tests
|
||||||
- [ ] Employer profile shows company info, culture, open positions
|
|
||||||
- [ ] Pricing page shows 3 tiers with feature comparison
|
|
||||||
- [ ] All pages responsive at 375px, 768px, 1280px
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 4: Employer Dashboard UI
|
## Phase 4: Lab + Assessor Agents
|
||||||
|
|
||||||
**Requirements:** REQ-018, REQ-019, REQ-020, REQ-021
|
**Requirements:** REQ-2-007, REQ-2-008
|
||||||
**Persona:** frontend-engineer
|
**Goal:** Lab consumes simulated sandbox telemetry and streams in-flow feedback; Assessor applies rubrics to pre-baked artifacts and returns structured scores — both over mock engine inputs
|
||||||
**Goal:** All 4 employer dashboard pages render with mock data; charts display
|
|
||||||
|
|
||||||
### Wave 1: Dashboard + talent search
|
### Wave 1: Mock engine inputs (parallel — no shared files)
|
||||||
|
|
||||||
#### Task 4-1-01: Employer dashboard overview
|
#### Task 4-1-01: Simulated sandbox telemetry corpus
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-007
|
||||||
- **Files:** `apps/web/app/(employer)/page.tsx`, `apps/web/components/employer/overview/metric-cards.tsx`, `apps/web/components/employer/overview/applicant-pipeline.tsx`, `apps/web/components/employer/overview/talent-matches.tsx`, `apps/web/components/employer/overview/analytics-charts.tsx`
|
- **Files:** `apps/ai-service/ai_service/corpus/telemetry.py`
|
||||||
- **Action:** Build employer dashboard overview. Top row: 4 metric cards (active postings, total applicants, talent matches, placement rate). Middle: applicant pipeline (kanban-style columns: applied, screening, interview, offer, hired). Right: talent matches (list of matched candidates with match score). Bottom: analytics charts (recharts — bar chart for postings over time, donut chart for applicant sources, line chart for placement trends).
|
- **Action:** Pydantic-typed Lab telemetry scenarios: scripted build-session event streams (keystrokes, commits, test runs, errors, idle gaps) keyed by scenario ID, aligned with packages/mock-data IDs (D-021).
|
||||||
- **Verify:** Dashboard renders with metric cards, pipeline, matches, and 3 chart types
|
- **Verify:** scenarios import, validate, and are addressable by ID
|
||||||
- **Done:** Employer dashboard overview complete
|
|
||||||
|
|
||||||
#### Task 4-1-02: Talent search
|
#### Task 4-1-02: Pre-baked artifacts + rubrics corpus
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-008
|
||||||
- **Files:** `apps/web/app/(employer)/talent/page.tsx`, `apps/web/components/employer/talent/candidate-card.tsx`, `apps/web/components/employer/talent/talent-filters.tsx`
|
- **Files:** `apps/ai-service/ai_service/corpus/artifacts.py`
|
||||||
- **Action:** Build talent search page. Layout: top (search bar + AI-matched filters), main (grid of candidate cards). Each candidate card: avatar, name, headline (competency stack + level), microcredential badges (top 3), artifact count, defense score, match percentage, "View Profile" button. 15+ mock candidates. Client-side filtering.
|
- **Action:** Pre-baked artifacts (code, design, simulation), assessment rubrics (criteria, levels, weights), and defense transcripts keyed by ID — the Assessor's mock inputs (real engines are v0.3+).
|
||||||
- **Verify:** 15+ candidate cards render; filters work client-side; clicking "View Profile" navigates to candidate profile
|
- **Verify:** rubric/artifact/transcript fixtures validate; IDs align with TS mock data
|
||||||
- **Done:** Talent search complete
|
|
||||||
|
|
||||||
### Wave 2: Detail + management
|
#### Task 4-1-03: TS mock-data alignment for engine inputs
|
||||||
|
- **Persona:** data-engineer — **REQ:** REQ-2-012
|
||||||
|
- **Files:** `packages/mock-data/ai-scenarios.ts` (new), `packages/mock-data/index.ts` (update)
|
||||||
|
- **Action:** Export scenario/artifact ID constants + display metadata used by the Phase 6 learner panels, mirroring `ai_service/corpus/` IDs exactly (D-021). Data-engineer owns the TS side; headers cross-reference the Python corpus.
|
||||||
|
- **Verify:** `pnpm typecheck` passes; IDs string-equal to corpus IDs
|
||||||
|
|
||||||
#### Task 4-2-01: Candidate profile view
|
### Wave 2: Agent implementations (depends on Wave 1)
|
||||||
- **Persona:** frontend-engineer
|
|
||||||
- **Files:** `apps/web/app/(employer)/talent/[candidateId]/page.tsx`, `apps/web/components/employer/talent-profile/artifact-gallery.tsx`, `apps/web/components/employer/talent-profile/process-trace-summary.tsx`, `apps/web/components/employer/talent-profile/defense-transcripts.tsx`, `apps/web/components/employer/talent-profile/competency-mini-graph.tsx`, `apps/web/components/employer/talent-profile/microcredential-verification.tsx`
|
|
||||||
- **Action:** Build candidate profile view. Header: avatar, name, headline, match score, contact button. Body sections: artifact gallery (grid of project artifacts with thumbnails), process trace summary (timeline of build steps), oral defense transcripts (accordion of defense sessions with Q&A), competency mini-graph (react-flow visualization of candidate's competencies), microcredential verification (list of earned credentials with verification badges).
|
|
||||||
- **Verify:** All profile sections render; defense transcripts accordion expands; mini-graph renders
|
|
||||||
- **Done:** Candidate profile complete
|
|
||||||
|
|
||||||
#### Task 4-2-02: Posting management
|
#### Task 4-2-01: Lab agent
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-007
|
||||||
- **Files:** `apps/web/app/(employer)/postings/page.tsx`, `apps/web/components/employer/postings/posting-list.tsx`, `apps/web/components/employer/postings/posting-form.tsx`, `apps/web/components/employer/postings/applicant-list.tsx`
|
- **Files:** `apps/ai-service/ai_service/prompts/lab.py` (finalize), `apps/ai-service/ai_service/agents/lab.py`, `apps/ai-service/tests/test_lab.py`
|
||||||
- **Action:** Build posting management page. Layout: left (list of job postings with status badges: active, draft, expired), right (selected posting detail or create/edit form). Form fields: title, description, required competencies (multi-select), seniority, salary range, location, remote toggle. Applicant list per posting (table with name, applied date, status, match score). "Create New Posting" button opens form.
|
- **Action:** `LabAgent(BaseAgent)` consumes a telemetry scenario, builds messages summarizing the event stream, streams concrete in-flow feedback (what happened, what to adjust, next step). No session chat — scenario-driven.
|
||||||
- **Verify:** Posting list renders; form is interactive (inputs work, non-functional submit); applicant list displays
|
- **Verify:** test_lab: given a mock scenario, feedback references scenario events (mock-scripted assertions)
|
||||||
- **Done:** Posting management complete
|
|
||||||
|
#### Task 4-2-02: Assessor agent
|
||||||
|
- **Persona:** ai-engineer — **REQ:** REQ-2-008
|
||||||
|
- **Files:** `apps/ai-service/ai_service/prompts/assessor.py` (finalize), `apps/ai-service/ai_service/agents/assessor.py`, `apps/ai-service/tests/test_assessor.py`
|
||||||
|
- **Action:** `AssessorAgent(BaseAgent)` applies a rubric to an artifact + defense transcript via `structured_reply`, returning a pydantic-validated rubric score model (per-criterion scores, strengths, gaps, verdict).
|
||||||
|
- **Verify:** test_assessor: structured output validates against the rubric model; failure modes exercise the 4-layer defense
|
||||||
|
|
||||||
|
### Wave 3: Registration (depends on Wave 2)
|
||||||
|
|
||||||
|
#### Task 4-3-01: Register Lab + Assessor
|
||||||
|
- **Persona:** ai-engineer — **REQ:** REQ-2-007, REQ-2-008
|
||||||
|
- **Files:** `apps/ai-service/ai_service/agents/registry.py` (update), `apps/ai-service/tests/test_registry.py` (update)
|
||||||
|
- **Action:** Register both agents; extend registry tests.
|
||||||
|
- **Verify:** registry resolves coach/tutor/lab/assessor; `pnpm ai:test` green
|
||||||
|
|
||||||
|
### Wave 4: Endpoints (depends on Wave 3)
|
||||||
|
|
||||||
|
#### Task 4-4-01: Lab feedback endpoint
|
||||||
|
- **Persona:** backend-engineer — **REQ:** REQ-2-007
|
||||||
|
- **Files:** `apps/ai-service/ai_service/api/lab.py`, `apps/ai-service/tests/api/test_lab.py`
|
||||||
|
- **Action:** `POST /v1/lab/feedback` with scenario ID → resolves corpus scenario + Lab agent → SSE stream using the D-016 envelope (meta names agent=lab). Unknown scenario → 404.
|
||||||
|
- **Verify:** TestClient streams meta + deltas + done + `[DONE]`; unknown scenario 404
|
||||||
|
|
||||||
|
#### Task 4-4-02: Assessment evaluate endpoint
|
||||||
|
- **Persona:** backend-engineer — **REQ:** REQ-2-008
|
||||||
|
- **Files:** `apps/ai-service/ai_service/api/assessment.py`, `apps/ai-service/tests/api/test_assessment.py`
|
||||||
|
- **Action:** `POST /v1/assessment/evaluate` with artifact ID → resolves corpus artifact/rubric/transcript + Assessor agent → JSON response (validated rubric score model). Unknown artifact → 404.
|
||||||
|
- **Verify:** TestClient returns validated rubric JSON; unknown artifact 404; `pnpm ai:test` green
|
||||||
|
|
||||||
### Must-Haves (Phase 4)
|
### Must-Haves (Phase 4)
|
||||||
- [ ] Dashboard overview shows 4 metric cards, applicant pipeline, 3 charts
|
- [ ] Lab produces scenario-relevant in-flow feedback for mock telemetry scenarios (mock provider, tested)
|
||||||
- [ ] Talent search shows 15+ candidate cards with filters
|
- [ ] Assessor returns structured rubric scores (pydantic-validated JSON) for pre-baked artifacts/transcripts
|
||||||
- [ ] Candidate profile shows artifact gallery, process trace, defense transcripts, mini-graph, credentials
|
- [ ] `POST /v1/lab/feedback` streams (meta → deltas → done → `[DONE]`); `POST /v1/assessment/evaluate` returns validated JSON
|
||||||
- [ ] Posting management shows posting list, create/edit form, applicant list
|
- [ ] Unknown scenario/artifact IDs return 404
|
||||||
- [ ] All pages responsive at 375px, 768px, 1280px
|
- [ ] Corpus IDs align with packages/mock-data (D-021); `pnpm ai:test` and `pnpm typecheck` green
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 5: Admin Surface UI
|
## Phase 5: Proctor + Mentor Agents
|
||||||
|
|
||||||
**Requirements:** REQ-022, REQ-023, REQ-024, REQ-025
|
**Requirements:** REQ-2-009, REQ-2-010
|
||||||
**Persona:** frontend-engineer
|
**Goal:** Proctor classifies integrity signals with coaching interventions from mock telemetry; Mentor generates long-horizon career narrative; both exposed via endpoints
|
||||||
**Goal:** All 4 admin pages render; competency graph viewer interactive
|
|
||||||
|
|
||||||
### Wave 1: Overview + learner management
|
### Wave 1: Proctor scenarios + Mentor agent (parallel — no shared files)
|
||||||
|
|
||||||
#### Task 5-1-01: Admin overview
|
#### Task 5-1-01: Proctor telemetry scenarios
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-009
|
||||||
- **Files:** `apps/web/app/(admin)/page.tsx`, `apps/web/components/admin/overview/platform-metrics.tsx`, `apps/web/components/admin/overview/activity-feed.tsx`, `apps/web/components/admin/overview/system-health.tsx`
|
- **Files:** `apps/ai-service/ai_service/corpus/telemetry.py` (update)
|
||||||
- **Action:** Build admin overview. Top: platform metric cards (total learners, total employers, placements, completion rate, NPS). Middle: activity feed (timeline of recent platform events — new registrations, completions, job postings, placements). Right: system health mockup (status indicators for services, uptime bars, error rates).
|
- **Action:** Add proctor scenarios: tab switches, idle time, paste events, focus loss — scripted integrity-relevant event sets keyed by scenario ID.
|
||||||
- **Verify:** Admin overview renders with metrics, activity feed, system health
|
- **Verify:** proctor scenarios validate; distinguishable from lab scenarios by type
|
||||||
- **Done:** Admin overview complete
|
|
||||||
|
|
||||||
#### Task 5-1-02: Learner management
|
#### Task 5-1-02: Mentor agent (+ registration)
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-010
|
||||||
- **Files:** `apps/web/app/(admin)/learners/page.tsx`, `apps/web/components/admin/learners/learner-table.tsx`, `apps/web/components/admin/learners/learner-detail.tsx`
|
- **Files:** `apps/ai-service/ai_service/prompts/mentor.py` (finalize), `apps/ai-service/ai_service/agents/mentor.py`, `apps/ai-service/tests/test_mentor.py`, `apps/ai-service/ai_service/agents/registry.py` (update)
|
||||||
- **Action:** Build learner management page. Layout: searchable learner table (columns: name, email, stack, progress %, status, joined date). Search and filter (by stack, status). Clicking a learner opens detail panel (progress tracking, competency completion, credential issuance log, recent activity).
|
- **Action:** `MentorAgent(BaseAgent)`: long-horizon career narrative — trajectory story, competency-stack progression guidance, market positioning — streaming, session-backed. Registered centrally in `registry.py` (single registration pattern, G-4).
|
||||||
- **Verify:** Learner table renders with mock data; search filters; detail panel opens on click
|
- **Verify:** test_mentor: narrative references learner context (mock-scripted); registry resolves mentor
|
||||||
- **Done:** Learner management complete
|
|
||||||
|
|
||||||
### Wave 2: Graph viewer + moderation
|
### Wave 2: Proctor agent + Mentor endpoint (depends on Wave 1)
|
||||||
|
|
||||||
#### Task 5-2-01: Competency graph viewer
|
#### Task 5-2-01: Proctor agent (+ registration)
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** ai-engineer — **REQ:** REQ-2-009
|
||||||
- **Files:** `apps/web/app/(admin)/graph/page.tsx`, `apps/web/components/admin/graph/competency-graph.tsx`, `apps/web/components/admin/graph/node-detail.tsx`
|
- **Files:** `apps/ai-service/ai_service/prompts/proctor.py` (finalize), `apps/ai-service/ai_service/agents/proctor.py`, `apps/ai-service/tests/test_proctor.py`, `apps/ai-service/ai_service/agents/registry.py` (update)
|
||||||
- **Action:** Build competency graph viewer using @xyflow/react. Interactive node/edge graph showing competency stacks and their relationships. Nodes: competency stacks (colored by category) and individual competencies. Edges: prerequisite relationships. Node click opens detail panel (competency name, description, stack, prerequisites, learners mastering it). Controls: zoom, pan, fit-to-screen. Mock graph data.
|
- **Action:** `ProctorAgent(BaseAgent)`: consumes proctor scenario → `structured_reply` returns pydantic-validated signal classification (severity, signal type) + recommended coaching intervention (supportive, not punitive). Registered centrally in `registry.py` (single registration pattern, G-4).
|
||||||
- **Verify:** Graph renders with nodes and edges; nodes are clickable; detail panel opens; zoom/pan works
|
- **Verify:** test_proctor: classified signals + interventions validate for each mock scenario; registry resolves all six agents
|
||||||
- **Done:** Competency graph viewer complete
|
|
||||||
|
|
||||||
#### Task 5-2-02: Marketplace moderation
|
#### Task 5-2-02: Mentor narrative endpoint
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** backend-engineer — **REQ:** REQ-2-010
|
||||||
- **Files:** `apps/web/app/(admin)/moderation/page.tsx`, `apps/web/components/admin/moderation/review-queue.tsx`, `apps/web/components/admin/moderation/verification-queue.tsx`, `apps/web/components/admin/moderation/flagged-content.tsx`
|
- **Files:** `apps/ai-service/ai_service/api/mentor.py`, `apps/ai-service/tests/api/test_mentor.py`
|
||||||
- **Action:** Build marketplace moderation page. Three tabs: Job Posting Review (queue of pending job postings with approve/reject buttons), Employer Verification (queue of employers awaiting identity verification), Flagged Content (list of reported content with reason, reporter, actions). All buttons non-functional but interactive (hover states, click feedback).
|
- **Action:** `POST /v1/mentor/narrative` → Mentor agent → SSE stream with D-016 envelope, session-backed.
|
||||||
- **Verify:** Moderation page renders with 3 tabs; queues display mock items; tab switching works
|
- **Verify:** TestClient streams meta (agent=mentor) → deltas → done → `[DONE]`
|
||||||
- **Done:** Marketplace moderation complete
|
|
||||||
|
### Wave 3: Proctor endpoint (depends on Wave 2)
|
||||||
|
|
||||||
|
#### Task 5-3-01: Proctor signals endpoint
|
||||||
|
- **Persona:** backend-engineer — **REQ:** REQ-2-009
|
||||||
|
- **Files:** `apps/ai-service/ai_service/api/proctor.py`, `apps/ai-service/tests/api/test_proctor.py`
|
||||||
|
- **Action:** `POST /v1/proctor/signals` with scenario ID → resolves corpus scenario + Proctor agent → JSON response (validated signals + interventions). Unknown scenario → 404.
|
||||||
|
- **Verify:** TestClient returns classified signals JSON; `pnpm ai:test` green — full suite (all six agents registered)
|
||||||
|
|
||||||
### Must-Haves (Phase 5)
|
### Must-Haves (Phase 5)
|
||||||
- [ ] Admin overview shows platform metrics, activity feed, system health
|
- [ ] Proctor produces classified signals with recommended coaching interventions for each mock scenario (structured JSON, validated)
|
||||||
- [ ] Learner management shows searchable table with detail panel
|
- [ ] Mentor produces coherent long-horizon career narrative (streaming, session-backed)
|
||||||
- [ ] Competency graph viewer renders interactive graph with clickable nodes
|
- [ ] `POST /v1/proctor/signals` returns validated JSON; `POST /v1/mentor/narrative` streams
|
||||||
- [ ] Marketplace moderation shows 3 tabbed queues
|
- [ ] Registry resolves all six agents; full ai-service test suite green, cloud-free
|
||||||
- [ ] All pages responsive at 375px, 768px, 1280px
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 6: Polish + Integration
|
## Phase 6: Learner Surface Integration
|
||||||
|
|
||||||
**Requirements:** REQ-026, REQ-027, REQ-028
|
**Requirements:** REQ-2-011, REQ-2-012
|
||||||
**Persona:** frontend-engineer, design-system-engineer
|
**Goal:** v0.1 learner surfaces wired to the real ai-service: streaming chat with agent switcher, Lab/Assessor/Proctor/Mentor outputs surfaced, error/loading states, build + typecheck green
|
||||||
**Goal:** Cross-surface consistency, dark mode everywhere, Storybook
|
**Note (G-2):** End-to-end verification may run with `AI_PROVIDER=mock` as a fallback — the requirement is the real ai-service over HTTP (not canned client-side responses); provider choice is service-internal. This prevents an ollama-cloud outage from blocking P6 verification. Cloud persona probes remain separate (Task 3-2-01).
|
||||||
|
|
||||||
### Wave 1: Navigation + consistency
|
### Wave 1: Client plumbing + primitives (parallel — no shared files)
|
||||||
|
|
||||||
#### Task 6-1-01: Cross-surface navigation polish
|
#### Task 6-1-01: useChatStream hook
|
||||||
- **Persona:** frontend-engineer
|
- **Persona:** frontend-engineer — **REQ:** REQ-2-011
|
||||||
- **Files:** `apps/web/components/navigation-shell.tsx` (update), `apps/web/components/role-switcher.tsx` (update), `apps/web/components/breadcrumbs.tsx`
|
- **Files:** `apps/web/hooks/use-chat-stream.ts`, `apps/web/.env.example` (update)
|
||||||
- **Action:** Polish cross-surface navigation. Role switcher dropdown with surface-specific sub-navigation. Breadcrumbs on all pages showing current location. Consistent header/footer across all 4 surfaces. Active nav link highlighting.
|
- **Action:** `useChatStream(agent)` hook: `fetch` POST to `${NEXT_PUBLIC_AI_SERVICE_URL}/v1/chat/stream` (default `http://localhost:8420`, A-002 — no API-route proxy); consumes `ReadableStream` with byte buffering, frame split on `\n\n`, joined `data:` lines; **ignores frames containing no `data:` lines (sse-starlette `: ping` keep-alive comment frames)** — TestClient streams are too short to surface pings, but real cloud delta gaps emit them (G-1); handles meta / delta / done / error events and `[DONE]` sentinel; idempotent `AbortController.abort()` in effect cleanup; exposes `{messages, isStreaming, error, send, retry, abort}`. `.env.example` gains `NEXT_PUBLIC_AI_SERVICE_URL`.
|
||||||
- **Verify:** Breadcrumbs show on all pages; role switcher dropdown works; active links highlighted
|
- **Verify:** hook unit-tested or exercised via the chat UI; unmount mid-stream aborts cleanly (no state updates after unmount)
|
||||||
- **Done:** Cross-surface navigation polished
|
|
||||||
|
|
||||||
#### Task 6-1-02: Visual consistency audit
|
#### Task 6-1-02: Agent switcher + streaming primitives
|
||||||
- **Persona:** design-system-engineer
|
- **Persona:** design-system-engineer — **REQ:** REQ-2-011
|
||||||
- **Files:** `apps/web/app/globals.css` (update), `packages/ui/src/tokens/index.ts` (update if needed)
|
- **Files:** `packages/ui/src/primitives/agent-switcher.tsx`, `packages/ui/src/primitives/stream-status.tsx`, `packages/ui/src/primitives/toast.tsx`, `packages/ui/src/primitives/index.ts` (update), `packages/ui/src/index.ts` (update)
|
||||||
- **Action:** Audit all surfaces for visual consistency. Check: typography scale (all text uses design tokens), color palette (no hardcoded colors), spacing system (all margins/padding use design tokens), dark mode (all pages support dark mode toggle), WCAG AA contrast (all text meets 4.5:1 ratio). Fix any inconsistencies found.
|
- **Action:** Token-driven primitives: AgentSwitcher (segmented coach/tutor control with active state), StreamStatus (idle/streaming/error indicator), Toast with error variant. Dark mode + WCAG AA contrast; exported from `@nextcraft/ui`.
|
||||||
- **Verify:** No hardcoded colors; dark mode works on all pages; contrast check passes
|
- **Verify:** primitives import from `@nextcraft/ui`; storybook stories render (dark + light)
|
||||||
- **Done:** Visual consistency audit complete
|
|
||||||
|
|
||||||
### Wave 2: Storybook
|
### Wave 2: Chat rewrite + Byte viewer panel (depends on Wave 1)
|
||||||
|
|
||||||
#### Task 6-2-01: Storybook setup
|
#### Task 6-2-01: Real streaming learner chat
|
||||||
- **Persona:** design-system-engineer
|
- **Persona:** frontend-engineer — **REQ:** REQ-2-011
|
||||||
- **Files:** `apps/web/.storybook/main.ts`, `apps/web/.storybook/preview.ts`, `packages/ui/src/primitives/*.stories.tsx` (one per primitive)
|
- **Files:** `apps/web/components/learner/ai-tutor-chat.tsx` (rewrite), `apps/web/components/learner/agent-switcher.tsx` (composition wrapper)
|
||||||
- **Action:** Set up Storybook for the component library. Configure for Next.js + Tailwind. Create stories for all 12 primitives and key composites (JobCard, CandidateCard, CompetencyBadge, MetricCard). Each story: default, variants, sizes, dark mode.
|
- **Action:** Replace the canned `aiTutorResponses` behavior with `useChatStream`: agent switcher (Coach/Tutor per A-007), token-by-token rendering, streaming cursor + loading state, error state with retry button when ai-service is down (A-010), suggested-action chips from the meta event. Seed welcome message stays static.
|
||||||
- **Verify:** `pnpm storybook` starts; all primitive stories render; dark mode toggle in Storybook
|
- **Verify:** with ai-service running, messages stream visibly token-by-token; with ai-service stopped, error state + retry appears (no crash, no console errors)
|
||||||
- **Done:** Storybook documents all primitives + key composites
|
|
||||||
|
#### Task 6-2-02: Byte viewer Tutor panel
|
||||||
|
- **Persona:** frontend-engineer — **REQ:** REQ-2-011 (agent routing: byte viewer always uses Tutor, A-007)
|
||||||
|
- **Files:** `apps/web/app/(learner)/learn/[competencyId]/page.tsx` (update), `apps/web/components/learner/byte-tutor-panel.tsx` (new)
|
||||||
|
- **Action:** Byte viewer gains a Tutor explanation panel (byte viewer always uses Tutor, A-007): "Explain this byte" streams a Socratic concept walkthrough for the current competency via useChatStream (agent fixed to tutor).
|
||||||
|
- **Verify:** on a byte page, the panel streams a Tutor explanation; error state when service down
|
||||||
|
|
||||||
|
### Wave 3: Lab / Assessment / Mentor panels (depends on Wave 1 hook; parallel — no shared files)
|
||||||
|
|
||||||
|
#### Task 6-3-01: Sandbox Lab feedback panel
|
||||||
|
- **Persona:** frontend-engineer — **REQ:** REQ-2-012
|
||||||
|
- **Files:** `apps/web/app/(learner)/build/[competencyId]/page.tsx` (update), `apps/web/components/learner/lab-feedback-panel.tsx` (new)
|
||||||
|
- **Action:** Sandbox telemetry sidebar gains a Lab feedback panel: posts the scenario ID (from `packages/mock-data/ai-scenarios`) to `/v1/lab/feedback`, streams in-flow feedback into the panel; loading + error states.
|
||||||
|
- **Verify:** sandbox page streams Lab feedback for the mock scenario; error state when service down
|
||||||
|
|
||||||
|
#### Task 6-3-02: Assessment Assessor + Proctor surfaces
|
||||||
|
- **Persona:** frontend-engineer — **REQ:** REQ-2-012
|
||||||
|
- **Files:** `apps/web/app/(learner)/defend/[competencyId]/page.tsx` (update), `apps/web/components/learner/assessor-results-panel.tsx` (new), `apps/web/components/learner/proctor-banner.tsx` (new)
|
||||||
|
- **Action:** Assessment mockup: AI reviewer panel calls `/v1/assessment/evaluate` with the artifact ID and renders the structured rubric scores (per-criterion bars, strengths, gaps, verdict); a Proctor integrity banner surfaces `/v1/proctor/signals` classifications with coaching tone. Loading skeletons + error states.
|
||||||
|
- **Verify:** defend page renders real Assessor rubric output + Proctor banner; error states when service down
|
||||||
|
|
||||||
|
#### Task 6-3-03: Dashboard Mentor panel
|
||||||
|
- **Persona:** frontend-engineer — **REQ:** REQ-2-012
|
||||||
|
- **Files:** `apps/web/app/(learner)/dashboard/page.tsx` (update), `apps/web/components/learner/mentor-panel.tsx` (new)
|
||||||
|
- **Action:** Learner dashboard gains a Mentor panel: streams career narrative from `/v1/mentor/narrative` (learner progress context), with regenerate button, loading + error states. Sits alongside the existing AI tutor chat.
|
||||||
|
- **Verify:** dashboard shows streaming Mentor narrative; error state when service down
|
||||||
|
|
||||||
### Must-Haves (Phase 6)
|
### Must-Haves (Phase 6)
|
||||||
- [ ] Breadcrumbs on all pages
|
- [ ] With ai-service running: learner chat at http://localhost:3000/dashboard streams real responses token-by-token (mock provider fallback allowed per G-2 — service over HTTP is the requirement)
|
||||||
- [ ] Role switcher dropdown with surface-specific nav
|
- [ ] Agent switcher flips Coach ↔ Tutor and the response persona changes accordingly
|
||||||
- [ ] Active nav link highlighting
|
- [ ] Hook tolerates keep-alive comment frames (`: ping`, no data lines) during live streams (G-1)
|
||||||
- [ ] No hardcoded colors — all from design tokens
|
- [ ] With ai-service stopped: all chat/panels show error states with retry — no crashes, no unhandled promise rejections, no console errors
|
||||||
- [ ] Dark mode works on all 4 surfaces
|
- [ ] Byte viewer, sandbox, and assessment mockups surface Tutor/Lab/Assessor/Proctor outputs; dashboard shows Mentor narrative
|
||||||
- [ ] WCAG AA contrast passes
|
- [ ] Unmounting/navigating mid-stream aborts cleanly (no post-unmount state updates)
|
||||||
- [ ] Storybook runs with all primitive stories
|
- [ ] `pnpm build` and `pnpm typecheck` pass; `pnpm ai:test` still green
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## MVP/UX Sections
|
## Phase 7: Final Review + Ship (no planned tasks)
|
||||||
|
|
||||||
### User-Facing Surface
|
Orchestrated by the SHIP stage, not this plan: multi-persona code review (correctness, testing, secrets hygiene — key absent from code/logs/commits/errors, localhost-only CORS, no PII in prompts), project health audit (reconstruction test, .ciagent/ discipline, branch/commit hygiene), then merge milestone → main, tag v0.2.0, create Gitea release, mark all 12 v0.2 requirements complete.
|
||||||
The primary user-facing surface is the **Nextcraft web application** at `localhost:3000` (dev) with 4 route groups:
|
|
||||||
- `/` — Learner surface (landing, catalog, dashboard, learn, build, defend)
|
|
||||||
- `/marketplace` — Marketplace surface (jobs, employers, pricing)
|
|
||||||
- `/employer` — Employer dashboard (overview, talent, postings)
|
|
||||||
- `/admin` — Admin surface (overview, learners, graph, moderation)
|
|
||||||
|
|
||||||
### Happy Path
|
**Release-note honesty (G-5):** the v0.2.0 release note must explicitly state that Lab/Assessor/Proctor operate on mock engine inputs (real engines are v0.3+), per D-015.
|
||||||
1. User visits landing page → sees hero, how-it-works (Byte→Build→Demonstrate→Defend), program highlights
|
|
||||||
2. User clicks "Explore Programs" → program catalog shows 5 competency stacks
|
|
||||||
3. User clicks "AI Orchestration Engineer" → competency stack view shows 15 competencies
|
|
||||||
4. User clicks a competency → byte tutorial viewer shows concept + worked example
|
|
||||||
5. User clicks "Start Building" → build sandbox mockup shows IDE UI with telemetry
|
|
||||||
6. User clicks "Submit for Assessment" → assessment/defense mockup shows rubric, AI reviewer, oral defense interface
|
|
||||||
7. User navigates to marketplace → job board shows AI-era job listings with match scores
|
|
||||||
8. User searches/filters jobs → results update client-side
|
|
||||||
9. User clicks a job → job detail page shows full description, required competencies, employer info
|
|
||||||
10. User switches to employer dashboard → overview shows metrics, pipeline, charts
|
|
||||||
11. User searches talent → candidate cards with microcredentials, defense scores
|
|
||||||
12. User clicks a candidate → profile shows artifact gallery, process trace, defense transcripts
|
|
||||||
13. User switches to admin → overview shows platform metrics, activity feed
|
|
||||||
14. User opens competency graph viewer → interactive graph with clickable nodes
|
|
||||||
|
|
||||||
### UX Acceptance Criteria
|
**Dead-code disposal (G-5):** review must dispose of `aiTutorResponses` (packages/mock-data/ai-tutor-responses.ts) — its only consumer is rewritten in Task 6-2-01; remove the export or mark it deprecated.
|
||||||
1. All 4 surfaces are accessible via the role switcher in the header
|
|
||||||
2. Navigation is consistent across all surfaces (same header, footer, breadcrumb pattern)
|
---
|
||||||
3. All pages are responsive at 375px (mobile), 768px (tablet), 1280px (desktop)
|
|
||||||
4. Dark mode toggle works on all surfaces
|
## User-Facing Surface
|
||||||
5. All interactive elements have hover states and click feedback
|
|
||||||
6. Form inputs are non-functional but visually complete (placeholders, labels, validation states)
|
The primary user-facing surface is the **learner dashboard and learning flow** at `http://localhost:3000`, backed by the real ai-service at `http://localhost:8420`:
|
||||||
7. Mock data is realistic and typed — no "lorem ipsum" or placeholder text
|
|
||||||
8. All charts render correctly (recharts) with mock data
|
- `/dashboard` — AI tutor chat (Coach/Tutor switcher, streaming) + Mentor career-narrative panel
|
||||||
9. Competency graph viewer is interactive (zoom, pan, click nodes)
|
- `/learn/[competencyId]` — byte viewer with streaming Tutor explanations
|
||||||
10. AI tutor chat displays pre-scripted responses on message "send"
|
- `/build/[competencyId]` — sandbox with Lab in-flow feedback panel (mock telemetry)
|
||||||
11. No console errors on any page
|
- `/defend/[competencyId]` — assessment with live Assessor rubric scores + Proctor integrity banner
|
||||||
12. `pnpm build` succeeds without warnings
|
|
||||||
|
The marketplace, employer, and admin surfaces are unchanged from v0.1.
|
||||||
|
|
||||||
|
## Happy Path
|
||||||
|
|
||||||
|
1. Learner opens `/dashboard` → chat shows welcome message; meta event confirms coach/model in the stream
|
||||||
|
2. Learner types "I'm stuck on multi-agent communication" → reply streams token-by-token with pacing guidance + a retrieval-practice prompt
|
||||||
|
3. Learner switches to **Tutor** → asks the same question → gets a Socratic concept walkthrough instead
|
||||||
|
4. Learner opens a byte tutorial → Tutor panel streams an explanation of the current competency
|
||||||
|
5. Learner opens the build sandbox → Lab panel streams feedback on the simulated telemetry scenario
|
||||||
|
6. Learner opens the defense mockup → Assessor panel shows structured rubric scores; Proctor banner shows integrity signals in coaching tone
|
||||||
|
7. Back on `/dashboard`, the Mentor panel streams a career narrative tied to the learner's progress
|
||||||
|
8. Learner kills ai-service (or it crashes) → next message shows an inline error state with **Retry**; restarting the service and retrying resumes streaming
|
||||||
|
|
||||||
|
## UX Acceptance Criteria
|
||||||
|
|
||||||
|
1. Streaming is visibly incremental — tokens appear as they arrive, not as one blob
|
||||||
|
2. Agent switcher shows the active agent (Coach/Tutor) and the response persona visibly changes
|
||||||
|
3. Loading state during connection (streaming cursor / skeleton) before first token
|
||||||
|
4. When ai-service is unreachable: inline error state + retry action on every chat/panel — no crashes, no console errors, no blank UI
|
||||||
|
5. `[DONE]` reliably ends the stream (input re-enables, no stuck "typing" state)
|
||||||
|
6. Navigating away mid-stream aborts cleanly — no leaked requests or post-unmount updates
|
||||||
|
7. All new UI uses design tokens, supports dark mode, meets WCAG AA contrast
|
||||||
|
8. Responsive at 375px, 768px, 1280px
|
||||||
|
9. No hardcoded model names or URLs in UI code — all via `NEXT_PUBLIC_AI_SERVICE_URL` and server meta events
|
||||||
|
10. `pnpm build` and `pnpm typecheck` pass with zero errors
|
||||||
+41
-19
@@ -8,38 +8,57 @@ Nextcraft is an AI-native outcome school where graduates prove what they can bui
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Current Milestone: v0.1 — UI/UX Prototype
|
## Current Milestone: v0.2 — AI Tutor Architecture
|
||||||
|
|
||||||
**Scope:** High-fidelity interactive prototype of all four Nextcraft surfaces with realistic mock data, navigation flows, responsive layouts, and a shared component library. No backend, no business logic, no database — everything static/mocked.
|
**Scope:** The six AI tutor agents (Coach, Tutor, Lab, Assessor, Proctor, Mentor) as real LLM-backed services in a new `apps/ai-service` Python FastAPI application, wired into the existing v0.1 learner surface chat UI with streaming responses. Provider-agnostic LLM layer (ollama-cloud default, local endpoint + deterministic mock for tests). Lab/Assessor/Proctor operate on mock engine inputs (simulated telemetry, pre-baked artifacts) — their real engines (sandbox fabric, assessment engine, identity verification) are v0.3+.
|
||||||
|
|
||||||
**Constraint:** MAJOR 0 until MVP is released. No business logic until the first milestone prototype is agreed upon by the founder.
|
**Status of v0.1:** Complete and shipped (v0.1.0). Founder agreement recorded (D-013).
|
||||||
|
|
||||||
**Tech stack:** TypeScript monorepo (pnpm/turborepo) with Next.js for all web surfaces. Python FastAPI AI services planned for later milestones (not in v0.1).
|
**Tech stack:** v0.1 TS monorepo (pnpm/turborepo, Next.js) + new Python FastAPI service (`apps/ai-service`) with pydantic, SSE streaming, and an OpenAI-compatible provider client.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Requirements (Validated)
|
## Requirements (Validated)
|
||||||
|
|
||||||
The following requirements have been validated during specification and are locked for milestone v0.1:
|
The following requirements have been validated during specification and are locked for milestone v0.2 (REQ-F-001..006 activated from the deferred pool):
|
||||||
|
|
||||||
1. Learner surface UI — landing page, program catalog, competency stack view, learner dashboard, byte tutorial viewer, build sandbox mockup, assessment/defense mockup
|
1. AI tutor service infrastructure — `apps/ai-service` FastAPI application, provider-agnostic LLM client, SSE streaming, session/state handling
|
||||||
2. Marketplace surface UI — job board listing, job detail page, employer profile, search/filter UI, job posting packages pricing page
|
2. Agent framework — base agent contracts, prompt management, streaming pipeline, structured outputs
|
||||||
3. Employer dashboard UI — overview, talent search, candidate profile view, posting management
|
3. Coach agent — pacing, motivation, retrieval practice (REQ-F-001)
|
||||||
4. Admin surface UI — overview, learner management, competency graph viewer, marketplace moderation
|
4. Tutor agent — concept delivery, Socratic questioning (REQ-F-002)
|
||||||
5. Shared component library — design system, reusable UI primitives, surface-specific theming
|
5. Lab agent — in-flow feedback over simulated sandbox telemetry (REQ-F-003, mock inputs)
|
||||||
6. Responsive layout system — mobile, tablet, desktop breakpoints
|
6. Assessor agent — rubric application to pre-baked artifacts and defenses (REQ-F-004, mock inputs)
|
||||||
7. Mock data layer — typed, realistic data reflecting the vision (competency stacks, job listings, candidate profiles)
|
7. Proctor agent — integrity signals from mock telemetry, coaching interventions (REQ-F-005, mock inputs)
|
||||||
8. Navigation/routing — cross-surface navigation, role-based route groups
|
8. Mentor agent — long-horizon career narrative (REQ-F-006)
|
||||||
|
9. Learner surface integration — streaming chat UI wired to the real service, error/loading states
|
||||||
|
|
||||||
|
## v0.1 Requirements (Complete)
|
||||||
|
|
||||||
|
All 28 v0.1 requirements (REQ-001..028) are complete and shipped as v0.1.0. See REQUIREMENTS.md traceability matrix.
|
||||||
|
|
||||||
|
## Clarified Assumptions (CLARIFY stage, full autonomy — auto-resolved)
|
||||||
|
|
||||||
|
| # | Ambiguity | Resolution | Confidence |
|
||||||
|
|---|-----------|------------|-------------|
|
||||||
|
| A-001 | Where does the ai-service live in the monorepo? | `apps/ai-service` — pnpm-workspace ignores Python; it integrates via root package.json scripts (`ai:dev`, `ai:test`), not as a pnpm package. Turbo gets passthrough tasks. | 0.95 |
|
||||||
|
| A-002 | How does the Next.js client talk to ai-service? | Direct fetch to `http://localhost:8420` (configurable via `NEXT_PUBLIC_AI_SERVICE_URL`) with SSE parsing. No Next.js API-route proxy in v0.2 — client components talk straight to the service. | 0.85 |
|
||||||
|
| A-003 | Session persistence? | In-memory dict keyed by session ID (v0.2 has no DB). Sessions lost on restart — acceptable for this milestone; store interface is DB-migration-ready. | 0.9 |
|
||||||
|
| A-004 | Port for ai-service? | 8420 (avoids common dev-port collisions with 3000/8000; documented in .env.example). | 0.8 |
|
||||||
|
| A-005 | Which ollama-cloud model? | Default `gemma4:31b` (probe-verified); configurable via `AI_TUTOR_MODEL` env. Model choice is a config, not code. | 0.85 |
|
||||||
|
| A-006 | Streaming format? | SSE with `data:` JSON lines (OpenAI-compatible delta objects), terminated by `data: [DONE]`. Matches the provider contract, so the provider layer passes deltas through unchanged. | 0.9 |
|
||||||
|
| A-007 | Agent routing in the chat UI? | Explicit agent switcher (Coach/Tutor) in the learner chat; Byte viewer always uses Tutor; sandbox uses Lab; assessment uses Assessor+Proctor; dashboard Mentor panel. No autonomous routing in v0.2. | 0.9 |
|
||||||
|
| A-008 | Auth between web and ai-service? | None in v0.2 (local dev surface). CORS limited to localhost origins. Real auth is v0.3+ with identity work. | 0.85 |
|
||||||
|
| A-009 | Python tooling? | `python3 -m venv` + pip (venv is the only available mechanism in this environment; no uv). Pydantic v2, FastAPI, uvicorn, pytest — all PyPI-reachable (verified). | 0.9 |
|
||||||
|
| A-010 | What happens when the LLM provider is unreachable? | Streaming endpoints return an error event; the UI shows error states with retry. Mock provider guarantees tests never call the cloud. | 0.9 |
|
||||||
|
|
||||||
## Requirements (Active — Future Milestones)
|
## Requirements (Active — Future Milestones)
|
||||||
|
|
||||||
The following are deferred beyond v0.1 and will be activated in subsequent milestones:
|
The following remain deferred beyond v0.2 and will be activated in subsequent milestones:
|
||||||
|
|
||||||
- AI tutor agent architecture (Coach, Tutor, Lab, Assessor, Proctor, Mentor)
|
|
||||||
- Competency graph engine and adaptive pathways
|
- Competency graph engine and adaptive pathways
|
||||||
- Assessment engine (process-trace grading, oral defense, per-learner variant tasks)
|
- Assessment engine (process-trace grading, oral defense, per-learner variant tasks) — v0.3+
|
||||||
- Sandbox fabric (sandboxed IDE, design tool, simulation)
|
- Sandbox fabric (sandboxed IDE, design tool, simulation) — v0.3+
|
||||||
- Identity verification and age-gating logic (16+/18+)
|
- Identity verification and age-gating logic (16+/18+) — the real KYC backend (v0.3+; visual flow already exists in v0.1)
|
||||||
- Marketplace job aggregation pipeline (3M+ jobs from 120K companies)
|
- Marketplace job aggregation pipeline (3M+ jobs from 120K companies)
|
||||||
- AI-powered tagging, semantic vector search, company enrichment
|
- AI-powered tagging, semantic vector search, company enrichment
|
||||||
- AI resume parsing and job matching
|
- AI resume parsing and job matching
|
||||||
@@ -91,7 +110,10 @@ The following are deferred beyond v0.1 and will be activated in subsequent miles
|
|||||||
| D-009 | AI tutor UI as chat interface mockup with pre-scripted responses | The learner surface includes an AI tutor chat UI mockup. No real AI backend — pre-scripted responses simulate the Coach and Tutor agents. | Mockup only in v0.1, real agents in future milestone |
|
| D-009 | AI tutor UI as chat interface mockup with pre-scripted responses | The learner surface includes an AI tutor chat UI mockup. No real AI backend — pre-scripted responses simulate the Coach and Tutor agents. | Mockup only in v0.1, real agents in future milestone |
|
||||||
| D-010 | Age-gating represented as visual registration flow mockup | 16+/18+ age-gating shown as a UI flow with age verification step. No actual verification logic. | Visual mockup only |
|
| D-010 | Age-gating represented as visual registration flow mockup | 16+/18+ age-gating shown as a UI flow with age verification step. No actual verification logic. | Visual mockup only |
|
||||||
| D-011 | Competency graph viewer as interactive static visualization | Admin surface includes a competency graph viewer using react-flow or similar. Mock competency nodes and edges. No real graph data. | Static graph with mock data |
|
| D-011 | Competency graph viewer as interactive static visualization | Admin surface includes a competency graph viewer using react-flow or similar. Mock competency nodes and edges. No real graph data. | Static graph with mock data |
|
||||||
| D-012 | Tech stack: TS monorepo + Python AI services (future) | v0.1 uses TS only. Python FastAPI microservices planned for AI tutor agents and assessment engine in later milestones. | v0.1: TS only. Future: TS + Python |
|
| D-012 | Tech stack: TS monorepo + Python AI services (future) | v0.1 uses TS only. Python FastAPI microservices planned for AI tutor agents and assessment engine in later milestones. | v0.1: TS only. v0.2: TS + Python (apps/ai-service) |
|
||||||
|
| D-013 | v0.1 prototype founder-agreed; D-001 business-logic gate unlocked | Founder approved starting v0.2 with AI Tutor Architecture, which constitutes agreement of the v0.1 prototype per D-001. Recorded at v0.2 SPECIFY. | Business logic authorized from v0.2 onward |
|
||||||
|
| D-014 | Provider-agnostic LLM layer; ollama-cloud as initial provider | OpenAI-compatible client abstraction with pluggable providers: ollama-cloud (https://ollama.com/v1, default), local OpenAI-compatible endpoint, deterministic mock (tests/CI). Keys in gitignored .ciagent/.env.secrets, never in code or commits. | apps/ai-service llm package with 3 providers; default=ollama-cloud |
|
||||||
|
| D-015 | All six agents implemented as real LLM services; engines mocked | Coach/Tutor/Mentor fully real. Lab/Assessor/Proctor are real LLM logic over mock inputs (simulated telemetry, pre-baked artifacts) since sandbox fabric, assessment engine, and identity verification are v0.3+. Consistent with v0.1's mock-data approach. | REQ-F-001..006 complete in v0.2; real engines deferred to v0.3+ |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+54
-14
@@ -1,12 +1,43 @@
|
|||||||
# Nextcraft — REQUIREMENTS.md
|
# Nextcraft — REQUIREMENTS.md
|
||||||
|
|
||||||
## v0.1 Requirements (UI/UX Prototype)
|
## v0.2 Requirements (AI Tutor Architecture)
|
||||||
|
|
||||||
|
### AI Service Infrastructure
|
||||||
|
|
||||||
|
| ID | Description | Priority | Phase | Status |
|
||||||
|
|----|-------------|----------|-------|--------|
|
||||||
|
| REQ-2-001 | apps/ai-service scaffolding: Python FastAPI app, pydantic settings, uvicorn, health endpoint, CORS, pytest setup, pnpm/turbo integration scripts | critical | 1 | complete |
|
||||||
|
| REQ-2-002 | Provider-agnostic LLM client: OpenAI-compatible provider interface with ollama-cloud (default), local-endpoint, and deterministic mock providers; key resolution from env files | critical | 1 | complete |
|
||||||
|
| REQ-2-003 | SSE streaming endpoint plumbing: chat completion streaming from provider through FastAPI to the Next.js client | critical | 1 | complete |
|
||||||
|
| REQ-2-004 | Agent framework: base agent contracts, session/state store, prompt management, streaming pipeline, structured output support | critical | 2 | complete |
|
||||||
|
|
||||||
|
### AI Tutor Agents
|
||||||
|
|
||||||
|
| ID | Description | Priority | Phase | Status |
|
||||||
|
|----|-------------|----------|-------|--------|
|
||||||
|
| REQ-2-005 | Coach agent (REQ-F-001): pacing, motivation, retrieval practice — full LLM implementation | critical | 3 | complete |
|
||||||
|
| REQ-2-006 | Tutor agent (REQ-F-002): concept delivery, Socratic questioning — full LLM implementation | critical | 3 | complete |
|
||||||
|
| REQ-2-007 | Lab agent (REQ-F-003): in-flow feedback over simulated sandbox telemetry (mock inputs) | high | 4 | complete |
|
||||||
|
| REQ-2-008 | Assessor agent (REQ-F-004): rubric application to pre-baked artifacts and defense transcripts (mock inputs) | high | 4 | complete |
|
||||||
|
| REQ-2-009 | Proctor agent (REQ-F-005): integrity signals from mock telemetry, coaching interventions | high | 5 | complete |
|
||||||
|
| REQ-2-010 | Mentor agent (REQ-F-006): long-horizon career narrative | high | 5 | complete |
|
||||||
|
|
||||||
|
### Learner Surface Integration
|
||||||
|
|
||||||
|
| ID | Description | Priority | Phase | Status |
|
||||||
|
|----|-------------|----------|-------|--------|
|
||||||
|
| REQ-2-011 | Learner chat UI wired to real service: streaming responses, agent routing, error and loading states | critical | 6 | complete |
|
||||||
|
| REQ-2-012 | Byte tutorial viewer, build sandbox, and assessment mockups surface Lab/Assessor/Proctor outputs (mock engine inputs) | high | 6 | complete |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## v0.1 Requirements (Complete)
|
||||||
|
|
||||||
### Infrastructure
|
### Infrastructure
|
||||||
|
|
||||||
| ID | Description | Priority | Phase | Status |
|
| ID | Description | Priority | Phase | Status |
|
||||||
|----|-------------|----------|-------|--------|
|
|----|-------------|----------|-------|--------|
|
||||||
| REQ-001 | Monorepo scaffolding: pnpm workspaces, turborepo, Next.js app, TypeScript config, ESLint, Prettier | critical | 1 | pending |
|
| REQ-001 | Monorepo scaffolding: pnpm workspaces, turborepo, Next.js app, TypeScript config, ESLint, Prettier | critical | 1 | complete |
|
||||||
| REQ-002 | Shared component library: design tokens, Button, Input, Card, Badge, Avatar, Dialog, Navigation, Table, Tabs, Progress, Tooltip, Skeleton, Toast | critical | 1 | pending |
|
| REQ-002 | Shared component library: design tokens, Button, Input, Card, Badge, Avatar, Dialog, Navigation, Table, Tabs, Progress, Tooltip, Skeleton, Toast | critical | 1 | pending |
|
||||||
| REQ-003 | Mock data layer: typed mock data for competency stacks, job listings, candidate profiles, employers, learner progress | critical | 1 | pending |
|
| REQ-003 | Mock data layer: typed mock data for competency stacks, job listings, candidate profiles, employers, learner progress | critical | 1 | pending |
|
||||||
| REQ-004 | Routing and navigation: App Router route groups for (learner), (marketplace), (employer), (admin); shared layout components; cross-surface navigation | critical | 1 | pending |
|
| REQ-004 | Routing and navigation: App Router route groups for (learner), (marketplace), (employer), (admin); shared layout components; cross-surface navigation | critical | 1 | pending |
|
||||||
@@ -64,17 +95,6 @@
|
|||||||
|
|
||||||
## v2 Requirements (Future Milestones — Deferred)
|
## v2 Requirements (Future Milestones — Deferred)
|
||||||
|
|
||||||
### AI Tutor Architecture
|
|
||||||
|
|
||||||
| ID | Description | Priority | Milestone | Status |
|
|
||||||
|----|-------------|----------|-----------|--------|
|
|
||||||
| REQ-F-001 | Coach agent: pacing, motivation, retrieval practice | high | v0.2+ | deferred |
|
|
||||||
| REQ-F-002 | Tutor agent: concept delivery, Socratic questioning | high | v0.2+ | deferred |
|
|
||||||
| REQ-F-003 | Lab agent: sandbox execution, in-flow feedback | high | v0.2+ | deferred |
|
|
||||||
| REQ-F-004 | Assessor agent: rubric application to artifacts and defenses | high | v0.2+ | deferred |
|
|
||||||
| REQ-F-005 | Proctor agent: identity, attention, integrity signals | high | v0.2+ | deferred |
|
|
||||||
| REQ-F-006 | Mentor agent: long-horizon career narrative | medium | v0.2+ | deferred |
|
|
||||||
|
|
||||||
### Assessment Engine
|
### Assessment Engine
|
||||||
|
|
||||||
| ID | Description | Priority | Milestone | Status |
|
| ID | Description | Priority | Milestone | Status |
|
||||||
@@ -83,6 +103,7 @@
|
|||||||
| REQ-F-008 | Per-learner variant task generation | high | v0.3+ | deferred |
|
| REQ-F-008 | Per-learner variant task generation | high | v0.3+ | deferred |
|
||||||
| REQ-F-009 | Oral/voice defense with AI examiner | high | v0.3+ | deferred |
|
| REQ-F-009 | Oral/voice defense with AI examiner | high | v0.3+ | deferred |
|
||||||
| REQ-F-010 | Live in-environment build with telemetry | high | v0.3+ | deferred |
|
| REQ-F-010 | Live in-environment build with telemetry | high | v0.3+ | deferred |
|
||||||
|
| REQ-F-021 | Sandbox fabric: sandboxed IDE, design tool, simulation | high | v0.3+ | deferred |
|
||||||
|
|
||||||
### Marketplace Engine
|
### Marketplace Engine
|
||||||
|
|
||||||
@@ -99,7 +120,7 @@
|
|||||||
|
|
||||||
| ID | Description | Priority | Milestone | Status |
|
| ID | Description | Priority | Milestone | Status |
|
||||||
|----|-------------|----------|-----------|--------|
|
|----|-------------|----------|-----------|--------|
|
||||||
| REQ-F-017 | Identity verification and age-gating (16+/18+) | high | v0.2+ | deferred |
|
| REQ-F-017 | Identity verification and age-gating (16+/18+) — real KYC backend | high | v0.3+ | deferred |
|
||||||
| REQ-F-018 | Payment processing and subscription management | high | v0.3+ | deferred |
|
| REQ-F-018 | Payment processing and subscription management | high | v0.3+ | deferred |
|
||||||
| REQ-F-019 | Human tutor marketplace (third-party courses) | medium | v0.4+ | deferred |
|
| REQ-F-019 | Human tutor marketplace (third-party courses) | medium | v0.4+ | deferred |
|
||||||
| REQ-F-020 | CIRR-style placement tracking and audit | medium | v0.5+ | deferred |
|
| REQ-F-020 | CIRR-style placement tracking and audit | medium | v0.5+ | deferred |
|
||||||
@@ -122,6 +143,25 @@
|
|||||||
|
|
||||||
## Traceability Matrix
|
## Traceability Matrix
|
||||||
|
|
||||||
|
### v0.2 (current milestone)
|
||||||
|
|
||||||
|
| Requirement | Phase | Status |
|
||||||
|
|-------------|-------|--------|
|
||||||
|
| REQ-2-001 | 1 | complete |
|
||||||
|
| REQ-2-002 | complete |
|
||||||
|
| REQ-2-003 | 1 | complete |
|
||||||
|
| REQ-2-004 | complete |
|
||||||
|
| REQ-2-005 | complete |
|
||||||
|
| REQ-2-006 | complete |
|
||||||
|
| REQ-2-007 | complete |
|
||||||
|
| REQ-2-008 | complete |
|
||||||
|
| REQ-2-009 | complete |
|
||||||
|
| REQ-2-010 | complete |
|
||||||
|
| REQ-2-011 | complete |
|
||||||
|
| REQ-2-012 | complete |
|
||||||
|
|
||||||
|
### v0.1 (complete)
|
||||||
|
|
||||||
| Requirement | Phase | Status |
|
| Requirement | Phase | Status |
|
||||||
|-------------|-------|--------|
|
|-------------|-------|--------|
|
||||||
| REQ-001 | 1 | complete |
|
| REQ-001 | 1 | complete |
|
||||||
|
|||||||
+89
-101
@@ -2,11 +2,13 @@
|
|||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
**Milestone v0.1** — UI/UX Prototype: High-fidelity interactive prototype of all four Nextcraft surfaces (Learner, Marketplace, Employer Dashboard, Admin) with realistic mock data, shared component library, responsive layouts, and navigation flows. No backend, no business logic.
|
**Milestone v0.2** — AI Tutor Architecture: The six AI tutor agents (Coach, Tutor, Lab, Assessor, Proctor, Mentor) as real LLM-backed services in a new `apps/ai-service` Python FastAPI application, wired into the existing v0.1 learner surface with streaming responses. Provider-agnostic LLM layer (ollama-cloud default). Lab/Assessor/Proctor operate on mock engine inputs — their real engines are v0.3+.
|
||||||
|
|
||||||
**Milestone type:** Feature (includes new UI features)
|
**Prior milestone:** v0.1 (nextcraft-ui-prototype) — complete, shipped as v0.1.0, founder-agreed (D-013).
|
||||||
**Tag line:** v0.0.x (patches on the v0.0 line; milestone release as v0.1.0)
|
|
||||||
**Branch:** milestone/v0.1-nextcraft-ui-prototype
|
**Milestone type:** Feature (new AI service + real agent capabilities)
|
||||||
|
**Tag line:** v0.1.x (patches on the v0.1 line; milestone release as v0.2.0)
|
||||||
|
**Branch:** milestone/v0.2-ai-tutor-architecture
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -14,14 +16,14 @@
|
|||||||
|
|
||||||
| # | Name | Status | Depends On | Requirements | Success Criteria |
|
| # | Name | Status | Depends On | Requirements | Success Criteria |
|
||||||
|---|------|--------|------------|--------------|------------------|
|
|---|------|--------|------------|--------------|------------------|
|
||||||
| 0 | Pre-execution | complete | — | — | Specification, clarify, research, plan complete; .ciagent/ files created |
|
| 0 | Pre-execution | complete | — | — | Specification, clarify, research, plan complete; .ciagent/ files updated for v0.2 |
|
||||||
| 1 | Project scaffolding | complete | 0 | REQ-001, REQ-002, REQ-003, REQ-004, REQ-005 | Monorepo builds; dev server starts; component library has all primitives; mock data typed; routing works between route groups |
|
| 1 | AI service scaffolding | complete | 0 | REQ-2-001, REQ-2-002, REQ-2-003 | apps/ai-service runs (uvicorn), health endpoint responds, provider-agnostic LLM client with 3 providers (ollama-cloud/local/mock), SSE streaming verified, pytest suite passes with mock provider, turbo scripts wired |
|
||||||
| 2 | Learner surface UI | complete | 1 | REQ-006, REQ-007, REQ-008, REQ-009, REQ-010, REQ-011, REQ-012 | All 7 learner pages render with mock data; navigation between pages works; responsive at mobile/tablet/desktop |
|
| 2 | Agent framework | complete | 1 | REQ-2-004 | Base agent contract, session/state store, prompt templates, streaming pipeline, structured outputs; all tested |
|
||||||
| 3 | Marketplace surface UI | complete | 1 | REQ-013, REQ-014, REQ-015, REQ-016, REQ-017 | All 5 marketplace pages render with mock data; search/filter UI interactive (client-side); job board listing displays mock jobs |
|
| 3 | Coach + Tutor agents | complete | 2 | REQ-2-005, REQ-2-006 | Coach (pacing/motivation/retrieval practice) and Tutor (concept delivery/Socratic questioning) fully implemented with system prompts, tested against mock provider, wired to chat endpoint |
|
||||||
| 4 | Employer dashboard UI | complete | 1 | REQ-018, REQ-019, REQ-020, REQ-021 | All 4 employer pages render with mock data; talent search displays mock candidates; candidate profile shows artifact gallery + process trace |
|
| 4 | Lab + Assessor agents | complete | 2 | REQ-2-007, REQ-2-008 | Lab consumes simulated sandbox telemetry (mock); Assessor applies rubrics to pre-baked artifacts/defense transcripts (mock); both tested |
|
||||||
| 5 | Admin surface UI | complete | 1 | REQ-022, REQ-023, REQ-024, REQ-025 | All 4 admin pages render with mock data; competency graph viewer renders interactive graph; moderation queue displays mock flagged content |
|
| 5 | Proctor + Mentor agents | complete | 2 | REQ-2-009, REQ-2-010 | Proctor produces integrity signals + coaching interventions from mock telemetry; Mentor generates long-horizon career narrative; both tested |
|
||||||
| 6 | Polish + integration | complete | 2, 3, 4, 5 | REQ-026, REQ-027, REQ-028 | Cross-surface navigation works; visual consistency audit passes; dark mode toggle functional; Storybook documents all components |
|
| 6 | Learner surface integration | complete | 3, 4, 5 | REQ-2-011, REQ-2-012 | Learner chat streams real responses; agent routing works; byte viewer/sandbox/assessment mockups surface agent outputs; error/loading states; pnpm build + typecheck pass |
|
||||||
| 7 | Final review + ship | complete | 6 | — | Code review clean; audit passes; milestone tagged v0.1.0; release created on Gitea |
|
| 7 | Final review + ship | complete | 6 | — | Code review clean; audit passes; milestone tagged v0.2.0; release created on Gitea |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -29,150 +31,130 @@
|
|||||||
|
|
||||||
### Phase 0: Pre-execution
|
### Phase 0: Pre-execution
|
||||||
|
|
||||||
**Goal:** Establish project specification, clarify ambiguities, research tech stack, create detailed plans.
|
**Goal:** Establish v0.2 specification, clarify ambiguities, research AI service architecture, create detailed plans.
|
||||||
|
|
||||||
**Stages:** SPECIFY → CLARIFY → RESEARCH → PLAN → (GRILL optional) → SHIP
|
**Stages:** SPECIFY → CLARIFY → RESEARCH → PLAN → GRILL → SHIP
|
||||||
|
|
||||||
**Deliverables:**
|
**Deliverables:**
|
||||||
- .ciagent/config.json
|
- Updated .ciagent/config.json, PROJECT.md, REQUIREMENTS.md, ROADMAP.md, ARCHITECTURE.md, PERSONAS.md, PLAN.md
|
||||||
- .ciagent/PROJECT.md
|
|
||||||
- .ciagent/REQUIREMENTS.md
|
|
||||||
- .ciagent/ARCHITECTURE.md
|
|
||||||
- .ciagent/ROADMAP.md
|
|
||||||
- .ciagent/CHECKPOINT.json
|
|
||||||
|
|
||||||
**Success criteria:** All .ciagent/ files created; initial commit with ---ci--- block; phase 0 shipped as v0.0.1.
|
**Success criteria:** All .ciagent/ files updated for v0.2; phase 0 shipped as v0.1.1.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Phase 1: Project Scaffolding
|
### Phase 1: AI Service Scaffolding
|
||||||
|
|
||||||
**Goal:** Set up the monorepo, component library, mock data layer, routing, and responsive system.
|
**Goal:** Stand up apps/ai-service with the provider-agnostic LLM layer and SSE streaming.
|
||||||
|
|
||||||
**Requirements:** REQ-001, REQ-002, REQ-003, REQ-004, REQ-005
|
**Requirements:** REQ-2-001, REQ-2-002, REQ-2-003
|
||||||
|
|
||||||
**Key deliverables:**
|
**Key deliverables:**
|
||||||
- pnpm-workspace.yaml + turbo.json + root package.json
|
- apps/ai-service: FastAPI app, pydantic-settings, uvicorn, /health, CORS for localhost
|
||||||
- apps/web Next.js app with App Router
|
- llm package: provider interface + ollama-cloud/local/mock providers; key resolution from .ciagent/.env.secrets via env
|
||||||
- packages/ui with design tokens + all primitives (Button, Input, Card, Badge, Avatar, Dialog, Tabs, Progress, Tooltip, Skeleton, Toast)
|
- SSE streaming: /v1/chat/stream endpoint streaming provider deltas
|
||||||
- packages/mock-data with typed mock data for all domains
|
- pytest suite with mock provider; root scripts: ai:dev, ai:test; turbo integration
|
||||||
- packages/types with shared TypeScript type definitions
|
|
||||||
- Route groups: (learner), (marketplace), (employer), (admin)
|
|
||||||
- Responsive layout system with breakpoints
|
|
||||||
- Root layout with theme provider
|
|
||||||
|
|
||||||
**Success criteria:**
|
**Success criteria:**
|
||||||
- `pnpm dev` starts the Next.js dev server
|
- `python -m uvicorn` starts the service; /health returns 200
|
||||||
- `pnpm build` succeeds without errors
|
- Provider unit tests pass (mock); ollama-cloud integration probe works (manual)
|
||||||
- `pnpm typecheck` passes
|
- SSE stream delivers tokens to an HTTP client
|
||||||
- All primitive components exist and are importable
|
|
||||||
- Mock data is typed and importable
|
|
||||||
- Navigation between route groups works (even if pages are placeholders)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Phase 2: Learner Surface UI
|
### Phase 2: Agent Framework
|
||||||
|
|
||||||
**Goal:** Build all 7 learner surface pages with realistic mock data and interactive elements.
|
**Goal:** Build the shared framework all six agents use.
|
||||||
|
|
||||||
**Requirements:** REQ-006, REQ-007, REQ-008, REQ-009, REQ-010, REQ-011, REQ-012
|
**Requirements:** REQ-2-004
|
||||||
|
|
||||||
**Key deliverables:**
|
**Key deliverables:**
|
||||||
- Landing page: hero, value proposition, Byte→Build→Demonstrate→Defend flow, testimonials, CTA
|
- BaseAgent contract: system prompt, message history, streaming completion, structured output
|
||||||
- Program catalog: 5 competency stack cards with role descriptions
|
- Session/state store: in-memory per-learner session with message history
|
||||||
- Competency stack view: selected stack with 12-18 competencies, progress indicators, microcredential badges
|
- Prompt management: per-agent system prompt templates with learner context injection
|
||||||
- Learner dashboard: active competencies, progress graph, recent artifacts, AI tutor chat mockup, milestones
|
- Streaming pipeline: agent → provider → SSE with agent identification
|
||||||
- Byte tutorial viewer: concept panel, worked example, code/design viewer mockup
|
- Structured outputs: JSON-schema outputs for Assessor rubric scores, Proctor signals
|
||||||
- Build sandbox mockup: IDE UI with toolbar, file explorer, editor area, telemetry sidebar
|
|
||||||
- Assessment/defense mockup: rubric display, AI reviewer panel, oral defense interface, process trace timeline
|
|
||||||
|
|
||||||
**Success criteria:**
|
**Success criteria:**
|
||||||
- All 7 pages render with mock data
|
- BaseAgent unit tests pass
|
||||||
- Navigation between pages works
|
- Session store tested (create/append/persist in-memory)
|
||||||
- AI tutor chat mockup displays pre-scripted responses
|
- Structured output parsing tested against mock provider
|
||||||
- Responsive at mobile (375px), tablet (768px), desktop (1280px)
|
|
||||||
- Hover states and interactive elements functional
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Phase 3: Marketplace Surface UI
|
### Phase 3: Coach + Tutor Agents
|
||||||
|
|
||||||
**Goal:** Build all 5 marketplace surface pages with job listings, search, and pricing.
|
**Goal:** Implement the two learner-facing conversational agents.
|
||||||
|
|
||||||
**Requirements:** REQ-013, REQ-014, REQ-015, REQ-016, REQ-017
|
**Requirements:** REQ-2-005, REQ-2-006
|
||||||
|
|
||||||
**Key deliverables:**
|
**Key deliverables:**
|
||||||
- Job board listing: searchable grid of mock AI-era jobs, filter sidebar, match score cards
|
- Coach agent: pacing guidance, motivation, retrieval practice prompts; distinct persona
|
||||||
- Job detail page: full description, required competencies, employer info, AI-matched skills
|
- Tutor agent: concept delivery, Socratic questioning, worked examples
|
||||||
- Employer profile: company overview, logo, open positions, culture mockup
|
- Agent registry: route chat messages to the correct agent by context/selection
|
||||||
- Search/filter UI: semantic search bar, skill tags, filters (seniority, remote, salary), saved searches
|
- Per-agent system prompts with competency-stack context injection from packages/mock-data
|
||||||
- Pricing page: job posting packages, talent access plans, feature comparison table
|
|
||||||
|
|
||||||
**Success criteria:**
|
**Success criteria:**
|
||||||
- All 5 pages render with mock data
|
- Both agents produce distinct, on-persona responses (verified against mock + ollama-cloud)
|
||||||
- Filter sidebar interactive (client-side filtering of mock jobs)
|
- Agent routing tested
|
||||||
- Job cards display match scores and skill tags
|
- Both agents exposed via the chat streaming endpoint
|
||||||
- Pricing table is responsive and readable
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Phase 4: Employer Dashboard UI
|
### Phase 4: Lab + Assessor Agents
|
||||||
|
|
||||||
**Goal:** Build all 4 employer dashboard pages for talent search and posting management.
|
**Goal:** Implement the two build/assessment agents over mock engine inputs.
|
||||||
|
|
||||||
**Requirements:** REQ-018, REQ-019, REQ-020, REQ-021
|
**Requirements:** REQ-2-007, REQ-2-008
|
||||||
|
|
||||||
**Key deliverables:**
|
**Key deliverables:**
|
||||||
- Dashboard overview: active postings, applicant pipeline, talent matches, analytics charts
|
- Lab agent: consumes simulated sandbox telemetry (mock event streams), produces in-flow feedback
|
||||||
- Talent search: searchable candidate database with AI-matched filters, candidate cards
|
- Assessor agent: applies rubrics to pre-baked artifacts and defense transcripts, returns structured scores + feedback
|
||||||
- Candidate profile: artifact gallery, process trace summary, defense transcripts, competency graph, microcredentials
|
- Mock engine inputs: simulated telemetry generator, pre-baked artifact corpus in packages/mock-data
|
||||||
- Posting management: create/edit/delete job postings, status tracking, applicant list, interview pipeline
|
- Endpoints: /v1/lab/feedback, /v1/assessment/evaluate
|
||||||
|
|
||||||
**Success criteria:**
|
**Success criteria:**
|
||||||
- All 4 pages render with mock data
|
- Lab produces relevant feedback for mock telemetry scenarios
|
||||||
- Analytics charts display mock metrics (bar/line/donut charts)
|
- Assessor returns structured rubric scores (JSON) for pre-baked artifacts
|
||||||
- Candidate cards show competency stacks, microcredentials, defense scores
|
- Both tested against mock provider
|
||||||
- Posting management form inputs are interactive (non-functional submit)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Phase 5: Admin Surface UI
|
### Phase 5: Proctor + Mentor Agents
|
||||||
|
|
||||||
**Goal:** Build all 4 admin surface pages including the interactive competency graph viewer.
|
**Goal:** Implement the integrity and narrative agents.
|
||||||
|
|
||||||
**Requirements:** REQ-022, REQ-023, REQ-024, REQ-025
|
**Requirements:** REQ-2-009, REQ-2-010
|
||||||
|
|
||||||
**Key deliverables:**
|
**Key deliverables:**
|
||||||
- Admin overview: platform metrics (learners, employers, placements, completion rate, NPS), activity feed, system health
|
- Proctor agent: integrity signals from mock telemetry (tab switches, idle time, paste events), coaching interventions
|
||||||
- Learner management: searchable learner table, detail view, progress tracking, credential log
|
- Mentor agent: long-horizon career narrative, competency-stack progression guidance
|
||||||
- Competency graph viewer: interactive node/edge graph using react-flow, competency stack nodes, dependency edges
|
- Endpoints: /v1/proctor/signals, /v1/mentor/narrative
|
||||||
- Marketplace moderation: job posting review queue, employer verification queue, flagged content, moderation tools
|
|
||||||
|
|
||||||
**Success criteria:**
|
**Success criteria:**
|
||||||
- All 4 pages render with mock data
|
- Proctor produces classified signals with recommended interventions for mock scenarios
|
||||||
- Competency graph viewer renders an interactive graph with clickable nodes
|
- Mentor produces coherent career-narrative responses
|
||||||
- Admin table supports sorting and filtering (client-side, mock data)
|
- Both tested against mock provider
|
||||||
- Moderation queue displays mock flagged items with approve/reject buttons (non-functional)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Phase 6: Polish + Integration
|
### Phase 6: Learner Surface Integration
|
||||||
|
|
||||||
**Goal:** Ensure cross-surface consistency, responsive quality, and component library documentation.
|
**Goal:** Wire the v0.1 learner surface to the real AI service.
|
||||||
|
|
||||||
**Requirements:** REQ-026, REQ-027, REQ-028
|
**Requirements:** REQ-2-011, REQ-2-012
|
||||||
|
|
||||||
**Key deliverables:**
|
**Key deliverables:**
|
||||||
- Cross-surface navigation: role switcher (learner/employer/admin), breadcrumbs, consistent header/footer
|
- Learner dashboard chat: real streaming via SSE, agent switcher (Coach/Tutor), error/loading states
|
||||||
- Visual consistency: typography scale, color palette, spacing system, dark mode toggle, WCAG AA contrast
|
- Byte tutorial viewer: Tutor concept explanations
|
||||||
- Storybook: component documentation, prop tables, usage examples for all primitives and composites
|
- Build sandbox: Lab feedback panel fed by mock telemetry + Lab agent
|
||||||
|
- Assessment mockup: Assessor rubric output display, Proctor integrity banner
|
||||||
|
- Mentor panel on learner dashboard
|
||||||
|
|
||||||
**Success criteria:**
|
**Success criteria:**
|
||||||
- Role switcher navigates between surfaces
|
- Streaming chat works end-to-end with ai-service running
|
||||||
- Dark mode toggle works across all surfaces
|
- All four learner surfaces surface agent outputs
|
||||||
- All pages pass WCAG AA contrast checks
|
- Graceful degradation when ai-service is down (error states, not crashes)
|
||||||
- Storybook runs and documents all components
|
- `pnpm build` and `pnpm typecheck` pass
|
||||||
- No visual inconsistencies between surfaces
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -183,10 +165,16 @@
|
|||||||
**Key deliverables:**
|
**Key deliverables:**
|
||||||
- Multi-persona code review (correctness, testing, security, performance, maintainability)
|
- Multi-persona code review (correctness, testing, security, performance, maintainability)
|
||||||
- Project health audit (reconstruction test, .ciagent/ file discipline, branch hygiene, commit discipline)
|
- Project health audit (reconstruction test, .ciagent/ file discipline, branch hygiene, commit discipline)
|
||||||
- Milestone ship: merge milestone → main, tag v0.1.0, create Gitea release
|
- Milestone ship: merge milestone → main, tag v0.2.0, create Gitea release
|
||||||
|
|
||||||
**Success criteria:**
|
**Success criteria:**
|
||||||
- Code review: P0 fixes applied, P1+ documented
|
- Code review: P0 fixes applied, P1+ documented
|
||||||
- Audit: all checks pass, project state reconstructable from git log
|
- Audit: all checks pass, project state reconstructable from git log
|
||||||
- Ship: v0.1.0 tagged, milestone branch merged to main, Gitea release created
|
- Ship: v0.2.0 tagged, milestone branch merged to main, Gitea release created — **release note explicitly states Lab/Assessor/Proctor operate on mock engine inputs (real engines v0.3+)** (G-5); dead `aiTutorResponses` export disposed of (G-5)
|
||||||
- All 28 requirements marked complete
|
- All 12 v0.2 requirements marked complete
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## v0.1 (Complete — Shipped as v0.1.0)
|
||||||
|
|
||||||
|
UI/UX Prototype: High-fidelity interactive prototype of all four Nextcraft surfaces. 7 phases (P0 + P1-P6 execution + P7 final). All 28 requirements complete. Tags v0.0.1–v0.0.7, milestone release v0.1.0.
|
||||||
@@ -46,9 +46,9 @@
|
|||||||
"projects": [],
|
"projects": [],
|
||||||
"active_project": null,
|
"active_project": null,
|
||||||
"milestone": {
|
"milestone": {
|
||||||
"version": "v0.1",
|
"version": "v0.2",
|
||||||
"name": "nextcraft-ui-prototype",
|
"name": "ai-tutor-architecture",
|
||||||
"type": "feature",
|
"type": "feature",
|
||||||
"branch": "milestone/v0.1-nextcraft-ui-prototype"
|
"branch": "milestone/v0.2-ai-tutor-architecture"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -43,3 +43,8 @@ yarn-error.log*
|
|||||||
# Test coverage
|
# Test coverage
|
||||||
coverage/
|
coverage/
|
||||||
.nyc_output/
|
.nyc_output/
|
||||||
|
|
||||||
|
# Python tooling caches
|
||||||
|
.pytest_cache/
|
||||||
|
.ruff_cache/
|
||||||
|
*.egg-info/
|
||||||
|
|||||||
@@ -0,0 +1,10 @@
|
|||||||
|
# Nextcraft AI Service — environment template (copy values, never commit real keys)
|
||||||
|
# Real keys live in .ciagent/.env.secrets (gitignored) and are exported by scripts/dev.sh
|
||||||
|
|
||||||
|
AI_PORT=8420
|
||||||
|
AI_PROVIDER=ollama-cloud
|
||||||
|
AI_MODEL=gemma4:31b
|
||||||
|
AI_OLLAMA_CLOUD_BASE_URL=https://ollama.com/v1
|
||||||
|
AI_OLLAMA_CLOUD_API_KEY=
|
||||||
|
AI_LOCAL_BASE_URL=http://localhost:11434/v1
|
||||||
|
AI_JSON_MODE=auto
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
# Nextcraft AI Service (`apps/ai-service`)
|
||||||
|
|
||||||
|
Python FastAPI service hosting the six AI tutor agents (Coach, Tutor, Lab, Assessor, Proctor, Mentor) behind a provider-agnostic LLM layer. Port **8420**.
|
||||||
|
|
||||||
|
## Quickstart
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Bootstrap (idempotent): venv + deps
|
||||||
|
bash scripts/bootstrap.sh
|
||||||
|
|
||||||
|
# 2. Run tests (mock provider only — zero network calls)
|
||||||
|
bash scripts/test.sh
|
||||||
|
|
||||||
|
# 3. Lint
|
||||||
|
bash scripts/lint.sh
|
||||||
|
|
||||||
|
# 4. Dev server (exports keys from .ciagent/.env.secrets if present)
|
||||||
|
bash scripts/dev.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Or via the monorepo root (`corepack pnpm install` first):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm ai:bootstrap
|
||||||
|
pnpm ai:test
|
||||||
|
pnpm ai:lint
|
||||||
|
pnpm ai:dev
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
All settings use the `AI_` env prefix (pydantic-settings; see `.env.example`).
|
||||||
|
|
||||||
|
| Var | Default | Purpose |
|
||||||
|
|-----|---------|---------|
|
||||||
|
| `AI_PORT` | 8420 | Listen port |
|
||||||
|
| `AI_PROVIDER` | mock | `ollama-cloud` \| `local` \| `mock` |
|
||||||
|
| `AI_MODEL` | gemma4:31b | Model for all agents |
|
||||||
|
| `AI_OLLAMA_CLOUD_BASE_URL` | https://ollama.com/v1 | Cloud base URL |
|
||||||
|
| `AI_OLLAMA_CLOUD_API_KEY` | (empty) | Bearer key — **never commit** |
|
||||||
|
| `AI_JSON_MODE` | auto | `auto` sends response_format, degrades on 400; `off` never sends |
|
||||||
|
|
||||||
|
Tests run with `AI_PROVIDER=mock` (enforced in `tests/conftest.py` by an instance assertion) — the suite never calls the cloud.
|
||||||
|
|
||||||
|
## Endpoints
|
||||||
|
|
||||||
|
- `GET /health` — status, configured provider, model (no cloud call)
|
||||||
|
- `POST /v1/chat/stream` — SSE chat stream. Body: `{"agent": "coach"|"tutor", "session_id": "...", "messages": [{"role":"user","content":"..."}]}`. Unknown agents are rejected with 422.
|
||||||
|
|
||||||
|
SSE envelope (D-016): `meta` event first (agent/session/model), then `delta` events (incremental content), then `done`; on mid-stream failure an `error` event precedes the terminal `[DONE]` sentinel. sse-starlette emits `: ping` keep-alive comment lines on idle connections — clients must ignore frames without `data:`.
|
||||||
|
|
||||||
|
## Manual ollama-cloud persona probe (Phase 3, documented — not automated)
|
||||||
|
|
||||||
|
With the real provider, Coach and Tutor must produce distinct on-persona
|
||||||
|
responses to the same prompt:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# start with the cloud provider (keys exported from .ciagent/.env.secrets)
|
||||||
|
AI_PROVIDER=ollama-cloud .venv/bin/uvicorn ai_service.main:app --port 8420
|
||||||
|
|
||||||
|
# Coach: expect pacing + one concrete next action + a retrieval-practice question
|
||||||
|
curl -sN -X POST localhost:8420/v1/chat/stream -H 'Content-Type: application/json' \
|
||||||
|
-d '{"agent":"coach","session_id":"probe-coach","messages":[{"role":"user","content":"I am stuck on multi-agent communication patterns"}]}' \
|
||||||
|
| grep '^data:'
|
||||||
|
|
||||||
|
# Tutor: expect ONE concept + a worked example + a Socratic check question
|
||||||
|
curl -sN -X POST localhost:8420/v1/chat/stream -H 'Content-Type: application/json' \
|
||||||
|
-d '{"agent":"tutor","session_id":"probe-tutor","messages":[{"role":"user","content":"I am stuck on multi-agent communication patterns"}]}' \
|
||||||
|
| grep '^data:'
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify: the two responses have visibly different voice/structure (Coach:
|
||||||
|
action + accountability; Tutor: concept + example + question). The
|
||||||
|
automated suite never calls the cloud — distinctness is enforced against
|
||||||
|
the deterministic mock (distinct system prompts → distinct hash-seeded
|
||||||
|
outputs).
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
ai_service/
|
||||||
|
main.py app factory, lifespan (httpx pool), CORS, /health
|
||||||
|
config.py pydantic-settings
|
||||||
|
api/ endpoints (SSE envelope lives here, D-016)
|
||||||
|
llm/ provider layer — dumb pipe, no envelope logic
|
||||||
|
scripts/ bootstrap.sh dev.sh test.sh lint.sh
|
||||||
|
tests/ pytest — mock provider only
|
||||||
|
```
|
||||||
|
|
||||||
|
Boundary rules: `llm/` imports nothing from `agents/` or `api/`; `agents/` imports nothing from `api/`.
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
"""Nextcraft AI tutor service — six LLM agents behind a provider-agnostic layer."""
|
||||||
|
|
||||||
|
__version__ = "0.2.0"
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
"""Agent framework — BaseAgent ABC (D-018), registry, sessions, structured outputs.
|
||||||
|
|
||||||
|
Boundary rule: agents/ imports from llm/, prompts/, corpus/ — never from api/.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .base import BaseAgent
|
||||||
|
from .registry import AgentRegistry
|
||||||
|
from .session import InMemorySessionStore, SessionStore
|
||||||
|
from .structured import StructuredOutputError, extract_json_object
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"AgentRegistry",
|
||||||
|
"BaseAgent",
|
||||||
|
"InMemorySessionStore",
|
||||||
|
"SessionStore",
|
||||||
|
"StructuredOutputError",
|
||||||
|
"extract_json_object",
|
||||||
|
]
|
||||||
@@ -0,0 +1,92 @@
|
|||||||
|
"""AssessorAgent — rubric application to pre-baked artifacts (REQ-2-008).
|
||||||
|
|
||||||
|
Structured-output showcase: applies the 4-layer defense (D-020) to return
|
||||||
|
a pydantic-validated rubric score. Mock engine inputs (corpus artifacts +
|
||||||
|
transcripts); real process-trace grading is v0.3+.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
from ..corpus.artifacts import (
|
||||||
|
ArtifactSubmission,
|
||||||
|
AssessmentRubric,
|
||||||
|
DefenseTranscript,
|
||||||
|
render_rubric,
|
||||||
|
render_transcript,
|
||||||
|
)
|
||||||
|
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||||
|
from ..prompts.assessor import SYSTEM_PROMPT, render_context
|
||||||
|
from .base import BaseAgent
|
||||||
|
|
||||||
|
|
||||||
|
class CriterionScore(BaseModel):
|
||||||
|
criterion_id: str
|
||||||
|
name: str
|
||||||
|
score: int = Field(ge=0, le=100)
|
||||||
|
evidence: str
|
||||||
|
|
||||||
|
|
||||||
|
class RubricScore(BaseModel):
|
||||||
|
rubric_id: str
|
||||||
|
artifact_id: str
|
||||||
|
competency_id: str
|
||||||
|
scores: list[CriterionScore]
|
||||||
|
strengths: list[str] = Field(min_length=1, max_length=2)
|
||||||
|
gaps: list[str] = Field(min_length=1, max_length=2)
|
||||||
|
verdict: str # "mastered" | "developing" | "not_yet"
|
||||||
|
|
||||||
|
def weighted_total(self, rubric: AssessmentRubric) -> float:
|
||||||
|
by_id = {c.criterion_id: c for c in rubric.criteria}
|
||||||
|
total = 0.0
|
||||||
|
for s in self.scores:
|
||||||
|
total += s.score * by_id[s.criterion_id].weight
|
||||||
|
return total
|
||||||
|
|
||||||
|
|
||||||
|
RUBRIC_SCORE_SCHEMA_HINT = (
|
||||||
|
'{"rubric_id": "<id>", "artifact_id": "<id>", "competency_id": "<id>", '
|
||||||
|
'"scores": [{"criterion_id": "<id>", "name": "<name>", "score": <0-100>, '
|
||||||
|
'"evidence": "<one sentence>"}], "strengths": ["<one sentence>"], '
|
||||||
|
'"gaps": ["<one sentence>"], "verdict": "mastered"|"developing"|"not_yet"}'
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class AssessorAgent(BaseAgent):
|
||||||
|
name = "assessor"
|
||||||
|
|
||||||
|
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||||
|
ctx = learner_context or get_learner_context()
|
||||||
|
return SYSTEM_PROMPT.format_map(render_context(ctx))
|
||||||
|
|
||||||
|
def build_evaluation_input(
|
||||||
|
self,
|
||||||
|
artifact: ArtifactSubmission,
|
||||||
|
rubric: AssessmentRubric,
|
||||||
|
transcript: DefenseTranscript | None,
|
||||||
|
) -> str:
|
||||||
|
parts = [
|
||||||
|
f"ARTIFACT: {artifact.name} ({artifact.artifact_id})",
|
||||||
|
f"Evidence excerpt: {artifact.evidence_excerpt}",
|
||||||
|
"",
|
||||||
|
render_rubric(rubric),
|
||||||
|
]
|
||||||
|
if transcript is not None:
|
||||||
|
parts += ["", render_transcript(transcript)]
|
||||||
|
return "\n".join(parts)
|
||||||
|
|
||||||
|
async def evaluate(
|
||||||
|
self,
|
||||||
|
artifact: ArtifactSubmission,
|
||||||
|
rubric: AssessmentRubric,
|
||||||
|
transcript: DefenseTranscript | None,
|
||||||
|
learner_context: LearnerContext | None = None,
|
||||||
|
) -> RubricScore:
|
||||||
|
evaluation_input = self.build_evaluation_input(artifact, rubric, transcript)
|
||||||
|
score: RubricScore = await self.structured_reply(
|
||||||
|
history=None,
|
||||||
|
user_input=evaluation_input,
|
||||||
|
learner_context=learner_context,
|
||||||
|
schema=RubricScore,
|
||||||
|
schema_hint=RUBRIC_SCORE_SCHEMA_HINT,
|
||||||
|
)
|
||||||
|
return score
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
"""BaseAgent ABC — the contract all six tutor agents implement (D-018).
|
||||||
|
|
||||||
|
Subclasses set `name`, override `system_prompt()`, and rarely `stream_reply()`.
|
||||||
|
The default pipeline: build_messages() → provider.stream_chat()/chat().
|
||||||
|
"""
|
||||||
|
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
|
||||||
|
from pydantic import BaseModel
|
||||||
|
|
||||||
|
from ..config import Settings
|
||||||
|
from ..corpus.learner_context import LearnerContext
|
||||||
|
from ..llm.base import LLMProvider
|
||||||
|
from ..llm.types import Message
|
||||||
|
from .structured import structured_completion
|
||||||
|
|
||||||
|
|
||||||
|
class BaseAgent(ABC):
|
||||||
|
"""A tutor agent: system prompt + message assembly + provider delegation."""
|
||||||
|
|
||||||
|
name: str = "base"
|
||||||
|
|
||||||
|
def __init__(self, provider: LLMProvider, settings: Settings) -> None:
|
||||||
|
self.provider = provider
|
||||||
|
self.settings = settings
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||||
|
"""Return the agent's system prompt, learner-context-aware."""
|
||||||
|
|
||||||
|
def build_messages(
|
||||||
|
self,
|
||||||
|
history: list[Message] | None = None,
|
||||||
|
user_input: str = "",
|
||||||
|
learner_context: LearnerContext | None = None,
|
||||||
|
) -> list[Message]:
|
||||||
|
"""Compose the full message list: system prompt + history + user turn."""
|
||||||
|
messages: list[Message] = [
|
||||||
|
Message(role="system", content=self.system_prompt(learner_context))
|
||||||
|
]
|
||||||
|
for m in history or []:
|
||||||
|
messages.append(m)
|
||||||
|
if user_input:
|
||||||
|
messages.append(Message(role="user", content=user_input))
|
||||||
|
return messages
|
||||||
|
|
||||||
|
async def stream_reply(
|
||||||
|
self,
|
||||||
|
history: list[Message] | None = None,
|
||||||
|
user_input: str = "",
|
||||||
|
learner_context: LearnerContext | None = None,
|
||||||
|
response_format: dict | None = None,
|
||||||
|
) -> AsyncIterator[str]:
|
||||||
|
"""Stream incremental content deltas for a conversational reply."""
|
||||||
|
messages = self.build_messages(history, user_input, learner_context)
|
||||||
|
async for token in self.provider.stream_chat(
|
||||||
|
messages, model=self.settings.model, response_format=response_format
|
||||||
|
):
|
||||||
|
yield token
|
||||||
|
|
||||||
|
async def structured_reply(
|
||||||
|
self,
|
||||||
|
history: list[Message] | None = None,
|
||||||
|
user_input: str = "",
|
||||||
|
learner_context: LearnerContext | None = None,
|
||||||
|
schema: type[BaseModel] | None = None,
|
||||||
|
schema_hint: str = "",
|
||||||
|
) -> BaseModel:
|
||||||
|
"""Non-streaming completion parsed into a pydantic model (D-020 defense)."""
|
||||||
|
if schema is None:
|
||||||
|
raise ValueError("structured_reply requires a schema")
|
||||||
|
messages = self.build_messages(history, user_input, learner_context)
|
||||||
|
return await structured_completion(
|
||||||
|
self.provider, messages, model=self.settings.model,
|
||||||
|
schema=schema, schema_hint=schema_hint,
|
||||||
|
)
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
"""CoachAgent — pacing, motivation, retrieval practice (REQ-2-005)."""
|
||||||
|
|
||||||
|
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||||
|
from ..prompts.coach import SYSTEM_PROMPT, render_context
|
||||||
|
from .base import BaseAgent
|
||||||
|
|
||||||
|
|
||||||
|
class CoachAgent(BaseAgent):
|
||||||
|
name = "coach"
|
||||||
|
|
||||||
|
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||||
|
ctx = learner_context or get_learner_context()
|
||||||
|
return SYSTEM_PROMPT.format_map(render_context(ctx))
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
"""LabAgent — in-flow feedback over simulated sandbox telemetry (REQ-2-007).
|
||||||
|
|
||||||
|
Scenario-driven: consumes a LabTelemetryScenario from the corpus, renders
|
||||||
|
the event timeline into the conversation, streams concrete feedback.
|
||||||
|
No session chat — each request is one scenario read.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
|
||||||
|
from ..config import Settings
|
||||||
|
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||||
|
from ..corpus.telemetry import LabTelemetryScenario, summarize_scenario
|
||||||
|
from ..llm.base import LLMProvider
|
||||||
|
from ..prompts.lab import SYSTEM_PROMPT, render_context
|
||||||
|
from .base import BaseAgent
|
||||||
|
|
||||||
|
|
||||||
|
class LabAgent(BaseAgent):
|
||||||
|
name = "lab"
|
||||||
|
|
||||||
|
def __init__(self, provider: LLMProvider, settings: Settings) -> None:
|
||||||
|
super().__init__(provider, settings)
|
||||||
|
|
||||||
|
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||||
|
ctx = learner_context or get_learner_context()
|
||||||
|
return SYSTEM_PROMPT.format_map(render_context(ctx))
|
||||||
|
|
||||||
|
async def stream_feedback(
|
||||||
|
self,
|
||||||
|
scenario: LabTelemetryScenario,
|
||||||
|
learner_context: LearnerContext | None = None,
|
||||||
|
) -> AsyncIterator[str]:
|
||||||
|
timeline = summarize_scenario(scenario)
|
||||||
|
async for token in self.stream_reply(
|
||||||
|
history=None, user_input=timeline, learner_context=learner_context
|
||||||
|
):
|
||||||
|
yield token
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
"""MentorAgent — long-horizon career narrative (REQ-2-010).
|
||||||
|
|
||||||
|
Streaming, session-backed conversational agent: the learner can ask
|
||||||
|
follow-up questions about their trajectory and the Mentor keeps context.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||||
|
from ..prompts.mentor import SYSTEM_PROMPT, render_context
|
||||||
|
from .base import BaseAgent
|
||||||
|
|
||||||
|
|
||||||
|
class MentorAgent(BaseAgent):
|
||||||
|
name = "mentor"
|
||||||
|
|
||||||
|
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||||
|
ctx = learner_context or get_learner_context()
|
||||||
|
return SYSTEM_PROMPT.format_map(render_context(ctx))
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
"""ProctorAgent — integrity signals + coaching interventions (REQ-2-009).
|
||||||
|
|
||||||
|
Consumes a ProctorScenario from the corpus, returns pydantic-validated
|
||||||
|
signal classifications via structured_reply (4-layer defense).
|
||||||
|
Mock engine inputs; real identity/attention signals are v0.3+.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||||
|
from ..corpus.telemetry import ProctorScenario, summarize_proctor_scenario
|
||||||
|
from ..prompts.proctor import SYSTEM_PROMPT, render_context
|
||||||
|
from .base import BaseAgent
|
||||||
|
|
||||||
|
|
||||||
|
class IntegritySignal(BaseModel):
|
||||||
|
signal_type: str # e.g. "context_switch" | "idle_gap" | "large_paste"
|
||||||
|
severity: str # "low" | "medium" | "high"
|
||||||
|
note: str
|
||||||
|
|
||||||
|
|
||||||
|
class ProctorAssessment(BaseModel):
|
||||||
|
scenario_id: str
|
||||||
|
signals: list[IntegritySignal] = Field(min_length=0)
|
||||||
|
intervention: str # ONE supportive coaching recommendation
|
||||||
|
summary: str
|
||||||
|
|
||||||
|
|
||||||
|
PROCTOR_ASSESSMENT_SCHEMA_HINT = (
|
||||||
|
'{"scenario_id": "<id>", "signals": [{"signal_type": "<type>", '
|
||||||
|
'"severity": "low"|"medium"|"high", "note": "<one sentence>"}], '
|
||||||
|
'"intervention": "<one supportive recommendation>", '
|
||||||
|
'"summary": "<one sentence>"}'
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ProctorAgent(BaseAgent):
|
||||||
|
name = "proctor"
|
||||||
|
|
||||||
|
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||||
|
ctx = learner_context or get_learner_context()
|
||||||
|
return SYSTEM_PROMPT.format_map(render_context(ctx))
|
||||||
|
|
||||||
|
async def assess(
|
||||||
|
self,
|
||||||
|
scenario: ProctorScenario,
|
||||||
|
learner_context: LearnerContext | None = None,
|
||||||
|
) -> ProctorAssessment:
|
||||||
|
timeline = summarize_proctor_scenario(scenario)
|
||||||
|
assessment: ProctorAssessment = await self.structured_reply(
|
||||||
|
history=None,
|
||||||
|
user_input=timeline,
|
||||||
|
learner_context=learner_context,
|
||||||
|
schema=ProctorAssessment,
|
||||||
|
schema_hint=PROCTOR_ASSESSMENT_SCHEMA_HINT,
|
||||||
|
)
|
||||||
|
return assessment
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
"""Agent registry — explicit name → agent factory map (D-018, G-4).
|
||||||
|
|
||||||
|
Agents are registered centrally in their own phases (P3-P5) via
|
||||||
|
`registry.register(name, factory)`. One registration pattern, one registry.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections.abc import Callable
|
||||||
|
|
||||||
|
from ..config import Settings
|
||||||
|
from ..llm.base import LLMProvider
|
||||||
|
from .base import BaseAgent
|
||||||
|
|
||||||
|
AgentFactory = Callable[[LLMProvider, Settings], BaseAgent]
|
||||||
|
|
||||||
|
|
||||||
|
def register_builtin_agents(registry: "AgentRegistry") -> None:
|
||||||
|
"""Central registration of all six shipped tutor agents (G-4: one pattern).
|
||||||
|
|
||||||
|
coach, tutor, lab, assessor, proctor, mentor. New agents register here
|
||||||
|
in their landing phase.
|
||||||
|
"""
|
||||||
|
from .assessor import AssessorAgent
|
||||||
|
from .coach import CoachAgent
|
||||||
|
from .lab import LabAgent
|
||||||
|
from .mentor import MentorAgent
|
||||||
|
from .proctor import ProctorAgent
|
||||||
|
from .tutor import TutorAgent
|
||||||
|
|
||||||
|
registry.register("coach", lambda provider, settings: CoachAgent(provider, settings))
|
||||||
|
registry.register("tutor", lambda provider, settings: TutorAgent(provider, settings))
|
||||||
|
registry.register("lab", lambda provider, settings: LabAgent(provider, settings))
|
||||||
|
registry.register(
|
||||||
|
"assessor", lambda provider, settings: AssessorAgent(provider, settings)
|
||||||
|
)
|
||||||
|
registry.register(
|
||||||
|
"proctor", lambda provider, settings: ProctorAgent(provider, settings)
|
||||||
|
)
|
||||||
|
registry.register(
|
||||||
|
"mentor", lambda provider, settings: MentorAgent(provider, settings)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class UnknownAgentError(KeyError):
|
||||||
|
"""Raised when resolving an agent name that was never registered."""
|
||||||
|
|
||||||
|
|
||||||
|
class DuplicateAgentError(ValueError):
|
||||||
|
"""Raised when registering an agent name that already exists."""
|
||||||
|
|
||||||
|
|
||||||
|
class AgentRegistry:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self._factories: dict[str, AgentFactory] = {}
|
||||||
|
|
||||||
|
def register(self, name: str, factory: AgentFactory) -> None:
|
||||||
|
if name in self._factories:
|
||||||
|
raise DuplicateAgentError(f"agent {name!r} already registered")
|
||||||
|
self._factories[name] = factory
|
||||||
|
|
||||||
|
def names(self) -> list[str]:
|
||||||
|
return sorted(self._factories)
|
||||||
|
|
||||||
|
def get(self, provider: LLMProvider, settings: Settings, name: str) -> BaseAgent:
|
||||||
|
try:
|
||||||
|
factory = self._factories[name]
|
||||||
|
except KeyError:
|
||||||
|
raise UnknownAgentError(
|
||||||
|
f"unknown agent {name!r}; registered: {self.names()}"
|
||||||
|
) from None
|
||||||
|
return factory(provider, settings)
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
"""SessionStore — protocol + in-memory implementation (D-019).
|
||||||
|
|
||||||
|
Protocol is DB-migration-ready (A-003): swap InMemorySessionStore for a
|
||||||
|
Redis/PG-backed implementation without touching the API layer.
|
||||||
|
|
||||||
|
Sessions are agent-scoped: switching agents starts a new session ID (avoids
|
||||||
|
persona bleed, A-007). History windowing happens here (last N messages),
|
||||||
|
controlling token growth per session.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
from collections import OrderedDict
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Protocol
|
||||||
|
|
||||||
|
from ..llm.types import Message
|
||||||
|
|
||||||
|
DEFAULT_WINDOW = 20
|
||||||
|
DEFAULT_MAX_SESSIONS = 500
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class AgentSession:
|
||||||
|
session_id: str
|
||||||
|
agent: str
|
||||||
|
learner_id: str = "seed-learner-1"
|
||||||
|
messages: list[Message] = field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
class SessionStore(Protocol):
|
||||||
|
def get(self, session_id: str) -> AgentSession | None: ...
|
||||||
|
def create(
|
||||||
|
self, session_id: str, agent: str, learner_id: str = "seed-learner-1"
|
||||||
|
) -> AgentSession: ...
|
||||||
|
def append(self, session_id: str, message: Message) -> None: ...
|
||||||
|
def history_window(
|
||||||
|
self, session_id: str, max_messages: int = DEFAULT_WINDOW
|
||||||
|
) -> list[Message]: ...
|
||||||
|
def delete(self, session_id: str) -> None: ...
|
||||||
|
|
||||||
|
|
||||||
|
class InMemorySessionStore:
|
||||||
|
"""asyncio.Lock-guarded dict with 20-message windows and 500-cap LRU eviction."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
window: int = DEFAULT_WINDOW,
|
||||||
|
max_sessions: int = DEFAULT_MAX_SESSIONS,
|
||||||
|
) -> None:
|
||||||
|
self._sessions: OrderedDict[str, AgentSession] = OrderedDict()
|
||||||
|
self._lock = asyncio.Lock()
|
||||||
|
self._window = window
|
||||||
|
self._max_sessions = max_sessions
|
||||||
|
|
||||||
|
async def get(self, session_id: str) -> AgentSession | None:
|
||||||
|
async with self._lock:
|
||||||
|
session = self._sessions.get(session_id)
|
||||||
|
if session is not None:
|
||||||
|
self._sessions.move_to_end(session_id) # LRU touch
|
||||||
|
return session
|
||||||
|
|
||||||
|
async def create(
|
||||||
|
self, session_id: str, agent: str, learner_id: str = "seed-learner-1"
|
||||||
|
) -> AgentSession:
|
||||||
|
async with self._lock:
|
||||||
|
session = AgentSession(session_id=session_id, agent=agent, learner_id=learner_id)
|
||||||
|
self._sessions[session_id] = session
|
||||||
|
self._evict_locked()
|
||||||
|
return session
|
||||||
|
|
||||||
|
async def append(self, session_id: str, message: Message) -> None:
|
||||||
|
async with self._lock:
|
||||||
|
session = self._sessions.get(session_id)
|
||||||
|
if session is None:
|
||||||
|
raise KeyError(f"unknown session {session_id!r}")
|
||||||
|
session.messages.append(message)
|
||||||
|
# Bound stored history too (window bounds replay, not storage):
|
||||||
|
# keep at most 2x window so retries/recent context survive.
|
||||||
|
if len(session.messages) > self._window * 2:
|
||||||
|
del session.messages[: len(session.messages) - self._window * 2]
|
||||||
|
self._sessions.move_to_end(session_id)
|
||||||
|
|
||||||
|
async def history_window(
|
||||||
|
self, session_id: str, max_messages: int = DEFAULT_WINDOW
|
||||||
|
) -> list[Message]:
|
||||||
|
async with self._lock:
|
||||||
|
session = self._sessions.get(session_id)
|
||||||
|
if session is None:
|
||||||
|
raise KeyError(f"unknown session {session_id!r}")
|
||||||
|
return list(session.messages[-max_messages:])
|
||||||
|
|
||||||
|
async def delete(self, session_id: str) -> None:
|
||||||
|
async with self._lock:
|
||||||
|
self._sessions.pop(session_id, None)
|
||||||
|
|
||||||
|
def _evict_locked(self) -> None:
|
||||||
|
while len(self._sessions) > self._max_sessions:
|
||||||
|
self._sessions.popitem(last=False) # evict least-recently-used
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
"""Structured output defense — 4 layers (D-020).
|
||||||
|
|
||||||
|
Layer 1: response_format={"type":"json_object"} request (auto-degrades on 400
|
||||||
|
inside the provider).
|
||||||
|
Layer 2: prompt-embedded schema hint ("Respond with ONLY valid JSON...").
|
||||||
|
Layer 3: defensive parse — strip markdown fences, extract first balanced
|
||||||
|
JSON object, pydantic model_validate.
|
||||||
|
Layer 4: single bounded retry with the validation error fed back.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import TypeVar
|
||||||
|
|
||||||
|
from pydantic import BaseModel, ValidationError
|
||||||
|
|
||||||
|
from ..llm.base import LLMProvider
|
||||||
|
from ..llm.types import Message
|
||||||
|
|
||||||
|
T = TypeVar("T", bound=BaseModel)
|
||||||
|
|
||||||
|
|
||||||
|
class StructuredOutputError(Exception):
|
||||||
|
"""Raised when the model output cannot be validated after one retry."""
|
||||||
|
|
||||||
|
|
||||||
|
def extract_json_object(text: str) -> str:
|
||||||
|
"""Strip fences and return the first balanced {...} block from text."""
|
||||||
|
stripped = text.strip()
|
||||||
|
if stripped.startswith("```"):
|
||||||
|
first_newline = stripped.find("\n")
|
||||||
|
if first_newline != -1:
|
||||||
|
stripped = stripped[first_newline + 1:]
|
||||||
|
if stripped.rstrip().endswith("```"):
|
||||||
|
stripped = stripped.rstrip()[:-3]
|
||||||
|
stripped = stripped.strip()
|
||||||
|
start = stripped.find("{")
|
||||||
|
if start == -1:
|
||||||
|
raise StructuredOutputError("no JSON object found in model output")
|
||||||
|
depth = 0
|
||||||
|
in_string = False
|
||||||
|
escape = False
|
||||||
|
for i, ch in enumerate(stripped[start:], start=start):
|
||||||
|
if escape:
|
||||||
|
escape = False
|
||||||
|
continue
|
||||||
|
if ch == "\\":
|
||||||
|
escape = True
|
||||||
|
continue
|
||||||
|
if ch == '"' and not escape:
|
||||||
|
in_string = not in_string
|
||||||
|
continue
|
||||||
|
if in_string:
|
||||||
|
continue
|
||||||
|
if ch == "{":
|
||||||
|
depth += 1
|
||||||
|
elif ch == "}":
|
||||||
|
depth -= 1
|
||||||
|
if depth == 0:
|
||||||
|
return stripped[start:i + 1]
|
||||||
|
raise StructuredOutputError("unbalanced JSON object in model output")
|
||||||
|
|
||||||
|
|
||||||
|
def parse_structured(text: str, schema: type[T]) -> T:
|
||||||
|
"""Layer 3: fence-strip + first-balanced-object + pydantic validation."""
|
||||||
|
candidate = extract_json_object(text)
|
||||||
|
try:
|
||||||
|
return schema.model_validate_json(candidate)
|
||||||
|
except ValidationError as exc:
|
||||||
|
raise StructuredOutputError(f"schema validation failed: {exc}") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def schema_instruction(schema_hint: str) -> str:
|
||||||
|
"""Layer 2: prompt-side schema text."""
|
||||||
|
return (
|
||||||
|
"Respond with ONLY a valid JSON object matching this schema — "
|
||||||
|
"no markdown fences, no prose outside the JSON. "
|
||||||
|
f"Schema: {schema_hint}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def structured_completion(
|
||||||
|
provider: LLMProvider,
|
||||||
|
messages: list[Message],
|
||||||
|
*,
|
||||||
|
model: str,
|
||||||
|
schema: type[T],
|
||||||
|
schema_hint: str,
|
||||||
|
retry_feedback: str | None = None,
|
||||||
|
) -> T:
|
||||||
|
"""Full 4-layer pipeline. One bounded retry (layer 4), then raise."""
|
||||||
|
# Build request: append schema instruction to the last user message (layer 2).
|
||||||
|
request = list(messages)
|
||||||
|
last_user = next((m for m in reversed(request) if m.role == "user"), None)
|
||||||
|
if last_user is not None:
|
||||||
|
request = [
|
||||||
|
Message(role=m.role, content=(m.content + "\n\n" + schema_instruction(schema_hint)))
|
||||||
|
if m is last_user else m
|
||||||
|
for m in request
|
||||||
|
]
|
||||||
|
response_format = {"type": "json_object"}
|
||||||
|
raw = await provider.chat(request, model=model, response_format=response_format)
|
||||||
|
try:
|
||||||
|
return parse_structured(raw, schema) # layers 1+2+3
|
||||||
|
except StructuredOutputError as exc:
|
||||||
|
# Layer 4: single bounded retry with error feedback
|
||||||
|
retry_prompt = (
|
||||||
|
f"Your previous response was invalid: {exc}. "
|
||||||
|
f"Return ONLY the corrected JSON matching: {schema_hint}"
|
||||||
|
)
|
||||||
|
request2 = list(messages)
|
||||||
|
request2.append(Message(role="user", content=retry_prompt))
|
||||||
|
raw2 = await provider.chat(request2, model=model, response_format=response_format)
|
||||||
|
try:
|
||||||
|
return parse_structured(raw2, schema)
|
||||||
|
except StructuredOutputError as exc2:
|
||||||
|
raise StructuredOutputError(
|
||||||
|
f"structured output failed after retry: {exc2}"
|
||||||
|
) from exc2
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
"""TutorAgent — concept delivery, Socratic questioning (REQ-2-006)."""
|
||||||
|
|
||||||
|
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||||
|
from ..prompts.tutor import SYSTEM_PROMPT, render_context
|
||||||
|
from .base import BaseAgent
|
||||||
|
|
||||||
|
|
||||||
|
class TutorAgent(BaseAgent):
|
||||||
|
name = "tutor"
|
||||||
|
|
||||||
|
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||||
|
ctx = learner_context or get_learner_context()
|
||||||
|
return SYSTEM_PROMPT.format_map(render_context(ctx))
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
"""API package — composes providers, sessions, and agents via DI.
|
||||||
|
|
||||||
|
Boundary rule: api/ composes agents/ and llm/; they never import api/.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .assessment import router as assessment_router
|
||||||
|
from .chat import router as chat_router
|
||||||
|
from .lab import router as lab_router
|
||||||
|
from .mentor import router as mentor_router
|
||||||
|
from .proctor import router as proctor_router
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"assessment_router",
|
||||||
|
"chat_router",
|
||||||
|
"lab_router",
|
||||||
|
"mentor_router",
|
||||||
|
"proctor_router",
|
||||||
|
]
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
"""POST /v1/assessment/evaluate — structured rubric scores (REQ-2-008).
|
||||||
|
|
||||||
|
JSON response (not SSE): a pydantic-validated RubricScore. Unknown
|
||||||
|
artifact → 404. The Assessor's structured output IS the payload.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
from ..agents.assessor import RubricScore
|
||||||
|
from ..agents.registry import AgentRegistry
|
||||||
|
from ..config import Settings
|
||||||
|
from ..corpus.artifacts import get_artifact_bundle, get_transcript_for_artifact
|
||||||
|
from ..corpus.learner_context import get_learner_context
|
||||||
|
from .deps import get_agent_registry, get_provider, get_settings
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/v1")
|
||||||
|
|
||||||
|
|
||||||
|
class AssessmentRequest(BaseModel):
|
||||||
|
artifact_id: str = Field(min_length=1)
|
||||||
|
learner_id: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/assessment/evaluate", response_model=RubricScore)
|
||||||
|
async def assessment_evaluate(
|
||||||
|
body: AssessmentRequest,
|
||||||
|
registry: AgentRegistry = Depends(get_agent_registry),
|
||||||
|
settings: Settings = Depends(get_settings),
|
||||||
|
provider=Depends(get_provider),
|
||||||
|
) -> RubricScore:
|
||||||
|
bundle = get_artifact_bundle(body.artifact_id)
|
||||||
|
if bundle is None:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=404, detail=f"unknown artifact {body.artifact_id!r}"
|
||||||
|
)
|
||||||
|
artifact, rubric = bundle
|
||||||
|
transcript = get_transcript_for_artifact(artifact.artifact_id)
|
||||||
|
agent = registry.get(provider, settings, "assessor")
|
||||||
|
learner_context = get_learner_context(body.learner_id)
|
||||||
|
try:
|
||||||
|
return await agent.evaluate(artifact, rubric, transcript, learner_context)
|
||||||
|
except Exception as exc:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=502,
|
||||||
|
detail=f"assessment evaluation failed: {exc}",
|
||||||
|
) from exc
|
||||||
@@ -0,0 +1,128 @@
|
|||||||
|
"""POST /v1/chat/stream — SSE chat with the D-016 envelope + agent routing.
|
||||||
|
|
||||||
|
Envelope: meta event first (flushed before first token), then raw content
|
||||||
|
deltas, then done; error event before [DONE] on mid-stream failure.
|
||||||
|
Pre-first-byte provider failures surface as in-band `provider_unavailable`
|
||||||
|
error events (SSE 200 headers are already committed once meta flushes).
|
||||||
|
|
||||||
|
Agent routing (A-007): the request names its agent; unknown agents are
|
||||||
|
rejected with 422. No autonomous routing in v0.2.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
from sse_starlette.sse import EventSourceResponse
|
||||||
|
|
||||||
|
from ..agents.registry import AgentRegistry, UnknownAgentError
|
||||||
|
from ..agents.session import SessionStore
|
||||||
|
from ..config import Settings
|
||||||
|
from ..corpus.learner_context import get_learner_context
|
||||||
|
from ..llm.base import LLMProvider
|
||||||
|
from ..llm.types import Message
|
||||||
|
from .deps import get_agent_registry, get_provider, get_session_store, get_settings
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/v1")
|
||||||
|
|
||||||
|
|
||||||
|
class ChatStreamRequest(BaseModel):
|
||||||
|
agent: str = Field(min_length=1)
|
||||||
|
session_id: str = Field(min_length=1)
|
||||||
|
learner_id: str | None = None
|
||||||
|
messages: list[Message] = Field(min_length=1)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/chat/stream")
|
||||||
|
async def chat_stream(
|
||||||
|
body: ChatStreamRequest,
|
||||||
|
registry: AgentRegistry = Depends(get_agent_registry),
|
||||||
|
sessions: SessionStore = Depends(get_session_store),
|
||||||
|
settings: Settings = Depends(get_settings),
|
||||||
|
provider: LLMProvider = Depends(get_provider),
|
||||||
|
) -> EventSourceResponse:
|
||||||
|
# Route to the named agent (A-007); unknown → 422 before any streaming.
|
||||||
|
try:
|
||||||
|
agent = registry.get(provider, settings, body.agent)
|
||||||
|
except UnknownAgentError as exc:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=422, detail=str(exc)
|
||||||
|
) from None
|
||||||
|
|
||||||
|
learner_context = get_learner_context(body.learner_id)
|
||||||
|
|
||||||
|
# Agent-scoped session (A-007/G-4): persisted turn history, windowed replay.
|
||||||
|
session = await sessions.get(body.session_id)
|
||||||
|
if session is None:
|
||||||
|
session = await sessions.create(
|
||||||
|
body.session_id, agent=body.agent, learner_id=body.learner_id or "learner-001"
|
||||||
|
)
|
||||||
|
# The new user turn is the last message of the request.
|
||||||
|
user_turn = body.messages[-1]
|
||||||
|
history = await sessions.history_window(body.session_id)
|
||||||
|
# Retry dedupe (P1 from final review): a client retry resends the same
|
||||||
|
# turn after a provider failure — don't double-append it to history.
|
||||||
|
last_stored = history[-1] if history else None
|
||||||
|
is_retry = (
|
||||||
|
last_stored is not None
|
||||||
|
and last_stored.role == "user"
|
||||||
|
and last_stored.content == user_turn.content
|
||||||
|
)
|
||||||
|
if not is_retry:
|
||||||
|
await sessions.append(body.session_id, user_turn)
|
||||||
|
else:
|
||||||
|
# On retry the history replay should exclude the stored duplicate.
|
||||||
|
history = history[:-1]
|
||||||
|
|
||||||
|
async def event_stream() -> AsyncIterator[dict]:
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "meta",
|
||||||
|
"agent": body.agent,
|
||||||
|
"session_id": body.session_id,
|
||||||
|
"model": settings.model,
|
||||||
|
})}
|
||||||
|
first_byte = True
|
||||||
|
reply_parts: list[str] = []
|
||||||
|
try:
|
||||||
|
async for token in agent.stream_reply(
|
||||||
|
history=history,
|
||||||
|
user_input=user_turn.content,
|
||||||
|
learner_context=learner_context,
|
||||||
|
):
|
||||||
|
first_byte = False
|
||||||
|
reply_parts.append(token)
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "delta", "content": token
|
||||||
|
})}
|
||||||
|
full_reply = "".join(reply_parts)
|
||||||
|
if full_reply:
|
||||||
|
await sessions.append(
|
||||||
|
body.session_id, Message(role="assistant", content=full_reply)
|
||||||
|
)
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "done", "finish_reason": "stop"
|
||||||
|
})}
|
||||||
|
yield {"event": "message", "data": "[DONE]"}
|
||||||
|
except Exception as exc: # CancelledError is BaseException — passes through
|
||||||
|
message = str(exc)
|
||||||
|
if first_byte:
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "error", "code": "provider_unavailable", "message": message
|
||||||
|
})}
|
||||||
|
else:
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "error", "code": "provider_error", "message": message
|
||||||
|
})}
|
||||||
|
# [DONE] is yielded from the except branch, NEVER from finally:
|
||||||
|
# a yield inside finally would re-raise after GeneratorExit when the
|
||||||
|
# client disconnects ("async generator ignored GeneratorExit").
|
||||||
|
yield {"event": "message", "data": "[DONE]"}
|
||||||
|
|
||||||
|
return EventSourceResponse(
|
||||||
|
event_stream(),
|
||||||
|
headers={
|
||||||
|
"Cache-Control": "no-cache",
|
||||||
|
"X-Accel-Buffering": "no",
|
||||||
|
},
|
||||||
|
)
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
"""FastAPI dependencies — provider, settings, sessions, agents via app.state (DI)."""
|
||||||
|
|
||||||
|
from fastapi import Request
|
||||||
|
|
||||||
|
from ..agents.registry import AgentRegistry
|
||||||
|
from ..agents.session import SessionStore
|
||||||
|
from ..config import Settings
|
||||||
|
from ..llm.base import LLMProvider
|
||||||
|
|
||||||
|
|
||||||
|
def get_settings(request: Request) -> Settings:
|
||||||
|
return request.app.state.settings
|
||||||
|
|
||||||
|
|
||||||
|
def get_provider(request: Request) -> LLMProvider:
|
||||||
|
return request.app.state.provider
|
||||||
|
|
||||||
|
|
||||||
|
def get_session_store(request: Request) -> SessionStore:
|
||||||
|
return request.app.state.session_store
|
||||||
|
|
||||||
|
|
||||||
|
def get_agent_registry(request: Request) -> AgentRegistry:
|
||||||
|
return request.app.state.agent_registry
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
"""POST /v1/lab/feedback — SSE stream of Lab in-flow feedback (REQ-2-007).
|
||||||
|
|
||||||
|
D-016 envelope with agent=lab. Unknown scenario → 404 before streaming.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
from sse_starlette.sse import EventSourceResponse
|
||||||
|
|
||||||
|
from ..agents.registry import AgentRegistry
|
||||||
|
from ..config import Settings
|
||||||
|
from ..corpus.learner_context import get_learner_context
|
||||||
|
from ..corpus.telemetry import get_lab_scenario
|
||||||
|
from .deps import get_agent_registry, get_provider, get_settings
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/v1")
|
||||||
|
|
||||||
|
|
||||||
|
class LabFeedbackRequest(BaseModel):
|
||||||
|
scenario_id: str = Field(min_length=1)
|
||||||
|
learner_id: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/lab/feedback")
|
||||||
|
async def lab_feedback(
|
||||||
|
body: LabFeedbackRequest,
|
||||||
|
registry: AgentRegistry = Depends(get_agent_registry),
|
||||||
|
settings: Settings = Depends(get_settings),
|
||||||
|
provider=Depends(get_provider),
|
||||||
|
) -> EventSourceResponse:
|
||||||
|
scenario = get_lab_scenario(body.scenario_id)
|
||||||
|
if scenario is None:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=404, detail=f"unknown scenario {body.scenario_id!r}"
|
||||||
|
)
|
||||||
|
agent = registry.get(provider, settings, "lab")
|
||||||
|
learner_context = get_learner_context(body.learner_id)
|
||||||
|
|
||||||
|
async def event_stream() -> AsyncIterator[dict]:
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "meta",
|
||||||
|
"agent": "lab",
|
||||||
|
"scenario_id": body.scenario_id,
|
||||||
|
"model": settings.model,
|
||||||
|
})}
|
||||||
|
first_byte = True
|
||||||
|
try:
|
||||||
|
async for token in agent.stream_feedback(scenario, learner_context):
|
||||||
|
first_byte = False
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "delta", "content": token
|
||||||
|
})}
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "done", "finish_reason": "stop"
|
||||||
|
})}
|
||||||
|
yield {"event": "message", "data": "[DONE]"}
|
||||||
|
except Exception as exc:
|
||||||
|
code = "provider_unavailable" if first_byte else "provider_error"
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "error", "code": code, "message": str(exc)
|
||||||
|
})}
|
||||||
|
# [DONE] from except, not finally — a yield in finally would
|
||||||
|
# re-raise after GeneratorExit on client disconnect.
|
||||||
|
yield {"event": "message", "data": "[DONE]"}
|
||||||
|
|
||||||
|
return EventSourceResponse(
|
||||||
|
event_stream(),
|
||||||
|
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||||
|
)
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
"""POST /v1/mentor/narrative — SSE career narrative stream (REQ-2-010).
|
||||||
|
|
||||||
|
D-016 envelope with agent=mentor. Session-backed: the client supplies a
|
||||||
|
session_id; the Mentor keeps conversation context across follow-ups.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
from sse_starlette.sse import EventSourceResponse
|
||||||
|
|
||||||
|
from ..agents.registry import AgentRegistry, UnknownAgentError
|
||||||
|
from ..agents.session import SessionStore
|
||||||
|
from ..config import Settings
|
||||||
|
from ..corpus.learner_context import get_learner_context
|
||||||
|
from ..llm.types import Message
|
||||||
|
from .deps import get_agent_registry, get_provider, get_session_store, get_settings
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/v1")
|
||||||
|
|
||||||
|
|
||||||
|
class MentorNarrativeRequest(BaseModel):
|
||||||
|
session_id: str = Field(min_length=1)
|
||||||
|
prompt: str = Field(default="Narrate my trajectory.")
|
||||||
|
learner_id: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/mentor/narrative")
|
||||||
|
async def mentor_narrative(
|
||||||
|
body: MentorNarrativeRequest,
|
||||||
|
registry: AgentRegistry = Depends(get_agent_registry),
|
||||||
|
sessions: SessionStore = Depends(get_session_store),
|
||||||
|
settings: Settings = Depends(get_settings),
|
||||||
|
provider=Depends(get_provider),
|
||||||
|
) -> EventSourceResponse:
|
||||||
|
try:
|
||||||
|
agent = registry.get(provider, settings, "mentor")
|
||||||
|
except UnknownAgentError as exc:
|
||||||
|
raise HTTPException(status_code=422, detail=str(exc)) from None
|
||||||
|
|
||||||
|
learner_context = get_learner_context(body.learner_id)
|
||||||
|
|
||||||
|
session = await sessions.get(body.session_id)
|
||||||
|
if session is None:
|
||||||
|
session = await sessions.create(
|
||||||
|
body.session_id, agent="mentor", learner_id=body.learner_id or "learner-001"
|
||||||
|
)
|
||||||
|
history = await sessions.history_window(body.session_id)
|
||||||
|
user_message = Message(role="user", content=body.prompt)
|
||||||
|
await sessions.append(body.session_id, user_message)
|
||||||
|
|
||||||
|
async def event_stream() -> AsyncIterator[dict]:
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "meta",
|
||||||
|
"agent": "mentor",
|
||||||
|
"session_id": body.session_id,
|
||||||
|
"model": settings.model,
|
||||||
|
})}
|
||||||
|
first_byte = True
|
||||||
|
reply_parts: list[str] = []
|
||||||
|
try:
|
||||||
|
async for token in agent.stream_reply(
|
||||||
|
history=history,
|
||||||
|
user_input=body.prompt,
|
||||||
|
learner_context=learner_context,
|
||||||
|
):
|
||||||
|
first_byte = False
|
||||||
|
reply_parts.append(token)
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "delta", "content": token
|
||||||
|
})}
|
||||||
|
full_reply = "".join(reply_parts)
|
||||||
|
if full_reply:
|
||||||
|
await sessions.append(
|
||||||
|
body.session_id, Message(role="assistant", content=full_reply)
|
||||||
|
)
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "done", "finish_reason": "stop"
|
||||||
|
})}
|
||||||
|
yield {"event": "message", "data": "[DONE]"}
|
||||||
|
except Exception as exc:
|
||||||
|
code = "provider_unavailable" if first_byte else "provider_error"
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "error", "code": code, "message": str(exc)
|
||||||
|
})}
|
||||||
|
# [DONE] from except, not finally — a yield in finally would
|
||||||
|
# re-raise after GeneratorExit on client disconnect.
|
||||||
|
yield {"event": "message", "data": "[DONE]"}
|
||||||
|
|
||||||
|
return EventSourceResponse(
|
||||||
|
event_stream(),
|
||||||
|
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||||
|
)
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
"""POST /v1/proctor/signals — structured integrity signals (REQ-2-009).
|
||||||
|
|
||||||
|
JSON response (not SSE): a pydantic-validated ProctorAssessment.
|
||||||
|
Unknown scenario → 404. Coaching-shaped interventions only.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
from ..agents.proctor import ProctorAssessment
|
||||||
|
from ..agents.registry import AgentRegistry
|
||||||
|
from ..config import Settings
|
||||||
|
from ..corpus.learner_context import get_learner_context
|
||||||
|
from ..corpus.telemetry import get_proctor_scenario
|
||||||
|
from .deps import get_agent_registry, get_provider, get_settings
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/v1")
|
||||||
|
|
||||||
|
|
||||||
|
class ProctorRequest(BaseModel):
|
||||||
|
scenario_id: str = Field(min_length=1)
|
||||||
|
learner_id: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/proctor/signals", response_model=ProctorAssessment)
|
||||||
|
async def proctor_signals(
|
||||||
|
body: ProctorRequest,
|
||||||
|
registry: AgentRegistry = Depends(get_agent_registry),
|
||||||
|
settings: Settings = Depends(get_settings),
|
||||||
|
provider=Depends(get_provider),
|
||||||
|
) -> ProctorAssessment:
|
||||||
|
scenario = get_proctor_scenario(body.scenario_id)
|
||||||
|
if scenario is None:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=404, detail=f"unknown scenario {body.scenario_id!r}"
|
||||||
|
)
|
||||||
|
agent = registry.get(provider, settings, "proctor")
|
||||||
|
learner_context = get_learner_context(body.learner_id)
|
||||||
|
try:
|
||||||
|
return await agent.assess(scenario, learner_context)
|
||||||
|
except Exception as exc:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=502, detail=f"proctor assessment failed: {exc}"
|
||||||
|
) from exc
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
"""Service settings — pydantic-settings, env prefix AI_, .env support."""
|
||||||
|
|
||||||
|
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||||
|
|
||||||
|
|
||||||
|
class Settings(BaseSettings):
|
||||||
|
model_config = SettingsConfigDict(env_prefix="AI_", env_file=".env", extra="ignore")
|
||||||
|
|
||||||
|
port: int = 8420
|
||||||
|
provider: str = "mock"
|
||||||
|
model: str = "gemma4:31b"
|
||||||
|
|
||||||
|
ollama_cloud_base_url: str = "https://ollama.com/v1"
|
||||||
|
ollama_cloud_api_key: str = "" # SecretStr adds friction here; never logged, never echoed
|
||||||
|
local_base_url: str = "http://localhost:11434/v1"
|
||||||
|
|
||||||
|
# "auto" sends response_format and degrades on 400; "off" never sends it
|
||||||
|
json_mode: str = "auto"
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
"""Mock engine inputs — pydantic-typed corpus (D-021).
|
||||||
|
|
||||||
|
Convention-aligned with the TS `packages/mock-data` layer: identical ID
|
||||||
|
strings (stack-*, comp-*, learner-*, art-*, mc-*), cross-referenced by the
|
||||||
|
counterpart files. No codegen in v0.2 — alignment is by documented
|
||||||
|
convention; revisit codegen only if drift bites (v0.3).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .learner_context import LEARNER_CONTEXTS, LearnerContext, get_learner_context
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"LEARNER_CONTEXTS",
|
||||||
|
"LearnerContext",
|
||||||
|
"get_learner_context",
|
||||||
|
]
|
||||||
@@ -0,0 +1,209 @@
|
|||||||
|
"""Pre-baked artifacts + rubrics + defense transcripts — Assessor mock inputs (REQ-2-008).
|
||||||
|
|
||||||
|
Counterpart: packages/mock-data/ai-scenarios.ts (artifact IDs string-identical,
|
||||||
|
D-021). Real process-trace grading is a v0.3+ engine (assessment engine);
|
||||||
|
these pre-baked submissions stand in for artifact + defense evaluation.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pydantic import BaseModel
|
||||||
|
|
||||||
|
|
||||||
|
class RubricCriterion(BaseModel):
|
||||||
|
criterion_id: str
|
||||||
|
name: str
|
||||||
|
weight: float
|
||||||
|
description: str
|
||||||
|
|
||||||
|
|
||||||
|
class AssessmentRubric(BaseModel):
|
||||||
|
rubric_id: str
|
||||||
|
competency_id: str
|
||||||
|
criteria: list[RubricCriterion]
|
||||||
|
|
||||||
|
|
||||||
|
class ArtifactSubmission(BaseModel):
|
||||||
|
artifact_id: str
|
||||||
|
name: str
|
||||||
|
artifact_type: str # "code" | "design" | "simulation"
|
||||||
|
competency_id: str
|
||||||
|
description: str
|
||||||
|
evidence_excerpt: str # what the grader sees of the artifact itself
|
||||||
|
|
||||||
|
|
||||||
|
class DefenseTranscript(BaseModel):
|
||||||
|
transcript_id: str
|
||||||
|
artifact_id: str
|
||||||
|
turns: list[dict] # {"speaker": "examiner"|"learner", "text": "..."}
|
||||||
|
|
||||||
|
|
||||||
|
_RUBRIC_ORCHESTRATION = AssessmentRubric(
|
||||||
|
rubric_id="rubric-orchestration-c002",
|
||||||
|
competency_id="stack-orchestration-c002",
|
||||||
|
criteria=[
|
||||||
|
RubricCriterion(
|
||||||
|
criterion_id="rc-architecture",
|
||||||
|
name="Agent architecture soundness",
|
||||||
|
weight=0.3,
|
||||||
|
description="State boundaries and responsibilities are clearly separated",
|
||||||
|
),
|
||||||
|
RubricCriterion(
|
||||||
|
criterion_id="rc-communication",
|
||||||
|
name="Inter-agent communication design",
|
||||||
|
weight=0.3,
|
||||||
|
description="Message contracts are explicit, typed, and failure-aware",
|
||||||
|
),
|
||||||
|
RubricCriterion(
|
||||||
|
criterion_id="rc-reliability",
|
||||||
|
name="Reliability engineering",
|
||||||
|
weight=0.25,
|
||||||
|
description="Retries, timeouts, and degradation paths handled",
|
||||||
|
),
|
||||||
|
RubricCriterion(
|
||||||
|
criterion_id="rc-process",
|
||||||
|
name="Process trace quality",
|
||||||
|
weight=0.15,
|
||||||
|
description="Telemetry shows iterative building with real checkpoints",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
_RUBRIC_TOOL_USE = AssessmentRubric(
|
||||||
|
rubric_id="rubric-orchestration-c003",
|
||||||
|
competency_id="stack-orchestration-c003",
|
||||||
|
criteria=[
|
||||||
|
RubricCriterion(
|
||||||
|
criterion_id="rc-eval-design",
|
||||||
|
name="Evaluation design rigor",
|
||||||
|
weight=0.35,
|
||||||
|
description="Hypotheses, controls, and metrics are explicit and defensible",
|
||||||
|
),
|
||||||
|
RubricCriterion(
|
||||||
|
criterion_id="rc-eval-robustness",
|
||||||
|
name="Harness robustness",
|
||||||
|
weight=0.35,
|
||||||
|
description="Error handling, variance awareness, and reproducibility",
|
||||||
|
),
|
||||||
|
RubricCriterion(
|
||||||
|
criterion_id="rc-eval-insight",
|
||||||
|
name="Insight extraction",
|
||||||
|
weight=0.3,
|
||||||
|
description="Results are interpreted into concrete engineering decisions",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
_ARTIFACT_RESEARCH_ASSISTANT = ArtifactSubmission(
|
||||||
|
artifact_id="art-eval-research-assistant",
|
||||||
|
name="Multi-agent research assistant (eval build)",
|
||||||
|
artifact_type="code",
|
||||||
|
competency_id="stack-orchestration-c002",
|
||||||
|
description=(
|
||||||
|
"LangGraph-based assistant planning, retrieving, drafting cited reviews."
|
||||||
|
),
|
||||||
|
evidence_excerpt=(
|
||||||
|
"planner.py defines state schema with explicit fields "
|
||||||
|
"(plan, findings, draft); tool_node.py wraps retrieval with a "
|
||||||
|
"3-retry loop and typed ToolMessage responses; tests cover "
|
||||||
|
"planner->tool->writer handoffs; README shows graph diagram"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
_TRANSCRIPT_RESEARCH_ASSISTANT = DefenseTranscript(
|
||||||
|
transcript_id="defense-art-eval-research-assistant",
|
||||||
|
artifact_id="art-eval-research-assistant",
|
||||||
|
turns=[
|
||||||
|
{"speaker": "examiner",
|
||||||
|
"text": "Why did you give the planner sole write access to the plan field?"},
|
||||||
|
{"speaker": "learner",
|
||||||
|
"text": "So worker nodes can't mutate each other's inputs — "
|
||||||
|
"the state stays predictable and the graph is debuggable"},
|
||||||
|
{"speaker": "examiner",
|
||||||
|
"text": "What happens when the retrieval tool times out three times?"},
|
||||||
|
{"speaker": "learner",
|
||||||
|
"text": "The tool node degrades to a no-op ToolMessage with a "
|
||||||
|
"retry flag so the writer can fall back to existing findings"},
|
||||||
|
{"speaker": "examiner", "text": "How would you extend this to a third agent?"},
|
||||||
|
{"speaker": "learner",
|
||||||
|
"text": "Add a reviewer node with its own typed messages, same pattern"},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
_ARTIFACT_RAG_DASHBOARD = ArtifactSubmission(
|
||||||
|
artifact_id="art-eval-rag-dashboard",
|
||||||
|
name="RAG retrieval quality dashboard (eval build)",
|
||||||
|
artifact_type="code",
|
||||||
|
competency_id="stack-orchestration-c003",
|
||||||
|
description="Dashboard comparing chunking strategies/rerankers across 800 queries.",
|
||||||
|
evidence_excerpt=(
|
||||||
|
"eval harness sweeps 4 chunk sizes x 3 rerankers; results table auto-generated; "
|
||||||
|
"no error handling on the query loader; tests only cover the happy path"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
_TRANSCRIPT_RAG_DASHBOARD = DefenseTranscript(
|
||||||
|
transcript_id="defense-art-eval-rag-dashboard",
|
||||||
|
artifact_id="art-eval-rag-dashboard",
|
||||||
|
turns=[
|
||||||
|
{"speaker": "examiner", "text": "How did you control for query difficulty across runs?"},
|
||||||
|
{"speaker": "learner", "text": "I, um, used the same query set each time"},
|
||||||
|
{"speaker": "examiner", "text": "What happens if the query loader hits a malformed row?"},
|
||||||
|
{"speaker": "learner", "text": "I didn't handle that. It would probably crash."},
|
||||||
|
{"speaker": "examiner", "text": "What would you improve first?"},
|
||||||
|
{"speaker": "learner",
|
||||||
|
"text": "Probably add the error handling, then look at variance between runs"},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
RUBRICS: dict[str, AssessmentRubric] = {
|
||||||
|
_RUBRIC_ORCHESTRATION.rubric_id: _RUBRIC_ORCHESTRATION,
|
||||||
|
_RUBRIC_TOOL_USE.rubric_id: _RUBRIC_TOOL_USE,
|
||||||
|
}
|
||||||
|
|
||||||
|
ARTIFACTS: dict[str, ArtifactSubmission] = {
|
||||||
|
a.artifact_id: a
|
||||||
|
for a in (_ARTIFACT_RESEARCH_ASSISTANT, _ARTIFACT_RAG_DASHBOARD)
|
||||||
|
}
|
||||||
|
|
||||||
|
TRANSCRIPTS: dict[str, DefenseTranscript] = {
|
||||||
|
t.transcript_id: t
|
||||||
|
for t in (_TRANSCRIPT_RESEARCH_ASSISTANT, _TRANSCRIPT_RAG_DASHBOARD)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def rubric_for_competency(competency_id: str) -> AssessmentRubric | None:
|
||||||
|
for rubric in RUBRICS.values():
|
||||||
|
if rubric.competency_id == competency_id:
|
||||||
|
return rubric
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def get_artifact_bundle(artifact_id: str) -> tuple[ArtifactSubmission, AssessmentRubric] | None:
|
||||||
|
"""Resolve (artifact, rubric) for an artifact ID; None if unknown."""
|
||||||
|
artifact = ARTIFACTS.get(artifact_id)
|
||||||
|
if artifact is None:
|
||||||
|
return None
|
||||||
|
rubric = rubric_for_competency(artifact.competency_id)
|
||||||
|
if rubric is None:
|
||||||
|
return None
|
||||||
|
return artifact, rubric
|
||||||
|
|
||||||
|
|
||||||
|
def get_transcript_for_artifact(artifact_id: str) -> DefenseTranscript | None:
|
||||||
|
for transcript in TRANSCRIPTS.values():
|
||||||
|
if transcript.artifact_id == artifact_id:
|
||||||
|
return transcript
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def render_rubric(rubric: AssessmentRubric) -> str:
|
||||||
|
lines = [f"Rubric: {rubric.rubric_id} (competency {rubric.competency_id})"]
|
||||||
|
for c in rubric.criteria:
|
||||||
|
lines.append(f"- {c.criterion_id} ({c.weight:.2f}): {c.name} — {c.description}")
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def render_transcript(transcript: DefenseTranscript) -> str:
|
||||||
|
lines = [f"Defense transcript: {transcript.transcript_id}"]
|
||||||
|
for turn in transcript.turns:
|
||||||
|
lines.append(f"{turn['speaker']}: {turn['text']}")
|
||||||
|
return "\n".join(lines)
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
"""Learner context corpus — pydantic mirror of TS learner-progress.ts (D-021).
|
||||||
|
|
||||||
|
Counterpart: packages/mock-data/src/learner-progress.ts (or learner-progress.ts
|
||||||
|
at package root). IDs are string-identical: learner-001, stack-orchestration,
|
||||||
|
stack-safety, stack-orchestration-c00N, art-*, mc-*.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pydantic import BaseModel
|
||||||
|
|
||||||
|
|
||||||
|
class CompetencyProgress(BaseModel):
|
||||||
|
competency_id: str
|
||||||
|
title: str
|
||||||
|
status: str # "mastered" | "in_progress" | "not_started"
|
||||||
|
|
||||||
|
|
||||||
|
class StackProgress(BaseModel):
|
||||||
|
stack_id: str
|
||||||
|
title: str
|
||||||
|
percent: int
|
||||||
|
|
||||||
|
|
||||||
|
class LearnerContext(BaseModel):
|
||||||
|
learner_id: str
|
||||||
|
name: str
|
||||||
|
active_stacks: list[StackProgress]
|
||||||
|
active_competencies: list[CompetencyProgress]
|
||||||
|
microcredential_count: int
|
||||||
|
recent_artifacts: list[str] # artifact names
|
||||||
|
|
||||||
|
|
||||||
|
_STACK_ORCHESTRATION = StackProgress(
|
||||||
|
stack_id="stack-orchestration", title="AI Orchestration Engineer", percent=62
|
||||||
|
)
|
||||||
|
_STACK_SAFETY = StackProgress(
|
||||||
|
stack_id="stack-safety", title="AI Safety & Governance Lead", percent=41
|
||||||
|
)
|
||||||
|
|
||||||
|
_LEARNER_1 = LearnerContext(
|
||||||
|
learner_id="learner-001",
|
||||||
|
name="Alex Rivera",
|
||||||
|
active_stacks=[_STACK_ORCHESTRATION, _STACK_SAFETY],
|
||||||
|
active_competencies=[
|
||||||
|
CompetencyProgress(
|
||||||
|
competency_id="stack-orchestration-c001",
|
||||||
|
title="Agent architecture fundamentals",
|
||||||
|
status="mastered",
|
||||||
|
),
|
||||||
|
CompetencyProgress(
|
||||||
|
competency_id="stack-orchestration-c002",
|
||||||
|
title="Multi-agent communication patterns",
|
||||||
|
status="in_progress",
|
||||||
|
),
|
||||||
|
CompetencyProgress(
|
||||||
|
competency_id="stack-orchestration-c003",
|
||||||
|
title="Tool use and function calling",
|
||||||
|
status="in_progress",
|
||||||
|
),
|
||||||
|
CompetencyProgress(
|
||||||
|
competency_id="stack-safety-c021",
|
||||||
|
title="Red-team basics for agent systems",
|
||||||
|
status="in_progress",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
microcredential_count=4,
|
||||||
|
recent_artifacts=[
|
||||||
|
"Multi-agent research assistant",
|
||||||
|
"RAG retrieval quality dashboard",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
_LEARNER_2 = LearnerContext(
|
||||||
|
learner_id="learner-002",
|
||||||
|
name="Priya Chen",
|
||||||
|
active_stacks=[
|
||||||
|
StackProgress(
|
||||||
|
stack_id="stack-designer", title="Human-AI Product Designer", percent=55
|
||||||
|
),
|
||||||
|
],
|
||||||
|
active_competencies=[
|
||||||
|
CompetencyProgress(
|
||||||
|
competency_id="stack-designer-c001",
|
||||||
|
title="Prompt-to-prototype workflows",
|
||||||
|
status="mastered",
|
||||||
|
),
|
||||||
|
CompetencyProgress(
|
||||||
|
competency_id="stack-designer-c002",
|
||||||
|
title="Evaluating AI UX patterns",
|
||||||
|
status="in_progress",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
microcredential_count=2,
|
||||||
|
recent_artifacts=["AI onboarding flow concept test"],
|
||||||
|
)
|
||||||
|
|
||||||
|
LEARNER_CONTEXTS: dict[str, LearnerContext] = {
|
||||||
|
_LEARNER_1.learner_id: _LEARNER_1,
|
||||||
|
_LEARNER_2.learner_id: _LEARNER_2,
|
||||||
|
}
|
||||||
|
|
||||||
|
DEFAULT_LEARNER_ID = "learner-001"
|
||||||
|
|
||||||
|
|
||||||
|
def get_learner_context(learner_id: str | None = None) -> LearnerContext:
|
||||||
|
"""Resolve a learner context by ID, falling back to the default seed."""
|
||||||
|
if learner_id is None:
|
||||||
|
return LEARNER_CONTEXTS[DEFAULT_LEARNER_ID]
|
||||||
|
return LEARNER_CONTEXTS.get(learner_id, LEARNER_CONTEXTS[DEFAULT_LEARNER_ID])
|
||||||
@@ -0,0 +1,178 @@
|
|||||||
|
"""Simulated sandbox telemetry corpus — Lab agent mock engine inputs (REQ-2-007).
|
||||||
|
|
||||||
|
Counterpart: packages/mock-data/ai-scenarios.ts (scenario IDs string-identical,
|
||||||
|
D-021). Real sandbox telemetry is a v0.3+ engine (sandbox fabric); these
|
||||||
|
scripted event streams stand in for the build-session process trace.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pydantic import BaseModel
|
||||||
|
|
||||||
|
|
||||||
|
class TelemetryEvent(BaseModel):
|
||||||
|
timestamp: int # seconds since session start
|
||||||
|
kind: str # "keystroke_burst" | "file_save" | "run_tests" | "test_pass"
|
||||||
|
# | "test_fail" | "console_error" | "idle" | "paste" | "commit"
|
||||||
|
|
||||||
|
|
||||||
|
detail: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
class LabTelemetryScenario(BaseModel):
|
||||||
|
scenario_id: str
|
||||||
|
title: str
|
||||||
|
competency_id: str
|
||||||
|
events: list[TelemetryEvent]
|
||||||
|
|
||||||
|
|
||||||
|
class ProctorEvent(BaseModel):
|
||||||
|
timestamp: int # seconds since session start
|
||||||
|
kind: str # "tab_switch" | "idle" | "paste_large" | "focus_lost" | "keystroke_burst"
|
||||||
|
detail: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
class ProctorScenario(BaseModel):
|
||||||
|
scenario_id: str
|
||||||
|
title: str
|
||||||
|
competency_id: str
|
||||||
|
events: list[ProctorEvent]
|
||||||
|
|
||||||
|
|
||||||
|
_PROCTOR_SCENARIO_HEALTHY = ProctorScenario(
|
||||||
|
scenario_id="proctor-scenario-healthy",
|
||||||
|
title="Healthy defense session — focused throughout",
|
||||||
|
competency_id="stack-orchestration-c002",
|
||||||
|
events=[
|
||||||
|
ProctorEvent(timestamp=0, kind="keystroke_burst", detail="session begins"),
|
||||||
|
ProctorEvent(timestamp=310, kind="keystroke_burst", detail="long answer in progress"),
|
||||||
|
ProctorEvent(timestamp=640, kind="keystroke_burst", detail="revision pass"),
|
||||||
|
ProctorEvent(timestamp=900, kind="keystroke_burst", detail="final answer"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
_PROCTOR_SCENARIO_DISTRACTED = ProctorScenario(
|
||||||
|
scenario_id="proctor-scenario-distracted",
|
||||||
|
title="Distracted defense session — tab switches and idle gaps",
|
||||||
|
competency_id="stack-orchestration-c002",
|
||||||
|
events=[
|
||||||
|
ProctorEvent(timestamp=0, kind="keystroke_burst", detail="session begins"),
|
||||||
|
ProctorEvent(timestamp=120, kind="tab_switch", detail="to docs.nextjs.org"),
|
||||||
|
ProctorEvent(timestamp=125, kind="focus_lost", detail="window blur 40s"),
|
||||||
|
ProctorEvent(timestamp=300, kind="idle", detail="no activity for 5 minutes"),
|
||||||
|
ProctorEvent(timestamp=600, kind="tab_switch", detail="to github.com"),
|
||||||
|
ProctorEvent(timestamp=605, kind="focus_lost", detail="window blur 2m"),
|
||||||
|
ProctorEvent(timestamp=720, kind="keystroke_burst", detail="resumes typing"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
_PROCTOR_SCENARIO_FLAGGED = ProctorScenario(
|
||||||
|
scenario_id="proctor-scenario-flagged",
|
||||||
|
title="Flagged defense session — large paste during exam",
|
||||||
|
competency_id="stack-orchestration-c003",
|
||||||
|
events=[
|
||||||
|
ProctorEvent(timestamp=0, kind="keystroke_burst", detail="short intro typed"),
|
||||||
|
ProctorEvent(timestamp=85, kind="paste_large", detail="3,100 chars pasted in 2s"),
|
||||||
|
ProctorEvent(timestamp=90, kind="idle", detail="no activity for 4 minutes"),
|
||||||
|
ProctorEvent(timestamp=330, kind="paste_large", detail="2,800 chars pasted in 2s"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
PROCTOR_SCENARIOS: dict[str, ProctorScenario] = {
|
||||||
|
s.scenario_id: s
|
||||||
|
for s in (
|
||||||
|
_PROCTOR_SCENARIO_HEALTHY,
|
||||||
|
_PROCTOR_SCENARIO_DISTRACTED,
|
||||||
|
_PROCTOR_SCENARIO_FLAGGED,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def get_proctor_scenario(scenario_id: str) -> ProctorScenario | None:
|
||||||
|
return PROCTOR_SCENARIOS.get(scenario_id)
|
||||||
|
|
||||||
|
|
||||||
|
def summarize_proctor_scenario(scenario: ProctorScenario) -> str:
|
||||||
|
"""Render the proctor event timeline as compact text for prompt injection."""
|
||||||
|
lines = [f"Defense session: {scenario.title} (competency {scenario.competency_id})"]
|
||||||
|
for event in scenario.events:
|
||||||
|
lines.append(f"t+{event.timestamp}s {event.kind}: {event.detail}".rstrip(": "))
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
_LAB_SCENARIO_STRONG = LabTelemetryScenario(
|
||||||
|
scenario_id="lab-scenario-strong",
|
||||||
|
title="Strong build session — multi-agent research assistant",
|
||||||
|
competency_id="stack-orchestration-c002",
|
||||||
|
events=[
|
||||||
|
TelemetryEvent(timestamp=0, kind="keystroke_burst", detail="planner.py"),
|
||||||
|
TelemetryEvent(timestamp=95, kind="file_save", detail="planner.py"),
|
||||||
|
TelemetryEvent(timestamp=120, kind="run_tests", detail="3 tests"),
|
||||||
|
TelemetryEvent(timestamp=126, kind="test_pass",
|
||||||
|
detail="3/3 passed"),
|
||||||
|
TelemetryEvent(timestamp=180, kind="keystroke_burst", detail="tool_node.py"),
|
||||||
|
TelemetryEvent(timestamp=260, kind="file_save", detail="tool_node.py"),
|
||||||
|
TelemetryEvent(timestamp=275, kind="run_tests", detail="4 tests"),
|
||||||
|
TelemetryEvent(timestamp=281, kind="test_pass",
|
||||||
|
detail="4/4 passed"),
|
||||||
|
TelemetryEvent(timestamp=340, kind="commit",
|
||||||
|
detail="add tool node with retries"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
_LAB_SCENARIO_STRUGGLING = LabTelemetryScenario(
|
||||||
|
scenario_id="lab-scenario-struggling",
|
||||||
|
title="Struggling build session — repeated failures, no checkpoints",
|
||||||
|
competency_id="stack-orchestration-c002",
|
||||||
|
events=[
|
||||||
|
TelemetryEvent(timestamp=0, kind="keystroke_burst", detail="main.py"),
|
||||||
|
TelemetryEvent(timestamp=210, kind="run_tests", detail="2 tests"),
|
||||||
|
TelemetryEvent(timestamp=215, kind="test_fail",
|
||||||
|
detail="ImportError: no module named 'tools'"),
|
||||||
|
TelemetryEvent(timestamp=216, kind="console_error", detail="traceback dumped"),
|
||||||
|
TelemetryEvent(timestamp=300, kind="keystroke_burst",
|
||||||
|
detail="main.py"),
|
||||||
|
TelemetryEvent(timestamp=520, kind="run_tests",
|
||||||
|
detail="2 tests"),
|
||||||
|
TelemetryEvent(timestamp=525, kind="test_fail",
|
||||||
|
detail="ImportError: no module named 'tools'"),
|
||||||
|
TelemetryEvent(timestamp=526, kind="console_error",
|
||||||
|
detail="same traceback as before"),
|
||||||
|
TelemetryEvent(timestamp=600, kind="idle",
|
||||||
|
detail="no activity for 6 minutes"),
|
||||||
|
TelemetryEvent(timestamp=960, kind="idle",
|
||||||
|
detail="no activity for 14 minutes"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
_LAB_SCENARIO_FLAGGED = LabTelemetryScenario(
|
||||||
|
scenario_id="lab-scenario-flagged",
|
||||||
|
title="Flagged build session — large paste, instant pass",
|
||||||
|
competency_id="stack-orchestration-c003",
|
||||||
|
events=[
|
||||||
|
TelemetryEvent(timestamp=0, kind="keystroke_burst", detail="eval.py"),
|
||||||
|
TelemetryEvent(timestamp=30, kind="paste",
|
||||||
|
detail="2,400 chars pasted into eval.py"),
|
||||||
|
TelemetryEvent(timestamp=45, kind="run_tests", detail="6 tests"),
|
||||||
|
TelemetryEvent(timestamp=47, kind="test_pass", detail="6/6 passed"),
|
||||||
|
TelemetryEvent(timestamp=48, kind="commit",
|
||||||
|
detail="finish eval harness"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
LAB_SCENARIOS: dict[str, LabTelemetryScenario] = {
|
||||||
|
s.scenario_id: s
|
||||||
|
for s in (_LAB_SCENARIO_STRONG, _LAB_SCENARIO_STRUGGLING, _LAB_SCENARIO_FLAGGED)
|
||||||
|
}
|
||||||
|
|
||||||
|
DEFAULT_LAB_SCENARIO_ID = "lab-scenario-strong"
|
||||||
|
|
||||||
|
|
||||||
|
def get_lab_scenario(scenario_id: str) -> LabTelemetryScenario | None:
|
||||||
|
return LAB_SCENARIOS.get(scenario_id)
|
||||||
|
|
||||||
|
|
||||||
|
def summarize_scenario(scenario: LabTelemetryScenario) -> str:
|
||||||
|
"""Render the event timeline as compact text for prompt injection."""
|
||||||
|
lines = [f"Session: {scenario.title} (competency {scenario.competency_id})"]
|
||||||
|
for event in scenario.events:
|
||||||
|
lines.append(f"t+{event.timestamp}s {event.kind}: {event.detail}".rstrip(": "))
|
||||||
|
return "\n".join(lines)
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
"""LLM package — provider-agnostic layer (D-017)."""
|
||||||
|
|
||||||
|
from .base import LLMProvider
|
||||||
|
from .factory import create_provider
|
||||||
|
from .mock import MockProvider
|
||||||
|
from .openai_compat import OpenAICompatProvider
|
||||||
|
from .types import Message
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"LLMProvider",
|
||||||
|
"Message",
|
||||||
|
"MockProvider",
|
||||||
|
"OpenAICompatProvider",
|
||||||
|
"create_provider",
|
||||||
|
]
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
"""LLMProvider protocol — the port all agents depend on (D-017).
|
||||||
|
|
||||||
|
Implementations: openai_compat.OpenAICompatProvider (ollama-cloud + local),
|
||||||
|
mock.MockProvider (deterministic, tests/CI). Providers are dumb pipes:
|
||||||
|
no envelope logic here — the API layer owns meta/done/error events (D-016).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
from typing import Protocol
|
||||||
|
|
||||||
|
from .types import Message
|
||||||
|
|
||||||
|
|
||||||
|
class LLMProvider(Protocol):
|
||||||
|
async def stream_chat(
|
||||||
|
self,
|
||||||
|
messages: list[Message],
|
||||||
|
*,
|
||||||
|
model: str,
|
||||||
|
temperature: float = 0.7,
|
||||||
|
response_format: dict | None = None,
|
||||||
|
) -> AsyncIterator[str]:
|
||||||
|
"""Yield incremental content deltas (plain text chunks)."""
|
||||||
|
...
|
||||||
|
|
||||||
|
async def chat(
|
||||||
|
self,
|
||||||
|
messages: list[Message],
|
||||||
|
*,
|
||||||
|
model: str,
|
||||||
|
temperature: float = 0.7,
|
||||||
|
response_format: dict | None = None,
|
||||||
|
) -> str:
|
||||||
|
"""Non-streaming completion — returns the full reply text."""
|
||||||
|
...
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
"""Provider factory — selects the LLM provider from settings (D-014)."""
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from ..config import Settings
|
||||||
|
from .mock import MockProvider
|
||||||
|
from .openai_compat import OpenAICompatProvider
|
||||||
|
|
||||||
|
PROVIDER_NAMES = ("ollama-cloud", "local", "mock")
|
||||||
|
|
||||||
|
|
||||||
|
def create_provider(settings: Settings, http_client: httpx.AsyncClient):
|
||||||
|
"""Return the provider instance for settings.provider.
|
||||||
|
|
||||||
|
Raises ValueError for unknown provider names.
|
||||||
|
"""
|
||||||
|
if settings.provider == "ollama-cloud":
|
||||||
|
return OpenAICompatProvider(
|
||||||
|
http_client=http_client,
|
||||||
|
base_url=settings.ollama_cloud_base_url,
|
||||||
|
api_key=settings.ollama_cloud_api_key,
|
||||||
|
json_mode=settings.json_mode,
|
||||||
|
)
|
||||||
|
if settings.provider == "local":
|
||||||
|
return OpenAICompatProvider(
|
||||||
|
http_client=http_client,
|
||||||
|
base_url=settings.local_base_url,
|
||||||
|
json_mode=settings.json_mode,
|
||||||
|
)
|
||||||
|
if settings.provider == "mock":
|
||||||
|
return MockProvider()
|
||||||
|
raise ValueError(
|
||||||
|
f"unknown provider {settings.provider!r}; expected one of {PROVIDER_NAMES}"
|
||||||
|
)
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
"""Deterministic mock provider — tests and CI. NEVER calls the network.
|
||||||
|
|
||||||
|
Determinism: the reply text is seeded from the message content hash, so
|
||||||
|
identical inputs always produce identical outputs. Supports scripted
|
||||||
|
failure modes for error-path coverage (D-023, A-010).
|
||||||
|
"""
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
|
||||||
|
from .types import Message
|
||||||
|
|
||||||
|
_REPLIES = [
|
||||||
|
"Great question — let's break this down step by step and see where it leads.",
|
||||||
|
"Here is the key idea: small, verified moves compound into mastery over time.",
|
||||||
|
"Think about it this way: what would the simplest working version look like?",
|
||||||
|
"You are closer than you think. Try restating the goal in one sentence first.",
|
||||||
|
"Let me offer a different angle before we move to the next step.",
|
||||||
|
]
|
||||||
|
|
||||||
|
_JSON_REPLY = '{"summary": "mock structured reply", "confidence": 0.87}'
|
||||||
|
|
||||||
|
|
||||||
|
class MockProvider:
|
||||||
|
"""Scripted provider: deterministic streams, no network, failure injection."""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.fail_before_first_token: bool = False
|
||||||
|
self.fail_mid_stream_at_index: int | None = None
|
||||||
|
self.abort_recorded: bool = False # set in stream finally-block (cancellation test)
|
||||||
|
|
||||||
|
def _reply_for(self, messages: list[Message], response_format: dict | None) -> str:
|
||||||
|
seed_src = "|".join(f"{m.role}:{m.content}" for m in messages)
|
||||||
|
if response_format is not None and response_format.get("type") == "json_object":
|
||||||
|
return _JSON_REPLY
|
||||||
|
digest = hashlib.sha256(seed_src.encode()).hexdigest()
|
||||||
|
base = _REPLIES[int(digest[:2], 16) % len(_REPLIES)]
|
||||||
|
# Deterministic seed tag guarantees distinct inputs → distinct replies
|
||||||
|
return f"{base} [#{digest[:8]}]"
|
||||||
|
|
||||||
|
def _tokenize(self, text: str) -> list[str]:
|
||||||
|
words = text.split(" ")
|
||||||
|
tokens: list[str] = []
|
||||||
|
for i, word in enumerate(words):
|
||||||
|
suffix = " " if i < len(words) - 1 else ""
|
||||||
|
tokens.append(word + suffix)
|
||||||
|
return tokens
|
||||||
|
|
||||||
|
async def stream_chat(
|
||||||
|
self,
|
||||||
|
messages: list[Message],
|
||||||
|
*,
|
||||||
|
model: str,
|
||||||
|
temperature: float = 0.7,
|
||||||
|
response_format: dict | None = None,
|
||||||
|
) -> AsyncIterator[str]:
|
||||||
|
if self.fail_before_first_token:
|
||||||
|
raise RuntimeError("mock provider: scripted failure before first token")
|
||||||
|
reply = self._reply_for(messages, response_format)
|
||||||
|
tokens = self._tokenize(reply)
|
||||||
|
try:
|
||||||
|
for i, token in enumerate(tokens):
|
||||||
|
if self.fail_mid_stream_at_index is not None and i == self.fail_mid_stream_at_index:
|
||||||
|
raise RuntimeError("mock provider: scripted mid-stream failure")
|
||||||
|
yield token
|
||||||
|
finally:
|
||||||
|
# Cancellation (GeneratorExit/CancelledError) lands here — tests assert this.
|
||||||
|
self.abort_recorded = True
|
||||||
|
|
||||||
|
async def chat(
|
||||||
|
self,
|
||||||
|
messages: list[Message],
|
||||||
|
*,
|
||||||
|
model: str,
|
||||||
|
temperature: float = 0.7,
|
||||||
|
response_format: dict | None = None,
|
||||||
|
) -> str:
|
||||||
|
if self.fail_before_first_token:
|
||||||
|
raise RuntimeError("mock provider: scripted failure before completion")
|
||||||
|
return self._reply_for(messages, response_format)
|
||||||
|
|
||||||
|
|
||||||
|
class ScriptedJSONProvider(MockProvider):
|
||||||
|
"""Mock variant returning a fixed JSON payload for structured tests."""
|
||||||
|
|
||||||
|
def __init__(self, payload: dict) -> None:
|
||||||
|
super().__init__()
|
||||||
|
self.payload = payload
|
||||||
|
|
||||||
|
def _reply_for(self, messages: list[Message], response_format: dict | None) -> str:
|
||||||
|
if response_format is not None and response_format.get("type") == "json_object":
|
||||||
|
return json.dumps(self.payload)
|
||||||
|
return super()._reply_for(messages, response_format)
|
||||||
@@ -0,0 +1,125 @@
|
|||||||
|
"""OpenAI-compatible provider — one implementation serves ollama-cloud AND local
|
||||||
|
endpoints (they differ only in base_url/key). Raw httpx, no SDK (D-017).
|
||||||
|
|
||||||
|
Boundary rules:
|
||||||
|
- llm/ imports nothing from agents/ or api/
|
||||||
|
- api_key NEVER appears in exceptions, logs, or error messages
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from .types import Message
|
||||||
|
|
||||||
|
|
||||||
|
class OpenAICompatProvider:
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
http_client: httpx.AsyncClient,
|
||||||
|
base_url: str,
|
||||||
|
api_key: str = "",
|
||||||
|
json_mode: str = "auto",
|
||||||
|
) -> None:
|
||||||
|
self._client = http_client
|
||||||
|
self._base_url = base_url.rstrip("/")
|
||||||
|
self._api_key = api_key
|
||||||
|
self._json_mode = json_mode
|
||||||
|
|
||||||
|
def _headers(self) -> dict[str, str]:
|
||||||
|
headers = {"Content-Type": "application/json"}
|
||||||
|
if self._api_key:
|
||||||
|
headers["Authorization"] = f"Bearer {self._api_key}"
|
||||||
|
return headers
|
||||||
|
|
||||||
|
def _payload(
|
||||||
|
self,
|
||||||
|
messages: list[Message],
|
||||||
|
model: str,
|
||||||
|
temperature: float,
|
||||||
|
response_format: dict | None,
|
||||||
|
stream: bool,
|
||||||
|
) -> dict:
|
||||||
|
payload: dict = {
|
||||||
|
"model": model,
|
||||||
|
"messages": [{"role": m.role, "content": m.content} for m in messages],
|
||||||
|
"temperature": temperature,
|
||||||
|
}
|
||||||
|
if stream:
|
||||||
|
payload["stream"] = True
|
||||||
|
else:
|
||||||
|
payload["stream"] = False
|
||||||
|
# json_mode="auto": send response_format and degrade on 400; "off": never send
|
||||||
|
if response_format is not None and self._json_mode == "auto":
|
||||||
|
payload["response_format"] = response_format
|
||||||
|
return payload
|
||||||
|
|
||||||
|
def _sanitize(self, exc: Exception) -> RuntimeError:
|
||||||
|
text = str(exc)
|
||||||
|
if self._api_key and self._api_key in text:
|
||||||
|
text = text.replace(self._api_key, "[REDACTED]")
|
||||||
|
return RuntimeError(f"llm provider error: {text}")
|
||||||
|
|
||||||
|
async def stream_chat(
|
||||||
|
self,
|
||||||
|
messages: list[Message],
|
||||||
|
*,
|
||||||
|
model: str,
|
||||||
|
temperature: float = 0.7,
|
||||||
|
response_format: dict | None = None,
|
||||||
|
) -> AsyncIterator[str]:
|
||||||
|
payload = self._payload(messages, model, temperature, response_format, stream=True)
|
||||||
|
try:
|
||||||
|
async with self._client.stream(
|
||||||
|
"POST", f"{self._base_url}/chat/completions",
|
||||||
|
json=payload, headers=self._headers(),
|
||||||
|
) as response:
|
||||||
|
response.raise_for_status()
|
||||||
|
async for line in response.aiter_lines():
|
||||||
|
if not line or not line.startswith("data:"):
|
||||||
|
continue # keep-alive comments (": ping"), empty lines
|
||||||
|
data = line.removeprefix("data:").strip()
|
||||||
|
if data == "[DONE]":
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
chunk = json.loads(data)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
continue # malformed line — tolerate (ollama-cloud quirks)
|
||||||
|
choices = chunk.get("choices") or []
|
||||||
|
if not choices:
|
||||||
|
continue
|
||||||
|
content = (choices[0].get("delta") or {}).get("content")
|
||||||
|
if content:
|
||||||
|
yield content
|
||||||
|
except httpx.HTTPError as exc:
|
||||||
|
raise self._sanitize(exc) from exc
|
||||||
|
|
||||||
|
async def chat(
|
||||||
|
self,
|
||||||
|
messages: list[Message],
|
||||||
|
*,
|
||||||
|
model: str,
|
||||||
|
temperature: float = 0.7,
|
||||||
|
response_format: dict | None = None,
|
||||||
|
) -> str:
|
||||||
|
payload = self._payload(messages, model, temperature, response_format, stream=False)
|
||||||
|
try:
|
||||||
|
response = await self._client.post(
|
||||||
|
f"{self._base_url}/chat/completions",
|
||||||
|
json=payload, headers=self._headers(),
|
||||||
|
)
|
||||||
|
if response.status_code == 400 and "response_format" in payload:
|
||||||
|
# json_mode auto-degrade (D-020 layer 1): retry once without it
|
||||||
|
payload.pop("response_format")
|
||||||
|
response = await self._client.post(
|
||||||
|
f"{self._base_url}/chat/completions",
|
||||||
|
json=payload, headers=self._headers(),
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
data = response.json()
|
||||||
|
return (data["choices"][0]["message"]["content"]) or ""
|
||||||
|
except httpx.HTTPError as exc:
|
||||||
|
raise self._sanitize(exc) from exc
|
||||||
|
except (KeyError, ValueError) as exc:
|
||||||
|
raise self._sanitize(exc) from exc
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
"""LLM layer types — messages.
|
||||||
|
|
||||||
|
Boundary rule: nothing in llm/ imports from agents/ or api/.
|
||||||
|
|
||||||
|
Providers yield plain str deltas (providers-as-pipes, D-016/D-017);
|
||||||
|
the OpenAI chunk shape lives only at the wire level inside
|
||||||
|
openai_compat.py. ChatDelta/ChoiceDelta were removed in Phase 3 after
|
||||||
|
two verification cycles confirmed no consumers (P2-a finding).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pydantic import BaseModel
|
||||||
|
|
||||||
|
|
||||||
|
class Message(BaseModel):
|
||||||
|
role: str
|
||||||
|
content: str
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
"""FastAPI app factory — lifespan, CORS, health, routers."""
|
||||||
|
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from fastapi.middleware.cors import CORSMiddleware
|
||||||
|
|
||||||
|
from .agents.registry import AgentRegistry, register_builtin_agents
|
||||||
|
from .agents.session import InMemorySessionStore
|
||||||
|
from .api import (
|
||||||
|
assessment_router,
|
||||||
|
chat_router,
|
||||||
|
lab_router,
|
||||||
|
mentor_router,
|
||||||
|
proctor_router,
|
||||||
|
)
|
||||||
|
from .config import Settings
|
||||||
|
from .llm import create_provider
|
||||||
|
|
||||||
|
|
||||||
|
def create_app(settings: Settings | None = None) -> FastAPI:
|
||||||
|
settings = settings or Settings()
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def lifespan(app: FastAPI):
|
||||||
|
# Shared HTTP client pool (D-017): 10s connect / 300s read for cloud TTFT
|
||||||
|
timeout = httpx.Timeout(connect=10.0, read=300.0, write=30.0, pool=10.0)
|
||||||
|
app.state.http_client = httpx.AsyncClient(timeout=timeout)
|
||||||
|
app.state.settings = settings
|
||||||
|
app.state.provider = create_provider(settings, app.state.http_client)
|
||||||
|
app.state.session_store = InMemorySessionStore()
|
||||||
|
app.state.agent_registry = AgentRegistry()
|
||||||
|
register_builtin_agents(app.state.agent_registry)
|
||||||
|
yield
|
||||||
|
await app.state.http_client.aclose()
|
||||||
|
|
||||||
|
app = FastAPI(title="Nextcraft AI Service", version="0.2.0", lifespan=lifespan)
|
||||||
|
|
||||||
|
# A-008: localhost-only CORS, no credentials
|
||||||
|
app.add_middleware(
|
||||||
|
CORSMiddleware,
|
||||||
|
allow_origins=["http://localhost:3000", "http://127.0.0.1:3000"],
|
||||||
|
allow_methods=["GET", "POST", "OPTIONS"],
|
||||||
|
allow_headers=["Content-Type"],
|
||||||
|
allow_credentials=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
@app.get("/health")
|
||||||
|
async def health() -> dict:
|
||||||
|
return {
|
||||||
|
"status": "ok",
|
||||||
|
"provider": settings.provider,
|
||||||
|
"model": settings.model,
|
||||||
|
}
|
||||||
|
|
||||||
|
app.include_router(chat_router)
|
||||||
|
app.include_router(lab_router)
|
||||||
|
app.include_router(assessment_router)
|
||||||
|
app.include_router(mentor_router)
|
||||||
|
app.include_router(proctor_router)
|
||||||
|
return app
|
||||||
|
|
||||||
|
|
||||||
|
app = create_app()
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
"""Prompt library — prompts are code: versioned in git, reviewed like code (D-018).
|
||||||
|
|
||||||
|
Each module exposes a `versioned SYSTEM_PROMPT` constant and a
|
||||||
|
`render_context(learner_context) -> dict` for str.format_map injection.
|
||||||
|
Final personas land in Phases 3-5; these are the initial drafts.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .coach import SYSTEM_PROMPT as COACH_PROMPT
|
||||||
|
from .coach import render_context as render_coach
|
||||||
|
from .mentor import SYSTEM_PROMPT as MENTOR_PROMPT
|
||||||
|
from .mentor import render_context as render_mentor
|
||||||
|
from .tutor import SYSTEM_PROMPT as TUTOR_PROMPT
|
||||||
|
from .tutor import render_context as render_tutor
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"COACH_PROMPT",
|
||||||
|
"MENTOR_PROMPT",
|
||||||
|
"TUTOR_PROMPT",
|
||||||
|
"render_coach",
|
||||||
|
"render_mentor",
|
||||||
|
"render_tutor",
|
||||||
|
]
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
"""Assessor agent prompt — rubric application to artifacts + defenses (REQ-2-008).
|
||||||
|
|
||||||
|
Final persona (Phase 4). Assessor is a rigorous, fair grader: scores each
|
||||||
|
criterion with evidence, cites what the learner did, returns ONLY valid
|
||||||
|
JSON matching the rubric schema.
|
||||||
|
Version: assessor-v2 (final for v0.2).
|
||||||
|
"""
|
||||||
|
|
||||||
|
SYSTEM_PROMPT = """You are Assessor, the grading agent of Nextcraft, an AI-native competency school.
|
||||||
|
Learner: {learner_name}.
|
||||||
|
|
||||||
|
You receive: (a) an artifact evidence excerpt, (b) its defense transcript,
|
||||||
|
and (c) the rubric for the competency. Your job:
|
||||||
|
- Score EVERY rubric criterion from 0-100, justified by evidence you can
|
||||||
|
point to in the artifact or transcript.
|
||||||
|
- Cite what the learner did ("the 3-retry loop in the tool node"), not
|
||||||
|
what they should have done — except in gaps, where the missed work goes.
|
||||||
|
- Strengths: the two strongest evidence points, each one sentence.
|
||||||
|
- Gaps: the two most important missed opportunities, each one sentence.
|
||||||
|
- Verdict: "mastered" | "developing" | "not_yet" — judged against the
|
||||||
|
rubric weights, honestly.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
- Rigorous but fair. A polished artifact with a weak defense is NOT mastery.
|
||||||
|
- Respond with ONLY a valid JSON object matching the provided schema —
|
||||||
|
no markdown fences, no prose outside the JSON."""
|
||||||
|
|
||||||
|
PROMPT_VERSION = "assessor-v2"
|
||||||
|
|
||||||
|
|
||||||
|
def render_context(learner_context) -> dict:
|
||||||
|
return {"learner_name": learner_context.name}
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
"""Coach agent prompt — pacing, motivation, retrieval practice (REQ-2-005).
|
||||||
|
|
||||||
|
Final persona (Phase 3). Coach is an accountability partner: warm,
|
||||||
|
action-oriented, allergic to fluff. Always ends with exactly one next action
|
||||||
|
and weaves retrieval practice into every reply.
|
||||||
|
Version: coach-v2 (final for v0.2).
|
||||||
|
"""
|
||||||
|
|
||||||
|
SYSTEM_PROMPT = """You are Coach, the pacing and motivation agent of Nextcraft,
|
||||||
|
an AI-native competency school.
|
||||||
|
|
||||||
|
Learner: {learner_name}
|
||||||
|
Active stack: {stacks}
|
||||||
|
Current focus: {progress}
|
||||||
|
|
||||||
|
Your style:
|
||||||
|
- Warm, direct, allergic to fluff. Two short paragraphs maximum.
|
||||||
|
- Pacing: name the learner's next concrete step in their current competency.
|
||||||
|
- Motivation: tie effort to their trajectory — what this unlocks, specifically.
|
||||||
|
- Retrieval practice: before introducing anything new, ask the learner to
|
||||||
|
recall or apply something they already covered (one pointed question).
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
- End with exactly ONE clear next action phrased as a command ("Post your
|
||||||
|
plan for the orchestrator retry loop before starting").
|
||||||
|
- Never lecture; never list more than two options.
|
||||||
|
- If the learner is stuck or frustrated, slow down and shrink the step."""
|
||||||
|
|
||||||
|
PROMPT_VERSION = "coach-v2"
|
||||||
|
|
||||||
|
|
||||||
|
def render_context(learner_context) -> dict:
|
||||||
|
stacks = ", ".join(f"{s.title} ({s.percent}%)" for s in learner_context.active_stacks)
|
||||||
|
in_progress = [
|
||||||
|
c for c in learner_context.active_competencies if c.status == "in_progress"
|
||||||
|
]
|
||||||
|
progress = (
|
||||||
|
f"{in_progress[0].title} ({in_progress[0].competency_id})"
|
||||||
|
if in_progress
|
||||||
|
else "no competency currently in progress"
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"learner_name": learner_context.name,
|
||||||
|
"stacks": stacks or "none yet",
|
||||||
|
"progress": progress,
|
||||||
|
}
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
"""Lab agent prompt — in-flow feedback over sandbox telemetry (REQ-2-007).
|
||||||
|
|
||||||
|
Final persona (Phase 4). Lab is a pragmatic build partner: reads the
|
||||||
|
telemetry timeline, names the one most useful adjustment, gives one
|
||||||
|
concrete next step. Scenario-driven; no session chat.
|
||||||
|
Version: lab-v2 (final for v0.2).
|
||||||
|
"""
|
||||||
|
|
||||||
|
SYSTEM_PROMPT = """You are Lab, the in-flow feedback agent watching a learner
|
||||||
|
build in the Nextcraft sandbox.
|
||||||
|
Learner: {learner_name}. Active stack: {stacks}.
|
||||||
|
|
||||||
|
You receive a telemetry timeline of the learner's build session below.
|
||||||
|
Your job, in order:
|
||||||
|
1. Say what the telemetry shows — name the specific events that matter.
|
||||||
|
2. Name the single most useful adjustment (one thing, not a list).
|
||||||
|
3. Give one concrete next step phrased as a command.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
- Be specific to the events you see. If tests failed twice with the same
|
||||||
|
error, say so. If there is a long idle gap, name it.
|
||||||
|
- If the session looks healthy, say so briefly and set the next challenge.
|
||||||
|
- If something looks off (e.g., a huge paste followed by instant success),
|
||||||
|
treat it as a coaching moment, not an accusation — suggest a quick
|
||||||
|
self-check that would prove understanding.
|
||||||
|
- Three short paragraphs maximum. No headers, no bullet lists."""
|
||||||
|
|
||||||
|
PROMPT_VERSION = "lab-v2"
|
||||||
|
|
||||||
|
|
||||||
|
def render_context(learner_context) -> dict:
|
||||||
|
stacks = ", ".join(f"{s.title} ({s.percent}%)" for s in learner_context.active_stacks)
|
||||||
|
return {
|
||||||
|
"learner_name": learner_context.name,
|
||||||
|
"stacks": stacks or "none yet",
|
||||||
|
}
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
"""Mentor agent prompt — long-horizon career narrative (REQ-2-010).
|
||||||
|
|
||||||
|
Final persona (Phase 5). Mentor is a wise career guide: connects today's
|
||||||
|
competencies and artifacts to a long-horizon AI-era trajectory.
|
||||||
|
Version: mentor-v2 (final for v0.2).
|
||||||
|
"""
|
||||||
|
|
||||||
|
SYSTEM_PROMPT = """You are Mentor, the long-horizon career agent of Nextcraft,
|
||||||
|
an AI-native competency school.
|
||||||
|
|
||||||
|
Learner: {learner_name}
|
||||||
|
Active stacks: {stacks}
|
||||||
|
Current focus: {progress}
|
||||||
|
Microcredentials earned: {microcredentials}
|
||||||
|
Recent artifacts: {artifacts}
|
||||||
|
|
||||||
|
Your job: narrate the learner's trajectory in two to three paragraphs:
|
||||||
|
1. Where they are now — what their competency progress and artifacts say
|
||||||
|
about them as a builder (specific, evidence-based).
|
||||||
|
2. What their current stack unlocks next — name the next competency or
|
||||||
|
microcredential worth chasing and the role it points toward.
|
||||||
|
3. How they position in the AI-era labor market — which employer problems
|
||||||
|
their profile already answers.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
- Forward-looking and concrete. No fortune-telling, no flattery.
|
||||||
|
- Reference their artifacts by name at least once.
|
||||||
|
- Write like a mentor writing to one person, not a career-services brochure."""
|
||||||
|
|
||||||
|
PROMPT_VERSION = "mentor-v2"
|
||||||
|
|
||||||
|
|
||||||
|
def render_context(learner_context) -> dict:
|
||||||
|
stacks = ", ".join(f"{s.title} ({s.percent}%)" for s in learner_context.active_stacks)
|
||||||
|
in_progress = [
|
||||||
|
c for c in learner_context.active_competencies if c.status == "in_progress"
|
||||||
|
]
|
||||||
|
progress = (
|
||||||
|
f"{in_progress[0].title} ({in_progress[0].competency_id})"
|
||||||
|
if in_progress
|
||||||
|
else "no competency currently in progress"
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"learner_name": learner_context.name,
|
||||||
|
"stacks": stacks or "none yet",
|
||||||
|
"progress": progress,
|
||||||
|
"microcredentials": str(learner_context.microcredential_count),
|
||||||
|
"artifacts": ", ".join(learner_context.recent_artifacts) or "none yet",
|
||||||
|
}
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
"""Proctor agent prompt — integrity signals with coaching interventions (REQ-2-009).
|
||||||
|
|
||||||
|
Final persona (Phase 5). Proctor is a supportive observer, never punitive:
|
||||||
|
classifies signals, recommends ONE coaching intervention. Assume good
|
||||||
|
faith — most signals have innocent explanations.
|
||||||
|
Version: proctor-v2 (final for v0.2).
|
||||||
|
"""
|
||||||
|
|
||||||
|
SYSTEM_PROMPT = """You are Proctor, the integrity-support agent of Nextcraft,
|
||||||
|
an AI-native competency school.
|
||||||
|
Learner: {learner_name}.
|
||||||
|
|
||||||
|
You receive a telemetry timeline of defense-session events (tab switches,
|
||||||
|
idle gaps, large pastes, focus loss, keystroke bursts). Your job:
|
||||||
|
- Classify EACH notable signal: type (e.g. "context_switch", "idle_gap",
|
||||||
|
"large_paste"), severity ("low" | "medium" | "high"), and a one-sentence
|
||||||
|
note citing the event (timestamps and details).
|
||||||
|
- Recommend exactly ONE supportive coaching intervention for the session
|
||||||
|
overall — never punitive, never accusatory. Frame around helping the
|
||||||
|
learner succeed, e.g. "offer a short break", "invite them to explain
|
||||||
|
the pasted section in their own words".
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
- Assume good faith. Tab switches to documentation are normal engineering.
|
||||||
|
- Idle gaps are often thinking. Only unusual patterns deserve higher severity.
|
||||||
|
- A large paste during an assessment deserves "high" severity but the
|
||||||
|
intervention stays coaching-shaped: verification, not punishment.
|
||||||
|
- Respond with ONLY a valid JSON object matching the provided schema —
|
||||||
|
no markdown fences, no prose outside the JSON."""
|
||||||
|
|
||||||
|
PROMPT_VERSION = "proctor-v2"
|
||||||
|
|
||||||
|
|
||||||
|
def render_context(learner_context) -> dict:
|
||||||
|
return {"learner_name": learner_context.name}
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
"""Tutor agent prompt — concept delivery, Socratic questioning (REQ-2-006).
|
||||||
|
|
||||||
|
Final persona (Phase 3). Tutor is a patient expert teacher: one concept at
|
||||||
|
a time, worked example first, Socratic check before moving on.
|
||||||
|
Version: tutor-v2 (final for v0.2).
|
||||||
|
"""
|
||||||
|
|
||||||
|
SYSTEM_PROMPT = """You are Tutor, the concept-delivery agent of Nextcraft,
|
||||||
|
an AI-native competency school.
|
||||||
|
|
||||||
|
Learner: {learner_name}
|
||||||
|
Active stack: {stacks}
|
||||||
|
Current focus: {progress}
|
||||||
|
|
||||||
|
Your style:
|
||||||
|
- Teach exactly ONE concept per reply. Never more.
|
||||||
|
- Structure: (1) name the concept in one sentence, (2) give a short worked
|
||||||
|
example (5-8 lines) the learner can trace, (3) ask ONE Socratic question
|
||||||
|
that checks whether they can apply it to a slightly different case.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
- Never dump walls of text. If the concept needs more than ~150 words, teach
|
||||||
|
only its first slice and promise the rest after the learner answers.
|
||||||
|
- If the learner's last message reveals a misconception, correct it gently
|
||||||
|
before teaching.
|
||||||
|
- If the learner answers your question, evaluate the answer explicitly
|
||||||
|
(right / partly right / not yet) before the next concept."""
|
||||||
|
|
||||||
|
PROMPT_VERSION = "tutor-v2"
|
||||||
|
|
||||||
|
|
||||||
|
def render_context(learner_context) -> dict:
|
||||||
|
stacks = ", ".join(f"{s.title} ({s.percent}%)" for s in learner_context.active_stacks)
|
||||||
|
in_progress = [
|
||||||
|
c for c in learner_context.active_competencies if c.status == "in_progress"
|
||||||
|
]
|
||||||
|
progress = (
|
||||||
|
f"{in_progress[0].title} ({in_progress[0].competency_id})"
|
||||||
|
if in_progress
|
||||||
|
else "no competency currently in progress"
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"learner_name": learner_context.name,
|
||||||
|
"stacks": stacks or "none yet",
|
||||||
|
"progress": progress,
|
||||||
|
}
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
{
|
||||||
|
"name": "@nextcraft/ai-service",
|
||||||
|
"private": true,
|
||||||
|
"version": "0.2.0",
|
||||||
|
"scripts": {
|
||||||
|
"dev": "bash scripts/dev.sh",
|
||||||
|
"test": "bash scripts/test.sh",
|
||||||
|
"bootstrap": "bash scripts/bootstrap.sh",
|
||||||
|
"lint": "bash scripts/lint.sh"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
[build-system]
|
||||||
|
requires = ["setuptools>=68"]
|
||||||
|
build-backend = "setuptools.build_meta"
|
||||||
|
|
||||||
|
[project]
|
||||||
|
name = "nextcraft-ai-service"
|
||||||
|
version = "0.2.0"
|
||||||
|
description = "Nextcraft AI tutor service — six LLM agents behind a provider-agnostic layer"
|
||||||
|
requires-python = ">=3.11"
|
||||||
|
dependencies = [
|
||||||
|
"fastapi>=0.141,<0.142",
|
||||||
|
"uvicorn>=0.52,<0.53",
|
||||||
|
"pydantic>=2.13,<2.14",
|
||||||
|
"pydantic-settings>=2.15,<2.16",
|
||||||
|
"httpx>=0.28,<0.29",
|
||||||
|
"sse-starlette>=3.4,<3.5",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
dev = [
|
||||||
|
"pytest>=9.1,<10",
|
||||||
|
"pytest-asyncio>=1.4,<2",
|
||||||
|
"ruff>=0.14",
|
||||||
|
]
|
||||||
|
|
||||||
|
[tool.setuptools.packages.find]
|
||||||
|
include = ["ai_service*"]
|
||||||
|
|
||||||
|
[tool.pytest.ini_options]
|
||||||
|
asyncio_mode = "auto"
|
||||||
|
testpaths = ["tests"]
|
||||||
|
|
||||||
|
[tool.ruff]
|
||||||
|
line-length = 100
|
||||||
|
target-version = "py311"
|
||||||
|
|
||||||
|
[tool.ruff.lint]
|
||||||
|
select = ["E", "F", "W", "I", "UP", "B"]
|
||||||
|
# B008: Depends() in argument defaults is the idiomatic FastAPI DI pattern
|
||||||
|
ignore = ["B008"]
|
||||||
Executable
+30
@@ -0,0 +1,30 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Idempotent bootstrap: create venv + install deps.
|
||||||
|
# Handles Debian systems without python3-venv/ensurepip via --without-pip + get-pip.
|
||||||
|
set -euo pipefail
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
APP_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||||
|
VENV="$APP_DIR/.venv"
|
||||||
|
|
||||||
|
mkdir -p "$HOME/.cache/ciagent"
|
||||||
|
|
||||||
|
if [ ! -x "$VENV/bin/python3" ]; then
|
||||||
|
if python3 -m venv "$VENV" 2>/dev/null; then
|
||||||
|
:
|
||||||
|
else
|
||||||
|
# No ensurepip available — create bare venv and bootstrap pip separately.
|
||||||
|
python3 -m venv --without-pip "$VENV"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ ! -x "$VENV/bin/pip" ]; then
|
||||||
|
GET_PIP="$HOME/.cache/ciagent/get-pip.py"
|
||||||
|
if [ ! -f "$GET_PIP" ]; then
|
||||||
|
curl -sSf --max-time 60 https://bootstrap.pypa.io/get-pip.py -o "$GET_PIP"
|
||||||
|
fi
|
||||||
|
"$VENV/bin/python3" "$GET_PIP" --quiet
|
||||||
|
fi
|
||||||
|
|
||||||
|
"$VENV/bin/pip" install --quiet --upgrade pip
|
||||||
|
"$VENV/bin/pip" install --quiet -e "$APP_DIR[dev]"
|
||||||
|
echo "bootstrap complete: $VENV"
|
||||||
Executable
+26
@@ -0,0 +1,26 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Dev server: export secrets (if present) then run uvicorn on :8420.
|
||||||
|
set -euo pipefail
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
APP_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||||
|
REPO_ROOT="$(cd "$APP_DIR/../.." && pwd)"
|
||||||
|
VENV="$APP_DIR/.venv"
|
||||||
|
|
||||||
|
if [ ! -x "$VENV/bin/uvicorn" ]; then
|
||||||
|
echo "venv missing — run scripts/bootstrap.sh first" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
SECRETS="$REPO_ROOT/.ciagent/.env.secrets"
|
||||||
|
if [ -f "$SECRETS" ]; then
|
||||||
|
while IFS='=' read -r key value; do
|
||||||
|
case "$key" in
|
||||||
|
OLLAMA_API_KEY) export AI_OLLAMA_CLOUD_API_KEY="$value" ;;
|
||||||
|
OLLAMA_BASE_URL) export AI_OLLAMA_CLOUD_BASE_URL="$value" ;;
|
||||||
|
AI_TUTOR_MODEL) export AI_MODEL="$value" ;;
|
||||||
|
esac
|
||||||
|
done < "$SECRETS"
|
||||||
|
fi
|
||||||
|
|
||||||
|
cd "$APP_DIR"
|
||||||
|
exec "$VENV/bin/uvicorn" ai_service.main:app --reload --port 8420
|
||||||
Executable
+14
@@ -0,0 +1,14 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Lint: ruff check over the ai-service tree.
|
||||||
|
set -euo pipefail
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
APP_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||||
|
VENV="$APP_DIR/.venv"
|
||||||
|
|
||||||
|
if [ ! -x "$VENV/bin/ruff" ]; then
|
||||||
|
echo "venv missing — run scripts/bootstrap.sh first" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
cd "$APP_DIR"
|
||||||
|
exec "$VENV/bin/ruff" check .
|
||||||
Executable
+14
@@ -0,0 +1,14 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Test runner: pytest via venv — mock provider only, zero network calls.
|
||||||
|
set -euo pipefail
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
APP_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||||
|
VENV="$APP_DIR/.venv"
|
||||||
|
|
||||||
|
if [ ! -x "$VENV/bin/pytest" ]; then
|
||||||
|
echo "venv missing — run scripts/bootstrap.sh first" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
cd "$APP_DIR"
|
||||||
|
exec "$VENV/bin/pytest" -q "$@"
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
"""Assessor agent tests — structured rubric scores (REQ-2-008).
|
||||||
|
|
||||||
|
The Assessor is the structured-output showcase: tests use ScriptedJSONProvider
|
||||||
|
for valid payloads and exercise the 4-layer defense failure modes.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ai_service.agents.assessor import AssessorAgent, RubricScore
|
||||||
|
from ai_service.agents.structured import StructuredOutputError
|
||||||
|
from ai_service.config import Settings
|
||||||
|
from ai_service.corpus.artifacts import (
|
||||||
|
get_artifact_bundle,
|
||||||
|
get_transcript_for_artifact,
|
||||||
|
render_rubric,
|
||||||
|
)
|
||||||
|
from ai_service.corpus.learner_context import get_learner_context
|
||||||
|
from ai_service.llm.mock import MockProvider, ScriptedJSONProvider
|
||||||
|
|
||||||
|
VALID_SCORE = {
|
||||||
|
"rubric_id": "rubric-orchestration-c002",
|
||||||
|
"artifact_id": "art-eval-research-assistant",
|
||||||
|
"competency_id": "stack-orchestration-c002",
|
||||||
|
"scores": [
|
||||||
|
{"criterion_id": "rc-architecture", "name": "Agent architecture soundness",
|
||||||
|
"score": 92, "evidence": "Explicit state schema with planner-only write access"},
|
||||||
|
{"criterion_id": "rc-communication", "name": "Inter-agent communication design",
|
||||||
|
"score": 88, "evidence": "Typed ToolMessage responses with retry flags"},
|
||||||
|
{"criterion_id": "rc-reliability", "name": "Reliability engineering",
|
||||||
|
"score": 85, "evidence": "3-retry loop with degradation path"},
|
||||||
|
{"criterion_id": "rc-process", "name": "Process trace quality",
|
||||||
|
"score": 90, "evidence": "Iterative saves with passing test checkpoints"},
|
||||||
|
],
|
||||||
|
"strengths": ["Clean state boundaries", "Failure-aware tool wrapping"],
|
||||||
|
"gaps": ["No reviewer node yet", "Graph diagram only in README"],
|
||||||
|
"verdict": "mastered",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def make_assessor(provider=None) -> AssessorAgent:
|
||||||
|
return AssessorAgent(provider or MockProvider(), Settings(provider="mock"))
|
||||||
|
|
||||||
|
|
||||||
|
def get_bundle(artifact_id="art-eval-research-assistant"):
|
||||||
|
bundle = get_artifact_bundle(artifact_id)
|
||||||
|
assert bundle is not None
|
||||||
|
return bundle
|
||||||
|
|
||||||
|
|
||||||
|
async def test_evaluate_returns_validated_rubric_score():
|
||||||
|
provider = ScriptedJSONProvider(VALID_SCORE)
|
||||||
|
assessor = make_assessor(provider)
|
||||||
|
artifact, rubric = get_bundle()
|
||||||
|
transcript = get_transcript_for_artifact(artifact.artifact_id)
|
||||||
|
result = await assessor.evaluate(artifact, rubric, transcript)
|
||||||
|
assert isinstance(result, RubricScore)
|
||||||
|
assert result.verdict == "mastered"
|
||||||
|
assert len(result.scores) == 4
|
||||||
|
assert result.weighted_total(rubric) == pytest.approx(
|
||||||
|
92 * 0.3 + 88 * 0.3 + 85 * 0.25 + 90 * 0.15
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_evaluate_rejects_invalid_schema_after_retry():
|
||||||
|
"""Plain MockProvider returns non-rubric JSON → 4-layer defense exhausts
|
||||||
|
its single retry and raises StructuredOutputError."""
|
||||||
|
assessor = make_assessor(MockProvider())
|
||||||
|
artifact, rubric = get_bundle()
|
||||||
|
transcript = get_transcript_for_artifact(artifact.artifact_id)
|
||||||
|
with pytest.raises(StructuredOutputError):
|
||||||
|
await assessor.evaluate(artifact, rubric, transcript)
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_evaluation_input_carries_all_inputs():
|
||||||
|
assessor = make_assessor()
|
||||||
|
artifact, rubric = get_bundle()
|
||||||
|
transcript = get_transcript_for_artifact(artifact.artifact_id)
|
||||||
|
text = assessor.build_evaluation_input(artifact, rubric, transcript)
|
||||||
|
assert artifact.name in text
|
||||||
|
assert artifact.evidence_excerpt in text
|
||||||
|
assert "rc-architecture" in text # rubric rendered
|
||||||
|
assert "examiner:" in text # transcript rendered
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_evaluation_input_without_transcript():
|
||||||
|
assessor = make_assessor()
|
||||||
|
artifact, rubric = get_bundle()
|
||||||
|
text = assessor.build_evaluation_input(artifact, rubric, None)
|
||||||
|
assert artifact.name in text
|
||||||
|
assert "examiner:" not in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_system_prompt_names_assessor_persona():
|
||||||
|
prompt = make_assessor().system_prompt(get_learner_context())
|
||||||
|
assert "Assessor" in prompt
|
||||||
|
assert "ONLY" in prompt # JSON-only instruction
|
||||||
|
|
||||||
|
|
||||||
|
def test_rubric_render_in_prompt_is_complete():
|
||||||
|
"""The rubric passed to the model lists every criterion (fair grading)."""
|
||||||
|
artifact, rubric = get_bundle()
|
||||||
|
text = render_rubric(rubric)
|
||||||
|
assert text.count("rc-") == len(rubric.criteria)
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
"""BaseAgent contract tests — stub agent + mock provider."""
|
||||||
|
|
||||||
|
|
||||||
|
from ai_service.agents.base import BaseAgent
|
||||||
|
from ai_service.config import Settings
|
||||||
|
from ai_service.corpus.learner_context import get_learner_context
|
||||||
|
from ai_service.llm.mock import MockProvider
|
||||||
|
from ai_service.llm.types import Message
|
||||||
|
|
||||||
|
|
||||||
|
class StubAgent(BaseAgent):
|
||||||
|
name = "stub"
|
||||||
|
|
||||||
|
def system_prompt(self, learner_context=None) -> str:
|
||||||
|
return "You are Stub. Answer briefly."
|
||||||
|
|
||||||
|
|
||||||
|
def make_agent() -> StubAgent:
|
||||||
|
return StubAgent(MockProvider(), Settings(provider="mock"))
|
||||||
|
|
||||||
|
|
||||||
|
async def test_build_messages_composition():
|
||||||
|
agent = make_agent()
|
||||||
|
history = [Message(role="user", content="earlier"), Message(role="assistant", content="reply")]
|
||||||
|
messages = agent.build_messages(history, "new question")
|
||||||
|
assert messages[0].role == "system"
|
||||||
|
assert messages[0].content == "You are Stub. Answer briefly."
|
||||||
|
assert [m.content for m in messages[1:]] == ["earlier", "reply", "new question"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_stream_reply_yields_deltas():
|
||||||
|
agent = make_agent()
|
||||||
|
tokens = [t async for t in agent.stream_reply(user_input="hello")]
|
||||||
|
assert len(tokens) >= 1
|
||||||
|
assert all(isinstance(t, str) for t in tokens)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_stream_reply_with_history_and_context():
|
||||||
|
agent = make_agent()
|
||||||
|
ctx = get_learner_context()
|
||||||
|
history = [Message(role="user", content="earlier"), Message(role="assistant", content="ok")]
|
||||||
|
tokens = [t async for t in agent.stream_reply(history, "next", ctx)]
|
||||||
|
assert tokens
|
||||||
|
|
||||||
|
|
||||||
|
async def test_structured_reply_requires_schema():
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
agent = make_agent()
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
await agent.structured_reply(user_input="x", schema=None)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_name_defaults():
|
||||||
|
assert make_agent().name == "stub"
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
"""Coach agent tests — persona, message assembly, streaming (REQ-2-005)."""
|
||||||
|
|
||||||
|
from ai_service.agents.coach import CoachAgent
|
||||||
|
from ai_service.config import Settings
|
||||||
|
from ai_service.corpus.learner_context import get_learner_context
|
||||||
|
from ai_service.llm.mock import MockProvider
|
||||||
|
from ai_service.llm.types import Message
|
||||||
|
|
||||||
|
|
||||||
|
def make_coach() -> CoachAgent:
|
||||||
|
return CoachAgent(MockProvider(), Settings(provider="mock"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_system_prompt_includes_learner_context():
|
||||||
|
coach = make_coach()
|
||||||
|
prompt = coach.system_prompt(get_learner_context())
|
||||||
|
assert "Alex Rivera" in prompt
|
||||||
|
assert "AI Orchestration Engineer (62%)" in prompt
|
||||||
|
assert "retrieval practice" in prompt.lower()
|
||||||
|
assert "one clear next action" in prompt.lower()
|
||||||
|
|
||||||
|
|
||||||
|
def test_system_prompt_marks_current_focus():
|
||||||
|
coach = make_coach()
|
||||||
|
prompt = coach.system_prompt(get_learner_context())
|
||||||
|
assert "Multi-agent communication patterns" in prompt
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_messages_system_history_user():
|
||||||
|
coach = make_coach()
|
||||||
|
history = [
|
||||||
|
Message(role="user", content="earlier"),
|
||||||
|
Message(role="assistant", content="reply"),
|
||||||
|
]
|
||||||
|
messages = coach.build_messages(history, "what next?", get_learner_context())
|
||||||
|
assert messages[0].role == "system"
|
||||||
|
assert "Coach" in messages[0].content
|
||||||
|
assert [m.content for m in messages[1:]] == ["earlier", "reply", "what next?"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_stream_reply_yields_deltas():
|
||||||
|
coach = make_coach()
|
||||||
|
ctx = get_learner_context()
|
||||||
|
tokens = [t async for t in coach.stream_reply(user_input="hello", learner_context=ctx)]
|
||||||
|
assert tokens
|
||||||
|
assert all(isinstance(t, str) for t in tokens)
|
||||||
|
|
||||||
|
|
||||||
|
def test_agent_name():
|
||||||
|
assert make_coach().name == "coach"
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
"""Learner context corpus tests — D-021 ID alignment + prompt rendering."""
|
||||||
|
|
||||||
|
from ai_service.corpus.learner_context import (
|
||||||
|
LEARNER_CONTEXTS,
|
||||||
|
get_learner_context,
|
||||||
|
)
|
||||||
|
from ai_service.prompts.assessor import render_context as render_assessor
|
||||||
|
from ai_service.prompts.coach import SYSTEM_PROMPT as COACH_PROMPT
|
||||||
|
from ai_service.prompts.coach import render_context as render_coach
|
||||||
|
from ai_service.prompts.lab import SYSTEM_PROMPT as LAB_PROMPT
|
||||||
|
from ai_service.prompts.lab import render_context as render_lab
|
||||||
|
from ai_service.prompts.mentor import SYSTEM_PROMPT as MENTOR_PROMPT
|
||||||
|
from ai_service.prompts.mentor import render_context as render_mentor
|
||||||
|
from ai_service.prompts.proctor import SYSTEM_PROMPT as PROCTOR_PROMPT
|
||||||
|
from ai_service.prompts.proctor import render_context as render_proctor
|
||||||
|
from ai_service.prompts.tutor import SYSTEM_PROMPT as TUTOR_PROMPT
|
||||||
|
from ai_service.prompts.tutor import render_context as render_tutor
|
||||||
|
|
||||||
|
|
||||||
|
def test_default_learner_resolves():
|
||||||
|
ctx = get_learner_context()
|
||||||
|
assert ctx.learner_id == "learner-001"
|
||||||
|
assert ctx.name == "Alex Rivera"
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_learner_falls_back_to_default():
|
||||||
|
assert get_learner_context("nobody").learner_id == "learner-001"
|
||||||
|
|
||||||
|
|
||||||
|
def test_ids_align_with_ts_mock_data():
|
||||||
|
# D-021: identical ID strings to packages/mock-data (learner-progress.ts)
|
||||||
|
ctx = get_learner_context("learner-001")
|
||||||
|
stack_ids = {s.stack_id for s in ctx.active_stacks}
|
||||||
|
assert {"stack-orchestration", "stack-safety"} <= stack_ids
|
||||||
|
competency_ids = {c.competency_id for c in ctx.active_competencies}
|
||||||
|
assert "stack-orchestration-c001" in competency_ids
|
||||||
|
|
||||||
|
|
||||||
|
def test_all_prompt_modules_render_without_keyerror():
|
||||||
|
ctx = get_learner_context()
|
||||||
|
for render in (render_coach, render_tutor, render_mentor, render_assessor):
|
||||||
|
values = render(ctx)
|
||||||
|
assert isinstance(values, dict)
|
||||||
|
assert "learner_name" in values
|
||||||
|
assert values["learner_name"] == "Alex Rivera"
|
||||||
|
|
||||||
|
|
||||||
|
def test_prompts_format_map_with_rendered_context():
|
||||||
|
ctx = get_learner_context()
|
||||||
|
for prompt, render in (
|
||||||
|
(COACH_PROMPT, render_coach),
|
||||||
|
(TUTOR_PROMPT, render_tutor),
|
||||||
|
(MENTOR_PROMPT, render_mentor),
|
||||||
|
):
|
||||||
|
rendered = prompt.format_map(render(ctx))
|
||||||
|
assert "Alex Rivera" in rendered
|
||||||
|
assert "{" not in rendered # all placeholders filled
|
||||||
|
|
||||||
|
|
||||||
|
def test_lab_and_proctor_prompts_render():
|
||||||
|
ctx = get_learner_context()
|
||||||
|
# Each module renders through its OWN render_context (its own placeholders).
|
||||||
|
assert "Alex Rivera" in LAB_PROMPT.format_map(render_lab(ctx))
|
||||||
|
assert "Alex Rivera" in PROCTOR_PROMPT.format_map(render_proctor(ctx))
|
||||||
|
for prompt, render in ((LAB_PROMPT, render_lab), (PROCTOR_PROMPT, render_proctor)):
|
||||||
|
rendered = prompt.format_map(render(ctx))
|
||||||
|
assert "{" not in rendered # all placeholders filled
|
||||||
|
|
||||||
|
|
||||||
|
def test_two_seed_learners_exist():
|
||||||
|
assert set(LEARNER_CONTEXTS) == {"learner-001", "learner-002"}
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
"""Telemetry + artifacts corpus tests (REQ-2-007/008 inputs, D-021)."""
|
||||||
|
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ai_service.corpus.artifacts import (
|
||||||
|
ARTIFACTS,
|
||||||
|
RUBRICS,
|
||||||
|
get_artifact_bundle,
|
||||||
|
get_transcript_for_artifact,
|
||||||
|
render_rubric,
|
||||||
|
render_transcript,
|
||||||
|
rubric_for_competency,
|
||||||
|
)
|
||||||
|
from ai_service.corpus.telemetry import (
|
||||||
|
LAB_SCENARIOS,
|
||||||
|
get_lab_scenario,
|
||||||
|
summarize_scenario,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_lab_scenarios_addressable_by_id():
|
||||||
|
for scenario_id in (
|
||||||
|
"lab-scenario-strong",
|
||||||
|
"lab-scenario-struggling",
|
||||||
|
"lab-scenario-flagged",
|
||||||
|
):
|
||||||
|
scenario = get_lab_scenario(scenario_id)
|
||||||
|
assert scenario is not None
|
||||||
|
assert scenario.scenario_id == scenario_id
|
||||||
|
|
||||||
|
|
||||||
|
def test_lab_scenarios_have_distinct_event_profiles():
|
||||||
|
strong = get_lab_scenario("lab-scenario-strong")
|
||||||
|
struggling = get_lab_scenario("lab-scenario-struggling")
|
||||||
|
kinds = lambda s: {e.kind for e in s.events} # noqa: E731
|
||||||
|
assert "test_pass" in kinds(strong)
|
||||||
|
assert "test_fail" in kinds(struggling)
|
||||||
|
assert "idle" in kinds(struggling)
|
||||||
|
assert "paste" in kinds(get_lab_scenario("lab-scenario-flagged"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_summarize_scenario_mentions_events():
|
||||||
|
text = summarize_scenario(get_lab_scenario("lab-scenario-struggling"))
|
||||||
|
assert "test_fail" in text
|
||||||
|
assert "ImportError" in text
|
||||||
|
assert "stack-orchestration-c002" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_scenario_returns_none():
|
||||||
|
assert get_lab_scenario("lab-scenario-ghost") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_artifacts_and_rubrics_resolve():
|
||||||
|
bundle = get_artifact_bundle("art-eval-research-assistant")
|
||||||
|
assert bundle is not None
|
||||||
|
artifact, rubric = bundle
|
||||||
|
assert artifact.competency_id == "stack-orchestration-c002"
|
||||||
|
assert rubric.rubric_id == "rubric-orchestration-c002"
|
||||||
|
assert len(rubric.criteria) == 4
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_artifact_returns_none():
|
||||||
|
assert get_artifact_bundle("art-eval-ghost") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_transcripts_pair_with_artifacts():
|
||||||
|
for artifact_id in ARTIFACTS:
|
||||||
|
transcript = get_transcript_for_artifact(artifact_id)
|
||||||
|
assert transcript is not None
|
||||||
|
assert transcript.artifact_id == artifact_id
|
||||||
|
assert len(transcript.turns) >= 4
|
||||||
|
|
||||||
|
|
||||||
|
def test_rubric_render_mentions_all_criteria():
|
||||||
|
rubric = rubric_for_competency("stack-orchestration-c002")
|
||||||
|
text = render_rubric(rubric)
|
||||||
|
for criterion in rubric.criteria:
|
||||||
|
assert criterion.criterion_id in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_transcript_render_has_both_speakers():
|
||||||
|
transcript = get_transcript_for_artifact("art-eval-rag-dashboard")
|
||||||
|
text = render_transcript(transcript)
|
||||||
|
assert "examiner:" in text
|
||||||
|
assert "learner:" in text
|
||||||
|
|
||||||
|
|
||||||
|
_TS_SOURCE_CANDIDATES = [
|
||||||
|
Path(__file__).resolve().parents[4] / "packages" / "mock-data" / "ai-scenarios.ts",
|
||||||
|
Path(__file__).resolve().parents[2] / "ai-scenarios.ts",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _ts_source() -> Path:
|
||||||
|
for candidate in _TS_SOURCE_CANDIDATES:
|
||||||
|
if candidate.exists():
|
||||||
|
return candidate
|
||||||
|
pytest.skip("ai-scenarios.ts not found in this checkout layout")
|
||||||
|
|
||||||
|
|
||||||
|
def test_corpus_ids_align_with_ts_mock_data():
|
||||||
|
"""D-021: Python corpus IDs string-identical to ai-scenarios.ts."""
|
||||||
|
content = _ts_source().read_text()
|
||||||
|
ts_ids = re.findall(r"id:\s*'([^']+)'", content)
|
||||||
|
ts_scenarios = ts_ids[: len(LAB_SCENARIOS)]
|
||||||
|
ts_artifacts = ts_ids[len(LAB_SCENARIOS):]
|
||||||
|
assert sorted(ts_scenarios) == sorted(LAB_SCENARIOS), (
|
||||||
|
f"scenario IDs drifted: py={sorted(LAB_SCENARIOS)} ts={sorted(ts_scenarios)}"
|
||||||
|
)
|
||||||
|
assert sorted(ts_artifacts) == sorted(ARTIFACTS), (
|
||||||
|
f"artifact IDs drifted: py={sorted(ARTIFACTS)} ts={sorted(ts_artifacts)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_rubric_weights_sum_to_one():
|
||||||
|
"""Every rubric's criteria weights must sum to exactly 1.0."""
|
||||||
|
for rubric in RUBRICS.values():
|
||||||
|
total = sum(c.weight for c in rubric.criteria)
|
||||||
|
assert total == pytest.approx(1.0), (
|
||||||
|
f"{rubric.rubric_id} weights sum to {total}, expected 1.0"
|
||||||
|
)
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
"""Lab agent tests — scenario-driven streaming feedback (REQ-2-007)."""
|
||||||
|
|
||||||
|
from ai_service.agents.lab import LabAgent
|
||||||
|
from ai_service.config import Settings
|
||||||
|
from ai_service.corpus.learner_context import get_learner_context
|
||||||
|
from ai_service.corpus.telemetry import get_lab_scenario, summarize_scenario
|
||||||
|
from ai_service.llm.mock import MockProvider
|
||||||
|
|
||||||
|
|
||||||
|
def make_lab() -> LabAgent:
|
||||||
|
return LabAgent(MockProvider(), Settings(provider="mock"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_system_prompt_names_lab_persona():
|
||||||
|
prompt = make_lab().system_prompt(get_learner_context())
|
||||||
|
assert "Lab" in prompt
|
||||||
|
assert "Alex Rivera" in prompt
|
||||||
|
assert "telemetry" in prompt.lower()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_stream_feedback_mentions_scenario_events():
|
||||||
|
"""Mock-scripted: feedback text derives from scenario timeline input —
|
||||||
|
distinct scenarios produce distinct (deterministic) replies."""
|
||||||
|
lab = make_lab()
|
||||||
|
ctx = get_learner_context()
|
||||||
|
strong = get_lab_scenario("lab-scenario-strong")
|
||||||
|
struggling = get_lab_scenario("lab-scenario-struggling")
|
||||||
|
|
||||||
|
strong_reply = "".join([t async for t in lab.stream_feedback(strong, ctx)])
|
||||||
|
struggling_reply = "".join([t async for t in lab.stream_feedback(struggling, ctx)])
|
||||||
|
assert strong_reply
|
||||||
|
assert strong_reply != struggling_reply # scenario-driven, not canned
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_evaluation_messages_carry_timeline():
|
||||||
|
lab = make_lab()
|
||||||
|
scenario = get_lab_scenario("lab-scenario-flagged")
|
||||||
|
timeline = summarize_scenario(scenario)
|
||||||
|
messages = lab.build_messages(None, timeline, get_learner_context())
|
||||||
|
assert messages[0].role == "system"
|
||||||
|
assert "paste" in messages[-1].content
|
||||||
|
assert "2,400 chars" in messages[-1].content
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
"""Mentor agent tests — career narrative, session-backed (REQ-2-010)."""
|
||||||
|
|
||||||
|
from ai_service.agents.mentor import MentorAgent
|
||||||
|
from ai_service.config import Settings
|
||||||
|
from ai_service.corpus.learner_context import get_learner_context
|
||||||
|
from ai_service.llm.mock import MockProvider
|
||||||
|
from ai_service.llm.types import Message
|
||||||
|
|
||||||
|
|
||||||
|
def make_mentor() -> MentorAgent:
|
||||||
|
return MentorAgent(MockProvider(), Settings(provider="mock"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_system_prompt_carries_full_learner_context():
|
||||||
|
prompt = make_mentor().system_prompt(get_learner_context())
|
||||||
|
assert "Alex Rivera" in prompt
|
||||||
|
assert "AI Orchestration Engineer (62%)" in prompt
|
||||||
|
assert "Multi-agent research assistant" in prompt # artifacts by name
|
||||||
|
assert "4" in prompt # microcredential count
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_messages_system_history_user():
|
||||||
|
mentor = make_mentor()
|
||||||
|
history = [Message(role="user", content="what next?"),
|
||||||
|
Message(role="assistant", content="trajectory...")]
|
||||||
|
messages = mentor.build_messages(history, "tell me more", get_learner_context())
|
||||||
|
assert messages[0].role == "system"
|
||||||
|
assert "Mentor" in messages[0].content
|
||||||
|
assert [m.content for m in messages[1:]] == ["what next?", "trajectory...", "tell me more"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_stream_reply_mentions_learner_context_in_output():
|
||||||
|
"""Mock-scripted: narrative derives from context-injected messages —
|
||||||
|
different learner contexts produce distinct (deterministic) replies."""
|
||||||
|
mentor = make_mentor()
|
||||||
|
alex = get_learner_context("learner-001")
|
||||||
|
priya = get_learner_context("learner-002")
|
||||||
|
alex_reply = "".join(
|
||||||
|
[t async for t in mentor.stream_reply(user_input="narrate", learner_context=alex)]
|
||||||
|
)
|
||||||
|
priya_reply = "".join(
|
||||||
|
[t async for t in mentor.stream_reply(user_input="narrate", learner_context=priya)]
|
||||||
|
)
|
||||||
|
assert alex_reply
|
||||||
|
assert alex_reply != priya_reply
|
||||||
|
|
||||||
|
|
||||||
|
def test_agent_name():
|
||||||
|
assert make_mentor().name == "mentor"
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
"""Proctor agent tests — structured integrity signals (REQ-2-009)."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ai_service.agents.proctor import ProctorAgent, ProctorAssessment
|
||||||
|
from ai_service.agents.structured import StructuredOutputError
|
||||||
|
from ai_service.config import Settings
|
||||||
|
from ai_service.corpus.learner_context import get_learner_context
|
||||||
|
from ai_service.corpus.telemetry import (
|
||||||
|
PROCTOR_SCENARIOS,
|
||||||
|
get_proctor_scenario,
|
||||||
|
summarize_proctor_scenario,
|
||||||
|
)
|
||||||
|
from ai_service.llm.mock import MockProvider, ScriptedJSONProvider
|
||||||
|
|
||||||
|
VALID_ASSESSMENT = {
|
||||||
|
"scenario_id": "proctor-scenario-distracted",
|
||||||
|
"signals": [
|
||||||
|
{"signal_type": "context_switch", "severity": "low",
|
||||||
|
"note": "Tab switch to docs at t+120s — normal engineering behavior"},
|
||||||
|
{"signal_type": "idle_gap", "severity": "medium",
|
||||||
|
"note": "5-minute idle at t+300s followed by more tab switches"},
|
||||||
|
],
|
||||||
|
"intervention": "Offer a short break and ask the learner to restate their answer plan",
|
||||||
|
"summary": "Distracted but explainable session; coach the focus pattern, don't flag it",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def make_proctor(provider=None) -> ProctorAgent:
|
||||||
|
return ProctorAgent(provider or MockProvider(), Settings(provider="mock"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_proctor_scenarios_addressable_and_distinct_type():
|
||||||
|
healthy = get_proctor_scenario("proctor-scenario-healthy")
|
||||||
|
flagged = get_proctor_scenario("proctor-scenario-flagged")
|
||||||
|
assert healthy is not None and flagged is not None
|
||||||
|
kinds = lambda s: {e.kind for e in s.events} # noqa: E731
|
||||||
|
assert "tab_switch" in kinds(get_proctor_scenario("proctor-scenario-distracted"))
|
||||||
|
assert "paste_large" in kinds(flagged)
|
||||||
|
assert not kinds(healthy) & {"tab_switch", "paste_large", "focus_lost"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_proctor_scenario_none():
|
||||||
|
assert get_proctor_scenario("proctor-scenario-ghost") is None
|
||||||
|
|
||||||
|
|
||||||
|
async def test_assess_returns_validated_signals():
|
||||||
|
provider = ScriptedJSONProvider(VALID_ASSESSMENT)
|
||||||
|
proctor = make_proctor(provider)
|
||||||
|
scenario = get_proctor_scenario("proctor-scenario-distracted")
|
||||||
|
result = await proctor.assess(scenario, get_learner_context())
|
||||||
|
assert isinstance(result, ProctorAssessment)
|
||||||
|
assert len(result.signals) == 2
|
||||||
|
assert result.signals[0].severity == "low"
|
||||||
|
assert "break" in result.intervention.lower()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_assess_rejects_invalid_after_retry():
|
||||||
|
proctor = make_proctor(MockProvider()) # non-schema JSON
|
||||||
|
scenario = get_proctor_scenario("proctor-scenario-healthy")
|
||||||
|
with pytest.raises(StructuredOutputError):
|
||||||
|
await proctor.assess(scenario, get_learner_context())
|
||||||
|
|
||||||
|
|
||||||
|
def test_system_prompt_is_coaching_not_punitive():
|
||||||
|
prompt = make_proctor().system_prompt(get_learner_context())
|
||||||
|
assert "never punitive" in prompt.lower()
|
||||||
|
assert "good faith" in prompt.lower()
|
||||||
|
assert "ONLY" in prompt # JSON-only instruction
|
||||||
|
|
||||||
|
|
||||||
|
def test_timeline_summary_carries_events():
|
||||||
|
scenario = get_proctor_scenario("proctor-scenario-flagged")
|
||||||
|
text = summarize_proctor_scenario(scenario)
|
||||||
|
assert "paste_large" in text
|
||||||
|
assert "3,100 chars" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_all_three_proctor_scenarios_exist():
|
||||||
|
assert set(PROCTOR_SCENARIOS) == {
|
||||||
|
"proctor-scenario-healthy",
|
||||||
|
"proctor-scenario-distracted",
|
||||||
|
"proctor-scenario-flagged",
|
||||||
|
}
|
||||||
@@ -0,0 +1,132 @@
|
|||||||
|
"""Agent registry tests — register/get round-trip, error paths (G-4), builtins."""
|
||||||
|
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ai_service.agents.base import BaseAgent
|
||||||
|
from ai_service.agents.coach import CoachAgent
|
||||||
|
from ai_service.agents.registry import (
|
||||||
|
AgentRegistry,
|
||||||
|
DuplicateAgentError,
|
||||||
|
UnknownAgentError,
|
||||||
|
register_builtin_agents,
|
||||||
|
)
|
||||||
|
from ai_service.agents.tutor import TutorAgent
|
||||||
|
from ai_service.config import Settings
|
||||||
|
from ai_service.llm.mock import MockProvider
|
||||||
|
|
||||||
|
|
||||||
|
class DummyAgent(BaseAgent):
|
||||||
|
name = "dummy"
|
||||||
|
|
||||||
|
def system_prompt(self, learner_context=None) -> str:
|
||||||
|
return "dummy"
|
||||||
|
|
||||||
|
|
||||||
|
def make_factory():
|
||||||
|
def factory(provider, settings):
|
||||||
|
return DummyAgent(provider, settings)
|
||||||
|
return factory
|
||||||
|
|
||||||
|
|
||||||
|
def test_register_and_get():
|
||||||
|
registry = AgentRegistry()
|
||||||
|
registry.register("dummy", make_factory())
|
||||||
|
agent = registry.get(MockProvider(), Settings(provider="mock"), "dummy")
|
||||||
|
assert isinstance(agent, DummyAgent)
|
||||||
|
assert agent.name == "dummy"
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_agent_raises():
|
||||||
|
registry = AgentRegistry()
|
||||||
|
with pytest.raises(UnknownAgentError):
|
||||||
|
registry.get(MockProvider(), Settings(provider="mock"), "ghost")
|
||||||
|
|
||||||
|
|
||||||
|
def test_duplicate_registration_raises():
|
||||||
|
registry = AgentRegistry()
|
||||||
|
registry.register("dummy", make_factory())
|
||||||
|
with pytest.raises(DuplicateAgentError):
|
||||||
|
registry.register("dummy", make_factory())
|
||||||
|
|
||||||
|
|
||||||
|
def test_names_sorted():
|
||||||
|
registry = AgentRegistry()
|
||||||
|
registry.register("zeta", make_factory())
|
||||||
|
registry.register("alpha", make_factory())
|
||||||
|
assert registry.names() == ["alpha", "zeta"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_builtin_agents_register_and_resolve():
|
||||||
|
registry = AgentRegistry()
|
||||||
|
register_builtin_agents(registry)
|
||||||
|
assert set(registry.names()) >= {"coach", "tutor"}
|
||||||
|
settings = Settings(provider="mock")
|
||||||
|
coach = registry.get(MockProvider(), settings, "coach")
|
||||||
|
tutor = registry.get(MockProvider(), settings, "tutor")
|
||||||
|
assert isinstance(coach, CoachAgent)
|
||||||
|
assert isinstance(tutor, TutorAgent)
|
||||||
|
|
||||||
|
|
||||||
|
def test_lab_and_assessor_resolve_via_registry():
|
||||||
|
"""Phase 4: lab + assessor registered centrally (Task 4-3-01)."""
|
||||||
|
from ai_service.agents.assessor import AssessorAgent
|
||||||
|
from ai_service.agents.lab import LabAgent
|
||||||
|
|
||||||
|
registry = AgentRegistry()
|
||||||
|
register_builtin_agents(registry)
|
||||||
|
assert {"lab", "assessor"} <= set(registry.names())
|
||||||
|
settings = Settings(provider="mock")
|
||||||
|
lab = registry.get(MockProvider(), settings, "lab")
|
||||||
|
assessor = registry.get(MockProvider(), settings, "assessor")
|
||||||
|
assert isinstance(lab, LabAgent)
|
||||||
|
assert isinstance(assessor, AssessorAgent)
|
||||||
|
|
||||||
|
|
||||||
|
def test_proctor_and_mentor_resolve_via_registry():
|
||||||
|
"""Phase 5: proctor + mentor registered centrally (Tasks 5-1-02/5-2-01)."""
|
||||||
|
from ai_service.agents.mentor import MentorAgent
|
||||||
|
from ai_service.agents.proctor import ProctorAgent
|
||||||
|
|
||||||
|
registry = AgentRegistry()
|
||||||
|
register_builtin_agents(registry)
|
||||||
|
settings = Settings(provider="mock")
|
||||||
|
proctor = registry.get(MockProvider(), settings, "proctor")
|
||||||
|
mentor = registry.get(MockProvider(), settings, "mentor")
|
||||||
|
assert isinstance(proctor, ProctorAgent)
|
||||||
|
assert isinstance(mentor, MentorAgent)
|
||||||
|
|
||||||
|
|
||||||
|
def test_registry_resolves_all_six_agents():
|
||||||
|
"""Must-Have (Phase 5): the full roster — coach/tutor/lab/assessor/proctor/mentor."""
|
||||||
|
from ai_service.agents.assessor import AssessorAgent
|
||||||
|
from ai_service.agents.coach import CoachAgent
|
||||||
|
from ai_service.agents.lab import LabAgent
|
||||||
|
from ai_service.agents.mentor import MentorAgent
|
||||||
|
from ai_service.agents.proctor import ProctorAgent
|
||||||
|
from ai_service.agents.tutor import TutorAgent
|
||||||
|
|
||||||
|
registry = AgentRegistry()
|
||||||
|
register_builtin_agents(registry)
|
||||||
|
assert registry.names() == ["assessor", "coach", "lab", "mentor", "proctor", "tutor"]
|
||||||
|
settings = Settings(provider="mock")
|
||||||
|
expected = {
|
||||||
|
"coach": CoachAgent,
|
||||||
|
"tutor": TutorAgent,
|
||||||
|
"lab": LabAgent,
|
||||||
|
"assessor": AssessorAgent,
|
||||||
|
"proctor": ProctorAgent,
|
||||||
|
"mentor": MentorAgent,
|
||||||
|
}
|
||||||
|
for name, cls in expected.items():
|
||||||
|
agent = registry.get(MockProvider(), settings, name)
|
||||||
|
assert isinstance(agent, cls), f"{name} resolved to {type(agent).__name__}"
|
||||||
|
assert agent.name == name
|
||||||
|
|
||||||
|
|
||||||
|
def test_builtin_registration_is_idempotent_safe():
|
||||||
|
"""Duplicate registration raises — builtin bootstrap must be called once."""
|
||||||
|
registry = AgentRegistry()
|
||||||
|
register_builtin_agents(registry)
|
||||||
|
with pytest.raises(DuplicateAgentError):
|
||||||
|
register_builtin_agents(registry)
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
"""SessionStore tests — create/append/window/LRU/agent scoping (D-019)."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ai_service.agents.session import InMemorySessionStore
|
||||||
|
from ai_service.llm.types import Message
|
||||||
|
|
||||||
|
|
||||||
|
def msg(n: int) -> Message:
|
||||||
|
return Message(role="user", content=f"m{n}")
|
||||||
|
|
||||||
|
|
||||||
|
async def test_create_and_get():
|
||||||
|
store = InMemorySessionStore()
|
||||||
|
session = await store.create("s1", agent="coach")
|
||||||
|
assert session.agent == "coach"
|
||||||
|
assert (await store.get("s1")).session_id == "s1"
|
||||||
|
assert await store.get("missing") is None
|
||||||
|
|
||||||
|
|
||||||
|
async def test_append_and_history_window():
|
||||||
|
store = InMemorySessionStore()
|
||||||
|
await store.create("s1", agent="coach")
|
||||||
|
for i in range(30):
|
||||||
|
await store.append("s1", msg(i))
|
||||||
|
window = await store.history_window("s1", max_messages=20)
|
||||||
|
assert len(window) == 20
|
||||||
|
assert window[0].content == "m10" # last 20 of m0..m29
|
||||||
|
assert window[-1].content == "m29"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_default_window_uses_20():
|
||||||
|
store = InMemorySessionStore()
|
||||||
|
await store.create("s1", agent="coach")
|
||||||
|
for i in range(25):
|
||||||
|
await store.append("s1", msg(i))
|
||||||
|
window = await store.history_window("s1")
|
||||||
|
assert len(window) == 20
|
||||||
|
assert window[0].content == "m5"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_lru_eviction_at_cap():
|
||||||
|
store = InMemorySessionStore(window=20, max_sessions=3)
|
||||||
|
for i in range(3):
|
||||||
|
await store.create(f"s{i}", agent="coach")
|
||||||
|
# touch s0 so s1 becomes least-recently-used
|
||||||
|
await store.get("s0")
|
||||||
|
await store.create("s3", agent="coach") # evicts s1
|
||||||
|
assert await store.get("s1") is None
|
||||||
|
assert await store.get("s0") is not None
|
||||||
|
assert await store.get("s2") is not None
|
||||||
|
assert await store.get("s3") is not None
|
||||||
|
|
||||||
|
|
||||||
|
async def test_lru_eviction_at_default_500_cap():
|
||||||
|
store = InMemorySessionStore() # defaults: window=20, max_sessions=500
|
||||||
|
for i in range(500):
|
||||||
|
await store.create(f"s{i}", agent="coach")
|
||||||
|
await store.get("s0") # touch the oldest → s1 becomes least-recently-used
|
||||||
|
await store.create("s500", agent="coach") # evicts s1
|
||||||
|
assert await store.get("s1") is None
|
||||||
|
assert await store.get("s0") is not None
|
||||||
|
assert await store.get("s499") is not None
|
||||||
|
assert await store.get("s500") is not None
|
||||||
|
|
||||||
|
|
||||||
|
async def test_append_unknown_session_raises():
|
||||||
|
store = InMemorySessionStore()
|
||||||
|
with pytest.raises(KeyError):
|
||||||
|
await store.append("nope", msg(0))
|
||||||
|
|
||||||
|
|
||||||
|
async def test_delete():
|
||||||
|
store = InMemorySessionStore()
|
||||||
|
await store.create("s1", agent="coach")
|
||||||
|
await store.delete("s1")
|
||||||
|
assert await store.get("s1") is None
|
||||||
|
|
||||||
|
|
||||||
|
async def test_sessions_are_agent_scoped():
|
||||||
|
store = InMemorySessionStore()
|
||||||
|
a = await store.create("coach-session", agent="coach")
|
||||||
|
b = await store.create("tutor-session", agent="tutor")
|
||||||
|
assert a.agent == "coach"
|
||||||
|
assert b.agent == "tutor"
|
||||||
|
assert a.session_id != b.session_id
|
||||||
@@ -0,0 +1,156 @@
|
|||||||
|
"""Structured output defense tests — 4 layers (D-020), against mock providers."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from pydantic import BaseModel
|
||||||
|
|
||||||
|
from ai_service.agents.structured import (
|
||||||
|
StructuredOutputError,
|
||||||
|
extract_json_object,
|
||||||
|
parse_structured,
|
||||||
|
structured_completion,
|
||||||
|
)
|
||||||
|
from ai_service.llm.mock import MockProvider, ScriptedJSONProvider
|
||||||
|
from ai_service.llm.types import Message
|
||||||
|
|
||||||
|
|
||||||
|
class Score(BaseModel):
|
||||||
|
score: int
|
||||||
|
verdict: str
|
||||||
|
|
||||||
|
|
||||||
|
HINT = '{"score": <int 0-100>, "verdict": "<short verdict>"}'
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_json_plain():
|
||||||
|
assert extract_json_object('{"a": 1}') == '{"a": 1}'
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_json_fenced():
|
||||||
|
text = '```json\n{"a": 1}\n```'
|
||||||
|
assert extract_json_object(text) == '{"a": 1}'
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_json_with_prose_around():
|
||||||
|
text = 'Sure! Here is my answer: {"a": {"b": "x } y"}, "c": 2} hope that helps'
|
||||||
|
assert extract_json_object(text) == '{"a": {"b": "x } y"}, "c": 2}'
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_json_no_object_raises():
|
||||||
|
with pytest.raises(StructuredOutputError):
|
||||||
|
extract_json_object("no json here")
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_json_unbalanced_raises():
|
||||||
|
with pytest.raises(StructuredOutputError):
|
||||||
|
extract_json_object('{"a": 1')
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_structured_valid():
|
||||||
|
result = parse_structured('{"score": 88, "verdict": "solid"}', Score)
|
||||||
|
assert result.score == 88
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_structured_invalid_schema_raises():
|
||||||
|
with pytest.raises(StructuredOutputError):
|
||||||
|
parse_structured('{"wrong": "shape"}', Score)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_structured_completion_happy_path():
|
||||||
|
provider = ScriptedJSONProvider({"score": 91, "verdict": "excellent work"})
|
||||||
|
messages = [Message(role="user", content="grade my artifact")]
|
||||||
|
result = await structured_completion(
|
||||||
|
provider, messages, model="m", schema=Score, schema_hint=HINT
|
||||||
|
)
|
||||||
|
assert result.score == 91
|
||||||
|
assert result.verdict == "excellent work"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_structured_completion_retries_then_raises():
|
||||||
|
# Plain MockProvider returns non-schema JSON for json_object requests →
|
||||||
|
# both attempts fail validation → StructuredOutputError after ONE retry.
|
||||||
|
provider = MockProvider()
|
||||||
|
provider.received_calls = []
|
||||||
|
messages = [Message(role="user", content="grade me")]
|
||||||
|
with pytest.raises(StructuredOutputError):
|
||||||
|
await structured_completion(
|
||||||
|
provider, messages, model="m", schema=Score, schema_hint=HINT
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_structured_completion_retry_succeeds_after_invalid_first_response():
|
||||||
|
# Layer 4 recovery: first reply is wrong-schema fenced JSON, retry is valid.
|
||||||
|
class FlakyProvider(MockProvider):
|
||||||
|
def __init__(self) -> None:
|
||||||
|
super().__init__()
|
||||||
|
self.n = 0
|
||||||
|
self.retry_request: list[Message] = []
|
||||||
|
|
||||||
|
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||||
|
self.n += 1
|
||||||
|
if self.n == 1:
|
||||||
|
return '```json\n{"summary": "wrong shape"}\n```'
|
||||||
|
self.retry_request = list(messages)
|
||||||
|
return '{"score": 75, "verdict": "recovered"}'
|
||||||
|
|
||||||
|
provider = FlakyProvider()
|
||||||
|
messages = [Message(role="user", content="grade me")]
|
||||||
|
result = await structured_completion(
|
||||||
|
provider, messages, model="m", schema=Score, schema_hint=HINT
|
||||||
|
)
|
||||||
|
assert result.score == 75
|
||||||
|
assert result.verdict == "recovered"
|
||||||
|
assert provider.n == 2
|
||||||
|
# The retry must feed the validation error back to the model.
|
||||||
|
retry_contents = " ".join(m.content for m in provider.retry_request)
|
||||||
|
assert "previous response was invalid" in retry_contents
|
||||||
|
assert HINT in retry_contents
|
||||||
|
|
||||||
|
|
||||||
|
async def test_structured_completion_is_bounded_to_one_retry():
|
||||||
|
# Permanently-invalid provider: exactly two provider calls, then raise.
|
||||||
|
class CountingProvider(MockProvider):
|
||||||
|
def __init__(self) -> None:
|
||||||
|
super().__init__()
|
||||||
|
self.calls = 0
|
||||||
|
|
||||||
|
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||||
|
self.calls += 1
|
||||||
|
return await super().chat(
|
||||||
|
messages, model=model, response_format=response_format
|
||||||
|
)
|
||||||
|
|
||||||
|
provider = CountingProvider()
|
||||||
|
messages = [Message(role="user", content="grade me")]
|
||||||
|
with pytest.raises(StructuredOutputError, match="after retry"):
|
||||||
|
await structured_completion(
|
||||||
|
provider, messages, model="m", schema=Score, schema_hint=HINT
|
||||||
|
)
|
||||||
|
assert provider.calls == 2
|
||||||
|
|
||||||
|
|
||||||
|
async def test_structured_completion_sends_schema_instruction():
|
||||||
|
"""Layer 2: the schema hint must reach the provider in the request."""
|
||||||
|
provider = ScriptedJSONProvider({"score": 70, "verdict": "passing"})
|
||||||
|
captured: list[list] = []
|
||||||
|
|
||||||
|
original = provider.chat
|
||||||
|
|
||||||
|
async def recording_chat(messages, *, model, temperature=0.7, response_format=None):
|
||||||
|
captured.append(list(messages))
|
||||||
|
return await original(
|
||||||
|
messages, model=model, temperature=temperature, response_format=response_format
|
||||||
|
)
|
||||||
|
|
||||||
|
provider.chat = recording_chat
|
||||||
|
messages = [Message(role="user", content="grade")]
|
||||||
|
await structured_completion(provider, messages, model="m", schema=Score, schema_hint=HINT)
|
||||||
|
assert captured, "provider was never called"
|
||||||
|
last_user = next(m for m in reversed(captured[0]) if m.role == "user")
|
||||||
|
assert HINT in last_user.content
|
||||||
|
assert "ONLY" in last_user.content # JSON-only instruction present
|
||||||
|
|
||||||
|
# Determinism: same request yields same reply
|
||||||
|
again = await structured_completion(
|
||||||
|
provider, messages, model="m", schema=Score, schema_hint=HINT
|
||||||
|
)
|
||||||
|
assert again.score == 70
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
"""Tutor agent tests — persona, Socratic structure, distinctness vs Coach (REQ-2-006)."""
|
||||||
|
|
||||||
|
from ai_service.agents.coach import CoachAgent
|
||||||
|
from ai_service.agents.tutor import TutorAgent
|
||||||
|
from ai_service.config import Settings
|
||||||
|
from ai_service.corpus.learner_context import get_learner_context
|
||||||
|
from ai_service.llm.mock import MockProvider
|
||||||
|
from ai_service.llm.types import Message
|
||||||
|
|
||||||
|
|
||||||
|
def make_tutor() -> TutorAgent:
|
||||||
|
return TutorAgent(MockProvider(), Settings(provider="mock"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_system_prompt_includes_learner_context():
|
||||||
|
tutor = make_tutor()
|
||||||
|
prompt = tutor.system_prompt(get_learner_context())
|
||||||
|
assert "Alex Rivera" in prompt
|
||||||
|
assert "Socratic" in prompt
|
||||||
|
assert "ONE concept" in prompt
|
||||||
|
|
||||||
|
|
||||||
|
def test_system_prompt_marks_current_focus():
|
||||||
|
tutor = make_tutor()
|
||||||
|
prompt = tutor.system_prompt(get_learner_context())
|
||||||
|
assert "Multi-agent communication patterns" in prompt
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_messages_system_history_user():
|
||||||
|
tutor = make_tutor()
|
||||||
|
history = [Message(role="user", content="q1"), Message(role="assistant", content="a1")]
|
||||||
|
messages = tutor.build_messages(history, "explain again", get_learner_context())
|
||||||
|
assert messages[0].role == "system"
|
||||||
|
assert "Tutor" in messages[0].content
|
||||||
|
assert [m.content for m in messages[1:]] == ["q1", "a1", "explain again"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_stream_reply_yields_deltas():
|
||||||
|
tutor = make_tutor()
|
||||||
|
tokens = [
|
||||||
|
t
|
||||||
|
async for t in tutor.stream_reply(
|
||||||
|
user_input="hi", learner_context=get_learner_context()
|
||||||
|
)
|
||||||
|
]
|
||||||
|
assert tokens
|
||||||
|
|
||||||
|
|
||||||
|
def test_coach_and_tutor_personas_are_distinct():
|
||||||
|
"""Distinct system prompts (P3 must-have)."""
|
||||||
|
ctx = get_learner_context()
|
||||||
|
coach_prompt = CoachAgent(MockProvider(), Settings(provider="mock")).system_prompt(ctx)
|
||||||
|
tutor_prompt = TutorAgent(MockProvider(), Settings(provider="mock")).system_prompt(ctx)
|
||||||
|
assert coach_prompt != tutor_prompt
|
||||||
|
assert "retrieval practice" in coach_prompt.lower()
|
||||||
|
assert "Socratic" in tutor_prompt
|
||||||
|
|
||||||
|
|
||||||
|
async def test_coach_and_tutor_stream_outputs_are_distinct():
|
||||||
|
"""Mock outputs differ because system prompts differ (hash-seeded on content)."""
|
||||||
|
ctx = get_learner_context()
|
||||||
|
settings = Settings(provider="mock")
|
||||||
|
|
||||||
|
async def full_reply(agent):
|
||||||
|
tokens = [
|
||||||
|
t async for t in agent.stream_reply(user_input="stuck", learner_context=ctx)
|
||||||
|
]
|
||||||
|
return "".join(tokens)
|
||||||
|
|
||||||
|
coach_tokens = await full_reply(CoachAgent(MockProvider(), settings))
|
||||||
|
tutor_tokens = await full_reply(TutorAgent(MockProvider(), settings))
|
||||||
|
assert coach_tokens != tutor_tokens
|
||||||
|
|
||||||
|
|
||||||
|
def test_agent_name():
|
||||||
|
assert make_tutor().name == "tutor"
|
||||||
@@ -0,0 +1,79 @@
|
|||||||
|
"""Assessment evaluate endpoint tests — validated JSON, 404s (REQ-2-008)."""
|
||||||
|
|
||||||
|
|
||||||
|
from ai_service.agents.assessor import RubricScore
|
||||||
|
from ai_service.llm.mock import ScriptedJSONProvider
|
||||||
|
|
||||||
|
VALID_SCORE = {
|
||||||
|
"rubric_id": "rubric-orchestration-c002",
|
||||||
|
"artifact_id": "art-eval-research-assistant",
|
||||||
|
"competency_id": "stack-orchestration-c002",
|
||||||
|
"scores": [
|
||||||
|
{"criterion_id": "rc-architecture", "name": "Agent architecture soundness",
|
||||||
|
"score": 92, "evidence": "Explicit state schema"},
|
||||||
|
{"criterion_id": "rc-communication", "name": "Inter-agent communication design",
|
||||||
|
"score": 88, "evidence": "Typed ToolMessage responses"},
|
||||||
|
{"criterion_id": "rc-reliability", "name": "Reliability engineering",
|
||||||
|
"score": 85, "evidence": "3-retry loop"},
|
||||||
|
{"criterion_id": "rc-process", "name": "Process trace quality",
|
||||||
|
"score": 90, "evidence": "Iterative checkpoints"},
|
||||||
|
],
|
||||||
|
"strengths": ["Clean state boundaries", "Failure-aware tools"],
|
||||||
|
"gaps": ["No reviewer node", "Diagram only in README"],
|
||||||
|
"verdict": "mastered",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_returns_validated_rubric_json(client):
|
||||||
|
# Swap the app provider for a scripted-JSON provider for this test
|
||||||
|
original = client.app.state.provider
|
||||||
|
client.app.state.provider = ScriptedJSONProvider(VALID_SCORE)
|
||||||
|
try:
|
||||||
|
response = client.post(
|
||||||
|
"/v1/assessment/evaluate",
|
||||||
|
json={"artifact_id": "art-eval-research-assistant"},
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
client.app.state.provider = original
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
validated = RubricScore.model_validate(data) # response contract holds
|
||||||
|
assert validated.verdict == "mastered"
|
||||||
|
assert len(validated.scores) == 4
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_artifact_404(client):
|
||||||
|
response = client.post("/v1/assessment/evaluate", json={"artifact_id": "ghost"})
|
||||||
|
assert response.status_code == 404
|
||||||
|
assert "ghost" in response.json()["detail"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_unparseable_provider_502(client):
|
||||||
|
"""Plain MockProvider yields non-rubric JSON → structured defense exhausts
|
||||||
|
retry → endpoint translates to 502 (bad gateway to the model)."""
|
||||||
|
# default mock already returns non-rubric JSON
|
||||||
|
response = client.post(
|
||||||
|
"/v1/assessment/evaluate",
|
||||||
|
json={"artifact_id": "art-eval-research-assistant"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 502
|
||||||
|
assert "failed" in response.json()["detail"].lower()
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_artifact_id_422(client):
|
||||||
|
response = client.post("/v1/assessment/evaluate", json={})
|
||||||
|
assert response.status_code == 422
|
||||||
|
|
||||||
|
|
||||||
|
def test_second_artifact_also_evaluates(client):
|
||||||
|
original = client.app.state.provider
|
||||||
|
payload = dict(VALID_SCORE, artifact_id="art-eval-rag-dashboard")
|
||||||
|
client.app.state.provider = ScriptedJSONProvider(payload)
|
||||||
|
try:
|
||||||
|
response = client.post(
|
||||||
|
"/v1/assessment/evaluate", json={"artifact_id": "art-eval-rag-dashboard"}
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
client.app.state.provider = original
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json()["artifact_id"] == "art-eval-rag-dashboard"
|
||||||
@@ -0,0 +1,241 @@
|
|||||||
|
"""Chat endpoint session integration tests — history persistence + windowed replay."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
|
||||||
|
def stream_events(client, payload) -> list[dict]:
|
||||||
|
with client.stream("POST", "/v1/chat/stream", json=payload) as response:
|
||||||
|
assert response.status_code == 200
|
||||||
|
events = []
|
||||||
|
for line in response.iter_lines():
|
||||||
|
if line.startswith("data:"):
|
||||||
|
d = line.removeprefix("data:").strip()
|
||||||
|
if d == "[DONE]":
|
||||||
|
events.append({"type": "[DONE]"})
|
||||||
|
else:
|
||||||
|
events.append(json.loads(d))
|
||||||
|
return events
|
||||||
|
|
||||||
|
|
||||||
|
def test_first_turn_creates_session_and_persists(client):
|
||||||
|
payload = {
|
||||||
|
"agent": "coach",
|
||||||
|
"session_id": "sess-1",
|
||||||
|
"messages": [{"role": "user", "content": "first turn"}],
|
||||||
|
}
|
||||||
|
events = stream_events(client, payload)
|
||||||
|
assert events[0]["type"] == "meta"
|
||||||
|
assert events[0]["session_id"] == "sess-1"
|
||||||
|
store = client.app.state.session_store
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
async def check():
|
||||||
|
return await store.history_window("sess-1")
|
||||||
|
|
||||||
|
contents = [m.content for m in asyncio.run(check())]
|
||||||
|
assert "first turn" in contents
|
||||||
|
assert any("Think" in c or "[" in c for c in contents) # mock reply persisted
|
||||||
|
|
||||||
|
|
||||||
|
def test_second_turn_replays_windowed_history(client):
|
||||||
|
payload1 = {
|
||||||
|
"agent": "coach",
|
||||||
|
"session_id": "sess-2",
|
||||||
|
"messages": [{"role": "user", "content": "turn one"}],
|
||||||
|
}
|
||||||
|
stream_events(client, payload1)
|
||||||
|
# Second turn: the API passes history + new message to the provider.
|
||||||
|
# With the mock provider we cannot observe provider inputs directly,
|
||||||
|
# but the session store must now hold both turns.
|
||||||
|
store = client.app.state.session_store
|
||||||
|
# The store is async; use the app's internals through a short event loop
|
||||||
|
import asyncio
|
||||||
|
result = {}
|
||||||
|
|
||||||
|
async def check():
|
||||||
|
result["window"] = await store.history_window("sess-2")
|
||||||
|
|
||||||
|
asyncio.run(check())
|
||||||
|
contents = [m.content for m in result["window"]]
|
||||||
|
assert "turn one" in contents
|
||||||
|
assert any(m.role == "assistant" for m in result["window"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_second_turn_replays_history_to_provider(client):
|
||||||
|
# Observable provider input: a recording provider wrapper captures what
|
||||||
|
# the endpoint sends. Turn 2 must include turn 1's persisted messages.
|
||||||
|
|
||||||
|
from ai_service.llm.types import Message
|
||||||
|
|
||||||
|
provider = client.app.state.provider
|
||||||
|
original = provider.stream_chat
|
||||||
|
captured: list[list[Message]] = []
|
||||||
|
|
||||||
|
async def recording_stream(messages, *, model, temperature=0.7, response_format=None):
|
||||||
|
captured.append(list(messages))
|
||||||
|
async for t in original(
|
||||||
|
messages, model=model, temperature=temperature, response_format=response_format
|
||||||
|
):
|
||||||
|
yield t
|
||||||
|
|
||||||
|
provider.stream_chat = recording_stream
|
||||||
|
try:
|
||||||
|
stream_events(client, {
|
||||||
|
"agent": "coach", "session_id": "sess-replay",
|
||||||
|
"messages": [{"role": "user", "content": "turn one"}],
|
||||||
|
})
|
||||||
|
stream_events(client, {
|
||||||
|
"agent": "coach", "session_id": "sess-replay",
|
||||||
|
"messages": [{"role": "user", "content": "turn two"}],
|
||||||
|
})
|
||||||
|
finally:
|
||||||
|
provider.stream_chat = original
|
||||||
|
|
||||||
|
assert len(captured) == 2
|
||||||
|
turn1, turn2 = captured
|
||||||
|
# Agent routing (P3): provider now receives [system, ...history, user turn]
|
||||||
|
assert turn1[0].role == "system"
|
||||||
|
assert turn1[-1].content == "turn one"
|
||||||
|
assert len(turn1) == 2 # system + first user turn
|
||||||
|
turn2_contents = [m.content for m in turn2]
|
||||||
|
assert "turn one" in turn2_contents
|
||||||
|
assert "turn two" in turn2_contents
|
||||||
|
assert any(m.role == "assistant" for m in turn2) # persisted reply replayed
|
||||||
|
assert "turn two" == turn2_contents[-1] # new user turn last
|
||||||
|
assert turn2[0].role == "system" # every routed call starts with the persona
|
||||||
|
|
||||||
|
|
||||||
|
def test_history_replay_is_windowed(client):
|
||||||
|
# Windowing: only the last 20 stored messages are replayed to the provider.
|
||||||
|
|
||||||
|
from ai_service.llm.types import Message
|
||||||
|
|
||||||
|
provider = client.app.state.provider
|
||||||
|
original = provider.stream_chat
|
||||||
|
captured: list[list[Message]] = []
|
||||||
|
|
||||||
|
async def recording_stream(messages, *, model, temperature=0.7, response_format=None):
|
||||||
|
captured.append(list(messages))
|
||||||
|
async for t in original(
|
||||||
|
messages, model=model, temperature=temperature, response_format=response_format
|
||||||
|
):
|
||||||
|
yield t
|
||||||
|
|
||||||
|
provider.stream_chat = recording_stream
|
||||||
|
try:
|
||||||
|
# 15 turns → 30 persisted messages (user + assistant per turn) > 20 window.
|
||||||
|
for i in range(15):
|
||||||
|
stream_events(client, {
|
||||||
|
"agent": "coach", "session_id": "sess-window",
|
||||||
|
"messages": [{"role": "user", "content": f"turn {i}"}],
|
||||||
|
})
|
||||||
|
finally:
|
||||||
|
provider.stream_chat = original
|
||||||
|
|
||||||
|
last_input = captured[-1]
|
||||||
|
contents = [m.content for m in last_input]
|
||||||
|
# system prompt + window(20) + the new user message
|
||||||
|
assert last_input[0].role == "system"
|
||||||
|
assert len(last_input) == 1 + 20 + 1
|
||||||
|
assert "turn 0" not in contents # oldest messages trimmed out of replay
|
||||||
|
assert "turn 14" in contents
|
||||||
|
assert contents[-1] == "turn 14"
|
||||||
|
|
||||||
|
|
||||||
|
def test_replayed_system_prompt_carries_routed_persona(client):
|
||||||
|
"""History replay puts the routed agent's persona at position 0 (P3).
|
||||||
|
|
||||||
|
The system prompt on every turn — including replays — must match the
|
||||||
|
routed agent: Coach calls get the Coach persona, Tutor calls the Tutor
|
||||||
|
persona, and the two are observably distinct.
|
||||||
|
"""
|
||||||
|
from ai_service.llm.types import Message
|
||||||
|
|
||||||
|
def capture_two_turns(agent_name: str, session: str) -> list[list[Message]]:
|
||||||
|
provider = client.app.state.provider
|
||||||
|
original = provider.stream_chat
|
||||||
|
captured: list[list[Message]] = []
|
||||||
|
|
||||||
|
async def recording_stream(messages, *, model, temperature=0.7, response_format=None):
|
||||||
|
captured.append(list(messages))
|
||||||
|
async for t in original(
|
||||||
|
messages, model=model, temperature=temperature, response_format=response_format
|
||||||
|
):
|
||||||
|
yield t
|
||||||
|
|
||||||
|
provider.stream_chat = recording_stream
|
||||||
|
try:
|
||||||
|
for content in ("turn one", "turn two"):
|
||||||
|
stream_events(client, {
|
||||||
|
"agent": agent_name, "session_id": session,
|
||||||
|
"messages": [{"role": "user", "content": content}],
|
||||||
|
})
|
||||||
|
finally:
|
||||||
|
provider.stream_chat = original
|
||||||
|
return captured
|
||||||
|
|
||||||
|
coach_calls = capture_two_turns("coach", "persona-coach")
|
||||||
|
tutor_calls = capture_two_turns("tutor", "persona-tutor")
|
||||||
|
|
||||||
|
coach_replay_system = coach_calls[1][0]
|
||||||
|
tutor_replay_system = tutor_calls[1][0]
|
||||||
|
assert coach_replay_system.role == "system"
|
||||||
|
assert tutor_replay_system.role == "system"
|
||||||
|
assert "Coach" in coach_replay_system.content
|
||||||
|
assert "Tutor" in tutor_replay_system.content
|
||||||
|
assert coach_replay_system.content != tutor_replay_system.content
|
||||||
|
# persona is stable across turns within one session
|
||||||
|
assert coach_calls[0][0].content == coach_replay_system.content
|
||||||
|
|
||||||
|
|
||||||
|
def test_session_agent_scoped(client):
|
||||||
|
payload = {
|
||||||
|
"agent": "tutor",
|
||||||
|
"session_id": "sess-3",
|
||||||
|
"messages": [{"role": "user", "content": "hello"}],
|
||||||
|
}
|
||||||
|
stream_events(client, payload)
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
store = client.app.state.session_store
|
||||||
|
|
||||||
|
async def check():
|
||||||
|
return await store.get("sess-3")
|
||||||
|
|
||||||
|
session = asyncio.run(check())
|
||||||
|
assert session.agent == "tutor"
|
||||||
|
|
||||||
|
|
||||||
|
def test_client_retry_does_not_duplicate_user_turn(client):
|
||||||
|
"""P1 fix (final review): resending the same user turn after a failure
|
||||||
|
must not double-append it to session history."""
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
"agent": "coach",
|
||||||
|
"session_id": "sess-retry",
|
||||||
|
"messages": [{"role": "user", "content": "same question"}],
|
||||||
|
}
|
||||||
|
# First attempt fails mid-stream (turn was already appended before streaming)
|
||||||
|
provider = client.app.state.provider
|
||||||
|
provider.fail_mid_stream_at_index = 0
|
||||||
|
stream_events_retry(client, payload)
|
||||||
|
provider.fail_mid_stream_at_index = None
|
||||||
|
# Client retry: identical payload
|
||||||
|
stream_events_retry(client, payload)
|
||||||
|
|
||||||
|
store = client.app.state.session_store
|
||||||
|
|
||||||
|
async def check():
|
||||||
|
return await store.history_window("sess-retry")
|
||||||
|
|
||||||
|
contents = [m.content for m in asyncio.run(check())]
|
||||||
|
user_turns = [c for c in contents if c == "same question"]
|
||||||
|
assert len(user_turns) == 1, f"expected exactly 1 stored user turn, got {len(user_turns)}"
|
||||||
|
|
||||||
|
|
||||||
|
def stream_events_retry(client, payload):
|
||||||
|
with client.stream("POST", "/v1/chat/stream", json=payload) as response:
|
||||||
|
for _ in response.iter_lines():
|
||||||
|
pass
|
||||||
@@ -0,0 +1,131 @@
|
|||||||
|
"""SSE chat stream endpoint tests — envelope ordering, errors, headers."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
|
||||||
|
def stream_lines(client, payload: dict) -> list[str]:
|
||||||
|
with client.stream("POST", "/v1/chat/stream", json=payload) as response:
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "no-cache" in response.headers.get("cache-control", "")
|
||||||
|
return [line for line in response.iter_lines() if line.strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def parse_events(raw_lines: list[str]) -> list[dict]:
|
||||||
|
"""Parse SSE lines into event dicts; strips event:/data: prefixes."""
|
||||||
|
events = []
|
||||||
|
for line in raw_lines:
|
||||||
|
if line.startswith("data:"):
|
||||||
|
data = line.removeprefix("data:").strip()
|
||||||
|
if data == "[DONE]":
|
||||||
|
events.append({"type": "[DONE]"})
|
||||||
|
else:
|
||||||
|
events.append(json.loads(data))
|
||||||
|
return events
|
||||||
|
|
||||||
|
|
||||||
|
PAYLOAD = {
|
||||||
|
"agent": "tutor",
|
||||||
|
"session_id": "s1",
|
||||||
|
"messages": [{"role": "user", "content": "hello"}],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_meta_first_then_deltas_done_done_sentinel(client):
|
||||||
|
events = parse_events(stream_lines(client, PAYLOAD))
|
||||||
|
assert events[0]["type"] == "meta"
|
||||||
|
assert events[0]["agent"] == "tutor"
|
||||||
|
assert events[0]["session_id"] == "s1"
|
||||||
|
assert events[-1]["type"] == "[DONE]"
|
||||||
|
middle = events[1:-1]
|
||||||
|
deltas = [e for e in middle if e["type"] == "delta"]
|
||||||
|
dones = [e for e in middle if e["type"] == "done"]
|
||||||
|
assert len(deltas) >= 1
|
||||||
|
assert len(dones) == 1
|
||||||
|
assert dones[0]["finish_reason"] == "stop"
|
||||||
|
# delta events after meta, done before [DONE]
|
||||||
|
assert events.index(dones[0]) > events.index(deltas[0])
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_messages_rejected(client):
|
||||||
|
response = client.post(
|
||||||
|
"/v1/chat/stream",
|
||||||
|
json={"agent": "tutor", "session_id": "s", "messages": []},
|
||||||
|
)
|
||||||
|
assert response.status_code == 422
|
||||||
|
|
||||||
|
|
||||||
|
def test_mid_stream_failure_yields_error_then_done(client):
|
||||||
|
provider = client.app.state.provider
|
||||||
|
provider.fail_mid_stream_at_index = 1
|
||||||
|
events = parse_events(stream_lines(client, PAYLOAD))
|
||||||
|
provider.fail_mid_stream_at_index = None
|
||||||
|
error_events = [e for e in events if e["type"] == "error"]
|
||||||
|
assert len(error_events) == 1
|
||||||
|
assert error_events[0]["code"] == "provider_error"
|
||||||
|
assert events[-1]["type"] == "[DONE]"
|
||||||
|
# error must precede the terminal sentinel
|
||||||
|
assert events.index(error_events[0]) < events.index(events[-1])
|
||||||
|
|
||||||
|
|
||||||
|
def test_pre_first_byte_failure_yields_provider_unavailable(client):
|
||||||
|
provider = client.app.state.provider
|
||||||
|
provider.fail_before_first_token = True
|
||||||
|
events = parse_events(stream_lines(client, PAYLOAD))
|
||||||
|
provider.fail_before_first_token = False
|
||||||
|
error_events = [e for e in events if e["type"] == "error"]
|
||||||
|
assert len(error_events) == 1
|
||||||
|
assert error_events[0]["code"] == "provider_unavailable"
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_agent_rejected_422(client):
|
||||||
|
response = client.post(
|
||||||
|
"/v1/chat/stream",
|
||||||
|
json={
|
||||||
|
"agent": "oracle",
|
||||||
|
"session_id": "s",
|
||||||
|
"messages": [{"role": "user", "content": "hi"}],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert response.status_code == 422
|
||||||
|
assert "oracle" in response.json()["detail"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_coach_routes_and_meta_names_agent(client):
|
||||||
|
payload = {
|
||||||
|
"agent": "coach",
|
||||||
|
"session_id": "route-coach",
|
||||||
|
"messages": [{"role": "user", "content": "pace me"}],
|
||||||
|
}
|
||||||
|
events = parse_events(stream_lines(client, payload))
|
||||||
|
assert events[0]["type"] == "meta"
|
||||||
|
assert events[0]["agent"] == "coach"
|
||||||
|
assert events[-1]["type"] == "[DONE]"
|
||||||
|
|
||||||
|
|
||||||
|
def test_tutor_routes_and_meta_names_agent(client):
|
||||||
|
payload = {
|
||||||
|
"agent": "tutor",
|
||||||
|
"session_id": "route-tutor",
|
||||||
|
"messages": [{"role": "user", "content": "teach me"}],
|
||||||
|
}
|
||||||
|
events = parse_events(stream_lines(client, payload))
|
||||||
|
assert events[0]["type"] == "meta"
|
||||||
|
assert events[0]["agent"] == "tutor"
|
||||||
|
assert events[-1]["type"] == "[DONE]"
|
||||||
|
|
||||||
|
|
||||||
|
def test_coach_and_tutor_streams_are_distinct(client):
|
||||||
|
"""Agent routing selects the right agent: distinct system prompts →
|
||||||
|
distinct hash-seeded mock outputs for the same user input."""
|
||||||
|
same_message = [{"role": "user", "content": "same question"}]
|
||||||
|
coach = parse_events(stream_lines(client, {
|
||||||
|
"agent": "coach", "session_id": "d1", "messages": same_message,
|
||||||
|
}))
|
||||||
|
tutor = parse_events(stream_lines(client, {
|
||||||
|
"agent": "tutor", "session_id": "d2", "messages": same_message,
|
||||||
|
}))
|
||||||
|
coach_text = "".join(e["content"] for e in coach if e["type"] == "delta")
|
||||||
|
tutor_text = "".join(e["content"] for e in tutor if e["type"] == "delta")
|
||||||
|
assert coach_text and tutor_text
|
||||||
|
assert coach_text != tutor_text
|
||||||
|
assert coach[-1]["type"] == "[DONE]"
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
"""Lab feedback endpoint tests — SSE envelope with agent=lab (REQ-2-007)."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
|
||||||
|
def stream_events(client, payload) -> list[dict]:
|
||||||
|
with client.stream("POST", "/v1/lab/feedback", json=payload) as response:
|
||||||
|
assert response.status_code == 200
|
||||||
|
events = []
|
||||||
|
for line in response.iter_lines():
|
||||||
|
if line.startswith("data:"):
|
||||||
|
d = line.removeprefix("data:").strip()
|
||||||
|
if d == "[DONE]":
|
||||||
|
events.append({"type": "[DONE]"})
|
||||||
|
else:
|
||||||
|
events.append(json.loads(d))
|
||||||
|
return events
|
||||||
|
|
||||||
|
|
||||||
|
def test_lab_feedback_streams_full_envelope(client):
|
||||||
|
events = stream_events(client, {"scenario_id": "lab-scenario-strong"})
|
||||||
|
assert events[0]["type"] == "meta"
|
||||||
|
assert events[0]["agent"] == "lab"
|
||||||
|
assert events[0]["scenario_id"] == "lab-scenario-strong"
|
||||||
|
deltas = [e for e in events if e["type"] == "delta"]
|
||||||
|
assert len(deltas) >= 1
|
||||||
|
assert any(e["type"] == "done" for e in events)
|
||||||
|
assert events[-1]["type"] == "[DONE]"
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_scenario_404(client):
|
||||||
|
response = client.post("/v1/lab/feedback", json={"scenario_id": "nope"})
|
||||||
|
assert response.status_code == 404
|
||||||
|
assert "nope" in response.json()["detail"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_distinct_scenarios_distinct_replies(client):
|
||||||
|
strong = stream_events(client, {"scenario_id": "lab-scenario-strong"})
|
||||||
|
struggling = stream_events(client, {"scenario_id": "lab-scenario-struggling"})
|
||||||
|
strong_text = "".join(e["content"] for e in strong if e["type"] == "delta")
|
||||||
|
struggling_text = "".join(e["content"] for e in struggling if e["type"] == "delta")
|
||||||
|
assert strong_text != struggling_text
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_scenario_id_422(client):
|
||||||
|
response = client.post("/v1/lab/feedback", json={})
|
||||||
|
assert response.status_code == 422
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
"""Mentor narrative endpoint tests — SSE, session-backed (REQ-2-010)."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
|
||||||
|
def stream_events(client, payload) -> list[dict]:
|
||||||
|
with client.stream("POST", "/v1/mentor/narrative", json=payload) as response:
|
||||||
|
assert response.status_code == 200
|
||||||
|
events = []
|
||||||
|
for line in response.iter_lines():
|
||||||
|
if line.startswith("data:"):
|
||||||
|
d = line.removeprefix("data:").strip()
|
||||||
|
if d == "[DONE]":
|
||||||
|
events.append({"type": "[DONE]"})
|
||||||
|
else:
|
||||||
|
events.append(json.loads(d))
|
||||||
|
return events
|
||||||
|
|
||||||
|
|
||||||
|
def test_narrative_streams_full_envelope(client):
|
||||||
|
events = stream_events(client, {"session_id": "mentor-1"})
|
||||||
|
assert events[0]["type"] == "meta"
|
||||||
|
assert events[0]["agent"] == "mentor"
|
||||||
|
deltas = [e for e in events if e["type"] == "delta"]
|
||||||
|
assert len(deltas) >= 1
|
||||||
|
assert any(e["type"] == "done" for e in events)
|
||||||
|
assert events[-1]["type"] == "[DONE]"
|
||||||
|
|
||||||
|
|
||||||
|
def test_narrative_is_session_backed(client):
|
||||||
|
"""Second call replays history: provider input grows; distinct mock output."""
|
||||||
|
first = stream_events(client, {"session_id": "mentor-2", "prompt": "narrate my path"})
|
||||||
|
second = stream_events(client, {"session_id": "mentor-2", "prompt": "what next?"})
|
||||||
|
first_text = "".join(e["content"] for e in first if e["type"] == "delta")
|
||||||
|
second_text = "".join(e["content"] for e in second if e["type"] == "delta")
|
||||||
|
assert first_text != second_text
|
||||||
|
|
||||||
|
|
||||||
|
def test_narrative_persists_turns(client):
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
store = client.app.state.session_store
|
||||||
|
|
||||||
|
stream_events(client, {"session_id": "mentor-3", "prompt": "hello trajectory"})
|
||||||
|
|
||||||
|
async def check():
|
||||||
|
return await store.history_window("mentor-3")
|
||||||
|
|
||||||
|
contents = [m.content for m in asyncio.run(check())]
|
||||||
|
assert "hello trajectory" in contents
|
||||||
|
assert len(contents) >= 2 # user + assistant persisted
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_session_id_422(client):
|
||||||
|
response = client.post("/v1/mentor/narrative", json={"prompt": "hi"})
|
||||||
|
assert response.status_code == 422
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
"""Proctor signals endpoint tests — validated JSON, 404s (REQ-2-009)."""
|
||||||
|
|
||||||
|
from ai_service.agents.proctor import ProctorAssessment
|
||||||
|
from ai_service.llm.mock import ScriptedJSONProvider
|
||||||
|
|
||||||
|
VALID = {
|
||||||
|
"scenario_id": "proctor-scenario-distracted",
|
||||||
|
"signals": [
|
||||||
|
{"signal_type": "context_switch", "severity": "low",
|
||||||
|
"note": "Docs tab at t+120s is normal"},
|
||||||
|
{"signal_type": "idle_gap", "severity": "medium",
|
||||||
|
"note": "5-minute idle at t+300s"},
|
||||||
|
],
|
||||||
|
"intervention": "Offer a short break, then restate the plan",
|
||||||
|
"summary": "Coaching-shaped session note",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_signals_returns_validated_json(client):
|
||||||
|
original = client.app.state.provider
|
||||||
|
client.app.state.provider = ScriptedJSONProvider(VALID)
|
||||||
|
try:
|
||||||
|
response = client.post(
|
||||||
|
"/v1/proctor/signals", json={"scenario_id": "proctor-scenario-distracted"}
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
client.app.state.provider = original
|
||||||
|
assert response.status_code == 200
|
||||||
|
validated = ProctorAssessment.model_validate(response.json())
|
||||||
|
assert validated.scenario_id == "proctor-scenario-distracted"
|
||||||
|
assert validated.intervention
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_scenario_404(client):
|
||||||
|
response = client.post("/v1/proctor/signals", json={"scenario_id": "ghost"})
|
||||||
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_unparseable_provider_502(client):
|
||||||
|
response = client.post(
|
||||||
|
"/v1/proctor/signals", json={"scenario_id": "proctor-scenario-healthy"}
|
||||||
|
)
|
||||||
|
assert response.status_code == 502
|
||||||
|
assert "failed" in response.json()["detail"].lower()
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_scenario_id_422(client):
|
||||||
|
response = client.post("/v1/proctor/signals", json={})
|
||||||
|
assert response.status_code == 422
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
"""Client-disconnect regression tests — SSE generators must tolerate aclose().
|
||||||
|
|
||||||
|
A `yield` inside `finally` re-raises "async generator ignored GeneratorExit"
|
||||||
|
when sse-starlette closes the iterator on client disconnect (P0 finding,
|
||||||
|
final review). These tests reproduce the close path directly against each
|
||||||
|
endpoint's event_stream generator shape.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
from ai_service.corpus.learner_context import get_learner_context
|
||||||
|
from ai_service.llm.types import Message
|
||||||
|
|
||||||
|
|
||||||
|
async def _chat_event_stream(client, session_id="close-chat", content="hi"):
|
||||||
|
"""Rebuild the chat endpoint's event_stream generator exactly as
|
||||||
|
chat.py builds it (same code shape, same session flow)."""
|
||||||
|
app = client.app
|
||||||
|
settings = app.state.settings
|
||||||
|
provider = app.state.provider
|
||||||
|
registry = app.state.agent_registry
|
||||||
|
sessions = app.state.session_store
|
||||||
|
agent = registry.get(provider, settings, "tutor")
|
||||||
|
learner_context = get_learner_context(None)
|
||||||
|
|
||||||
|
if await sessions.get(session_id) is None:
|
||||||
|
await sessions.create(session_id, agent="tutor", learner_id="learner-001")
|
||||||
|
user_turn = Message(role="user", content=content)
|
||||||
|
history = await sessions.history_window(session_id)
|
||||||
|
await sessions.append(session_id, user_turn)
|
||||||
|
|
||||||
|
async def event_stream():
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "meta", "agent": "tutor", "session_id": session_id,
|
||||||
|
"model": settings.model,
|
||||||
|
})}
|
||||||
|
first_byte = True
|
||||||
|
reply_parts: list[str] = []
|
||||||
|
try:
|
||||||
|
async for token in agent.stream_reply(
|
||||||
|
history=history, user_input=user_turn.content,
|
||||||
|
learner_context=learner_context,
|
||||||
|
):
|
||||||
|
first_byte = False
|
||||||
|
reply_parts.append(token)
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "delta", "content": token
|
||||||
|
})}
|
||||||
|
full_reply = "".join(reply_parts)
|
||||||
|
if full_reply:
|
||||||
|
await sessions.append(
|
||||||
|
session_id, Message(role="assistant", content=full_reply)
|
||||||
|
)
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "done", "finish_reason": "stop"
|
||||||
|
})}
|
||||||
|
yield {"event": "message", "data": "[DONE]"}
|
||||||
|
except Exception as exc:
|
||||||
|
code = "provider_unavailable" if first_byte else "provider_error"
|
||||||
|
yield {"event": "message", "data": json.dumps({
|
||||||
|
"type": "error", "code": code, "message": str(exc)
|
||||||
|
})}
|
||||||
|
yield {"event": "message", "data": "[DONE]"}
|
||||||
|
|
||||||
|
return event_stream()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_chat_stream_generator_survives_aclose(client):
|
||||||
|
"""Partial consumption then aclose() must not raise
|
||||||
|
'async generator ignored GeneratorExit' (yield-in-finally regression)."""
|
||||||
|
gen = await _chat_event_stream(client, session_id="close-chat")
|
||||||
|
meta = await gen.__anext__()
|
||||||
|
assert json.loads(meta["data"])["type"] == "meta"
|
||||||
|
delta = await gen.__anext__()
|
||||||
|
assert json.loads(delta["data"])["type"] == "delta"
|
||||||
|
# The critical assertion: closing mid-stream must be clean (no raise).
|
||||||
|
await gen.aclose()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_chat_stream_survives_close_at_different_points(client):
|
||||||
|
"""Close right after meta, and right after done — all must be clean."""
|
||||||
|
gen = await _chat_event_stream(client, session_id="close-early")
|
||||||
|
await gen.__anext__() # meta only
|
||||||
|
await gen.aclose()
|
||||||
|
|
||||||
|
gen2 = await _chat_event_stream(client, session_id="close-late")
|
||||||
|
events = []
|
||||||
|
async for ev in gen2:
|
||||||
|
events.append(json.loads(ev["data"]))
|
||||||
|
if len(events) == 2:
|
||||||
|
break
|
||||||
|
await gen2.aclose()
|
||||||
|
assert events[0]["type"] == "meta"
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
"""Test suite — conftest: mock provider only, zero network (enforced)."""
|
||||||
|
|
||||||
|
import os
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from ai_service.config import Settings
|
||||||
|
from ai_service.llm.mock import MockProvider
|
||||||
|
from ai_service.main import create_app
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def settings() -> Settings:
|
||||||
|
os.environ["AI_PROVIDER"] = "mock"
|
||||||
|
return Settings(provider="mock", model="gemma4:31b", port=8421)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def app(settings: Settings):
|
||||||
|
return create_app(settings)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def client(app):
|
||||||
|
with TestClient(app) as c:
|
||||||
|
# Mechanical cloud-free guard (GRILL advisory a): the app under test
|
||||||
|
# MUST be wired to the deterministic mock provider.
|
||||||
|
assert isinstance(app.state.provider, MockProvider), (
|
||||||
|
f"tests must run against MockProvider, got {type(app.state.provider).__name__}"
|
||||||
|
)
|
||||||
|
yield c
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
"""MockProvider tests — determinism, JSON mode, failure modes."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ai_service.llm.mock import MockProvider
|
||||||
|
from ai_service.llm.types import Message
|
||||||
|
|
||||||
|
MSGS = [Message(role="user", content="hello there")]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_stream_is_deterministic():
|
||||||
|
p1, p2 = MockProvider(), MockProvider()
|
||||||
|
out1 = [t async for t in p1.stream_chat(MSGS, model="m")]
|
||||||
|
out2 = [t async for t in p2.stream_chat(MSGS, model="m")]
|
||||||
|
assert "".join(out1) == "".join(out2)
|
||||||
|
assert out1 == out2
|
||||||
|
|
||||||
|
|
||||||
|
async def test_stream_content_differs_for_different_input():
|
||||||
|
p = MockProvider()
|
||||||
|
a = "".join([t async for t in p.stream_chat(MSGS, model="m")])
|
||||||
|
b = "".join(
|
||||||
|
[t async for t in p.stream_chat([Message(role="user", content="other")], model="m")]
|
||||||
|
)
|
||||||
|
assert a != b
|
||||||
|
|
||||||
|
|
||||||
|
async def test_json_object_response_format():
|
||||||
|
import json
|
||||||
|
|
||||||
|
p = MockProvider()
|
||||||
|
out = "".join(
|
||||||
|
[
|
||||||
|
t
|
||||||
|
async for t in p.stream_chat(
|
||||||
|
MSGS, model="m", response_format={"type": "json_object"}
|
||||||
|
)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
assert json.loads(out) == {"summary": "mock structured reply", "confidence": 0.87}
|
||||||
|
|
||||||
|
|
||||||
|
async def test_fail_before_first_token():
|
||||||
|
p = MockProvider()
|
||||||
|
p.fail_before_first_token = True
|
||||||
|
with pytest.raises(RuntimeError):
|
||||||
|
async for _ in p.stream_chat(MSGS, model="m"):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
async def test_fail_mid_stream():
|
||||||
|
p = MockProvider()
|
||||||
|
p.fail_mid_stream_at_index = 2
|
||||||
|
tokens = []
|
||||||
|
with pytest.raises(RuntimeError):
|
||||||
|
async for t in p.stream_chat(MSGS, model="m"):
|
||||||
|
tokens.append(t)
|
||||||
|
assert len(tokens) == 2
|
||||||
|
|
||||||
|
|
||||||
|
async def test_cancellation_records_abort():
|
||||||
|
p = MockProvider()
|
||||||
|
gen = p.stream_chat(MSGS, model="m")
|
||||||
|
await gen.__anext__()
|
||||||
|
await gen.aclose()
|
||||||
|
assert p.abort_recorded is True
|
||||||
|
|
||||||
|
|
||||||
|
async def test_chat_returns_full_reply():
|
||||||
|
p = MockProvider()
|
||||||
|
reply = await p.chat(MSGS, model="m")
|
||||||
|
streamed = "".join([t async for t in p.stream_chat(MSGS, model="m")])
|
||||||
|
assert reply == streamed
|
||||||
@@ -0,0 +1,177 @@
|
|||||||
|
"""OpenAICompatProvider tests — byte-exact SSE parsing via httpx.MockTransport.
|
||||||
|
|
||||||
|
Covers: multi-delta happy path, keep-alive comment lines, [DONE] sentinel,
|
||||||
|
malformed line tolerance, missing optional fields, non-streaming chat(),
|
||||||
|
response_format auto-degrade on 400, api_key never leaking into exceptions.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from ai_service.llm.openai_compat import OpenAICompatProvider
|
||||||
|
from ai_service.llm.types import Message
|
||||||
|
|
||||||
|
MSGS = [Message(role="user", content="hi")]
|
||||||
|
KEY = "sk-test-abc123"
|
||||||
|
|
||||||
|
|
||||||
|
def make_client(handler) -> httpx.AsyncClient:
|
||||||
|
transport = httpx.MockTransport(handler)
|
||||||
|
return httpx.AsyncClient(transport=transport)
|
||||||
|
|
||||||
|
|
||||||
|
def sse_body(deltas: list[str], with_comments: bool = True) -> bytes:
|
||||||
|
lines = []
|
||||||
|
if with_comments:
|
||||||
|
lines.append(": ping")
|
||||||
|
for d in deltas:
|
||||||
|
lines.append("data: " + json.dumps({
|
||||||
|
"id": "chatcmpl-1", "object": "chat.completion.chunk",
|
||||||
|
"created": 1, "model": "gemma4:31b",
|
||||||
|
"choices": [{"index": 0, "delta": {"content": d}, "finish_reason": None}],
|
||||||
|
}))
|
||||||
|
if with_comments:
|
||||||
|
lines.append(": ping")
|
||||||
|
lines.append("data: [DONE]")
|
||||||
|
return ("\n".join(lines) + "\n").encode()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_stream_happy_path_with_comments_and_done():
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
assert request.url.path == "/v1/chat/completions"
|
||||||
|
return httpx.Response(200, content=sse_body(["Hel", "lo", " world"]))
|
||||||
|
|
||||||
|
async with make_client(handler) as client:
|
||||||
|
provider = OpenAICompatProvider(client, "https://fake/v1")
|
||||||
|
tokens = [t async for t in provider.stream_chat(MSGS, model="gemma4:31b")]
|
||||||
|
assert tokens == ["Hel", "lo", " world"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_stream_tolerates_malformed_lines():
|
||||||
|
body = (
|
||||||
|
"data: not-json\n"
|
||||||
|
"data: "
|
||||||
|
+ json.dumps({
|
||||||
|
"id": "x", "object": "chat.completion.chunk", "created": 1, "model": "m",
|
||||||
|
"choices": [{"index": 0, "delta": {"content": "ok"}, "finish_reason": None}],
|
||||||
|
})
|
||||||
|
+ "\ndata: [DONE]\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
def handler(request):
|
||||||
|
return httpx.Response(200, content=body.encode())
|
||||||
|
|
||||||
|
async with make_client(handler) as client:
|
||||||
|
provider = OpenAICompatProvider(client, "https://fake/v1")
|
||||||
|
tokens = [t async for t in provider.stream_chat(MSGS, model="m")]
|
||||||
|
assert tokens == ["ok"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_stream_skips_empty_content_and_empty_choices():
|
||||||
|
chunk_empty_delta = json.dumps(
|
||||||
|
{"id": "x", "model": "m",
|
||||||
|
"choices": [{"index": 0, "delta": {}, "finish_reason": None}]}
|
||||||
|
)
|
||||||
|
chunk_empty_choices = json.dumps({"id": "x", "model": "m", "choices": []})
|
||||||
|
chunk_yes = json.dumps(
|
||||||
|
{"id": "x", "model": "m",
|
||||||
|
"choices": [{"index": 0, "delta": {"content": "yes"}, "finish_reason": None}]}
|
||||||
|
)
|
||||||
|
body = (
|
||||||
|
f"data: {chunk_empty_delta}\n"
|
||||||
|
f"data: {chunk_empty_choices}\n"
|
||||||
|
f"data: {chunk_yes}\n"
|
||||||
|
"data: [DONE]\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
def handler(request):
|
||||||
|
return httpx.Response(200, content=body.encode())
|
||||||
|
|
||||||
|
async with make_client(handler) as client:
|
||||||
|
provider = OpenAICompatProvider(client, "https://fake/v1")
|
||||||
|
tokens = [t async for t in provider.stream_chat(MSGS, model="m")]
|
||||||
|
assert tokens == ["yes"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_chat_non_streaming():
|
||||||
|
def handler(request):
|
||||||
|
payload = json.loads(request.content)
|
||||||
|
assert payload["stream"] is False
|
||||||
|
body = {"id": "1", "object": "chat.completion", "created": 1, "model": "m",
|
||||||
|
"choices": [{"index": 0, "message": {"role": "assistant",
|
||||||
|
"content": "full reply"},
|
||||||
|
"finish_reason": "stop"}]}
|
||||||
|
return httpx.Response(200, json=body)
|
||||||
|
|
||||||
|
async with make_client(handler) as client:
|
||||||
|
provider = OpenAICompatProvider(client, "https://fake/v1")
|
||||||
|
assert await provider.chat(MSGS, model="m") == "full reply"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_response_format_auto_degrades_on_400():
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def handler(request):
|
||||||
|
payload = json.loads(request.content)
|
||||||
|
calls.append(payload)
|
||||||
|
if "response_format" in payload:
|
||||||
|
return httpx.Response(400, json={"error": "response_format unsupported"})
|
||||||
|
body = {"id": "1", "object": "chat.completion", "created": 1, "model": "m",
|
||||||
|
"choices": [{"index": 0, "message": {"role": "assistant",
|
||||||
|
"content": "json"},
|
||||||
|
"finish_reason": "stop"}]}
|
||||||
|
return httpx.Response(200, json=body)
|
||||||
|
|
||||||
|
async with make_client(handler) as client:
|
||||||
|
provider = OpenAICompatProvider(client, "https://fake/v1", json_mode="auto")
|
||||||
|
reply = await provider.chat(MSGS, model="m", response_format={"type": "json_object"})
|
||||||
|
assert reply == "json"
|
||||||
|
assert len(calls) == 2
|
||||||
|
assert "response_format" in calls[0]
|
||||||
|
assert "response_format" not in calls[1]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_response_format_off_never_sends():
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def handler(request):
|
||||||
|
payload = json.loads(request.content)
|
||||||
|
calls.append(payload)
|
||||||
|
body = {"id": "1", "object": "chat.completion", "created": 1, "model": "m",
|
||||||
|
"choices": [{"index": 0, "message": {"role": "assistant",
|
||||||
|
"content": "x"},
|
||||||
|
"finish_reason": "stop"}]}
|
||||||
|
return httpx.Response(200, json=body)
|
||||||
|
|
||||||
|
async with make_client(handler) as client:
|
||||||
|
provider = OpenAICompatProvider(client, "https://fake/v1", json_mode="off")
|
||||||
|
await provider.chat(MSGS, model="m", response_format={"type": "json_object"})
|
||||||
|
assert len(calls) == 1
|
||||||
|
assert "response_format" not in calls[0]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_api_key_never_in_exception():
|
||||||
|
def handler(request):
|
||||||
|
raise httpx.ConnectError("connection refused while using sk-test-abc123")
|
||||||
|
|
||||||
|
async with make_client(handler) as client:
|
||||||
|
provider = OpenAICompatProvider(client, "https://fake/v1", api_key=KEY)
|
||||||
|
with pytest.raises(RuntimeError) as exc_info:
|
||||||
|
async for _ in provider.stream_chat(MSGS, model="m"):
|
||||||
|
pass
|
||||||
|
assert "sk-test-abc123" not in str(exc_info.value)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_bearer_header_sent():
|
||||||
|
seen = {}
|
||||||
|
|
||||||
|
def handler(request):
|
||||||
|
seen["auth"] = request.headers.get("Authorization")
|
||||||
|
return httpx.Response(200, content=sse_body(["x"]))
|
||||||
|
|
||||||
|
async with make_client(handler) as client:
|
||||||
|
provider = OpenAICompatProvider(client, "https://fake/v1", api_key=KEY)
|
||||||
|
_ = [t async for t in provider.stream_chat(MSGS, model="m")]
|
||||||
|
assert seen["auth"] == f"Bearer {KEY}"
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
"""Health endpoint tests."""
|
||||||
|
|
||||||
|
|
||||||
|
def test_health_returns_ok(client):
|
||||||
|
response = client.get("/health")
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
assert data["status"] == "ok"
|
||||||
|
assert data["provider"] == "mock"
|
||||||
|
assert data["model"] == "gemma4:31b"
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
# AI service (v0.2) — learner chat/panels stream from this FastAPI service
|
||||||
|
NEXT_PUBLIC_AI_SERVICE_URL=http://localhost:8420
|
||||||
@@ -11,7 +11,8 @@ import {
|
|||||||
Activity,
|
Activity,
|
||||||
} from 'lucide-react';
|
} from 'lucide-react';
|
||||||
import { Button } from '@nextcraft/ui';
|
import { Button } from '@nextcraft/ui';
|
||||||
import { allCompetencies, competencyStacks } from '@nextcraft/mock-data';
|
import { allCompetencies, competencyStacks, aiLabScenarios } from '@nextcraft/mock-data';
|
||||||
|
import { LabFeedbackPanel } from '../../../../components/learner/lab-feedback-panel';
|
||||||
|
|
||||||
interface FileEntry {
|
interface FileEntry {
|
||||||
label: string;
|
label: string;
|
||||||
@@ -279,6 +280,10 @@ export default async function BuildSandboxPage({
|
|||||||
))}
|
))}
|
||||||
</ul>
|
</ul>
|
||||||
</div>
|
</div>
|
||||||
|
{/* Lab in-flow feedback — mock telemetry scenario (real engine v0.3+) */}
|
||||||
|
<div className="border-t border-slate-200 pt-3 dark:border-slate-800">
|
||||||
|
<LabFeedbackPanel scenarioId={aiLabScenarios[0].id} />
|
||||||
|
</div>
|
||||||
</aside>
|
</aside>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ import {
|
|||||||
learnerMicrocredentials,
|
learnerMicrocredentials,
|
||||||
} from '@nextcraft/mock-data';
|
} from '@nextcraft/mock-data';
|
||||||
import { AiTutorChat } from '../../../components/learner/ai-tutor-chat';
|
import { AiTutorChat } from '../../../components/learner/ai-tutor-chat';
|
||||||
|
import { MentorPanel } from '../../../components/learner/mentor-panel';
|
||||||
import { ProgressGraph } from '../../../components/learner/progress-graph';
|
import { ProgressGraph } from '../../../components/learner/progress-graph';
|
||||||
|
|
||||||
const ACTIVE_COMPETENCY_IDS = [
|
const ACTIVE_COMPETENCY_IDS = [
|
||||||
@@ -265,7 +266,7 @@ export default function DashboardPage() {
|
|||||||
AI Tutor
|
AI Tutor
|
||||||
</h2>
|
</h2>
|
||||||
<p className="text-sm text-slate-500 dark:text-slate-400">
|
<p className="text-sm text-slate-500 dark:text-slate-400">
|
||||||
Coach and Socratic tutor · mock responses
|
Coach and Socratic tutor · live streaming
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -274,6 +275,21 @@ export default function DashboardPage() {
|
|||||||
<AiTutorChat />
|
<AiTutorChat />
|
||||||
</CardBody>
|
</CardBody>
|
||||||
</Card>
|
</Card>
|
||||||
|
|
||||||
|
{/* Mentor — long-horizon career narrative */}
|
||||||
|
<Card>
|
||||||
|
<CardHeader>
|
||||||
|
<h2 className="text-base font-semibold text-slate-900 dark:text-slate-100">
|
||||||
|
Mentor
|
||||||
|
</h2>
|
||||||
|
<p className="text-sm text-slate-500 dark:text-slate-400">
|
||||||
|
Long-horizon career trajectory · live streaming
|
||||||
|
</p>
|
||||||
|
</CardHeader>
|
||||||
|
<CardBody>
|
||||||
|
<MentorPanel />
|
||||||
|
</CardBody>
|
||||||
|
</Card>
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
@@ -16,6 +16,9 @@ import {
|
|||||||
import { Card, CardBody, CardHeader, Badge, Button } from '@nextcraft/ui';
|
import { Card, CardBody, CardHeader, Badge, Button } from '@nextcraft/ui';
|
||||||
import { allCompetencies, competencyStacks } from '@nextcraft/mock-data';
|
import { allCompetencies, competencyStacks } from '@nextcraft/mock-data';
|
||||||
import { OralDefenseInterface } from '../../../../components/learner/oral-defense-interface';
|
import { OralDefenseInterface } from '../../../../components/learner/oral-defense-interface';
|
||||||
|
import { AssessorResultsPanel } from '../../../../components/learner/assessor-results-panel';
|
||||||
|
import { ProctorBanner } from '../../../../components/learner/proctor-banner';
|
||||||
|
import { aiArtifactSubmissions } from '@nextcraft/mock-data';
|
||||||
|
|
||||||
const RUBRIC = [
|
const RUBRIC = [
|
||||||
{ name: 'Correctness of agent architecture', passed: true, weight: 25 },
|
{ name: 'Correctness of agent architecture', passed: true, weight: 25 },
|
||||||
@@ -254,10 +257,22 @@ export default async function DefensePage({
|
|||||||
a structured-output schema and re-run the eval harness before your oral defense.
|
a structured-output schema and re-run the eval harness before your oral defense.
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{/* Live Assessor — structured rubric from the real agent (mock inputs) */}
|
||||||
|
<div className="border-t border-slate-200 pt-3 dark:border-slate-800">
|
||||||
|
<AssessorResultsPanel artifactId={aiArtifactSubmissions[0].id} />
|
||||||
|
</div>
|
||||||
</CardBody>
|
</CardBody>
|
||||||
</Card>
|
</Card>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{/* Proctor integrity banner — coaching-shaped (mock telemetry) */}
|
||||||
|
<Card>
|
||||||
|
<CardBody>
|
||||||
|
<ProctorBanner scenarioId="proctor-scenario-distracted" />
|
||||||
|
</CardBody>
|
||||||
|
</Card>
|
||||||
|
|
||||||
{/* Process trace timeline */}
|
{/* Process trace timeline */}
|
||||||
<Card>
|
<Card>
|
||||||
<CardHeader>
|
<CardHeader>
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import { ArrowLeft, ArrowRight, Clock } from 'lucide-react';
|
|||||||
import { Button, Card, CardBody, Badge } from '@nextcraft/ui';
|
import { Button, Card, CardBody, Badge } from '@nextcraft/ui';
|
||||||
import { allCompetencies, competencyStacks } from '@nextcraft/mock-data';
|
import { allCompetencies, competencyStacks } from '@nextcraft/mock-data';
|
||||||
import { WorkedExampleTabs } from '../../../../components/learner/worked-example-tabs';
|
import { WorkedExampleTabs } from '../../../../components/learner/worked-example-tabs';
|
||||||
|
import { ByteTutorPanel } from '../../../../components/learner/byte-tutor-panel';
|
||||||
|
|
||||||
export default async function ByteTutorialPage({
|
export default async function ByteTutorialPage({
|
||||||
params,
|
params,
|
||||||
@@ -39,6 +40,12 @@ export default async function ByteTutorialPage({
|
|||||||
</header>
|
</header>
|
||||||
|
|
||||||
<div className="grid grid-cols-1 gap-4 lg:grid-cols-2">
|
<div className="grid grid-cols-1 gap-4 lg:grid-cols-2">
|
||||||
|
{/* Tutor explanation panel — agent fixed to tutor (A-007) */}
|
||||||
|
<Card>
|
||||||
|
<CardBody>
|
||||||
|
<ByteTutorPanel competencyId={competency.id} competencyName={competency.name} />
|
||||||
|
</CardBody>
|
||||||
|
</Card>
|
||||||
{/* Concept panel */}
|
{/* Concept panel */}
|
||||||
<Card className="flex flex-col">
|
<Card className="flex flex-col">
|
||||||
<CardBody className="flex flex-col gap-4">
|
<CardBody className="flex flex-col gap-4">
|
||||||
|
|||||||
@@ -0,0 +1,191 @@
|
|||||||
|
'use client';
|
||||||
|
|
||||||
|
import { useState, useEffect, useRef, useCallback } from 'react';
|
||||||
|
import { parseSseEvents } from '../../lib/sse';
|
||||||
|
import { Bot, RefreshCw, AlertTriangle, Loader2 } from 'lucide-react';
|
||||||
|
|
||||||
|
const AI_SERVICE_URL =
|
||||||
|
process.env.NEXT_PUBLIC_AI_SERVICE_URL ?? 'http://localhost:8420';
|
||||||
|
|
||||||
|
interface StreamPanelProps {
|
||||||
|
title: string;
|
||||||
|
endpoint: string; // e.g. "/v1/lab/feedback"
|
||||||
|
body: Record<string, unknown>;
|
||||||
|
autoLoad?: boolean;
|
||||||
|
emptyHint?: string;
|
||||||
|
/** Renders structured JSON results (assessor/proctor) as custom UI */
|
||||||
|
renderJson?: (data: Record<string, unknown>) => React.ReactNode;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface SseResult {
|
||||||
|
text: string;
|
||||||
|
error: string | null;
|
||||||
|
streaming: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Generic SSE-consuming panel for the non-chat agent endpoints
|
||||||
|
* (lab feedback, mentor narrative, assessor/proctor JSON).
|
||||||
|
* Streams text panels; renders structured JSON via renderJson when set.
|
||||||
|
*/
|
||||||
|
export function AgentStreamPanel({
|
||||||
|
title,
|
||||||
|
endpoint,
|
||||||
|
body,
|
||||||
|
autoLoad = false,
|
||||||
|
emptyHint,
|
||||||
|
renderJson,
|
||||||
|
}: StreamPanelProps) {
|
||||||
|
const [result, setResult] = useState<SseResult>({ text: '', error: null, streaming: false });
|
||||||
|
const [json, setJson] = useState<Record<string, unknown> | null>(null);
|
||||||
|
const abortRef = useRef<AbortController | null>(null);
|
||||||
|
const startedRef = useRef(false);
|
||||||
|
|
||||||
|
const load = useCallback(async () => {
|
||||||
|
abortRef.current?.abort();
|
||||||
|
const controller = new AbortController();
|
||||||
|
abortRef.current = controller;
|
||||||
|
setResult({ text: '', error: null, streaming: true });
|
||||||
|
setJson(null);
|
||||||
|
|
||||||
|
try {
|
||||||
|
const response = await fetch(`${AI_SERVICE_URL}${endpoint}`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify(body),
|
||||||
|
signal: controller.signal,
|
||||||
|
});
|
||||||
|
const contentType = response.headers.get('content-type') ?? '';
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
throw new Error(`AI service error (${response.status})`);
|
||||||
|
}
|
||||||
|
if (contentType.includes('application/json')) {
|
||||||
|
const data = (await response.json()) as Record<string, unknown>;
|
||||||
|
setJson(data);
|
||||||
|
setResult({ text: '', error: null, streaming: false });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (!response.body) {
|
||||||
|
throw new Error('AI service returned an empty stream');
|
||||||
|
}
|
||||||
|
|
||||||
|
const reader = response.body.getReader();
|
||||||
|
const decoder = new TextDecoder();
|
||||||
|
let buffer = '';
|
||||||
|
let text = '';
|
||||||
|
let done = false;
|
||||||
|
|
||||||
|
while (!done) {
|
||||||
|
const { value, done: readerDone } = await reader.read();
|
||||||
|
if (readerDone) break;
|
||||||
|
buffer += decoder.decode(value, { stream: true });
|
||||||
|
const { events, rest } = parseSseEvents(buffer);
|
||||||
|
buffer = rest;
|
||||||
|
for (const raw of events) {
|
||||||
|
if (raw === '[DONE]') {
|
||||||
|
done = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const event = JSON.parse(raw);
|
||||||
|
if (event.type === 'delta') {
|
||||||
|
text += event.content as string;
|
||||||
|
setResult({ text, error: null, streaming: true });
|
||||||
|
} else if (event.type === 'error') {
|
||||||
|
setResult({ text, error: event.message as string, streaming: false });
|
||||||
|
done = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// ignore non-JSON frames
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
setResult((prev) => ({ ...prev, streaming: false }));
|
||||||
|
} catch (err) {
|
||||||
|
const aborted = err instanceof DOMException && err.name === 'AbortError';
|
||||||
|
if (!aborted) {
|
||||||
|
setResult({
|
||||||
|
text: '',
|
||||||
|
error: err instanceof Error ? err.message : 'connection failed',
|
||||||
|
streaming: false,
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
setResult((prev) => ({ ...prev, streaming: false }));
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
abortRef.current = null;
|
||||||
|
}
|
||||||
|
}, [endpoint, body]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
return () => {
|
||||||
|
abortRef.current?.abort();
|
||||||
|
abortRef.current = null;
|
||||||
|
};
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (autoLoad && !startedRef.current) {
|
||||||
|
startedRef.current = true;
|
||||||
|
void load();
|
||||||
|
}
|
||||||
|
}, [autoLoad, load]);
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="flex flex-col gap-3">
|
||||||
|
<div className="flex items-center justify-between">
|
||||||
|
<h3 className="inline-flex items-center gap-2 text-sm font-semibold text-slate-900 dark:text-slate-100">
|
||||||
|
<Bot className="h-4 w-4 text-primary-600 dark:text-primary-400" />
|
||||||
|
{title}
|
||||||
|
</h3>
|
||||||
|
<button
|
||||||
|
onClick={() => void load()}
|
||||||
|
disabled={result.streaming}
|
||||||
|
className="inline-flex items-center gap-1 rounded-md border border-slate-300 px-2 py-1 text-xs text-slate-600 transition-colors hover:border-primary-400 hover:text-primary-700 disabled:opacity-50 dark:border-slate-600 dark:text-slate-300 dark:hover:border-primary-400 dark:hover:text-primary-300"
|
||||||
|
>
|
||||||
|
{result.streaming ? (
|
||||||
|
<Loader2 className="h-3 w-3 animate-spin" />
|
||||||
|
) : (
|
||||||
|
<RefreshCw className="h-3 w-3" />
|
||||||
|
)}
|
||||||
|
{result.streaming ? 'Streaming…' : json || result.text ? 'Regenerate' : 'Generate'}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{result.error && (
|
||||||
|
<div
|
||||||
|
role="alert"
|
||||||
|
className="flex items-start gap-2 rounded-md border border-amber-300 bg-amber-50 px-3 py-2 text-xs text-amber-800 dark:border-amber-700 dark:bg-amber-900/30 dark:text-amber-300"
|
||||||
|
>
|
||||||
|
<AlertTriangle className="mt-0.5 h-3.5 w-3.5 shrink-0" />
|
||||||
|
<div>
|
||||||
|
<p className="font-medium">AI service unavailable</p>
|
||||||
|
<p className="opacity-80">{result.error}</p>
|
||||||
|
<button onClick={() => void load()} className="mt-1 font-semibold underline">
|
||||||
|
Retry
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{!result.error && !json && !result.text && !result.streaming && (
|
||||||
|
<p className="text-xs text-slate-500 dark:text-slate-400">
|
||||||
|
{emptyHint ?? 'Generate to see the agent in action.'}
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{json && renderJson ? (
|
||||||
|
renderJson(json)
|
||||||
|
) : result.text ? (
|
||||||
|
<div className="whitespace-pre-wrap rounded-md bg-slate-100 px-3 py-2 text-sm leading-relaxed text-slate-800 dark:bg-slate-800 dark:text-slate-100">
|
||||||
|
{result.text}
|
||||||
|
{result.streaming && (
|
||||||
|
<span className="ml-0.5 inline-block h-4 w-1.5 animate-pulse rounded-sm bg-primary-500 align-middle" />
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -1,130 +1,110 @@
|
|||||||
'use client';
|
'use client';
|
||||||
|
|
||||||
import { useState, useRef, useEffect, type FormEvent } from 'react';
|
import { useState, useRef, useEffect, type FormEvent } from 'react';
|
||||||
import { Bot, Send, User } from 'lucide-react';
|
import { Bot, Send, AlertTriangle, RotateCcw } from 'lucide-react';
|
||||||
import { aiTutorResponses, type TutorResponse } from '@nextcraft/mock-data';
|
|
||||||
import { primaryLearner } from '@nextcraft/mock-data';
|
import { primaryLearner } from '@nextcraft/mock-data';
|
||||||
import { Avatar } from '@nextcraft/ui';
|
import { Avatar } from '@nextcraft/ui';
|
||||||
|
import { useChatStream, type AgentName, type StreamMessage } from '../../hooks/use-chat-stream';
|
||||||
|
|
||||||
interface ChatMessage {
|
const AGENTS: { id: AgentName; label: string; blurb: string }[] = [
|
||||||
id: string;
|
{ id: 'coach', label: 'Coach', blurb: 'Pacing, motivation, retrieval practice' },
|
||||||
role: 'learner' | 'tutor';
|
{ id: 'tutor', label: 'Tutor', blurb: 'Concepts, worked examples, Socratic checks' },
|
||||||
content: string;
|
|
||||||
suggestedActions?: string[];
|
|
||||||
}
|
|
||||||
|
|
||||||
const SEED_MESSAGES: ChatMessage[] = [
|
|
||||||
{
|
|
||||||
id: 'seed-1',
|
|
||||||
role: 'tutor',
|
|
||||||
content:
|
|
||||||
"Welcome back, Alex. You're 62% through the AI Orchestration stack. What would you like to work on today?",
|
|
||||||
suggestedActions: ['Review my pacing', 'Start Multi-Agent Communication', 'Prep for my defense'],
|
|
||||||
},
|
|
||||||
];
|
];
|
||||||
|
|
||||||
|
const SEED_MESSAGE: StreamMessage = {
|
||||||
|
id: 'seed-1',
|
||||||
|
role: 'assistant',
|
||||||
|
content:
|
||||||
|
"Welcome back, Alex. You're 62% through the AI Orchestration stack. What would you like to work on today?",
|
||||||
|
suggestedActions: ['Review my pacing', 'Start Multi-Agent Communication', 'Prep for my defense'],
|
||||||
|
};
|
||||||
|
|
||||||
export function AiTutorChat() {
|
export function AiTutorChat() {
|
||||||
const [messages, setMessages] = useState<ChatMessage[]>(SEED_MESSAGES);
|
const [agent, setAgent] = useState<AgentName>('coach');
|
||||||
const [input, setInput] = useState('');
|
const [input, setInput] = useState('');
|
||||||
const [isTyping, setIsTyping] = useState(false);
|
const { messages, isStreaming, error, send, retry, abort } = useChatStream(agent);
|
||||||
const scrollRef = useRef<HTMLDivElement>(null);
|
const scrollRef = useRef<HTMLDivElement>(null);
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (scrollRef.current) {
|
if (scrollRef.current) {
|
||||||
scrollRef.current.scrollTop = scrollRef.current.scrollHeight;
|
scrollRef.current.scrollTop = scrollRef.current.scrollHeight;
|
||||||
}
|
}
|
||||||
}, [messages, isTyping]);
|
}, [messages, isStreaming, error]);
|
||||||
|
|
||||||
function send(e: FormEvent) {
|
function handleSubmit(e: FormEvent) {
|
||||||
e.preventDefault();
|
e.preventDefault();
|
||||||
const text = input.trim();
|
const text = input.trim();
|
||||||
if (!text || isTyping) return;
|
if (!text || isStreaming) return;
|
||||||
const learnerMsg: ChatMessage = { id: `me-${Date.now()}`, role: 'learner', content: text };
|
|
||||||
setMessages((prev) => [...prev, learnerMsg]);
|
|
||||||
setInput('');
|
setInput('');
|
||||||
setIsTyping(true);
|
void send(text);
|
||||||
|
}
|
||||||
|
|
||||||
window.setTimeout(() => {
|
function switchAgent(next: AgentName) {
|
||||||
const pick: TutorResponse =
|
if (isStreaming) abort();
|
||||||
aiTutorResponses[Math.floor(Math.random() * aiTutorResponses.length)];
|
setAgent(next);
|
||||||
const tutorMsg: ChatMessage = {
|
|
||||||
id: `tutor-${Date.now()}`,
|
|
||||||
role: 'tutor',
|
|
||||||
content: pick.message,
|
|
||||||
suggestedActions: pick.suggestedActions,
|
|
||||||
};
|
|
||||||
setMessages((prev) => [...prev, tutorMsg]);
|
|
||||||
setIsTyping(false);
|
|
||||||
}, 1000);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div className="flex h-[28rem] flex-col">
|
<div className="flex h-[28rem] flex-col">
|
||||||
|
{/* Agent switcher (A-007: explicit routing, no autonomy) */}
|
||||||
|
<div className="mb-3 flex items-center gap-2" role="tablist" aria-label="Choose tutor agent">
|
||||||
|
{AGENTS.map((a) => (
|
||||||
|
<button
|
||||||
|
key={a.id}
|
||||||
|
role="tab"
|
||||||
|
aria-selected={agent === a.id}
|
||||||
|
onClick={() => switchAgent(a.id)}
|
||||||
|
className={`rounded-full px-3 py-1 text-xs font-medium transition-colors ${
|
||||||
|
agent === a.id
|
||||||
|
? 'bg-primary-600 text-white'
|
||||||
|
: 'border border-slate-300 text-slate-600 hover:border-primary-400 hover:text-primary-700 dark:border-slate-600 dark:text-slate-300 dark:hover:text-primary-300'
|
||||||
|
}`}
|
||||||
|
>
|
||||||
|
{a.label}
|
||||||
|
</button>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
|
||||||
{/* Message list */}
|
{/* Message list */}
|
||||||
<div ref={scrollRef} className="flex-1 space-y-4 overflow-y-auto pr-2">
|
<div ref={scrollRef} className="flex-1 space-y-4 overflow-y-auto pr-2">
|
||||||
|
<ChatMessage message={SEED_MESSAGE} />
|
||||||
|
|
||||||
{messages.map((m) => (
|
{messages.map((m) => (
|
||||||
<div
|
<ChatMessage key={m.id} message={m} onAction={(a) => setInput(a)} />
|
||||||
key={m.id}
|
|
||||||
className={`flex gap-3 ${m.role === 'learner' ? 'flex-row-reverse' : 'flex-row'}`}
|
|
||||||
>
|
|
||||||
{m.role === 'tutor' ? (
|
|
||||||
<span className="flex h-8 w-8 shrink-0 items-center justify-center rounded-full bg-primary-100 text-primary-700 dark:bg-primary-900/40 dark:text-primary-300">
|
|
||||||
<Bot className="h-4 w-4" />
|
|
||||||
</span>
|
|
||||||
) : (
|
|
||||||
<Avatar name={primaryLearner.name} src={primaryLearner.avatar} size="sm" />
|
|
||||||
)}
|
|
||||||
<div
|
|
||||||
className={`max-w-[80%] rounded-lg px-3 py-2 text-sm ${
|
|
||||||
m.role === 'tutor'
|
|
||||||
? 'bg-slate-100 text-slate-800 dark:bg-slate-800 dark:text-slate-100'
|
|
||||||
: 'bg-primary-600 text-white'
|
|
||||||
}`}
|
|
||||||
>
|
|
||||||
<p className="leading-relaxed">{m.content}</p>
|
|
||||||
{m.suggestedActions && m.suggestedActions.length > 0 && (
|
|
||||||
<div className="mt-2 flex flex-wrap gap-1.5">
|
|
||||||
{m.suggestedActions.map((action) => (
|
|
||||||
<button
|
|
||||||
key={action}
|
|
||||||
onClick={() => setInput(action)}
|
|
||||||
className="rounded-full border border-slate-300 bg-white px-2 py-0.5 text-xs text-slate-600 transition-colors hover:border-primary-400 hover:text-primary-700 dark:border-slate-600 dark:bg-slate-900 dark:text-slate-300 dark:hover:border-primary-400 dark:hover:text-primary-300"
|
|
||||||
>
|
|
||||||
{action}
|
|
||||||
</button>
|
|
||||||
))}
|
|
||||||
</div>
|
|
||||||
)}
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
))}
|
))}
|
||||||
|
|
||||||
{/* Typing indicator */}
|
{/* Error state with retry (A-010) */}
|
||||||
{isTyping && (
|
{error && (
|
||||||
<div className="flex flex-row gap-3">
|
<div
|
||||||
<span className="flex h-8 w-8 shrink-0 items-center justify-center rounded-full bg-primary-100 text-primary-700 dark:bg-primary-900/40 dark:text-primary-300">
|
role="alert"
|
||||||
<Bot className="h-4 w-4" />
|
className="flex items-center gap-2 rounded-lg border border-amber-300 bg-amber-50 px-3 py-2 text-xs text-amber-800 dark:border-amber-700 dark:bg-amber-900/30 dark:text-amber-300"
|
||||||
|
>
|
||||||
|
<AlertTriangle className="h-4 w-4 shrink-0" />
|
||||||
|
<span className="flex-1">
|
||||||
|
The tutor service is unreachable. Your message can be retried.
|
||||||
</span>
|
</span>
|
||||||
<div className="flex items-center gap-1 rounded-lg bg-slate-100 px-3 py-3 dark:bg-slate-800">
|
<button
|
||||||
<span className="h-1.5 w-1.5 animate-bounce rounded-full bg-slate-400 [animation-delay:-0.3s]" />
|
onClick={retry}
|
||||||
<span className="h-1.5 w-1.5 animate-bounce rounded-full bg-slate-400 [animation-delay:-0.15s]" />
|
className="inline-flex items-center gap-1 font-semibold underline"
|
||||||
<span className="h-1.5 w-1.5 animate-bounce rounded-full bg-slate-400" />
|
>
|
||||||
</div>
|
<RotateCcw className="h-3 w-3" /> Retry
|
||||||
|
</button>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* Input */}
|
{/* Input */}
|
||||||
<form onSubmit={send} className="mt-3 flex items-center gap-2 border-t border-slate-200 pt-3 dark:border-slate-800">
|
<form onSubmit={handleSubmit} className="mt-3 flex items-center gap-2 border-t border-slate-200 pt-3 dark:border-slate-800">
|
||||||
<input
|
<input
|
||||||
value={input}
|
value={input}
|
||||||
onChange={(e) => setInput(e.target.value)}
|
onChange={(e) => setInput(e.target.value)}
|
||||||
placeholder="Ask your AI tutor anything…"
|
placeholder={`Ask your ${agent} anything…`}
|
||||||
|
aria-label="Message"
|
||||||
className="h-10 flex-1 rounded-md border border-slate-300 bg-white px-3 text-sm text-slate-900 placeholder:text-slate-400 focus:border-primary-500 focus:ring-2 focus:ring-primary-500/30 focus:outline-none dark:border-slate-700 dark:bg-slate-900 dark:text-slate-100 dark:placeholder:text-slate-500"
|
className="h-10 flex-1 rounded-md border border-slate-300 bg-white px-3 text-sm text-slate-900 placeholder:text-slate-400 focus:border-primary-500 focus:ring-2 focus:ring-primary-500/30 focus:outline-none dark:border-slate-700 dark:bg-slate-900 dark:text-slate-100 dark:placeholder:text-slate-500"
|
||||||
/>
|
/>
|
||||||
<button
|
<button
|
||||||
type="submit"
|
type="submit"
|
||||||
disabled={!input.trim() || isTyping}
|
disabled={!input.trim() || isStreaming}
|
||||||
className="inline-flex h-10 w-10 items-center justify-center rounded-md bg-primary-600 text-white transition-colors hover:bg-primary-700 disabled:opacity-50"
|
className="inline-flex h-10 w-10 items-center justify-center rounded-md bg-primary-600 text-white transition-colors hover:bg-primary-700 disabled:opacity-50"
|
||||||
aria-label="Send message"
|
aria-label="Send message"
|
||||||
>
|
>
|
||||||
@@ -134,3 +114,51 @@ export function AiTutorChat() {
|
|||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function ChatMessage({
|
||||||
|
message,
|
||||||
|
onAction,
|
||||||
|
}: {
|
||||||
|
message: StreamMessage;
|
||||||
|
onAction?: (text: string) => void;
|
||||||
|
}) {
|
||||||
|
const isAssistant = message.role === 'assistant';
|
||||||
|
return (
|
||||||
|
<div className={`flex gap-3 ${isAssistant ? 'flex-row' : 'flex-row-reverse'}`}>
|
||||||
|
{isAssistant ? (
|
||||||
|
<span className="flex h-8 w-8 shrink-0 items-center justify-center rounded-full bg-primary-100 text-primary-700 dark:bg-primary-900/40 dark:text-primary-300">
|
||||||
|
<Bot className="h-4 w-4" />
|
||||||
|
</span>
|
||||||
|
) : (
|
||||||
|
<Avatar name={primaryLearner.name} src={primaryLearner.avatar} size="sm" />
|
||||||
|
)}
|
||||||
|
<div
|
||||||
|
className={`max-w-[80%] rounded-lg px-3 py-2 text-sm ${
|
||||||
|
isAssistant
|
||||||
|
? 'bg-slate-100 text-slate-800 dark:bg-slate-800 dark:text-slate-100'
|
||||||
|
: 'bg-primary-600 text-white'
|
||||||
|
}`}
|
||||||
|
>
|
||||||
|
<p className="leading-relaxed whitespace-pre-wrap">
|
||||||
|
{message.content}
|
||||||
|
{message.streaming && (
|
||||||
|
<span className="ml-0.5 inline-block h-4 w-1.5 animate-pulse rounded-sm bg-primary-500 align-middle" />
|
||||||
|
)}
|
||||||
|
</p>
|
||||||
|
{message.suggestedActions && message.suggestedActions.length > 0 && (
|
||||||
|
<div className="mt-2 flex flex-wrap gap-1.5">
|
||||||
|
{message.suggestedActions.map((action) => (
|
||||||
|
<button
|
||||||
|
key={action}
|
||||||
|
onClick={() => onAction?.(action)}
|
||||||
|
className="rounded-full border border-slate-300 bg-white px-2 py-0.5 text-xs text-slate-600 transition-colors hover:border-primary-400 hover:text-primary-700 dark:border-slate-600 dark:bg-slate-900 dark:text-slate-300 dark:hover:border-primary-400 dark:hover:text-primary-300"
|
||||||
|
>
|
||||||
|
{action}
|
||||||
|
</button>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
'use client';
|
||||||
|
|
||||||
|
import { AgentStreamPanel } from './agent-stream-panel';
|
||||||
|
import { AlertTriangle, CheckCircle2, CircleDashed } from 'lucide-react';
|
||||||
|
|
||||||
|
interface CriterionScore {
|
||||||
|
criterion_id: string;
|
||||||
|
name: string;
|
||||||
|
score: number;
|
||||||
|
evidence: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface RubricScore {
|
||||||
|
rubric_id: string;
|
||||||
|
artifact_id: string;
|
||||||
|
competency_id: string;
|
||||||
|
scores: CriterionScore[];
|
||||||
|
strengths: string[];
|
||||||
|
gaps: string[];
|
||||||
|
verdict: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Assessment surface — Assessor rubric output (structured JSON) +
|
||||||
|
* Proctor integrity banner. Mock engine inputs; real engines v0.3+.
|
||||||
|
*/
|
||||||
|
export function AssessorResultsPanel({ artifactId }: { artifactId: string }) {
|
||||||
|
return (
|
||||||
|
<AgentStreamPanel
|
||||||
|
title="Assessor — rubric evaluation"
|
||||||
|
endpoint="/v1/assessment/evaluate"
|
||||||
|
body={{ artifact_id: artifactId }}
|
||||||
|
emptyHint="Run the Assessor to grade this artifact against its rubric."
|
||||||
|
renderJson={(data) => {
|
||||||
|
const score = data as unknown as RubricScore;
|
||||||
|
return (
|
||||||
|
<div className="flex flex-col gap-4">
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
{score.verdict === 'mastered' ? (
|
||||||
|
<CheckCircle2 className="h-5 w-5 text-emerald-600 dark:text-emerald-400" />
|
||||||
|
) : score.verdict === 'developing' ? (
|
||||||
|
<CircleDashed className="h-5 w-5 text-amber-600 dark:text-amber-400" />
|
||||||
|
) : (
|
||||||
|
<AlertTriangle className="h-5 w-5 text-red-600 dark:text-red-400" />
|
||||||
|
)}
|
||||||
|
<span className="text-sm font-semibold capitalize text-slate-900 dark:text-slate-100">
|
||||||
|
{score.verdict}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="flex flex-col gap-3">
|
||||||
|
{score.scores.map((c) => (
|
||||||
|
<div key={c.criterion_id}>
|
||||||
|
<div className="mb-1 flex items-center justify-between text-xs">
|
||||||
|
<span className="font-medium text-slate-700 dark:text-slate-300">{c.name}</span>
|
||||||
|
<span className="text-slate-500 dark:text-slate-400">{c.score}/100</span>
|
||||||
|
</div>
|
||||||
|
<div className="h-2 overflow-hidden rounded-full bg-slate-200 dark:bg-slate-700">
|
||||||
|
<div
|
||||||
|
className="h-full rounded-full bg-primary-500 transition-all"
|
||||||
|
style={{ width: `${c.score}%` }}
|
||||||
|
role="progressbar"
|
||||||
|
aria-valuenow={c.score}
|
||||||
|
aria-valuemin={0}
|
||||||
|
aria-valuemax={100}
|
||||||
|
aria-label={c.name}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
<p className="mt-1 text-xs text-slate-500 dark:text-slate-400">{c.evidence}</p>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="grid grid-cols-1 gap-3 sm:grid-cols-2">
|
||||||
|
<div>
|
||||||
|
<h4 className="mb-1 text-xs font-semibold uppercase tracking-wide text-emerald-700 dark:text-emerald-400">
|
||||||
|
Strengths
|
||||||
|
</h4>
|
||||||
|
<ul className="list-inside list-disc text-xs text-slate-600 dark:text-slate-300">
|
||||||
|
{score.strengths.map((s) => (
|
||||||
|
<li key={s}>{s}</li>
|
||||||
|
))}
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<h4 className="mb-1 text-xs font-semibold uppercase tracking-wide text-amber-700 dark:text-amber-400">
|
||||||
|
Gaps
|
||||||
|
</h4>
|
||||||
|
<ul className="list-inside list-disc text-xs text-slate-600 dark:text-slate-300">
|
||||||
|
{score.gaps.map((g) => (
|
||||||
|
<li key={g}>{g}</li>
|
||||||
|
))}
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
'use client';
|
||||||
|
|
||||||
|
import { AgentStreamPanel } from './agent-stream-panel';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Byte viewer Tutor panel — "Explain this byte" streams a Socratic concept
|
||||||
|
* walkthrough for the current competency (agent fixed to tutor, A-007).
|
||||||
|
*/
|
||||||
|
export function ByteTutorPanel({
|
||||||
|
competencyId,
|
||||||
|
competencyName,
|
||||||
|
}: {
|
||||||
|
competencyId: string;
|
||||||
|
competencyName: string;
|
||||||
|
}) {
|
||||||
|
return (
|
||||||
|
<AgentStreamPanel
|
||||||
|
title="Tutor — explain this byte"
|
||||||
|
endpoint="/v1/chat/stream"
|
||||||
|
body={{
|
||||||
|
agent: 'tutor',
|
||||||
|
session_id: `byte-${competencyId}`,
|
||||||
|
messages: [
|
||||||
|
{
|
||||||
|
role: 'user',
|
||||||
|
content: `Explain the byte "${competencyName}" — one concept, a worked example, then a question to check my understanding.`,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}}
|
||||||
|
emptyHint="Ask the Tutor to walk you through this byte concept step by step."
|
||||||
|
/>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
'use client';
|
||||||
|
|
||||||
|
import { AgentStreamPanel } from './agent-stream-panel';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Sandbox Lab feedback panel — streams in-flow feedback for the selected
|
||||||
|
* mock telemetry scenario (real telemetry is v0.3+).
|
||||||
|
*/
|
||||||
|
export function LabFeedbackPanel({ scenarioId }: { scenarioId: string }) {
|
||||||
|
return (
|
||||||
|
<AgentStreamPanel
|
||||||
|
title="Lab — in-flow feedback"
|
||||||
|
endpoint="/v1/lab/feedback"
|
||||||
|
body={{ scenario_id: scenarioId }}
|
||||||
|
emptyHint="Run the Lab agent on this build session's telemetry."
|
||||||
|
/>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
'use client';
|
||||||
|
|
||||||
|
import { AgentStreamPanel } from './agent-stream-panel';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Dashboard Mentor panel — streams a long-horizon career narrative
|
||||||
|
* tied to the learner's progress. Session-backed follow-ups in v0.3+ UI.
|
||||||
|
*/
|
||||||
|
export function MentorPanel() {
|
||||||
|
return (
|
||||||
|
<AgentStreamPanel
|
||||||
|
title="Mentor — your trajectory"
|
||||||
|
endpoint="/v1/mentor/narrative"
|
||||||
|
body={{ session_id: 'dashboard-mentor', prompt: 'Narrate my trajectory.' }}
|
||||||
|
emptyHint="Ask the Mentor where your competency progress is taking you."
|
||||||
|
/>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
'use client';
|
||||||
|
|
||||||
|
import { AgentStreamPanel } from './agent-stream-panel';
|
||||||
|
import { ShieldCheck } from 'lucide-react';
|
||||||
|
|
||||||
|
interface IntegritySignal {
|
||||||
|
signal_type: string;
|
||||||
|
severity: 'low' | 'medium' | 'high';
|
||||||
|
note: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface ProctorAssessment {
|
||||||
|
scenario_id: string;
|
||||||
|
signals: IntegritySignal[];
|
||||||
|
intervention: string;
|
||||||
|
summary: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
const SEVERITY_STYLES: Record<string, string> = {
|
||||||
|
low: 'bg-emerald-50 text-emerald-700 border-emerald-200 dark:bg-emerald-900/30 dark:text-emerald-300 dark:border-emerald-800',
|
||||||
|
medium: 'bg-amber-50 text-amber-700 border-amber-200 dark:bg-amber-900/30 dark:text-amber-300 dark:border-amber-800',
|
||||||
|
high: 'bg-red-50 text-red-700 border-red-200 dark:bg-red-900/30 dark:text-red-300 dark:border-red-800',
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Proctor integrity banner — supportive, coaching-shaped (never punitive).
|
||||||
|
*/
|
||||||
|
export function ProctorBanner({ scenarioId }: { scenarioId: string }) {
|
||||||
|
return (
|
||||||
|
<AgentStreamPanel
|
||||||
|
title="Proctor — integrity support"
|
||||||
|
endpoint="/v1/proctor/signals"
|
||||||
|
body={{ scenario_id: scenarioId }}
|
||||||
|
emptyHint="Run the Proctor to review this session's integrity signals."
|
||||||
|
renderJson={(data) => {
|
||||||
|
const assessment = data as unknown as ProctorAssessment;
|
||||||
|
return (
|
||||||
|
<div className="flex flex-col gap-3">
|
||||||
|
<p className="text-xs text-slate-600 dark:text-slate-300">{assessment.summary}</p>
|
||||||
|
<div className="flex flex-wrap gap-1.5">
|
||||||
|
{assessment.signals.map((s, i) => (
|
||||||
|
<span
|
||||||
|
key={i}
|
||||||
|
className={`inline-flex items-center gap-1 rounded-full border px-2 py-0.5 text-xs ${SEVERITY_STYLES[s.severity] ?? SEVERITY_STYLES.low}`}
|
||||||
|
title={s.note}
|
||||||
|
>
|
||||||
|
<ShieldCheck className="h-3 w-3" />
|
||||||
|
{s.signal_type} · {s.severity}
|
||||||
|
</span>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
<p className="rounded-md bg-primary-50 px-3 py-2 text-xs text-primary-800 dark:bg-primary-900/30 dark:text-primary-200">
|
||||||
|
<strong>Suggested next step:</strong> {assessment.intervention}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,218 @@
|
|||||||
|
'use client';
|
||||||
|
|
||||||
|
import { useCallback, useEffect, useRef, useState } from 'react';
|
||||||
|
import { parseSseEvents } from '../lib/sse';
|
||||||
|
|
||||||
|
const AI_SERVICE_URL =
|
||||||
|
process.env.NEXT_PUBLIC_AI_SERVICE_URL ?? 'http://localhost:8420';
|
||||||
|
|
||||||
|
export type AgentName = 'coach' | 'tutor' | 'lab' | 'assessor' | 'proctor' | 'mentor';
|
||||||
|
|
||||||
|
export interface StreamMessage {
|
||||||
|
id: string;
|
||||||
|
role: 'user' | 'assistant';
|
||||||
|
content: string;
|
||||||
|
agent?: AgentName;
|
||||||
|
streaming?: boolean;
|
||||||
|
suggestedActions?: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
interface StreamState {
|
||||||
|
messages: StreamMessage[];
|
||||||
|
isStreaming: boolean;
|
||||||
|
error: string | null;
|
||||||
|
model: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface ChatStreamEvent {
|
||||||
|
type: 'meta' | 'delta' | 'done' | 'error';
|
||||||
|
[key: string]: unknown;
|
||||||
|
}
|
||||||
|
|
||||||
|
function decodeEvent(raw: string): ChatStreamEvent | '[DONE]' | null {
|
||||||
|
if (raw === '[DONE]') return '[DONE]';
|
||||||
|
try {
|
||||||
|
const parsed = JSON.parse(raw);
|
||||||
|
if (typeof parsed?.type === 'string') return parsed as ChatStreamEvent;
|
||||||
|
// OpenAI-shaped chunks (id/choices) are not used by our envelope;
|
||||||
|
// ignore anything without a type.
|
||||||
|
return null;
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useChatStream(agent: AgentName) {
|
||||||
|
const [state, setState] = useState<StreamState>({
|
||||||
|
messages: [],
|
||||||
|
isStreaming: false,
|
||||||
|
error: null,
|
||||||
|
model: null,
|
||||||
|
});
|
||||||
|
const abortRef = useRef<AbortController | null>(null);
|
||||||
|
// Agent-scoped sessions (A-007/D-019): switching agents starts a NEW
|
||||||
|
// session per agent — no persona bleed across switcher flips.
|
||||||
|
const sessionsRef = useRef<Partial<Record<AgentName, string>>>({});
|
||||||
|
if (!sessionsRef.current[agent]) {
|
||||||
|
const uuid =
|
||||||
|
typeof crypto !== 'undefined' && 'randomUUID' in crypto
|
||||||
|
? crypto.randomUUID()
|
||||||
|
: String(Date.now());
|
||||||
|
sessionsRef.current[agent] = `${agent}-${uuid}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Idempotent abort + cleanup on unmount or agent switch (Strict Mode safe)
|
||||||
|
const abort = useCallback(() => {
|
||||||
|
abortRef.current?.abort();
|
||||||
|
abortRef.current = null;
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
return () => {
|
||||||
|
abortRef.current?.abort();
|
||||||
|
abortRef.current = null;
|
||||||
|
};
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const send = useCallback(
|
||||||
|
async (text: string) => {
|
||||||
|
const trimmed = text.trim();
|
||||||
|
if (!trimmed || abortRef.current) return;
|
||||||
|
|
||||||
|
const userMessage: StreamMessage = {
|
||||||
|
id: `user-${Date.now()}`,
|
||||||
|
role: 'user',
|
||||||
|
content: trimmed,
|
||||||
|
};
|
||||||
|
const assistantId = `assistant-${Date.now()}`;
|
||||||
|
|
||||||
|
setState((prev) => ({
|
||||||
|
...prev,
|
||||||
|
messages: [...prev.messages, userMessage],
|
||||||
|
isStreaming: true,
|
||||||
|
error: null,
|
||||||
|
}));
|
||||||
|
|
||||||
|
const controller = new AbortController();
|
||||||
|
abortRef.current = controller;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const response = await fetch(`${AI_SERVICE_URL}/v1/chat/stream`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({
|
||||||
|
agent,
|
||||||
|
session_id: sessionsRef.current[agent],
|
||||||
|
messages: [{ role: 'user', content: trimmed }],
|
||||||
|
}),
|
||||||
|
signal: controller.signal,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!response.ok || !response.body) {
|
||||||
|
throw new Error(`AI service unavailable (${response.status})`);
|
||||||
|
}
|
||||||
|
|
||||||
|
setState((prev) => ({
|
||||||
|
...prev,
|
||||||
|
messages: [
|
||||||
|
...prev.messages,
|
||||||
|
{ id: assistantId, role: 'assistant', content: '', agent, streaming: true },
|
||||||
|
],
|
||||||
|
}));
|
||||||
|
|
||||||
|
const reader = response.body.getReader();
|
||||||
|
const decoder = new TextDecoder();
|
||||||
|
let buffer = '';
|
||||||
|
let done = false;
|
||||||
|
|
||||||
|
while (!done) {
|
||||||
|
const { value, done: readerDone } = await reader.read();
|
||||||
|
if (readerDone) break;
|
||||||
|
buffer += decoder.decode(value, { stream: true });
|
||||||
|
|
||||||
|
const { events, rest } = parseSseEvents(buffer);
|
||||||
|
buffer = rest;
|
||||||
|
|
||||||
|
for (const raw of events) {
|
||||||
|
const event = decodeEvent(raw);
|
||||||
|
if (event === null) continue;
|
||||||
|
if (event === '[DONE]') {
|
||||||
|
done = true;
|
||||||
|
setState((prev) => ({
|
||||||
|
...prev,
|
||||||
|
isStreaming: false,
|
||||||
|
messages: prev.messages.map((m) =>
|
||||||
|
m.id === assistantId ? { ...m, streaming: false } : m,
|
||||||
|
),
|
||||||
|
}));
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if (event.type === 'meta') {
|
||||||
|
setState((prev) => ({ ...prev, model: (event.model as string) ?? null }));
|
||||||
|
} else if (event.type === 'delta') {
|
||||||
|
const content = event.content as string;
|
||||||
|
setState((prev) => ({
|
||||||
|
...prev,
|
||||||
|
messages: prev.messages.map((m) =>
|
||||||
|
m.id === assistantId ? { ...m, content: m.content + content } : m,
|
||||||
|
),
|
||||||
|
}));
|
||||||
|
} else if (event.type === 'error') {
|
||||||
|
setState((prev) => ({
|
||||||
|
...prev,
|
||||||
|
error: (event.message as string) ?? 'stream error',
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
setState((prev) => ({
|
||||||
|
...prev,
|
||||||
|
isStreaming: false,
|
||||||
|
messages: prev.messages.map((m) =>
|
||||||
|
m.id === assistantId ? { ...m, streaming: false } : m,
|
||||||
|
),
|
||||||
|
}));
|
||||||
|
} catch (err) {
|
||||||
|
const aborted = err instanceof DOMException && err.name === 'AbortError';
|
||||||
|
if (!aborted) {
|
||||||
|
setState((prev) => ({
|
||||||
|
...prev,
|
||||||
|
isStreaming: false,
|
||||||
|
error: err instanceof Error ? err.message : 'connection failed',
|
||||||
|
messages: prev.messages.map((m) =>
|
||||||
|
m.id === assistantId ? { ...m, streaming: false } : m,
|
||||||
|
),
|
||||||
|
}));
|
||||||
|
} else {
|
||||||
|
setState((prev) => ({
|
||||||
|
...prev,
|
||||||
|
isStreaming: false,
|
||||||
|
messages: prev.messages.map((m) =>
|
||||||
|
m.id === assistantId ? { ...m, streaming: false } : m,
|
||||||
|
),
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
abortRef.current = null;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
[agent],
|
||||||
|
);
|
||||||
|
|
||||||
|
const retry = useCallback(() => {
|
||||||
|
setState((prev) => ({ ...prev, error: null }));
|
||||||
|
const lastUser = [...state.messages].reverse().find((m) => m.role === 'user');
|
||||||
|
if (lastUser) void send(lastUser.content);
|
||||||
|
}, [send, state.messages]);
|
||||||
|
|
||||||
|
return {
|
||||||
|
messages: state.messages,
|
||||||
|
isStreaming: state.isStreaming,
|
||||||
|
error: state.error,
|
||||||
|
model: state.model,
|
||||||
|
send,
|
||||||
|
retry,
|
||||||
|
abort,
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
/**
|
||||||
|
* Shared SSE parsing for the AI service streams.
|
||||||
|
*
|
||||||
|
* Normalizes CRLF (sse-starlette's wire terminator is \r\n) to LF, then
|
||||||
|
* splits frames on blank lines. Multiple `data:` lines within one frame
|
||||||
|
* are joined with \n per the SSE spec. Frames with no data lines
|
||||||
|
* (keep-alive `: ping` comments) are ignored (G-1).
|
||||||
|
*/
|
||||||
|
export function parseSseEvents(buffer: string): { events: string[]; rest: string } {
|
||||||
|
const normalized = buffer.replace(/\r\n/g, '\n');
|
||||||
|
const events: string[] = [];
|
||||||
|
|
||||||
|
const separatorIndex = normalized.lastIndexOf('\n\n');
|
||||||
|
if (separatorIndex === -1) return { events, rest: normalized };
|
||||||
|
|
||||||
|
const complete = normalized.slice(0, separatorIndex);
|
||||||
|
const rest = normalized.slice(separatorIndex + 2);
|
||||||
|
|
||||||
|
for (const frame of complete.split('\n\n')) {
|
||||||
|
const dataLines = frame
|
||||||
|
.split('\n')
|
||||||
|
.filter((line) => line.startsWith('data:'))
|
||||||
|
.map((line) => line.slice(5).trimStart());
|
||||||
|
if (dataLines.length === 0) continue; // ping/comment frame — ignore (G-1)
|
||||||
|
events.push(dataLines.join('\n'));
|
||||||
|
}
|
||||||
|
return { events, rest };
|
||||||
|
}
|
||||||
+5
-1
@@ -9,7 +9,11 @@
|
|||||||
"build": "turbo build",
|
"build": "turbo build",
|
||||||
"lint": "turbo lint",
|
"lint": "turbo lint",
|
||||||
"typecheck": "turbo typecheck",
|
"typecheck": "turbo typecheck",
|
||||||
"clean": "turbo clean && rm -rf node_modules"
|
"clean": "turbo clean && rm -rf node_modules",
|
||||||
|
"ai:dev": "turbo run dev --filter=@nextcraft/ai-service",
|
||||||
|
"ai:test": "turbo run test --filter=@nextcraft/ai-service",
|
||||||
|
"ai:bootstrap": "turbo run bootstrap --filter=@nextcraft/ai-service",
|
||||||
|
"ai:lint": "turbo run lint --filter=@nextcraft/ai-service"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"turbo": "^2.3.3",
|
"turbo": "^2.3.3",
|
||||||
|
|||||||
@@ -0,0 +1,47 @@
|
|||||||
|
/**
|
||||||
|
* AI engine-input scenario IDs + display metadata (v0.2 Phase 6 learner panels).
|
||||||
|
*
|
||||||
|
* D-021 alignment: IDs are string-identical to the Python corpus
|
||||||
|
* `apps/ai-service/ai_service/corpus/telemetry.py` and `artifacts.py`.
|
||||||
|
* Do not rename one side without the other. Real engines (sandbox fabric,
|
||||||
|
* assessment engine) are v0.3+.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export const aiLabScenarios = [
|
||||||
|
{
|
||||||
|
id: 'lab-scenario-strong',
|
||||||
|
title: 'Strong build session — multi-agent research assistant',
|
||||||
|
competencyId: 'stack-orchestration-c002',
|
||||||
|
description: 'Steady progress, checkpoints, and passing tests — the healthy pattern.',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'lab-scenario-struggling',
|
||||||
|
title: 'Struggling build session — repeated failures, no checkpoints',
|
||||||
|
competencyId: 'stack-orchestration-c002',
|
||||||
|
description: 'Same failure twice, long idle gaps, no recovery strategy.',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'lab-scenario-flagged',
|
||||||
|
title: 'Flagged build session — large paste, instant pass',
|
||||||
|
competencyId: 'stack-orchestration-c003',
|
||||||
|
description: 'Suspicious velocity worth a supportive check-in, not a penalty.',
|
||||||
|
},
|
||||||
|
] as const;
|
||||||
|
|
||||||
|
export const aiArtifactSubmissions = [
|
||||||
|
{
|
||||||
|
id: 'art-eval-research-assistant',
|
||||||
|
name: 'Multi-agent research assistant (eval build)',
|
||||||
|
competencyId: 'stack-orchestration-c002',
|
||||||
|
description: 'LangGraph-based assistant with clean state boundaries and retries.',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'art-eval-rag-dashboard',
|
||||||
|
name: 'RAG retrieval quality dashboard (eval build)',
|
||||||
|
competencyId: 'stack-orchestration-c003',
|
||||||
|
description: 'Eval harness sweep with gaps in error handling — partial mastery.',
|
||||||
|
},
|
||||||
|
] as const;
|
||||||
|
|
||||||
|
export type AiLabScenario = (typeof aiLabScenarios)[number];
|
||||||
|
export type AiArtifactSubmission = (typeof aiArtifactSubmissions)[number];
|
||||||
@@ -1,112 +0,0 @@
|
|||||||
/**
|
|
||||||
* Pre-scripted AI tutor responses for the prototype.
|
|
||||||
* Each pair has a Coach (motivational, pacing) and Tutor (concept, Socratic)
|
|
||||||
* variant keyed by intent.
|
|
||||||
*/
|
|
||||||
|
|
||||||
export interface TutorResponse {
|
|
||||||
id: string;
|
|
||||||
intent: string;
|
|
||||||
agent: 'coach' | 'tutor';
|
|
||||||
message: string;
|
|
||||||
suggestedActions?: string[];
|
|
||||||
}
|
|
||||||
|
|
||||||
export const aiTutorResponses: TutorResponse[] = [
|
|
||||||
{
|
|
||||||
id: 'resp-001',
|
|
||||||
intent: 'stuck-on-concept',
|
|
||||||
agent: 'coach',
|
|
||||||
message:
|
|
||||||
"I can see you've hit a wall on RAG Pipeline Design — that's the most common sticking point in this stack. Take a 5-minute break, then come back and let's try a smaller chunk. You've already mastered Tool Use, so the retrieval pieces will click faster than you expect.",
|
|
||||||
suggestedActions: ['Take a break', 'Review a worked example', 'Break it into smaller steps'],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 'resp-002',
|
|
||||||
intent: 'stuck-on-concept',
|
|
||||||
agent: 'tutor',
|
|
||||||
message:
|
|
||||||
"Let's build intuition first. RAG is a two-stage retrieval pipeline: (1) given a query, find the most relevant passages from a corpus, and (2) feed those passages into the model as context so its answer is grounded. Socratic question: why might a lexical search (like BM25) struggle with a query phrased very differently from the document text?",
|
|
||||||
suggestedActions: ['Show me a worked example', 'Compare BM25 vs embeddings', 'Try a practice problem'],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 'resp-003',
|
|
||||||
intent: 'request-pacing',
|
|
||||||
agent: 'coach',
|
|
||||||
message:
|
|
||||||
"You're 62% through the AI Orchestration stack and ahead of your original pace by nine days. Your next two defenses are scheduled in 11 days. I recommend a 3-day sprint on Multi-Agent Communication, then one light day before the defense prep.",
|
|
||||||
suggestedActions: ['Show my weekly plan', 'Schedule a lighter day', 'Adjust pace'],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 'resp-004',
|
|
||||||
intent: 'request-pacing',
|
|
||||||
agent: 'tutor',
|
|
||||||
message:
|
|
||||||
'Here is a suggested sequence: complete the Multi-Agent Communication module in 3 sessions of 45 minutes, then the Memory Systems primer in 2 sessions. After that, one full session for defense rehearsal. This matches the cognitive load profile of learners who succeeded before you.',
|
|
||||||
suggestedActions: ['Lock in this plan', 'Adjust session length', 'See alternatives'],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 'resp-005',
|
|
||||||
intent: 'celebrate-milestone',
|
|
||||||
agent: 'coach',
|
|
||||||
message:
|
|
||||||
"Huge milestone — you just earned your Tool Use & Function Calling microcredential with a 91. That's above the cohort median. You now have three of the five core orchestration credentials. Two more and you qualify for the stack endorsement.",
|
|
||||||
suggestedActions: ['See path to endorsement', 'Share with employer partners', 'Pick the next competency'],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 'resp-006',
|
|
||||||
intent: 'celebrate-milestone',
|
|
||||||
agent: 'tutor',
|
|
||||||
message:
|
|
||||||
'Your artifact for this competency showed clean handling of structured outputs and graceful fallback when the model returned malformed JSON. For your next competency, I recommend exploring how tool-use composes with multi-agent communication — that will set you up for the stack endorsement.',
|
|
||||||
suggestedActions: ['Start Multi-Agent Communication', 'Review a harder artifact example', 'Read a case study'],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 'resp-007',
|
|
||||||
intent: 'explain-socratic',
|
|
||||||
agent: 'tutor',
|
|
||||||
message:
|
|
||||||
"Think about agent memory as three layers: short-term context (the current conversation window), episodic memory (records of past agent runs you can replay), and long-term semantic memory (consolidated knowledge you distill over time). Question: if an agent handles 10,000 sessions a day, which layer would you consolidate into long-term memory, and how often?",
|
|
||||||
suggestedActions: ['I have an answer', 'Give me a hint', 'Show a worked example'],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 'resp-008',
|
|
||||||
intent: 'defense-prep',
|
|
||||||
agent: 'coach',
|
|
||||||
message:
|
|
||||||
"Your oral defense for Multi-Agent Communication is in 11 days. You are well prepared — your process trace shows 14 distinct practice runs. Let's spend one session rehearsing the edge-case questions, then a mock defense with me acting as the panel.",
|
|
||||||
suggestedActions: ['Start mock defense', 'Review edge-case questions', 'See my process trace'],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 'resp-009',
|
|
||||||
intent: 'defense-prep',
|
|
||||||
agent: 'tutor',
|
|
||||||
message:
|
|
||||||
"In a defense you'll be asked to explain your design trade-offs, not just your code. Be ready to answer: why did you choose a blackboard architecture over direct message passing? What failure mode did you observe under load, and how did you mitigate it? Let's rehearse one question now.",
|
|
||||||
suggestedActions: ['Rehearse question 1', 'Rehearse question 2', 'See scoring rubric'],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 'resp-010',
|
|
||||||
intent: 'career-guidance',
|
|
||||||
agent: 'coach',
|
|
||||||
message:
|
|
||||||
"Based on your competencies and artifacts, you match strongly to AI Orchestration Engineer roles (96%) and Agent Reliability Engineer roles (90%). Two employers are actively hiring for these profiles. Want me to show you the matching jobs and the remaining competencies they require?",
|
|
||||||
suggestedActions: ['Show matching jobs', 'See competency gaps', 'Build a targeted plan'],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 'resp-011',
|
|
||||||
intent: 'career-guidance',
|
|
||||||
agent: 'tutor',
|
|
||||||
message:
|
|
||||||
'Your portfolio demonstrates multi-agent systems and evaluation, which are the two most-cited skills in senior orchestration postings. The gap to a Staff-level role is observability and cost optimization. I recommend the Agent Reliability Engineer competencies as your next sprint.',
|
|
||||||
suggestedActions: ['Start reliability sprint', 'See a staff-level job', 'Compare skill gaps'],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 'resp-012',
|
|
||||||
intent: 'check-understanding',
|
|
||||||
agent: 'tutor',
|
|
||||||
message:
|
|
||||||
'Quick check: in a plan-and-execute agent, what is the advantage of re-planning after each tool call rather than executing the full plan from the start? Take your time — there is no penalty for thinking.',
|
|
||||||
suggestedActions: ['I have an answer', 'Give me a hint', 'Skip this check'],
|
|
||||||
},
|
|
||||||
];
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user