docs(milestone): complete v0.2-ai-tutor-architecture
---ci--- phase: 7 milestone: v0.2 status: complete requirements: covered: [REQ-2-001, REQ-2-002, REQ-2-003, REQ-2-004, REQ-2-005, REQ-2-006, REQ-2-007, REQ-2-008, REQ-2-009, REQ-2-010, REQ-2-011, REQ-2-012] partial: [] ---/ci--- Milestone v0.2 (ai-tutor-architecture) merged to main. Escalation record (audit remediation, durable): P1 executor delegation failed twice (empty subagent results, zero files created); auto-resolved at full autonomy to inline execution with identical plan fidelity (commit 3271373, reflog-only after phase branch squash-delete).
This commit is contained in:
CIAgent
@@ -0,0 +1,10 @@
|
||||
# Nextcraft AI Service — environment template (copy values, never commit real keys)
|
||||
# Real keys live in .ciagent/.env.secrets (gitignored) and are exported by scripts/dev.sh
|
||||
|
||||
AI_PORT=8420
|
||||
AI_PROVIDER=ollama-cloud
|
||||
AI_MODEL=gemma4:31b
|
||||
AI_OLLAMA_CLOUD_BASE_URL=https://ollama.com/v1
|
||||
AI_OLLAMA_CLOUD_API_KEY=
|
||||
AI_LOCAL_BASE_URL=http://localhost:11434/v1
|
||||
AI_JSON_MODE=auto
|
||||
@@ -0,0 +1,90 @@
|
||||
# Nextcraft AI Service (`apps/ai-service`)
|
||||
|
||||
Python FastAPI service hosting the six AI tutor agents (Coach, Tutor, Lab, Assessor, Proctor, Mentor) behind a provider-agnostic LLM layer. Port **8420**.
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
# 1. Bootstrap (idempotent): venv + deps
|
||||
bash scripts/bootstrap.sh
|
||||
|
||||
# 2. Run tests (mock provider only — zero network calls)
|
||||
bash scripts/test.sh
|
||||
|
||||
# 3. Lint
|
||||
bash scripts/lint.sh
|
||||
|
||||
# 4. Dev server (exports keys from .ciagent/.env.secrets if present)
|
||||
bash scripts/dev.sh
|
||||
```
|
||||
|
||||
Or via the monorepo root (`corepack pnpm install` first):
|
||||
|
||||
```bash
|
||||
pnpm ai:bootstrap
|
||||
pnpm ai:test
|
||||
pnpm ai:lint
|
||||
pnpm ai:dev
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
All settings use the `AI_` env prefix (pydantic-settings; see `.env.example`).
|
||||
|
||||
| Var | Default | Purpose |
|
||||
|-----|---------|---------|
|
||||
| `AI_PORT` | 8420 | Listen port |
|
||||
| `AI_PROVIDER` | mock | `ollama-cloud` \| `local` \| `mock` |
|
||||
| `AI_MODEL` | gemma4:31b | Model for all agents |
|
||||
| `AI_OLLAMA_CLOUD_BASE_URL` | https://ollama.com/v1 | Cloud base URL |
|
||||
| `AI_OLLAMA_CLOUD_API_KEY` | (empty) | Bearer key — **never commit** |
|
||||
| `AI_JSON_MODE` | auto | `auto` sends response_format, degrades on 400; `off` never sends |
|
||||
|
||||
Tests run with `AI_PROVIDER=mock` (enforced in `tests/conftest.py` by an instance assertion) — the suite never calls the cloud.
|
||||
|
||||
## Endpoints
|
||||
|
||||
- `GET /health` — status, configured provider, model (no cloud call)
|
||||
- `POST /v1/chat/stream` — SSE chat stream. Body: `{"agent": "coach"|"tutor", "session_id": "...", "messages": [{"role":"user","content":"..."}]}`. Unknown agents are rejected with 422.
|
||||
|
||||
SSE envelope (D-016): `meta` event first (agent/session/model), then `delta` events (incremental content), then `done`; on mid-stream failure an `error` event precedes the terminal `[DONE]` sentinel. sse-starlette emits `: ping` keep-alive comment lines on idle connections — clients must ignore frames without `data:`.
|
||||
|
||||
## Manual ollama-cloud persona probe (Phase 3, documented — not automated)
|
||||
|
||||
With the real provider, Coach and Tutor must produce distinct on-persona
|
||||
responses to the same prompt:
|
||||
|
||||
```bash
|
||||
# start with the cloud provider (keys exported from .ciagent/.env.secrets)
|
||||
AI_PROVIDER=ollama-cloud .venv/bin/uvicorn ai_service.main:app --port 8420
|
||||
|
||||
# Coach: expect pacing + one concrete next action + a retrieval-practice question
|
||||
curl -sN -X POST localhost:8420/v1/chat/stream -H 'Content-Type: application/json' \
|
||||
-d '{"agent":"coach","session_id":"probe-coach","messages":[{"role":"user","content":"I am stuck on multi-agent communication patterns"}]}' \
|
||||
| grep '^data:'
|
||||
|
||||
# Tutor: expect ONE concept + a worked example + a Socratic check question
|
||||
curl -sN -X POST localhost:8420/v1/chat/stream -H 'Content-Type: application/json' \
|
||||
-d '{"agent":"tutor","session_id":"probe-tutor","messages":[{"role":"user","content":"I am stuck on multi-agent communication patterns"}]}' \
|
||||
| grep '^data:'
|
||||
```
|
||||
|
||||
Verify: the two responses have visibly different voice/structure (Coach:
|
||||
action + accountability; Tutor: concept + example + question). The
|
||||
automated suite never calls the cloud — distinctness is enforced against
|
||||
the deterministic mock (distinct system prompts → distinct hash-seeded
|
||||
outputs).
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
ai_service/
|
||||
main.py app factory, lifespan (httpx pool), CORS, /health
|
||||
config.py pydantic-settings
|
||||
api/ endpoints (SSE envelope lives here, D-016)
|
||||
llm/ provider layer — dumb pipe, no envelope logic
|
||||
scripts/ bootstrap.sh dev.sh test.sh lint.sh
|
||||
tests/ pytest — mock provider only
|
||||
```
|
||||
|
||||
Boundary rules: `llm/` imports nothing from `agents/` or `api/`; `agents/` imports nothing from `api/`.
|
||||
@@ -0,0 +1,3 @@
|
||||
"""Nextcraft AI tutor service — six LLM agents behind a provider-agnostic layer."""
|
||||
|
||||
__version__ = "0.2.0"
|
||||
@@ -0,0 +1,18 @@
|
||||
"""Agent framework — BaseAgent ABC (D-018), registry, sessions, structured outputs.
|
||||
|
||||
Boundary rule: agents/ imports from llm/, prompts/, corpus/ — never from api/.
|
||||
"""
|
||||
|
||||
from .base import BaseAgent
|
||||
from .registry import AgentRegistry
|
||||
from .session import InMemorySessionStore, SessionStore
|
||||
from .structured import StructuredOutputError, extract_json_object
|
||||
|
||||
__all__ = [
|
||||
"AgentRegistry",
|
||||
"BaseAgent",
|
||||
"InMemorySessionStore",
|
||||
"SessionStore",
|
||||
"StructuredOutputError",
|
||||
"extract_json_object",
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
"""AssessorAgent — rubric application to pre-baked artifacts (REQ-2-008).
|
||||
|
||||
Structured-output showcase: applies the 4-layer defense (D-020) to return
|
||||
a pydantic-validated rubric score. Mock engine inputs (corpus artifacts +
|
||||
transcripts); real process-trace grading is v0.3+.
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..corpus.artifacts import (
|
||||
ArtifactSubmission,
|
||||
AssessmentRubric,
|
||||
DefenseTranscript,
|
||||
render_rubric,
|
||||
render_transcript,
|
||||
)
|
||||
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||
from ..prompts.assessor import SYSTEM_PROMPT, render_context
|
||||
from .base import BaseAgent
|
||||
|
||||
|
||||
class CriterionScore(BaseModel):
|
||||
criterion_id: str
|
||||
name: str
|
||||
score: int = Field(ge=0, le=100)
|
||||
evidence: str
|
||||
|
||||
|
||||
class RubricScore(BaseModel):
|
||||
rubric_id: str
|
||||
artifact_id: str
|
||||
competency_id: str
|
||||
scores: list[CriterionScore]
|
||||
strengths: list[str] = Field(min_length=1, max_length=2)
|
||||
gaps: list[str] = Field(min_length=1, max_length=2)
|
||||
verdict: str # "mastered" | "developing" | "not_yet"
|
||||
|
||||
def weighted_total(self, rubric: AssessmentRubric) -> float:
|
||||
by_id = {c.criterion_id: c for c in rubric.criteria}
|
||||
total = 0.0
|
||||
for s in self.scores:
|
||||
total += s.score * by_id[s.criterion_id].weight
|
||||
return total
|
||||
|
||||
|
||||
RUBRIC_SCORE_SCHEMA_HINT = (
|
||||
'{"rubric_id": "<id>", "artifact_id": "<id>", "competency_id": "<id>", '
|
||||
'"scores": [{"criterion_id": "<id>", "name": "<name>", "score": <0-100>, '
|
||||
'"evidence": "<one sentence>"}], "strengths": ["<one sentence>"], '
|
||||
'"gaps": ["<one sentence>"], "verdict": "mastered"|"developing"|"not_yet"}'
|
||||
)
|
||||
|
||||
|
||||
class AssessorAgent(BaseAgent):
|
||||
name = "assessor"
|
||||
|
||||
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||
ctx = learner_context or get_learner_context()
|
||||
return SYSTEM_PROMPT.format_map(render_context(ctx))
|
||||
|
||||
def build_evaluation_input(
|
||||
self,
|
||||
artifact: ArtifactSubmission,
|
||||
rubric: AssessmentRubric,
|
||||
transcript: DefenseTranscript | None,
|
||||
) -> str:
|
||||
parts = [
|
||||
f"ARTIFACT: {artifact.name} ({artifact.artifact_id})",
|
||||
f"Evidence excerpt: {artifact.evidence_excerpt}",
|
||||
"",
|
||||
render_rubric(rubric),
|
||||
]
|
||||
if transcript is not None:
|
||||
parts += ["", render_transcript(transcript)]
|
||||
return "\n".join(parts)
|
||||
|
||||
async def evaluate(
|
||||
self,
|
||||
artifact: ArtifactSubmission,
|
||||
rubric: AssessmentRubric,
|
||||
transcript: DefenseTranscript | None,
|
||||
learner_context: LearnerContext | None = None,
|
||||
) -> RubricScore:
|
||||
evaluation_input = self.build_evaluation_input(artifact, rubric, transcript)
|
||||
score: RubricScore = await self.structured_reply(
|
||||
history=None,
|
||||
user_input=evaluation_input,
|
||||
learner_context=learner_context,
|
||||
schema=RubricScore,
|
||||
schema_hint=RUBRIC_SCORE_SCHEMA_HINT,
|
||||
)
|
||||
return score
|
||||
@@ -0,0 +1,77 @@
|
||||
"""BaseAgent ABC — the contract all six tutor agents implement (D-018).
|
||||
|
||||
Subclasses set `name`, override `system_prompt()`, and rarely `stream_reply()`.
|
||||
The default pipeline: build_messages() → provider.stream_chat()/chat().
|
||||
"""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ..config import Settings
|
||||
from ..corpus.learner_context import LearnerContext
|
||||
from ..llm.base import LLMProvider
|
||||
from ..llm.types import Message
|
||||
from .structured import structured_completion
|
||||
|
||||
|
||||
class BaseAgent(ABC):
|
||||
"""A tutor agent: system prompt + message assembly + provider delegation."""
|
||||
|
||||
name: str = "base"
|
||||
|
||||
def __init__(self, provider: LLMProvider, settings: Settings) -> None:
|
||||
self.provider = provider
|
||||
self.settings = settings
|
||||
|
||||
@abstractmethod
|
||||
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||
"""Return the agent's system prompt, learner-context-aware."""
|
||||
|
||||
def build_messages(
|
||||
self,
|
||||
history: list[Message] | None = None,
|
||||
user_input: str = "",
|
||||
learner_context: LearnerContext | None = None,
|
||||
) -> list[Message]:
|
||||
"""Compose the full message list: system prompt + history + user turn."""
|
||||
messages: list[Message] = [
|
||||
Message(role="system", content=self.system_prompt(learner_context))
|
||||
]
|
||||
for m in history or []:
|
||||
messages.append(m)
|
||||
if user_input:
|
||||
messages.append(Message(role="user", content=user_input))
|
||||
return messages
|
||||
|
||||
async def stream_reply(
|
||||
self,
|
||||
history: list[Message] | None = None,
|
||||
user_input: str = "",
|
||||
learner_context: LearnerContext | None = None,
|
||||
response_format: dict | None = None,
|
||||
) -> AsyncIterator[str]:
|
||||
"""Stream incremental content deltas for a conversational reply."""
|
||||
messages = self.build_messages(history, user_input, learner_context)
|
||||
async for token in self.provider.stream_chat(
|
||||
messages, model=self.settings.model, response_format=response_format
|
||||
):
|
||||
yield token
|
||||
|
||||
async def structured_reply(
|
||||
self,
|
||||
history: list[Message] | None = None,
|
||||
user_input: str = "",
|
||||
learner_context: LearnerContext | None = None,
|
||||
schema: type[BaseModel] | None = None,
|
||||
schema_hint: str = "",
|
||||
) -> BaseModel:
|
||||
"""Non-streaming completion parsed into a pydantic model (D-020 defense)."""
|
||||
if schema is None:
|
||||
raise ValueError("structured_reply requires a schema")
|
||||
messages = self.build_messages(history, user_input, learner_context)
|
||||
return await structured_completion(
|
||||
self.provider, messages, model=self.settings.model,
|
||||
schema=schema, schema_hint=schema_hint,
|
||||
)
|
||||
@@ -0,0 +1,13 @@
|
||||
"""CoachAgent — pacing, motivation, retrieval practice (REQ-2-005)."""
|
||||
|
||||
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||
from ..prompts.coach import SYSTEM_PROMPT, render_context
|
||||
from .base import BaseAgent
|
||||
|
||||
|
||||
class CoachAgent(BaseAgent):
|
||||
name = "coach"
|
||||
|
||||
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||
ctx = learner_context or get_learner_context()
|
||||
return SYSTEM_PROMPT.format_map(render_context(ctx))
|
||||
@@ -0,0 +1,37 @@
|
||||
"""LabAgent — in-flow feedback over simulated sandbox telemetry (REQ-2-007).
|
||||
|
||||
Scenario-driven: consumes a LabTelemetryScenario from the corpus, renders
|
||||
the event timeline into the conversation, streams concrete feedback.
|
||||
No session chat — each request is one scenario read.
|
||||
"""
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from ..config import Settings
|
||||
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||
from ..corpus.telemetry import LabTelemetryScenario, summarize_scenario
|
||||
from ..llm.base import LLMProvider
|
||||
from ..prompts.lab import SYSTEM_PROMPT, render_context
|
||||
from .base import BaseAgent
|
||||
|
||||
|
||||
class LabAgent(BaseAgent):
|
||||
name = "lab"
|
||||
|
||||
def __init__(self, provider: LLMProvider, settings: Settings) -> None:
|
||||
super().__init__(provider, settings)
|
||||
|
||||
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||
ctx = learner_context or get_learner_context()
|
||||
return SYSTEM_PROMPT.format_map(render_context(ctx))
|
||||
|
||||
async def stream_feedback(
|
||||
self,
|
||||
scenario: LabTelemetryScenario,
|
||||
learner_context: LearnerContext | None = None,
|
||||
) -> AsyncIterator[str]:
|
||||
timeline = summarize_scenario(scenario)
|
||||
async for token in self.stream_reply(
|
||||
history=None, user_input=timeline, learner_context=learner_context
|
||||
):
|
||||
yield token
|
||||
@@ -0,0 +1,17 @@
|
||||
"""MentorAgent — long-horizon career narrative (REQ-2-010).
|
||||
|
||||
Streaming, session-backed conversational agent: the learner can ask
|
||||
follow-up questions about their trajectory and the Mentor keeps context.
|
||||
"""
|
||||
|
||||
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||
from ..prompts.mentor import SYSTEM_PROMPT, render_context
|
||||
from .base import BaseAgent
|
||||
|
||||
|
||||
class MentorAgent(BaseAgent):
|
||||
name = "mentor"
|
||||
|
||||
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||
ctx = learner_context or get_learner_context()
|
||||
return SYSTEM_PROMPT.format_map(render_context(ctx))
|
||||
@@ -0,0 +1,57 @@
|
||||
"""ProctorAgent — integrity signals + coaching interventions (REQ-2-009).
|
||||
|
||||
Consumes a ProctorScenario from the corpus, returns pydantic-validated
|
||||
signal classifications via structured_reply (4-layer defense).
|
||||
Mock engine inputs; real identity/attention signals are v0.3+.
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||
from ..corpus.telemetry import ProctorScenario, summarize_proctor_scenario
|
||||
from ..prompts.proctor import SYSTEM_PROMPT, render_context
|
||||
from .base import BaseAgent
|
||||
|
||||
|
||||
class IntegritySignal(BaseModel):
|
||||
signal_type: str # e.g. "context_switch" | "idle_gap" | "large_paste"
|
||||
severity: str # "low" | "medium" | "high"
|
||||
note: str
|
||||
|
||||
|
||||
class ProctorAssessment(BaseModel):
|
||||
scenario_id: str
|
||||
signals: list[IntegritySignal] = Field(min_length=0)
|
||||
intervention: str # ONE supportive coaching recommendation
|
||||
summary: str
|
||||
|
||||
|
||||
PROCTOR_ASSESSMENT_SCHEMA_HINT = (
|
||||
'{"scenario_id": "<id>", "signals": [{"signal_type": "<type>", '
|
||||
'"severity": "low"|"medium"|"high", "note": "<one sentence>"}], '
|
||||
'"intervention": "<one supportive recommendation>", '
|
||||
'"summary": "<one sentence>"}'
|
||||
)
|
||||
|
||||
|
||||
class ProctorAgent(BaseAgent):
|
||||
name = "proctor"
|
||||
|
||||
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||
ctx = learner_context or get_learner_context()
|
||||
return SYSTEM_PROMPT.format_map(render_context(ctx))
|
||||
|
||||
async def assess(
|
||||
self,
|
||||
scenario: ProctorScenario,
|
||||
learner_context: LearnerContext | None = None,
|
||||
) -> ProctorAssessment:
|
||||
timeline = summarize_proctor_scenario(scenario)
|
||||
assessment: ProctorAssessment = await self.structured_reply(
|
||||
history=None,
|
||||
user_input=timeline,
|
||||
learner_context=learner_context,
|
||||
schema=ProctorAssessment,
|
||||
schema_hint=PROCTOR_ASSESSMENT_SCHEMA_HINT,
|
||||
)
|
||||
return assessment
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Agent registry — explicit name → agent factory map (D-018, G-4).
|
||||
|
||||
Agents are registered centrally in their own phases (P3-P5) via
|
||||
`registry.register(name, factory)`. One registration pattern, one registry.
|
||||
"""
|
||||
|
||||
from collections.abc import Callable
|
||||
|
||||
from ..config import Settings
|
||||
from ..llm.base import LLMProvider
|
||||
from .base import BaseAgent
|
||||
|
||||
AgentFactory = Callable[[LLMProvider, Settings], BaseAgent]
|
||||
|
||||
|
||||
def register_builtin_agents(registry: "AgentRegistry") -> None:
|
||||
"""Central registration of all six shipped tutor agents (G-4: one pattern).
|
||||
|
||||
coach, tutor, lab, assessor, proctor, mentor. New agents register here
|
||||
in their landing phase.
|
||||
"""
|
||||
from .assessor import AssessorAgent
|
||||
from .coach import CoachAgent
|
||||
from .lab import LabAgent
|
||||
from .mentor import MentorAgent
|
||||
from .proctor import ProctorAgent
|
||||
from .tutor import TutorAgent
|
||||
|
||||
registry.register("coach", lambda provider, settings: CoachAgent(provider, settings))
|
||||
registry.register("tutor", lambda provider, settings: TutorAgent(provider, settings))
|
||||
registry.register("lab", lambda provider, settings: LabAgent(provider, settings))
|
||||
registry.register(
|
||||
"assessor", lambda provider, settings: AssessorAgent(provider, settings)
|
||||
)
|
||||
registry.register(
|
||||
"proctor", lambda provider, settings: ProctorAgent(provider, settings)
|
||||
)
|
||||
registry.register(
|
||||
"mentor", lambda provider, settings: MentorAgent(provider, settings)
|
||||
)
|
||||
|
||||
|
||||
class UnknownAgentError(KeyError):
|
||||
"""Raised when resolving an agent name that was never registered."""
|
||||
|
||||
|
||||
class DuplicateAgentError(ValueError):
|
||||
"""Raised when registering an agent name that already exists."""
|
||||
|
||||
|
||||
class AgentRegistry:
|
||||
def __init__(self) -> None:
|
||||
self._factories: dict[str, AgentFactory] = {}
|
||||
|
||||
def register(self, name: str, factory: AgentFactory) -> None:
|
||||
if name in self._factories:
|
||||
raise DuplicateAgentError(f"agent {name!r} already registered")
|
||||
self._factories[name] = factory
|
||||
|
||||
def names(self) -> list[str]:
|
||||
return sorted(self._factories)
|
||||
|
||||
def get(self, provider: LLMProvider, settings: Settings, name: str) -> BaseAgent:
|
||||
try:
|
||||
factory = self._factories[name]
|
||||
except KeyError:
|
||||
raise UnknownAgentError(
|
||||
f"unknown agent {name!r}; registered: {self.names()}"
|
||||
) from None
|
||||
return factory(provider, settings)
|
||||
@@ -0,0 +1,98 @@
|
||||
"""SessionStore — protocol + in-memory implementation (D-019).
|
||||
|
||||
Protocol is DB-migration-ready (A-003): swap InMemorySessionStore for a
|
||||
Redis/PG-backed implementation without touching the API layer.
|
||||
|
||||
Sessions are agent-scoped: switching agents starts a new session ID (avoids
|
||||
persona bleed, A-007). History windowing happens here (last N messages),
|
||||
controlling token growth per session.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from collections import OrderedDict
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Protocol
|
||||
|
||||
from ..llm.types import Message
|
||||
|
||||
DEFAULT_WINDOW = 20
|
||||
DEFAULT_MAX_SESSIONS = 500
|
||||
|
||||
|
||||
@dataclass
|
||||
class AgentSession:
|
||||
session_id: str
|
||||
agent: str
|
||||
learner_id: str = "seed-learner-1"
|
||||
messages: list[Message] = field(default_factory=list)
|
||||
|
||||
|
||||
class SessionStore(Protocol):
|
||||
def get(self, session_id: str) -> AgentSession | None: ...
|
||||
def create(
|
||||
self, session_id: str, agent: str, learner_id: str = "seed-learner-1"
|
||||
) -> AgentSession: ...
|
||||
def append(self, session_id: str, message: Message) -> None: ...
|
||||
def history_window(
|
||||
self, session_id: str, max_messages: int = DEFAULT_WINDOW
|
||||
) -> list[Message]: ...
|
||||
def delete(self, session_id: str) -> None: ...
|
||||
|
||||
|
||||
class InMemorySessionStore:
|
||||
"""asyncio.Lock-guarded dict with 20-message windows and 500-cap LRU eviction."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
window: int = DEFAULT_WINDOW,
|
||||
max_sessions: int = DEFAULT_MAX_SESSIONS,
|
||||
) -> None:
|
||||
self._sessions: OrderedDict[str, AgentSession] = OrderedDict()
|
||||
self._lock = asyncio.Lock()
|
||||
self._window = window
|
||||
self._max_sessions = max_sessions
|
||||
|
||||
async def get(self, session_id: str) -> AgentSession | None:
|
||||
async with self._lock:
|
||||
session = self._sessions.get(session_id)
|
||||
if session is not None:
|
||||
self._sessions.move_to_end(session_id) # LRU touch
|
||||
return session
|
||||
|
||||
async def create(
|
||||
self, session_id: str, agent: str, learner_id: str = "seed-learner-1"
|
||||
) -> AgentSession:
|
||||
async with self._lock:
|
||||
session = AgentSession(session_id=session_id, agent=agent, learner_id=learner_id)
|
||||
self._sessions[session_id] = session
|
||||
self._evict_locked()
|
||||
return session
|
||||
|
||||
async def append(self, session_id: str, message: Message) -> None:
|
||||
async with self._lock:
|
||||
session = self._sessions.get(session_id)
|
||||
if session is None:
|
||||
raise KeyError(f"unknown session {session_id!r}")
|
||||
session.messages.append(message)
|
||||
# Bound stored history too (window bounds replay, not storage):
|
||||
# keep at most 2x window so retries/recent context survive.
|
||||
if len(session.messages) > self._window * 2:
|
||||
del session.messages[: len(session.messages) - self._window * 2]
|
||||
self._sessions.move_to_end(session_id)
|
||||
|
||||
async def history_window(
|
||||
self, session_id: str, max_messages: int = DEFAULT_WINDOW
|
||||
) -> list[Message]:
|
||||
async with self._lock:
|
||||
session = self._sessions.get(session_id)
|
||||
if session is None:
|
||||
raise KeyError(f"unknown session {session_id!r}")
|
||||
return list(session.messages[-max_messages:])
|
||||
|
||||
async def delete(self, session_id: str) -> None:
|
||||
async with self._lock:
|
||||
self._sessions.pop(session_id, None)
|
||||
|
||||
def _evict_locked(self) -> None:
|
||||
while len(self._sessions) > self._max_sessions:
|
||||
self._sessions.popitem(last=False) # evict least-recently-used
|
||||
@@ -0,0 +1,117 @@
|
||||
"""Structured output defense — 4 layers (D-020).
|
||||
|
||||
Layer 1: response_format={"type":"json_object"} request (auto-degrades on 400
|
||||
inside the provider).
|
||||
Layer 2: prompt-embedded schema hint ("Respond with ONLY valid JSON...").
|
||||
Layer 3: defensive parse — strip markdown fences, extract first balanced
|
||||
JSON object, pydantic model_validate.
|
||||
Layer 4: single bounded retry with the validation error fed back.
|
||||
"""
|
||||
|
||||
from typing import TypeVar
|
||||
|
||||
from pydantic import BaseModel, ValidationError
|
||||
|
||||
from ..llm.base import LLMProvider
|
||||
from ..llm.types import Message
|
||||
|
||||
T = TypeVar("T", bound=BaseModel)
|
||||
|
||||
|
||||
class StructuredOutputError(Exception):
|
||||
"""Raised when the model output cannot be validated after one retry."""
|
||||
|
||||
|
||||
def extract_json_object(text: str) -> str:
|
||||
"""Strip fences and return the first balanced {...} block from text."""
|
||||
stripped = text.strip()
|
||||
if stripped.startswith("```"):
|
||||
first_newline = stripped.find("\n")
|
||||
if first_newline != -1:
|
||||
stripped = stripped[first_newline + 1:]
|
||||
if stripped.rstrip().endswith("```"):
|
||||
stripped = stripped.rstrip()[:-3]
|
||||
stripped = stripped.strip()
|
||||
start = stripped.find("{")
|
||||
if start == -1:
|
||||
raise StructuredOutputError("no JSON object found in model output")
|
||||
depth = 0
|
||||
in_string = False
|
||||
escape = False
|
||||
for i, ch in enumerate(stripped[start:], start=start):
|
||||
if escape:
|
||||
escape = False
|
||||
continue
|
||||
if ch == "\\":
|
||||
escape = True
|
||||
continue
|
||||
if ch == '"' and not escape:
|
||||
in_string = not in_string
|
||||
continue
|
||||
if in_string:
|
||||
continue
|
||||
if ch == "{":
|
||||
depth += 1
|
||||
elif ch == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return stripped[start:i + 1]
|
||||
raise StructuredOutputError("unbalanced JSON object in model output")
|
||||
|
||||
|
||||
def parse_structured(text: str, schema: type[T]) -> T:
|
||||
"""Layer 3: fence-strip + first-balanced-object + pydantic validation."""
|
||||
candidate = extract_json_object(text)
|
||||
try:
|
||||
return schema.model_validate_json(candidate)
|
||||
except ValidationError as exc:
|
||||
raise StructuredOutputError(f"schema validation failed: {exc}") from exc
|
||||
|
||||
|
||||
def schema_instruction(schema_hint: str) -> str:
|
||||
"""Layer 2: prompt-side schema text."""
|
||||
return (
|
||||
"Respond with ONLY a valid JSON object matching this schema — "
|
||||
"no markdown fences, no prose outside the JSON. "
|
||||
f"Schema: {schema_hint}"
|
||||
)
|
||||
|
||||
|
||||
async def structured_completion(
|
||||
provider: LLMProvider,
|
||||
messages: list[Message],
|
||||
*,
|
||||
model: str,
|
||||
schema: type[T],
|
||||
schema_hint: str,
|
||||
retry_feedback: str | None = None,
|
||||
) -> T:
|
||||
"""Full 4-layer pipeline. One bounded retry (layer 4), then raise."""
|
||||
# Build request: append schema instruction to the last user message (layer 2).
|
||||
request = list(messages)
|
||||
last_user = next((m for m in reversed(request) if m.role == "user"), None)
|
||||
if last_user is not None:
|
||||
request = [
|
||||
Message(role=m.role, content=(m.content + "\n\n" + schema_instruction(schema_hint)))
|
||||
if m is last_user else m
|
||||
for m in request
|
||||
]
|
||||
response_format = {"type": "json_object"}
|
||||
raw = await provider.chat(request, model=model, response_format=response_format)
|
||||
try:
|
||||
return parse_structured(raw, schema) # layers 1+2+3
|
||||
except StructuredOutputError as exc:
|
||||
# Layer 4: single bounded retry with error feedback
|
||||
retry_prompt = (
|
||||
f"Your previous response was invalid: {exc}. "
|
||||
f"Return ONLY the corrected JSON matching: {schema_hint}"
|
||||
)
|
||||
request2 = list(messages)
|
||||
request2.append(Message(role="user", content=retry_prompt))
|
||||
raw2 = await provider.chat(request2, model=model, response_format=response_format)
|
||||
try:
|
||||
return parse_structured(raw2, schema)
|
||||
except StructuredOutputError as exc2:
|
||||
raise StructuredOutputError(
|
||||
f"structured output failed after retry: {exc2}"
|
||||
) from exc2
|
||||
@@ -0,0 +1,13 @@
|
||||
"""TutorAgent — concept delivery, Socratic questioning (REQ-2-006)."""
|
||||
|
||||
from ..corpus.learner_context import LearnerContext, get_learner_context
|
||||
from ..prompts.tutor import SYSTEM_PROMPT, render_context
|
||||
from .base import BaseAgent
|
||||
|
||||
|
||||
class TutorAgent(BaseAgent):
|
||||
name = "tutor"
|
||||
|
||||
def system_prompt(self, learner_context: LearnerContext | None = None) -> str:
|
||||
ctx = learner_context or get_learner_context()
|
||||
return SYSTEM_PROMPT.format_map(render_context(ctx))
|
||||
@@ -0,0 +1,18 @@
|
||||
"""API package — composes providers, sessions, and agents via DI.
|
||||
|
||||
Boundary rule: api/ composes agents/ and llm/; they never import api/.
|
||||
"""
|
||||
|
||||
from .assessment import router as assessment_router
|
||||
from .chat import router as chat_router
|
||||
from .lab import router as lab_router
|
||||
from .mentor import router as mentor_router
|
||||
from .proctor import router as proctor_router
|
||||
|
||||
__all__ = [
|
||||
"assessment_router",
|
||||
"chat_router",
|
||||
"lab_router",
|
||||
"mentor_router",
|
||||
"proctor_router",
|
||||
]
|
||||
@@ -0,0 +1,47 @@
|
||||
"""POST /v1/assessment/evaluate — structured rubric scores (REQ-2-008).
|
||||
|
||||
JSON response (not SSE): a pydantic-validated RubricScore. Unknown
|
||||
artifact → 404. The Assessor's structured output IS the payload.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..agents.assessor import RubricScore
|
||||
from ..agents.registry import AgentRegistry
|
||||
from ..config import Settings
|
||||
from ..corpus.artifacts import get_artifact_bundle, get_transcript_for_artifact
|
||||
from ..corpus.learner_context import get_learner_context
|
||||
from .deps import get_agent_registry, get_provider, get_settings
|
||||
|
||||
router = APIRouter(prefix="/v1")
|
||||
|
||||
|
||||
class AssessmentRequest(BaseModel):
|
||||
artifact_id: str = Field(min_length=1)
|
||||
learner_id: str | None = None
|
||||
|
||||
|
||||
@router.post("/assessment/evaluate", response_model=RubricScore)
|
||||
async def assessment_evaluate(
|
||||
body: AssessmentRequest,
|
||||
registry: AgentRegistry = Depends(get_agent_registry),
|
||||
settings: Settings = Depends(get_settings),
|
||||
provider=Depends(get_provider),
|
||||
) -> RubricScore:
|
||||
bundle = get_artifact_bundle(body.artifact_id)
|
||||
if bundle is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"unknown artifact {body.artifact_id!r}"
|
||||
)
|
||||
artifact, rubric = bundle
|
||||
transcript = get_transcript_for_artifact(artifact.artifact_id)
|
||||
agent = registry.get(provider, settings, "assessor")
|
||||
learner_context = get_learner_context(body.learner_id)
|
||||
try:
|
||||
return await agent.evaluate(artifact, rubric, transcript, learner_context)
|
||||
except Exception as exc:
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"assessment evaluation failed: {exc}",
|
||||
) from exc
|
||||
@@ -0,0 +1,128 @@
|
||||
"""POST /v1/chat/stream — SSE chat with the D-016 envelope + agent routing.
|
||||
|
||||
Envelope: meta event first (flushed before first token), then raw content
|
||||
deltas, then done; error event before [DONE] on mid-stream failure.
|
||||
Pre-first-byte provider failures surface as in-band `provider_unavailable`
|
||||
error events (SSE 200 headers are already committed once meta flushes).
|
||||
|
||||
Agent routing (A-007): the request names its agent; unknown agents are
|
||||
rejected with 422. No autonomous routing in v0.2.
|
||||
"""
|
||||
|
||||
import json
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
from sse_starlette.sse import EventSourceResponse
|
||||
|
||||
from ..agents.registry import AgentRegistry, UnknownAgentError
|
||||
from ..agents.session import SessionStore
|
||||
from ..config import Settings
|
||||
from ..corpus.learner_context import get_learner_context
|
||||
from ..llm.base import LLMProvider
|
||||
from ..llm.types import Message
|
||||
from .deps import get_agent_registry, get_provider, get_session_store, get_settings
|
||||
|
||||
router = APIRouter(prefix="/v1")
|
||||
|
||||
|
||||
class ChatStreamRequest(BaseModel):
|
||||
agent: str = Field(min_length=1)
|
||||
session_id: str = Field(min_length=1)
|
||||
learner_id: str | None = None
|
||||
messages: list[Message] = Field(min_length=1)
|
||||
|
||||
|
||||
@router.post("/chat/stream")
|
||||
async def chat_stream(
|
||||
body: ChatStreamRequest,
|
||||
registry: AgentRegistry = Depends(get_agent_registry),
|
||||
sessions: SessionStore = Depends(get_session_store),
|
||||
settings: Settings = Depends(get_settings),
|
||||
provider: LLMProvider = Depends(get_provider),
|
||||
) -> EventSourceResponse:
|
||||
# Route to the named agent (A-007); unknown → 422 before any streaming.
|
||||
try:
|
||||
agent = registry.get(provider, settings, body.agent)
|
||||
except UnknownAgentError as exc:
|
||||
raise HTTPException(
|
||||
status_code=422, detail=str(exc)
|
||||
) from None
|
||||
|
||||
learner_context = get_learner_context(body.learner_id)
|
||||
|
||||
# Agent-scoped session (A-007/G-4): persisted turn history, windowed replay.
|
||||
session = await sessions.get(body.session_id)
|
||||
if session is None:
|
||||
session = await sessions.create(
|
||||
body.session_id, agent=body.agent, learner_id=body.learner_id or "learner-001"
|
||||
)
|
||||
# The new user turn is the last message of the request.
|
||||
user_turn = body.messages[-1]
|
||||
history = await sessions.history_window(body.session_id)
|
||||
# Retry dedupe (P1 from final review): a client retry resends the same
|
||||
# turn after a provider failure — don't double-append it to history.
|
||||
last_stored = history[-1] if history else None
|
||||
is_retry = (
|
||||
last_stored is not None
|
||||
and last_stored.role == "user"
|
||||
and last_stored.content == user_turn.content
|
||||
)
|
||||
if not is_retry:
|
||||
await sessions.append(body.session_id, user_turn)
|
||||
else:
|
||||
# On retry the history replay should exclude the stored duplicate.
|
||||
history = history[:-1]
|
||||
|
||||
async def event_stream() -> AsyncIterator[dict]:
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "meta",
|
||||
"agent": body.agent,
|
||||
"session_id": body.session_id,
|
||||
"model": settings.model,
|
||||
})}
|
||||
first_byte = True
|
||||
reply_parts: list[str] = []
|
||||
try:
|
||||
async for token in agent.stream_reply(
|
||||
history=history,
|
||||
user_input=user_turn.content,
|
||||
learner_context=learner_context,
|
||||
):
|
||||
first_byte = False
|
||||
reply_parts.append(token)
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "delta", "content": token
|
||||
})}
|
||||
full_reply = "".join(reply_parts)
|
||||
if full_reply:
|
||||
await sessions.append(
|
||||
body.session_id, Message(role="assistant", content=full_reply)
|
||||
)
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "done", "finish_reason": "stop"
|
||||
})}
|
||||
yield {"event": "message", "data": "[DONE]"}
|
||||
except Exception as exc: # CancelledError is BaseException — passes through
|
||||
message = str(exc)
|
||||
if first_byte:
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "error", "code": "provider_unavailable", "message": message
|
||||
})}
|
||||
else:
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "error", "code": "provider_error", "message": message
|
||||
})}
|
||||
# [DONE] is yielded from the except branch, NEVER from finally:
|
||||
# a yield inside finally would re-raise after GeneratorExit when the
|
||||
# client disconnects ("async generator ignored GeneratorExit").
|
||||
yield {"event": "message", "data": "[DONE]"}
|
||||
|
||||
return EventSourceResponse(
|
||||
event_stream(),
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"X-Accel-Buffering": "no",
|
||||
},
|
||||
)
|
||||
@@ -0,0 +1,24 @@
|
||||
"""FastAPI dependencies — provider, settings, sessions, agents via app.state (DI)."""
|
||||
|
||||
from fastapi import Request
|
||||
|
||||
from ..agents.registry import AgentRegistry
|
||||
from ..agents.session import SessionStore
|
||||
from ..config import Settings
|
||||
from ..llm.base import LLMProvider
|
||||
|
||||
|
||||
def get_settings(request: Request) -> Settings:
|
||||
return request.app.state.settings
|
||||
|
||||
|
||||
def get_provider(request: Request) -> LLMProvider:
|
||||
return request.app.state.provider
|
||||
|
||||
|
||||
def get_session_store(request: Request) -> SessionStore:
|
||||
return request.app.state.session_store
|
||||
|
||||
|
||||
def get_agent_registry(request: Request) -> AgentRegistry:
|
||||
return request.app.state.agent_registry
|
||||
@@ -0,0 +1,72 @@
|
||||
"""POST /v1/lab/feedback — SSE stream of Lab in-flow feedback (REQ-2-007).
|
||||
|
||||
D-016 envelope with agent=lab. Unknown scenario → 404 before streaming.
|
||||
"""
|
||||
|
||||
import json
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
from sse_starlette.sse import EventSourceResponse
|
||||
|
||||
from ..agents.registry import AgentRegistry
|
||||
from ..config import Settings
|
||||
from ..corpus.learner_context import get_learner_context
|
||||
from ..corpus.telemetry import get_lab_scenario
|
||||
from .deps import get_agent_registry, get_provider, get_settings
|
||||
|
||||
router = APIRouter(prefix="/v1")
|
||||
|
||||
|
||||
class LabFeedbackRequest(BaseModel):
|
||||
scenario_id: str = Field(min_length=1)
|
||||
learner_id: str | None = None
|
||||
|
||||
|
||||
@router.post("/lab/feedback")
|
||||
async def lab_feedback(
|
||||
body: LabFeedbackRequest,
|
||||
registry: AgentRegistry = Depends(get_agent_registry),
|
||||
settings: Settings = Depends(get_settings),
|
||||
provider=Depends(get_provider),
|
||||
) -> EventSourceResponse:
|
||||
scenario = get_lab_scenario(body.scenario_id)
|
||||
if scenario is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"unknown scenario {body.scenario_id!r}"
|
||||
)
|
||||
agent = registry.get(provider, settings, "lab")
|
||||
learner_context = get_learner_context(body.learner_id)
|
||||
|
||||
async def event_stream() -> AsyncIterator[dict]:
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "meta",
|
||||
"agent": "lab",
|
||||
"scenario_id": body.scenario_id,
|
||||
"model": settings.model,
|
||||
})}
|
||||
first_byte = True
|
||||
try:
|
||||
async for token in agent.stream_feedback(scenario, learner_context):
|
||||
first_byte = False
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "delta", "content": token
|
||||
})}
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "done", "finish_reason": "stop"
|
||||
})}
|
||||
yield {"event": "message", "data": "[DONE]"}
|
||||
except Exception as exc:
|
||||
code = "provider_unavailable" if first_byte else "provider_error"
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "error", "code": code, "message": str(exc)
|
||||
})}
|
||||
# [DONE] from except, not finally — a yield in finally would
|
||||
# re-raise after GeneratorExit on client disconnect.
|
||||
yield {"event": "message", "data": "[DONE]"}
|
||||
|
||||
return EventSourceResponse(
|
||||
event_stream(),
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
@@ -0,0 +1,95 @@
|
||||
"""POST /v1/mentor/narrative — SSE career narrative stream (REQ-2-010).
|
||||
|
||||
D-016 envelope with agent=mentor. Session-backed: the client supplies a
|
||||
session_id; the Mentor keeps conversation context across follow-ups.
|
||||
"""
|
||||
|
||||
import json
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
from sse_starlette.sse import EventSourceResponse
|
||||
|
||||
from ..agents.registry import AgentRegistry, UnknownAgentError
|
||||
from ..agents.session import SessionStore
|
||||
from ..config import Settings
|
||||
from ..corpus.learner_context import get_learner_context
|
||||
from ..llm.types import Message
|
||||
from .deps import get_agent_registry, get_provider, get_session_store, get_settings
|
||||
|
||||
router = APIRouter(prefix="/v1")
|
||||
|
||||
|
||||
class MentorNarrativeRequest(BaseModel):
|
||||
session_id: str = Field(min_length=1)
|
||||
prompt: str = Field(default="Narrate my trajectory.")
|
||||
learner_id: str | None = None
|
||||
|
||||
|
||||
@router.post("/mentor/narrative")
|
||||
async def mentor_narrative(
|
||||
body: MentorNarrativeRequest,
|
||||
registry: AgentRegistry = Depends(get_agent_registry),
|
||||
sessions: SessionStore = Depends(get_session_store),
|
||||
settings: Settings = Depends(get_settings),
|
||||
provider=Depends(get_provider),
|
||||
) -> EventSourceResponse:
|
||||
try:
|
||||
agent = registry.get(provider, settings, "mentor")
|
||||
except UnknownAgentError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from None
|
||||
|
||||
learner_context = get_learner_context(body.learner_id)
|
||||
|
||||
session = await sessions.get(body.session_id)
|
||||
if session is None:
|
||||
session = await sessions.create(
|
||||
body.session_id, agent="mentor", learner_id=body.learner_id or "learner-001"
|
||||
)
|
||||
history = await sessions.history_window(body.session_id)
|
||||
user_message = Message(role="user", content=body.prompt)
|
||||
await sessions.append(body.session_id, user_message)
|
||||
|
||||
async def event_stream() -> AsyncIterator[dict]:
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "meta",
|
||||
"agent": "mentor",
|
||||
"session_id": body.session_id,
|
||||
"model": settings.model,
|
||||
})}
|
||||
first_byte = True
|
||||
reply_parts: list[str] = []
|
||||
try:
|
||||
async for token in agent.stream_reply(
|
||||
history=history,
|
||||
user_input=body.prompt,
|
||||
learner_context=learner_context,
|
||||
):
|
||||
first_byte = False
|
||||
reply_parts.append(token)
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "delta", "content": token
|
||||
})}
|
||||
full_reply = "".join(reply_parts)
|
||||
if full_reply:
|
||||
await sessions.append(
|
||||
body.session_id, Message(role="assistant", content=full_reply)
|
||||
)
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "done", "finish_reason": "stop"
|
||||
})}
|
||||
yield {"event": "message", "data": "[DONE]"}
|
||||
except Exception as exc:
|
||||
code = "provider_unavailable" if first_byte else "provider_error"
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "error", "code": code, "message": str(exc)
|
||||
})}
|
||||
# [DONE] from except, not finally — a yield in finally would
|
||||
# re-raise after GeneratorExit on client disconnect.
|
||||
yield {"event": "message", "data": "[DONE]"}
|
||||
|
||||
return EventSourceResponse(
|
||||
event_stream(),
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
@@ -0,0 +1,44 @@
|
||||
"""POST /v1/proctor/signals — structured integrity signals (REQ-2-009).
|
||||
|
||||
JSON response (not SSE): a pydantic-validated ProctorAssessment.
|
||||
Unknown scenario → 404. Coaching-shaped interventions only.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..agents.proctor import ProctorAssessment
|
||||
from ..agents.registry import AgentRegistry
|
||||
from ..config import Settings
|
||||
from ..corpus.learner_context import get_learner_context
|
||||
from ..corpus.telemetry import get_proctor_scenario
|
||||
from .deps import get_agent_registry, get_provider, get_settings
|
||||
|
||||
router = APIRouter(prefix="/v1")
|
||||
|
||||
|
||||
class ProctorRequest(BaseModel):
|
||||
scenario_id: str = Field(min_length=1)
|
||||
learner_id: str | None = None
|
||||
|
||||
|
||||
@router.post("/proctor/signals", response_model=ProctorAssessment)
|
||||
async def proctor_signals(
|
||||
body: ProctorRequest,
|
||||
registry: AgentRegistry = Depends(get_agent_registry),
|
||||
settings: Settings = Depends(get_settings),
|
||||
provider=Depends(get_provider),
|
||||
) -> ProctorAssessment:
|
||||
scenario = get_proctor_scenario(body.scenario_id)
|
||||
if scenario is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"unknown scenario {body.scenario_id!r}"
|
||||
)
|
||||
agent = registry.get(provider, settings, "proctor")
|
||||
learner_context = get_learner_context(body.learner_id)
|
||||
try:
|
||||
return await agent.assess(scenario, learner_context)
|
||||
except Exception as exc:
|
||||
raise HTTPException(
|
||||
status_code=502, detail=f"proctor assessment failed: {exc}"
|
||||
) from exc
|
||||
@@ -0,0 +1,18 @@
|
||||
"""Service settings — pydantic-settings, env prefix AI_, .env support."""
|
||||
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(env_prefix="AI_", env_file=".env", extra="ignore")
|
||||
|
||||
port: int = 8420
|
||||
provider: str = "mock"
|
||||
model: str = "gemma4:31b"
|
||||
|
||||
ollama_cloud_base_url: str = "https://ollama.com/v1"
|
||||
ollama_cloud_api_key: str = "" # SecretStr adds friction here; never logged, never echoed
|
||||
local_base_url: str = "http://localhost:11434/v1"
|
||||
|
||||
# "auto" sends response_format and degrades on 400; "off" never sends it
|
||||
json_mode: str = "auto"
|
||||
@@ -0,0 +1,15 @@
|
||||
"""Mock engine inputs — pydantic-typed corpus (D-021).
|
||||
|
||||
Convention-aligned with the TS `packages/mock-data` layer: identical ID
|
||||
strings (stack-*, comp-*, learner-*, art-*, mc-*), cross-referenced by the
|
||||
counterpart files. No codegen in v0.2 — alignment is by documented
|
||||
convention; revisit codegen only if drift bites (v0.3).
|
||||
"""
|
||||
|
||||
from .learner_context import LEARNER_CONTEXTS, LearnerContext, get_learner_context
|
||||
|
||||
__all__ = [
|
||||
"LEARNER_CONTEXTS",
|
||||
"LearnerContext",
|
||||
"get_learner_context",
|
||||
]
|
||||
@@ -0,0 +1,209 @@
|
||||
"""Pre-baked artifacts + rubrics + defense transcripts — Assessor mock inputs (REQ-2-008).
|
||||
|
||||
Counterpart: packages/mock-data/ai-scenarios.ts (artifact IDs string-identical,
|
||||
D-021). Real process-trace grading is a v0.3+ engine (assessment engine);
|
||||
these pre-baked submissions stand in for artifact + defense evaluation.
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class RubricCriterion(BaseModel):
|
||||
criterion_id: str
|
||||
name: str
|
||||
weight: float
|
||||
description: str
|
||||
|
||||
|
||||
class AssessmentRubric(BaseModel):
|
||||
rubric_id: str
|
||||
competency_id: str
|
||||
criteria: list[RubricCriterion]
|
||||
|
||||
|
||||
class ArtifactSubmission(BaseModel):
|
||||
artifact_id: str
|
||||
name: str
|
||||
artifact_type: str # "code" | "design" | "simulation"
|
||||
competency_id: str
|
||||
description: str
|
||||
evidence_excerpt: str # what the grader sees of the artifact itself
|
||||
|
||||
|
||||
class DefenseTranscript(BaseModel):
|
||||
transcript_id: str
|
||||
artifact_id: str
|
||||
turns: list[dict] # {"speaker": "examiner"|"learner", "text": "..."}
|
||||
|
||||
|
||||
_RUBRIC_ORCHESTRATION = AssessmentRubric(
|
||||
rubric_id="rubric-orchestration-c002",
|
||||
competency_id="stack-orchestration-c002",
|
||||
criteria=[
|
||||
RubricCriterion(
|
||||
criterion_id="rc-architecture",
|
||||
name="Agent architecture soundness",
|
||||
weight=0.3,
|
||||
description="State boundaries and responsibilities are clearly separated",
|
||||
),
|
||||
RubricCriterion(
|
||||
criterion_id="rc-communication",
|
||||
name="Inter-agent communication design",
|
||||
weight=0.3,
|
||||
description="Message contracts are explicit, typed, and failure-aware",
|
||||
),
|
||||
RubricCriterion(
|
||||
criterion_id="rc-reliability",
|
||||
name="Reliability engineering",
|
||||
weight=0.25,
|
||||
description="Retries, timeouts, and degradation paths handled",
|
||||
),
|
||||
RubricCriterion(
|
||||
criterion_id="rc-process",
|
||||
name="Process trace quality",
|
||||
weight=0.15,
|
||||
description="Telemetry shows iterative building with real checkpoints",
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
_RUBRIC_TOOL_USE = AssessmentRubric(
|
||||
rubric_id="rubric-orchestration-c003",
|
||||
competency_id="stack-orchestration-c003",
|
||||
criteria=[
|
||||
RubricCriterion(
|
||||
criterion_id="rc-eval-design",
|
||||
name="Evaluation design rigor",
|
||||
weight=0.35,
|
||||
description="Hypotheses, controls, and metrics are explicit and defensible",
|
||||
),
|
||||
RubricCriterion(
|
||||
criterion_id="rc-eval-robustness",
|
||||
name="Harness robustness",
|
||||
weight=0.35,
|
||||
description="Error handling, variance awareness, and reproducibility",
|
||||
),
|
||||
RubricCriterion(
|
||||
criterion_id="rc-eval-insight",
|
||||
name="Insight extraction",
|
||||
weight=0.3,
|
||||
description="Results are interpreted into concrete engineering decisions",
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
_ARTIFACT_RESEARCH_ASSISTANT = ArtifactSubmission(
|
||||
artifact_id="art-eval-research-assistant",
|
||||
name="Multi-agent research assistant (eval build)",
|
||||
artifact_type="code",
|
||||
competency_id="stack-orchestration-c002",
|
||||
description=(
|
||||
"LangGraph-based assistant planning, retrieving, drafting cited reviews."
|
||||
),
|
||||
evidence_excerpt=(
|
||||
"planner.py defines state schema with explicit fields "
|
||||
"(plan, findings, draft); tool_node.py wraps retrieval with a "
|
||||
"3-retry loop and typed ToolMessage responses; tests cover "
|
||||
"planner->tool->writer handoffs; README shows graph diagram"
|
||||
),
|
||||
)
|
||||
|
||||
_TRANSCRIPT_RESEARCH_ASSISTANT = DefenseTranscript(
|
||||
transcript_id="defense-art-eval-research-assistant",
|
||||
artifact_id="art-eval-research-assistant",
|
||||
turns=[
|
||||
{"speaker": "examiner",
|
||||
"text": "Why did you give the planner sole write access to the plan field?"},
|
||||
{"speaker": "learner",
|
||||
"text": "So worker nodes can't mutate each other's inputs — "
|
||||
"the state stays predictable and the graph is debuggable"},
|
||||
{"speaker": "examiner",
|
||||
"text": "What happens when the retrieval tool times out three times?"},
|
||||
{"speaker": "learner",
|
||||
"text": "The tool node degrades to a no-op ToolMessage with a "
|
||||
"retry flag so the writer can fall back to existing findings"},
|
||||
{"speaker": "examiner", "text": "How would you extend this to a third agent?"},
|
||||
{"speaker": "learner",
|
||||
"text": "Add a reviewer node with its own typed messages, same pattern"},
|
||||
],
|
||||
)
|
||||
|
||||
_ARTIFACT_RAG_DASHBOARD = ArtifactSubmission(
|
||||
artifact_id="art-eval-rag-dashboard",
|
||||
name="RAG retrieval quality dashboard (eval build)",
|
||||
artifact_type="code",
|
||||
competency_id="stack-orchestration-c003",
|
||||
description="Dashboard comparing chunking strategies/rerankers across 800 queries.",
|
||||
evidence_excerpt=(
|
||||
"eval harness sweeps 4 chunk sizes x 3 rerankers; results table auto-generated; "
|
||||
"no error handling on the query loader; tests only cover the happy path"
|
||||
),
|
||||
)
|
||||
|
||||
_TRANSCRIPT_RAG_DASHBOARD = DefenseTranscript(
|
||||
transcript_id="defense-art-eval-rag-dashboard",
|
||||
artifact_id="art-eval-rag-dashboard",
|
||||
turns=[
|
||||
{"speaker": "examiner", "text": "How did you control for query difficulty across runs?"},
|
||||
{"speaker": "learner", "text": "I, um, used the same query set each time"},
|
||||
{"speaker": "examiner", "text": "What happens if the query loader hits a malformed row?"},
|
||||
{"speaker": "learner", "text": "I didn't handle that. It would probably crash."},
|
||||
{"speaker": "examiner", "text": "What would you improve first?"},
|
||||
{"speaker": "learner",
|
||||
"text": "Probably add the error handling, then look at variance between runs"},
|
||||
],
|
||||
)
|
||||
|
||||
RUBRICS: dict[str, AssessmentRubric] = {
|
||||
_RUBRIC_ORCHESTRATION.rubric_id: _RUBRIC_ORCHESTRATION,
|
||||
_RUBRIC_TOOL_USE.rubric_id: _RUBRIC_TOOL_USE,
|
||||
}
|
||||
|
||||
ARTIFACTS: dict[str, ArtifactSubmission] = {
|
||||
a.artifact_id: a
|
||||
for a in (_ARTIFACT_RESEARCH_ASSISTANT, _ARTIFACT_RAG_DASHBOARD)
|
||||
}
|
||||
|
||||
TRANSCRIPTS: dict[str, DefenseTranscript] = {
|
||||
t.transcript_id: t
|
||||
for t in (_TRANSCRIPT_RESEARCH_ASSISTANT, _TRANSCRIPT_RAG_DASHBOARD)
|
||||
}
|
||||
|
||||
|
||||
def rubric_for_competency(competency_id: str) -> AssessmentRubric | None:
|
||||
for rubric in RUBRICS.values():
|
||||
if rubric.competency_id == competency_id:
|
||||
return rubric
|
||||
return None
|
||||
|
||||
|
||||
def get_artifact_bundle(artifact_id: str) -> tuple[ArtifactSubmission, AssessmentRubric] | None:
|
||||
"""Resolve (artifact, rubric) for an artifact ID; None if unknown."""
|
||||
artifact = ARTIFACTS.get(artifact_id)
|
||||
if artifact is None:
|
||||
return None
|
||||
rubric = rubric_for_competency(artifact.competency_id)
|
||||
if rubric is None:
|
||||
return None
|
||||
return artifact, rubric
|
||||
|
||||
|
||||
def get_transcript_for_artifact(artifact_id: str) -> DefenseTranscript | None:
|
||||
for transcript in TRANSCRIPTS.values():
|
||||
if transcript.artifact_id == artifact_id:
|
||||
return transcript
|
||||
return None
|
||||
|
||||
|
||||
def render_rubric(rubric: AssessmentRubric) -> str:
|
||||
lines = [f"Rubric: {rubric.rubric_id} (competency {rubric.competency_id})"]
|
||||
for c in rubric.criteria:
|
||||
lines.append(f"- {c.criterion_id} ({c.weight:.2f}): {c.name} — {c.description}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def render_transcript(transcript: DefenseTranscript) -> str:
|
||||
lines = [f"Defense transcript: {transcript.transcript_id}"]
|
||||
for turn in transcript.turns:
|
||||
lines.append(f"{turn['speaker']}: {turn['text']}")
|
||||
return "\n".join(lines)
|
||||
@@ -0,0 +1,108 @@
|
||||
"""Learner context corpus — pydantic mirror of TS learner-progress.ts (D-021).
|
||||
|
||||
Counterpart: packages/mock-data/src/learner-progress.ts (or learner-progress.ts
|
||||
at package root). IDs are string-identical: learner-001, stack-orchestration,
|
||||
stack-safety, stack-orchestration-c00N, art-*, mc-*.
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class CompetencyProgress(BaseModel):
|
||||
competency_id: str
|
||||
title: str
|
||||
status: str # "mastered" | "in_progress" | "not_started"
|
||||
|
||||
|
||||
class StackProgress(BaseModel):
|
||||
stack_id: str
|
||||
title: str
|
||||
percent: int
|
||||
|
||||
|
||||
class LearnerContext(BaseModel):
|
||||
learner_id: str
|
||||
name: str
|
||||
active_stacks: list[StackProgress]
|
||||
active_competencies: list[CompetencyProgress]
|
||||
microcredential_count: int
|
||||
recent_artifacts: list[str] # artifact names
|
||||
|
||||
|
||||
_STACK_ORCHESTRATION = StackProgress(
|
||||
stack_id="stack-orchestration", title="AI Orchestration Engineer", percent=62
|
||||
)
|
||||
_STACK_SAFETY = StackProgress(
|
||||
stack_id="stack-safety", title="AI Safety & Governance Lead", percent=41
|
||||
)
|
||||
|
||||
_LEARNER_1 = LearnerContext(
|
||||
learner_id="learner-001",
|
||||
name="Alex Rivera",
|
||||
active_stacks=[_STACK_ORCHESTRATION, _STACK_SAFETY],
|
||||
active_competencies=[
|
||||
CompetencyProgress(
|
||||
competency_id="stack-orchestration-c001",
|
||||
title="Agent architecture fundamentals",
|
||||
status="mastered",
|
||||
),
|
||||
CompetencyProgress(
|
||||
competency_id="stack-orchestration-c002",
|
||||
title="Multi-agent communication patterns",
|
||||
status="in_progress",
|
||||
),
|
||||
CompetencyProgress(
|
||||
competency_id="stack-orchestration-c003",
|
||||
title="Tool use and function calling",
|
||||
status="in_progress",
|
||||
),
|
||||
CompetencyProgress(
|
||||
competency_id="stack-safety-c021",
|
||||
title="Red-team basics for agent systems",
|
||||
status="in_progress",
|
||||
),
|
||||
],
|
||||
microcredential_count=4,
|
||||
recent_artifacts=[
|
||||
"Multi-agent research assistant",
|
||||
"RAG retrieval quality dashboard",
|
||||
],
|
||||
)
|
||||
|
||||
_LEARNER_2 = LearnerContext(
|
||||
learner_id="learner-002",
|
||||
name="Priya Chen",
|
||||
active_stacks=[
|
||||
StackProgress(
|
||||
stack_id="stack-designer", title="Human-AI Product Designer", percent=55
|
||||
),
|
||||
],
|
||||
active_competencies=[
|
||||
CompetencyProgress(
|
||||
competency_id="stack-designer-c001",
|
||||
title="Prompt-to-prototype workflows",
|
||||
status="mastered",
|
||||
),
|
||||
CompetencyProgress(
|
||||
competency_id="stack-designer-c002",
|
||||
title="Evaluating AI UX patterns",
|
||||
status="in_progress",
|
||||
),
|
||||
],
|
||||
microcredential_count=2,
|
||||
recent_artifacts=["AI onboarding flow concept test"],
|
||||
)
|
||||
|
||||
LEARNER_CONTEXTS: dict[str, LearnerContext] = {
|
||||
_LEARNER_1.learner_id: _LEARNER_1,
|
||||
_LEARNER_2.learner_id: _LEARNER_2,
|
||||
}
|
||||
|
||||
DEFAULT_LEARNER_ID = "learner-001"
|
||||
|
||||
|
||||
def get_learner_context(learner_id: str | None = None) -> LearnerContext:
|
||||
"""Resolve a learner context by ID, falling back to the default seed."""
|
||||
if learner_id is None:
|
||||
return LEARNER_CONTEXTS[DEFAULT_LEARNER_ID]
|
||||
return LEARNER_CONTEXTS.get(learner_id, LEARNER_CONTEXTS[DEFAULT_LEARNER_ID])
|
||||
@@ -0,0 +1,178 @@
|
||||
"""Simulated sandbox telemetry corpus — Lab agent mock engine inputs (REQ-2-007).
|
||||
|
||||
Counterpart: packages/mock-data/ai-scenarios.ts (scenario IDs string-identical,
|
||||
D-021). Real sandbox telemetry is a v0.3+ engine (sandbox fabric); these
|
||||
scripted event streams stand in for the build-session process trace.
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class TelemetryEvent(BaseModel):
|
||||
timestamp: int # seconds since session start
|
||||
kind: str # "keystroke_burst" | "file_save" | "run_tests" | "test_pass"
|
||||
# | "test_fail" | "console_error" | "idle" | "paste" | "commit"
|
||||
|
||||
|
||||
detail: str = ""
|
||||
|
||||
|
||||
class LabTelemetryScenario(BaseModel):
|
||||
scenario_id: str
|
||||
title: str
|
||||
competency_id: str
|
||||
events: list[TelemetryEvent]
|
||||
|
||||
|
||||
class ProctorEvent(BaseModel):
|
||||
timestamp: int # seconds since session start
|
||||
kind: str # "tab_switch" | "idle" | "paste_large" | "focus_lost" | "keystroke_burst"
|
||||
detail: str = ""
|
||||
|
||||
|
||||
class ProctorScenario(BaseModel):
|
||||
scenario_id: str
|
||||
title: str
|
||||
competency_id: str
|
||||
events: list[ProctorEvent]
|
||||
|
||||
|
||||
_PROCTOR_SCENARIO_HEALTHY = ProctorScenario(
|
||||
scenario_id="proctor-scenario-healthy",
|
||||
title="Healthy defense session — focused throughout",
|
||||
competency_id="stack-orchestration-c002",
|
||||
events=[
|
||||
ProctorEvent(timestamp=0, kind="keystroke_burst", detail="session begins"),
|
||||
ProctorEvent(timestamp=310, kind="keystroke_burst", detail="long answer in progress"),
|
||||
ProctorEvent(timestamp=640, kind="keystroke_burst", detail="revision pass"),
|
||||
ProctorEvent(timestamp=900, kind="keystroke_burst", detail="final answer"),
|
||||
],
|
||||
)
|
||||
|
||||
_PROCTOR_SCENARIO_DISTRACTED = ProctorScenario(
|
||||
scenario_id="proctor-scenario-distracted",
|
||||
title="Distracted defense session — tab switches and idle gaps",
|
||||
competency_id="stack-orchestration-c002",
|
||||
events=[
|
||||
ProctorEvent(timestamp=0, kind="keystroke_burst", detail="session begins"),
|
||||
ProctorEvent(timestamp=120, kind="tab_switch", detail="to docs.nextjs.org"),
|
||||
ProctorEvent(timestamp=125, kind="focus_lost", detail="window blur 40s"),
|
||||
ProctorEvent(timestamp=300, kind="idle", detail="no activity for 5 minutes"),
|
||||
ProctorEvent(timestamp=600, kind="tab_switch", detail="to github.com"),
|
||||
ProctorEvent(timestamp=605, kind="focus_lost", detail="window blur 2m"),
|
||||
ProctorEvent(timestamp=720, kind="keystroke_burst", detail="resumes typing"),
|
||||
],
|
||||
)
|
||||
|
||||
_PROCTOR_SCENARIO_FLAGGED = ProctorScenario(
|
||||
scenario_id="proctor-scenario-flagged",
|
||||
title="Flagged defense session — large paste during exam",
|
||||
competency_id="stack-orchestration-c003",
|
||||
events=[
|
||||
ProctorEvent(timestamp=0, kind="keystroke_burst", detail="short intro typed"),
|
||||
ProctorEvent(timestamp=85, kind="paste_large", detail="3,100 chars pasted in 2s"),
|
||||
ProctorEvent(timestamp=90, kind="idle", detail="no activity for 4 minutes"),
|
||||
ProctorEvent(timestamp=330, kind="paste_large", detail="2,800 chars pasted in 2s"),
|
||||
],
|
||||
)
|
||||
|
||||
PROCTOR_SCENARIOS: dict[str, ProctorScenario] = {
|
||||
s.scenario_id: s
|
||||
for s in (
|
||||
_PROCTOR_SCENARIO_HEALTHY,
|
||||
_PROCTOR_SCENARIO_DISTRACTED,
|
||||
_PROCTOR_SCENARIO_FLAGGED,
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
def get_proctor_scenario(scenario_id: str) -> ProctorScenario | None:
|
||||
return PROCTOR_SCENARIOS.get(scenario_id)
|
||||
|
||||
|
||||
def summarize_proctor_scenario(scenario: ProctorScenario) -> str:
|
||||
"""Render the proctor event timeline as compact text for prompt injection."""
|
||||
lines = [f"Defense session: {scenario.title} (competency {scenario.competency_id})"]
|
||||
for event in scenario.events:
|
||||
lines.append(f"t+{event.timestamp}s {event.kind}: {event.detail}".rstrip(": "))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
_LAB_SCENARIO_STRONG = LabTelemetryScenario(
|
||||
scenario_id="lab-scenario-strong",
|
||||
title="Strong build session — multi-agent research assistant",
|
||||
competency_id="stack-orchestration-c002",
|
||||
events=[
|
||||
TelemetryEvent(timestamp=0, kind="keystroke_burst", detail="planner.py"),
|
||||
TelemetryEvent(timestamp=95, kind="file_save", detail="planner.py"),
|
||||
TelemetryEvent(timestamp=120, kind="run_tests", detail="3 tests"),
|
||||
TelemetryEvent(timestamp=126, kind="test_pass",
|
||||
detail="3/3 passed"),
|
||||
TelemetryEvent(timestamp=180, kind="keystroke_burst", detail="tool_node.py"),
|
||||
TelemetryEvent(timestamp=260, kind="file_save", detail="tool_node.py"),
|
||||
TelemetryEvent(timestamp=275, kind="run_tests", detail="4 tests"),
|
||||
TelemetryEvent(timestamp=281, kind="test_pass",
|
||||
detail="4/4 passed"),
|
||||
TelemetryEvent(timestamp=340, kind="commit",
|
||||
detail="add tool node with retries"),
|
||||
],
|
||||
)
|
||||
|
||||
_LAB_SCENARIO_STRUGGLING = LabTelemetryScenario(
|
||||
scenario_id="lab-scenario-struggling",
|
||||
title="Struggling build session — repeated failures, no checkpoints",
|
||||
competency_id="stack-orchestration-c002",
|
||||
events=[
|
||||
TelemetryEvent(timestamp=0, kind="keystroke_burst", detail="main.py"),
|
||||
TelemetryEvent(timestamp=210, kind="run_tests", detail="2 tests"),
|
||||
TelemetryEvent(timestamp=215, kind="test_fail",
|
||||
detail="ImportError: no module named 'tools'"),
|
||||
TelemetryEvent(timestamp=216, kind="console_error", detail="traceback dumped"),
|
||||
TelemetryEvent(timestamp=300, kind="keystroke_burst",
|
||||
detail="main.py"),
|
||||
TelemetryEvent(timestamp=520, kind="run_tests",
|
||||
detail="2 tests"),
|
||||
TelemetryEvent(timestamp=525, kind="test_fail",
|
||||
detail="ImportError: no module named 'tools'"),
|
||||
TelemetryEvent(timestamp=526, kind="console_error",
|
||||
detail="same traceback as before"),
|
||||
TelemetryEvent(timestamp=600, kind="idle",
|
||||
detail="no activity for 6 minutes"),
|
||||
TelemetryEvent(timestamp=960, kind="idle",
|
||||
detail="no activity for 14 minutes"),
|
||||
],
|
||||
)
|
||||
|
||||
_LAB_SCENARIO_FLAGGED = LabTelemetryScenario(
|
||||
scenario_id="lab-scenario-flagged",
|
||||
title="Flagged build session — large paste, instant pass",
|
||||
competency_id="stack-orchestration-c003",
|
||||
events=[
|
||||
TelemetryEvent(timestamp=0, kind="keystroke_burst", detail="eval.py"),
|
||||
TelemetryEvent(timestamp=30, kind="paste",
|
||||
detail="2,400 chars pasted into eval.py"),
|
||||
TelemetryEvent(timestamp=45, kind="run_tests", detail="6 tests"),
|
||||
TelemetryEvent(timestamp=47, kind="test_pass", detail="6/6 passed"),
|
||||
TelemetryEvent(timestamp=48, kind="commit",
|
||||
detail="finish eval harness"),
|
||||
],
|
||||
)
|
||||
|
||||
LAB_SCENARIOS: dict[str, LabTelemetryScenario] = {
|
||||
s.scenario_id: s
|
||||
for s in (_LAB_SCENARIO_STRONG, _LAB_SCENARIO_STRUGGLING, _LAB_SCENARIO_FLAGGED)
|
||||
}
|
||||
|
||||
DEFAULT_LAB_SCENARIO_ID = "lab-scenario-strong"
|
||||
|
||||
|
||||
def get_lab_scenario(scenario_id: str) -> LabTelemetryScenario | None:
|
||||
return LAB_SCENARIOS.get(scenario_id)
|
||||
|
||||
|
||||
def summarize_scenario(scenario: LabTelemetryScenario) -> str:
|
||||
"""Render the event timeline as compact text for prompt injection."""
|
||||
lines = [f"Session: {scenario.title} (competency {scenario.competency_id})"]
|
||||
for event in scenario.events:
|
||||
lines.append(f"t+{event.timestamp}s {event.kind}: {event.detail}".rstrip(": "))
|
||||
return "\n".join(lines)
|
||||
@@ -0,0 +1,15 @@
|
||||
"""LLM package — provider-agnostic layer (D-017)."""
|
||||
|
||||
from .base import LLMProvider
|
||||
from .factory import create_provider
|
||||
from .mock import MockProvider
|
||||
from .openai_compat import OpenAICompatProvider
|
||||
from .types import Message
|
||||
|
||||
__all__ = [
|
||||
"LLMProvider",
|
||||
"Message",
|
||||
"MockProvider",
|
||||
"OpenAICompatProvider",
|
||||
"create_provider",
|
||||
]
|
||||
@@ -0,0 +1,35 @@
|
||||
"""LLMProvider protocol — the port all agents depend on (D-017).
|
||||
|
||||
Implementations: openai_compat.OpenAICompatProvider (ollama-cloud + local),
|
||||
mock.MockProvider (deterministic, tests/CI). Providers are dumb pipes:
|
||||
no envelope logic here — the API layer owns meta/done/error events (D-016).
|
||||
"""
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Protocol
|
||||
|
||||
from .types import Message
|
||||
|
||||
|
||||
class LLMProvider(Protocol):
|
||||
async def stream_chat(
|
||||
self,
|
||||
messages: list[Message],
|
||||
*,
|
||||
model: str,
|
||||
temperature: float = 0.7,
|
||||
response_format: dict | None = None,
|
||||
) -> AsyncIterator[str]:
|
||||
"""Yield incremental content deltas (plain text chunks)."""
|
||||
...
|
||||
|
||||
async def chat(
|
||||
self,
|
||||
messages: list[Message],
|
||||
*,
|
||||
model: str,
|
||||
temperature: float = 0.7,
|
||||
response_format: dict | None = None,
|
||||
) -> str:
|
||||
"""Non-streaming completion — returns the full reply text."""
|
||||
...
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Provider factory — selects the LLM provider from settings (D-014)."""
|
||||
|
||||
import httpx
|
||||
|
||||
from ..config import Settings
|
||||
from .mock import MockProvider
|
||||
from .openai_compat import OpenAICompatProvider
|
||||
|
||||
PROVIDER_NAMES = ("ollama-cloud", "local", "mock")
|
||||
|
||||
|
||||
def create_provider(settings: Settings, http_client: httpx.AsyncClient):
|
||||
"""Return the provider instance for settings.provider.
|
||||
|
||||
Raises ValueError for unknown provider names.
|
||||
"""
|
||||
if settings.provider == "ollama-cloud":
|
||||
return OpenAICompatProvider(
|
||||
http_client=http_client,
|
||||
base_url=settings.ollama_cloud_base_url,
|
||||
api_key=settings.ollama_cloud_api_key,
|
||||
json_mode=settings.json_mode,
|
||||
)
|
||||
if settings.provider == "local":
|
||||
return OpenAICompatProvider(
|
||||
http_client=http_client,
|
||||
base_url=settings.local_base_url,
|
||||
json_mode=settings.json_mode,
|
||||
)
|
||||
if settings.provider == "mock":
|
||||
return MockProvider()
|
||||
raise ValueError(
|
||||
f"unknown provider {settings.provider!r}; expected one of {PROVIDER_NAMES}"
|
||||
)
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Deterministic mock provider — tests and CI. NEVER calls the network.
|
||||
|
||||
Determinism: the reply text is seeded from the message content hash, so
|
||||
identical inputs always produce identical outputs. Supports scripted
|
||||
failure modes for error-path coverage (D-023, A-010).
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from .types import Message
|
||||
|
||||
_REPLIES = [
|
||||
"Great question — let's break this down step by step and see where it leads.",
|
||||
"Here is the key idea: small, verified moves compound into mastery over time.",
|
||||
"Think about it this way: what would the simplest working version look like?",
|
||||
"You are closer than you think. Try restating the goal in one sentence first.",
|
||||
"Let me offer a different angle before we move to the next step.",
|
||||
]
|
||||
|
||||
_JSON_REPLY = '{"summary": "mock structured reply", "confidence": 0.87}'
|
||||
|
||||
|
||||
class MockProvider:
|
||||
"""Scripted provider: deterministic streams, no network, failure injection."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.fail_before_first_token: bool = False
|
||||
self.fail_mid_stream_at_index: int | None = None
|
||||
self.abort_recorded: bool = False # set in stream finally-block (cancellation test)
|
||||
|
||||
def _reply_for(self, messages: list[Message], response_format: dict | None) -> str:
|
||||
seed_src = "|".join(f"{m.role}:{m.content}" for m in messages)
|
||||
if response_format is not None and response_format.get("type") == "json_object":
|
||||
return _JSON_REPLY
|
||||
digest = hashlib.sha256(seed_src.encode()).hexdigest()
|
||||
base = _REPLIES[int(digest[:2], 16) % len(_REPLIES)]
|
||||
# Deterministic seed tag guarantees distinct inputs → distinct replies
|
||||
return f"{base} [#{digest[:8]}]"
|
||||
|
||||
def _tokenize(self, text: str) -> list[str]:
|
||||
words = text.split(" ")
|
||||
tokens: list[str] = []
|
||||
for i, word in enumerate(words):
|
||||
suffix = " " if i < len(words) - 1 else ""
|
||||
tokens.append(word + suffix)
|
||||
return tokens
|
||||
|
||||
async def stream_chat(
|
||||
self,
|
||||
messages: list[Message],
|
||||
*,
|
||||
model: str,
|
||||
temperature: float = 0.7,
|
||||
response_format: dict | None = None,
|
||||
) -> AsyncIterator[str]:
|
||||
if self.fail_before_first_token:
|
||||
raise RuntimeError("mock provider: scripted failure before first token")
|
||||
reply = self._reply_for(messages, response_format)
|
||||
tokens = self._tokenize(reply)
|
||||
try:
|
||||
for i, token in enumerate(tokens):
|
||||
if self.fail_mid_stream_at_index is not None and i == self.fail_mid_stream_at_index:
|
||||
raise RuntimeError("mock provider: scripted mid-stream failure")
|
||||
yield token
|
||||
finally:
|
||||
# Cancellation (GeneratorExit/CancelledError) lands here — tests assert this.
|
||||
self.abort_recorded = True
|
||||
|
||||
async def chat(
|
||||
self,
|
||||
messages: list[Message],
|
||||
*,
|
||||
model: str,
|
||||
temperature: float = 0.7,
|
||||
response_format: dict | None = None,
|
||||
) -> str:
|
||||
if self.fail_before_first_token:
|
||||
raise RuntimeError("mock provider: scripted failure before completion")
|
||||
return self._reply_for(messages, response_format)
|
||||
|
||||
|
||||
class ScriptedJSONProvider(MockProvider):
|
||||
"""Mock variant returning a fixed JSON payload for structured tests."""
|
||||
|
||||
def __init__(self, payload: dict) -> None:
|
||||
super().__init__()
|
||||
self.payload = payload
|
||||
|
||||
def _reply_for(self, messages: list[Message], response_format: dict | None) -> str:
|
||||
if response_format is not None and response_format.get("type") == "json_object":
|
||||
return json.dumps(self.payload)
|
||||
return super()._reply_for(messages, response_format)
|
||||
@@ -0,0 +1,125 @@
|
||||
"""OpenAI-compatible provider — one implementation serves ollama-cloud AND local
|
||||
endpoints (they differ only in base_url/key). Raw httpx, no SDK (D-017).
|
||||
|
||||
Boundary rules:
|
||||
- llm/ imports nothing from agents/ or api/
|
||||
- api_key NEVER appears in exceptions, logs, or error messages
|
||||
"""
|
||||
|
||||
import json
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
import httpx
|
||||
|
||||
from .types import Message
|
||||
|
||||
|
||||
class OpenAICompatProvider:
|
||||
def __init__(
|
||||
self,
|
||||
http_client: httpx.AsyncClient,
|
||||
base_url: str,
|
||||
api_key: str = "",
|
||||
json_mode: str = "auto",
|
||||
) -> None:
|
||||
self._client = http_client
|
||||
self._base_url = base_url.rstrip("/")
|
||||
self._api_key = api_key
|
||||
self._json_mode = json_mode
|
||||
|
||||
def _headers(self) -> dict[str, str]:
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if self._api_key:
|
||||
headers["Authorization"] = f"Bearer {self._api_key}"
|
||||
return headers
|
||||
|
||||
def _payload(
|
||||
self,
|
||||
messages: list[Message],
|
||||
model: str,
|
||||
temperature: float,
|
||||
response_format: dict | None,
|
||||
stream: bool,
|
||||
) -> dict:
|
||||
payload: dict = {
|
||||
"model": model,
|
||||
"messages": [{"role": m.role, "content": m.content} for m in messages],
|
||||
"temperature": temperature,
|
||||
}
|
||||
if stream:
|
||||
payload["stream"] = True
|
||||
else:
|
||||
payload["stream"] = False
|
||||
# json_mode="auto": send response_format and degrade on 400; "off": never send
|
||||
if response_format is not None and self._json_mode == "auto":
|
||||
payload["response_format"] = response_format
|
||||
return payload
|
||||
|
||||
def _sanitize(self, exc: Exception) -> RuntimeError:
|
||||
text = str(exc)
|
||||
if self._api_key and self._api_key in text:
|
||||
text = text.replace(self._api_key, "[REDACTED]")
|
||||
return RuntimeError(f"llm provider error: {text}")
|
||||
|
||||
async def stream_chat(
|
||||
self,
|
||||
messages: list[Message],
|
||||
*,
|
||||
model: str,
|
||||
temperature: float = 0.7,
|
||||
response_format: dict | None = None,
|
||||
) -> AsyncIterator[str]:
|
||||
payload = self._payload(messages, model, temperature, response_format, stream=True)
|
||||
try:
|
||||
async with self._client.stream(
|
||||
"POST", f"{self._base_url}/chat/completions",
|
||||
json=payload, headers=self._headers(),
|
||||
) as response:
|
||||
response.raise_for_status()
|
||||
async for line in response.aiter_lines():
|
||||
if not line or not line.startswith("data:"):
|
||||
continue # keep-alive comments (": ping"), empty lines
|
||||
data = line.removeprefix("data:").strip()
|
||||
if data == "[DONE]":
|
||||
return
|
||||
try:
|
||||
chunk = json.loads(data)
|
||||
except json.JSONDecodeError:
|
||||
continue # malformed line — tolerate (ollama-cloud quirks)
|
||||
choices = chunk.get("choices") or []
|
||||
if not choices:
|
||||
continue
|
||||
content = (choices[0].get("delta") or {}).get("content")
|
||||
if content:
|
||||
yield content
|
||||
except httpx.HTTPError as exc:
|
||||
raise self._sanitize(exc) from exc
|
||||
|
||||
async def chat(
|
||||
self,
|
||||
messages: list[Message],
|
||||
*,
|
||||
model: str,
|
||||
temperature: float = 0.7,
|
||||
response_format: dict | None = None,
|
||||
) -> str:
|
||||
payload = self._payload(messages, model, temperature, response_format, stream=False)
|
||||
try:
|
||||
response = await self._client.post(
|
||||
f"{self._base_url}/chat/completions",
|
||||
json=payload, headers=self._headers(),
|
||||
)
|
||||
if response.status_code == 400 and "response_format" in payload:
|
||||
# json_mode auto-degrade (D-020 layer 1): retry once without it
|
||||
payload.pop("response_format")
|
||||
response = await self._client.post(
|
||||
f"{self._base_url}/chat/completions",
|
||||
json=payload, headers=self._headers(),
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
return (data["choices"][0]["message"]["content"]) or ""
|
||||
except httpx.HTTPError as exc:
|
||||
raise self._sanitize(exc) from exc
|
||||
except (KeyError, ValueError) as exc:
|
||||
raise self._sanitize(exc) from exc
|
||||
@@ -0,0 +1,16 @@
|
||||
"""LLM layer types — messages.
|
||||
|
||||
Boundary rule: nothing in llm/ imports from agents/ or api/.
|
||||
|
||||
Providers yield plain str deltas (providers-as-pipes, D-016/D-017);
|
||||
the OpenAI chunk shape lives only at the wire level inside
|
||||
openai_compat.py. ChatDelta/ChoiceDelta were removed in Phase 3 after
|
||||
two verification cycles confirmed no consumers (P2-a finding).
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class Message(BaseModel):
|
||||
role: str
|
||||
content: str
|
||||
@@ -0,0 +1,65 @@
|
||||
"""FastAPI app factory — lifespan, CORS, health, routers."""
|
||||
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
import httpx
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from .agents.registry import AgentRegistry, register_builtin_agents
|
||||
from .agents.session import InMemorySessionStore
|
||||
from .api import (
|
||||
assessment_router,
|
||||
chat_router,
|
||||
lab_router,
|
||||
mentor_router,
|
||||
proctor_router,
|
||||
)
|
||||
from .config import Settings
|
||||
from .llm import create_provider
|
||||
|
||||
|
||||
def create_app(settings: Settings | None = None) -> FastAPI:
|
||||
settings = settings or Settings()
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
# Shared HTTP client pool (D-017): 10s connect / 300s read for cloud TTFT
|
||||
timeout = httpx.Timeout(connect=10.0, read=300.0, write=30.0, pool=10.0)
|
||||
app.state.http_client = httpx.AsyncClient(timeout=timeout)
|
||||
app.state.settings = settings
|
||||
app.state.provider = create_provider(settings, app.state.http_client)
|
||||
app.state.session_store = InMemorySessionStore()
|
||||
app.state.agent_registry = AgentRegistry()
|
||||
register_builtin_agents(app.state.agent_registry)
|
||||
yield
|
||||
await app.state.http_client.aclose()
|
||||
|
||||
app = FastAPI(title="Nextcraft AI Service", version="0.2.0", lifespan=lifespan)
|
||||
|
||||
# A-008: localhost-only CORS, no credentials
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["http://localhost:3000", "http://127.0.0.1:3000"],
|
||||
allow_methods=["GET", "POST", "OPTIONS"],
|
||||
allow_headers=["Content-Type"],
|
||||
allow_credentials=False,
|
||||
)
|
||||
|
||||
@app.get("/health")
|
||||
async def health() -> dict:
|
||||
return {
|
||||
"status": "ok",
|
||||
"provider": settings.provider,
|
||||
"model": settings.model,
|
||||
}
|
||||
|
||||
app.include_router(chat_router)
|
||||
app.include_router(lab_router)
|
||||
app.include_router(assessment_router)
|
||||
app.include_router(mentor_router)
|
||||
app.include_router(proctor_router)
|
||||
return app
|
||||
|
||||
|
||||
app = create_app()
|
||||
@@ -0,0 +1,22 @@
|
||||
"""Prompt library — prompts are code: versioned in git, reviewed like code (D-018).
|
||||
|
||||
Each module exposes a `versioned SYSTEM_PROMPT` constant and a
|
||||
`render_context(learner_context) -> dict` for str.format_map injection.
|
||||
Final personas land in Phases 3-5; these are the initial drafts.
|
||||
"""
|
||||
|
||||
from .coach import SYSTEM_PROMPT as COACH_PROMPT
|
||||
from .coach import render_context as render_coach
|
||||
from .mentor import SYSTEM_PROMPT as MENTOR_PROMPT
|
||||
from .mentor import render_context as render_mentor
|
||||
from .tutor import SYSTEM_PROMPT as TUTOR_PROMPT
|
||||
from .tutor import render_context as render_tutor
|
||||
|
||||
__all__ = [
|
||||
"COACH_PROMPT",
|
||||
"MENTOR_PROMPT",
|
||||
"TUTOR_PROMPT",
|
||||
"render_coach",
|
||||
"render_mentor",
|
||||
"render_tutor",
|
||||
]
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Assessor agent prompt — rubric application to artifacts + defenses (REQ-2-008).
|
||||
|
||||
Final persona (Phase 4). Assessor is a rigorous, fair grader: scores each
|
||||
criterion with evidence, cites what the learner did, returns ONLY valid
|
||||
JSON matching the rubric schema.
|
||||
Version: assessor-v2 (final for v0.2).
|
||||
"""
|
||||
|
||||
SYSTEM_PROMPT = """You are Assessor, the grading agent of Nextcraft, an AI-native competency school.
|
||||
Learner: {learner_name}.
|
||||
|
||||
You receive: (a) an artifact evidence excerpt, (b) its defense transcript,
|
||||
and (c) the rubric for the competency. Your job:
|
||||
- Score EVERY rubric criterion from 0-100, justified by evidence you can
|
||||
point to in the artifact or transcript.
|
||||
- Cite what the learner did ("the 3-retry loop in the tool node"), not
|
||||
what they should have done — except in gaps, where the missed work goes.
|
||||
- Strengths: the two strongest evidence points, each one sentence.
|
||||
- Gaps: the two most important missed opportunities, each one sentence.
|
||||
- Verdict: "mastered" | "developing" | "not_yet" — judged against the
|
||||
rubric weights, honestly.
|
||||
|
||||
Rules:
|
||||
- Rigorous but fair. A polished artifact with a weak defense is NOT mastery.
|
||||
- Respond with ONLY a valid JSON object matching the provided schema —
|
||||
no markdown fences, no prose outside the JSON."""
|
||||
|
||||
PROMPT_VERSION = "assessor-v2"
|
||||
|
||||
|
||||
def render_context(learner_context) -> dict:
|
||||
return {"learner_name": learner_context.name}
|
||||
@@ -0,0 +1,46 @@
|
||||
"""Coach agent prompt — pacing, motivation, retrieval practice (REQ-2-005).
|
||||
|
||||
Final persona (Phase 3). Coach is an accountability partner: warm,
|
||||
action-oriented, allergic to fluff. Always ends with exactly one next action
|
||||
and weaves retrieval practice into every reply.
|
||||
Version: coach-v2 (final for v0.2).
|
||||
"""
|
||||
|
||||
SYSTEM_PROMPT = """You are Coach, the pacing and motivation agent of Nextcraft,
|
||||
an AI-native competency school.
|
||||
|
||||
Learner: {learner_name}
|
||||
Active stack: {stacks}
|
||||
Current focus: {progress}
|
||||
|
||||
Your style:
|
||||
- Warm, direct, allergic to fluff. Two short paragraphs maximum.
|
||||
- Pacing: name the learner's next concrete step in their current competency.
|
||||
- Motivation: tie effort to their trajectory — what this unlocks, specifically.
|
||||
- Retrieval practice: before introducing anything new, ask the learner to
|
||||
recall or apply something they already covered (one pointed question).
|
||||
|
||||
Rules:
|
||||
- End with exactly ONE clear next action phrased as a command ("Post your
|
||||
plan for the orchestrator retry loop before starting").
|
||||
- Never lecture; never list more than two options.
|
||||
- If the learner is stuck or frustrated, slow down and shrink the step."""
|
||||
|
||||
PROMPT_VERSION = "coach-v2"
|
||||
|
||||
|
||||
def render_context(learner_context) -> dict:
|
||||
stacks = ", ".join(f"{s.title} ({s.percent}%)" for s in learner_context.active_stacks)
|
||||
in_progress = [
|
||||
c for c in learner_context.active_competencies if c.status == "in_progress"
|
||||
]
|
||||
progress = (
|
||||
f"{in_progress[0].title} ({in_progress[0].competency_id})"
|
||||
if in_progress
|
||||
else "no competency currently in progress"
|
||||
)
|
||||
return {
|
||||
"learner_name": learner_context.name,
|
||||
"stacks": stacks or "none yet",
|
||||
"progress": progress,
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
"""Lab agent prompt — in-flow feedback over sandbox telemetry (REQ-2-007).
|
||||
|
||||
Final persona (Phase 4). Lab is a pragmatic build partner: reads the
|
||||
telemetry timeline, names the one most useful adjustment, gives one
|
||||
concrete next step. Scenario-driven; no session chat.
|
||||
Version: lab-v2 (final for v0.2).
|
||||
"""
|
||||
|
||||
SYSTEM_PROMPT = """You are Lab, the in-flow feedback agent watching a learner
|
||||
build in the Nextcraft sandbox.
|
||||
Learner: {learner_name}. Active stack: {stacks}.
|
||||
|
||||
You receive a telemetry timeline of the learner's build session below.
|
||||
Your job, in order:
|
||||
1. Say what the telemetry shows — name the specific events that matter.
|
||||
2. Name the single most useful adjustment (one thing, not a list).
|
||||
3. Give one concrete next step phrased as a command.
|
||||
|
||||
Rules:
|
||||
- Be specific to the events you see. If tests failed twice with the same
|
||||
error, say so. If there is a long idle gap, name it.
|
||||
- If the session looks healthy, say so briefly and set the next challenge.
|
||||
- If something looks off (e.g., a huge paste followed by instant success),
|
||||
treat it as a coaching moment, not an accusation — suggest a quick
|
||||
self-check that would prove understanding.
|
||||
- Three short paragraphs maximum. No headers, no bullet lists."""
|
||||
|
||||
PROMPT_VERSION = "lab-v2"
|
||||
|
||||
|
||||
def render_context(learner_context) -> dict:
|
||||
stacks = ", ".join(f"{s.title} ({s.percent}%)" for s in learner_context.active_stacks)
|
||||
return {
|
||||
"learner_name": learner_context.name,
|
||||
"stacks": stacks or "none yet",
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Mentor agent prompt — long-horizon career narrative (REQ-2-010).
|
||||
|
||||
Final persona (Phase 5). Mentor is a wise career guide: connects today's
|
||||
competencies and artifacts to a long-horizon AI-era trajectory.
|
||||
Version: mentor-v2 (final for v0.2).
|
||||
"""
|
||||
|
||||
SYSTEM_PROMPT = """You are Mentor, the long-horizon career agent of Nextcraft,
|
||||
an AI-native competency school.
|
||||
|
||||
Learner: {learner_name}
|
||||
Active stacks: {stacks}
|
||||
Current focus: {progress}
|
||||
Microcredentials earned: {microcredentials}
|
||||
Recent artifacts: {artifacts}
|
||||
|
||||
Your job: narrate the learner's trajectory in two to three paragraphs:
|
||||
1. Where they are now — what their competency progress and artifacts say
|
||||
about them as a builder (specific, evidence-based).
|
||||
2. What their current stack unlocks next — name the next competency or
|
||||
microcredential worth chasing and the role it points toward.
|
||||
3. How they position in the AI-era labor market — which employer problems
|
||||
their profile already answers.
|
||||
|
||||
Rules:
|
||||
- Forward-looking and concrete. No fortune-telling, no flattery.
|
||||
- Reference their artifacts by name at least once.
|
||||
- Write like a mentor writing to one person, not a career-services brochure."""
|
||||
|
||||
PROMPT_VERSION = "mentor-v2"
|
||||
|
||||
|
||||
def render_context(learner_context) -> dict:
|
||||
stacks = ", ".join(f"{s.title} ({s.percent}%)" for s in learner_context.active_stacks)
|
||||
in_progress = [
|
||||
c for c in learner_context.active_competencies if c.status == "in_progress"
|
||||
]
|
||||
progress = (
|
||||
f"{in_progress[0].title} ({in_progress[0].competency_id})"
|
||||
if in_progress
|
||||
else "no competency currently in progress"
|
||||
)
|
||||
return {
|
||||
"learner_name": learner_context.name,
|
||||
"stacks": stacks or "none yet",
|
||||
"progress": progress,
|
||||
"microcredentials": str(learner_context.microcredential_count),
|
||||
"artifacts": ", ".join(learner_context.recent_artifacts) or "none yet",
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
"""Proctor agent prompt — integrity signals with coaching interventions (REQ-2-009).
|
||||
|
||||
Final persona (Phase 5). Proctor is a supportive observer, never punitive:
|
||||
classifies signals, recommends ONE coaching intervention. Assume good
|
||||
faith — most signals have innocent explanations.
|
||||
Version: proctor-v2 (final for v0.2).
|
||||
"""
|
||||
|
||||
SYSTEM_PROMPT = """You are Proctor, the integrity-support agent of Nextcraft,
|
||||
an AI-native competency school.
|
||||
Learner: {learner_name}.
|
||||
|
||||
You receive a telemetry timeline of defense-session events (tab switches,
|
||||
idle gaps, large pastes, focus loss, keystroke bursts). Your job:
|
||||
- Classify EACH notable signal: type (e.g. "context_switch", "idle_gap",
|
||||
"large_paste"), severity ("low" | "medium" | "high"), and a one-sentence
|
||||
note citing the event (timestamps and details).
|
||||
- Recommend exactly ONE supportive coaching intervention for the session
|
||||
overall — never punitive, never accusatory. Frame around helping the
|
||||
learner succeed, e.g. "offer a short break", "invite them to explain
|
||||
the pasted section in their own words".
|
||||
|
||||
Rules:
|
||||
- Assume good faith. Tab switches to documentation are normal engineering.
|
||||
- Idle gaps are often thinking. Only unusual patterns deserve higher severity.
|
||||
- A large paste during an assessment deserves "high" severity but the
|
||||
intervention stays coaching-shaped: verification, not punishment.
|
||||
- Respond with ONLY a valid JSON object matching the provided schema —
|
||||
no markdown fences, no prose outside the JSON."""
|
||||
|
||||
PROMPT_VERSION = "proctor-v2"
|
||||
|
||||
|
||||
def render_context(learner_context) -> dict:
|
||||
return {"learner_name": learner_context.name}
|
||||
@@ -0,0 +1,46 @@
|
||||
"""Tutor agent prompt — concept delivery, Socratic questioning (REQ-2-006).
|
||||
|
||||
Final persona (Phase 3). Tutor is a patient expert teacher: one concept at
|
||||
a time, worked example first, Socratic check before moving on.
|
||||
Version: tutor-v2 (final for v0.2).
|
||||
"""
|
||||
|
||||
SYSTEM_PROMPT = """You are Tutor, the concept-delivery agent of Nextcraft,
|
||||
an AI-native competency school.
|
||||
|
||||
Learner: {learner_name}
|
||||
Active stack: {stacks}
|
||||
Current focus: {progress}
|
||||
|
||||
Your style:
|
||||
- Teach exactly ONE concept per reply. Never more.
|
||||
- Structure: (1) name the concept in one sentence, (2) give a short worked
|
||||
example (5-8 lines) the learner can trace, (3) ask ONE Socratic question
|
||||
that checks whether they can apply it to a slightly different case.
|
||||
|
||||
Rules:
|
||||
- Never dump walls of text. If the concept needs more than ~150 words, teach
|
||||
only its first slice and promise the rest after the learner answers.
|
||||
- If the learner's last message reveals a misconception, correct it gently
|
||||
before teaching.
|
||||
- If the learner answers your question, evaluate the answer explicitly
|
||||
(right / partly right / not yet) before the next concept."""
|
||||
|
||||
PROMPT_VERSION = "tutor-v2"
|
||||
|
||||
|
||||
def render_context(learner_context) -> dict:
|
||||
stacks = ", ".join(f"{s.title} ({s.percent}%)" for s in learner_context.active_stacks)
|
||||
in_progress = [
|
||||
c for c in learner_context.active_competencies if c.status == "in_progress"
|
||||
]
|
||||
progress = (
|
||||
f"{in_progress[0].title} ({in_progress[0].competency_id})"
|
||||
if in_progress
|
||||
else "no competency currently in progress"
|
||||
)
|
||||
return {
|
||||
"learner_name": learner_context.name,
|
||||
"stacks": stacks or "none yet",
|
||||
"progress": progress,
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"name": "@nextcraft/ai-service",
|
||||
"private": true,
|
||||
"version": "0.2.0",
|
||||
"scripts": {
|
||||
"dev": "bash scripts/dev.sh",
|
||||
"test": "bash scripts/test.sh",
|
||||
"bootstrap": "bash scripts/bootstrap.sh",
|
||||
"lint": "bash scripts/lint.sh"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=68"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "nextcraft-ai-service"
|
||||
version = "0.2.0"
|
||||
description = "Nextcraft AI tutor service — six LLM agents behind a provider-agnostic layer"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"fastapi>=0.141,<0.142",
|
||||
"uvicorn>=0.52,<0.53",
|
||||
"pydantic>=2.13,<2.14",
|
||||
"pydantic-settings>=2.15,<2.16",
|
||||
"httpx>=0.28,<0.29",
|
||||
"sse-starlette>=3.4,<3.5",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=9.1,<10",
|
||||
"pytest-asyncio>=1.4,<2",
|
||||
"ruff>=0.14",
|
||||
]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
include = ["ai_service*"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
asyncio_mode = "auto"
|
||||
testpaths = ["tests"]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
target-version = "py311"
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "W", "I", "UP", "B"]
|
||||
# B008: Depends() in argument defaults is the idiomatic FastAPI DI pattern
|
||||
ignore = ["B008"]
|
||||
Executable
+30
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
# Idempotent bootstrap: create venv + install deps.
|
||||
# Handles Debian systems without python3-venv/ensurepip via --without-pip + get-pip.
|
||||
set -euo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
APP_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
VENV="$APP_DIR/.venv"
|
||||
|
||||
mkdir -p "$HOME/.cache/ciagent"
|
||||
|
||||
if [ ! -x "$VENV/bin/python3" ]; then
|
||||
if python3 -m venv "$VENV" 2>/dev/null; then
|
||||
:
|
||||
else
|
||||
# No ensurepip available — create bare venv and bootstrap pip separately.
|
||||
python3 -m venv --without-pip "$VENV"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ ! -x "$VENV/bin/pip" ]; then
|
||||
GET_PIP="$HOME/.cache/ciagent/get-pip.py"
|
||||
if [ ! -f "$GET_PIP" ]; then
|
||||
curl -sSf --max-time 60 https://bootstrap.pypa.io/get-pip.py -o "$GET_PIP"
|
||||
fi
|
||||
"$VENV/bin/python3" "$GET_PIP" --quiet
|
||||
fi
|
||||
|
||||
"$VENV/bin/pip" install --quiet --upgrade pip
|
||||
"$VENV/bin/pip" install --quiet -e "$APP_DIR[dev]"
|
||||
echo "bootstrap complete: $VENV"
|
||||
Executable
+26
@@ -0,0 +1,26 @@
|
||||
#!/usr/bin/env bash
|
||||
# Dev server: export secrets (if present) then run uvicorn on :8420.
|
||||
set -euo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
APP_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
REPO_ROOT="$(cd "$APP_DIR/../.." && pwd)"
|
||||
VENV="$APP_DIR/.venv"
|
||||
|
||||
if [ ! -x "$VENV/bin/uvicorn" ]; then
|
||||
echo "venv missing — run scripts/bootstrap.sh first" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SECRETS="$REPO_ROOT/.ciagent/.env.secrets"
|
||||
if [ -f "$SECRETS" ]; then
|
||||
while IFS='=' read -r key value; do
|
||||
case "$key" in
|
||||
OLLAMA_API_KEY) export AI_OLLAMA_CLOUD_API_KEY="$value" ;;
|
||||
OLLAMA_BASE_URL) export AI_OLLAMA_CLOUD_BASE_URL="$value" ;;
|
||||
AI_TUTOR_MODEL) export AI_MODEL="$value" ;;
|
||||
esac
|
||||
done < "$SECRETS"
|
||||
fi
|
||||
|
||||
cd "$APP_DIR"
|
||||
exec "$VENV/bin/uvicorn" ai_service.main:app --reload --port 8420
|
||||
Executable
+14
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
# Lint: ruff check over the ai-service tree.
|
||||
set -euo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
APP_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
VENV="$APP_DIR/.venv"
|
||||
|
||||
if [ ! -x "$VENV/bin/ruff" ]; then
|
||||
echo "venv missing — run scripts/bootstrap.sh first" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cd "$APP_DIR"
|
||||
exec "$VENV/bin/ruff" check .
|
||||
Executable
+14
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
# Test runner: pytest via venv — mock provider only, zero network calls.
|
||||
set -euo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
APP_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
VENV="$APP_DIR/.venv"
|
||||
|
||||
if [ ! -x "$VENV/bin/pytest" ]; then
|
||||
echo "venv missing — run scripts/bootstrap.sh first" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cd "$APP_DIR"
|
||||
exec "$VENV/bin/pytest" -q "$@"
|
||||
@@ -0,0 +1,103 @@
|
||||
"""Assessor agent tests — structured rubric scores (REQ-2-008).
|
||||
|
||||
The Assessor is the structured-output showcase: tests use ScriptedJSONProvider
|
||||
for valid payloads and exercise the 4-layer defense failure modes.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from ai_service.agents.assessor import AssessorAgent, RubricScore
|
||||
from ai_service.agents.structured import StructuredOutputError
|
||||
from ai_service.config import Settings
|
||||
from ai_service.corpus.artifacts import (
|
||||
get_artifact_bundle,
|
||||
get_transcript_for_artifact,
|
||||
render_rubric,
|
||||
)
|
||||
from ai_service.corpus.learner_context import get_learner_context
|
||||
from ai_service.llm.mock import MockProvider, ScriptedJSONProvider
|
||||
|
||||
VALID_SCORE = {
|
||||
"rubric_id": "rubric-orchestration-c002",
|
||||
"artifact_id": "art-eval-research-assistant",
|
||||
"competency_id": "stack-orchestration-c002",
|
||||
"scores": [
|
||||
{"criterion_id": "rc-architecture", "name": "Agent architecture soundness",
|
||||
"score": 92, "evidence": "Explicit state schema with planner-only write access"},
|
||||
{"criterion_id": "rc-communication", "name": "Inter-agent communication design",
|
||||
"score": 88, "evidence": "Typed ToolMessage responses with retry flags"},
|
||||
{"criterion_id": "rc-reliability", "name": "Reliability engineering",
|
||||
"score": 85, "evidence": "3-retry loop with degradation path"},
|
||||
{"criterion_id": "rc-process", "name": "Process trace quality",
|
||||
"score": 90, "evidence": "Iterative saves with passing test checkpoints"},
|
||||
],
|
||||
"strengths": ["Clean state boundaries", "Failure-aware tool wrapping"],
|
||||
"gaps": ["No reviewer node yet", "Graph diagram only in README"],
|
||||
"verdict": "mastered",
|
||||
}
|
||||
|
||||
|
||||
def make_assessor(provider=None) -> AssessorAgent:
|
||||
return AssessorAgent(provider or MockProvider(), Settings(provider="mock"))
|
||||
|
||||
|
||||
def get_bundle(artifact_id="art-eval-research-assistant"):
|
||||
bundle = get_artifact_bundle(artifact_id)
|
||||
assert bundle is not None
|
||||
return bundle
|
||||
|
||||
|
||||
async def test_evaluate_returns_validated_rubric_score():
|
||||
provider = ScriptedJSONProvider(VALID_SCORE)
|
||||
assessor = make_assessor(provider)
|
||||
artifact, rubric = get_bundle()
|
||||
transcript = get_transcript_for_artifact(artifact.artifact_id)
|
||||
result = await assessor.evaluate(artifact, rubric, transcript)
|
||||
assert isinstance(result, RubricScore)
|
||||
assert result.verdict == "mastered"
|
||||
assert len(result.scores) == 4
|
||||
assert result.weighted_total(rubric) == pytest.approx(
|
||||
92 * 0.3 + 88 * 0.3 + 85 * 0.25 + 90 * 0.15
|
||||
)
|
||||
|
||||
|
||||
async def test_evaluate_rejects_invalid_schema_after_retry():
|
||||
"""Plain MockProvider returns non-rubric JSON → 4-layer defense exhausts
|
||||
its single retry and raises StructuredOutputError."""
|
||||
assessor = make_assessor(MockProvider())
|
||||
artifact, rubric = get_bundle()
|
||||
transcript = get_transcript_for_artifact(artifact.artifact_id)
|
||||
with pytest.raises(StructuredOutputError):
|
||||
await assessor.evaluate(artifact, rubric, transcript)
|
||||
|
||||
|
||||
def test_build_evaluation_input_carries_all_inputs():
|
||||
assessor = make_assessor()
|
||||
artifact, rubric = get_bundle()
|
||||
transcript = get_transcript_for_artifact(artifact.artifact_id)
|
||||
text = assessor.build_evaluation_input(artifact, rubric, transcript)
|
||||
assert artifact.name in text
|
||||
assert artifact.evidence_excerpt in text
|
||||
assert "rc-architecture" in text # rubric rendered
|
||||
assert "examiner:" in text # transcript rendered
|
||||
|
||||
|
||||
def test_build_evaluation_input_without_transcript():
|
||||
assessor = make_assessor()
|
||||
artifact, rubric = get_bundle()
|
||||
text = assessor.build_evaluation_input(artifact, rubric, None)
|
||||
assert artifact.name in text
|
||||
assert "examiner:" not in text
|
||||
|
||||
|
||||
def test_system_prompt_names_assessor_persona():
|
||||
prompt = make_assessor().system_prompt(get_learner_context())
|
||||
assert "Assessor" in prompt
|
||||
assert "ONLY" in prompt # JSON-only instruction
|
||||
|
||||
|
||||
def test_rubric_render_in_prompt_is_complete():
|
||||
"""The rubric passed to the model lists every criterion (fair grading)."""
|
||||
artifact, rubric = get_bundle()
|
||||
text = render_rubric(rubric)
|
||||
assert text.count("rc-") == len(rubric.criteria)
|
||||
@@ -0,0 +1,55 @@
|
||||
"""BaseAgent contract tests — stub agent + mock provider."""
|
||||
|
||||
|
||||
from ai_service.agents.base import BaseAgent
|
||||
from ai_service.config import Settings
|
||||
from ai_service.corpus.learner_context import get_learner_context
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
|
||||
class StubAgent(BaseAgent):
|
||||
name = "stub"
|
||||
|
||||
def system_prompt(self, learner_context=None) -> str:
|
||||
return "You are Stub. Answer briefly."
|
||||
|
||||
|
||||
def make_agent() -> StubAgent:
|
||||
return StubAgent(MockProvider(), Settings(provider="mock"))
|
||||
|
||||
|
||||
async def test_build_messages_composition():
|
||||
agent = make_agent()
|
||||
history = [Message(role="user", content="earlier"), Message(role="assistant", content="reply")]
|
||||
messages = agent.build_messages(history, "new question")
|
||||
assert messages[0].role == "system"
|
||||
assert messages[0].content == "You are Stub. Answer briefly."
|
||||
assert [m.content for m in messages[1:]] == ["earlier", "reply", "new question"]
|
||||
|
||||
|
||||
async def test_stream_reply_yields_deltas():
|
||||
agent = make_agent()
|
||||
tokens = [t async for t in agent.stream_reply(user_input="hello")]
|
||||
assert len(tokens) >= 1
|
||||
assert all(isinstance(t, str) for t in tokens)
|
||||
|
||||
|
||||
async def test_stream_reply_with_history_and_context():
|
||||
agent = make_agent()
|
||||
ctx = get_learner_context()
|
||||
history = [Message(role="user", content="earlier"), Message(role="assistant", content="ok")]
|
||||
tokens = [t async for t in agent.stream_reply(history, "next", ctx)]
|
||||
assert tokens
|
||||
|
||||
|
||||
async def test_structured_reply_requires_schema():
|
||||
import pytest
|
||||
|
||||
agent = make_agent()
|
||||
with pytest.raises(ValueError):
|
||||
await agent.structured_reply(user_input="x", schema=None)
|
||||
|
||||
|
||||
async def test_name_defaults():
|
||||
assert make_agent().name == "stub"
|
||||
@@ -0,0 +1,50 @@
|
||||
"""Coach agent tests — persona, message assembly, streaming (REQ-2-005)."""
|
||||
|
||||
from ai_service.agents.coach import CoachAgent
|
||||
from ai_service.config import Settings
|
||||
from ai_service.corpus.learner_context import get_learner_context
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
|
||||
def make_coach() -> CoachAgent:
|
||||
return CoachAgent(MockProvider(), Settings(provider="mock"))
|
||||
|
||||
|
||||
def test_system_prompt_includes_learner_context():
|
||||
coach = make_coach()
|
||||
prompt = coach.system_prompt(get_learner_context())
|
||||
assert "Alex Rivera" in prompt
|
||||
assert "AI Orchestration Engineer (62%)" in prompt
|
||||
assert "retrieval practice" in prompt.lower()
|
||||
assert "one clear next action" in prompt.lower()
|
||||
|
||||
|
||||
def test_system_prompt_marks_current_focus():
|
||||
coach = make_coach()
|
||||
prompt = coach.system_prompt(get_learner_context())
|
||||
assert "Multi-agent communication patterns" in prompt
|
||||
|
||||
|
||||
def test_build_messages_system_history_user():
|
||||
coach = make_coach()
|
||||
history = [
|
||||
Message(role="user", content="earlier"),
|
||||
Message(role="assistant", content="reply"),
|
||||
]
|
||||
messages = coach.build_messages(history, "what next?", get_learner_context())
|
||||
assert messages[0].role == "system"
|
||||
assert "Coach" in messages[0].content
|
||||
assert [m.content for m in messages[1:]] == ["earlier", "reply", "what next?"]
|
||||
|
||||
|
||||
async def test_stream_reply_yields_deltas():
|
||||
coach = make_coach()
|
||||
ctx = get_learner_context()
|
||||
tokens = [t async for t in coach.stream_reply(user_input="hello", learner_context=ctx)]
|
||||
assert tokens
|
||||
assert all(isinstance(t, str) for t in tokens)
|
||||
|
||||
|
||||
def test_agent_name():
|
||||
assert make_coach().name == "coach"
|
||||
@@ -0,0 +1,71 @@
|
||||
"""Learner context corpus tests — D-021 ID alignment + prompt rendering."""
|
||||
|
||||
from ai_service.corpus.learner_context import (
|
||||
LEARNER_CONTEXTS,
|
||||
get_learner_context,
|
||||
)
|
||||
from ai_service.prompts.assessor import render_context as render_assessor
|
||||
from ai_service.prompts.coach import SYSTEM_PROMPT as COACH_PROMPT
|
||||
from ai_service.prompts.coach import render_context as render_coach
|
||||
from ai_service.prompts.lab import SYSTEM_PROMPT as LAB_PROMPT
|
||||
from ai_service.prompts.lab import render_context as render_lab
|
||||
from ai_service.prompts.mentor import SYSTEM_PROMPT as MENTOR_PROMPT
|
||||
from ai_service.prompts.mentor import render_context as render_mentor
|
||||
from ai_service.prompts.proctor import SYSTEM_PROMPT as PROCTOR_PROMPT
|
||||
from ai_service.prompts.proctor import render_context as render_proctor
|
||||
from ai_service.prompts.tutor import SYSTEM_PROMPT as TUTOR_PROMPT
|
||||
from ai_service.prompts.tutor import render_context as render_tutor
|
||||
|
||||
|
||||
def test_default_learner_resolves():
|
||||
ctx = get_learner_context()
|
||||
assert ctx.learner_id == "learner-001"
|
||||
assert ctx.name == "Alex Rivera"
|
||||
|
||||
|
||||
def test_unknown_learner_falls_back_to_default():
|
||||
assert get_learner_context("nobody").learner_id == "learner-001"
|
||||
|
||||
|
||||
def test_ids_align_with_ts_mock_data():
|
||||
# D-021: identical ID strings to packages/mock-data (learner-progress.ts)
|
||||
ctx = get_learner_context("learner-001")
|
||||
stack_ids = {s.stack_id for s in ctx.active_stacks}
|
||||
assert {"stack-orchestration", "stack-safety"} <= stack_ids
|
||||
competency_ids = {c.competency_id for c in ctx.active_competencies}
|
||||
assert "stack-orchestration-c001" in competency_ids
|
||||
|
||||
|
||||
def test_all_prompt_modules_render_without_keyerror():
|
||||
ctx = get_learner_context()
|
||||
for render in (render_coach, render_tutor, render_mentor, render_assessor):
|
||||
values = render(ctx)
|
||||
assert isinstance(values, dict)
|
||||
assert "learner_name" in values
|
||||
assert values["learner_name"] == "Alex Rivera"
|
||||
|
||||
|
||||
def test_prompts_format_map_with_rendered_context():
|
||||
ctx = get_learner_context()
|
||||
for prompt, render in (
|
||||
(COACH_PROMPT, render_coach),
|
||||
(TUTOR_PROMPT, render_tutor),
|
||||
(MENTOR_PROMPT, render_mentor),
|
||||
):
|
||||
rendered = prompt.format_map(render(ctx))
|
||||
assert "Alex Rivera" in rendered
|
||||
assert "{" not in rendered # all placeholders filled
|
||||
|
||||
|
||||
def test_lab_and_proctor_prompts_render():
|
||||
ctx = get_learner_context()
|
||||
# Each module renders through its OWN render_context (its own placeholders).
|
||||
assert "Alex Rivera" in LAB_PROMPT.format_map(render_lab(ctx))
|
||||
assert "Alex Rivera" in PROCTOR_PROMPT.format_map(render_proctor(ctx))
|
||||
for prompt, render in ((LAB_PROMPT, render_lab), (PROCTOR_PROMPT, render_proctor)):
|
||||
rendered = prompt.format_map(render(ctx))
|
||||
assert "{" not in rendered # all placeholders filled
|
||||
|
||||
|
||||
def test_two_seed_learners_exist():
|
||||
assert set(LEARNER_CONTEXTS) == {"learner-001", "learner-002"}
|
||||
@@ -0,0 +1,124 @@
|
||||
"""Telemetry + artifacts corpus tests (REQ-2-007/008 inputs, D-021)."""
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from ai_service.corpus.artifacts import (
|
||||
ARTIFACTS,
|
||||
RUBRICS,
|
||||
get_artifact_bundle,
|
||||
get_transcript_for_artifact,
|
||||
render_rubric,
|
||||
render_transcript,
|
||||
rubric_for_competency,
|
||||
)
|
||||
from ai_service.corpus.telemetry import (
|
||||
LAB_SCENARIOS,
|
||||
get_lab_scenario,
|
||||
summarize_scenario,
|
||||
)
|
||||
|
||||
|
||||
def test_lab_scenarios_addressable_by_id():
|
||||
for scenario_id in (
|
||||
"lab-scenario-strong",
|
||||
"lab-scenario-struggling",
|
||||
"lab-scenario-flagged",
|
||||
):
|
||||
scenario = get_lab_scenario(scenario_id)
|
||||
assert scenario is not None
|
||||
assert scenario.scenario_id == scenario_id
|
||||
|
||||
|
||||
def test_lab_scenarios_have_distinct_event_profiles():
|
||||
strong = get_lab_scenario("lab-scenario-strong")
|
||||
struggling = get_lab_scenario("lab-scenario-struggling")
|
||||
kinds = lambda s: {e.kind for e in s.events} # noqa: E731
|
||||
assert "test_pass" in kinds(strong)
|
||||
assert "test_fail" in kinds(struggling)
|
||||
assert "idle" in kinds(struggling)
|
||||
assert "paste" in kinds(get_lab_scenario("lab-scenario-flagged"))
|
||||
|
||||
|
||||
def test_summarize_scenario_mentions_events():
|
||||
text = summarize_scenario(get_lab_scenario("lab-scenario-struggling"))
|
||||
assert "test_fail" in text
|
||||
assert "ImportError" in text
|
||||
assert "stack-orchestration-c002" in text
|
||||
|
||||
|
||||
def test_unknown_scenario_returns_none():
|
||||
assert get_lab_scenario("lab-scenario-ghost") is None
|
||||
|
||||
|
||||
def test_artifacts_and_rubrics_resolve():
|
||||
bundle = get_artifact_bundle("art-eval-research-assistant")
|
||||
assert bundle is not None
|
||||
artifact, rubric = bundle
|
||||
assert artifact.competency_id == "stack-orchestration-c002"
|
||||
assert rubric.rubric_id == "rubric-orchestration-c002"
|
||||
assert len(rubric.criteria) == 4
|
||||
|
||||
|
||||
def test_unknown_artifact_returns_none():
|
||||
assert get_artifact_bundle("art-eval-ghost") is None
|
||||
|
||||
|
||||
def test_transcripts_pair_with_artifacts():
|
||||
for artifact_id in ARTIFACTS:
|
||||
transcript = get_transcript_for_artifact(artifact_id)
|
||||
assert transcript is not None
|
||||
assert transcript.artifact_id == artifact_id
|
||||
assert len(transcript.turns) >= 4
|
||||
|
||||
|
||||
def test_rubric_render_mentions_all_criteria():
|
||||
rubric = rubric_for_competency("stack-orchestration-c002")
|
||||
text = render_rubric(rubric)
|
||||
for criterion in rubric.criteria:
|
||||
assert criterion.criterion_id in text
|
||||
|
||||
|
||||
def test_transcript_render_has_both_speakers():
|
||||
transcript = get_transcript_for_artifact("art-eval-rag-dashboard")
|
||||
text = render_transcript(transcript)
|
||||
assert "examiner:" in text
|
||||
assert "learner:" in text
|
||||
|
||||
|
||||
_TS_SOURCE_CANDIDATES = [
|
||||
Path(__file__).resolve().parents[4] / "packages" / "mock-data" / "ai-scenarios.ts",
|
||||
Path(__file__).resolve().parents[2] / "ai-scenarios.ts",
|
||||
]
|
||||
|
||||
|
||||
def _ts_source() -> Path:
|
||||
for candidate in _TS_SOURCE_CANDIDATES:
|
||||
if candidate.exists():
|
||||
return candidate
|
||||
pytest.skip("ai-scenarios.ts not found in this checkout layout")
|
||||
|
||||
|
||||
def test_corpus_ids_align_with_ts_mock_data():
|
||||
"""D-021: Python corpus IDs string-identical to ai-scenarios.ts."""
|
||||
content = _ts_source().read_text()
|
||||
ts_ids = re.findall(r"id:\s*'([^']+)'", content)
|
||||
ts_scenarios = ts_ids[: len(LAB_SCENARIOS)]
|
||||
ts_artifacts = ts_ids[len(LAB_SCENARIOS):]
|
||||
assert sorted(ts_scenarios) == sorted(LAB_SCENARIOS), (
|
||||
f"scenario IDs drifted: py={sorted(LAB_SCENARIOS)} ts={sorted(ts_scenarios)}"
|
||||
)
|
||||
assert sorted(ts_artifacts) == sorted(ARTIFACTS), (
|
||||
f"artifact IDs drifted: py={sorted(ARTIFACTS)} ts={sorted(ts_artifacts)}"
|
||||
)
|
||||
|
||||
|
||||
def test_rubric_weights_sum_to_one():
|
||||
"""Every rubric's criteria weights must sum to exactly 1.0."""
|
||||
for rubric in RUBRICS.values():
|
||||
total = sum(c.weight for c in rubric.criteria)
|
||||
assert total == pytest.approx(1.0), (
|
||||
f"{rubric.rubric_id} weights sum to {total}, expected 1.0"
|
||||
)
|
||||
@@ -0,0 +1,42 @@
|
||||
"""Lab agent tests — scenario-driven streaming feedback (REQ-2-007)."""
|
||||
|
||||
from ai_service.agents.lab import LabAgent
|
||||
from ai_service.config import Settings
|
||||
from ai_service.corpus.learner_context import get_learner_context
|
||||
from ai_service.corpus.telemetry import get_lab_scenario, summarize_scenario
|
||||
from ai_service.llm.mock import MockProvider
|
||||
|
||||
|
||||
def make_lab() -> LabAgent:
|
||||
return LabAgent(MockProvider(), Settings(provider="mock"))
|
||||
|
||||
|
||||
def test_system_prompt_names_lab_persona():
|
||||
prompt = make_lab().system_prompt(get_learner_context())
|
||||
assert "Lab" in prompt
|
||||
assert "Alex Rivera" in prompt
|
||||
assert "telemetry" in prompt.lower()
|
||||
|
||||
|
||||
async def test_stream_feedback_mentions_scenario_events():
|
||||
"""Mock-scripted: feedback text derives from scenario timeline input —
|
||||
distinct scenarios produce distinct (deterministic) replies."""
|
||||
lab = make_lab()
|
||||
ctx = get_learner_context()
|
||||
strong = get_lab_scenario("lab-scenario-strong")
|
||||
struggling = get_lab_scenario("lab-scenario-struggling")
|
||||
|
||||
strong_reply = "".join([t async for t in lab.stream_feedback(strong, ctx)])
|
||||
struggling_reply = "".join([t async for t in lab.stream_feedback(struggling, ctx)])
|
||||
assert strong_reply
|
||||
assert strong_reply != struggling_reply # scenario-driven, not canned
|
||||
|
||||
|
||||
def test_build_evaluation_messages_carry_timeline():
|
||||
lab = make_lab()
|
||||
scenario = get_lab_scenario("lab-scenario-flagged")
|
||||
timeline = summarize_scenario(scenario)
|
||||
messages = lab.build_messages(None, timeline, get_learner_context())
|
||||
assert messages[0].role == "system"
|
||||
assert "paste" in messages[-1].content
|
||||
assert "2,400 chars" in messages[-1].content
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Mentor agent tests — career narrative, session-backed (REQ-2-010)."""
|
||||
|
||||
from ai_service.agents.mentor import MentorAgent
|
||||
from ai_service.config import Settings
|
||||
from ai_service.corpus.learner_context import get_learner_context
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
|
||||
def make_mentor() -> MentorAgent:
|
||||
return MentorAgent(MockProvider(), Settings(provider="mock"))
|
||||
|
||||
|
||||
def test_system_prompt_carries_full_learner_context():
|
||||
prompt = make_mentor().system_prompt(get_learner_context())
|
||||
assert "Alex Rivera" in prompt
|
||||
assert "AI Orchestration Engineer (62%)" in prompt
|
||||
assert "Multi-agent research assistant" in prompt # artifacts by name
|
||||
assert "4" in prompt # microcredential count
|
||||
|
||||
|
||||
def test_build_messages_system_history_user():
|
||||
mentor = make_mentor()
|
||||
history = [Message(role="user", content="what next?"),
|
||||
Message(role="assistant", content="trajectory...")]
|
||||
messages = mentor.build_messages(history, "tell me more", get_learner_context())
|
||||
assert messages[0].role == "system"
|
||||
assert "Mentor" in messages[0].content
|
||||
assert [m.content for m in messages[1:]] == ["what next?", "trajectory...", "tell me more"]
|
||||
|
||||
|
||||
async def test_stream_reply_mentions_learner_context_in_output():
|
||||
"""Mock-scripted: narrative derives from context-injected messages —
|
||||
different learner contexts produce distinct (deterministic) replies."""
|
||||
mentor = make_mentor()
|
||||
alex = get_learner_context("learner-001")
|
||||
priya = get_learner_context("learner-002")
|
||||
alex_reply = "".join(
|
||||
[t async for t in mentor.stream_reply(user_input="narrate", learner_context=alex)]
|
||||
)
|
||||
priya_reply = "".join(
|
||||
[t async for t in mentor.stream_reply(user_input="narrate", learner_context=priya)]
|
||||
)
|
||||
assert alex_reply
|
||||
assert alex_reply != priya_reply
|
||||
|
||||
|
||||
def test_agent_name():
|
||||
assert make_mentor().name == "mentor"
|
||||
@@ -0,0 +1,84 @@
|
||||
"""Proctor agent tests — structured integrity signals (REQ-2-009)."""
|
||||
|
||||
import pytest
|
||||
|
||||
from ai_service.agents.proctor import ProctorAgent, ProctorAssessment
|
||||
from ai_service.agents.structured import StructuredOutputError
|
||||
from ai_service.config import Settings
|
||||
from ai_service.corpus.learner_context import get_learner_context
|
||||
from ai_service.corpus.telemetry import (
|
||||
PROCTOR_SCENARIOS,
|
||||
get_proctor_scenario,
|
||||
summarize_proctor_scenario,
|
||||
)
|
||||
from ai_service.llm.mock import MockProvider, ScriptedJSONProvider
|
||||
|
||||
VALID_ASSESSMENT = {
|
||||
"scenario_id": "proctor-scenario-distracted",
|
||||
"signals": [
|
||||
{"signal_type": "context_switch", "severity": "low",
|
||||
"note": "Tab switch to docs at t+120s — normal engineering behavior"},
|
||||
{"signal_type": "idle_gap", "severity": "medium",
|
||||
"note": "5-minute idle at t+300s followed by more tab switches"},
|
||||
],
|
||||
"intervention": "Offer a short break and ask the learner to restate their answer plan",
|
||||
"summary": "Distracted but explainable session; coach the focus pattern, don't flag it",
|
||||
}
|
||||
|
||||
|
||||
def make_proctor(provider=None) -> ProctorAgent:
|
||||
return ProctorAgent(provider or MockProvider(), Settings(provider="mock"))
|
||||
|
||||
|
||||
def test_proctor_scenarios_addressable_and_distinct_type():
|
||||
healthy = get_proctor_scenario("proctor-scenario-healthy")
|
||||
flagged = get_proctor_scenario("proctor-scenario-flagged")
|
||||
assert healthy is not None and flagged is not None
|
||||
kinds = lambda s: {e.kind for e in s.events} # noqa: E731
|
||||
assert "tab_switch" in kinds(get_proctor_scenario("proctor-scenario-distracted"))
|
||||
assert "paste_large" in kinds(flagged)
|
||||
assert not kinds(healthy) & {"tab_switch", "paste_large", "focus_lost"}
|
||||
|
||||
|
||||
def test_unknown_proctor_scenario_none():
|
||||
assert get_proctor_scenario("proctor-scenario-ghost") is None
|
||||
|
||||
|
||||
async def test_assess_returns_validated_signals():
|
||||
provider = ScriptedJSONProvider(VALID_ASSESSMENT)
|
||||
proctor = make_proctor(provider)
|
||||
scenario = get_proctor_scenario("proctor-scenario-distracted")
|
||||
result = await proctor.assess(scenario, get_learner_context())
|
||||
assert isinstance(result, ProctorAssessment)
|
||||
assert len(result.signals) == 2
|
||||
assert result.signals[0].severity == "low"
|
||||
assert "break" in result.intervention.lower()
|
||||
|
||||
|
||||
async def test_assess_rejects_invalid_after_retry():
|
||||
proctor = make_proctor(MockProvider()) # non-schema JSON
|
||||
scenario = get_proctor_scenario("proctor-scenario-healthy")
|
||||
with pytest.raises(StructuredOutputError):
|
||||
await proctor.assess(scenario, get_learner_context())
|
||||
|
||||
|
||||
def test_system_prompt_is_coaching_not_punitive():
|
||||
prompt = make_proctor().system_prompt(get_learner_context())
|
||||
assert "never punitive" in prompt.lower()
|
||||
assert "good faith" in prompt.lower()
|
||||
assert "ONLY" in prompt # JSON-only instruction
|
||||
|
||||
|
||||
def test_timeline_summary_carries_events():
|
||||
scenario = get_proctor_scenario("proctor-scenario-flagged")
|
||||
text = summarize_proctor_scenario(scenario)
|
||||
assert "paste_large" in text
|
||||
assert "3,100 chars" in text
|
||||
|
||||
|
||||
def test_all_three_proctor_scenarios_exist():
|
||||
assert set(PROCTOR_SCENARIOS) == {
|
||||
"proctor-scenario-healthy",
|
||||
"proctor-scenario-distracted",
|
||||
"proctor-scenario-flagged",
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
"""Agent registry tests — register/get round-trip, error paths (G-4), builtins."""
|
||||
|
||||
|
||||
import pytest
|
||||
|
||||
from ai_service.agents.base import BaseAgent
|
||||
from ai_service.agents.coach import CoachAgent
|
||||
from ai_service.agents.registry import (
|
||||
AgentRegistry,
|
||||
DuplicateAgentError,
|
||||
UnknownAgentError,
|
||||
register_builtin_agents,
|
||||
)
|
||||
from ai_service.agents.tutor import TutorAgent
|
||||
from ai_service.config import Settings
|
||||
from ai_service.llm.mock import MockProvider
|
||||
|
||||
|
||||
class DummyAgent(BaseAgent):
|
||||
name = "dummy"
|
||||
|
||||
def system_prompt(self, learner_context=None) -> str:
|
||||
return "dummy"
|
||||
|
||||
|
||||
def make_factory():
|
||||
def factory(provider, settings):
|
||||
return DummyAgent(provider, settings)
|
||||
return factory
|
||||
|
||||
|
||||
def test_register_and_get():
|
||||
registry = AgentRegistry()
|
||||
registry.register("dummy", make_factory())
|
||||
agent = registry.get(MockProvider(), Settings(provider="mock"), "dummy")
|
||||
assert isinstance(agent, DummyAgent)
|
||||
assert agent.name == "dummy"
|
||||
|
||||
|
||||
def test_unknown_agent_raises():
|
||||
registry = AgentRegistry()
|
||||
with pytest.raises(UnknownAgentError):
|
||||
registry.get(MockProvider(), Settings(provider="mock"), "ghost")
|
||||
|
||||
|
||||
def test_duplicate_registration_raises():
|
||||
registry = AgentRegistry()
|
||||
registry.register("dummy", make_factory())
|
||||
with pytest.raises(DuplicateAgentError):
|
||||
registry.register("dummy", make_factory())
|
||||
|
||||
|
||||
def test_names_sorted():
|
||||
registry = AgentRegistry()
|
||||
registry.register("zeta", make_factory())
|
||||
registry.register("alpha", make_factory())
|
||||
assert registry.names() == ["alpha", "zeta"]
|
||||
|
||||
|
||||
def test_builtin_agents_register_and_resolve():
|
||||
registry = AgentRegistry()
|
||||
register_builtin_agents(registry)
|
||||
assert set(registry.names()) >= {"coach", "tutor"}
|
||||
settings = Settings(provider="mock")
|
||||
coach = registry.get(MockProvider(), settings, "coach")
|
||||
tutor = registry.get(MockProvider(), settings, "tutor")
|
||||
assert isinstance(coach, CoachAgent)
|
||||
assert isinstance(tutor, TutorAgent)
|
||||
|
||||
|
||||
def test_lab_and_assessor_resolve_via_registry():
|
||||
"""Phase 4: lab + assessor registered centrally (Task 4-3-01)."""
|
||||
from ai_service.agents.assessor import AssessorAgent
|
||||
from ai_service.agents.lab import LabAgent
|
||||
|
||||
registry = AgentRegistry()
|
||||
register_builtin_agents(registry)
|
||||
assert {"lab", "assessor"} <= set(registry.names())
|
||||
settings = Settings(provider="mock")
|
||||
lab = registry.get(MockProvider(), settings, "lab")
|
||||
assessor = registry.get(MockProvider(), settings, "assessor")
|
||||
assert isinstance(lab, LabAgent)
|
||||
assert isinstance(assessor, AssessorAgent)
|
||||
|
||||
|
||||
def test_proctor_and_mentor_resolve_via_registry():
|
||||
"""Phase 5: proctor + mentor registered centrally (Tasks 5-1-02/5-2-01)."""
|
||||
from ai_service.agents.mentor import MentorAgent
|
||||
from ai_service.agents.proctor import ProctorAgent
|
||||
|
||||
registry = AgentRegistry()
|
||||
register_builtin_agents(registry)
|
||||
settings = Settings(provider="mock")
|
||||
proctor = registry.get(MockProvider(), settings, "proctor")
|
||||
mentor = registry.get(MockProvider(), settings, "mentor")
|
||||
assert isinstance(proctor, ProctorAgent)
|
||||
assert isinstance(mentor, MentorAgent)
|
||||
|
||||
|
||||
def test_registry_resolves_all_six_agents():
|
||||
"""Must-Have (Phase 5): the full roster — coach/tutor/lab/assessor/proctor/mentor."""
|
||||
from ai_service.agents.assessor import AssessorAgent
|
||||
from ai_service.agents.coach import CoachAgent
|
||||
from ai_service.agents.lab import LabAgent
|
||||
from ai_service.agents.mentor import MentorAgent
|
||||
from ai_service.agents.proctor import ProctorAgent
|
||||
from ai_service.agents.tutor import TutorAgent
|
||||
|
||||
registry = AgentRegistry()
|
||||
register_builtin_agents(registry)
|
||||
assert registry.names() == ["assessor", "coach", "lab", "mentor", "proctor", "tutor"]
|
||||
settings = Settings(provider="mock")
|
||||
expected = {
|
||||
"coach": CoachAgent,
|
||||
"tutor": TutorAgent,
|
||||
"lab": LabAgent,
|
||||
"assessor": AssessorAgent,
|
||||
"proctor": ProctorAgent,
|
||||
"mentor": MentorAgent,
|
||||
}
|
||||
for name, cls in expected.items():
|
||||
agent = registry.get(MockProvider(), settings, name)
|
||||
assert isinstance(agent, cls), f"{name} resolved to {type(agent).__name__}"
|
||||
assert agent.name == name
|
||||
|
||||
|
||||
def test_builtin_registration_is_idempotent_safe():
|
||||
"""Duplicate registration raises — builtin bootstrap must be called once."""
|
||||
registry = AgentRegistry()
|
||||
register_builtin_agents(registry)
|
||||
with pytest.raises(DuplicateAgentError):
|
||||
register_builtin_agents(registry)
|
||||
@@ -0,0 +1,86 @@
|
||||
"""SessionStore tests — create/append/window/LRU/agent scoping (D-019)."""
|
||||
|
||||
import pytest
|
||||
|
||||
from ai_service.agents.session import InMemorySessionStore
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
|
||||
def msg(n: int) -> Message:
|
||||
return Message(role="user", content=f"m{n}")
|
||||
|
||||
|
||||
async def test_create_and_get():
|
||||
store = InMemorySessionStore()
|
||||
session = await store.create("s1", agent="coach")
|
||||
assert session.agent == "coach"
|
||||
assert (await store.get("s1")).session_id == "s1"
|
||||
assert await store.get("missing") is None
|
||||
|
||||
|
||||
async def test_append_and_history_window():
|
||||
store = InMemorySessionStore()
|
||||
await store.create("s1", agent="coach")
|
||||
for i in range(30):
|
||||
await store.append("s1", msg(i))
|
||||
window = await store.history_window("s1", max_messages=20)
|
||||
assert len(window) == 20
|
||||
assert window[0].content == "m10" # last 20 of m0..m29
|
||||
assert window[-1].content == "m29"
|
||||
|
||||
|
||||
async def test_default_window_uses_20():
|
||||
store = InMemorySessionStore()
|
||||
await store.create("s1", agent="coach")
|
||||
for i in range(25):
|
||||
await store.append("s1", msg(i))
|
||||
window = await store.history_window("s1")
|
||||
assert len(window) == 20
|
||||
assert window[0].content == "m5"
|
||||
|
||||
|
||||
async def test_lru_eviction_at_cap():
|
||||
store = InMemorySessionStore(window=20, max_sessions=3)
|
||||
for i in range(3):
|
||||
await store.create(f"s{i}", agent="coach")
|
||||
# touch s0 so s1 becomes least-recently-used
|
||||
await store.get("s0")
|
||||
await store.create("s3", agent="coach") # evicts s1
|
||||
assert await store.get("s1") is None
|
||||
assert await store.get("s0") is not None
|
||||
assert await store.get("s2") is not None
|
||||
assert await store.get("s3") is not None
|
||||
|
||||
|
||||
async def test_lru_eviction_at_default_500_cap():
|
||||
store = InMemorySessionStore() # defaults: window=20, max_sessions=500
|
||||
for i in range(500):
|
||||
await store.create(f"s{i}", agent="coach")
|
||||
await store.get("s0") # touch the oldest → s1 becomes least-recently-used
|
||||
await store.create("s500", agent="coach") # evicts s1
|
||||
assert await store.get("s1") is None
|
||||
assert await store.get("s0") is not None
|
||||
assert await store.get("s499") is not None
|
||||
assert await store.get("s500") is not None
|
||||
|
||||
|
||||
async def test_append_unknown_session_raises():
|
||||
store = InMemorySessionStore()
|
||||
with pytest.raises(KeyError):
|
||||
await store.append("nope", msg(0))
|
||||
|
||||
|
||||
async def test_delete():
|
||||
store = InMemorySessionStore()
|
||||
await store.create("s1", agent="coach")
|
||||
await store.delete("s1")
|
||||
assert await store.get("s1") is None
|
||||
|
||||
|
||||
async def test_sessions_are_agent_scoped():
|
||||
store = InMemorySessionStore()
|
||||
a = await store.create("coach-session", agent="coach")
|
||||
b = await store.create("tutor-session", agent="tutor")
|
||||
assert a.agent == "coach"
|
||||
assert b.agent == "tutor"
|
||||
assert a.session_id != b.session_id
|
||||
@@ -0,0 +1,156 @@
|
||||
"""Structured output defense tests — 4 layers (D-020), against mock providers."""
|
||||
|
||||
import pytest
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ai_service.agents.structured import (
|
||||
StructuredOutputError,
|
||||
extract_json_object,
|
||||
parse_structured,
|
||||
structured_completion,
|
||||
)
|
||||
from ai_service.llm.mock import MockProvider, ScriptedJSONProvider
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
|
||||
class Score(BaseModel):
|
||||
score: int
|
||||
verdict: str
|
||||
|
||||
|
||||
HINT = '{"score": <int 0-100>, "verdict": "<short verdict>"}'
|
||||
|
||||
|
||||
def test_extract_json_plain():
|
||||
assert extract_json_object('{"a": 1}') == '{"a": 1}'
|
||||
|
||||
|
||||
def test_extract_json_fenced():
|
||||
text = '```json\n{"a": 1}\n```'
|
||||
assert extract_json_object(text) == '{"a": 1}'
|
||||
|
||||
|
||||
def test_extract_json_with_prose_around():
|
||||
text = 'Sure! Here is my answer: {"a": {"b": "x } y"}, "c": 2} hope that helps'
|
||||
assert extract_json_object(text) == '{"a": {"b": "x } y"}, "c": 2}'
|
||||
|
||||
|
||||
def test_extract_json_no_object_raises():
|
||||
with pytest.raises(StructuredOutputError):
|
||||
extract_json_object("no json here")
|
||||
|
||||
|
||||
def test_extract_json_unbalanced_raises():
|
||||
with pytest.raises(StructuredOutputError):
|
||||
extract_json_object('{"a": 1')
|
||||
|
||||
|
||||
def test_parse_structured_valid():
|
||||
result = parse_structured('{"score": 88, "verdict": "solid"}', Score)
|
||||
assert result.score == 88
|
||||
|
||||
|
||||
def test_parse_structured_invalid_schema_raises():
|
||||
with pytest.raises(StructuredOutputError):
|
||||
parse_structured('{"wrong": "shape"}', Score)
|
||||
|
||||
|
||||
async def test_structured_completion_happy_path():
|
||||
provider = ScriptedJSONProvider({"score": 91, "verdict": "excellent work"})
|
||||
messages = [Message(role="user", content="grade my artifact")]
|
||||
result = await structured_completion(
|
||||
provider, messages, model="m", schema=Score, schema_hint=HINT
|
||||
)
|
||||
assert result.score == 91
|
||||
assert result.verdict == "excellent work"
|
||||
|
||||
|
||||
async def test_structured_completion_retries_then_raises():
|
||||
# Plain MockProvider returns non-schema JSON for json_object requests →
|
||||
# both attempts fail validation → StructuredOutputError after ONE retry.
|
||||
provider = MockProvider()
|
||||
provider.received_calls = []
|
||||
messages = [Message(role="user", content="grade me")]
|
||||
with pytest.raises(StructuredOutputError):
|
||||
await structured_completion(
|
||||
provider, messages, model="m", schema=Score, schema_hint=HINT
|
||||
)
|
||||
|
||||
|
||||
async def test_structured_completion_retry_succeeds_after_invalid_first_response():
|
||||
# Layer 4 recovery: first reply is wrong-schema fenced JSON, retry is valid.
|
||||
class FlakyProvider(MockProvider):
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.n = 0
|
||||
self.retry_request: list[Message] = []
|
||||
|
||||
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
self.n += 1
|
||||
if self.n == 1:
|
||||
return '```json\n{"summary": "wrong shape"}\n```'
|
||||
self.retry_request = list(messages)
|
||||
return '{"score": 75, "verdict": "recovered"}'
|
||||
|
||||
provider = FlakyProvider()
|
||||
messages = [Message(role="user", content="grade me")]
|
||||
result = await structured_completion(
|
||||
provider, messages, model="m", schema=Score, schema_hint=HINT
|
||||
)
|
||||
assert result.score == 75
|
||||
assert result.verdict == "recovered"
|
||||
assert provider.n == 2
|
||||
# The retry must feed the validation error back to the model.
|
||||
retry_contents = " ".join(m.content for m in provider.retry_request)
|
||||
assert "previous response was invalid" in retry_contents
|
||||
assert HINT in retry_contents
|
||||
|
||||
|
||||
async def test_structured_completion_is_bounded_to_one_retry():
|
||||
# Permanently-invalid provider: exactly two provider calls, then raise.
|
||||
class CountingProvider(MockProvider):
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.calls = 0
|
||||
|
||||
async def chat(self, messages, *, model, temperature=0.7, response_format=None):
|
||||
self.calls += 1
|
||||
return await super().chat(
|
||||
messages, model=model, response_format=response_format
|
||||
)
|
||||
|
||||
provider = CountingProvider()
|
||||
messages = [Message(role="user", content="grade me")]
|
||||
with pytest.raises(StructuredOutputError, match="after retry"):
|
||||
await structured_completion(
|
||||
provider, messages, model="m", schema=Score, schema_hint=HINT
|
||||
)
|
||||
assert provider.calls == 2
|
||||
|
||||
|
||||
async def test_structured_completion_sends_schema_instruction():
|
||||
"""Layer 2: the schema hint must reach the provider in the request."""
|
||||
provider = ScriptedJSONProvider({"score": 70, "verdict": "passing"})
|
||||
captured: list[list] = []
|
||||
|
||||
original = provider.chat
|
||||
|
||||
async def recording_chat(messages, *, model, temperature=0.7, response_format=None):
|
||||
captured.append(list(messages))
|
||||
return await original(
|
||||
messages, model=model, temperature=temperature, response_format=response_format
|
||||
)
|
||||
|
||||
provider.chat = recording_chat
|
||||
messages = [Message(role="user", content="grade")]
|
||||
await structured_completion(provider, messages, model="m", schema=Score, schema_hint=HINT)
|
||||
assert captured, "provider was never called"
|
||||
last_user = next(m for m in reversed(captured[0]) if m.role == "user")
|
||||
assert HINT in last_user.content
|
||||
assert "ONLY" in last_user.content # JSON-only instruction present
|
||||
|
||||
# Determinism: same request yields same reply
|
||||
again = await structured_completion(
|
||||
provider, messages, model="m", schema=Score, schema_hint=HINT
|
||||
)
|
||||
assert again.score == 70
|
||||
@@ -0,0 +1,76 @@
|
||||
"""Tutor agent tests — persona, Socratic structure, distinctness vs Coach (REQ-2-006)."""
|
||||
|
||||
from ai_service.agents.coach import CoachAgent
|
||||
from ai_service.agents.tutor import TutorAgent
|
||||
from ai_service.config import Settings
|
||||
from ai_service.corpus.learner_context import get_learner_context
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
|
||||
def make_tutor() -> TutorAgent:
|
||||
return TutorAgent(MockProvider(), Settings(provider="mock"))
|
||||
|
||||
|
||||
def test_system_prompt_includes_learner_context():
|
||||
tutor = make_tutor()
|
||||
prompt = tutor.system_prompt(get_learner_context())
|
||||
assert "Alex Rivera" in prompt
|
||||
assert "Socratic" in prompt
|
||||
assert "ONE concept" in prompt
|
||||
|
||||
|
||||
def test_system_prompt_marks_current_focus():
|
||||
tutor = make_tutor()
|
||||
prompt = tutor.system_prompt(get_learner_context())
|
||||
assert "Multi-agent communication patterns" in prompt
|
||||
|
||||
|
||||
def test_build_messages_system_history_user():
|
||||
tutor = make_tutor()
|
||||
history = [Message(role="user", content="q1"), Message(role="assistant", content="a1")]
|
||||
messages = tutor.build_messages(history, "explain again", get_learner_context())
|
||||
assert messages[0].role == "system"
|
||||
assert "Tutor" in messages[0].content
|
||||
assert [m.content for m in messages[1:]] == ["q1", "a1", "explain again"]
|
||||
|
||||
|
||||
async def test_stream_reply_yields_deltas():
|
||||
tutor = make_tutor()
|
||||
tokens = [
|
||||
t
|
||||
async for t in tutor.stream_reply(
|
||||
user_input="hi", learner_context=get_learner_context()
|
||||
)
|
||||
]
|
||||
assert tokens
|
||||
|
||||
|
||||
def test_coach_and_tutor_personas_are_distinct():
|
||||
"""Distinct system prompts (P3 must-have)."""
|
||||
ctx = get_learner_context()
|
||||
coach_prompt = CoachAgent(MockProvider(), Settings(provider="mock")).system_prompt(ctx)
|
||||
tutor_prompt = TutorAgent(MockProvider(), Settings(provider="mock")).system_prompt(ctx)
|
||||
assert coach_prompt != tutor_prompt
|
||||
assert "retrieval practice" in coach_prompt.lower()
|
||||
assert "Socratic" in tutor_prompt
|
||||
|
||||
|
||||
async def test_coach_and_tutor_stream_outputs_are_distinct():
|
||||
"""Mock outputs differ because system prompts differ (hash-seeded on content)."""
|
||||
ctx = get_learner_context()
|
||||
settings = Settings(provider="mock")
|
||||
|
||||
async def full_reply(agent):
|
||||
tokens = [
|
||||
t async for t in agent.stream_reply(user_input="stuck", learner_context=ctx)
|
||||
]
|
||||
return "".join(tokens)
|
||||
|
||||
coach_tokens = await full_reply(CoachAgent(MockProvider(), settings))
|
||||
tutor_tokens = await full_reply(TutorAgent(MockProvider(), settings))
|
||||
assert coach_tokens != tutor_tokens
|
||||
|
||||
|
||||
def test_agent_name():
|
||||
assert make_tutor().name == "tutor"
|
||||
@@ -0,0 +1,79 @@
|
||||
"""Assessment evaluate endpoint tests — validated JSON, 404s (REQ-2-008)."""
|
||||
|
||||
|
||||
from ai_service.agents.assessor import RubricScore
|
||||
from ai_service.llm.mock import ScriptedJSONProvider
|
||||
|
||||
VALID_SCORE = {
|
||||
"rubric_id": "rubric-orchestration-c002",
|
||||
"artifact_id": "art-eval-research-assistant",
|
||||
"competency_id": "stack-orchestration-c002",
|
||||
"scores": [
|
||||
{"criterion_id": "rc-architecture", "name": "Agent architecture soundness",
|
||||
"score": 92, "evidence": "Explicit state schema"},
|
||||
{"criterion_id": "rc-communication", "name": "Inter-agent communication design",
|
||||
"score": 88, "evidence": "Typed ToolMessage responses"},
|
||||
{"criterion_id": "rc-reliability", "name": "Reliability engineering",
|
||||
"score": 85, "evidence": "3-retry loop"},
|
||||
{"criterion_id": "rc-process", "name": "Process trace quality",
|
||||
"score": 90, "evidence": "Iterative checkpoints"},
|
||||
],
|
||||
"strengths": ["Clean state boundaries", "Failure-aware tools"],
|
||||
"gaps": ["No reviewer node", "Diagram only in README"],
|
||||
"verdict": "mastered",
|
||||
}
|
||||
|
||||
|
||||
def test_evaluate_returns_validated_rubric_json(client):
|
||||
# Swap the app provider for a scripted-JSON provider for this test
|
||||
original = client.app.state.provider
|
||||
client.app.state.provider = ScriptedJSONProvider(VALID_SCORE)
|
||||
try:
|
||||
response = client.post(
|
||||
"/v1/assessment/evaluate",
|
||||
json={"artifact_id": "art-eval-research-assistant"},
|
||||
)
|
||||
finally:
|
||||
client.app.state.provider = original
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
validated = RubricScore.model_validate(data) # response contract holds
|
||||
assert validated.verdict == "mastered"
|
||||
assert len(validated.scores) == 4
|
||||
|
||||
|
||||
def test_unknown_artifact_404(client):
|
||||
response = client.post("/v1/assessment/evaluate", json={"artifact_id": "ghost"})
|
||||
assert response.status_code == 404
|
||||
assert "ghost" in response.json()["detail"]
|
||||
|
||||
|
||||
def test_unparseable_provider_502(client):
|
||||
"""Plain MockProvider yields non-rubric JSON → structured defense exhausts
|
||||
retry → endpoint translates to 502 (bad gateway to the model)."""
|
||||
# default mock already returns non-rubric JSON
|
||||
response = client.post(
|
||||
"/v1/assessment/evaluate",
|
||||
json={"artifact_id": "art-eval-research-assistant"},
|
||||
)
|
||||
assert response.status_code == 502
|
||||
assert "failed" in response.json()["detail"].lower()
|
||||
|
||||
|
||||
def test_missing_artifact_id_422(client):
|
||||
response = client.post("/v1/assessment/evaluate", json={})
|
||||
assert response.status_code == 422
|
||||
|
||||
|
||||
def test_second_artifact_also_evaluates(client):
|
||||
original = client.app.state.provider
|
||||
payload = dict(VALID_SCORE, artifact_id="art-eval-rag-dashboard")
|
||||
client.app.state.provider = ScriptedJSONProvider(payload)
|
||||
try:
|
||||
response = client.post(
|
||||
"/v1/assessment/evaluate", json={"artifact_id": "art-eval-rag-dashboard"}
|
||||
)
|
||||
finally:
|
||||
client.app.state.provider = original
|
||||
assert response.status_code == 200
|
||||
assert response.json()["artifact_id"] == "art-eval-rag-dashboard"
|
||||
@@ -0,0 +1,241 @@
|
||||
"""Chat endpoint session integration tests — history persistence + windowed replay."""
|
||||
|
||||
import json
|
||||
|
||||
|
||||
def stream_events(client, payload) -> list[dict]:
|
||||
with client.stream("POST", "/v1/chat/stream", json=payload) as response:
|
||||
assert response.status_code == 200
|
||||
events = []
|
||||
for line in response.iter_lines():
|
||||
if line.startswith("data:"):
|
||||
d = line.removeprefix("data:").strip()
|
||||
if d == "[DONE]":
|
||||
events.append({"type": "[DONE]"})
|
||||
else:
|
||||
events.append(json.loads(d))
|
||||
return events
|
||||
|
||||
|
||||
def test_first_turn_creates_session_and_persists(client):
|
||||
payload = {
|
||||
"agent": "coach",
|
||||
"session_id": "sess-1",
|
||||
"messages": [{"role": "user", "content": "first turn"}],
|
||||
}
|
||||
events = stream_events(client, payload)
|
||||
assert events[0]["type"] == "meta"
|
||||
assert events[0]["session_id"] == "sess-1"
|
||||
store = client.app.state.session_store
|
||||
|
||||
import asyncio
|
||||
|
||||
async def check():
|
||||
return await store.history_window("sess-1")
|
||||
|
||||
contents = [m.content for m in asyncio.run(check())]
|
||||
assert "first turn" in contents
|
||||
assert any("Think" in c or "[" in c for c in contents) # mock reply persisted
|
||||
|
||||
|
||||
def test_second_turn_replays_windowed_history(client):
|
||||
payload1 = {
|
||||
"agent": "coach",
|
||||
"session_id": "sess-2",
|
||||
"messages": [{"role": "user", "content": "turn one"}],
|
||||
}
|
||||
stream_events(client, payload1)
|
||||
# Second turn: the API passes history + new message to the provider.
|
||||
# With the mock provider we cannot observe provider inputs directly,
|
||||
# but the session store must now hold both turns.
|
||||
store = client.app.state.session_store
|
||||
# The store is async; use the app's internals through a short event loop
|
||||
import asyncio
|
||||
result = {}
|
||||
|
||||
async def check():
|
||||
result["window"] = await store.history_window("sess-2")
|
||||
|
||||
asyncio.run(check())
|
||||
contents = [m.content for m in result["window"]]
|
||||
assert "turn one" in contents
|
||||
assert any(m.role == "assistant" for m in result["window"])
|
||||
|
||||
|
||||
def test_second_turn_replays_history_to_provider(client):
|
||||
# Observable provider input: a recording provider wrapper captures what
|
||||
# the endpoint sends. Turn 2 must include turn 1's persisted messages.
|
||||
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
provider = client.app.state.provider
|
||||
original = provider.stream_chat
|
||||
captured: list[list[Message]] = []
|
||||
|
||||
async def recording_stream(messages, *, model, temperature=0.7, response_format=None):
|
||||
captured.append(list(messages))
|
||||
async for t in original(
|
||||
messages, model=model, temperature=temperature, response_format=response_format
|
||||
):
|
||||
yield t
|
||||
|
||||
provider.stream_chat = recording_stream
|
||||
try:
|
||||
stream_events(client, {
|
||||
"agent": "coach", "session_id": "sess-replay",
|
||||
"messages": [{"role": "user", "content": "turn one"}],
|
||||
})
|
||||
stream_events(client, {
|
||||
"agent": "coach", "session_id": "sess-replay",
|
||||
"messages": [{"role": "user", "content": "turn two"}],
|
||||
})
|
||||
finally:
|
||||
provider.stream_chat = original
|
||||
|
||||
assert len(captured) == 2
|
||||
turn1, turn2 = captured
|
||||
# Agent routing (P3): provider now receives [system, ...history, user turn]
|
||||
assert turn1[0].role == "system"
|
||||
assert turn1[-1].content == "turn one"
|
||||
assert len(turn1) == 2 # system + first user turn
|
||||
turn2_contents = [m.content for m in turn2]
|
||||
assert "turn one" in turn2_contents
|
||||
assert "turn two" in turn2_contents
|
||||
assert any(m.role == "assistant" for m in turn2) # persisted reply replayed
|
||||
assert "turn two" == turn2_contents[-1] # new user turn last
|
||||
assert turn2[0].role == "system" # every routed call starts with the persona
|
||||
|
||||
|
||||
def test_history_replay_is_windowed(client):
|
||||
# Windowing: only the last 20 stored messages are replayed to the provider.
|
||||
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
provider = client.app.state.provider
|
||||
original = provider.stream_chat
|
||||
captured: list[list[Message]] = []
|
||||
|
||||
async def recording_stream(messages, *, model, temperature=0.7, response_format=None):
|
||||
captured.append(list(messages))
|
||||
async for t in original(
|
||||
messages, model=model, temperature=temperature, response_format=response_format
|
||||
):
|
||||
yield t
|
||||
|
||||
provider.stream_chat = recording_stream
|
||||
try:
|
||||
# 15 turns → 30 persisted messages (user + assistant per turn) > 20 window.
|
||||
for i in range(15):
|
||||
stream_events(client, {
|
||||
"agent": "coach", "session_id": "sess-window",
|
||||
"messages": [{"role": "user", "content": f"turn {i}"}],
|
||||
})
|
||||
finally:
|
||||
provider.stream_chat = original
|
||||
|
||||
last_input = captured[-1]
|
||||
contents = [m.content for m in last_input]
|
||||
# system prompt + window(20) + the new user message
|
||||
assert last_input[0].role == "system"
|
||||
assert len(last_input) == 1 + 20 + 1
|
||||
assert "turn 0" not in contents # oldest messages trimmed out of replay
|
||||
assert "turn 14" in contents
|
||||
assert contents[-1] == "turn 14"
|
||||
|
||||
|
||||
def test_replayed_system_prompt_carries_routed_persona(client):
|
||||
"""History replay puts the routed agent's persona at position 0 (P3).
|
||||
|
||||
The system prompt on every turn — including replays — must match the
|
||||
routed agent: Coach calls get the Coach persona, Tutor calls the Tutor
|
||||
persona, and the two are observably distinct.
|
||||
"""
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
def capture_two_turns(agent_name: str, session: str) -> list[list[Message]]:
|
||||
provider = client.app.state.provider
|
||||
original = provider.stream_chat
|
||||
captured: list[list[Message]] = []
|
||||
|
||||
async def recording_stream(messages, *, model, temperature=0.7, response_format=None):
|
||||
captured.append(list(messages))
|
||||
async for t in original(
|
||||
messages, model=model, temperature=temperature, response_format=response_format
|
||||
):
|
||||
yield t
|
||||
|
||||
provider.stream_chat = recording_stream
|
||||
try:
|
||||
for content in ("turn one", "turn two"):
|
||||
stream_events(client, {
|
||||
"agent": agent_name, "session_id": session,
|
||||
"messages": [{"role": "user", "content": content}],
|
||||
})
|
||||
finally:
|
||||
provider.stream_chat = original
|
||||
return captured
|
||||
|
||||
coach_calls = capture_two_turns("coach", "persona-coach")
|
||||
tutor_calls = capture_two_turns("tutor", "persona-tutor")
|
||||
|
||||
coach_replay_system = coach_calls[1][0]
|
||||
tutor_replay_system = tutor_calls[1][0]
|
||||
assert coach_replay_system.role == "system"
|
||||
assert tutor_replay_system.role == "system"
|
||||
assert "Coach" in coach_replay_system.content
|
||||
assert "Tutor" in tutor_replay_system.content
|
||||
assert coach_replay_system.content != tutor_replay_system.content
|
||||
# persona is stable across turns within one session
|
||||
assert coach_calls[0][0].content == coach_replay_system.content
|
||||
|
||||
|
||||
def test_session_agent_scoped(client):
|
||||
payload = {
|
||||
"agent": "tutor",
|
||||
"session_id": "sess-3",
|
||||
"messages": [{"role": "user", "content": "hello"}],
|
||||
}
|
||||
stream_events(client, payload)
|
||||
import asyncio
|
||||
|
||||
store = client.app.state.session_store
|
||||
|
||||
async def check():
|
||||
return await store.get("sess-3")
|
||||
|
||||
session = asyncio.run(check())
|
||||
assert session.agent == "tutor"
|
||||
|
||||
|
||||
def test_client_retry_does_not_duplicate_user_turn(client):
|
||||
"""P1 fix (final review): resending the same user turn after a failure
|
||||
must not double-append it to session history."""
|
||||
import asyncio
|
||||
|
||||
payload = {
|
||||
"agent": "coach",
|
||||
"session_id": "sess-retry",
|
||||
"messages": [{"role": "user", "content": "same question"}],
|
||||
}
|
||||
# First attempt fails mid-stream (turn was already appended before streaming)
|
||||
provider = client.app.state.provider
|
||||
provider.fail_mid_stream_at_index = 0
|
||||
stream_events_retry(client, payload)
|
||||
provider.fail_mid_stream_at_index = None
|
||||
# Client retry: identical payload
|
||||
stream_events_retry(client, payload)
|
||||
|
||||
store = client.app.state.session_store
|
||||
|
||||
async def check():
|
||||
return await store.history_window("sess-retry")
|
||||
|
||||
contents = [m.content for m in asyncio.run(check())]
|
||||
user_turns = [c for c in contents if c == "same question"]
|
||||
assert len(user_turns) == 1, f"expected exactly 1 stored user turn, got {len(user_turns)}"
|
||||
|
||||
|
||||
def stream_events_retry(client, payload):
|
||||
with client.stream("POST", "/v1/chat/stream", json=payload) as response:
|
||||
for _ in response.iter_lines():
|
||||
pass
|
||||
@@ -0,0 +1,131 @@
|
||||
"""SSE chat stream endpoint tests — envelope ordering, errors, headers."""
|
||||
|
||||
import json
|
||||
|
||||
|
||||
def stream_lines(client, payload: dict) -> list[str]:
|
||||
with client.stream("POST", "/v1/chat/stream", json=payload) as response:
|
||||
assert response.status_code == 200
|
||||
assert "no-cache" in response.headers.get("cache-control", "")
|
||||
return [line for line in response.iter_lines() if line.strip()]
|
||||
|
||||
|
||||
def parse_events(raw_lines: list[str]) -> list[dict]:
|
||||
"""Parse SSE lines into event dicts; strips event:/data: prefixes."""
|
||||
events = []
|
||||
for line in raw_lines:
|
||||
if line.startswith("data:"):
|
||||
data = line.removeprefix("data:").strip()
|
||||
if data == "[DONE]":
|
||||
events.append({"type": "[DONE]"})
|
||||
else:
|
||||
events.append(json.loads(data))
|
||||
return events
|
||||
|
||||
|
||||
PAYLOAD = {
|
||||
"agent": "tutor",
|
||||
"session_id": "s1",
|
||||
"messages": [{"role": "user", "content": "hello"}],
|
||||
}
|
||||
|
||||
|
||||
def test_meta_first_then_deltas_done_done_sentinel(client):
|
||||
events = parse_events(stream_lines(client, PAYLOAD))
|
||||
assert events[0]["type"] == "meta"
|
||||
assert events[0]["agent"] == "tutor"
|
||||
assert events[0]["session_id"] == "s1"
|
||||
assert events[-1]["type"] == "[DONE]"
|
||||
middle = events[1:-1]
|
||||
deltas = [e for e in middle if e["type"] == "delta"]
|
||||
dones = [e for e in middle if e["type"] == "done"]
|
||||
assert len(deltas) >= 1
|
||||
assert len(dones) == 1
|
||||
assert dones[0]["finish_reason"] == "stop"
|
||||
# delta events after meta, done before [DONE]
|
||||
assert events.index(dones[0]) > events.index(deltas[0])
|
||||
|
||||
|
||||
def test_empty_messages_rejected(client):
|
||||
response = client.post(
|
||||
"/v1/chat/stream",
|
||||
json={"agent": "tutor", "session_id": "s", "messages": []},
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
|
||||
def test_mid_stream_failure_yields_error_then_done(client):
|
||||
provider = client.app.state.provider
|
||||
provider.fail_mid_stream_at_index = 1
|
||||
events = parse_events(stream_lines(client, PAYLOAD))
|
||||
provider.fail_mid_stream_at_index = None
|
||||
error_events = [e for e in events if e["type"] == "error"]
|
||||
assert len(error_events) == 1
|
||||
assert error_events[0]["code"] == "provider_error"
|
||||
assert events[-1]["type"] == "[DONE]"
|
||||
# error must precede the terminal sentinel
|
||||
assert events.index(error_events[0]) < events.index(events[-1])
|
||||
|
||||
|
||||
def test_pre_first_byte_failure_yields_provider_unavailable(client):
|
||||
provider = client.app.state.provider
|
||||
provider.fail_before_first_token = True
|
||||
events = parse_events(stream_lines(client, PAYLOAD))
|
||||
provider.fail_before_first_token = False
|
||||
error_events = [e for e in events if e["type"] == "error"]
|
||||
assert len(error_events) == 1
|
||||
assert error_events[0]["code"] == "provider_unavailable"
|
||||
|
||||
|
||||
def test_unknown_agent_rejected_422(client):
|
||||
response = client.post(
|
||||
"/v1/chat/stream",
|
||||
json={
|
||||
"agent": "oracle",
|
||||
"session_id": "s",
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 422
|
||||
assert "oracle" in response.json()["detail"]
|
||||
|
||||
|
||||
def test_coach_routes_and_meta_names_agent(client):
|
||||
payload = {
|
||||
"agent": "coach",
|
||||
"session_id": "route-coach",
|
||||
"messages": [{"role": "user", "content": "pace me"}],
|
||||
}
|
||||
events = parse_events(stream_lines(client, payload))
|
||||
assert events[0]["type"] == "meta"
|
||||
assert events[0]["agent"] == "coach"
|
||||
assert events[-1]["type"] == "[DONE]"
|
||||
|
||||
|
||||
def test_tutor_routes_and_meta_names_agent(client):
|
||||
payload = {
|
||||
"agent": "tutor",
|
||||
"session_id": "route-tutor",
|
||||
"messages": [{"role": "user", "content": "teach me"}],
|
||||
}
|
||||
events = parse_events(stream_lines(client, payload))
|
||||
assert events[0]["type"] == "meta"
|
||||
assert events[0]["agent"] == "tutor"
|
||||
assert events[-1]["type"] == "[DONE]"
|
||||
|
||||
|
||||
def test_coach_and_tutor_streams_are_distinct(client):
|
||||
"""Agent routing selects the right agent: distinct system prompts →
|
||||
distinct hash-seeded mock outputs for the same user input."""
|
||||
same_message = [{"role": "user", "content": "same question"}]
|
||||
coach = parse_events(stream_lines(client, {
|
||||
"agent": "coach", "session_id": "d1", "messages": same_message,
|
||||
}))
|
||||
tutor = parse_events(stream_lines(client, {
|
||||
"agent": "tutor", "session_id": "d2", "messages": same_message,
|
||||
}))
|
||||
coach_text = "".join(e["content"] for e in coach if e["type"] == "delta")
|
||||
tutor_text = "".join(e["content"] for e in tutor if e["type"] == "delta")
|
||||
assert coach_text and tutor_text
|
||||
assert coach_text != tutor_text
|
||||
assert coach[-1]["type"] == "[DONE]"
|
||||
@@ -0,0 +1,47 @@
|
||||
"""Lab feedback endpoint tests — SSE envelope with agent=lab (REQ-2-007)."""
|
||||
|
||||
import json
|
||||
|
||||
|
||||
def stream_events(client, payload) -> list[dict]:
|
||||
with client.stream("POST", "/v1/lab/feedback", json=payload) as response:
|
||||
assert response.status_code == 200
|
||||
events = []
|
||||
for line in response.iter_lines():
|
||||
if line.startswith("data:"):
|
||||
d = line.removeprefix("data:").strip()
|
||||
if d == "[DONE]":
|
||||
events.append({"type": "[DONE]"})
|
||||
else:
|
||||
events.append(json.loads(d))
|
||||
return events
|
||||
|
||||
|
||||
def test_lab_feedback_streams_full_envelope(client):
|
||||
events = stream_events(client, {"scenario_id": "lab-scenario-strong"})
|
||||
assert events[0]["type"] == "meta"
|
||||
assert events[0]["agent"] == "lab"
|
||||
assert events[0]["scenario_id"] == "lab-scenario-strong"
|
||||
deltas = [e for e in events if e["type"] == "delta"]
|
||||
assert len(deltas) >= 1
|
||||
assert any(e["type"] == "done" for e in events)
|
||||
assert events[-1]["type"] == "[DONE]"
|
||||
|
||||
|
||||
def test_unknown_scenario_404(client):
|
||||
response = client.post("/v1/lab/feedback", json={"scenario_id": "nope"})
|
||||
assert response.status_code == 404
|
||||
assert "nope" in response.json()["detail"]
|
||||
|
||||
|
||||
def test_distinct_scenarios_distinct_replies(client):
|
||||
strong = stream_events(client, {"scenario_id": "lab-scenario-strong"})
|
||||
struggling = stream_events(client, {"scenario_id": "lab-scenario-struggling"})
|
||||
strong_text = "".join(e["content"] for e in strong if e["type"] == "delta")
|
||||
struggling_text = "".join(e["content"] for e in struggling if e["type"] == "delta")
|
||||
assert strong_text != struggling_text
|
||||
|
||||
|
||||
def test_missing_scenario_id_422(client):
|
||||
response = client.post("/v1/lab/feedback", json={})
|
||||
assert response.status_code == 422
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Mentor narrative endpoint tests — SSE, session-backed (REQ-2-010)."""
|
||||
|
||||
import json
|
||||
|
||||
|
||||
def stream_events(client, payload) -> list[dict]:
|
||||
with client.stream("POST", "/v1/mentor/narrative", json=payload) as response:
|
||||
assert response.status_code == 200
|
||||
events = []
|
||||
for line in response.iter_lines():
|
||||
if line.startswith("data:"):
|
||||
d = line.removeprefix("data:").strip()
|
||||
if d == "[DONE]":
|
||||
events.append({"type": "[DONE]"})
|
||||
else:
|
||||
events.append(json.loads(d))
|
||||
return events
|
||||
|
||||
|
||||
def test_narrative_streams_full_envelope(client):
|
||||
events = stream_events(client, {"session_id": "mentor-1"})
|
||||
assert events[0]["type"] == "meta"
|
||||
assert events[0]["agent"] == "mentor"
|
||||
deltas = [e for e in events if e["type"] == "delta"]
|
||||
assert len(deltas) >= 1
|
||||
assert any(e["type"] == "done" for e in events)
|
||||
assert events[-1]["type"] == "[DONE]"
|
||||
|
||||
|
||||
def test_narrative_is_session_backed(client):
|
||||
"""Second call replays history: provider input grows; distinct mock output."""
|
||||
first = stream_events(client, {"session_id": "mentor-2", "prompt": "narrate my path"})
|
||||
second = stream_events(client, {"session_id": "mentor-2", "prompt": "what next?"})
|
||||
first_text = "".join(e["content"] for e in first if e["type"] == "delta")
|
||||
second_text = "".join(e["content"] for e in second if e["type"] == "delta")
|
||||
assert first_text != second_text
|
||||
|
||||
|
||||
def test_narrative_persists_turns(client):
|
||||
import asyncio
|
||||
|
||||
store = client.app.state.session_store
|
||||
|
||||
stream_events(client, {"session_id": "mentor-3", "prompt": "hello trajectory"})
|
||||
|
||||
async def check():
|
||||
return await store.history_window("mentor-3")
|
||||
|
||||
contents = [m.content for m in asyncio.run(check())]
|
||||
assert "hello trajectory" in contents
|
||||
assert len(contents) >= 2 # user + assistant persisted
|
||||
|
||||
|
||||
def test_missing_session_id_422(client):
|
||||
response = client.post("/v1/mentor/narrative", json={"prompt": "hi"})
|
||||
assert response.status_code == 422
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Proctor signals endpoint tests — validated JSON, 404s (REQ-2-009)."""
|
||||
|
||||
from ai_service.agents.proctor import ProctorAssessment
|
||||
from ai_service.llm.mock import ScriptedJSONProvider
|
||||
|
||||
VALID = {
|
||||
"scenario_id": "proctor-scenario-distracted",
|
||||
"signals": [
|
||||
{"signal_type": "context_switch", "severity": "low",
|
||||
"note": "Docs tab at t+120s is normal"},
|
||||
{"signal_type": "idle_gap", "severity": "medium",
|
||||
"note": "5-minute idle at t+300s"},
|
||||
],
|
||||
"intervention": "Offer a short break, then restate the plan",
|
||||
"summary": "Coaching-shaped session note",
|
||||
}
|
||||
|
||||
|
||||
def test_signals_returns_validated_json(client):
|
||||
original = client.app.state.provider
|
||||
client.app.state.provider = ScriptedJSONProvider(VALID)
|
||||
try:
|
||||
response = client.post(
|
||||
"/v1/proctor/signals", json={"scenario_id": "proctor-scenario-distracted"}
|
||||
)
|
||||
finally:
|
||||
client.app.state.provider = original
|
||||
assert response.status_code == 200
|
||||
validated = ProctorAssessment.model_validate(response.json())
|
||||
assert validated.scenario_id == "proctor-scenario-distracted"
|
||||
assert validated.intervention
|
||||
|
||||
|
||||
def test_unknown_scenario_404(client):
|
||||
response = client.post("/v1/proctor/signals", json={"scenario_id": "ghost"})
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
def test_unparseable_provider_502(client):
|
||||
response = client.post(
|
||||
"/v1/proctor/signals", json={"scenario_id": "proctor-scenario-healthy"}
|
||||
)
|
||||
assert response.status_code == 502
|
||||
assert "failed" in response.json()["detail"].lower()
|
||||
|
||||
|
||||
def test_missing_scenario_id_422(client):
|
||||
response = client.post("/v1/proctor/signals", json={})
|
||||
assert response.status_code == 422
|
||||
@@ -0,0 +1,93 @@
|
||||
"""Client-disconnect regression tests — SSE generators must tolerate aclose().
|
||||
|
||||
A `yield` inside `finally` re-raises "async generator ignored GeneratorExit"
|
||||
when sse-starlette closes the iterator on client disconnect (P0 finding,
|
||||
final review). These tests reproduce the close path directly against each
|
||||
endpoint's event_stream generator shape.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from ai_service.corpus.learner_context import get_learner_context
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
|
||||
async def _chat_event_stream(client, session_id="close-chat", content="hi"):
|
||||
"""Rebuild the chat endpoint's event_stream generator exactly as
|
||||
chat.py builds it (same code shape, same session flow)."""
|
||||
app = client.app
|
||||
settings = app.state.settings
|
||||
provider = app.state.provider
|
||||
registry = app.state.agent_registry
|
||||
sessions = app.state.session_store
|
||||
agent = registry.get(provider, settings, "tutor")
|
||||
learner_context = get_learner_context(None)
|
||||
|
||||
if await sessions.get(session_id) is None:
|
||||
await sessions.create(session_id, agent="tutor", learner_id="learner-001")
|
||||
user_turn = Message(role="user", content=content)
|
||||
history = await sessions.history_window(session_id)
|
||||
await sessions.append(session_id, user_turn)
|
||||
|
||||
async def event_stream():
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "meta", "agent": "tutor", "session_id": session_id,
|
||||
"model": settings.model,
|
||||
})}
|
||||
first_byte = True
|
||||
reply_parts: list[str] = []
|
||||
try:
|
||||
async for token in agent.stream_reply(
|
||||
history=history, user_input=user_turn.content,
|
||||
learner_context=learner_context,
|
||||
):
|
||||
first_byte = False
|
||||
reply_parts.append(token)
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "delta", "content": token
|
||||
})}
|
||||
full_reply = "".join(reply_parts)
|
||||
if full_reply:
|
||||
await sessions.append(
|
||||
session_id, Message(role="assistant", content=full_reply)
|
||||
)
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "done", "finish_reason": "stop"
|
||||
})}
|
||||
yield {"event": "message", "data": "[DONE]"}
|
||||
except Exception as exc:
|
||||
code = "provider_unavailable" if first_byte else "provider_error"
|
||||
yield {"event": "message", "data": json.dumps({
|
||||
"type": "error", "code": code, "message": str(exc)
|
||||
})}
|
||||
yield {"event": "message", "data": "[DONE]"}
|
||||
|
||||
return event_stream()
|
||||
|
||||
|
||||
async def test_chat_stream_generator_survives_aclose(client):
|
||||
"""Partial consumption then aclose() must not raise
|
||||
'async generator ignored GeneratorExit' (yield-in-finally regression)."""
|
||||
gen = await _chat_event_stream(client, session_id="close-chat")
|
||||
meta = await gen.__anext__()
|
||||
assert json.loads(meta["data"])["type"] == "meta"
|
||||
delta = await gen.__anext__()
|
||||
assert json.loads(delta["data"])["type"] == "delta"
|
||||
# The critical assertion: closing mid-stream must be clean (no raise).
|
||||
await gen.aclose()
|
||||
|
||||
|
||||
async def test_chat_stream_survives_close_at_different_points(client):
|
||||
"""Close right after meta, and right after done — all must be clean."""
|
||||
gen = await _chat_event_stream(client, session_id="close-early")
|
||||
await gen.__anext__() # meta only
|
||||
await gen.aclose()
|
||||
|
||||
gen2 = await _chat_event_stream(client, session_id="close-late")
|
||||
events = []
|
||||
async for ev in gen2:
|
||||
events.append(json.loads(ev["data"]))
|
||||
if len(events) == 2:
|
||||
break
|
||||
await gen2.aclose()
|
||||
assert events[0]["type"] == "meta"
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Test suite — conftest: mock provider only, zero network (enforced)."""
|
||||
|
||||
import os
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from ai_service.config import Settings
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.main import create_app
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def settings() -> Settings:
|
||||
os.environ["AI_PROVIDER"] = "mock"
|
||||
return Settings(provider="mock", model="gemma4:31b", port=8421)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def app(settings: Settings):
|
||||
return create_app(settings)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client(app):
|
||||
with TestClient(app) as c:
|
||||
# Mechanical cloud-free guard (GRILL advisory a): the app under test
|
||||
# MUST be wired to the deterministic mock provider.
|
||||
assert isinstance(app.state.provider, MockProvider), (
|
||||
f"tests must run against MockProvider, got {type(app.state.provider).__name__}"
|
||||
)
|
||||
yield c
|
||||
@@ -0,0 +1,73 @@
|
||||
"""MockProvider tests — determinism, JSON mode, failure modes."""
|
||||
|
||||
import pytest
|
||||
|
||||
from ai_service.llm.mock import MockProvider
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
MSGS = [Message(role="user", content="hello there")]
|
||||
|
||||
|
||||
async def test_stream_is_deterministic():
|
||||
p1, p2 = MockProvider(), MockProvider()
|
||||
out1 = [t async for t in p1.stream_chat(MSGS, model="m")]
|
||||
out2 = [t async for t in p2.stream_chat(MSGS, model="m")]
|
||||
assert "".join(out1) == "".join(out2)
|
||||
assert out1 == out2
|
||||
|
||||
|
||||
async def test_stream_content_differs_for_different_input():
|
||||
p = MockProvider()
|
||||
a = "".join([t async for t in p.stream_chat(MSGS, model="m")])
|
||||
b = "".join(
|
||||
[t async for t in p.stream_chat([Message(role="user", content="other")], model="m")]
|
||||
)
|
||||
assert a != b
|
||||
|
||||
|
||||
async def test_json_object_response_format():
|
||||
import json
|
||||
|
||||
p = MockProvider()
|
||||
out = "".join(
|
||||
[
|
||||
t
|
||||
async for t in p.stream_chat(
|
||||
MSGS, model="m", response_format={"type": "json_object"}
|
||||
)
|
||||
]
|
||||
)
|
||||
assert json.loads(out) == {"summary": "mock structured reply", "confidence": 0.87}
|
||||
|
||||
|
||||
async def test_fail_before_first_token():
|
||||
p = MockProvider()
|
||||
p.fail_before_first_token = True
|
||||
with pytest.raises(RuntimeError):
|
||||
async for _ in p.stream_chat(MSGS, model="m"):
|
||||
pass
|
||||
|
||||
|
||||
async def test_fail_mid_stream():
|
||||
p = MockProvider()
|
||||
p.fail_mid_stream_at_index = 2
|
||||
tokens = []
|
||||
with pytest.raises(RuntimeError):
|
||||
async for t in p.stream_chat(MSGS, model="m"):
|
||||
tokens.append(t)
|
||||
assert len(tokens) == 2
|
||||
|
||||
|
||||
async def test_cancellation_records_abort():
|
||||
p = MockProvider()
|
||||
gen = p.stream_chat(MSGS, model="m")
|
||||
await gen.__anext__()
|
||||
await gen.aclose()
|
||||
assert p.abort_recorded is True
|
||||
|
||||
|
||||
async def test_chat_returns_full_reply():
|
||||
p = MockProvider()
|
||||
reply = await p.chat(MSGS, model="m")
|
||||
streamed = "".join([t async for t in p.stream_chat(MSGS, model="m")])
|
||||
assert reply == streamed
|
||||
@@ -0,0 +1,177 @@
|
||||
"""OpenAICompatProvider tests — byte-exact SSE parsing via httpx.MockTransport.
|
||||
|
||||
Covers: multi-delta happy path, keep-alive comment lines, [DONE] sentinel,
|
||||
malformed line tolerance, missing optional fields, non-streaming chat(),
|
||||
response_format auto-degrade on 400, api_key never leaking into exceptions.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from ai_service.llm.openai_compat import OpenAICompatProvider
|
||||
from ai_service.llm.types import Message
|
||||
|
||||
MSGS = [Message(role="user", content="hi")]
|
||||
KEY = "sk-test-abc123"
|
||||
|
||||
|
||||
def make_client(handler) -> httpx.AsyncClient:
|
||||
transport = httpx.MockTransport(handler)
|
||||
return httpx.AsyncClient(transport=transport)
|
||||
|
||||
|
||||
def sse_body(deltas: list[str], with_comments: bool = True) -> bytes:
|
||||
lines = []
|
||||
if with_comments:
|
||||
lines.append(": ping")
|
||||
for d in deltas:
|
||||
lines.append("data: " + json.dumps({
|
||||
"id": "chatcmpl-1", "object": "chat.completion.chunk",
|
||||
"created": 1, "model": "gemma4:31b",
|
||||
"choices": [{"index": 0, "delta": {"content": d}, "finish_reason": None}],
|
||||
}))
|
||||
if with_comments:
|
||||
lines.append(": ping")
|
||||
lines.append("data: [DONE]")
|
||||
return ("\n".join(lines) + "\n").encode()
|
||||
|
||||
|
||||
async def test_stream_happy_path_with_comments_and_done():
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
assert request.url.path == "/v1/chat/completions"
|
||||
return httpx.Response(200, content=sse_body(["Hel", "lo", " world"]))
|
||||
|
||||
async with make_client(handler) as client:
|
||||
provider = OpenAICompatProvider(client, "https://fake/v1")
|
||||
tokens = [t async for t in provider.stream_chat(MSGS, model="gemma4:31b")]
|
||||
assert tokens == ["Hel", "lo", " world"]
|
||||
|
||||
|
||||
async def test_stream_tolerates_malformed_lines():
|
||||
body = (
|
||||
"data: not-json\n"
|
||||
"data: "
|
||||
+ json.dumps({
|
||||
"id": "x", "object": "chat.completion.chunk", "created": 1, "model": "m",
|
||||
"choices": [{"index": 0, "delta": {"content": "ok"}, "finish_reason": None}],
|
||||
})
|
||||
+ "\ndata: [DONE]\n"
|
||||
)
|
||||
|
||||
def handler(request):
|
||||
return httpx.Response(200, content=body.encode())
|
||||
|
||||
async with make_client(handler) as client:
|
||||
provider = OpenAICompatProvider(client, "https://fake/v1")
|
||||
tokens = [t async for t in provider.stream_chat(MSGS, model="m")]
|
||||
assert tokens == ["ok"]
|
||||
|
||||
|
||||
async def test_stream_skips_empty_content_and_empty_choices():
|
||||
chunk_empty_delta = json.dumps(
|
||||
{"id": "x", "model": "m",
|
||||
"choices": [{"index": 0, "delta": {}, "finish_reason": None}]}
|
||||
)
|
||||
chunk_empty_choices = json.dumps({"id": "x", "model": "m", "choices": []})
|
||||
chunk_yes = json.dumps(
|
||||
{"id": "x", "model": "m",
|
||||
"choices": [{"index": 0, "delta": {"content": "yes"}, "finish_reason": None}]}
|
||||
)
|
||||
body = (
|
||||
f"data: {chunk_empty_delta}\n"
|
||||
f"data: {chunk_empty_choices}\n"
|
||||
f"data: {chunk_yes}\n"
|
||||
"data: [DONE]\n"
|
||||
)
|
||||
|
||||
def handler(request):
|
||||
return httpx.Response(200, content=body.encode())
|
||||
|
||||
async with make_client(handler) as client:
|
||||
provider = OpenAICompatProvider(client, "https://fake/v1")
|
||||
tokens = [t async for t in provider.stream_chat(MSGS, model="m")]
|
||||
assert tokens == ["yes"]
|
||||
|
||||
|
||||
async def test_chat_non_streaming():
|
||||
def handler(request):
|
||||
payload = json.loads(request.content)
|
||||
assert payload["stream"] is False
|
||||
body = {"id": "1", "object": "chat.completion", "created": 1, "model": "m",
|
||||
"choices": [{"index": 0, "message": {"role": "assistant",
|
||||
"content": "full reply"},
|
||||
"finish_reason": "stop"}]}
|
||||
return httpx.Response(200, json=body)
|
||||
|
||||
async with make_client(handler) as client:
|
||||
provider = OpenAICompatProvider(client, "https://fake/v1")
|
||||
assert await provider.chat(MSGS, model="m") == "full reply"
|
||||
|
||||
|
||||
async def test_response_format_auto_degrades_on_400():
|
||||
calls = []
|
||||
|
||||
def handler(request):
|
||||
payload = json.loads(request.content)
|
||||
calls.append(payload)
|
||||
if "response_format" in payload:
|
||||
return httpx.Response(400, json={"error": "response_format unsupported"})
|
||||
body = {"id": "1", "object": "chat.completion", "created": 1, "model": "m",
|
||||
"choices": [{"index": 0, "message": {"role": "assistant",
|
||||
"content": "json"},
|
||||
"finish_reason": "stop"}]}
|
||||
return httpx.Response(200, json=body)
|
||||
|
||||
async with make_client(handler) as client:
|
||||
provider = OpenAICompatProvider(client, "https://fake/v1", json_mode="auto")
|
||||
reply = await provider.chat(MSGS, model="m", response_format={"type": "json_object"})
|
||||
assert reply == "json"
|
||||
assert len(calls) == 2
|
||||
assert "response_format" in calls[0]
|
||||
assert "response_format" not in calls[1]
|
||||
|
||||
|
||||
async def test_response_format_off_never_sends():
|
||||
calls = []
|
||||
|
||||
def handler(request):
|
||||
payload = json.loads(request.content)
|
||||
calls.append(payload)
|
||||
body = {"id": "1", "object": "chat.completion", "created": 1, "model": "m",
|
||||
"choices": [{"index": 0, "message": {"role": "assistant",
|
||||
"content": "x"},
|
||||
"finish_reason": "stop"}]}
|
||||
return httpx.Response(200, json=body)
|
||||
|
||||
async with make_client(handler) as client:
|
||||
provider = OpenAICompatProvider(client, "https://fake/v1", json_mode="off")
|
||||
await provider.chat(MSGS, model="m", response_format={"type": "json_object"})
|
||||
assert len(calls) == 1
|
||||
assert "response_format" not in calls[0]
|
||||
|
||||
|
||||
async def test_api_key_never_in_exception():
|
||||
def handler(request):
|
||||
raise httpx.ConnectError("connection refused while using sk-test-abc123")
|
||||
|
||||
async with make_client(handler) as client:
|
||||
provider = OpenAICompatProvider(client, "https://fake/v1", api_key=KEY)
|
||||
with pytest.raises(RuntimeError) as exc_info:
|
||||
async for _ in provider.stream_chat(MSGS, model="m"):
|
||||
pass
|
||||
assert "sk-test-abc123" not in str(exc_info.value)
|
||||
|
||||
|
||||
async def test_bearer_header_sent():
|
||||
seen = {}
|
||||
|
||||
def handler(request):
|
||||
seen["auth"] = request.headers.get("Authorization")
|
||||
return httpx.Response(200, content=sse_body(["x"]))
|
||||
|
||||
async with make_client(handler) as client:
|
||||
provider = OpenAICompatProvider(client, "https://fake/v1", api_key=KEY)
|
||||
_ = [t async for t in provider.stream_chat(MSGS, model="m")]
|
||||
assert seen["auth"] == f"Bearer {KEY}"
|
||||
@@ -0,0 +1,10 @@
|
||||
"""Health endpoint tests."""
|
||||
|
||||
|
||||
def test_health_returns_ok(client):
|
||||
response = client.get("/health")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "ok"
|
||||
assert data["provider"] == "mock"
|
||||
assert data["model"] == "gemma4:31b"
|
||||
@@ -0,0 +1,2 @@
|
||||
# AI service (v0.2) — learner chat/panels stream from this FastAPI service
|
||||
NEXT_PUBLIC_AI_SERVICE_URL=http://localhost:8420
|
||||
@@ -11,7 +11,8 @@ import {
|
||||
Activity,
|
||||
} from 'lucide-react';
|
||||
import { Button } from '@nextcraft/ui';
|
||||
import { allCompetencies, competencyStacks } from '@nextcraft/mock-data';
|
||||
import { allCompetencies, competencyStacks, aiLabScenarios } from '@nextcraft/mock-data';
|
||||
import { LabFeedbackPanel } from '../../../../components/learner/lab-feedback-panel';
|
||||
|
||||
interface FileEntry {
|
||||
label: string;
|
||||
@@ -279,6 +280,10 @@ export default async function BuildSandboxPage({
|
||||
))}
|
||||
</ul>
|
||||
</div>
|
||||
{/* Lab in-flow feedback — mock telemetry scenario (real engine v0.3+) */}
|
||||
<div className="border-t border-slate-200 pt-3 dark:border-slate-800">
|
||||
<LabFeedbackPanel scenarioId={aiLabScenarios[0].id} />
|
||||
</div>
|
||||
</aside>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@ import {
|
||||
learnerMicrocredentials,
|
||||
} from '@nextcraft/mock-data';
|
||||
import { AiTutorChat } from '../../../components/learner/ai-tutor-chat';
|
||||
import { MentorPanel } from '../../../components/learner/mentor-panel';
|
||||
import { ProgressGraph } from '../../../components/learner/progress-graph';
|
||||
|
||||
const ACTIVE_COMPETENCY_IDS = [
|
||||
@@ -265,7 +266,7 @@ export default function DashboardPage() {
|
||||
AI Tutor
|
||||
</h2>
|
||||
<p className="text-sm text-slate-500 dark:text-slate-400">
|
||||
Coach and Socratic tutor · mock responses
|
||||
Coach and Socratic tutor · live streaming
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
@@ -274,6 +275,21 @@ export default function DashboardPage() {
|
||||
<AiTutorChat />
|
||||
</CardBody>
|
||||
</Card>
|
||||
|
||||
{/* Mentor — long-horizon career narrative */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<h2 className="text-base font-semibold text-slate-900 dark:text-slate-100">
|
||||
Mentor
|
||||
</h2>
|
||||
<p className="text-sm text-slate-500 dark:text-slate-400">
|
||||
Long-horizon career trajectory · live streaming
|
||||
</p>
|
||||
</CardHeader>
|
||||
<CardBody>
|
||||
<MentorPanel />
|
||||
</CardBody>
|
||||
</Card>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -16,6 +16,9 @@ import {
|
||||
import { Card, CardBody, CardHeader, Badge, Button } from '@nextcraft/ui';
|
||||
import { allCompetencies, competencyStacks } from '@nextcraft/mock-data';
|
||||
import { OralDefenseInterface } from '../../../../components/learner/oral-defense-interface';
|
||||
import { AssessorResultsPanel } from '../../../../components/learner/assessor-results-panel';
|
||||
import { ProctorBanner } from '../../../../components/learner/proctor-banner';
|
||||
import { aiArtifactSubmissions } from '@nextcraft/mock-data';
|
||||
|
||||
const RUBRIC = [
|
||||
{ name: 'Correctness of agent architecture', passed: true, weight: 25 },
|
||||
@@ -254,10 +257,22 @@ export default async function DefensePage({
|
||||
a structured-output schema and re-run the eval harness before your oral defense.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{/* Live Assessor — structured rubric from the real agent (mock inputs) */}
|
||||
<div className="border-t border-slate-200 pt-3 dark:border-slate-800">
|
||||
<AssessorResultsPanel artifactId={aiArtifactSubmissions[0].id} />
|
||||
</div>
|
||||
</CardBody>
|
||||
</Card>
|
||||
</div>
|
||||
|
||||
{/* Proctor integrity banner — coaching-shaped (mock telemetry) */}
|
||||
<Card>
|
||||
<CardBody>
|
||||
<ProctorBanner scenarioId="proctor-scenario-distracted" />
|
||||
</CardBody>
|
||||
</Card>
|
||||
|
||||
{/* Process trace timeline */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
|
||||
@@ -4,6 +4,7 @@ import { ArrowLeft, ArrowRight, Clock } from 'lucide-react';
|
||||
import { Button, Card, CardBody, Badge } from '@nextcraft/ui';
|
||||
import { allCompetencies, competencyStacks } from '@nextcraft/mock-data';
|
||||
import { WorkedExampleTabs } from '../../../../components/learner/worked-example-tabs';
|
||||
import { ByteTutorPanel } from '../../../../components/learner/byte-tutor-panel';
|
||||
|
||||
export default async function ByteTutorialPage({
|
||||
params,
|
||||
@@ -39,6 +40,12 @@ export default async function ByteTutorialPage({
|
||||
</header>
|
||||
|
||||
<div className="grid grid-cols-1 gap-4 lg:grid-cols-2">
|
||||
{/* Tutor explanation panel — agent fixed to tutor (A-007) */}
|
||||
<Card>
|
||||
<CardBody>
|
||||
<ByteTutorPanel competencyId={competency.id} competencyName={competency.name} />
|
||||
</CardBody>
|
||||
</Card>
|
||||
{/* Concept panel */}
|
||||
<Card className="flex flex-col">
|
||||
<CardBody className="flex flex-col gap-4">
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
'use client';
|
||||
|
||||
import { useState, useEffect, useRef, useCallback } from 'react';
|
||||
import { parseSseEvents } from '../../lib/sse';
|
||||
import { Bot, RefreshCw, AlertTriangle, Loader2 } from 'lucide-react';
|
||||
|
||||
const AI_SERVICE_URL =
|
||||
process.env.NEXT_PUBLIC_AI_SERVICE_URL ?? 'http://localhost:8420';
|
||||
|
||||
interface StreamPanelProps {
|
||||
title: string;
|
||||
endpoint: string; // e.g. "/v1/lab/feedback"
|
||||
body: Record<string, unknown>;
|
||||
autoLoad?: boolean;
|
||||
emptyHint?: string;
|
||||
/** Renders structured JSON results (assessor/proctor) as custom UI */
|
||||
renderJson?: (data: Record<string, unknown>) => React.ReactNode;
|
||||
}
|
||||
|
||||
interface SseResult {
|
||||
text: string;
|
||||
error: string | null;
|
||||
streaming: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Generic SSE-consuming panel for the non-chat agent endpoints
|
||||
* (lab feedback, mentor narrative, assessor/proctor JSON).
|
||||
* Streams text panels; renders structured JSON via renderJson when set.
|
||||
*/
|
||||
export function AgentStreamPanel({
|
||||
title,
|
||||
endpoint,
|
||||
body,
|
||||
autoLoad = false,
|
||||
emptyHint,
|
||||
renderJson,
|
||||
}: StreamPanelProps) {
|
||||
const [result, setResult] = useState<SseResult>({ text: '', error: null, streaming: false });
|
||||
const [json, setJson] = useState<Record<string, unknown> | null>(null);
|
||||
const abortRef = useRef<AbortController | null>(null);
|
||||
const startedRef = useRef(false);
|
||||
|
||||
const load = useCallback(async () => {
|
||||
abortRef.current?.abort();
|
||||
const controller = new AbortController();
|
||||
abortRef.current = controller;
|
||||
setResult({ text: '', error: null, streaming: true });
|
||||
setJson(null);
|
||||
|
||||
try {
|
||||
const response = await fetch(`${AI_SERVICE_URL}${endpoint}`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(body),
|
||||
signal: controller.signal,
|
||||
});
|
||||
const contentType = response.headers.get('content-type') ?? '';
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`AI service error (${response.status})`);
|
||||
}
|
||||
if (contentType.includes('application/json')) {
|
||||
const data = (await response.json()) as Record<string, unknown>;
|
||||
setJson(data);
|
||||
setResult({ text: '', error: null, streaming: false });
|
||||
return;
|
||||
}
|
||||
if (!response.body) {
|
||||
throw new Error('AI service returned an empty stream');
|
||||
}
|
||||
|
||||
const reader = response.body.getReader();
|
||||
const decoder = new TextDecoder();
|
||||
let buffer = '';
|
||||
let text = '';
|
||||
let done = false;
|
||||
|
||||
while (!done) {
|
||||
const { value, done: readerDone } = await reader.read();
|
||||
if (readerDone) break;
|
||||
buffer += decoder.decode(value, { stream: true });
|
||||
const { events, rest } = parseSseEvents(buffer);
|
||||
buffer = rest;
|
||||
for (const raw of events) {
|
||||
if (raw === '[DONE]') {
|
||||
done = true;
|
||||
break;
|
||||
}
|
||||
try {
|
||||
const event = JSON.parse(raw);
|
||||
if (event.type === 'delta') {
|
||||
text += event.content as string;
|
||||
setResult({ text, error: null, streaming: true });
|
||||
} else if (event.type === 'error') {
|
||||
setResult({ text, error: event.message as string, streaming: false });
|
||||
done = true;
|
||||
break;
|
||||
}
|
||||
} catch {
|
||||
// ignore non-JSON frames
|
||||
}
|
||||
}
|
||||
}
|
||||
setResult((prev) => ({ ...prev, streaming: false }));
|
||||
} catch (err) {
|
||||
const aborted = err instanceof DOMException && err.name === 'AbortError';
|
||||
if (!aborted) {
|
||||
setResult({
|
||||
text: '',
|
||||
error: err instanceof Error ? err.message : 'connection failed',
|
||||
streaming: false,
|
||||
});
|
||||
} else {
|
||||
setResult((prev) => ({ ...prev, streaming: false }));
|
||||
}
|
||||
} finally {
|
||||
abortRef.current = null;
|
||||
}
|
||||
}, [endpoint, body]);
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
abortRef.current?.abort();
|
||||
abortRef.current = null;
|
||||
};
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
if (autoLoad && !startedRef.current) {
|
||||
startedRef.current = true;
|
||||
void load();
|
||||
}
|
||||
}, [autoLoad, load]);
|
||||
|
||||
return (
|
||||
<div className="flex flex-col gap-3">
|
||||
<div className="flex items-center justify-between">
|
||||
<h3 className="inline-flex items-center gap-2 text-sm font-semibold text-slate-900 dark:text-slate-100">
|
||||
<Bot className="h-4 w-4 text-primary-600 dark:text-primary-400" />
|
||||
{title}
|
||||
</h3>
|
||||
<button
|
||||
onClick={() => void load()}
|
||||
disabled={result.streaming}
|
||||
className="inline-flex items-center gap-1 rounded-md border border-slate-300 px-2 py-1 text-xs text-slate-600 transition-colors hover:border-primary-400 hover:text-primary-700 disabled:opacity-50 dark:border-slate-600 dark:text-slate-300 dark:hover:border-primary-400 dark:hover:text-primary-300"
|
||||
>
|
||||
{result.streaming ? (
|
||||
<Loader2 className="h-3 w-3 animate-spin" />
|
||||
) : (
|
||||
<RefreshCw className="h-3 w-3" />
|
||||
)}
|
||||
{result.streaming ? 'Streaming…' : json || result.text ? 'Regenerate' : 'Generate'}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{result.error && (
|
||||
<div
|
||||
role="alert"
|
||||
className="flex items-start gap-2 rounded-md border border-amber-300 bg-amber-50 px-3 py-2 text-xs text-amber-800 dark:border-amber-700 dark:bg-amber-900/30 dark:text-amber-300"
|
||||
>
|
||||
<AlertTriangle className="mt-0.5 h-3.5 w-3.5 shrink-0" />
|
||||
<div>
|
||||
<p className="font-medium">AI service unavailable</p>
|
||||
<p className="opacity-80">{result.error}</p>
|
||||
<button onClick={() => void load()} className="mt-1 font-semibold underline">
|
||||
Retry
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{!result.error && !json && !result.text && !result.streaming && (
|
||||
<p className="text-xs text-slate-500 dark:text-slate-400">
|
||||
{emptyHint ?? 'Generate to see the agent in action.'}
|
||||
</p>
|
||||
)}
|
||||
|
||||
{json && renderJson ? (
|
||||
renderJson(json)
|
||||
) : result.text ? (
|
||||
<div className="whitespace-pre-wrap rounded-md bg-slate-100 px-3 py-2 text-sm leading-relaxed text-slate-800 dark:bg-slate-800 dark:text-slate-100">
|
||||
{result.text}
|
||||
{result.streaming && (
|
||||
<span className="ml-0.5 inline-block h-4 w-1.5 animate-pulse rounded-sm bg-primary-500 align-middle" />
|
||||
)}
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -1,130 +1,110 @@
|
||||
'use client';
|
||||
|
||||
import { useState, useRef, useEffect, type FormEvent } from 'react';
|
||||
import { Bot, Send, User } from 'lucide-react';
|
||||
import { aiTutorResponses, type TutorResponse } from '@nextcraft/mock-data';
|
||||
import { Bot, Send, AlertTriangle, RotateCcw } from 'lucide-react';
|
||||
import { primaryLearner } from '@nextcraft/mock-data';
|
||||
import { Avatar } from '@nextcraft/ui';
|
||||
import { useChatStream, type AgentName, type StreamMessage } from '../../hooks/use-chat-stream';
|
||||
|
||||
interface ChatMessage {
|
||||
id: string;
|
||||
role: 'learner' | 'tutor';
|
||||
content: string;
|
||||
suggestedActions?: string[];
|
||||
}
|
||||
|
||||
const SEED_MESSAGES: ChatMessage[] = [
|
||||
{
|
||||
id: 'seed-1',
|
||||
role: 'tutor',
|
||||
content:
|
||||
"Welcome back, Alex. You're 62% through the AI Orchestration stack. What would you like to work on today?",
|
||||
suggestedActions: ['Review my pacing', 'Start Multi-Agent Communication', 'Prep for my defense'],
|
||||
},
|
||||
const AGENTS: { id: AgentName; label: string; blurb: string }[] = [
|
||||
{ id: 'coach', label: 'Coach', blurb: 'Pacing, motivation, retrieval practice' },
|
||||
{ id: 'tutor', label: 'Tutor', blurb: 'Concepts, worked examples, Socratic checks' },
|
||||
];
|
||||
|
||||
const SEED_MESSAGE: StreamMessage = {
|
||||
id: 'seed-1',
|
||||
role: 'assistant',
|
||||
content:
|
||||
"Welcome back, Alex. You're 62% through the AI Orchestration stack. What would you like to work on today?",
|
||||
suggestedActions: ['Review my pacing', 'Start Multi-Agent Communication', 'Prep for my defense'],
|
||||
};
|
||||
|
||||
export function AiTutorChat() {
|
||||
const [messages, setMessages] = useState<ChatMessage[]>(SEED_MESSAGES);
|
||||
const [agent, setAgent] = useState<AgentName>('coach');
|
||||
const [input, setInput] = useState('');
|
||||
const [isTyping, setIsTyping] = useState(false);
|
||||
const { messages, isStreaming, error, send, retry, abort } = useChatStream(agent);
|
||||
const scrollRef = useRef<HTMLDivElement>(null);
|
||||
|
||||
useEffect(() => {
|
||||
if (scrollRef.current) {
|
||||
scrollRef.current.scrollTop = scrollRef.current.scrollHeight;
|
||||
}
|
||||
}, [messages, isTyping]);
|
||||
}, [messages, isStreaming, error]);
|
||||
|
||||
function send(e: FormEvent) {
|
||||
function handleSubmit(e: FormEvent) {
|
||||
e.preventDefault();
|
||||
const text = input.trim();
|
||||
if (!text || isTyping) return;
|
||||
const learnerMsg: ChatMessage = { id: `me-${Date.now()}`, role: 'learner', content: text };
|
||||
setMessages((prev) => [...prev, learnerMsg]);
|
||||
if (!text || isStreaming) return;
|
||||
setInput('');
|
||||
setIsTyping(true);
|
||||
void send(text);
|
||||
}
|
||||
|
||||
window.setTimeout(() => {
|
||||
const pick: TutorResponse =
|
||||
aiTutorResponses[Math.floor(Math.random() * aiTutorResponses.length)];
|
||||
const tutorMsg: ChatMessage = {
|
||||
id: `tutor-${Date.now()}`,
|
||||
role: 'tutor',
|
||||
content: pick.message,
|
||||
suggestedActions: pick.suggestedActions,
|
||||
};
|
||||
setMessages((prev) => [...prev, tutorMsg]);
|
||||
setIsTyping(false);
|
||||
}, 1000);
|
||||
function switchAgent(next: AgentName) {
|
||||
if (isStreaming) abort();
|
||||
setAgent(next);
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="flex h-[28rem] flex-col">
|
||||
{/* Agent switcher (A-007: explicit routing, no autonomy) */}
|
||||
<div className="mb-3 flex items-center gap-2" role="tablist" aria-label="Choose tutor agent">
|
||||
{AGENTS.map((a) => (
|
||||
<button
|
||||
key={a.id}
|
||||
role="tab"
|
||||
aria-selected={agent === a.id}
|
||||
onClick={() => switchAgent(a.id)}
|
||||
className={`rounded-full px-3 py-1 text-xs font-medium transition-colors ${
|
||||
agent === a.id
|
||||
? 'bg-primary-600 text-white'
|
||||
: 'border border-slate-300 text-slate-600 hover:border-primary-400 hover:text-primary-700 dark:border-slate-600 dark:text-slate-300 dark:hover:text-primary-300'
|
||||
}`}
|
||||
>
|
||||
{a.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
{/* Message list */}
|
||||
<div ref={scrollRef} className="flex-1 space-y-4 overflow-y-auto pr-2">
|
||||
<ChatMessage message={SEED_MESSAGE} />
|
||||
|
||||
{messages.map((m) => (
|
||||
<div
|
||||
key={m.id}
|
||||
className={`flex gap-3 ${m.role === 'learner' ? 'flex-row-reverse' : 'flex-row'}`}
|
||||
>
|
||||
{m.role === 'tutor' ? (
|
||||
<span className="flex h-8 w-8 shrink-0 items-center justify-center rounded-full bg-primary-100 text-primary-700 dark:bg-primary-900/40 dark:text-primary-300">
|
||||
<Bot className="h-4 w-4" />
|
||||
</span>
|
||||
) : (
|
||||
<Avatar name={primaryLearner.name} src={primaryLearner.avatar} size="sm" />
|
||||
)}
|
||||
<div
|
||||
className={`max-w-[80%] rounded-lg px-3 py-2 text-sm ${
|
||||
m.role === 'tutor'
|
||||
? 'bg-slate-100 text-slate-800 dark:bg-slate-800 dark:text-slate-100'
|
||||
: 'bg-primary-600 text-white'
|
||||
}`}
|
||||
>
|
||||
<p className="leading-relaxed">{m.content}</p>
|
||||
{m.suggestedActions && m.suggestedActions.length > 0 && (
|
||||
<div className="mt-2 flex flex-wrap gap-1.5">
|
||||
{m.suggestedActions.map((action) => (
|
||||
<button
|
||||
key={action}
|
||||
onClick={() => setInput(action)}
|
||||
className="rounded-full border border-slate-300 bg-white px-2 py-0.5 text-xs text-slate-600 transition-colors hover:border-primary-400 hover:text-primary-700 dark:border-slate-600 dark:bg-slate-900 dark:text-slate-300 dark:hover:border-primary-400 dark:hover:text-primary-300"
|
||||
>
|
||||
{action}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
<ChatMessage key={m.id} message={m} onAction={(a) => setInput(a)} />
|
||||
))}
|
||||
|
||||
{/* Typing indicator */}
|
||||
{isTyping && (
|
||||
<div className="flex flex-row gap-3">
|
||||
<span className="flex h-8 w-8 shrink-0 items-center justify-center rounded-full bg-primary-100 text-primary-700 dark:bg-primary-900/40 dark:text-primary-300">
|
||||
<Bot className="h-4 w-4" />
|
||||
{/* Error state with retry (A-010) */}
|
||||
{error && (
|
||||
<div
|
||||
role="alert"
|
||||
className="flex items-center gap-2 rounded-lg border border-amber-300 bg-amber-50 px-3 py-2 text-xs text-amber-800 dark:border-amber-700 dark:bg-amber-900/30 dark:text-amber-300"
|
||||
>
|
||||
<AlertTriangle className="h-4 w-4 shrink-0" />
|
||||
<span className="flex-1">
|
||||
The tutor service is unreachable. Your message can be retried.
|
||||
</span>
|
||||
<div className="flex items-center gap-1 rounded-lg bg-slate-100 px-3 py-3 dark:bg-slate-800">
|
||||
<span className="h-1.5 w-1.5 animate-bounce rounded-full bg-slate-400 [animation-delay:-0.3s]" />
|
||||
<span className="h-1.5 w-1.5 animate-bounce rounded-full bg-slate-400 [animation-delay:-0.15s]" />
|
||||
<span className="h-1.5 w-1.5 animate-bounce rounded-full bg-slate-400" />
|
||||
</div>
|
||||
<button
|
||||
onClick={retry}
|
||||
className="inline-flex items-center gap-1 font-semibold underline"
|
||||
>
|
||||
<RotateCcw className="h-3 w-3" /> Retry
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Input */}
|
||||
<form onSubmit={send} className="mt-3 flex items-center gap-2 border-t border-slate-200 pt-3 dark:border-slate-800">
|
||||
<form onSubmit={handleSubmit} className="mt-3 flex items-center gap-2 border-t border-slate-200 pt-3 dark:border-slate-800">
|
||||
<input
|
||||
value={input}
|
||||
onChange={(e) => setInput(e.target.value)}
|
||||
placeholder="Ask your AI tutor anything…"
|
||||
placeholder={`Ask your ${agent} anything…`}
|
||||
aria-label="Message"
|
||||
className="h-10 flex-1 rounded-md border border-slate-300 bg-white px-3 text-sm text-slate-900 placeholder:text-slate-400 focus:border-primary-500 focus:ring-2 focus:ring-primary-500/30 focus:outline-none dark:border-slate-700 dark:bg-slate-900 dark:text-slate-100 dark:placeholder:text-slate-500"
|
||||
/>
|
||||
<button
|
||||
type="submit"
|
||||
disabled={!input.trim() || isTyping}
|
||||
disabled={!input.trim() || isStreaming}
|
||||
className="inline-flex h-10 w-10 items-center justify-center rounded-md bg-primary-600 text-white transition-colors hover:bg-primary-700 disabled:opacity-50"
|
||||
aria-label="Send message"
|
||||
>
|
||||
@@ -133,4 +113,52 @@ export function AiTutorChat() {
|
||||
</form>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function ChatMessage({
|
||||
message,
|
||||
onAction,
|
||||
}: {
|
||||
message: StreamMessage;
|
||||
onAction?: (text: string) => void;
|
||||
}) {
|
||||
const isAssistant = message.role === 'assistant';
|
||||
return (
|
||||
<div className={`flex gap-3 ${isAssistant ? 'flex-row' : 'flex-row-reverse'}`}>
|
||||
{isAssistant ? (
|
||||
<span className="flex h-8 w-8 shrink-0 items-center justify-center rounded-full bg-primary-100 text-primary-700 dark:bg-primary-900/40 dark:text-primary-300">
|
||||
<Bot className="h-4 w-4" />
|
||||
</span>
|
||||
) : (
|
||||
<Avatar name={primaryLearner.name} src={primaryLearner.avatar} size="sm" />
|
||||
)}
|
||||
<div
|
||||
className={`max-w-[80%] rounded-lg px-3 py-2 text-sm ${
|
||||
isAssistant
|
||||
? 'bg-slate-100 text-slate-800 dark:bg-slate-800 dark:text-slate-100'
|
||||
: 'bg-primary-600 text-white'
|
||||
}`}
|
||||
>
|
||||
<p className="leading-relaxed whitespace-pre-wrap">
|
||||
{message.content}
|
||||
{message.streaming && (
|
||||
<span className="ml-0.5 inline-block h-4 w-1.5 animate-pulse rounded-sm bg-primary-500 align-middle" />
|
||||
)}
|
||||
</p>
|
||||
{message.suggestedActions && message.suggestedActions.length > 0 && (
|
||||
<div className="mt-2 flex flex-wrap gap-1.5">
|
||||
{message.suggestedActions.map((action) => (
|
||||
<button
|
||||
key={action}
|
||||
onClick={() => onAction?.(action)}
|
||||
className="rounded-full border border-slate-300 bg-white px-2 py-0.5 text-xs text-slate-600 transition-colors hover:border-primary-400 hover:text-primary-700 dark:border-slate-600 dark:bg-slate-900 dark:text-slate-300 dark:hover:border-primary-400 dark:hover:text-primary-300"
|
||||
>
|
||||
{action}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
'use client';
|
||||
|
||||
import { AgentStreamPanel } from './agent-stream-panel';
|
||||
import { AlertTriangle, CheckCircle2, CircleDashed } from 'lucide-react';
|
||||
|
||||
interface CriterionScore {
|
||||
criterion_id: string;
|
||||
name: string;
|
||||
score: number;
|
||||
evidence: string;
|
||||
}
|
||||
|
||||
interface RubricScore {
|
||||
rubric_id: string;
|
||||
artifact_id: string;
|
||||
competency_id: string;
|
||||
scores: CriterionScore[];
|
||||
strengths: string[];
|
||||
gaps: string[];
|
||||
verdict: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Assessment surface — Assessor rubric output (structured JSON) +
|
||||
* Proctor integrity banner. Mock engine inputs; real engines v0.3+.
|
||||
*/
|
||||
export function AssessorResultsPanel({ artifactId }: { artifactId: string }) {
|
||||
return (
|
||||
<AgentStreamPanel
|
||||
title="Assessor — rubric evaluation"
|
||||
endpoint="/v1/assessment/evaluate"
|
||||
body={{ artifact_id: artifactId }}
|
||||
emptyHint="Run the Assessor to grade this artifact against its rubric."
|
||||
renderJson={(data) => {
|
||||
const score = data as unknown as RubricScore;
|
||||
return (
|
||||
<div className="flex flex-col gap-4">
|
||||
<div className="flex items-center gap-2">
|
||||
{score.verdict === 'mastered' ? (
|
||||
<CheckCircle2 className="h-5 w-5 text-emerald-600 dark:text-emerald-400" />
|
||||
) : score.verdict === 'developing' ? (
|
||||
<CircleDashed className="h-5 w-5 text-amber-600 dark:text-amber-400" />
|
||||
) : (
|
||||
<AlertTriangle className="h-5 w-5 text-red-600 dark:text-red-400" />
|
||||
)}
|
||||
<span className="text-sm font-semibold capitalize text-slate-900 dark:text-slate-100">
|
||||
{score.verdict}
|
||||
</span>
|
||||
</div>
|
||||
|
||||
<div className="flex flex-col gap-3">
|
||||
{score.scores.map((c) => (
|
||||
<div key={c.criterion_id}>
|
||||
<div className="mb-1 flex items-center justify-between text-xs">
|
||||
<span className="font-medium text-slate-700 dark:text-slate-300">{c.name}</span>
|
||||
<span className="text-slate-500 dark:text-slate-400">{c.score}/100</span>
|
||||
</div>
|
||||
<div className="h-2 overflow-hidden rounded-full bg-slate-200 dark:bg-slate-700">
|
||||
<div
|
||||
className="h-full rounded-full bg-primary-500 transition-all"
|
||||
style={{ width: `${c.score}%` }}
|
||||
role="progressbar"
|
||||
aria-valuenow={c.score}
|
||||
aria-valuemin={0}
|
||||
aria-valuemax={100}
|
||||
aria-label={c.name}
|
||||
/>
|
||||
</div>
|
||||
<p className="mt-1 text-xs text-slate-500 dark:text-slate-400">{c.evidence}</p>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div className="grid grid-cols-1 gap-3 sm:grid-cols-2">
|
||||
<div>
|
||||
<h4 className="mb-1 text-xs font-semibold uppercase tracking-wide text-emerald-700 dark:text-emerald-400">
|
||||
Strengths
|
||||
</h4>
|
||||
<ul className="list-inside list-disc text-xs text-slate-600 dark:text-slate-300">
|
||||
{score.strengths.map((s) => (
|
||||
<li key={s}>{s}</li>
|
||||
))}
|
||||
</ul>
|
||||
</div>
|
||||
<div>
|
||||
<h4 className="mb-1 text-xs font-semibold uppercase tracking-wide text-amber-700 dark:text-amber-400">
|
||||
Gaps
|
||||
</h4>
|
||||
<ul className="list-inside list-disc text-xs text-slate-600 dark:text-slate-300">
|
||||
{score.gaps.map((g) => (
|
||||
<li key={g}>{g}</li>
|
||||
))}
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}}
|
||||
/>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
'use client';
|
||||
|
||||
import { AgentStreamPanel } from './agent-stream-panel';
|
||||
|
||||
/**
|
||||
* Byte viewer Tutor panel — "Explain this byte" streams a Socratic concept
|
||||
* walkthrough for the current competency (agent fixed to tutor, A-007).
|
||||
*/
|
||||
export function ByteTutorPanel({
|
||||
competencyId,
|
||||
competencyName,
|
||||
}: {
|
||||
competencyId: string;
|
||||
competencyName: string;
|
||||
}) {
|
||||
return (
|
||||
<AgentStreamPanel
|
||||
title="Tutor — explain this byte"
|
||||
endpoint="/v1/chat/stream"
|
||||
body={{
|
||||
agent: 'tutor',
|
||||
session_id: `byte-${competencyId}`,
|
||||
messages: [
|
||||
{
|
||||
role: 'user',
|
||||
content: `Explain the byte "${competencyName}" — one concept, a worked example, then a question to check my understanding.`,
|
||||
},
|
||||
],
|
||||
}}
|
||||
emptyHint="Ask the Tutor to walk you through this byte concept step by step."
|
||||
/>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
'use client';
|
||||
|
||||
import { AgentStreamPanel } from './agent-stream-panel';
|
||||
|
||||
/**
|
||||
* Sandbox Lab feedback panel — streams in-flow feedback for the selected
|
||||
* mock telemetry scenario (real telemetry is v0.3+).
|
||||
*/
|
||||
export function LabFeedbackPanel({ scenarioId }: { scenarioId: string }) {
|
||||
return (
|
||||
<AgentStreamPanel
|
||||
title="Lab — in-flow feedback"
|
||||
endpoint="/v1/lab/feedback"
|
||||
body={{ scenario_id: scenarioId }}
|
||||
emptyHint="Run the Lab agent on this build session's telemetry."
|
||||
/>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
'use client';
|
||||
|
||||
import { AgentStreamPanel } from './agent-stream-panel';
|
||||
|
||||
/**
|
||||
* Dashboard Mentor panel — streams a long-horizon career narrative
|
||||
* tied to the learner's progress. Session-backed follow-ups in v0.3+ UI.
|
||||
*/
|
||||
export function MentorPanel() {
|
||||
return (
|
||||
<AgentStreamPanel
|
||||
title="Mentor — your trajectory"
|
||||
endpoint="/v1/mentor/narrative"
|
||||
body={{ session_id: 'dashboard-mentor', prompt: 'Narrate my trajectory.' }}
|
||||
emptyHint="Ask the Mentor where your competency progress is taking you."
|
||||
/>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
'use client';
|
||||
|
||||
import { AgentStreamPanel } from './agent-stream-panel';
|
||||
import { ShieldCheck } from 'lucide-react';
|
||||
|
||||
interface IntegritySignal {
|
||||
signal_type: string;
|
||||
severity: 'low' | 'medium' | 'high';
|
||||
note: string;
|
||||
}
|
||||
|
||||
interface ProctorAssessment {
|
||||
scenario_id: string;
|
||||
signals: IntegritySignal[];
|
||||
intervention: string;
|
||||
summary: string;
|
||||
}
|
||||
|
||||
const SEVERITY_STYLES: Record<string, string> = {
|
||||
low: 'bg-emerald-50 text-emerald-700 border-emerald-200 dark:bg-emerald-900/30 dark:text-emerald-300 dark:border-emerald-800',
|
||||
medium: 'bg-amber-50 text-amber-700 border-amber-200 dark:bg-amber-900/30 dark:text-amber-300 dark:border-amber-800',
|
||||
high: 'bg-red-50 text-red-700 border-red-200 dark:bg-red-900/30 dark:text-red-300 dark:border-red-800',
|
||||
};
|
||||
|
||||
/**
|
||||
* Proctor integrity banner — supportive, coaching-shaped (never punitive).
|
||||
*/
|
||||
export function ProctorBanner({ scenarioId }: { scenarioId: string }) {
|
||||
return (
|
||||
<AgentStreamPanel
|
||||
title="Proctor — integrity support"
|
||||
endpoint="/v1/proctor/signals"
|
||||
body={{ scenario_id: scenarioId }}
|
||||
emptyHint="Run the Proctor to review this session's integrity signals."
|
||||
renderJson={(data) => {
|
||||
const assessment = data as unknown as ProctorAssessment;
|
||||
return (
|
||||
<div className="flex flex-col gap-3">
|
||||
<p className="text-xs text-slate-600 dark:text-slate-300">{assessment.summary}</p>
|
||||
<div className="flex flex-wrap gap-1.5">
|
||||
{assessment.signals.map((s, i) => (
|
||||
<span
|
||||
key={i}
|
||||
className={`inline-flex items-center gap-1 rounded-full border px-2 py-0.5 text-xs ${SEVERITY_STYLES[s.severity] ?? SEVERITY_STYLES.low}`}
|
||||
title={s.note}
|
||||
>
|
||||
<ShieldCheck className="h-3 w-3" />
|
||||
{s.signal_type} · {s.severity}
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
<p className="rounded-md bg-primary-50 px-3 py-2 text-xs text-primary-800 dark:bg-primary-900/30 dark:text-primary-200">
|
||||
<strong>Suggested next step:</strong> {assessment.intervention}
|
||||
</p>
|
||||
</div>
|
||||
);
|
||||
}}
|
||||
/>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,218 @@
|
||||
'use client';
|
||||
|
||||
import { useCallback, useEffect, useRef, useState } from 'react';
|
||||
import { parseSseEvents } from '../lib/sse';
|
||||
|
||||
const AI_SERVICE_URL =
|
||||
process.env.NEXT_PUBLIC_AI_SERVICE_URL ?? 'http://localhost:8420';
|
||||
|
||||
export type AgentName = 'coach' | 'tutor' | 'lab' | 'assessor' | 'proctor' | 'mentor';
|
||||
|
||||
export interface StreamMessage {
|
||||
id: string;
|
||||
role: 'user' | 'assistant';
|
||||
content: string;
|
||||
agent?: AgentName;
|
||||
streaming?: boolean;
|
||||
suggestedActions?: string[];
|
||||
}
|
||||
|
||||
interface StreamState {
|
||||
messages: StreamMessage[];
|
||||
isStreaming: boolean;
|
||||
error: string | null;
|
||||
model: string | null;
|
||||
}
|
||||
|
||||
interface ChatStreamEvent {
|
||||
type: 'meta' | 'delta' | 'done' | 'error';
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
function decodeEvent(raw: string): ChatStreamEvent | '[DONE]' | null {
|
||||
if (raw === '[DONE]') return '[DONE]';
|
||||
try {
|
||||
const parsed = JSON.parse(raw);
|
||||
if (typeof parsed?.type === 'string') return parsed as ChatStreamEvent;
|
||||
// OpenAI-shaped chunks (id/choices) are not used by our envelope;
|
||||
// ignore anything without a type.
|
||||
return null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export function useChatStream(agent: AgentName) {
|
||||
const [state, setState] = useState<StreamState>({
|
||||
messages: [],
|
||||
isStreaming: false,
|
||||
error: null,
|
||||
model: null,
|
||||
});
|
||||
const abortRef = useRef<AbortController | null>(null);
|
||||
// Agent-scoped sessions (A-007/D-019): switching agents starts a NEW
|
||||
// session per agent — no persona bleed across switcher flips.
|
||||
const sessionsRef = useRef<Partial<Record<AgentName, string>>>({});
|
||||
if (!sessionsRef.current[agent]) {
|
||||
const uuid =
|
||||
typeof crypto !== 'undefined' && 'randomUUID' in crypto
|
||||
? crypto.randomUUID()
|
||||
: String(Date.now());
|
||||
sessionsRef.current[agent] = `${agent}-${uuid}`;
|
||||
}
|
||||
|
||||
// Idempotent abort + cleanup on unmount or agent switch (Strict Mode safe)
|
||||
const abort = useCallback(() => {
|
||||
abortRef.current?.abort();
|
||||
abortRef.current = null;
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
abortRef.current?.abort();
|
||||
abortRef.current = null;
|
||||
};
|
||||
}, []);
|
||||
|
||||
const send = useCallback(
|
||||
async (text: string) => {
|
||||
const trimmed = text.trim();
|
||||
if (!trimmed || abortRef.current) return;
|
||||
|
||||
const userMessage: StreamMessage = {
|
||||
id: `user-${Date.now()}`,
|
||||
role: 'user',
|
||||
content: trimmed,
|
||||
};
|
||||
const assistantId = `assistant-${Date.now()}`;
|
||||
|
||||
setState((prev) => ({
|
||||
...prev,
|
||||
messages: [...prev.messages, userMessage],
|
||||
isStreaming: true,
|
||||
error: null,
|
||||
}));
|
||||
|
||||
const controller = new AbortController();
|
||||
abortRef.current = controller;
|
||||
|
||||
try {
|
||||
const response = await fetch(`${AI_SERVICE_URL}/v1/chat/stream`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
agent,
|
||||
session_id: sessionsRef.current[agent],
|
||||
messages: [{ role: 'user', content: trimmed }],
|
||||
}),
|
||||
signal: controller.signal,
|
||||
});
|
||||
|
||||
if (!response.ok || !response.body) {
|
||||
throw new Error(`AI service unavailable (${response.status})`);
|
||||
}
|
||||
|
||||
setState((prev) => ({
|
||||
...prev,
|
||||
messages: [
|
||||
...prev.messages,
|
||||
{ id: assistantId, role: 'assistant', content: '', agent, streaming: true },
|
||||
],
|
||||
}));
|
||||
|
||||
const reader = response.body.getReader();
|
||||
const decoder = new TextDecoder();
|
||||
let buffer = '';
|
||||
let done = false;
|
||||
|
||||
while (!done) {
|
||||
const { value, done: readerDone } = await reader.read();
|
||||
if (readerDone) break;
|
||||
buffer += decoder.decode(value, { stream: true });
|
||||
|
||||
const { events, rest } = parseSseEvents(buffer);
|
||||
buffer = rest;
|
||||
|
||||
for (const raw of events) {
|
||||
const event = decodeEvent(raw);
|
||||
if (event === null) continue;
|
||||
if (event === '[DONE]') {
|
||||
done = true;
|
||||
setState((prev) => ({
|
||||
...prev,
|
||||
isStreaming: false,
|
||||
messages: prev.messages.map((m) =>
|
||||
m.id === assistantId ? { ...m, streaming: false } : m,
|
||||
),
|
||||
}));
|
||||
break;
|
||||
}
|
||||
if (event.type === 'meta') {
|
||||
setState((prev) => ({ ...prev, model: (event.model as string) ?? null }));
|
||||
} else if (event.type === 'delta') {
|
||||
const content = event.content as string;
|
||||
setState((prev) => ({
|
||||
...prev,
|
||||
messages: prev.messages.map((m) =>
|
||||
m.id === assistantId ? { ...m, content: m.content + content } : m,
|
||||
),
|
||||
}));
|
||||
} else if (event.type === 'error') {
|
||||
setState((prev) => ({
|
||||
...prev,
|
||||
error: (event.message as string) ?? 'stream error',
|
||||
}));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
setState((prev) => ({
|
||||
...prev,
|
||||
isStreaming: false,
|
||||
messages: prev.messages.map((m) =>
|
||||
m.id === assistantId ? { ...m, streaming: false } : m,
|
||||
),
|
||||
}));
|
||||
} catch (err) {
|
||||
const aborted = err instanceof DOMException && err.name === 'AbortError';
|
||||
if (!aborted) {
|
||||
setState((prev) => ({
|
||||
...prev,
|
||||
isStreaming: false,
|
||||
error: err instanceof Error ? err.message : 'connection failed',
|
||||
messages: prev.messages.map((m) =>
|
||||
m.id === assistantId ? { ...m, streaming: false } : m,
|
||||
),
|
||||
}));
|
||||
} else {
|
||||
setState((prev) => ({
|
||||
...prev,
|
||||
isStreaming: false,
|
||||
messages: prev.messages.map((m) =>
|
||||
m.id === assistantId ? { ...m, streaming: false } : m,
|
||||
),
|
||||
}));
|
||||
}
|
||||
} finally {
|
||||
abortRef.current = null;
|
||||
}
|
||||
},
|
||||
[agent],
|
||||
);
|
||||
|
||||
const retry = useCallback(() => {
|
||||
setState((prev) => ({ ...prev, error: null }));
|
||||
const lastUser = [...state.messages].reverse().find((m) => m.role === 'user');
|
||||
if (lastUser) void send(lastUser.content);
|
||||
}, [send, state.messages]);
|
||||
|
||||
return {
|
||||
messages: state.messages,
|
||||
isStreaming: state.isStreaming,
|
||||
error: state.error,
|
||||
model: state.model,
|
||||
send,
|
||||
retry,
|
||||
abort,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
/**
|
||||
* Shared SSE parsing for the AI service streams.
|
||||
*
|
||||
* Normalizes CRLF (sse-starlette's wire terminator is \r\n) to LF, then
|
||||
* splits frames on blank lines. Multiple `data:` lines within one frame
|
||||
* are joined with \n per the SSE spec. Frames with no data lines
|
||||
* (keep-alive `: ping` comments) are ignored (G-1).
|
||||
*/
|
||||
export function parseSseEvents(buffer: string): { events: string[]; rest: string } {
|
||||
const normalized = buffer.replace(/\r\n/g, '\n');
|
||||
const events: string[] = [];
|
||||
|
||||
const separatorIndex = normalized.lastIndexOf('\n\n');
|
||||
if (separatorIndex === -1) return { events, rest: normalized };
|
||||
|
||||
const complete = normalized.slice(0, separatorIndex);
|
||||
const rest = normalized.slice(separatorIndex + 2);
|
||||
|
||||
for (const frame of complete.split('\n\n')) {
|
||||
const dataLines = frame
|
||||
.split('\n')
|
||||
.filter((line) => line.startsWith('data:'))
|
||||
.map((line) => line.slice(5).trimStart());
|
||||
if (dataLines.length === 0) continue; // ping/comment frame — ignore (G-1)
|
||||
events.push(dataLines.join('\n'));
|
||||
}
|
||||
return { events, rest };
|
||||
}
|
||||
Reference in New Issue
Block a user