# Evals Studio

> [View this product on lib-pro](https://lib-pro-3f8437.gitlab.io/libs/evals-studio) · The eval pipeline senior AI engineering leaders are expected to run.

**Outcome:** A production-grade eval pipeline that turns 'I haven't built an eval pipeline' into 'yes — here it is.'

A production-grade evaluation pipeline for agentic AI systems. M1 code-complete: real agent runtime with tool calls + structured outputs + access controls, deterministic scorers (schema, citations, access), a runner with regression detection, Fastify service, SQLite state store, and 3 starter golden-set cases. The architecture is shaped to support M2–M6 (golden set expansion, LLM-as-judge, regression tracking, feedback loop) without rewrites.

**Price:** £40 — lifetime · **Category:** Reference Implementation · **Tech:** TypeScript, Node 20, Fastify, OpenAI SDK, better-sqlite3, Zod, Markdown

| Metric | Value |
|--------|-------|
| Milestone | M1 |
| Starter cases | 3 |
| Deterministic scorers | 5+ |
| Roadmap | M2–M6 |

## Quick start

```bash
tar -xzf evals-studio-0.2.0.tgz && cd evals-studio-0.2.0 && npm install
```

- # Install
- npm install
- # Smoke test (no LLM needed)
- npm test
- # Run the M1 golden set
- npm run eval
- # Or start the Fastify service
- npm start

## Code preview

```tsx
/**
 * Deterministic scorers. Pure functions — given a case + output,
 * the score is fully traceable. The rationale string explains exactly
 * why a case failed, which makes a "failed" debuggable in 30 seconds.
 *
 * The pattern: deterministic checks for shape, citations, and access.
 * LLM-as-judge for quality (in M3). This pack ships the deterministic
 * half, which is the part most teams skip — and that's why their
 * pipelines are untrustworthy.
 */

import type { Scorer, ScoreRecord, ScorerContext } from "./types.js";
import type { ResearchOutput } from "../agent/types.js";

const passed = (
  scorer: string,
  score: number,
  reason: string
): ScoreRecord => ({ scorer, passed: score === 1, score, reason });

/**
 * Structural validity: the output matches the ResearchOutput schema and
 * (in answer mode) has at least one citation; (in refuse mode) has
 * refused=true with a non-empty reason.
 */
export const schemaScorer: Scorer = ({ case: c, output }): ScoreRecord => {
  if (typeof output.answer !== "string") {
    return passed("schema", 0, "answer must be a string");
  }
  if (output.refused) {
    if (typeof output.refusal_reason !== "string" || output.refusal_reason.length === 0) {
      return passed("schema", 0, "refusal_reason required when refused=true");
    }
    return passed("schema", 1, "refusal shape ok");
  }
  // ... (continues with citation checks, access checks, etc.)
};

```

## Use cases

### 1. Closing the AI eval gap

On a senior AI engineering interview track. Hiring managers ask 'have you built an eval pipeline?' This is the answer.

**Outcome:** From 'I'm learning' to 'yes — here it is.'

### 2. Production eval for your own AI team

Building an AI agent product. Need to know if a model upgrade actually improves things or just shuffles failures. Drop your agent in, run the golden set.

**Outcome:** Catch regressions before users do.

### 3. Differentiator for client LLM work

Selling AI consulting. Most consultants ship prompts; you ship a pipeline that proves the outputs are good. The architecture is the same shape Big Tech uses.

**Outcome:** Premium pricing because you can defend the work.

### 4. Calibrate an LLM-as-judge

Need a cheap, fast way to grade 1000s of outputs. Build it on this framework — the deterministic scorers are your ground truth, the LLM judge is calibrated against them.

**Outcome:** Automated grading that doesn't lie.

## Built for

- Senior AI engineering leaders preparing for system-design interviews
- AI engineering managers building eval pipelines for their team
- Consultants selling LLM work who need defensible quality measurement
- AI product teams who want to catch quality regressions before users
- Researchers building reproducible eval harnesses for papers

## Highlights

### Architecture, not vendor lock-in

The eval harness is generic. The agent (`src/agent/agent.ts`) is a swap-in module. Replace your agent — same pipeline, same scorers, same UI.

### Deterministic, not vibe-based

Every scorer is a pure function with a rationale string. Failed cases are debuggable in 30 seconds — 'this failed because citation src-003 was missing.'

### Regression-aware, not one-shot

Every run is persisted to SQLite. The runner compares to the previous baseline. Drift between model upgrades is loud, not silent.

## What's in the box

- `Agent runtime` — replace with your own
- `Deterministic scorers` — schema, citations, access
- `Runner + regression detection` — vs prior baseline
- `Fastify service` — POST /runs, GET /runs/:id
- `SQLite state store` — single file, no infra
- `3 starter golden cases` — happy path + 2 violations
- `Pre-built dist/` — runs without a build step
- `Architecture docs` — M2–M6 roadmap built in

## Adapters included

These skills were authored for pi-agent, but we include drop-in adapters for the major AI coding tools.

| Tool | Adapter location | What you get |
|------|------------------|--------------|
| pi-agent | `adapters/pi-agent/` | SKILL.md drop-in |
| Claude Code | `adapters/claude-code/` | CLAUDE.md fragments |
| Cline / Continue | `adapters/cline/` | System-prompt rules |
| Aider / generic | `adapters/generic/` | INSTRUCTIONS.md snippets |

**Same playbook, different packaging.** The adapters translate the methodology into the format each tool expects. Install once, swap agents without re-learning the method.

## Features

- Agent runtime with real tool calls + structured outputs + access controls
- Deterministic scorers (schema, citations, access controls) — pure functions, traceable
- Runner with run history + regression detection against prior baseline
- SQLite persistence (single file, no infra) — better-sqlite3
- Fastify API service — POST /runs, GET /runs/:id, GET /regressions
- CLI for running the golden set from your terminal
- 3 starter golden-set cases (happy path, access violation, required citation)
- Pre-built dist/ — runs without a build step
- Architecture docs — shaped to support M2–M6 without rewrites
- Adapters for Claude Code, Cursor, Cline, Aider
- TypeScript on Node 20, Fastify 4, OpenAI SDK
- Open-source upgrade path (M1 is MIT upstream, this is a curated package)

## FAQ

### Is this a complete eval pipeline or just a foundation?

It's M1 code-complete — a working pipeline, scorer framework, runner, Fastify API, and 3 starter cases. The full 6-milestone vision includes golden-set expansion (M2), LLM-as-judge with calibration (M3), regression tracking (M4), feedback loop (M5), and an interview artefact (M6). The architecture is shaped to support all of them without rewrites.

### How do I use this with my own agent?

Edit `src/agent/agent.ts` to wire your LLM tool calling + structured output. The `AgentRuntimeDeps` interface is the contract — implement `dispatchToolCall` and pass your tool schemas. The runner drives the loop; the scorers evaluate the output.

### What LLM does it use?

The agent runtime uses the OpenAI SDK, which works with any OpenAI-compatible endpoint. Default: local `http://ai_gpu:88/v1` (Qwen3.6-35B-A3B). Configure via `OPENAI_BASE_URL` and `OPENAI_API_KEY` env vars.

### Why SQLite? Postgres seems more 'production'.

SQLite is a single file. No infra. No auth. No replicas. No migrations. For an eval pipeline that runs a few hundred times a day, it's the right tool. The schema is portable — when you outgrow SQLite, `pg_dump` in Postgres and update the connection layer. The architecture is the value.

### What's the operating cost?

Static cost: $0. The LLM call is the only per-run cost. With a local Qwen 3 8B model, that's free. With GPT-4o-mini, roughly $0.01 per case. With GPT-4o, roughly $0.10 per case. The pipeline itself is zero.

### How is this different from Langfuse / LangSmith / Helicone?

Those are observability + production tracing tools. This is an eval pipeline. Different concern. You can run this pack alongside Langfuse — the eval pipeline uses this pack's golden set + scorers; Langfuse watches production traffic. They complement each other.

### Why is the upstream MIT but this pack is commercial?

The upstream `evals-studio` repo is a public artefact — the whole point is showing what's possible. This pack is a curated, packaged version with adapters, examples, and a clear path from M1 to M6. The upstream code is the architecture; the pack is the deliverable.