Reference Implementation

Evals Studio

Name: Evals Studio
Brand: lib-pro
SKU: evals-studio
Price: 40.00 GBP
Availability: InStock

The eval pipeline senior AI engineering leaders are expected to run.

A production-grade eval pipeline that turns 'I haven't built an eval pipeline' into 'yes — here it is.'

See use cases ↓

One-time paymentLifetime updatesCommercial license

Milestone

Starter cases

Deterministic scorers

M2–M6

Roadmap

What you'll be able to do

Concrete outcomes, not vague promises.

Closing the AI eval gap

On a senior AI engineering interview track. Hiring managers ask 'have you built an eval pipeline?' This is the answer.

From 'I'm learning' to 'yes — here it is.'

Production eval for your own AI team

Building an AI agent product. Need to know if a model upgrade actually improves things or just shuffles failures. Drop your agent in, run the golden set.

Catch regressions before users do.

Differentiator for client LLM work

Selling AI consulting. Most consultants ship prompts; you ship a pipeline that proves the outputs are good. The architecture is the same shape Big Tech uses.

Premium pricing because you can defend the work.

Calibrate an LLM-as-judge

Need a cheap, fast way to grade 1000s of outputs. Build it on this framework — the deterministic scorers are your ground truth, the LLM judge is calibrated against them.

Automated grading that doesn't lie.

Built for

If any of these are you, this is for you.

Senior AI engineering leaders preparing for system-design interviewsAI engineering managers building eval pipelines for their teamConsultants selling LLM work who need defensible quality measurementAI product teams who want to catch quality regressions before usersResearchers building reproducible eval harnesses for papers

Built withTypeScriptNode 20FastifyOpenAI SDKbetter-sqlite3ZodMarkdown

Why this playbook works

Three things that make this different from a wall of prompts.

Architecture, not vendor lock-in

The eval harness is generic. The agent (`src/agent/agent.ts`) is a swap-in module. Replace your agent — same pipeline, same scorers, same UI.

Deterministic, not vibe-based

Every scorer is a pure function with a rationale string. Failed cases are debuggable in 30 seconds — 'this failed because citation src-003 was missing.'

Regression-aware, not one-shot

Every run is persisted to SQLite. The runner compares to the previous baseline. Drift between model upgrades is loud, not silent.

Quick start

Drop the package into your project and start using the commands.

terminal

$tar -xzf evals-studio-0.2.0.tgz && cd evals-studio-0.2.0 && npm install

$# Install

$npm install

$# Smoke test (no LLM needed)

$npm test

$# Run the M1 golden set

$npm run eval

$# Or start the Fastify service

$npm start

What's in the box

Agent runtime
replace with your own
Deterministic scorers
schema, citations, access
Runner + regression detection
vs prior baseline
Fastify service
POST /runs, GET /runs/:id
SQLite state store
single file, no infra
3 starter golden cases
happy path + 2 violations
Pre-built dist/
runs without a build step
Architecture docs
M2–M6 roadmap built in

Everything in the pack

What you get the moment your payment clears.

Agent runtime with real tool calls + structured outputs + access controls

Deterministic scorers (schema, citations, access controls) — pure functions, traceable

Runner with run history + regression detection against prior baseline

SQLite persistence (single file, no infra) — better-sqlite3

Fastify API service — POST /runs, GET /runs/:id, GET /regressions

CLI for running the golden set from your terminal

3 starter golden-set cases (happy path, access violation, required citation)

Pre-built dist/ — runs without a build step

Architecture docs — shaped to support M2–M6 without rewrites

Adapters for Claude Code, Cursor, Cline, Aider

TypeScript on Node 20, Fastify 4, OpenAI SDK

Open-source upgrade path (M1 is MIT upstream, this is a curated package)

Adapters included

These skills were authored for pi-agent, but we include drop-in adapters for the major AI coding tools.

Tool	Adapter location	What you get
pi-agent	`adapters/pi-agent/`	SKILL.md drop-in
Claude Code	`adapters/claude-code/`	CLAUDE.md fragments
Cline / Continue	`adapters/cline/`	System-prompt rules
Aider / generic	`adapters/generic/`	INSTRUCTIONS.md snippets

Same playbook, different packaging. The adapters translate the methodology into the format each tool expects. Install once, swap agents without re-learning the method.

Full documentation

Everything in the package, every prop, every pattern.

Evals Studio

The evaluation pipeline senior AI engineering leaders are expected to run. Open the box, replace the agent with yours, ship.

Built and battle-tested. The architecture that turns "I haven't built an eval pipeline" into "yes — here it is."

What's Inside

The pack ships the M1 milestone of a six-milestone vision. M1 is the foundation: working agent runtime, deterministic scorers, runner, Fastify API, SQLite state, 3 starter golden-set cases. The architecture is shaped to support M2 through M6 without rewrites.

Working code

Component	What it does
`src/agent/`	The agent under test — real tool calls, structured outputs, access-control refusals. Replace this with your own agent.
`src/evals/runner.ts`	Run loop, trace persistence, regression detection against prior baseline
`src/evals/scorers.ts`	Deterministic scorers — schema validity, expected tool calls, access controls, citation coverage
`src/evals/store.ts`	SQLite persistence (single file, no infra)
`src/evals/cli.ts`	Run the golden set from your terminal
`src/api/server.ts`	Fastify service — POST /runs, GET /runs/:id, GET /regressions
`data/golden/m1.jsonl`	3 starter cases (happy path, access-control violation, required citation)

Pre-built

dist/ — pre-built JavaScript so the service runs without a build step
tests/ — scorer smoke tests (no LLM needed)

Documentation

Doc	Content
`docs/architecture.md`	The full diagram + component rationale
`docs/deploy.md`	Deploy to the homelab (Docker Swarm stack)
`docs/2026-07-14-m1-foundations.md`	The M1 milestone plan you can build on

Adapters

Drop-in rules for AI coders — let your tool answer questions about evals with the same vocabulary as the pack:

adapters/pi-agent/ — direct copy
adapters/claude-code/ — CLAUDE.md fragments
adapters/cline/ — system-prompt rules
adapters/generic/ — INSTRUCTIONS.md snippets

Quick Start

# 1. Install
npm install

# 2. Smoke test the scorers (no LLM needed)
npm test

# 3. Run the golden set from the terminal
npm run eval

# 4. Or run the Fastify service
npm run build
npm start

# 5. Deploy (homelab)
docker build -t evals-studio .
# Add a service block to /PI/PROJECTS/projects/docker-compose.yml
# and deploy via the Portainer stack editor.

How to Use It With Your Own Agent

The eval pipeline is generic. The agent is replaceable. To use this pack with your own agent:

Drop in your agent. Edit src/agent/agent.ts to wire your LLM tool calling + structured output. The AgentRuntimeDeps interface is the contract — implement dispatchToolCall and you're done.
Define your eval rubric. Replace the scorers in src/evals/scorers.ts with rubric for your domain. The pattern: deterministic checks for shape/citations/access, an LLM-as-judge for quality.
Curate your golden set. Edit data/golden/m1.jsonl (or create a new file) with 20–30 hand-curated cases covering happy path, edge cases, and violations.
Run it. npm run eval for terminal, npm start for the API.

The Core Idea

The pipeline is the architecture. The agents are replaceable.

Most eval pipelines die because they conflate the harness with the agent. This pack separates them — the agent is a swap-in module, the pipeline is the long-lived asset. Replace the agent when your model changes. Replace the pipeline when your eval methodology changes. Don't replace both at once.

Roadmap

The full vision has 6 milestones:

M1 — Foundations. Repo, stack, "hello eval" running. ✅ Shipped in this pack.
M2 — Golden set + deterministic scoring. 20–30 hand-curated cases. The 3 starter cases here are the seed.
M3 — LLM-as-judge. Rubric designed; judge calibrated against human spot-check on 10 cases.
M4 — Regression tracking. Run history, delta-from-baseline, failure clustering.
M5 — Feedback loop. Sampled production traces → eval set. The closed loop.
M6 — Interview artefact. Case study written.

You get M1 working out of the box. The patterns are in place — M2–M6 are the next 4–6 weeks of work, and the architecture supports them without rewrites.

What's in the download

A .tgz containing:

evals-studio-0.2.0/
├── README.md
├── OVERVIEW.md
├── LICENSE.md
├── package.json
├── tsconfig.json
├── Dockerfile
├── CLAUDE.md
├── src/
│   ├── agent/                 ← agent runtime (replace with your own)
│   ├── evals/                 ← runner, scorers, store, CLI
│   ├── api/                   ← Fastify service
│   └── ui/                    ← (placeholder; extend with your dashboard)
├── dist/                      ← pre-built JavaScript (runs out of the box)
├── data/golden/               ← starter eval cases
├── data/runs/                 ← SQLite state store
├── tests/                     ← scorer smoke tests
├── docs/                      ← architecture + deploy + roadmap
└── adapters/                  ← AI coder drop-ins

Updates

Free for life. Re-download for:

New golden-set cases (as the M2 milestone ships)
New scorers (scoring rubric improvements)
New adapters (new AI coding tools)
Bug fixes

What This Isn't

Not a turnkey "AI eval as a service." The pipeline is the architecture; you bring the agents.
Not a single-shot test harness. The pipeline is built for repeated runs + regression tracking.
Not a replacement for production monitoring. The LLM-as-judge in production is a sibling concern; this pack gives you the eval half.

Common questions

Things people ask before buying.

Is this a complete eval pipeline or just a foundation?

It's M1 code-complete — a working pipeline, scorer framework, runner, Fastify API, and 3 starter cases. The full 6-milestone vision includes golden-set expansion (M2), LLM-as-judge with calibration (M3), regression tracking (M4), feedback loop (M5), and an interview artefact (M6). The architecture is shaped to support all of them without rewrites.

How do I use this with my own agent?

Edit `src/agent/agent.ts` to wire your LLM tool calling + structured output. The `AgentRuntimeDeps` interface is the contract — implement `dispatchToolCall` and pass your tool schemas. The runner drives the loop; the scorers evaluate the output.

What LLM does it use?

The agent runtime uses the OpenAI SDK, which works with any OpenAI-compatible endpoint. Default: local `http://ai_gpu:88/v1` (Qwen3.6-35B-A3B). Configure via `OPENAI_BASE_URL` and `OPENAI_API_KEY` env vars.

Why SQLite? Postgres seems more 'production'.

SQLite is a single file. No infra. No auth. No replicas. No migrations. For an eval pipeline that runs a few hundred times a day, it's the right tool. The schema is portable — when you outgrow SQLite, `pg_dump` in Postgres and update the connection layer. The architecture is the value.

What's the operating cost?

Static cost: $0. The LLM call is the only per-run cost. With a local Qwen 3 8B model, that's free. With GPT-4o-mini, roughly $0.01 per case. With GPT-4o, roughly $0.10 per case. The pipeline itself is zero.

How is this different from Langfuse / LangSmith / Helicone?

Those are observability + production tracing tools. This is an eval pipeline. Different concern. You can run this pack alongside Langfuse — the eval pipeline uses this pack's golden set + scorers; Langfuse watches production traffic. They complement each other.

Why is the upstream MIT but this pack is commercial?

The upstream `evals-studio` repo is a public artefact — the whole point is showing what's possible. This pack is a curated, packaged version with adapters, examples, and a clear path from M1 to M6. The upstream code is the architecture; the pack is the deliverable.

£40 — lifetime

One purchase. Yours forever.

Pay once via Stripe. Get the complete package instantly. Free updates for the same major version. Commercial license included.

Secure payment via Stripe · Instant download · 30-day money-back if it doesn't save you time