# LLM Evaluation Engineering

> [View this product on lib-pro](https://lib-pro-3f8437.gitlab.io/libs/llm-evaluation) · The methodology that catches regressions before users do.

**Outcome:** Ship LLM features with quantitative feedback, not vibes.

5 skills for shipping LLM features with quantitative feedback loops: the eval funnel (deterministic → LLM-as-judge → human), golden set construction, scorer patterns, calibration, and eval-driven development. The same methodology Hamel Husain teaches in his $4,200 Maven course — packaged as a SKILL.md bundle you can apply to any eval system (Braintrust, LangSmith, Arize, Helicone, or your own).

**Price:** £40 — lifetime · **Category:** Production AI Skills · **Tech:** pi-agent, Claude Code, Cursor, Cline, Markdown, Eval vendors (vendor-agnostic)

| Metric | Value |
|--------|-------|
| Skills | 5 |
| Adapters | 5 |
| Worked examples | 2 |

## Quick start

```bash
tar -xzf llm-evaluation-0.1.0.tgz && cd llm-evaluation-0.1.0
```

- # Pick your agent host
- ls adapters/
- # pi-agent: cp -r skills/* ~/.pi/agent/skills/
- # Claude Code: cp -r adapters/claude-code/commands/* .claude/commands/

## Code preview

```
# llm-evaluation — The Eval Funnel

All you get from using prefab evals is you don't know what they actually do.
— Hamel Husain

The eval funnel:

1. **Deterministic** (cheap, fast, reliable) — schema, regex, exact-match, citation-presence. The workhorses.
2. **LLM-as-judge** (expensive, slower, needs calibration) — quality, nuance, open-ended answers. Calibrate against human ground truth.
3. **Human review** (slowest, most expensive, ground truth) — spot-check, edge cases, new failure modes.

Every eval system should walk the funnel. Most ship the wrong way (LLM-as-judge first, no calibration, no ground truth).

## The 5 Skills

| Skill | What it teaches |
|---|---|
| llm-evaluation | The funnel, when to use what, the eval mindset |
| eval-design | What to measure, how to write criteria, golden sets |
| deterministic-scorers | Schema checks, regex, exact-match, citation-presence |
| llm-as-judge | When LLM judges work, calibration, prompt design, pitfalls |
| eval-driven-dev | Eval-first PRs, baseline tracking, regression detection |

## The Core Idea

> Build evals from real failures, not imagined ones. Run evals on every change. Compare to the prior baseline.
```

## Use cases

### 1. You're shipping an LLM feature

You added a new prompt. You want to know if quality went up or down. No eval system, just vibes and a few manual checks.

**Outcome:** Build a 20-case golden set, run it on every PR, never ship a regression again.

### 2. You're evaluating a new model

GPT-5 just dropped. You want to know if it's better than Sonnet for your use case. The vendor says yes. Your gut says maybe.

**Outcome:** Run your golden set on both models. See the numbers. Decide with data, not vibes.

### 3. You're scaling AI features

Three teams, six features, dozens of prompts. You can't manually review every change. Quality is drifting.

**Outcome:** Standardize on the eval funnel. Every team runs the same workflow. Quality becomes measurable.

## Built for

- AI engineers shipping their first LLM feature
- AI engineering managers who need a team-wide eval standard
- Senior engineers who want to defend AI work with data
- Indie hackers shipping AI products who can't afford a quality incident

## Highlights

### The eval funnel

Deterministic first (cheap, fast, reliable). LLM-as-judge second (expensive, needs calibration). Human review last (ground truth). Every eval system should walk the funnel.

### Golden sets from real failures

The best golden sets are built from real prod failures, not imagined test cases. The pack teaches how to harvest failures into your eval set.

### Eval-first PRs

Run evals before merging. Compare to the prior baseline. Catch regressions before they reach users. The workflow, not the tool.

## What's in the box

- `5 skills` — eval funnel, design, scorers, judge, workflow
- `5 tool adapters` — pi-agent, Claude Code, Cursor, Cline, generic
- `2 worked examples` — eval design + regression detection
- `Methodology` — vendor-agnostic
- `Calibration patterns` — for LLM-as-judge
- `Golden set construction` — from real failures

## Adapters included

These skills were authored for pi-agent, but we include drop-in adapters for the major AI coding tools.

| Tool | Adapter location | What you get |
|------|------------------|--------------|
| pi-agent | `adapters/pi-agent/` | SKILL.md drop-in |
| Claude Code | `adapters/claude-code/` | .claude/commands/*.md + CLAUDE.md |
| Cursor | `adapters/cursor/` | .cursorrules rule sets |
| Cline / Continue | `adapters/cline/` | System-prompt rules |
| Aider / generic | `adapters/generic/` | INSTRUCTIONS.md snippets |

**Same playbook, different packaging.** The adapters translate the methodology into the format each tool expects. Install once, swap agents without re-learning the method.

## Features

- llm-evaluation skill — master orchestrator + the eval funnel
- eval-design skill — what to measure, criteria writing, golden sets from real failures
- deterministic-scorers skill — schema checks, regex, exact-match, citation-presence
- llm-as-judge skill — when LLM judges work, calibration, prompt design, bias mitigation
- eval-driven-dev skill — eval-first PRs, baseline tracking, regression detection
- 5 tool adapters — pi-agent, Claude Code, Cursor, Cline, generic
- 2 worked examples — eval design + regression detection
- Methodology, not vendor lock-in — works with Braintrust, LangSmith, Arize, Helicone, or your own

## FAQ

### Is this a Braintrust / LangSmith / Helicone replacement?

No. The pack is methodology — what to measure, how to design evals, how to detect regressions. The vendors are infrastructure — they run the evals. Use both: this pack for the playbook, your vendor for execution.

### How is this different from Hamel Husain's Maven course?

Hamel's course is the gold standard. This pack distills the same methodology into a SKILL.md bundle you can drop into your agent. The course is 6 weeks of video; the pack is the same playbook, condensed, agent-loadable.

### I don't have any evals yet. Where do I start?

Start with eval-design. It teaches you to write criteria, build a golden set from real failures, and pick the right granularity. From there, deterministic-scorers first (cheap), LLM-as-judge second (calibrate against human).