All products
Production AI Skills

LLM Evaluation Engineering

The methodology that catches regressions before users do.

Ship LLM features with quantitative feedback, not vibes.
See use cases ↓
One-time paymentLifetime updatesCommercial license

5

Skills

5

Adapters

2

Worked examples

What you'll be able to do

Concrete outcomes, not vague promises.

01

You're shipping an LLM feature

You added a new prompt. You want to know if quality went up or down. No eval system, just vibes and a few manual checks.

Build a 20-case golden set, run it on every PR, never ship a regression again.
02

You're evaluating a new model

GPT-5 just dropped. You want to know if it's better than Sonnet for your use case. The vendor says yes. Your gut says maybe.

Run your golden set on both models. See the numbers. Decide with data, not vibes.
03

You're scaling AI features

Three teams, six features, dozens of prompts. You can't manually review every change. Quality is drifting.

Standardize on the eval funnel. Every team runs the same workflow. Quality becomes measurable.

Built for

If any of these are you, this is for you.

AI engineers shipping their first LLM featureAI engineering managers who need a team-wide eval standardSenior engineers who want to defend AI work with dataIndie hackers shipping AI products who can't afford a quality incident
Built withpi-agentClaude CodeCursorClineMarkdownEval vendors (vendor-agnostic)

Why this playbook works

Three things that make this different from a wall of prompts.

The eval funnel

Deterministic first (cheap, fast, reliable). LLM-as-judge second (expensive, needs calibration). Human review last (ground truth). Every eval system should walk the funnel.

Golden sets from real failures

The best golden sets are built from real prod failures, not imagined test cases. The pack teaches how to harvest failures into your eval set.

Eval-first PRs

Run evals before merging. Compare to the prior baseline. Catch regressions before they reach users. The workflow, not the tool.

Quick start

Drop the package into your project and start using the commands.

terminal
$tar -xzf llm-evaluation-0.1.0.tgz && cd llm-evaluation-0.1.0
$# Pick your agent host
$ls adapters/
$# pi-agent: cp -r skills/* ~/.pi/agent/skills/
$# Claude Code: cp -r adapters/claude-code/commands/* .claude/commands/

What's in the box

  • 5 skills
    eval funnel, design, scorers, judge, workflow
  • 5 tool adapters
    pi-agent, Claude Code, Cursor, Cline, generic
  • 2 worked examples
    eval design + regression detection
  • Methodology
    vendor-agnostic
  • Calibration patterns
    for LLM-as-judge
  • Golden set construction
    from real failures

Everything in the pack

What you get the moment your payment clears.

llm-evaluation skill — master orchestrator + the eval funnel
eval-design skill — what to measure, criteria writing, golden sets from real failures
deterministic-scorers skill — schema checks, regex, exact-match, citation-presence
llm-as-judge skill — when LLM judges work, calibration, prompt design, bias mitigation
eval-driven-dev skill — eval-first PRs, baseline tracking, regression detection
5 tool adapters — pi-agent, Claude Code, Cursor, Cline, generic
2 worked examples — eval design + regression detection
Methodology, not vendor lock-in — works with Braintrust, LangSmith, Arize, Helicone, or your own

Adapters included

These skills were authored for pi-agent, but we include drop-in adapters for the major AI coding tools.

ToolAdapter locationWhat you get
pi-agentadapters/pi-agent/SKILL.md drop-in
Claude Codeadapters/claude-code/.claude/commands/*.md + CLAUDE.md
Cursoradapters/cursor/.cursorrules rule sets
Cline / Continueadapters/cline/System-prompt rules
Aider / genericadapters/generic/INSTRUCTIONS.md snippets

Same playbook, different packaging. The adapters translate the methodology into the format each tool expects. Install once, swap agents without re-learning the method.

Full documentation

Everything in the package, every prop, every pattern.

LLM Evaluation Engineering

The methodology that catches regressions before users do.

Built and battle-tested. 5 skills covering the eval mindset, deterministic scorers, LLM-as-judge calibration, golden sets, and eval-driven development. Adapters for pi-agent, Claude Code, Cursor, Cline, and any tool that reads a system-prompt file.

What's Inside

SkillPurpose
llm-evaluationMaster orchestrator — the eval funnel, when to use what, the eval mindset
eval-designDesigning evals: what to measure, how to write criteria, golden set construction
deterministic-scorersNon-LLM-as-judge: schema checks, regex, exact-match, citation-presence
llm-as-judgeWhen LLM judges make sense, calibration, prompt design, pitfalls
eval-driven-devWorkflow: eval-first PRs, baseline tracking, regression detection

Quick Start

# 1. Extract
tar -xzf llm-evaluation-0.1.0.tgz
cd llm-evaluation-0.1.0

# 2. Pick your agent host
ls adapters/

# pi-agent (recommended)
mkdir -p ~/.pi/agent/skills
cp -r skills/* ~/.pi/agent/skills/
# /reload in pi to pick them up

# Claude Code
mkdir -p .claude/commands
cp -r adapters/claude-code/commands/* .claude/commands/
cat adapters/claude-code/CLAUDE.md >> CLAUDE.md

The Core Idea

The single sentence that ties this whole pack together:

All you get from using prefab evals is you don't know what they actually do. — Hamel Husain

The pack teaches the eval funnel: deterministic checks first (cheap, fast, reliable), LLM-as-judge second (expensive, slower, needs calibration), human review last (slowest, most expensive, ground truth). Every eval system should walk the funnel — most ship the wrong way.

The 5 Skills in Detail

llm-evaluation — The orchestrator

  • The eval funnel (deterministic → LLM → human)
  • When to use what
  • The eval-driven development loop

eval-design — What to measure

  • Writing criteria that mean something
  • Golden set construction from real failures
  • Coverage: happy path, edge cases, adversarial cases

deterministic-scorers — The cheap layer

  • Schema validation
  • Exact-match, regex, substring
  • Citation-presence, refusal detection
  • These are the workhorses — use them first

llm-as-judge — The expensive layer

  • When LLM judges make sense (quality, nuance)
  • Calibration against human ground truth
  • Prompt design for judges
  • Bias mitigation

eval-driven-dev — The workflow

  • Eval-first PRs (eval before feature)
  • Baseline tracking
  • Regression detection (vs prior baseline)
  • CI integration

Adapters Included

Works in every major AI coding tool:

ToolAdapter locationWhat you get
pi-agentadapters/pi-agent/SKILL.md drop-in
Claude Codeadapters/claude-code/.claude/commands/*.md + CLAUDE.md fragments
Cursoradapters/cursor/.cursorrules rule sets
Cline / Continueadapters/cline/System-prompt rules
Aider / genericadapters/generic/INSTRUCTIONS.md snippets

Updates

Free for life. Re-download for new scoring patterns, new LLM-as-judge techniques, new baseline tracking approaches.

What This Isn't

  • Not a vendor replacement — works with Braintrust, LangSmith, Arize, Helicone
  • Not eval-as-a-service — methodology you apply to your own evals
  • Not prebuilt eval sets — teaches you to build your own from real failures

What's in the download

llm-evaluation-0.1.0/
├── README.md
├── OVERVIEW.md
├── METHODOLOGY.md
├── INSTALL.md
├── LICENSE.md
├── package.json
├── skills/                ← 5 SKILL.md files
│   ├── llm-evaluation/
│   ├── eval-design/
│   ├── deterministic-scorers/
│   ├── llm-as-judge/
│   └── eval-driven-dev/
├── adapters/              ← drop-in for AI coders
│   ├── pi-agent/
│   ├── claude-code/
│   ├── cursor/
│   ├── cline/
│   └── generic/
├── examples/              ← 2 worked examples
└── docs/                  ← diagrams + decision trees

Common questions

Things people ask before buying.

Is this a Braintrust / LangSmith / Helicone replacement?

No. The pack is methodology — what to measure, how to design evals, how to detect regressions. The vendors are infrastructure — they run the evals. Use both: this pack for the playbook, your vendor for execution.

How is this different from Hamel Husain's Maven course?

Hamel's course is the gold standard. This pack distills the same methodology into a SKILL.md bundle you can drop into your agent. The course is 6 weeks of video; the pack is the same playbook, condensed, agent-loadable.

I don't have any evals yet. Where do I start?

Start with eval-design. It teaches you to write criteria, build a golden set from real failures, and pick the right granularity. From there, deterministic-scorers first (cheap), LLM-as-judge second (calibrate against human).

£40 — lifetime

One purchase. Yours forever.

Pay once via Stripe. Get the complete package instantly. Free updates for the same major version. Commercial license included.

Secure payment via Stripe · Instant download · 30-day money-back if it doesn't save you time