Evaluating AI Systems
Short answer: Without evaluation you are not improving a system, you are redecorating it. "That looks better" is not a measurement, and it is the only signal most teams have.
Related guides: Designing an Agent Harness produces the run logs evaluation depends on. Building AI Skills covers the regression inputs every skill should carry.
The Problem With Vibes
The default workflow: tweak the prompt, run it once, read the output, decide it seems better, ship.
Three things break this:
- Output is stochastic. The same input twice gives different results. One sample tells you almost nothing.
- You only look at the case you were thinking about. The regression is in the case you were not.
- You are the worst possible judge of your own edit. You wrote it expecting improvement.
The fix is not elaborate. Thirty stored examples and a scoring script outperform intuition immediately.
Start With a Golden Set
A golden set is a fixed collection of inputs with known-good outputs or explicit acceptance criteria.
Building one:
| Step | Detail |
|---|---|
| 1. Collect real inputs | From actual usage. Invented examples are systematically too clean. |
| 2. Aim for 20–50 to start | Enough to catch regressions; small enough to actually build |
| 3. Stratify deliberately | ~60% typical, ~30% hard, ~10% should-refuse-or-flag |
| 4. Record expected output or criteria | Exact match where possible; a rubric where not |
| 5. Version it | It is a test fixture. It belongs in the repo. |
Include the ugly cases: empty inputs, contradictory instructions, data that does not exist, ambiguous requests. Those are where systems fail in production, and they are systematically missing from hand-written test sets.
The best source is your own failure log. Every time the system gets something wrong in real use, that input becomes a permanent test case. The set grows exactly where it needs to.
Choosing a Scoring Method
Match the method to the output type. Using a judge where exact match would work is slow and adds noise.
| Output type | Method | Notes |
|---|---|---|
| Classification, routing, extraction | Exact match / F1 | Cheap, deterministic. Use wherever possible. |
| Structured JSON | Schema validation + field match | Validate shape first, then values |
| Numbers pulled from a source | Compare against the source | Catches hallucination directly |
| Freeform prose | Rubric + LLM-as-judge | Slower, noisier, sometimes the only option |
| Retrieval quality | Recall@k / precision@k | Was the right document even fetched? |
| Agent trajectories | Did it call the right tools, in a sane order? | Process matters, not just the answer |
That last row is underused. An agent reaching the right answer by luck after twelve wrong tool calls is fragile even though the output scored well.
LLM-as-Judge, Done Carefully
Using a model to grade output is practical and genuinely useful — but it fails silently in specific, well-documented ways.
Make it work:
- Write an explicit rubric. Not "is this good?" — a numbered criteria list with what each score means.
- Score criteria separately. Accuracy, completeness, format, tone as separate fields. A single blended score hides which one moved.
- Require a reason before the score. Reasoning first, then the number, in that field order.
- Use a different model or a fresh context than the one that produced the output.
- Calibrate against humans once. Score 20 items both ways. If they disagree badly, fix the rubric before trusting the judge on anything.
Known biases to control for:
| Bias | Effect | Mitigation |
|---|---|---|
| Length | Longer answers score higher | Add a concision criterion |
| Position | First option in a pair wins | Randomize order; run both orderings |
| Self-preference | Models favor their own style | Judge with a different model |
| Confidence | Assertive prose beats hedged accuracy | Score accuracy against a source, not by feel |
An uncalibrated judge produces numbers that feel like data and are not. Validate it against human judgment before letting it gate anything.
Regression Testing
Once a golden set exists, wire it in.
change instruction / model / tool
↓
run full golden set
↓
compare against last baseline
↓
┌──────┴───────┐
score dropped score held or rose
↓ ↓
investigate promote to new baseline
Practical rules:
- Run the whole set, not the case you were fixing. The point is the cases you were not thinking about.
- Run 3–5 samples per input and record mean and variance. High variance is itself a finding.
- Store every run. Comparing two runs requires having both.
- Gate on the aggregate, review the diffs. A 2% aggregate move can hide one catastrophic case.
- Re-baseline deliberately, never automatically.
Run it in CI on every change to instructions, tools, or model version. The version fields in your run log are what let you attribute a change to a cause.
Calibration Against Reality
Offline scores measure agreement with your golden set. They do not tell you whether the system is right about the world.
For anything predictive, log the prediction and check it later:
| Step | Detail |
|---|---|
| 1. Log every prediction | With timestamp, inputs, and stated confidence |
| 2. Wait for the outcome | Whatever the real feedback loop is |
| 3. Compare, bucketed by confidence | Of the "80% confident" calls, how many were right? |
| 4. Adjust | Systematic over-confidence is correctable once measured |
A system claiming 90% confidence that is right 60% of the time is worse than one that says "uncertain," because people act on the first. This is the difference between a model output and decision support.
What to Track Over Time
| Metric | Question |
|---|---|
| Golden set score | Are we better than last release? |
| Score variance | Are we consistent? |
| Refusal / flag rate | Does it admit when it does not know? |
| Grounding rate | What share of claims cite a retrieved source? |
| Tool-call efficiency | Calls per successful completion — is it thrashing? |
| Stop-condition mix | What share end on budget rather than success? |
| Human override rate | How often does review change the output? |
| Cost per completion | Is quality improving faster than spend? |
Human override rate is the most honest single number available. If reviewers rewrite most outputs, the system is a draft generator regardless of what it scores.
Evaluation Checklist
- Golden set exists, versioned in the repo, 20+ real inputs
- Set includes hard cases and should-refuse cases
- Scoring method matches output type; exact match used where possible
- Any judge has an explicit rubric with per-criterion scores
- Judge calibrated against human scoring at least once
- 3+ samples per input; variance recorded, not just the mean
- Full set runs automatically on instruction, tool, or model change
- Baselines stored; promotion is a deliberate act
- Predictions logged for later comparison against outcomes
- Human override rate tracked
- Failures from production are added back to the golden set
Where This Leaves You
Four layers, each with its own failure mode: instructions, tools, the harness, and evaluation. A system missing any one of them will work in a demo and disappoint in production — and the layer that is missing tells you exactly which disappointment you will get.
Get in touch if you are standing up an eval pipeline and want to compare notes.