Evaluating AI Systems

You cannot improve what you do not measure. When a feature is powered by a foundation model, “it seemed to work when I tried it” is not evidence — it’s a vibe. This lesson is the door into the AI-engineering craft, and the craft begins here: an AI engineer measures everything. Evals are how you prove a feature is good, catch it the moment it stops being good, and ship changes without crossing your fingers.

Why “it looks good” is not good enough — the eval gap

☺ Like you’re 10: Trying a chatbot once and saying “nice, it works” is like tasting one spoon of soup and declaring the whole pot perfect. One spoon can’t tell you the pot is too salty, or that yesterday’s pot was better.

When you build a normal feature, you know it works because you wrote a test: add(2, 2) returns 4, always, forever. Foundation-model features break that comfort in two ways. First, the model is non-deterministic — ask it the same thing twice and you may get two different wordings. Second, and worse, “correct” is fuzzy. There’s rarely one right answer to “summarize this ticket” or “draft a friendly reply.” There are many good answers, many mediocre ones, and many confidently-wrong ones that read just fine.

Contrast this with the classical ML you may have met: there, a model predicts a house price or a spam label, and you have the true answer sitting in a column, so you compute a clean number — accuracy, precision, RMSE, loss. Even evaluating a fine-tune leans on a held-out set with known-good labels. Generative output has no such column. “Is this summary good?” doesn’t reduce to a subtraction. That gap — between how obviously-scorable classical ML is and how slippery generative quality is — is the eval gap, and closing it is the whole job of this lesson.

The failure mode this creates is seductive. You demo the feature, it dazzles, everyone ships. Then real users send inputs you never tried, the model drifts when the provider updates it, someone tweaks the prompt to fix one bug and silently breaks three others — and nobody notices, because there was never a scorecard. Vibes don’t catch regressions. Measurement does.

The eval dataset — golden sets & regression sets

☺ Like you’re 10: A golden set is your special box of practice questions where you already know the best answers. Every time you change the recipe, you re-run the same box — so you’re comparing apples to apples, not to whatever you happened to type today.

An eval needs something fixed to run against. That’s the eval dataset (often called a golden set or golden dataset): a curated collection of representative inputs, each paired with what a great output looks like — an exact expected answer, a reference to compare against, or at minimum a rubric describing what “good” means. The golden set is the ruler you hold up to every version of your system.

Where do the examples come from? The best source is real production traffic, not your imagination:

• Seed from reality. Log real user inputs (scrubbed of anything sensitive) and pull a representative sample — common cases and the weird edges. Inputs you’d never dream up are exactly the ones that break things.
• Cover the distribution. Deliberately include easy, hard, ambiguous, adversarial, and “no good answer exists” cases. A golden set of only easy questions gives you a flattering, useless score.
• Curate the expected output. A human (ideally a domain expert) decides the ideal answer or the acceptance criteria for each item. This labeling is real work — but it’s the asset that makes every future eval possible.

The second flavor is the regression set. Every time you find a bug — the model mishandled a date, leaked a system instruction, botched a currency — you distill that failure into a small, permanent test case and add it to a set that must always pass. Regression tests lock in fixes: they’re your guarantee that a bug you squashed last month can never quietly crawl back in when you change a prompt or swap a model.

Graders — how a machine scores a fuzzy answer

☺ Like you’re 10: A grader is the thing that marks the test. Some questions have one right answer, so a simple checker works. Others need judgment, so you either write a checklist — or you ask a second smart friend to grade the first friend’s work. But that grader-friend has quirks you have to watch.

Once you have inputs and expected outputs, you need a grader: something that turns each output into a score. Graders come in a spectrum, from rigid and cheap to flexible and tricky.

• Exact / rule-based checks. When there’s a crisp right answer, code it. Does the output exactly equal the expected string? Match a regex? Parse as valid JSON? Contain the required field? Fall under a length limit? These are fast, free, deterministic, and 100% trustworthy — always prefer them when the question allows. Anything you can check with a rule, you should. (For grading structured output specifically, see structured outputs; for retrieval quality, retrieval engineering has its own metrics.)
• Rubric grading. When the answer is open-ended, break “good” into a checklist: Does it answer the question? Is it grounded in the sources? Is the tone right? Under 100 words? Each criterion scores separately, then you sum. Rubrics turn a mushy “is it good?” into several sharper questions — and each of those is easier to grade consistently, whether by a human or a machine.
• LLM-as-judge. When you can’t write a rule and can’t afford a human on every run, you use a strong model as the grader: hand it the input, the output, and a clear rubric, and ask it to score. This scales judgment cheaply and works surprisingly well — but a model judge has biases you must actively defend against.

The two biases that bite hardest with LLM-as-judge:

• Position bias. When comparing two answers, a judge tends to favor whichever one it sees first (or sometimes last), regardless of quality. Defend by running each comparison both ways — A-then-B and B-then-A — and only counting a win when the judge agrees in both orders.
• Verbosity bias. Judges over-reward longer, more elaborate answers even when a crisp one is better. Defend by putting explicit length/conciseness criteria in the rubric, and by not letting the judge see which answer is longer as a proxy for “more effort.”

A powerful trick is pairwise comparison: instead of asking “rate this answer 1–10” (models are wobbly at absolute scores), ask “which of these two answers is better, A or B?” Relative judgments are far more reliable than absolute ones. You run your new version against the old one head-to-head across the golden set and report a win rate. And crucially — you calibrate the judge itself: have humans grade a sample, then check the judge agrees with them. If your judge doesn’t match human judgment on cases you already trust, its scores are noise.

# PSEUDOCODE — an LLM-as-judge grader with position-bias defense
def judge_pairwise(question, answer_a, answer_b, rubric):
    # ask BOTH orders to cancel position bias
    v1 = judge_model(prompt(question, rubric, first=answer_a, second=answer_b))
    v2 = judge_model(prompt(question, rubric, first=answer_b, second=answer_a))
    if v1.winner == "first" and v2.winner == "second":
        return "A wins"          # judge agreed A is better, both orders
    if v1.winner == "second" and v2.winner == "first":
        return "B wins"
    return "tie / inconsistent"  # disagreed → don't trust it

# CALIBRATE before you rely on it
assert agreement(judge_model, human_labels) > 0.8   # else the judge is noise

What to measure — the metrics that matter

☺ Like you’re 10: A good answer isn’t just “right.” It also has to stick to the facts you gave it, come out in the shape you asked for, stay safe, and not cost a fortune or take forever. So you keep score on all of those at once, not just one.

“Is it good?” is really several questions. A production eval tracks a handful of dimensions, and it’s a mistake to collapse them into one number. The core set:

Faithfulness (also called groundedness) deserves special attention because it’s the direct measure of hallucination. In any retrieval or document-based feature, the question isn’t just “is the answer plausible?” but “does every statement trace back to the sources we provided?” You grade it by giving the judge the answer and the source context and asking it to flag any claim that isn’t supported. High faithfulness is the number that tells you your RAG system is trustworthy rather than merely fluent.

And note the last two rows. Cost and latency are first-class metrics, measured on every eval run right alongside quality. A version that’s 1% more accurate but twice as expensive and twice as slow is often a worse engineering choice — and you can only see that tradeoff if you’ve been measuring all three together. The cost & latency lesson goes deep on tuning them.

Offline vs online — evals in CI, then A/B tests in production

☺ Like you’re 10: First you test the sandwich in your kitchen before anyone eats it (offline). Then, once it passes, you serve it to a few real customers and watch whether they actually like it more than the old one (online). Both matter — the kitchen test can’t tell you everything real people will do.

There are two arenas for evals, and mature teams use both.

Offline evals run against your fixed golden set, before any user is involved — typically in CI (continuous integration), the automated pipeline that runs on every code or prompt change. This is where evals become a release gate: you set pass-rate thresholds (“task success ≥ 90%, faithfulness ≥ 95%, zero regression failures”) and the pipeline blocks the deploy if a change drops below them. This is the single highest-leverage habit in AI engineering — it turns “I hope this prompt change didn’t break anything” into a hard, automatic guarantee, exactly like a unit-test suite gates normal code.

Online evals happen once the change is live, because the fixed golden set can never fully predict the messy real world. The tools here:

• Canaries. Roll the change out to a tiny slice of traffic first and watch its metrics. If error rates, cost, or latency spike, you roll back before most users ever see it.
• A/B tests. Send some users the new version and some the old, then compare real outcomes — thumbs-up rate, task completion, escalation to a human, retention. This measures what the golden set can’t: whether real people are actually better served.
• Human review of production. Continuously sample real live outputs and have humans (or a calibrated judge) grade them. This is how you catch drift and discover the new failure modes that then become fresh golden-set and regression cases — closing the loop.

Offline tells you a change is safe to try; online tells you it’s actually better. You need both. The operational machinery for canaries, rollouts, and monitoring lives in production & ops, and evals are a core stage of any real AI pipeline.

Pitfalls — how evals quietly lie to you

☺ Like you’re 10: Even the scorecard can trick you. If you practice on the exact same ten questions forever, you’ll memorize them and think you’re a genius. If the grader has a favorite, it’ll pick wrong. And ten questions is too few to trust anyway. So you have to check the checker.

Evals are powerful, but a badly-built eval gives you false confidence — which is worse than no eval at all. Watch for these:

• Overfitting to the eval set. If you tune your prompt over and over against the same golden set, you eventually optimize for those exact examples rather than the real task — the classic overfitting trap from classical ML, reborn. Keep a held-out set you tune less against, and refresh the golden set with new real traffic so it can’t go stale.
• Leaking test items into the prompt. If an example (and its ideal answer) from your golden set ends up in your few-shot examples or system prompt, the model has effectively seen the answer key. Your score soars and means nothing. Keep eval data strictly separate from anything the system is given at runtime.
• Trusting an uncalibrated judge. An LLM-as-judge you never checked against humans may be confidently, systematically wrong — favoring long answers, missing subtle hallucinations, rewarding a house style. Always calibrate the judge against human labels before you let it gate anything.
• Tiny datasets. A ten-item golden set can swing wildly on noise; a single flaky example flips your pass rate. You need enough examples that the score is stable and a real regression stands out from run-to-run wobble. Small sets give you precise-looking numbers that mean nothing.

The meta-lesson is Timmy’s creed: be suspicious of your own measurements. An eval that always passes may be too easy, leaking answers, or measuring the wrong thing. A good eval occasionally fails and catches real problems — that’s how you know it’s doing its job.