Evaluating and Testing Claude Applications

A traditional unit test assumes a function returns the same output for the same input. Claude doesn't work that way — so "it looked right when I tried it" is not a test suite.

Why non-determinism breaks the old testing playbook

☺ Like you're 10: A vending machine always gives the same chips for B4. Claude is more like asking a friend the same question twice — the answer shifts a little each time, so you can't test it the vending-machine way.

A classic test asserts assert add(2, 2) == 4 and is done forever. A call to Claude can return slightly different wording on every run, even at the same temperature, and a single prompt tweak — a rephrased instruction, a reordered example, one new system sentence — can silently shift behavior on inputs nobody happened to re-check. Nothing crashes, no exception fires. The app just quietly gets a little worse at edge cases.

That's why evaluation ("evals") is a first-class part of building with Claude, not an afterthought. An eval suite turns "I think this prompt change is fine" into "I have evidence this change didn't regress task fidelity, tone, or safety on a representative sample of real inputs." Anthropic frames success criteria around SMART properties — Specific, Measurable, Achievable, Relevant — applied across whichever dimensions matter for your app: task fidelity, consistency, relevance and coherence, tone and style, privacy preservation, context utilization, latency, and price. Most real applications need to track more than one of these at once.

Building an eval dataset that reflects reality

☺ Like you're 10: Don't just write down the easy quiz questions you already know the answer to — sneak in the weird ones too, like the blank page and the trick question, because those are the ones most likely to trip things up later.

An eval dataset isn't a handful of prompts you personally liked the answer to. It's a set of representative inputs — drawn from real usage where possible — paired with what a good answer looks like. Critically, "what a good answer looks like" is usually a set of expected properties, not one exact string, because Claude can phrase a correct answer many different ways.

{
  "id": "refund-policy-001",
  "input": "Can I get a refund if I bought the wrong size?",
  "expected_properties": [
    "States the actual return window from the policy, not a made-up one",
    "Does not promise a refund method the store doesn't support",
    "Tone is helpful and non-defensive",
    "Asks for order number only if it isn't already provided"
  ]
}

Two design principles are worth internalizing. First, mirror your real-world task distribution — don't just write the easy cases you'd expect to pass. Second, deliberately include edge cases: irrelevant or nonexistent input, unusually long input, and harmful or ambiguous input. Those are exactly the inputs a prompt regression is most likely to break.

On volume: automated, slightly-noisier evals run at scale beat a handful of hand-graded examples, because statistical signal from more cases outweighs the precision of grading each one perfectly. A commonly cited starting point is roughly 50-100 cases for signal you can trust, scaling toward the low hundreds for a path that's critical to get right. You don't have to write every case by hand — seed a small set yourself and ask Claude to generate additional, diverse test cases from it.

Grading approaches: exact match to LLM-as-judge

Once you have a dataset, you need something to score outputs against it automatically. Anthropic's testing guidance lays out several grading patterns, from cheap and deterministic to flexible and subjective:

The simplest of these is exact match:

def evaluate_exact_match(model_output, correct_answer):
    return model_output.strip().lower() == correct_answer.lower()

For anything subjective — tone, helpfulness, adherence to a style guide — you can't string-match your way to a score. That's where LLM-as-judge comes in: use a second call to Claude to grade the first call's output against a rubric.

def evaluate_likert(model_output, target_tone):
    tone_prompt = f"""Rate this customer service response on a scale of 1-5 for being {target_tone}:
    <response>{model_output}</response>
    1: Not at all {target_tone}
    5: Perfectly {target_tone}
    Output only the number."""
    response = client.messages.create(
        model="claude-opus-5",
        max_tokens=50,
        messages=[{"role": "user", "content": tone_prompt}],
    )
    return int(next(block.text for block in response.content if block.type == "text").strip())

async function evaluateLikert(modelOutput, targetTone) {
  const tonePrompt = `Rate this customer service response on a scale of 1-5 for being ${targetTone}:
<response>${modelOutput}</response>
1: Not at all ${targetTone}
5: Perfectly ${targetTone}
Output only the number.`;

  const response = await client.messages.create({
    model: "claude-opus-5",
    max_tokens: 50,
    messages: [{ role: "user", content: tonePrompt }],
  });

  const textBlock = response.content.find((block) => block.type === "text");
  return parseInt(textBlock.text.trim(), 10);
}

Notice this judge is graded with claude-opus-5 — deliberately a different, more capable model than whatever cheaper model might be generating the responses under test. That's documented best practice, covered below. Other best practices for LLM-based grading: give the judge a clear, detailed rubric with examples of each score level; let the judge reason before it answers, then constrain the final output ("output only the number," or "respond with correct or incorrect") so scores are easy to parse and aggregate; and validate the grader's own reliability — check that it scores consistently — before trusting it at scale.

The Console's Evaluate tool in Workbench

Anthropic's Console ships a built-in Evaluate tool inside the Workbench for exactly this workflow, no external tooling required to get started. It requires at least one {{variable}} in your prompt template, which becomes the slot your test cases fill in. You can import test cases from a CSV or use the Console's "Generate Test Case" feature to have Claude draft more from a seed, run the entire suite in one click, and compare outputs from multiple prompt versions side by side against the same test set — which is precisely how you catch a regression before it ships. Built-in grading uses a 5-point scale.

Regression testing with promptfoo

☺ Like you're 10: Think of it as a spelling-test machine for prompts — you feed it a stack of questions and the expected properties, it runs both the old and new prompt through every question, and it tells you exactly where the new one did worse.

For eval suites that live in version control and run in CI on every prompt change, promptfoo is the tool most teams reach for. It's an open-source (MIT-licensed, no paid tier), local-first CLI and library built for test-driven LLM development. It runs a config-driven matrix of prompts across models and providers — 60+ providers are supported, including Anthropic — scores each combination against assertions you define, and can output results to the terminal, a web UI, or a CI/CD pipeline. It also has dedicated workflows beyond plain regression testing, including RAG evaluation and red-teaming (jailbreak and adversarial-input detection, plus compliance-oriented testing).

The promptfoo workflow mirrors the eval dataset approach above: define test cases that capture your core use cases and known failure modes, point the config at your prompt(s) and provider(s), run the eval, analyze results, and iterate. A minimal config sketch (promptfoo accepts YAML or JSON) looks like this:

{
  "prompts": ["prompts/support_reply.txt"],
  "providers": ["claude-sonnet-5", "claude-haiku-4-5"],
  "tests": [
    {
      "vars": {
        "ticket": "My order #4521 never arrived and it's been two weeks."
      },
      "assert": [
        { "type": "contains", "value": "#4521" },
        {
          "type": "llm-rubric",
          "value": "Apologizes for the delay, does not promise a delivery date, and offers a concrete next step (refund or reship)."
        }
      ]
    },
    {
      "vars": { "ticket": "" },
      "assert": [
        { "type": "llm-rubric", "value": "Asks the customer to describe their issue rather than guessing at one." }
      ]
    }
  ]
}

The second test case here is an edge case — an empty ticket — the kind of input that's easy to forget by hand but trivial to include once it's just another row in a config file. Run the suite from the same integrated terminal you're already using for the rest of the project:

npx promptfoo eval
npx promptfoo view   # open the results in a local web UI

Wire that promptfoo eval command into your CI pipeline (or a pre-merge hook) and a prompt change that regresses on any assertion fails the build before it reaches production, the same way a broken unit test would.

LLM-as-judge pitfalls, mitigations, and shipping safely

☺ Like you're 10: Would you trust a student to grade their own test fairly? Probably not — they'd be a little generous with themselves. Judge models have the same soft spot for their own family's answers.

Using a second Claude call to grade the first is powerful, but current research on LLM-as-judge documents real biases worth designing around. Judge models show self-preference bias — a tendency to favor outputs from themselves or their own model family even when applying an objective-looking rubric, with studies finding a judge can be roughly 50% more likely to incorrectly mark a failing rubric item as satisfied when it's grading its own output. Two related biases are position bias (favoring whichever answer appears first or second) and verbosity bias (conflating a longer answer with a better one).

Mitigations documented in the literature and in Anthropic's own guidance: use a different, ideally stronger, model as judge than the one being evaluated (Anthropic's stated best practice); run multi-judge ensembles across different model families, which improves alignment with human rankings without fully eliminating self-preference bias; randomize answer order across multiple permutations to cancel out position bias; and use calibrated, multi-dimensional rubrics that explicitly penalize redundancy and reward conciseness — shown to roughly halve prediction error compared to an uncalibrated rubric. None of this replaces spot-checking: periodically pull a sample of judge scores and read the underlying outputs yourself to confirm the judge and your own read of "good" still agree.

That pipeline — run the same test set through the old and new prompt, grade both the same way, compare — is the whole point. It's the difference between a prompt change that's a guess and one that's a decision backed by evidence.