takeaway notes

How LLM Evals Work

10 min · ai engineering, evals, testing, llm judges

Watch on YouTube

After this you can stop judging prompt changes by trying three examples. You can build an eval from a dataset of real failures and edge cases, your actual app as the task, and a grader that scores each answer. You know to use code graders wherever a plain assertion works, an LLM judge with a rubric for the fuzzy parts, and humans to calibrate the judge. You can run each case several times and read the pass rate, gate pull requests on the score, and feed production failures back into the dataset.

Concepts

The vibes check

Theory

Normal tests check code, but a model's answers aren't code, so a prompt change can make things worse while nothing crashes and every test stays green. You try three examples; your users send thousands. A change that fixes one case can quietly break others, and there is no stack trace for slightly worse.

Practical example: The support bot that forgot goodbye

You fix how a support bot answers the refund question, and now it forgets how to say goodbye. Nobody notices for days, because the wrong answer comes in a very polite tone.

Anatomy of an eval

Theory

An eval has three parts: a dataset, a task and a grader. The dataset is a list of inputs with what a good answer looks like; the task is your actual app; the grader scores each answer as pass, fail or a number. Average the scores and you get one number to compare before and after a change.

Practical example: One row for a refund bot

A customer asks: can I get my money back after thirty days? The expected answer: no, the window is fourteen days (a made-up policy), but here is how to contact support. Run the bot, grade the reply, write down pass or fail, then loop over a few hundred rows.

dataset = [
    {"input": "Can I get my money back after 30 days?", "must_mention": "14 days"},
]

def call_bot(question):  # stub: put your real app here (same prompt, model, code)
    return "No, the refund window is 14 days, but here is how to contact support."

def grade(reply, row):  # pass or fail
    return row["must_mention"] in reply

scores = [grade(call_bot(row["input"]), row) for row in dataset]
print(sum(scores) / len(scores))

The golden dataset

Theory

Test cases come from your failures, not your imagination: every time a user hits a bad answer, save that input. Start with twenty or thirty real, messy examples, then add weird inputs, boring easy cases, and cases where the right answer is I don't know. Model providers' own guides say a bigger set graded automatically beats a small set graded by hand.

Practical example: Grow the set from real traffic

Keep the typos and the rage, then add an empty input, a giant input and a user typing in all caps. Keep a few easy cases too: they tell you when you broke something basic.

Graders

Theory

There are three kinds of grader, and you want the cheapest one that works. Code graders (exact match, a regex, a schema check) are fast, free and reliable. An LLM judge scores the fuzzy parts against a rubric, and a human checks that the judge isn't talking nonsense.

Practical example: Code first, judge for the squishy parts

Check the format with code. For politeness or faithfulness, hand a judge the question, the answer and a rubric (did it cite the policy, did it stay on topic) and ask for a short reason plus a score; the reason makes failures easier to debug.

import json, re

MAX_CHARS = 500  # pick your own limit

def code_grade(reply):
    try:
        json.loads(reply)  # does it parse as JSON?
    except ValueError:
        return False
    has_order = re.search(r"ORD-\d+", reply) is not None  # mentions the order number?
    return has_order and len(reply) <= MAX_CHARS  # short enough?

print(code_grade('{"answer": "Order ORD-1234 ships today"}'))

Judge biases

Theory

Researchers have measured biases in LLM judges: position bias (favoring whichever answer came first), verbosity bias (longer answers win for being long) and self-preference (rating answers in its own style higher). These come from the 2023 MT-Bench paper, and their size varies by model and prompt. Judges are still useful; you calibrate them like any other tool.

Practical example: Cheap fixes

Swap the order of the two answers and score twice, give the judge a clear rubric, and spot-check it against human labels. Twenty human-labeled examples is a good start; if the judge disagrees a lot, fix the rubric first.

Run it more than once

Theory

Models sample, so the same input can give a different answer from run to run. Even with the dials turned down, providers don't promise identical outputs (a seed setting is best-effort, not a guarantee). So run each case several times and look at the rate, not one lucky pass.

Practical example: 98 versus 42, same bot

A bot that gets a case right 3 times out of 4 scores about 98 percent if any one of three tries may pass, and about 42 percent if all three must pass (assuming independent tries). For a code helper one good try may be enough; customer answers need it every time.

p = 0.75  # the bot gets this case right 3 times out of 4
tries = 3

any_pass = 1 - (1 - p) ** tries  # need just one pass out of three
all_pass = p ** tries            # need all three to pass

print(round(any_pass, 2), round(all_pass, 2))  # 0.98 0.42

CI and production

Theory

Run evals on every pull request that changes a prompt, and fail the build when the score drops below your baseline. Keep the full suite for nightly runs and a smaller fast subset for every pull request. Save scores per model version, because providers update models and your numbers can move on their own.

Practical example: Close the loop

Sample real production traffic, grade it, and feed the failures back into the dataset. Fail in production, save the case, never fail that way again.

Checklist

Gotchas

References