Eval sets: how to evaluate LLM output

How LLM output is evaluated

LLM output is evaluated with an eval set: a fixed collection of real cases from your own operation, each carrying an expected outcome, run against the system on demand and re-run after every change. Evaluation is then a comparison - what the system produced against what the case says it should produce - so “is the output good?” stops being an opinion and becomes a list: which cases pass, which fail, and what changed since the last run. Two things that feel like evaluation are not. Reading a handful of outputs and judging them by eye is sampling, and the next section is about the failures it reliably misses. And a public benchmark score is a fact about a model on someone else’s cases, not about your system on yours - a model that tops a leaderboard can still fail your invoice formats every day. The eval set is also the reason this page sits in this library at all: it is one of the tests a client team must be able to re-run alone, inside the handover method these tests belong to, because a team that cannot evaluate the system’s output cannot safely change the system.

The failures human review misses are authentication and data-sanity failures

The case for evals over human review is not speed. It is that the two see different failures, and human review sees the wrong ones. A person reads LLM output for plausibility - tone, fluency, whether the answer looks right - and LLM output is fluent even when it is wrong, so plausibility is exactly the property that survives a real failure. In the studio’s experience, the failures humans most often miss are authentication and data-sanity failures: the system is quietly working against the wrong account or with expired access, or the data feeding it is stale, incomplete or from the wrong source, and the output reads as well as it ever did. A reviewer approves it because nothing about the text says anything is wrong; the defect is upstream of the words. An eval catches these because it does not read for plausibility - it checks the output against an expected outcome computed from known inputs, and an answer built on the wrong account or bad data does not match, however well it reads. This is why the position here is that strict, well-designed evals covering the edge cases are what keep a production system running: they are the only reviewer that never gets talked into a fluent wrong answer.

What a handed-over eval docket contains

An eval set worth handing over is unique to the company it tests. It is built from that company’s needs and workflows - its documents, its customers, its failure history - which is why a docket cannot be bought generic or copied from another engagement, and why receiving one is receiving something no benchmark supplies. The typical docket covers all the major and minor use cases plus the edge cases, and it is backtested on verified company data: the expected outcomes are checked against records the company knows to be true, so the bar the system is measured against is real before the system is measured against it.

The last row is the one that makes the other three trustworthy. An eval set whose expected outcomes were never verified against real data tests the system against a guess, and a system that passes it has proven nothing.

What LLM evals are, and what testing an LLM app involves

LLM evals, as practitioners use the term, are exactly the runs described above: the system’s output on known cases compared against expected outcomes, repeatable at will. LLM evaluation in the academic sense - benchmark suites, leaderboards, capability scores - answers a different question, which model to reach for, and answers it about models in general rather than your application in particular. The two are routinely confused because they share a word. Testing an LLM app is wider than either, because an application is more than its model: it authenticates against accounts, pulls data from sources, calls tools, and hands results to people or other systems. The deterministic parts of that pipeline are tested the way software has always been tested. The model-driven parts are where conventional testing breaks down - the same input can produce differing output, and there is no compiler to object to a bad change - and the eval set is the instrument that fills the gap. In a working system the two meet in one habit: nothing that changes the system’s behaviour goes to production without the docket run against it, which is what makes change safe rather than brave.

The docket changes hands at handover, and keeps growing after it

Within a complete AI handover the eval set is one artefact among several - the full list sits with the artefact inventory this one belongs to - but it is the one this studio treats as inseparable from the system itself, because a system without its docket can be operated but not safely changed. Handing it over means the client team receives the cases, the expected outcomes and the means to run them in their own environment, not a report of the last run. And the handover is of a living thing, not a finished one: workflows change, new edge cases arrive with new customers and new data, and each production surprise that was not in the docket belongs in it by the following week. The team that owns the system owns the docket, and the docket’s growth is the record of that ownership being exercised. A buyer evaluating a vendor can use this as a plain test before any build starts: ask how the system’s output will be evaluated, who will hold the eval set at the end, and what data the expected outcomes will be verified against. A vendor with real answers is offering a system you can run; a vendor without them is offering output you will have to keep taking on trust.

Foxnut Studios works on briefs like this one from Bengaluru and Paris. If you want the shape of that before you talk to anyone, here is how an AI engagement is scoped and priced.