How to evaluate AI systems: a practical guide to evals

An eval is a repeatable measurement of whether an AI system does what you need on inputs you care about. You define a set of cases, run the system on them, and score the outputs against an explicit definition of correct. The output of an eval is a number or a distribution you can compare across versions, not a feeling that the demo looked good.

Evals are the backbone of reliable AI because the alternative is shipping on vibes. A language model is nondeterministic, sensitive to prompt wording, and easy to regress with a small change upstream. Without a measurement you trust, every change is a coin flip and every incident is a surprise. With one, you can change a prompt, swap a model, or tighten a retrieval step and see the effect before it reaches a user.

The useful mental model is a spectrum. On one end sit fast, cheap, deterministic checks you run on every commit. On the other sit slow, expensive, human-graded reviews you run occasionally. A serious system uses both, plus a third layer that watches real traffic in production. The skill is deciding what to measure and how much confidence each layer buys you.

Offline evaluation runs against a fixed dataset in a controlled setting, before anything ships. You know the inputs, you have decided what correct looks like, and you can rerun the exact same suite a hundred times. Offline evals are how you gate a release: a new version has to match or beat the current one on the suite before it goes out.

Online evaluation measures the system on live traffic after it ships. Real inputs are messier and more varied than any dataset you can assemble in advance, and user behavior tells you things a static test cannot: which answers get thumbs-down, where users rephrase and retry, where they abandon. Online signals catch the failure modes you did not think to write a test for.

Neither replaces the other. Offline evals give you fast, controlled, reproducible feedback but only cover cases you anticipated. Online evals cover reality but arrive late, are noisy, and often lack a ground-truth label. The pattern that works is a loop: real failures found online become new offline test cases, so the suite grows toward the distribution your users actually send.

A golden dataset is a curated set of inputs paired with known-good outputs or a clear grading rule. It is the reference the whole offline suite depends on, so its quality caps the quality of everything downstream. Build it from real or realistic inputs, cover the common cases and the known hard edges, and keep it version-controlled so a change to the dataset is a reviewable event, not a silent shift.

The metric has to match the task, because there is no single score that means good. Retrieval steps are measured with recall and precision at k, and whether the right document made it into context at all. Classification and extraction use accuracy, precision, recall, and F1 against labels. Freeform generation is harder: exact match is too strict, and surface-overlap scores like BLEU or ROUGE correlate poorly with whether the answer is actually correct or useful.

Pick the metric before you build the dataset, not after you see the scores. Deciding what counts as success up front keeps the evaluation honest and stops the common failure of choosing whichever metric happens to make the current version look best.

For freeform outputs where writing a rule is impractical, a common approach is to use a language model to grade the output against a rubric. LLM-as-judge scales human-style judgment cheaply: you give the judge the input, the response, and a scoring rubric, and it returns a score with a reason. Done carefully, it correlates well enough with human ratings to be useful as a fast signal.

It also has real pitfalls, and treating the judge's score as ground truth is the most expensive mistake. Judges show position bias, favoring the first option in a pairwise comparison. They show verbosity bias, rating longer answers higher regardless of quality. They can favor outputs from models similar to themselves, and they drift when the underlying judge model is updated, which silently moves your baseline.

The mitigations are concrete. Use pairwise comparison rather than absolute 1-to-10 scores, which humans and models both score inconsistently. Randomize option order and average over both orderings to cancel position bias. Write a specific rubric with examples rather than asking for a vague quality rating. Most importantly, validate the judge against a human-labeled sample and measure their agreement, so you know how much to trust it before you rely on it to gate releases.

Regression testing is the offline suite run as a gate. Every prompt change, model swap, dependency bump, or retrieval tweak runs against the golden dataset in CI, and the change ships only if scores hold. This is what stops the quiet decay where a fix for one case breaks three others nobody noticed. Track scores over time so a slow slide is as visible as a sudden drop.

Human review stays in the loop for what automation cannot judge reliably: nuanced correctness, tone, safety, and the cases the judge disagrees with itself on. You do not review everything. You review a sampled slice, the low-confidence cases, and anything users flagged, and you feed those labels back into both the golden dataset and the judge validation set.

Production monitoring closes the loop on live traffic. Watch operational metrics (latency, error rate, cost, token usage) alongside quality signals (user feedback, retry and abandonment rates, guardrail triggers, and cheap automated checks run on a sample of real responses). Alert on drift, because input distributions shift and a system that scored well last quarter can quietly degrade. The failures you catch here become tomorrow's offline test cases.

Start smaller than feels rigorous. Twenty to fifty real cases with clear expected outcomes beats a thousand synthetic ones, and you can build them in an afternoon. Run them on your current system to set a baseline, then never change the system without rerunning them. That single discipline, a baseline you defend on every change, is most of the value.

From there the practice grows in layers: automated metrics where a rule fits, a validated LLM judge where it does not, human review on a sampled slice, and monitoring on production traffic that feeds new cases back into the suite. This is exactly the harness that separates a system you can operate from a demo that happened to work once. It is the layer most teams skip, and where Stallwart builds the evaluation and monitoring in as part of the system, not as an afterthought, so reliability is measured rather than hoped for.