AI Evals: How to Actually Measure Agent Quality

Ship an AI agent to production without an eval suite and you're flying on vibes. The demo worked. The team liked the outputs in Slack. Then a prompt tweak that seemed harmless quietly drops task success from 91% to 74%, and nobody notices until support tickets pile up. This is the single most common failure mode in applied AI work today, and it's almost entirely preventable with a discipline that has existed in machine learning for decades but gets treated as optional when teams move fast with LLMs: evaluation.

"Evals" is the term of art for the practice of systematically measuring how well an AI system performs a task, using a fixed set of test cases and a repeatable scoring method. For simple chatbots, evals can be almost an afterthought — check a few outputs, eyeball the tone, ship it. For agents — systems that plan, call tools, maintain state across turns, and take actions with real consequences — evals are the only thing standing between "it seemed to work" and "it works."

This post is about what a real eval suite looks like for an agent, how it differs from evaluating a single LLM call, and where teams go wrong when they build one.

What an eval actually is

An eval, at minimum, has three parts:

1. A task set — a collection of representative inputs (and often expected outputs or acceptance criteria) that reflect what the agent will actually be asked to do.
2. A grading method — a way to decide, for each task, whether the agent's output was correct, acceptable, or a failure.
3. A metric — an aggregation of grades into a number or set of numbers you can track over time and compare across versions.

That's it structurally. The hard part isn't the scaffolding — it's building a task set that actually represents production, a grader that's trustworthy, and a metric that tells you something you can act on.

It's worth being precise about what evals are not. They are not unit tests in the traditional sense, because LLM outputs are non-deterministic and open-ended — there's rarely a single correct string to assert against. They are not A/B tests, because A/B tests measure real user behavior in production after you've already shipped, while evals are meant to catch regressions before shipping. And they are not the same as monitoring or observability, which tracks what's happening in live traffic rather than testing against a controlled set. Evals sit upstream of all three: they're the gate you use to decide whether a change is safe to expose to a monitoring pipeline or an A/B test in the first place.

Evals vs. testing vs. monitoring

Why agent evals are harder than model evals

Evaluating a single LLM call — "does this prompt produce a good summary?" — is already nontrivial. Evaluating an agent is a different order of difficulty, because an agent's output isn't one response, it's a trajectory: a sequence of decisions, tool calls, intermediate reasoning, and state changes that eventually terminate in some final result (or don't terminate at all).

A few things make this genuinely harder than classic model evaluation:

• Multiple valid paths. Two agents can both correctly answer "book me a flight to Chicago under $300" by calling different tools in a different order, checking different data sources, or asking different clarifying questions. Grading on the final answer alone misses whether the process was sound, safe, or efficient.
• Compounding errors. A small mistake early in a multi-step trajectory (misreading a date, misparsing a tool's return value) can cascade into a completely wrong final answer three steps later. A single pass/fail grade on the outcome tells you almost nothing about where the failure originated.
• Tool use correctness. Agents call external tools — APIs, databases, code execution, search. An eval needs to check not just whether the final answer was right, but whether the agent called the right tool, with the right arguments, and correctly interpreted the result.
• Non-termination and runaway loops. Agents can get stuck retrying the same failed action, burn through token/cost budgets, or never produce a final answer at all. This is a failure mode that doesn't exist for single-turn LLM calls and needs its own detection.
• Environment state. Many agent tasks change the state of a database, a filesystem, or a live system. Grading sometimes has to inspect the resulting state, not just the text the agent produced ("did the record actually get updated correctly," not "did the agent say it updated the record").

This is why serious agent eval frameworks grade at the level of the trajectory, not just the final message: did the agent take a reasonable sequence of actions, did it recover from errors, did it stay within budget, and did the end state match the goal.

Building blocks of a real eval suite

1. The task set

The task set is where most eval suites quietly fail before they even start. Two traps show up constantly:

• Too small and too easy. Ten hand-picked happy-path examples will pass on almost any reasonable agent and tell you nothing about the 5% of cases that generate support tickets.
• Disconnected from production. Tasks written by the engineering team in isolation, rather than sourced from real user queries, logs, or support escalations, tend to systematically miss the messy, ambiguous, or adversarial inputs that actually show up in the wild.

A workable task set usually blends three sources:

• Golden examples — hand-curated, high-confidence cases with known-correct outputs, used as a stable baseline.
• Production samples — real (anonymized) queries pulled from logs, including edge cases, weird phrasing, and multi-turn conversations that broke something in the past.
• Adversarial/red-team cases — inputs deliberately designed to probe safety boundaries, prompt injection resistance, and graceful failure under ambiguous or malicious instructions.

Task sets should also be stratified by difficulty and category, not treated as one undifferentiated pile. A 90% aggregate pass rate can hide a 40% failure rate on the one task category — say, refund policy edge cases — that matters most to the business.

2. The grading method

There are three broad ways to grade agent output, and most mature eval suites use a mix of all three:

Rule-based grading is the right choice whenever the task has a checkable ground truth: did the SQL query return the correct rows, did the JSON conform to the schema, was the correct tool called with the correct arguments. Use it wherever it's possible — it's cheaper and more trustworthy than any LLM judge.

LLM-as-judge fills the gap for anything more subjective: was the tone appropriate, was the explanation clear, did the agent correctly refuse an unsafe request without being unnecessarily unhelpful. The catch is that an LLM judge is itself an LLM call and inherits LLM failure modes — it can be inconsistent across runs, biased toward longer or more confident-sounding answers, and fooled by superficially plausible but wrong reasoning. Any team relying on LLM-as-judge should periodically validate the judge's scores against human review on a sample, and treat disagreement between the two as a signal to refine the rubric, not just to trust the judge more.

Human review is the fallback of record. It's too slow to run on every build, but it's essential for calibrating the automated graders and for periodic spot checks on high-stakes categories (safety, compliance, anything touching money or user trust).

3. Metrics that are actually actionable

Aggregate pass rate is the metric everyone starts with and the one that's least useful on its own. A single number going from 88% to 85% tells you something got worse; it does not tell you what, where, or how badly. More useful metrics, tracked alongside the headline number:

• Task success rate, broken out by category — so a regression in one workflow doesn't hide inside a healthy overall average.
• Trajectory efficiency — number of steps, tool calls, or tokens used to reach a correct answer; a model that gets the right answer in 12 steps when 4 would do is burning cost and latency even if it "passes."
• Tool-call accuracy — the rate at which the agent selects the correct tool and supplies correct arguments, independent of whether the final answer happened to be right anyway.
• Failure mode classification — bucketing failures into categories (wrong tool, hallucinated fact, gave up early, exceeded budget, unsafe output) so engineering effort goes where it will actually move the number.
• Latency and cost per task — quality without a cost/latency budget is an incomplete picture for anything that has to run at scale.

Why this matters now

Agent evals have moved from a nice-to-have to a load-bearing part of the development loop for a structural reason: as agents take on more autonomous, multi-step responsibility — executing code, modifying records, spending budget, contacting customers — the cost of an undetected regression scales with the amount of unsupervised action the agent takes. A single-turn chatbot that gives a slightly worse answer disappoints one user. An agent that autonomously mishandles a multi-step refund workflow, or silently loops on a failed tool call and burns through an API budget, produces damage that compounds before anyone notices.

At the same time, the tooling for building agents has become good enough that teams can stand up a working prototype in days, which creates a gap: the barrier to building an agent has dropped faster than the discipline of evaluating one has spread. Teams that treat evals as an afterthought find out about failures from users instead of from a test suite — the same lesson software engineering learned about testing decades ago, now replaying itself for a class of system where failures are harder to reproduce and errors compound across steps instead of surfacing immediately.

The practical upshot is that eval infrastructure is no longer something only frontier labs building foundation models need. Any team shipping an agent that takes real actions needs a version of this discipline, scaled to their size — even if that means twenty hand-curated tasks and a spreadsheet rather than a full evaluation platform.

Practical implications for teams building agents

For teams actually building and shipping agents, a few operational patterns consistently separate the ones that scale safely from the ones that don't:

1. Build the eval suite before the agent is "done," not after. Evals written after the fact tend to be shaped around whatever the agent already does well, which defeats the purpose. Draft the task set from expected use cases before or alongside initial development.
2. Gate deployments on eval results, not on demo quality. A change should not ship to production until it clears the eval suite at or above the current baseline. This is the same discipline as a CI test gate, applied to quality rather than correctness of code.
3. Keep a frozen regression set separate from the set you iterate against. If engineers tune prompts against the same tasks used to measure success, the eval suite stops measuring generalization and starts measuring overfitting to the test set — a version of Goodhart's law that shows up constantly in prompt engineering.
4. Version the eval suite alongside the agent. As new failure modes get discovered in production, add them to the task set. An eval suite that never grows stops catching new categories of regression.
5. Instrument trajectories, not just outcomes. Log the full sequence of tool calls, intermediate reasoning, and state changes for every eval run, not just pass/fail. When something fails, the trajectory log is what tells you why.
6. Budget for the cost of evaluation itself. LLM-as-judge grading and large task sets are not free — running a few thousand graded tasks against a frontier judge model on every release has real dollar and latency cost. Size the suite to the risk of the workflow it protects.

Limitations and open questions

Evals are not a solved problem, and it's worth being honest about where the discipline still falls short:

• LLM judges are not neutral arbiters. They carry their own training biases, can be gamed by verbose or confident-sounding outputs, and their reliability varies by task domain. A judge that's well-calibrated for summarization quality may be poorly calibrated for judging code correctness or safety refusals.
• Static task sets go stale. User behavior, adversarial techniques, and the agent's own capabilities all shift over time. A task set frozen a year ago may no longer reflect the distribution of real queries, giving a false sense of stability.
• Trajectory-level grading is expensive and immature. Tooling for automatically scoring whether an agent's process, not just its final answer, was sound is far less standardized than outcome grading, and many teams still fall back to checking only the final output because it's easier.
• There's no universal metric for "good agent behavior." Unlike classification accuracy or BLEU scores in older ML paradigms, agent quality is inherently multidimensional — correctness, efficiency, safety, and cost trade off against each other, and reasonable teams weight them differently depending on the use case.
• Evals can't fully substitute for production monitoring. Even an excellent eval suite tests against a finite, curated distribution. Real users find inputs no task set anticipated, which is why evals and live monitoring have to work together rather than one replacing the other.

What to watch next

The eval space is still consolidating around shared practice rather than shared tooling, which means a few trends are worth tracking:

• Standardized trajectory-grading formats. As more teams build multi-step agents, expect more shared conventions (and eventually tooling) for scoring the process an agent took, not just its final output — closing the gap that currently makes trajectory evaluation the hardest and least mature part of the stack.
• Judge calibration becoming its own discipline. Teams are increasingly treating "is our LLM judge trustworthy" as a question requiring its own evaluation, rather than assuming a capable model makes a good grader by default.
• Continuous eval pipelines tied to production sampling. Rather than a static task set run occasionally, more mature setups are building pipelines that continuously pull fresh production samples into the eval loop, keeping the task distribution current automatically.
• Cost-aware evaluation. As agents get judged on efficiency as well as correctness, expect metrics that jointly optimize for task success and resource use to become standard rather than optional.

FAQ

What's the difference between AI evals and traditional software testing?

Traditional software tests check deterministic logic against fixed assertions — same input, same expected output, every time. Evals grade probabilistic, open-ended outputs against a rubric or acceptance criteria, often using another model or a human as the judge, because there's rarely a single "correct" string to match against.

Do I need evals if I'm just using an off-the-shelf AI agent tool?

Yes, if you're customizing its prompts, tools, or configuration for your own workflow. The underlying model's general capability doesn't tell you whether your specific configuration handles your specific tasks correctly — that's exactly what an eval suite tied to your own task set is for.

How many test cases do I need for a useful eval suite?

There's no fixed number, but a suite in the dozens rather than single digits is a reasonable starting point, stratified across the main categories of task your agent handles plus known edge cases. What matters more than raw count is coverage: every workflow and failure mode you care about should have several representative cases, not just the happy path.

Can LLM-as-judge grading be trusted on its own?

Not without validation. LLM judges are useful for scaling evaluation of subjective quality, but they should be periodically checked against human review on a sample of cases, and any systematic disagreement should prompt a rubric revision rather than automatic trust in the judge.

How often should I update my eval task set?

Whenever a new failure mode shows up in production, in support tickets, or in a red-team exercise, add it to the suite. Treat the task set as a living regression log rather than a document you write once and leave static.

What's a trajectory in the context of agent evals?

It's the full sequence of steps an agent takes to complete a task — its intermediate reasoning, tool calls, arguments, and the results it receives — not just the final answer it returns. Grading the trajectory, rather than only the outcome, is what lets you tell whether an agent got the right answer for the right reasons or by accident.

Why do agents that pass evals still fail in production?

Because the eval task set is finite and the production input distribution isn't. Real users generate phrasing, edge cases, and multi-turn context an eval suite didn't anticipate, which is why evals should be paired with ongoing production monitoring rather than treated as a one-time gate.

Teams building agents that need this kind of eval infrastructure set up properly can find hands-on help at Woyce Technologies.