How to Evaluate AI Agents: Evals, Metrics and LLM Judges

Q: True or false: 'Using a frontier model as a judge to score agents, since it agrees with humans often, you can treat it directly as absolutely correct ground truth.'

False。LLM-as-judge has systematic position / verbosity / self-enhancement bias; it's only a scalable approximation, to be calibrated with order-swapping + human spot-checks, not treated as ground truth.

A working demo ≠ production-ready: the first thing is being able to 'measure'

An Agent passing once in a demo doesn't mean it's stable under real traffic. To ship to production, the first thing is to build evaluation — on two legs: offline eval (run before launch / in CI, fully reproducible, zero impact on users) + online monitoring (watch real behavior after launch; it catches distribution drift, but problems usually surface only after they've already hit users). The two are complementary — you need both.

Your eval set doesn't have to be big at first. Anthropic's advice: start with 20–50 tasks drawn from real failures — the effect size is large early on, so a small sample reveals problems. Each task should be clear enough that 'two domain experts judging independently land on the same pass / fail,' with a reference solution, covering both positive and negative cases.

⚠️The biggest mistake: fixating on a single 'accuracy' number. HELM noted long ago that holistic evaluation needs many metrics in parallel (accuracy / robustness / calibration / safety…). Looking only at accuracy misses the most lethal safety and reliability problems.

How to grade: execution-based evaluation + LLM judges (which are biased)

There are two solid ways to grade an agent. One is execution-based evaluation: don't compare text similarity — check whether the result is actually right. SWE-bench has the model fix real GitHub-repo issues and runs unit tests to decide if it truly solved them (at release, even the strongest model passed only ~2%); τ-bench simulates multi-turn 'tool-agent-user' dialogue and uses pass^k (the same task done right across k repeats) to measure reliability — doing it right once isn't the skill; doing it right every time is.

The other is using a strong LLM as a judge (LLM-as-judge): the original study (2023, using GPT-4 of that era) found judge-human agreement above 80%, scaling to replace some manual scoring. But it's systematically biased: position bias (favoring the earlier answer), verbosity bias (favoring longer answers), self-enhancement bias (favoring the answer the judge model itself produced). So don't treat the judge as ground truth — calibrate with order-swapping and human spot-checks.

💡Observability to match: use the OpenTelemetry GenAI semantic conventions to record each call's tokens / model / latency / cost uniformly (standard span attributes) — instrument once, works across vendors — so you can actually see what the agent is doing in production.

自测 · 学完检查一下

想真正动手做题、记进度、攒连胜？到互动课里练。

Which statement about an agent's 'offline eval' and 'online monitoring' is correct?

答案：They're complementary: offline is reproducible and good as a pre-launch gate; online shows real behavior but problems often already hit users

Offline eval runs pre-launch / in CI, is reproducible and zero-impact; online monitoring shows real ground truth and catches drift but is reactive. Complementary — you need both.

What's the core of 'execution-based evaluation' in benchmarks like SWE-bench and τ-bench?

答案：Checking whether the result is right (run unit tests / complete a multi-turn task), not text similarity

Execution-based evaluation checks real results (SWE-bench runs unit tests, τ-bench checks multi-turn task completion) — far more reliable than text matching.

True or false: 'Using a frontier model as a judge to score agents, since it agrees with humans often, you can treat it directly as absolutely correct ground truth.'

答案：False

LLM-as-judge has systematic position / verbosity / self-enhancement bias; it's only a scalable approximation, to be calibrated with order-swapping + human spot-checks, not treated as ground truth.