The test oracle problem, again

An old problem with a new shape

Every test assumes something can tell you whether the output was right. In 1982 Elaine Weyuker pointed out that this assumption breaks more often than people admit, and she called programs where checking the answer is impractical non-testable. Testing research calls that something the test oracle. A 2015 survey of the field concluded that when automated oracles fall short, a person's judgement is still the last resort.

Agents bring the problem back to the front. A coding agent can fix a bug three different ways. A support agent can give an answer that's correct but unhelpful, or helpful but against policy. Comparing the output to an expected string checks none of that. So before I ask whether an agent passed, I try to write down which of three things I'm checking: was it correct, was it useful and did it stay inside the rules.

Every kind of grader fails somewhere

Executable checks are the best option when you can get them. SWE-bench grades a patch by running the repository's own tests. The tests tied to the issue have to start passing, and the tests that already passed (a median of 51 per task) have to keep passing. That second set is easy to overlook. It's the hard constraint, don't break what worked, sitting right next to the outcome check.

Where outputs are open-ended, people turn to a model as the judge. Zheng et al. found that GPT-4 agreed with human raters on chat answers 85% of the time (ties excluded), about as often as the humans agreed with each other at 81%. The same paper documents how judges go wrong. They favor whichever answer is shown first, they can be won over by longer answers and there's some sign they prefer answers from their own model, though the authors say the data on that is thin.

Human review is still the reference, but it's slow and people disagree too. Anthropic's guide to agent evals lays out the tradeoff well. Code graders are fast and repeatable but brittle when a valid answer looks different. Model graders are flexible but need checking against people, and human graders are the most trusted and the most expensive.

Disagreement is information

When two reviewers score the same transcript differently, the tempting move is to average them and move on. Aroyo and Welty argue that's often a mistake: disagreement can mean the item really is ambiguous, and averaging hides that. I'd rather keep those examples and look at them. Usually they show the rubric didn't say what we thought it said.

Same with scores. One number hides whether the agent was decent everywhere or great on half the tasks and broken on the rest. Rubrics that score one dimension at a time, with an honest "can't tell" option, make that visible. The Anthropic guide recommends close to exactly that.

Build it in layers, then keep revising

What I end up with is layers. Hard constraints first, checked by code wherever possible: didn't touch files outside its scope, didn't break existing tests, didn't call tools it wasn't allowed to. Then outcome checks specific to the task. Then rubric grading, by a model or a person, for the parts nobody can check mechanically.

And the rubric is never finished. The same Anthropic post describes one model's score on a benchmark going from 42% to 95% once problems with the grading and the agent's setup were fixed. That's an extreme case but the lesson isn't. When a real failure slips past your evaluation, the oracle was wrong too, and it deserves the same fix as the agent.