A Loop Is Only as Good as Its Sensor

I made one change to an agent and its score went up 27 points. My own release gate blocked it.

The gate was right. The agent reads company updates and records their revenue figures in a database, and my change had taught it to look for forward-looking markers before treating a figure as realized. The marker match ran on substrings, so the word unexpected contained expect, and a revenue figure reported "despite unexpected churn" got filed as a projection and never written. On the slice that measured whether the correct value actually reached the database, the pass rate dropped from 100% to 67%. The average climbed anyway, because there were more of the other kind of case.

So: a 27-point improvement that would have shipped a silent data-loss bug. I had planned on one change and one release. It took me a while to stop being annoyed at the gate and start being interested in it.

The sensor is inside the plant

Improving an agent is a control loop. There is a plant, an actuator, a controller, and a sensor, labeled below. I didn't invent this framing. It was handed to me at the start of a project, adapted from Kyle Mistele's talk on loop engineering (opens in new tab), and I mostly nodded at it. Taking it literally is what changed how I worked.

In that vocabulary, almost every bad agent evaluation is a bad sensor. And the commonest way to build a bad one is to let it read the plant's own account of itself. The agent says it filed the record — did anything get filed? The agent cites a source — does the source say that? When the sensor reads the transcript instead of the state, you are measuring the plant with an instrument the plant controls.

A whole class of evaluation failure is instrumental before it is statistical. More trials won't help, and neither will tighter confidence intervals. You don't trust a thermometer you've never put in ice water, so the verifier gets calibrated against outcomes whose correct verdict you already know, before the agent runs at all. The fixture that matters most there is the near miss: the plausible, half-finished attempt, because incomplete work is the shape real failures take. In a study of three issue-solving tools (opens in new tab), nearly 30% of the patches that passed SWE-bench Verified's tests behaved differently from the developers' own fix, and more than a quarter of those were wrong on inspection. For the wrong ones, with an oracle that weak, the benchmark was measuring its own verification.

Three turns of the same screw

What I didn't expect is that fixing the instrument keeps introducing new bias.

Start with the obvious suite: a set of cases where the agent is supposed to act. Every case rewards acting. An agent that always acts scores well, and you have measured eagerness rather than judgment.

So add negative cases, situations where the right answer is to leave things alone. My suite for a ticket-closing agent had a few. Then calibration caught something. A wrong answer I had planted, in which the agent claimed to have escalated the ticket and changed nothing, passed when it should have failed. Leaving the ticket open was necessary but not sufficient, and my case design had never said so. The suite that punished eagerness now rewarded silence just as thoroughly.

So weight the cases by risk and gate on the risky slice rather than the average, since averages are where a dangerous regression hides. That works. It also has a failure mode I found only because I went looking: when nearly every high-risk case is a don't-act case, a reflex agent that never acts scores 8 of 9 on precisely the slices guarding the gate.

Each of the three fixes introduced a new bias, and each bias pointed at whatever I had just started measuring. I have no general solution, only a habit: after any change to an evaluation, ask which degenerate agent now passes it.

A suite where everything passes has stopped measuring. The number has become a fact about my case set.

What it costs

Sealing a suite means hashing every case, fixture, and verifier file so a comparison can prove both runs faced identical bytes. Every edit to a single case then forces a version bump and a re-run of everything. Calibration means writing wrong answers on purpose and committing to their verdicts before you are allowed to measure anything at all.

When four agents each built an evaluation for the same target, two of them without sealing or a calibration gate, the disciplined runs cost two to four times the wall clock and roughly three times the tokens. Both undisciplined runs also produced a working harness, and one of them gated releases on critical cases. Neither could prove that two of its runs had faced the same cases, and neither had tested its verifier against an agent that claimed success and changed nothing.

On a later agent, with the harness adapted to it, eighteen candidate changes went through its gate. Three were released.

The fifteen refusals are what I learned that month. Most of them were specific claims I would otherwise have believed. One blocked a change that raised an extractor's mean score from 0.36 to 0.57 and broke the one case even the baseline had passed. The first comparison released it, and the gate refused it once I had marked that case critical.

The best thing that happened wasn't planned. One of the undisciplined runs from the four-agent comparison found a defect my case set had no case for: the substring bug from the opening. My suite had never tested a realized figure sitting next to churn language, so my comparison had reported that first candidate as a clean win. I added the case, re-sealed with a version bump, and re-ran everything. The release decision inverted from release to block. That is the loop improving the evaluation instead of the target.

When something looks like it's working, the useful question is what read the state, and whether it could have told you otherwise. The honest headline for the harness is that its case population is known to be incomplete, and I would rather publish that sentence than the 27 points.