A Loop Is Only as Good as Its Sensor

I made one change to an agent and its score went up 27 points. My own release gate blocked it.

The first thing I did was check whether the gate was broken. It wasn't. The change had taught the agent to look for forward-looking markers in a document before treating a revenue figure as realized, and the marker match ran on substrings, so the word unexpected contained expect. Any sentence with "unexpected" in it got filed as a projection. On the slice that measured whether the correct value actually reached the database, the pass rate dropped from 100% to 67%. The average climbed anyway, because there were more of the other kind of case.

So: a 27-point improvement that would have shipped a silent data-loss bug. It took me longer than I want to admit to stop being annoyed at the gate and start being interested in it.

The sensor is inside the plant

Improving an agent is a control loop. There is a plant (the agent, and the environment it acts on), an actuator (a change you make), a controller (you, choosing what to try next), and a sensor (whatever tells you what happened). I didn't invent this framing. It was handed to me at the start of a project and I mostly nodded at it. Taking it literally is what changed how I worked.

Once the vocabulary is in place the failure mode is hard to unsee: almost every bad agent evaluation is a bad sensor. And the commonest way to build a bad one is to let it read the plant's own account of itself. The agent says it filed the record — did anything get filed? The agent cites a source — does the source say that? A claim about a write is not a write, and a citation is not support. When the sensor reads the transcript instead of the state, you are measuring the plant with an instrument the plant controls.

Which means a whole class of evaluation failure is instrumental before it is statistical. More trials won't help, and neither will tighter confidence intervals. You don't trust a thermometer you've never put in ice water, so the verifier gets calibrated against outcomes whose correct verdict you already know, before the agent runs at all. The fixture that matters most there is the near miss: the plausible, half-finished attempt, because incomplete work is the shape real failures take. Roughly a third of SWE-bench Verified instances with passing patches turned out to have test suites too weak to catch an incomplete one. Those numbers were never about the models.

Three turns of the same screw

What I didn't expect is that fixing the instrument keeps introducing new bias.

Start with the obvious suite: a set of cases where the agent is supposed to act. Every case rewards acting. An agent that always acts scores well, and you have measured eagerness rather than judgment.

So add negative cases, situations where the right answer is to leave things alone. Mine had a few. Then calibration caught something. An agent that read the ticket, understood nothing, and walked away passed the negative cases cleanly. Not closing was necessary but not sufficient, and my case design had never said so. The suite that punished eagerness now rewarded silence just as thoroughly.

So weight the cases by risk and gate on the risky slice rather than the average, since averages are exactly where a dangerous regression hides. That works. It also has a failure mode I found only because I went looking: when nearly every high-risk case is a don't-act case, a reflex agent that never acts scores 8 of 9 on precisely the slices guarding the gate.

Three fixes, three new biases, each one pointing at whatever I had just started measuring. This is Goodhart's law a level down — not the metric being gamed by the agent, but the instrument being bent by its own repair. I have no general solution. What I have is a habit: after any change to an evaluation, ask which degenerate agent now passes it.

A related tell, which took me a while to accept: a suite where everything passes has stopped measuring. The number has quietly become a fact about my case set.

What it costs

This discipline is slow, and I would rather say so than pretend otherwise.

Sealing a suite means hashing every case, fixture and verifier file so a comparison can prove both runs faced identical bytes. It also means every edit to a single case forces a version bump and a re-run of everything. Calibration means writing wrong answers on purpose and committing to their verdicts before you are allowed to measure anything at all.

When four agents worked the same target and two of them skipped all of it, the disciplined runs cost two to four times the wall clock and roughly three times the tokens. Both undisciplined runs also produced a working harness. The difference wasn't that one side had a harness and the other didn't. It was that only one side's harness could say no.

Eighteen candidate changes went through that gate. Three were released.

The fifteen refusals are what I actually learned that month. Each one was a specific claim I would otherwise have believed. A gate that never says no isn't a gate; it's a formality with a pass rate.

The best thing that happened wasn't planned. One of the agents working without any of this machinery found a defect my case set had no case for: the substring bug from the opening. My suite had never tested a realized figure sitting next to churn language, so my comparison had reported that first candidate as a clean win. I added the case, re-sealed with a version bump, and re-ran everything. The release decision inverted from release to block.

That is the loop improving the evaluation instead of the target, and it is worth more than the tidy story it replaced. It also means the honest headline for that project is that the case population is known to be incomplete. I would rather publish that sentence than the 27 points.

When something looks like it's working, the useful question is what read the state, and whether it was capable of telling you otherwise. I hit the same problem from the other direction while profiling a GPU kernel: a number that was reproducible, statistically clean, and measuring the wrong interval. It was the same failure in a different instrument, and it took me just as long to see.