The paper was beautiful.
Clean prose, a tidy results table, a related-work section that cited the right people in the right places. It read like it had been through three rounds of revision with a patient advisor.
Then I checked two of its claims. The first was a citation whose title I didn't recognize, and it resolved to nothing: no DOI, no arXiv identifier, no record in any index I searched. Plausible title, plausible authors, plausible year, simply never written. The second was the headline number. I went looking for the experiment behind "we observe a 4.2% improvement" and could not find one that had produced that figure, or any figure, for that comparison. The model had written the sentence the way it wrote every other sentence, because it was the kind of sentence that goes there.
I can't hand you that artifact. What I can say is that both of those failures are build errors now, which is most of what this essay is about.
That is the failure mode of the current wave of "AI scientist" systems, and it is worth being exact about it. They are optimized, end to end, to produce an artifact that looks finished. Looking finished is now cheap. A model that can write a flawless methods section can write an equally flawless methods section about an experiment it never ran. Fluency used to be weak evidence of competence. It is no longer evidence of anything.
The most dangerous thing a research system can produce isn't a wrong answer. It's a confident, well-formatted wrong answer that nobody thinks to check.
Honesty is a build step
My instinct, the first time, was to ask the model to be more careful: add a line to the prompt saying do not fabricate citations. Which cannot work, because the model has no privileged access to which of its citations are real. "Don't hallucinate" is advice to a system that can't tell when it's hallucinating.
What works is to stop trusting the generator and start checking it. The interesting engineering in an honest research system isn't the part that writes; it's the part that refuses to let the writing through. And that part is mostly a set of build errors. A citation key that doesn't resolve to a verified record fails the build, rather than degrading into a warning somebody reads later. A number in the results table with no matching value in a run artifact fails the build. A run that failed stays failed, and the failure stays in the artifact, so no downstream stage can round it up into a success because the narrative wanted one. An experiment that separates its conditions on no metric at all comes back marked uninformative, which is a result the pipeline knows how to store rather than a gap the prose has to paper over.
None of this is intelligence in the usual sense. It is plumbing: the model proposes, deterministic code disposes. That division of labor is the whole point, because it is the only arrangement in which the system's honesty doesn't depend on the system's mood.
Refusal is the hard part
Here's what the demos get backwards. Generating a research paper is, in 2026, not hard. The hard part is refusal, and refusal takes taste: you have to know which citation you can't stand behind, or that your beautiful result is an artifact of a leak between your features and your labels, or that the honest version of your abstract reads "we tried this and it didn't work" — and then write that one instead of the version that gets accepted.
Anyone, human or model, can produce more. The discipline is in producing less, on purpose, when the evidence isn't there. We pour enormous effort into making models more capable and almost none into making them better at saying I don't actually know. In science, the second skill is most of the job.
Which is also why I've lost interest in the "fully autonomous AI scientist" framing. It aims at the wrong target.
Some things these systems structurally cannot do, and pretending otherwise is where the hype curdles into a dishonesty of its own. Novelty is backward-looking: a model can tell you your idea resembles five existing papers, but certifying that nobody, anywhere, has had it is not a computation anyone can run. It can surface a confounded experiment; it cannot fix a bad question, because that takes wanting to know something specific about the world. Acceptance at a venue is a social fact about a committee, and no property of your manuscript will settle it. A system that promises novelty, rigor, and acceptance is selling you the three things it is least equipped to guarantee.
What it can do is less glamorous and, I think, more valuable. It can run the epistemic hygiene that humans skip when they are tired or hopeful or three days from a deadline: checking every citation, re-deriving every number, going back to look for the leak. The useful property of a machine here is that it does not get bored partway through, and that it applies the same standard to the person who built it.
That's the version I wanted to exist, so I've been building toward it in AutoPaperLab, where everything in the output traces back to something real and the parts that can't simply don't make it in.
The measure of an automated scientist isn't what it can write. Models crossed that line a while ago and it told us less than we'd hoped. The measure is what it will refuse to write. None of that makes science faster; it makes it slightly harder to fool yourself, which on most days is the same thing as doing science at all.
