为什么多数“AI 自动修 bug”演示撑不过生产环境

Every few months another demo shows an AI agent opening a pull request that closes a bug ticket, and the reaction is the same: impressive, and also — would you actually trust this in production, unattended? Most teams' honest answer is no, and the reason isn't model capability. It's that closing the loop from "bug reported" to "fix merged" without a human in the middle requires solving a problem most of these demos don't even acknowledge: the model can make its own success metric go green without the underlying bug actually being fixed. This isn't a hypothetical concern. METR's RE-Bench research measured coding agents gaming their own evaluation metrics roughly 30% of the time, and found that simply telling the agent not to cheat in the prompt only prevents it 5–30% of the time — the rest of the time, the instruction is ignored under pressure to show a passing result. If you're building a system that fixes bugs with no human reviewing every step, this is not a side issue. It's the entire engineering problem. Over the last year I've been building and operating exactly this kind of system in a real production environment — not a demo, a pipeline that has processed real bug reports and merged real fixes into a live, multi-service platform. I'm not going to walk through the architecture here — some of it is genuinely novel and I'm keeping the specifics to myself for now — but I want to share the general lessons, because the failure mode itself is universal to anyone attempting this. The core problem nobody's demo addresses Any system that lets an AI both write a fix and judge whether that fix is good has a built-in conflict of interest. The model's incentive is to produce something that looks done, and "looks done" is a much easier bar to clear than "is actually correct." Without a structural counterweight, autonomous bug-fixing systems drift toward the easy bar, not the real one — quietly, and often invisibly to whoever's watching the dashboard. What actually holds up, at a principle level Without getting into our specific implementation, a few categories of mitigation are the ones I'd tell anyone building this kind of system to take seriously: Separate the writer from the judge, structurally. If the same model (or the same prompt lineage) both writes and reviews a fix, its blind spots are correlated with themselves. An architecturally independent second opinion — a different provider, a different context, ideally both — catches a different set of mistakes than self-review ever will. Don't let the model touch the rules of the game. Any system where the model can edit its own tests, its own CI config, or its own success criteria will eventually — not maybe, eventually — find that it's easier to weaken the bar than to clear it. This has to be enforced outside the model's own reach, not requested of it politely in a prompt. Require multiple independent signals to agree before calling anything "done." One green checkmark is not evidence. Several different, independently-sourced checks agreeing is much harder to fake by accident or under optimization pressure. Default to escalating to a human, not defaulting to proceeding. Every point where the system is uncertain, where money is being spent, or where something hard-to-reverse is about to happen should stop and ask, not guess and continue. The absence of a human catching an error is not the same thing as the system being correct — conflating those two is where I've seen teams get burned. The part I think matters most The single most valuable engineering decisions in a system like this are usually not the features that shipped — they're the capabilities that were built, tested, and then deliberately left switched off, because the team couldn't yet prove they were safe to turn on. That restraint is unglamorous and doesn't show up in a demo, but it's the actual difference between a system you can trust unattended and one that just hasn't failed publicly yet. The honest caveat None of this eliminates the need for engineers. It changes where their time goes — away from the repetitive "reproduce, fix, review" cycle on routine bugs, toward the judgment calls a well-designed system is built to escalate rather than guess at. A system that claims to remove humans from the loop entirely is usually the one you should trust least. Anton Shorygin is a QA Lead with 10+ years in software quality and test automation, currently building automated detection and remediation systems for a production software platform.