AI 模型能发现坏代码，却对好代码误报

Kaggle Benchmarking Challenge Submission I wanted to know one thing. When an AI model reads a data-science tutorial, does it check the code or believe the caption? So I built Blog vs Bytecode, a Kaggle benchmark of 28 short data-science snippets. Each pairs a piece of code with a blog-style claim about it. Half the claims are honest and the method is sound. The other half hide a real mistake: a scaler fit before the train/test split, a feature built from the target, shuffled cross-validation on a time series, accuracy quoted on data that is 99% one class, a threshold tuned on the test set. The model returns one verdict, OK or PROBLEM, then says why. The set is balanced 15/13, so a model that just yells PROBLEM at everything lands near 50% and gets caught. Then I ran it against a spread of current models on Kaggle. The first thing it caught was not a model. It was the harness. The harness was lying about half the models Kaggle runs the models for you on its Model Proxy. When I pulled the raw results, many of the answers were empty. The run finished, but no response was captured. Grading an empty answer as wrong made strong models look broken. gpt-6-astra showed 14%. claude-opus-5 showed 4%. DeepSeek-R1 showed 17%. None of it was real. I re-ran the same prompts, with the same grader, through a second gateway that actually returns the text. DeepSeek-R1 went from 17% to 100%. Same model, same questions, same scoring. The only thing that changed was a harness that handed back the answer. That is the uncomfortable lesson for a benchmarking contest: a benchmark that does not audit its own capture is measuring the plumbing, not the model. So the scorer now flags every empty response and drops it, instead of counting it as wrong. What the models actually do With capture fixed, the picture is more interesting than "big good, small bad." They catch bad code, then cry wolf on the good code. The strong models are near-perfect at flagging a genuine flaw. Where they drop points is the opposite error: calling sound code a PROBLEM. Claude Sonnet 5 and Gemini 3.8 Flash each miss one clean item, both by over-flagging. I expected models to be too trusting. Today's frontier models are the other thing, a nervous reviewer that cannot leave a correct snippet alone. Scale flips the bias. The small model inverts it. Gemma 4 31B keeps 85% on the clean code but its recall on the hidden flaws falls to 20%. It is the one that actually trusts the prose: it reads the confident claim, agrees, then walks past the leak. So the failure mode is not fixed. Big models over-flag, small models under-flag, for opposite reasons. Reasoning is worth about a third of the score and you can see where. Grok 4.20 scores 100% with reasoning on and 68% with it off. The 32 points do not come off evenly. They come off the subtle pairs built to separate understanding from pattern-matching: a parameter-free transform before the split is fine, a fitted scaler before the split is a leak; filling a missing value with a constant is fine, filling it with the column mean is a leak. Turn reasoning off and the model stops telling "before the split" apart from "using numbers from the split." It matches the position of a line instead of what the line computes. The leaderboard Model Accuracy Catches flaws Clears clean code Gemini 3.1 Pro 100% 100% 100% Gemini 3.7 Flash 100% 100% 100% Grok 4.20 (reasoning) 100% 100% 100% DeepSeek-R1 100% 100% 100% GLM-5 100% 100% 100% Claude Sonnet 5 96% 100% 92% Gemini 3.8 Flash 96% 100% 92% Claude Haiku 4.5 86% 100% 69% Grok 4.20 (no reasoning) 68% 100% 31% Gemma 4 31B 50% 20% 85% Try it The benchmark is public on Kaggle: https://www.kaggle.com/benchmarks/tasks/zkasuran/blog-vs-bytecode/1 . Every item, its verdict and the grader are open. Kaggle captures each model's answer verbatim, so you can read exactly where it went wrong. Point any model on the Kaggle model list at it. Honest notes Two backends were used: Kaggle's Model Proxy for the models it captured cleanly and an OpenAI-compatible gateway for the ones the proxy dropped, with the identical prompt and grader. GLM-5 was measured in its FP8 build. GLM-5 and Gemini 3.5 Flash Lite are scored on the items the proxy captured. Grades are on the verdict. The written reason is captured too. On the top models it names the right failure almost every time. AI assistance (Claude) was used to build the benchmark and draft this post. The design, the items, the grading and the analysis were reviewed and verified by the author, including reading the raw model outputs to catch the capture bug above.