两份 AI 代码审查基准结论相左，却在评测维度上一致

The top of the Google results for "best AI code review tools" is a wall of vendor listicles. Most of them rank the publisher's own product first and support that rank with feature bullets, not numbers. Two pages on that SERP actually publish a method, and they are worth reading together because they come to opposite conclusions about the winner. Two evaluations, two winners LinearB's benchmark measured 16 bugs across two phases and scored each reviewer on four dimensions: competency, clarity, configurability, and developer experience. The write-up reports that LinearB produced the best signal-to-noise ratio and foregrounds statefulness, the ability to withdraw or revise a comment once a later commit has made it stale. In that telling, CodeRabbit caught the most total issues but generated heavy noise, flagging the same pattern repeatedly without context. GitHub Copilot delivered consistently relevant suggestions but with shallow context, missing multi-file reasoning. Graphite Diamond performed weakest on detection. DeepSource's comparison measured a similar category with a different instrument. It ran every tool against the OpenSSF CVE Benchmark, a public dataset of 200+ real production vulnerabilities, and publishes F1 scores. DeepSource reports the highest F1 at 84.51%. CodeRabbit scores 36.19% F1 in that telling. Greptile's self-reported 82% catch rate, DeepSource notes, comes from an internal benchmark of 50 PRs across five repositories that is not independently validated. Two vendors, two methods, two winners, and both write-ups rank their own product first. That is normal, and it is not a reason to dismiss either page. It is a reason to read the method instead of the winner. The two evaluations are incommensurable. LinearB's dataset is built in-house, and the bug list and scoring criteria live in a downloadable whitepaper. DeepSource's dataset is public, but it is a security corpus: OpenSSF CVEs score whether a reviewer detects real recorded vulnerabilities. That is a different question from whether a reviewer writes clear, usable comments. The same tool that wins on signal-to-noise can trail on F1 against a security corpus, because the two metrics punish different failure modes. Where the two benchmarks agree Reading both pages side by side is worth it because of the quarters where they do not disagree. Both independently land on the same dimensions separating a useful reviewer from a noisy one. Signal-to-noise ratio. Both pages make this the deciding quality. A reviewer that finds 90% of issues but buries them in hundreds of comments per PR is worse than one that finds 70% and says it once. LinearB calls it the best reviewer says more with less. DeepSource puts it near the headline of its methodology. This metric is not really about whether the model catches bugs. It is about trust. Developers stop reading comments they have learned to ignore, and once that happens the reviewer is adding noise to every PR without contributing anything. Statefulness across commits. Pull requests change between commits. A reviewer that treats each commit as a fresh start makes developers re-litigate issues they already resolved. LinearB measured this directly: the reviewers that withdraw outdated comments and revise their opinion after a fix score predictably higher on developer experience. The tools that restart from zero on every commit pay for it in review time, because a fix that should close a thread just opens a new one. Configurability. Both pages raise this independently. Teams have different tolerances for verbosity and different standards, and reviewers that let teams tune rules, tone, and enforcement through configuration fit into existing workflows instead of demanding new ones. LinearB specifically notes that YAML-defined rules and slash commands correlated with a smoother developer experience. This is the axis teams that care about their own coding standards should watch closely. It separates a tool that adapts to your rules from one that applies generic rules to everyone. Reviewers in this bucket, Kodus among them, put rules and signal ahead of raw finding count, which is exactly the configurability axis both benchmarks flag as the one that determines whether a tool gets adopted at all. Time to first useful signal. LinearB measures the average time from PR open to the first correct, actionable comment. A fast but wrong comment is worse than a slow correct one, which is why the metric exists: it stops vendors from optimizing for response speed at the expense of accuracy. It is a genuinely useful thing to time in your own workflow, because it tells you how long a developer waits before the reviewer does something they trust. What to take away You will not get a winner from these two pages, and you should not expect one. What you get is a checklist for evaluating reviewers internally. Hold a vendor to three questions that both published write-ups imply. What is the dataset, and can you inspect it? DeepSource gives you OpenSSF CVEs. LinearB gives you a 16-bug in-house list inside a whitepaper. A number without a method you can verify is marketing, not evidence. Note that neither of these ranks an independent benchmark where the publisher is not also a contestant; that is a separate, harder bar, and one an exploratory arXiv benchmark I covered in an earlier article on this feed tries to clear by scoring reviewers against an independent annotation workflow rather than a vendor's own product. Which metric, and what does it punish? F1 penalizes false positives alongside false negatives, which is partly why the safety numbers in a security corpus diverge from a DevEx-style evaluation. Make the metric match the failure mode you are buying against. If you lose more time to false positives than to missed bugs, optimize for precision first. If a misses slip to production, weight recall. Is the reviewer stateful and configurable, or does it restart and shout? On that axis, the two competitors who disagree about the winner agree completely. That agreement is the most transferable finding in either document, because it predicts how a tool will behave in your review cycle regardless of which vendor happens to come out on top of its own benchmark. The wider pattern The broader pattern is worth naming. Generation got fast and cheap, so the market filled with reviewers. Validation got the opposite treatment. Most marketing still optimizes for volume of output, not for whether the output is worth reading. The two ranked pages that included a method are the exception, and they independently reach the same conclusion: lean reviews, fewer comments, rules you control, and a reviewer that remembers what it already said. That convergence across competitors is worth more than either winner claim. Stamp: claims checked 2026-09 against the LinearB and DeepSource pages linked above.