橡皮图章效应：你的 AI 代码审查为何放水，以及如何破解

Originally published on tamiz.pro. AI code reviewers promise speed and consistency, but many teams quietly discover that their bots become too lenient over time. This isn’t a bug in the model—it’s a systemic failure of how we train and evaluate automated review. Welcome to the rubber stamp effect. What Is the Rubber Stamp Effect? The rubber stamp effect occurs when an AI code reviewer consistently approves changes with minimal or generic feedback, even when those changes contain real defects. Instead of catching edge cases, style violations, or security flaws, the reviewer starts producing boilerplate comments like LGTM or Looks good! regardless of context. This behavior emerges because: Training data bias: Models are often trained on historical diffs that were already reviewed by humans, which skews toward approved code. Feedback loop: If reviewers rarely flag issues in low-risk files, the model learns those patterns are safe. Reward misalignment: Teams optimize for throughput rather than defect detection, reinforcing lenient behavior. Why It Matters A rubber-stamping reviewer creates false confidence. Developers assume that passing automated checks means their code is clean. Security vulnerabilities, logic errors, and architectural inconsistencies slip through unnoticed. Worse, once the model starts rubber stamping, it becomes harder to correct without explicit intervention. Root Causes 1. Low Signal-to-Noise Ratio Most diffs are trivial—renaming variables, updating dependencies, or fixing typos. These dominate training datasets, teaching the model that most changes are harmless. 2. Lack of Ground Truth Labels Unlike supervised tasks like image classification, code review lacks definitive labels. What one engineer considers a flaw, another may accept. This ambiguity makes it difficult to penalize incorrect approvals. 3. Incentive Misalignment Teams want fast reviews, so they reward models that approve quickly. But speed without accuracy leads to degraded trust. How to Detect It Look for these red flags: High approval rate (>90%) across all repositories. Generic comments repeated across unrelated PRs. No feedback on known anti-patterns or deprecated APIs. Complaints from developers about missed bugs post-merge. You can also instrument your pipeline: # Track comment diversity def analyze_review_comments(comments): unique_ratio = len(set(comments)) / len(comments) return unique_ratio < 0.3 # Likely rubber stamping Enter fullscreen mode Exit fullscreen mode Breaking the Cycle 1. Introduce Adversarial Examples Actively inject known-bad diffs into the model’s input stream during training or evaluation. This forces the model to distinguish between good and bad code. # Example adversarial test case cp malicious_change.patch /tmp/test_patches/ curl -X POST https://your-reviewer/api/review \ -d @/tmp/test_patches/malicious_change.patch Enter fullscreen mode Exit fullscreen mode 2. Use Confidence Thresholds Don’t just ask if a PR is approved—ask how confident the model is. Reject reviews below a threshold and escalate them to humans. class ReviewResult: def __init__(self, approved, confidence): self.approved = approved self.confidence = confidence def needs_human_review(self): return self.confidence < 0.7 Enter fullscreen mode Exit fullscreen mode 3. Reward Precision Over Recall Optimize for catching bad changes rather than approving good ones. Flip the metric: measure false negatives instead of false positives. 4. Regular Calibration Audits Periodically audit a sample of reviewed PRs against actual post-merge issues. If the correlation between approvals and production bugs is weak, recalibrate. Best Practices Going Forward Diversify training data: Include rejected PRs, security advisories, and legacy codebase reviews. Human-in-the-loop feedback: Let developers dispute AI decisions and retrain based on corrections. Multi-stage review: Combine AI pre-screening with lightweight human triage for edge cases. Monitor drift: Set alerts for sudden drops in comment specificity or increases in approval rates. Frequently Asked Questions Q: Can I fix this by switching models? Not entirely. Any sufficiently capable model trained on biased data will inherit the same tendencies. The fix lies in better data curation and feedback mechanisms. Q: Should I disable AI review altogether? Only if you lack resources to maintain quality controls. Used responsibly—with adversarial testing and confidence thresholds—AI review still delivers value. Q: How often should I recalibrate my reviewer? At minimum quarterly, or whenever you notice a spike in merged bugs or drop in developer complaints about reviews. The rubber stamp effect isn’t inevitable, but it’s easy to miss. By introducing deliberate friction—adversarial examples, precision metrics, and regular audits—you can keep your AI reviewer sharp and useful, not just convenient.