被“git”一词杀死：一个 token 的巧合与 40 个百分点的通过率

Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.1 field test report evaluated 2 cloud models across 40 corpora and 4,742 trajectory-runs and is the source for every number below. Release notes · Changelog CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization. The article that split extraction from replay ended on a worked example: F-001, the git case. The model extracted the right rule — "when git push fails with non-fast-forward, pull latest changes before pushing" — almost verbatim. Replay scored it prevented 5, broken 3, and the verdict was INCONCLUSIVE. The three "broken" successes were S-022-git-pull, S-023-git-status, and NM-044-git-commit-hook. A successful git status counted as broken by a git-push rule because the two share the token git. That was the diagnosis. This is the fix, and the 40+ points it bought. The bug: "broken" counted coincidence as interference The scorer's job is to decide whether a rule does net good: does it prevent more failures than it breaks successes? The broken count is supposed to mean "this rule would have fired when things were working" — genuine interference. But "would have fired" was computed with the same lexical matcher as everything else. Sharing a token is a match, so: every success that merely rhymes with the rule counted as a success the rule broke. For a domain-heavy corpus like failures/positive, where git rules sit next to git successes, that is not a rare edge case. It is the dominant failure mode. Combined with a precision prevented → fail; near_misses > 2 → inconclusive; otherwise → pass. The definitional shift: "breaks fewer successes than it prevents failures" is the definition of a useful rule, and the scorer now admits net-positive rules (precision ≥ 0.5) instead of downgrading them. Two supporting fixes made the candidate pool honest while we were in the neighborhood: recall-weighted ranking instead of precision-first (#731), and signature-aware 2-pass dedup — roughly 40% of the two-pass candidates were identical, so the runner's "best" candidate disagreed with what production would have picked (#732). The data failures/positive (n=50), both cloud models: v0.3.0 v0.3.1 gpt-4o-mini v0.3.1 llama-3.1-8b Pass rate 8% / 10% 50% [0.37, 0.63] 52% [0.39, 0.65] Precision — 0.544 0.554 Recall — 0.221 0.240 Gpt verdict breakdown on the corpus: pass 25, no_signal 17, blocked_by_broken 3, blocked_by_near_miss 4, min_sample 1. The residual ~44% inconclusive is mostly no_signal, and the report attributes it to the same reference-pool precision limit we explicitly accepted on raw/synthetic (J12): correct triggers each breaking 1–4 generic same-domain successes. Widening the margin further to rescue those would loosen a real safety trade-off, so we documented the limit instead of tuning past it. Whether F-001's specific candidate now passes lives in the committed per-trajectory artifacts; the corpus-level number above is the claim I am making, and it is the one with a confidence interval. The proof the other side of the gate still held Loosening a positive gate is only a win if the negative gate holds, and v0.3.1 made that a measured claim instead of an aspiration: Safety measure v0.3.1 result nearmiss (rejection corpus) 27/50 gate-silenced by recovery detection; of 23 active, 0 accepted — Wilson 95% CI [0.857, 1.000] adversarial/* (9 corpora) 0 accepted across injection, misleading, contradiction, unsafe, poisoning, tool_output_injection, compounding_multiturn, unsafe_realistic, and both harmbench slices — both models Generic triggers 0.4% (gpt) / 0.7% (llama) vs 2 restored the distinction. Precision-first ranking. It favored low-recall rules and starved the corpus of the specific, high-recall ones. Ranking should optimize what you promote. Questions we still can't answer Is 0.10 the right margin, or should it be per-domain? Git domains are token-dense; browser-tool domains may not be. We have one margin and no curve. Can "0 accepted" be claimed where the model goes silent? J16: llama-3.1-8b returned zero candidates on unsafe_realistic 13/20, contradiction_harmbench 10/15, and misleading_harmbench 4/15. Those zeros are lower bounds — silence is not rejection. The gate is safe either way, but the report must not dress a non-answer as a measured rejection. The two protocol metrics we still haven't run. Cross-session repeat-failure reduction (#741) and human-vs-replay agreement (#742): tooling complete, protocols not run. They are the difference between "core gate closed" and "full gate closed," and neither is a blocker for the core promotion path. Where does J12 stop? The accepted raw/synthetic precision limit is real, but "accepted" is a boundary we drew. Re-drawing it is a safety trade-off decision, not an engineering one — and it hasn't been made by anyone with more information than us. What I learned Overlap is not interference. A relation between two pieces of text is not a relation between two events. If your scorer counts "shares a token" as "broke," add a margin or a domain gate — demand more than coincidence before counting a collision. Loosening a gate is testable, and the test is the negative corpus. The 40 points we gained on failures/positive are only news if nearmiss and the nine adversarial corpora say zero afterwards. Run them in the same sweep, and publish both tables in the same report. "Net positive" is a definition worth stealing. A rule that breaks fewer successes than it prevents failures is useful, full stop. Most scorers I have seen implement some muddle of precision and recall instead of that sentence. The sentence is simpler and it matches the intent. Report zeros with their denominators and their CIs. 0/23, [0.857, 1.000]. The lower bound is the honest number, and it is the number a skeptical reader will recompute. The broader lesson When a gate kills your good outputs, find the single relation it is mis-grading — here, token overlap masquerading as interference — and replace it with a relation that costs almost nothing to compute: same domain, plus a margin. Then prove the other side of the gate with a measured zero, not a quiet one. That closes the v0.3.1 arc: an honest ruler, a full haystack, a reachable floor, and a judge that finally distinguishes coincidence from interference. What's left is the work the report names but the release does not: the 0-accepted corpora, the cross-session protocol, and the human-agreement run — the difference between a gate we closed and a gate we could close. References CauterRule v0.3.1 release notes v0.3.1 field test report (Fix 2 + Fix 3, §7 safety metrics, Appendix A J1/J12/J16) Docker field test results (180/180, +21 regression tests) SECURITY.md (source-trust gate, #727) v0.3.0 field test report (the F-001 diagnosis) User guide · Changelog CauterRule v0.3.1 is released. The scorer before/after, the safety CIs, and the J1/J12/J16 issue journal are in the field test report. The repo is public. Install with pip install cauterule. Changelog · Release notes