无后见之明的时间证据资格基准

Kaggle Benchmarking Challenge Submission This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked In drug-discovery target evaluation, the hardest discipline is not letting hindsight leak into a judgment that's supposed to be made at an earlier point in time. Given a piece of clinical/genetic evidence and a decision date, the only question that should matter is: was this evidence actually available before that date? Everything else, including whether we now know the drug succeeded or failed, is irrelevant, and if it sneaks in, it invalidates the judgment. That's a simple rule (evidence_date ... tags before the answer, and the structured-output parser tried to parse the entire response (thinking block included) as JSON and choked. This is a genuinely useful, unglamorous insight for anyone benchmarking reasoning models with structured-output tooling: you need to strip the block before parsing, or the model can be right and still register as a total failure. What surprised me overall: I built this benchmark to catch models getting seduced by dramatic content into misjudging dates. That specific failure mode never showed up, not once, in any model, in any run. What showed up instead were two failure modes I hadn't designed for: run-to-run instability in a small model, and an infrastructure mismatch with a reasoning model's output format. Both are more useful findings than the one I went looking for, because they're the kind of thing you only find by actually running the benchmark against a diverse model set instead of stopping at the first clean result. What I'd measure next: whether gpt-oss-20b's instability is temperature-driven (rerun at temperature 0 to see if it stabilizes), and whether stripping tags before parsing resolves deepseek-r1 entirely or whether its judgment quality on the trap items specifically is also worse once it's actually scorable. My Benchmark https://www.kaggle.com/code/jeonghosong/new-benchmark-task-fe5fd