大模型
dev.to ·
@Elio Coppola
荷兰法院判决预测中把目标泄漏从 92% 降至 0.1%，并构建 Triple-A MCP Server
作者称在 609,715 份荷兰法院判决上，原始文本约 92% 直接包含判决结果（如“法院驳回请求”），导致模型只是复读答案。其做法是在训练前剥离 dictum、摘要行和结果宣告句，并持续测量残余结果标记，使泄漏降至约 0.1%。模型选用 LightGBM 而非 LLM，理由是无需 GPU、推理低于 10ms、树结构可追溯，且置信度低于 55% 时返回“不确定”。5 折交叉验证的 out-of-fold 结果为总体准确率 78.2%（多数类基线 43.7%）、Macro-F1 77.1%；刑法 82.6%、行政法 81.0%、民法 71.0%。该流程被封装为零外部依赖的 MCP server，提供基准统计、泄漏检测和结果风险分类三个工具，代码以 Apache 2.0 开源。上述数据与 Glama 评级均为作者自述，尚无独立验证。
AI推荐理由 作者自述在 609,715 份荷兰法院判决上把目标泄漏从 92% 降到 0.1%，并给出 5 折交叉验证的 out-of-fold 指标（总体准确率 78.2%，多数类基线 43.7%，Macro-F1 77.1%）及分领域结果，还开源了零依赖 MCP server。具体新信息是泄漏清洗流程、LightGBM 选型理由（<10ms 推理、可追溯、置信度低于 55% 返回“不确定”）和三个 MCP 工具。局限：数据、指标和 Glama 评级均为作者自述，未提供独立复现或第三方验证，安装/审计标签不能证明实际效果；MCP 接口本身对 Agent 集成有参考价值。
72
正文
When building AI for court outcome prediction, there is a massive hidden trap that invalidates most benchmarks: target leakage.
In Dutch court rulings, roughly 92% of raw texts contain the actual outcome verbatim (dictum or conclusion sentences like "the court dismisses the claim"). If you feed raw text to a model, it does not learn legal logic. It simply learns to read the answer back to you.
Here is how we solved this across 609,715 cases, built a traceable LightGBM model, and exposed it as a zero-dependency open-core MCP server.
1. The Pre-Training Cut: 92% to 0.1% Leakage
Before training any classifier, we implemented a strict sanitization step:
The dictum, summary lines, and outcome-announcing phrases are stripped from the text.
We continuously measure residual outcome markers.
Result: Leakage dropped from 92% to 0.1% (around 1 in 1,000 texts).
Only on this sanitized dataset did we train.
2. Why LightGBM Instead of an LLM
We intentionally picked LightGBM over deep neural networks or fine-tuned LLMs:
Fast and cheap: Sub-10ms inference without GPUs.
Traceable: Clear tree structures and feature importance.
Deterministic calibration: If confidence drops below 55%, the model does not guess. It returns "insufficient certainty".
3. Benchmark on 609,715 Cases (Out-of-Fold)
Evaluated through 5-fold cross-validation, strictly measured out-of-fold:
Overall Accuracy: 78.2% (against a 43.7% majority baseline)
Macro-F1: 77.1%
Per-Class F1:
Dismissed: 0.827
Partly granted: 0.726
Granted: 0.761
Domain Breakdown:
Criminal Law (n=105,151): 82.6% accuracy, 0.804 macro-F1 (strongest performance)
Administrative Law (n=316,273): 81.0% accuracy, 0.685 macro-F1 (high accuracy, but government victory is the majority class)
Civil Law (n=188,177): 71.0% accuracy, 0.656 macro-F1 (most complex due to factual nuances)
4. Model Context Protocol (MCP) Interface
To make this accessible to AI assistants (Claude, Cursor, autonomous agents), we wrapped the pipeline into an MCP server.
Zero external dependencies: Single Python file using only standard library (sys, json, urllib).
Audited on Glama: Triple-A rating (5/5 on coherence and completeness).
3 Tools:
rechtspraak_cijfers (keyless): Benchmark statistics and baseline metrics.
lekkage_check (keyless): Paste any legal text to test for outcome leakage before and after the cut.
voorspel_uitkomst (key required): Outcome risk classification.
Links
GitHub Repo (Apache 2.0): https://github.com/rechtssysteem-ai/rechtssysteem-mcp
Live Benchmark: https://rechtssysteem.ai/benchmark
Glama Audit: https://glama.ai/mcp/servers/@rechtssysteem-ai/rechtssysteem-mcp
Keyless API: https://api.rechtssysteem.ai/cijfers
Disclaimer: Not legal advice. Built as an open, verifiable yardstick for legal tech developers and agents.