IDKMesh 实验：25 个验证器面板的有效规模仅约 1.00

Generation got cheap. Trustworthy review did not. So we add reviewers. More eyes on the PR, more verifiers in the gate, a panel of LLM judges instead of one. The assumption underneath is that each additional reviewer adds independent evidence. That assumption is measurable. I measured it, and it did not hold. What IDKMesh is IDKMesh is an open-source research project (Apache-2.0, Python 3.11+) asking how humans, AI agents, tools, and heterogeneous compute can coordinate on uncertain goals and turn proposals into verified useful work. It is a research preview, not production software, and the repository is deliberately conservative about what it claims: every number below traces to a committed experiment record you can re-run. The thesis in one line: reviewer count is not independent evidence count. How it works The coordination model treats generation as cheap and verification as the scarce resource: Work arrives as a bounded Work Unit that must declare its security bounds. Replaceable workers attempt it. Multiple attempts are expected; candidates are disposable. A separate, verifier-owned evaluation plan judges those candidates. Worker completion is not acceptance. Results become reproducible evidence - result manifests, verification results, provenance digests. Integration is an explicit decision gated on that evidence, not on a green impression. The bundled demo makes the boundary concrete: it runs the real validators against committed fixtures and deliberately rejects four invalid ones - including a task with no security contract, and a worker result that accepts itself. Measuring a gate idkmesh gate-audit takes verdicts you have already collected and reports what the panel is actually worth. Here is the bundled example: { "schema": "gate-audit-report-v0.1", "gate_id": "example-five-verifier-gate", "evidence_class": "synthetic", "inputs": { "candidates": 15, "verifiers": 5, "known_good": 8, "known_bad": 4 }, "panel": { "nominal_votes": 5, "mean_verifier_accuracy": 0.8, "mean_pairwise_error_correlation": 0.0916, "effective_votes": 1.6944, "heuristic_n_eff": 3.6588 }, "probes": { "total": 3, "breached": 2, "breach_rate": 0.667 }, "warnings": ["2/3 seeded known-bad probes were accepted by the panel"] } Enter fullscreen mode Exit fullscreen mode Read the third key before the interesting one: "evidence_class": "synthetic". Those five reviewers are invented, and 1.69 is a demonstration of the arithmetic, not a measurement of anything real. The project keeps a hard line between synthetic demonstration and observed evidence - so here is the observed side. The observed result Experiment E017 built a panel where every verifier is a program, not a model: a partial test oracle that draws inputs from one named region of a problem's input domain (tiny, small, large, extreme, duplicate) and accepts a candidate only if it matches a reference implementation on all of them. 5 regions x 5 seeds = 25 verifiers, run over a 72-candidate corpus whose ground truth comes from executing hidden tests. Every verifier's Youden J is significantly positive after Bonferroni correction - mean accuracy 0.7956. So the panel genuinely works, individually. Error correlation between them: same region (declared dependent) mean rho = +0.8924 diff region (declared independent) mean rho = +0.5263 all pairs mean rho = +0.5873 Enter fullscreen mode Exit fullscreen mode Verifiers that share no declared attribute still share 53% of their errors. A metadata group boundary is not an independence boundary. Then the part that matters: 25-verifier majority error : 0.2083 single verifier : 0.2044 measured effective size : 1.00 (of 25 nominal) N/(1+(N-1)rho) heuristic : 1.66 Enter fullscreen mode Exit fullscreen mode The panel was worth no more than one of its members. Twenty-five nominal votes, an effective size of 1.00, and the standard correlation correction still overstated it by 1.66x. The whole measurement ran in about 5 seconds on one laptop. Why the usual correction doesn't rescue this The familiar fix is to discount for correlation with N_eff = N/(1+(N-1)rho). Experiment E015 tested that heuristic across a parameter grid and found it exact at rho=0 and rho=1, wrong in between, and - importantly - the sign of its error is not fixed. It is conservative where verifiers are weak, which is where conservatism is cheap. It is optimistic where verifiers are accurate and share modest dependence, which is exactly the regime you care about. At p=0.90, rho=0.125 the measured effective size saturates at 4.60 against a heuristic asymptote of 8.00 - and where an independent 9-verifier panel would deliver balanced error 0.000891, the real panel delivers 0.0125. Fourteen times worse. What this does not claim Being precise about scope, because it is easy to over-read: No AI review panel has been measured in this repository. E017's verifiers are programs. Whether LLM reviewers exhibit the same correlation structure is an open question - and the one I most want data on. It is a research preview. Not production software. It is not on PyPI yet, so installation is from a clone. Try it git clone https://github.com/MSKazemi/idkmesh && cd idkmesh python -m venv .venv && source .venv/bin/activate python -m pip install -r requirements-phase0.txt python scripts/demo.py Enter fullscreen mode Exit fullscreen mode No model account or API key needed. Repository: https://github.com/MSKazemi/idkmesh If you have verdict logs from a real review gate - human reviewers, LLM judges, or CI checks - pointing gate-audit at them is the experiment I would most like to see someone else run. Especially if it disagrees with the above.