当 AI 系统被设计成挑战自身决策时会发生什么？

Kaggle Benchmarking Challenge Submission This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked I started with a question: What happens if an AI system is designed so that it can try to prove its own decision wrong? I did not want to build another benchmark that rewards a model for saying “I may be wrong”, producing longer self-critiques, or running more checks. A useful system has to decide whether checking is actually worth the cost. So I built Decision-Conditioned Falsification, a deterministic environment containing three possible hidden mechanisms. The model knows the possible mechanisms, but not which one is active. It can compose experiments from primitive operations, observe the result, and eventually commit to an operational action. Experiments cost points. Bad operational decisions can cost much more. And crucially: more information is not always better. In some configurations, testing is optimal. In others, the experiment costs more than the information is worth. Sometimes the best choice is a robust action that works across several possible worlds. Sometimes it is rational to act on the most likely mechanism without testing at all. The benchmark therefore does not reward: number of experiments, amount of self-criticism, number of hypothesis revisions, information gained, or how often the model changes its mind. It rewards: net operational utility. The public benchmark contains six decision configurations across three hidden mechanisms, producing 18 isolated episodes per model. The environment itself is deterministic. The uncertainty comes from what the model does with incomplete information. Different experiments also distinguish different mechanisms. There is no single cheap test that reveals everything. So the real benchmark question becomes: What is worth finding out before acting? Models Tested The public Kaggle benchmark currently contains results from four model runs: GPT-5.4 nano Gemini 3.5 Flash Claude Haiku 4.5 Gemini 3.7 Flash Gemini 3.7 Flash was the model used during Kaggle task creation, and I preserved that run rather than silently discarding it. The other three were selected before final evaluation to provide several model families instead of filling the leaderboard with variants from a single provider. The public leaderboard compares the models on the stable benchmark. But the part that changed my thinking came from using the same environment to test progressively stronger ways of making falsification part of the system itself. Findings 1. Telling the model to challenge itself was not enough The first controlled experiment compared two systems using the same model, environment, tools, budgets and output structure. The key difference was simple: one condition explicitly instructed the model to consider evidence capable of contradicting assumptions supporting its intended plan. The result was: Explicit falsification − Structured planner: −1.45 points 95% paired bootstrap interval: [−2.93, +0.01] I did not observe a robust aggregate benefit from making counterevidence-seeking explicit. That immediately raised another possibility. Perhaps the problem was not willingness to falsify an assumption, but the model's ability to calculate what its own hypothesis actually predicts. The traces supported that concern: models sometimes declared outcomes that were physically impossible under every mechanism in the environment. So instead of making the prompt more insistent, I changed the system. 2. I made hypotheses executable I introduced a deterministic function that could answer a question like: If mechanism W1 were true, what would this experiment produce? That computation was kept strictly separate from observing the hidden world. Conceptually: predict(hypothesis, experiment) ≠ run_experiment(experiment) Enter fullscreen mode Exit fullscreen mode predict calculated the consequences of a hypothetical mechanism using public rules. run_experiment interacted with the actual hidden machine. The prediction tool could not reveal the true world, calculate value of information, recommend an experiment, or choose an action. It seemed much closer to the AI-native architecture I had originally imagined. Then something unexpected happened. The model almost never used it. Across 360 assisted episodes, executable prediction was invoked once. The prespecified factorial interaction was: −2.71 points 95% interval: [−5.33, −0.60] But that result should not be read as the effect of using executable hypotheses. The capability was almost never activated. What the experiment actually demonstrated was more basic: Making a capability available inside an agent does not mean that capability participates in its decisions. So I removed that choice too. 3. I put the consequences directly into the decision path In the final experiment, the model first produced a provisional decision. Before that decision became an action, the system automatically calculated its consequences under the public mechanisms and placed them into the model's context. The model could still: keep the original decision, replace it, perform a real experiment, or act immediately. Only the hypothetical calculation became automatic. But there was an obvious confound: perhaps any improvement would simply come from letting the model think twice. So the control condition received the same additional review step, but without the computed consequences. The study used: 10 complete blocks 180 paired units 360 trajectories The main result: Integrated consequences − Neutral review: −0.39 points 95% paired bootstrap interval: [−1.53, +0.80] At the first review step, where both branches began from exactly the same provisional decision, the difference in decision value was: −0.088 points 95% interval: [−0.454, +0.402] The projection was actually delivered in 170 of 180 applicable first reviews. So this time the result could not be explained by the model simply failing to open the tool. The correct hypothetical consequences were in the decision context. I still did not observe an average improvement. The interval does not prove equivalence, and the result is limited to this environment, model and protocol. But it changed the question for me. Correct epistemic machinery is not automatically valuable decision machinery This became the main insight of the project. It is easy to collapse several different properties into one idea: The system can check something ↓ The check is logically correct ↓ The check reaches the model ↓ The model uses it well ↓ The final decision improves Enter fullscreen mode Exit fullscreen mode Those are not the same thing. Across these experiments: explicitly asking for counterevidence did not reliably improve outcomes; an executable prediction mechanism could exist and remain almost completely unused; automatically supplying correct hypothetical consequences guaranteed delivery, but still did not demonstrate better average decisions. This does not show that self-falsification is useless. It shows that an epistemic safeguard should not be considered successful merely because the safeguard itself is correct. It has to improve what the system eventually does. That changes the engineering target. Instead of: Make the model doubt itself. I would formulate it as: Build a path from hypothesis, to evidence, to action — and measure whether that path is worth its cost. The hardest part was not finding counterexamples One thing surprised me more than the headline comparisons. The difficult decision was often simply: Should I run another experiment at all? The exact solver exposed both directions of failure. Models sometimes stopped while another experiment still had positive decision value. They also sometimes paid for experiments when acting immediately was already better. This is why the benchmark keeps three ideas separate: information gain, falsification, and decision value. They are not interchangeable. An experiment can reveal a lot and still be irrelevant to the current decision. A hypothesis can be falsified without changing the best action. A model can even obtain enough evidence and still choose the wrong final action. A benchmark that scores only whether the model eventually discovered the hidden mechanism would miss all of this. What I would measure next I would not immediately add another self-critique stage or a more aggressive verification prompt. The interesting open problem is deeper: How should evidence become causally connected to action inside an AI-native system? The experiments here progressively made counterevidence easier to request, easier to calculate, and finally impossible to ignore at the interface level. That still did not guarantee better decisions. Understanding that last connection — between correct evidence and changed action — looks more important to me now than simply building stronger checking mechanisms. Limits This is deliberately a small, controlled environment. There are only three known mechanism families. Utilities are synthetic. The models are not inventing new scientific theories. The controlled architectural experiments also used one model family and ten-block samples, so the bootstrap intervals remain approximate. The experiments were separate protocol versions and should not be pooled into one effect estimate. These results therefore do not establish claims about all language models, all agents, or scientific reasoning in general. That limitation is intentional. I wanted an experiment capable of telling me that an architecture I found appealing did not produce the improvement I expected. Otherwise, it would not have been much of a falsification experiment. My Benchmark Decision-Conditioned Falsification 👉 https://www.kaggle.com/benchmarks/danielecangi/decision-conditioned-falsification The public benchmark evaluates models over six configurations and three hidden mechanisms. Its primary score is prior-weighted net operational utility, averaged across the six configurations. The exact solver is used only for post-decision diagnostics. It never tells the evaluated model: which hidden mechanism is active, which experiment to perform, or which final action is optimal. One display note: Kaggle currently renders the raw task score in the leaderboard table using percentage-style scaling, while the benchmark summary displays the underlying value correctly. I kept the frozen task unchanged and use the underlying score throughout the benchmark documentation. The benchmark is published under Apache 2.0. I started with: Can an AI system prove its own decision wrong when doing so costs something? After building it, I would ask something slightly different: Can we design the path from hypothesis, to evidence, to action so that discovering you are wrong actually improves what you do next? For me, that became the more interesting benchmark.