你的六个 AI 模型其实只有两个：相关错误的数学

On October 9, Vals AI published a test of whether a team of coding agents is worth the price [1]. They had GPT 6 Sol and Claude Opus 5.5 build 50 web apps from specifications, each model alone and as a team in which a lead hands parts of the build to subagents, at medium and at maximum reasoning effort. The teams cost 1.8 to 5.1 times as much. Of the four team-versus-single comparisons, one was statistically significant: Sol at medium effort, which gained 7.3 points. For Sol, raising the single agent's reasoning effort from medium to max added 11.4 points, more than the 7.3 a team added at medium effort [1]. That study is about splitting a job into parts, so it says nothing directly about whether agents make the same mistakes. It did leave me with a question a mathematician can attack. When you add another model to a vote, how many independent opinions do you actually add? The plan is to look at what a vote promises, simulate how it can fail, put a formula on the failure, and then measure it on six real models. What a vote is supposed to buy The classical answer is Condorcet's jury theorem, from 1785 [2]. Suppose every voter is right with probability p > 1/2 and the voters are independent. The chance that the majority of k voters is right is P(majority right)=∑j=⌊k/2⌋+1k(kj) pj(1−p)k−jP(\text{majority right}) = \sum_{j=\lfloor k/2 \rfloor + 1}^{k} \binom{k}{j}\, p^{j}(1-p)^{k-j}P(majority right)=j=⌊k/2⌋+1∑k​(jk​)pj(1−p)k−j and it climbs toward 1 as k grows. With p = 0.7, three voters are right 78.4% of the time, five 83.7%, nine 90.1%, and twenty-five 98.3%. Figure 1 plots the formula for three levels of individual accuracy. Chance that the majority is right for independent voters, by number of voters and individual accuracy. Image by author Every number in that paragraph leans on the word independent. Two models that make the same mistake cast one vote between them, whatever the ballot count says. The catch: friends who studied from the same notes Imagine asking five friends a trivia question and going with the majority. If they know different things, one friend's wrong guess is likely to be outvoted by the others. Now suppose all five studied from the same notes, and the notes contain a mistake. Everyone repeats it, and the majority is wrong with total confidence. Five people, one source, one vote. Language models are not friends, but the worry carries over. Models trained on overlapping text, with similar designs, could plausibly share blind spots. Whether they do is an empirical question, and it is the one this article measures. A small simulation To see the difference, I simulated two panels of five models. Every model is right 70% of the time. In the first panel each model's mistakes land on questions independently of the others. In the second the mistakes overlap: some questions are hard for everyone, so the models tend to fail on them together. Figure 2 shows 40 questions from each panel. Each row is a model, each column is a question, and orange means wrong. The bottom row is the majority vote. Two simulated panels of five models, each right 70% of the time. Left: mistakes land independently. Right: mistakes overlap. Orange means wrong. Image by author Per model, the two panels are identical. The majority vote is not. With independent mistakes it is right 83.8% of the time, which is what Condorcet's formula predicts (83.7%). With overlapping mistakes it is right 74.4% of the time. More than nine points of accuracy disappear without any single model getting worse, and the panel is worth fewer opinions than it has members: 5.0 effective votes in the first case, 2.2 in the second. The next section explains where those two numbers come from. From a picture to a number A picture is not a measurement, so here is the standard way to put a number on it. For each model and each question, record a 1 if the model got it wrong and a 0 if it got it right. The correlation between two such columns is the phi coefficient, φ\varphiφ. It is 0 when two models' mistakes are unrelated and 1 when they fail on exactly the same questions. Take k voters whose error indicators E1 through EkE_1 \text{ through } E_kE1​ through Ek​ each have variance σ2\sigma^2σ2 and whose pairs all have correlation φ\varphiφ. The variance of their average error is Var(1k∑i=1kEi)=σ2k[1+(k−1)φ]\mathrm{Var}\left(\frac{1}{k}\sum_{i=1}^{k} E_i\right) = \frac{\sigma^2}{k}\left[1 + (k-1)\varphi\right]Var(k1​i=1∑k​Ei​)=kσ2​[1+(k−1)φ] Independent voters would give σ2/n\sigma^2/nσ2/n for n of them, so define the effective number of votes as the n that matches: neff=k1+(k−1)φn_{\mathrm{eff}} = \frac{k}{1 + (k-1)\varphi}neff​=1+(k−1)φk​ If φ=0,neff=k\varphi = 0 , n_{\mathrm{eff}} = kφ=0,neff​=kand every voter counts. As k grows, neff,1/φn_{\mathrm{eff}} , 1/\varphineff​,1/φ and stops there. With φ=0.3\varphi = 0.3φ=0.3, seven models are worth 2.5 votes, and no number of extra models like them would be worth more than 3.3. Survey statisticians use the same arithmetic for clustered samples, where 1+(k−1)φ1 + (k-1)\varphi1+(k−1)φ is what Kish called the design effect [3]. What a given correlation is worth The formula is easy to apply, so here it is as a lookup table. Rows are the average error correlation, columns are the number of models, and each cell is the number of effective votes. The last column is the ceiling that no number of extra models can beat. Average error correlation 3 models 5 models 10 models Ceiling 0.05 2.7 4.2 6.9 20 0.1 2.5 3.6 5.3 10 0.2 2.1 2.8 3.6 5 0.3 1.9 2.3 2.7 3.3 0.5 1.5 1.7 1.8 2 A correlation of 0.3 sounds modest, yet it makes ten models behave like fewer than three. In the simulation above, the overlapping panel has an average correlation of 0.31, which puts its five models at 2.2 effective votes. Figure 3 draws the same table as curves. Each one bends toward its ceiling, and the higher the correlation, the sooner it flattens. Effective votes against the number of models for four error correlations. Dotted lines mark each ceiling. Image by author What has already been measured Two recent papers measured this on language models. Kim et al. compared more than 350 models and found that, on one leaderboard dataset, two models agree 60% of the time when both are wrong, and that larger, more accurate models have highly correlated errors even across developers [4]. Kohli measured panels of LLM judges with a Kish effective sample size and found that nine frontier models from seven families carried about two independent votes' worth of information on natural-language inference [5]. A recent post repeated that measurement with five judges on 300 sentence pairs and landed inside Kohli's range [6]. I wanted a version anyone can rerun and get the same numbers, up to floating-point noise, so I used open models and read their answers as probabilities instead of sampling them. I also added a control for one bias that can fake agreement: every model gets its own random order of the answer options. The setup Figure 4 lays out the whole experiment before the details. The exam The test is MMLU, 57 multiple-choice subjects that range from elementary mathematics to law [7]. I drew 1,500 questions spread evenly over the subjects and set aside the questions whose options refer to each other ('all of the above'), because shuffling the options would break them. The models The panel is six open models of 3 to 4 billion parameters from five developers (Table 1). I also planned to include Llama 3.2 3B, but its gated download failed, so it is not in the panel. Open weights let me read exact probabilities instead of sampling answers. Reading answers without sampling Each model gets the standard five-shot MMLU prompt (three or four examples for about 3.5% of questions where five would not fit in 2,048 tokens), and I read the probability it assigns to the next token being A, B, C or D. Nothing in the main experiment is sampled, so a rerun should give the same answers, up to floating-point noise on near-ties. Exact ties between two options are rare and go to the earlier one. Removing the letter bias One detail matters more than it looks. Earlier work found that LLMs prefer particular option letters, mostly because the model assigns more probability to certain letter tokens a priori [8]. Two models that both lean toward A would agree on wrong answers for reasons unrelated to what they know. To take that out, every model sees each question with its own random ordering of the options, and I map its answer back to the original option before comparing anything. The weakest model I tried shows why this matters: a 1-billion-parameter model from AI2 put 74% of its answers on A or B, even though the right answer was equally likely to land on any letter. As a control, I also ran everyone on the original order. The measurement The measurement itself is a few lines: python import numpy as npdef effective_votes(pred, answer): # pred: (n_questions, k_models) array of the option each model chose # answer: (n_questions,) array of the correct option errors = (pred != answer[:, None]).astype(float) # 1 where a model is wrong phi = np.corrcoef(errors.T) # pairwise error correlation k = errors.shape[1] phi_bar = (phi.sum() - k) / (k * (k - 1)) # average of the off-diagonal entries return k / (1 + (k - 1) * phi_bar), phi_bar The function does four things. It marks every wrong answer, correlates those marks for every pair of models, averages the pair correlations, and puts that average into the formula. Everything else I ran (the prompts, the option shuffling, the bootstrap intervals) is plumbing around it. How often do they fail together? Table 1. Accuracy on 1,500 MMLU questions, with each model shown its own random option order. Intervals are 95% Wilson intervals. The last row was left out of the panel analysis because it scored under 50%. Model Developer Accuracy 95% interval Qwen3-4B Qwen 72.7% 70.4 to 74.9 Phi-4-mini-instruct Microsoft 67.5% 65.1 to 69.8 Qwen2.5-3B-Instruct Qwen 67.3% 64.9 to 69.6 Granite 4.1 3B IBM 66.5% 64.0 to 68.8 SmolLM3-3B Hugging Face 60.9% 58.4 to 63.3 Falcon3-3B-Instruct TII 58.3% 55.8 to 60.7 OLMo-2-0425-1B-Instruct AI2 40.0% 37.5 to 42.5 The most accurate model in the panel scores 72.7% and the least accurate 58.3%, against 25% for guessing among four options. Figure 5 is a seating chart of the panel: each cell is one pair of models, and the darker it is, the more often those two miss the same questions. Error correlation (φ) for every pair of models. Darker means the two fail on the same questions more often. Image by author The average pairwise error correlation, which I will write as φˉ\bar{\varphi}φˉ​, is 0.41 (95% bootstrap interval 0.39 to 0.44), where 0 would mean the mistakes are unrelated. No pair is anywhere near 0: all 15 fall between 0.33 and 0.48. The one pair from the same developer, Qwen2.5 and Qwen3, sits at 0.45, and the 14 pairs from different developers average 0.41. When two models are both wrong, they choose the same wrong answer 59% of the time, close to the 60% that Kim et al. report for one leaderboard dataset [4]. Random wrong answers would agree about a third of the time. How many votes did the panel cast? Putting φˉ\bar{\varphi}φˉ​ into the formula, the six models are worth neff=1.96n_{\mathrm{eff}} = 1.96neff​=1.96 independent votes (95% interval 1.88 to 2.04). A second estimate that does not assume every pair is equally correlated, k divided by the largest eigenvalue of the correlation matrix, gives 1.96 as well. Figure 6 shows neffn_{\mathrm{eff}}neff​ for every panel size, with the ceiling 1/φˉ1/\bar{\varphi}1/φˉ​ = 2.4 as the dotted line. Each added model contributes less than the one before: going from one model to two adds 0.4 effective votes, and going from five to six adds 0.07. Effective number of independent votes against the number of models. The dashed diagonal is what independent voters would give. Image by author Votes only matter if they change the answer, so I also ran the vote. For every possible panel of each size I took the most common answer and compared its accuracy with what independent voters of the same individual accuracy would score. I computed that baseline exactly, assuming a voter who is wrong picks uniformly among the three wrong options, and I used odd panel sizes so that ties do not blur the picture. Plurality-vote accuracy against panel size, compared with independent voters (dashed) and with a version where only subject difficulty is shared (dotted). Image by author With five models the independent baseline is 87.1% and the real vote scored 71.2% (95% interval 69.2 to 73.2), a gap of about 16 points. All six models together scored 71.8%, against 90.3% for independent voters. The best single model, Qwen3-4B, scored 72.7%, and none of the 57 possible panels of two to six models beat it when each model casts one vote. The best panel, three models, tied it at 1,091 of 1,500 questions. Counting only each model's top choice throws information away, so I also tried averaging the models' probabilities. That does better: all six together reach 73.8%, a gain of 1.1 points over Qwen3-4B with a 95% interval of -0.8 to 3.0 points, and the best of the 57 panels, picked after the fact, reaches 74.1%. Neither is distinguishable from the best model alone, and both fall well short of the 90% that independent voters would reach by plain voting. All six models agreed on 35.8% of the questions, and when they did the shared answer was wrong 5.8% of the time (31 of 537 questions). Independent voters would agree on only 7.8% of the questions, and would be wrong 0.008% of the time when they did. All six models were wrong on 112 questions (7.5%), which independent voters with the same accuracies would manage on about two. Where the shared error comes from Some of this correlation is not surprising. A hard subject is hard for every model, and that alone produces correlated errors. To see how much, I shuffled each model's results across questions within each subject, separately for every model. That keeps every model's exact accuracy in every subject and destroys any agreement on individual questions. With only subject difficulty left, the average error correlation falls from 0.41 to 0.09, so subject difficulty accounts for about a fifth of it and most of the rest sits at the level of individual questions. Figure 8 puts that next to the letter-bias control. With a shared option order the average correlation is 0.45, against 0.41 when each model sees its own random order. The gap between those two bars is what sharing the option order adds, about 0.04, and a paired bootstrap puts that difference between 0.02 and 0.06. Letter preferences were mild inside the panel: no model put more than 30% of its answers on any one letter, where an even split is 25%. Average pairwise error correlation with a shared option order, with each model's own random order, with only subject difficulty shared, and for independent voters. Image by author What this cannot do is separate a shared blind spot from a question that is simply hard for everyone, because both look like several models failing on the same item. The benchmark adds its own noise: an estimated 6.49% of MMLU questions contain errors [9], and a wrong answer key marks every model that knows better as wrong together. What if you ask one model many times? Self-consistency samples several answers from one model and keeps the most common one [10]. A cheap stand-in is to draw the answer letter from each model's own probabilities, several times, and vote. Figure 9 compares that with voting across different models. Effective votes (left) and accuracy gained over a single vote (right) for different models and for one model sampled repeatedly. Image by author Six samples from one model are worth 1.4 effective votes on average (range 1.3 to 1.7), and eight samples are worth 1.5, against 2.0 for six different models. The accuracy gain tells the same story. A vote of five samples from one model gains 2.2 points over a single sample's expected accuracy on average (range 1.3 to 3.6), while five different models gain 5.7 points over the average single model. The right panel of Figure 9 shows gains rather than accuracy levels, because a sampled answer is less accurate than a model's top choice and the two lines would otherwise start in different places. A letter drawn from a probability is not a sampled chain of thought, and real self-consistency samples reasoning paths. This comparison says little about how well real self-consistency works. It only compares two sources of votes under the same plain readout. What this does and does not say about agent teams Vals tested teams that split a job into parts [1]. I tested voters who answer the same question, on multiple-choice exams, with small open models. None of this is a verdict on agent teams. What carries over is the arithmetic: a vote is worth the independent information behind it, and the number that measures that can be computed on any labeled set where each model's mistakes are recorded. If you run a panel of judges, or a team in which agents check each other's work, compute φˉ\bar{\varphi}φˉ​ on a few hundred items with known answers before adding a member. A new model from a different developer is not automatically a new opinion. In this panel, switching developers lowered the error correlation only a little (0.45 for the one same-developer pair, 0.41 on average across developers), and no pair of models fell below 0.33. One same-developer pair is too few to say how far that generalizes. The ceiling 1/φˉ1/\bar{\varphi}1/φˉ​ also says where to stop adding voters and start adding something else: a tool call, a retrieval step, a test that can actually be run, a person. Frontier models behind APIs, chain-of-thought answers in place of letter probabilities, and the manually re-checked subset of MMLU [9] would each change these numbers in ways I can't predict from this run. Counting models is easy. Counting how many of them can be wrong in different ways takes one correlation matrix, and it is worth computing before the next model joins the panel. ··· References [1] Katrina Drozdov, Do Agent Teams Pay Off? A Case Study on Vibe Code Bench (2026), Vals AI. https://www.vals.ai/blogs/multi-agent-vibe-code-bench [2] Marquis de Condorcet, Essai sur l'application de l'analyse à la probabilité des décisions rendues à la pluralité des voix (1785), Imprimerie Royale [3] Leslie Kish, Survey Sampling (1965), John Wiley & Sons [4] Elliot Myunghoon Kim, Avi Garg, Kenny Peng, Nikhil Garg, Correlated Errors in Large Language Models (2025), ICML 2025. https://arxiv.org/abs/2506.07962 [5] Guneet Kohli, Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels (2026), arXiv. https://arxiv.org/abs/2605.29800 [6] David J. Knudson, My Four AI Judges Were Worth 1.6 Votes (2026), Level Up Coding. https://levelup.gitconnected.com/my-four-ai-judges-were-worth-1-6-votes-96a84e337405 [7] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt, Measuring Massive Multitask Language Understanding (2021), ICLR 2021. https://arxiv.org/abs/2009.03300 (MIT license) [8] Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, Minlie Huang, Large Language Models Are Not Robust Multiple Choice Selectors (2024), ICLR 2024. https://arxiv.org/abs/2309.03882 [9] Aryo Pradipta Gema et al., Are We Done with MMLU? (2025), NAACL 2025. https://arxiv.org/abs/2406.04127 [10] Xuezhi Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models (2023), ICLR 2023. https://arxiv.org/abs/2203.11171 The Bias Variance Tradeoff and How it Shapes The LLMs of Today Artificial Intelligence What makes chatGPT so good? What are the architectural assumptions behind the success and pitfalls of large language models like chatGPT? Michael Zakhary November 2, 20246 min read Agentic AI from First Principles: Reflection Agentic AI From theory to code: building feedback loops that improve LLM accuracy Mariya Mansurova October 24, 202520 min read Building a Chat App with LangChain, LLMs, and Streamlit for Complex SQL Database Interaction Sql Build and deploy a chat application for complex database interaction with LangChain agents. Hamza Gharbi February 9, 202417 min read Building a Сustom MCP Chatbot Artificial Intelligence Understanding all the details of the model context protocol Mariya Mansurova July 10, 202522 min read Your Next ‘Large’ Language Model Might Not Be Large After All Artificial Intelligence A 27M-parameter model just outperformed giants like DeepSeek R1, o3-mini, and Claude 3.7 on reasoning tasks Moulik Gupta November 23, 20259 min read Preference Alignment for Everyone! Frugal RLHF with multi-adapter PPO on Amazon SageMaker Aris Tsakpinis November 8, 202427 min read Build an AI Agent to Explore Your Data Catalog with Natural Language Artificial Intelligence Leverage LLMs to query your Databricks Data Catalog Fabiana Clemente June 16, 20256 min read Llama-2 vs. Llama-3: a Tic-Tac-Toe Battle Between Models Programming Making a non-scientific benchmark with Python and Llama-CPP Dmitrii Eliuseev April 27, 202410 min read How to Find and Solve Valuable Generative AI Use Cases Product Management 80% of AI projects fail due to poor use cases or technical knowledge. Gen AI reduced complexity, and now we must pick the right battles. Teemu Sormunen June 18, 20246 min read