智能体
arxiv.org ·
语言模型群体在推理任务中重演人类审议时高估共识
该 arXiv 预印本将 100 组留出的人类 Wason 任务讨论群体与匹配的 LLM agent 群体进行对照，用同一套评分代码评估。人类全共识率随评分定义不同在 24.0%–57.0% 之间，而 agent 群体在基于提交（n=98）和参与度匹配（n=45）两种敏感性分析中分别高出约 34–44 个百分点；去除可记忆答案后 reasoning 模式群体近乎一致但多落在错误答案上。作者指出模拟共识并不追踪集体准确率，信念锚定的 agent 群体在该设定下是人类群体结果分布的有偏估计。
AI推荐理由 arXiv 预印本，作者 Tengfei Shao，摘要给出具体实验设计与量化结果：用 100 组留出的人类 Wason 任务群体与匹配的 LLM agent 群体对照，人类全共识率在不同评分定义下为 24.0%–57.0%，agent 群体在基于提交（n=98）和参与度匹配（n=45）两种敏感性分析中分别高出 34.0/43.9 和 34.1/44.4 个百分点，且去除可记忆答案后 reasoning 模式几乎一致但多为错误答案。新信息在于揭示 LLM 群体模拟人类审议时会系统性高估共识且不追踪集体准确率，对多智能体仿真与集体智能研究有方法学警示。局限是单一作者预印本、尚未经同行评审，仅覆盖 Wason 任务一种场景，不能外推到所有审议任务。
78
正文
Computer Science > Artificial Intelligence
arXiv:2609.20543 (cs)
[Submitted on 17 Sep 2026]
Title:Language-model groups overstate consensus when replaying human deliberation on a reasoning task
Authors:Tengfei Shao
View a PDF of the paper titled Language-model groups overstate consensus when replaying human deliberation on a reasoning task, by Tengfei Shao
View PDF
Abstract:Full-consensus rates are often treated as indicators of collective cognition, yet depend on how participation and final states are operationalized. We replayed 100 held-out human Wason groups with matched large language model (LLM) agent groups, seeding one belief-anchored agent per participant's pre-discussion answer and scoring agents and people with the same code. Across human scoring definitions, estimates ranged from 24.0% to 57.0%; about one fifth of participants never posted, whereas agents almost always did. Agent groups remained more consensual in two post-unblinding sensitivity analyses: the submit-based comparison (n = 98) yielded gaps of 34.0 and 43.9 percentage points for chat and reasoning modes, and the participation-matched comparison (n = 45) yielded gaps of 34.1 and 44.4 points. These complementary routes reduced different measurement asymmetries yet converged within 0.5 percentage points. The gap persisted without early stopping and under a reparameterization removing the memorizable answer; reasoning-mode groups then agreed nearly unanimously, mostly on incorrect answers. Simulated consensus did not track collective accuracy, and belief-anchored agent groups were biased estimators of the human group-outcome distribution in this setting. These analyses provide a scoring-explicit basis for assessing simulated-group estimates of human deliberative outcomes.
Comments:
Subjects:
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); Multiagent Systems (cs.MA)
Cite as:
arXiv:2609.20543 [cs.AI]
(or
arXiv:2609.20543v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.20543
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Tengfei Shao [view email]
[v1]
Thu, 17 Sep 2026 15:11:08 UTC (851 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Language-model groups overstate consensus when replaying human deliberation on a reasoning task, by Tengfei Shao
View PDF
view license
Current browse context:
cs.AI
new
|
recent
| 2026-09
Change to browse by:
cs
cs.CL
cs.CY
cs.MA
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)