论文研究
arxiv.org ·
语言模型中免训练自报告置信度的分析
该 arXiv 预印本分析三种免训练置信度信号：随答案口头表达的置信度、事后 P(True)、以及同一 100 道 TriviaQA 问题上与三次额外生成的一致性，覆盖两个模型家族。结果显示，直接口头表达是意外强的基线，经基准错误审计后正确性预测 AUROC 达 0.956 和 0.937；三样本一致性明显更弱（0.765 和 0.790），与口头置信度的固定插值无统计可靠增益。一个模型 9 个错误中 4 个、另一个 8 个中 2 个获得全样本一致支持，说明自一致性可能放大共享误解。用等价提示对同一固定答案重新诱导置信度，平均变化 0.043–0.084，在 0.8 阈值下翻转 4%–9% 的决策。对 100 条带置信度标签的传记声明的探索性审计发现，被支持与被反驳声明之间的置信度差距有限。作者认为，有用的自报告仍对诱导方式、相关错误和基准噪声敏感。
AI推荐理由 arXiv 预印本，作者与摘要明确，实验设置（100 道 TriviaQA、两个模型家族、三种免训练置信度信号）具体可查，信息密度较高。核心新信息是：直接口头表达的置信度经基准错误审计后 AUROC 达 0.956/0.937，明显强于三样本一致性（0.765/0.790），固定插值无统计可靠增益；同一答案用等价提示重新诱导置信度平均变化 0.043–0.084，在 0.8 阈值下翻转 4%–9% 决策；自一致性会放大共享误解（某模型 9 个错误中 4 个获全样本支持）。局限是仅 100 题、两个模型家族，属预印本未经同行评审，传记声明审计为探索性。对 LLM 置信度评估与不确定性研究有参考价值，但非产品级落地证据。
72
正文
Computer Science > Computation and Language
arXiv:2609.20541 (cs)
[Submitted on 17 Sep 2026]
Title:An Analysis of Training-Free Self-Reported Confidence in Language Models
Authors:Lukas Meyer, Sofia Rossi, Wei Chen, Thomas Laurent, Yiming Li
View a PDF of the paper titled An Analysis of Training-Free Self-Reported Confidence in Language Models, by Lukas Meyer and 4 other authors
View PDF
HTML (experimental)
Abstract:Large language models can report a numerical confidence together with generated content, but it is unclear whether this report is more than calibrated rhetoric. We analyze three training-free signals: confidence verbalized with the answer, post-hoc $P(\mathrm{True})$, and agreement with three additional generations on the same 100 TriviaQA questions for two model families. Direct verbalization is a surprisingly strong baseline: after auditing benchmark errors, it reaches AUROC 0.956 and 0.937 for correctness prediction. Three-sample agreement is substantially weaker (0.765 and 0.790), and a fixed interpolation with verbalized confidence has no statistically reliable benefit. Four of nine errors from one model and two of eight from the other receive unanimous sample support, showing that self-consistency can amplify shared misconceptions. Re-eliciting confidence for the same fixed answers with equivalent prompts changes scores by 0.043 to 0.084 on average and flips 4\% to 9\% of decisions at a 0.8 threshold. An exploratory audit of 100 confidence-tagged biography claims further finds only a modest confidence gap between supported and contradicted claims. These results argue that useful self-reports remain sensitive to elicitation, correlated errors, and benchmark noise.
Comments:
Subjects:
Computation and Language (cs.CL)
Cite as:
arXiv:2609.20541 [cs.CL]
(or
arXiv:2609.20541v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.20541
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Jay Chen [view email]
[v1]
Thu, 17 Sep 2026 15:10:28 UTC (102 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled An Analysis of Training-Free Self-Reported Confidence in Language Models, by Lukas Meyer and 4 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)