大模型
arxiv.org ·
对齐悖论：后训练如何放大语言模型中的自信幻觉
该 arXiv 论文提出“对齐悖论”，指出后训练对齐本身是 LLM 高置信幻觉的主要驱动因素。在五个模型家族的事实性基准上，未对齐基座模型在长尾事实查询中很少产生高置信错误，而指令微调模型将高置信错误（p≥0.95）放大约 10 至 35 倍。Logit Lens 逐层探测显示，这种过度自信出现在后层，错误答案 margin 在早期和中间层接近零后扩大到 4.0 以上。作者据此提出在 DPO 中加入熵依赖的 margin bound，在 Mistral-7B 多轮实验中，相比标准 DPO 将高置信错误最多降低 35.3%，同时保持通用推理基准表现。
AI推荐理由 arXiv 论文，摘要给出具体量化结果：在五个模型家族的事实性基准上，指令微调模型的高置信错误（p≥0.95）比未对齐基座模型放大 10–35 倍；Logit Lens 逐层探测显示过度自信出现在后层，错误答案 margin 超过 4.0；并在 Mistral-7B 多轮实验中用熵依赖的 margin bound 改进 DPO，使高置信错误相对标准 DPO 最多降低 35.3%，同时保持通用推理基准表现。新信息在于把自信幻觉归因于后训练对齐本身并提出可操作的约束方法，对模型可靠性与对齐研究有直接价值。局限是结论基于摘要所述实验，尚未见独立复现，且仅报告单一模型的多轮实验。
82
正文
Computer Science > Computation and Language
arXiv:2609.32617 (cs)
[Submitted on 26 Sep 2026]
Title:The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models
Authors:Qingjia Huang, Yakai Li, Jianguo Wu, Qihang Zhou, Aimin Yu, Xiaoqi Jia, Luping Ma, Weijuan Zhang
View a PDF of the paper titled The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models, by Qingjia Huang and 7 other authors
View PDF
HTML (experimental)
Abstract:Large language models (LLMs) can produce factually incorrect answers with high confidence, undermining their reliability and limiting the effectiveness of uncertainty-based error detection. While prior research attributes confident hallucinations to factors such as missing knowledge in training data, reasoning errors, or stochastic decoding, we uncover that post-training alignment itself is a primary driver of these errors, a phenomenon we call the \textbf{Alignment Paradox}. Across five model families evaluated on factual benchmarks, unaligned base models produce few high-confidence errors on long-tail factual queries, whereas instruction-tuned models multiply high-confidence errors ($p \ge 0.95$) by more than an order of magnitude (10$\times$ to 35$\times$). Layer-wise probing with the Logit Lens reveals that this overconfidence emerges in late layers, where wrong-answer margins expand past 4.0 points after remaining near zero across early and intermediate layers. These findings motivate limiting margin growth during post-training. We implement this principle through an entropy-dependent margin bound in direct preference optimization (DPO). In multi-epoch experiments with Mistral-7B, the bounded objective reduces high-confidence errors by up to 35.3\% relative to standard DPO while maintaining performance on evaluated general reasoning benchmarks. These results show that bounded margins mitigate confident hallucinations during post-training.
Comments:
Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:
arXiv:2609.32617 [cs.CL]
(or
arXiv:2609.32617v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.32617
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Yakai Li [view email]
[v1]
Sat, 26 Sep 2026 13:39:15 UTC (1,003 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models, by Qingjia Huang and 7 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
cs.AI
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)