大模型
arxiv.org ·
多模态大语言模型中的对齐幻觉
论文对 13 个多模态大语言模型（覆盖 5 个家族、0.5B 至 72B 参数）的视觉流做受控干预：将投影器输出的视觉 token 替换为高斯噪声后任务准确率明显下降，但 CKA、SVCCA、MIR 和主角度余弦等四种常用标量对齐指标无法稳定区分被破坏的视觉流与原始流，作者称之为“对齐幻觉”，并将其归因于语言模型共享通路中各向异性的 MLP 降维把视觉与文本 token 拉向共同输出方向，形成权重诱导的对齐。为此作者提出主角度间隙（PA gap），即前两个主角度余弦之差，用于区分权重诱导相似性与多方向视觉结构；在渐进式视觉破坏下，PA gap 与任务准确率的一致性优于所考察的标量指标，在结构化但无关的图像下还能揭示内部几何与任务准确率分离的情形。作者认为 MLLM 的内部视觉-文本对齐应被理解为语言模型内部视觉流的几何诊断，而非内容级跨模态交互的直接代理，需结合受控任务证据来校准。
AI推荐理由 arXiv 预印本，摘要给出了明确的方法与实验设计：对 13 个 MLLM（5 个家族，0.5B–72B）做视觉流受控干预，发现 CKA、SVCCA、MIR 和主角度余弦等标量对齐指标无法区分被高斯噪声替换的视觉 token，并提出 PA gap 作为区分权重诱导相似性与多方向视觉结构的诊断量。这是对多模态对齐评估方法的具体质疑与新指标，对 MLLM 可解释性和评测有实际参考价值；局限是仅预印本、未经同行评审，且摘要未给出完整数值结果。
78
正文
Computer Science > Computer Vision and Pattern Recognition
arXiv:2609.30210 (cs)
[Submitted on 24 Sep 2026]
Title:The Alignment Illusion in Multimodal Large Language Models
Authors:Hong-Han Wang, Yuntao Wang, Hu Ding
View a PDF of the paper titled The Alignment Illusion in Multimodal Large Language Models, by Hong-Han Wang and 2 other authors
View PDF
HTML (experimental)
Abstract:Layer-wise visual-text similarity in Multimodal Large Language Models (MLLMs) is widely interpreted as evidence that the language model progressively integrates visual content into a shared representation space. This reading rests on the assumption that scalar alignment scores reflect content-level cross-modal interaction. To test this assumption, we apply controlled interventions to the visual stream. Across 13 MLLMs from five families spanning 0.5B to 72B parameters, replacing projector-output visual tokens with Gaussian noise sharply reduces task accuracy, yet four standard scalar measures (CKA, SVCCA, MIR, and the leading principal-angle cosine) fail to consistently separate the corrupted stream from the original. We call this failure the alignment illusion and trace it to the shared language-model pathway: anisotropic MLP down-projections pull visual and text tokens toward common output directions, producing weight-induced alignment. Because this component is essentially one-dimensional, we introduce the principal-angle gap (PA gap), defined as the difference between the top two principal-angle cosines, which separates weight-induced similarity from multi-directional visual structure. Under graded visual corruption, the PA gap tracks task accuracy more consistently than the scalar scores we consider; under a structured but irrelevant image, it further exposes regimes in which internal geometry and task accuracy come apart. Internal visual-text alignment in MLLMs is therefore best read as a geometric diagnostic of the visual stream inside the language model rather than a direct proxy for content-level cross-modal interaction, and is most informative when calibrated by controlled task evidence.
Comments:
Subjects:
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
Cite as:
arXiv:2609.30210 [cs.CV]
(or
arXiv:2609.30210v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2609.30210
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Hong-Han Wang [view email]
[v1]
Thu, 24 Sep 2026 17:42:29 UTC (5,766 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled The Alignment Illusion in Multimodal Large Language Models, by Hong-Han Wang and 2 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CV
new
|
recent
| 2026-09
Change to browse by:
cs
cs.LG
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)