大模型
arxiv.org ·
通过硬接地偏好监督减少多模态大语言模型中的幻觉
该论文研究将后训练监督集中在“硬接地边界”——即偏好与拒绝回答语义接近、但在视觉证据支持上存在差异的情形——以降低多模态幻觉。方法在冻结视觉编码器和跨模态对齐路径的前提下，先用 SFT 建立解码器侧多模态行为，再构造硬接地偏好对进行 DPO，覆盖细粒度识别、空间推理、OCR、证据模糊或不足以及假前提查询。所得 DPO 模型在 DocVQA、TextVQA、MMBench 和 VQAv2 上较 SFT 初始化一致提升，并在约 839 个额外多模态样本上，由三个独立 LLM 评审给出 6.2–8.2 个百分点的成对胜率提升。作者同时发布 HardVQA-DPO 资源，含 3K 以上硬接地 SFT 样本和初始 DPO 偏好对。结果表明，在固定多模态表示下仅做解码器侧适配即可改善部分接地失败，同时保持更广泛的多模态能力。
AI推荐理由 arXiv 论文（v4，2026-09-26 修订），提出在冻结视觉编码器与跨模态对齐路径下，先用 SFT 建立解码器多模态行为，再针对“硬接地边界”构造偏好对做 DPO，覆盖细粒度识别、空间推理、OCR、证据不足与假前提查询。在 DocVQA、TextVQA、MMBench、VQAv2 上较 SFT 初始化一致提升，并在约 839 个额外样本上由三个独立 LLM 评审给出 6.2–8.2 个百分点的成对胜率提升；同时发布 HardVQA-DPO 数据集（3K+ 硬接地 SFT 样本及初始 DPO 偏好对）。局限：仅解码器侧适配、视觉表示固定，LLM 评审胜率不等同于人工验证，数据集规模仍有限。
72
正文
Computer Science > Computer Vision and Pattern Recognition
arXiv:2605.16411 (cs)
[Submitted on 13 May 2026 (v1), last revised 26 Sep 2026 (this version, v4)]
Title:Reducing Hallucination in Multimodal Large Language Models through Hard Grounding Preference Supervision
Authors:Qinwu Xu
View a PDF of the paper titled Reducing Hallucination in Multimodal Large Language Models through Hard Grounding Preference Supervision, by Qinwu Xu
View PDF
HTML (experimental)
Abstract:Hallucination remains a major challenge in vision-language models (VLMs), particularly when linguistically plausible responses are unsupported by visual evidence. We study whether multimodal hallucination can be reduced by concentrating post-training supervision on hard grounding boundaries, where preferred and rejected responses are semantically close but differ in their support from observable visual evidence. Under a frozen visual encoder and cross-modal alignment pathway, we first use supervised fine-tuning (SFT) to establish broad decoder-side multimodal behavior and then construct hard grounding preference pairs for Direct Preference Optimization (DPO). These pairs target evidence utilization, calibration, and grounding consistency across fine-grained recognition, spatial reasoning, OCR, ambiguous or insufficient evidence, and false-premise queries. The resulting DPO model consistently improves over its SFT initialization on DocVQA, TextVQA, MMBench, and VQAv2. Across approximately 839 additional multimodal examples, it further achieves 6.2--8.2 percentage-point higher pairwise win rates under three independent LLM judges. We additionally release HardVQA-DPO, a curated and growing resource with more than 3K hard-grounding SFT examples and an initial set of DPO preference this http URL://huggingface.co/datasets/vlmgrounding/hardvqa-sft-dpo These results show that decoder-side adaptation alone can improve a meaningful subset of grounding failures under a fixed multimodal representation, while preserving broader multimodal capabilities.
Subjects:
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Databases (cs.DB); Machine Learning (cs.LG)
Cite as:
arXiv:2605.16411 [cs.CV]
(or
arXiv:2605.16411v4 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2605.16411
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Qinwu Xu [view email]
[v1]
Wed, 13 May 2026 15:37:51 UTC (3,496 KB)
[v2]
Thu, 6 Aug 2026 13:01:18 UTC (3,452 KB)
[v3]
Tue, 11 Aug 2026 16:06:17 UTC (28,234 KB)
[v4]
Sat, 26 Sep 2026 16:27:15 UTC (35,589 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Reducing Hallucination in Multimodal Large Language Models through Hard Grounding Preference Supervision, by Qinwu Xu
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CV
new
|
recent
| 2026-05
Change to browse by:
cs
cs.AI
cs.CL
cs.DB
cs.LG
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)