大模型
arxiv.org ·
学习预测多模态大语言模型中的中间层注意力以进行视觉令牌剪枝
该论文针对 MLLM 视觉 token 剪枝中固定中间层次优、且获取注意力需先处理大量视觉 token 的问题，提出 Middle-layer Attention Prediction（MAP）：通过 Question Contrastive Teacher Selection 按样本选择教师层，并将该层注意力蒸馏到轻量预测器，从多模态输入特征估计 token 重要性，在首个语言模型层前结合多样性准则完成剪枝。在 LLaVA-NeXT-7B 的十个基准上，仅保留 5.56% 视觉 token 时保留 97.5% 原始性能，端到端加速 3.09x。
AI推荐理由 arXiv 论文（v2，2026-09-09 修订），提出 MAP 方法：用 Question Contrastive Teacher Selection 为每个样本挑选教师层，再蒸馏到轻量预测器，在首个语言模型层之前完成视觉 token 剪枝。摘要给出具体量化结果：LLaVA-NeXT-7B 上十个基准保留 97.5% 性能、仅用 5.56% 视觉 token、端到端加速 3.09x，信息密度高且可验证。局限是仅在单一模型与基准集上报告，未提供代码或第三方复现证据。
78
正文
Computer Science > Artificial Intelligence
arXiv:2608.06411 (cs)
[Submitted on 4 Aug 2026 (v1), last revised 9 Sep 2026 (this version, v2)]
Title:Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning
Authors:Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang, Hao Geng, Minjun Yu
View a PDF of the paper titled Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning, by Yuyao Sun and 5 other authors
View PDF
HTML (experimental)
Abstract:Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by the cost of processing numerous visual tokens. Visual token pruning can reduce this cost, but requires accurate token importance estimates. Recent studies have demonstrated that text-to-vision attention from middle language model layers can effectively guide visual token pruning, typically using attention from a predefined middle layer to select the visual tokens to retain. Two problems therefore remain. First, our analysis shows that the layer whose attention is most responsive to the question varies substantially across samples, making a fixed layer suboptimal. Second, obtaining attention from the appropriate middle layer requires processing numerous visual tokens through several language model layers, by which point considerable computation has already been spent. To address both problems, we propose Middle-layer Attention Prediction (MAP), which uses Question Contrastive Teacher Selection to identify a sample-specific teacher layer by contrasting attention under the original and reference questions, and distills attention from the selected layer into a lightweight predictor that estimates visual token importance from multi-modal input features. During inference, MAP combines the predicted importance scores with a diversity criterion to prune visual tokens before the first language model layer. Thus, MAP requires no attention maps for pruning and remains compatible with existing inference acceleration techniques. Across ten benchmarks on LLaVA-NeXT-7B, MAP retains 97.5% of the unpruned model performance with only 5.56% of the visual tokens, yielding a 3.09x end-to-end speedup.
Subjects:
Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
Cite as:
arXiv:2608.06411 [cs.AI]
(or
arXiv:2608.06411v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.06411
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Yuyao Sun [view email]
[v1]
Tue, 4 Aug 2026 05:44:07 UTC (1,680 KB)
[v2]
Wed, 9 Sep 2026 13:23:20 UTC (1,680 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning, by Yuyao Sun and 5 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new
|
recent
| 2026-08
Change to browse by:
cs
cs.CV
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)