大模型
arxiv.org ·
AdaVSkip：面向高效多模态大语言模型推理的跨层自适应视觉令牌跳过
该论文指出 MLLM 推理中视觉 token 的冗余程度与分布随输入变化，且在 self-attention 与 MLP 模块间存在差异。为此提出 AdaVSkip，在每层设置两个轻量路由器，独立决定视觉 token 是否跳过 self-attention 和 MLP，从而形成输入相关的计算路径；由于路由决策离散不可微，采用两阶段训练：第一阶段冻结主干、以模块必要性分数为监督目标训练路由器，第二阶段用强化学习结合答案正确性奖励与跳过一致性奖励优化路由。在 LLaVA-NeXT-7B 上报告 FLOPs 降低 53.2% 且保持原平均性能，叠加视觉 token 压缩后降低 91.2%、保留 97.2% 平均性能，并在三个 MLLM 主干上验证。
AI推荐理由 arXiv 新提交论文（v1，2026-09-14），提出 AdaVSkip：为每层配备两个轻量路由器，分别决定视觉 token 是否经过 self-attention 与 MLP 模块，形成输入相关的视觉计算路径；针对路由决策离散不可微的问题，采用两阶段训练（Stage I 冻结主干、用模块必要性分数做监督训练路由器；Stage II 用强化学习结合答案正确性奖励与跳过一致性奖励优化路由）。在 LLaVA-NeXT-7B 上报告 FLOPs 降低 53.2% 且保持原模型平均性能，与视觉 token 压缩结合后降低 91.2%、保留 97.2% 平均性能，并在三个 MLLM 主干上验证。局限：摘要未给出具体评测基准、延迟/显存实测与代码开源信息，效果为作者自报，尚待复现。
78
正文
Computer Science > Computer Vision and Pattern Recognition
arXiv:2609.15131 (cs)
[Submitted on 14 Sep 2026]
Title:AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLMs Inference
Authors:Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang
View a PDF of the paper titled AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLMs Inference, by Yuyao Sun and 3 other authors
View PDF
HTML (experimental)
Abstract:Multimodal large language models (MLLMs) require substantial computation to process numerous visual tokens across all transformer layers. Most methods for efficient MLLM inference exploit horizontal redundancy by compressing visual tokens. Beyond token reduction, recent studies exploit vertical redundancy through early exit or fixed-layer skipping. However, we find that the extent and distribution of this redundancy vary across inputs and differ between self-attention and MLP modules. Motivated by these observations, we propose AdaVSkip, which equips each layer with two lightweight routers that independently determine whether visual tokens pass through by or skip the self-attention and MLP modules. These decisions collectively define an input-specific visual-computation path, but their discrete and non-differentiable nature makes learning effective paths challenging. To address this challenge, we develop a progressive two-stage training framework that updates only the routers while keeping the backbone frozen. Stage I establishes an initial routing policy through supervised training with input-specific targets derived from module-wise necessity scores. To further align the routing policy with task performance, Stage II uses reinforcement learning to optimize routing decisions with direct feedback from generated answers. It combines an answer correctness reward with a skip-consistency reward that discourages excessive retention of visual-token computation. Across three MLLM backbones, AdaVSkip maintains strong task performance with substantially less computation. On LLaVA-NeXT-7B, AdaVSkip reduces FLOPs by 53.2\% while preserving the original model's average performance. Combining it with visual token compression increases this reduction to 91.2\%, while retaining 97.2\% of the original performance on average.
Subjects:
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Cite as:
arXiv:2609.15131 [cs.CV]
(or
arXiv:2609.15131v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2609.15131
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Yuyao Sun [view email]
[v1]
Mon, 14 Sep 2026 07:07:51 UTC (2,354 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLMs Inference, by Yuyao Sun and 3 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CV
new
|
recent
| 2026-09
Change to browse by:
cs
cs.AI
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)