大模型
arxiv.org ·
FlexEE：面向卸载感知 LLM 推理的自推测与 KV 兼容早退框架
该论文提出 FlexEE，一种面向资源受限与权重卸载式 LLM 推理的早退框架。它通过逐层退出监督实现可靠的中间层预测，在 Top-K 局部词表上做自推测解码以低成本决定是否退出，并用动态隐状态管理保证 KV-cache 正确与内存感知执行。作者报告在 Llama2-7B 和 Llama3-8B 上，0%/50% 权重卸载条件下分别取得最高 1.27×/3.16× 和 1.25×/2.83× 的端到端加速，且精度下降很小。
AI推荐理由 arXiv 新提交论文，提出 FlexEE 早退框架，针对 offloading 场景的 LLM 推理，结合逐层退出监督、Top-K 局部词表自推测解码和 KV-cache 兼容的动态隐状态管理；摘要给出 Llama2-7B 与 Llama3-8B 在 0%/50% 权重卸载下最高 1.27×/3.16× 与 1.25×/2.83× 端到端加速，并称精度损失极小。信息密度较高、来源可查，但结果仅为作者自报，尚无第三方复现或代码链接，实际收益需进一步验证。
72
正文
Computer Science > Artificial Intelligence
arXiv:2609.17008 (cs)
[Submitted on 15 Sep 2026]
Title:FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference
Authors:Qihu Xie, Ziwei Li, Yi Kang
View a PDF of the paper titled FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference, by Qihu Xie and 2 other authors
View PDF
HTML (experimental)
Abstract:Large language model (LLM) inference is often constrained by both computation and memory, especially in offloading-based deployments where model weights are transferred across memory hierarchies during autoregressive decoding. In this setting, reducing the number of executed layers can lower per-token latency while also avoiding costly weight movement. Motivated by this observation, we present FlexEE, an early exiting framework for resource-constrained and offloading-based LLM inference. FlexEE makes early exiting practical for LLM decoding through layer-wise exit supervision for reliable intermediate-layer prediction, self-speculative decoding over a Top-K local vocabulary for low-cost exit decisions, and dynamic hidden state management for KV-cache-correct and memory-aware execution. Across generative and downstream tasks, FlexEE enables efficient early exit with minimal accuracy degradation, delivering up to 1.27$\times$/3.16$\times$ and 1.25$\times$/2.83$\times$ end-to-end speedups on Llama2-7B and Llama3-8B under 0\%/50\% weight offloading, respectively.
Subjects:
Artificial Intelligence (cs.AI)
Cite as:
arXiv:2609.17008 [cs.AI]
(or
arXiv:2609.17008v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.17008
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Qihu Xie [view email]
[v1]
Tue, 15 Sep 2026 11:20:23 UTC (534 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference, by Qihu Xie and 2 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)