智能体
arxiv.org ·
HBM 与高带宽闪存的冷热分层用于智能体 LLM 服务
该论文针对智能体 LLM 多轮会话中 KV 状态被逐出导致重算或慢速传输的问题，利用高带宽闪存（HBF）容量远大于 HBM 但读能耗高、写耐久有限的特点，按访问模式将 KV 状态分层：频繁读取的热集放 HBM，仅在会话恢复时读取的冷池放 HBF。在 Qwen3-Coder-30B-A3B 的智能体负载上，方案实现 14ms 的 token 间隔、在预填充基础上仅增加约 0.1ms 恢复延迟，每 GPU 可承载的并发会话数提升 24 倍，并使每 8-GPU 节点读功耗较全部 KV 放闪存降低 7.6kW，作者据此认为 HBF 是 HBM 的冷层补充而非替代。
AI推荐理由 arXiv 硬件架构论文，提出将 HBM 与高带宽闪存（HBF）按冷热分层用于智能体 LLM 服务：热 KV 集放 HBM、冷池放 HBF。摘要给出具体数据（Qwen3-Coder-30B-A3B 上 TBT 14ms、恢复延迟约 0.1ms、每 GPU 并发会话数提升 24 倍、每 8-GPU 节点读功耗降低 7.6kW），信息密度高、问题定义清晰，对 Agent 推理的显存与成本瓶颈有直接参考价值。局限在于为单篇论文的仿真/设计结果，尚无独立复现或生产部署证据，且 HBF 的写耐久与读能耗约束仍需验证。
82
正文
Computer Science > Hardware Architecture
arXiv:2609.25782 (cs)
[Submitted on 22 Sep 2026]
Title:Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving
Authors:Jongjin Baek, Won Ji, Seungjae Yoo, Joo-Young Kim
View a PDF of the paper titled Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving, by Jongjin Baek and 3 other authors
View PDF
HTML (experimental)
Abstract:Large language model (LLM) serving is increasingly agentic, with multi-turn sessions that idle between actions yet must retain their full context. Limited GPU memory capacity forces inactive KV states to be evicted, so resuming a session incurs either costly recomputation or slow interconnect transfers. To address this, high bandwidth flash (HBF)-an on-package 3D-NAND memory offering orders-of-magnitude greater capacity than high bandwidth memory (HBM) at comparable read bandwidth-has emerged as a strong candidate. However, its high read energy and limited write endurance make it impractical to serve all KV traffic. Fortunately, our analysis shows that agentic KV states exhibit distinct access patterns: a small hot set is read for every decoding step, while a large cold pool is read only when a paused session resumes. Exploiting this, we place the hot set in HBM and the cold pool in HBF, forming a hot-cold KV hierarchy within the GPU memory tier. On agentic workloads with Qwen3-Coder-30B-A3B, our design delivers 14 ms time-between-tokens (TBT) and adds only $\approx$0.1 ms of resume latency on top of prefill, while hosting $24\times$ more concurrent sessions per GPU. By confining HBM to the hot set, our design also cuts read power by 7.6 kW per 8-GPU node relative to serving all KV from flash-establishing HBF as a cold-tier complement to HBM rather than its replacement.
Subjects:
Hardware Architecture (cs.AR)
Cite as:
arXiv:2609.25782 [cs.AR]
(or
arXiv:2609.25782v1 [cs.AR] for this version)
https://doi.org/10.48550/arXiv.2609.25782
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Journal reference:
vol. 25, no. 2, pp. 355-358, July-Dec. 2026
Related DOI:
https://doi.org/10.1109/LCA.2026.3729099
Focus to learn more
DOI(s) linking to related resources
Submission history
From: Jongjin Baek [view email]
[v1]
Tue, 22 Sep 2026 07:18:00 UTC (331 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving, by Jongjin Baek and 3 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AR
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)