大模型
arxiv.org ·
InferScale：面向个性化 LLM 服务的 GPU 原生 KV 注入
论文提出 InferScale，一种 GPU 原生 LLM 记忆系统：预计算每条记忆事实的 KV 表示并与语义嵌入一同存于 GPU，服务时检索相关事实并将其 KV 直接注入 vLLM 的 paged cache，从而以可复用 KV 状态替代重复 prompt prefill。为支持旋转位置嵌入下的动态记忆组装，作者提出 Chunked RoPE（存储未旋转的 key，注入时施加服务时位置）和 Context-Window Encoding（编码记忆事实时附带少量前序对话上下文，仅缓存目标事实 KV）。实现基于 vLLM 的 KV-connector 接口，无需修改引擎或微调模型。在 LoCoMo 上对三个开源模型评测：k=50 时 TTFT 降低 72-79%（3.6-4.8x），准确率 60.3%（Mem0 为 63.3%），并发负载下吞吐为 3.7-4.5x。
AI推荐理由 arXiv 论文（v2，2026-09-27 修订），提出 InferScale：将个性化记忆事实的 KV 表示预计算并直接注入 vLLM 的 paged cache，替代重复 prefill；配套 Chunked RoPE 与 Context-Window Encoding 解决动态记忆组装和跨事实上下文缺失问题。在 LoCoMo 三个开源模型上，k=50 时 TTFT 降低 72-79%（3.6-4.8x），准确率 60.3% 对 Mem0 的 63.3%，并发吞吐 3.7-4.5x，且无需改引擎或微调。局限：准确率略低于 Mem0，评测仅限 LoCoMo 与三个开源模型，尚无生产环境验证。
82
正文
Computer Science > Distributed, Parallel, and Cluster Computing
arXiv:2607.27090 (cs)
[Submitted on 29 Jul 2026 (v1), last revised 27 Sep 2026 (this version, v2)]
Title:InferScale: GPU-Native KV Injection for Personalized LLM Serving
Authors:Peter Li, Prashant Pandey
View a PDF of the paper titled InferScale: GPU-Native KV Injection for Personalized LLM Serving, by Peter Li and Prashant Pandey
View PDF
HTML (experimental)
Abstract:Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.g., Mem0, MemGPT, and Zep) retrieve a relevant subset of this memory and inject it into the prompt, forcing the serving engine to repeatedly prefill the same content. As the retrieval budget grows, time-to-first-token (TTFT) increases even though the underlying memory is reused across requests.
We present InferScale, a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state. InferScale precomputes each memory fact's KV representation, stores it alongside a semantic embedding on the GPU, retrieves relevant facts at serving time, and injects their KV directly into vLLM's paged cache. To support dynamically assembled memories under rotary position embeddings, we introduce Chunked RoPE, which stores keys before rotation and applies their serving-time positions during injection. However, encoding memory facts independently omits the cross-fact context available during joint prefilling. We mitigate this with Context-Window Encoding, which encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV.
InferScale is implemented through vLLM's KV-connector interface, requiring neither engine modifications nor model fine-tuning. Across three open-weight models on LoCoMo, InferScale keeps TTFT nearly constant as the retrieval budget increases: at k=50 it reduces TTFT by 72-79% (3.6-4.8x), achieves 60.3% accuracy versus 63.3% for Mem0 without serving-time recomputation, and delivers 3.7-4.5x the throughput under concurrent load. Reusable KV state thus decouples memory-conditioned serving latency from retrieved-context size while preserving application quality.
Subjects:
Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)
Cite as:
arXiv:2607.27090 [cs.DC]
(or
arXiv:2607.27090v2 [cs.DC] for this version)
https://doi.org/10.48550/arXiv.2607.27090
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Prashant Pandey [view email]
[v1]
Wed, 29 Jul 2026 16:18:22 UTC (251 KB)
[v2]
Sun, 27 Sep 2026 20:18:24 UTC (252 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled InferScale: GPU-Native KV Injection for Personalized LLM Serving, by Peter Li and Prashant Pandey
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.DC
new
|
recent
| 2026-07
Change to browse by:
cs
cs.LG
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)