大模型
arxiv.org ·
追踪大语言模型中的计算密度
论文提出 s-Trace 方法，用规模为 s 的子图近似完整模型输出，从而估计 LLM 的有效计算量。实验发现 LLM 计算呈两阶段组织：早期层的小型稀疏子图即可重建输出分布头部，随后由后期层（越来越多为注意力头）逐步细化；每个输入所需的计算量与模型不确定性相关，稀疏子图编码 unigram 频率等浅层统计。作者据此认为 LLM 有效计算具有一致的模块化结构。
AI推荐理由 arXiv 正式论文（v2，2026-09-09 修订），作者来自 Corentin Kervadec、Marco Baroni、Gemma Boleda 等，摘要给出明确方法（s-Trace）与可复现发现：LLM 有效计算呈两阶段模块化组织，早期层稀疏子图可重建输出分布头部，后期层注意力头逐步细化；所需计算量与模型不确定性相关，稀疏子图编码 unigram 频率等浅层统计。属于机制可解释性方向的具体新证据，对推理效率与模型分析有参考价值；局限是仅摘要层面，未提供代码或跨任务验证细节。
72
正文
Computer Science > Computation and Language
arXiv:2605.27033 (cs)
[Submitted on 26 May 2026 (v1), last revised 9 Sep 2026 (this version, v2)]
Title:Tracing Computation Density in LLMs
Authors:Corentin Kervadec, Iuliia Lysova, Iuri Macocco, Marco Baroni, Gemma Boleda
View a PDF of the paper titled Tracing Computation Density in LLMs, by Corentin Kervadec and 4 other authors
View PDF
HTML (experimental)
Abstract:Transformer-based large language models (LLMs) are comprised of billions of parameters arranged in deep and wide computational graphs, but it is not clear that they exploit their full capacity for all inputs. We introduce the s-Trace method to efficiently estimate a subgraph of size s that approximates a full model output. With this method, we find the computation in a variety of LLMs to be organized in two distinct phases. A small subgraph mostly composed of early-layer nodes can reconstruct the head of the full model output distribution. Adding further nodes, mostly located in later layers and increasingly consisting of attention heads, leads to incremental refinements in approximating the full output distribution. We find moreover that the amount of necessary computation per input correlates with model uncertainty, and that sparser subgraphs encode shallow statistics, such as unigram frequency. Overall, our results suggest a consistent modular organization in effective LLM computation, with a sparse early-layer core providing a rough prediction that is further refined through denser computations in later layers.
Comments:
Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as:
arXiv:2605.27033 [cs.CL]
(or
arXiv:2605.27033v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.27033
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Corentin Kervadec [view email]
[v1]
Tue, 26 May 2026 13:55:08 UTC (2,311 KB)
[v2]
Wed, 9 Sep 2026 13:35:13 UTC (5,405 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Tracing Computation Density in LLMs, by Corentin Kervadec and 4 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-05
Change to browse by:
cs
cs.AI
cs.LG
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)