大模型
arxiv.org ·
大型语言模型在上下文中形成信念状态几何
论文研究 LLM 上下文学习（ICL）所依赖的内部表征：以隐马尔可夫模型（HMM）生成的数据提示模型，探测其对隐状态后验分布（belief state）的编码。在 6 个开源 LLM 与 40 个具有非平凡信念结构的 HMM 组合上，belief state 可从残差流激活中线性解码，探针 R² 峰值在 0.83–0.99 之间，分布于早层至晚层。作者进一步对探针识别出的子空间做 patching 与 steering 干预，下游预测质量接近未干预模型，而对照条件显著退化。结果提供表征层证据，表明开源 LLM 的 ICL 近似于对上下文推断生成模型的最优贝叶斯预测，并将输入分布结构与激活几何的关联从玩具网络扩展到生产级 LLM。
AI推荐理由 arXiv 论文，作者明确、方法可验证：用 40 个 HMM 提示 6 个开源 LLM，发现 belief state 可从残差流激活中线性解码（探针 R² 0.83–0.99），并通过 patching/steering 干预验证功能相关性。为 ICL 近似贝叶斯预测提供表征层证据，属机制可解释性方向的具体新结果；局限是仅在受控 HMM 设定与开源模型上验证，摘要未给出跨任务泛化或规模外推证据。
78
正文
Computer Science > Machine Learning
arXiv:2609.17376 (cs)
[Submitted on 15 Sep 2026]
Title:Large Language Models Develop Belief State Geometry In-Context
Authors:Daniel Balcells, Andrew Jun Lee, Chirag Rastogi, Paul M. Riechers, Adam Shai, Xavier Poncini
View a PDF of the paper titled Large Language Models Develop Belief State Geometry In-Context, by Daniel Balcells and 5 other authors
View PDF
Abstract:Large language models (LLMs) trained on next-token prediction exhibit remarkable in-context learning (ICL) abilities, yet the representations that support ICL remain poorly understood. We consider such representations in a controlled setting: prompting LLMs with data emitted from hidden Markov models (HMMs) and probing for the corresponding belief state -- the posterior distribution over the HMM's hidden states given the observed token history. Across six open-source LLMs prompted with data from 40 HMMs selected for non-trivial belief structure, we find that belief states are linearly decodable from residual stream activations, with peak probe $R^2$-values from 0.83-0.99 across HMM and LLM combinations, ranging from early to late layers. To establish functional relevance, we intervene directly on the probe-identified subspace via patching and steering, resulting in downstream prediction quality on the order of the untampered model, while controls degrade performance substantially. Together, these results provide representation-level evidence that ICL in open-source LLMs approximates optimal Bayesian prediction over a context-inferred generative model. More broadly, our findings extend prior results linking input-distribution structure to activation geometry: from toy networks trained explicitly on HMM data to production-scale LLMs.
Comments:
Subjects:
Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as:
arXiv:2609.17376 [cs.LG]
(or
arXiv:2609.17376v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.17376
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Xavier Poncini [view email]
[v1]
Tue, 15 Sep 2026 16:17:59 UTC (14,642 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Large Language Models Develop Belief State Geometry In-Context, by Daniel Balcells and 5 other authors
View PDF
TeX Source
view license
Current browse context:
cs.LG
new
|
recent
| 2026-09
Change to browse by:
cs
cs.CL
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
IArxiv recommender toggle
IArxiv Recommender
(What is IArxiv?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)