大模型
arxiv.org ·
FTB Graph：多语言语言模型中首令牌广播器与语言身份头回路的确定与验证
该论文对 GPT-2、BLOOM-560M、Pythia-1B/2.8B 和 Qwen2.5-1.5B Base/Instruct 六种模型架构进行端到端结构电路分析，使用 Edge Attribution Patching（EAP）配合 FP16 主动钳制，并以 2000 条候选边搜索上限做精确激活补丁验证，提取驱动首令牌语言广播的有向无环图。结果显示广播枢纽多位于深层或中深层，Pythia-2.8B 与 BLOOM-560M 证据最强；Pythia-1B 扩展到 2.8B 时节点参与扩大但验证边预算相近，拓扑更稀疏；Qwen2.5-1.5B base 与 instruct 电路保持 84.7% Jaccard 相似度（含第 27 层枢纽），提示首令牌路由主要在预训练阶段形成并被指令微调保留。此外，多数模型中 EAP 分数与精确补丁增量仅弱相关，说明 FP16 下线性梯度近似可能与因果干预结果背离，需精确补丁验证。
AI推荐理由 arXiv 预印本，方法描述具体（EAP + FP16 主动钳制、2000 条候选边上限、精确激活补丁验证），覆盖 GPT-2、BLOOM-560M、Pythia-1B/2.8B、Qwen2.5-1.5B Base/Instruct 六种架构，并给出 Qwen2.5 base 与 instruct 电路 84.7% Jaccard 相似度、EAP 分数与精确补丁增量弱相关等可验证结论。局限：作者自述 GPT-2 与 Pythia-1B 证据较弱，且为预印本未经同行评审，结论尚待复现。
72
正文
Computer Science > Artificial Intelligence
arXiv:2609.30954 (cs)
[Submitted on 25 Sep 2026]
Title:FTB Graph: Determining and Validating First-token Broadcasters and Language-Identity Head Circuits in Multilingual Language Models
Authors:Arjun Pillai, Christian Hoang, Anjelo Laroza
View a PDF of the paper titled FTB Graph: Determining and Validating First-token Broadcasters and Language-Identity Head Circuits in Multilingual Language Models, by Arjun Pillai and 2 other authors
View PDF
HTML (experimental)
Abstract:Large language models operating in multilingual contexts must resolve target response languages early in generation, yet the causal circuitry governing first-token language identity decisions remains poorly mapped. We present an end-to-end structural circuit analysis across six model architectures spanning four families: GPT-2, BLOOM-560M, Pythia-1B/2.8B, and Qwen2.5-1.5B Base/Instruct. Using Edge Attribution Patching (EAP) with FP16 active clamping, followed by exact activation patching verification with a 2,000-candidate-edge search ceiling, we extract directed acyclic graphs driving first-token language broadcasting. Across the standalone models, we observe deep or mid-to-deep broadcasting hubs, though the evidence is strongest for Pythia-2.8B and BLOOM-560M because GPT-2 and Pythia-1B leave few out-of-graph heads for comparison, while both Qwen2.5-1.5B variants invert the necessity check. Scaling from Pythia-1B to 2.8B expands node participation while maintaining a similar verified edge budget, producing sparser topology. The Qwen2.5-1.5B base and instruct circuits retain 84.7% Jaccard similarity, including the Layer 27 hub, indicating that first-token routing is largely established during pretraining and preserved by instruction tuning. Finally, EAP scores correlate weakly with exact patching deltas across most models, showing that linear gradient approximations can diverge from causal interventions in FP16 and motivating exact-patching verification for reliable circuit discovery.
Comments:
Subjects:
Artificial Intelligence (cs.AI)
Cite as:
arXiv:2609.30954 [cs.AI]
(or
arXiv:2609.30954v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.30954
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Christian Hoang [view email]
[v1]
Fri, 25 Sep 2026 08:05:03 UTC (20,553 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled FTB Graph: Determining and Validating First-token Broadcasters and Language-Identity Head Circuits in Multilingual Language Models, by Arjun Pillai and 2 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)