大模型
arxiv.org ·
死权重，活信号：冻结语言模型的前馈图
该论文提出一种前馈图架构，把多个异构冻结大模型当作计算节点，通过共享连续隐空间与学习到的线性投影通信，投影矩阵借助残差流注入钩子联合反向传播优化。三个小冻结模型（Llama-3.2-1B、Qwen2.5-1.5B、Gemma-2-2B）编码输入，聚合信号注入两个较大冻结模型（Phi-3-mini、Mistral-7B），其表示再送入轻量 cross-attention 输出节点。在约 12B 冻结参数下仅训练 17.6M 参数，取得 ARC-Challenge 87.3%、OpenBookQA 82.8%、MMLU 67.2%，分别超过最佳单一组成模型 11.4、6.2、1.2 个百分点，也超过参数匹配的冻结单模型学习分类器 9.1、5.2、6.7 个百分点；作者称跨多个冻结模型边界的梯度流可训练，且输出节点在无显式监督下形成选择性路由。
AI推荐理由 arXiv 论文（v2，2026-09-11 修订），摘要给出明确架构与可验证基准数字：以 Llama-3.2-1B、Qwen2.5-1.5B、Gemma-2-2B 三个冻结小模型为节点，经共享连续隐空间与学习到的线性投影，将聚合信号注入 Phi-3-mini 与 Mistral-7B 两个冻结大模型，输出端为轻量 cross-attention 节点；仅 17.6M 可训练参数对约 12B 冻结参数，在 ARC-Challenge 87.3%、OpenBookQA 82.8%、MMLU 67.2%，分别比最佳单一组成模型高 11.4、6.2、1.2 个百分点，比参数匹配的冻结单模型学习分类器高 9.1、5.2、6.7 个百分点，并称跨多个冻结模型边界的梯度流经验证可训练、输出节点在无显式监督下出现选择性路由。局限：结果均为论文自报，尚无第三方复现或代码/权重公开信息，摘要未说明评测细节与统计显著性；对多模型组合与隐空间对齐方向有参考价值，但不宜视为已确立的通用结论。
78
正文
Computer Science > Machine Learning
arXiv:2604.08335 (cs)
[Submitted on 9 Apr 2026 (v1), last revised 11 Sep 2026 (this version, v2)]
Title:Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models
Authors:Marcus Armstrong, Navid Ayoobi, Arjun Mukherjee
View a PDF of the paper titled Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models, by Marcus Armstrong and 2 other authors
View PDF
HTML (experimental)
Abstract:We present a feedforward graph architecture in which heterogeneous frozen large language models serve as computational nodes, communicating through a shared continuous latent space via learned linear projections. Building on recent work demonstrating geometric compatibility between independently trained LLM latent spaces~\cite{armstrong2026thinking}, we extend this finding from static two-model steering to end-to-end trainable multi-node graphs, where projection matrices are optimized jointly via backpropagation through residual stream injection hooks. Three small frozen models (Llama-3.2-1B, Qwen2.5-1.5B, Gemma-2-2B) encode the input into a shared latent space whose aggregate signal is injected into two larger frozen models (Phi-3-mini, Mistral-7B), whose representations feed a lightweight cross-attention output node. With only 17.6M trainable parameters against approximately 12B frozen, the architecture achieves 87.3\% on ARC-Challenge, 82.8\% on OpenBookQA, and 67.2\% on MMLU, outperforming the best single constituent model by 11.4, 6.2, and 1.2 percentage points respectively, and outperforming parameter-matched learned classifiers on frozen single models by 9.1, 5.2, and 6.7 points. Gradient flow through multiple frozen model boundaries is empirically verified to be tractable, and the output node develops selective routing behavior across layer-2 nodes without explicit supervision.
Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:
arXiv:2604.08335 [cs.LG]
(or
arXiv:2604.08335v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2604.08335
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Marcus Armstrong [view email]
[v1]
Thu, 9 Apr 2026 15:07:44 UTC (116 KB)
[v2]
Fri, 11 Sep 2026 17:42:22 UTC (122 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models, by Marcus Armstrong and 2 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.LG
new
|
recent
| 2026-04
Change to browse by:
cs
cs.AI
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
IArxiv recommender toggle
IArxiv Recommender
(What is IArxiv?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)