大模型
website ·
[持续跟踪] GPT-6 Astra 被曝采用循环 Transformer 架构
Sebastian Raschka 对 OpenAI Astra 采用循环 Transformer 的传闻进行剖析，指出该架构本质是复用层堆叠以增加深度而不增加参数，并引用 Nanbeige 4.2 和 Mixture-of-recursions 论文说明其原理与局限。
AI推荐理由 帮助AI从业者理解循环Transformer的真实机制，避免被炒作误导，并了解其计算成本与推理透明性的影响。
67
浏览 20,515
点赞 121
评论 4
转发 20
正文
推荐阅读！
GPT-6 Astra + 循环深度/循环变压器
Sebastian Raschka@rasbt
今天我的时间线上有很多关于OpenAI Astra模型的炒作。显然，这源于The Information的一篇新文章，该文章称Astra是一个“循环深度或循环变压器”。
阅读新的或不同的方法总是很有趣（包括关于封闭实验室可能正在做什么的传言），但让我们稍微澄清一下。
例如，大约两个月前，我分享了Nanbeige的架构细节，其中“Nanbeige4.2-3B是从头开始在28T个令牌上预训练的，使用循环变压器，重用层堆栈以在不增加参数的情况下增加容量。”
是的，就是这样。循环变压器的想法只是在变压器块中重用层。
以Nanbeige为例，主要思想是重用相同的22层堆栈（即变压器块）两次而不是一次。因此，它有效地将22层架构扩展到44层，但没有复制权重。
简单来说，这大致使模型的大小翻倍（如果我们暂时忽略嵌入和输出层）。但不需要2倍的存储和内存来托管这个模型，它保持相同的大小，因为我们重用组件。然而，在计算方面几乎贵了2倍，因为我们通过几乎两倍的层数运行嵌入文本。
为什么？在Nanbeige 4.2技术报告中，研究人员发现两次传递提供了最佳的权衡，并保留了标准架构约75%的令牌效率。（更多次传递几乎没有带来收益，但使训练更慢且更昂贵。）
虽然据我所知，Nanbeige 4.2是第一个采用这种方法的重要开放权重模型，但这个想法可以追溯到NeurIPS论文“递归混合：学习动态递归深度以实现自适应令牌级计算”。实际上，这篇论文提出了一种更复杂的机制，添加了一个学习的路由器，决定每个令牌是接收一次、两次还是更多次传递。因此，简单的令牌可以提前退出，而更难的令牌则获得额外的计算。
总之，Astra可能是一个非常好的模型，但这不应该与“循环变压器方面”有关，这只是一个微小的架构调整。
此外，关于“新技术以某种方式掩盖了AI的部分或全部推理，即所谓的‘思维链’”的说法，对于循环变压器方法来说不一定正确。The Information的记者可能指的是其他技术，或者误解了循环变压器方法。
重用层本身并不会抑制可见的思维链。它在下一个令牌发出之前向隐藏状态添加计算，就像普通的变压器层一样。
但根据我们掌握的信息，这里唯一合理的解释是，如果模型使用更多这样的循环传递，它可能需要生成更少的中间推理令牌。因此，更多的计算发生在无法作为文本读取的潜在激活中。但如果我们扩大模型规模，比如GPT 5.6 Luna -> GPT 5.6 Sol，我们也会得到同样的效果。
[🔗 查看引用的推文](https://x.com/rasbt/status/2095141254958858496)
查看原文（中文正文为 AI 翻译）
推荐阅读！
GPT-6 Astra + Recurrent depth /Looped transformer
Sebastian Raschka@rasbt
A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
Also, the statement "the new technique works in a way that obscures some or all of the AI's reasoning, otherwise known as 'chain-of-thought'" is not necessarily true with respect to the looped transformer method. It's possible that The Information journalist refers to some other technique or misunderstood the looped transformer method.
Reusing layers does not by itself suppress visible chain of thought. It adds computation in hidden states before the next token is emitted, just as ordinary transformer layers do.
But based on the information we have, the only plausible interpretation here is that if a model uses more of these recurrent passes, it may need to generate fewer intermediate reasoning tokens. So then more of its computation happens in latent activations that cannot be read as text. But we would get the same effect if we were scaling up the model size, like GPT 5.6 Luna -> GPT 5.6 Sol.
[🔗 View Quoted Tweet](https://x.com/rasbt/status/2095141254958858496)