大模型
x.com ·
@StanfordAILab
Stanford AI Lab 提出 CSBP：面向扩散 LLM 的上下文分片块并行训练策略
Stanford AI Lab 转述 Tarun Suresh 等人的研究，提出 Context-Sharded Block Parallelism（CSBP）分布式并行策略，用于加速扩散 LLM 的长上下文训练。团队报告称 DFlash2 推测解码草稿训练提速 7.59 倍、块扩散微调提速 1.61 倍、自回归转块扩散适配提速 1.33 倍，并称在相同 GPU 小时下模型在 SWE-bench Verified 和 Terminal-Bench Lite 上得分更高。该工作已在 Turbo-dLLM 分布式训练库中开源，相关数据尚待独立验证。
AI推荐理由 Stanford AI Lab 转述 Tarun Suresh 等人的工作，提出 Context-Sharded Block Parallelism（CSBP）分布式并行策略，用于扩散 LLM 训练，报告 DFlash2 推测解码草稿训练提速 7.59 倍、块扩散微调提速 1.61 倍、自回归转块扩散适配提速 1.33 倍，并称相同 GPU 小时下模型在 SWE-bench Verified 与 Terminal-Bench Lite 上得分更高，已在 Turbo-dLLM 库开源。数据具体且指向开源实现，但为团队自述，尚未见独立复现。
78
浏览 2,013
点赞 19
评论 1
转发 2
正文
Introducing Context-Sharded Block Parallelism (CSBP), a new distributed parallelism strategy enabling significant training efficiency for diffusion LLMs, with gains that grow with context length 🚀
⚡ 7.59× faster DFlash2 speculative decoding drafter training
⚡ 1.61× faster block diffusion fine-tuning
⚡ 1.33× faster autoregressive → block diffusion adaptation
Tarun Suresh @ COLM 2026@TarunSures41845
Diffusion LLMs and speculative decoding promise much faster agents. Yet agents need long contexts, and training on them is painfully slow.
Introducing Context-Sharded Block Parallelism (CSBP), a new distributed parallelism strategy unlocking significant training efficiency for diffusion LLMs, with speedup gains growing with context length 🚀
⚡ 7.59× faster DFlash2 speculative decoding drafter training
⚡ 1.61× faster block diffusion fine-tuning
⚡ 1.33× faster autoregressive → block diffusion adaptation
With the same GPU hours, models trained with CSBP score higher on SWE-bench Verified and Terminal-Bench Lite 👑
Open-sourced in Turbo-dLLM, our new optimized distributed training library.
Advised by [@Azaliamirh](https://x.com/Azaliamirh) and with an amazing team: [@PranshuChatur11](https://x.com/PranshuChatur11)[@hangoo\_kang](https://x.com/hangoo_kang)[@pshroff\_](https://x.com/pshroff_)[@ishanskhare](https://x.com/ishanskhare)[@KumbongHermann](https://x.com/KumbongHermann)
[🔗 View Quoted Tweet](https://x.com/TarunSures41845/status/2101016123278029303)