大模型
arxiv.org ·
GeoMesh：工作负载均衡且符号压缩的地理分布式 LLM 训练
该论文提出 GeoMesh，一个面向异构 GPU 的同步地理分布式训练框架。它通过为每块 GPU 分配适配的 batch size 与 inner steps 实现负载均衡，让更快的 GPU 承担更多有效工作；同时以压缩的符号伪梯度（附带轻量 magnitude 与 token count）交换梯度，将通信量降低近 32 倍。在异构 GPU 与 Azure 衍生 WAN 环境下，GeoMesh 相比代表性基线将达到目标困惑度的时间最多缩短 70.2%，并将 straggler 与 WAN 引起的 GPU 空闲分别最多降低 8.0 倍和 5.6 倍，同时保持可比的 zero-shot 准确率。
AI推荐理由 arXiv 新提交论文（2026-09-16），提出 GeoMesh 同步地理分布式训练框架，针对异构 GPU 与广域网场景：按 GPU 分配 batch size 与 inner steps 做负载均衡，并用基于符号的压缩伪梯度（含轻量 magnitude 与 token count）将通信量降低近 32 倍；在异构 GPU 与 Azure 衍生 WAN 上，time-to-target perplexity 相比代表性基线最多降低 70.2%，straggler 与 WAN 导致的 GPU 空闲分别最多降低 8.0x 和 5.6x，同时保持可比的 zero-shot 准确率。局限：摘要未给出具体模型规模、数据集与完整实验配置，效果为作者自报，尚待复现。
78
正文
Computer Science > Distributed, Parallel, and Cluster Computing
arXiv:2609.18388 (cs)
[Submitted on 16 Sep 2026]
Title:GeoMesh: Workload-Balanced and Sign-Compressed Geo-Distributed LLM Training
Authors:Changyong Shin, Jaerim Park, Minchul Kang, Younghun Go, Zhixiong Niu, Yongqiang Xiong, Gyeongsik Yang, Chuck Yoo
View a PDF of the paper titled GeoMesh: Workload-Balanced and Sign-Compressed Geo-Distributed LLM Training, by Changyong Shin and 7 other authors
View PDF
HTML (experimental)
Abstract:Large language models are increasingly trained on GPUs distributed across multiple regions, but geo-distributed training is challenging in practice. Real clusters often contain GPUs with different speeds and memory capacities, and they communicate over slow wide-area networks. Our analysis shows that this creates serious problems: existing synchronous methods preserve stable updates, but fast GPUs wait up to 20.9% of their runtime for slower ones, and all workers spend, on average, 65.8% of their runtime on synchronization. Recent asynchronous methods reduce waiting time but worsen the model accuracy due to stale updates. To address the problems, we present GeoMesh, a synchronous geo-distributed training framework for heterogeneous GPUs. GeoMesh balances per-worker workloads by assigning each GPU a suitable batch size and number of inner steps, so faster GPUs do more useful work instead of waiting. It also reduces communication volume by nearly 32x by exchanging compressed sign-based pseudo-gradients with lightweight magnitude and token count. Across heterogeneous GPUs and Azure-derived WAN, GeoMesh reduces time-to-target perplexity by up to 70.2% over representative baselines and lowers straggler- and WAN-induced GPU idle by up to 8.0x and 5.6x, respectively, while preserving comparable zero-shot accuracy.
Comments:
Subjects:
Distributed, Parallel, and Cluster Computing (cs.DC)
Cite as:
arXiv:2609.18388 [cs.DC]
(or
arXiv:2609.18388v1 [cs.DC] for this version)
https://doi.org/10.48550/arXiv.2609.18388
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Changyong Shin [view email]
[v1]
Wed, 16 Sep 2026 09:41:39 UTC (332 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled GeoMesh: Workload-Balanced and Sign-Compressed Geo-Distributed LLM Training, by Changyong Shin and 7 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.DC
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)