智能体
arxiv.org ·
当模型退役时：开源应用中 LLM 迁移的实证研究
该论文挖掘 GitHub 上迁移离开 OpenAI、Anthropic、Google 官方弃用模型与端点的提交，并与厂商公告和关停日期匹配。在 2024-2026 年 17,703 个非 fork 仓库的 22,555 次提交中，5,139 次匹配到官方事件，两名独立标注者对 300 条分层样本验证（kappa 0.89-0.95）并据此重加权。结果显示约 82%（95% CI 79-84）的迁移在关停日期之后才提交，即应用已开始失败；该比例随通知期变化，Anthropic 60-114 天通知下为 89%，OpenAI 一年期 Assistants API 通知下仅 13%，通知期每增加一个 e 倍，关停后迁移几率约降低四分之三。94% 的迁移应用硬编码模型标识符，迁移工作量从仅提示词应用的中位 6 行新增代码到微调应用的近 700 行，仅 8% 迁移时更换厂商。作者公开数据集与流程，并讨论弃用政策、LLM 产品依赖风险评估与工具化启示。
AI推荐理由 arXiv 实证研究，基于 22,555 次提交、5,139 次匹配官方退役事件，双人标注 kappa 0.89-0.95，数据与流程公开，事实明确、信息密度高。核心新信息：约 82%（95% CI 79-84）的迁移发生在模型关停之后，即应用已开始失败才动手；迁移比例与厂商通知期强相关（Anthropic 60-114 天通知为 89%，OpenAI 一年期 Assistants API 通知仅 13%）；94% 应用硬编码模型标识符；仅 8% 迁移时更换厂商。对 LLM 依赖风险、弃用策略与工具链有直接参考价值。局限：样本限于 GitHub 开源应用，结论外推至闭源商业产品需谨慎。
82
正文
Computer Science > Software Engineering
arXiv:2609.31288 (cs)
[Submitted on 25 Sep 2026]
Title:When the Model Retires: An Empirical Study of LLM Migration in Open-Source Applications
Authors:Hyungjin Lukas Kim
View a PDF of the paper titled When the Model Retires: An Empirical Study of LLM Migration in Open-Source Applications, by Hyungjin Lukas Kim
View PDF
HTML (experimental)
Abstract:Applications built on commercial large language model (LLM) APIs depend on model versions that providers retire on their own schedule, with notice periods ranging from one year to two weeks. We ask what actually happens to applications when a model is retired. We mine GitHub for commits that migrate away from officially deprecated models and endpoints of OpenAI, Anthropic, and Google, matching each commit to the provider's published announcement and shutdown dates. From 22,555 commits in 17,703 non-fork repositories (2024-2026), 5,139 are matched to an official event; two independent coders validated a stratified sample of 300 (kappa = 0.89-0.95), and we reweight all estimates by their labels. We find that an estimated 82% (95% CI 79-84) of migrations away from retired models were committed after the shutdown date - after the application had started failing - regardless of repository popularity, prior retirement experience, or the presence of a provider-abstraction layer. The share tracks the provider's notice policy: 89% for Anthropic's 60-114-day notices versus 13% for OpenAI's one-year Assistants API notice, and each e-fold increase in notice length reduces the odds of post-shutdown migration by about three quarters. Model identifiers are hard-coded in 94% of migrating applications, migration effort scales from a median of 6 added lines for prompt-only applications to nearly 700 for fine-tuned ones, and only 8% of migrations switch provider. We release the dataset and pipeline and discuss implications for deprecation policy, dependency-risk assessment of LLM products, and tooling.
Comments:
Subjects:
Software Engineering (cs.SE)
ACM classes:
D.2.7; D.2.8
Cite as:
arXiv:2609.31288 [cs.SE]
(or
arXiv:2609.31288v1 [cs.SE] for this version)
https://doi.org/10.48550/arXiv.2609.31288
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Hyungjin Kim Dr. [view email]
[v1]
Fri, 25 Sep 2026 13:57:50 UTC (201 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled When the Model Retires: An Empirical Study of LLM Migration in Open-Source Applications, by Hyungjin Lukas Kim
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.SE
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)