论文研究
arxiv.org ·
2023至2026年医学大语言模型研究中不断扩大的评估差距
该 arXiv 预印本统计了 PubMed 2023年1月至2026年6月十四个临床领域的 11,628 条记录，发现文献量增长 45 倍，但仅 2.5% 采用随机、对照或前瞻性设计。研究将“评估滞后”定义为从研究命名的最新模型发布到论文发表的时间，该指标从 1.33 个季度扩大到 6.08 个季度；在固定模型构成的对照下，迁移到更新模型仅抵消 56% 的漂移（95% CI 50-65）。随机试验评估的模型中位年龄比其他设计老 4.6 个季度（P = 3×10^-19），62% 的随机试验评估的是已停用模型家族。作者认为严谨性与时效性存在张力，且该张力更多源于模型选择而非研究周期。
AI推荐理由 arXiv 预印本，摘要给出明确量化证据：PubMed 2023年1月至2026年6月检索到 11,628 条记录、增长 45 倍，仅 2.5% 采用随机对照或前瞻性设计，评估滞后从 1.33 个季度扩大到 6.08 个季度，随机试验所评估模型中位老 4.6 个季度，62% 的随机试验评估的是已停用模型家族。这些具体数字对医学 LLM 评估方法学有实际参考价值，也提示临床证据与模型迭代速度的结构性矛盾。局限在于仅为预印本、未经同行评审，且摘要未说明检索与筛选细节，结论需谨慎对待。
78
正文
Computer Science > Computation and Language
arXiv:2609.11770 (cs)
[Submitted on 10 Sep 2026]
Title:The widening evaluation gap in medical large language model research 2023 to 2026
Authors:Raad Bin Tareaf, Murad Al-Rajab, Samia Loucif
View a PDF of the paper titled The widening evaluation gap in medical large language model research 2023 to 2026, by Raad Bin Tareaf and 2 other authors
View PDF
HTML (experimental)
Abstract:Large language models are superseded every few quarters; clinical evidence takes years. We asked whether medical research is keeping pace with the systems it evaluates. PubMed returned 11,628 records for January 2023 to June 2026 across fourteen clinical domains, growing 45-fold; 2.5% used a randomised, controlled or prospective design. Evaluation lag, from a study's newest named model release to its own publication, widened from 1.33 to 6.08 quarters. Because discontinued models age mechanically, we benchmarked this against a counterfactual holding model composition fixed: migration to newer systems offset only 56% of the drift (95% CI 50-65). Randomised trials evaluated models a median 4.6 quarters older than other designs (P = 3 x 10^-19), yet among studies naming a model still under development no design differed from any other; 62% of randomised trials evaluated a discontinued family. Rigour and currency are in tension, and that tension reflects model selection rather than research timelines.
Subjects:
Computation and Language (cs.CL)
Cite as:
arXiv:2609.11770 [cs.CL]
(or
arXiv:2609.11770v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.11770
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Raad Bin Tareaf Prof.Dr. [view email]
[v1]
Thu, 10 Sep 2026 16:25:01 UTC (446 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled The widening evaluation gap in medical large language model research 2023 to 2026, by Raad Bin Tareaf and 2 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)