大模型
arxiv.org ·
日本卒中大语言模型评估：面向日语安全卒中护理的多轮对话基准
该 arXiv 预印本提出 Japanese Stroke LLM Evaluation，一个针对日语卒中诊疗的多轮对话基准，由神经外科专家设计 10 个病例，由认证神经外科医生扮演模拟患者并评分，覆盖病史采集与处置阶段，并将可能直接危及生命的错误定义为 critical mistake，安全阈值为总体至少 80% 且零 critical mistake。2025 年 10 月和 2026 年 6 月共评测 18 个模型：Claude Fable 5 以 87.4% 且零 critical mistake 居首，Claude Opus 4.7 为 80.3%，GLM-5.2 为 75.6%，仅两个模型达到安全阈值；11 个模型共出现 17 次 critical mistake，如 t-PA 前未确认化验或血糖、气道未稳定即手术、遗漏颈部血管评估及超适应证使用 t-PA。病史采集提问数与得分相关（r=0.648，p=0.007）。作者指出该基准样本量小、使用模拟患者，仍需真实病例进一步评估。
AI推荐理由 arXiv 预印本，由神经外科专家而非 LLM-as-judge 构建 10 例日语卒中多轮对话基准，含病史采集与处置两阶段评分，并定义危及生命的 critical mistake 与 80% 安全阈值。18 个模型在 2025 年 10 月和 2026 年 6 月两次评测：Claude Fable 5 最高 87.4% 且零 critical mistake，Claude Opus 4.7 为 80.3%，GLM-5.2 为 75.6%，仅两个模型达标；11 个模型共犯 17 次 critical mistake，包括 t-PA 前未确认化验或血糖、气道未稳定即手术、遗漏颈部血管评估、超适应证使用 t-PA。局限是仅 10 例、模拟患者而非真实病例，作者也指出仍需真实世界验证；模型名称与时间点来自论文摘要，未独立核实。
78
正文
Computer Science > Computation and Language
arXiv:2609.16739 (cs)
[Submitted on 15 Sep 2026]
Title:Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models
Authors:Keisuke Masuda, Kazutaka Yatsushiro, Hirohumi Iwamoto, Hirofumi Hirano, Ryosuke Hanaya
View a PDF of the paper titled Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models, by Keisuke Masuda and 4 other authors
View PDF
Abstract:Background: Large language models (LLMs) have achieved physician-comparable performance on multiple-choice medical knowledge examinations, but their capabilities in clinical history taking, urgency assessment, and safety remain insufficiently evaluated. We proposed Japanese Stroke LLM Evaluation, a multi-turn conversational benchmark for stroke care in Japanese, and evaluated LLM performance and safety under practice-oriented conditions. Methods: We created 10 stroke and related-condition cases and evaluated LLMs in multi-turn Japanese conversations. The LLM acted as physician, while a board-certified neurosurgeon acted as simulated patient and evaluator. Each case comprised history-taking and action phases scored using pre-specified criteria. Errors that could directly threaten life were defined as critical mistakes. The safety threshold was at least 80% overall with zero critical mistakes. Eighteen models were evaluated in October 2025 and June 2026. Results: Claude Fable 5 achieved the highest score (87.4%) with zero critical mistakes, followed by Claude Opus 4.7 (80.3%) and GLM-5.2 (75.6%). Two leaders met the safety threshold. Eleven models made 17 critical mistakes, including failure to confirm laboratory results or blood glucose before t-PA, surgery before airway stabilization, omission of cervical vascular evaluation, and t-PA outside its indication. History-taking question count correlated with history-taking score (r = 0.648, p = 0.007). Conclusions: Japanese Stroke LLM Evaluation provides a benchmark for LLM performance under practice-oriented conditions, including a cap on history-taking questions. Cases and evaluations were created by neurosurgical specialists rather than using an LLM-as-judge approach. Performance improved across cloud-based and on-premise models in 2026, with some exceeding the safety threshold. Further evaluation using real-world cases is required.
Subjects:
Computation and Language (cs.CL)
Cite as:
arXiv:2609.16739 [cs.CL]
(or
arXiv:2609.16739v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.16739
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Keisuke Masuda [view email]
[v1]
Tue, 15 Sep 2026 07:13:49 UTC (588 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models, by Keisuke Masuda and 4 other authors
View PDF
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)