大模型
arxiv.org ·
SAFARI：面向 LLM 辅助危害分析与风险评估的工业基准
该论文提出 SAFARI（Safety-Aware Functional Automotive Risk Inference），称其为首个面向 ISO 26262 汽车危害分析与风险评估（HARA）的工业基准，包含 3000 条去标识化工业 HARA 案例，覆盖开放式危害分析与标准依据的风险评估两项耦合任务，并提出首个参考锚定的 LLM-as-a-judge 评估协议。对九个前沿模型的实验显示，模型常能生成看似合理的危害叙述，但在 ISO 26262 风险分类上仍较弱，最佳 ASIL macro-F1 仅 0.261；Chain-of-Thought 提示收益有限，且常降低分类风险评估表现。错误分析将主要失败定位为危害生成中的场景关键上下文遗漏与风险评估中的可控性误判，提示专家监督应集中于此。数据集需通过文中链接获取。
AI推荐理由 arXiv 新论文，提出首个面向 ISO 26262 汽车危害分析与风险评估（HARA）的工业基准 SAFARI，含 3000 条去标识化工业案例，并给出首个参考锚定的 LLM-as-a-judge 评估协议。实验显示九个前沿模型在 ISO 26262 风险分类上表现薄弱，最佳 ASIL macro-F1 仅 0.261，且 Chain-of-Thought 提示收益有限甚至损害分类评估，错误分析定位到危害生成中的场景关键上下文遗漏与风险评估中的可控性误判。具体新信息是基准、评估协议与量化失败模式，对安全关键场景下 LLM 落地有参考价值；局限是数据集需另行申请获取，且结论限于该基准与所测模型。
78
正文
Computer Science > Computation and Language
arXiv:2609.20584 (cs)
[Submitted on 17 Sep 2026]
Title:SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment
Authors:Chenxi Wu, Zimu Wang, Haiyang Zhang, Wei Wang, Zhijie Xu
View a PDF of the paper titled SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment, by Chenxi Wu and 4 other authors
View PDF
HTML (experimental)
Abstract:Large language models (LLMs) are increasingly considered for safety-critical engineering, yet their reliability in regulated functional-safety workflows remains underexplored. We introduce SAFARI (Safety-Aware Functional Automotive Risk Inference), the first industrial benchmark for LLM-assisted automotive Hazard Analysis and Risk Assessment (HARA) under ISO 26262. It contains 3,000 de-identified industrial HARA cases and evaluates two coupled tasks: open-ended hazard analysis and standards-grounded risk assessment. To evaluate open-ended HARA artifacts, we propose the first reference-anchored LLM-as-a-judge protocol with high expert correlation. Experiments with nine frontier LLMs show that models often produce plausible hazard narratives but remain weak at ISO 26262 risk classification, with the best ASIL macro-F1 reaching only 0.261. Chain-of-Thought prompting provides limited benefit and often degrades categorical risk assessment. Error analysis further localizes major failures to scenario-critical context omissions during hazard generation and to controllability misjudgments during risk assessment, indicating where expert oversight should be concentrated. The dataset can be obtained from this https URL.
Comments:
Subjects:
Computation and Language (cs.CL)
Cite as:
arXiv:2609.20584 [cs.CL]
(or
arXiv:2609.20584v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.20584
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Zimu Wang [view email]
[v1]
Thu, 17 Sep 2026 15:37:51 UTC (408 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment, by Chenxi Wu and 4 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)