智能体
arxiv.org ·
基于LLM的系统支持工业测试失败分析：经验报告
该研究在 Westermo Network Technologies AB 落地工具增强的 LLM 系统，用于夜间测试失败的根因分析（RCA），实现单 agent 与编排式多 agent 两种配置，均可访问测试元数据与日志。探索性工业案例研究使用两个真实失败场景，6 名从业者通过问卷与焦点小组评估报告，并从 120 次重复执行中采集运营指标，覆盖感知正确性、推理质量、修复现实性、清晰度、有用性、信任以及成本、时长和一致性。结果显示两种配置在两个场景中均无一致的感知质量优势；单 agent 生成报告更快、成本更低，是更实用的基线，多 agent 架构的潜在收益仍需在更复杂场景中评估。
AI推荐理由 arXiv 经验报告，来自 Westermo Network Technologies AB 的工业案例：用单 agent 与编排式多 agent 两种配置对夜间测试失败做根因分析（RCA），由 6 名从业者通过问卷和焦点小组评估两个真实失败场景，并采集 120 次重复运行的运营指标（成本、时长、一致性）。核心新信息是：两种配置在两个场景中均未表现出一致的从业者感知质量优势，单 agent 更快、成本更低，是更实用的基线；多 agent 架构的潜在收益需在更复杂场景中进一步验证。局限是仅两个场景、单一企业，结论为探索性，不能外推为通用结论。
72
正文
Computer Science > Software Engineering
arXiv:2609.21843 (cs)
[Submitted on 18 Sep 2026]
Title:Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report
Authors:Eric Jansson, Per Strandberg, Thomas Sörensen, Eduard Paul Enoiu, Wasif Afzal
View a PDF of the paper titled Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report, by Eric Jansson and 4 other authors
View PDF
HTML (experimental)
Abstract:This study examines tool-augmented Large Language Model (LLM) systems for supporting Root Cause Analysis (RCA) of nightly test failures at Westermo Network Technologies AB. Nightly test executions produce heterogeneous test data and logs that practitioners currently inspect manually across multiple sources. We implemented an RCA workflow in single-agent and orchestrated multi-agent configurations, both with access to test metadata and logs. An exploratory industrial case study used two real failure scenarios. Six practitioners evaluated the scenario reports through a survey and focus group, and operational measurements were collected from 120 repeated executions. The evaluation covered practitioner-perceived correctness, reasoning quality, fix realism, clarity, usefulness, and trust, as well as cost, duration, and consistency. Neither configuration showed a consistent practitioner-perceived quality advantage across the two scenarios. The single agent system generated reports faster and at lower cost, making it the more practical baseline in this context. The potential benefits of agent architectures require further evaluation in more complex scenarios.
Comments:
Subjects:
Software Engineering (cs.SE)
Cite as:
arXiv:2609.21843 [cs.SE]
(or
arXiv:2609.21843v1 [cs.SE] for this version)
https://doi.org/10.48550/arXiv.2609.21843
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Per Strandberg [view email]
[v1]
Fri, 18 Sep 2026 14:39:42 UTC (870 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report, by Eric Jansson and 4 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.SE
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)