大模型
arxiv.org ·
基于LLM的代码漏洞修复中的指标失效：一项实证研究与变更感知筛选方法
论文认为 compile rate 对单函数漏洞修复是不可靠指标，并用 203 个 Big-Vul 漏洞函数、3 个开源代码 LLM（350M 至 6.7B 参数）和 3 种提示策略进行五组受控实验：compile rate 对显著改善代码的干预几乎不敏感；约 64% 编译失败源于评测框架与数据集因素而非模型；同一补丁在单个编译器标准开关下波动 1.8 至 2.7 倍；模型排序与参考相似度指标相反；作为优化目标时编译器反馈循环会提高 compile rate 却降低与人工修复的相似度，并出现删除式和占位式非修复。整函数 CodeBLEU 同样失效，未修改的漏洞输入得分高于所有模型。作者提出只对编辑区域计分的 diff_F1 变更感知筛选，对空操作给零分、对部分删除式作弊补丁接近零分，但仍会认可真实的部分编辑，可作为深入执行级分析前的低成本筛选，并说明其并非修复质量指标及不足。
AI推荐理由 arXiv 实证论文，用 203 个 Big-Vul 漏洞函数、3 个开源代码 LLM（350M–6.7B）和 3 种提示策略做五组对照实验，指出 compile rate 作为单函数漏洞修复指标不可靠：约 64% 编译失败不归因于模型、同一补丁在单个编译器标准开关下波动 1.8–2.7 倍、模型排序与参考相似度指标相反，且编译器反馈循环会奖励删除/占位式非修复。同时提出只对编辑区域计分的 diff_F1 变更感知筛选，并明确其不是修复质量指标、存在局限。对 LLM 代码安全评测方法有直接参考价值，但结论限于 C/C++ 单函数场景与所测小模型，需进一步验证。
78
正文
Computer Science > Software Engineering
arXiv:2609.26749 (cs)
[Submitted on 22 Sep 2026]
Title:Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen
Authors:Om Nepal, Sushant Aryal, Oluseyi Olukola, Nick Rahimi
View a PDF of the paper titled Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen, by Om Nepal and 3 other authors
View PDF
HTML (experimental)
Abstract:Large language models (LLMs) are increasingly applied to the automated repair of C/C++ security vulnerabilities, and compile rate is a commonly reported proxy for progress: whether the generated patch compiles. We argue that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and we support this with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies. Compile rate (i) barely responds to an intervention that substantially improves the generated code; (ii) is dominated by evaluation-harness and dataset artifacts rather than model quality, with about 64% of compile failures not attributable to the model, a share that is nearly invariant across models; (iii) shifts by 1.8 to 2.7 times on identical patches under a single compiler-standard flag, with zero regressions; (iv) ranks the three models in the opposite order to reference-similarity metrics; and (v) rewards non-repairs when used as an optimization target, since a compiler-feedback loop raises compile rate while similarity to the human fix falls, with manual inspection finding deletion- and placeholder-style non-repairs among the newly compiling outputs. The natural fallback, whole-function CodeBLEU, also fails: an unchanged copy of the vulnerable input outscores every model. We also examine diff_F1, a change-aware screen that scores only the edited region. It gives exactly zero credit to a no-op and near-zero credit to some, though not all, of the deletion-based gaming patches we observed, while still crediting genuine partial edits, so it may serve as a cheap screen before deeper, execution-based analysis. It is not a repair-quality metric, and we report where it falls short. Our findings argue for change-aware, execution-grounded evaluation of LLM-based vulnerability repair.
Comments:
Subjects:
Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR)
ACM classes:
D.2.5; I.2.2
Cite as:
arXiv:2609.26749 [cs.SE]
(or
arXiv:2609.26749v1 [cs.SE] for this version)
https://doi.org/10.48550/arXiv.2609.26749
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Om Nepal [view email]
[v1]
Tue, 22 Sep 2026 17:32:05 UTC (49 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen, by Om Nepal and 3 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.SE
new
|
recent
| 2026-09
Change to browse by:
cs
cs.AI
cs.CR
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)