大模型
arxiv.org ·
CodeAssay：面向 LLM 代码生成的带审计真值多指标基准
该论文提出 CodeAssay，一个包含 185 个 Python 任务、覆盖 10 个软件工程类别的分类优先基准，结合审计后的 ground truth、公开测试（生成与修复）、隐藏测试（评分）、基于变异的测试套件验证以及选定的代码属性度量。审计后重新评分改变了 1890 个正确性标签中的 170 个（9.0%），并将最佳与最差模型差距从 11.9 个百分点扩大到 23.7 个百分点；完整与隐藏测试套件的变异得分分别为 82.6% 和 74.8%。在 7 个专有 LLM 上，标准提示正确率为 77.3%–98.9%，21 对模型中有 12 对差异显著。在 14 个模型-提示配置均解决的 120 个任务上，没有模型在所有选定代码属性上表现最佳。安全导向提示未显著改变正确性或一致减少静态分析发现，但增加了所有模型的程序长度和圈复杂度。
AI推荐理由 arXiv 论文（cs.SE，v2 修订版），提出 CodeAssay 基准：185 个 Python 任务、10 个软件工程类别，采用审计后的 ground truth、公开测试用于生成与修复、隐藏测试用于评分、基于变异的测试套件验证及代码属性度量。具体新信息包括：审计后重新评分改变了 1890 个正确性标签中的 170 个（9.0%），最佳与最差模型差距从 11.9 个百分点扩大到 23.7 个百分点；完整与隐藏测试套件的变异得分分别为 82.6% 和 74.8%；7 个专有 LLM 标准提示正确率 77.3%–98.9%，21 对模型中有 12 对差异显著；在 14 个模型-提示配置均解决的 120 个任务上，没有模型在所有选定代码属性上表现最佳；安全导向提示未显著改变正确性或一致减少静态分析发现，反而增加程序长度和圈复杂度。局限：仅覆盖 Python 任务，结论基于论文自述，尚未见独立复现。
78
正文
Computer Science > Software Engineering
arXiv:2608.03535 (cs)
[Submitted on 4 Aug 2026 (v1), last revised 17 Sep 2026 (this version, v2)]
Title:CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation
Authors:Shahbaz Siddeeq, Muhammad Waseem, Umar Subhan Malhi, Pekka Abrahamsson
View a PDF of the paper titled CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation, by Shahbaz Siddeeq and 3 other authors
View PDF
HTML (experimental)
Abstract:Large Language Models are increasingly evaluated for code generation using test-based benchmarks. The validity of such evaluations depends on the reliability of their references and tests, while test-based correctness captures only part of the observable properties of generated code. We present CodeAssay, a taxonomy-first benchmark of 185 Python tasks across ten software-engineering categories. It combines audited ground truth, public tests for generation and repair, hidden tests for grading, mutation-based test-suite validation, and selected code-property measures. Regrading fixed model outputs after the audit changed 170 of 1,890 correctness labels (9.0%) and increased the measured best-to-worst model spread from 11.9 to 23.7 percentage points, although aggregate correctness remained nearly unchanged. The complete and hidden test suites achieved mutation scores of 82.6% and 74.8%, respectively. Across seven proprietary LLMs, standard-prompt correctness ranged from 77.3% to 98.9%, with significant differences in 12 of 21 model pairs. On the 120 tasks solved by all 14 model-prompt configurations, no model performed best across all selected code properties. A security-focused prompt produced no significant change in correctness or consistent reduction in the selected static-analysis findings, while increasing program length and cyclomatic complexity across all models. These findings show that reliable evaluation of LLM-generated code requires validated ground truth, protected tests, and multiple explicitly interpreted measures. CodeAssay provides a reproducible basis for evidence-based model evaluation in AI-augmented software development.
Comments:
Subjects:
Software Engineering (cs.SE)
Cite as:
arXiv:2608.03535 [cs.SE]
(or
arXiv:2608.03535v2 [cs.SE] for this version)
https://doi.org/10.48550/arXiv.2608.03535
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Shahbaz Siddeeq [view email]
[v1]
Tue, 4 Aug 2026 12:15:17 UTC (1,926 KB)
[v2]
Thu, 17 Sep 2026 11:31:20 UTC (2,787 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation, by Shahbaz Siddeeq and 3 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.SE
new
|
recent
| 2026-08
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)