大模型
arxiv.org ·
LLM 评分器在哪里成功、在哪里失效：来自两门计算机科学考试的证据
论文用两门计算机科学考试（计算机视觉 570 名双评学生、171 种配置；机器学习 1038 名双评学生、162 种配置）评估 LLM 自动评分。最佳配置平均绝对误差 1.64/35，低于两名人类评分者之间的 2.61/35。但提示词中的“严格评分”前言会显著破坏稳定性：17 个开源权重模型中有 14 个 MAE 升至 8 以上，3 个完全停止评分，问题主要来自两句扣分指令，其中“never give partial credit”单独就使 3 个被测模型中的 2 个停止评分；三家厂商的闭源旗舰模型仅改变校准、未脱离区间。在第二门考试中该脆弱性可复现，但影响方向因考试而异。对约 3900 条评分样本做轻量 LoRA 微调后，5 个小开源模型达到或超过人类评分者水平，对三种严厉提示的敏感度几乎消失（MAE ≤ 0.32）。作者公开匿名数据集、完整消融网格及评分、微调与分析流水线。
AI推荐理由 arXiv 论文，摘要给出具体实验规模（计算机视觉考试 570 名双评学生、171 种配置；机器学习考试 1038 名双评学生、162 种配置）和可量化结果（最佳 MAE 1.64/35，低于两名人类评分者之间的 2.61/35），并公开匿名数据集、消融网格与流水线，事实明确、信息密度高。核心新信息是提示词中“严格评分”前言（尤其“never give partial credit”一句）会让 17 个开源权重模型中的 14 个偏离评分区间、3 个完全停止评分，而闭源旗舰模型仅改变校准；轻量 LoRA 微调（约 3900 条评分样本）可修复该脆弱性。局限在于结论来自两门课程考试，方向具有考试特异性，不能直接外推到所有评分场景。
78
正文
Computer Science > Computation and Language
arXiv:2609.29333 (cs)
[Submitted on 24 Sep 2026]
Title:Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams
Authors:Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh, Salman Khan, Naeemullah Khan
View a PDF of the paper titled Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams, by Ali Habibullah and 3 other authors
View PDF
HTML (experimental)
Abstract:One long-form exam in a large course costs hundreds of grader-hours, and qualified graders are scarce; LLM graders are a tempting alternative. To show its pitfalls we grade a practical Computer Vision exam ($570$ dual-graded students) under $171$ configurations spanning closed and open-weights models; the best reaches mean absolute error $1.64/35$, below the $2.61/35$ two human graders achieve against each other. The catch is the prompt: a short ''strict grader'' preamble drives $14$ of $17$ open-weights models out of the graded band ($\text{MAE} \ge 8$), three stopping grading altogether. The damage traces to the preamble's two credit-withholding sentences, not to tone or model scale; one of them, ''never give partial credit'', alone makes two of three probed models stop grading. The closed flagships of three vendors shift calibration under it but stay in the band. In $162$ further configurations on a second, independent Machine Learning exam from another course ($1{,}038$ dual-graded students), the preamble worsens ten models, moving three out of the band into collapse and one into refusal, yet improves seven whose neutral prompts over-mark: the vulnerability replicates, but its direction is exam-specific. Light LoRA fine-tuning repairs it: one adapter on the two exams' pooled $\sim 3{,}900$ graded examples brings five small open models to parity or better with a human grader in agreement with the grader pair, and sensitivity to the three harsh personas nearly vanishes ($\le 0.32$ MAE). We release the anonymised dataset, full ablation grid, and grading, fine-tuning and analysis pipelines.
Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)
Cite as:
arXiv:2609.29333 [cs.CL]
(or
arXiv:2609.29333v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.29333
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Ali Habibullah [view email]
[v1]
Thu, 24 Sep 2026 10:07:57 UTC (247 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams, by Ali Habibullah and 3 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
cs.AI
cs.CY
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)