大模型
arxiv.org ·
LLM 作为评判者的有效性在物理评估形式中强烈依赖任务类型
该 arXiv 论文（v3）评估 LLM-as-a-judge 在物理评估中的有效性，覆盖结构化题、论文和科学图表三类形式，对比 GPT-5.2、Grok 4.1、Claude Opus 4.5、DeepSeek-V3.2、Gemini 3 Pro 及委员会聚合与人工评分，设置盲评、提供解答、错误解答和锚定四种条件，并区分绝对准确率与排序一致性。结果显示有效性强烈依赖任务类型：结构化题（n=771、n=1151）排序一致性稳健（Spearman ρ>0.6），官方解答降低误差；错误解答损害绝对准确率但不破坏排序；论文评分（275 篇）盲评更严苛且波动更大，加评分指引不改善排序，锚定样本使均值接近人工但排序一致性仍近零；代码图表元素（n=1400）排序一致性高（ρ>0.83）且校准近线性。作者指出人工基准的可靠性会限制对 AI-人类一致性的结论。
AI推荐理由 arXiv 论文（v3，2026-09-26 修订），实验设计明确：在盲评、提供解答、错误解答和锚定四种条件下，用 GPT-5.2、Grok 4.1、Claude Opus 4.5、DeepSeek-V3.2、Gemini 3 Pro 及委员会聚合对三类物理评估形式（结构化题、论文、科学图表）与人工评分对比，并区分绝对准确率与排序一致性。核心新信息是 LLM 评判有效性强烈依赖任务类型：结构化题和代码图表排序一致性高（ρ>0.6、ρ>0.83），论文评分排序一致性接近零且锚定样本只压缩方差、不改善排序。局限是样本规模有限（论文仅 275 篇）、结论限于物理教育场景，且摘要未给出人工基准可靠性细节。
78
正文
Physics > Physics Education
arXiv:2603.14732 (physics)
[Submitted on 16 Mar 2026 (v1), last revised 26 Sep 2026 (this version, v3)]
Title:LLM-as-a-judge validity is strongly task-dependent across physics assessment formats
Authors:Will Yeadon, Tom Hardy, Paul Mackay, Elise Agra
View a PDF of the paper titled LLM-as-a-judge validity is strongly task-dependent across physics assessment formats, by Will Yeadon and 2 other authors
View PDF
HTML (experimental)
Abstract:As large language models (LLMs) are increasingly considered for automated assessment and feedback, understanding when LLM marking is valid is essential. We evaluate LLM-as-a-judge marking across four settings spanning three assessment formats - structured questions, written essays, and scientific plots - comparing GPT-5.2, Grok 4.1, Claude Opus 4.5, DeepSeek-V3.2, Gemini 3 Pro, and committee aggregations against human markers under blind, solution-provided, false-solution, and anchored conditions. We distinguish absolute accuracy from rank-order agreement, since a marking system can match the distribution of human marks while failing to order responses by quality. Across task types, performance is sharply task-dependent. For blind university exam questions ($n=771$) and secondary and university structured questions ($n=1151$), models show robust rank-order agreement with human markers (Spearman $\rho > 0.6$), with official solutions reducing error and strengthening agreement. False solutions degrade absolute accuracy, showing that models defer to provided references, but leave rank-ordering intact. Essay marking behaves fundamentally differently. Across $n=55$ scripts, each containing five essays ($n=275$ essays total), blind AI marking is harsher and more variable than human marking and adding marking guidance does not improve rank-order agreement. Anchored exemplars shift the AI mean close to the human mean and compress variance below the human standard deviation, but rank-order agreement remains near-zero. For code-based plot elements ($n=1400$), models achieve high rank-order agreement ($\rho > 0.83$) with near-linear calibration. LLM marking validity depended strongly on the assessment task, and this held for every contemporary model tested. The results also show that the reliability of the human benchmark constrains the claims that can be made about AI-human agreement.
Comments:
Subjects:
Physics Education (physics.ed-ph); Computation and Language (cs.CL)
Cite as:
arXiv:2603.14732 [physics.ed-ph]
(or
arXiv:2603.14732v3 [physics.ed-ph] for this version)
https://doi.org/10.48550/arXiv.2603.14732
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Will Yeadon [view email]
[v1]
Mon, 16 Mar 2026 02:09:06 UTC (2,259 KB)
[v2]
Tue, 30 Jun 2026 16:52:37 UTC (2,507 KB)
[v3]
Sat, 26 Sep 2026 14:43:47 UTC (2,434 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled LLM-as-a-judge validity is strongly task-dependent across physics assessment formats, by Will Yeadon and 2 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
physics.ed-ph
new
|
recent
| 2026-03
Change to browse by:
cs
cs.CL
physics
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)