大模型
arxiv.org ·
DIAL：具有自适应人类偏好校准的位置去偏 LLM 评判器
该论文提出 DIAL 统一框架，利用大量 LLM 比较与有限人类比较，分离评判器特有的位置效应、学习去偏 LLM 偏好中的共享结构，并自适应校准至人类偏好目标。理论分析涵盖潜在 LLM 偏好、位置效应与人类校准的可识别性，兼顾 LLM 锚定与有限人类证据的自适应估计，以及校准后人类偏好的固定权重不确定性量化。实验在受控模拟和三个人类偏好基准上分别评估位置去偏与人类对齐，显示 DIAL 对不平衡响应顺序稳健、在标签有限时仍能获得强人类对齐排序，并在 LLM 信息不完善时向人类证据自适应。真实数据研究收集了 21 个 LLM 评判器在两种展示顺序下超过 41 万条判断。
AI推荐理由 arXiv 新论文，提出 DIAL 框架，将大量 LLM 比较与有限人类比较结合，分离评判器的位置效应、学习去偏后的共享偏好结构并自适应校准到人类偏好目标；理论部分覆盖潜在偏好/位置效应/人类校准的可识别性、自适应估计与固定权重不确定性量化，并在三个人类偏好基准上分别评估去偏与对齐，还收集了 21 个 LLM 评判器、两种展示顺序下超过 41 万条判断的数据。信息密度高、方法可验证，对 LLM-as-judge 评测可靠性有直接价值；局限是仅基于摘要，未提供具体指标数值，实际效果需看全文。
78
正文
Computer Science > Artificial Intelligence
arXiv:2609.31215 (cs)
[Submitted on 25 Sep 2026]
Title:DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration
Authors:Zesheng Cai, Yingqi Fan, Sichang Chen, Jin-Hong Du
View a PDF of the paper titled DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration, by Zesheng Cai and Yingqi Fan and Sichang Chen and Jin-Hong Du
View PDF
HTML (experimental)
Abstract:Large language models (LLMs) as a judge enable scalable evaluation, but their judgments can be sensitive to response order and, even after removing such position effects, can still diverge systematically from human this http URL introduce DIAL, a unified framework that combines abundant LLM comparisons with limited human comparisons to separate judge-specific position effects, learn shared structure in position-debiased LLM preferences, and adaptively calibrate that structure toward the human preference target. Theoretically, we study three aspects of DIAL: (i) identification of latent LLM preferences, position effects, and human calibration; (ii) adaptive estimation that balances LLM anchoring against limited human evidence; and (iii) fixed-weight uncertainty quantification for the calibrated human preference. Empirically, we evaluate position debiasing and human alignment separately in controlled simulations and on three human-preference benchmarks, showing that DIAL remains robust to unbalanced response order, achieves strong human-aligned rankings with limited labels, and adapts toward human evidence when LLM information is imperfect. Our real-data study collects over 410K judgments from 21 LLM judges in both display orders, providing a resource for future studies of LLM-judge bias, heterogeneity, and human alignment.
Subjects:
Artificial Intelligence (cs.AI); Applications (stat.AP); Methodology (stat.ME); Machine Learning (stat.ML)
Cite as:
arXiv:2609.31215 [cs.AI]
(or
arXiv:2609.31215v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.31215
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Jin-Hong Du [view email]
[v1]
Fri, 25 Sep 2026 12:56:36 UTC (4,952 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration, by Zesheng Cai and Yingqi Fan and Sichang Chen and Jin-Hong Du
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new
|
recent
| 2026-09
Change to browse by:
cs
stat
stat.AP
stat.ME
stat.ML
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)