大模型
arxiv.org ·
LLM 偏好对齐的零阶范式
论文提出 Comparison-based Preference Optimization（ComPO），一种基于比较 oracle 的零阶对齐方法，不直接优化可微偏好损失，而是从偏好对中提取方向信息。作者为基本离线方案在平滑性、梯度稀疏性和 oracle 与潜在目标兼容性假设下建立了收敛保证，并引入在线 ComPO，利用无标注策略生成做相对参考策略的 reverse-KL 控制，在局部覆盖与分布内成对奖励准确率条件下给出性能保证。实验覆盖 Mistral、Llama、Gemma-2、Qwen3 和 Gemma-3，报告在长度控制胜率等指标上优于现有直接对齐方法，成对诊断结果与缓解 likelihood displacement 一致。
AI推荐理由 arXiv 新论文，提出基于比较 oracle 的零阶偏好对齐方法 ComPO，并给出离线与在线方案的收敛/性能保证，在 Mistral、Llama、Gemma-2、Qwen3、Gemma-3 上报告了相对现有直接对齐方法的改进（含长度控制胜率）。新信息在于用比较信息而非可微偏好损失来缓解 likelihood displacement，并附理论保证；局限是摘要未给出具体数值、算力开销与复现细节，效果仍需独立验证。
78
正文
Computer Science > Computation and Language
arXiv:2609.19144 (cs)
[Submitted on 16 Sep 2026]
Title:A Zeroth-Order Paradigm for LLM Preference Alignment
Authors:Peter Chen, Xi Chen, Wotao Yin, Tianyi Lin
View a PDF of the paper titled A Zeroth-Order Paradigm for LLM Preference Alignment, by Peter Chen and 3 other authors
View PDF
HTML (experimental)
Abstract:Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.
Comments:
Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as:
arXiv:2609.19144 [cs.CL]
(or
arXiv:2609.19144v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.19144
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Peter Chen [view email]
[v1]
Wed, 16 Sep 2026 17:59:35 UTC (126 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled A Zeroth-Order Paradigm for LLM Preference Alignment, by Peter Chen and 3 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
cs.AI
cs.LG
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)