论文研究
arxiv.org ·
从自由能视角检测大语言模型中的预训练数据
该论文针对预训练数据检测中高似然既可能来自训练暴露也可能来自强泛化的问题，提出在预测损失与预测熵的联合空间中使用倾斜边界，而非仅似然的水平边界。作者分析表明熵校正可保留期望成员信号并降低方差，并将均值-方差分析推广到非零平均熵差情形；该熵调整分数可解释为 Helmholtz 自由能，由此提出 Energy Transfer Detection（ETD）。实验称 ETD 取得最佳平均检测性能，平均 AUROC 最高提升 3.5%，TPR@5%FPR 最高提升 5.1%，并在多种设置下保持稳健。
AI推荐理由 arXiv 新论文，提出 Energy Transfer Detection（ETD），从自由能视角检测 LLM 预训练数据：用预测损失与预测熵的倾斜边界替代仅似然检测，并给出熵校正保持成员信号、降低方差的均值-方差分析。摘要报告平均 AUROC 最高提升 3.5%、TPR@5%FPR 最高提升 5.1%，但为作者自述实验结果，尚无独立复现或代码验证。对数据溯源、隐私与评测有实际参考价值，属方法类增量改进而非范式突破。
72
正文
Computer Science > Machine Learning
arXiv:2609.21888 (cs)
[Submitted on 18 Sep 2026]
Title:Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective
Authors:Chenye Ke, Zirui Liu, Qi Liu, Yan Zhuang, Jintao Zhang, Zhenya Huang, Shijin Wang
View a PDF of the paper titled Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective, by Chenye Ke and 6 other authors
View PDF
HTML (experimental)
Abstract:Detecting pretraining data in large language models is challenging because high likelihood can reflect either training exposure or strong generalization. In the joint space of prediction loss and predictive entropy, a likelihood-only detector uses a horizontal boundary and can mistake predictable non-members for members. Motivated by this, we introduce an inclined boundary that evaluates prediction loss relative to predictive entropy. Our analysis shows that entropy correction can preserve the expected membership signal while reducing its variance, thereby improving standardized member--non-member separation. We further extend the mean--variance analysis to the more general setting with a nonzero mean entropy gap. Interestingly, this entropy-adjusted score admits a Helmholtz free-energy interpretation, leading to Energy Transfer Detection (ETD), which views pretraining data detection from a macroscopic residual free-energy transfer perspective. Extensive experiments show that ETD achieves the best average detection performance, improving average AUROC by up to 3.5\% and TPR@5\%FPR by up to 5.1\%, while remaining robust across diverse settings.
Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as:
arXiv:2609.21888 [cs.LG]
(or
arXiv:2609.21888v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.21888
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Chenye Ke [view email]
[v1]
Fri, 18 Sep 2026 15:16:51 UTC (2,812 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective, by Chenye Ke and 6 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.LG
new
|
recent
| 2026-09
Change to browse by:
cs
cs.AI
cs.CL
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
IArxiv recommender toggle
IArxiv Recommender
(What is IArxiv?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)