大模型
arxiv.org ·
语言模型的测谎测试：读取模型不愿揭示的知识
论文提出 Probe of Internal Recognition (PIR)，借鉴取证学的 Concealed Information Test，在模型内部状态中读取其识别为正确的候选答案，无需参考模型或标注真值。在 Gemma、Qwen、Llama、Mistral、Phi 五个家族八个模型上平衡准确率达 0.70–0.87，高于未知项基线 0.28–0.40 与随机 0.25；在提示欺骗、训练出的 sandbagging、密码锁定与 circuit-broken checkpoint 等隐藏形式下识别率为 0.85–0.93。当 unlearning 真正移除知识时识别率降至未知问题水平，可用于区分模型“不愿答”与“不能答”，支持 sandbagging 审计与 unlearning 验证。
AI推荐理由 arXiv 新论文，提出 Probe of Internal Recognition (PIR) 方法，借用取证学的 Concealed Information Test，从模型内部状态读取其识别为正确的候选答案，无需参考模型或标注真值语料。在 Gemma、Qwen、Llama、Mistral、Phi 五个家族八个模型上，平衡准确率 0.70–0.87，高于未知项基线 0.28–0.40 与随机 0.25；在提示欺骗、训练出的 sandbagging、外部密码锁定与 circuit-broken checkpoint 等隐藏形式下识别率 0.85–0.93；当 unlearning 真正移除知识时识别率降至未知问题水平。该工作为 sandbagging 审计与 unlearning 验证提供了可区分“不愿答”与“不能答”的因果信号，并声称可扩展到自由生成。局限：仅摘要，未见代码、复现细节与同行评审，样本模型范围有限，实际审计可用性待验证。
78
正文
Computer Science > Artificial Intelligence
arXiv:2609.21996 (cs)
[Submitted on 18 Sep 2026]
Title:A Lie Detector Test for Language Models: Reading Knowledge a Model Won't Reveal
Authors:Hiskias Dingeto
View a PDF of the paper titled A Lie Detector Test for Language Models: Reading Knowledge a Model Won't Reveal, by Hiskias Dingeto
View PDF
HTML (experimental)
Abstract:Large language models can hold knowledge they do not report. A model may sandbag on a capability evaluation, or answer against what it internally knows, and its outputs alone cannot tell whether it is hiding an answer or simply does not have one. We borrow the Concealed Information Test, a forensic method that identifies guilty knowledge by presenting a suspect with the true detail among plausible decoys and measuring a stronger response to the item they recognize. Our method, Probe of Internal Recognition (PIR), does the same inside a model. It presents a question with its candidate answers and reads, from the model's internal states, which candidate the model recognizes as correct. PIR is reference-free, needing no honest reference model and no labeled truth corpus. Across eight models from five families (Gemma, Qwen, Llama, Mistral, and Phi), PIR recovers the recognized answer at 0.70 to 0.87 balanced accuracy, well above the 0.28 to 0.40 unknown-item baseline and the 0.25 chance rate. It stays readable across every form of concealment we test, from prompted deception and trained sandbagging to external password-locked and circuit-broken checkpoints, with recognition between 0.85 and 0.93. When the model hides a known answer, recognition stays high. When unlearning removes the knowledge, recognition drops to the level of a question the model never knew. PIR therefore separates a model that will not answer from one that cannot, which supports sandbagging audits and unlearning verification. The signal is causal, adds information beyond black-box behavioral cues, and extends from multiple-choice questions to free-form generation.
Subjects:
Artificial Intelligence (cs.AI)
Cite as:
arXiv:2609.21996 [cs.AI]
(or
arXiv:2609.21996v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.21996
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Hiskias Dingeto Dr [view email]
[v1]
Fri, 18 Sep 2026 16:57:00 UTC (310 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled A Lie Detector Test for Language Models: Reading Knowledge a Model Won't Reveal, by Hiskias Dingeto
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)