大模型
arxiv.org ·
大语言模型中的领域特定幻觉检测
该论文提出多信号幻觉检测流水线，结合微调 DeBERTa-v3 分类、Monte Carlo Dropout 不确定性量化和温度缩放校准，用于响应级幻觉检测。在 HaluEval 基准上，通用域任务取得 F1=0.915、AUROC=0.977，各任务 F1 为 QA 0.97、摘要 0.96、对话 0.82；MC Dropout 推理将准确率提升至 93.2%。上下文消融显示，移除知识上下文后摘要 F1 下降 24%，表明模型进行的是真实蕴含推理而非利用表面模式；学习曲线显示 25% 训练数据即可达到全量性能的 77%。此外，作者用 DPO 微调 Qwen2.5-0.5B 生成器，使其幻觉率从 85.5% 降至 37.7%（相对下降 55.9%）。跨域评估显示通用域训练迁移到 SciFact 生物医学基准效果较差（F1=0.52），而域内微调的 PubMedBERT 达到 F1=0.63、AUROC=0.81，说明域匹配预训练是最有效的适配策略。代码与模型已公开。
AI推荐理由 arXiv 新提交论文，摘要给出明确方法（DeBERTa-v3 微调 + MC Dropout 不确定性 + 温度校准）与可核验指标：HaluEval 上 F1=0.915、AUROC=0.977，MC Dropout 推理准确率 93.2%；上下文消融显示去除知识上下文后摘要任务 F1 下降 24%，支持其做真实蕴含推理而非表面模式；学习曲线显示 25% 训练数据可达全量性能的 77%。还报告用 DPO 将 Qwen2.5-0.5B 生成器幻觉率从 85.5% 降至 37.7%，以及跨域迁移差（SciFact 上 F1=0.52）、PubMedBERT 域内微调 F1=0.63/AUROC=0.81。局限：均为作者自报结果，尚未见独立复现；跨域泛化弱，实际部署需域内微调。
78
正文
Computer Science > Computation and Language
arXiv:2609.11878 (cs)
[Submitted on 10 Sep 2026]
Title:Domain-Specific Hallucination Detection in Large Language Models
Authors:Varun Teja Chundru, Debasmita Biswas
View a PDF of the paper titled Domain-Specific Hallucination Detection in Large Language Models, by Varun Teja Chundru and 1 other authors
View PDF
HTML (experimental)
Abstract:Large language models generate fluent text that can contain unfaithful claims -- a phenomenon known as hallucination. We present a multi-signal detection pipeline combining fine-tuned DeBERTa-v3 classification, Monte Carlo (MC) Dropout uncertainty quantification, and temperature-scaled calibration for response-level hallucination detection. Evaluated on the HaluEval benchmark, our pipeline achieves F1=0.915 and AUROC=0.977 on general-domain tasks, with per-task F1 scores of 0.97 (QA), 0.96 (Summarization), and 0.82 (Dialogue). MC Dropout inference further improves accuracy to 93.2%. A context ablation study confirms the model performs genuine entailment reasoning rather than exploiting surface patterns, with summarization F1 dropping 24% when knowledge context is removed. Learning curve analysis reveals that 25% of training data captures 77% of full-data performance. Beyond detection, we apply Direct Preference Optimization (DPO) to a Qwen2.5-0.5B generator, reducing its hallucination rate from 85.5% to 37.7% (55.9% relative reduction) as measured by our detector. Cross-domain evaluation on the SciFact biomedical benchmark shows that general-domain training transfers poorly (F1=0.52), motivating domain-specific fine-tuning. PubMedBERT fine-tuned on SciFact achieves F1=0.63 and AUROC=0.81, demonstrating that domain-matched pre-training is the strongest adaptation strategy. Code and models are available at this https URL
Comments:
Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
ACM classes:
I.2.7
Cite as:
arXiv:2609.11878 [cs.CL]
(or
arXiv:2609.11878v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.11878
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Varun Teja Chundru [view email]
[v1]
Thu, 10 Sep 2026 17:45:36 UTC (155 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Domain-Specific Hallucination Detection in Large Language Models, by Varun Teja Chundru and 1 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
cs.AI
cs.LG
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)