大语言模型拒绝的语义稳定性如何？测量局部安全边界中的混淆

Computer Science > Computation and Language arXiv:2512.01037 (cs) [Submitted on 30 Nov 2025 (v1), last revised 12 Sep 2026 (this version, v3)] Title:How Semantically Stable Are LLM Refusals? Measuring Confusion in Local Safety Boundaries Authors:Riad Ahmed Anonto, Md Labid Al Nahiyan, Md Tanvir Hassan View a PDF of the paper titled How Semantically Stable Are LLM Refusals? Measuring Confusion in Local Safety Boundaries, by Riad Ahmed Anonto and 2 other authors View PDF HTML (experimental) Abstract:As safety alignment becomes standard in large language models, refusal behavior has become an important part of model reliability. However, models may still reject benign prompts, especially when the wording resembles risky content. Existing evaluations usually report global scores, such as false rejection rate or compliance rate. These scores are useful, but they treat each prompt independently. As a result, they miss local inconsistency, where a model accepts one phrasing of an intent but rejects a close paraphrase. This makes it difficult to understand whether refusals are only frequent or also semantically unstable. We address this gap by introducing Semantic Confusion, a failure mode that captures contradictory refusal decisions across meaning-preserving paraphrases. We build ParaGuard, a 10k-prompt corpus of controlled paraphrase clusters that keep intent fixed while varying surface form. We also propose three model-agnostic token-level metrics: Confusion Index, Confusion Rate, and Confusion Depth. These metrics compare each rejected prompt with its nearest accepted neighbors using token embeddings, next-token probabilities, and perplexity signals. Experiments across diverse model families show that global false rejection rate can hide important structure in the refusal boundary. Our metrics reveal cases where confusion is spread broadly, cases where it appears only in specific semantic regions, and cases where stricter refusal does not lead to more local inconsistency. These findings show that refusal evaluation should measure not only how often a model refuses, but also how consistently it refuses across nearby paraphrases. This gives developers a practical signal for reducing false refusals while preserving safety. Comments: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI) Cite as: arXiv:2512.01037 [cs.CL] (or arXiv:2512.01037v3 [cs.CL] for this version) https://doi.org/10.48550/arXiv.2512.01037 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Riad Ahmed Anonto [view email] [v1] Sun, 30 Nov 2025 19:11:45 UTC (755 KB) [v2] Fri, 19 Dec 2025 11:00:50 UTC (747 KB) [v3] Sat, 12 Sep 2026 20:08:58 UTC (747 KB) Full-text links: Access Paper: View a PDF of the paper titled How Semantically Stable Are LLM Refusals? Measuring Confusion in Local Safety Boundaries, by Riad Ahmed Anonto and 2 other authors View PDF HTML (experimental) TeX Source view license Current browse context: cs.CL new | recent | 2025-12 Change to browse by: cs cs.AI References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Loading... BibTeX formatted citation × loading... Data provided by: Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv (What is alphaXiv?) Links to Code Toggle CatalyzeX Code Finder for Papers (What is CatalyzeX?) DagsHub Toggle DagsHub (What is DagsHub?) GotitPub Toggle Gotit.pub (What is GotitPub?) Huggingface Toggle Hugging Face (What is Huggingface?) ScienceCast Toggle ScienceCast (What is ScienceCast?) Demos Demos Replicate Toggle Replicate (What is Replicate?) Spaces Toggle Hugging Face Spaces (What is Spaces?) Spaces Toggle TXYZ.AI (What is TXYZ.AI?) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower (What are Influence Flowers?) Core recommender toggle CORE Recommender (What is CORE?) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs. Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)