大模型
arxiv.org ·
测量大型语言模型指令中的语用影响
该 arXiv 论文研究“语用框架”对 LLM 指令遵循的影响，即“这很紧急”“作为你的主管”等措辞在不改变任务内容的情况下改变模型行为。作者提出三部分框架：将框架语境与任务规格分离的指令-框架分解、把 400 个框架实例归入 13 种策略和 4 类机制的分类法，以及通过指令优先级可观测偏移来量化影响的基于优先级的测量方法。在 5 个不同家族与规模的开源权重模型上评测发现，语用框架会造成指令优先级的系统性偏移，且不同策略的有效性排序跨模型高度一致，表明对语用框架的敏感性是指令微调系统的结构化行为属性。
AI推荐理由 arXiv 正式论文（v2，2026-09-11 修订），作者含 Omri Abend、Eduard Hovy、Lea Frermann 等知名研究者，摘要信息密度高、方法框架明确。新信息在于把“语用框架”（如“这很紧急”“作为你的主管”）作为可测量属性：提出指令-框架分解、400 个实例归入 13 种策略/4 类机制的分类法，以及基于优先级的测量方法，并在 5 个不同家族与规模的开源权重模型上发现指令优先级出现系统性偏移、策略有效性排序跨模型高度一致。局限是仅评测 5 个开源权重模型，摘要未给出具体效应量、数据集与统计细节，也未说明是否开源代码；对 prompt 工程、指令遵循评测与提示注入安全有直接参考价值，但属学术测量框架而非可直接落地的产品能力。
72
正文
Computer Science > Computation and Language
arXiv:2602.21223 (cs)
[Submitted on 2 Feb 2026 (v1), last revised 11 Sep 2026 (this version, v2)]
Title:Measuring Pragmatic Influence in Large Language Model Instructions
Authors:Yilin Geng, Omri Abend, Eduard Hovy, Lea Frermann
View a PDF of the paper titled Measuring Pragmatic Influence in Large Language Model Instructions, by Yilin Geng and 3 other authors
View PDF
HTML (experimental)
Abstract:It is not only what we ask large language models (LLMs) to do that matters, but also how we ask them. Phrases like ``This is urgent'' or ``As your supervisor'' can shift model behavior without altering task content. We study this effect as pragmatic framing, contextual cues that shape directive interpretation rather than task specification. While prior work exploits such cues for prompt optimization or probes them as security vulnerabilities, pragmatic framing itself has received comparatively little attention as a target of controlled measurement in instruction following. To support its systematic study as a measurable property, we introduce a framework that combines three components: directive-framing decomposition separating framing context from task specification; a taxonomy organizing 400 instantiations of framing into 13 strategies across 4 mechanism clusters; and priority-based measurement that quantifies influence through observable shifts in directive prioritization. Evaluating five open-weight LLMs across different families and scales, we find that pragmatic framing produces systematic shifts in directive prioritization, and the effectiveness ranking of different strategies proves highly consistent across models. This reveals that susceptibility to pragmatic framing is a structured behavioral property of instruction-tuned systems. Measuring this susceptibility is a prerequisite for any deliberate response to it, and this work provides the framework to do so.
Comments:
Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:
arXiv:2602.21223 [cs.CL]
(or
arXiv:2602.21223v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2602.21223
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Yilin Geng [view email]
[v1]
Mon, 2 Feb 2026 06:52:37 UTC (542 KB)
[v2]
Fri, 11 Sep 2026 10:44:17 UTC (1,171 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Measuring Pragmatic Influence in Large Language Model Instructions, by Yilin Geng and 3 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-02
Change to browse by:
cs
cs.AI
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)