大模型
arxiv.org ·
从政策文件到结构化调查回复：评估用于政策监测的大型语言模型
该 arXiv 论文提出将 LLM 作为“AI 受访者”，用长上下文 in-context learning 管线把公开网络政策文本映射到预定义调查类别（政策工具、目标群体、主题领域），并引入二级 LLM 做相关性与证据验证，再与人工回复对比。基于多国数据集的评估显示，结构化指标上 LLM 与人工一致率为 84-95%，但自由文本字段仍有差异，模型倾向提供更详细的程序性描述。作者认为这支持人机混合的政策监测流程，可提升效率与可扩展性，但仍需人工验证与语境解读。
AI推荐理由 arXiv 论文，来源可信、摘要信息密度高，给出具体评估方法（长上下文 in-context learning 抽取管线、二级 LLM 验证、与人工回复对比）和量化结果（结构化指标一致率 84-95%）。新信息在于把 LLM 用作政策监测的“AI 受访者”生成结构化调查回复，并指出自由文本字段仍存在差异、模型倾向给出更详细的程序性描述，需人工验证。局限是仅基于摘要，未提供数据集规模、国家覆盖和具体误差细节，且属应用评估而非模型能力突破。
72
正文
Computer Science > Computation and Language
arXiv:2609.29370 (cs)
[Submitted on 24 Sep 2026]
Title:From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoring
Authors:Carolyn Cole, Matthias Deschryvere, Toqeer Ehsan, Arash Hajikhani
View a PDF of the paper titled From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoring, by Carolyn Cole and 3 other authors
View PDF
HTML (experimental)
Abstract:Science, technology, and innovation policies are crucial for competitiveness, yet their diversity and scale make them difficult to map and monitor consistently. Existing approaches rely heavily on manual survey efforts, which are costly and challenging to scale across countries. Large language models (LLMs) enable new possibilities for extracting and structuring information from long and unstructured policy documents. This paper presents an application of LLMs as "AI respondents" for generating structured survey responses from policy texts. We develop a data extraction pipeline based on long-context in-context learning to map information from public web sources into predefined survey categories, including policy instruments, target groups, and thematic areas. The pipeline integrates a validation step using a secondary LLM to assess relevance and evidence, alongside comparisons with human-provided responses. Using a multi-country dataset, we evaluate the alignment between LLM-generated and human-generated outputs through overlap measures and cross-validation. Results show that LLMs achieve high agreement for structured indicators (84-95%), while differences remain in free-text fields, where models tend to provide more detailed procedural descriptions. These findings highlight the potential of hybrid human-AI workflows for policy monitoring, improving both efficiency and scalability while maintaining the need for human validation and contextual interpretation.
Comments:
Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:
arXiv:2609.29370 [cs.CL]
(or
arXiv:2609.29370v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.29370
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Related DOI:
https://doi.org/10.1007/978-981-92-2480-7_20
Focus to learn more
DOI(s) linking to related resources
Submission history
From: Toqeer Ehsan [view email]
[v1]
Thu, 24 Sep 2026 10:49:37 UTC (135 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoring, by Carolyn Cole and 3 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
cs.AI
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)