智能体
arxiv.org ·
PIA-Bench：迈向基于大语言模型的自动化隐私影响评估
该论文提出 PIA-Bench，称其为首个评估 LLM 在真实隐私影响评估（PIA）任务上表现的开放基准。作者审计了美国联邦机构发布的 499 份专家撰写的 PIA，整理出 73 份结构化 PIA，包含 451 条隐私风险和 831 条缓解措施，用于测试 LLM 评估复杂系统隐私风险并提出缓解措施的能力。结果显示现成 LLM 能产出有意义的评估，但论文未披露具体准确率，作者呼吁改进面向 LLM Agent 的领域工作流、建设可问责的 LLM 基础设施并制定新的 PIA 质量标准。
AI推荐理由 arXiv 新论文，提出首个面向真实隐私影响评估（PIA）的开放基准 PIA-Bench：审计了美国联邦机构发布的 499 份专家撰写的 PIA，整理出 73 份结构化 PIA，含 451 条隐私风险和 831 条缓解措施，用于评测 LLM 评估隐私风险与提出缓解措施的能力。摘要称现成 LLM 能给出有意义的评估，但未给出具体准确率数字，也未证明可实际替代人工 PIA；作者呼吁改进面向 LLM Agent 的领域工作流与可问责基础设施。价值在于为 LLM 在合规/隐私场景的评测提供了可复用的数据集与任务定义，局限是结论仍停留在‘有潜力、需改进’层面。
72
正文
Computer Science > Cryptography and Security
arXiv:2609.12571 (cs)
[Submitted on 11 Sep 2026]
Title:PIA-Bench: Towards Automated Privacy Impact Assessment with Large Language Models
Authors:Jiamin Zheng, Hao-Ping Lee, Luo Mai, Jingjie Li
View a PDF of the paper titled PIA-Bench: Towards Automated Privacy Impact Assessment with Large Language Models, by Jiamin Zheng and 3 other authors
View PDF
HTML (experimental)
Abstract:Privacy impact assessment (PIA) is a critical instrument for institutions to proactively identify privacy risks and develop mitigation strategies before system deployment. While mandated across regulatory and institutional contexts, executing PIA requires extensive privacy and technical expertise, posing a particular challenge for teams without access to such resources. Prior work shows the potential of leveraging large language models (LLMs) to assist practitioners' privacy decisions, but little is known about how accurately and reliably LLMs can automate PIA. To this end, we develop PIA-Bench, the first open benchmark for evaluating LLMs on real-world PIAs. We first audited 499 expert-authored PIAs published by US federal agencies and curated 73 structured PIAs, comprising a total of 451 privacy risk and 831 mitigation items, to evaluate LLMs' ability to assess privacy risks and propose mitigations of complex systems. Our results show that off-the-shelf LLMs produce meaningful assessments and identify avenues for future improvement. Finally, we call for improving domain-specific workflows for LLM agents, developing accountable LLM infrastructure, and designing new quality standards for PIAs.
Subjects:
Cryptography and Security (cs.CR)
Cite as:
arXiv:2609.12571 [cs.CR]
(or
arXiv:2609.12571v1 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2609.12571
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Jiamin Zheng [view email]
[v1]
Fri, 11 Sep 2026 08:16:03 UTC (265 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled PIA-Bench: Towards Automated Privacy Impact Assessment with Large Language Models, by Jiamin Zheng and 3 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CR
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)