大模型
arxiv.org ·
PhysDox：在生理传感协议物理可行性审计上对 LLM 进行基准测试
该论文提出 PhysDox，一个面向生物医学协议的物理可行性审计基准，包含 683 条专家整理的 Gold 集和 5000 条 Silver 集，覆盖六个传感域。任务分两阶段：先将协议判为有效、轻微或致命，再对致命违规做约束级诊断。在 6 个 LLM 与 4 种推理策略下，Stage-1 macro-F1 峰值仅 53.0；端到端评估中 oracle 诊断因级联错误而失效。错误分析显示模型存在 scaffold bias，将流程完整性误当作物理有效性，隐式约束漏检率约为显式硬件违规的 2 倍（ρ=0.81）。
AI推荐理由 arXiv 论文，提出 PhysDox 基准，含 683 条专家标注 Gold 集与 5000 条 Silver 集，覆盖六个生理传感域，将任务形式化为严重度检测与致命违规的约束级诊断两阶段评估。摘要给出具体结果：6 个 LLM、4 种推理策略下 Stage-1 macro-F1 峰值仅 53.0，端到端评估中 oracle 诊断因级联错误而崩溃，并报告 scaffold bias 及隐式约束漏检率约为显式硬件违规的 2 倍（ρ=0.81）。信息密度高、指标明确，对 LLM 在实验设计中的物理可行性审计有实际评测价值；局限是仅摘要片段，未提供完整实验细节与代码可用性证据。
72
正文
Computer Science > Human-Computer Interaction
arXiv:2606.05003 (cs)
[Submitted on 3 Jun 2026 (v1), last revised 23 Sep 2026 (this version, v2)]
Title:PhysDox: Benchmarking LLMs on Physical Feasibility Auditing of Physiological Sensing Protocols
Authors:He Liu, Boyuan Gu, Shuaiqi Cheng, Haiyang Sun, Siyu You, Xuming Hu
View a PDF of the paper titled PhysDox: Benchmarking LLMs on Physical Feasibility Auditing of Physiological Sensing Protocols, by He Liu and Boyuan Gu and Shuaiqi Cheng and Haiyang Sun and Siyu You and Xuming Hu
View PDF
HTML (experimental)
Abstract:Large language models (LLMs) increasingly assist in experimental design, yet fluent protocols often remain physically infeasible. We introduce PhysDox, a physical feasibility auditing benchmark for biomedical protocols comprising a 683-sample expert-curated Gold set and a 5,000-sample Silver set across six sensing domains. We formulate the task as a two-stage evaluation: severity detection classifying protocols as valid, minor, or fatal, followed by the constraint-level diagnosis of fatal violations. Evaluating 6 LLMs across 4 inference strategies yields a peak Stage-1 macro-F1 of only 53.0. Moreover, strong oracle diagnosis collapses during end-to-end evaluation due to correlated cascade errors. Error analysis reveals scaffold bias, where models conflate procedural completeness with physical validity. Consequently, implicit constraints exhibit a 2 times higher miss rate than explicit hardware violations, supported by strong statistical correlation at $\rho{=}0.81$ and $p{
new
|
recent
| 2026-06
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)