智能体
arxiv.org ·
基于应用手册的任务：揭示语言模型在长时程程序性推理中的差距
论文提出 TAM 基准，用于评估长时程程序性推理。任务取自 ICD-10-CM 临床编码（将病症映射为诊断码）和美国联邦量刑（计算犯罪量刑指南中的犯罪等级）两个真实领域，标签经人工验证；每个任务需遵循含数万条规则的权威手册，跨不同章节执行一系列相互依赖的步骤以得出精确答案。作者在 GPT-5 上评估通用提示方法，包括检索增强生成、ReAct 式提示和一种 agent harness 基线，发现最佳精确匹配率仍极低：ICD-10-CM 编码为 1%，量刑任务为 15.5%。结果表明当前基准可能高估 LLM 的推理能力，并遗漏了可靠遵循长规则化流程这一关键挑战。TAM 数据与代码已公开。
AI推荐理由 arXiv 论文，提出 TAM 基准，用 ICD-10-CM 临床编码和美国联邦量刑两个真实领域任务评估长时程程序性推理，任务需遵循数万条规则的权威手册并执行跨章节的相互依赖步骤，标签经人工验证。在 GPT-5 上测试 RAG、ReAct 式提示和 agent harness 基线，最佳精确匹配仅 1%（ICD-10-CM）和 15.5%（量刑），指出当前基准可能高估 LLM 推理能力。数据与代码公开。局限是仅摘要，未提供各方法细节与误差分析。
82
正文
Computer Science > Computation and Language
arXiv:2609.13005 (cs)
[Submitted on 11 Sep 2026]
Title:Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models
Authors:Utkarsh Soni, Syed Shariyar Murtaza, Yifan Nie, Sachin Chandrasekhar, Eugene Wen
View a PDF of the paper titled Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models, by Utkarsh Soni and 4 other authors
View PDF
HTML (experimental)
Abstract:Large language models (LLMs) have achieved strong performance on a wide range of natural language tasks, and recent benchmarks suggest that they are increasingly adept at multi-hop reasoning. However, these benchmarks are typically short-horizon, requiring only a small number of retrieval or inference steps, and provide limited evidence of reliability on real-world tasks that involve following manuals spanning hundreds of pages with complex, interdependent guidelines. In this paper, we introduce Tasks over Application Manuals (TAM), a benchmark for evaluating long-horizon procedural reasoning. We construct TAM by curating real-world tasks from two domains: ICD-10-CM clinical coding (mapping medical conditions to diagnostic codes) and U.S. federal sentencing (computing crime sentencing guideline outcomes, specifically offense levels), with human-validated labels. Each task requires following an authoritative manual with tens of thousands of rules and executing a sequence of interdependent steps across different sections to produce an exact answer. We evaluate general-purpose prompting approaches, including retrieval-augmented generation, ReAct-style prompting, and an agent-harness baseline on GPT-5, and find that the best exact-match performance remains extremely low: 1% on ICD-10-CM coding and 15.5% on sentencing tasks. These results show that current benchmarks may overestimate LLM reasoning ability and miss a key challenge: reliably following long, rule-based procedures. The complete TAM data and code are publicly available.
Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:
arXiv:2609.13005 [cs.CL]
(or
arXiv:2609.13005v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.13005
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Related DOI:
https://doi.org/10.1145/3799682.3841039
Focus to learn more
DOI(s) linking to related resources
Submission history
From: Utkarsh Soni [view email]
[v1]
Fri, 11 Sep 2026 16:04:00 UTC (6,503 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models, by Utkarsh Soni and 4 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
cs.AI
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)