大模型
arxiv.org ·
哪种 LLM 最适合将自然语言目标转换为 PDDL？
该论文实证评估当前 LLM 能否把视频游戏测试者用非正式语言书写的自然语言测试目标，可靠转换为适合经典规划的规范 PDDL 目标。作者设计了专门提示模板，在真实领域基准上系统比较六个当代模型在正确性、速度和错误倾向上的表现：所有模型正确率均超过 92%，Gemini 2.5 Flash 以 96% 的正确率和最少的假阳性居首，GPT-4.1 响应速度最快。研究同时指出模型间仍存在关键差异，语言歧义和领域表示局限会导致偶发失败，说明 LLM 作为自然语言目标与自动规划流程之间桥梁仍有待完善。
AI推荐理由 arXiv 论文，作者与摘要明确，给出可验证的评测设置：用视频游戏测试者书写的非正式自然语言目标，经专门设计的提示模板转换为 PDDL，对六个当代模型在正确率、速度和错误倾向上做系统评估。具体新信息是 Gemini 2.5 Flash 正确率最高达 96% 且假阳性最少，GPT-4.1 响应速度最快，所有模型正确率均超过 92%；局限是仍存在语言歧义与领域表示不足导致的偶发失败。属于自然语言到形式化规划这一 Agent/规划方向的实证工作，对开发者和研究者有参考价值，但样本为特定游戏测试领域，结论外推需谨慎。
72
正文
Computer Science > Artificial Intelligence
arXiv:2609.18731 (cs)
[Submitted on 16 Sep 2026]
Title:Which LLM is Best for Translating Natural Language Goals to PDDL
Authors:Tomas Balyo, Lukas Chrpa, G. Michael Youngblood
View a PDF of the paper titled Which LLM is Best for Translating Natural Language Goals to PDDL, by Tomas Balyo and Lukas Chrpa and G. Michael Youngblood
View PDF
HTML (experimental)
Abstract:Bridging the gap between human intent and machine execution remains a challenge in automated planning, where expressing goals in formal languages like PDDL restricts accessibility to non-experts. This paper empirically evaluates whether current Large Language Models (LLMs) can reliably translate natural language testing goals, written in informal language by video game testers, into well-formed PDDL targets suitable for classical planning. We present a carefully designed prompt template, integrating insights from iterative experimentation, aimed at maximizing both accuracy and response coherence from multiple state-of-the-art LLMs. Six contemporary models are systematically assessed on correctness, speed, and error tendencies using real-world, domain-specific benchmarks. All models demonstrate high correctness, exceeding 92\%, with Gemini 2.5 Flash achieving the highest accuracy at 96\% and the lowest incidence of false positives, while GPT-4.1 leads in response speed. Despite these advances, critical distinctions exist in model performance, and occasional failures arise from language ambiguity and limitations in domain representation. Our analysis underscores both the significant progress and ongoing gaps in enabling LLMs to act as robust bridges between natural language objectives and automated planning pipelines.
Subjects:
Artificial Intelligence (cs.AI)
Cite as:
arXiv:2609.18731 [cs.AI]
(or
arXiv:2609.18731v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.18731
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Tomas Balyo [view email]
[v1]
Wed, 16 Sep 2026 14:29:57 UTC (51 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Which LLM is Best for Translating Natural Language Goals to PDDL, by Tomas Balyo and Lukas Chrpa and G. Michael Youngblood
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)