大模型
arxiv.org ·
双向评分：LLM 能生成却无法可靠解析 MRS
该 arXiv 论文基于英语资源语法（ERG）的 MRS 语义表示，重建 10K 句测试集，对 Claude Sonnet 4.5 与 Claude Opus 5 做双向评测。MRS 转文本方向，Opus 以 76.3 BLEU 超过在 72k 对上训练的系统（66.1），接近百万额外对训练系统（77.2）；Sonnet 为 65.7，借助 ACE 候选提升至 69.6。文本转 MRS 方向，两模型在谓词与论元图上仅得 57.2 和 65.5 F1，ACE 为 91.0，精确匹配约 1%。作者据此指出，生成分数本身不足以证明模型理解形式语义表示。
AI推荐理由 arXiv 论文，作者 Soham Dan，摘要给出明确实验设置与可验证数字：在重建的 10K 句测试集上，Claude Opus 5 由 MRS 生成英文达 76.3 BLEU，高于其 72k 对训练系统（66.1 BLEU），接近百万额外对训练系统（77.2 BLEU）；Sonnet 4.5 为 65.7 BLEU，加入 ACE 候选后升至 69.6。反向解析任务中两模型仅 57.2/65.5 F1，远低于 ACE 的 91.0，精确匹配约 1%。核心新信息是生成分数高不等于理解形式语义表示，对 LLM 语义能力评估有直接参考价值；局限是仅两个模型、单一 ERG/MRS 基准，且为预印本未经同行评审。
78
正文
Computer Science > Computation and Language
arXiv:2609.30071 (cs)
[Submitted on 24 Sep 2026]
Title:Scoring Both Directions: LLMs realize the MRS they cannot reliably parse
Authors:Soham Dan
View a PDF of the paper titled Scoring Both Directions: LLMs realize the MRS they cannot reliably parse, by Soham Dan
View PDF
HTML (experimental)
Abstract:The English Resource Grammar (ERG) is a hand-written computational grammar of English. Given a sentence, its processor, ACE, produces a formal meaning representation called Minimal Recursion Semantics (MRS): a graph of the sentence's predicates and their arguments. The grammar is bidirectional and can also turn an MRS back into an English sentence. \citet{hajdik2019} used the ERG's treebank to build a benchmark for that generation task, MRS to text, and trained sequence-to-sequence models to solve it. The parsing task, text to MRS, can be tested on the same sentences. We reconstruct their 10K-sentence test split, and score two large language models, Claude Sonnet~4.5 and Claude Opus~5, in both directions against their trained systems and against ACE, with no task-specific training. Given an MRS and three examples, Opus writes the sentence at 76.3 BLEU, ten points above their system trained on 72k pairs (66.1 BLEU), and comparable to their system trained on a million extra pairs (77.2 BLEU). Sonnet scores 65.7 BLEU, and letting it choose among ACE's own candidate sentences lifts it to 69.6, while a pooled judge that keeps Opus's own sentence among the candidates adds 0.6 points (77.0 BLEU). In the parsing direction, however, the models fall far behind ACE: asked for the MRS of the same sentences, they reach 57.2 (Sonnet) and 65.5 (Opus) F$_1$ on the graph's predicates and arguments against 91.0 for ACE, and exact-match the gold on about 1\% of sentences. We characterize the failure modes for the parsing tasks, and conclude that a generation score alone does not show that models understand formal semantic representations.
Subjects:
Computation and Language (cs.CL)
Cite as:
arXiv:2609.30071 [cs.CL]
(or
arXiv:2609.30071v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.30071
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Soham Dan [view email]
[v1]
Thu, 24 Sep 2026 16:27:54 UTC (159 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Scoring Both Directions: LLMs realize the MRS they cannot reliably parse, by Soham Dan
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)