Scoring Both Directions: LLMs realize the MRS they cannot reliably parse
cs.CL
Quick verdict
Claims prompted LLMs match trained realizers on MRS→text but fail parsing (65.5 vs 91.0 EDM); well-controlled, though stats and model coverage are thin.
Summary
The paper evaluates whether large language models can handle Minimal Recursion Semantics (MRS) in both directions — realizing MRS into English and parsing English into MRS — using the English Resource Grammar (ERG) and its processor ACE as the reference system on both ends. The author reconstructs the Hajdik et al. (2019) 10K-sentence test split from the Redwoods treebank (10,201 items; 9,932 scoreable) and prompts Claude Sonnet 4.5 and Opus 5 with three exemplars, no task-specific training. Realization: Opus reaches 76.3 BLEU, about ten points above the 72k-pair trained seq2seq (66.1) and near the million-pair system (77.2); Sonnet hits 65.7. Judge/rerank hybrids lift Sonnet to 69.6 and Opus to 77.0. Round-trip parses confirm the generated sentences largely preserve meaning (85.4/89.8 spans-off EDM vs 91.0 for gold). Parsing: the same models reach only 57.2/65.5 EDM F1 against ACE's 91.0, with exact match on ~1% (ACE: 50.5%). Calibration inputs and failure-mode analysis show errors concentrate in the ERG's predicate inventory and argument attachment, not format or role labels. Memorization controls (verbatim probe, noun substitution on 140 pairs) find small effects. Limitations: two proprietary models, one prompting setup, ~10% parsing sample, no significance testing; no code release stated.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- With three exemplars and no task-specific training, Opus 5 realizes ERG MRS to English at 76.34 BLEU, about ten points above the 72k-pair trained seq2seq and essentially level with the million-pair silver-data system.
- Reranking hybrids that let the model judge ACE's n-best candidates lift Sonnet to 69.64 and Opus to 76.95 BLEU, with the grammar's help showing diminishing returns for the stronger model.
- On the reverse task, the same models reach only 57.2/65.5 spans-off EDM F1 against ACE's 91.0 and exact-match the gold MRS on about 1% of sentences.
- The parsing failures lie in the ERG's predicate inventory and argument attachment, not in output format or role-label choice.
- A high generation score alone is not evidence that a model understands a formal semantic representation.
Strengths
Bidirectional evaluation on identical items
The same reconstructed Hajdik et al. (2019) split (10,201 items; 9,932 scoreable) is scored MRS→text and text→MRS with the grammar as reference both ways (§1-2, Tables 1 vs 3), isolating the produce/understand asymmetry far better than the cross-benchmark comparisons it improves on (Ettinger et al. 2023; West et al. 2024).
Metric calibration rows make scores interpretable
Table 3 scores gold with every predicate name junked (79.2 Smatch but 0.0 on names/EDM names) and a constant MRS (33.1 Smatch) against gold, showing concretely why Smatch flatters structure-only answers and why spans-off EDM is the correct primary metric.
BLEU is not taken at face value
Memorization is controlled, not assumed away
Failure modes are quantified, not anecdotal
Weaknesses
No inferential statistics
Uncontrolled decoding across models
ACE baseline is in-sample
No contamination control in the parsing direction
Parsing rests on a ~10% sample
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks one cleanly scoped question: can prompted LLMs handle the English Resource Grammar's (ERG) Minimal Recursion Semantics in both directions — realization (MRS→text) and analysis (text→MRS) — on the identical ~10K-sentence test split Hajdik et al. (2019) built from the Redwoods treebank? Three claims anchor it (§1, §5): (1) with three exemplars and no task-specific training, Opus 5 realizes at 76.34 BLEU, about ten points above the 72k-pair trained seq2seq (66.11) and essentially level with the million-pair silver-data system (77.17), while Sonnet 4.5 matches the former (65.65); (2) judge/rerank hybrids over ACE's n-best lift Sonnet to 69.64 and Opus to 76.95; (3) in the parsing direction the same models collapse to 57.2/65.5 spans-off EDM F1 against ACE's 91.0, exact-matching gold on 0.6%/1.4% versus 50.5%. The conclusion — generation scores are not evidence of representational understanding — is modestly stated and follows from the data.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper show about LLMs and formal semantics?
It tests Claude Sonnet 4.5 and Opus 5 on the English Resource Grammar's MRS in both directions. The models write good English from MRS (Opus 76.3 BLEU, above a system trained on 72k pairs) but largely fail to produce the gold MRS from a sentence (57-66 EDM F1 vs ACE's 91, exact match on ~1%), so generating from a representation does not imply understanding it.
Which models and baselines were compared?
Prompted Claude Sonnet 4.5 and Opus 5 (three examples, no fine-tuning) against the Hajdik et al. (2019) seq2seq systems (66.1 BLEU trained on 72k gold pairs; 77.2 with ~1M silver pairs) and the ACE processor of the ERG, which realizes at 61.8 BLEU and parses at 91.0 EDM F1.
Does high BLEU mean the model understood the MRS graph?
The authors argue no. Round-trip parsing shows the generated sentences mostly preserve meaning (85-90 EDM), but when asked for the MRS itself the same models miss the grammar's exact predicate names and argument attachments, so surface generation quality is not evidence of representational competence.
Is code or data available for this paper?
The paper fully specifies its pipeline (ACE 0.9.31 with ERG 1214, Hajdik's public data conversion, PyDelphin 1.11, SacreBLEU 2.6, and verbatim prompts in Appendix A), but the reviewed text does not mention a code or data release, and the LLM results require paid access to proprietary Anthropic APIs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract in the supplied metadata is truncated mid-sentence, but the extracted body contains the complete abstract, all sections, references and Appendix A.
- Figure 1 is available only via its caption; the annotated example image itself cannot be verified from the extracted text.
- Tables are flattened inline in the extraction; Table 3's eight metric columns were parsed from running text, so individual cell values carry minor mis-attribution risk.
- No code/data availability statement appears in the extracted text; availability was judged accordingly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.