Larger Context Window, Fewer Overcorrections: Optimizing Prompts and Batching for Minimal-Edit Grammatical Error Correction
Kateryna Karpo, Artem Chernodub
cs.CL
Quick verdict
Prompt-based GEC SOTA (78.32 F0.5 BEA-2019, 0.38 from fine-tuned SOTA) via taxonomy, batching, prompt opt; solid empirics but single-run, mechanism unverified.
Summary
The paper attacks overcorrection in minimal-edit grammatical error correction (GEC) by prompted LLMs, without fine-tuning. Three components are combined: (1) taxonomy-based instructions enumerating 25 ERRANT error categories on top of minimal-edit constraints; (2) batching multiple sentences into one prompt, which acts as a regularizer against overcorrection — edit rates fall and precision rises as batch size grows (optima B=15 for high-capacity models, B=30–60 for medium, B=5 for Qwen3-8B), hypothesized to stem from attention dilution; (3) human-in-the-loop LLM-assisted prompt optimization implemented as a Claude Code skill, tuned on BEA-2019 dev. Across seven models (six commercial, plus open-weight Qwen3-8B) and two benchmarks, the best configuration (Gemini 3.1-Pro, optimized prompt A.6.4, B=15) reaches F0.5=78.32 on BEA-2019 test — a prompt-based SOTA 0.38 points below the fine-tuned single-model SOTA (Staruch et al., 2025; 78.70) — and 67.08 on CoNLL-2014. The taxonomy helps only the strongest models; batching carries a fluency cost visible in declining Scribendi scores, and Claude Sonnet 4.6's edit rate rises rather than falls with batch size. Key limitations: single runs without variance estimates, possible pretraining contamination of the public benchmarks and the BEA-derived few-shot examples, an unverified attention-dilution mechanism, and prompt optimization restricted to medium-capacity and open-weight models.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Taxonomy-based instructions enumerating 25 ERRANT categories bound the edit space and improve F0.5 only for the strongest models while acting as noise for others (Table 1).
- Packing multiple sentences into one prompt acts as a targeted regularizer against overcorrection, reducing edit rates and improving F0.5 across most models (Fig. 2, Tables 4–5).
- LLM-assisted prompt optimization yields F0.5=78.32 on BEA-2019 test with Gemini 3.1-Pro — a prompt-based SOTA within 0.38 points of the fine-tuned single-model SOTA (Staruch et al., 2025).
- The batching effect is hypothesized to arise from attention dilution induced by the bounded capacity of self-attention scores (presented as an unverified conjecture).
Strengths
Comprehensive, single-variable-controlled experimental grid
Seven models in two capability tiers plus open-weight Qwen3-8B; five manual prompts (A.1–A.5); batch sweep B∈{1,2,5,15,30,60,120} on the 4,384-sentence BEA-2019 dev; optimized prompts evaluated cross-model (Tables 1, 2, 4, 5). The Appendix C skill enforces one-change-per-iteration with per-ERRANT-category attribution, an unusually disciplined optimization protocol.
Novel, practical core finding
Using the input batch as an anti-overcorrection regularizer is new to minimal-edit GEC. The behavioral evidence is well instrumented: Qwen3-8B's 0-edit share grows 43.8%→80.7% from B=1 to B=120, precision rises monotonically for all swept models, and word-edit distributions are given per (model, B) cell (Tables 4–5) — while batching simultaneously cuts API cost and latency.
Landmark held-out result
Unusual transparency and candor
Weaknesses
No variance, no significance testing anywhere
Contamination exposure undermines the parity framing
Mechanism claim untested and slightly overgeneralized
Dev-set selection with demonstrated negative transfer
Reproducibility friction on commercial models
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets minimal-edit GEC with API-accessed LLMs, motivated by users who cannot fine-tune (§1). Three empirical claims: (i) 25-category ERRANT taxonomy instructions bound the edit space productively only for the strongest models (Table 1: Claude Opus 4.6 dev F0.5 rises 58.75→59.28 from A.3 to A.5, but GPT-5.4 falls 57.47→55.47); (ii) packing B sentences per prompt is a targeted anti-overcorrection regularizer — edit rates fall and precision rises with B (Fig. 2, Tables 4–5); (iii) LLM-assisted prompt optimization lifts Gemini 3.1-Pro to F0.5=78.32 on BEA-2019 test, 0.38 below the fine-tuned single-model SOTA (Staruch et al., 2025). The attention-dilution mechanism is framed as a hypothesis and admitted unverified (Limitation 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What F0.5 does this prompting method achieve on BEA-2019, and how close is it to fine-tuned systems?
F0.5=78.32 with Gemini 3.1-Pro, the optimized prompt A.6.4, and batch size 15 — the best prompt-based result reported, just 0.38 points below the fine-tuned single-model SOTA of 78.70 (Staruch et al., 2025), while matching its precision (82.29 vs 82.28). On CoNLL-2014 test it reaches 67.08.
Why does batching multiple sentences into one prompt reduce overcorrection in GEC?
Empirically, larger batches shift model behavior toward leaving sentences untouched and raise precision (e.g., Qwen3-8B's share of unedited sentences grows from 43.8% at batch size 1 to 80.7% at 120). The authors hypothesize an attention-dilution effect — a bounded self-attention budget per token — but this mechanism is not experimentally verified, and one model (Claude Sonnet 4.6) actually makes more edits at larger batches.
Does listing the 25 ERRANT error categories in the prompt help every model?
No. The taxonomy improves the strongest models (Claude Opus 4.6 and Gemini 3.1-Pro on dev) but degrades several others, including GPT-5.4; the paper itself concludes the effect is model-dependent, which motivates its per-model prompt optimization step.
Is the code, data, or prompt set from this paper available?
Yes. The paper states that code, prompts, and outputs are publicly available at github.com/katerynkarpo/llm-en-gec, and all nine prompts plus the full prompt-optimization skill are reproduced in the paper's appendices. Note the commercial model endpoints are identified only by preview aliases for two of three providers.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 appear only as captions in the extraction (graphic content not rendered); every quantitative claim in this review was verified against Tables 1–5, which contain the underlying numbers.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.