Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary, Maike Züfle, Sara Rajaee, Pinzhen Chen, Jannis Vamvas, Sara Papi, et al.
cs.CL
Quick verdict
社区构建的MT基准测试，包含3456个难例外加基于规则的验证器，将前沿模型与人类排名区分开来；尽管依赖未量化的LLM验证器，依然坚实。
Summary
Last Translation Benchmark是一个社区构建的“活”集合，包含3456个经同行评审的难翻译例子（文本、图像、音频、视频；涵盖109种主要语言），每个例子都带有参考翻译和手工制作的验证规则；当一个翻译通过所有规则时，LLM验证器会判定其成功。通过专用平台（244位贡献者，166个机构）收集的示例被接受，需满足9/10个平台模型未通过至少一条规则，而人类参考翻译通过；每个例子成本约0.12美元。在前29个系统中，很少有多样化的前沿模型能通过规则：在Gemma 4作为验证器的情况下，Gemini 3.1 Pro通过了43.8%，大多数系统低于15%，而人类参考达到约99%；ChrF、COMET/MetricX和通用LLM判断将前沿输出评分在71-87之间，且验证器的 Kendall tau-b 模型排名稳定性为86.9，与人类排名的一致性高于判断或指标，并且比这些更小的自我偏见。规则作为预言机，将LLM翻译从7.2%跃升至89.8%。作者诱导了一个翻译难度分类法，并分析了跨目标语言的示例转移。局限性：以英语为主（73%非英语→英语），收集期间显示的模型也出现在结果中，没有置信区间，验证器与人类的对应关系仅在317个例子上进行了验证。
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Each LTB example carries handcrafted verification rules, enabling reproducible, actionable automatic evaluation via an LLM verifier whose pass rate is the reported metric (Sec. 1.3, Sec. 2).
- State-of-the-art models rarely satisfy the rules: best model Gemini 3.1 Pro reaches only 43.8% verifier pass rate (Gemma 4 verifier) vs. ~99% for human references (Main Table 2).
- Verification-rule evaluation is more stable across judge choice, better aligned with human rankings, and less self-biased than generic LLM judges or neural metrics (Tables 4-5).
- Frontier LLMs largely cannot identify what makes an example hard: providing gold rules lifts translation success from 7.2% to 89.8%, while LLM-generated rules give only 12.9% (Table 3).
Strengths
Scale and breadth with difficulty built into acceptance
3,456 accepted examples across 109 main languages from 244 contributors at 166 institutions (Sec. 3.1, Table 1); acceptance requires 8 of 10 platform models to fail at least one rule while the human reference passes (Sec. 2, Fig. 1), at ~$0.12 per accepted example.
The evaluation paradigm is validated empirically, not just asserted
Verifier pass rate is cross-verifier stable (Table 4 Kendall tau-b 86.9 diagonal vs. 71.3 for judges and 35.1 for metrics), agrees best with human rankings, and shows smaller self-bias than LLM-as-judge (Table 5: e.g., Gemini 3.1 Pro +8.9% as judge vs. −8.9% as verifier).
Informative negative results and ablations
Diagnostic depth beyond a leaderboard
Weaknesses
Selection circularity: difficulty is defined against models that also appear in the results
The verifier is itself an LLM and its per-item validity is thin
No uncertainty quantification anywhere
Severe English pivot and numeric presentation issues
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Last Translation Benchmark (LTBv1)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Last Translation Benchmark (LTB)?
A community-built, continuously updated machine translation benchmark of 3,456 peer-reviewed hard examples (text, images, audio, video) across 109 main languages. Unlike standard test sets, each example ships with a correct reference translation and handcrafted verification rules, and an LLM verifier scores a system by the percentage of examples where its translation passes all rules (LTBv1, September 2026).
How well do current translation models perform on LTB?
Poorly by design. In the paper's headline table, the best model, Gemini 3.1 Pro, passes only 43.8% of examples with Gemma 4 as verifier; most systems score under 15%, Google Translate about 3%, while the human reference translations pass roughly 99%. Generic metrics and LLM judges rate the same frontier outputs 71-87, illustrating their poor discrimination on hard cases.
How is LTB different from WMT test sets or adversarial MT benchmarks?
WMT test sets are sampled from natural text and are saturating; adversarial sets are automatically generated and often unnatural. LTB examples are human-authored from real usage, peer-reviewed by a fluent reviewer, and accepted only if most current models fail at least one targeted rule while the human reference passes — so difficulty is verified, not assumed.
Can I contribute examples or submit my model to the leaderboard?
Yes. Submissions go through the project's web platform (last-translation-benchmark.vilda.net); after 10 accepted examples you are invited as a dataset co-author. The dataset is on Hugging Face (zouhar/last-translation-benchmark, CC BY 4.0) with MIT-licensed code on GitHub, and the leaderboard accepts 'blind' and 'oracle' submissions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The multi-panel Main Table 2 (~30 rows × 19 columns) and Table 4 were recovered from flattened PDF text; row/column attribution of individual numeric cells could not be independently double-checked against the rendered PDF.
- The text ends cleanly after Appendix C / Example 22; no truncation is claimed beyond that point. Figures 1-2 and the examples were extracted as inline text, so layout-dependent details (e.g., exact error-span marks in Figure 2) are approximations.
- Table 4's identical HUMAN with-rules/without-rules rows (90.5/34.9/16.2) may be an extraction artifact of the flattened table; flagged as an inconsistency/question rather than asserted as a paper error.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.