The Blindness of Document-Level Translation Evaluation
Ahrii Kim, Vilém Zouhar, Chanjun Park, Seong-heum Kim
cs.CL · cs.HC
Quick verdict
Claims document-level MT evaluation protocols fail to register coherence; large blinded counterfactual study is rigorous, though scope is one language pair.
Summary
The paper tests whether document-level presentation in MT evaluation actually elicits document-level judgment. Using the WMT 2025 English→Korean test set (23 documents, 9 MT systems plus a human reference), the authors built a counterfactual MIX condition in which each document is assembled from segments of different systems, preserving document layout while destroying cross-segment coherence. Ten professional translators produced 18,420 Error Span Annotation judgments across SEG (isolated), DOC (coherent), and MIX conditions; 14 automatic metrics were run in parallel. Equivalence testing (TOST) shows human scores, system rankings, and error annotations are statistically indistinguishable between DOC and MIX, and all metrics yield identical DOC/MIX system rankings (τ=1.00), with several document-level metrics even scoring MIX marginally higher. Blinded manipulation checks confirm the disruption is perceptible (87.3% of raters identified the coherent passage). Behavioral measures show presentation does change annotator behavior (faster annotation, divergent agreement patterns), yet nothing registers in the recorded output. The authors conclude the evaluation protocol, not the annotator, is blind, and that segment-level gold data may not support document-level progress. Limitations: single language pair, literary domain rests on two source documents, only one incoherence type tested, and several document-level metrics suffered input truncation.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under the ESA protocol, human scores, system rankings, and error annotations are statistically equivalent between coherent (DOC) and deliberately incoherent (MIX) documents (TOST p<0.001 at ε=3 / εerr=0.5).
- All 14 automatic metrics produce identical system rankings on DOC and MIX (τ=1.00), and document-level metrics move in the wrong direction, scoring MIX marginally higher.
- The equivalence is not explained by perception or context length: blinded raters identify coherent passages in 87.3% of trials, and LLM judges with effectively unbounded context fail identically.
- Document presentation changes annotator behavior (76s/61s/56s per segment for SEG/DOC/MIX; inverted per-document vs per-segment IAA pattern) but this change never reaches the recorded output — the protocol, not the annotator, is blind.
- Because gold data under segment-centric protocols cannot register document-level quality, reported document-level progress is hard to separate from segment-level progress, and new elicitation protocols are required.
Strengths
Real equivalence testing, not failed rejection
TOST with pre-specified margins (ε=3 ≈ 0.2 SD on the 100-point ESA scale; εerr=0.5 ≈ 20% of the 2.3-error baseline), with robustness across paired/independent designs and ε∈{3,5,7} — all 18 cells p<0.001 (Table 8, Appendix B.1).
Two-stage manipulation checks make the null interpretable
Study 1 (486 judgments) shows pooled systems are distinguishable in 85–94% of trials even for maximally overlapping pairs; Study 2 (102 passage pairs, 3 blind raters) shows the assembled MIX reads as inconsistent: 87.3% DOC identification, Krippendorff's α=0.73, binomial p<0.001 (Section 4.3, Appendix A.2).
Scale, blinding, and confound control
Unusually candid self-audit
Artifacts released for verification
Weaknesses
Single language pair limits the sweep of the conclusion
'All 14 metrics' includes truncation-compromised variants
Human H2 formally fails its pre-set criterion
One perturbation family only
Effective sample and domain balance are thin
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether document-level presentation in MT human evaluation actually elicits document-level judgment, and whether document-level automatic metrics measure coherence. The construct under test is the ESA protocol (Kocmi et al., 2024b) as deployed in WMT-style campaigns, plus 14 automatic metrics. The central claim: DOC (coherent) and MIX (deliberately incoherent, system-mixed) documents receive statistically equivalent scores, rankings, and error annotations; since blinded readers perceive the incoherence (87.3% in Section 4.3's Study 2), the failure is located in the protocol, not the annotator. The claim is deliberately scoped: one language pair (En→Ko), one protocol (ESA), one perturbation family (system mixing = consistency breaks). The authors correctly do not claim that all document-level phenomena fail to register — internal errors like mistranslated anaphora are untouched by the manipulation (Limitations, 'Scope of the counterfactual').
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'the blindness of document-level translation evaluation' mean?
The paper shows that when annotators evaluate full documents, their recorded scores and error marks do not change even when the 'document' is secretly assembled from different machine translation systems and is internally incoherent. Document presentation exists, but the evaluation protocol gives annotators no channel to record document-level problems — so the protocol is 'blind' to coherence, not the human raters, who perceive it easily (87.3% identification in blind tests).
How did they create incoherent documents that raters still scored normally?
The MIX condition takes a real document and replaces each sentence with a translation from a different MT system, keeping the document layout. In Korean, speech level is marked on every sentence-final verb, so mixing systems creates jarring register shifts within a passage. Blind readers could reliably tell which passage was coherent, yet official ESA scores, system rankings, and error counts were statistically indistinguishable between coherent and mixed documents.
Does this mean document-level MT evaluation is useless?
The paper's claim is narrower: current segment-centric protocols (like Error Span Annotation used in WMT-style evaluation) do not measure document-level quality, so progress reported under them is hard to distinguish from segment-level progress aggregated at the document level. The authors call for new protocols that give annotators a way to record document-level judgments, not for abandoning document-level evaluation.
Was the study only done for Korean?
Yes — all 18,420 expert annotations cover English→Korean only, deliberately chosen because Korean's discourse features (pro-drop, honorifics, cross-sentence entity consistency) should make coherence easiest to detect. The authors state that replication across typologically diverse language pairs is the primary next step and that their finding is scoped to English→Korean under ESA.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2/3 and Tables 2/6 are layout-scrambled by PDF extraction (axis labels and panel captions interleave with prose); all numbers cited in this review were cross-checked against table text and body prose.
- No truncation observed: the extracted text ends with a complete sentence in Appendix C.3, and references plus Appendices A–C appear in full.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.