Discourse Dependency: A Continuous Criterion for Translation Difficulty
Ahrii Kim, Chanjun Park, Seong-heum Kim
cs.CL
Quick verdict
Proposes DDP, a source-side discourse-distance criterion for MT difficulty; convincing validation and En-Ko human eval, but untested beyond one language pair.
Summary
This paper introduces discourse dependency (DDP), a metric-free, source-side measure of translation difficulty: the maximum distance, in segments, that a segment must look back to resolve its entity mentions and pronouns, computed from NER (four entity types) and dependency-filtered referential pronouns with chain accumulation. Validated against OntoNotes 5.0 gold coreference on 80 documents (2,143 segments), DDP correlates with gold distance (Pearson r=0.81) and errs one-sidedly in 99.2% of segments (86.4% underestimates, 12.8% exact, 0.8% overestimates from NER false positives), so high-DDP segments reliably need long-range context. Profiling WMT24++ and WMT25 shows both skew toward low-DDP segments (literary means 52.8 and 222.6 versus news 9.7 and social 3.8 in WMT24++), variation that domain labels conceal. In an English-Korean post-editing setup with six LLMs and five context-injection strategies, human preference for professional post-edits grows monotonically with DDP (Spearman rho=0.28, p=1.2e-9, n=498); at DDP>=15 raters prefer human revisions by +0.35 (Wilcoxon p=3.0e-5, CI [+0.18,+0.52]) while XCOMET separates nothing (p=0.40). Full-document context costs 40-84x tokens without gain. Main limitations: human evaluation is confined to English-Korean with non-randomly selected, high-DDP-oversampled subsets; the anchor inventory excludes bridging anaphora and lexical cohesion; inter-annotator agreement is modest (Kendall tau=0.37).
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- DDP, computed source-side from entity re-mentions and pronominal coreference, is a one-sided (lower-bound) estimate of a segment's contextual dependency in 99.2% of segments, validated at Pearson r=0.81 against OntoNotes gold coreference.
- WMT24++ and WMT25 are heavily skewed toward low-DDP segments (WMT24++ literary mean 52.8 vs news 9.7, social 3.8; WMT25 literary 222.6), and domain labels do not capture within-domain DDP variation.
- In English-Korean APE, human preference for professional post-edits over all five LLM context-injection strategies grows monotonically with DDP, becoming significant at DDP≥15 (+0.35, Wilcoxon p=3.0e-5).
- Automatic metrics, including document-level ones, are flat across the DDP spectrum and cannot separate strategies that humans clearly differentiate (XCOMET p=0.40), exposing a metric blind spot precisely where context matters.
Strengths
Continuous source-side difficulty criterion with a clear conceptual payoff
DDP (Section 3) is metric-free and computable before any translation exists, unlike metric-derived difficulty selection (Section 2 explicitly contrasts with Zouhar et al. 2025/2026); this enables prospective test-set design rather than post-hoc filtering.
Validation measures error directionality, not just correlation
Appendix A.1 quantifies the lower-bound property against OntoNotes 5.0 gold coreference (80 docs, 2,143 segments): Pearson r=0.81, 86.4% underestimates, 12.8% exact, 0.8% overestimates attributed to NER false positives — the abstract's 99.2% one-sided claim is grounded, and the narrative-genre gap (r=0.74) is honestly attributed to unmodeled bridging/lexical chains.
Tool robustness is ablated, not assumed
Headline finding is tested, not just plotted
Useful negative results with cost accounting
Weaknesses
All human evidence comes from one language pair
Human evaluation subsets are non-randomly sampled
Narrow anchor inventory limits the 'difficulty' reading
Modest annotator agreement with thin replication
Small and literary-concentrated corpus
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets an unmeasured axis of MT evaluation difficulty: how far a segment must reach back into its document to resolve entities and pronouns. It formalizes this as discourse dependency (DDP) (Section 3), profiles WMT24++ and WMT25 with it (Section 3.3, Figure 1, Tables 5-6), and uses it to stratify an English-Korean post-editing study comparing five context-injection strategies across six LLMs (Sections 4-5). Four claims are advanced: DDP is a one-sided lower bound on contextual dependency; both benchmarks skew toward low-DDP segments; human preference for professional post-edits grows monotonically with DDP while automatic metrics stay flat; and context beyond a sequential window does not help.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is discourse dependency (DDP) in translation?
DDP is a score proposed by this paper to measure how hard a segment is to translate because of discourse: it is the maximum distance, counted in sentences, that a segment must look back into its document to find what its named entities or pronouns refer to. It is computed from the source text only, using NER and pronoun-resolution rules, and needs no reference translation or model scores.
Do current MT benchmarks like WMT contain hard discourse segments?
According to the paper's analysis, mostly no: in WMT24++ English-Korean, news segments average DDP 9.7 and social 3.8, with 62.3% of social segments needing no prior context at all; only literary texts reach high dependency (mean 52.8, max 171; WMT25 literary mean 222.6, max 482). Domain labels do not capture this variation.
Can LLMs actually use document context when translating?
In the paper's English-Korean post-editing experiments with six LLMs and five context-injection strategies, giving more or differently selected context did not help on contextually demanding segments: even full-document context (40-84x more tokens) brought no consistent gain, and on segments with DDP≥15 human raters significantly preferred professional human post-edits while automatic metrics like XCOMET detected no difference.
Is the DDP code or data available?
The paper states that the DDP-annotated benchmark statistics will be released under CC-BY 4.0 and the code under MIT, with a GitHub repository referenced in the paper; the underlying WMT24++/WMT25 test sets are public, though the OntoNotes coreference validation data requires an LDC license.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with three marked [...sampling gap...] regions; content inside them (e.g., the body of Table 2, the tail of Section 3.2, the head of Limitations, Appendix B.4-B.5) was not visible to this review and may exist in the full paper.
- Figures are present only as extracted axis labels and captions; chart shapes were judged from captions and reported statistics rather than visual inspection, and annotator staffing details (inside a sampling gap, Appendix B.5) could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.