Evaluating Communicative Success in Machine-Translated Conversation
Faiz Ghifari Haznitrama, Alice Oh
cs.CL
Quick verdict
Checklist benchmark for interpreter-MT communicative success; judge validation is thorough, but model rankings carry an admitted context-asymmetry confound.
Summary
The paper introduces a three-layer checklist-and-judge framework for evaluating interpreter agents — machine translation systems mediating live conversation — across semantic (L1), pragmatic (L2), and cultural-social (L3) dimensions. Built on 5,624 OpenSubtitles-derived scenarios in Arabic, Bengali, Indonesian, and Korean (6 pairs, 12 directions), it benchmarks 10 interpreter setups single-turn and evaluates all pairs in scripted and live multi-turn modes with simulated users. All setups decline from L1 to L3, with overall pass rates from 91.3% (Gemini 3.1 Pro) to 24.2% (SeamlessM4T v2). For stronger interpreters, within-system correlations with CometKiwi, GEMBA-MQM, MQM-APE, and RATE collapse: 10.3–12.2% of metric top-quartile outputs still fail the checklist, while 62.0–69.6% of bottom-quartile outputs pass. A prompt ablation raises pooled scores from 36.4% (direct, no context) to 51.3% (full cultural-context brief). Validation includes adversarial judge tests (DeepSeek V4 Pro disqualified by a 58-point polarity drop), cross-judge agreement, self-preference checks, and IRB-approved human annotation (75.4% agreement, κ=0.403). Main limitations, mostly self-acknowledged: MT baselines receive no context while LLM interpreters do, confounding capability comparisons; annotator agreement is low and the judge under-scores by up to 18.8 points after calibration; OpenSubtitles contamination and scripted dialogue limit generalization.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 3
● low 0
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing MT metrics measure fidelity rather than communicative success and, within strong interpreters, are near-uninformative about checklist failures (within-system correlation collapses; 10.3-12.2% of top-quartile outputs still fail).
- Every evaluated interpreter setup degrades from semantic (L1) to pragmatic (L2) to cultural-social (L3) success, in the same ordering for all systems.
- Turn-level scoring systematically overstates whole-conversation success; conversation-level scores land below turn-level rates for every system.
- Scenario context, structured specification-aware briefs, and pair-specific cultural context materially raise communicative success (36.4% → 51.3% pooled), with gains concentrated in stronger setups.
- The checklist judge survives adversarial validation (dropout, rewording, polarity, planted violations) and human calibration well enough to serve as primary judge, with DeepSeek V4 Pro disqualified.
Strengths
Deep, multi-angle judge validation
Adversarial battery (input dropout ~2-3%, rewording flips 2.2-3.0%, polarity inversion, planted violations), cross-judge κ (0.462-0.602 overall, ~0.92-0.96 on L1), and a genuine circularity check (§F.4: Gemini judge Δ=+0.3, p=0.67 on its own outputs; −2.4, i.e., stricter, vs DeepSeek).
Quantified orthogonality to MT metrics, with power analysis
Table 10 shows 10.3-12.2% of CometKiwi/GEMBA-MQM/MQM-APE/RATE top-quartile Gemini-Pro outputs score ≤0.6 on the checklist while 62.0-69.6% of bottom-quartile outputs reach ≥0.9; §J.2 correctly argues n≈5,600/system with CIs ≤±0.03 makes the near-null within-system correlation precise, and demonstrates pooling inflates alignment via between-system variance.
Scale and full multi-turn coverage
Unusually candid self-auditing
Weaknesses
Headline rankings confounded by context asymmetry
Layer decline partly constructed by instruction
Absolute scores tied to one vendor's model and noisy human calibration
Ablation gains not cleanly attributable
Contamination and scripted data acknowledged but unquantified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a real evaluation gap: MT systems increasingly act as live interpreters, yet are judged by sentence-level fidelity metrics. It proposes a three-layer checklist-and-judge framework (L1 semantic, L2 pragmatic, L3 cultural-social) applied per translated turn and, in multi-turn mode, per conversation. Central claims: (1) every setup declines from L1 to L3; (2) standard MT metrics (CometKiwi, GEMBA-MQM, MQM-APE, RATE) lose within-system correlation with communicative success as interpreter strength rises; (3) turn-level scoring overstates whole-conversation success; (4) context and elicitation — scenario context, specification-aware briefs, pair-specific cultural context — materially raise success (36.4% → 51.3% pooled, Table 1). Claims are mostly scoped and hedged, backed by an unusually candid Limitations section.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'communicative success' mean in machine translation evaluation?
It asks whether a translated message achieves what the speaker intended in the conversation — preserving intent, tone, naturalness, and social/cultural appropriateness for the listener — rather than only matching the source wording. This paper measures it with binary, scenario-specific checklist items at three layers: semantic, pragmatic, and cultural-social.
Why do standard MT metrics like BLEU or COMET fail for interpreter-style translation?
Because they score fidelity to the source, they can rate a fluent, semantically accurate translation highly even when it uses the wrong politeness level or erases social hierarchy. In this paper, for the strongest interpreters, 10.3-12.2% of outputs in a metric's top quartile still fail the communicative checklist, and within-system correlation between metric scores and checklist pass rates is near zero.
Which languages and models are covered by this benchmark?
Arabic, Bengali, Indonesian, and Korean, in all six pairwise combinations and twelve translation directions, over 5,624 OpenSubtitles-derived scenarios. Ten interpreter setups are compared, including Gemini 3.1 Pro and Flash Lite, DeepSeek V4, GPT-5.4 Mini, Qwen3.5 Flash, Google Translate, NLLB-3.3B, Tiny Aya, and SeamlessM4T v2.
Can I trust the absolute pass rates the benchmark reports?
Treat them as conservative and model-relative. The chosen LLM judge agrees with native-speaker annotators 75.4% of the time (κ=0.403, moderate), and a calibration cascade suggests the judge under-scores by 11.8-18.8 points in three of four languages. The authors also document that MT baselines received no context while LLM interpreters did, so cross-family rankings mix capability with context access.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction is truncated: parts of §4.1 and several appendices (E.2-E.3, I, H.2-H.3, K) are omitted, and the Figure 3 numeric grid is only partially recoverable from garbled chart text.
- Figures 1-9 cannot be inspected directly; numeric claims are taken from embedded chart labels and tables, and non-Latin examples in Table 11 show OCR corruption.
- 2026-dated systems (Gemini 3.1, GPT-5.4, DeepSeek V4, Qwen3.5, Grok 4.3, RATE) are cited to model cards/preprints not verifiable from this text; the authors themselves note no official card exists for the Grok build used as a secondary judge.
- Availability claims (scenarios, system outputs, scoring scripts, prompts) rest on the paper's statements; no repository contents were inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.