TransClean: A Benchmark for Detecting and Extracting Clean Translations from Large Language Model Outputs
cs.CL
Quick verdict
First benchmark for extracting clean translations from noisy LLM outputs; thorough analysis, but LLM-annotated silver labels raise circularity concerns.
Summary
The paper addresses translation noise — extra text (language labels, explanations, bilingual repetitions) in LLM translation outputs that distorts MT evaluation. Analyzing 792,000 outputs from 12 open-weight LLMs across 22 language pairs and 3 prompt templates, the authors identify 12 recurring noise patterns grouped into formatting and content noise; pattern-frequency estimates from Claude Opus 4.6 correlate with a gemma-4 replication (Spearman rho=0.84). They build TransClean, a 9,900-instance benchmark pairing noisy outputs with clean translations: 8,800 synthetic instances generated by injecting patterns at empirically observed frequencies (gold references), plus 1,100 authentic noisy instances curated from real outputs with silver labels from three-LLM majority voting (GPT-5-mini, Qwen3.5-122B-A10B, gemma-4-31B-it). Benchmarking a span-based extractor (COMET-KIWI scoring of line-split spans) against LLM extractors shows noise detection is near-saturated (96-100% for the strongest methods) but exact-match extraction is hard: the best method (Qwen extractor) reaches 54.07% on synthetic and 52.18% on curated data, while span-based extraction collapses from 50.59% to 15.82% on authentic noise. Main limitations: silver labels rest on partial manual verification (only 143/315 disagreements checked); Qwen3.5-122B-A10B serves as both annotator and extractor, risking circularity; the exact-match metric penalizes valid paraphrases; noise-rate estimation relies on English-keyword regexes; proprietary models are absent.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Translation noise is prevalent and strongly model-dependent, affecting roughly 3% to 99.7% of outputs across the 12 studied LLMs (Table 1).
- Twelve recurring noise patterns fall into formatting and content categories, with content noise dominating (explanation ~33%, alternatives ~15%, off-topic ~12%; Table 2).
- TransClean is the first benchmark for clean translation extraction, combining 8,800 synthetic instances with gold clean references and 1,100 curated authentic instances with silver labels.
- Noise detection is near-saturated while exact-match extraction remains hard: the best method achieves 54.07% (synthetic) and 52.18% (curated) accuracy (Table 4).
- LLM-based extraction generalizes better to authentic noise than span-based QE extraction, which collapses from 50.59% to 15.82% on curated data (Table 4).
Strengths
First dedicated benchmark for a real, unsolved evaluation problem
Formalizes 'clean translation extraction' (§1) and delivers 9,900 noisy/clean pairs: 8,800 synthetic instances with gold reference translations plus 1,100 curated authentic instances, enabling both controlled and realistic evaluation (§3, Table 4).
Large-scale empirical grounding with cross-validated taxonomy
792,000 outputs (12 open-weight models x 22 LPs x 3,000 sentences, four corpus sources, Table A.1); 12 noise patterns in two categories (Table 2), with frequency estimates replicated by a second LLM yielding Spearman rho=0.84 (§2.3, footnote 5).
Clean control split exposes over-editing
Clear central finding separating detection from extraction
High reproducibility transparency
Weaknesses
Annotator-extractor circularity
Silver-label validity rests on partial verification
Exact match as the sole extraction metric
External validity of the prevalence study
Thin statistical reporting and small validation samples
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets "translation noise": any content in an LLM machine-translation output beyond the intended target translation — language labels, verbose preambles, explanations, bilingual repetition, wrong-language text. Its claims are: (1) noise is prevalent and model-dependent (3.24%–99.73% across models and prompts, Table 1); (2) twelve recurring patterns fall into formatting and content categories (Table 2); (3) TransClean is the first benchmark for clean translation extraction, with 9,900 noisy/clean pairs; (4) noise detection is near-saturated while exact-match extraction is hard (best 54.07%/52.18%, Table 4); (5) LLM-based extraction generalizes to authentic noise better than span-based QE extraction. The task is deliberately post-hoc: extraction methods see only the noisy output, no reference, and no model retraining is assumed — a realistic constraint for proprietary deployments.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is translation noise in LLM translation outputs?
Any extra text an LLM adds beyond the translation itself — e.g., 'Here is the translation:' preambles, language labels like 'Chinese:', word-by-word explanations, bilingual repetition, or answers in the wrong language. It breaks MT evaluation pipelines that assume outputs contain only the translation.
What is the TransClean benchmark and how big is it?
A 9,900-instance benchmark of noisy/clean translation pairs released on GitHub: 8,800 synthetic instances where noise was injected into reference translations at empirically observed frequencies, plus 1,100 authentic noisy outputs from real LLM runs with silver (model-voted) clean labels, covering 22 language pairs.
How well do current methods remove translation noise?
Detecting whether an output is noisy is nearly solved (96-100% accuracy for the strongest evaluated methods), but exactly recovering the clean translation is not: the best LLM extractor reaches 54.07% on synthetic and 52.18% on authentic data, and a COMET-KIWI span-based method drops to 15.82% on authentic noise.
Which LLMs produce the noisiest translations?
Among the 12 open-weight models studied, gemma-3-27b-it was by far the noisiest (up to 99.73% noisy outputs under one prompt), while Qwen3 variants and the translation-specialized Tower-Plus-72B were cleanest at roughly 3-5%.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.