TACTICS: Taxonomy-Aware Intelligent Corpus Sampling for Machine Translation
Prasanth Bathala, Anubhav Shrimal, Sukhdeep Singh Kharbhanda, Pradyumna Lanka, Rohit Dhaipule
cs.CL · cs.LG
Quick verdict
TACTICS recasts MT evaluation sampling as taxonomy coverage optimization; well-motivated and unusually transparent about its bias, but with cross-table numeric inconsistencies, classifier-quality and circularity concerns.
Summary
TACTICS argues that random evaluation samples faithfully reproduce a corpus but under-represent the rare phenomena that discriminate MT systems. It induces a hierarchical taxonomy from locale style guides via an LLM extract-audit-refine loop, labels every segment (regex rules for deterministic categories, an LLM classifier for semantic ones), and greedily selects B=5,000 segments in three levels: coverage seeds (every category represented), CF-IDF-scored document-level coherence, and chi-squared distributional fidelity to corpus category composition and edit rate. On four proprietary localization directions, TACTICS beats random, DSIR, and ClusterCLIP on rare-category coverage (average Cov20 56.3 to 73.3) and coherence while keeping Pearson r 0.949-0.982 with the corpus distribution. Where a real system gap exists it recovers the true ranking from fewer segments (paired |t| ratio 1.19-1.63 on en-hi; 100% recovery on ja-zh at 1600-2400 segments), and on the en-fr null control it raises no false verdict. This diagnostic power is bought with deliberately biased corpus-average estimates (DEFF up to 66.5), requiring pairing with a random sample. Limitations: fully proprietary data, no code release, classifier agreement only kappa=0.51, and unexplained inconsistencies between Table 2 and Tables 6-7/Appendix E for en-fr.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Selecting evaluation segments against a taxonomy induced from locale style guides raises rare-category coverage at a fixed budget (average Cov20 from 56.3 with random to 73.3 with TACTICS) relative to random, DSIR, and ClusterCLIP.
- TACTICS recovers the true system ranking from fewer segments than random wherever a real quality gap exists (paired |t| ratio 1.19-1.63 on en->hi; 100% recovery on ja->zh at 1600-2400 segments vs. 62-93% for random).
- This diagnostic power is an intentional trade: TACTICS is a deliberately biased estimator of corpus-average quality (bias^2/MSE near 100%), so it should be paired with a separate random sample for aggregate estimation.
- On the one null-control direction (en->fr), TACTICS raises no false verdict, supporting the claim that it never signals a difference where none exists.
- Lexical and embedding-based selectors (DSIR, ClusterCLIP) fail to capture prescriptive localization phenomena and are unstable across directions, with distributional MSE varying by orders of magnitude (Table 2).
Strengths
Well-motivated reframing of evaluation-set construction
The 'representative is not diagnostic' thesis (Sections 1, 5) is clearly argued and positioned against WMT test-set design and FLORES-style sampling (Section 2.1), and against output-dependent selection methods (tinyBenchmarks, anchor points, Zouhar et al. 2025) that do not transfer to new systems.
Novel grounding of sampling strata in prescriptive style guides
Stage 1 (Section 3.2.1, Appendix I) induces deterministic vs. semantic categories from locale style guides; the fr-FR spot-check table (Table 10) concretely maps rules like date and currency formats to regex or LLM categories, making the taxonomy auditable rather than an embedding-artifact.
Unusually transparent reporting of the coverage-fidelity trade-off
Includes a genuine null control and an ablation-resolved efficiency experiment
Multi-layer empirical support
Weaknesses
Internal numeric inconsistencies across tables
Potential circularity between sampler and evaluated system
Taxonomy label quality is only moderate and unvalidated against humans
The 'never signals a difference where none exists' guarantee rests on one null
Downstream results lack uncertainty quantification and system details are vague
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TACTICS and how does it choose which sentences to use for MT evaluation?
TACTICS (Taxonomy-Aware Coverage-opTimized Intelligent Corpus Sampling) is a method for building compact machine-translation evaluation sets. It first derives a hierarchy of quality phenomena (terminology, date/currency formats, tone, etc.) from locale style guides using LLM agents, labels every corpus segment against that taxonomy using rules plus an LLM classifier, and then greedily selects about 5,000 segments that jointly cover every category (including rare ones), keep documents intact for context, and stay reasonably close to the full corpus's category and difficulty distribution.
Is TACTICS better than random sampling for MT evaluation?
For detecting differences between systems, yes, in the paper's experiments: TACTICS-covered samples recovered true system rankings from fewer segments (e.g., on ja-zh it reached significance at 1,600-2,400 segments where random needed more), and it covered far more rare phenomena (average 73.3 vs. 56.3 categories with at least 20 sampled segments). For estimating a system's average quality over the whole corpus, random sampling is better — TACTICS is intentionally biased, and the authors recommend using both together.
What trade-off does TACTICS make compared with a representative random sample?
TACTICS guarantees coverage of rare style-guide phenomena but deliberately departs from the corpus distribution: it has higher distributional error (Pearson r 0.949-0.982 instead of ~1.0) and much higher error when estimating corpus-average system scores. In exchange, a fixed evaluation budget yields far more diagnostic signal about where systems differ.
Can I reproduce TACTICS or apply it to my own corpus?
Not directly from this paper: the evaluation corpus and style guides are proprietary and no code is released; the authors promise a public companion study on open benchmarks and style guides. Conceptually, applying TACTICS requires locale style guides, an LLM-based taxonomy induction loop, a segment classifier, and a constrained greedy sampler — all described procedurally in the paper but without released implementations.
Which languages and systems were evaluated with TACTICS?
Four localization translation directions — English to French, English to Hindi, Japanese to English, and Japanese to Chinese — on an internal Amazon post-editing corpus of 57K-84K segments each. The compared systems were a dedicated neural MT system (NMTbase) and an LLM translation pipeline (LLMMT, a Gemma-based model prompted with summarized style guides), with a human MQM audit as ground truth for ranking recovery.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (Figure 1-5, especially the sample-efficiency curves in Figure 3 and taxonomy diagrams) were not available as images; analysis relies on captions and surrounding text.
- Several numeric inconsistencies were found across tables (en->fr in Table 2 vs. Appendix E/Table 6; 'en->ja' row label in Table 6); these may stem from text-extraction or paper-version issues but are unresolvable from the provided text, which lowers confidence.
- Appendices F, G, H and the limitations section were included in the extraction, but no external materials (style guides, prompts beyond those quoted, code) could be consulted, and no network access was used.
- The public benchmark used for the taxonomy-consistency checks (Section 4.3) is not named in the extracted text, so that part of the validation could not be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.