TeochewBench: A Human-Reviewed Benchmark for Teochew Hanzi Translation
cs.CL
Quick verdict
Well-documented 300-item Teochew translation benchmark with a clever Hanzi-copy control, but single-reviewer labels and no stated data release limit impact.
Summary
TeochewBench is a 300-item evaluation set for translating written Teochew (in Chinese characters) into Mandarin Chinese and English, spanning five content categories from basic vocabulary to idiomatic and culturally specific expressions. Candidates were compiled with reference to public dictionaries (e.g., PUJDICT) and machine-assisted organization, then reviewed item-by-item and revised by a native Teochew speaker, with two additional speakers verifying an unspecified subset. The paper evaluates 11 official post-trained checkpoints from five families (7 of 11 Qwen) under greedy decoding with fixed prompts, plus two Qwen base checkpoints for diagnostics, yielding 7,800 predictions. Scoring uses a locally implemented chrF-style metric, multiset character F1, and exact match with max-over-references, plus a Hanzi-copy control that quantifies source-overlap inflation in the Mandarin direction (67.69 chrF-style on low-specificity items, exact match 0.524). Mean scores fall from 69.25 (low specificity) to 27.52 (high), with the high tier showing the smallest cross-model dispersion; Qwen3.5-27B leads at 60.63. Main limitations: no stated public release of data, code, or predictions; all specificity labels come from one reviewer without inter-annotator agreement; 240 of 300 items have a single English reference; and the small high tier (33 items) plus a specificity-category association (Cramer's V = 0.292) complicate attribution of tier effects.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- TeochewBench provides a 300-item, human-reviewed benchmark with Mandarin and English references for evaluating LLM translation from written Teochew.
- Mean chrF-style scores fall from 69.25 on low-specificity items to 27.52 on high-specificity items across 11 post-trained checkpoints, and the high tier forms a shared low-scoring region with the smallest cross-model dispersion.
- Surface Hanzi overlap substantially inflates automatic scores in the Mandarin direction: the Hanzi-copy control earns 67.69 chrF-style and 0.524 exact match on low-specificity items.
- Qwen3.5-27B achieves the best overall chrF-style score (60.63) among 11 evaluated checkpoints, with non-Qwen models Gemma-3-27B-IT and GLM-4-32B-0414 ranking third and fourth.
- Post-trained checkpoints materially outperform their base counterpart on the same protocol (Qwen3.5-4B-Base 47.98 vs Qwen3.5-4B 51.86), while Qwen3-4B-Base degenerates under instruction-based prompting (3.28).
Strengths
Copy control directly addresses the shared-script confound
Section 4.3/5.3 quantifies how much automatic scores in the Mandarin direction reflect copying rather than translation: the unchanged-source control scores 67.69 chrF-style and 0.524 exact match on low-specificity items (versus the best model's 60.21), and item-level copy chrF correlates with specificity at Spearman rho = -0.634. Requiring this control in Sinitic evaluations is a transferable methodological contribution.
Unusually transparent inference and scoring protocol
Appendix A specifies exact system/user prompts, output-processing rules (whitespace, think-tag stripping, label and quote removal), vLLM 0.22.1 and Transformers 5.11.0, bfloat16, greedy decoding, seed 0, bootstrap seed 20260825, and per-checkpoint deployment logging - sufficient to re-run the evaluation given the data.
Honest scoping of what the numbers do and do not show
Stratified analysis with a falsifiable rubric
Cross-family signal despite Qwen-heavy model set
Weaknesses
No stated release of the dataset, code, or predictions
All stratified analyses rest on single-annotator labels
Convenience model pool with heavy family imbalance
Metric validity is asserted, not demonstrated
Thin and unbalanced tail tiers weaken headline claims
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TeochewBench?
TeochewBench is a benchmark of 300 human-reviewed Teochew expressions written in Chinese characters, used to test how well large language models translate Teochew into Mandarin Chinese and English. Items span five categories, from basic vocabulary to idioms and culturally specific expressions.
Which AI model translated Teochew best in TeochewBench?
Qwen3.5-27B scored highest with an overall chrF-style score of 60.63, ahead of Qwen2.5-72B-Instruct (56.61), Gemma-3-27B-IT (56.36), and GLM-4-32B-0414 (55.82). All models scored much lower on Teochew-specific idioms and culturally loaded expressions, where the mean dropped to 27.52.
Why does the paper include a Hanzi-copy control?
Written Teochew shares many characters with Mandarin Chinese, so a system that simply copies the source text earns a chrF-style score of 67.69 on low-specificity items - higher than the best real model. The control shows how much automatic scores can reflect character overlap rather than genuine translation, and the authors argue it should always be reported for Mandarin-direction evaluation.
Is the TeochewBench dataset publicly available?
The reviewed paper text contains no repository, license, or download link for the dataset, model predictions, or scoring code; it mentions documenting future revisions in a dataset version history but provides no availability statement, which is a notable gap for a benchmark paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract in the extracted text is truncated mid-sentence; the missing clause is consistent with body results (low-tier mean 69.25 falling to 27.52 for high).
- No figures appear in the extraction; all tables were rendered inline as text and appear complete, but appendix formatting may differ from the PDF.
- The dataset, labels, and model outputs could not be independently inspected; this review relies entirely on the authors' descriptions.
- Model names and versions (e.g., Qwen3.5, Seed-OSS-36B, vLLM 0.22.1) could not be verified against public releases from the text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.