Do Cantonese-Adapted Language Models Better Predict Cantonese Reading? A Cross-Model Eye-Tracking Evaluation
Ziqi Zhang, Emmanuele Chersoni, Mohammad Momenian
cs.CL
Quick verdict
Claims heavier Cantonese adaptation predicts reading better; careful, honest, but <2% gains, confounded adaptation contrasts, and no significance testing.
Summary
This empirical study asks whether Cantonese-specific language-model training improves the prediction of Cantonese reading behavior. The authors extract four information-theoretic predictors - lexical surprisal, POS surprisal, entropy before the target, and entropy reduction - from two within-family model pairs: CKIP GPT-2 Tiny versus its lightly Cantonese-adapted derivative JED351, and Qwen2.5-7B versus CantoneseLLM-7B, which received extensive Cantonese continued pretraining and instruction tuning. Predictors are evaluated with CatBoost regression on the Cantonese portion of the MCFIX eye-tracking corpus (about 10,011 word observations from 30 native speakers reading a Cantonese translation of The Little Prince under natural and task-specific reading), against a conventional no-LLM baseline, using five-fold cross-validation repeated over five seeds. CantoneseLLM-7B yields the largest gains for lexical surprisal and the joint four-metric model (e.g., a 1.251 ms improvement in total fixation duration), with Qwen2.5-7B second; the lightly adapted JED351 shows no advantage over its CKIP base, while entropy reduction favors CKIP. All effects are modest, below 2% of baseline MAE after normalization. Main limitations: the adaptation contrasts confound vocabulary, continued-pretraining, and instruction-tuning changes; the five-seed robustness counts carry no significance tests; and the data cover a single text and participant sample.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- All four LLM-derived information-theoretic metrics improve prediction of Cantonese fixation durations beyond conventional lexical, positional, POS, and dependency predictors, but the largest joint gain stays below 2% of baseline MAE after normalization.
- Extensively adapted CantoneseLLM-7B consistently outperforms its base Qwen2.5-7B on lexical surprisal and the joint four-metric model, with positive contrasts across all five split seeds (e.g., 0.132/0.066/0.430 ms on FFD/SFD/TFD).
- Lightly adapted JED351 shows no overall predictive advantage over its CKIP base model, indicating light Cantonese adaptation is insufficient.
- Entropy reduction breaks the ranking and favors the unadapted CKIP base (e.g., 0.362 ms on TFD versus roughly 0.16 ms for the other three models), dissociating target-word predictability from uncertainty-updating fit.
- Task-specific reading (TSR) yields larger four-metric FFD improvements than natural reading (NR) for all four models, while SFD/TFD task patterns are model-specific.
Strengths
Within-family design controls architecture and scale
Both contrasts hold model family fixed (Table 1): JED351 descends from CKIP GPT-2 Tiny and CantoneseLLM-7B from Qwen2.5-7B, so the adaptation comparison is not immediately confounded by architecture within each pair.
Complementary metric battery with shared construction
Four IT measures (lexical surprisal, POS surprisal, entropy before, entropy reduction) cover pre-target uncertainty, target expectation, and updating; POS surprisal is built on a common Cifu/PyCantonese candidate inventory so it is comparable across models (Section 4.1).
Robustness discipline uncommon in this genre
Honest effect-size calibration
Underexplored evaluation target
Weaknesses
Adaptation contrasts are multiply confounded
No inferential statistics anywhere
Entropy measures not cross-model comparable
Tokenization-boundary artifact disclosed but not analyzed
Narrow stimulus and sample; code not yet available
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv:2609.02163v2) asks whether Cantonese-specific training improves the psycholinguistic alignment of language models, operationalized as the ability of information-theoretic (IT) predictors to explain eye-tracking measures. Two within-family contrasts are studied: CKIP GPT-2 Tiny versus its Cantonese-adapted derivative JED351 (vocabulary patching plus about 50 MB of Cantonese Wikipedia fine-tuning), and Qwen2.5-7B versus CantoneseLLM-7B (continued pretraining on HK news/web text plus instruction tuning on 75,000 pairs). Four predictors per model - lexical surprisal (Eq. 1), POS surprisal (Eq. 2), entropy before the target (Eq. 3), and next-token entropy reduction (Eq. 4) - are added to CatBoost regressions of FFD/SFD/TFD on the Cantonese MCFIX corpus (Section 3.1; 10,011 rows = 5,007 NR + 5,004 TSR after quantifying an 86-row intersection loss). The central claims are deliberately calibrated: IT metrics help incrementally (<2% baseline MAE), extensive adaptation is associated with stronger surprisal-based fit, light adaptation is not, and rankings depend on the measure, with entropy reduction favoring CKIP.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did Cantonese-adapted language models predict Cantonese reading better?
Partially. In this study the heavily adapted CantoneseLLM-7B gave the largest gains on lexical surprisal and the joint four-metric model (for example a 1.251 ms improvement in total fixation duration), but the lightly adapted JED351 did not beat its base model CKIP, all gains were under 2% of baseline error, and entropy reduction actually favored the unadapted CKIP.
What data does the study use?
The Cantonese portion of the MCFIX eye-tracking corpus: about 10,011 word-level observations from 30 native Cantonese speakers reading a Cantonese translation of The Little Prince under two conditions - natural reading and a task-specific information-search reading - with first, second, and total fixation durations as outcomes.
Which information-theoretic measures were compared across models?
Four measures computed from each language model for every word: lexical surprisal (how unexpected the word is), POS surprisal (how unexpected its grammatical category is), entropy before the word (the model's uncertainty about the next token), and entropy reduction (how much observing the word sharpens the distribution over what comes next).
Does more Cantonese training always mean better psycholinguistic fit?
Not in this paper: the benefit depended on adaptation depth and on the measure. Extensive adaptation (continued pretraining plus instruction tuning) helped surprisal-based fit, light adaptation (vocabulary patching plus small-corpus fine-tuning) did not, and the uncertainty-updating measure favored a base model with no Cantonese adaptation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 appear in the extraction as unformatted numeric grids with ambiguous cell mapping; every figure value cited in this review was taken from the prose or the appendix tables rather than parsed figure layouts.
- The full-precision metric-correlation coefficients of Figure 6 are stated to live in a supplementary CSV file that is not part of the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.