Made in Hungary: Comments on the performance of generative language models
M\'aty\'as Osv\'ath, Enik\H{o} H\'eja, No\'emi Ligeti-Nagy
cs.CL
Quick verdict
Hungarian LLM adaptations show contamination and capability loss; this critique is largely sound, though its Qwen3>Racka reversal partly stems from truncation.
Summary
This critical review examines three Hungarian generative language model initiatives - OTP (GPT-3-style bilingual pretraining), PULI/ChatPULI (continued pretraining plus SFT of Llama-3.1/Qwen2.5), and Racka-4B (LoRA-based adaptation of Qwen3-4B) - and argues their reported results are unreliable. The authors re-evaluate models on the HuGME benchmark under recommended inference settings, finding Qwen3-4B beats its Hungarian-adapted Racka-4B on 9 of 11 tasks, contradicting Csibi et al., and measure forgetting on MMLU-Redux, C-Eval, GPQA-Diamond, GSM8K, IFEval and HumanEval, showing capability loss in every tested adapted model, most severely Racka (HumanEval 91.95 to 29.88). They document contamination (ChatPULI fine-tuned on 7,344 HuLU prompts; OTP benchmarks built from its own pretraining corpus), protocol problems (Racka scored on the HuLU validation split with undisclosed script modifications; reasoning disabled in the OpenHuEval evaluation), 2020-era data pipelines lacking quality filtering and decontamination, missing ablations, absent Hungarian post-training, and uneven cost/carbon reporting (Racka's 1,290 kg CO2eq reconstructed within 3.5%). Limitations: several HuGME metrics reward degenerate outputs, Racka was truncated at 8,192 tokens (996.7 average unfinished reasoning traces on MMLU), OTP models could not be tested (weights unreleased), and no significance testing accompanies the 5-run means.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under recommended inference settings, Qwen3-4B outperforms its Hungarian adaptation Racka-4B on 9/11 HuGME tasks and all six capability benchmarks, contradicting Csibi et al. 2026.
- Train-test contamination is documented in ChatPULI (7,344 HuLU prompts in SFT data) and structurally present in OTP, whose hand-crafted benchmarks were built from its own pretraining corpus.
- All three Hungarian pipelines use a 2020-era curation stack lacking model-based quality filtering, decontamination, composition reporting and ablations, and no Hungarian preference-tuning data or RLHF/RLVR post-training exists.
- Continual pretraining causes measurable capability loss in every tested adapted model, most severely Racka-4B (HumanEval 91.95 to 29.88), despite LoRA's claimed forgetting mitigation.
- Carbon and cost reporting is uneven: Racka's 1,290 kg CO2eq is consistent with hyperscale PUE (reconstruction within 3.5%), while ChatPULI and OTP disclose no energy figures.
Strengths
Independent re-evaluation reverses a published claim
Under Qwen3's recommended sampling, Qwen3-4B beats Racka-4B on 9/11 HuGME tasks (Table 1) and on all six capability benchmarks (Table 3: GSM8K 94.27 vs 67.82; HumanEval 91.95 vs 29.88), with failure modes quantified (996.7 vs 315.4 unclosed reasoning traces on MMLU).
Concrete, checkable contamination evidence
ChatPULI's own paper reports 7,344 HuLU prompts in its SFT data; OTP's sentiment and topic-classification benchmarks were drawn from the authors' own pretraining corpus; Racka's HuLU numbers use the validation split, stated only in a table caption, with modified evaluation scripts left undescribed.
First forgetting measurements for these adaptations
Quantified cost and carbon reconstruction
Systemic diagnosis beyond individual papers
Weaknesses
Headline reversal partially confounded by token truncation
Degenerate-output-prone metrics used in the authors' own comparison
No statistical inference
OTP claims rest on textual analysis alone
Undisclosed conflict of interest
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does Racka-4B actually beat Qwen3-4B on Hungarian tasks?
According to this re-analysis, no. When both models run with Qwen's recommended sampling settings, Qwen3-4B scores higher than Racka-4B on 9 of 11 HuGME tasks and on all six capability benchmarks tested (e.g., GSM8K 94.27 vs 67.82; HumanEval 91.95 vs 29.88). Part of Racka's deficit comes from failing to finish its reasoning within the 8,192-token limit (about 997 unfinished answers per 1,000 on MMLU), so the original paper's superiority claim is not supported under these settings - though the truncation caveat means the exact gap size remains open.
What data contamination was found in the Hungarian language models?
The ChatPULI paper itself reports including 7,344 HuLU benchmark prompts in its fine-tuning data, which inflates benchmark scores even with rephrased items. The OTP models' hand-crafted sentiment and topic-classification tests were built from the authors' own pretraining corpus, creating a possible home advantage over competing systems. Racka-4B was evaluated on HuLU's public validation split rather than the official leaderboard test set, with modified evaluation scripts that were never described, making its reported numbers incomparable to published HuLU results.
How much did it cost and how much CO2 to train these Hungarian models?
The paper estimates Racka-4B used about 18,386 A100 GPU-hours (~8.08 MWh, ~1.24 tCO2eq, notional cost ~EUR 20,500); PULI Base about EUR 4,300 and PULI Trio-Q Base about EUR 9,900. The original Racka paper reported 1,290 kg CO2eq, which this reconstruction matches within 3.5% assuming an energy-efficient facility (PUE 1.1). OTP's energy use cannot be estimated because SambaNova never published the SN10 RDU's power draw, and ChatPULI reported no training duration or energy figures.
Who wrote this critique and is it independent?
It is by researchers at the ELTE Research Centre for Linguistics (NYTUD) in Budapest - the same institute behind the HuLU and HuGME benchmarks and co-authors of the PULI/ChatPULI models. They do criticize their own ChatPULI work, including its benchmark contamination, but readers should note that the main re-evaluation of the competing Racka model relies on the authors' own HuGME benchmark, and the paper contains no conflict-of-interest statement.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (score differences) is referenced but not analyzable from the extracted text; its interpretation relies on the prose description.
- Footnote URLs (HuLU leaderboard, Qwen documentation, evaluation scripts, Komondor pricing) were not verified.
- The accuracy of the critique's factual claims about the three evaluated papers could not be independently checked against those papers.
- Appendix A.2 concerns HuGME evaluations; token limits and prompts for the Table 3 lm-eval-harness runs are only partially specified in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.