Polish ModernBERT: The Long and Short of Polish Language Understanding
Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata, Małgorzata Grębowiec
cs.CL
Quick verdict
Polish ModernBERT + new LongContext benchmark show real gains (9.68 pts Base-8K) but a translated benchmark and no significance tests temper the claim.
Summary
The paper introduces Polish ModernBERT, a family of four Polish encoder-only models (Base 149M / Large 475M, each in 512-token and 8K-context variants) built by adapting the ModernBERT recipe through four staged 512-token pretraining stages selected via KLEJ validation, followed by 8K context extension that raises the global RoPE base to 160,000; training used about 44.5B Polish tokens on 8xGH200. The authors also release LongContext, a five-task Polish benchmark (SCOTUS topic and decision direction, ECtHR-PL violation prediction, BookSummary factual consistency) machine-translated from English with GLM-4.6 and spot-checked by one annotator on 50 examples per task. Across 30 Polish NLU tasks with five fine-tuning seeds, the family achieves the best aggregate results among evaluated Polish encoders (83.99 Base-8K, 85.11 Large-8K overall averages). On LongContext, Base-8K improves over Polish RoBERTa-8K from 67.47 to 77.15 with 22% fewer parameters; Large-8K improves from 75.88 to 78.49. Inference latency and peak memory favor ModernBERT in both settings. Base-8K also tops PIRB retrieval among encoders under 300M parameters (55.36 NDCG@10), though RoBERTa-8K-Large slightly leads at Large scale (58.42 vs 58.16). Main limitations: translated-benchmark validation rests on single-annotator spot-checks, possible cross-lingual contamination from public English sources, non-controlled model comparisons (including different tokenizers per scale), and no statistical significance tests.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Polish ModernBERT achieves the best aggregate performance among evaluated Polish encoders across 30 tasks, with overall averages of 83.99 (Base-8K) and 85.11 (Large-8K).
- The 8K variants improve LongContext averages over matched Polish RoBERTa-8K baselines from 67.47 to 77.15 (Base) and 75.88 to 78.49 (Large).
- Base-8K delivers the long-context gain with 22% fewer parameters (149M vs 190M) plus lower latency and peak memory in both 512-token and 8K settings.
- Polish ModernBERT-8K-Base achieves the best PIRB retrieval result among evaluated encoders below 300M parameters (55.36 vs 54.10 NDCG@10).
- The staged recipe-selection procedure (WWM, corpus refinement, annealing, zero attention dropout) outperforms direct transfer of the original ModernBERT recipe (87.00 vs 83.73 on the 15-task validation suite).
Strengths
Broad, multi-seed evaluation with matched baselines
30 tasks across four groups (9 KLEJ, 6 FinBench, 10 Other, 5 LongContext), five fine-tuning seeds each, sample SDs for matched Polish comparisons (Table 15), plus multilingual EuroBERT/mmBERT/XLM-R baselines and a 41-dataset retrieval evaluation (PIRB, Table 5).
Unusually transparent training documentation with a negative control
Appendix B reports every staged recipe decision (Table 8: cosine vs WSD, WWM@0.25, curated corpus, zero attention dropout 86.62→87.00) and the direct-transfer ModernBERT baseline that failed to improve (83.73 vs 87.00 on 15-task validation, Appendix B.1); corpus cleaning and dedup are fully specified (Appendix A.2).
Large and internally consistent Base-scale long-context effect
Carefully scoped claims and honest limitations
Rigorous efficiency measurement protocol
Weaknesses
Benchmark validity rests on very thin validation
No statistical significance testing
Non-controlled comparisons with tokenizer confound
Untested cross-lingual contamination asymmetry
Recipe selection metric overlaps the test suite
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper introduces Polish ModernBERT — four Polish encoder-only models spanning Base (149M) and Large (475M) scales, each in 512-token and 8,192-token context variants (Table 1) — plus LongContext, a new five-task Polish benchmark for document-level understanding (SCOTUS-DOM/DEC, ECtHR-PL-AVA/VA, BookSummary). Central claims: best aggregate performance among evaluated Polish encoders across 30 tasks (83.99 Base-8K and 85.11 Large-8K overall averages, Tables 3–4, five fine-tuning seeds each); large long-context gains over matched Polish RoBERTa-8K baselines (67.47→77.15 Base, 75.88→78.49 Large); lower latency and peak memory (Appendix D); and the best sub-300M PIRB retrieval result (55.36 NDCG@10, Table 5). Claims are consistently scoped — notably, retrieval superiority is claimed only below 300M parameters, correctly, because RoBERTa-8K-Large wins at Large scale (58.42 vs 58.16).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Polish ModernBERT?
A family of four open Polish encoder-only language models — Base (149M parameters) and Large (475M) scales, each in a 512-token and an 8,192-token context version — built on the ModernBERT architecture and pretrained on about 44.5 billion Polish tokens, with a long-context continuation stage that raises the global RoPE base from 10,000 to 160,000. The checkpoints are slated for release on Hugging Face.
How much better is Polish ModernBERT than Polish RoBERTa?
Across the paper's 30 Polish NLU tasks its overall averages are the highest among the evaluated Polish encoders: 83.99 for Base-8K and 85.11 for Large-8K. The largest gap is on the five long-context tasks, where Base-8K scores 77.15 versus 67.47 for Polish RoBERTa-8K (about +9.7 points) while using 22% fewer parameters. On standard 512-token short-text tasks the advantage is smaller (roughly +1.2 points at Base scale), and at Large scale the two families are close.
Is the new LongContext benchmark reliable?
It is a five-task Polish suite (SCOTUS topic and decision direction, ECtHR-PL human-rights violation prediction, BookSummary factual consistency) machine-translated from public English datasets with GLM-4.6. Quality control consisted of one annotator spot-checking 50 examples per task plus automatic filters for degenerate outputs. The authors themselves call this a quality check rather than full human validation, so translation artifacts and occasional label noise may remain.
Does Polish ModernBERT work well for retrieval?
Yes at Base scale: after a shared contrastive fine-tuning protocol, Base-8K scores 55.36 mean NDCG@10 on the 41-dataset PIRB Polish retrieval benchmark, the best among evaluated encoders under 300M parameters. At Large scale it is essentially tied with Polish RoBERTa-8K-Large, which slightly wins (58.42 vs 58.16).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are present in the extraction only as axis labels, tick values, and captions; their plotted content (e.g. per-bucket curves, quality-efficiency frontier) could not be visually assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.