Size Matters: Foundation Model for Czech HTML documents
Martin Dvo\v{r}\'ak, V\'it Tlusto\v{s}, Artyom Voronin, Martin Habrovec, Kate\v{r}ina Podlesn\'a, Barbora Ri\v{s}ov\'a, Josef Von\'a\v{s}ek
cs.CL
Quick verdict
Seznam.cz presents HTML-LM, a 154M-parameter ModernBERT-based encoder for Czech web pages that reportedly beats 8B models on five internal Czech tasks via HTML-aware training, distillation, and long context; claims are plausible but rest entirely on private benchmarks without significance testing.
Summary
The paper introduces HTML-LM, a 154M-parameter encoder (ModernBERT architecture, 22 layers, 8192-token context) trained from scratch on 100M web documents (53% Czech domains) with three objectives: masked language modeling, bag-of-words prediction from [CLS], and a teacher-weighted contrastive distillation (SoftCSE variant) from Qwen3-Embedding-8B and Seznam's internal SeLLMa 8B. An aggressive preprocessing pipeline retains under 5% of original tokens, and a custom 57K WordPiece tokenizer assigns dedicated tokens to the 100 most frequent HTML tags. Evaluated by training small frozen-backbone MLP heads on five proprietary Czech tasks (article type, Curlie category, adult content, product pages, web-spam regression), HTML-LM reaches a normalized aggregate metric (NMM) of 0.7001 versus 0.6571 for Qwen3-Embedding-8B and 0.6544 for OpenAI text-embeddings-3-small, while using roughly 1/50th the parameters. Ablations show NMM rises with context length up to 8192 tokens and that preserving HTML markup beats plain text (+0.0064 NMM). Main limitations: all benchmarks and training data are internal to Seznam.cz, no statistical significance or variance is reported, the NMM metric is non-standard, and the release is weights-only under CC BY-NC 4.0, limiting reproducibility and independent verification.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- HTML-LM (154M params) sets a new state of the art for classification and regression on Czech web documents, beating larger encoders (jina-embeddings-v3, OpenAI text-embeddings-3) and 8B LLMs (Qwen3-Embedding, SeLLMa) with NMM 0.7001 vs 0.6571 for the best competitor.
- Effective HTML document processing requires both a context window of at least 2048 tokens (ideally 8192) and explicit preservation of HTML structure in the input.
- A multi-objective pretraining recipe (MLM + bag-of-words + teacher-weighted contrastive distillation) on 100M web documents yields a universal document embedding that transfers to five distinct downstream tasks via a small frozen-backbone MLP head.
- Aggressive DOM-based preprocessing that retains fewer than 5% of original tokens preserves the information needed for downstream accuracy, making processing economical at production scale (thousands of documents per second).
Strengths
Strong efficiency-performance trade-off with production evidence
A 154M model outperforms Qwen3-Embedding-8B (NMM 0.7001 vs 0.6571, Table 2) while being ~50x smaller and is deployed in production at thousands of documents per second, grounding the efficiency claim in real deployment rather than speculation.
Careful tokenizer engineering with transparent comparison
Section 3.1/Table 1 compares seven tokenizers on document fit rates at 512/4096/8192 tokens plus vocabulary utilization; the dedicated top-100-tag vocabulary choice is empirically justified (top-50 worse, >100 no gain), and the tokenizer achieves the best coverage among practical competitors (83.7% at 4096 tokens).
Ablations isolate the two claimed design drivers
Sensible multi-task loss normalization
Weaknesses
Evaluation entirely on proprietary internal benchmarks
No statistical significance, variance, or dataset sizes reported
NMM metric is non-standard and baseline-relative
Potential pretraining/evaluation entanglement
Tokenizer claims partially undercut by own table
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is HTML-LM and who made it?
HTML-LM is a 154-million-parameter language model for representing HTML web documents as embeddings, built by the Czech search engine Seznam.cz on the ModernBERT architecture. It is trained on 100 million web pages (about half Czech) and released on Hugging Face under the CC BY-NC 4.0 license (non-commercial use).
How does HTML-LM perform compared to larger models?
According to the paper, on five internal Czech web-page tasks (article type, directory category, adult-content, product-page, and web-spam detection) HTML-LM scores a normalized aggregate of 0.7001 versus 0.6571 for Qwen3-Embedding-8B and 0.6544 for OpenAI's text-embeddings-3-small — despite being roughly 50 times smaller than the 8B model. These results come from the authors' own benchmarks and have not been independently verified.
Why preserve HTML tags instead of using plain text?
The authors ran a controlled experiment (same architecture and training, only the input differs) showing that keeping HTML markup improves their aggregate metric by 0.0064 — an effect they estimate as comparable to doubling model size from 75M to 154M parameters. Tags convey structure (navigation, forms, media) that plain text loses.
Can I use HTML-LM commercially?
No. The model weights are released under CC BY-NC 4.0, which permits attribution-compliant non-commercial use only. The training data, preprocessing code, and benchmark datasets described in the paper are not released, so full reproduction is not currently possible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text appears complete (all sections, tables 1–4, and references present), but figures, if any existed beyond tables, could not be inspected.
- The Version of Record is published in TSD 2026 (Springer LNCS); this review covers the arXiv v1 preprint text only.
- All evaluation is on proprietary Seznam.cz benchmarks, so the review can assess internal consistency but not external validity of the SOTA claim.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.