Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training
Lukas Borggren, Jenny Kunz, Marco Kuhlmann
cs.CL · cs.AI · cs.LG
Quick verdict
Credible CPT study for Swedish journalism: 8.5B-token corpus, new 6-task benchmark; replay+LoRA key, but proprietary data/models limit reproducibility.
Summary
This paper reports the first comprehensive study of continued pre-training (CPT) for adapting LLMs to Swedish journalism. From 21.7 million articles published by Bonnier News (1991–2026), the authors curate BonCorpus — 14.6 million articles, 8.5 billion tokens — via cleaning, filtering, and MinHash-LSH plus Levenshtein deduplication, and build BonEval, a six-task benchmark covering headline, lead, and summary generation, topic classification, salient-entity recognition, and multiple-choice quiz. They adapt Ministral 3 3B and 8B base models with full fine-tuning (FFT), LoRA, and LLaMA Pro, diluting BonCorpus with 10% each of code, English, and Swedish replay data. At 3B, BonCorpus alone degrades performance (36.98 vs base 39.73) while the full BCES mixture scores 41.48; LoRA plus an instruct vector is best (42.27). The 8B BonLM mirrors this: BonEval average 46.02 for LoRA+IV versus 41.59 for base, with gains in generative tasks and factual Quiz knowledge but slight declines on discriminative Topic and Entity; on Swedish EuroEval both variants fall below base (57.39/45.86 vs 58.73), driven by steep MMLU and HellaSwag regressions. Instruct vectors help LoRA models yet hurt FFT models, partly attributed to smaller parameter drift. Key limitations: no data, code, or models released; a single model family; unknown overlap between base-model pre-training data and the benchmark; automatic metrics only.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Continued pre-training on curated Swedish news data improves in-domain performance only when paired with experience replay (3B: BonCorpus-only 36.98 vs BCES 41.48 vs base 39.73).
- CPT consistently improves generative tasks and factual Quiz knowledge at both model scales, but slightly degrades discriminative Topic and Entity performance.
- Instruct vectors improve LoRA-trained models but impair full-fine-tuned models; LoRA+IV achieves the best BonEval averages (3B 42.27; 8B 46.02 vs base 41.59).
- The existing Swedish EuroEval suite fails to reflect in-domain gains, demonstrating that targeted language- and domain-specific evaluation is necessary to guide adaptation.
Strengths
Novel dual-gap coverage with industrial-scale data
First comprehensive CPT study for journalism in Swedish, built on 21.7M Bonnier News articles reduced to a 14.6M-article / 8.5B-token corpus (§3.1), with the curation pipeline fully documented (Appendix A: fastText LID > 0.9, MinHash LSH at 0.25, Levenshtein < 0.2 dedup).
Contamination-aware benchmark construction
BonEval examples come only from 2022+ articles that are excluded from training, with task-specific filters (length caps, entity-label sanitisation, removal of time-dependent quiz items; §3.2, Appendix B, Table 5).
Systematic ablations with a mechanistic probe
New benchmark validated against independent tasks
Negative results reported honestly
Weaknesses
Recipe selection on a statistically unestablished margin
Unquantified SweDN contamination
Confounded attribution of factual-knowledge gains
Under-explained IV mechanism, over-generalised conclusion
Zero external verifiability
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper attacks two gaps at once: journalism as a rarely treated target domain for continued pre-training (CPT), and Swedish as the last major Scandinavian language without a methodologically detailed CPT study. Four central claims appear in §1 and are revisited in §6: (1) CPT on a curated news corpus helps only with experience replay; (2) gains appear in generation and factual knowledge but not discriminative tasks; (3) instruct vectors (IV) are compatible with LoRA-trained but not full-fine-tuned (FFT) models; (4) the existing Swedish EuroEval suite misses in-domain gains, so targeted evaluation is essential. All artifacts (BonCorpus, BonEval, BonLM) are proprietary and withheld (Limitations), so every claim below rests on the paper's internal reporting.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is BonEval and what tasks does it include?
BonEval is a Swedish journalism benchmark built by the authors from Bonnier News articles published 2022 or later and excluded from training data. It contains six tasks: headline generation, lead-paragraph generation, bullet-point summarisation, 19-way topic classification, salient-entity recognition, and multiple-choice quiz questions, evaluated EuroEval-style with few-shot prompting using CHRF3++, Matthews correlation, and micro-F1.
Does continued pre-training on news data hurt a model's general abilities?
In this study, yes, on general Swedish benchmarks: the 8B adapted models score 57.39 (LoRA) and 45.86 (FFT) versus 58.73 for their base model on EuroEval, mainly because MMLU drops from 58.10 to 53.34/30.62 and HellaSwag from 43.68 to 36.82/17.66. Natively Swedish tasks were roughly preserved under LoRA, and in-domain BonEval performance improved.
Can I download BonCorpus, BonEval, or BonLM?
No. The authors state that for business and legal reasons they cannot release the proprietary news data, code, or models; BonLM is intended for internal deployment within Bonnier News' editorial tools. This makes the study a blueprint rather than a reproducible result.
What is an instruct vector in this paper?
It is the parameter difference between a base model and its instruction-tuned descendant, computed by subtracting the base from the instruct model. The authors add this vector to their continued-pre-trained models: it improved LoRA-trained models (e.g., 3B BonEval 41.32 → 42.27) but degraded full-fine-tuned models (41.48 → 39.07), a difference they link to smaller parameter drift in LoRA training.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction is complete from the abstract through Appendix E (Table 13 with confidence intervals); no truncation detected at the end of the provided text.
- Underline/bold 'section-best' markers in the tables are lost in plain-text extraction; best-model designations in this review were verified numerically from the reported values.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.