Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting
Fırat Öncel, Salman Hussain Ali, Mirco Ravanelli, Cem Subakan, Çağatay Yıldız
cs.CL
Quick verdict
Multi-family empirical study: time-incremental CPT adds post-cutoff knowledge at near-zero forgetting cost; credible, but gains are net, not absolute.
Summary
This paper studies continued pretraining (CPT) as a mechanism for updating open-weight LLMs with knowledge that postdates their training cutoff, using FineWeb-Edu dumps drawn strictly from after each model's cutoff rather than the disjoint data streams assumed by classic continual learning. Six models spanning three families and four scales (OLMo2-1B/7B, Llama-3.2-1B/3B, Llama-3.1-8B, Gemma-3-1B) are trained on two slices (score≥4, 6B tokens; score≥3.5, 40B tokens) and evaluated on Daily Oracle news QA split at each cutoff plus thirteen downstream tasks under the OLMES protocol. After-cutoff Macro-F1 improves heterogeneously (OLMo2-7B +0.09, Gemma-3-1B +0.07, Llama family less), with gains tracking tokens-per-parameter (ρ=−0.63, not significant at n=6); five of six models also improve on pre-cutoff questions (mean +0.027, p=0.034). Macro-average downstream accuracy stays within 0.01 of base (TOST 90% CI [−0.001,+0.007]), though sample-level analysis shows 9–16% pre-cutoff True/False flips and a worst-case CommonsenseQA drop the text puts at 0.06. The proposed recipe: data quality dominates quantity; knowledge and capability learning-rate optima differ roughly tenfold (validated in depth only on Gemma-3-1B); LoRA at rank≥128 matches full CPT; gains survive SFT while DPO effects are family-dependent (two models only). Main limitations: a single primary temporal benchmark, learning-rate selection on the same surface used for headline claims, and limited sweep depth.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Continued pretraining strictly on post-cutoff web data acquires new knowledge heterogeneously across families (OLMo2-7B +0.09, Gemma3-1B +0.07 after-cutoff Macro-F1) while improving pre-cutoff recall for five of six models, so catastrophic forgetting is not the dominant failure mode in the overlapping-crawl regime.
- The capability cost is statistically negligible: macro-average accuracy across thirteen tasks stays within 0.01 of base for every model, with TOST equivalence (90% CI [−0.001,+0.007]).
- At the sample level acquisition and forgetting are tightly coupled (forg ≈ 0.60·acq + 0.025, r=0.68 over 191 configurations), so 'no forgetting' means net-positive, not zero answer flipping.
- The operational recipe: a curated 6B-token score≥4 slice matches a 40B score≥3.5 slice; knowledge-acquisition and capability LR optima differ ~10×; LoRA at rank≥128 matches full CPT; and CPT gains survive SFT, with DPO's effect family-dependent.
Strengths
Realistic deployment regime with candid causal accounting
Section 2 quantifies cross-snapshot overlap (~30–40% novel URLs per dump) and the Discussion explicitly concedes the absence of forgetting is 'unsurprising' given implicit replay via recrawled URLs — the paper argues against its own most convenient explanation rather than hiding it.
Sample-level acquisition/forgetting decomposition
Section 3.2 (Table 2, Figure 8) fits forg = 0.60·acq + 0.025 across 191 configurations (r=0.68), admits even plastic models flip 9–16% of before-cutoff TF answers, and reframes 'no forgetting' as net-positive — an unusually honest treatment of the headline claim.
Anti-artifact controls and fourth-family replication
Above-norm statistical discipline
Actionable, replicated recipe
Weaknesses
All acquisition evidence flows through one auto-generated benchmark
Winner's-curse risk in 'best-performing setting' reporting
Abstract outruns the saturation evidence
LR recipe claim is single-model
Internal tension between claimed and tabulated worst-case cost
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether time-incremental continued pretraining (CPT) — training open-weight LLMs strictly on web data postdating their knowledge cutoff — acquires new knowledge without the catastrophic forgetting predicted by the continual-learning literature. The reframing is the core contribution: web crawls naturally overlap across snapshots (Section 2 reports ~30–40% novel URLs per dump), so the disjoint-stream assumption underlying prior forgetting results does not describe deployment reality. Claims are organized around four questions: acquisition (heterogeneous but positive; five of six models also improve pre-cutoff), cost (≤0.01 macro-average across thirteen tasks), recipe (quality > quantity; ~10× LR separation; LoRA rank≥128 suffices), and deployment (gains survive SFT; DPO family-dependent). A fifth, fainter claim — that plasticity tracks tokens-per-parameter saturation — is promoted to the abstract but is the weakest-supported result in the paper.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does continued pretraining on new web data cause catastrophic forgetting in LLMs?
Not in this paper's realistic setting. Training six open-weight models (1B–8B, three families) strictly on post-cutoff FineWeb-Edu dumps kept macro-average accuracy on thirteen downstream tasks within 0.01 of the base model, and five of six models actually improved on pre-cutoff factual questions. The caveat: at the individual-answer level 9–16% of pre-cutoff True/False answers still flipped; gains are net, not absolute. The authors argue catastrophic forgetting is mainly a property of disjoint-data regimes, and real web crawls naturally replay old content via recrawled URLs.
What learning rate should I use for continued pretraining of an LLM?
The paper recommends a moderate rate around 3e-5, tuned against a held-out temporal slice, when factual currency is the goal. Its nine-point sweep on Gemma-3-1B found knowledge acquisition peaks near 3e-5 and collapses at higher rates, while downstream benchmark accuracy keeps improving at rates that already damage factual recall — the two optima differ by roughly an order of magnitude. Important caveat: this analysis was done in depth on one 1B model only, so treat the number as a starting point, not a universal constant.
Can LoRA be used for continued pretraining and knowledge updates?
Yes, at sufficient rank. Rank 128–256 LoRA applied to attention and MLP projections matched full continued pretraining on new-knowledge acquisition for Gemma-3-1B and nearly matched it on Llama-3.2-3B (after-cutoff Macro-F1 0.534 vs 0.540 for full CPT). The benefit is GPU-memory savings plus easy rollback via untouched base weights — not speed, since wall-clock time was comparable because full forward and backward passes are still required.
What is the Daily Oracle benchmark and how is it used in this paper?
Daily Oracle (Dai et al., 2025) is a continuously regenerated evaluation that harvests True/False and multiple-choice question-answer pairs from each day's news, so any model can be scored against its own knowledge cutoff date. This paper uses it as the primary temporal benchmark: questions are split into before-cutoff and after-cutoff sets per model, evaluated under the OLMES protocol, and scored with Macro-F1 to handle class imbalance. It is the sole source of the paper's knowledge-acquisition numbers, which is also its main evaluation limitation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded values extracted as garbled run-on strings (e.g., Figures 1, 3, 5–7, 11); quantitative judgments rely primarily on Tables 4–8 and Appendix tables, which extracted cleanly.
- Table 6 column alignment is uncertain in places (duplicate 'Avg' header, possible column shifts); apparent contradictions with §3.3 claims are flagged as reviewer questions rather than asserted as errors.
- Some training details (batch size, sequence length, per-run token counts) are not present in the extracted text; the extracted text may be truncated relative to the full PDF.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.