Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training
cs.AI
Quick verdict
Controlled 30-allocation sweep: moderate mid-training coverage is best per domain and gaps resist SFT; unusually candid, but single model and benchmark.
Summary
The paper asks whether per-domain data coverage chosen at mid-training can later be undone by alignment (SFT and RL). Using Qwen3-8B-Base — with a mid-training-only replication at Qwen3-4B — the authors train 30 coverage allocations across the five-domain simplex of rule-disjoint KOR-Bench domains (24 sweep configurations plus 6 held out, five seeds each, ~1.5B tokens per mid-training run, 630 training runs in total). Three findings emerge. First, every domain has an interior coverage optimum: the moderate 10–40% band is best for all five domains (calibrated permutation test for quadratic interiority P≈0.010), with fitted peaks between 9.9% and 35.1% at 8B that shift with model scale. Second, a fixed-budget compensatory SFT raises 116/120 cells (mean +4.32pp) yet bridges 0/240 domain pairs at a 5pp threshold (30/240 at a 10% ratio), behaves almost identically to an equal-budget uniform-SFT control, and a permutation null would bridge 13.8±3.3 and 77.9±8.5 pairs (P<0.001). Third, zero coverage collapses mid-training-only accuracy, but a FineWeb-Edu-only control shows the collapse is co-mingled with generic distributional drift. Main limitations: simplex coupling makes the optima mixture-level marginals with a saddle joint surface, the exploratory θ* gain (+4.36pp) is marginal under an uncorrected Welch test, and everything is scoped to one model family, benchmark, and recipe.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Every KOR-Bench domain exhibits an interior mid-training coverage optimum, with the moderate 10–40% band best for all five domains (permutation test P≈0.010) and fitted 8B peaks between 9.9% and 35.1%.
- A fixed-budget compensatory SFT pass raises accuracy in 116/120 cells (mean +4.32 pp) yet bridges 0/240 pairwise domain gaps at a 5 pp threshold, behaves like an equal-budget uniform-SFT control, and places gains in a gap-preserving way per a permutation null (P<0.001).
- Zero mid-training coverage collapses mid-training-only accuracy even with a high prior, but the FineWeb-Edu-only control shows the collapse is co-mingled with generic distributional drift.
- The five per-domain optima are mixture-level marginals, not the coordinates of a jointly optimal mixture: the compositional surface reads as a saddle in every domain, though that classification is provisional.
- The exploratory θ* allocation attains the largest full-pipeline gain (+4.36 pp vs +0.80/+0.64 pp) but is marginal under an uncorrected Welch test and is selected from the same sweep.
Strengths
Design and compute scale with genuine out-of-sample testing
630 primary runs (160 mid-trainings of ~1.5B tokens each, 420 SFT passes, 50 RL runs; Appendix B.1), five seeds per configuration, and six allocations withheld from all curve fits yet carried through the complete mid+SFT+RL pipeline: held-out residuals 1.0–1.8 pp for four domains (3.0 pp for Operation) and non-closure reproduced at 0/60 pairs (Appendix F.4).
Calibrated statistics rather than raw p-values
Interiority is tested by whole-row permutation over the 42-allocation pool (P≈0.010; independent shuffling P≈0.005, Appendix F.1), and closure is judged against a permutation null reallocating the observed gains (13.8±3.3 vs observed 0/240 at 5 pp; P<0.001, Appendix G.2). The authors even report the unfavorable diagnostic that split-Gaussians manufacture interior peaks on 64% of permuted datasets, motivating the 3-parameter quadratic test.
Self-adversarial controls
Headline does not rest on parametric fits
Compositional honesty
Weaknesses
Marginals, not mechanisms — and the saddle is undecidable
θ* headline rests on the weakest inference chain
The actionable output is scale-dependent and unvalidated
'Alignment-resistant' is essentially an SFT-family result
External benchmarks share exposure with training
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper studies one decision in the multi-stage LLM pipeline: per-domain data coverage at mid-training. Using Qwen3-8B-Base (plus a mid-training-only Qwen3-4B replication), five semantically rule-disjoint KOR-Bench domains (ciphers, operations, logic, counterfactual, puzzles), and 30 coverage allocations spanning the five-domain simplex (24 sweep points + 6 held-out, five seeds each), it advances three claims: (1) every domain has an interior coverage optimum, with the moderate 10–40% band best for all five (permutation test P≈0.010, Appendix F.1); (2) coverage-induced gaps survive a fixed-budget alignment pass — compensatory SFT raises 116/120 cells (mean +4.32 pp) yet bridges 0/240 domain pairs at 5 pp, matching a uniform-SFT control, while a permutation null would bridge 13.8±3.3 pairs (P<0.001, Appendix G.2); (3) zero coverage collapses mid-training-only accuracy, though a FineWeb-Edu-only control shows the collapse co-mingled with generic drift (§5.3). An exploratory θ* allocation yields the largest full-pipeline gain (+4.36 vs +0.80/+0.64 pp, Table 1) but is admitted to be marginal under an uncorrected Welch test. The abstract and §5 enforce scope discipline unusually well: all claims are explicitly confined to one model family, one benchmark, and one recipe.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is mid-training and why does its data mixture matter?
Mid-training is the stage between broad pretraining and instruction alignment (SFT/RL), where models receive curated continued-pretraining data. This paper shows in a controlled setting (Qwen3-8B-Base, five KOR-Bench reasoning domains) that the per-domain share of that data has non-monotonic effects — a moderate 10–40% share is best for every domain tested — and that these coverage choices leave accuracy fingerprints that later SFT and RL mostly do not erase.
Can SFT or RL fix a bad mid-training data mixture?
In this paper's tested setting — a fixed ~3-epoch SFT budget on Qwen3-8B-Base checkpoints — no: compensatory SFT reweighted toward deficient domains raised accuracy in 116 of 120 cells but closed 0 of 240 pairwise domain gaps at a 5-point threshold, and behaved almost identically to a uniform-SFT control. The authors scope this carefully to one policy family, one budget, and a short 200-step RL leg; sharper reweighting does close some gaps but costs roughly half the average gain.
What fraction of training data should each domain get during mid-training?
The paper finds an interior optimum per domain, with the moderate 10–40% band best for all five domains and fitted peaks between 9.9% and 35.1% at the 8B scale. However, the peaks shift with model scale (a 4B replication moved them by up to −17.5 points), the domains share a fixed budget so the optima are not jointly achievable, and the authors explicitly warn these are descriptive marginals, not validated allocation rules.
Is the paper claiming alignment can never repair coverage gaps?
No. The paper is explicit that it tested one compensatory-SFT policy family, one fixed budget, and a short RL stage (200 GSPO steps, +1.1 to +2.3 points) in one model family on one benchmark family. What it does establish is that, in that setting, the average gain from SFT is real (+4.32 points) but is placed in a systematically gap-preserving way — a random reallocation of the same gains would have closed significantly more gaps.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with explicit '[...sampling gap...]' markers; skipped sections include parts of §4.3, the §4.2 continuation, appendices C.5–C.7, and parts of Appendix G.4/Table 15. The review relies on the visible content, which covers all three headline findings, Table 1, and appendices B, C (partial), E, F, G, H, I.
- No code or repository availability statement appears in the visible text — only 'released per-seed data' is referenced (Appendix I). codeAvailable=false reflects the visible text; an availability statement could sit in a sampled gap.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.