Stress-testing Alignment Midtraining
Sid Baines, Jonathan Bostock, Maria Angelica Martinez, Andrew Draganov, David Africa, Daniel Tan
cs.CL · cs.AI
Quick verdict
Stress-tests show alignment midtraining steers motivation but breaks under 2% conflicting finetuning data; well-scaled yet one seed per cell limits certainty.
Summary
This paper empirically stress-tests alignment midtraining (AMT): continuing a base model's pretraining on large volumes of synthetic alignment-relevant documents before instruction tuning and elicitation fine-tuning. Using a fictional 'Dispatch' ship-allocation task (egalitarian Charter vs profit-maximizing Coin) and a fictional Python 4 programming language, the authors midtrain gemma-3-4b/12b/27b and GLM-4.5-Air (110B parameters) on 1M-1B tokens (mixed 1:1 with Dolmino replay), then apply instruction tuning and elicitation via SFT or GRPO RL. Midtraining steers motivation under ambiguous finetuning (90%/92% conformance on GLM-4.5-Air), but replacing 2% of finetuning examples with conflicting demonstrations reverses preferences (90% to 13%, 92% to 46%) while chat evaluations cannot distinguish the models. Generalization to rules held out of finetuning is weak (19% to 53% GLM; 26% to 37% gemma-27b); removing worked demonstrations from the midtraining corpus cuts the uplift (factors 0.73/0.35); GRPO preserves the midtrained motivation worse than SFT. A re-run of Model Spec Midtraining finds its robustness conclusion inconsistent with the original Figure 4 and released checkpoints. Limitations: one training seed per cell, highly synthetic settings, and deliberately minimal instruction tuning. The authors conclude public evidence is insufficient to treat AMT as a robust alignment solution.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Midtraining steers model motivation when finetuning data is motivationally ambiguous: with 8,192 ambiguous EFT demonstrations, Charter-midtrained and Coin-midtrained GLM-4.5-Air models follow their midtrained motivation 90% and 92% of the time respectively (Fig 2a).
- Replacing just 164 of 8,192 (2%) EFT examples with conflicting-motivation demonstrations reverses the preference (Charter 90% to 13%, Coin 92% to 46%), meaning roughly 45K-80K conflicting finetuning tokens override 190M midtraining tokens.
- Midtraining generalizes only weakly to rules held out of finetuning (held-out clause adherence 19% to 53% on GLM-4.5-Air, 26% to 37% on gemma-3-27b), and filtering worked demonstrations out of the midtraining corpus reduces this uplift by factors of 0.73 (GLM) and 0.35 (gemma-27b).
- Switching elicitation from SFT to GRPO-based RL substantially diminishes behavioral use of the midtrained motivation even though the model still verbally states it.
- Based on these and related results, public evidence is insufficient to claim alignment midtraining solves the structural difficulties of aligning capable AI systems.
Strengths
Unusual public scale on both axes
Full-weight midtraining of up to 110B parameters (GLM-4.5-Air) with up to 1B midtraining tokens, across three model families and four sizes (gemma-3-4b/12b/27b, GLM-4.5-Air), spanning over an order of magnitude in each dimension (Sec 2.1, Fig 3) — far beyond most public AMT work, which uses small models.
Carefully controlled synthetic setting
The Dispatch task is fictional (Qalvori Charter vs Coin), programmatically generable so Charter/Coin agreement or conflict is arbitrary (App A.3); evaluation uses held-out templates, crew names and ports; corpus generation uses a planner blind to the target motivation, a four-LLM writer pool with seeded mixture weights, model-judge pass/fail plus mechanical filters (5-gram Jaccard 0.72 dedup, 80.6%/76.2% acceptance rates in App F.1.1).
Multi-axis stress-testing rather than a single demonstration
External replication with released checkpoints
Negative results reported with side-effect accounting
Weaknesses
One training seed per cell in nearly all experiments
External validity rests on synthetic settings and a minimal pipeline
Threshold and dose asymmetries undercut scale generalization
Internal eval-draw inconsistencies
No hypothesis testing for main comparisons
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether alignment midtraining (AMT) — continuing pretraining on alignment-relevant synthetic documents before instruction tuning — actually repairs two structural weaknesses of post-training: insufficient context coverage and underdetermined (ambiguous) finetuning data. It tests this at unusual public scale: gemma-3-4b/12b/27b and GLM-4.5-Air (110B), midtraining doses 1M-1B tokens, full-weight training, mixed 1:1 with Dolmino replay, then instruction tuning (Dolci-Instruct-SFT) and elicitation finetuning (SFT or GRPO). Central claims: (1) AMT steers motivation under ambiguous EFT (Fig 2a: 90% Charter / 92% Coin on GLM-4.5-Air); (2) 2% conflicting EFT (164 of 8,192 examples) reverses this (90% to 13%, 92% to 46%); (3) generalization to rules held out of EFT is weak (19% to 53% GLM, 26% to 37% gemma-27b, Sec 3.2); (4) removing worked demonstrations from the midtraining corpus cuts the uplift by factors 0.73/0.35 (Sec 3.3); (5) GRPO-based elicitation keeps the verbal motivation but not the behavior (Sec 3.4). Conclusion: public evidence is insufficient that AMT addresses distribution-shift alignment.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is alignment midtraining (AMT)?
Alignment midtraining is a proposed training stage inserted between a model's initial pretraining and its final post-training (instruction tuning and RLHF). The model continues pretraining-style learning on large volumes of documents encoding desired facts, rules, or motivations, with the goal of shaping the underlying prior so that later alignment training generalizes better to situations not directly demonstrated.
Does alignment midtraining work, according to this paper?
Only partially. In the paper's fictional Dispatch setting, midtraining reliably steers which motivation a model follows when the finetuning data is ambiguous between two motivations (90-92% conformance on a 110B model). But the effects proved brittle: replacing just 2% of finetuning examples with conflicting demonstrations reversed behavior, generalization to rules never shown in finetuning was weak (19-53%), and RL-based elicitation largely erased the behavioral effect while the model still verbally endorsed the trained motivation.
How much conflicting finetuning data is needed to override alignment midtraining?
In these experiments, 164 conflicting examples out of 8,192 (2%, roughly 28K-80K tokens) were enough to reverse the midtrained preference across three model families, including GLM-4.5-Air with 110B parameters trained on 190M midtraining tokens. Notably, standard chat-based evaluations could not distinguish the contaminated models from clean ones.
Which models and how many tokens were used in the study?
The authors midtrained gemma-3-4b, gemma-3-12b, gemma-3-27b, and GLM-4.5-Air (about 110B parameters) on 1 million to 1 billion tokens of synthetic alignment documents (a fictional trade-dispatch charter versus profit motive, plus a fictional Python 4 programming language), always mixed 1:1 with Dolmino replay data, followed by instruction tuning and task-specific elicitation fine-tuning.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Sections 3.4 (RL results) and part of Section 4 were truncated mid-sentence in the extracted text; the GRPO findings are taken from narrative claims without the underlying numbers.
- Figures appear as flattened text dumps of axis labels and values; exact layouts and some bar-level values could not be independently verified.
- Several appendix subsections were partially omitted (B.5, B.7, the B.8 multi-seed study, part of D.8), so seed-variance evidence could not be assessed.
- No explicit code or data availability statement appears in the extracted text; codeAvailable=false reflects absence of evidence rather than confirmed unavailability.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.