Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M
Dushyant Rajput, Nirdesh Chauhan, Siddharth Kosaraju
cs.AI · cs.CL
Quick verdict
Controlled 53M→110M scaling plus honest ablation ladder; gains credible but single-seed, and the keep/drop efficiency score is never defined.
Summary
The authors scale a deliberately conventional sub-150M pretraining recipe from JugnuLM-53M to JugnuLM-110M, changing only the geometry to a deep-thin 23-layer × 576-hidden Qwen3-style decoder trained on FineWeb-Edu. The 110M baseline (R0) improves every reported metric — BLiMP 78.14→81.25, ARC-Easy 51.43→52.48, WikiText-2 byte-perplexity 2.04→1.95 — despite seeing fewer tokens (~8B vs ~12B), and its BLiMP essentially matches GPT-X2-125M (81.28) at ~12% fewer parameters. A keep-if-it-beats-prior-rung ablation ladder then adds value residuals (R1: ARC-Easy 54.67) and the Muon optimizer (R2: 56.10), both kept; a diverse data blend (R3) improves BLiMP and perplexity but drops ARC-Easy to 53.62, and two logit-distillation settings from a SmolLM2-1.7B teacher trade ARC against perplexity (R4a 56.99 at 2.178 byte-ppl; R4b 1.917 at 55.47) — honest negatives, with R2 shipped as champion. Limitations: single-seed runs compared at roughly one to two standard errors without formal tests; leaderboard comparator scores not re-evaluated under the authors' harness; an undefined 'efficiency score' drives keep/drop decisions; token accounting varies (8B/8.4B/10B); and Haidass-143M (60.23 ARC-Easy) still leads on reasoning. Code, configs, and all seven checkpoints are public.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 17 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Scaling the fixed recipe from 53M to 110M with a deep-thin 23×576 geometry improves every reported metric (BLiMP 78.14→81.25, ARC-Easy 51.43→52.48, byte-ppl 2.04→1.95) despite fewer training tokens (~8B vs ~12B), attributing the gain to capacity and depth rather than data.
- The 110M baseline's BLiMP of 81.25 essentially matches GPT-X2-125M's 81.28 at ~12% fewer parameters.
- Value residuals (R1) followed by the Muon optimizer (R2) lift ARC-Easy cumulatively by +3.6 to 56.10 at a near-flat BLiMP, and R2 is the best kept stack.
- A diverse data blend (R3) improves BLiMP (81.79) and byte-ppl (1.909) but costs 2.48 ARC-Easy points, showing ARC-Easy at this scale tracks FineWeb-Edu's educational filtering rather than raw diversity.
- Offline top-16 logit distillation from SmolLM2-1.7B either buys class-leading ARC-Easy (R4a: 56.99) at severe perplexity cost (2.178) or restores perplexity but loses the gain (R4b: 55.47, 1.917), so KD is not a clean win at this teacher–student gap.
Strengths
Genuinely controlled scaling design
Table 1 shows 53M and 110M share the entire recipe (GQA+QK-Norm, RMSNorm, RoPE, SwiGLU, z-loss, SmolLM2 tokenizer, FineWeb-Edu); only geometry (8L×512 → 23L×576) and token budget change, cleanly isolating capacity/depth from method or data.
Honest negative results with mechanisms
R3 (best BLiMP 81.79 and byte-ppl 1.909 of any rung, yet ARC-Easy −2.48 to 53.62) and R4a/R4b are reported and dropped, with a specific, testable mechanism offered (ARC-Easy tracks FineWeb-Edu's educational filtering, not diversity).
Full reproducibility package
Real evaluation pitfall documented
Informative optimizer dynamics
Weaknesses
Single-seed decisions at ~1–2 standard errors
The deciding metric is never defined
Comparators are leaderboard numbers, not re-evaluations
Token accounting is inconsistent
Narrow evaluation suite
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a deliberately conventional scaling and ablation study in the sub-150M parameter regime. The authors scale JugnuLM-53M to JugnuLM-110M (R0), holding the recipe fixed (Qwen3-style decoder: GQA, RoPE, SwiGLU, RMSNorm, QK-Norm, z-loss; FineWeb-Edu sample-10BT) and changing only the geometry to a deep-thin 23-layer × 576-hidden design following MobileLLM. Central claims: (1) the 110M model improves every reported metric — BLiMP 78.14→81.25, ARC-Easy 51.43→52.48, WikiText-2 byte-ppl 2.04→1.95 — despite a smaller token budget (~8B vs ~12B), so the gain is capacity/depth rather than data; (2) its BLiMP essentially matches GPT-X2-125M (81.28) at ~12% fewer parameters; (3) an ablation ladder identifies value residuals (R1) and Muon (R2) as keepers (cumulative +3.6 ARC-Easy to 56.10) while a data blend (R3) and two logit-distillation settings (R4a/R4b) are honest negatives.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is JugnuLM-110M and does it really match GPT-X2-125M?
JugnuLM-110M is a 109.7M-parameter Qwen3-style decoder trained by AltSlate Labs on FineWeb-Edu, described in arXiv 2609.14715. Its BLiMP score of 81.25 essentially matches GPT-X2-125M's 81.28 with about 12% fewer parameters and fewer training tokens (~8B vs ~12B). On ARC-Easy, however, the base 110M (52.48) and even the improved R2 stack (56.10) still trail GPT-X2-125M (57.07), and Haidass-143M leads the field at 60.23.
What are value residuals and the Muon optimizer in the JugnuLM ablation ladder?
Value residuals (R1, from ResFormer) give every layer's value vectors a learned-gated residual from the first layer (eV = V + λV1); they lifted ARC-Easy from 52.48 to 54.67 at essentially unchanged BLiMP. Muon (R2) is a Newton–Schulz-orthogonalized-momentum optimizer applied to the 2D hidden matrices (81.4M params) while AdamW keeps the embedding/head, norm gains, and gates (28.3M params); stacked on R1 it lifted ARC-Easy to 56.10 with a small −0.32 BLiMP dip.
Why was the diverse data blend (R3) rejected when it had the best perplexity?
The R3 blend (55% FineWeb-Edu, 35% DCLM-baseline, 10% FineMath) achieved the best BLiMP (81.79) and best byte-perplexity (1.909) of any rung, but ARC-Easy fell 2.48 points to 53.62 — the metric that gates the ladder ranking. The authors conclude that at this scale ARC-Easy depends on FineWeb-Edu's educational filtering, so diluting it with general-web and math data removed relevant signal the additions did not replace.
Are the JugnuLM models, code, and checkpoints publicly available?
Yes. Code and configs (including value_residual.py and muon.py) are at github.com/AltSlate-Labs/jugnu, and weights plus model cards for JugnuLM-53M and all 110M rungs (R0, R1, R2, R3, R4a, R4b) are on Hugging Face under the altslate organization. The paper also gives the exact EleutherAI lm-eval command used for BLiMP, ARC-Easy, and WikiText-2.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 appear only as extracted axis labels, legends, and captions in the text; curve shapes and exact clusterings could not be visually inspected.
- Comparator scores (GPT-X2-125M, Haidass-143M, Supra-50M-Instruct) are leaderboard-reported per the paper and could not be independently verified from the provided text.
- Section 7's evaluation command is truncated mid-line in extraction ('trust remote code=Tru 6'); the meaning is recoverable but the exact flags are uncertain.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.