Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs
Jingtan Wang, Arun Verma, Xiaoqiang Lin, Zhengyuan Liu, Nancy F. Chen, Daniela Rus, Bryan Kian Hsiang Low
cs.CL · cs.AI · cs.LG
Quick verdict
Credible empirical study: near-optimal SFT–RL allocation regions are wide and transfer from small to larger models, but key numbers rest on single seeds and scale caps at 14B.
Summary
This empirical study tackles how to split a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL) in LLM post-training. Rather than chasing a single optimal ratio, the authors define the "near-optimal region": all allocation ratios within a tolerance (2–10%) of peak performance. Experiments cover four tasks (math, instruction following, summarization, helpfulness) across Llama 3, Qwen 2.5, and Qwen 3 families up to 14B parameters, using DPO and GRPO as RL methods, plus a cost-asymmetry analysis where SFT data costs more than preference data. Core findings: near-optimal regions are indeed wide (at 10% tolerance, most tasks cover 55–75% of the allocation space), widen with model scale, and transfer reliably from small proxies to larger targets, so the proposed recipe succeeds 94.3% and 97.1% of the time at 5% and 10% tolerance while costing roughly 2.5× less than sweeping the target model. Key limitations: all training is in-distribution on task-specific data, budgets cap at 15k examples, LoRA replaces full fine-tuning, headline results are single-seed with multi-seed validation on only two tasks, task saturation and peak-performance effects of the relative-tolerance definition can inflate regions, and helpfulness is scored by a GPT-2-scale reward model. A methodologically self-aware study that should be confirmed by independent replications before practice adopts it.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under a fixed relative tolerance of 2–10%, the set of near-optimal SFT–RL allocations is wide rather than a single sharp optimum: at 10% tolerance most tasks cover 55–75% of the allocation space (Sec. 3.2).
- The near-optimal region generally widens with model scale at fixed tolerance, and near-optimal regions transfer reliably from small proxy models to larger targets whereas exact optima transfer inconsistently (Sec. 3.3).
- A three-step recipe (5-point proxy sweep, choose tolerance, pick region midpoint) satisfies the chosen tolerance on the target model in 94.3% of cases at ε=5% and 97.1% at ε=10% (Sec. 3.5).
- The conclusions hold across Llama 3, Qwen 2.5, and Qwen 3 families, four tasks, and both DPO (off-policy) and GRPO (on-policy) post-training (Secs. 3.2–3.3).
- When SFT annotations cost ρ× more than RL data (ρ∈{1,2,5,10}), the near-optimal region widens further and transfer trends persist (Sec. 3.4).
Strengths
Well-posed reframing with a concrete decision object
Sec. 2.1 Eq. (2) defines the ε-near-optimal region as the set of ratios within a relative tolerance of peak performance, which matches what practitioners actually choose (an allocation, not a point) and degrades gracefully to the point optimum as ε→0.
Broad, multi-axis empirical sweep
Three families (Llama 3, Qwen 2.5, Qwen 3) up to 14B (App. D), four tasks (math/IF/summarization/helpfulness), both DPO (off-policy) and GRPO (on-policy, Sec. 3.3/Fig. 5), and a cost-asymmetry extension (Sec. 3.4, ρ∈{1,2,5,10}), so the qualitative claims rest on many independent settings.
Systematic robustness program
Actionable recipe with quantified success rates and honest accounting
Candor about confounds and limits
Weaknesses
Single-seed headline statistics
Transfer metric is prone to optimistic degeneracy
Weak evaluators for two of four tasks
Scope is narrower than the title implies
The recipe is never validated out-of-sample
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Scaling Near-Optimal SFT–RL Annotation Budget Allocation from Small to Large LLMs
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main result of this paper on SFT vs RL data budget allocation?
The paper finds that when splitting a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL), there is rarely one uniquely best ratio. Instead, a wide set of allocations performs within a few percent (2–10%) of the best observed one. These "near-optimal regions" widen as models get bigger and can be identified cheaply on a small proxy model (e.g., 1B) and then applied to a larger target model (up to 14B), avoiding an expensive full grid search.
How much compute does the proposed proxy-model strategy save?
Per the paper's cost accounting (App. E.7): a 5-point ratio sweep on a 1B proxy plus one 8B run costs about 200 GPU-minutes, versus roughly 500 GPU-minutes for a full 5-point sweep directly on the 8B model — about a 2.5× saving. The proxy step acts as insurance: a blind fixed ratio of 0.5 misses the 5%-tolerance near-optimal region up to 80% of the time (Llama HelpSteer), while the proxy recipe hits the target's region 94.3% (ε=5%) and 97.1% (ε=10%) of the time.
What are the main limitations of the study?
All training is in-distribution on task-specific data (no out-of-distribution generalization is tested), budgets are capped at 15k annotated examples, LoRA is used instead of full fine-tuning, and model scale tops out at 14B. Only DPO and GRPO are tested (no PPO/SimPO), headline results are single-seed (with 3-seed validation on just two Llama-3 tasks), and helpfulness is scored by a small GPT-2-scale reward model.
Which models, tasks, and methods were used in the experiments?
Model families: Llama 3 (1B/3B/8B), Qwen 2.5 (1.5B/3B/7B, plus one 14B run), and Qwen 3 (1.7B/4B/8B). Tasks: GSM8K math accuracy, IFEval instruction following, ROUGE-L summarization on Reddit TL;DR, and a reward-model score on HelpSteer. RL methods: DPO (off-policy preference optimization) and GRPO (on-policy with verifiable rewards), both preceded by SFT, all with LoRA rank 32.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction is complete through Appendix H and the reference list; figures (Figs. 1–26, Tabs. 6–8) are referenced with captions but their plotted pixel values are not in the text, so per-point numbers cited here come only from in-text statements and tables.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.