FrogNano: Training a 4B Coding Agent via Online Task Synthesis
Minseon Kim, Zhengyan Shi, Emiliano Penaloza, Christopher Cui, Roger Creus Castanyer, Maryam Hashemzadeh, Isadora White, Jonathan Light, et al.
cs.AI
Quick verdict
Claims distillation-free RL on ~1,500 synthetic tasks makes Qwen3.5-4B a competitive agent (61.5% SWE-bench Verified); credible recipe, but validation-set reuse, base-score discrepancies, and missing significance testing temper the headline claims.
Summary
FrogNano (Microsoft Research) is a 4B coding agent post-trained from Qwen3.5-4B purely with reinforcement learning on ~1,500 synthetic software-engineering tasks, with no SFT or imitation of stronger models' trajectories. Two ingredients matter most: Leaf, a lightweight Claude-Code-style harness whose adoption lifts the base model from 8.3% to 37.2% on SWE-bench Verified, and TaskPilot, an online synthesis pipeline that generates issue, gold patch, and hidden fail-to-pass tests from repository snapshots (SWE-rebench, Scale-SWE), calibrates each candidate to the current policy's success rate (0.5 target, later 0–0.5), refining or discarding out-of-band tasks. Five iterations of ~300 tasks yield 61.5% on SWE-bench Verified — explicitly the paper's validation set — plus 37.6% SWE-bench Pro, 31.1% Terminal-Bench 2.0, and 47.3% PatchEval-Verified on held-out sets. Gains transfer to mini-SWE-agent, and pass@k analysis suggests genuine capability-boundary growth rather than reshaping. A two-stage reward-hacking audit found 2.5–3.07% attempted hacks, none effective. Main limitations: headline numbers lack confidence intervals or significance tests; the base score appears inconsistently as 39.4% and 43.0%; and weights or code release is not announced in the text.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A 4B model (Qwen3.5-4B) post-trained purely with RL on ~1,500 synthetic tasks reaches 61.5% on SWE-bench Verified, 37.6% on SWE-bench Pro, 31.1% on Terminal-Bench 2.0, and 47.3% on PatchEval-Verified, without SFT or imitation of stronger models' trajectories.
- Online task synthesis calibrated to the current policy's learnability frontier (TaskPilot) outperforms training on ~300 learnability-filtered real tasks (49.1% vs 48.0%) and on a 1,500-task real mixture (61.5% vs 53.4%).
- The agent harness is a first-order factor for compact models: Leaf lifts Qwen3.5-4B from 8.3% to 37.2% on SWE-bench Verified while leaving MiniMax-M2.5 unchanged (66.5%).
- Iterative RL expands the capability frontier rather than reshaping existing pass@k into pass@1, as shown by a k-invariant gap over the base model (Figure 14).
- No confirmed reward hack was effective: 2.5% of trajectories contain attempts, all blocked by harness and infrastructure.
Strengths
Harness ablation is the most valuable finding
Switching Qwen3.5-4B from the R2E-Gym harness to Leaf lifts SWE-bench Verified from 8.3% to 37.2% (Section 1/2), while MiniMax-M2.5 is indifferent (66.5% on both) — a clean, practically important demonstration that interface design dominates at small scale.
Controlled synthetic-vs-real data comparison
Figure 1: ~300 policy-filtered real SWE-rebench tasks reach 48.0% vs 49.1% after iteration 1 (~300 synthetic tasks); the 1,500-task real mixture reaches 53.4% vs FrogNano's 61.5% — directly supporting the core synthetic-only claim.
Evidence against pure pass@k reshaping
Exemplary reward-hacking audit
Generalization beyond the training harness
Weaknesses
SWE-bench Verified is both validation set and headline metric
Internal base-score discrepancy (39.4 vs 43.0)
No uncertainty quantification on headline deltas
Cross-model comparisons are not harness-controlled
Contamination/decontamination not addressed in the provided text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
FrogNano (arXiv:2609.07925v4) — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FrogNano and how well does it score on SWE-bench Verified?
FrogNano is Microsoft Research's 4B-parameter coding agent, post-trained from Qwen3.5-4B using only reinforcement learning on ~1,500 synthetic software-engineering tasks (no SFT and no imitation of larger models' trajectories). It resolves 61.5% of SWE-bench Verified (the paper's validation set), 37.6% of SWE-bench Pro, 31.1% of Terminal-Bench 2.0, and 47.3% of PatchEval-Verified, and costs about $0.21 per SWE-bench Verified task in the authors' setup.
How was FrogNano trained without using larger models?
Three ingredients: (1) Leaf, a lightweight Claude-Code-style harness with five typed tools (read, write, edit, glob, bash) — it alone lifts the base model from 8.3% to 37.2% on SWE-bench Verified; (2) TaskPilot, an online pipeline that generates issue + gold patch + hidden fail-to-pass tests from repository snapshots and keeps only tasks the current policy solves at a target rate (0.5 for iterations 1–4), refining tasks that are too easy or too hard; (3) five RL iterations (DPPO, ~300 tasks each) with a success-gated length penalty. Note that a stronger generator model authors the tasks in iteration 5 — stronger models author data but never provide solution trajectories to imitate.
Is the 61.5% SWE-bench Verified number trustworthy?
Treat it as somewhat optimistic: the paper explicitly uses SWE-bench Verified as its validation set, meaning iteration launches, calibration targets, and the iteration-5 redesign were decided by watching that benchmark, and the training distribution is Python-heavy like Verified. Cleaner generalization evidence is the held-out sets (37.6% Pro, 31.1% Terminal-Bench 2.0, 47.3% PatchEval). The paper also reports the base model at both 39.4% and 43.0% in different places, an unexplained discrepancy.
Can I reproduce FrogNano or download the model?
Reproduction is expensive: training used a single node with 8 NVIDIA B200 GPUs plus access to SWE-rebench/Scale-SWE repository snapshots and a strong task-generator model. The provided text announces no code or weight release (a project page exists at microsoft.github.io/debug-gym). The evaluation protocol is fully specified (Leaf harness, 3 seeds, temperature 0.6, 150 steps, 131k-token context), so a re-evaluation would be feasible if checkpoints are released.
Does the RL actually add new capability or just reshape existing skills?
The paper's pass@k analysis supports genuine frontier expansion: the gap between FrogNano and the base Qwen3.5-4B stays roughly constant as k grows from 1 to 8, meaning FrogNano is not merely distilling high pass@k into pass@1. It also explicitly avoids distillation: no SFT and no imitation of stronger models' trajectories in the main pipeline.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with '[...sampling gap...]' markers; content in skipped sections (parts of Sections 3.2, 5.2 figures, Appendix A benchmark detail, Appendix F.1 settings) was not visible, so judgments about those parts rely on surrounding text.
- The provided text contains no explicit code or weight-release statement; 'codeAvailable: false' reflects absence of such a statement, not a confirmed closed release.
- Per-model competitor values behind Figure 2's claims (e.g., GPT-5 mini, Grok 4, Opus 4.1) are not present in the provided text; the comparison claim was assessed on the figure description only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.