CodeMidas: Scaling Agentic Coding RL Environments from Code Itself
Bowen Ye, Lei Li, Shicheng Li, Zihao Yue, Linghao Zhang, Hanglong Lv, Yuanxin Liu, Wenhan Ma, et al.
cs.AI
Quick verdict
Claims source code alone can yield 5,545 RL environments that lift MiMo-V2.5 on five benchmarks; gains plausible but single-run, no significance tests.
Summary
CodeMidas is an agentic pipeline that turns implemented functionality in existing open-source codebases into executable reinforcement-learning environments for coding agents, using source code as its only task-specific input. Four modules—task design, test construction grounded in execution of the original code, execution-consistency checks (fail-to-pass verified across six fresh containers), and post-rollout filtering (leakage probes, solution review, rollout outcome filtering)—reduce 22,575 candidate sources to 5,545 verified tasks from 3,185 codebases across 23 languages and 15 domains. GRPO training of MiMo-V2.5 on these tasks improves all five external benchmarks: SWE-bench Pro 50.3→54.4, DeepSWE 10.0→21.7, ProgramBench Almost Solved 4.5→21.5, RepoZero C2Rust 40.5→51.8, and Terminal-Bench v2.1 63.7→72.2. Ablations show monotonic gains from 1k→3k→5,545 filtered tasks, with the 5,545-task pool beating an 8k unfiltered sample (DeepSWE 21.70 vs 17.11). Behavioral analysis finds more pre-edit exploration (27.2→40.1 calls), higher drafting ratios (0.358→0.629), and more diverse self-verification (2.03→2.53 distinct commands), with agent-written checks associated with +4.2pp pass rates (95% CI 1.8–6.6). Main limitations: single-run results without significance testing, unquantified LLM-judge filtering quality, only task-level disjointness from benchmarks, a single base model, and no stated data or code release.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Source code alone suffices to construct thousands of executable, verified RL tasks — 5,545 from 3,185 codebases across 23 languages — without issues, PRs, commits, existing tests, or written documentation.
- GRPO training on these tasks improves MiMo-V2.5 on all five external benchmarks, e.g., DeepSWE 10.0→21.7, ProgramBench Almost Solved 4.5→21.5, Terminal-Bench v2.1 63.7→72.2.
- Filtering and cleaning matter more than raw scale: the 5,545-task filtered pool beats an 8k unfiltered pool on SWE-bench Pro, DeepSWE, and CodeMidas Val.
- RL induces generalizable behavioral shifts — more codebase exploration, more reasoning-grounded drafting, more diverse self-verification — visible on held-out benchmarks, with agent-executed checks associated with +4.2pp success (95% CI 1.8-6.6).
Strengths
Genuinely artifact-free task construction at scale
Table 1 positions CodeMidas as the only listed pipeline requiring no issues, PRs, commits, existing tests, or written descriptions, with 23-language coverage vs 1-20 for eight prior pipelines; the Figure 1 funnel quantifies retention at every stage (22,575 → 16,027 → 12,746 → 11,930 → 8,173 → 5,545).
Verifier hygiene treated as a first-class pipeline stage
§3.3 enforces a fail-to-pass transition (2 starting-state containers fail, 4 reference passes) in fresh containers; §3.4 adds adversarial leakage probes, solution review, and outcome filtering. §5.1 empirically justifies this: the 8k vanilla pool built before cleaning/filtering loses to the 5,545-task pool on all three ablation evals (DeepSWE 17.11 vs 21.70; Val 40.24 vs 44.73).
Broad transfer across task types
Behavior-level evidence beyond score deltas
Weaknesses
No statistical treatment of headline gains
LLM-judge filtering is unvalidated
Contamination checked only at task level
No head-to-head against artifact-derived training data
Single base model, no release statement
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CodeMidas (arXiv 2609.22068) is a method paper from Xiaomi's LLM Core team (with Peking University, HKU, and Renmin University) proposing an agentic pipeline that converts implemented functionality in existing codebases into executable RL environments for coding agents, with source code as the only task-specific input — no issues, PRs, commits, pre-existing tests, or written descriptions (Table 1 positions it against eight prior pipelines and claims the broadest language coverage, 23). The central claims are: (1) artifact-free environments can be produced at scale with reliable verifiers (5,545 tasks from 3,185 codebases, 15 domains); (2) GRPO training on them transfers broadly — all five external benchmarks improve, from +4.1pp on SWE-bench Pro to +17.0 on ProgramBench (Almost Solved) and +11.7pp on DeepSWE; (3) quality filtering beats raw scale (§5.1); (4) RL shifts measurable agent behaviors that also appear on held-out tasks (§5.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CodeMidas?
CodeMidas is a pipeline from Xiaomi's LLM Core team (with Peking University, HKU, and Renmin University) that automatically turns functionality already implemented in open-source codebases into reinforcement-learning training environments for coding agents. Unlike prior pipelines that need issues, pull requests, commits, existing tests, or documentation, it uses only the source code: agents select functionality, remove its implementation to create a task, build tests by executing the original code, and filter out leaky or unreliable environments. The result is 5,545 verifiable tasks from 3,185 repositories across 23 programming languages.
How much does training on CodeMidas improve coding agents?
After GRPO training on the 5,545 tasks, the MiMo-V2.5 model improved on all five external benchmarks: SWE-bench Pro from 50.3% to 54.4%, DeepSWE from 10.0% to 21.7%, ProgramBench Almost Solved from 4.5 to 21.5, RepoZero C2Rust from 40.5% to 51.8%, and Terminal-Bench v2.1 from 63.7% to 72.2%. Note these are single-run numbers without statistical significance tests.
Is the CodeMidas dataset or code publicly available?
The reviewed text does not state any release of the dataset, pipeline code, or container images, and training depends on Xiaomi's MiMo-V2.5 base model. Independent reproduction would currently require rebuilding the pipeline from scratch; the paper does describe every pipeline stage and hyperparameter in enough detail to attempt that.
Why do fewer filtered tasks beat more unfiltered tasks in this paper?
In the paper's ablation, an 8,000-task pool sampled before cleaning and filtering scored lower than the 5,545-task filtered set on SWE-bench Pro (53.81 vs 54.40), DeepSWE (17.11 vs 21.70), and CodeMidas Val (40.24 vs 44.73), and even a 3,000-task filtered subset beat the 8k pool. The authors attribute this to unreliable verifiers — leaked solutions, tests that accept wrong implementations, unstable executions — which their consistency checks and post-rollout filtering remove before RL training.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were extracted as text dumps; numeric labels were readable but layout could not be visually inspected (e.g., exact learning-curve shapes in Figs. 6-7).
- The text ends after Appendix B; any further appendices (e.g., prompt templates or per-benchmark evaluation details) appear truncated.
- Paper metadata lists 8 authors while the full text shows about 20; affiliation attribution here follows the full-text author block.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.