Two-Stage Reinforcement Learning for Sound and Adversarial Test Generation in Code LLMs
Jiacheng Xu, Wentao Zhang, Zhiyi Lyu, Fuxiang Zhang, Chaojie Wang, Yang Liu, Bo An
cs.CL · cs.LG
Quick verdict
Claims two-stage RL makes an LLM generate sound, adversarial tests; gains over SFT/reward-model selection on TACO/LCB look real but lack variance reporting.
Summary
The paper proposes Test Cases Scaling (TCS), a two-stage reinforcement-learning framework that trains a code LLM to generate test cases that are both sound (consistent with a ground-truth reference solution) and adversarial (able to expose plausible incorrect programs). Using GRPO under one shared policy acting as both solver and verifier, Stage 1 rewards tests verified against the reference solution; Stage 2 restricts a rolling policy-aligned buffer to the solver's current failure modes and rewards counterexample tests. A theoretical bound (Proposition 1) caps the mis-selection probability of pass-count selection at (N−1)exp(−K(δ−α)²/2) under independence and δ>α. On TACO (6,318 curated training problems, 1,000-problem validation) and LiveCodeBench, DeepSeek-R1-Distill-Qwen-1.5B/7B trained with TCS reach pass@1 of 16.47/30.56 on LiveCodeBench without public tests, versus 13.32/25.53 for joint SFT and 10.01/21.46 for the base models; Best-of-N selection with self-generated tests yields the largest gains (7B average 44.57). Generated tests also help select outputs from stronger external LLMs. Main limitations: no variance estimates or significance tests are reported; the δ>α assumption is never empirically measured; TACO evaluation is in-distribution; only Qwen-distill backbones are tested; and the hard stage switch is acknowledged as suboptimal.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Effective test generation for code LLMs requires both soundness control and candidate-conditioned adversariality, best learned through a staged reward curriculum.
- TCS's two-stage RL improves both training-time pass@1 and inference-time Best-of-N selection over base models and joint offline SFT on TACO and LiveCodeBench.
- Under independent sampling and δ>α, pass-count selection with self-generated tests mis-selects with probability at most (N−1)exp(−K(δ−α)²/2).
- The learned test generator transfers to selecting among outputs of stronger external LLMs.
Strengths
Well-motivated problem decomposition
Secs. 1-3 split test quality into soundness (Exec(C*,I)=O) and candidate-conditioned adversariality, and the two-stage curriculum directly addresses the sparsity of counterexample rewards that makes joint optimization brittle (Sec. 3.2).
Fair offline baseline design
The SFT baseline (Sec. 4.2) already receives execution-verified, candidate-conditioned adversarial supervision following Sol-Ver, so the TCS-vs-SFT gap isolates the value of online policy-aligned RL rather than label quality.
Theory with actionable knobs
Honest reporting of test limits
Consistent gains across scales and both uses
Weaknesses
No statistical rigor
Theory-practice gap
Reward-hacking surface in Stage 2
Limited generality
Baseline coverage
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper studies automatic test-case generation for code LLMs in an execution-verifiable post-training setting: a ground-truth solution C* is available to validate generated tests. Central claims: (1) effective test generation requires both soundness (Exec(C*,I)=O) and candidate-conditioned adversariality (failing plausible incorrect code); (2) the proposed TCS framework — Stage 1 learns ground-truth-verified tests, Stage 2 learns counterexample tests against the solver's current failure modes from a rolling policy-aligned buffer — improves both pass@1 and inference-time selection on TACO and LiveCodeBench; (3) Proposition 1 gives an exponential reliability bound (N−1)exp(−K(δ−α)²/2) for pass-count selection; (4) the learned verifier transfers to selecting outputs of stronger external LLMs. The claims are stated precisely and, as far as the visible text shows, matched by experiments.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TCS (Test Cases Scaling)?
TCS is a two-stage reinforcement-learning framework that trains one code LLM as both a solution solver and a test-case verifier. Stage 1 learns to generate tests consistent with a ground-truth reference solution; Stage 2 learns counterexample tests targeting the solver's current failure modes via a rolling policy-aligned buffer. The generated tests are then used at inference time to rank candidate programs by how many tests they pass.
How much does TCS improve code generation results?
On LiveCodeBench without public tests, pass@1 improves from 10.01 to 16.47 for the 1.5B model and from 21.46 to 30.56 for the 7B model relative to the DeepSeek-R1-Distill-Qwen base models, and beats a joint code-test SFT baseline (13.32 and 25.53). Using the generated tests for Best-of-N selection raises the 7B cross-benchmark average to 44.57.
Why train test generation in two stages instead of one objective?
Because a useful test must be both sound (correct for the reference solution) and discriminative (fail wrong code). Optimizing both at once is brittle: soundness-only rewards yield trivial tests, while counterexample rewards are too sparse to learn from from scratch. The curriculum provides dense feedback first, then adversarial pressure.
Does the paper release code or models?
The provided text contains no code or model release statement. Training relies on public assets (TACO, LiveCodeBench, DeepSeek-R1-Distill-Qwen backbones), and prompt templates plus the training algorithm are given in the appendices, so reimplementation is possible in principle.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Middle sections were sampled out of the provided excerpt: parts of Sec. 3.2.1 (buffer mechanics), Sec. 3.3 (inference rule Eq. 5), the bodies of reward equations Eqs. 3-4, related work, and Appendix Table 4 were not visible; assessments of those parts rely on surrounding visible text only.
- No code/data availability statement appears anywhere in the provided excerpt; codeAvailable=false reflects the excerpt, not a verified absence in the full paper.
- The LiveCodeBench evaluation subset size and RL compute/hardware details are not visible in the excerpt.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.