ExecCritic: Learn to Test, Test to Improve for Coding Agents
Leitian Tao, Baolin Peng, Haorui Wang, Hang Wang, Hao Cheng, Wenlin Yao, Qianhui Wu, Tao Ge, et al.
cs.AI · cs.CL · cs.SE
Quick verdict
Claims frozen-test Test/Repair RL lifts SWE-bench Verified 61.2→72.6%; sound design, but no significance tests and headline uses oracle-feedback training.
Summary
ExecCritic targets a coupling problem in repository-level repair: when one agent trajectory writes both the patch and its validating test, shared misinterpretations yield false confidence. The framework splits the loop into a Test agent that authors a repository-native test bundle (patch, command, behavior contract), a fail-closed harness that qualifies the bundle by requiring a clean failure on the buggy checkout and freezes it, and a Repair agent that revises a source-only patch from bounded execution feedback. Both roles instantiate Qwen-3.5-35B-A3B and are trained separately: the Test agent gets SFT on 5,000 DeepSeek-V4-Flash trajectories plus GRPO rewarding Base-to-Gold success and discrimination of labeled candidate patches; Repair-agent RL rewards direct solving and feedback-guided revision, using one fixed oracle fail-to-pass test as training feedback. On SWE-bench Verified, test quality decides whether feedback helps: with the base Qwen Repair agent fixed, base-model tests drop resolution from 61.2% to 57.3% while GPT-5.6-sol tests raise it to 65.3%. Test-agent training lifts Base-to-Gold success from 22.2% to 62.2%; Repair training lifts no-test Round-0 from 61.2% to 68.3%; the composed system reaches 72.6% (+11.4). Limitations: results are three-run means without significance testing, the headline configuration trains with privileged oracle feedback (generated-test training scores 70.9%), and gains partly reflect extra compute, as the authors acknowledge.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Generated-test feedback can help or harm the same Repair agent depending on test quality: holding the base Qwen Repair agent fixed, base-model tests drop SWE-bench Verified resolution from 61.2% to 57.3% while GPT-5.6-sol tests raise it to 65.3%.
- Role-specific post-training raises the Qwen-3.5-35B-A3B Test agent's Base-to-Gold success from 22.2% to 62.2% via SFT followed by GRPO with discrimination rewards.
- Composing the post-trained Test and Repair agents reaches 72.6% on SWE-bench Verified, +11.4 points over the 61.2% no-test baseline, without stronger-model or oracle feedback at evaluation time (though Repair training used a fixed oracle F2P test).
- Keeping the generated test fixed and hidden during repair, under a fail-closed harness, reduces trajectory-level coupling between repair and validation evidence.
Strengths
Honest negative result leads the motivation
Section 5.2.1 / Figure 5b shows the same base Qwen Repair agent drops from 61.2% to 57.3% with base-model tests before showing gains with better tests — the paper's core insight is demonstrated by its own harm case, not buried.
Architecturally clean decoupling with fail-closed semantics
Sections 3 and B.1 (Eq. 7-12) formally separate qualification (validity + clean Base failure) from behavioral verdicts; Base-gate failures retain the Round-0 patch in the full denominator (Table 9c), and forbidden test modification earns zero reward (Table 8d), closing the obvious gaming channel.
Staged, attributable training gains
Exceptional reproducibility documentation
Transparent self-caveats
Weaknesses
No statistical uncertainty anywhere
Headline number depends on privileged training feedback
Compute confound acknowledged but unquantified
Proxy metrics for test quality
Thin generalization evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ExecCritic (arXiv:2609.09133) is a method paper for repository-level automated repair. Its central diagnosis is trajectory-level coupling: when one agent trajectory produces both the source patch and the validation evidence, a shared misinterpretation lets a wrong patch pass a wrong test (Section 2, Eq. 1). The remedy is a test-verify-revise scaffold — Test agent, fail-closed harness, Repair agent — plus a role-specific RL recipe. Headline claims: (1) generated-test feedback helps or harms the same Repair agent depending on test quality (61.2% → 57.3% with base-model tests vs → 65.3% with GPT-5.6-sol tests, Repair held fixed, Figure 5b / Table 2); (2) role-specific post-training lifts Qwen-3.5-35B-A3B Test-agent Base-to-Gold success from 22.2% to 62.2% (Table 1); (3) the composed system reaches 72.6% on SWE-bench Verified, +11.4pp over the no-test 61.2% baseline. Scope is one benchmark family (Verified; Pro and Multilingual in secondary analyses), one backbone for the trained roles, and three-run averaged repair evaluation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does giving a coding agent its own tests always improve repair success?
No. ExecCritic shows that with the base Qwen repair agent held fixed, tests generated by the same base model reduce SWE-bench Verified resolution from 61.2% to 57.3%, because weak tests encode wrong behavioral targets and can validate a wrong patch. Only higher-quality tests help — tests from GPT-5.6-sol raise resolution to 65.3%. Test quality, not test presence, determines whether execution feedback helps.
What score does ExecCritic achieve on SWE-bench Verified?
The composed system with both post-trained Qwen-3.5-35B-A3B agents reaches 72.6% resolved, up 11.4 points from the 61.2% no-test baseline, without stronger-model or oracle feedback at evaluation time. Note the repair agent's training used a fixed oracle fail-to-pass test as feedback source; training with generated tests instead scores 70.9% after revision.
How does ExecCritic stop the repair agent from gaming the generated test?
The test bundle is produced by a separate Test agent, qualified by a fail-closed harness (artifact validity plus a clean failure on the buggy checkout), then frozen: the Repair agent cannot see or edit test sources, only receives bounded execution feedback, and any forbidden test modification earns zero reward during training. Final correctness is decided by the official evaluator.
Is ExecCritic's code available?
Yes, at github.com/MSR-Orchard/execcritic. The release covers launchers, rewards, the test-submission protocol, repair loops, and evaluation drivers, with verbatim prompts documented in the paper. Datasets, model checkpoints, runtime installations, and sandbox credentials remain external, so full replication requires substantial additional infrastructure.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked '[...sampling gap...]' sections; the values of Table 4 and Table 5 (cross-language Test-agent analysis) and parts of Sections 5.3 and A.4-A.5 were skipped, so no conclusion in this review relies on those skipped contents.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.