RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
Peng Xia, Rujun Han, Zifeng Wang, Yanfei Chen, Yufan Zhuang, Yoonho Lee, Chengsong Huang, Han Yu, et al.
cs.LG · cs.AI · cs.CL
Quick verdict
Adds proposal/selection regularizers to agent-harness self-improvement; OOD transfer gains look real across 8 benchmarks, yet no significance tests reported.
Summary
The paper studies overfitting in automated agent-harness evolution: an LLM proposer iteratively edits the prompts, control flow, tooling, and memory around a frozen backbone using feedback from a finite evolve set, a practical form of recursive self-improvement. RRSI regularizes both sides of this loop. Proposal-side, a cosine-annealed budget caps bundled edits per round (an L0-style analogy), credit assignment conditions on the full edit history, and stalled searches explore untried components. Selection-side, a critic rejects benchmark-specific edits before scoring, acceptance requires clearing a noise-calibrated floor, cost growth must be justified by measured gain, and unproductive components are pruned. Across eight benchmarks in coding, agentic workspace, and engineering design, RRSI trades smaller evolve-set gains (+6.0 on Terminal-Bench 2.1, +1.1 on Harvey LAB with Claude Opus 4.8) for consistent held-out improvements: +1.8 on SWE-bench Verified, +2.3 on Harvey's in-distribution hold-out, +3.5 to +4.7 on three OOD agentic suites, and +4.3 on Frontier-Eng, while using about 30% fewer policy tokens than unregularized evolution (2.42 vs 3.80 million per trial). Four baselines keep little OOD gain; ablations show both regularizer groups drive transfer. Results replicate under Gemini 3.5 Flash (+14.1 evolve) and the harness transfers to an unseen weaker model. Limitations: no significance tests or seed variance, judge-policy entanglement in agentic suites, and many hand-set hyperparameters.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Unregularized harness evolution achieves the highest evolve-set score (92.8 on Harvey LAB) but its OOD average (40.3) lands within one point of the unevolved harness (39.7), demonstrating adaptive overfitting.
- RRSI's proposal-side (annealed edit budget, history-aware credit, structured exploration) and selection-side (leakage critic, noise floor, cost rule, pruning) regularizers shift the trade from evolve-fit to transfer, improving all six held-out splits by up to +4.7 OOD points.
- RRSI's harness is the cheapest of all evolved arms at 2.42M policy tokens/trial, about 30% below unregularized evolution (3.80M), and dominates all four baselines in the cost-transfer plane.
- The gains are policy-agnostic: they replicate under Gemini 3.5 Flash (+14.1 evolve, +2.2 SWE-bench) and the coding harness transfers to an unseen weaker backbone (Terminal-Bench 11.2→14.6 with Gemini 3.1 Flash Lite).
Strengths
Well-grounded problem framing
Sec 2 casts harness evolution as adaptive empirical optimization over a reused evolve set (Eq. 2) and connects it to adaptive data analysis (Dwork et al., 2015); the failure mode is demonstrated, not just asserted — Fig 1(a) shows prior methods' evolve gains do not transfer, several ending below H0.
Unusually careful evaluation hygiene
Fixed evolve/ID/OOD partitions (Harvey LAB 120/40 of 160, App A.3), all arms re-measured against H0 in the same evaluation window (Sec 4.2), leakage critic runs before scoring so leaking candidates never earn inflated evolve credit (Sec 3.3), EngDesign's 61 evolve tasks are excluded from Frontier-Eng (App A.8), and the deterministic-verifier engineering domain closes the judge-mediation route (Sec 4.2).
Ablations directly support the mechanism
Strongest anti-artifact evidence in this subfield
Internal numerical consistency
Weaknesses
No inferential statistics anywhere
Judge-policy entanglement in the agentic domain
Claim-level precision lapses
Reproducibility barriers and unverified baselines
Many interacting hand-set rules, no sensitivity analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a well-defined failure mode: automated harness evolution — an LLM proposer iteratively editing prompts, control flow, tools, and memory around a frozen backbone — behaves like adaptive empirical optimization on a small, reused evolve set (Sec 2, Eq. 2), and prior methods show evolve-set gains that shrink or invert out of distribution (Fig 1a: AHE and TTHE end below H0). The authors propose RRSI, which keeps the edit space open but regularizes the search trajectory: proposal-side (cosine-annealed edit budget, full-history credit assignment, stall-triggered exploration) and selection-side (pre-scoring leakage critic, noise-calibrated acceptance floor, cost-justified growth, windowed pruning). Three headline claims: (1) unregularized evolution's OOD average (40.3, Table 2) lands within a point of H0 (39.7) despite the best evolve score (92.8); (2) RRSI improves all six held-out splits — +1.8 SWE-bench Verified, +2.3 Harvey ID hold-out, +3.5 to +4.7 on three OOD agentic suites, +4.3 Frontier-Eng (Fig 3) — at 2.42M policy tokens/trial, ~30% below unregularized evolution's 3.80M; (3) gains are policy-agnostic (Tables 3–4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RRSI?
RRSI (Regularized Recursive Self-Improvement) is a Google Cloud AI Research framework that lets an LLM agent improve its own harness — the prompts, control flow, tools, and memory around a frozen model — while adding regularization constraints (edit budgets, leakage screening, noise-adjusted acceptance floors, cost rules, pruning) so improvements transfer to unseen benchmarks instead of overfitting the tasks used during evolution.
Why do self-improving agents overfit their training tasks?
Because the evolution loop repeatedly reuses feedback from the same finite task set: candidates proposed in later rounds depend on earlier measurements of the same tasks, so the search can encode benchmark-specific patterns, chase evaluation noise, or accumulate complexity. The paper shows unregularized evolution reaches the highest evolve-set score (92.8) yet scores out of distribution about the same as the untouched starting harness (40.3 vs 39.7).
Does the evolved harness work with models it was not optimized for?
Partially yes. A coding harness evolved with Gemini 3.5 Flash still raised Terminal-Bench 2.1 accuracy from 11.2 to 14.6 when run with the smaller, never-seen Gemini 3.1 Flash Lite, and gains replicated across two policy families (Claude Opus 4.8 and Gemini 3.5 Flash), though absolute gains shrink with weaker backbones.
Is RRSI's code available?
The paper points to github.com/google-research/rrsi and a project website (regularized-rsi.com) hosting round-by-round evolution records with harness diffs; this review could not verify what is actually released. Independent replication requires paid frontier-model APIs (Claude Opus 4.8 / Gemini) over thousands of agent rollouts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are only partially parseable from extracted text; Figure 1(a) scatter and Figure 4 panel values were read from embedded numeric labels only.
- External artifacts (github.com/google-research/rrsi, regularized-rsi.com) could not be checked; codeAvailable reflects the paper's stated links, not verified contents.
- The arXiv v2 text appears complete through Appendix E, but any supplementary material beyond it was not available.
- The '22.9%' figure in Sec 1 could not be reconciled with Table 1; it is flagged as unverifiable rather than judged an outright error.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.