ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement
Siwei Wu, Jincheng Ren, Yizhi Li, Haau-Sing Li, Chengran Yang, Yuxuan Zhang, Weicheng Gu, Jian Yang, et al.
cs.CL
Quick verdict
Benchmark-disjoint, modular harness RSI shows consistent TB2.0/SWE gains; sound design, but single-run results and missing contrastive ablation weaken claims.
Summary
ModularRSI addresses generalizable recursive self-improvement (RSI) of agent harnesses — the scaffolding governing agent loops, tool use, and context handling — via three coupled ideas. First, evolution runs on a benchmark-disjoint dataset: the authors curate 2,000 executable Harbor-format tasks from external sources (GitHub, Hugging Face, Kaggle, Linux docs) with multi-stage quality control including executable validation, no-op-reward checks, and LLM similarity screening against downstream benchmarks. Second, contrastive trajectory analysis pairs successful and failed rollouts of the same task, aggregating cross-task evidence to isolate systematic harness deficiencies rather than task-specific fixes. Third, the harness is decomposed into five modules (Agent Loop, Tool Use, Observation Management, Context Management, Task Completion Detection) evolved independently under validation gates, then integrated. Evolving on 120 TB-related and 120 SWE-related tasks with DeepSeek-V4-Flash models, the frozen harness lifts TerminalBench 2.0 accuracy from 47.57 to 52.43 and SWE-Bench Verified from 73.40 to 76.45 in-domain, with positive cross-domain transfer in both directions and consistent gains across GLM-5.2 and MiniMax-2.5 backbones. Joint or non-modular evolution underperforms the baseline (44.19/46.44 vs 47.57), supporting the modularization argument. Limitations: no dedicated ablation isolating the contrastive component (acknowledged), no significance testing or variance reporting on small benchmarks (89 TB2.0 tasks), and only 240 of 2,000 curated instances actually used for evolution.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Contrasting successful and failed trajectories of the same task, aggregated across tasks, converts coarse task-level rewards into localized, generalizable harness-improvement signals (Sec. 3.3).
- Independent module-wise evolution followed by cross-module integration outperforms joint or whole-harness evolution, which falls below baseline (Table 4: 52.43 vs 44.19/46.44 vs baseline 47.57 on TB2.0).
- A harness evolved on a benchmark-disjoint set improves frozen-harness accuracy on unseen in-domain and cross-domain evaluations (TB2.0 47.57→52.43; SWE-Bench Verified 73.40→76.45) and transfers across foundation models (Table 3).
- Under the benchmark-disjoint protocol, prior RSI methods (AHE, Meta-Harness) improve the Terminus-2 baseline by ≈1 pp while ModularRSI gains ≈5.6 pp (Table 6: 67.42 vs 62.54/62.92 vs 61.79).
Strengths
Clean controlled evidence for modularization
Table 4 shows joint all-module evolution (44.19) and non-modular evolution (46.44) both fall below the 47.57 baseline on TerminalBench 2.0 while ModularRSI reaches 52.43 — a direct test of the interference hypothesis, not a bare ablation.
Benchmark-disjoint protocol attacks contamination head-on
Sec. 4 curates 2,000 Harbor-format tasks from external sources with executable validation, a no-op-submission reward check, manual review, and LLM similarity screening; Table 1 positions this against 11 prior harness-RSI works that evolve on benchmark data.
Generalization tested along three axes
Unusual methodological transparency
Reliability and efficiency gains, not just accuracy
Weaknesses
No statistical rigor
Missing ablation of the title component
Baseline inconsistency across tables
Dataset contribution under-exercised
Cost parity in Table 6 is asserted, not measured
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ModularRSI (arXiv 2609.14857) is a method paper on recursive self-improvement (RSI) of agent harnesses — the scaffolding governing an LLM agent's loop control, tool invocation, observation filtering, context handling, and stop conditions. The authors target three coupled failure modes in prior harness RSI: evolution on benchmark-derived data (contamination), credit ambiguity in single-trajectory updates, and unlocalized modification of monolithic harnesses. Central claims: (1) contrasting successful and failed rollouts of the same task, aggregated across tasks, yields localized, transferable improvement signals (Sec. 3.3); (2) five-module decomposition with restricted modification scopes beats joint or whole-harness evolution (Sec. 3.2, Table 4); (3) a benchmark-disjoint 2,000-task evolution set enables genuine generalization testing (Sec. 4). Evaluation covers TerminalBench 2.0 (89 tasks) and SWE-Bench Verified (500 tasks), plus cross-domain transfer (Table 2), cross-model transfer (Table 3), and a comparison with AHE and Meta-Harness under the disjoint protocol (Table 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ModularRSI in simple terms?
ModularRSI is a framework that lets an LLM coding agent improve its own harness — the scaffolding code controlling its loop, tool calls, context handling, and stop conditions — by learning from execution experience. It contrasts successful and failed runs of the same task to find systematic harness weaknesses, evolves five harness modules separately, and keeps only changes that pass code checks, an overfitting review, and real execution tests.
How much does ModularRSI improve agent performance?
On TerminalBench 2.0 the evolved harness raises accuracy from 47.57 to 52.43 with DeepSeek-V4-Flash-Preview; on SWE-Bench Verified it rises from 73.40 to 76.45. It also helps other models (GLM-5.2: 59.55→61.80; MiniMax-2.5: 41.57→44.94) and transfers across domains — a harness evolved only on terminal tasks still lifts SWE-Bench Verified to 75.80. These are single-run results without reported significance tests.
How does ModularRSI avoid benchmark contamination?
The authors built a separate evolution set of 2,000 executable tasks sourced from GitHub, Hugging Face, Kaggle, and Linux documentation rather than from evaluation benchmarks. Each task passed executable validation, a check that a no-op submission earns no reward, manual review, and LLM-based semantic-similarity screening against downstream benchmarks. The evolved harness is frozen before evaluation.
Is the code and data available?
The paper states all code and the 2,000-task evolution dataset are available at github.com/IQuestLab/ModularRSI, with abridged evolution and review prompts in the appendices. Note that reported results used only 240 of the 2,000 curated tasks for the actual evolution runs, and replication depends on proprietary LLM APIs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-6 are only partially recoverable from the extracted text (axis labels and legends garbled); numeric values attributed to figures rely on inline text, and the Fig. 3 delta may be an extraction artifact.
- The metadata author list is shorter than the byline in the extracted text (which adds R. Batista-Navarro, C. Zhang, X. Liu, M. Zhou, B. Dai, C. Lin); this review reflects the embedded text.
- Appendix listings are abridged by the authors themselves, so prompt-level reproduction fidelity cannot be fully verified from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.