SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness
Haozhe Liu, Tian Ye, Sensen Gao, Qihang Cao, Yitong Li, Yitong Li, Mingchen Zhuge, Duomin Wang, et al.
cs.AI
Quick verdict
Auto-research finds a harness matching Pi at ~45% less tokens on EdgeBench; solid isolation protocol, but single-run stats and TB4 capability drop temper it.
Summary
SoL-Pi is an agent harness discovered by an AI-driven auto-research loop that optimizes token efficiency at the harness layer rather than the model layer. A research agent inspects execution traces of the Pi harness, proposes candidate mechanisms, and validates them across roughly 535 executable environments (495 GitHub issue-PR repository tasks plus 40 verifier-driven synthetic tasks), spanning about 3,000 runs and 60,000 interactions. Four mechanisms survive pre-registered capability and efficiency gates: Action Fusion, Online Context Compact, ObservationPack, and Evidence-Preserving Reducer. On the 51-task EdgeBench, SoL-Pi stays within ~2.8 points of Pi's scores while cutting recorded token traffic by 44.7-49.0% and API cost by roughly a third across GPT-5.6 Sol and Opus 5, and transfers to the unseen Opus 5 backend without re-search. However, on Terminal-Bench 4 SoL-Pi solves 15/63 tasks versus 18/63 for Pi and Codex, and on IMO 2026 it passes 3/6 versus 5/6 for Codex. Main limitations: all results are single-run point estimates without variance or significance tests, the search's own API cost is unreported, and cost conclusions depend on August 2026 API pricing.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An AI-driven auto-research loop over 535 executable environments (~152 directions, ~3,000 runs, >60,000 interactions) discovers four harness mechanisms that survive pre-registered capability and efficiency gates.
- On 51-task EdgeBench, SoL-Pi matches Pi within ~2.8 score points while reducing recorded token traffic by 44.7-49.0% and API cost by about one third across GPT-5.6 Sol and Opus 5.
- Mechanisms developed solely on GPT-5.6 Sol transfer to Opus 5 without re-search, retaining 94.3% of Pi's average score (Table 2).
- Efficiency gains extend to Terminal-Bench 4 and IMO 2026 (lower cost per solved/passed task) and to multi-agent swarms (26.8% cost reduction vs a Pi-worker swarm, though the single agent is cheapest).
Strengths
Search-evaluation isolation is rigorously engineered
Capability metrics, tolerances and efficiency metrics are fixed before search and 'strictly isolated from the optimizing agent's control' (§2.1); EdgeBench is frozen and evaluated only after candidate freeze, with failed validation rejecting candidates without feedback (Fig. 2-3) — directly addressing the harness-overfitting failure mode documented by Wang et al. [20].
Add-one ablation with transparent activation analysis
Table 4 shows every single mechanism reduces total tokens under both backends; Figures 6-7 report trigger rate, trigger intensity and per-trigger efficiency gain, and honestly note that interaction effects are not isolated and that ObservationPack overlaps with Evidence-Preserving Reducer.
Cross-backend transfer evidence
Honest framing in several places
Consistent cost accounting across benchmarks
Weaknesses
No statistical uncertainty anywhere
Capability regression beyond EdgeBench is underweighted
Search cost never accounted
Mechanism specification depth is thin in the main text
Pricing- and cache-sensitive conclusions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SoL-Pi (arXiv 2609.20519, NVIDIA/NTU/MIT) claims that RSI-inspired auto-research loops, run at the agent-harness layer, discover token-efficiency mechanisms that transfer beyond their development environments. The search spans ~152 proposed directions across six families, 535 executable environments (495 GitHub issue–PR repository tasks with hidden fail-to-pass regression tests; 40 verifier-driven synthetic tasks), ~3,000 runs, and >60,000 agent–environment interactions (§1-2.3). Four mechanisms survive: Action Fusion, Online Context Compact, ObservationPack, and Evidence-Preserving Reducer (§2.4). Central claims: on 51-task EdgeBench, SoL-Pi matches Pi within ~2.8 score points while cutting recorded token traffic 44.7-49.0% and API cost ~one third across GPT-5.6 Sol and Opus 5; mechanisms transfer to the unseen Opus 5 backend without re-search (94.3% of Pi's score, Table 2). Estimated hourly savings of $8.75-$13.50 vs native Codex/Claude Code are abstract-level projections from the same cost tables.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SoL-Pi?
SoL-Pi is an open coding-agent harness (built on the Pi harness) whose efficiency mechanisms were discovered automatically: an AI research agent inspected execution traces, proposed harness changes, and tested them across ~535 software environments. Four mechanisms survived: Action Fusion (merges an edit and its test/build command into one tool call), Online Context Compact (compacts context at plan-step boundaries under a cost gate), ObservationPack (stores observations outside the context), and Evidence-Preserving Reducer (compresses delegated reading while keeping decision-relevant evidence).
How much does SoL-Pi save in tokens and cost?
On the 51-task EdgeBench with GPT-5.6 Sol and Opus 5, SoL-Pi reduced recorded token traffic by 44.7-49.0% and API cost by roughly one third versus the Pi harness, at scores within ~2.8 points of Pi. Against native Codex/Claude Code harnesses the savings are larger (about 50-54% cost cut). The paper estimates hourly savings of $8.75-$13.50 vs native harnesses, using API prices from August 17, 2026.
Does SoL-Pi lose accuracy compared to Pi or Codex?
On EdgeBench it stays close to Pi (42.0 vs 44.8 average score on GPT-5.6 Sol; 42.2 vs 44.8 on Opus 5), and its 'Performance' configuration (one best mechanism) actually scores higher (47.2 and 50.5). But on Terminal-Bench 4 it solves 15 of 63 tasks versus 18 for both Pi and Codex, and on IMO 2026 it passes 3 of 6 problems versus 5 for Codex — so maximum solve-rate users may prefer the baselines.
How was SoL-Pi discovered, and could the gains be benchmark overfitting?
It was found by scaling AI-run auto-research loops over roughly 150 proposed directions and 535 executable environments (~3,000 runs), with capability metrics fixed before search and a strict freeze-then-evaluate protocol: the held-out EdgeBench was never used during search and failed validation rejected candidates without feedback. Cross-backend transfer to Opus 5 without re-search is the paper's main evidence against overfitting, though all results are single runs without statistical variance.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 text was extracted with heavy duplication and garbling; figure-level numbers were cross-checked against tables 1-3, but figure-only details may be imprecise.
- Appendix, prompts, and mechanism pseudocode are absent from the extracted text; implementation-level judgments are limited to the main text's narrative descriptions.
- All cost figures assume API prices as of Aug 17, 2026 (paper's table note) and could not be independently verified.
- Code and blog links are listed in the author block but their contents could not be verified offline.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.