Ecdysis: Efficient and Effective Training of Runtime Harnesses for LLM Agents
Ruiqing Yue, Yu Cui, Zhuoyu Sun, Sicheng Pan, Xianhong Xue, Tingyu Li, Ting Li, Wenzhuo Zhu, et al.
cs.SE · cs.AI
Quick verdict
Cross-task failure aggregation + multi-agent refinement improves LLM harness evolution; solid ablations, but a single baseline and no significance testing.
Summary
The paper introduces Ecdysis, a framework for training the runtime harnesses of LLM agents through failure-driven evolution. Its premise: an observed failure may stem from model limitations (warranting model-specific accommodation, MSA) or genuine harness deficiencies (harness-level repair, HLR), and single failures cannot reliably distinguish the two. Ecdysis aggregates failure evidence across task instances each evolution round, prioritizes patterns recurring across distinct tasks, and runs a collaborative refinement stage (FDCR: Analyst, Critic, Engineer, Moderator) to produce structured modification specifications before a coding agent edits the harness. Trained on Qwen3-8B over tau2-Airline, tau2-Retail, and AgentBench, then frozen and evaluated across five LLMs, Ecdysis (w/ FDCR) reaches 69.56% average held-out accuracy versus 58.67% for Self-Evolution (+18.56% relative) and 60.22% for the human-optimized baseline, with up to 1.84x faster training and 12.78% lower inference token consumption. The honest ablation shows aggregation alone raises pooled MSA to 75.61% (94.44% on Airline); FDCR cuts it to 45.45%. Limitations: the headline comparison uses a single evolution baseline; no significance tests on roughly sixty-task subsets with three trials; MSA/HLR labels were annotated solely by five authors; some inference runtimes regress (e.g., Llama-3.1-8B on Airline); and the five-task reduced-data variant drops Pass^3 from 60% to 45%.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Failures recurring across distinct task instances provide stronger inductive evidence of systematic harness deficiencies than isolated failures.
- Ecdysis (w/ FDCR) raises average held-out accuracy to 69.56% versus 58.67% for Self-Evolution (+18.56% relative) across 15 model-dataset combinations.
- Batch-level failure aggregation speeds harness training (up to 1.84x on tau2-Airline) and reduces inference-time token consumption by 12.78% versus Self-Evolution.
- Aggregation alone increases model-specific accommodation (pooled MSA 75.61% vs SE's 60.00%), while adding FDCR reduces it to 45.45% and improves cross-LLM generalization.
- Recurrence-aware curation trains on five tasks at $0.291 versus $5.763 for the full set, with near-full average accuracy.
Strengths
Clear, actionable conceptual framing (MSA vs HLR)
Section 3.1 formalizes the split between model-specific accommodation and harness-level repair via the parameter t, giving the field a reusable lens on why naive failure-driven evolution misevolves; the SE-below-initialization result (Qwen3-8B tau2-Airline: 35.00% vs Human-Aug. 55.00%) substantiates it empirically.
Honest, informative ablation
Table 10 shows aggregation alone worsens pooled MSA to 75.61% (94.44% on tau2-Airline) versus SE's 60.00%; reporting that the proposed aggregation hurts without FDCR is transparent and strengthens the causal story.
Genuine cross-LLM transfer protocol
Exemplary cost and process accounting
Novel diagnostic use of training tasks
Weaknesses
Headline claim rests on one evolution baseline
No statistical testing on small, high-variance subsets
Central MSA metric is author-annotated
Efficiency narrative hides per-cell regressions
Data-efficiency and robustness claims outrun the evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Ecdysis (arXiv:2609.11677v2) addresses failure-driven self-evolution of runtime harnesses for LLM agents — the surrounding code (planning, tool interaction, context management) that is iteratively edited while model weights stay frozen. Its thesis: an observed failure may stem from model limitations (model-specific accommodation, MSA) or genuine harness deficiencies (harness-level repair, HLR), and single failures cannot distinguish the two. The paper claims three things: (1) cross-task failure recurrence is an efficient evidence-selection signal; (2) combining batch aggregation with Failure-Driven Collaborative Refinement (FDCR) yields +18.56% relative accuracy over Self-Evolution (SE), up to 1.84x faster training, and 12.78% fewer inference tokens; (3) the combination cuts the share of MSA decisions from 75.61% to 45.45%. The abstract hedges correctly ('up to'), but 'over existing harness evolution' generalizes from exactly one baseline.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Ecdysis and what problem does it solve?
It is a framework that trains an LLM agent's runtime harness (the surrounding planning/tool/context code) through failure-driven evolution while model weights stay frozen. Before editing the harness, it aggregates failure evidence across multiple task instances and only acts on failure patterns that recur across distinct tasks, then refines the proposed changes via a multi-agent analysis stage (FDCR).
How much better is Ecdysis than prior harness-evolution methods?
Across 15 combinations of five LLMs (Qwen3-8B/14B/32B, MiniMax-M2.7, Llama-3.1-8B) and three benchmarks (tau2-Airline, tau2-Retail, AgentBench), the paper reports 69.56% average held-out accuracy versus 58.67% for the Self-Evolution baseline, a relative gain of 18.56%, plus training speedups up to 1.84x and 12.78% fewer inference tokens. The comparison covers only that one evolution baseline.
What do MSA and HLR mean in this paper?
MSA (model-specific accommodation) is a harness change that papers over a limitation of the current model; HLR (harness-level repair) fixes a genuine defect in the runtime mechanism itself. Ecdysis's goal is to shift evolution from MSA toward HLR: in the paper's audit, MSA drops from 75.61% to 45.45% of modification decisions when its full method is used.
Is code available for Ecdysis?
The paper points to github.com/cuiyu-ai/Ecdysis. All models are accessed via APIs at temperature 0 (task models include Qwen3, MiniMax-M2.7, Llama-3.1-8B; the coding agent uses DeepSeek-V4-Pro), so reproduction depends on those endpoints remaining available.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Provided metadata lists 8 authors; the extracted paper byline lists 14 (adds Yi Chen, Yifei Liu, Baohan Huang, Zhe Cui, Haibin Zhang, Cong Zuo).
- The embedded text runs from abstract through Appendix E.3 and appears complete; the truncated metadata abstract ('...evolution wh') is a metadata artifact, and the full abstract is present in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.