RESKILL: Explicit Failure Attribution and Structured Repair for Interactive Language Agents
Mengyi Deng, Xin Li, Duyi Pan, Zilin Wang, Zhiwei Li, Zhijiang Guo, Wei Wang
cs.CL
Quick verdict
Proposes structured, hypothesis-weighted skill repair for LLM agents; gains are consistent across six settings but small and lack statistical testing.
Summary
The paper frames post-failure skill repair for LLM agents as a repair credit-assignment problem. RESKILL maintains an explicit repair state: the frozen model proposes weighted failure hypotheses, a pool of candidate local patches from four sources, and a binary coverage matrix linking patches to hypotheses; the framework selects the patch with maximal weighted coverage, retests it in the environment, and after a failed retest updates hypotheses and weights before the next round. Model weights stay frozen; only the external skill file is edited. Evaluation covers ALFWorld (274 tasks, repair budget R=7) and AgentGym TextCraft (R=4) with Qwen3-1.7B, Qwen2.5-3B, and Qwen3-4B, against Direct Repair and Hypothesis Repair under matched first-pass trajectories, skill files, and budgets. RESKILL attains the best final success in all six settings — e.g., ALFWorld overall 33.2 vs 30.7 (Direct) for Qwen3-1.7B, TextCraft 29.4 vs 17.5/22.8 for Qwen2.5-3B — averaging +3.7/+3.3 points over the baselines; gains shrink as first-pass accuracy rises (Qwen3-4B TextCraft: 50.7 vs 48.1). Limitations: no statistical testing or seed variance despite margins as small as 1.4 points; no ablation isolating coverage-based selection or the outcome-conditioned update; the coverage matrix is self-assigned by the same model; the TextCraft Qwen3-4B setting uses a different depth subset and transcript policy; and no code release is stated.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Post-failure skill repair should be treated as a credit-assignment problem over competing failure hypotheses, candidate patches, and retest outcomes.
- RESKILL achieves the highest final repair success in all six benchmark-model settings, averaging +3.7 points over Direct Repair and +3.3 over Hypothesis Repair.
- Explicit failure attribution alone is insufficient; durable improvement requires integrating attribution with coverage-based patch selection and persistent retest-conditioned updates.
- The advantage concentrates where failures admit multiple plausible explanations and shrinks as first-pass model accuracy approaches ceiling.
Strengths
Fully specified, auditable method
Algorithm 1, the variable table (Table 6), and the prompt contracts in Appendix F completely specify hypotheses, coverage matrix, selection rule, and outcome-conditioned update; the Table 5 TextCraft case study walks the audit trail end-to-end from evidence to selected patch and retest.
Controlled, honestly reported comparisons
Within each setting, Direct, Hypothesis, and RESKILL share first-pass trajectories, initial skill files, evaluator, and decoding configs (Sec 4.1, A.2-A.3); Sec 4.2 explicitly flags where Direct Repair wins (Qwen2.5-3B Pick 50.0 vs 47.0 and Clean 67.2 vs 58.6, Qwen3-1.7B Heat 30.8 vs 23.1).
Frozen-model design cleanly attributes gains
Budget-resolved result reporting
Directionally consistent effect across settings
Weaknesses
No uncertainty quantification
Mechanism and candidate-pool confound not separated
Self-referential coverage judgments
Heterogeneous TextCraft protocol for the 4B setting
Reference and artifact hygiene
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
RESKILL (arXiv 2609.15684, HKUST-GZ) targets post-failure repair of reusable agent skills. The authors argue that one-shot reflection (Reflexion, Self-Refine-style rewriting) conflates diagnosis, repair generation, and repair selection, and reframe repair as credit assignment over ambiguous failure explanations. Three central claims: (1) an explicit repair state — weighted failure hypotheses, a multi-source candidate patch pool, a patch-hypothesis coverage matrix, and retest-conditioned updates — improves repair; (2) RESKILL attains the highest final success in all six benchmark-model settings, averaging +3.7 points over Direct Repair and +3.3 over Hypothesis Repair (Table 1); (3) attribution alone is insufficient — durable gains require integrating it with selection and persistent updates.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RESKILL?
RESKILL is a framework for repairing the reusable skills of LLM-based agents after a failed task rollout. Instead of asking the model to rewrite skills in one shot, it keeps an explicit record of failure hypotheses (with weights), candidate skill patches, which patches address which hypotheses, and retest results, and uses that record to choose and update repairs across rounds. The model itself is never fine-tuned; only the external skill file changes.
How much does RESKILL improve over simpler repair baselines?
Across six benchmark-model settings (ALFWorld and TextCraft with Qwen3-1.7B, Qwen2.5-3B, and Qwen3-4B), RESKILL posts the best final repair success in each, averaging 3.7 percentage points above direct repair and 3.3 points above hypothesis-conditioned repair. The largest single gain is on TextCraft with Qwen2.5-3B (29.4 vs 17.5/22.8); margins are much smaller for the strongest model, e.g., 50.7 vs 48.1 on TextCraft with Qwen3-4B.
Is code or data available for RESKILL?
The paper does not state a public code release. However, it uses public benchmarks (ALFWorld and AgentGym TextCraft), open-weight Qwen models, and its appendices give detailed decoding parameters, evaluation configurations, and full prompt templates, so independent reimplementation is feasible though non-trivial.
Does RESKILL train or fine-tune the agent model?
No. The language model is frozen throughout; repair operates entirely on the external skill file via local patches (inserting or revising skill sections), and improvements come from better repair decisions plus environment retests under a fixed retest budget.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are available only as captions and axis tick text in the extraction; curve shapes were not independently verifiable.
- Appendix C is an empty heading in the extracted text, and Table 4's referenced reproduction commands and task-level logs are not actually included.
- Tables were reconstructed from linearized text; minor digit-alignment risk exists despite cross-checking against in-text numbers.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.