Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures
Harsh Raj, David Lee, Anas Mahmoud, Renxiong Wang, Razvan-Gabriel Dumitru, Chenguang Wang, Tong Zhao, Yunzhong He, et al.
cs.AI · cs.HC · cs.LG · cs.SE
Quick verdict
Continual Search iteratively pushes LLM judges to keep searching; solid gains on long traces (F1 0.349→0.498), but prompt asymmetry and weak stats temper it.
Summary
The paper reframes root-cause attribution (RCA) for LLM-agent failures as an evidence-search problem. Single-turn LLM judges, it argues, commit early and leave most of a large execution log unread. Continual Search continues the same judge session over four turns, instructing it to challenge its standing diagnosis and inspect previously unexamined evidence; the control, Passive Continuation, merely asks it to reconfirm. Across three long-horizon benchmarks (MegaRCA-Mix, TRAIL, TELBench) and two frontier judges, Continual Search outperforms both the single-turn baseline and Passive Continuation: GPT-5.5's MegaRCA-Mix F1 rises from 0.349 to 0.498 (Passive 0.401), and TRAIL Weighted F1 from 0.426 to 0.500. Gains track deduplicated observation tokens and artifact coverage (70.8%→97.4%), and exceed self-consistency and heterogeneous judge panels despite lower cost than self-consistency. The authors also introduce MegaRCA-Mix's design: 50 human-annotated Harbor Index trials with a median 286K-token record. Lower-tier judges with Continual Search approach top-tier performance (Sonnet-5 ≈ Fable-5 on TRAIL). On short trajectories (AgentRx, Who&When), continued pressure degrades accuracy (e.g., Who&When 0.548→0.452). Limitations: no significance testing despite n=50 on the headline benchmark; heavily engineered, asymmetric continuation prompts; label noise acknowledged; no code or data release statement.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- One-shot LLM judges commit to a diagnosis early, so root-cause attribution over long execution logs is better modeled as an iterative evidence-search problem.
- Continual Search improves attribution on all three long-horizon benchmarks for both judges (e.g., GPT-5.5 MegaRCA-Mix F1 0.349→0.498 vs 0.401 for Passive Continuation; TRAIL Weighted F1 0.426→0.500) and beats self-consistency and heterogeneous judge panels (Table 4).
- The gains come from acquiring previously unexamined evidence — deduplicated observation tokens and artifact coverage (70.8%→97.4%) — not from extra test-time compute, since self-consistency costs more and regresses (Table 4).
- Search substitutes for model scale: with Continual Search, Sonnet-5 matches Fable-5 on TRAIL at ~76% of its compute, and GLM-4.7 rises from 0.268 to 0.473 Weighted F1 (Section 4.2).
- On short trajectories where turn 1 already reads everything, continued pressure flips correct answers and degrades accuracy (Table 3), delimiting when the method applies.
Strengths
Gains are tied to a measured mechanism, not just scores
Section 5.3/Fig. 7 shows deduplicated (13-gram) observation tokens grow 42K→58K under Continual Search vs only 48K under Passive on TRAIL; Appendix A.1 shows artifact coverage rising 70.8%→97.4% while Passive plateaus at 77.0%; the GAIA2 qualitative case (Fig. 10) shows an unread config.json overturning a trajectory-only misattribution.
Honest negative results consistent with the paper's own theory
Table 3 reports that on short trajectories GPT-5.5 degrades under continuation (AgentRx 0.345→0.241; Who&When 0.548→0.452), and Opus-4.8 regresses on Who&When (0.373→0.325) — exactly what the search-framing predicts when no unread evidence remains.
Unusual transparency for a prompt-based method
MegaRCA-Mix addresses a real evaluation gap
Compute-aware baseline comparison
Weaknesses
Prompt-asymmetry confound at the heart of the comparison
No inferential statistics anywhere
MegaRCA-Mix annotation reliability undocumented
Reproducibility depends on unreleased assets and proprietary judges
Cost multiple is real and only implicitly traded off
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This Scale AI paper targets root-cause attribution (RCA): given a failed agent execution record, identify the first consequential error and assign the fault (model, harness, environment, grader). The core claim is that over logs reaching hundreds of thousands of tokens, one-shot LLM judges commit early and leave most evidence unread, making RCA fundamentally a search problem. The proposed fix, **Continual Search**, keeps the same judge session alive for four turns and instructs it to challenge its standing diagnosis and examine unread evidence; **Passive Continuation** (ask the judge to reconfirm) is the control. Claims are tested on four public benchmarks (TRAIL, TELBench, AgentRx, Who&When) plus a new benchmark, **MegaRCA-Mix** (50 Harbor Index trials, human-annotated, median 286K tokens), with secondary claims that search compensates for model scale and that a taxonomy classifier under Continual Search becomes a competitive benchmark auditor (Section 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Continual Search for agent failure analysis?
It is a prompting framework (Scale AI, arXiv 2609.13463) that keeps an LLM judge's session open over four turns and instructs it, each turn, to challenge its current root-cause diagnosis and examine evidence it has not yet read — instead of accepting a one-shot judgment over a possibly huge execution log.
How much does Continual Search improve root-cause attribution accuracy?
On MegaRCA-Mix it raised GPT-5.5's F1 from 0.349 to 0.498 after four turns (passive reconfirmation reached only 0.401); on TRAIL it raised GPT-5.5's Weighted F1 from 0.426 to 0.500. But on short trajectories (AgentRx, Who&When) it reduced accuracy, since there is little left to read.
What is the MegaRCA-Mix benchmark?
A set of 50 failed agent trials drawn from Harbor Index execution logs, each human-annotated with the first consequential error under an interaction-centric failure taxonomy. The median execution record is about 286K tokens — far larger than prior RCA benchmarks (2.4K–100K tokens) — and includes trial configs, verifier outputs, and sandbox artifacts.
Does a stronger or more expensive LLM judge fix long-trace diagnosis without Continual Search?
Not according to the paper: self-consistency (four independent samples) cost more per trajectory ($8.66 vs $6.24 for GPT-5.5 on TRAIL) yet scored below the single-pass baseline, and raising per-turn reasoning effort showed flat or inverse returns. Reading previously unread evidence, not extra compute, drove the gains.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text ends immediately after Appendix B.3's Eq. (7); any content beyond that point (e.g., further appendix sections or acknowledgments) is absent from the extraction — nothing before it appears cut mid-sentence.
- Figures are present only as embedded axis/label text, so per-turn curve values (Figs. 3–6) cannot be read precisely; turn-wise numbers cited in this review come from body text and Tables 2–5, which are complete.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.