EDGE: Error Dependency Graph-Guided Multi-Error Attribution in Multi-Agent LLM Systems
Jun Hou, Priya Pitre, Yi Fang, Xuan Wang
cs.AI
Quick verdict
Counterfactually validated error-dependency graph guides two-stage multi-error attribution; broad TRAIL/MAST gains but single-run, LLM-judge-mediated evidence.
Summary
EDGE is a method paper on multi-error attribution in failed LLM-agent traces. Instead of predicting a single root cause, the authors build a corpus-level error dependency graph over taxonomy categories: Suppes screening (temporal precedence plus probability raising) and CAPRI-style AIC pruning propose candidate edges, which are validated by counterfactual rollout — patching the source error span with category-specific templates, re-running or simulating the trace, and using two LLM judges for repair verification and downstream-effect estimation. Validated edges (GV) are unioned with score-filtered correlation edges into an inference graph Gτ that guides a two-stage LLM-as-judge detector: Stage 1 predicts an initial error set; Stage 2 verifies downstream errors suggested by trace-conditioned graph edges. On TRAIL (148 traces, span-level) and MAST (393 AG2 traces, trace-level multi-label), across eleven backbones, EDGE raises category-level F1 in most settings — e.g., GPT-oss-120B on TRAIL-GAIA from 25.76 to 37.20 — though Gemini-2.5-Flash loses 6.81 F1 on TRAIL-GAIA and MAST accuracy drops slightly for most models. An 80/20 held-out split and transfers to adapted Who&When-style prompts support generality. Main limitations: all results are single-run point estimates without significance testing; the 'causal' validation is LLM-judge-mediated (MAST validation yield 47.8%, ~70% not-observable labels); and localization gains stay near-flat on long SWE-Bench traces.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Framing multi-error attribution as dependency-aware diagnosis improves category-level multi-error attribution on TRAIL and MAST for most evaluated models and settings.
- A two-stage detector that conditions graph-suggested downstream hypotheses on Stage-1 predictions outperforms injecting the same graph as static single-pass context (+CG).
- Counterfactual rollout validation produces a trustworthy causal subgraph GV suitable for explanation, repair prioritization, and propagation analysis.
- The inferred graph transfers to unseen traces (80/20 split) and to adapted Who&When-style prompting strategies.
Strengths
The two-stage conditioning is properly ablated against graph access alone
Table 2's +CG control feeds the identical Gτ as a static single-pass context block and loses to EDGE in nearly every cell (e.g., GPT-oss-20B TRAIL-GAIA 12.49 vs 33.28 F1), showing the gain comes from trace-conditioned Stage-2 verification, not from mere graph presence — Appendix A.4 also explains the context-budget mechanism on long traces.
An intervention layer goes beyond co-occurrence, with honest role separation
Candidate edges are tested by patch-and-reroll with rule-based postchecks (~4% patch failure, Table 6), a repair verifier (Judge A) and an effect evaluator (Judge B); keeping Gτ for prediction and GV for explanation is an epistemically clean split the paper consistently maintains.
Breadth of evaluation and robustness checks
Candid limitations section
Weaknesses
No variance or significance reporting anywhere
'Causal' edges rest on thin, LLM-judged evidence
Counterintuitive cells under-analyzed
Headline graph is fit on the full evaluation corpus
MAST temporal order is auto-annotated and unvalidated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EDGE (arXiv 2609.01360, Virginia Tech) targets multi-error attribution: given a failed LLM-agent trace containing several interacting errors, predict the full set of error categories (and, for TRAIL, their span locations). The authors claim three things: (1) multi-error attribution is best framed as dependency-aware diagnosis rather than isolated per-error prediction; (2) a corpus-level error dependency graph — partially validated by counterfactual rollout — injected through a two-stage LLM-as-judge detector improves category-level attribution on TRAIL and MAST across most of eleven backbones; (3) the intervention-validated subgraph GV is a trustworthy explanatory artifact for repair prioritization. The paper positions itself against single-root-cause lines (Who&When, AgentDebug, CHIEF, CDC-MAS) and imports progression-modeling machinery from cancer genomics (Suppes screening, CAPRI, TRONCO).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the EDGE paper do?
EDGE is a framework for attributing failures in LLM agent systems: when a failed agent trace contains multiple related errors, EDGE builds a dependency graph over error categories (statistical screening plus counterfactual 'patch-and-rerun' validation) and uses it to guide a two-stage LLM judge that detects all errors in the trace, not just one root cause.
What benchmarks does EDGE evaluate on?
TRAIL — 148 agent traces from GAIA and SWE-Bench with span-level multi-error annotations (841 error instances, 19 categories) — and MAST/MAD — 393 AutoGen multi-agent conversation traces with trace-level multi-label annotations over 13 error categories.
Does EDGE improve error localization in agent traces?
Mostly it improves category-level detection. Location accuracy improves on TRAIL-GAIA (e.g., GPT-oss-120B Loc 16.92 to 28.00) but localization stays near-flat on the long-context SWE-Bench split; the authors state the graph helps more with detecting which errors occur than with localizing them.
Is EDGE's code and data available?
Yes — code, prompts, and patch libraries are at github.com/JuneHou/EDGE, built on the public TRAIL and MAST/MAD datasets. The authors caution that reported numbers are single-run LLM outputs and may not replicate exactly.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with three marked '[...sampling gap...]' breaks (within §4.1-4.2, after the references, and inside Appendix A.7); judgments relying solely on those regions were avoided.
- Figure content (Figs. 1-6) is available only through captions; the validated graphs' topology in Fig. 5 could not be inspected.
- PDF extraction garbles some table column alignment; cited numbers were cross-checked against surrounding prose and repeated appendix tables where possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.