GraphEcho: Structural Redundancy and Evidence Provenance in LLM Graph Agents
Sikun Wang, Yixi Zhou, Lei Fan, Fan Zhang
cs.AI · cs.CL
Quick verdict
Controlled benchmark separating path redundancy from provenance in LLM graph agents; rigorous stats, but one training seed and narrow models temper conclusions.
Summary
GraphEcho is a benchmark probing whether LLM graph agents mistake multiple graph paths from one evidential origin for independent corroboration. Using 800 fictional synthetic claims across four domains, it independently varies path count and provenance count under matched path templates and wording, evaluating both static judgments (path-multiplicity effect PME, independent-source effect ISE) and active exploration within a six-walk budget (echo walk rate, source recall). A SciFact conversion (188 claims, 584 instances) tests transfer. Static effects are model-dependent: Qwen3-4B responds far more to redundant paths (PME5 +7.47) than to distinct origins (ISE5 +0.42), while GPT-5-mini shows the reverse and flips verdicts 23.6% of the time. All four frozen agents raise echo walk rates under redundant support (+5.80 to +38.84 pp) while accuracy moves in both directions. Provenance-aware post-training (PAPT: GRPO on Qwen3-4B) adds +41.5 pp synthetic accuracy and cuts echo rate by ~37 pp, but lowers source recall and loses 3.55 pp SciFact accuracy. Provenance metadata and counting instructions barely change behavior; collapsing equivalent paths (M3) nearly eliminates redundancy sensitivity without training. Limitations include a single training seed, convention-based equal-weight verdict labels, and no multi-source acquisition test on real scientific data.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Redundant supporting paths (same origin, more paths) raise echo walk rates in all four evaluated frozen agents (+5.80 to +38.84 pp), while verdict accuracy moves in both directions.
- Static sensitivity to path redundancy versus distinct origins is model-dependent: Qwen3-4B responds mainly to redundancy (PME5 +7.47 vs ISE5 +0.42), whereas Qwen3-8B and GPT-5-mini respond mainly to origins.
- Provenance-aware post-training raises synthetic agent accuracy by 41.5 pp over frozen Qwen3-4B and cuts echo rate by ~37 pp, but lowers source recall and loses 3.55 pp on SciFact transfer.
- Inference-time provenance metadata and counting instructions barely change behavior, while collapsing equivalent paths (M3) nearly eliminates redundancy sensitivity at much lower cost than training.
- Reduced repetition and effective evidence use are distinct objectives: an agent can learn to stop repeating itself while leaving needed distinct evidence unvisited.
Strengths
Genuinely controlled intervention design
The sSkP notation (Section 3.1) and Figure 1 manipulations hold path templates, evidence wording, and path lengths fixed while changing only provenance assignment, cleanly separating structural multiplicity (Npath) from source multiplicity (Nprov) — a distinction prior RAG-redundancy work (Ross et al., 2026) conflates. Construction checks in Appendix A verify atom/path counts and matched content.
Acquisition metrics expose what accuracy conceals
The EWR/USR/UPC decomposition (Section 5.3) reveals the paper's most interesting finding: PAPT cuts echo rate by 36.9 pp partly by stopping early, reaching only about one-third of C3's six origins versus more than four-fifths for the frozen policy (Section 6.3). Section 6.6's stopping-ambiguity analysis (C2 vs C3 identical observed pairs, different unseen evidence) explains why correct source counting can coexist with wrong verdicts.
Informative cheap-control baselines
Statistical care unusual for benchmark papers
Reward ablations show competing objectives
Weaknesses
Narrow model coverage with mismatched decoding
All policy comparisons rest on one training seed
Transfer evaluation cannot test multi-source acquisition
Validity-confounded headline estimate for Qwen3-1.7B
No code or data availability statement
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the GraphEcho benchmark?
GraphEcho is a benchmark that tests whether LLM-based graph agents mistake multiple graph paths tracing back to the same evidential source for independent corroboration. It separately varies the number of paths and the number of distinct sources while keeping evidence wording fixed, using 800 fictional synthetic claims plus a conversion of the SciFact scientific claim-verification dataset.
Do LLM agents treat repeated evidence paths as independent support?
It depends on the model. All four evaluated agents (Qwen3-1.7B/4B/8B and GPT-5-mini) repeated more walks when redundant supporting paths appeared, but accuracy moved in both directions. In static judgments, Qwen3-4B was far more sensitive to redundant paths than to genuinely distinct sources, while Qwen3-8B and GPT-5-mini showed the opposite pattern.
What is PAPT (provenance-aware post-training) and did it work?
PAPT fine-tunes Qwen3-4B with GRPO using rewards for correct verdicts, accurate source counting, and distinct-source coverage, plus penalties for repeated evidence acquisition and cost. It raised synthetic accuracy by 41.5 percentage points and nearly eliminated repeated walks, but it also stopped exploring earlier: source recall dropped and SciFact accuracy fell by 3.6 points, showing that stopping repetition is not the same as using evidence well.
Can you fix redundancy sensitivity without fine-tuning?
Mostly not with prompting: displaying source identifiers or instructing the model to count distinct origins changed behavior very little. The most effective cheap intervention was preprocessing that collapses equivalent paths into one, which nearly eliminated redundancy effects and achieved near-perfect source recall in the benchmark environment.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 were analyzed only from captions and embedded alt-text; their quantitative content (e.g., per-condition curves in Figure 4, training trajectories in Figure 5) could not be inspected directly.
- No code or data availability statement was found in the extracted text; the reproducibility rating assumes the described protocol is accurate but the release status is unverified.
- Appendix content beyond Tables 6-12 (e.g., full confidence curves in Appendix B) is only summarizable from in-text references.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.