SearchAtlas: Analyzing Agentic Search Strategies via Evidential Query Graphs
Jiacheng Sang, Mengyuan Li, Sanxing Chen, Yukun Huang, Yu Feng, Bhuwan Dhingra
cs.CL
Quick verdict
Validated query-DAG framework exposes search-agent process failures and beats LLM judges, but gold set is small, author-built, and no significance tests.
Summary
SearchAtlas converts LLM search-agent trajectories into query-to-query directed acyclic graphs whose typed edges — constraint-use, evidence-use, failure-response, and prior-knowledge — trace how retrieved evidence propagates from issued queries to the final answer. Construction pairs deterministic preprocessing (query/result parsing, provenance windows, failure detection) with LLM-based per-query attribution and minimal parent-set pruning. Against 100 human-adjudicated trajectory DAGs spanning three benchmarks and five agents, the pipeline reaches a macro edge F1 of 0.860 with GPT-5.2 attribution (0.814–0.860 across four constructor LLMs), whereas direct one-shot graph generation collapses to 0.072 F1. From these graphs the authors derive three diagnostics conditioned on question constraint type — answer-path topology, constraint grounding, and prior-knowledge reliance — and show across 1,350 trajectories from five agents (WebSailor, MiroThinker, TYDP, TYDP-GPT5, TYDP-Qwen3) on BrowseComp, WebWalker-Hard, and DeepSearchQA that a composite score separates correct from incorrect trajectories within each agent at macro ROC-AUC 0.840–0.856, beating GPT-5.2 judges given the raw trajectory (77.6% vs 73.8% accuracy; 0.772 vs 0.705 macro-F1) or the ordered query list. Ablations confirm DAG-localized grounding and answer-localized PK attribution outperform trajectory-wide variants. Key limitations: the gold set is small and built by the authors, the dominant evidence-use edges score only 0.769 F1, and headline comparisons lack confidence intervals or significance tests.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SearchAtlas automatically converts search-agent trajectories into evidential query DAGs with macro edge F1 0.860 against 100 human-adjudicated graphs, stable across four attribution LLMs (0.814-0.860).
- Three question-type-conditioned diagnostics (answer-path topology, constraint grounding, prior-knowledge reliance) composite into a score ranking correct vs incorrect trajectories within each agent at macro ROC-AUC 0.840-0.856 across 1,350 trajectories from five agents on three benchmarks.
- The DAG-derived thresholded score predicts correctness better than GPT-5.2 judges given the raw trajectory (77.6% vs 73.8% accuracy; 0.772 vs 0.705 macro-F1) or the ordered query list (67.7%).
- Process quality and answer accuracy carry complementary information: audited divergence cases show coherent-but-wrong support structures (high-scoring errors) and over-search or prior-knowledge shortcut successes (low-scoring correct runs).
Strengths
Evidence-gated construction validated against a substantial human gold package
The staged pipeline (deterministic preprocessing + provenance-gated LLM attribution, Appendix A.10) is contrasted with a direct one-shot GPT-5.2 baseline that collapses to 0.072 edge F1 and 0.455 repeat Jaccard (Table 17). Validation covers 100 human-adjudicated DAGs, 4,109 query nodes, 7,432 gold edge pairs across all four edge types (Tables 9/13), with per-type scores disclosed: constraint-use 0.957 F1, evidence-use 0.769, PK 0.758, failure-response 0.759 (Table 12).
Localization ablations isolate graph structure from raw search effort
Answer-backbone grounding beats trajectory-wide coverage (AUC 0.764 vs 0.459 on BrowseComp-sequential, Tables 4/31); answer-localized PK beats trajectory-wide PK by +0.063 to +0.119 AUC (Table 32); and the DAG score beats a compact raw-log classifier (query count, visits, trace length, redundancy) by +0.080 to +0.315 AUC (Table 35), ruling out the 'more search = better score' confound.
Constructor sensitivity and downstream robustness across attribution LLMs
Honest disclosure of costs, failure modes, and weakest channels
Weaknesses
No uncertainty quantification on any headline comparison
Author-built gold standard with no inter-annotator edge agreement
Dominant edge type is the noisiest, with no error-propagation analysis
Regime conditioning is partially circular and benchmark-coupled
Judge baselines use a single model, the same as the main constructor
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SearchAtlas (arXiv 2609.10901, Duke NLP) targets the gap between outcome-only evaluation of LLM search agents and process-level auditing. It converts a raw trajectory into a query-to-query DAG whose typed edges — constraint-use (q0→qi), evidence-use (qi→v), failure-response, and prior-knowledge (PK→v) — make evidence propagation explicit from question to answer. Three claims: (1) the two-stage pipeline (deterministic preprocessing + LLM attribution with minimal parent-set cover) reconstructs human-adjudicated graphs at macro edge F1 0.860; (2) three question-type-conditioned diagnostics (answer-path topology, constraint grounding, prior-knowledge reliance) composite into a score ranking correct vs incorrect trajectories at macro ROC-AUC 0.840–0.856 across five agents and three benchmarks; (3) this signal beats GPT-5.2 judges given the raw trajectory (77.6% vs 73.8% accuracy, 0.772 vs 0.705 macro-F1, Tables 3/27) or the ordered query list, and captures information not reducible to accuracy (Section 5.5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SearchAtlas?
SearchAtlas is a framework from Duke NLP that converts the full trajectory of an LLM search agent (queries, retrieved results, page visits, reasoning, final answer) into a directed acyclic graph of queries. Typed edges (constraint-use, evidence-use, failure-response, prior-knowledge) show how evidence flows from earlier queries to later ones and to the final answer, enabling post-hoc analysis of search strategy and localization of process failures to specific queries.
How accurate is SearchAtlas's automatic graph construction?
Against 100 human-adjudicated trajectory graphs spanning three benchmarks and five agents, the pipeline achieves a macro edge F1 of 0.860 (precision 0.895, recall 0.852) with GPT-5.2 attribution, and 0.814-0.860 across four different attribution LLMs. Prompting GPT-5.2 to build the graph in one shot reaches only 0.072 F1, showing the staged pipeline is essential.
Can SearchAtlas predict whether a search agent's answer is correct?
Its composite score of three graph diagnostics separates correct from incorrect trajectories within each agent at macro ROC-AUC 0.840-0.856 across 1,350 trajectories. Thresholded predictions reach 77.6% accuracy, higher than GPT-5.2 judges reading the raw trajectory (73.8%) or the ordered query list (67.7%), though the paper reports no confidence intervals for these margins.
Which agents and benchmarks does SearchAtlas analyze?
Five agent configurations — WebSailor (32B), MiroThinker (30B), Tongyi DeepResearch (30B), and the Tongyi scaffold with GPT-5 or Qwen3-32B backbones — run on BrowseComp (150 sequential-constraint questions), WebWalker-Hard-English (70 parallel-constraint questions), and DeepSearchQA (25 questions per regime), for 1,350 trajectories in total.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels (Figures 1-9) are images; only captions and some embedded metric values were extractable, so the qualitative DAG examples were assessed via captions and surrounding body text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.