Benchmarking Hybrid Deep Research Across Database Querying and Web Search
Ruofan Wu, Peiran Xu, Xiaolong Li, Fan Shu, Soyoung Yoon, Yite Wang, Xiaodong Yu, Boyi Liu, et al.
cs.CL
Quick verdict
First web-search+SQL hybrid research benchmark; rigorous anti-leak construction and a convincing handoff gap, but LLM-judge scoring and no CIs cap precision.
Summary
The paper introduces HybridDeepResearch, a benchmark of 380 tool-dependent tasks requiring agents to combine web search with SQL querying over LiveSQLBench-Base-Lite databases, drawing web evidence from Wikipedia, Wikidata, and FineWeb-10BT. Tasks follow three patterns — SQL2S (a database anchor constrains subsequent search), S2SQL (a search-derived entity becomes a SQL predicate), and Parallel (exact intersection of two candidate sets) — with a balanced 120-example hard subset. Construction uses dual-grounded entity anchoring, role-aware symmetric generation from a 35-pattern SQL catalog, controlled clue blurring, and a 'hybrid lock' single-tool ablation that discards items solvable by one modality alone, followed by author review. Evaluating Qwen3.5 (4B-397B), GLM-5.2-FP4, GPT-5, and Claude-Sonnet-4.6 under smolagents and MiroFlow scaffolds with real-web versus fixed-corpus retrieval, eight trials per task, GLM-5.2 reaches 75.79 Pass@8 and 49.24 Avg@8 on the full set (MiroFlow, Corpus Search); on the hard subset GPT-5 tops Pass@8 at 54.17 and Claude-Sonnet-4.6 tops Avg@8 at 28.12. Directional patterns are markedly harder than Parallel (GLM-5.2: 62.07 vs 93.28 Pass@8). MiroFlow raises Pass@8 but not Avg@8, with a 41.3% timeout rate. Limitations include only 58 S2SQL items, author-only human review, an LLM judge without validated agreement, and no confidence intervals or significance tests.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- HybridDeepResearch is, to the authors' knowledge, the first deep-research benchmark requiring both web search and SQL execution for a complete, verifiable answer.
- Even frontier systems reach only about 50-54% Pass@8 (GPT-5 54.17, Claude-Sonnet-4.6 52.50, GLM-5.2-FP4 50.83) on the balanced 120-task hard subset, with overall Avg@8 below 30% (Table 3).
- Directional handoff patterns are substantially harder than Parallel intersection — e.g., GLM-5.2-FP4 under MiroFlow scores 62.07 S2SQL and 69.46 SQL2S vs 93.28 Parallel Pass@8 (Table 2).
- The multi-agent MiroFlow scaffold raises Pass@8 but not Avg@8, trading a 41.3% timeout rate and ~2.3x tool calls for broader exploration (Appendix G.1).
Strengths
Genuine gap with precise task semantics
Three patterns with explicit information-flow definitions (Section 3, Figure 2) target constraint-preserving handoffs that prior hybrid QA (HybridQA, TAT-QA; Appendix H.4) leaves open; both tools are strictly necessary, enforced by single-tool ablation.
Layered anti-shortcut construction
Closed-book prefilter, BM25 resistance, clue blurring ('2012'→'the 2010s'), answer-cardinality checks (Appendix B.1) and the 'hybrid lock' discard step (Section 4.3) directly attack single-source leakage — the classic failure mode of hybrid benchmarks.
Decomposed, broad evaluation
Honest cost accounting
Reproducibility-first design
Weaknesses
LLM-judge scoring unvalidated
No uncertainty quantification
Hard-subset selection circularity
Author-only human review
Narrow proprietary evaluation grid
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
HybridDeepResearch (arXiv 2609.09410) is a 380-task benchmark requiring agents to jointly execute SQL over nine LiveSQLBench-Base-Lite databases and search web evidence drawn from Wikipedia, Wikidata, and FineWeb-10BT. Three handoff patterns structure the set: SQL2S (203 tasks; a SQL singleton anchors later search), S2SQL (58; a search-recovered entity becomes a SQL predicate), and Parallel (119; exact intersection of two candidate sets), plus a balanced 120-question hard subset with 40 questions per pattern. Central claims: it is the first deep-research benchmark where both modalities are necessary for a verifiable answer; frontier systems reach only ~50-54% Pass@8 and below 30% Avg@8 on the hard subset; directional handoffs are far harder than parallel intersection; and multi-agent scaffolds raise Pass@8 without raising Avg@8.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is HybridDeepResearch?
A benchmark of 380 tasks where AI agents must combine web search with SQL database querying to produce a single verifiable answer. Tasks are organized into three patterns — SQL2S (database result anchors web search), S2SQL (web finding becomes a SQL filter), and Parallel (intersect two candidate sets) — plus a balanced 120-question hard subset. Code and data are on GitHub and Hugging Face.
How well do GPT-5 and Claude perform on it?
On the 120-question hard subset with 8 attempts per task, GPT-5 solves 54.17% at least once (best Pass@8) and Claude-Sonnet-4.6 achieves the best average success rate at 28.12%. No model exceeds ~30% average success, showing that coordinating web search with SQL is still an unsolved problem for current agents.
Why are directional tasks (SQL2S/S2SQL) harder than Parallel tasks?
Directional tasks require transferring an exact constraint from one tool into the other without losing it; typical failures are issuing a generic web query after correct SQL, or writing SQL without the searched entity as a filter. Parallel tasks only need two independent candidate sets intersected, which models do far better — for example GLM-5.2 scores 93.28 Pass@8 on Parallel versus 62.07-69.46 on the directional patterns.
Can models cheat by guessing or using only one tool?
The authors ran a single-tool ablation called the 'hybrid lock': an evaluator agent attempts every question twice, once with only web search and once with only SQL; any item solvable by a single modality is discarded. A closed-book prefilter also removes questions answerable from the model's own memory, and controlled clue blurring blocks single-clue shortcuts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's per-model bar values are not embedded in the extracted text (chart-internal numbers absent); hard-subset figures cited here come from Table 3 and surrounding prose.
- The Figure 4 construction-diagram text is interleaved with surrounding layout text in extraction (typical PDF figure noise); its content is nonetheless fully described in Section 4.2 prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.