Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents
Minghao Guo, Meng Cao, Sui Zhao, Siyu Ning, Xin Wang, Haoze Zhao, Jiaxuan Yang, Haihong Hao, et al.
cs.AI
Quick verdict
A 102-item multimodal deep-research benchmark with dependency-aware scoring; carefully built and honestly reported, but small n and LLM-judging cap claims.
Summary
The paper introduces Mr.LHDR, a benchmark for evaluating multimodal deep research agents on long-horizon, open-web investigations. Its 102 questions span eight categories and are built from hidden Node-Relation graphs: each requires an average of 12.1 necessary intermediate conclusions with a mean dependency depth of 10.4, and each must contain non-text evidence (images, maps, PDFs, charts, video frames) that changes the reasoning state. Beyond final-answer accuracy (OA), the authors propose the Dependency-Aware Checklist Score (DACS), which recursively withholds credit for a stated conclusion whose annotated prerequisites are unmet on human-verified dependency DAGs. Evaluating 25 systems — tool-augmented and tool-free VLMs, deep research endpoints, and DeerFlow agents — the best run, GPT-5.5, reaches only 43.1% OA and 34.3% SA, and o3 Deep Research 32.4% OA. A controlled image ablation shows DACS rises 12.6 points with images, and SA declines monotonically with checklist length. Reporting is statistically careful (bootstrap CIs, 12-judge robustness with Fleiss' κ 0.940), and construction passed four human audit phases. Main limitations: only 102 items with wide confidence intervals; all process metrics rest on an LLM judge with no human calibration; each system ran once in a single request; and deep-research endpoints received images as URL text rather than attached bytes, a possible confound.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Even the strongest system (GPT-5.5) reaches only 43.1% OA and 34.3% SA on Mr.LHDR, so final-answer accuracy substantially overstates complete, dependency-consistent task success.
- Withholding images from the same model lowers DACS by 12.6 points (21.6%→34.2% with images), showing non-text evidence materially contributes to the reasoning chain.
- Across checklist-length ranges with at least 20 items, SA decreases monotonically with chain length for every representative model.
- DACS, which withholds credit when annotated prerequisites are unsatisfied, exposes dependency-inconsistent conclusions that unordered checklist scoring (CS) overestimates.
- Per-model rankings on the 102-item set are mostly not statistically separable; only the GPT-5.5 vs o4-mini ordering exceeds 95% bootstrap stability.
Strengths
Genuinely dependency-aware metric
DACS (Eqs. 4–5) recursively conditions credit on annotated prerequisites over human-verified DAGs; the CS−DACS gap (3.0–8.9 points across systems, Fig. 11) demonstrates it captures failures that unordered checklist scoring (CS) hides.
Unusually honest uncertainty reporting
Table 9 gives item-bootstrap CIs for all 25 systems (GPT-5.5 OA [33.3, 52.9]); E.1 states that only the leading pair is separable above 95% (96.4% of resamples) and frames conclusions as broad capability patterns rather than a ranking.
Extensive judge-robustness analysis
Disciplined construction pipeline
Controlled multimodal ablation with paired inference
Weaknesses
Small, wide-interval evaluation set
LLM judge with no human calibration
Confound in the deep-research comparison
Single-run, single-request protocol
Statement-based scoring conflates style with failure
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Mr.LHDR (arXiv:2609.11318v2, cs.AI) is a benchmark paper for evaluating multimodal deep research agents on long-horizon, open-web investigations. The authors make three claims: (1) existing browsing benchmarks test medium-horizon research — MM-BrowseComp averages 3.0 checklist items per question — while real investigations need long, dependency-linked evidence chains; (2) their 102 items, each built from a hidden Node-Relation graph (mean 12.1 necessary intermediate conclusions, mean dependency depth 10.4, at least one non-text evidence element that changes the reasoning state), make this regime measurable; and (3) four metrics — OA, SA, CS, and the new Dependency-Aware Checklist Score (DACS) — separate final-answer correctness from complete, dependency-consistent execution. Headline results (Table 4): GPT-5.5 leads at 43.1% OA / 34.3% SA / 73.6% CS / 70.3% DACS; o3 Deep Research, the best dedicated system by OA, reaches 32.4% / 19.6% / 56.1% / 49.8%.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Mr.LHDR benchmark?
Mr.LHDR (Multimodal real-world Long-Horizon Deep Research) is a benchmark of 102 open-web research questions across eight categories. Each question is built from a hidden dependency graph requiring an average of 12.1 intermediate conclusions (mean dependency depth 10.4) and must include non-text evidence such as images, maps, PDFs, charts, or video frames that changes the reasoning. Answers are short and verifiable, e.g., a name, year, or number.
What is DACS (Dependency-Aware Checklist Score)?
DACS scores a model response against an annotated checklist of necessary intermediate conclusions organized as a dependency graph. A conclusion earns credit only if it is stated correctly and all of its annotated prerequisites are also satisfied, so a downstream statement built on missing steps receives no credit — unlike a plain checklist score that treats items as unordered and independent.
How well do current AI models perform on Mr.LHDR?
Far from saturation. The best overall run, GPT-5.5, reaches 43.1% final-answer accuracy (OA) but only 34.3% strict accuracy (SA); the best deep-research endpoint by OA, o3 Deep Research, reaches 32.4% OA and 19.6% SA. Removing images cuts DACS by 12.6 points, showing visual evidence is integral to solving the tasks.
Is the Mr.LHDR dataset and evaluation code public?
The paper states code and data are available at github.com/minghaoguo20/Mr-LHDR, and the release plan includes the evaluator, the exact judge prompt, verified dependency graphs, source URLs, image assets, and regression tests. Hidden reasoning paths and full source chains are withheld to resist shortcutting and contamination.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: three '[...sampling gap...]' regions skip parts of Section 3.2.1, the transition around Sections 4.3.4–5, Appendix A.2–A.4, part of E.3, and the first case study; no conclusion in this review relies on content inside those gaps.
- Figures are available only as extracted text and captions; the visual content of figures (e.g., Fig. 3, 6, 12) could not be inspected directly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.