DI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents
Jiangyun Zhang, Kristen Surrao, Torpong Nitayanont, Yupei Zhang, Roopali Singh, Zhiyu Chen, Julia Huang, Zhou Tang, et al.
cs.AI
Quick verdict
Proposes a pipeline generating DI benchmarks with SQL-derived gold answers; construction is credible, but synthetic rules and small cells limit generality.
Summary
DI-Bench is a pipeline that automatically generates benchmarks for enterprise data-intelligence agents, targeting questions that require both database computation and retrieval of business rules. It builds an artifact linkage graph over tables, metrics, rules, and dimensions; the key rule-to-metric edge is validated by counterfactual SQL re-execution, with graph-selected rules changing answers in 82% of cases versus 29% for semantic-similarity and 22% for random pairing. Gold answers are computed by executing parameterized SQL; questions are generated backward from answers by DeepSeek V3.2; and a two-stage validator (regex filters plus an eight-dimension LLM judge) admits tasks, with a 100-task human audit reporting 99% quality for admitted tasks. Applied to the Olist e-commerce and PKDD'99 banking datasets, the pipeline yields 731 tasks across nine analysis types. Evaluating Claude Opus 4.6 (57.9%), DeepSeek V3.2 (50.9%), Claude Haiku 4.5 (49.0%), and Qwen3-80B (47.9%), accuracy falls from 82% on retrieval-only questions to 50% with analytics and 32% when a retrieved rule must modify the computation; oracle rule provision adds only 0.2 points while removing rules adds 20.4, isolating rule application as the bottleneck. Limitations include synthetic LLM-written rules, very small rare-type cells (counterfactual N=21, decomposition N=20), exact-match scoring, and evaluation on two domains with a single agent architecture.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- DI-Bench generates in-domain data-intelligence benchmarks with deterministic, execution-derived gold answers from any relational database plus a rule-document corpus.
- Graph-based rule selection pairs rules that genuinely change computed answers far more often than semantic or random pairing (82% vs 29%/22% on N=175 matched skeletons).
- Four frontier models show a knowledge-to-computation gap: accuracy falls from 82% (retrieval-only) to 50% (plus analytics) to 32% (plus rule-modified computation).
- Oracle rule provision adds only +0.2 points while removing the rule adds +20.4, showing rule application — not retrieval — is the bottleneck.
- The pipeline reduces estimated construction effort from ~420 to ~22 human-hours for the 731-task benchmark.
Strengths
Execution-derived ground truth
Gold answers come from running parameterized SQL with rule-encoded modifications on the real database (§3.3), and a candidate is retained only if re-executing its stored SQL reproduces the answer (§3.2) — avoiding the LLM-fabricated-answer weakness of prior generated benchmarks the paper itself cites (§1, Li et al. 2025).
Causal, not similarity-based, rule selection
Table 8: on 175 identical task skeletons, graph-selected rule pairings change the computed answer 82% of the time versus 29% (semantic similarity) and 22% (random) — a direct measurement that the rule→metric edge captures causal effect rather than topical overlap (Appendix C).
Clean isolation of the failure mode
Transparent validation audit
Human-solvability guard
Weaknesses
Difficulty partly by construction
Very small cells for the hardest types
Scoring and variance under-specified
Shared-model confound across four roles
Thin human baseline and unmeasured cost claim
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
DI-Bench (Amazon; arXiv 2609.05776) is a benchmark-generation pipeline for enterprise data-intelligence (DI) agents — settings where answering requires SQL computation over a relational database plus retrieval of natural-language business rules that change how metrics are computed (§2). The paper claims three contributions: the pipeline; a 731-task benchmark over Olist e-commerce (449 tasks) and PKDD'99 Czech banking (282); and an empirical finding that four frontier models drop from 82% (retrieval-only) to 50% (retrieval + analytics) to 32% (when a retrieved rule must modify the computation) (§4.2, Figure 2). A secondary claim is cost: ~420 estimated manual hours shrink to ~22 verification hours (Appendix G).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DI-Bench?
DI-Bench is a pipeline from Amazon researchers that automatically generates evaluation benchmarks for enterprise data-intelligence agents — LLM agents that must query a relational database and also retrieve business-rule documents that change how metrics are computed. Applied to two public datasets it produced a 731-task benchmark whose gold answers are computed by executing SQL, not by an LLM.
What is the paper's main finding?
Across four models (Claude Opus 4.6 57.9%, DeepSeek V3.2 50.9%, Claude Haiku 4.5 49.0%, Qwen3-80B 47.9% overall accuracy), scores fall from 82% on retrieval-only questions to 50% when database computation is added and 32% when a retrieved business rule must modify the computation. Giving the rule directly (oracle) barely helps (+0.2 points), so applying rules — not finding them — is the bottleneck.
What data is DI-Bench built on, and is it real enterprise data?
Two public relational datasets: the Olist Brazilian e-commerce dataset (~100K orders, 9 tables) and the PKDD'99 Czech banking dataset (8 tables). The metric catalogs (29 metrics per domain) and 17 rule documents per domain are fictional, LLM-generated augmentations, because no public dataset ships both relational data and business-rule documentation with open release rights.
Are the DI-Bench dataset and pipeline publicly available?
The paper marks DI-Bench as releasing both the generation pipeline and the fixed 731-task dataset (Table 1's artifact legend), and both underlying datasets are public. The authors note LLM-based generation stages are not reproducible bit-for-bit, though non-LLM randomized stages use seed 42.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- No repository or artifact URL appears in the provided text; release is asserted only via Table 1's artifact legend and Appendix L's reference to 'the released configuration'.
- Figure 2's per-model values appear as inline digit strings from PDF extraction (e.g., 89/76/85/78); they are legible but come from chart labels, not a formatted table.
- The provided text ends with the completion of Appendix L; references and appendices A–L appear intact, so no material truncation was observed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.