BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligence
Chuxuan Hu, Yeye He, Penny Zhou, Wee Hyong Tok, Daniel Kang, Surajit Chaudhuri
cs.LG · cs.AI · cs.CL · cs.DB
Quick verdict
Claims first end-to-end BI benchmark and agent with large, statistically careful gains; credible, but 100-query scale and one 8B backbone limit generality.
Summary
The paper introduces BI-Bench, a 100-query benchmark of real business questions harvested from over 3,000 crawled Power BI (.pbix) projects, where ground-truth result tables are exported from actual user dashboards and verified through a manual-plus-LLM-assisted process costing over 400 person-hours. Evaluating 24 models and systems, the authors find frontier LLMs answer fewer than half of the queries in SQL settings, and top NL2SQL models with over 70% BIRD accuracy fall below 18% on BI-Bench. They propose BI-Agent, an LLM orchestrator invoking data-management tools for transformation prediction, join prediction, table search, and code execution; tools raise accuracy by 5-16 points typically, and by 40 points for GPT-OSS-120B in SQL (5.3% to 45.3%). A trajectory-synthesis post-training framework (SFT with a GPT-4o teacher, then GRPO RL) lifts Qwen3-8B from 5.2% to 35.0% with tools in SQL, at roughly 54x lower cost than frontier models, with gains transferring to Spider 2.0. Statistical care is high: paired t-tests, bootstrap CIs, Benjamini-Hochberg FDR control, and project-level clustering checks. Limitations: the benchmark spans only ~82 unique projects, is Power BI- and English-only, derives queries from curator-paraphrased visualization titles, and post-training uses a single 8B backbone.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- BI-Bench is the first benchmark of end-to-end BI built from real Power BI projects, and frontier LLMs answer fewer than half of its SQL-setting queries.
- SOTA NL2SQL models exceeding 70% accuracy on BIRD fall below 18% on BI-Bench, and Spider 2.0 agents reach only ~24-26%, showing existing benchmarks miss BI's data-preparation challenges.
- BI-Agent's data-management tools raise accuracy by 14 points on average in SQL (11 in Python), significantly in 19 of 20 model-setting pairs, up to +40.0 for GPT-OSS-120B in SQL.
- Trajectory-synthesis post-training (SFT then GRPO) lifts Qwen3-8B from 5.2% to 35.0% (SQL, with tools) at roughly 54x lower cost than frontier models, with gains transferring out-of-domain to Spider 2.0.
Strengths
First end-to-end BI benchmark built from real artifacts, with unusually transparent curation
100 (query, ground-truth table) pairs from 3K+ crawled .pbix projects; §4 documents the sample-screen-verify loop and an LLM-assisted check that only triggers human re-review (23.1% re-reviewed, 14.4% repaired, 5.2% discarded, 3.5% retained as verified-but-unsolvable), explicitly not filtering by model solvability; 400+ person-hours reported.
Statistical rigor well above field norm
10 runs per configuration, paired t-tests (n=100), 95% paired-bootstrap CIs (10,000 resamples), Benjamini-Hochberg FDR at q=0.05 across all 36 comparisons, and a project-level robustness check over 82 unique projects (footnote 9) that preserved every significant result.
Convincing demonstration that existing benchmarks do not transfer
Quantified leakage control and OOD validation for post-training
Tool design grounded in prior data-management algorithms, with sensitivity analyses
Weaknesses
Narrow benchmark scale and domain
All post-training evidence rests on one backbone
Headline numbers are best-case selections
Tool-vs-prompting confound not isolated
Incomplete verifiability of some sections in the available text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes three linked contributions: (1) BI-Bench, a 100-query benchmark of end-to-end BI built from over 3,000 crawled real Power BI (.pbix) projects; (2) BI-Agent, an LLM orchestrator that delegates table search, transformation, join, and code-execution subtasks to specialized data-management tools; (3) a post-training framework that synthesizes trajectories from real BI projects for SFT (GPT-4o teacher) and RL (GRPO). Central claims: frontier LLMs answer fewer than half of BI-Bench queries in SQL settings (best no-tool SQL accuracy is o4-mini at 48.2%, Table 3); tool augmentation and post-training yield large, statistically significant gains (up to +40.0 points for GPT-OSS-120B in SQL; Qwen3-8B lifted from 5.2% to 35.0% with tools); and post-training gains transfer out-of-domain to Spider 2.0 (20.5x over base). The authors scope 'end-to-end' honestly as raw-table-to-result-table (§3.2), explicitly complementary to curated enterprise BI rather than a replacement.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is BI-Bench?
BI-Bench is a benchmark of 100 real business questions with ground-truth result tables, extracted from over 3,000 real Power BI project files. It tests whether AI systems can do the full BI workflow — finding relevant tables, reshaping messy tables (e.g., unpivoting pivot-style tables), predicting joins, and computing the answer — rather than just writing SQL over clean, analysis-ready tables.
Why do strong NL2SQL models fail on BI-Bench?
NL2SQL benchmarks like BIRD assume clean, relational, analysis-ready tables. BI-Bench's raw inputs include pivot-style tables that need reshaping, cryptic ID columns that make join prediction hard, and many irrelevant tables. The paper's Table 4 shows the top open-source BIRD models (over 70% accuracy there) score between 6% and 17.3% on BI-Bench.
How much does BI-Agent improve accuracy?
Per the paper's Tables 3-4, adding data-management tools to vanilla LLMs raises accuracy by 14 points on average in SQL (11 in Python), with the largest jump being GPT-OSS-120B in SQL from 5.3% to 45.3%. Post-training Qwen3-8B with SFT and reinforcement learning reaches 35.0% with tools — comparable to much larger models at roughly 1/54 of the per-query cost.
Is the code and data for this paper available?
Yes. The paper states that code and data are released at github.com/Hu-Chuxuan/bi-agent. The post-training route uses open models (Qwen3-8B trained on 2x H100 GPUs for under $200 total), so independent re-derivation of the open-model results is plausible; results for proprietary models require paid API access.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text is truncated: parts of §3.1, the ground-truth-augmentation subsection of §4 (cut mid-example), the opening of §5, and §6.4 (reward ablation) are missing, so some methodology details could not be verified.
- Figures 1-11 are only partially represented in the text; Figure 10/11 numbers appear as fragments, so figure-based claims (e.g., tool stacking, latency) rest on the surrounding narrative.
- Tables 3 and 4 were extractable and anchor the review, but some section narratives around them were unavailable.
- The paper evaluates then-current 2026 models (GPT-5.5, DeepSeek-V4-Pro, Kimi-K2.6); specific model rankings will age quickly even if the benchmark and protocol remain useful.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.