Benchmarking Language Models for Statistical Problem Formulation
Chen Wang, Junzhe Zhao, Xin Cong, Wanlu Deng, Ke Deng
cs.AI
Quick verdict
First benchmark for LLM statistical problem formulation; careful construction, but headline model gaps lack significance tests and match metric has ~10% error.
Summary
The paper introduces StatFormBench, a benchmark for statistical problem formulation: mapping an informal natural-language request plus heterogeneous data to (i) a statistical problem type and (ii) the required variables with roles (predictor/response/both/N/A). The task splits into Statistical Problem Classification over a 20 coarse/85 fine taxonomy and Variable Identification & Role Assignment. The 1,013 samples derive from five statistics textbooks (629) and a GouXiongHui data-science case library (384; Chinese-origin, translated, values simulated by GPT-5); LLM-generated initial labels were human-verified — two PhD review rounds plus adjudication for textbooks, one expert for cases. Zero-shot evaluation of 14 LLMs shows best fine-grained accuracy 72.0 (Gemini 3.1 Pro) and best variable Jaccard 63.2 (Claude Opus 4.6), achieved by different models; category definitions add 1.9–3.8 points while few-shot prompting helps inconsistently. Performance splits by source: textbook classification is easy (ACCFG 74.1–84.1) but variable extraction hard (JCV 33.6–51.0); the case library reverses this. Weaknesses: no significance tests or confidence intervals for closely spaced rankings; exact-value matching has a self-measured 10–11.7% error rate comparable to headline gaps; initial labels came from evaluated model families; category distribution is heavily imbalanced. Data and code are released on Hugging Face and GitHub.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Statistical problem formulation — inferring the problem type and required variables from informal requests — is a distinct LLM capability not previously benchmarked.
- StatFormBench provides 1,013 human-verified samples spanning 20 coarse and 85 fine statistical categories from five textbooks and a data-science case library.
- The best zero-shot models reach only 72.0 fine-grained classification accuracy and 63.2 variable-set Jaccard, achieved by different models, so no model reliably performs formulation.
- Enhanced prompting — category definitions and few-shot examples — yields only limited or inconsistent gains across the two subtasks.
Strengths
Novel, clearly motivated task decomposition
Sections 2–3 carve out upstream formulation from prior benchmarks that presuppose a given analysis target (StatQA, StatEval, DA-Code); the two-subtask split with question-conditional roles (predictor/response/both/N/A, Sec. 3.2) is crisp and reusable.
Human verification above benchmark norms
All 629 textbook samples passed two PhD-level review rounds plus senior adjudication, and all 384 case samples one expert review (Sec. 4.3); variable labels are grounded in reference answers rather than scenario mentions (Sec. 4.2).
Metric honesty
Broad, scale-stratified model coverage
Actionable error analysis
Weaknesses
No uncertainty quantification
Headline gaps sit inside the metric's own error
Label provenance overlaps evaluated models
Severe category imbalance limits fine-grained claims
Ecological validity of scenario rewriting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper formalizes **statistical problem formulation**: given an informal request triple (background, question, data), output (i) a statistical problem type over a two-level taxonomy (20 coarse, 85 fine; Appendix B) and (ii) a variable set with per-variable roles (predictor/response/both/N/A). It claims to be the first benchmark for this upstream step, distinguishing itself from StatQA, StatEval, QRData and execution benchmarks (DA-Code, ScienceAgentBench, BLADE) that presuppose a specified analysis target (Section 2). StatFormBench contains 1,013 samples: 629 from five statistics textbooks and 384 from a GouXiongHui case library. Key findings (Table 1): the best zero-shot fine-grained accuracy of 72.0 (Gemini 3.1 Pro) and the best variable Jaccard of 63.2 (Claude Opus 4.6) come from different models; coarse accuracy exceeds fine-grained for all 14 models; category-definition prompting adds 1.9–3.8 ACCFG points with mixed effects on variable metrics (Table 3); few-shot gains are inconsistent (Figures 5, 8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is StatFormBench?
A benchmark of 1,013 realistic scenarios testing whether large language models can turn an informal data-analysis request into (a) the correct statistical problem type, chosen from a taxonomy of 20 coarse and 85 fine categories, and (b) the minimal set of variables with their analytical roles. It was built by a Tsinghua statistics group from five statistics textbooks and a data-science case library, with PhD-level human verification of every label.
Which LLM is best at statistical problem formulation?
In the paper's zero-shot evaluation of 14 models, Gemini 3.1 Pro leads fine-grained problem classification (72.0 accuracy) while Claude Opus 4.6 leads variable-set overlap (63.2 Jaccard) and variable recall. No single model leads on both subtasks, which is the paper's central finding.
Why are the benchmark scores relatively low?
Scenarios are deliberately rewritten to hide statistical terminology and include irrelevant details, and the variable metric requires exact value matches, which the authors estimate wrongly rejects about 10% of semantically correct matches even on a hard subset. Models also confuse adjacent categories, such as model comparison versus model interpretation.
Is the StatFormBench dataset and code public?
The paper states that the data is released on Hugging Face (THU-CongLab/StatFormBench) and the evaluation code on GitHub (THU-CongLab/StatFormBench); all construction and evaluation prompts are reproduced in Appendix C of the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Bar-chart values in Figures 4, 5, 8 and 9 are not recoverable from the text extraction; all chart-derived numbers cited in this review come from Figure 3's text labels or prose statements.
- Table 4's two-case layout is garbled by PDF extraction; error-case details were reconstructed from the surrounding prose in Section 6.4.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.