CausalVerify: An Execution-Grounded Benchmark for LLM Causal Inference Workflows
Yonghong Zhang, Ricardo Correia, Isabel M. Parra, Yong Xie
cs.AI · cs.CL · econ.EM
Quick verdict
Solid execution-grounded benchmark showing runnable-but-wrong LLM econometrics code; well audited, though LLM-consensus labels and R-only scope limit.
Summary
CausalVerify is a benchmark testing whether LLM-written econometric analysis code actually recovers the target causal estimate, rather than merely running. It pairs 259 published economics papers (Experiment A: text-level method-family and direction agreement against four-LLM consensus labels) with 100 fixed-seed synthetic scenarios realising CSV datasets for DID, event study, IV, and RDD (Experiment B: models write R code; the executed estimate must match a canonical estimator on the same data within 50% relative error — layer L2b+). Across seven LLMs, L2b+ pass rates span 10%–88%; 66 of 426 executing workflows (15.5%) return wrong estimates. Execution-based ranking aligns strongly with execution-grounded correctness (Kendall τ=0.81, Spearman ρ=0.93) while text-direction scoring does not (τ∈[−0.20, 0.10]); this gap survives leave-one-model-out, leave-one-design-out, tolerance sweeps, and a Llama-3.3-70B robustness run. Retrospective self-confidence does not reliably separate correct from incorrect workflows. Human audits validate the extraction instrument (50 cells, 88.6% decision agreement) and reveal high ambiguity in Exp A consensus labels (direction κ=0.294). Main limitations: Exp A ground truth is LLM consensus overlapping the evaluated panel, the headline metric depends on an in-house scorer, results rest on 100 single-seed scenarios without confidence intervals, the default tolerance is generous, and only the R backend is released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- 66 of 426 executing LLM workflows (15.5%) return a wrong treatment-effect estimate despite runnable code, a failure invisible to execution-only and text-based scoring.
- Model ranking by execution (L2b) agrees with execution-grounded correctness (L2b+) at Kendall τ=0.81 / Spearman ρ=0.93, whereas text-direction scoring (L4) does not (τ∈[−0.20, 0.10]).
- L2b+ pass rates span 10%–88% across seven LLMs at 50% tolerance, and the execution–correctness alignment is robust to leave-one-out, design-family ablation, tolerance sweep, and a Llama-3.3-70B robustness run.
- Retrospective self-reported confidence does not reliably separate correct from incorrect workflows under the evaluated prompt.
- Exp A over 259 real papers is only an agreement diagnostic: its consensus labels are ambiguous (human direction κ=0.294) and overlap the evaluated panel, so correctness claims are confined to Experiment B.
Strengths
Verification target matches the real failure mode
L2b+ compares the executed coefficient against a canonical estimator on the same realised dataset, catching the runs-but-wrong failure that execution-only checks (L2b) and text scoring (L4) miss: 66/426 workflows (15.5%) are wrong after executing, and conditional P(L2b+|L2b) ranges from 22.2% (Kimi) to 100% (Gemini) in Table 6.
The core negative result is robust across every ablation shown
The L2b↔L2b+ vs L4 misalignment (τ=0.81/ρ=0.93 vs τ∈[−0.20, 0.10]) survives leave-one-model-out (Kendall 0.733–0.867, Table 8), leave-one-design-out (τ∈[0.81, 1.00], pooled wrong rate 14.0–16.1%, Table 11), a 25% tolerance sweep (pass rates 9%–84%), and an eighth robustness model (Llama-3.3-70B: τ=0.714).
Audited measurement instrument and unusually honest claim boundaries
Deterministic, releasable artifact
Weaknesses
Exp A ground truth is LLM consensus overlapping the evaluated panel
Headline metric depends on an in-house scorer
No uncertainty quantification on headline pass rates
Generous default tolerance and a single canonical path per family
Backend and calibration-coverage gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CausalVerify targets one precise verification question: after an LLM interprets a study, picks a quasi-experimental design, writes analysis code, and reports a treatment effect, does the executed workflow recover a benchmark-defined target estimate? The paper separates four signals often conflated elsewhere — textual interpretation (Exp A, L3/L4), code executability (L2b), numerical coefficient recovery (L2b+), and self-assessed confidence — and confines claims to single-shot workflows in four design families (DID, event study, IV, RDD) under an R backend. Central claims: 66 of 426 executing workflows (15.5%) return wrong estimates; execution ranking aligns with L2b+ (Kendall τ=0.81, Spearman ρ=0.93) while text-direction scoring does not (τ∈[−0.20, 0.10]); retrospective confidence fails to separate correct from incorrect workflows; L2b+ pass rates span 10%–88% across seven models. The claims apparatus is consistently hedged — L2b+ is repeatedly framed as "agreement with a benchmark-fixed reference implementation," not general causal correctness.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does CausalVerify measure?
It measures whether LLM-written econometric analysis code, when actually executed, recovers a benchmark-defined target treatment-effect estimate — not just whether the model names the right method or produces code that runs. Correctness (layer L2b+) means the extracted coefficient matches a canonical estimator computed on the same dataset within a 50% relative-error tolerance, across four design families (difference-in-differences, event study, instrumental variables, regression discontinuity).
Which models were evaluated and how well did they do?
Seven LLMs were evaluated on 100 synthetic scenarios. Final L2b+ pass rates ranged from 10% (Kimi) to 88% (Opus) at the 50% tolerance. Notably, 66 of the 426 workflows whose code executed still returned a wrong estimate, and per-family results show sharp weaknesses (e.g., Sonnet 12% on IV, Gemini 0% on RDD).
Is the benchmark's data and code available?
Yes. The paper states that code, data, cached model outputs, the 100 realised CSV datasets with canonical-estimator outputs, scorer decision records, and a datasheet are released on GitHub (causalverify/causalverify) with checksums, so deterministic scoring can be rerun without new LLM calls.
Can CausalVerify scores be read as general causal-reasoning ability?
No. The authors explicitly confine claims to single-shot workflows in four quasi-experimental design families under an R backend. Matching, propensity-score, doubly-robust, mediation, and heterogeneous-effect workflows, design selection from ambiguous field constraints, and other programming languages are outside the benchmark's scope.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The excerpt contains four marked [...sampling gap...] regions; parts of Section 3, the Section 5 primary-results narrative, and Appendix A/B/D/E details were not visible — Exp A per-model results and artifact checksums could not be verified, though every number cited in this review appears in the visible text.
- The metadata abstract is cut mid-sentence, but the in-text abstract is complete; no truncation at the true end of the provided text is claimed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.