CausalArena: Benchmarking Causal Discovery in the Foundation Model Era
Zi-Rong Li, Si-Yang Liu, Tian-Zuo Wang, Han-Jia Ye
cs.LG
Quick verdict
CausalArena benchmarks 18 causal-discovery methods on 1200 synthetic, semantic, and formula SCMs; a well-built arena, though rankings lack statistical tests.
Summary
The paper introduces CausalArena, a unified and evolvable benchmark for causal discovery comprising 1,200 executable structural causal models: 1,000 synthetic SCMs spanning 10 graph families, 16 mechanism families, and 14 root and 15 noise distributions; 100 semantically grounded operational SCMs from 10 real-world domains; and 100 formula-grounded scientific SCMs anchored in explicit equations. All families compile to a shared specification and one observational/interventional protocol (1,000 observation rows; 800 observation plus 200 interventional rows; five replicates; F1/SHD and related metrics). The authors evaluate 18 methods, from the RandomRegress control and classical search (PC, GIES, IGSP, CDIS) through continuous optimization (NOTEARS, DAGMA, SDCD) to seven pretrained causal foundation models (AVICI, SEA, Arrow, CauScale, CDFM, FoundCause, TabCausal). Results show substantial ranking instability: FoundCause leads observation-only synthetic (F1 0.62), semantic (0.63), and formula (0.52) slices but drops to 0.21 on 13 real-world tables, where CDFM (0.39 obs-only) and TabCausal (0.46 obs+int) lead. Factor analyses attribute performance to graph topology, dimension, mechanisms, noise, and root dependencies; runtime fits and an accuracy-cost frontier are also reported. Limitations include absence of statistical significance testing, an LLM-agent-generated qualitative behavior table, a fixed-budget obs+int protocol that confounds evidence type with sample composition, and an overlap analysis reliant on self-documented pretraining coverage. Half of each family is public; the rest is reserved for held-out leaderboard evaluation.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Rankings of classical, neural, and pretrained causal discovery methods shift substantially across Synthetic, Semantic, Formula, and real-data regimes, so no method dominates every slice.
- Pretraining-evaluation overlap makes fixed public SCM benchmarks difficult to interpret for causal discovery foundation models, motivating an evolvable arena with reserved held-out SCMs.
- A shared executable SCM specification can jointly provide breadth (1,000 synthetic), grounding (100 semantic + 100 formula-grounded), and diagnosability (factor-level slice reporting) under one protocol.
- FoundCause leads observation-only slices (F1 0.62/0.63/0.52 on Synthetic/Semantic/Formula) but drops to 0.21 F1 on real tables, where TabCausal (0.46 obs+int) and CDFM (0.39 obs-only) lead.
Strengths
Three complementary SCM families under one executable spec
1,200 SCMs (1,000 synthetic, 100 semantic operational, 100 formula-grounded) compile to a shared specification and one obs/int protocol; Table 1 positions CausalArena as the only listed resource combining regenerable SCMs, breadth, grounding, scientific mechanisms, and shared protocol simultaneously.
Explicit pretraining-overlap design
Section 7.2 reserves 500 synthetic, 50 semantic, 50 formula SCMs as held-out leaderboard material with versioned progressive release, encourages disclosure of pretraining use, and Section 6.2 quantifies ID-to-OOD sensitivity (Eq. 10, Fig. 16: spans of ~0.01-0.03 F1, AVICI widest at 0.32-0.35).
Unusually complete protocol documentation
Diagnosable failure attribution
Real-data external check exposes transfer failure
Weaknesses
No inferential statistics behind headline ranking claims
LLM-generated qualitative method profiles
obs+int protocol conflates evidence type with sample composition
Overlap diagnostic is self-referential and coarse
Real-data check mixes full and repair configurations
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CausalArena (arXiv 2609.11897) is a benchmark paper for causal discovery evaluation in the foundation-model era. The authors frame four gaps in current practice — breadth, freshness, grounding, diagnosability — plus a distinctive problem: pretrained causal discovery foundation models (CDFMs) may score well simply because their pretraining environments resemble the test SCMs. Their response is a unified arena of 1,200 executable structural causal models (1,000 synthetic; 100 semantic operational from 10 real-world domains; 100 formula-grounded scientific from 10 domains), all compiled to one specification and one observational/interventional protocol, comparing 18 methods: the RandomRegress control; graphical search (PC, GIES, IGSP, CDIS); functional methods (DAS, LiNGAM); continuous optimization (DAGMA, NOTEARS, NOTEARS-MLP, SDCD); and pretrained models (Arrow, AVICI, CauScale, CDFM, FoundCause, SEA, TabCausal). Central claims: (1) rankings shift substantially across Synthetic/Semantic/Formula/real regimes; (2) pretraining-evaluation overlap makes fixed public benchmarks hard to interpret; (3) an evolvable, versioned release (half public, half reserved) mitigates that overlap.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CausalArena?
CausalArena is a benchmark for causal discovery that combines 1,200 executable structural causal models — 1,000 synthetic, 100 semantically grounded operational scenarios from 10 real-world domains, and 100 formula-grounded scientific scenarios — under one shared observational and interventional evaluation protocol. Half of each family is publicly released on Hugging Face; the rest is reserved for a held-out leaderboard with versioned updates.
Which causal discovery method performs best on CausalArena?
No single method dominates. FoundCause scores highest on observation-only synthetic (F1 0.62), semantic (0.63), and formula (0.52) slices, but drops to 0.21 on real-world tables, where CDFM leads observation-only (0.39) and TabCausal leads the observation-plus-intervention setting (0.46). Rankings change substantially across benchmark families, which is the paper's central finding.
Why does pretraining-evaluation overlap matter for causal discovery benchmarks?
Pretrained causal discovery foundation models learn from large collections of synthetic SCMs. If a benchmark's SCMs or generators resemble the model's training distribution — or later enter it once public — benchmark scores may reflect memorized priors rather than genuine causal structure recovery, making results on fixed synthetic suites hard to interpret.
Is the CausalArena data available?
Yes. The paper links a public Hugging Face dataset (LAMDA-Tabular/CausalArena) releasing half of each SCM family (500 synthetic, 50 semantic, 50 formula-grounded SCMs) plus a leaderboard space. The remaining audited SCMs are held out for leaderboard evaluation and scheduled for progressive release in versioned updates. Fixed wrapper configurations for all 18 evaluated methods are documented in the paper's appendix.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: middle sections (parts of the real-data results in Section 5.2, the middle of the Section 6.2 OOD derivation, the end of Section 7.2, and the numeric appendix result tables C.1-C.3 / Tables 7-13) were skipped and marked as sampling gaps; this review is based on the visible head and tail portions, which include Table 2, Table 5, Appendix A-B, and the main analyses.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.