PRISM-BN: A Controlled Corpus and Benchmark for Text-to-Parameterized Bayesian Network Extraction
Amartya Bhattacharya, Nikhil Singh, Nikhil Singh, Neeti Pokhriyal, Soroush Vosoughi
cs.CL
Quick verdict
Carefully built, honest benchmark for text-to-parameterized-BN extraction; key caveat: reference BNs are LLM-generated and not externally validated.
Summary
The paper introduces PRISM-BN, a benchmark of 5,054 natural-language descriptions paired with fully parameterized discrete Bayesian networks, derived from 50 Wikipedia-seeded backbone graphs across five domains (Economics, Environment, Geopolitics, Society, Technology). A construction pipeline, PRISM, elicits marginal and local joint distributions from Claude Sonnet 4.6 and recovers normalized CPDs analytically, then re-roots connected subgraphs into self-contained examples. Evaluation uses a four-phase protocol — semantic node alignment (Llama 3.3 70B judge), state alignment, edge scoring, and column-wise CPD-KL — with state, edge, and CPD metrics conditional on earlier alignment steps. Across six LLM extractors, Node F1 spans 0.566-0.826 with near-ceiling recall (0.972-0.987) but low precision (0.395-0.711), showing over-generation is the dominant failure; conditional Edge F1 reaches 0.902-0.966; CPD-KL spans 1.107-3.139, and every model underperforms uniform (0.27) and parent-ignorant marginal (0.20) anchors. Rankings hold under an independently generated GPT-5.5 reference set and a small human pilot (10 BNs, 2 annotators). Main limitations: reference BNs and descriptions are LLM-generated and never validated against empirical data; the semantic judge lacks human validation; conditional metrics with unreported survival counts can overstate end-to-end performance.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PRISM-BN is the first corpus pairing natural-language descriptions with fully parameterized discrete Bayesian networks at scale: 5,054 examples derived from 50 Wikipedia-seeded backbones across five domains.
- The marginal-first PRISM pipeline elicits marginals and local joint distributions and recovers normalized CPDs analytically, avoiding the documented failure modes of direct CPD elicitation.
- Across six LLM extractors, node over-generation dominates structural error (recall 0.972-0.987 vs precision 0.395-0.711), while conditional edge recovery is strong (Edge F1 0.902-0.966).
- All six LLMs underperform uniform (0.27) and parent-ignorant marginal (0.20) calibration anchors on CPD-KL (best 1.107), showing structural extraction and probabilistic calibration are decoupled.
- Relative performance trends persist under an independently generated GPT-5.5 reference set and a two-annotator, ten-network human pilot.
Strengths
Fills a genuine resource gap
No prior resource pairs text with fully parameterized BNs at scale: bnlearn has <50 networks without text (§1), LLM causal extraction is qualitative (Kıcıman et al. 2023; Long et al. 2022), and CLADDER/BLInD/QUITE test reasoning over given networks rather than extraction (§2). PRISM-BN's 5,054 paired examples with variables, states, edges, root priors, and full multi-parent CPDs are a first.
Marginal-first construction avoids known elicitation failures
§3.2 recovers CPDs analytically by normalizing elicited pairwise/local joints (Eqs. 1-2) instead of asking LLMs for full CPDs — a documented failure mode (round-number bias, normalization violations). The k>=4 Naive Bayes fallback (Eq. 3) affects only ~0.1% of subgraphs, so its conditional-independence distortion is negligible.
Calibration anchors give CPD-KL an interpretable scale and yield a sharp negative result
Robustness checks beyond a single reference generator
Unusually disciplined scope statements
Weaknesses
Label provenance is circular and never empirically validated
The semantic judge is unvalidated
Conditional metrics with unreported survival counts
Effective diversity is much smaller than the instance count
Thin human validation and minimal CPD prompting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PRISM-BN addresses a real gap: no existing resource pairs natural-language text with fully parameterized discrete Bayesian networks at scale. The bnlearn repository holds fewer than 50 expert networks without text (§1); LLM causal-extraction work (Kıcıman et al. 2023; Long et al. 2022) is qualitative; CLADDER, BLInD, and QUITE test reasoning over given networks, not extraction (§2). The paper's central claims are: (1) a 5,054-example corpus built from 50 Wikipedia-seeded backbones across five domains; (2) a marginal-first construction pipeline (PRISM) that elicits marginals and local joints from Claude Sonnet 4.6 and recovers normalized CPDs analytically (§3.2); and (3) a four-phase benchmark showing that node over-generation dominates structural error while CPD estimation lags trivial anchors. The authors are unusually disciplined about scope, repeatedly stating that labels are 'internally consistent benchmark targets, not externally validated causal estimates' (Abstract, §1, §5, §8, §9).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PRISM-BN?
PRISM-BN is a benchmark dataset of 5,054 natural-language descriptions, each paired with a fully parameterized discrete Bayesian network (variables, states, directed edges, root priors, and complete conditional probability tables). Examples were generated by an LLM pipeline called PRISM from 50 Wikipedia-seeded backbone graphs across five domains: economics, environment, geopolitics, society, and technology.
Are the probabilities in PRISM-BN real-world validated?
No. The authors state explicitly that the reference structures, marginals, and CPDs are internally consistent benchmark targets produced by their construction pipeline (using Claude Sonnet 4.6), not estimates validated against observational or interventional data. CPD-KL scores measure agreement with this reference parameterization, not real-world causal accuracy.
Which LLM is best at extracting Bayesian networks from text according to the paper?
DeepSeek-V3 leads on Node F1 (0.826) and conditional Edge F1 (0.966), but Claude Haiku 4.5 achieves the lowest CPD-KL (1.107) despite ranking second on node extraction. The paper's point is that structural skill and probabilistic calibration are decoupled — and that every tested model still loses to a parent-ignorant marginal predictor (CPD-KL 0.20) on parameter quality.
Why are the edge scores so high when node scores are much lower?
State, edge, and CPD metrics are conditional: they are computed only after a semantic judge (Llama 3.3 70B) successfully aligns predicted nodes to reference nodes, and CPDs are scored only on correctly recovered edges. End-to-end performance is bounded by Node F1 (0.566-0.826), so the conditional Edge F1 of 0.90-0.97 does not describe what a naive end-to-end pipeline would achieve.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are present only as captions in the extracted text; histogram shapes and the confidence bands in Figure 4 could not be visually inspected.
- Appendix H reports only partial numbers for the GPT-5.5 replication; the full per-model tables for that setting were not in the extract.
- Review is based solely on the provided text extraction; the released corpus, code, and prompts beyond Appendix B/C could not be checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.