Contextual Causality with Large Language Models: A Survey
Yiheng Zhao, Jun Yan, Chengming Hu
cs.CL · cs.AI
Quick verdict
Survey maps LLM contextual causality with a semantic/intervention/counterfactual taxonomy; clear roadmap, though narrative synthesis lacks systematic search.
Summary
This survey organizes the emerging literature on contextual causality — causal relations grounded in specific natural-language situations — for large language models. The authors argue prior surveys center on non-contextual causality (causal discovery/inference) and propose a three-part taxonomy: semantic causality (what a text expresses: ECI, causal judgment, attribution, strength estimation), intervention causality (what outcome follows an action: effect prediction), and counterfactual causality (what would have happened otherwise: counterfactual reasoning). Reviewing evaluation studies, they report causal hallucination (over-attribution by GPT-series on ECI/CJ), causal neglect in Qwen-series, sensitivity to misleading evidence in small models, and sharp degradation on graded strength assessment (CRAB); Table 3 benchmarks derive almost entirely from news, Wikipedia, stories, and commonsense questions. Methodologically, they cover training-free approaches (multi-agent ECI decomposition, knowledge injection) and CoT fine-tuning, criticizing computational overhead, knowledge-coverage dependence, and task specificity. Section 4 argues benchmarks are intrinsic-only, never measuring whether causal ability improves real decisions. Future directions include real-world and long-horizon agent-trajectory benchmarks, fine-grained process evaluation, and post-training data construction. Limitations: purely narrative synthesis without quantitative meta-analysis; the literature collection protocol (Appendix B) lacks systematic search criteria; and the taxonomy reframes classical causal theory rather than extending it.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Prior surveys on LLM causality center on non-contextual causality (discovery, inference), leaving contextual causality without systematic treatment (Table 1).
- Contextual causality decomposes into semantic, intervention, and counterfactual categories, each defined by a core causal question, required capabilities, tasks, and practical uses (Table 2).
- Current LLMs exhibit distinct causal failure modes — causal hallucination, Qwen-series causal neglect, sensitivity to misleading evidence — and degrade sharply on graded strength estimation (CRAB) and complex causal structures.
- Existing benchmarks draw mostly from news, Wikipedia, stories, and commonsense questions and are intrinsic-only, leaving real-world domains and decision-improvement unmeasured (§4, Table 3).
- No unified post-training dataset spans the three categories, and RLVR remains underexplored despite the easy verifiability of many contextual causal tasks (§3.4, §5).
Strengths
Unified taxonomy with explicit practical grounding
Table 2 maps each of the three categories to core causal question, capability, task, and practical use (e.g., causal attribution → responsibility assignment; CDS → causal filtering and evidence grounding), a level of operational detail absent from the non-contextual surveys it critiques in Table 1.
Intrinsic vs. extrinsic evaluation framing (§4)
The observation that every surveyed benchmark measures whether an LLM can identify causality but none measures whether that ability improves downstream decisions is a genuinely useful analytical lens that the prior critical review (Yang et al., 2024b) did not develop.
Integration of agentic failure attribution
Evidence-anchored failure-mode catalogue
Honest scoping and tangible appendix
Weaknesses
No systematic literature collection protocol
Purely narrative, non-quantitative synthesis
Thin boundary between intervention and counterfactual categories
Undefined, self-favoring comparative rubric in Table 1
Editorial hygiene lapses
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This survey (arXiv:2609.22409, Sep 2026) targets "contextual causality": causal relations grounded in concrete natural-language situations, as opposed to variable-level causal discovery/inference. The positioning claim — that seven methodology-oriented surveys and two task-specific surveys leave contextual causality without systematic treatment — is argued via Table 1, where the authors' own survey is the only row marked with full coverage (✓) across all three proposed categories. The contributions are: (1) a taxonomy (§2, Table 2) splitting contextual causality into semantic, intervention, and counterfactual categories, each characterized by core causal question, required capability, representative tasks, and practical uses; (2) a review of evaluation studies, benchmarks, and training-free/training-based methods (§3); (3) a benchmark-gap analysis framed as intrinsic vs. extrinsic evaluation (§4); (4) five future directions (§5). The paper is organizational rather than experimental; it contributes no new data, models, or code.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'contextual causality' in this survey?
Causal relations grounded in a specific natural-language situation — e.g., whether this particular patient's cancer was caused by smoking — as opposed to general variable-level relations like 'smoking raises lung-cancer risk.' The survey splits it into semantic causality (what the text states or implies), intervention causality (what outcome an action would produce), and counterfactual causality (what would have happened otherwise).
Do LLMs understand cause and effect, according to this survey?
Partially. The surveyed evidence shows strong performance on simple binary causal judgment and simple counterfactual benchmarks (e.g., CRASS), but also causal hallucination (over-attributing causality) in GPT-series models, causal neglect in some Qwen models, sensitivity to misleading context in small models, and sharp drops on graded strength estimation (CRAB) and complex causal structures.
Which benchmarks exist for evaluating LLM causal reasoning in context?
The survey catalogs roughly 30 in Table 3, including MAVEN-ERE, EventRelBench and CRAB (semantic causality), METER (a multi-level benchmark spanning all three categories), CaLQuest (Portuguese effect prediction), CRASS, IfQA and TimeTravel (counterfactual reasoning), and agent-trace benchmarks such as Who&When, TracerTraj-2.5K, MP-Bench and MAST-Data.
Does the survey release code, data, or a curated paper list?
No. The paper reports no companion code, dataset, or machine-readable bibliography; it is a narrative survey with prompt examples per task in Appendix A and a venue-based literature collection scope in Appendix B.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Tables 1–3 are extracted as flattened text; the ✓/○/✗ matrix in Table 1 may be misaligned, so per-survey coverage marks were read cautiously.
- Benchmark-level details (sizes, splits, metrics, reported scores) are not in the paper, limiting assessment of individual benchmark quality.
- The extraction appears complete through references and both appendices, but no figures beyond tables could be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.