On Epistemic Diversity in Large Language Models
Elisabeth Kirsten, Nicole Krämer, Muhammad Bilal Zafar
cs.CL
Quick verdict
Formalizes epistemic diversity (coverage of valid answer spaces) for LLM evaluation; conceptually rich, experiments suggestive but proxy- and judge-dependent.
Summary
The paper introduces 'epistemic diversity' as an evaluation dimension for LLMs: the range of valid answers, explanations, and reasoning strategies a model exposes, distinguished from group-based diversity. The authors formalize coverage over a universal answer set, propose a 2x2 typology of query specification versus answer-space cardinality, and distinguish four interaction protocols: accessible (default sampling), latent (exclusion-based probing), multi-turn, and multi-output. They operationalize two quadrants across ten models: six professions with finite Wikipedia reference lists (36 prompts, 100 samples) and nine theorems with multiple proof strategies. Frontier models cover only 2-24% of reference lists (e.g., 6% of physicists, 2% of composers), concentrate heavily on canonical figures like Einstein, and rarely exceed a handful of proof strategies. Accessible diversity is 1.9-4 unique valid names in ten samples; explicit diversity-seeking protocols recover substantially more (multi-turn averages 8.2 names), with Wilcoxon tests confirming gains over accessible sampling (p=0.002). Temperature scaling does not resolve narrowness. A WildChat annotation suggests 75% of real queries admit multiple valid answers. Limitations include reliance on Wikipedia proxies, dependence on LLM-judge annotation (human-validated on only 100 outputs), proofs evaluated without correctness checking, and only two of four typology quadrants tested; no mitigation experiments are run.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Epistemic diversity - coverage of valid answers, explanations, and reasoning strategies - is a distinct evaluation dimension for LLMs that group-based diversity does not capture.
- Frontier LLMs exhibit epistemic narrowness: across ten models, combined coverage of Wikipedia reference lists reaches only 2-24%, with heavy concentration on canonical figures such as Einstein.
- Measured diversity depends strongly on the interaction protocol: default sampling surfaces 1.9-4 unique valid names in ten samples, while explicit diversity-seeking protocols recover far more (multi-turn averaging 8.2 names), with an accessible-latent ratio as low as 0.01.
- Increased stochasticity (temperature 0.5-1.5) and small phrasing variations do not resolve narrowness, indicating the effect reflects model priors rather than sampling noise.
- Three quarters of 500 real WildChat queries admit multiple valid answers and over 75% are underspecified, so epistemic diversity is practically relevant beyond curated prompts.
Strengths
Genuinely new conceptual distinction, well grounded
Section 2 and Figure 1 cleanly separate epistemic diversity (what knowledge/reasoning is made available) from demographic/cultural/political diversity, showing group diversity does not imply epistemic diversity; the grounding in Longino, Kellert, Harding, and Zollman is substantive, not decorative.
Protocol taxonomy is a reusable methodological contribution
Separating accessible, latent (exclusion-based), multi-turn, and multi-output diversity (Section 3.1, Appendix B) and comparing them at a matched budget of ten responses (Table 5) isolates interface effects from model priors; multi-turn (8.2 names avg) matching latent (7.8) at the same budget is an actionable finding.
Unusually careful statistics for this literature
Honest failure-mode analysis and human validation
Real-world relevance check
Weaknesses
Coverage numbers lack a defensible threshold and the proxy may drive them
Proofs are classified without correctness verification
Single-judge pipeline with thin validation
The headline accessible-latent ratio is an unstable metric
Framework generality outruns the evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This COLM 2026 paper introduces 'epistemic diversity' as an evaluation dimension for LLMs: the range of valid answers, explanations, and reasoning strategies a model exposes to users (Sections 1-2). The authors explicitly separate it from group-based diversity via Figure 1 - a system can represent all groups yet surface only a canonical subset of the knowledge those groups hold. Definition 1 formalizes narrowness as coverage |A intersect A'| below a threshold tau; a 2x2 typology (Figure 3) crosses question specification with answer-space cardinality; and four interaction protocols (accessible, latent, multi-turn, multi-output) separate what users see by default from what deliberate probing recovers (Section 3.1, Appendix B). Central claims: (1) epistemic diversity is a distinct, necessary evaluation axis; (2) frontier models are epistemically narrow, covering only 2-24% of reference lists; (3) measured diversity is protocol-dependent, with accessible/latent ratios as low as 0.01; (4) neither temperature (0.5-1.5) nor phrasing variations resolve narrowness.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is epistemic diversity in large language models?
As defined in this paper, it is the range of valid answers, explanations, and reasoning strategies an LLM exposes to users for a query - distinct from demographic or political representation. The authors find frontier models are often epistemically narrow: they collapse large spaces of equally valid answers onto a few canonical ones, such as always naming Einstein or always giving Euclid's proof.
How much of Wikipedia's list of physicists do LLMs actually cover?
Across ten models and all tested prompt formats combined, only 70 of the 1,077 individuals on Wikipedia's list of physicists (about 6%) were ever named. Coverage was worst for composers (121 of 4,983, about 2%) and best for women philosophers (77 of 316, about 24%).
Does raising the temperature make LLM outputs more epistemically diverse?
Not meaningfully in this paper's tests. Comparing Gemini 3 at temperatures 0.5, 1.0, and 1.5, higher temperature changed wording and framing but not which person or proof strategy was chosen, so cumulative coverage gains were limited. Explicitly asking for different answers worked much better than turning up randomness.
Is code available for this paper?
Yes. The authors state that the code for their experiments is available at github.com/aisoc-lab/llm-epistemic-diversity, including the judge prompts used for entity extraction and proof-strategy classification. Reproducing the headline numbers requires access to commercial model APIs (GPT-4o/5, Gemini) and January 2026 Wikipedia snapshots.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal data (the curves in Figures 2, 4-11) is not machine-readable from the PDF extraction; only captions, axis labels, and in-text claims about them were verifiable. All tables used in this review were present as text and fully parseable, and the extraction runs complete from the title page through Appendix E.7 with no evidence of further truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.