LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs
Muhammed Saeed, Simon Razniewski
cs.CL
Quick verdict
Demo: ~1.3M articles from LLM parametric memory; audit: 68.4% supported, silence not error dominates — strong open resource, stats deferred to companion paper.
Summary
This demonstration paper presents LLMPEDIA, a live, browsable encyclopedia of roughly 1.3 million articles materialized purely from the parametric memory of three LLM families — GPT-5-mini (~1M articles), DeepSeek-V3.2 and Llama-3.3-70B (~120K each) — through breadth-first wikilink expansion from the seed entity Vannevar Bush, with no retrieval at generation time. A stratified audit of 2,010 subjects and 20,092 atomic claims, checked against Wikipedia and a curated web tier of 133 quality-scored domains, labels each claim supported, refuted, or insufficient. On a uniform random sample of GPT-5-mini claims, 68.4% are supported, 1.2% refuted, and 30.5% unverifiable; the true rate falls from 94.0% at hop 1 to 56.0% at hop 6 while falsehood stays below 2%. Cross-model views reveal only 7.3% subject overlap, and persona experiments (left/neutral/conservative framings) shift emphasis — 37 significant Bonferroni-corrected Wilcoxon effects — with factual precision essentially unchanged. As a demonstrator it is compelling, but methodology and statistics largely reside in a companion paper: the gpt-4.1-nano judge's validation is deferred, cross-model scale is unequal, and headline rates carry no confidence intervals.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Materializing ~1.3M articles from parametric memory makes the availability bias of fixed benchmarks like MMLU measurable and browsable.
- For GPT-5-mini, the dominant long-form failure is unverifiability (30.5%) rather than falsehood (1.2%), with 68.4% of claims supported on a uniform random sample.
- True rate declines with BFS depth (94.0% at hop 1 to 56.0% at hop 6) while the false rate stays below 2%, showing evidence scarcity — not assertion error — drives the frontier.
- The three model families foreground largely different entities, sharing only 7.3% of subjects (entity Jaccard 0.17–0.22).
- Editorial personas shift lexical framing (37 significant Bonferroni-corrected Wilcoxon effects out of 648 comparisons) with factual precision essentially unchanged.
Strengths
Unusually open, large-scale artifact
~1.3M articles (GPT-5-mini ~1M; DeepSeek-V3.2 and Llama-3.3-70B ~120K each) with prompts, articles, verdicts, and code under CC BY 4.0, stable per-entity URLs (llmpedia.net/<model>/<Subject>.html), and a client-side index keeping a million static pages searchable (§4).
Anti-contamination design in verification
Evidence is retrieved using only the subject name so an article can never shape its own evidence (§2), and the supported/refuted/insufficient triad separates contradiction from silence — the distinction the availability-bias argument rests on.
The depth gradient makes the core finding inspectable
Candid, nearly complete limitations section
Personas as a deterministic, controlled variable
Weaknesses
All load-bearing methodology is deferred to the companion paper
Small proprietary judge on the hardest cases
No uncertainty quantification on headline rates
Cross-model comparison confounded by deliberate scale asymmetry
Wikipedia-based verification is partially circular
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a demonstration-track submission: an interactive walkthrough of LLMPEDIA (llmpedia.net), a ~1.3M-article encyclopedia materialized from parametric memory, with all methodology attributed to the companion paper (Saeed & Razniewski 2026, arXiv:2603.24080). Its claims are: (1) fixed benchmarks like MMLU suffer availability bias, and materialization exposes the unmeasured rest; (2) for GPT-5-mini on a uniform random sample, claims are 68.4% supported, 1.2% refuted, 30.5% unverifiable — "silence, not error" (§1); (3) factuality degrades with BFS depth — Table 2 shows true rate 94.0% (hop 1) → 80.3% (hop 3) → 56.0% (hop 6), random 68.4%, frontier (absent from Wikipedia, web evidence) 57.6% — while the false rate never exceeds 2.0% and unverifiable rises to 41.8% on the frontier; (4) the three model families share only 7.3% of subjects (entity Jaccard 0.17–0.22, §5.3); (5) editorial personas shift framing — 37 significant paired Wilcoxon effects after Bonferroni correction over 648 comparisons — with precision essentially unchanged (≤3.6 pp within a cell, §5.4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LLMPEDIA?
A live research website (llmpedia.net) hosting about 1.3 million encyclopedia articles written entirely from the internal (parametric) knowledge of GPT-5-mini, DeepSeek-V3.2, and Llama-3.3-70B, without web retrieval during generation. A sampled subset of claims is fact-checked against Wikipedia and 133 curated web domains, and visitors can browse every claim verdict at a stable URL.
How accurate is the LLM-generated encyclopedia?
On a uniform random sample of GPT-5-mini claims, 68.4% were supported by external evidence, only 1.2% were refuted, and 30.5% could not be verified either way. Verification covered a stratified audit of 2,010 subjects and 20,092 claims; most of the ~1.3M pages remain unaudited, and the unverifiable share grows with distance from the seed entity.
Can I use LLMPEDIA as a reference encyclopedia?
No. The authors describe it as a research and auditing artifact, not a reference work: every article is AI-generated, may be fabricated, conflated, or outdated, and a page's presence does not even guarantee its subject exists. The site carries this warning prominently.
Where are the full methodology and statistics?
In the companion paper (Saeed and Razniewski 2026, arXiv:2603.24080), which covers the sampling design, judge validation against human annotations and FActScore, and corpus-level findings. The demonstration paper describes the browsing experience and previews the headline numbers.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure contents are known only through their captions; the figure images themselves are not part of the text extraction.
- The extraction runs from the abstract through the reference list with no visible truncation; however, validation and sampling statistics cited from the companion paper (arXiv:2603.24080) are not contained in this text and could not be independently checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.