Modelpedia: A Catalog of Model Findings for the Meta-Science of AI
Franciszek Bernat, Dawid Płudowski, Michał Jan Włodarczyk, Luca Longo, Jianlong Zhou, Andreas Holzinger, Riccardo Guidotti, Wojciech Samek, et al.
cs.LG
Quick verdict
Paper presents LLM pipeline that catalogs >1000 model findings from ICLR 2024-25; promising infrastructure, but validation is LLM-only and recall unmeasured.
Summary
The paper introduces Modelpedia, an LLM-assisted framework that extracts third-party claims about existing AI models ('findings') from published papers and stores them in a structured public catalog, where each finding is linked to the model, dataset, method, concept, and source paper it concerns. A keyword-based scorer (seven vocabularies with weights and per-group ceilings, plus a reviewer-agreement gate) screens OpenReview metadata for ICLR 2024 and 2025, selecting roughly the top 15% (972 accepted papers: 521 from 2024, 451 from 2025). A Qwen3.8-27B-FP8 extractor with enforced entity-to-quote citations, deterministic citation checks, and a registry-matching step then produced 1026 findings, which a Claude Opus 5 validator nominally confirmed, disagreeing on the 'key metric' field for about 20% of records. Treating the catalog as data, the authors report a meta-analysis: correlational evidence dominates while interventional evidence concentrates in open-weight models; Llama, GPT, and Mistral families are the most analyzed, and vision evaluation concentrates on ImageNet and COCO. The design is careful — anti-hallucination constraints, public code and prompts, honest caveats per finding — but the evaluation is thin: validation relies on one LLM judging another with no human gold set, so precision is unquantified and the screening threshold's recall is unknown; the ~20% key-metric disagreements are not reported as resolved; and the web interface receives no user evaluation. A promising infrastructure contribution whose empirical claims remain preliminary.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Third-party claims about released AI models can be automatically extracted with enforced source citations and stored as first-class, structured findings linked to model, dataset, method, concept, and source.
- A keyword-screened prototype over accepted ICLR 2024/2025 papers yielded 1026 findings, validated by a second LLM with disagreement confined to the key-metric field in ~20% of records.
- The meta-analysis shows the community relies predominantly on correlational evidence, with interventional evidence concentrating in open-weight models.
- Llama, GPT, and Mistral families dominate model investigation, and vision evaluation concentrates on ImageNet and COCO while language benchmarks are more diverse.
- Model findings deserve a shared, citable, versioned catalog as infrastructure for the meta-science of AI.
Strengths
Serious anti-hallucination engineering
The extractor must attach a source quote to every entity, citations are deterministically checked by content-word overlap between steps, concepts come from a closed definition list, and a registry adopt/refuse stage prevents duplicates like 'GPT-4o' vs 'ChatGPT-4o' (Appendix D)
Fully transparent screening
Appendix C's Table 1 publishes all seven vocabularies' weights and ceilings plus a reviewer-agreement gate; Table 2 shows the top scorers are plausible finding-rich papers (e.g., 'Vision Transformers Need Registers', 23.5)
Honest scoping of what the catalog is
Actionable meta-analysis
Weaknesses
Validation is LLM-only and implausibly clean
Unknown recall of the top-15% screen
Single extractor/validator configuration, unclear checkpoint
Inconsistent conference scope across exhibits
No baselines or user study
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Modelpedia is a tool/infrastructure paper. It argues that third-party claims about released AI models — "findings" — should be first-class, citable, versioned units of knowledge (Section 4), and implements an LLM-assisted pipeline that extracts such findings from accepted ICLR 2024/2025 papers and links each to model, dataset, method, concept, and source, plus optionally typed related work (builds-on / compared-to / context, Appendix A). The three stated contributions are the conceptual design, the prototype implementation, and a meta-analysis of the resulting catalog. Positioning (Section 1) contrasts the work with surveys, live benchmarks (LiveCodeBench, LiveBench), and documentation artifacts (model cards, datasheets) — but not with existing LLM-based literature-mining systems, a notable omission. The meta-analysis is explicitly "predominantly" ICLR 2025 (Section 3), while Figure 5 (left) claims both conferences — a scope inconsistency addressed below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Modelpedia?
It is an open, searchable catalog that uses an LLM pipeline to automatically extract claims ('findings') about existing AI models from published papers, linking each finding to the model, dataset, method, concept, and source paper. Its first version covers accepted ICLR 2024 and 2025 papers and contains over a thousand findings; the code, prompts, and web service are public.
How were the findings extracted and validated?
A keyword scorer picked roughly the top 15% of accepted ICLR 2024/2025 papers (972 total). A Qwen3.8-27B-FP8 model extracted findings under strict rules requiring quotes from the source text, and a Claude Opus 5 validator reviewed them; about 20% of records showed discrepancies limited to the 'key metric' field. No human gold-standard annotation is reported.
What did the meta-analysis of the catalog find?
Most findings rely on correlational evidence rather than interventional experiments, and interventional work concentrates almost entirely on open-weight models. Llama, GPT, and Mistral families are the most analyzed models, while vision evaluations concentrate on ImageNet and COCO; language benchmarks are far more diverse.
Does Modelpedia verify whether the findings are true?
No. The authors state findings are presented 'as-is': the catalog records each claim, its evidence type, and a caveat, but does not judge whether the source paper's methodology is sound, leaving verification as an open problem for the community.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure graphics are images and were not extracted; all figure-level numbers cited come from captions and body text only.
- arXiv metadata lists 8 authors while the byline lists 9 (Przemysław Biecek added); the review assumes the byline.
- No truncation detected: the text ends cleanly with Appendix E figure captions (page 14).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.