PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models
cs.CL
Quick verdict
7-axis compact-VLM benchmark separates 97% of 861 pairs vs 44-80% for priors; rigorous judge audits, but artifacts unreleased and external validity untested.
Summary
The paper introduces PRISM-VLM, a benchmark for compact vision-language models (≤10B parameters or low-cost API tiers), built from 6,238 items recycled from 15 public benchmarks. Each item is scored on seven axes — task quality, instruction following, hallucination resistance, multi-question load, sycophancy resistance, counting, and visual discrimination — most involving per-item perturbations synthesized and graded by GPT-5 (effort=low) and human-verified; the seven axis means combine into a PScore via harmonic mean. Across 42 models, an item-level paired bootstrap separates 97.0% of the 861 model pairs, versus 44–80% for single prior benchmarks and 88.7% for their macro mean (81–88% when PScore is capped at matched item budgets). The sycophancy axis is nearly orthogonal to single-prompt accuracy (|ρ|≤0.26), and models with statistically indistinguishable PScores show sharply divergent per-axis profiles. Robustness checks include leave-one-axis-out, alternative aggregators, a cross-judge audit with Claude Sonnet 4.6 (all axes ρ≥0.96), and a human anchor (κ=0.97). Key limitations: code, prompts, and per-item annotations are promised only 'upon completion of internal review', so no result is yet independently verifiable; judge-generated perturbations graded within the same family retain bias risk with modest per-family audit samples; strongly correlated axes (mq–q ρ=0.95) reduce effective dimensionality; and the authors concede the separation claim has no demonstrated external validity for deployment outcomes.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PScore separates 97.0% of the 861 pairs among 42 compact VLMs under an item-level paired bootstrap, versus 44–80% for any single prior benchmark and 88.7% for the 15-benchmark macro mean.
- The sycophancy axis is nearly orthogonal to single-prompt accuracy (|ρ|≤0.26) and remains a distinct signal under a corrected sycophancy-only definition (ranking ρ=0.88, separation 95.9%).
- The harmonic mean makes PScore the least recoverable from task accuracy alone (r=+0.789 vs +0.865 arithmetic, +0.840 geometric).
- Cross-judge audit (Claude Sonnet 4.6, S=200/axis, all axes ρ≥0.96) and a human anchor (κ=0.97) indicate the leaderboard is not a judge-family artifact.
- Models with statistically indistinguishable PScores diverge sharply on per-axis profiles, so per-axis means should be the primary reporting unit.
Strengths
Separation is measured statistically, not by nominal score range
Item-level paired bootstrap (B=2,000, shared index multiset across all models and baselines, α=0.05) over all 861 pairs, with Benjamini–Hochberg (97.0%→96.9%) and Bonferroni (94.5%) corrections, per-model 95% CIs and rank CIs (Table 14: Gemini 3 Flash 0.719 [0.713, 0.724], rank-CI median width 1).
Unusually thorough judge-bias auditing
Cross-judge audit with a different family (Claude Sonnet 4.6, S=200 items/axis, all seven axes ρ≥0.96, Table 7), three paired generator-judge family cells (Table 19), a judge-swap control (Table 20), and an item-level human anchor (κ=0.97, Table 21) rather than rank correlation alone.
Honest self-decomposition of the headline sycophancy axis
Design choices are validated empirically, not asserted
Matched-budget control anticipates the sample-size objection
Weaknesses
No artifacts available at publication
Headline comparison partially conflates metric and item budget
Residual judge-dependence with limited item-level auditing
Effective dimensionality is well below seven
The leaderboard 'sycophancy' metric mostly measures early instability for weak models
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PRISM-VLM (arXiv 2609.27395) targets evaluation of compact VLMs — open-weight models ≤10B parameters and low-cost API tiers — arguing that inherited single-accuracy protocols either saturate (AI2D, GQA, OK-VQA, ScienceQA at the compact tier, per MMStar) or spend their score range on sub-domain difficulty rather than model separation (MMMU-Pro). The benchmark recycles 6,238 items from 15 public benchmarks, scores each along seven axes (q, if, h, mq, sc, c, vis), and aggregates by harmonic mean into PScore, evaluated on 42 models released March 2024–April 2026. Four central claims: (1) PScore separates 97.0% of the 861 model pairs under an item-level paired bootstrap, versus 44–80% for single prior benchmarks and 88.7% for the 15-benchmark macro mean; (2) the sc axis is nearly orthogonal to single-prompt accuracy (|ρ|≤0.26); (3) cross-judge and human audits rule out judge-family artifacts; (4) per-axis profiles, not the aggregate, should be the primary reporting unit for statistically indistinguishable models.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PRISM-VLM?
PRISM-VLM is a benchmark for compact vision-language models (open-weight models up to 10B parameters, or low-cost API tiers). It reuses 6,238 items from 15 public datasets and scores each along seven axes — task accuracy, instruction following, hallucination resistance, multi-question load, sycophancy resistance, counting, and visual discrimination — combining them into a single PScore via harmonic mean.
Which model tops the PRISM-VLM leaderboard?
Among the 42 compact models evaluated, Gemini 3 Flash ranks first with a PScore of 0.719; the weakest is Gemma-3-4B at 0.065, dragged down mainly by a near-zero sycophancy score (0.011) and weak multi-question performance.
How is PRISM-VLM different from MMMU or other VLM benchmarks?
Standard benchmarks give each model one accuracy number, which saturates at the compact tier. PRISM-VLM instead grades every item on seven behavioral and capability axes, most involving perturbations (misleading premises, bundled questions, planted false hints) synthesized by GPT-5. The claimed advantage is statistical: 97% of model pairs are reliably separated versus 44–80% for individual prior benchmarks.
Is PRISM-VLM's code and data available?
Not yet. The paper promises release of the pipeline (Apache-2.0), prompts, and per-item annotations (CC BY 4.0) only 'upon completion of internal review'; as of this preprint, none of the artifacts can be downloaded or independently verified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 and 18–22 are available only as extracted text and captions; visual content (correlation heatmaps, survival curves, worked samples) could not be inspected directly.
- All availability statements reflect the paper's own 'upon completion of internal review' status; codeAvailable=false reflects current, not promised, availability.
- Table 2 was extracted as a dense run-on block; values cited were spot-checked against in-text claims (e.g., Gemma-3-4B sc 0.011, Gemini 3 Flash 0.719) and are internally consistent, but individual cells were not exhaustively re-verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.