MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering
Erfan Nourbakhsh, Ke Yang, Anthony Rios
cs.CL
Quick verdict
Frozen-VLM linear probes beat prompting & medical VLMs on Med-VQA per this well-controlled study; supervised vs zero-shot fairness is the main caveat.
Summary
MedProb trains per-layer multinomial logistic-regression probes on frozen VLM hidden states (last input token) to predict multiple-choice Med-VQA answers, comparing against zero-shot prompting, agentic frameworks, and medical VLMs on PATH-VQA (500-sample test), SLAKE, and VQA-RAD, plus OmniMedVQA. Across 28 models, probing consistently beats prompting (InternVL3-1B: 79.32% vs 67.83% average accuracy; best probe Qwen2.5-VL-32B at 87.35%), exceeds the strongest medical VLM (InfiMed-RL-3B, 78.81%) and agentic baselines (UCAgents, 69.80%), and narrows the small-vs-large model gap. Across 14 matched general/medical pairs, medical adaptation does not consistently improve linear decodability (6/14 favor the general model); RL-based MediX-R1 is the exception (+11.31 pp at 2B). Probing is less sensitive to answer position (first-vs-last gaps 0.14-3.58 pp vs 0.44-10.49 pp under prompting). Controls are unusually thorough: option-likelihood scoring shows the probing advantage (78.6% vs 64.4%) exceeds generation artifacts; image-ablation/shuffle controls (6.2-18.3 pp drops) confirm visual dependence; McNemar tests with bootstrap CIs show 8/9 comparisons significant; MLP probes do not beat linear ones; PATH-VQA subsampling is seed-stable. Main limitations: probes use up to 1,000 labeled benchmark examples versus zero-shot baselines; evaluation is closed-ended MC with constructed distractors; test sets are small (263-500); and some appendix prompting scores (e.g., 6-10% on OmniMedVQA) suggest answer-extraction failures that may inflate the reported gap.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Linear probes on frozen VLM hidden states recover substantially more multiple-choice Med-VQA signal than zero-shot prompting, with the best probe (Qwen2.5-VL-32B) reaching 87.35% average accuracy versus 78.81% for the strongest medical VLM (InfiMed-RL-3B) and 69.80% for the best agentic system (UCAgents).
- The small-versus-large model gap is much smaller under probing than prompting, implying generation-based evaluation underestimates answer-relevant signal in smaller VLMs.
- Across 14 matched general/medical pairs, medical adaptation does not consistently improve linear decodability of answer signal (general wins 6/14); RL-based MediX-R1 is the exception (+11.31 pp at 2B, +3.72 pp at 30B).
- Free-text prompting exhibits answer-position bias up to 10.49 pp while probing's first-vs-last gap stays within 0.14-3.58 pp, so probing is far more stable though not immune.
- The probing advantage is not attributable to generation artifacts (option-likelihood ladder 59.8% -> 64.4% -> 78.6%) or textual shortcuts (image-shuffle controls drop accuracy 6.2-18.3 pp).
Strengths
Unusually thorough confound controls
Option-likelihood scoring (Appendix E) shows probing's advantage (78.6%) exceeds prompting (59.8%) and likelihood scoring (64.4%), ruling out generation artifacts; image-ablation and image-shuffle controls (Table 3) drop accuracy 6.2-18.3 pp, excluding textual-shortcut explanations; MLP probe ablation (Appendix L.1) shows the linear constraint is not the bottleneck.
Scale with a matched-pair design
28 VLMs including 14 matched general/medical pairs sharing backbone and vision encoder (Table 2, Table 6), isolating domain adaptation; per-layer probing curves for every pair (Appendix I) localize where medical adaptation helps or hurts.
Above-standard statistical transparency
Honest scope framing
Strong reproducibility documentation
Weaknesses
Supervised-vs-zero-shot asymmetry in headline comparisons
Constructed multiple-choice task simplifies the problem
Below-chance appendix prompting scores suggest baseline measurement error
Small test sets cap statistical power
Position-bias conclusion partly confounded
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MedProb reframes Med-VQA evaluation as a representational question: rather than asking what a VLM generates, it asks what is linearly decodable from frozen hidden states. The paper trains per-layer multinomial logistic regression probes on the last-token hidden state of 28 VLMs (Section 3), evaluates on PATH-VQA (500-example stratified test subsample), SLAKE (415, full test), and VQA-RAD (263), with an OmniMedVQA extension (Appendices F-G), and compares against zero-shot prompting, three agentic frameworks, and nine medical VLMs (Table 1). Three headline claims: (1) probing recovers substantially more answer signal than prompting and beats medical VLMs and agentic systems; (2) the small-vs-large model gap shrinks under probing; (3) across 14 matched general/medical pairs, medical adaptation does not consistently improve linear decodability. A fourth claim covers positional bias: prompting swings up to 10.49 pp with answer placement while probing stays within 0.14-3.58 pp (Figure 6). The authors are explicit that MedProb is a measurement instrument, not a clinical system.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MedProb and how does it work?
MedProb freezes a vision-language model (VLM), extracts the hidden state of the last input token at every transformer layer, and trains a simple multinomial logistic regression to predict the correct multiple-choice answer for medical VQA questions. No VLM parameters are updated; only the lightweight linear probe is trained, using up to 1,000 labeled examples from the benchmark's training split, with the best layer chosen on a separate validation set.
Does MedProb beat GPT-4 or Med-Gemini?
Not directly - those models are cited as background but not evaluated. The compared baselines are open medical VLMs (InfiMed-RL-3B, Aloe-Vision-72B-AR, MedGemma, etc.) and agentic systems (MAM, MMedAgent, UCAgents). The best probe (on Qwen2.5-VL-32B) reached 87.35% average accuracy versus 78.81% for the strongest evaluated medical VLM. Note the probe is supervised (up to 1,000 labels per benchmark) while those baselines were evaluated zero-shot.
Do I still need to fine-tune models for medical VQA?
For multiple-choice Med-VQA, the paper argues the answer is often already inside frozen general-purpose models: linear probing beat both medical VLMs and multi-agent pipelines on three benchmarks, and medical adaptation improved linear decodability in only 8 of 14 matched pairs (RL-trained MediX-R1 was the exception). This does not settle open-ended clinical generation, report writing, or explanation faithfulness.
Is the code and data available?
Yes. The paper links a public GitHub repository (github.com/erfan-nourbakhsh/MedProb). All datasets (PATH-VQA, SLAKE, VQA-RAD, and the open-access subset of OmniMedVQA) are public, and Hugging Face links for all 30 evaluated model checkpoints are listed in the appendix.
Why do probing scores look so much higher than prompting scores?
Three reasons supported by the paper: free-text generation introduces hallucinated reasoning chains that override correct latent answers; option-likelihood scoring shows only part of the gap comes from generation artifacts; and the probe is trained on 1,000 labeled examples while prompting is zero-shot, so part of the gain is minimal supervised elicitation rather than a like-for-like comparison.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: mid-Section 5 passages, parts of Appendices H, I and K, and reference-list middle entries were skipped by the sampler; no conclusion in this review depends on those sections.
- The Appendix H rejection-sampling results table is cut off mid-row at the end of the excerpt, so its outcome could not be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.