Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards
Kalash Shah, Kunal Singh, Snehan J, Shreyas Singh
cs.AI · cs.CL · cs.LG
Quick verdict
Claims rubric-based RL lifts medical reasoning (FV-30B: 50.1% HealthBench-Hard); gains look real but SOTA rests on self-run judge evals with no error bars.
Summary
The paper introduces Fathom-Vaidya (FV), a sequential RL framework for medical LLMs built on Qwen3-4B/8B/30B-A3B. Stage one targets diagnostic reasoning: after difficulty filtering, roughly 1,100-1,200 questions per source (MedQA, MedBullets, MedMCQA) are trained with binary-reward rule-based RL, then online rubric-guided RL in which GPT-5-thinking labels rollout errors (knowledge/causal/context) and severity weights form the reward; FV-8B reaches 28.3% on MedXpertQA-Text versus 17.4% for base Qwen3-8B. Stage two targets open-ended clinical dialogue: an actor-critic loop (GPT-5-thinking actor, Grok 3 critic) synthesizes 5,351 multi-turn conversations grounded in MedRedQA cases and HealthBench-Easy ICL examples, each paired with ICL-anchored multi-dimensional rubrics (Vaidya-Rubrics). GSPO training with gpt-4.1 as rubric judge lifts HealthBench-Hard by about 20-27 points; FV-30B-A3B reports 50.1% with the authors' system prompt, above their self-evaluated GPT-5-thinking baseline (43.79%; the model card claims 46.2). A clinician audit (500 samples, 8 clinicians) found 99.4% of dialogues valid and 97.7% of rubrics relevant. Weaknesses: no statistical testing or variance reporting despite sub-point ablation differences driving design choices (alpha selection: 27.76 vs 27.24), judge circularity between training reward and evaluation, no code/data release, and experiments confined to the Qwen3 family and text-only tasks.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Sequential rule-based then rubric-guided RL improves diagnostic reasoning without supervised CoT data: FV-8B scores 28.3% on MedXpertQA-Text vs 17.4% for Qwen3-8B (Tables 2-3).
- Training on 5,351 synthetic multi-turn conversations with multi-dimensional Vaidya rubrics lifts HealthBench-Hard by roughly 20-27 absolute points across 4B/8B/30B scales (Table 4).
- FV-30B-A3B achieves 50.1% on HealthBench-Hard with a system prompt, claimed to surpass proprietary baselines including GPT-5-thinking under the authors' evaluation protocol.
- The synthetic dialogue pipeline is reliable: 99.4% of 500 clinician-audited conversations were medically valid and 97.7% of rubrics relevant (Appendix B).
- Gains generalize out-of-distribution: +12.2 to +16.6 points on HealthBench-Professional and +1.32 to +6.68 on GPQA-Biology across scales (Table 5).
Strengths
Clinician-validated synthetic data pipeline
Appendix B: a 500-sample review by 8 clinicians (3 per sample, consensus-based) found 99.4% of dialogues medically valid and 97.7% of 5,570 rubrics relevant; error rubrics from GPT-5-thinking agreed with clinicians at 92.2% across 257 items — rare human grounding for synthetic RL data.
Transparent baseline-discrepancy reporting
The Table 4 footnote openly states that the authors' own OpenAI-API re-evaluation of gpt-5-thinking / gpt-5.2-thinking (38.58/39.95) diverges from the model cards (46.2/42.0), letting readers calibrate how much to trust the SOTA comparison.
Documented, principled reward design
Ablations isolate the dataset contribution
Multi-scale consistency
Weaknesses
No statistical uncertainty anywhere
LLM-judge circularity unquantified
Headline SOTA depends on protocol choices
Nothing released; pipeline not reproducible
No decontamination analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Fathom-Vaidya (FV) is a sequential two-stage RL recipe applied to Qwen3-4B/8B/30B-A3B. Stage 1 (§4.1) targets diagnostic QA: roughly 1,100-1,200 questions per source (MedQA, MedBullets, MedMCQA; Table 1) survive difficulty filtering (8 samples at T=1.0), split by pass rate into rule-based RL (0.4-0.7) and error-rubric RL (0.1-0.4). In the rubric stage, GPT-5-thinking labels knowledge/causal/context errors with severities 1-2, and reward = 15 − Σseverity/15, combined with an accuracy reward; the ablation (Table 6) lands on α = 1.0, i.e. pure rubric reward. Stage 2 (§3, §4.2) targets open-ended dialogue: an actor-critic loop (GPT-5-thinking actor, Grok 3 critic) produces 5,351 multi-turn conversations grounded in MedRedQA and HealthBench-Easy ICL, each scored by ICL-anchored rubrics in [−10,+10]; GSPO training uses gpt-4.1 as rubric judge (Eq. 4). Headline claims: +10.9 points MedXpertQA for FV-8B; +20-27 points HealthBench-Hard; FV-30B-A3B = 50.1% with system prompt, surpassing proprietary baselines including GPT-5-thinking.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Fathom-Vaidya?
A family of medical LLMs (FV-4B, FV-8B, FV-30B-A3B) fine-tuned from Qwen3 by Fractal AI Research using two-stage rubric-based reinforcement learning: first on multiple-choice medical questions for diagnostic reasoning, then on 5,351 synthetic multi-turn clinical conversations scored by multi-dimensional rubrics.
Does Fathom-Vaidya beat GPT-5 on medical benchmarks?
The authors report 50.1% on HealthBench-Hard (with their custom system prompt) versus 43.79% for GPT-5-thinking under their own re-evaluation. Note GPT-5's model card reports 46.2 and evaluation protocols differ, so the comparison depends on the authors' measurement setup rather than a single official leaderboard.
Is the code, dataset, or model checkpoint available?
The paper does not mention any public release of code, the Vaidya-Rubrics dataset, or model weights. The pipeline also depends on proprietary APIs (GPT-5-thinking, Grok 3, gpt-4.1), which makes independent reproduction difficult.
What benchmarks does the paper evaluate on?
MedXpertQA-Text, MedQA, MedCase Reasoning, and DiagnosisArena for diagnostic QA; HealthBench-Hard, HealthBench-Professional, and a 71-question GPQA-Biology subset for clinical reasoning and generalization.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 (theme distribution, rubric-generation scores, ablation bars, axis/theme breakdowns) are referenced but their content cannot be analyzed from the extracted text.
- The Table 4 footnote's respectively-ordering of 38.58/39.95 versus model-card 46.2/42.0 appears internally inconsistent with the table rows; treated as approximate when cited.
- Extraction may have omitted minor appendix material; the prompts that are included (D.1-D.3) appear complete.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.