Which Medical Questions Deserve Rationales? Perturbation-Sensitive Selection for Robust QA
Yuexin Wu, Dayou Yu, Vasile Rus
cs.CL · cs.AI
Quick verdict
Perturbation-based rationale selection for medical QA: accuracy gain significant only on AfriMed-QA; robustness gains consistent in direction; preliminary.
Summary
The paper reframes data selection for medical QA: given questions that already have answer labels, which should additionally receive rationale supervision under a fixed rationale-token budget? The proposed RMS-RSP appends each candidate rationale to the question, perturbs hidden states only at rationale tokens (RMS-scaled Gaussian noise at layers −2/−4/−8, four draws each, α=0.10) in an answer-trained MedGemma-4B-IT LoRA model, and ranks rationales by the root-mean-square shift of the gold-versus-best-distractor margin. Across five datasets (AfriMed-QA, MedExpQA, MedExQA, PubMedQA, MedMCQA), against Random, naive scorers (entropy, margin, length), five recent reasoning-data selectors, and a Signed-RSP ablation, RMS-RSP's locked-budget accuracy averages 60.61% versus 60.08% for Random, with the only statistically resolved gain on AfriMed-QA (+1.44 points, 95% CI [0.21, 2.77]); its Token-AUBC (0.5960) is below Random (0.6021) and LALP (0.6045). Its main empirical advantage is option-order robustness: REACC +1.91 and RECON +2.85 points over Random on average, directionally positive on all five datasets, though only AfriMed-QA resolves significance. An unbudgeted all-rationales reference reaches 63.74% macro accuracy but consumes 29–254× more rationale tokens and does not uniformly improve robustness. Limitations include a single 4B model, small pools with selected sets as small as four rationales, no multiple-comparison correction, and no code links in this draft.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under a fixed rationale-token budget, RMS-RSP's main reproducible advantage is improved invariance to answer-option reorderings (REACC +1.91, RECON +2.85 points vs Random on average), directionally consistent across all five datasets.
- Locked-budget accuracy gains are not universal: RMS-RSP averages 60.61% vs 60.08% for Random, with a statistically resolved gain only on AfriMed-QA (+1.44 points).
- The RMS aggregation trades clean accuracy for robustness relative to the signed-mean ablation (Signed-RSP: 60.87 macro accuracy but lower REACC/RECON).
- Training on all pool rationales raises macro accuracy to 63.74% but costs 29–254× more rationale tokens and does not uniformly improve robustness (loses REACC/RECON point estimates on PubMedQA).
- Rationale-token boundary sensitivity is a mechanism-aligned curation signal for candidate rationales, not a universal difficulty score or pre-annotation acquisition method.
Strengths
Claim–evidence alignment is unusually disciplined
Abstract, §5.1 and §6 explicitly disclaim universal accuracy gains and frame all-rationales as an unbudgeted reference, not a competitor; the strongest claim is scoped to permutation invariance, which Table 3/Table 9 support directionally on all five datasets.
Mechanistically targeted score with a clean ablation
§3.2 perturbs only rationale-token hidden states and scores the gold-vs-best-distractor margin shift, tying the signal to exactly what is acquired; the Signed-RSP ablation (§5.3) cleanly isolates aggregation: signed wins macro accuracy (60.87 vs 60.61) but loses robustness (REACC 40.23 vs 42.60).
Confounds actively controlled
Consistent robustness signal across heterogeneous datasets
Transparent statistical reporting
Weaknesses
Accuracy evidence rests on one dataset
Underpowered experimental scale
No sensitivity analysis for the score's own hyperparameters
No multiple-comparison correction
Bookkeeping inconsistencies and withheld artifacts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a resource-allocation question, not a modeling one: given medical QA items that already carry answer labels, which should additionally receive rationale supervision within a fixed rationale-token budget? The proposed selector, RMS-RSP (§3.2), appends each candidate rationale to the question, perturbs hidden states only at rationale tokens with RMS-scaled Gaussian noise (layers {−2,−4,−8}, K=4 draws each, α=0.10) in an answer-trained MedGemma-4B-IT LoRA model, and ranks candidates by the root-mean-square shift of the gold-versus-best-distractor logit margin. Selected rationales train a rationale branch with one-to-one answer-replay pairing (§3.3), preventing a selector from winning merely by changing answer-supervision volume. The claims are deliberately narrow: the abstract, §5.1 and §6 state that universal accuracy gains are not established and that the defensible result is improved invariance to answer-option reorderings. This claim–evidence alignment is the paper's most distinguishing property.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RMS-RSP?
A data-selection score for medical QA that decides which already-labeled questions should get rationale (explanation) supervision under a fixed token budget. It perturbs a model's hidden states only at the rationale's tokens and measures how much the margin between the correct answer and the strongest distractor changes; rationales whose representations most affect this boundary are selected first.
Does RMS-RSP improve medical QA accuracy?
Not reliably. Its average locked-budget accuracy is 60.61% vs 60.08% for random selection, and only one of five datasets (AfriMed-QA, +1.44 points) shows a statistically resolved gain. Its full-curve accuracy-per-token metric (Token-AUBC) is actually below random selection.
What does the paper actually show works?
Robustness to answer-option reordering: after training with RMS-RSP-selected rationales, accuracy under three option permutations (REACC) and semantic consistency (RECON) improve by 1.91 and 2.85 points on average over Random, with the same direction on all five datasets, though only AfriMed-QA reaches statistical significance.
Is the code or data available?
Not in this draft. Appendix E states that artifacts (selected UIDs, scores, predictions, per-run metrics) exist and that code and split manifests should be released with the submission, but the anonymous version omits repository links. The five datasets and MedGemma-4B-IT are public.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is extracted only as axis-label text; per-dataset interval claims in this review rely on Table 9 and §5.2, which are present in full.
- Appendix E's missing repository links reflect intentional anonymization stated in the paper, not an extraction loss.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.