Route, Don't Fix: Regime-Dependent Decoding Correction and a Trajectory-Gated Router for Reliable Clinical LLM Answer Selection
Zeyu Dong, Benjamin Wang, Joyee W. Jin
cs.CL · cs.AI · cs.LG
Quick verdict
Gated decoding correction keeps 8-9.5pt TruthfulQA gains with clinical benchmarks unharmed; sound locked evaluation, but routing leans on benchmark domain tags.
Summary
The paper argues that inference-time hallucination correction for LLMs is regime-dependent: a DoLa/DeLTa-style late-layer logit correction worth about 10 points on TruthfulQA is inert on clinical multiple-choice, where instruction tuning has already collapsed terminal entropy onto one option. It introduces ALTAS, a training-free router that reads terminal entropy and late-layer linearity (R²) from one extra prefill and routes each question between greedy decoding and the correction. Using Llama-3.2-3B, Llama-3.1-8B, and Qwen2.5-14B on TruthfulQA, MedHallu, MedQA, and PubMedQA (3,590 logged questions), the always-on correction gains +11.4/+10.0 points on TruthfulQA at 3B/8B (McNemar p<10⁻¹⁰) but degenerates at 14B on Qwen2.5 through a logit-lens transfer failure; ALTAS retains +9.5/+8.3 while every clinical delta stays within ±1 point and non-significant, at 6.5% latency overhead. Strengths include a leakage-conscious protocol with locked logs, paired bootstrap and McNemar tests, a per-question oracle showing the gate captures only 47-55% of available gain, and threshold sweeps. Main limitations: the load-bearing clinical/general label is taken from the benchmark (the lexical fallback recalls only 82% of clinical items), the 15-prompt R² calibration is one draw from flipping non-viable at 3B, evidence spans one model family plus one out-of-family failure, and the gate forfeits MedHallu's +11.8 always-on gain.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Inference-time hallucination correction is regime-dependent: a correction worth about 10 points on TruthfulQA is statistically indistinguishable from greedy decoding on clinical multiple-choice because instruction tuning has already collapsed terminal entropy onto one option.
- ALTAS routes per question between greedy decoding and a late-layer trajectory correction using only terminal entropy and late-layer R2 from one extra prefill, with no trained components and 6.5% latency overhead.
- ALTAS retains 8.3-9.5 of the roughly 10-point TruthfulQA gains at 3B/8B while keeping all clinical benchmarks within a ±1-point non-significant do-no-harm band at every scale.
- A per-question oracle shows the gate captures at most 47-55% of available gain on TruthfulQA, so the bottleneck is routing, not the corrector.
- The conclusions survive sweeps over the frozen thresholds, the cosine scoring rule, and substitution of a lexical fallback for the benchmark domain label, as measured on locked logs.
Strengths
Honest reporting of failures
The paper discloses its own gate's failures: all 1000 MedHallu items routed to greedy at 3B/8B despite a validated +11.8 always-on gain (Sec. VI), the Qwen2.5-14B operator collapse (MedQA 66.8 to 0.9, Table II), and an oracle gap capped at 47-55% recovery on TruthfulQA (Fig. 3b), explicitly locating future work in gating.
Rigorous, leakage-conscious evaluation
Locked logs over 3590 questions enable counterfactual sweeps without new runs (Sec. V); paired bootstrap with 5000 replicates plus exact McNemar tests (Sec. IV-A); thresholds calibrated label-free on 15 prompts with an explicit no-gold-label argument (Sec. III-C).
Do-no-harm statistically verified
Complete method specification
Sensitivity analysis beyond checkbox
Weaknesses
Domain label circularity
Gate forfeits a validated clinical gain
Out-of-family fragility of operator and viability test
Free-form gains partly scoring-dependent; fragile calibration
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets hallucination in clinical LLM question answering and argues against fixed inference-time correctors (DoLa, DeLTa, CoVe, ITI). Its central empirical claim (Sec. I, Table III): a correction worth about 10 points on TruthfulQA is indistinguishable from greedy decoding on MedQA/PubMedQA because instruction tuning concentrates terminal probability on one option. The methodological claim: a training-free router, ALTAS, reading terminal entropy H(N) and late-layer linearity R2 from one extra prefill keeps most of the TruthfulQA gain while causing no significant harm on clinical benchmarks. A diagnostic claim: a per-question oracle (Sec. IV-C, Fig. 3b) bounds any gate at roughly twice the gain ALTAS actually captures on TruthfulQA (47-55% recovery), locating future work in gating rather than correction. Claims are stated carefully; the abstract's do-no-harm band is supported at every tested scale, and the 14B failure is disclosed rather than hidden.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ALTAS and how does it work?
ALTAS is an inference-time routing method for LLM decoding. From one extra forward pass over the prompt it reads the terminal entropy and the linearity (R-squared) of late-layer logit trajectories, then decides per question whether to apply a DoLa/DeLTa-style late-layer correction or plain greedy decoding. Nothing is trained: the decision is three scalar threshold comparisons (R2 >= 0.50, entropy > 0.5 nats, plus a clinical/general tag).
Does ALTAS improve accuracy on medical QA benchmarks?
No, and that is by design. On MedQA, PubMedQA, and MedHallu its scores stay within one percentage point of greedy decoding and the differences are not statistically significant. Its measured gains of 8.3-9.5 points come from TruthfulQA with Llama-3.2-3B and Llama-3.1-8B, where the model's output distribution is less concentrated.
How much extra latency does the routing add?
One extra prefill pass, measured at 6.5% of greedy latency on Llama-3.2-3B (a 377 ms feature pass against a 5.82 s greedy median on one RTX 5060 laptop GPU). The corrected decoding branch itself runs about 1.68x slower than greedy, but it never fires on clinical questions in the reported experiments.
Does the method transfer to other model families?
Not yet. Threshold transfer is shown only within the Llama family (3B and 8B). On Qwen2.5-14B the correction itself degenerates — MedQA accuracy falls from 66.8% to 0.9% — because reading intermediate layers through the tied output head fails on that architecture, and the paper's R2 viability test did not predict the failure.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 appear only as OCR-approximated text; visual details (per-layer curves, CI bars) could not be independently checked, though all key numbers also appear in tables.
- The text references a released implementation but contains no repository URL; code availability is inferred from the paper's wording, not verified.
- Minor internal inconsistency noted: Sec. IV-B states the correction takes PubMedQA-14B to 36.6%, while Table II's correction column reads 32.6% (36.6 matches the DoLa+DeLTa ablation).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.