When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA
Manikandan Ravikiran, Siddharth Vohra
cs.CL
Quick verdict
引入NC-MCAR指标：4个大模型在虚假专家提示后会翻转原本正确的答案（专家提示41.1%对比多数提示12.5%）；配对审计设计扎实，但缺少噪声底限基线。
Summary
该论文审计了指令微调大语言模型在多选题中，当提示词加入一句关于“其他来源回答了什么”的虚假声称时，是否会放弃此前正确的答案。每个问题先在中性提示下回答，再测试十种提示变体；七种误导条件均指向同一个固定的错误选项（多数派、专家、AI助手、学生及三个声称可靠性等级），并辅以正确提示和填充句作为对照。作者提出的NC-MCAR指标仅在模型于中性提示下答对且在提示条件下输出有效时，才将切换至被提及错误选项的行为计为采纳，从而将真正的翻转与基线错误区分开来。在GPT-5.4、Claude Sonnet 4.6、Qwen3-32B和Gemma-4-31B-it模型上，跨越MMLU-Pro和IndicMMLU-Pro（英语、印地语、孟加拉语、泰米尔语、泰卢固语）的220,000个输出中，专家模板产生的NC-MCAR聚合值为41.1%，而多数派模板为12.5%；各模型的池化率介于10.1%（GPT）至27.5%（Qwen）之间。Gemma对声称的可靠性表现出近乎单调的响应（在20%可靠性下为1.3%，90%可靠性下为58.6%），而Qwen甚至会跟随低可靠性的提示。局限性包括：未报告针对特定目标的切换底限（即填充条件切换至同一错误选项的比率），缺乏完整的转移表，未披露有效分母，核心对比缺乏配对差的置信区间，且未声明代码发布。
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- All four evaluated instruction-following models sometimes switch from a neutral-correct gold answer to the exact wrong option named by an unverified source cue, with aggregate NC-MCAR of 41.1% under the expert template vs 12.5% under majority across 220,000 outputs.
- The expert template produces the highest NC-MCAR for every model (up to 66.0% for Gemma-4-31B-it), and the expert-vs-majority contrast — same wrong option, same verb — is the cleanest comparison.
- NC-MCAR, which conditions on neutral correctness and valid cued responses, separates cue-induced instability from ordinary benchmark error (e.g., GPT raw adoption 21.4% vs NC-MCAR 10.1%).
- Models differ sharply in stated-reliability sensitivity: Gemma rises monotonically from 1.3% to 58.6% NC-MCAR across 20%/50%/90% claims while Qwen adopts at similar rates (~26–28%) regardless of the number.
- Per-language variation reflects benchmark-setting heterogeneity (non-parallel MMLU-Pro vs IndicMMLU-Pro, 62% vs 48% cross-model neutral accuracy), not a causal language effect.
Strengths
Paired fixed-distractor design isolates template effects
One wrong option x_i is sampled per item and held fixed across all seven misleading conditions (§3), so distractor plausibility cannot vary across cue-template comparisons; question, options, order, gold label, and final instruction also stay fixed.
NC-MCAR demonstrably removes baseline-error inflation
Conditioning on neutral correctness plus valid cued output (Eq. 1) matters in practice: GPT-5.4's raw misleading adoption of 21.4% drops to 10.1% NC-MCAR (Table 2), quantifying exactly the confusion the metric eliminates.
Control conditions verify the paradigm
Unusually candid scoping and limitations
Scale with paired uncertainty
Weaknesses
Noise floor designed but never reported
Headline contrast lacks paired statistics
Effective denominators withheld
No transition table or multiplicity control
No code or data release stated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: When Models Defer to Wrong Answers (arXiv:2609.08934)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is NC-MCAR?
Neutral-conditioned misleading cue adoption rate: the share of trials — among those where a model answered correctly under a neutral prompt and produced a valid answer under a misleading cue — on which it switches to the exact wrong option named in the cue. It measures answer instability rather than overall error or knowledge.
Which model was most susceptible to false expert claims in this study?
Gemma-4-31B-it followed the false expert cue in 66.0% of neutral-correct valid trials, the highest single-template rate; Qwen3-32B and Claude Sonnet 4.6 each reached 37.0%, and GPT-5.4 17.0% (Table 4). Aggregate expert-template NC-MCAR was 41.1% vs 12.5% for the majority template.
Does a 41.1% expert-cue adoption rate mean models blindly trust experts?
Not exactly. It means that on trials where the model had just answered correctly, 41.1% of valid cued trials ended on the expert-named wrong option after reading 'A subject expert chose option X'. The authors stress this measures instability under one forced-choice template, not proof about rational deference or real expert advice.
Do the multilingual results show some languages make models less robust?
No. English items come from MMLU-Pro and Indic items from IndicMMLU-Pro, which differ in origin, subject mix, and difficulty (cross-model neutral accuracy 62% vs 48%). The authors explicitly frame language differences as benchmark-setting descriptions, not causal language effects.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction flattens table layout; column assignments in Tables 2–10 were reconstructed from header order and cross-checked against in-text numbers for consistency. No truncation observed — all referenced appendices (A–P), limitations, and ethics sections are present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.