Authority Bias in Conversational Search Engines for Academic Paper Recommendation
Uthman Jinadu, Parsa Ghazvinian, Anjila Budathoki, Benjamin M. Ampel, Rajshekhar Sunderraman, Yi Ding
cs.AI
Quick verdict
Counterfactual audit: LLM recommenders flip ~39% on authority metadata alone; rigorous, reproducible, but single-draw sampling noise may inflate flips.
Summary
This paper causally tests authority bias in LLM-based academic paper recommendation through a content-controlled counterfactual audit. Holding title and abstract fixed, the authors vary only authority metadata (venue, author h-indices, citations, affiliation) across original, flipped (high↔low swap), and boosted (mid-tier papers given elite metadata) conditions, over 250 queries on 25 CS topics and eight LLMs (five open-weight 7–9B models, three frontier closed-weight APIs), yielding 17,898 parsed runs. Authority weights come from a 15,000-run 1:N flip pilot fit with logistic regression and dominance analysis; venue prestige dominates (0.353) while affiliation is near zero. Headline findings: 39.2% of recommendations flip under swapped metadata and 21.7% under boosting; 68.2% of boosted-condition flips move toward higher authority, whereas flipped-condition flips are non-directional (40.9%). Susceptibility varies 2.83×, from gpt-5.4 (17.4%) to Llama 3.1:8b (49.2%), with frontier models most resistant. Mild anti-authority instructions backfire on all three frontier models at group level (+3.9pp, McNemar p=.02), while strong content-first prompting cuts flips 12.9pp. A say-do gap emerges: authority mentions drop −24.0pp against −12.9pp for behavior. Main limitations: a single-turn top-1 in-context setting, abstracts truncated to 400 characters, one draw per cell with no test-retest noise estimate, and CS-only coverage.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Holding title and abstract fixed, authority metadata alone changes 39.2% (flipped) and 21.7% (boosted) of top-1 recommendations across eight LLMs.
- Boosted-condition flips move toward higher authority (68.2%) while flipped-condition flips are non-directional (40.9%).
- Venue prestige is the dominant authority signal (weight 0.353) and institutional affiliation contributes near zero (0.031).
- Mild anti-authority prompting backfires on all three frontier models at group level (+3.9pp, McNemar p=.02), while strong content-first prompting cuts flips 12.9pp to a 31.4% residual.
- Debiasing suppresses authority mentions (−24.0pp) about twice as fast as authority-driven flips (−12.9pp), so surface auditing underestimates behavioral bias.
Strengths
Causal identification by construction
Content (title+abstract) is fixed within each item and the original condition serves as its own control, so recommendation changes are attributable to metadata without an external relevance gold standard (§3, §4.4, Table 2).
Derived and sensitivity-checked authority weights
Weights come from a 15,000-run 1:N flip pilot via z-scored logistic regression fused with dominance analysis (VIFs <5, stable per-model rank ordering, Table 16); Table 17 re-runs the direction analysis under six weighting schemes and the conclusion survives all but citations-only (p=0.31).
Scale plus full artifact release
Statistical candor
Diagnostic novelty
Weaknesses
No test-retest noise floor
Construct validity of 'bias'
'Directional' framing oversells a mixed result
Tier comparison confounds and small boost effect
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether conversational LLM recommenders judge academic papers on content or on authority signals, and answers with a content-controlled counterfactual audit. Central claims: (1) holding title and abstract fixed, authority metadata alone changes 39.2% of top-1 recommendations under the flipped (high↔low swap) condition and 21.7% under the boosted (mid-tier papers given elite metadata) condition, pooled across three instruction variants (Table 5); (2) the pull toward authority is directional under boosting — 68.2% of flips move toward higher composite authority — but non-directional under swapping (40.9%, Table 6); (3) susceptibility varies 2.83× across eight models, from gpt-5.4 (17.4%) to Llama 3.1:8b (49.2%), with all three frontier closed-weight models most resistant (Table 7); (4) mild anti-authority instructions backfire on the three frontier models at group level (+3.9pp, McNemar p=.02), while the strong content-first instruction cuts flips 12.9pp to a 31.4% residual (Tables 11–12); (5) a say-do gap: debiasing suppresses authority mentions (−24.0pp) roughly twice as fast as authority-driven behavior (−12.9pp, Table 13). Venue prestige dominates the derived weights (0.353); affiliation is near zero (0.031, Table 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is authority bias in LLM paper recommendation?
It is the tendency of an LLM recommender to favor papers based on authority signals — venue prestige, author h-indices, citation counts, affiliations — rather than content. This paper shows that when only the metadata changes (title and abstract fixed), 39.2% of top-1 recommendations change under a metadata swap and 21.7% under metadata inflation.
How large is the bias in frontier models like GPT-5.4 or Claude?
Smaller than in open-weight models but not zero: gpt-5.4 flips 22.4% and claude-sonnet-4-6 24.8% of the time under metadata swap, versus 63.2% for Llama 3.1:8b. Across the eight models tested, susceptibility varies 2.83-fold.
Can prompting reduce authority bias?
Only partially. A strong 'content-first' instruction cuts flips by 12.9pp (to a 31.4% residual), but a mild anti-authority instruction actually increases flips on all three frontier models at group level (+3.9pp). Models also stop mentioning authority (−24.0pp) faster than they stop acting on it (−12.9pp), so surface audits understate the bias.
Are the code and data available?
Yes. Code and prompt templates are on GitHub (jinaduuthman/Authority-Bias-In-Conversational-Search-Engine), and the full dataset of 17,898 per-run model responses (20,148 with the gpt-4o-mini ablation) is on Hugging Face (uthmanjinadu/authority-bias-paper-recommendation).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction is complete through Appendix G with no truncation detected; PDF line breaks split some words and numbers mid-token, which I reconstructed from context (low risk).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.