Receptiveness, Not Sycophancy: Distinguishing Engagement from Deference in Language Models
Calvin Isley, Johann Gaebler, Max Lamparth, Julia Minson, Sharad Goel
cs.CL · cs.AI · cs.HC
Quick verdict
Claims social-sycophancy metrics penalize desirable receptiveness; preregistered evidence is compelling, but overlap is partly definitional.
Summary
The paper argues that evaluations of "social sycophancy" in language models suffer a construct-validity problem because their indicators overlap with conversational receptiveness, a socially beneficial communication style. Using the Reddit AITA-YTA dataset behind ELEPHANT plus responses from GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.7 Flash, and Llama 4 Scout, the authors find total social-sycophancy scores correlate with their GPT-judged receptiveness measure at r = 0.64 (r = 0.69 on ELEPHANT's OEQ data, per the main text). Rewriting human comments with the H.E.A.R. framework—preserving verdicts in 97.9% of cases and raising receptiveness by 1.50 SD—shifts sycophancy scores markedly upward. In a preregistered Prolific experiment (n = 200; 100 item pairs), participants preferred receptive rewrites: quality ratings rose 1.50 points (human-origin) and 0.13 points (model-origin) on a 7-point scale, with parallel listening and advice-seeking preferences persisting among participants who judged the asker wrong. Finally, prompting models directly to be receptive increased substantive deference (Claude 0.47 to 0.56), whereas a draft-then-rewrite tool raised receptiveness 1.4-2.1 SD while keeping deference near invariance (0.48-0.50). Limitations: the overlap is partly definitional (validation and positivity index both constructs), rewrites may differ in fluency, outcomes are stated preferences, and all measures rest on LLM judges not validated against human labels.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Total social-sycophancy scores (ELEPHANT validation/indirectness/framing + Sharma positivity) correlate strongly with conversational receptiveness: r = 0.64 on AITA-YTA and r = 0.69 on OEQ per the main text.
- Rewriting human comments to be more receptive via H.E.A.R., while preserving verdicts in 97.9% of cases, substantially raises their measured social-sycophancy scores.
- In a preregistered experiment, participants prefer receptive rewrites (quality +1.50 points for human-origin items on a 7-point scale), including participants who themselves judge the asker to be in the wrong.
- Directly prompting models for receptiveness increases substantive deference (Claude 0.47 to 0.56), but a draft-then-rewrite tool raises receptiveness 1.4-2.1 SD while keeping deference near invariance (0.48-0.50), demonstrating 'receptive independence' is achievable.
Strengths
Interventional design cleanly isolates style from substance
H.E.A.R. rewrites preserve the substantive verdict in 97.9% of cases (95% CI [97.1%, 98.5%], verified by a separate comparator prompt, Sec. E.2) while raising receptiveness +1.50 SD [1.45, 1.54], so sycophancy-score shifts in Fig. 2 are attributable to communication style, not conclusions.
Preregistered, IRB-approved human experiment with released data
AsPredicted (ng22sb) and Harvard IRB25-1330 documented; 200 participants x 5 items over a 100-pair bank with >=8 ratings per item, randomized order, preregistered quality exclusion (4 dropped), and length-robustness regressions (Table S8) that leave estimates essentially unchanged.
Receptiveness judge is calibrated, not ad hoc
Policy-relevant mitigation with a clean spillover demonstration
Audit of inherited benchmark judges
Weaknesses
Partial circularity in the headline coupling
Every measurement layer is an unvalidated LLM judge
Fluency confound in rewrites
Internal inconsistencies and underpowered contrast tests
Limited external validity of preference outcomes
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a construct-validity problem: benchmarks for social sycophancy (validation, indirectness, framing sycophancy from ELEPHANT; positivity from Sharma et al. 2024) may conflate inappropriate deference with conversational receptiveness — hedging, acknowledging the other's perspective, emphasizing common ground — a style social psychology links to better disagreement outcomes. The organizing concept is the 2x2 in Table 1: a model can be substantively independent yet receptive, a combination the authors call receptive independence. Four claims are advanced: (1) total social-sycophancy scores track receptiveness (r = 0.64 on AITA-YTA; r = 0.69 on ELEPHANT's OEQ per the main text); (2) H.E.A.R.-based rewrites of human comments that preserve the verdict (97.9% preservation; receptiveness +1.50 SD, CI [1.45, 1.54]) nonetheless raise measured sycophancy (Fig. 2); (3) in a preregistered Prolific experiment participants prefer the receptive versions; (4) a draft-then-rewrite intervention increases receptiveness 1.4-2.1 SD without moving substantive deference Syc(pi) off invariance, whereas direct prompting does (Claude Sonnet 5: 0.47 to 0.56).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the difference between sycophancy and receptiveness in AI models?
Substantive sycophancy is when a model changes its actual judgment to match the user; social sycophancy covers surface behaviors like praise, validation, and hedged suggestions. Conversational receptiveness is the skill of acknowledging the other person's perspective while keeping your own view. This paper shows that social-sycophancy metrics cannot tell receptiveness apart from sycophancy, so they can penalize a communication style that human subjects actually prefer.
Did participants in the study prefer sycophantic AI responses?
No. The substantive conclusion was held fixed — the rewritten responses still said the poster was in the wrong in 97.9% of cases. Participants preferred the same verdict delivered more receptively: quality ratings rose 1.50 points for human comments and 0.13 points for model responses on a 7-point scale, and the preference persisted even among participants who themselves thought the original poster was in the wrong.
How can an LLM be receptive without becoming sycophantic?
The paper's draft-then-rewrite tool: the model first answers normally, fixing its substantive judgment, then a second pass rewrites that answer using the H.E.A.R. framework (hedge, emphasize agreement, acknowledge, reframe) while keeping the verdict. This raised receptiveness by 1.4-2.1 standard deviations while deference stayed near the invariance point (0.48-0.50), whereas simply prompting the model to be receptive leaked into deference (Claude 0.47 to 0.56).
Is the code and data for this paper available?
Yes. Code, prompts, and the experimental data (participant responses, stimuli, and scores) are released on GitHub (calisley/receptiveness-sycophancy), and the human study was preregistered on AsPredicted (aspredicted.org/ng22sb.pdf) with Harvard IRB approval (IRB25-1330).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction omitted middle content (part of the experiment-methods narrative under the 'Receptiveness Is Perceived' header), the reference-list tail, portions of Appendix F, and several prompt listings; figures are recovered as garbled text, so figure-level judgments rest on in-text numbers.
- The exact OEQ subset and judge version behind the main-text r = 0.69 versus Appendix r = 0.67 cannot be resolved from the extracted text.
- Model/version names (GPT-5.6 Terra/Luna, Claude Sonnet 5, Gemini 3.7 Flash) are as printed; no external verification was possible in an offline review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.