FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect
Hazel H. Kim, Andrew M. Bean, Guilherme Affonso Ferreira de Camargo, Shanyu Chauhan, Felix Drinkall, Jade Kosché, Chenyang Ma, Glory Nwaugbala, et al.
cs.CL
Quick verdict
A well-built 54k-question benchmark showing LLM answers flip with question framing (binary strict accuracy 0%); novelty and scale claims need firmer support.
Summary
The paper introduces FramingQA, a benchmark of 756 contexts and 54,756 questions probing whether LLM answers change when a fixed fact is asked about under different question framings. Framing bias is injected at three nested levels — root (leading question phrasing), propositional (a biased premise attached to a neutral question), and global (premise plus leading question) — in correct-bias and incorrect-bias directions, plain and negated forms, and three paraphrases, across law (LegalBench issue-spotting and hearsay), finance (Stack Exchange posts), medicine (case reports), and robotics (PARTNR). Nine open models (3.8B–70B; Gemma 3, LLaMA 3.1, Mistral, Phi) are evaluated with greedy decoding; robotics uses GPT-4.1 mini. Mean accuracy is moderate (binary 52.4–61.0%), but strict accuracy requiring every variant of a context to be correct collapses to 0.0% for all models in the binary setting (1.4–35.2% for multiple choice). In multiple choice, correct-bias phrasing lifts accuracy to 70–90% versus a 55–62% neutral baseline, while incorrect-bias drops to 48–62%. Observed strict accuracy exceeds the independence baseline by four to nine orders of magnitude, indicating errors concentrate in specific contexts. Scale trends are claimed but binary strict-pair numbers contradict them (Mistral 7B 8.3% vs 24B 1.1%). No significance tests, contamination analysis, or code-release statement is provided; closed models appear only in robotics.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Naming the gold option in a question raises accuracy and naming a distractor lowers it while evidence is held fixed (RQ1; MC: correct-bias 70-90% vs neutral 55-62% vs incorrect-bias 48-62%).
- Strong per-question accuracy coexists with near-zero compositional robustness: strict accuracy requiring all variants of a context to be correct is 0.0% for all nine models in the binary setting and 1.4-35.2% in MC (Table 3).
- Errors concentrate in specific contexts rather than occurring independently: observed strict accuracy exceeds the independence expectation by four to nine orders of magnitude (Table 4).
- Framing components combine non-additively: propositional-level bias contributes little once the question itself is framed, with global effects carrying root effects (delta-G minus delta-R averages -1.2 pp in MC).
- The compositional framing effect extends to embodied robotics, lowering task completion and increasing replanning under incorrect-bias instructions (Fig. 7).
Strengths
Verifiable, internally consistent dataset bookkeeping
Table 1's expansion algebra checks out against Table 2: 54x594 + 2x2,609 = 37,294 binary; 28x500 + 2,328 = 16,328 MC; 162x7 = 1,134 robotics; 662 neutral = 500 MC + 162 open. Cross-referenced label counts (4x50 issue-spotting; 14+19+29+12+20 hearsay slices; six finance categories summing to 200) also reconcile.
Clean negative-control design where agreement earns nothing
Fig. 2 and Table 5 show the key device: negation flips the gold answer while the named option stays fixed, and strict accuracy requires all phrasings of a context — so a model that simply agrees with the prompt's stance cannot score. This separates framing from trivial agreement-seeking better than most sycophancy setups.
An elegant selectivity analysis
Breadth including embodied evaluation with process metrics
Honest scoping
Weaknesses
The RQ2 scale claim is contradicted by the paper's own Table 3
No inferential statistics anywhere
Unexplained numeric inconsistency between Table 3 and Table 4
Contamination and release gaps
Ecological validity limits
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FramingQA (arXiv 2609.07448) is a benchmark paper asking whether LLM answers change when the same fact is asked about under differently framed questions. The authors claim three contributions (Section 1): the first benchmark to systematically evaluate framing effects across four high-stakes domains (law, medicine, finance, robotics); a unified framework spanning root-, propositional-, and global-level bias; and the demonstration that high mean accuracy coexists with near-zero compositional robustness across nine open models (3.8B-70B). Five research questions (RQ1-RQ5, Section 2) cover framing direction, scale, semantic systematicity, error selectivity, and structural composition. The paper usefully separates framing (bias inside the question's own semantics) from sycophancy (external social pressure), and Fig. 2 shows the core design trick: negation flips the gold answer while the named option stays fixed, so blind agreement earns nothing and strict accuracy requires all phrasings of a context to be correct.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the FramingQA benchmark?
FramingQA is a benchmark of 756 contexts and 54,756 questions testing whether large language models change their answers when the same underlying fact is asked about with differently framed questions — leading phrasings, injected biased premises, and negations — across law (LegalBench), finance (Stack Exchange), medicine (case reports), and robotics (PARTNR) tasks.
What did the FramingQA paper find?
Across nine open models (3.8B-70B), average accuracy stayed moderate, but strict accuracy — requiring every phrased variant of a context to be answered correctly — collapsed to 0.0% in the binary yes/no setting for every model (1.4-35.2% in multiple choice). Leading questions moved accuracy up (correct-bias 70-90% vs a 55-62% neutral baseline in multiple choice) or down (48-62%) depending on direction, and errors clustered in specific contexts.
Is the FramingQA dataset or code publicly available?
The paper text contains no code or data release statement. Construction recipes are described in detail — question templates, expansion algebra, and public sources (LegalBench, Stack Exchange API, medical case reports, PARTNR) — so a rebuild is possible, but the exact human-validated variant set is not stated to be downloadable.
How is framing different from sycophancy in LLM evaluation?
Sycophancy benchmarks apply social pressure outside the question (expressed opinions, emotional tone, pushback after a model responds). FramingQA embeds the bias inside the question's own semantics — its stance, presupposed premise, and negation — and designs items so that simply agreeing with the prompt's named option never earns credit, because the gold answer flips under negation while the named option stays fixed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3-7 appear in the extraction only as captions and inline text labels; figure-internal data (e.g., heatmap shading in Figs. 5-6, error-bar extents) was assessed solely through the prose descriptions.
- Table 4 rows are extracted linearly; the Exp./Act. value pairing follows the printed two-row header. The Table 3 vs Table 4 'All' mismatch is quoted as printed in the paper.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.