Epistemic orientation predicts legislative effectiveness among members of the US Congress
Segun Aroyehun, Stephan Lewandowsky, David Garcia
cs.CL
Quick verdict
Claims evidence-oriented speech (EMI) tracks ideology, holds across platforms, and predicts legislative effectiveness; solid associations, causality unproven.
Summary
This study asks whether the epistemic orientation of individual members of the US Congress — the balance of evidence-oriented versus intuition-oriented language — is a stable personal attribute and whether it relates to legislative behavior. Using the Evidence-Minus-Intuition (EMI) score, a hybrid measure combining ratings from three open-weight LLMs (Llama-3.1-8B, Qwen2.5-7B, Apertus-8B) with mGTE embedding similarity to evidence/intuition anchors, the authors score floor speeches from 1873 to 2024 and legislators' tweets from 2013 to 2022. Three results emerge: (1) more ideologically extreme legislators (absolute DW-NOMINATE) use less evidence-oriented language (b = -0.116 to -0.148 across specifications); (2) EMI is moderately consistent across floor and Twitter (r = 0.442) yet systematically lower on Twitter (b = -0.150); (3) EMI positively predicts Legislative Effectiveness Scores (b = 0.163, shrinking to 0.129 with topic and volume controls; N = 13,682 member-Congress observations). Mixed-effects models with legislator random intercepts, Congress/year-quarter fixed effects, and CR2 cluster-robust errors support the estimates. Limitations: all evidence is observational with an explicit non-causal disclaimer despite the title's 'predicts'; effect sizes are modest (EMI-LES r = 0.141); the ideology coefficient shifts across specifications whose samples also change; zero-value LES handling and the EMI standardization reference are unspecified; no code availability statement appears.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- More ideologically extreme members of Congress use less evidence-oriented language in floor speeches (Table 2: b = -0.148 in the fullest specification).
- EMI is moderately consistent across floor speeches and Twitter (r = 0.442), but is systematically lower on Twitter than on the floor (b = -0.150).
- Floor-speech EMI positively predicts Legislative Effectiveness Scores even after extensive political, institutional, demographic, topical, and volume controls (Table 6: b = 0.129 in the fullest specification, N = 13,682).
- Epistemic orientation is therefore a stable yet context-sensitive individual-level attribute of legislative communication.
Strengths
Well-validated, transparent measurement instrument
EMI averages three open-weight LLM ratings (Llama-3.1-8B, Qwen2.5-7B, Apertus-8B; prompt fully reproduced in Fig 4) with mGTE anchor similarity, and reaches AUC 0.825 on 592 human-annotated segments versus 0.791 for the prior word-embedding approach (DeLong Z = 2.59)
Effects survive communication-volume and topical controls
Table 6's fullest specification adds CAP topic composition plus log speech count and token count; the EMI coefficient survives at b = 0.129 [0.106, 0.152], addressing the obvious 'they just talk more' confound
Careful inference for repeated measures
Bidirectional cross-platform checks and honest causal disclaimer
Weaknesses
'Predicts' framing exceeds the contemporaneous associational design
Cross-model comparability is undermined by simultaneous sample changes
Outcome transformation for LES is never specified
Standardization reference for EMI is unspecified, affecting cross-platform comparisons
No code availability statement; validation inherited from prior work
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks three questions about individual members of the US Congress: (1) does ideological extremity relate to evidence-oriented language; (2) is epistemic orientation consistent across floor speeches and Twitter/X; (3) is EMI associated with Legislative Effectiveness Scores (LES). Data span Congressional Record speeches 1873-2024 (43rd-118th Congress) and member tweets 2013-2022, linked to DW-NOMINATE ideology, LES (Volden-Wiseman), and institutional/demographic metadata. Headline results: extremity negatively predicts EMI (Table 2, Model 3: b = -0.148, 95% CI [-0.175, -0.121]); cross-platform consistency (Pearson r = 0.442, CI [0.418, 0.466]) with EMI lower on Twitter (Table 5: b = -0.150, CI [-0.242, -0.057]); EMI predicts LES (Table 6: b = 0.163, falling to 0.129 with topic and volume controls, N = 13,682 member-Congress observations). The Discussion explicitly disclaims causality, though the title's 'predicts' is directional.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the EMI score used in this study?
EMI (Evidence-Minus-Intuition) is a text measure of how much language relies on verifiable facts and reasoning versus intuition, anecdotes, and emotional or belief-based appeal. In this paper it combines ratings from three open-weight language models with embedding similarity to evidence/intuition anchor terms; positive values mean more evidence-oriented language. It matched human annotations with AUC 0.825 on 592 segments.
Does evidence-based speech make lawmakers more effective?
No. The paper reports that members whose floor speeches are more evidence-oriented tend to have higher Legislative Effectiveness Scores (r = 0.141; regression coefficient 0.129 after extensive controls), but the authors themselves state the results do not permit causal inference — more effective legislators might simply speak differently, or unmeasured factors could drive both.
What data does the study use?
Congressional Record floor speeches from 1873 to 2024, tweets by members of Congress from 2013 to 2022, DW-NOMINATE ideology scores, Volden-Wiseman Legislative Effectiveness Scores, and institutional and biographical metadata (party, chamber, seniority, committee chairs, vote share, etc.).
Is the study reproducible / is code available?
The paper publishes its exact LLM prompt and uses open-weight models (Llama-3.1-8B, Qwen2.5-7B, Apertus-8B) plus the public mGTE embedder, and all underlying datasets are public. However, the provided text contains no explicit code release statement, and re-scoring 150 years of speeches requires substantial GPU compute.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 appear in the extraction only as captions and axis-label text; visual patterns (binning, party-wise fits) are assessed from captions and the reported table statistics, not from inspecting the plots themselves.
- The provided text contains no Data Availability or Code Availability statement; 'codeAvailable: false' reflects that verified absence, not a stated restriction on reuse.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.