Evaluating Financial Sentiment in the Age of AI
cs.LG
Quick verdict
Finds open-source LLMs match finance-specific sentiment models on benchmarks but not economically; a sound framing limited by a 15-firm, 221-event sample.
Summary
This paper asks whether general-purpose open-source large language models improve financial sentiment measurement over established finance-specific tools. The authors evaluate twelve models—three dictionary methods (VADER, TextBlob, Loughran–McDonald), five finance-specific transformers (FinBERT variants, FinancialBERT, DistilRoBERTa, RoBERTa-Large), and four open-source LLMs (Llama 3.2, Gemma 3, Qwen 2.5, Mistral)—on two criteria: linguistic validity against the Financial PhraseBank benchmark (2,264 unanimous-annotator sentences) and economic validity on 221 earnings announcements from 15 large U.S. firms (2022 to mid-2026), relating model sentiment to EPS surprises and next-day stock returns. Qwen 2.5 reaches 92.1% accuracy, statistically indistinguishable from FinBERT-Tone's 91.7% (McNemar test), and Claude Opus 4.5 (93.6%) is likewise statistically tied, suggesting open-source models close the benchmark gap. Economically, several models' sentiment correlates significantly with earnings surprises, but none correlates significantly with next-day returns; performance is strongest for extreme earnings beats and misses and weak for moderate surprises. Key limitations: the small, purposive firm sample, an event-level bootstrap that ignores within-firm dependence across the 221 events, raw rather than market-adjusted returns, unspecified LLM versions/prompts and sentiment-score construction, and no multiple-comparison correction. The linguistic-versus-economic validity distinction is the paper's most valuable contribution.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- General-purpose open-source LLMs (best: Qwen 2.5, 92.1% accuracy) match finance-specific transformers (FinBERT-Tone, 91.7%) on the Financial PhraseBank without task-specific fine-tuning, and Claude Opus 4.5 (93.6%) is statistically tied with the open models.
- Higher benchmark classification accuracy does not translate into stronger economic associations: no model's sentiment measure is significantly correlated with next-day stock returns.
- Several models produce sentiment significantly correlated with earnings surprises, indicating sentiment tracks firm fundamentals more than short-run market reactions.
- Sentiment informativeness is concentrated in announcements with extreme earnings beats or misses (top and bottom surprise quartiles, Figure 4) and is substantially weaker for moderate surprises.
Strengths
Dual-criterion evaluation framework
Section 4 and Figure 1 explicitly separate linguistic validity (agreement with expert labels on Financial PhraseBank, sentences_allagree, n=2,264) from economic validity (association with EPS surprises and next-day returns). This framing directly addresses a real flaw in common model-selection practice and is the paper's most transferable contribution.
Honest treatment of benchmark contamination
Section 5 correctly flags that RoBERTa-Large-Financial and DistilRoBERTa-Financial were fine-tuned on Financial PhraseBank, so their >90% accuracy partly reflects train/test overlap, and re-centers the comparison on uncontaminated models (Qwen 2.5 92.1% vs FinBERT-Tone 91.7%).
Paired statistical comparisons
Frontier closed-model benchmark
Candid scoping of external validity
Weaknesses
External validity of the economic leg
Dependence-ignoring inference
Outcome measurement is raw returns
Underdocumented LLM pipeline and score construction
No incremental-information test and no multiplicity control
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether general-purpose open-source LLMs improve financial sentiment measurement over three earlier generations of NLP: dictionaries (VADER, TextBlob, Loughran–McDonald), finance-specific transformers (FinBERT, FinBERT-Tone, FinancialBERT, DistilRoBERTa-Financial, RoBERTa-Large-Financial), and four open LLMs (Llama 3.2, Gemma 3, Qwen 2.5, Mistral). Evaluation is two-dimensional: linguistic validity on Financial PhraseBank sentences_allagree (n=2,264) and economic validity on 221 earnings announcements from 15 U.S. mega-caps (2022 to mid-2026), linking model sentiment to EPS surprise (percentage deviation from Yahoo Finance consensus) and next-day raw returns. Central claims: (i) Qwen 2.5's 92.1% accuracy is statistically indistinguishable from FinBERT-Tone's 91.7% (McNemar test); (ii) Claude Opus 4.5 (93.6% accuracy, 93.3% macro F1) is likewise statistically tied with the open models; (iii) no model's sentiment correlates significantly with next-day returns, though several do with earnings surprises; (iv) informativeness concentrates in the extreme surprise quartiles (Figure 4). The claims are clearly stated, and the linguistic-versus-economic validity framing is genuinely useful for the field.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do open-source LLMs like Qwen or Llama outperform FinBERT for financial sentiment analysis?
According to this paper, they perform comparably rather than better: Qwen 2.5 reached 92.1% accuracy on the Financial PhraseBank benchmark versus 91.7% for FinBERT-Tone — a statistically insignificant difference (McNemar test) — and the closed frontier model Claude Opus 4.5 (93.6%) was also statistically tied with the open-source models. Notably, none of the LLMs required task-specific fine-tuning.
Can financial sentiment models predict stock returns?
In this study, no. Across all twelve sentiment models tested on 221 earnings announcements from 15 large U.S. firms (2022 to mid-2026), none showed a statistically significant correlation with next-day stock returns. Several models did correlate significantly with earnings surprises, suggesting sentiment tracks company fundamentals more than short-run market reactions.
What is the difference between linguistic validity and economic validity of sentiment models?
Linguistic validity measures how closely a model reproduces expert human sentiment labels on benchmarks such as the Financial PhraseBank. Economic validity measures whether the model's sentiment scores correlate with economically meaningful outcomes like earnings surprises or stock returns. This paper's central point is that a model can excel on the first and still be uninformative for the second, so benchmark accuracy alone is a poor criterion for choosing a sentiment model for finance research.
What are the main limitations of this study?
The economic analysis covers only 221 earnings events from 15 purposely selected mega-cap firms, the bootstrap inference does not cluster by firm, next-day returns are raw rather than market-adjusted, and LLM model sizes, prompts, and the sentiment-score construction are not fully documented. These limit both generalizability to smaller firms and exact reproducibility.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Tables 2 and 3 contain no numeric cells in the extracted text and Figures 1-5 are images, so class-level F1s, correlation point estimates, and CI widths are taken only from the prose; exact table values could not be verified.
- The extracted text appears truncated: no data/code availability statement or appendix is present, and 'replication files' are referenced once without access details, so codeAvailable=false reflects absence of a stated link rather than confirmed unavailability.
- Figure 4's quartile-level values and Figure 2's error-rate magnitudes cannot be assessed from the text; judgments about their robustness rely on the prose description alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.