Enhancing Financial Question Answering: A Novel Benchmark Dataset of Banks' financial statements
Arianna Miola, Bruno Spaccavento, Lorenzo Silotto, Marco Bianchetti, Luca Cagliero
cs.CL · cs.AI · cs.CE · cs.IR · q-fin.GN
Quick verdict
FinRAG-QA: 999-question cross-bank RAG benchmark on bank reports; valuable practitioner-built resource, though evaluation lacks significance testing.
Summary
The paper introduces FinRAG-QA, a benchmark of 999 practitioner-curated questions on ten standardised banking indicators (e.g., CET1 ratio, total assets, fair-value tiers), grounded in 209 annual and Pillar 3 reports from 24 European and U.S. banks (2019-2023), with documents averaging about 198k words. Unlike prior financial QA corpora centred on U.S. filings and single-institution analysis, it targets cross-institutional retrieval over regulatory documents. The authors evaluate a multi-stage RAG pipeline: Azure Document Intelligence ingestion, hierarchical chunking (~150k chunks), contextual chunk enrichment via GPT-4.1, two embedding models (OpenAI text-embedding-3-large, VoyageAI voyage-3-large), Cohere rerank-3.5, and generation with GPT-4o versus GPT-o1-high. Contextual enrichment plus VoyageAI embeddings lifts NDCG@10 from 0.322 to 0.710 over the OpenAI baseline; conditional on the ground truth being retrieved, o1-high raises answer accuracy from 44.6% to 79.0% at roughly 20x latency (35.0s vs 1.8s). Reranking helps the weaker retrieval configuration but degrades the stronger one, and a single top-ranked chunk often beats larger contexts. Limitations: relevance labels are heuristic (ground-truth string matching within chunks), all models are proprietary with results a 2024-25 snapshot, and the analysis reports no statistical significance tests, confidence intervals, per-indicator error analysis, or open-model baselines.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- FinRAG-QA is the first financial QA benchmark combining cross-institutional scope, European/Pillar 3 regulatory coverage, and documents averaging ~198k words, longer than any prior financial QA resource.
- Contextual chunk enrichment combined with a retrieval-optimised embedding model raises NDCG@10 from 0.322 (OpenAI baseline) to 0.710 (VoyageAI contextualized).
- Conditional on the ground truth being retrieved, GPT-o1-high raises answer accuracy from 44.6% (GPT-4o) to 79.0%, at roughly 20x generation latency (35.0s vs 1.8s).
- Cross-encoder reranking (Cohere rerank-3.5) degrades retrieval when first-stage ranking is already strong, improving the weaker OpenAI setup but hurting the stronger VoyageAI setup.
- A single top-ranked chunk often yields higher generation accuracy than larger contexts (o1-high: 0.852 at k=1 vs 0.822 at k=10 on VoyageAI contextualized).
Strengths
Fills a documented gap with practitioner-driven, regulation-focused coverage
999 questions on 10 standardized indicators over 89 Pillar 3 + 120 annual reports of 24 EU/US banks (Sec. 3, Table 1) — prior resources (FinDER, FinanceBench, DocFinQA) are U.S.-filings-centric and single-institution, as the related-work section argues.
Longest documents of any financial QA resource
Mean 198,515 words/document, max 556,505, 150,437 chunks (Table 3) — a genuinely harder retrieval setting than DocFinQA's ~123k-word contexts.
Component-isolated ablations yield actionable, non-obvious findings
Realistic temporal structure and honest scoping
Open dataset with rich per-row metadata
Weaknesses
Relevance labels are heuristic, not human-judged
No inferential statistics anywhere
Rigid single-template queries limit ecological validity on the query side
End-to-end proprietary pipeline with unreported protocol details
Confusing percentage-point phrasing and evaluation-set opacity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FinRAG-QA (Sec. 3) contains 999 practitioner-curated questions on 10 standardized banking indicators, spanning 209 documents (120 annual reports, 89 Pillar 3) from 24 EU/US banks over 2019-2023, with a corpus average of 198,515 words per document (Table 3) — the longest in financial QA, exceeding DocFinQA's ~123k. The paper claims: (i) the dataset fills a cross-institutional, Pillar 3 gap left by U.S.-centric benchmarks (Sec. 2, Table 1); (ii) contextual enrichment plus VoyageAI embeddings lifts NDCG@10 from 0.322 to 0.710 (Table 4); (iii) GPT-o1-high raises conditional generation accuracy from 44.6% to 79.0% (+34.4pp) at ~20x latency (Table 5, Sec. 5.2); (iv) reranking degrades strong first-stage retrieval (0.710 to 0.498); (v) k=1 context often suffices for generation (o1-high: 0.852 at k=1). The claims are supported by the tables as raw deltas, but unit phrasing ('38.8% improvement' for a 0.388 point NDCG gain, Sec. 1) conflates percentage points with relative change (which would be ~120%).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FinRAG-QA?
A public benchmark dataset of 999 practitioner-written questions about ten standardised banking indicators (e.g., CET1 ratio, total assets, fair-value asset/liability tiers), each with a verified ground-truth value, grounded in 209 annual and Pillar 3 reports from 24 European and U.S. banks covering 2019-2023. It targets cross-bank retrieval over very long regulatory documents (averaging ~198k words).
How much does RAG improve financial QA on this benchmark?
With contextual chunk enrichment and VoyageAI embeddings, NDCG@10 rises from 0.322 to 0.710 versus an OpenAI-embedding baseline. Conditional on the ground truth being retrieved, GPT-o1-high answers 79.0% of questions correctly versus 44.6% for GPT-4o, but is roughly 20 times slower (35.0s vs 1.8s per answer).
Is reranking always useful in RAG pipelines?
Not in this study. Cohere rerank-3.5 improved the weaker OpenAI retrieval setup (about +5.1% average across k) but degraded the stronger VoyageAI setup (about -19.1%, NDCG@10 falling from 0.710 to 0.498), suggesting rerankers can hurt when the first-stage ranking is already accurate.
Can I reproduce the paper's results?
The dataset is open on GitHub, but the pipeline relies on paid proprietary services (Azure Document Intelligence, OpenAI GPT-4.1/GPT-4o/o1-high, VoyageAI embeddings, Cohere reranking), and prompts and decoding parameters are not reported. Exact numbers are therefore hard to reproduce; the dataset itself can be reused with open models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are not present in the extracted text (e.g., Section 3.1 cites '(see also 2)', evidently a figure reference); all tables (1-5) and body text through the references appear complete.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.