FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability
Wenqing Wang, Haitao Xiang, Xinyi Zhao, Mingming Yin, Ying Zhong, Zhaoxin Huan, Qiheng Zhou, Jin Zhu, et al.
cs.CL
Quick verdict
First financial benchmark to jointly score answers and evidence via atomic rubrics; construction rigorous, but single-run LLM-judged results warrant caution.
Summary
FinFIRST is a benchmark of 123 expert-authored financial search tasks designed to evaluate LLM agents on both final answers and the evidence supporting them. Tasks are built from an 18-field taxonomy, a six-axis coverage blueprint, and a registry of 138 financial sources, authored by over 50 finance professionals (with CICC investment-banking support) and filtered through a six-stage quality-control pipeline at a 9.78% acceptance rate. Each task carries an evidence-grounded reference package decomposed into 701 atomic rubric items (12,300 weighted points) across raw-information acquisition, source verification, and computation/answer formation. Fifteen model configurations run under a shared ReAct harness with web search, page visiting, and Python; a GLM-5.1 judge scores rubric items, validated against eight human experts (Cohen's kappa = 0.816 on 50 instances). Claude-Opus-5 leads atomic scoring (87.59%); GPT-5.6-Sol leads strict pass (71.54%, 88/123). Computation and answer formation lags raw-information acquisition for every system, and 17.48% of correct answers lack fully traceable evidence. Limitations: single run per model at temperature 1.0 with no variance or significance testing; incomplete coverage differs across models (117-122 of 123); the judge sees only final responses; and subgroup slice analyses rest on small question counts.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- FinFIRST is the first financial benchmark to jointly evaluate final answers and supporting evidence via atomic rubrics spanning raw-information acquisition, source verification, and computation/answer formation.
- Under a unified tool harness, Claude-Opus-5 achieves the highest atomic score (87.59%) while GPT-5.6-Sol attains the highest strict pass rate (71.54%, 88/123), showing high partial credit frequently reflects incomplete solutions.
- Computation and answer formation consistently lags raw-information acquisition across all 15 evaluated systems, with gaps from 5.27 to 21.80 percentage points.
- 17.48% of correct-answer attempts (201 of 1,150) lack a complete, verifiable evidence chain, so final-answer-only evaluation overstates fully traceable success by roughly one in six correct answers.
Strengths
Genuine gap filled by joint answer+evidence atomic scoring
Table 1 and §1 show FinSearchComp's 635 tasks use binary final-answer scoring that cannot localize errors; FinFIRST decomposes each task into rubric items across raw-information acquisition, source verification, and computation/answer formation (§2.2.3), enabling the stage-level diagnosis missing from all compared benchmarks.
Construction pipeline with real adversarial QC
The six-stage process (§2.3) includes independent re-solving by a second expert and adjudication 'against the strongest available evidence rather than by majority vote'; the 9.78% acceptance rate (123 retained from ~1,260 candidates, §2.4) and the rubric-audit rule banning criteria absent from the prompt address classic rubric-benchmark failure modes.
Transparent, internally consistent metric accounting
UCR is a novel and useful diagnostic
Honest failure accounting and validated judge
Weaknesses
Single run per configuration with no statistical inference
Judge-family overlap unexamined
Trajectory-blind judging limits the traceability claim
Coverage asymmetry confounds the strict-pass headline
Small subgroups undercut slice-analysis claims
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FinFIRST (arXiv 2609.25192, Ling Team / Inclusion AI with CICC investment-banking support) is a benchmark paper targeting a real, well-argued gap: financial search agents are scored almost exclusively on final answers, so a right-looking number can hide wrong sources, wrong reporting periods, or unsupported arithmetic. The authors claim three contributions: joint answer+evidence scoring via atomic rubrics, an expert-led demand-aligned construction pipeline, and a 15-model comparison under a unified toolset. Claims are scoped to V1 (123 tasks), with a V2 foreshadowed. The headline findings — Claude-Opus-5 tops Atomic (87.59%), GPT-5.6-Sol tops Strict Pass (71.54%, 88/123), and computation/answer formation lags retrieval for every system — are stated consistently with the presented tables.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FinFIRST?
FinFIRST is a benchmark of 123 expert-written financial search tasks for LLM agents. Unlike prior financial benchmarks that score only the final answer, each task is graded against atomic rubric criteria in three dimensions — raw-information acquisition, source verification, and computation/answer formation — so evaluators can see where in the research process an agent failed, not just whether the final number matched.
Which models perform best on FinFIRST?
In the paper's single-run evaluation of 15 configurations, Claude-Opus-5 achieves the highest atomic score (87.59%) and GPT-5.6-Sol the highest strict pass rate (71.54%, 88 of 123 tasks fully satisfied). Kimi-K3 and GLM-5.3 follow at roughly 80-84% on aggregate scores, and every system is weakest on computation and answer formation relative to information retrieval.
Where can I get the FinFIRST dataset and code?
The dataset is released on Hugging Face (huggingface.co/datasets/inclusionAI/FinFIRST), and the full rubric-judge prompt is published in Appendix C of the paper. The paper does not explicitly link a repository for the evaluation harness itself, so exact replication would require rebuilding the ReAct loop over SerpAPI search, Jina Reader page visits, and Python execution.
What is the unsupported-correct rate (UCR) reported in FinFIRST?
UCR measures how often a model produces a correct final answer without a complete, traceable evidence chain. Across 1,150 correct answers in the paper, 201 lacked full support — a micro-averaged UCR of 17.48%, meaning final-answer-only scoring overstates fully verifiable success by about one in six correct answers. GPT-5.6-Sol is lowest (8.70%) and Gemini-3.7-Flash highest (34.62%).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are described in prose but not analyzable as images; per-model slice values cited here come from the text summaries of Figure 3.
- The paper announces a forthcoming V2 with expanded tasks; this review covers V1 (123 tasks) only.
- Appendix tables (Table 5-7) were extracted, but PDF extraction may have lost minor formatting in Table 3; all cited numbers were cross-checked against surrounding prose.
- The dataset is released on Hugging Face, but no harness-code repository is stated in the text — 'codeAvailable: false' reflects code, not data.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.