FinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering
Xinyu Wang, Tung Sum Thomas Kwok, Tung Sum Thomas Kwok, Zhenghan Tai, Zhenghan Tai, Guang Cheng
cs.AI
Quick verdict
Frontier LLM agents rarely elicit ambiguity resolutions in financial QA (≤28.9% vs 95% ceiling); strong benchmark and diagnosis, preliminary interventions.
Summary
The paper introduces FinInteract, a bilingual (53 EN / 120 ZH) benchmark of 173 ambiguous financial questions built from SEC EDGAR, DocFinQA and CNINFO filings (61 companies, FY2022–2025). Each instance pairs a default and an intended interpretation across a five-category taxonomy (entity scope, metric definition, temporal scope, filing vintage, recognition policy), is constructed semi-automatically by a GPT-5 pipeline with fourteen sanity rules and a ten-trial adversarial verifier, and is validated by CFA/SFC-licensed annotators (Gwet's AC1 0.85). Re-grading identical model outputs against the default rather than the intended reading inflates GPT-4o's retrieval-augmented accuracy 3.1x (37.0% vs 12.1%) — the 'single-gold illusion' — which also replicates on AmbigQA (2.1–2.3x). Under ReAct interaction, models reach at most 28.9% (full scale; 44.0% in a seven-model pilot) versus a 93–95% context-oracle ceiling; interaction helps only GPT-5 (6.4%→20.2%) and Gemini and is flat or negative for ten of twelve models, while a human baseline reaches 30%. Category-conditioned prompting (+12 points on a 50-instance pilot) and category-guided SFT (AC@1 25→69) show the benchmark is actionable; scale, category imbalance and construction circularity remain limitations.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Re-grading identical outputs against the default rather than the intended reading inflates GPT-4o's retrieval-augmented accuracy 3.1x (37.0% vs 12.1%), the 'single-gold illusion', which replicates on AmbigQA at 2.1–2.3x across three models.
- Models answer at 93–95% once the intended interpretation is supplied but at most 28.9% at full scale when they must elicit it themselves, localizing the bottleneck to elicitation and integration rather than financial knowledge.
- Added interaction improves accuracy only for GPT-5 (6.4%→20.2%, p<0.001) and Gemini (+24 points) and is flat or negative for the other ten evaluated models; forced clarification actively hurts (GPT-5 drops to 1.5%).
- Entity, metric and temporal ambiguity are reliably targeted (AC@1 0.85–1.00) while recognition policy is never raised by any model or human annotator, on exploratory evidence from nine instances.
- Conditioning on the ambiguity category improves resolution at inference time (34.0%→46.0% pilot) and training time (AC@1 25→69 via category-guided SFT), and co-evolution is gated by category salience rather than model scale or representational decodability.
Strengths
The core construct is directly measured and replicated out-of-domain
Because every instance carries both readings, the 'single-gold illusion' is quantified by re-grading identical outputs (Table 6b): GPT-4o A+S scores 37.0% against the default vs 12.1% intended (3.1x); the effect replicates on AmbigQA across three models at 2.1–2.3x (Table 8), and a five-annotator study shows humans pick the default reading as gold 66% of the time — empirically grounding rather than assuming the premise.
Leak-safe, verifiable construction with expert validation
Fourteen code-level sanity rules (Table 15, R1–R14: answer absent from C, evidence spans ≤85% overlap, answers differ beyond grader tolerance) plus a ten-trial adversarial verifier (reject if ≥2/10 blind-solve the intended reading; measured blind-solve rate 0.7%, Table 4). Two CFA/SFC-licensed annotators validate on an eight-question protocol (Gwet's AC1 0.85, correctly chosen over Cohen's κ under label skew); cost is instrumented at $0.31/instance (Appendix G).
Unusually honest grading audits and robustness controls
Diagnostic depth beyond aggregate scores
Actionability shown at both inference and training time
Weaknesses
Three of five taxonomy categories are underpowered or empty
Construction circularity skews cross-vendor comparisons
The intervention evidence is statistically thin
Ambiguity-by-construction rewards always-asking
Incomplete metric reporting for the proprietary panel
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FinInteract (arXiv 2609.24002) is a benchmark paper targeting an evaluation blind spot the authors name the **single-gold illusion**: one gold answer per ambiguous financial question cannot distinguish an agent that resolved the ambiguity from one that guessed the common reading. The artifact is 173 bilingual instances (53 EN / 120 ZH) over 61 companies from SEC EDGAR, DocFinQA, and CNINFO filings (FY2022–2025), each pairing a default and an intended interpretation under a five-category taxonomy (entity scope n=94, metric definition n=63, recognition policy n=9, temporal scope n=7, filing vintage n=0; Table 4). Claims are scoped to three RQs — recognition (RQ1), category-level failure localization (RQ2), interventions (RQ3). The intended interpretation is a constructor-assigned hidden context; Appendix F correctly frames the task as recovering a specified interpretation, a proxy for — not identity with — real analyst intent disambiguation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'single-gold illusion' in this paper?
It is the blind spot of benchmarks that attach one gold answer to an ambiguous question: an agent that guesses the common (default) reading receives full credit even though it never resolved what the user actually meant. Because FinInteract stores both a default and an intended answer per question, the authors re-graded identical model outputs both ways: GPT-4o's retrieval-augmented accuracy is 37.0% against the default but only 12.1% against the intended reading — single-gold grading overstated its competence 3.1 times. The effect also replicates on the general-domain AmbigQA dataset (2.1–2.3x across three models).
How large is FinInteract and what does it contain?
173 instances: 53 English and 120 Chinese, covering 61 companies drawn from SEC EDGAR 10-K filings, DocFinQA passages, and Chinese A-share annual reports via CNINFO (FY2022–2025). Each question pairs a default and an intended interpretation under a five-category ambiguity taxonomy: entity scope (94 instances), metric definition (63), recognition policy (9), temporal scope (7), and filing vintage (0). Answers are programmatically verifiable against structured filing data.
How well do LLMs perform on FinInteract?
Poorly when they must ask for clarification: at most 28.9% accuracy under free interaction at full scale (best: Qwen3.5-35B-A3B; 44.0% for Claude-Sonnet-5 in a seven-vendor 50-instance pilot), versus 93–95% once the intended interpretation is handed to them, and a 30% human baseline. Adding interaction helps only GPT-5 (6.4%→20.2%) and Gemini; for ten of twelve evaluated models it is flat or hurts accuracy. The main failure mode is answering wrongly after asking the right question (misintegration), not failing to ask.
Is the benchmark, code and data publicly available?
The paper states that the benchmark, the full construction pipeline, and the evaluation and training code are released for research use. Reproducibility support includes construction cost accounting (~$0.31 per accepted instance, ~$54 for the current release), documented model versions, prompts, decoding settings, the fourteen sanity-check rules, and a versioned frozen v1 release with planned English-split expansion.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were recovered from text extraction only (axis values of Figures 6–11 read from embedded text); rendering detail such as exact bar positions could not be visually verified.
- Several appendix segments were omitted in extraction (parts of C.7, Appendix H details); assessment of those sections relies on cross-references in the main text.
- Model names and venues dated 2026 (GPT-5.6, Claude-Sonnet-5, InteractComp at ICML 2026, QuestBench at NeurIPS 2026) could not be externally verified in this offline review.
- Table 6(a) contains missing cells (–) for IR/AC@1/ECE of seven proprietary models; conclusions involving those diagnostics were drawn only from the populated rows.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.