IGT @ FinMMEval 2026 Task 2: Question-Type Prompting with Targeted Extraction for Multilingual Financial QA
cs.CL · cs.AI
Quick verdict
IGT routes PolyFiQA Task 2 questions to 8 handlers; ranks 3rd/12 (ROUGE-1 0.3071). Sound system report, but dev-tuned gains lack significance testing.
Summary
This CLEF 2026 working-notes paper describes the IGT system for PolyFiQA Task 2 (FinMMEval), multilingual financial QA over SEC filings and news in five languages for four companies. The core design routes 344 development questions, keyword-matched into eight types, to two handler families: regex/keyword extraction on raw filing text for structured numeric types (cash flow, R&D ratio, capex) and rule-based multilingual news passage selection for synthesis types (strategy, top-three focuses, capital allocation), all generated by Claude Sonnet 4 via AWS Bedrock. A key dataset observation is that 17-18 of 19 reference answers per synthesis type share an evidence-label prefix counted directly by ROUGE-1, which the system matches explicitly. On a fixed 152-question tuning subset, ROUGE-1 rose from ~0.247 (Haiku generic RAG) to ~0.395, with the largest gains from switching to Sonnet 4 (+0.049) and direct R&D-ratio computation (+0.054). On the official test set the system ranks 3rd of 12 (ROUGE-1 0.3071, precision 0.2821, recall 0.4044). Negative results are reported for ensemble selection and multi-hop HyDE retrieval. Main limitations: no significance testing for small per-type samples (n=19), a ~0.09 dev-to-test drop attributed to tuning overfitting, and heavy benchmark-specific calibration (label prefixes, format strings, company-specific keyword rules) that limits transfer.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Routing the 344 dev questions into eight type-specific handlers, with direct keyword/regex extraction for structured numeric types and rule-based news selection for synthesis types, outperforms generic RAG (dev ROUGE-1 ~0.395 vs ~0.247).
- Ground-truth references share evidence-label prefixes in 17-18 of 19 instances per synthesis type, and matching these prefixes yields direct ROUGE-1 gains (Table 1; FSE-label rows of Table 3).
- Dense retrieval structurally fails on pipe-delimited financial table chunks (all top-5 retrieved chunks were header rows, §4.3), motivating retrieval-free extraction; computing the R&D ratio in code was the largest per-type gain (~0.28 to ~0.582).
- Preserving source-language figures verbatim beats translation under ROUGE-1 on multilingual references (+0.017 overall, driven by Japanese and Spanish passages, §4.4 / §5.6).
- The system ranks 3rd of 12 on the official test set with ROUGE-1 0.3071 and the highest recall among top-5 teams (0.4044).
Strengths
Diagnostic-first design with a fully documented ablation trail
Table 3 logs every change with its delta on the fixed 152-question subset; the two dominant interventions (Sonnet 4 +0.049, dedicated R&D extraction +0.054) are explicitly separated from the eight smaller targeted fixes, and each is tied to an observed failure mode (§4.3, §5.1).
Honest analysis of metric/dataset leakage
Section 3 and Table 1 quantify that 17-18 of 19 references per synthesis type share an evidence-label prefix counted by ROUGE-1, including the capex footnote admitting the prompt label 'matches the majority of references' by design; §6 explicitly scopes this as benchmark-specific and non-transferable.
Instructive negative results, rarely reported in shared-task notes
Fine-grained error analysis
Strong reproducibility artifacts on paper
Weaknesses
No statistical uncertainty anywhere
Unverified overfitting explanation for the dev-test gap
Headline baseline conflates model swap with system design
ROUGE-1-only evidence despite demonstrating its failure
In-sample company-specific tuning
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a shared-task system paper (CLEF 2026 FinMMEval working notes) describing IGT, the author's entry for PolyFiQA Task 2: multilingual financial QA over one English SEC filing excerpt plus news in five languages (EN/ZH/JA/ES/EL) per question, for four companies (MSFT, HON, JNJ, UVV), scored by ROUGE-1 on <=100-word answers. Central claims: (1) the 344 development questions split into eight types across two regimes — structured numeric vs. synthesis — requiring different handlers (§3, Finding 1); (2) reference answers share evidence-label prefixes ("News Evidence:" / "Financial Statement Evidence:") in 17-18 of 19 instances per synthesis type, which ROUGE-1 rewards directly (Table 1); (3) the routed/extraction system reaches ~0.395 dev ROUGE-1 — a claimed 60% relative gain over a ~0.247 Haiku generic-RAG baseline — and ranks 3rd/12 on the official test (0.3071 / P 0.2821 / R 0.4044, Table 7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What rank did IGT achieve in FinMMEval 2026 Task 2?
The IGT system ranked 3rd of 12 ranked teams on the official PolyFiQA Task 2 test set, with ROUGE-1 0.3071, precision 0.2821, and recall 0.4044 — the highest recall among the top-5 teams (even above the 1st-place system's 0.3764).
Why does the system start answers with labels like 'Financial Statement Evidence:'?
PolyFiQA reference answers almost always begin with one of two evidence-label prefixes, and ROUGE-1 counts those tokens as overlap. The system prepends the majority label for each question type (measured as 17-18 of 19 references per synthesis type) to gain unigram overlap before content is compared.
Does the IGT system use retrieval-augmented generation (RAG)?
No. For structured numeric questions it replaces dense retrieval with direct keyword/regex extraction on raw filing text, after observing that the vector retriever returned pipe-delimited table header rows instead of data rows. Synthesis questions use rule-based selection of multilingual news passages rather than embedding retrieval.
Is the code and these results reproducible?
The paper links a public GitHub repository (chiuyuwen91/FinMMEval-2026) with extraction routines, prompt templates, and evaluation scripts, and pins the model (claude-sonnet-4-20250514, temperature 0). However, re-running the official numbers requires the organizers' task data and paid AWS Bedrock access, plus redoing company-specific tuning for any new filer.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's intermediate budget points are not recoverable from the PDF text extraction (only the labeled peaks 0.617 and 0.357 appear); the non-monotonicity claim is taken from the surrounding prose, not the curve data.
- The GitHub repository (chiuyuwen91/FinMMEval-2026) is cited but its contents were not verifiable from the text; code availability reflects the paper's claim only. No truncation detected — the provided text ends with the complete reference list and appendices.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.