STQA: A Benchmark for Stock-Focused Tabular Question Answering over Historical and Forecasted Data
Baoxu An, Wenmian Yang, Zhensheng Wang, Weijia Jia
cs.CL
Quick verdict
Claims the first unified stock QA benchmark over history+forecasts; valuable resource with clear ablations, but lacks significance tests and baselines.
Summary
The paper introduces STQA, a benchmark for natural-language question answering over stock data that unifies three task types: historical query reasoning (Text-to-SQL style), numerical forecasting, and reasoning over forecasts. Built on FNSPID tables for 4,417 stocks (2021-2023, 700 trading days each), it contains 31,400 QA pairs generated from 72 expert templates, paraphrased with GPT-oss-120B and manually reviewed, plus intent and slot annotations. The accompanying SQFRS baseline orchestrates a fine-tuned BERT SLU module, an LLM routing agent, SQL generation, PostgreSQL retrieval, and a Time-MoE forecaster with a 15-trading-day horizon. Across four open-weight LLM backbones, SQFRS beats a workflow-disabled Vanilla agent (average +8.84 pp QR accuracy, +12.26 pp FR accuracy); GEMMA 4-31B reaches 72.38% QR and 62.43% FR accuracy, while GPT-oss-120B attains the best forecasting MRE (0.0590). Incremental ground-truth injection identifies the forecaster as the dominant bottleneck: GT forecast data adds +19.85 pp FR for GEMMA (from the GT-SLU+WF+SQL setting), yet accuracy caps at 86.14%. On 100 human-written questions, performance drops sharply (QR -17.06 pp, FR -10.51 pp on average). Main limitations: no statistical significance testing, tiny human evaluations, internal-only baseline comparisons, and underspecified derivation of gold answers for forecast tasks.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- STQA is the first benchmark to unify natural-language historical querying, numerical forecasting, and forecast-based reasoning over stock tables, with 31,400 QA pairs over 4,417 stocks and intent/slot annotations.
- SQFRS's workflow-routed tool orchestration consistently outperforms autonomous agent invocation, with average gains of about +8.84 pp QR accuracy and +12.26 pp FR accuracy across four open-weight LLM backbones.
- The time-series forecaster is the dominant bottleneck of forecast-based reasoning: injecting ground-truth forecast data raises GEMMA 4-31B's FR accuracy by 19.85 pp (from the GT-SLU+WF+SQL setting) to only 86.14%, so even full supervision cannot close the gap.
- Current LLMs degrade substantially on human-written versus template-generated questions (QR -17.06 pp, FR -10.51 pp on average), and SQFRS with GEMMA 4-31B approaches stock-investor human FR accuracy on 120 questions (68.33% vs 69.17%).
Strengths
Genuinely new task composition with intent/slot supervision
Section 3 and Table 1 position STQA as the first benchmark combining historical SQL-style querying, numerical forecasting and forecast-based reasoning over 4,417 stock tables (31,400 QA pairs, 72 templates), where prior tabular QA datasets (WikiSQL, FinQA, FinDER) cover retrieval only.
Honest, incremental ground-truth bottleneck analysis
Section 5.3 and Appendix D (Tables 13-16) inject GT SLU/WF/SQL/forecast-data step by step: the decomposition for GEMMA 4-31B (FR 62.43 -> 63.57 -> 63.74 -> 66.29 -> 86.14%, i.e. +1.14/+0.17/+2.55/+19.85 pp) cleanly isolates the forecaster as the dominant error source and candidly shows reasoning still caps at 86.14% with oracle forecast data.
Template-vs-real-world gap is measured, not ignored
Careful rewriting quality control and controlled forecaster
Weaknesses
No statistical significance testing anywhere
Gold answers for forecasting tasks are underspecified
No external agent baselines
Tiny human studies with imprecise claims
Minor internal inconsistencies in dataset statistics
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
STQA (arXiv 2609.06117) is a benchmark-plus-baseline paper in financial NLP. The authors claim three contributions (Section 1): (1) the first benchmark unifying natural-language historical query reasoning, numerical forecasting, and forecast-based reasoning over stock tables, with intent/slot supervision; (2) SQFRS, an agent framework positioned explicitly as a transparent baseline rather than a novel algorithm; (3) experiments quantifying task difficulty for tool-augmented financial agents. The novelty claim ("to our knowledge, there is no benchmark...") is plausible but is argued against a thin comparison: Table 1 lists six datasets (WikiSQL, FinQA, FinDER, etc.), none of which combine retrieval and forecasting, so the gap is real but the survey of concurrent agent benchmarks is sparse.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the STQA benchmark?
STQA is a dataset of 31,400 natural-language question-answer pairs over daily stock tables (open, close, high, low, volume) of 4,417 US stocks from 2021-2023. It unifies three task types: historical query answering (SQL-style), numerical price forecasting for the next 15 trading days, and yes/no or extreme-value reasoning over forecasts. Each question carries intent and slot annotations, and the data/code are at github.com/xuxubaobaoan/STQA_Project.
What is SQFRS and how does it answer questions?
SQFRS is the authors' baseline agent system for STQA. A fine-tuned BERT module predicts the question's intent and slots; an LLM routing agent picks one of three workflows; LLM submodules generate SQL to fetch history from PostgreSQL; a Time-MoE model forecasts the next 15 trading days; and an LLM module either extracts the requested value from the forecast or performs logical reasoning over it, grounded strictly in tool outputs.
How well do current LLMs perform on STQA?
The best configuration (SQFRS with GEMMA 4-31B) reaches 72.38% accuracy on historical query reasoning and 62.43% on forecast-based reasoning; GPT-oss-120B achieves the lowest forecasting error (MRE 0.0590). On 100 questions written by humans rather than templates, accuracy drops by roughly 10-17 percentage points, and forecast-based reasoning remains the hardest task, approaching but not exceeding skilled human investors.
Is the STQA dataset and code publicly available?
Yes. The abstract states the dataset and code are available at https://github.com/xuxubaobaoan/STQA_Project, built on the public FNSPID financial dataset; the paper's appendices also publish all prompts, dataset statistics, and forecaster training details.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Chart figures (Figures 2-8) appear in the extraction only as axis labels and legends, so plotted curve values are unavailable; all quantitative judgments rely on the corresponding tables (Tables 2, 4, 5, 13-16), which are fully present.
- Prompt-template and case-study figures (Figures 8-19) are screenshots not rendered as text, so prompt wording and the six failure-case transcripts could not be verified, only their descriptions in Appendix F/G.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.