TabScope: Question-Adaptive Scope Selection for Table Question Answering
Yuxiang Wang, Junhao Gan, Jianzhong Qi
cs.CL · cs.AI
Quick verdict
Question-adaptive localize-vs-full scope selection for TableQA; best across WikiTQ/SLQA but key margins are ~1 pp and lack significance tests.
Summary
The paper argues that LLM accuracy loss on long-table QA is question-type dependent: lookup and local-reasoning questions benefit from table localization, while counting and comparison questions need full-table evidence. TabScope classifies each question into seven operation-defined types with an LLM and applies a validation-calibrated fixed policy to choose between localized reasoning (operation-aware decomposition: K=4 sampled row/column retrievals, confidence-weighted aggregation, compactness penalty, one refinement round) and full-table CoT. The authors also contribute WTQ-SUBTAB, silver reference sub-tables for all 4,344 WikiTQ test questions, and SLQA, a long-table benchmark (1,110 test questions; all tables exceed 4,096 serialized tokens, averaging ~9,786 tokens) built from Spider tables with GPT-4o-mini-generated, manually reviewed QA pairs. With GPT-5-mini and LLaMA-3.3-70B answer models, TabScope reaches 83.3/81.1 EM on WikiTQ and 73.4/62.8 on SLQA, beating decomposition baselines and full-table CoT, and leads all six cell/row/column metrics on WTQ-SUBTAB. Ablations show operation-aware retrieval matters most; oracle analysis leaves 3.0-5.4 pp headroom. Main limitations: no significance tests or variance estimates for gains as small as +0.3-1.3 pp on SLQA; SLQA is synthetic (LLM-generated questions); a 150 vs 200 inconsistency in the silver-calibration sample size; code and data not yet released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM accuracy degradation on long tables is question-type dependent: localization helps lookup/local-reasoning types while count and compare types benefit from full-table reasoning (Tables 4, 7, 8).
- Operation-aware decomposition with K=4 sampled, confidence-weighted retrieval and one refinement round outperforms prior decomposition methods on both answer accuracy and sub-table quality (Tables 2, 3).
- A validation-calibrated, type-based scope policy beats always-local, always-full, direct LLM routing, and a learned router (Tables 2, 12).
- SLQA, built from Spider tables exceeding 4,096 tokens, addresses the scarcity of long-table TableQA benchmarks (Table 1, App. D.1).
- Oracle scope selection leaves 5.4 pp (WikiTQ) and 3.0 pp (SLQA) headroom over the better fixed branch, so per-instance scope choice is a binding constraint (Table 11).
Strengths
The 'when to localize' claim is quantified, not asserted
Type-wise deltas (Tables 4, 7, 8) show localization helps lookup (+1.17 pp) but hurts count-general (−2.40 pp) on WikiTQ validation, and count-frequency flips sign with table size (−2.68 pp WikiTQ vs +2.22 pp SLQA); oracle scope selection (Table 11) bounds remaining headroom at 5.4 pp (WikiTQ) and 3.0 pp (SLQA).
Direct evaluation of intermediate evidence selection
WTQ-SUBTAB provides silver sub-tables for all 4,344 WikiTQ test questions; Table 3 shows TabScope leads all six metrics over DATER (cell F1 70.70 vs 68.00; row EM 70.74 vs 61.50), connecting QA gains to evidence quality rather than luck.
Controlled baseline protocol isolates the decomposition effect
SLQA fills a real long-table gap with human review
Candid selector comparison
Weaknesses
No statistical support for the headline adaptive gains
SLQA is synthetic with thin quality-control reporting
Internal inconsistency in silver-calibration sample size
Inference-cost asymmetry never quantified
Incomplete GPT-column baseline and no classifier error analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets TableQA over long tables with LLMs. Its central claim is that accuracy degradation with table size is question-type dependent, so a model should decide *when* to localize, not only *how* (Section 1, Figure 1: full-table CoT drops 13.0 pp from WikiTQ 80.3 to SLQA 67.3). Three contributions are claimed: (i) operation-aware decomposition with sampled, weighted retrieval plus refinement; (ii) a validation-calibrated type-based scope policy; (iii) two resources — WTQ-SUBTAB silver sub-tables for all 4,344 WikiTQ test questions, and SLQA, a long-table benchmark whose tables all exceed 4,096 serialized tokens (avg 9,786.2 tokens, 733.8 rows; 1,110 test questions, Table 1). Claims are properly scoped to two English benchmarks and two answer models (GPT-5-mini, LLaMA-3.3-70B), as the Limitations section acknowledges.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TabScope and what problem does it solve?
TabScope is a framework for table question answering with large language models. It classifies each question into seven operation types (e.g., lookup, counting, comparison) and then decides, via a fixed policy calibrated on validation data, whether to answer from a compact question-specific sub-table or from the full table. This addresses the finding that LLM accuracy drops on long tables, but the drop depends on question type: lookups benefit from removing irrelevant content, while counting and comparison questions need the whole table.
What is the SLQA benchmark?
SLQA is a long-table question answering benchmark introduced in this paper. Its tables come from Spider and all exceed 4,096 serialized tokens (averaging about 9,786 tokens and 734 rows per table). Question-answer pairs were generated by GPT-4o-mini from four evidence scopes (cell, row, column, sub-table) and then manually reviewed, with more than 70% accepted without changes. The test split contains 1,110 questions.
How much does TabScope improve accuracy over prior methods?
On WikiTQ it reaches 83.3% exact match with GPT-5-mini and 81.1% with LLaMA-3.3-70B; on SLQA 73.4% and 62.8%. It outperforms decomposition baselines such as DATER and Chain-of-Table and slightly beats full-table chain-of-thought, typically by around 1-2 points. On the WTQ-SUBTAB evidence test it leads all six cell/row/column F1 and exact-match metrics.
Is the code or the SLQA dataset available?
The paper states that code and datasets will be made available upon publication; at the time of this review they are not yet released. However, all prompt templates, the scope-selection policy table, and the benchmark construction procedures are printed in the paper's appendices, which would allow reimplementation in the meantime.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Mathematical notation is garbled by extraction: Eq. (4) renders as 'Support(R', C') = pRowScore(R') . ColScore(C')' with a stray leading 'p' (likely a lost radical), and Eq. (1) punctuation is mangled; formulas were interpreted from surrounding prose.
- Figures 1-3 appear as jumbled inline text in the extraction; the review relies on captions and the Section 3 prose rather than figure layout.
- The provided text ends after the Appendix E.2 Results paragraph on a complete sentence; any content that may follow that point in the published PDF is not visible in this extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.