LIMIT: Less Is More for Instruction Tuning in Text-to-SQL
Haoyuan Ma, Hengwei Liu, Linjuan Wu, Yongliang Shen, Yongliang Shen, Yongliang Shen, Weiming Lu, Weiming Lu
cs.AI
Quick verdict
Strong data-efficiency result: 796 curated samples give 69.1% BIRD EX on Qwen3-8B; pipeline well ablated, but schema-overlap and significance concerns remain.
Summary
The paper proposes LIMIT, a four-stage data-curation framework for Text-to-SQL instruction tuning built on the premise that a small, carefully selected training set can match large instruction corpora. LIMIT starts from the DB-Explore synthesis pool (24,457 candidates for BIRD; 29,237 for Spider), then (1) filters samples by difficulty — a weak model (Qwen3-4B, one greedy rollout) drops easy cases and a stronger model (Qwen3-Coder-30B, 8 rollouts) drops cases solved fewer than twice; (2) synthesizes chain-of-thought annotations with DeepSeek-R1, keeping the shortest execution-correct trace; (3) scores each sample with GPT-4o on adherence, input quality, SQL quality, and reasoning quality; and (4) runs a genetic algorithm that jointly maximizes schema coverage and composite quality under a size penalty. Fine-tuning Qwen3-8B with SFT (15 epochs) plus GRPO yields 66.9% execution accuracy on BIRD-Dev and 88.4% on Spider-Dev, rising to 69.1% and 88.9% with self-consistency@8 — reported as state-of-the-art among 7B-8B open-source models, surpassing DB-Explore (67.0% with self-consistency) despite using under 5% of its data. The appendix, however, shows full-data training slightly ahead (67.8% greedy, 69.6% majority vote), so LIMIT is primarily a cost-efficiency result (about 91% less GPU time). Limitations include no significance testing, no stated code/data release, and an unresolved question of whether training schemas overlap the evaluation databases.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Only 796 (BIRD) and 863 (Spider) curated samples suffice for Qwen3-8B with SFT + GRPO to reach 69.1% and 88.9% execution accuracy with self-consistency@8, the best reported among 7B-8B open-source models.
- Joint optimization of schema coverage and per-sample quality is necessary for Text-to-SQL data selection; removing the coverage constraint causes the largest ablation drop (-7.0 at SFT).
- Careful curation rather than scale drives Text-to-SQL learning: LIMIT uses under 5% of DB-Explore's data (796 vs 16,457) yet scores higher (69.1% vs 67.0% with self-consistency@8).
- The pipeline cuts GPU training time by ~91% (5.3h vs 61.5h on 4x A6000) at near-parity with full-data training.
Strengths
Genuine data-efficiency gain with transparent cost accounting
Appendix E reports 5.3h vs 61.5h training on 4x A6000 (~91% reduction) and itemizes API token costs (11.9M input/5.4M output for CoT; 4.9M/4.2M for scoring) — rare candor; Table 7 shows near-parity (66.9/69.1 vs 67.8/69.6) at 3% of the data.
Schema-coverage objective is a real methodological novelty for DB-grounded tasks
The GA fitness (Eq. 1: 0.2*Ctable + 0.3*Ccolumn + 0.5*score - 0.5*penalty) plus cardinality mutation addresses the unique coverage-vs-compactness tension; Table 5 shows removing DB cover rate is the most damaging ablation (58.4% SFT vs 65.4% full, -7.0).
Weight sensitivity analysis supports the GA configuration
Clean training-stage decomposition
Weaknesses
No statistical significance or variance reporting
Possible train/dev schema overlap is unaddressed
Missing equal-size random-selection control
Internal control undercuts the headline framing
Under-specified scoring and minor internal inconsistency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
LIMIT (Zhejiang University; arXiv 2609.24186, cs.AI, v1 Sep 2026) asks what minimal data volume suffices for Text-to-SQL instruction tuning and answers: 796 curated samples for BIRD and 863 for Spider. The headline claim: Qwen3-8B fine-tuned with SFT plus GRPO reaches 69.1% execution accuracy on BIRD-Dev and 88.9% on Spider-Dev with self-consistency@8 (66.9%/88.4% single-pass), positioned as state-of-the-art among 7B-8B open-source models and ahead of methods trained on roughly 20x more data (DB-Explore: 16,457 samples, 67.0% SC@8). Three contributions are claimed: formalizing joint schema-coverage/quality selection, a four-stage curation framework, and evidence that two orders of magnitude less data suffice.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What accuracy does LIMIT achieve on BIRD and Spider?
With Qwen3-8B, LIMIT reaches 66.9% execution accuracy on BIRD-Dev and 88.4% on Spider-Dev with single-pass inference; using 8-sample self-consistency voting, 69.1% and 88.9% respectively — which the authors report as the best among 7B-8B open-source fine-tuned models.
How many training examples does LIMIT actually use?
796 examples for BIRD and 863 for Spider, selected by a genetic algorithm from large synthesized pools (24,457 and 29,237 candidates respectively) after difficulty filtering. The selected sets cover 100% of tables and 86.8% of columns (BIRD).
Is less data really better than more data for Text-to-SQL fine-tuning?
Not exactly. The paper's own appendix (Table 7) shows training on all 24,457 candidates is slightly better: 67.8% vs 66.9% greedy and 69.6% vs 69.1% majority-vote on BIRD. LIMIT's real advantage is cost: about 91% less GPU training time (5.3 vs 61.5 hours on four A6000 GPUs) at near-equal accuracy.
What models and stages are involved in LIMIT training?
Qwen3-8B is fine-tuned with supervised learning for 15 epochs (LLaMA-Factory), then with GRPO reinforcement learning (VERL) using rewards for format, executability, result correctness, and length. Data curation additionally uses Qwen3-4B and Qwen3-Coder-30B for difficulty filtering, DeepSeek-R1 for chain-of-thought synthesis, and GPT-4o as quality judge.
Is the LIMIT code or dataset publicly available?
The paper text contains no code or data release statement. Reproduction would require the unreleased 24k-sample synthesis pool plus paid API access to DeepSeek-R1 and GPT-4o, so independent verification is currently difficult.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 4-8 (framework diagram, judge prompts, inference prompt) are images not recoverable from the extracted text; judge weights and GA population/generation parameters could not be verified.
- Table 4's coverage universe is ambiguous in the text; the schema-overlap risk is raised as a reviewer question, not confirmed as fact.
- Spider-side cost and coverage details are thinner than BIRD's in the extracted text.
- No code/data release statement found in the text; availability may exist in external channels (e.g., DB-Explore's repo).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.