SQL-Zero: Self-Evolving Text-to-SQL
Daniel Machado Pedrozo, Julia Soares Dollis, Bryan Lincoln Marques de Oliveira, Vinicius Alboneti Aguiar, Sávio Salvarino Teles de Oliveira, Telma Woerle de Lima Soares
cs.AI
Quick verdict
Claims annotation-free self-play lifts Text-to-SQL (+6.6/+7.3 BIRD); real vs base, unresolved vs gold control, no compounding — honest, carefully bounded.
Summary
SQL-Zero is a self-play training framework for Text-to-SQL that uses no annotated natural-language/SQL pairs: a challenger and a solver, both instantiated from the same base LLM (Qwen2.5-Coder-3B/7B-Instruct), co-evolve over BIRD training databases, with database execution as the only ground truth. The challenger proposes SQL-first question/query pairs calibrated to 'hard but solvable' difficulty against the current solver, filtered by execution determinism and deduplicated by masked SQL template; both roles are updated with GRPO in alternating turns, with a template-level repetition penalty preventing challenger diversity collapse. Under the OmniSQL evaluation pipeline, self-play improves BIRD dev over the zero-shot base by 6.6 points at 3B and 7.3 at 7B; the 3B arm beats a matched control trained on human BIRD gold over the same databases at every iteration, though McNemar's exact test leaves all three margins unresolved. Transfer is scale-dependent: at 3B every iteration exceeds the base on Spider test and Spider-Syn, whereas at 7B only the first iteration preserves transfer. Main limitations: all arms are single training runs; gains do not compound across iterations (entropy and diversity decay); CSC-SQL-3B and stronger 7B systems remain ahead; and the 3B third iteration is not budget-matched.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Execution-verified self-play trains competitive Text-to-SQL solvers with zero annotated NL–SQL pairs, improving BIRD dev by +6.6 points at 3B and +7.3 at 7B over the zero-shot bases.
- At 3B, self-play exceeds a matched control trained on human BIRD gold over the same databases at every iteration, though exact paired tests leave all three margins unresolved.
- Transfer to unseen databases and paraphrases survives every iteration at 3B but only the first iteration at 7B, where later turns trade transfer for in-domain fit.
- At 7B, naive static self-generation collapses (saturated GRPO groups), and the challenger's difficulty reward is the component that rescues training.
Strengths
Statistical candor well above field norm
§4–5 and Appendix B use McNemar exact tests with Holm correction, report binomial CIs (±2.5 BIRD dev, ±1.8 Spider test, ±2.8 Spider-Syn), set an explicit ~2-point non-conclusiveness threshold motivated by the Jin et al. 2026 annotation audit, and declare the headline self-play-vs-gold-control margins non-significant in the same paragraph that presents them.
Matched-control design isolates the effect of labels
§5 trains a control on all 8,390 human BIRD gold pairs over the same 64 training databases with identical GRPO settings and reasoning bonus, making labels the primary difference; pipeline fidelity is additionally gated by re-running released OmniSQL and Arctic checkpoints to within 0.9 points of published numbers (§4).
Leakage and checkpoint-selection controls are explicit
Mechanistic failure diagnostics accompany the negative result
Weaknesses
Single-run evidence for the central comparison
Gains do not compound and 7B pays in transfer
Still behind simpler gold-trained references
Budget mismatch confounds 3B iteration comparisons
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SQL-Zero (arXiv 2609.04697) proposes a proposer–solver self-play loop that trains Text-to-SQL solvers with zero annotated NL–SQL pairs: a challenger and a solver, both initialized from the same Qwen2.5-Coder-3B/7B-Instruct checkpoint, co-evolve over BIRD's 64 gated training databases with execution as the only ground truth. Central claims: (1) self-play adds +6.6 BIRD dev points at 3B and +7.3 at 7B over the zero-shot base; (2) at 3B it beats a matched control trained on human BIRD gold over the same databases — explicitly left statistically unresolved; (3) 3B transfer to Spider test and Spider-Syn survives every iteration while 7B transfer survives only iteration 1. Claims are carefully scoped in the abstract and §5, and the paper repeatedly flags its own non-significant margins — rare candor for this subfield.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does SQL-Zero beat supervised Text-to-SQL models?
No. SQL-Zero beats its own zero-shot base and a matched control trained on human BIRD gold (41.9 vs 43.7–45.3 BIRD dev at 3B), but gold-trained references remain ahead: CSC-SQL-3B scores 48.3 on BIRD dev, and at 7B OmniSQL (63.8) and Arctic-Text2SQL-R1 (68.5) substantially outperform SQL-Zero-7B (best 58.4). Also, its advantage over the matched control is not statistically significant (McNemar p≥0.14).
How can SQL-Zero train a Text-to-SQL model without any labeled examples?
It uses self-play: a challenger model generates SQL-first question/query pairs from database schemas alone, and each pair is validated by executing the candidate gold SQL against the database (it must be a deterministic SELECT with a non-empty result). The solver is then trained with GRPO using execution-match rewards only. Database execution replaces human annotation as the source of ground truth.
How much does SQL-Zero improve over the base model, and does it transfer?
On BIRD dev it improves over the zero-shot Qwen2.5-Coder base by 6.6 points at 3B (38.7→45.3) and 7.3 points at 7B (51.1→58.4). Transfer is scale-dependent: at 3B every iteration beats the base on Spider test (74.0–74.7 vs 68.0) and Spider-Syn (65.1–67.0 vs 56.5), but at 7B only the first iteration preserves transfer, and iterations 2–3 fall below the base on both transfer sets.
Is code or model availability provided for SQL-Zero?
Code, configurations, prompts, the database gate, and analysis scripts are released in an anonymized repository (anonymous.4open.science/r/sql-zero-artifact-478E). Model checkpoints are withheld during review to preserve anonymity, and the trainer's vendored code carries a CC BY-NC 4.0 license, so the released artifact is research-only.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.