Replacing Training with Memory: Listwise Selection for Text-to-SQL
Yeonseok Jeong, Soyoung Yoon, Seongjun Lee, Seung-won Hwang
cs.SE · cs.AI · cs.CL
Quick verdict
Training-free listwise SQL selection via memory+permutation; BIRD gains over R3-SQL recompute cleanly, yet ablation effects are small and parity claim weak.
Summary
MaP-SQL is a fine-tuning-free listwise selector for Text-to-SQL generate-execute-select pipelines. It replaces two roles of selector fine-tuning with inference-time strategies: (1) structured 'memories' — LLM-generated JSON specifications mapping natural-language phrases to schema elements, SQL operations, and expected outputs, distilled from training question-SQL pairs and retrieved with bge-m3 — serve as explicit selection criteria; (2) positional bias is mitigated by aggregating rankings over candidate permutations, where candidates are grouped by identical execution results, only within-group order is shuffled, and a paired t-distribution confidence score triggers an optional pointwise tie-break (Contextual-RM-32B). On BIRD-dev with fixed candidate pools from two open generators (Agentar-32B, Arctic-R1-7B) at n=8/32, it averages +2.02 execution-accuracy points over a reproduced R3-SQL with 2.92x fewer tokens (e.g., 73.08% vs 71.97% at n=32 Agentar); Spider-test and EHRSQL show +0.53/+0.68 with 2.12x/4.16x fewer tokens. Paired bootstrap/McNemar tests support the BIRD and Spider gains, but the EHRSQL gain is inconclusive (CI includes zero, p=0.125). Ablations attribute ~0.32 points to permutation and ~0.55 to memory; gains persist with a gpt-5.1-codex-mini selector (+0.33/+0.39). Limitations: absolute accuracy (70-73%) trails proprietary SOTA; baselines are reimplementations; the Table-7 parity with fine-tuned R3-SQL rests on a 0.06-point cross-pool comparison; ablation deltas lack confidence intervals.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The two major roles of listwise-selector fine-tuning — providing selection criteria and mitigating positional bias — can be replaced at inference time by retrieved structured memories and permutation-aggregated ranking, with no selector parameter updates.
- On BIRD-dev with identical candidate pools from two open generators at n=8/32, MaP-SQL averages +2.02 execution-accuracy points over a reproduced R3-SQL while using 2.92x fewer tokens and 6.54x fewer selector calls.
- Group-based permutation (fixing between-group order, shuffling only within execution-result groups) mitigates positional bias far more efficiently than global permutation, cutting the effective permutation space from O(n!) to O(g!).
- The pointwise tie-break is an optional enhancement rather than a dependency: without it MaP-SQL still outperforms R3-SQL (72.23% vs 71.51%, Table 8).
- Retrieved memories complement rather than duplicate a strong agentic code selector, adding +0.33/+0.39 points with gpt-5.1-codex-mini across two generators (Table 5).
Strengths
Verifiably consistent headline numbers
The abstract's +2.02 points / 2.92x tokens / 6.54x calls recompute exactly from Tables 2-3 row pairs (BIRD accuracy deltas 1.11/1.11/2.80/3.06; token ratios 1.97/3.84/1.71/4.15; call ratios 4.75/8.50/3.81/9.08), and Section 5.3's 9.07x/4.16x match Table 2's Arctic n=32 cells (216.59/23.85 calls; 510,881/122,985 tokens).
Unusually careful significance reporting
Table 9 pairs bootstrap 95% CIs with exact McNemar tests on question-level flips, and the authors themselves declare the EHRSQL improvement inconclusive (CI [-0.06, 1.43], p=0.125) instead of claiming it.
Large, well-instrumented efficiency gains
Broad robustness battery
Principled two-level aggregation design
Weaknesses
Weak parity claim with fine-tuned selector
Small component effects without uncertainty quantification
All key baselines are author reimplementations
Positional-bias mitigation is only partial
Memory quality never audited
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
MaP-SQL — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'fine-tuning-free' mean in MaP-SQL?
The selector LLM's parameters are never updated. However, the method still uses labeled question-SQL pairs from training splits: an LLM distills each pair into a structured JSON 'memory' describing how natural-language phrases map to schema elements, SQL operations, and expected output forms, and these memories are retrieved at test time as explicit selection criteria.
How much does MaP-SQL improve over R3-SQL?
On BIRD-dev, using identical candidate pools from two open generators (Agentar-32B, Arctic-R1-7B) at n=8 and n=32, it averages +2.02 execution-accuracy points (e.g., 73.08% vs 71.97% with Agentar at n=32) while using 2.92x fewer tokens and 6.54x fewer selector calls. The Spider-test (+0.53) and EHRSQL (+0.68) gains are smaller; the paper's own fixed-pool significance test found the EHRSQL gain not statistically significant (p=0.125).
Is code or data available for MaP-SQL?
The paper references a GitHub repository (github.com/ldilab/MaP-SQL). All three benchmarks (BIRD-dev, Spider-test, EHRSQL) are public, the selector (Qwen3-Coder-30B-A3B-Instruct) and tie-break reward model (Contextual-RM-32B) are publicly released, and experiments ran on a single RTX PRO 6000 GPU, so independent replication is plausible.
Why does MaP-SQL group candidates by execution results before permuting them?
Candidates whose SQL executes to identical results cannot be distinguished by execution alone, and majority voting favors them as a group. Fixing the between-group order and shuffling only within groups reduces the permutation space from O(n!) to O(g!) (about 2 groups on average at n=8) and, per Table 6, raises selection consistency to 29.04% versus 23.04% for shuffling all candidates globally.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction is complete from abstract through appendix prompts with no end truncation; only raster figure graphics are unavailable, but every Figure 3 value cited in this review appears in the Section 6.3 prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.