SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation
Mohammadhossein Malekpour, Mohamed Riahi, Maxime Lamothe, Amine Mhedhbi
cs.DB · cs.AI
Quick verdict
Query-mutation + EXP/EXR metrics for Text-to-SQL eval; deterministic core, careful checks, but small eval sets, no significance tests, 17%-vs-24% mismatch.
Summary
This method paper presents SQLMorph, a framework that treats Text-to-SQL evaluation as dynamic stress testing rather than static benchmark scoring. First, Join Query Expansion (JQE) adds valid joins to seed queries deterministically via schema graphs, transitive-redundancy checks, and join-query-graph isomorphism pruning; applied to BIRD dev it produces 6,873 generated and 58 pattern-novel expanded queries, raising average join-graph degree from 0.82 to 1.80 and cyclicity from 0.27% to 51.72%, while CHESS, DIN-SQL, and MAC-SQL fall to 44.8%, 32.8%, and 39.7% execution accuracy on the expanded queries. Second, Textual Query Augmentation (TQA) de-naturalizes schema and NL naming using the SNAILS classifier, causing EX drops up to 23.9 points (DIN-SQL, L/L setting), which stage-level analysis attributes to schema-linking recall collapse; notably, the abstract's 'up to 17%' claim is inconsistent with Table VII. Third, cell-level Execution Precision/Recall (EXP/EXR), validated against 15 single-error mutation operators, expose over- and under-prediction on 410 queries where all three systems fail binary EX. Main limitations: small evaluation samples (58 expansions, 30 seeds), no statistical significance testing, pre-reasoning-era systems on GPT-4o, closed-API dependence, and a single dataset.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- JQE deterministically generates join-hardened query variants, and system accuracy declines monotonically with join count, reaching ≤15% EX at higher join counts (Fig. 3).
- Controlled de-naturalization of schema and NL naming (TQA) cuts execution accuracy by up to 23.9 points on DIN-SQL (Table VII, L/L), driven by schema-linking recall collapse (Table VIII).
- EXP/EXR with cell-level matching decompose binary EX failures into over- vs under-prediction and differentiate the three systems on the 410 shared-failure BIRD dev queries (Table XI), where EX assigns all of them zero.
- Query mutation over existing NL–SQL pairs is a cheaper, more reproducible route to enterprise-like evaluation than building private benchmark sets.
Strengths
Genuinely deterministic, hallucination-resistant expansion core
JQE's SQL side is algorithmic (§IV.C–E, Algorithm 1): schema-graph candidate enumeration, power-set join-condition combinations, transitive-redundancy pruning, and JQG-isomorphism diversity pruning; only NL regeneration uses an LLM, and the pipeline yields a concrete, auditable 6,873-generated / 58-retained output (Table II).
Controlled mutation-based validation of the new metrics
§VI.C.1 constructs 15 atomic single-error operators (Table IX) with predicted effects and verifies them (Table X): e.g., limit_increase drives ∆EXP to −76 while ∆EXR stays 0, projection_drop hits recall only, confirming EXP/EXR separate over- vs under-prediction exactly as designed.
Stage-level mechanistic attribution for TQA degradation
Honest confound analysis in the join-complexity experiment
Reproducibility posture and transparent failure reporting
Weaknesses
Headline results rest on very small, non-representative samples
Internal numeric inconsistencies between abstract and tables
Dated systems undermine the 'state-of-the-art' framing
No floor calibration and no comparison against competing metrics
Single dataset, dev split only, with degraded pairs retained
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SQLMorph (arXiv 2609.08950, Polytechnique Montréal) reframes Text-to-SQL evaluation as controlled stress testing over existing NL–SQL pairs, with three contributions: (1) Join Query Expansion (JQE, §IV) deterministically adds valid joins to seed queries, claimed to 'double the average join degree' and expose monotonic degradation with join count; (2) Textual Query Augmentation (TQA, §V) de-naturalizes schema/NL identifiers via the SNAILS classifier, claimed in the abstract to cut accuracy 'by up to 17%' — though Table VII actually shows −23.9 for DIN-SQL under L/L; (3) cell-level Execution Precision/Recall metrics (EXP/EXR/F1, §VI) that decompose binary EX failures. All evaluation is on BIRD dev (1,534 pairs) with CHESS, DIN-SQL, and MAC-SQL on a GPT-4o backbone.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SQLMorph and what problem does it solve?
SQLMorph is an open-source framework for evaluating Text-to-SQL (natural-language-to-SQL) systems by automatically mutating existing test queries instead of hand-building new benchmarks. It adds valid joins to make queries structurally harder (Join Query Expansion), rewrites table and column names into abbreviated enterprise-style forms (Textual Query Augmentation), and scores outputs with cell-level Execution Precision and Recall (EXP/EXR) instead of the standard all-or-nothing Execution Accuracy.
How much does abbreviated schema naming hurt Text-to-SQL accuracy?
In the paper's BIRD dev experiments, renaming schema elements and their query mentions to abbreviated forms cut execution accuracy by roughly 7–24 percentage points depending on system and setting (Table VII: CHESS −7.0 to −8.8; MAC-SQL −9.7 to −11.7; DIN-SQL −23.4 to −23.9). Stage-level analysis attributes this mainly to a collapse in schema-linking recall. Note the abstract's 'up to 17%' figure does not match any table value, an internal inconsistency reviewers have flagged.
Is SQLMorph's code available, and can its results be reproduced?
The paper states the framework and all experimental scripts are open source at github.com/dais-polymtl/sql-morph, with fixed prompts and seeds, and the join-expansion and metric machinery is deterministic. However, the reported experiments run on OpenAI's closed GPT-4o and text-embedding-3-small APIs, so exact accuracy numbers may drift with provider-side model updates.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure content (Figs. 3–5, especially the Fig. 4 naturalness-distribution block) extracted as garbled interleaved labels/percentages; all conclusions cited in this review rest on Tables I–XI and inline numbers, which extracted cleanly.
- Math notation is lightly mangled by PDF extraction (e.g., ˆRi, ∆d̄) but unambiguous from context; no truncation was detected — the text runs complete through reference [41].
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.