text2ql: Multi-Target Natural Language Querying via a Language-Agnostic Intermediate Representation
cs.CL · cs.AI · cs.DB
Quick verdict
Proposes language-agnostic IR for SQL+GraphQL NL querying with zero-LLM mode; engineering is solid, but 100% exec-accuracy claims are unverified.
Summary
text2ql is an open-source Python framework that generates both SQL and GraphQL queries from natural language through a language-agnostic intermediate representation (QueryIR) and pluggable renderers. A deterministic, zero-LLM mode achieves a 3.2 ms median latency at zero API cost, while an LLM mode backed by gpt-4o-mini reports 62% exact match on a 50-query Spider sample and 70% on BIRD, with 84-91% execution accuracy. Every output carries a confidence score clamped to [0.15, 0.97], intended to enable cascade routing between modes. The authors present schema-aware prompting as the dominant accuracy lever (+18.4 points of EM over the no-schema ablation) and ship Spider/BIRD loaders and three evaluation modes as tooling. Main limitations are small 50-query samples per benchmark, an ambiguous measurement basis for the claimed 100% execution accuracy of the deterministic mode, and a GraphQL capability validated only structurally, without gold standards. The package is on PyPI under Apache 2.0; the paper appeared in the International Journal of Computer Applications.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 4
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single language-agnostic QueryIR plus pluggable renderers lets one seven-stage pipeline serve both SQL and GraphQL targets with new targets added via one IRRenderer subclass.
- The zero-LLM deterministic mode achieves 100% execution accuracy, sub-5 ms p50 latency (3.2 ms median measured), and zero API cost across all 100 sampled test cases.
- The LLM-backed mode with gpt-4o-mini reaches 62% Spider EM / 84% exec accuracy and 70% BIRD EM / 90% exec accuracy on 50-query samples, with BIRD's higher EM attributed to its evidence hints.
- Schema-aware prompting is the dominant accuracy lever, contributing +18.4 percentage points of EM over the no-schema baseline on both benchmarks (Table 5 ablation).
- A runtime confidence score in [0.15, 0.97] with a 0.75 cascade threshold routes straightforward queries to the deterministic mode, removing LLM calls entirely for them.
Strengths
Honest scoping of evaluation limits
Section 6.1 pre-registers ±7 pp sampling margins, Section 8.1 states N=50 vs full sets of 1,034/1,534 queries, Section 8.6 names concrete failure modes (aggregation misattribution ~8%, implicit join omission ~12%, filter value hallucination ~5% of errors) — candor unusual for a system paper.
Ablation isolates the claimed accuracy lever
Table 5's three-level design (no schema 43.6/51.6% EM → entity names only 52.4/60.2% → full NormalizedSchemaConfig 62.0/70.0%) cleanly supports the +18.4 pp schema-aware prompting claim on both benchmarks.
Genuinely differentiated operating point
Reusable evaluation infrastructure
Explicit metric critique
Weaknesses
Tiny, possibly non-comparable evaluation samples
Execution-accuracy metric is not defined and looks conflated with error-freeness
Related-work overclaims
GraphQL capability is essentially unvalidated
Internal arithmetic inconsistencies undermine confidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: text2ql (arXiv 2609.02115)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is text2ql and how accurate is it on Spider and BIRD?
text2ql is an open-source Python framework that converts natural-language questions into both SQL and GraphQL queries via an intermediate representation. On 50-query random samples it reports 62% exact match and 84% execution accuracy on Spider, and 70% exact match and 90% execution accuracy on BIRD using gpt-4o-mini; its rule-based deterministic mode scores 0% exact match (conservative query forms) but a claimed 100% execution accuracy with ~3 ms median latency. Note the samples are small and results are labeled indicative by the authors.
Why does text2ql's deterministic mode have 0% exact match but 100% execution accuracy?
The deterministic mode generates verbose but semantically conservative queries, which almost never match gold annotations verbatim, so exact match is 0% by design. The claimed 100% execution accuracy is harder to interpret: the paper does not describe the execution-comparison protocol, and since structural accuracy against gold is only 32–52%, the 100% figure likely reflects that outputs always execute (by fallback design) rather than that results always match gold — a key point reviewers should press on.
How does the text2ql confidence score work?
Every generated query gets a confidence score computed from additive signals (schema config present, entity resolved exactly/alias/semantically, field coverage, filters, aggregations, joins, ordering) minus validation penalties, clipped to [0.15, 0.97]. A score of 0.75 or higher is recommended for returning the deterministic result directly; lower scores trigger LLM mode. The paper presents no calibration data tying these scores to actual correctness.
Who publishes the International Journal of Computer Applications and is it a strong venue?
The International Journal of Computer Applications (IJCA, ISSN 0975-8887) is published by the Foundation of Computer Science (FCS), New York, USA. It is a rapid-publication venue not widely regarded as a selective peer-reviewed forum in the text-to-SQL community; readers should weigh this paper's claims against the small-sample evidence and the open-source code rather than venue reputation alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.