A Cost-Aware Agentic Architecture for NL-to-SQL over Nested Enterprise Schemas, with a New Benchmark
Yoga Sri Varshan Varadharajan, Ajay Yadav, Ritesh Goru, Prateek Chaudhury, Constantine Caramanis, Prateek Jain, Divyateja Pasupuleti, Sunil Kumar Pandey
cs.AI
Quick verdict
Claims a 900-query nested-schema NL2SQL benchmark and single-generation agent hitting 91.7% AC; plausible but self-built benchmark, LLM-judged, no released DB.
Summary
The paper makes two contributions for text-to-SQL over enterprise schemas with nested ARRAY[STRUCT] columns and typed link graphs, a regime the authors argue Spider 2.0-Snow does not measure. First, the DevRev NL2SQL benchmark: 900 LLM-generated, execution-verified queries over a proprietary production Snowflake schema (15 entity types, 2,036 array-of-struct columns), annotated with a seven-dimension Semantic Depth Score (SDS). Second, a single-generation agentic system combining LLM-driven iterative schema discovery over a curated knowledge graph, metadata-rich per-table retrieval, a 15+-class error taxonomy, a pre-execution LLM plan verifier, ten deterministic checkers, a dynamic cheatsheet, and LLM semantic validation. Using gpt-5.2, it reports 91.7% answer correctness (825/900) versus 37.1% for the best of three baselines (FlexSQL), with ablations attributing −15.0pp to metadata retrieval and −13.2pp to the error taxonomy. On the public 120-query Spider 2.0-Snow gold subset it reports 85.8% execution accuracy, below leaderboard leaders (94–97%). Main limitations: the benchmark and evaluation rest on a private production database whose release is only promised; the primary metric is an LLM judge with no reported human validation; baselines were built for flat schemas and lack the cheatsheet test-time learning; no confidence intervals or significance tests are reported anywhere.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing benchmarks, including Spider 2.0-Snow, do not measure nested ARRAY[STRUCT] traversal or polymorphic typed-link reasoning (Section 3.1, Table 1).
- The DevRev NL2SQL benchmark's 900 execution-verified queries plus the seven-dimension Semantic Depth Score fill this measurement gap (Section 3).
- The proposed single-generation agentic architecture attains 91.7% Answer Correctness on DevRev-900, 54.6pp above the best of three baselines (Table 2).
- Every architectural component contributes measurably, with metadata-enriched retrieval worth 15.0pp overall and 64.4pp on high-D7 queries (Figure 3).
- SDS is actionable: injecting dimension-targeted hints lifts ReFoRCE from 29.1% to 63.2% and APEX-SQL from 27.2% to 41.4% without architecture changes (Section 5.4).
Strengths
Identifies and operationalizes a genuine measurement gap
Section 3.1 and Table 1 document concretely that Spider 2.0-Snow has 0 declared sub-field schemas and 6,180 opaque VARIANT columns, while DevRev has 2,036 typed ARRAY[STRUCT] columns and 4-level nesting — a structural contrast that is verifiable and motivating.
Unusually complete pipeline and rubric documentation
Appendix C specifies the full generation pipeline (sub-schemas over 14 table combinations, a 25-formula book from top-SDS Spider 2.0 queries, six-slot planning with enforced category coverage, six-persona rewriting with preservation invariants) and reproduces the full SDS rubric (C.7, Table 5) and judge rubric (C.8, Table 6).
Internally consistent headline arithmetic
Ablations show differential, mechanism-linked effects
Honest subset framing and candid limitations
Weaknesses
Circular evaluation: self-built benchmark, LLM-judged primary metric
Comparison parity gaps favor the proposed system
No statistical uncertainty anywhere
Results effectively unverifiable by third parties
Survivorship bias in benchmark construction
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses NL-to-SQL over "nested enterprise schemas": production schemas with typed ARRAY[STRUCT] columns, polymorphic typed-link arrays, and up to four levels of nesting, instantiated in DevRev's production Snowflake schema (30 core entity types, 13,898 fields, 2,036 ARRAY[STRUCT] columns; the benchmark uses 15 entity types). Two claims are made. Claim 1: existing benchmarks do not measure this regime — well supported by Table 1 (Spider 2.0-Snow: 7,860 flat-DDL tables, 6,180 opaque VARIANT columns, zero declared sub-fields vs DevRev's 2,036). Claim 2: a cost-aware single-generation agentic system reaches 91.7% Answer Correctness (825/900), 54.6pp above FlexSQL's 37.1%, and is "competitive" at 85.8% EX (103/120) on Spider 2.0-Snow's gold subset. Claim 1 is the durable contribution; Claim 2 is weakened by evaluation circularity, comparison-parity gaps, and absent uncertainty quantification.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the DevRev NL2SQL benchmark?
A set of 900 natural-language questions over DevRev's production Snowflake schema, featuring nested ARRAY[STRUCT] columns and typed link arrays that benchmarks like Spider 2.0 lack. The queries were LLM-generated, verified by executing against the live database, and annotated with a seven-dimension Semantic Depth Score. The authors promise to release the queries, schema documentation, and scoring rubric on Hugging Face; the underlying production database itself is not part of the promised release.
How accurate is the proposed NL2SQL system?
On its own DevRev benchmark it reports 91.7% answer correctness (825/900 queries) versus 37.1% for the best of three external baselines, all using GPT-5.2. On the public 120-question Spider 2.0-Snow gold subset it reports 85.8% execution accuracy — solid, but below the leaderboard leaders at roughly 94–97% on the full 547-query set, and evaluated on a different (smaller) query set.
Why do strong baselines score so low (27–37%) on DevRev?
The baselines (APEX-SQL, FlexSQL, ReFoRCE) were designed for flat relational schemas and struggle with LATERAL FLATTEN over array-of-struct columns and polymorphic link traversal. The same systems score 63–73% on Spider 2.0-Snow, so the gap reflects regime mismatch rather than weak systems — compounded by the fact that the proposed system additionally gets metadata-enriched schema context and a cross-query 'cheatsheet' learning mechanism the baselines lack.
Can I reproduce the paper's results?
Not fully at present. The benchmark runs against a private production database that is not being released, the primary metric ends in an LLM judge with no reported human validation, and no code release is stated. The promised release covers queries, schema documentation, and evaluation rubrics only. The Spider 2.0-Snow portion uses public data (120 gold-answer queries) but would require reimplementing the full agentic system and paying for frontier-model API calls.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2 and 3 are embedded as raw plot text; the per-bucket and ablation values cited in this review come from the body text and appendix tables, which contain the same numbers.
- Several appendix tables (e.g., Tables 10–13) appear as inline text with scrambled layout; all numeric content was recoverable, and the extraction ends at the natural end of Section D.6 with no evidence of missing text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.