An Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime Database
Vigneshwar Ravi Rao, Rupesh Swarnakar, Fayeq Jeelani Syed†
cs.CL · cs.AI · cs.DB
Quick verdict
An honest engineering report: prompt-only LangGraph Text-to-SQL agent lifts EX to 60% on a self-built 100-question benchmark; absent baselines cap impact.
Summary
The paper presents a six-node LangGraph agent for Text-to-SQL over the Chicago Crime dataset (~8.5M records, 22 attributes), aimed at non-technical users: it filters irrelevant questions, fetches the live schema, generates PostgreSQL via Groq-hosted openai/gpt-oss-120b, validates queries with EXPLAIN dry runs (up to three retries), executes them, and narrates results in plain English. Optimization was prompt engineering only; no fine-tuning. On a self-built benchmark of 100 questions (30/40/30 Easy/Medium/Hard) with ground-truth SQL, the authors compare two prompt revisions of the same agent: V2 raises Valid SQL Rate from 87% to 93%, Execution Accuracy from 47% to 60% under a hybrid relational-equivalence metric (versus 12% to 19% under strict JSON matching), and mean Synthesis Quality from 3.91 to 4.34 out of 5. The largest gain is attributed to removing a LIMIT 10 instruction that truncated multi-row answers; residual failures stem from relevance-checker false rejections, ambiguous question semantics, and API rate limits. The authors candidly report the study's boundaries: no external baseline or public benchmark (Spider and BIRD were not run), one model with a single run per question, LLM-judged metrics without agreement statistics, and an unreleased hand-authored benchmark. Tables are internally consistent and the failure taxonomy is detailed, but the confounded V1-V2 changes, the judge-dependent headline metric, and missing baselines mean the numbers cannot be situated relative to the state of the art.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A six-node LangGraph agent combining EXPLAIN dry-run validation, bounded retries, and plain-English synthesis turns natural-language questions into executable PostgreSQL over the Chicago Crime database using prompt engineering alone.
- The revised prompt (V2) improves Valid SQL Rate from 87% to 93%, hybrid Execution Accuracy from 47% to 60% (strict JSON 12%→19%), and mean Synthesis Quality from 3.91 to 4.34/5 on a self-built 100-question benchmark.
- Removing the LIMIT 10 system-prompt instruction was the single largest improvement driver, chiefly by ending truncation of multi-row answers.
- Residual V2 failures stem from relevance-checker false rejections, ambiguous question semantics, and free-tier API rate limits rather than from SQL generation quality.
- Strict JSON-match grading systematically understates functional correctness, which the hybrid relational-equivalence metric captures more faithfully.
Strengths
Unusual transparency about scope
The abstract, §2.5 and §4.3 explicitly state that no external baseline was run, Spider/BIRD were not used, only one model with one run per question was evaluated, and judged metrics lack agreement statistics — rare candor that lets readers calibrate every number correctly.
Internally consistent quantitative reporting
Table 5 tier values aggregate exactly to the overall row (V1 EX 24+14+9 = 47%; V2 = 60%), Table 6's score distribution reproduces both SQ means precisely ((290+44+30+12+15)/100 = 3.91; (345+64+9+8+8)/100 = 4.34), and the SQ≥4 count of 85/100 in §4.1 matches Table 6 (69+16).
Actionable, ID-level failure taxonomy
Reproducibility-oriented engineering documentation
Weaknesses
Confounded V1→V2 ablation
Headline metric depends on an unspecified LLM judge
No external anchor of any kind
Null imputation alters the evaluated database
No variance or significance reporting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper presents a six-node LangGraph StateGraph agent that converts natural-language questions into PostgreSQL over the Chicago Crime dataset (~8.5M rows, 22 attributes): relevance check → live schema fetch → SQL generation (Groq-hosted openai/gpt-oss-120b) → EXPLAIN dry-run validation with ≤3 retries → execution → plain-English synthesis (Table 2). Optimization is prompt engineering only. The claims are: (1) an end-to-end architecture coupling a self-correction loop with a synthesis step; (2) prompt revision V2 beats V1 — VSR 87%→93%, hybrid EX 47%→60% (strict JSON 12%→19%), SQ 3.91→4.34/5 (Table 5); (3) removing the LIMIT 10 prompt instruction was the single largest driver; (4) residual failures are relevance rejections, ambiguity, and rate limits, not SQL generation. The authors explicitly disclaim architectural novelty and external comparability (abstract, §2.5, §4.3) — an unusual and commendable degree of candor.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What accuracy does this Text-to-SQL agent achieve on the Chicago Crime database?
On the authors' own 100-question benchmark, the improved prompt version (V2) produced valid executable SQL for 93% of questions and matched ground-truth results for 60% under a lenient relational-equivalence metric (only 19% under strict JSON matching). No public benchmark or external system comparison is reported.
Which LLM powers the agent, and was it fine-tuned?
It uses the openai/gpt-oss-120b model served through the Groq API at temperature 0. The system was optimized purely through prompt engineering; no model weights were fine-tuned.
What was the single biggest improvement between the two prompt versions?
Removing a LIMIT 10 instruction from the system prompt. In V1 it truncated multi-row answers (for example, 'list all distinct crime types' returned 10 of 31 types), and its removal drove much of the gain in execution accuracy and answer quality.
Is the benchmark or the code publicly available?
The paper states no public release. The 100-question benchmark with ground-truth SQL, the prompt texts, and the agent code are not published, which limits independent verification of the reported numbers.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 (system and agent architecture diagrams) are available only as captions; their content could not be verified from the extracted text.
- The 100 benchmark questions and their ground-truth SQL are not included, so difficulty stratification and per-item grading claims could not be independently checked.
- The exact V1/V2 prompt texts and the judge protocol are absent from the paper itself, limiting assessment of the grading procedure.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.