Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agents
cs.CL · cs.AI · cs.DB
Quick verdict
Rigorous four-arm ablation on DABStep: content, not scaffolding, drives context-layer gains; strong self-audited design, but one benchmark and single runs.
Summary
The paper decomposes the gains attributed to 'context layers' for text-to-SQL agents into semantic content, retrieval scaffolding, and pre-computed views, via a four-arm ablation on Adyen's DABStep benchmark across four models (GLM-5.3-Flash, DeepSeek-V4-Flash, Claude Sonnet 5, GPT-5.6). The instrument is a YAML data contract carrying domain semantics plus tool-enforced governance; a generator 'hollows' it, emptying all prose while keeping names, types, allow-lists, forbidden operations, and nine tools byte-identical. Restoring prose (hollow→contract) lifts hard-task accuracy from 13.9%→55.1% (GLM), 22.6%→56.6% (DeepSeek), 22.9%→68.4% (Sonnet 5), 37.0%→77.4% (GPT-5.6), exceeding the scaffolding step (schema→hollow: +0 to +15.1 points) by 1.8–6.6×. The contract beats the same manual pasted into a 24,177-character prompt by 13.7–44.6 hard points; on 97 tasks requiring all six fee clauses, contract SQL expresses them 39–98% of the time versus 3–9% for the prompt arm. Compiling the contract's SQL into DuckDB views yields gold on all 176 tasks they cover, framing arm accuracy as recovery of a proven ceiling (60.8–94.9%); the derivation gap shrinks from 39.2 to 5.1 points with capability. Ungoverned arms submitted 166 mutating statements; governed arms none. Limitations: gains confined to fee questions (294/332 hard tasks), single benchmark, single run per cell with a 23.4% flip rate, no multiplicity correction, three interpretation clauses exceed the manual, and reconstructed golds run 3–6 points lenient.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Semantic content dominates: restoring prose to the hollow contract adds 26.2–35.8 hard-task points on every model, 1.8–6.6× the scaffolding step.
- The contract beats the same knowledge pasted into the prompt by 13.7–44.6 hard points because its rule is one lookup away at the moment of use (SQL clause rates 39–98% vs 3–9%).
- The contract's SQL compiles to views that answer all 176 tasks they cover with gold; agent recovery of that ceiling rises from 60.8% to 94.9% with capability, so the derivation gap (39.2→5.1 points) is transient.
- Declared-but-unenforced governance rules stopped all mutating attempts (166 by ungoverned arms, zero by governed arms) without the enforcement layer ever firing.
- The gain is confined to the contract's domain (294/332 hard tasks are fee questions), so the benchmark cannot support general claims about analytics agents.
Strengths
Genuinely factorial control, auditable by construction
The hollow arm empties all prose while keeping names, types, allow-lists, forbidden operations and nine tools byte-identical; a test suite checks 6-gram overlap with manual.md in both directions, and transcripts confirm retrieval ran (272 lookup_domain calls on GLM's hollow arm, Section 3.3)
Compiled ceiling turns relative wins into absolute recovery
The contract's sql_expression fields compile to two DuckDB views reproducing gold on 176/176 tasks, ruling out a benchmark-favoring contract error and converting arm accuracy into recovery rates (60.8–94.9%, Section 5.3)
External validation instead of self-grading alone
Mechanism shown in behavior, not only outcomes
Exceptional candor about its own flaws
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks which component of a "context layer" — curated documentation an analytics agent fetches at query time — actually produces the reported gains: semantic content, retrieval scaffolding (tools, instructions, narrowed table surface), or pre-computed views. On DABStep with four models (glm-5.3-flash, deepseek-v4-flash, Claude Sonnet 5, gpt-5.6-sol), it runs a four-arm design: bare schema; a "hollow" contract (governed tools with all prose emptied); the vendor manual pasted into the prompt (24,177 chars); and a frozen YAML data contract (60,181 chars). Claims: content dominates scaffolding (content step 26.2–35.8 hard points, 1.8–6.6× the scaffolding step); the contract beats the same knowledge in-prompt by 13.7–44.6 hard points; the contract's own SQL compiles to gold on all 176 tasks its two DuckDB views cover, converting accuracy into a recovery rate (60.8–94.9%); declared-but-unenforced rules stopped 166 mutating statements by ungoverned arms (governed: zero). The gain is confined to fee questions, 294/332 hard tasks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
Across four models on DABStep, most of a context layer's accuracy gain comes from its semantic content, not its retrieval scaffolding: restoring prose to an emptied contract lifts hard-task accuracy by 26–36 points, while scaffolding with no content adds only 0–15 points.
What is a data contract in this paper?
A YAML artifact (60,181 characters) declaring a domain's semantics — metric descriptions with executable SQL expressions — plus governance rules (allowed tables, forbidden operations) enforced through nine tools. The agent reads the prose via one-call lookup tools like lookup_domain and lookup_metric.
How much better is the contract than pasting the manual into the prompt?
By 32.2, 13.7, 44.6 and 27.1 hard-task points on GLM, DeepSeek, Sonnet 5 and GPT-5.6 respectively. On one model (GLM) the contrast was also graded officially by the DABStep leaderboard: 51.9% vs 18.8% on hard tasks.
Is the code and data available?
Yes. The harness, the frozen digest-pinned contract, its hollow variant, and all 6,416 transcripts are on GitHub (flyersworder/agentic-data-contracts, release dabstep-eval-2026-09) with SHA-256 digests; the DABStep benchmark itself is public.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The metadata abstract string ends mid-sentence ('...and at the low'), but the complete abstract is present in the extracted full text; the body runs through Appendix H and all 38 references with no truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.