Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL
Jiamiao Liu, Dewen Qiao, Yu Zhang, Xuetao Chen
cs.LG · cs.CL
Quick verdict
Preregistered study: text-to-SQL conformal-abstention risk depends on the execution oracle; rigorous and expert-audited, but one benchmark, partly AI-labelled.
Summary
This preregistered study asks what a conformal abstention certificate for text-to-SQL actually guarantees when its calibration labels come from an execution oracle. On Spider-Realistic (508 questions), the authors run a 2x2 intervention - candidate partition and calibration labels each from the benchmark's shipped database or a distilled multi-instance suite - across four Qwen-lineage SQL checkpoints, three seeds and 200 split-halves. Swapping in the stricter suite oracle raises the certificate's held-out risk 2.73-10.23 points above the risk its own labels report at nominal 0.10 (Table 2). AI-assisted censuses attribute only 26.1-35.0% of suite rejections to semantic errors; a preregistered blinded expert audit supports the reference-defect flag (54.8% vs 20.0% of examined controls). Expert labels covering both oracle sides show the nominal-0.10 certificate carries 20.0 and 17.2 points of true judged risk; the stricter oracle errs in both directions. Consistency scores look better under the labels of the oracle that built their clusters (16/16 combinations). Limitations: one benchmark family, two Qwen lineages, census labels mostly AI-assigned, and per-question data not released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 0 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The held-out risk a conformal abstention certificate for text-to-SQL reports is set by the execution oracle that labels its calibration set: swapping the shipped database for the distilled multi-instance suite raises held-out risk 2.73-10.23 points above its own reported risk at nominal alpha 0.10 (Table 2).
- Most answers the stricter suite oracle rejects are not semantic errors (AI census: 26.1-35.0%), and a preregistered blinded expert audit supports its suspected reference-defect flag (54.8% of flagged questions defective vs 20.0% of examined controls).
- Under blinded expert labels covering both sides of the oracle, a certificate calibrated at nominal 0.10 carries 20.0 and 17.2 points of expert-judged risk on two checkpoints; the shipped database understates this on both, and the suite understates it on one and overstates it on the other.
- Every execution-consistency score looks better under the labels of the oracle that built its clusters (16/16 combinations); under expert labels the suite-partition AUROC gain holds on Kwai-32B (+6.96 points, p=0.004) but falls outside the expert interval on XiYan-32B.
Strengths
Preregistration discipline with honest negative results
Three registrations with content-hashed code, frozen entry and decision rules, realised outcome reported as O2 (D-A contrast fails the cross-lineage bar on XiYan-32B), and every post-hoc analysis explicitly labelled as such (Sections 3.5, A)
Labels on both sides of the oracle, not just the rejected side
Experts judged suite-accepted answers too, finding 17.7% and 15.1% of them wrong, so the certificate's expert-judged risk is read off rather than bracketed; both oracles sit far from experts (kappa 0.150-0.306, Section 7.1)
Direct quantification of oracle leniency at the benchmark layer
Oracle-alignment finding with a clean invariance null
Instrument checks including a negative control
Weaknesses
Census labels are single-AI-family with partial human validation
Defect-audit experts were not blind to the hypothesis
Vendor- and benchmark-narrow panel
Core data not released
Expert evidence is thin at the checkpoint level
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 0 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'certified against which oracle' mean in this paper?
It argues that a conformal abstention certificate for text-to-SQL only guarantees the risk level measured under whichever correctness oracle labelled its calibration set. Swapping the benchmark's shipped database for a stricter multi-instance execution oracle changes the reported held-out risk by 2.73-10.23 points, and expert labels give levels (about 17-20 points at nominal 0.10) that neither oracle reports.
Is the stricter multi-instance oracle more trustworthy than the shipped database?
Only 26.1-35.0% of rejected answers are judged semantic errors by an AI audit, with the rest attributed to underspecified questions, synthetic-instance artefacts or suspected defects in the benchmark's own reference queries. But the suite also accepts answers experts call wrong (15.1-17.7%), so it errs in both directions.
Can I reproduce this study?
The analysis code and preregistration documents are released, and candidate pools are regenerable from recorded prompts and seeds, but the per-question sufficient statistics, census labels and audit packages are not public; regenerating pools requires H100-class GPUs.
Does this mean conformal risk control is broken for text-to-SQL?
No. The certificate's mathematical guarantee still holds under the labels it was calibrated on. What the paper shows is that the semantic meaning of that guarantee depends on the oracle: expert-judged risk is far higher than what either oracle reports.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Middle sections (5.1-5.3, parts of Sections G, H, B, C) were omitted from the extracted text; review rests on the surviving tables, Section 4, 6, 7 and appendix tables
- Figures are referenced from captions only and could not be visually inspected
- Abstract in metadata is truncated mid-sentence; the in-text abstract was used instead
- Census label distributions by schema (Table A3, Figure A1) were only partially visible, so label-level judgements rely on the reported shares
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.