ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language
Hang Zhang, Chaokun Wang, Yuzhi Pan, Ziyao Zhong, Shuo Cao, Yue Xue, Zeyu Huang, Xingwei Zhou, et al.
cs.CL · cs.AI · cs.DB
Quick verdict
First multi-scenario Direct/Interactive NL-to-PL/SQL benchmark; strong design, but LLM-built gold tasks and no significance tests cap certainty.
Summary
The paper introduces ProcArena, presented as the first benchmark evaluating LLMs on natural-language-to-PL/SQL development across multiple scenarios, both single-turn Direct and multi-turn Interactive modes, and two dialects (PostgreSQL PL/pgSQL, Oracle PL/SQL). Tasks are built from PLForge and LogicCat seeds, hardened by iterative LLM-driven logic enhancement, then mapped into nine subscenarios (five Synthesis, four Editing) via scenario-specific adapters with execution-based validation. Interactive variants are derived by knowledge injection, information omission, and contradiction injection, evaluated under a budgeted Solver-User Simulator protocol (nine actions; submit terminates with no oracle feedback). The corpus totals 3,998 tasks over 157 databases. Across seven models, Gemini-3.1 Pro performs best at 62.2% (Direct) and 57.8% (Interactive) execution accuracy; interaction lowers accuracy on Synthesis but raises it on Editing (B3 repair: 67.5% to 75.5%). A performance-optimization track jointly reports acceleration and database work, exposing candidates that execute up to 4,445x the gold procedure's operations while looking fast on wall-clock. Main limitations: gold procedures and requirement rewrites are LLM-generated with only LLM-based semantic-alignment checks and no reported human validation of task correctness; each task runs once with no significance tests or confidence intervals; a single fixed model serves as User Simulator; B2 timing noise (1.42-2.05x run-to-run) is large relative to reported speedups.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ProcArena is the first benchmark evaluating LLMs on NL-to-PL/SQL across multiple development scenarios, both Direct and Interactive modes, and two dialects.
- The best models reach only 62.2% (Direct) and 57.8% (Interactive) execution accuracy, showing realistic NL-to-PL/SQL development remains unsolved.
- Multi-turn interaction lowers accuracy on Synthesis subscenarios but raises it on Editing ones (e.g., A2 and B3), because attachments and environment tools offset underspecification.
- Interactive failures concentrate in control flow and persistent-state effects (25.5-46.6% of tasks per model), error categories with no declarative-SQL analogue.
- Every interactive construction operator individually lowers EX on the same tasks, and the full pipeline lowers it most (Table 7).
Strengths
Fills a genuine evaluation gap
Table 3: prior NL-to-PL/SQL sets hold 209-300 tasks, one dialect, one scenario; ProcArena offers 3,998 tasks, 2 dialects, 9 subscenarios, 1,879 Interactive episodes, with mean 32.1 statements / 18.6 cyclomatic complexity vs PLForge Hard's 10.7/6.8. The scenario taxonomy is grounded in 1,535 Stack Overflow/DBA posts (48.7% from-scratch, 38.4% editing).
Execution metric catches what wall-clock hides
The B2 track jointly reports acceleration and work (Section 5.3): GLM-5.2's two worst PostgreSQL candidates perform 4,445x and 1,058x the gold procedure's database operations while its measured speedup is 0.97; work harmonic means fall to 0.02 (GLM-5.2) and 0.18 (GPT-5.4 Mini) despite acceleration near or above gold.
Carefully constrained interactive protocol
Component-level validation of design choices
Diagnostic error taxonomy
Weaknesses
LLM-built gold without human validation
Single-run evaluation, no inferential statistics
B2 timing noise comparable to effects
Single-simulator dependency
Contamination exposure unaddressed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ProcArena (PVLDB submission; arXiv 2609.06527) claims to be the first benchmark evaluating LLMs on natural-language-to-PL/SQL development across (a) nine development subscenarios, (b) single-turn Direct and multi-turn Interactive modes, and (c) two dialects (PostgreSQL PL/pgSQL, Oracle PL/SQL). The corpus holds 3,998 executable tasks over 157 databases, 1,879 of them Interactive (Table 3). Headline claims: best pooled execution accuracy is 62.2% (Direct) and 57.8% (Interactive), achieved by Gemini-3.1 Pro; interaction lowers accuracy on Synthesis but raises it on Editing; failures concentrate in control flow and persistent-state effects, two error categories without a declarative-SQL analogue. Prior coverage (PLForge Simple/Hard, ProcBench, WeBridge) is single-scenario, single-dialect, 209-300 tasks, with ProcArena procedures averaging 32.1 statements and 18.6 cyclomatic complexity versus PLForge Hard's 10.7/6.8 (Table 3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ProcArena?
ProcArena is an execution-based benchmark of 3,998 natural-language-to-PL/SQL tasks over 157 PostgreSQL and Oracle databases. It covers nine development subscenarios (from-scratch synthesis, document-grounded, example-guided, template instantiation, code composition, functional modification, performance optimization, code repair, code completion) in both single-turn Direct and multi-turn Interactive modes where the model clarifies an underspecified request with a simulated user.
Which LLMs perform best on ProcArena?
In the paper's evaluation of seven models, Gemini-3.1 Pro leads with 62.2% execution accuracy in Direct mode and 57.8% in Interactive mode, followed by GPT-5.5 at 61.6% and 57.0%; DeepSeek V4 Pro is last at 36.8% and 23.7%. Even the best model fails roughly a third of Direct tasks.
Is ProcArena publicly available?
Yes. The paper states that the benchmark, interaction traces, and construction pipeline are released at github.com/ZhanGHanG9991/ProcArena, with a project website at zeyuhuangzh.github.io/procarena-site.
Why does multi-turn interaction sometimes improve accuracy?
On Editing subscenarios the task attachment already contains most of the code being changed, and the Interactive mode's database tools help with grounding-heavy tasks: for Gemini-3.1 Pro, document-grounded synthesis (A2) rises from 57.0% to 68.1% and code repair (B3) from 67.5% to 75.5%. On Synthesis tasks, which lack such anchors, the underspecified requirement lowers accuracy.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text embedded in figures (Figures 1, 5-7, 9: axis labels, code listings, and numeric callouts) is interleaved and garbled by PDF extraction; all numbers cited in this review were taken from tables, captions, and body prose, not from figure-internal callouts.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.