Predicting Program Exit Code with LLMs and Programming Language Semantics
Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa, Junyi Jessy Li, Milos Gligoric
cs.PL · cs.AI · cs.CL · cs.SE
Quick verdict
PrEx benchmark: open-source LLMs answer from priors, not supplied semantics—sharp drops under shifts; good design, but means-only stats, no heuristic baseline.
Summary
The paper introduces Program Executability Prediction (PrEx): given a C* program, its EBNF grammar, and a complete operational semantics, an LLM must decide whether the program executes successfully or halts on a semantic error and, if it fails, name the violated rule. Starting from 491 valid C* programs taken from PLSemanticsBench, the authors generate 2,455 invalid counterparts through five ANTLR-based transformations (break/continue outside loop, divide-by-zero, modulo-by-zero, use-before-declaration), spread over Human-Written, LLM-Translated, and Fuzzer-Generated splits whose median length grows from 19 to 786 lines. Ten open-weight models (Ministral 3, Qwen2.5-Coder, DeepSeek-Qwen distills; 3B–32B) are tested under small-step and K-framework semantics plus KeywordSwap and KeywordObf shifts, three runs each. Under standard semantics the best models hit 99% on Human-Written programs, but accuracy collapses under shifts (largest drop 58 percentage points: Ministral 3 14B, K semantics, KeywordObf, 90%→32%) and on fuzzer programs (up to 55pp for Qwen2.5-Coder 14B-CoT); falsely declaring invalid programs valid is the dominant failure. The authors conclude models lean on pre-training priors rather than supplied rules. Limitations: means-only reporting without variance or significance tests, no heuristic baseline against surface-detectable injected errors, and coverage restricted to open models at or below 32B.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Given explicit operational semantics, open-source coding LLMs predict program executability from pre-training priors rather than by systematically applying the supplied rules.
- Semantic shifts cause large drops — median Human-Written declines of 19pp under KeywordSwap and 32pp under KeywordObf (§5.2) — indicating models override supplied symbol meanings with memorized ones.
- Accuracy degrades with program complexity: top-three models lose a mean 2.9–8.7pp from Human-Written to LLM-Translated and 19–33.3pp to Fuzzer-Generated (§5.1).
- False success — declaring invalid programs executable — is the dominant failure mode on LLM-Translated and Fuzzer-Generated splits (§5.5).
Strengths
Controlled prior-vs-semantics probe
KeywordSwap and KeywordObf (§3.3, Table 3) alter symbol meanings while execution stays defined by the supplied rules; the single-token obfuscation symbols explicitly fix the multi-tokenization confound the authors inherited from PLSemanticsBench.
Matched, systematic invalid corpus
2,455 invalid programs are generated from 491 valid ones via ANTLR parser-visitor transformations, each violating exactly one of five rules (§4, Table 5, Figure 4), keeping style and complexity comparable across valid and invalid classes.
Multi-axis factorial evaluation
Failure-mode depth beyond aggregates
Reproducibility posture
Weaknesses
No uncertainty quantification
Missing heuristic baseline
Prior-use is inferred, not measured
Metric conflation
Scope and label verification
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This LMPL '26 workshop paper introduces **Program Executability Prediction (PrEx)**: given a C* program, its EBNF grammar (Figure 2), and a complete operational semantics — small-step rules (S, Table 1) or K-framework rules (K, Table 2) — an LLM must answer `##success##` or `##error##`, plus the violated rule on failure. The task is positioned as a minimal probe separating semantic judgment from the trace-generation burden of the authors' prior PLSemanticsBench [39]. Three claims carry the paper: (1) open-source coding LLMs answer from pre-training priors rather than the supplied rules; (2) the KeywordSwap/KeywordObf shifts cause large drops; (3) degradation grows with program complexity across Human-Written, LLM-Translated, and Fuzzer-Generated splits.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the PrEx benchmark?
PrEx (Program Executability Prediction) asks a large language model to look at a small C-like program together with its formal grammar and complete operational semantics, and to predict whether the program runs to completion (##success##) or halts with a semantic error (##error##); on failure it must also name the formal rule violated. It tests whether models follow explicitly supplied rules or fall back on memorized coding patterns.
Do LLMs understand programming language semantics, according to this paper?
The authors find that open-source coding models (3B–32B) mostly do not follow supplied semantics: they score up to 99% on short human-written programs under standard semantics, but accuracy drops sharply (up to 58 percentage points) when operator meanings are swapped or replaced with novel symbols, and models often wrongly declare invalid programs executable. The conclusion is that these models lean on pre-training priors rather than the provided rules.
Is the PrEx dataset publicly available?
Yes. The paper states PrEx is available at github.com/EngineeringSoftware/prex. It contains 2,946 C* programs — 491 valid and 2,455 invalid — across Human-Written, LLM-Translated, and Fuzzer-Generated splits, with each invalid program violating exactly one of five semantic rules.
Which models were evaluated in this study?
Ten open-weight models: Ministral 3 (3B/8B/14B), Qwen2.5-Coder (3B/7B/14B/32B), and DeepSeek-Qwen models at 14B and 32B, tested with and without chain-of-thought prompting for non-reasoning models. No closed or frontier models were included.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 5's radar charts are extracted as garbled axis-label text; the per-error-type numeric values plotted there could not be read and are not cited in this review.
- Table cells carry minor PDF-extraction spacing noise (e.g., 'KeywordObf' variants); every number cited here was cross-checked against the surrounding prose where possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.