Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen
Om Nepal, Sushant Aryal, Oluseyi Olukola, Nick Rahimi
cs.SE · cs.AI · cs.CR
Quick verdict
Five controlled experiments convincingly show compile rate misleads in LLM vulnerability repair; valid for small models; diff_F1 screen unvalidated.
Summary
This empirical study argues that compile rate — the fraction of generated patches that build — is an unreliable proxy for progress in LLM-based single-function C/C++ vulnerability repair. The authors run five controlled experiments over 203 vulnerable functions from Big-Vul (seven CWE categories), three open-source base code LLMs (CodeGen-350M, DeepSeek-Coder 1.3B/6.7B), and three prompting strategies with five samples per cell. Compile rate (i) barely moves when a larger generation budget cuts truncation and raises CodeBLEU (Section 4.2); (ii) attributes only ~18% of failures to the model, with ~64% setup-caused (~44% missing project context, 15-19% a Big-Vul preprocessing artifact that strips return types); (iii) shifts 1.8-2.7x on identical patches under a single -std=gnu89 flag; (iv) ranks models opposite to whole-function CodeBLEU; and (v) rises under a compiler-feedback loop while similarity to the human fix falls, with manual inspection finding deletion- and placeholder-style non-repairs. A no-op copy of the vulnerable input scores 0.775 CodeBLEU, beating every model (0.46-0.68). The authors examine diff_F1, a change-aware screen scoring only the edited region: it gives zero credit to no-ops but up to 0.48 to some degenerate patches, and is unvalidated against human judgment. Main limitations: only small base models, one noisy dataset, no execution-based validation.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Compile rate in single-function vulnerability repair is dominated by evaluation artifacts: about 64% of failures are setup-caused (missing project context ~44%, Big-Vul return-type artifact 15-19%), a share nearly invariant across models spanning 350M to 6.7B parameters (Table 8).
- A single compiler-standard flag (-std=gnu89) shifts compile rate 1.8-2.7x on byte-identical patches with zero regressions, making reported numbers uninterpretable without the exact toolchain (Table 9).
- Whole-function CodeBLEU rewards preserved context rather than repair: a no-op copy of the vulnerable input outscores all three models (0.775 vs 0.463-0.675, Table 10).
- When compile rate becomes an explicit optimization target via a compiler-feedback loop, compile rate rises while similarity to the human fix falls, with deletion- and placeholder-style non-repairs among the newly compiling outputs (Section 4.6, Figure 4).
- diff_F1, a change-aware screen over the edited region, gives exactly zero credit to no-ops but is not a repair-quality metric, crediting some degenerate patches up to 0.48 (Sections 4.7, 6).
Strengths
Tightly controlled experimental design
Five experiments (Table 4) each isolate one failure mode; E1/E3/E5 are paired within-subject on a seeded, reproducible 70-function subset that 'reproduces the numbers of the subset exactly' (Section 3.1). E3 holds patches fixed, so the 1.8-2.7x -std flag effect (Table 9) is a deterministic property of the toolchain, not model noise.
A concrete, generalizable dataset finding
Section 3.2 documents that Big-Vul preprocessing itself strips return types in 96-100% of no-return-type functions, accounting for 15-19% of all compile failures (Table 8). This contaminates every compile-rate study built on Big-Vul and plausibly CVEfixes-style pipelines too.
Cheap construct-validity probes others can reuse
Unusual candor about its own metric
Honest handling of null instrument readings
Weaknesses
Model-scale ceiling weakens external validity
diff_F1 is exploratory and never validated
Statistical reporting is descriptive only
Feedback-loop evidence is thin
Single noisy dataset, partial coverage
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement critique, not a repair system: the paper asks whether the two cheap proxies that dominate single-function LLM vulnerability repair — compile rate and whole-function CodeBLEU — actually measure repair. On a stratified 203-function Big-Vul test set across seven CWE categories (Table 2), with three open-source base models (CodeGen-350M, DeepSeek-Coder 1.3B/6.7B) and three prompting strategies at five samples per cell (3,045 patches per model), the authors run five controlled experiments (Table 4). The claims: compile rate is insensitive to genuine improvement (E1, Section 4.2), dominated by setup artifacts (~64% of failures not model-caused, Table 8), toolchain-dependent (1.8-2.7x under one -std flag, Table 9), rank-inverted versus CodeBLEU (Table 5), and gameable under optimization (E5, Section 4.6). A secondary claim is that the similarity fallback fails too: a no-op copy of the input scores 0.775 CodeBLEU against 0.463-0.675 for all models (Table 10). diff_F1 is deliberately framed as a screen, not a metric — a framing the evidence mostly supports.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why does this paper say compile rate is a bad metric for LLM vulnerability repair?
Because when LLM-generated fixes for C/C++ vulnerabilities are evaluated as isolated single functions, most compile failures are not the model's fault. The study finds roughly 64% of compile failures come from the evaluation setup — about 44% from missing project context, 15-19% from a Big-Vul dataset preprocessing bug that strips return types — and only ~18% from malformed code written by the model. Worse, merely switching one compiler flag (-std=gnu89) changes the measured compile rate by 1.8-2.7x on the exact same patches, so the number mostly reflects the benchmark and toolchain rather than repair ability.
What is diff_F1 in this paper?
diff_F1 is a change-aware score that rates only the edited part of a patch: it compares the lines a generated patch adds and removes against the lines the human developer added and removed, using token-level F1, averaged over the sides that actually changed. A patch that changes nothing scores exactly 0, and an edit matching the developer's scores 1. The authors offer it as a cheap screen against non-edits, not as a repair-quality measure — it can still give meaningful credit (0.08-0.48 observed) to degenerate deletion patches, and it has not been validated against human judgment.
What dataset and models were used in the study?
203 vulnerable C/C++ functions drawn from the Big-Vul dataset across seven CWE categories (29 functions each), evaluated with three open-source base code models — CodeGen-350M-multi and DeepSeek-Coder base 1.3B and 6.7B — under zero-shot, few-shot, and chain-of-thought prompting, with five samples per model-prompt-function combination (3,045 patches per model). Compile checks used GCC 15.2.0 with an iterative stub-injection harness.
Would these conclusions hold for larger or instruction-tuned LLMs?
Possibly not in every detail — the authors explicitly state their results are established only for the 350M-6.7B base-model regime and cannot rule out different behavior for stronger models. However, they argue two findings are model-independent by construction: the compiler-flag sensitivity is measured on fixed patches with no model involvement, and the ~64% setup-caused failure share varies by only 0.8 percentage points across a nearly twentyfold model-size range, so a better model would still face the same harness and dataset artifacts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 3 are available only as embedded text/axis labels; per-round feedback-loop values were read approximately from axis tick labels.
- The abstract in the supplied metadata is truncated mid-sentence, but the full embedded text contains the complete claims, so no judgment relies on the truncated portion.
- The replication repository (github.com/OmNepal/llm-vulnrepair-metrics) was not inspected; code/data availability is taken from the paper's availability statement.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.