Automated Vulnerability Injection in Smart Contracts Using Large Language Models
Luca Migliaccio, Roberto Natella, Naghmeh Ivaki, Nuno Laranjeiro, Marco Vieira
cs.SE · cs.AI · cs.CR
Quick verdict
Claims LLM vulnerability injection in Solidity is feasible but low-yield: 32/193 survive validation; credible but small study (14 contracts, 1 evaluator).
Summary
The paper proposes a three-phase approach that uses open-source LLMs to inject known vulnerabilities into Solidity smart contracts, creating ground-truth datasets for evaluating detection tools. Phase I selects safe contracts; Phase II uses Qwen2.5-Coder-14B to score injectability of 49 OpenSCV vulnerability types per contract (10 runs per pair, 50% majority threshold) and Meta-Llama-3-8B to perform constrained in-place injections; Phase III filters candidates through compilation/execution, manual injection verification, business-logic verification, and manual exploitability confirmation. Applied to 14 contracts from SmartBugs' sb-wild subset on Kaggle P100 hardware, the pipeline produced 997 raw generations, 193 unique variants after textual deduplication (~80% duplicates), and 32 validated vulnerable contracts spanning 25 vulnerability types (16.58% survival). A manual check of 108 assessment decisions found 75.9% accuracy (95.5% precision, 73.6% recall; Wilson CI 67.3-83.2%). Survivors concentrate in structurally simpler contracts and localized Checking-class vulnerability types; business-logic preservation is the main bottleneck (89 down to 44). A demonstration on three static analyzers (Remix, Slither, Solhint) over the 19 analyzable contracts shows complementary but incomplete detection - Slither reaches precision 1.0 with recall 0.579, Solhint recall 0.706 - while 13 contracts fail entirely on legacy Solidity. Key limitations: small scale (14 contracts), a single human evaluator without inter-rater agreement, skewed defect-type coverage, and injector selection based on one vulnerability type.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM-based injection can produce verified vulnerable Solidity contracts with ground truth by construction, but yield is low: 32 of 193 unique variants (16.58%) survive the four-step validation.
- LLM injectability assessment is a conservative, high-precision filter (95.5% precision, 73.6% recall on a 108-decision manual sample) that reduces the search space but does not predict validation survival.
- Preserving business logic is the main bottleneck, causing the largest funnel drop (Step C: 89 to 44).
- The validated dataset exposes complementary and incomplete static-analyzer coverage: only 9 of 25 injected types are detected by any of Remix, Slither, or Solhint, and 13 of 32 contracts are unanalyzable due to legacy Solidity.
- LLM injection converges on a narrow set of patterns (about 80% duplicate generations), so additional runs yield diminishing diversity returns.
Strengths
Ground truth by construction with an unusually strict validation funnel
Section III-C defines per-step acceptance/rejection criteria for compilation/execution, injection verification, business-logic verification, and exploitability confirmation; Table I shows prior injectors (SolidiFI 7 types, MuSe 6, SGDL 7) skip business-logic checks, while this work targets 49 OpenSCV types and gates every candidate.
Candid statistical self-reporting
The assessment reliability check (Sec. IV-B) samples 108 of 6,860 decisions (1.57%), reports 75.9% accuracy with a 95% Wilson CI of [67.3%, 83.2%], and Section VI explicitly warns that tool metrics on n=19 are exploratory and single-contract-sensitive.
Negative results treated as findings
Baseline comparison with root-cause verification
Reproducibility provisions
Weaknesses
Very small effective scale
Single-evaluator ground truth without inter-rater agreement
Severe defect-class skew limits downstream use
Injector model selection rests on one vulnerability type
Approximate deduplication
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This arXiv preprint (2609.02624, cs.SE/cs.AI/cs.CR) proposes a three-phase approach for LLM-driven vulnerability injection into Solidity smart contracts: Phase I prepares safe contracts and setup, Phase II assesses injectability and performs injection with deduplication, Phase III validates candidates through a four-step funnel, and the resulting dataset is demonstrated on three static analyzers. The central claims are calibrated rather than grandiose: the approach is feasible, the yield is low (Abstract: feasible while exposing key limitations in scalability and diversity), business-logic preservation is the bottleneck, and the dataset exposes complementary but incomplete SAT coverage. This hedging discipline holds throughout and is the paper's most distinctive editorial feature.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How many validated vulnerable contracts did the approach produce?
32 contracts covering 25 of the 49 targeted OpenSCV vulnerability types. They come from 193 unique injected variants (997 raw generations were about 80% duplicates), for a 16.58% survival rate through the four-step validation pipeline.
Which LLMs were used for the injection pipeline?
Qwen2.5-Coder-14B (4-bit quantized) for injectability assessment and Meta-Llama-3-8B (8-bit) for injection, both run on a Kaggle P100 GPU with llama.cpp default generation settings. A Claude Sonnet 5 probe on three contracts served only as a complementary check.
How well do static analyzers detect the injected vulnerabilities?
On the 19 analyzable contracts: Slither reaches precision 1.000 with recall 0.579, Solhint precision 0.857 with recall 0.706, and Remix precision 0.786 with recall 0.688. Coverage is complementary and incomplete: only 9 of 25 injected types are detected by any tool, and 13 of 32 contracts fail on legacy Solidity versions.
Are the code and dataset available?
Yes. The paper states that all code, data, and scripts are publicly available in a replication package on Zenodo (DOI 10.5281/zenodo.19734199), including validation criteria and annotated accepted/rejected examples.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded values (Figs. 4, 5, 10, 11 heatmaps) extracted as interleaved axis labels; per-figure readings in this review rely on the body-text numbers, which are complete.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.