Exploring Automated Vulnerability Identification in JavaScript Code Using Large Language Models
Manit Kaushik, Ishir Bhardwaj, Pranav Gupta, Pankaj Jalote, Arun Balaji Buduru
cs.CR · cs.AI
Quick verdict
Claims LLMs beat SAST on snippet-level JS vuln detection (fine-tuned Gemini 60% vs ~0%); honest limits, but no significance tests on a 125-sample test set.
Summary
This empirical study evaluates whether large language models can identify vulnerabilities in isolated JavaScript code snippets, a setting where rule-based SAST tools reportedly fail for lack of project context. The authors assemble 1,125 real-world snippets with CWE labels from three public datasets (Kluban et al. 2022, CrossVul, Zenodo), balanced across five categories (CWE-74/78/79/89/400; 200 train/25 test each), and compare Gemini 1.5 Flash, GPT-4o Mini, and DeepSeek-R1-Distill-Llama-8B under zero-shot, chain-of-thought, and few-shot prompting plus standard and CoT fine-tuning, against Bearer and NodeJSScan. SAST baselines score near zero (Bearer 0.000 accuracy), while vanilla GPT-4o Mini reaches 34% multi-class accuracy. Standard fine-tuning yields the largest gains: Gemini 29%→60%, GPT-4o Mini 34%→49%; CoT fine-tuning lifts DeepSeek 8%→27%. CoT prompting helps GPT-4o Mini marginally (34%→35%) but hurts Gemini (29%→18%). Few-shot prompting most benefits polymorphic XSS (GPT-4o Mini 46%→76% single-class), and structured CWEs are easiest (SQL injection 82–84% zero-shot). Main limitations: the entire evaluation rests on a 125-sample test set with no significance tests, confidence intervals, or repeated runs; negative-sample construction for the binary task is undocumented; DeepSeek's standard-FT cell is an unexplained N/A; there are no learning-based baselines or code release; and the snippet setting itself favors LLMs over SAST, as the authors concede.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On isolated JavaScript snippets, LLMs vastly outperform the SAST tools Bearer and NodeJSScan (GPT-4o Mini 34% multi-class accuracy vs 0.000/0.008, Table 1), though LLM recall remains limited.
- Standard supervised fine-tuning on 1,000 CWE-labeled snippets lifts Gemini 1.5 Flash from 29% to 60% and GPT-4o Mini from 34% to 49% multi-class accuracy (Table 3).
- Prompting benefits are model-dependent: CoT prompting gives GPT-4o Mini a marginal gain (34%→35%) but degrades Gemini (29%→18%), and CoT fine-tuning helps DeepSeek (8%→27%) while underperforming standard FT for the commercial models.
- Few-shot prompting mainly helps polymorphic categories like XSS (GPT-4o Mini 46%→76% single-class, Table 4), while structured CWEs such as SQL injection are easiest zero-shot (82–84% accuracy).
Strengths
Fills a documented evaluation gap
Section 2 cites Sheng et al. (2025) on weak JavaScript coverage in LLM vulnerability-detection research; this is a focused multi-CWE JavaScript benchmark with commit-linked real-world snippets from Kluban et al. (ASE 2022), CrossVul (ESEC/FSE 2021), and Zenodo, rather than synthetic code.
Unusually broad configuration sweep with per-CWE breakdown
3 model families x {zero-shot, CoT, few-shot} x {vanilla, standard FT, CoT FT}, yielding 30 per-CWE rows in Table 4 plus multi-class Tables 1-3 — enough to surface model-dependent effects like CoT hurting Gemini (29%→18%).
Candid limitations discussion
Actionable negative result on CoT fine-tuning
Weaknesses
Entire evaluation rests on a 125-sample test set with no statistics
Binary-task negatives are undocumented
Unexplained missing ablation for DeepSeek
No learning-based baselines and no code/data release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an empirical study (Sections 1–6) of LLM-based vulnerability identification for isolated JavaScript snippets, benchmarked against SAST tools across four research questions: LLM vs SAST (RQ1), prompting effects (RQ2), fine-tuning effects (RQ3), and CWE-wise variation (RQ4). Headline claims: (1) LLMs beat Bearer and NodeJSScan by a wide margin on snippets — 34% multi-class accuracy for vanilla GPT-4o Mini vs 0.000/0.008 (Table 1); (2) standard fine-tuning roughly doubles Gemini 1.5 Flash accuracy, 29%→60% (Table 3); (3) prompting effects are model-dependent (Table 2); (4) few-shot mainly helps XSS while structured CWEs are easiest (Table 4: SQL injection 82–84% zero-shot). One scope oddity: Figure 1's caption advertises a 'patch generation' pipeline, but no patch-generation method or results appear anywhere in the text.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which LLM was best at finding JavaScript vulnerabilities in this study?
After fine-tuning on 1,000 labeled snippets, Gemini 1.5 Flash reached 60% accuracy on a 125-snippet test set spanning five CWE types (CWE-74/78/79/89/400); GPT-4o Mini reached 49%. Both far exceeded the SAST tools Bearer and NodeJSScan, which scored near zero on isolated snippets because rule-based analyzers need whole-project data-flow context.
Did the static analysis (SAST) tools really score zero?
In this snippet-level experiment, yes: Bearer scored 0.000 accuracy and NodeJSScan 0.008 on the multi-class task (Table 1). The authors caution this reflects an architectural mismatch — SAST tools trace data flow across a full project and are not built for isolated fragments — so they recommend LLMs complement, not replace, standard security analysis.
Does chain-of-thought prompting improve vulnerability detection?
It depends on the model in this study: CoT prompting nudged GPT-4o Mini from 34% to 35% accuracy but dropped Gemini 1.5 Flash from 29% to 18%. For fine-tuning, CoT-style supervision helped DeepSeek-R1-8B (8%→27%) but underperformed plain input-label fine-tuning for Gemini and GPT-4o Mini.
What dataset was used and can I get it?
The paper assembled 1,125 real-world JavaScript snippets (1,000 train / 125 test; 200/25 per category) with CWE labels drawn from three public sources: Kluban et al.'s 2022 JavaScript vulnerability set, CrossVul, and a Zenodo vulnerability dataset. The paper itself does not state a code or data release, so the assembled benchmark is not directly downloadable from the authors.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (experiment pipeline) is an image; only its caption is present in the extracted text, and the caption's mention of 'patch generation' corresponds to no results discussed in the body — its scope cannot be verified from text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.