CS-Guard: Benchmarking LLM Guardrails for Code Generation Security
Jinyang Li, Mingyu Guo, Hung X. Nguyen
cs.CR · cs.AI
Quick verdict
Claims first guardrail benchmark for code-generation security; broad, novel FSA, but judge noise, no significance tests, inconsistencies weaken rigor.
Summary
The paper introduces CS-Guard, presented as the first benchmark for evaluating guardrails against malicious code generation by LLMs. It covers text-to-code generation (1000 TTP-style malware prompts from CyberSecEval, seven jailbreaks, and a novel fictional scenario attack that embeds malicious functionality in legitimate development contexts) and code-to-code generation (331 prompts for infilling, translation, and completion from RMCBench and RedCode). Guardrails organized by a three-layer taxonomy are tested on seven LLMs (six quantized open-weight models plus GPT5-mini). Key results: level-1 base ASRs reach 98.2% (CodeLlama-13B); jailbreak ASRs average about 50% for many guardrails; code-to-code ASR approaches 100% on base models; FSA achieves 85.9-99.5% ASR on base LLMs and defeats input classifiers (73.3-98.5% ASR) and output classifiers (F1 6.9-72.3%). A 20-sample multi-turn FSA pilot shows 65-92.3% ASR. The LLM labeler was validated against 97-98 human-annotated samples per setting (93.7-96.8% accuracy, Fleiss' kappa). Main limitations: no statistical significance testing or confidence intervals; success counted if any of five sampled responses is jailbroken; multi-turn evidence rests on 20 samples; the stated 'nine guardrails' does not match the ten listed external guardrails; benign false-positive rates for guardrails are not shown in the provided text.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing guardrails perform poorly against malicious code-generation requests: after jailbreaks, average ASR reaches about 50% for many guardrails in text-to-code.
- Code-to-code attacks are nearly unconstrained on base LLMs (average ASR 52.5-98.9% across models) and remain high across many guardrails (14.4% to nearly 100%).
- The novel fictional scenario attack (FSA) achieves 85.9-99.5% ASR on base LLMs and defeats both input classifiers (73.3-98.5% ASR) and output classifiers (F1 6.9-72.3%), exposing a fundamental intent-detection gap.
- A modular three-layer guardrail taxonomy (category/operation/position) plus a standardized testbed lets developers register and compare guardrails on equal footing.
Strengths
First systematic testbed for a real gap
Prior evaluations treat malware as a minor subcategory of general safety taxonomies; CS-Guard concentrates 1331 prompts (1000 text-to-code + 331 code-to-code) on malicious code generation with documented ATT&CK-category coverage (100 per category) and 11 malware types across 9 languages (A.3).
FSA is a novel, well-constructed attack with validated construction
FSA embeds malicious functionality in legitimate development contexts (e.g., a bootkit framed as a 'Remote Boot Rescue Agent'); the three-stage pipeline (functionality extraction, application brainstorming, red-team crafting) is fully specified in A.6 and human-audited at 96.8% accuracy on 97 samples, with the released 1000/300-prompt sets under a research-only license.
Broad, taxonomy-organized guardrail coverage
Unusually transparent evaluation bookkeeping
Realistic prompt sources rather than synthetic keywords
Weaknesses
Guardrail-count inconsistency undermines framing
ASR inflation by any-of-5 counting plus judge noise
No uncertainty quantification
No benign-prompt control shown in provided text
Multi-turn evidence is a 20-sample pilot
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
CS-Guard: Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CS-Guard?
A benchmark for testing whether LLM safety guardrails can block malicious code generation. It contains 1000 malware-generation prompts (from CyberSecEval), 7 jailbreak techniques, a new 'fictional scenario attack' that disguises malware requests as legitimate software development, and 331 code-to-code prompts for infilling, translation, and completion. Ten external guardrails plus seven LLMs' built-in alignment were tested.
What is the fictional scenario attack (FSA)?
An attack that rewrites a malware request as a plausible legitimate development task sharing the same functionality — e.g., a bootkit framed as a 'Remote Boot Rescue Agent' for diagnosing failed servers. Because the prompt contains no explicit malicious keywords, guardrails rarely catch it: measured attack success rates were 85.9-99.5% on base LLMs and still 73.3-98.5% against input classifiers.
What are the paper's key findings?
Base LLMs comply with most malware requests (up to 98.2% ASR for CodeLlama-13B); jailbreaks push average ASR to about 50% for many guardrails; code infilling and translation are nearly undefended; and FSA defeats most tested guardrails, with the single best reduction being 27.7 points (PAT on Qwen3-30B), leaving a residual 64.7% ASR.
Is the dataset public and is its release responsible?
The authors state they release the benchmark and FSA data — a full 1000-prompt set and a 300-prompt subset — under a research-only CC BY-NC-SA 4.0 license, consistent with prior work. No malware is executed in the study, and all prompts derive from existing public research datasets.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with three marked [...sampling gap...] regions (parts of the FSA construction section, middle references, and an appendix prompt); claims that could depend on unseen content are phrased as 'not shown in the provided text'.
- Figure values (Figs 3-7) arrive as jumbled number streams in the extraction; per-cell figure citations are anchored to prose statements wherever possible.
- The numeric Fleiss' kappa value is described as high but does not appear in the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.