CodePoisonRAG: Knowledge Poisoning Attacks on Retrieval-Augmented Code Generation
Varun Gadey, Ziad Marey, Alexandra Dmitrienko
cs.CR · cs.LG
Quick verdict
Claims targeted RACG poisoning at 0.80-0.93 ASR; a convincing threat demo, but LLM-judge sensitivity and small per-CWE samples temper the conclusions.
Summary
The paper introduces CodePoisonRAG, a black-box knowledge-poisoning framework targeting retrieval-augmented code generation (RACG). Unlike prior work that retrieves already-vulnerable CVE samples, the attacker constructs one task-matched poisoned artifact per anticipated programming task: an attacker-chosen CWE is injected into a benign fixed-code snippet via a source-passthrough-sink taint chain, then false safety claims are added to comments ('Semantic Mislabeling'), with no knowledge of the victim's knowledge base, retriever, or generator. The authors build 85 poisoned artifacts covering ten CWE classes in Java (65, manually crafted) and C (20, LLM-generated), mixed into a 12,053-entry ReposVul corpus (0.7% poisoning ratio). All 85 artifacts appear in the Top-3 retrieved context, and ASR ranges 0.80-0.93 across Qwen 3.5 9B, Code Llama 13B, and DeepSeek-Coder-V2 16B; under the CodeGuarder defense ASR drops to 0.40-0.71. Ablations show taint-chain injection alone yields 77/85 while benign context yields only 25/85 target-weakness rate. Limitations: per-CWE samples are small (n=5-10) with no statistical testing; LLM-judge verdicts are sensitive to validator choice (Table IX: ASR 1.00 vs 0.80 on identical Qwen outputs); Java poisons are hand-crafted, limiting the automation claim; only one defense is evaluated; the dataset is promised only upon acceptance.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A black-box attacker can construct a single task-aligned poisoned artifact that both survives retrieval (Top-3) and induces an attacker-selected CWE in RACG outputs.
- With an aggregate 0.7% poisoning ratio (85 artifacts, ten CWEs, Java and C), ASR reaches 0.80–0.93 across three generators undefended and 0.40–0.71 under the CodeGuarder defense.
- Vulnerability injection via taint-chain modification is the dominant attack driver: 77/85 ASR without comment deception versus a 25/85 TWR baseline from benign context on Code Llama (Table VII).
- The attack withstands query paraphrasing (78/85, ASR 0.918) and affects non-target queries only when functional overlap triggers retrieval (6/50 overall, 6/8 conditional).
Strengths
Clear gap analysis and formal threat model
Section III-B formalizes the dual retrieval+generation objective (Eq. 6) and sharply differentiates targeted artifact construction from Lin et al.'s [17] opportunistic CVE-sample selection and ImportSnare's [16] dependency hijacking.
Broad, honest evaluation grid
Ten MITRE Top-25 CWEs across Java and C, three generators under defended/undefended settings, plus paraphrasing (IV-I), non-target impact (IV-K), judge comparison (IV-N), and an ablation cleanly separating taint injection (77/85) from the benign-context TWR baseline (25/85, Table VII).
Transparent validation methodology
Practically alarming attack economics
Weaknesses
No statistical inference on small samples
Headline ASR hinges on the LLM judge
Retrieval design risks circularity
Automation claim outpaces the evidence
Single defense evaluated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CodePoisonRAG (arXiv 2609.02774, cs.CR) formulates a targeted, black-box knowledge-poisoning attack on Retrieval-Augmented Code Generation (RACG). The claimed novelty over Lin et al. [17] (opportunistic selection of CVE-derived vulnerable samples) and ImportSnare [16] (dependency hijacking) is constructing a single task-matched artifact: given a benign fixed-code entry for an anticipated task, the attacker embeds a chosen CWE via a source–passthrough–sink taint chain (Vulnerability Injection) and appends false safety claims to comments (Semantic Mislabeling), with no access to the victim's knowledge base, retriever, re-ranker, generator, prompt, or defenses. Central claims: (1) one artifact per task at 0.7% corpus poisoning achieves perfect retrieval (85/85 Top-3) and ASR 0.80–0.93 across three generators (Tables II/III); (2) 0.40–0.71 ASR persists under CodeGuarder [14]; (3) taint-chain injection, not comment deception, is the dominant driver (Table VII); (4) the attack survives paraphrasing (Sec IV-I) and only mildly harms non-target queries (Sec IV-K).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CodePoisonRAG?
It is a knowledge-poisoning attack on retrieval-augmented code generation: an attacker plants a single task-matched, deliberately vulnerable code snippet (carrying a fake 'this code is safe' comment) into the corpus feeding a RAG-based code generator, so the LLM emits code with an attacker-chosen weakness (e.g., SQL injection) while still fulfilling the user's request.
How successful is the attack reported in the paper?
In the paper's setup, all 85 poisoned artifacts were retrieved within the Top-3 results, and attack success rates ranged from 0.80 to 0.93 across three open-weight LLMs (Qwen 3.5 9B, Code Llama 13B, DeepSeek-Coder-V2 16B). Under the CodeGuarder defense, success dropped but remained between 0.40 and 0.71, at a corpus poisoning ratio of only 0.7%.
What is 'Semantic Mislabeling' in this paper?
It is the second stage of the attack: adding comments or docstrings that falsely claim security measures (e.g., 'All file paths are validated and normalized') to a snippet whose vulnerable code is left unchanged, so the LLM treats the retrieved artifact as safe guidance and reproduces the flaw with less scrutiny.
Is the CodePoisonRAG dataset or code publicly available?
Not at the time of this review. The authors state they will release the 85-artifact poisoned dataset and its CWE-specific construction metadata upon paper acceptance; no code repository is announced in the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Fig. 1 (attack workflow diagram) is image-only in the extraction; its content was assessed from the caption and body text. All quantitative claims were verifiable from tables and prose, and the full abstract, references, and Appendix A were present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.