Pick Your Poison: Learning to Select Poison Sets for Stronger LLM Backdoor Attacks
Aashiq Muhamed, Mona T. Diab, Virginia Smith, Andrew Ilyas, Matthew Jagielski
cs.LG · cs.AI · cs.CL · cs.CR
Quick verdict
Shows poison-set choice swings LLM backdoor ASR 3–80%; SAILS's learned set scorer is convincing, but its +30pp headline uses unequal budgets vs +21pp matched.
Summary
The paper shows that the standard practice of evaluating LLM backdoor attacks with randomly sampled poison sets severely underestimates worst-case vulnerability: across three LLaMA-3-8B LoRA backdoor settings (refusal, command, compliance), poison sets of fixed size drawn from the same pool yield held-out attack success rates from 3% to 80%. The authors formalize poison selection as oracle-budgeted set optimization, argue that pointwise influence proxies suffer an irreducible additivity loss (over-optimizing them can even reduce true attack success via Goodhart effects), and propose SAILS: a DistilBERT set scorer trained on a few hundred finetune-and-evaluate oracle labels that cheaply ranks millions of candidate sets and audits only a small shortlist with real finetuning runs. At matched budget B=1500, SAILS reaches 72/92/67% ASR versus 56% average for the strongest influence baseline (TRAK+representer), a +21pp average gain (the abstract's +30pp compares against B=10 baselines). A scorer trained at mini scale transfers to full-scale finetuning (+41pp over random mean, Table 3). The pipeline extends to code generation (Qwen3-4B: 67% vs 47% random), WebShop agents (91% vs 84%), and API-only finetuning of Kimi-K2.5 (72% vs 46%); on SmolLM-360M it reaches 92% of oracle-guided RL's ASR at 1/18 the cost. Limitations: an artificial full-batch-gradient-descent oracle, tiny clean corpora (100–200 examples), thin statistical reporting (three seeds, no CIs), and dual-use risk.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across three LLaMA-3-8B LoRA backdoor settings, holding model, clean data, trigger, target behavior, and k fixed, the choice of poison set alone swings held-out ASR from 3% to 80%.
- Pointwise influence proxies carry an irreducible additivity loss: optimizing them (pool expansion, token-level text search) can raise the proxy score while true ASR falls or stays flat.
- SAILS, a DistilBERT set scorer trained on a few hundred oracle labels with propose–score–audit, outperforms the strongest influence baseline by +21pp average ASR at matched budget B=1500 (77% vs 56%, Table 4).
- A scorer trained on ~1500 mini-scale finetune-and-evaluate runs transfers to full-scale finetuning, beating random mean by +41pp and TRAK-greedy by +20pp on average (Table 3).
- The same content-only pipeline works under API-only access (Kimi-K2.5: 72% vs 46% at B=200), for code generation (Qwen3-4B: 67% vs 47% at B=1200), and for a WebShop shopping agent (91% vs 84%).
Strengths
Novel problem formalization with matched theory
Oracle-budgeted set optimization (Definition 1, §2.1) plus Theorem 1 (shortlist regret = best-set underestimation + smallest audited overestimation) and Proposition 3, which constructs pools where modular top-k selection is arbitrarily suboptimal — the theory directly motivates set-level learning.
Fair matched-budget control experiment
Table 4 gives the strongest proxy (TRAK+representer) the same B=1500 as SAILS (58/68/41 vs 72/92/67; +21pp average), pre-empting the obvious 'you just audited more sets' objection to Table 2.
Credible Goodhart evidence against pointwise proxies
Breadth across access regimes and domains
Unusually honest ablations and negative results
Weaknesses
Headline number uses unequal budgets
Artificial full-batch-gradient-descent oracle
Toy-scale clean data and pools
Thin statistical reporting
No defender-side validation of the dual-use argument
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper formalizes poison-set selection for LLM backdoor attacks as oracle-budgeted set optimization (Definition 1, §2.1) and introduces SAILS (§2.3): a DistilBERT set scorer over a canonical text serialization of each candidate set, trained on a few hundred finetune-and-evaluate 'oracle' labels, used to rank millions of candidate k-sets so that only a small ε-greedy shortlist is audited with real finetuning runs. Central claims: (1) poison-set choice alone swings held-out ASR from 3% to 80% across three LLaMA-3-8B LoRA settings (§1); (2) pointwise influence proxies suffer an irreducible additivity loss and aggressive optimization can reduce true ASR (§2.2, §3.4.1); (3) SAILS beats the strongest influence baselines by +30pp as stated in the abstract — but +21pp under the matched B=1500 budget of Table 4; (4) the scorer transfers from mini to full scale (Table 3); (5) the pipeline extends to code generation, an agentic setting, and API-only access (§3.3). Theory support: Theorem 1 bounds shortlist regret by the scorer's underestimation of the best set plus the smallest audited overestimation; Proposition 3 constructs pools where modular top-k is arbitrarily suboptimal.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SAILS and what problem does it address?
SAILS (Set-level Audit-Informed Iterated Learned Selection) is a method for choosing which training examples to poison in an LLM backdoor attack. It trains a small text model (DistilBERT) to predict how strong a whole poison set will be, using results from a few hundred real finetune-and-evaluate runs, then spends expensive finetuning only on top-ranked candidates. The stated motivation is defensive: random poison selection underestimates how vulnerable finetuning pipelines are to worst-case poisoning.
How much does the choice of poison examples affect a backdoor attack?
In the paper's three LLaMA-3-8B test settings, with the model, clean data, trigger, and number of poisoned examples all fixed, attack success ranged from 3% to 80% depending only on which poison set was chosen — so evaluations based on random sampling can badly underestimate worst-case risk.
Is the code for this paper available?
Yes. The paper links to github.com/aashiqmuhamed/poison-set-selection, and the appendix documents hyperparameters, prompts, datasets (Alpaca, StrongReject, NL2SH, WebShop), and compute costs (roughly 76 GPU-hours for one mini benchmark on H200 GPUs).
Does the optimized backdoor survive normal (non-deterministic) training?
Partially. The paper's main oracle uses full-batch gradient descent for reproducibility. Under mini-batch SGD, SAILS-selected sets kept a higher mean success rate than influence baselines, but results varied across seeds, and doubling the clean data drove full-batch success to 0% while individual SGD seeds still reached up to 71%.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Source text contains three '[...section omitted...]' gaps (parts of §3.1 and Tables 1–2, §F.5–F.6, §F.8–F.9); judgments rest on the extracted prose and visible tables.
- Figures are available only as text captions and axis labels; exact curves and per-seed spreads could not be inspected.
- Some per-baseline numbers (full Table 2 rows) appear only partially in the extraction; headline values were taken from prose passages that state them explicitly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.