RAG-Safety-Bench: Reliable Evaluation of Retrieval-Augmented LLM Safety
Adithiyan Rajan Indira Saravanan, Kathleen C. Fraser
cs.CL · cs.IR
Quick verdict
A controlled four-condition RAG safety benchmark across five small open LLMs; design is strong, but caveats: n=5 stats, single run, LLM-built data, judges disagree.
Summary
RAG-Safety-Bench measures how retrieval-augmented generation affects LLM safety while controlling retriever quality. The benchmark supplies document contexts directly under four conditions: non-RAG, oracle (answer-bearing document), on-topic (related documents without the answer), and random benign documents. The authors release 987 questions (346 balanced) across 20 harm subcategories, generated with Claude Sonnet-4.5 from Wikipedia and audited for answerability with GPT-4.1-mini. Across five small open-source models (Gemma-3-12B, Llama-3.1-8B, Ministral-3-8B, Phi-4-14B, Qwen-2.5-7B), harmfulness (majority vote of LlamaGuard, ShieldGemma, WildGuard) peaks in the oracle condition; benign and unsafe capability are negatively correlated (r = -0.70, five points); random safe documents yield the safest behavior, so long context alone does not cause unsafe output. Refusal analysis suggests many refusals stem from missing information rather than safety training. The strongest finding is model-specific: Qwen-2.5-7B stays above baseline harmfulness even on-topic, implying topic-related context alone can elicit harmful parametric knowledge. Limitations include small models, a single run, reliance on Claude models for benchmark creation and judging with 100-example human validation, and only moderate agreement between safety judges.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Baseline non-RAG safety does not transfer to RAG deployments: all five models give more accurate and more harmful answers to unsafe questions when an oracle document contains the answer (Table 3, Figure 2).
- Benign (MIRAGE) capability and unsafe capability in the oracle condition are negatively correlated (Pearson r = -0.70 across five models), so more capable models are not automatically less safe in RAG (Section 4.1).
- Long context alone does not cause unsafe output: random benign documents yield the lowest harmfulness for all models (0.3-3.2%, Figure 2), partly via insufficient-information refusals (Section 4.3).
- Topic-relevant, answer-free documents elicited above-baseline harmfulness in Qwen-2.5-7B (15.9% non-RAG to 17.3% on-topic), supporting parametric recall of harmful information (Table 3, Table 12).
- Automated safety judges disagree substantially (pairwise Cohen's kappa 0.47-0.48 except LlamaGuard-WildGuard at 0.66; Fleiss kappa 0.56), so evaluator choice materially affects safety conclusions (Section 4.4).
Strengths
Controlled four-condition design cleanly separates mechanisms
Non-RAG, oracle, on-topic, and random contexts are supplied directly (Table 1), eliminating retriever quality as a confounder and enabling the H1-H4 hypotheses to be tested one variable at a time (Sections 1, 3.2).
The Qwen-2.5-7B on-topic anomaly is carefully evidenced
Section 5 and Table 12 show Qwen refusing in non-RAG, answering outright in oracle, and — with the on-topic context audited as non-answerable (Table 8: 100% 'no') — still emitting procedural harmful steps; the parametric-recall interpretation is well supported.
Refusal vs. safety decoupling is an important conceptual contribution
Useful negative results with honest reporting
Release and provenance documentation are unusually complete
Weaknesses
Headline correlation rests on five data points with no inference
Oracle condition is partially confounded by the prompt template
Circularity: Claude generates the benchmark and judges the answers
Refusal classifier has moderate agreement and a documented bias
Random control is not fully standardized
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
RAG-Safety-Bench (arXiv:2609.11758) measures how retrieval-augmented generation affects LLM safety while removing retriever quality as a confounder: document contexts are supplied directly under four conditions (non-RAG; oracle, where one document contains the answer; on-topic, related but answer-free; random safe; Table 1). The paper pursues four hypotheses: H1 — benign capability predicts unsafe capability; H2 — oracle > on-topic > non-RAG harmfulness, with random isolating context-length effects; H3 — refusals in the random setting reflect missing information rather than safety; H4 — safety judges disagree. Headline claims: guardrails do not transfer to RAG, benign and unsafe capability are inversely related, and benign documents can elicit unsafe generation (scoped as model-specific in the abstract).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RAG-Safety-Bench?
A benchmark for testing whether retrieval-augmented generation (RAG) makes LLMs less safe. Instead of running a real retriever, it hands the model document sets under four controlled conditions: no documents, an oracle document containing the answer to a harmful question, topically related documents without the answer, and random safe documents. It contains 987 harmful questions (a 346-question balanced subset) across 20 harm categories, built from Wikipedia.
Does RAG make LLMs less safe, according to this paper?
Partly. Harmful responses peak when the retrieved documents literally contain the answer, for all five models tested. With safe documents the picture is mixed: for most models harmfulness returns to baseline, but Qwen-2.5-7B stayed above baseline even with topic-related documents that did not contain the answer — showing that model choice matters and guardrails do not automatically carry over into RAG settings.
Does adding random safe documents make a model unsafe?
No. Random benign documents produced the safest behavior across all five models (0.3-3.2% judged harmful). The paper also shows many refusals in that setting are because the model cannot find the information, not because of safety training.
Do different LLM safety judges agree with each other?
Only moderately. In this study LlamaGuard, ShieldGemma, and WildGuard showed pairwise Cohen's kappa of 0.47-0.48 (0.66 for LlamaGuard-WildGuard) and a four-way Fleiss kappa of 0.56. The authors recommend using multiple judges plus human oversight, since each judge is sensitive to different harm categories.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based on the publisher-provided text extraction; figures (Figures 1-10) are not rendered, so judgments relying on figure-internal values (e.g., per-category harmfulness detail) rest on the surrounding prose and tables.
- Appendix A.4 states the accuracy judge was Claude Opus-4.7 and A.5 used Claude Sonnet-4.6; model-name oddities (Opus-4.7, Ministral-3-8B, 2026-dated citations) are reproduced as printed in the source text and were not independently verifiable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.