arXiv paper2026-09-05· AI safety / NLP (LLM safety evaluation)· empirical study

Recall Is Not Protection: Evaluating Safety Monitors Against Model Compliance

Q: What does elicitable mean in this paper?

A harmful prompt is called elicitable if the target language model complies with it at least once out of eight responses sampled at temperature 1.0, with compliance decided by an LLM judge. Prompts the model never complies with are non-elicitable, and the paper argues that catching those adds little real protection because the model would have refused them anyway.

Q: What is the main finding about safety monitors like Llama Guard?

At a fixed 1% false-positive rate, all six tested monitors (activation probes, Llama Guard 3 8B, WildGuard, gpt-oss-safeguard-120B) detect harmful prompts the model would actually answer substantially worse than prompts the model would refuse: recall is 0.22 to 0.38 lower on the prompts models comply with, so headline recall overstates real-world protection.

Q: Which models and datasets were used in the study?

Three target models (Gemma-3-27B-IT, Qwen3-32B, Llama-3.3-70B-Instruct) were probed with 714 English harmful prompts from Aegis 2.0, HarmBench, and StrongREJECT. Monitors were linear and attention probes trained on WildGuardMix, plus Llama Guard 3 8B, WildGuard, and gpt-oss-safeguard-120B, all calibrated to 1% false positives on a WildChat benign pool.

Q: Is code or data available for this paper?

The reviewed text does not state a code release. However, the appendices give detailed reproduction information — probe training hyperparameters, calibration pool construction, verbatim harm policies, and rollout settings — and everything relies on publicly available datasets and model weights, though redoing rollouts and LLM judging requires GPU and API budget.

Sripad Karne

View original paper ↗arXiv:2609.05797

cs.CL

Quick verdict

Monitor recall drops 0.22–0.38 on prompts LLMs comply with; compelling evaluation critique, but small elicitable samples and LLM-judge labels limit certainty.

Summary

The paper argues that safety monitors for LLMs are evaluated with the wrong metric: recall against harmfulness labels counts catches of prompts the target model would have refused anyway, inflating apparent protection. The author introduces an elicitability protocol — eight temperature-1.0 samples per harmful prompt, with an LLM judge (Claude Opus 5, Opus 4.8 fallback) labeling complied/refused/deflected — and reports monitor recall separately on elicitable and non-elicitable prompts. On 714 English harmful prompts from Aegis 2.0, HarmBench, and StrongREJECT, six monitor configurations (linear and attention probes trained on WildGuardMix, Llama Guard 3 8B, WildGuard, and gpt-oss-safeguard-120B under two policies) all show recall 0.217–0.325 lower on elicitable prompts at 1% FPR on Gemma-3-27B-IT; probes on Qwen3-32B and Llama-3.3-70B show 0.30–0.38 gaps. Missed prompts are 2.8–5.6x more likely to be complied with; monitor scores correlate negatively with compliance rate (r -0.31 to -0.46). The gap holds across all 18 monitor-source and 24 monitor-FPR cells. Conservative handling of unjudged responses (3.9–4.2% residual) biases against the result. Limitations: English single-turn prompts only, no jailbreaks, input-side monitors only, no mechanism identified, small elicitable strata (83–138 prompts), and no code release stated.

Review complete

Assessment at a glance

AI safety / NLP (LLM safety evaluation)

risk flags identified

● high 0

● medium 3

● low 1

Reproducibility readinessmedium

Code & data availablenot stated

Review confidencehigh

AI text signal0/20 passages flagged

Deep research viableCode-reproducibleNeeds GPU

5 strengths · 5 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

Standard recall against harmfulness labels overstates the protection safety monitors provide, because it counts catches of prompts the target model would have refused anyway.
Across six monitor configurations and three model families, recall on elicitable prompts is 0.22-0.38 lower than on non-elicitable prompts at a fixed 1% false-positive rate.
Prompts a monitor misses are 2.8-5.6 times more likely to be complied with than prompts it catches, and monitor scores correlate negatively with per-prompt compliance rate (-0.31 to -0.46).
The failure is not an artifact of input modality, training corpus, or scale: it appears in activation probes, fine-tuned guards, and a 120B policy-conditioned reasoner, suggesting monitors key on prompt surface harmfulness rather than model behavior.

Strengths

Correct identification of a real metric flaw, with a direct measurement
Section 1-3.2 reframes monitor evaluation around model behavior: a catch only prevents harm if the model would comply. The elicitability split (8 temp-1.0 samples, comply at least once) is operationalized cleanly and computed with the monitor uninvolved, avoiding circularity.
Breadth and robustness of the sign result
Six monitor configurations across two modalities (probes, guard models) and three target families; the gap is positive in all 18 monitor-source cells (Table 3) and all 24 monitor-FPR cells from 0.5% to 5% (Appendix C), plus threshold-free Pearson correlations of -0.305 to -0.463 (all p < 10^-16).
Conservative bias handling on the judge
Unusually transparent implementation detail
Memorable qualitative mechanism sketch

Weaknesses

Small elicitable strata make several estimates fragile
Compliance ground truth rests on one LLM judge family
Uneven statistical reporting
Restricted external validity
No code or data release stated

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

The paper argues that recall against harmfulness labels is the wrong headline metric for input-side safety monitors, because a catch only prevents harm when the target model would have complied. It operationalizes *elicitability* (at least one compliance in eight temperature-1.0 samples, LLM-judged) and reports monitor recall split by that behavior. Central empirical claims: recall gaps of 0.217-0.325 across six monitor configurations on Gemma-3-27B-IT at 1% FPR (Table 1); 0.298-0.377 for linear probes on Qwen3-32B and Llama-3.3-70B (Table 2); missed prompts 2.8-5.6x more likely to be complied with; the gap positive in all 18 monitor-source cells (Table 3) and all 24 monitor-FPR cells from 0.5% to 5% FPR (Appendix C). The paper is explicitly a metric critique, not a monitor proposal, and Section 5 scopes claims to English, single-turn, non-adversarial prompts and input-side monitors.

Methodology Assessment

The design is clean and the obvious circularity objection — monitors defining their own strata — is avoided: elicitability is computed with the monitor uninvolved (Section 3.2). Several confound controls are well done. All monitors sit at a common 1% FPR on a 10,000-prompt WildChat pool with dedup and decontamination (Appendix A.4). Probes are trained on only 3,000 WildGuardMix prompts with class-conditional length matching; the length-only classifier at AUROC 0.502 rules out a length artifact (A.1). WildGuard doubles as a control sharing the probes' training corpus.

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 5 strengths & 5 weaknesses, analyzed in detail
4 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What does elicitable mean in this paper?

A harmful prompt is called elicitable if the target language model complies with it at least once out of eight responses sampled at temperature 1.0, with compliance decided by an LLM judge. Prompts the model never complies with are non-elicitable, and the paper argues that catching those adds little real protection because the model would have refused them anyway.

What is the main finding about safety monitors like Llama Guard?

At a fixed 1% false-positive rate, all six tested monitors (activation probes, Llama Guard 3 8B, WildGuard, gpt-oss-safeguard-120B) detect harmful prompts the model would actually answer substantially worse than prompts the model would refuse: recall is 0.22 to 0.38 lower on the prompts models comply with, so headline recall overstates real-world protection.

Which models and datasets were used in the study?

Three target models (Gemma-3-27B-IT, Qwen3-32B, Llama-3.3-70B-Instruct) were probed with 714 English harmful prompts from Aegis 2.0, HarmBench, and StrongREJECT. Monitors were linear and attention probes trained on WildGuardMix, plus Llama Guard 3 8B, WildGuard, and gpt-oss-safeguard-120B, all calibrated to 1% false positives on a WildChat benign pool.

Is code or data available for this paper?

The reviewed text does not state a code release. However, the appendices give detailed reproduction information — probe training hyperparameters, calibration pool construction, verbatim harm policies, and rollout settings — and everything relies on publicly available datasets and model weights, though redoing rollouts and LLM judging requires GPU and API budget.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper