An Empirical Measurement of Jailbreaking Evaluators
cs.CR · cs.LG
Quick verdict
Claims first unified six-jailbreak-evaluator comparison; controlled design, but dataset provenance favors JADES and no significance tests weaken conclusions.
Summary
The paper presents the first unified empirical comparison of six widely used jailbreak evaluators—HarmBench, JailbreakBench, JailbreakRadar, StrongReject, JADES, and JailMeter—against human judgments on two labeled datasets: JailbreakQR (400 prompt-response pairs, three-level labels, Krippendorff's alpha = 0.823) and JailMeter-Eva (330 non-refusal GPT-4o-mini responses, binary labels, 73 positives). To isolate evaluation methodology from judge quality, all prompt-based evaluators share one LLM backbone (an uncensored Qwen variant at temperature 1.0), while HarmBench uses its own fine-tuned classifier. JADES performs best on both datasets (92.3% accuracy, 5.8% FPR on JailbreakQR; 95.2% accuracy, 3.9% FPR on JailMeter-Eva), followed by StrongReject and HarmBench; holistic single-verdict judges JailbreakBench and JailbreakRadar lag, especially in precision on the class-imbalanced JailMeter-Eva. The authors conclude that structured evaluation procedures (decomposition, refusal gates, rubrics) agree better with humans and that evaluator choice materially shifts measured attack success. Limitations include English-only, moderate-sized data, noisy human references, and dataset provenance: each dataset originates from a team that also proposed one of the compared evaluators, and the study reports no significance testing for the observed gaps.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No prior study has evaluated all six evaluators (HarmBench, JailbreakBench, JailbreakRadar, StrongReject, JADES, JailMeter) on the same human-labeled data under a controlled setup.
- JADES achieves the best overall agreement with human judgments on both datasets (92.3% accuracy / 5.8% FPR on JailbreakQR; 95.2% / 3.9% on JailMeter-Eva), with StrongReject and HarmBench next.
- Structured evaluation procedures (decomposition, refusal gates, rubrics) agree better with human judgments than single holistic prompted verdicts.
- Evaluator choice is itself a major source of variation in measured jailbreak success, independent of the underlying judge model.
Strengths
First unified six-evaluator comparison on shared human labels
Sections 1 and 4 deliver what the fragmented literature lacks: HarmBench, JailbreakBench, JailbreakRadar, StrongReject, JADES and JailMeter scored against the same human annotations on two datasets (JailbreakQR n=400, JailMeter-Eva n=330), under one protocol with a common label mapping (Section 4.2).
Shared judge backbone isolates methodology from judge quality
Section 4.3 fixes one LLM (orcarouter/Qwen3.8-27B-Uncensored, temperature 1.0, native prompts retained) across all five prompt-based evaluators, so the observed spreads (e.g., JailbreakBench 81.2% F1 vs JADES 92.5% F1 on JailbreakQR) reflect evaluation design, not backbone differences.
Application-aware metric selection
Unusually honest limitations section
Cross-dataset consistency design
Weaknesses
Promised analyses are missing from the results
No statistical significance testing
Dataset provenance circularity favors JADES
Single, nonstandard judge backbone
Reference quality unverifiable on the harder dataset
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a meta-evaluation question: when jailbreak research relies on automated evaluators to decide whether an attack succeeded, how closely do the six evaluators recurring in the literature agree with human judgments? The six (Table 1) span the design space: HarmBench (fine-tuned Llama-2-13B behavior classifier), JailbreakBench (holistic prompted judge), JailbreakRadar (three-criterion prompted judge), StrongReject (refusal gate + convincingness/specificity rubric), JADES (decomposition into weighted sub-questions), and JailMeter (information-bottleneck evidence extraction + judgment). Central claims: (1) no prior study has evaluated all six on the same human-labeled data under a controlled setup; (2) JADES is best overall — 92.3% accuracy, 92.5% F1, 5.8% FPR on JailbreakQR and 95.2% accuracy, 89.3% F1, 3.9% FPR on JailMeter-Eva (Figure 1); (3) structured procedures beat single holistic verdicts; (4) evaluator choice materially changes measured attack success. Scope is deliberately narrow: no new attack or evaluator, purely methodological.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which jailbreak evaluator agrees best with human judgments?
According to this study, JADES — a decompositional scorer — agrees best with human labels on both JailbreakQR (92.3% accuracy, 5.8% false-positive rate) and JailMeter-Eva (95.2% accuracy, 3.9% FPR). StrongReject and HarmBench follow, while single-verdict LLM judges like JailbreakBench and JailbreakRadar lag, especially in precision on ambiguous non-refusal responses.
What are the JailbreakQR and JailMeter-Eva datasets?
Two human-labeled jailbreak evaluation sets. JailbreakQR contains 400 prompt-response pairs labeled failed, partially successful, or successful by three annotators (Krippendorff's alpha = 0.823), and was built by the JADES authors. JailMeter-Eva contains 330 non-refusal GPT-4o-mini responses with binary labels (73 successful jailbreaks) and was built by the JailMeter authors. This paper uses both as ground truth to score six automated evaluators.
Does the choice of jailbreak evaluator change reported attack success rates?
Yes. Because evaluators encode different definitions of jailbreak success, the same model responses receive different verdicts. Under a shared judge backbone, the six evaluators in this study still differed by roughly 8-11 accuracy points and 5-12 FPR points per dataset. Since false positives directly inflate measured attack success, evaluator choice can materially change conclusions about both attacks and defenses.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 was extracted as a flattened numeric grid; the row-to-evaluator mapping was cross-validated against values cited in the prose (e.g., JADES 92.3/92.5/5.8 on JailbreakQR, JailbreakBench F1 81.2, JailbreakBench 85.5/64.7/69.6 on JailMeter-Eva), but the rendered figure itself was not visually inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.