SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation
Fatih Deniz, Yazan Boshmaf, Issa Khalil
cs.CR · cs.LG · cs.SE
Quick verdict
Dynamic SSP benchmark exposing static-leaderboard failures via construct mixing; strong τ evidence but authors' own baseline and partly n.s. moderation leg.
Summary
The paper introduces SSP-Bench, a framework that dynamically generates evaluation items for large language models across four safety, security, and privacy services: safety, hallucination, over-refusal, and privacy. Candidate items are grounded in external sources (Wikipedia, the AIAAIC incident corpus, and the SPY PII dataset), filtered by scope, label, and quality gates, calibrated by a multi-model steering panel, and selected via multi-objective optimization over difficulty, separability, novelty, and diversity. Evaluating 24 models against the aiXamine static leaderboard, the authors report near-zero correlation for the static safety aggregate (Kendall's τ = −0.016) versus τ = 0.670 when restricted to seven adversarial-refusal tests, attributing the gap to construct mixing with two content-moderation tests; a HELM Safety replication on ten shared models supports the pattern. Further findings include Gemma-3 inverse safety scaling (93.8% at 1B versus 85.5% at 27B dynamically, compressed to a 1.1% static gap), a hallucination knowledge plateau, and pattern-specific privacy leakage (paraphrase leak 93.9% across models). Strengths include exact privacy ground truth, human item validation (κ = 0.85 on safety), and dual-judge cross-checks. Key limitations: the primary static baseline is the authors' own leaderboard; the moderation-negative correlation leg is statistically non-significant; and table names mask the smaller underlying model checkpoints (mini/haiku variants).
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Static safety leaderboards mix two constructs — adversarial refusal and content moderation — and their aggregate correlates near-zero with dynamic rankings (Kendall's τ = −0.016), while refusal-only tests recover τ = 0.670.
- SSP-Bench generates scope-consistent, externally grounded, difficulty-calibrated dynamic SSP benchmarks with cross-run ranking stability, achieving Q > 0.80 on all four services (Table 1).
- Dynamic evaluation reveals within-family regressions invisible to static scores, notably Gemma-3 safety dropping 8.3 points from 1B (93.8%) to 27B (85.5%) while the static gap is 1.1%.
- No model exceeds 53.9% safe rate on privacy across six attack patterns, with benign-framed tasks (paraphrase leak 93.9%) bypassing safety alignment while structured query formats trigger refusal.
Strengths
Quantitatively grounded construct-mixing diagnosis
Section 4.1 / Table 14 decomposes the static safety aggregate into seven adversarial-refusal tests (recovering Kendall's τ = 0.670, 95% CI [+0.478, +0.841]) and two content-moderation tests; the aggregate sits at τ = −0.016 [−0.394, +0.360]. Robustness is checked on grounded vs mutation subsets (τ = 0.732 / 0.735) and via an independent HELM replication on 10 models (Mean τ = +0.600), including a within-HELM negative XSTest correlation (τ = −0.511).
Externally grounded labels avoid self-labelling bias
Privacy ground truth is exact by construction (Faker PII substituted into SPY-annotated documents, Appendix F.1); hallucination answers are RAGAS-filtered against retrieved Wikipedia passages; safety/over-refusal items are human-validated on 200 items per service with κ = 0.85 (safety) and 99.0% agreement (over-refusal), Appendix D.
Multi-pronged stability and validity protocol
Actionable fine-grained privacy decomposition
Weaknesses
Primary static baseline is the authors' own leaderboard
The moderation-negative leg of construct mixing is not statistically significant
Table names mask the actual evaluated checkpoints
Key pipeline parameters not stated in the visible text
Artifact release unverifiable
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
SSP-Bench: Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SSP-Bench?
SSP-Bench is a framework from QCRI researchers that dynamically generates fresh test questions for evaluating large language models on safety, hallucination, over-refusal, and privacy, instead of reusing fixed public benchmarks. Labels are grounded in external sources (Wikipedia, a real-world AI incident database, synthetic PII documents), items pass validation gates, and a multi-objective selection keeps only items that distinguish models well.
Why do static safety leaderboards disagree with dynamic evaluation in this paper?
Because the static safety score averages two different things: seven adversarial-refusal tests (does the model refuse harmful prompts) and two content-moderation tests (does the output contain harmful text). A model that refuses a lot produces little text for moderation classifiers to flag, so the two scores pull in opposite directions. The aggregate correlates at Kendall's τ = −0.016 with dynamic rankings, while refusal-only tests alone reach τ = 0.670.
Which models were evaluated in SSP-Bench?
24 models accessed through OpenRouter between March and April 2026, including GPT-5-family and GPT-4o, Claude Haiku 4.5 and 3.5, Gemini 3 Flash preview, DeepSeek-R1, Grok 3/4, Llama-3.3-70B, Phi-4, and the Gemma-3 (1B to 27B) and Qwen 3.5 families. Notably, several table names in the paper refer to smaller variants (mini, haiku, flash) rather than flagship checkpoints.
Is SSP-Bench's code or data publicly available?
The paper states that code, generated benchmarks, and evaluation scripts are available, but the link is anonymized for review, so the release cannot yet be verified externally. The privacy benchmark is deterministic given a random seed, and exact model API identifiers are recorded in the appendix.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Section 2.3 (selection objective details), part of Section 3, the related-work section, and Appendix C (judge selection and dual-judge methodology) are truncated in the extracted text.
- Figure 1 (pipeline diagram) is referenced but not analyzable from the text extraction.
- Appendix B panel-composition criteria and Appendix E.3/E.4 tables are only partially visible, so some methodological judgments rely on main-text summaries.
- The artifact link is anonymized, so code/data availability is as claimed, not verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.