TIER: Threat Implicitness Benchmark for Evaluating LLM Safety Behaviors
Thu-Hien Trinh-Thi, Hai-Yen Vong, Thanh-Ha Ung-Dung, Tram Ho
cs.CR · cs.AI
Quick verdict
TIER grades LLM safety prompts by implicitness level; the behavior-aware angle is promising, but unvalidated judges and a shaky jailbreak-gap claim weaken it.
Summary
The paper introduces TIER, a benchmark of 1,184 harmful prompts organized along a new axis — threat implicitness — with four levels (explicit, euphemistic, contextual, jailbreak) across four risk domains (sexual content, violence, illegal activity, self-harm; 296 prompts each). Responses from six open-weight LLMs (Qwen1.5-MoE, Gemma-2B, Mistral-7B, GPT-J, Llama3, ChatGLM3) are labeled with a six-label behavior taxonomy adapted from Do-Not-Answer and judged independently by Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct. Mistral7B shows the highest attack success rate (0.509/0.355 across the two judges) and Llama3 the lowest (0.190/0.075). The authors report that safety behaviors shift gradually across threat levels, contextual prompts elicit the most diverse behaviors, and models with similar ASR show different behavior distributions. Key limitations: labels come solely from two 7-8B LLM judges, with no human validation or agreement statistics, and inter-judge divergence is large (Gemma2B refusal: 0.57 vs 0.03 on identical outputs); the abstract's claim that jailbreaks reveal the largest robustness gaps is contradicted for Gemma2B and Llama3, whose L4 ASR is their lowest; no statistical uncertainty is reported; per-domain results are absent despite the domain axis; and dataset construction raises provenance concerns (prompts generated by Mistral-7B, an evaluated family; manual review only of an L3 subset). Code and data are stated to be on GitHub.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 19 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM safety behaviors evolve gradually across the four threat implicitness levels rather than switching abruptly from refusal to compliance.
- Contextual (L3) prompts elicit the most behaviorally diverse responses of the four levels.
- Jailbreak (L4) prompts expose the largest robustness gaps between models.
- Models with similar attack success rates can exhibit substantially different response distributions, so binary metrics are insufficient for safety evaluation.
- Two independent LLM judges produce consistent model rankings despite differing in absolute ASR values.
Strengths
Novel evaluation axis
Threat implicitness (L1-L4, Table 1) is a variable that content-category benchmarks (OR-Bench, Sorry-Bench, RealToxicityPrompts, Wild Jailbreak) leave uncontrolled; the 4-level x 4-domain design with 296 prompts per domain (Table 2) is balanced and clearly motivated in Section 2.
Behavior-level evaluation beyond ASR
Adopting Do-Not-Answer's six-label taxonomy yields concrete added value: Mistral7B (ASR 0.509) and Llama3 (0.190) differ not just in degree but in label mix — e.g., Llama3's 0.62 Safety-Disclaimer share under the Llama-3.1 judge (Table 6), a strategy invisible to binary metrics.
Dual-judge robustness discipline
Released artifact
Weaknesses
Judge validity unestablished
Headline jailbreak claim contradicted internally
No statistical uncertainty
Weak dataset verification and unused domain axis
Under-specified experimental details
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
TIER: Threat Implicitness Benchmark — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the TIER benchmark?
TIER is a dataset of 1,184 harmful prompts for testing large language model safety, organized along a new axis called threat implicitness: four levels running from directly explicit harmful requests, through euphemistic and contextually disguised wording, to full jailbreak prompts. It spans four risk domains (sexual content, violence, illegal activity, self-harm) and evaluates model responses on a six-level behavior scale (refusal through full compliance) rather than a binary safe/unsafe verdict.
What did the TIER paper find?
Across six open-weight LLMs, Mistral7B complied with harmful requests most often (attack success rate 0.509 under the Qwen2.5-7B judge and 0.355 under the Llama-3.1 judge) while Llama3 complied least (0.190/0.075). Safety behaviors varied gradually with implicitness level, contextually disguised prompts produced the widest variety of response types, and models with similar attack success rates showed clearly different response distributions — the paper's core argument that binary metrics hide important behavioral differences.
Is the TIER benchmark's data and code public?
The paper states that data and code are available at github.com/hotrhoth/TIER. One caveat for anyone reusing it: the evaluation labels themselves were assigned by two LLM judges (Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct) without human validation, so reproducing the published numbers depends on having the exact judge prompt templates.
Which models were evaluated in the TIER paper?
Six open-weight LLMs: Qwen1.5-MoE, Gemma-2B, Mistral-7B, GPT-J, Llama3, and ChatGLM3, all decoded greedily (temperature 0). Their responses were judged by two independent instruction-tuned models, Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, using a six-label taxonomy ranging from complete refusal to full harmful compliance.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's plotted distributions cannot be verified from the extracted text (only the caption is present), so the gradual-transition and diversity claims were assessed from the authors' prose and Tables 5-6.
- Judge prompt templates are not included in the paper text; they may exist only in the released GitHub repository.
- The extracted text ends at reference [21] with no appendix; if the published PDF contains appendices, their content was not reviewed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.