Same Request, Different Boundary: Evaluating Cybersecurity Assistance across Conversational Contexts
Rui Yang, Yang Hong, Yichao Xu, Zhengyu Liu, Ziyang Li, Yinzhi Cao
cs.AI · cs.CR
Quick verdict
Accepted precedent lifts compliance +23.1pp, decomposition cuts it −45.1pp; paired design is sound but labels rest on automated coders, no human gold check.
Summary
The paper introduces 3R-Bench, a benchmark of 150 real-world cybersecurity requests (50 benign, 50 ambiguous, 50 high-risk) reconstructed from Security StackExchange posts, testing whether LLM safety boundaries shift with conversational context. Eight hosted models (Claude Opus and Sonnet 4.6, three GPT-5.6 variants, MiniMax M2.5, GLM-5, Qwen3-Next 80B) face three scenarios: direct requests (S1), a forged history in which the assistant ostensibly accepted or refused the same request (S2), and a six-message task decomposition ending in neutral or failure-feedback requests (S3). Responses are coded refuse/bounded/comply, with provider blocks tracked separately. In S2, compliance across 376 matched pairs rises from 62.0% (233) after refused history to 85.1% (320) after accepted history (+23.1 points, bootstrap CI +16.6 to +29.9); an unrelated-accepted control also gains +23.1 points, so the effect does not require target disclosure. In S3, compliance falls from 501/800 direct to 172/800 after dialogue (−41.1 points; −45.1 among 738 semantic pairs), and failure feedback recovers only +6.3 points in the 44-request exact-template control. Key limitations: labels come from automated coders (GPT-5.6 Sol, audited by Claude Opus 4.6, κ=.748) without human gold validation; item-level selection records are not released; and raw prompts, histories, and responses are withheld, blocking independent reproduction.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Prior assistant behavior alone shifts boundary decisions: on the same 50 ambiguous requests, compliance rises from 233/376 (62.0%) after refused history to 320/376 (85.1%) after accepted history, +23.1 points.
- The acceptance effect does not require target disclosure: task-unrelated accepted precedent adds +23.1 points over refused history, and target-specific acceptance adds a further +13.6.
- Contrary to multi-turn jailbreak expectations, six-message task decomposition sharply reduces compliance: 501/800 direct vs 172/800 after failure-ended dialogue (−41.1 points; −45.1 among 738 semantic pairs).
- Failure feedback recovers only a small fraction of the decomposition-induced loss: +6.3 points in the 44-request exact-template control, still far below direct presentation.
Strengths
Genuinely novel evaluation axis
First paired test of unchanged cybersecurity requests under controlled conversational states — S2 forged precedent, S3 decomposition (Table 1, Table 5); prior datasets (HarmBench, CyberSecEval 2, SORRY-Bench) test direct prompts only.
Counterintuitive, decision-relevant S3 finding
Six-message decomposition cuts compliance from 223/332 to 46/332 (−53.3 pp) in the exact-template control and from 501/800 to 172/800 (−41.1 pp) at scale — the opposite of the escalation assumption motivating multi-turn jailbreaks (Section 4.4).
Disciplined pairing and availability accounting
Control and robustness battery
Weaknesses
No human gold standard for outcome labels
Artifact blocks independent verification
Headline S2 estimate is length-confounded
Fragile small panels and sign-flipping sensitivity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
3R-Bench is a 150-request benchmark — 50 benign, 50 ambiguous, 50 high-risk — reconstructed from 66,443 crawled records and screened by an automated pass plus three cybersecurity experts into a balanced cohort (Figure 2). Eight hosted frontier models (Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.6 Sol/Terra/Luna, MiniMax M2.5, GLM-5, Qwen3-Next 80B) are tested under three conversational states while the terminal request stays fixed: direct presentation (S1), a forged five-message history in which the assistant ostensibly accepted or refused the same request (S2), and a seven-message decomposition ending in neutral or failure-feedback final turns (S3). Outcomes are R/B/C with provider blocks reported separately (Table 2). The central claims are bidirectional context-dependence: forged acceptance lifts compliance on 376 matched pairs from 62.0% (233) to 85.1% (320), +23.1 points; decomposition collapses it from 501/800 to 172/800 attempted (−41.1) and −45.1 points on the 738 semantic pairs.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 3R-Bench?
A benchmark of 150 real-world cybersecurity requests (50 benign, 50 ambiguous, 50 high-risk) reconstructed from public Security StackExchange posts, used to test how LLM safety boundaries change across three conversational scenarios: direct request, forged assistant history, and multi-turn task decomposition.
Does giving an LLM a fake history where it already complied make it comply with cybersecurity requests?
The paper finds yes: across eight models, compliance on the same 50 ambiguous requests rose from 62.0% after a fabricated refusal history to 85.1% after a fabricated acceptance history (+23.1 percentage points), and even acceptance of an unrelated request raised compliance by the same margin.
Do multi-turn jailbreak dialogues bypass LLM cybersecurity safeguards?
Not in this study: a six-message task decomposition reduced compliance from 501/800 direct responses to 172/800 (−41.1 points), and failure feedback ('this doesn't work, write another') recovered only about 6.3 points — the opposite of the common escalation expectation.
Can I reproduce the paper's results or download the dataset?
Only partially. The authors release outcome labels, artifact-local identifiers, aggregate statistics and a local verifier (CC BY 4.0), but deliberately withhold raw requests, forged histories and model responses, and item-level selection records; full independent reproduction of the numbers is not possible from the released artifact.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3–7 arrive in the extraction as jumbled axis labels and stray numbers; figure-level readings were taken from captions and body text, not the rendered plots.
- Per-model tables (Tables 7–8) extract as flattened number strings; all counts were legible, but cell-to-model alignment relies on the caption ordering.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.