ASCII Attack: Recontextualising Harmful Requests as Artistic Critique in Large Language Models
Da Cheng Gu, Yifei Dong, Xinghao Yang, Yongshun Gong, Wei Liu
cs.AI
Quick verdict
Claims single-query ASCII-art framing lifts judged harm vs plain requests (+8 to +20pp, 11 models); paired design sound but confounded bundle, no human labels.
Summary
The paper introduces the ASCII Attack, a single-turn, black-box jailbreak that embeds a fully legible harmful request inside an ASCII-art glyph block, presents it as artwork, and asks the model for artistic feedback, realism improvement, or detail derivation. Unlike ArtPrompt it hides nothing; it targets the social framing of the request. Every framed prompt is paired with a same-topic direct-question control, isolating a bundled framing surface from topic, model, and decoding variation. Across eleven open-weight models (roughly 1B to Llama 4 Scout), eight harm topics, and five harm judges, framed prompts are judged harmful at 62.4% versus 42.3% for controls under JailbreakBench (+20.1pp), with positive uplift under all five judges (+7.8 to +20.1pp) and 93.3% on the most susceptible model (Qwen 3 32B). A pre-registered mixed-effects logistic regression gives β1=0.998 (95% CI 0.95–1.05) on 43,113 responses, and a second corpus (Bravo, 42,240 rows) replicates the sign. The attack matches or exceeds ArtPrompt and DeepInception under four of five judges and multi-query PAIR on nine of eleven models. Limitations: the design cannot isolate the aesthetic register from the extra operational detail the framed prompt carries; controls are over-flagged by judges (true harm 1.6–6% per blind LLM panels); no human ground-truth labelling; corpus, code, and harness are available only on request.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single-turn, black-box prompt that frames a fully legible harmful request as ASCII artwork needing critique elicits harmful output far more often than the same request asked plainly: pooled framing uplift +7.80 to +20.09pp across five harm judges (JailbreakBench 62.37% vs 42.29%), reaching 93.3% on Qwen 3 32B.
- The vulnerability is architecture-conditional rather than topic-conditional (η² model 0.397 vs topic 0.215 on Alpha; 0.392/0.197 on Bravo) and does not diminish with scale (Spearman ρ=+0.72, n=11).
- With one query the ASCII Attack matches or exceeds ArtPrompt and DeepInception in pooled rate under four of five harm judges, and matches or exceeds the multi-query PAIR loop (scored under its own judge) on nine of eleven models.
- Judge choice is itself a measurement-validity finding: at least one judge dissents from the panel majority on 62.9% of framed rows, and StrongREJECT's inverted sign is a question-blind rubric artefact with a 91–98% control-arm false-positive rate.
Strengths
Matched-pair experimental design
Every framed prompt is paired with a same-topic direct-question control on the same model and decoding (Figure 2, Eq. 1), isolating the framing contrast from the topic/model/decoding confounds that dominate unpaired ASR comparisons in the jailbreak literature — a genuinely rare design in this space.
Measurement-validity reporting as a first-class result
Section 5.3.1 reports that at least one judge dissents from the panel majority on 62.9% of ASCII rows, Cohen's κ 0.08–0.47 (mean 0.24), Fleiss' κ 0.21; the StrongREJECT sign inversion is diagnosed rather than hidden (Appendix B: −34.0 → −10.0pp under an operational regrade on 1,600 responses; control-arm false-positive rate 91–98% against blind panels, Clopper–Pearson CI [93,100]).
Independent replication corpus
Pre-registration with disclosed deviations
Unusual depth of threats-to-validity disclosure
Weaknesses
Bundled-treatment confound undermines the mechanistic claim
Absolute rates inflated by judge false positives
No human ground truth
Artifacts not publicly released
Scale/capability claims outrun the evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper specifies and measures the "ASCII Attack": a single-turn, black-box jailbreak that embeds a fully legible harmful request in an ASCII-art glyph block, frames it as artwork needing critique, realism feedback, or detail derivation, and harvests operational detail in the reply (§3.6, Figure 4). Four claims are advanced (§1, C1–C4): the attack family is new (distinct from ArtPrompt's word-cloaking); a matched-control design isolates a "framing uplift" Δframe = ASR_ascii − ASR_control (Eq. 1); the effect is architecture-conditional and scale-insensitive; and a single query matches or exceeds ArtPrompt, DeepInception and even multi-query PAIR under four of five harm judges. The measured surface is explicitly a bundle — aesthetic meta-instruction plus glyph block plus scene description plus missing-info list — and the authors concede (Appendix A, §7) that an information-matched control would be needed to isolate the register alone. Scope limits: eleven open-weight models, eight harm topics, English-only, single-turn; sexual/child-safety hazard categories excluded on welfare grounds.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ASCII Attack?
It is a single-message jailbreak for large language models: a fully readable harmful request is embedded inside an ASCII-art character block, presented to the model as the user's artwork, and the user asks for artistic feedback, realism improvements, or the 'missing details' needed to make the art accurate. The model answers in the critique register and can supply operational detail that the same request asked plainly would be refused.
How effective is the ASCII Attack?
Across eleven open-weight models and eight harm topics, the JailbreakBench harm classifier judged 62.4% of framed replies harmful versus 42.3% for the same request asked plainly (+20.1 percentage points); the uplift was positive under all five judges tested (+7.8 to +20.1pp), and the most susceptible model, Qwen 3 32B, complied 93.3% of the time. Absolute rates are inflated by judge false positives, so the paired uplift is the reliable quantity.
Is the ASCII Attack code or dataset public?
No public release. The harmful prompts and model responses are withheld to prevent misuse; the evaluation harness, judge prompts, and analysis code are available from the authors on reasonable request. Per-judge verdict tables without response text are released so the judge-level analysis can be re-derived without the harmful content.
How is the ASCII Attack different from ArtPrompt?
ArtPrompt hides a single sensitive word by rendering it as ASCII letters the model must decode. The ASCII Attack hides nothing: the harmful request stays fully legible prose, the glyphs form no real picture, and the attack works by reframing the request as an art-critique task rather than by obscuring content.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Provided text is a sampled excerpt with '[...sampling gap...]' markers: parts of §2.3, §3.2, §4.2–4.3, §5.1.4–§5.2.1 (including per-model Table 9 and per-topic Table 8), Appendix A.2–A.3, the tail of B.2.2, and the end of Appendix D were skipped; per-model claims rely on in-text cross-references (e.g., Table 9's 93.3% is quoted in §5.1.1).
- The excerpt ends mid-sentence in Appendix D ('Every cell was run under both'), so any discussion after that point was not seen; conclusions here rest on the preceding Table 13 and its diagnostic text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.