SinoGlyphBench: A Diagnostic Benchmark for Chinese Glyph-Level Obfuscation in Language-Model Moderation
Yifan Wang, Zimu Wang, Suliu Qin, Changyu Zeng, Tong Chen, Siqi Chen, Yijie Lin, Lingyu Jiang, et al.
cs.CL
Quick verdict
Matched-pair benchmark shows glyph obfuscation degrades Chinese moderation across 12 models; small corpus and modest label agreement temper generality.
Summary
The paper introduces SinoGlyphBench, a diagnostic benchmark measuring how glyph-level obfuscation — character decomposition and cross-script substitution with visually similar symbols — degrades LLM- and MLLM-based Chinese content moderation. Source items from STATE-TOXICN, CNTP, and PCR-TOXICN are normalized to a four-label schema (hostile, abusive, benign, context-dependent); annotators mark label-critical 'semantic anchors', and each item yields matched original, anchor-only, background-only, and full-scope variants as Unicode text or rendered images. Human studies verify recoverability (99.13% exact character recovery across 572 forms) and a preference for cross-script forms. Across 176,916 paired evaluations of 12 models, obfuscation raises harmful false-negative and false-positive rates by 6.1 and 4.7 points, lowers four-way accuracy by 5.0 points, and retains 75.7% of originally correct decisions. Full-scope perturbation is most damaging; anchor-only hurts more than background-only, a gap that survives density adjustment (2.0 to 1.8 points); cross-script substitution is harder in text but not in image modality. Obfuscation-aware prompts and image rendering bring small, setting-dependent gains. Limitations include a small corpus (980 broad-panel and 157 strict-panel items), modest pre-reconciliation label agreement (κ = 0.333), reliance on hosted model endpoints with unverified deployments, and a license-conditional data release.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across 176,916 paired evaluations of 12 LLMs/MLLMs, glyph obfuscation increases harmful false-negative and false-positive rates by 6.1 and 4.7 points, lowers four-way accuracy by 5.0 points, and retains 75.7% of originally correct decisions (Table 1).
- Perturbing label-critical semantic anchors hurts more than perturbing background text (∆Acc 4.4 vs 2.1 points), and the gap survives density adjustment (2.0 → 1.8 points, App. D.2).
- Full-scope obfuscation is the most damaging condition (∆FNR +12.5 points; CondAcc 68.3%, Table 1).
- Cross-script substitution is harder than decomposition in text but easier in rendered-image form (strict-panel CondAcc 76.2 vs 83.6 in text; 80.7 vs 78.1 in images, Tables 2 and 9).
- Observable mismatches occur at multiple stages — visible-form reading, intended-message recovery, and final judgment — so no single-stage intervention removes the degradation (Figure 5, Table 12).
Strengths
Matched-pair design isolates the causal effect of obfuscation location
Each obfuscated condition is compared to an original differing only in scope, with model, prompt, modality, and rendering held fixed (§4.3); anchor-only vs background-only contrasts rest on explicit balance constraints (substitutable-count gap = 0 in the strict panel, §3.1), pre-empting the obvious 'more characters changed' confound.
Density adjustment strengthens the anchor claim
Appendix D.2 shows the anchor-minus-background ∆Acc gap moves only from 2.0 [1.6, 2.5] to 1.8 [1.1, 2.4] points across 124 matched pairs after adjusting for perturbation density, so the scope effect is not an artifact of transformation volume.
Human validation of perturbation quality is unusually thorough
Honest label-stratified reporting and interval-based inference
A genuinely interesting modality finding
Weaknesses
Small corpus, especially the strict panel driving headline contrasts
Modest gold-label reliability
Failure localization is strongly threshold-dependent
Metric asymmetry in the safety deltas
Reproducibility caveats on the model side and data release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SinoGlyphBench (arXiv:2609.05843) addresses one narrow, well-posed question: does rewriting Chinese with *recoverable* glyph-level transformations — component decomposition and cross-script substitution into visually similar symbols — change how LLM/MLLM moderators label the same content? The design holds the source item fixed and varies (i) perturbation location (semantic anchor / background / full scope), (ii) transformation type, (iii) density, and (iv) realization (Unicode text vs rendered image). Central claims are quantified in Table 1: +6.1 pp harmful FNR, +4.7 pp harmful FPR, −5.0 pp four-way accuracy, and 75.7% retention of originally correct decisions across 176,916 paired evaluations of 12 models. The Limitations section is candid that this diagnoses one capability slice, not deployment readiness (no multi-turn, screenshots, emoji, handwriting, or platform conventions).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SinoGlyphBench?
A diagnostic benchmark that measures how writing Chinese with non-canonical glyphs — decomposed characters or visually similar foreign symbols — changes the content-moderation decisions of large language models. Each obfuscated input is compared with a matched original under identical settings, and changes to label-critical words are separated from changes to surrounding text.
How much does glyph obfuscation hurt Chinese content moderation in this study?
Across 12 models and 176,916 paired evaluations, obfuscation raised harmful-content misses by 6.1 percentage points and false alarms by 4.7 points, lowered four-way accuracy by 5.0 points, and kept 75.7% of decisions that were correct on the original inputs. Obfuscating the entire text was worse than obfuscating only key words, which in turn was worse than obfuscating only background text.
Can humans still read the obfuscated Chinese in this benchmark?
Yes — human annotators recovered the exact source character from 99.13% of the benchmark's 572 obfuscated forms, so the evidence remains human-readable by design. Models, however, lost about a quarter of originally correct decisions, with errors appearing at the reading, message-recovery, and judgment stages.
Is the SinoGlyphBench dataset publicly available?
The paper points to a GitHub repository with the benchmark, transformation code, prompts, and documentation, but release is conditional on the licenses of the three source datasets; where redistribution is not permitted, the authors say they will provide construction scripts and derived metadata and direct users to the official sources.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction serializes tables and figures as inline run-on text (e.g., Table 1, Table 5, Figure 6); every value cited in this review was present in the extraction, but column alignment is inferred from layout order.
- The actual contents of the linked GitHub repository (what data and code are released today) could not be verified in this review; codeAvailable reflects the paper's statement only.
- Per-cell sample sizes behind the strict-panel tables (13–16) are not stated in the text; the observation about small denominators is an inference from value granularity, not a reported fact.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.