Benchmarking LLM Compliance with China AI Generated Content Regulations
Chenrui Cui, Hongye Fang, Lisha Song, Weichao Chen, Yue Zhu, Gang Xu
cs.CL · cs.MA
Quick verdict
A legally grounded benchmark for China's AIGC rules with an ALHF judge pipeline; promising though judge circularity and small-n correlations warrant caution.
Summary
The paper introduces a benchmark evaluating 20 LLMs against China's AI-generated content regulations, using 2,303 questions across six dimensions (2,100 from the public AAIBench suite plus 203 self-constructed constitutional questions) and reporting both compliance and refusal rates. Its methodological contribution is an 'Agent Learning from Human Feedback' (ALHF) pipeline: three LLM judges (Qwen3.5-plus, GPT-5.4-nano, Gemini-3.1-flash-lite) vote independently, each maintaining three layers of alignment memory (semantic, episodic, conflict) distilled from legal-expert corrections by DeepSeek-chat. On a 249-question held-out set, the multi-judge+ALHF configuration reaches 98.8% accuracy and Cohen's κ=0.927 against expert majority vote. Results show 5 of 11 international models exceed the 90% compliance threshold; Chinese models lead mainly on ideologically charged dimensions (constitutional compliance, socialist core values), with per-category chi-square p<0.001. A constitutional-internalization experiment (Pearson r=0.687, n=10 models) and incremental jailbreak tests (average compliance 92.8%→76.0%) round out the analysis. Key limitations: judges and the internalization scorer (Qwen2.5-32B) may inherit Chinese-context compliance biases that could inflate agreement with expert labels and CN-model scores; model release dates span llama2_13b to GPT-5.1 without confound control; the correlation lacks confidence intervals; no code release is committed beyond supplemental data.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Five of eleven international LLMs exceed the 90% compliance threshold when tested with standard Chinese questions, indicating gaps with Chinese models are smaller than commonly assumed.
- The compliance gap between Chinese and non-Chinese models concentrates in ideologically loaded categories — constitutional compliance and socialist core value violations — while converging on commercial legality.
- The ALHF multi-judge pipeline with hierarchical alignment memory agrees with legal-expert majority vote at 98.8% accuracy and Cohen's κ=0.927 on a 249-question holdout.
- Compliance rates positively correlate with constitutional internalization (Pearson r=0.687 across 10 models), suggesting the Constitution functions as a normative anchor for alignment.
- Coercive non-refusal jailbreak instructions degrade compliance for all models (average 92.8%→76.0%), whereas dangerous-persona emphasis triggers stronger safety behavior in Chinese models specifically.
Strengths
Legally grounded benchmark design
Built with a law school and anchored to TC260's Basic Security Requirements and GB/T 45654-2025 (semantic memory rule in Fig 2); it is, to the authors' claim, the first to add constitutional-compliance questions (203, in three sub-types) to a Chinese LLM safety benchmark.
Validated judge pipeline with ablation
Table 1 shows ALHF lifts single-judge (GPT-5.4-nano) recall from 0.591 to 0.818 and F1 from 0.650 to 0.766; multi-judge+ALHF reaches Acc 0.988, F1 0.933, Cohen's κ 0.927 against expert majority vote on a 249-question holdout, with expert Fleiss' κ=0.610 honestly reported.
Multi-angle empirical coverage
Transparent prompt documentation
Honest edge-case handling in memory design
Weaknesses
Judge circularity and potential alignment bias
Causal language resting on a small-n correlation
Uncontrolled release-date confound
Statistical reporting gaps
Partial reproducibility and thin dataset validation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper, a collaboration between Tongji University's CS and Law schools, introduces what it calls the first operational benchmark measuring how well 20 notable LLMs (11 US, 8 Chinese, 1 EU — Mistral) comply with China's AI-generated content regulations. The instrument spans 2,303 questions across six categories: 2,100 from the public AAIBench security suite plus 203 self-constructed constitutional questions (90 factual, 90 statute-comprehension, 23 scenario). It reports compliance and refusal rates (Figure 1), category-level chi-square breakdowns (Figure 3), a constitutional-internalization experiment (Table 2), and incremental jailbreak tests (Table 3). Central claims: 5/11 international models exceed the 90% compliance threshold; cross-country gaps concentrate in ideologically loaded categories; the ALHF judge pipeline matches legal experts at 98.8% accuracy; and compliance correlates with constitutional internalization (r=0.6866).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is this paper about?
It introduces a benchmark of 2,303 questions that measures how well 20 large language models comply with China's regulations on AI-generated content, scoring both compliance rates and refusal rates across six categories, including a novel constitutional-compliance dimension built by legal scholars.
Do international LLMs like GPT and Claude comply with Chinese content rules?
According to the paper, many do: 5 of the 11 tested international models exceed the 90% compliance threshold even with standard Chinese questions. The largest gaps versus Chinese models appear in ideologically sensitive categories — constitutional compliance and socialist core values — while behavior converges on commercial-legal topics.
How does the paper judge compliance automatically?
Three LLM judges (Qwen3.5-plus, GPT-5.4-nano, Gemini-3.1-flash-lite) vote independently, each maintaining three layers of memory (semantic rules, past cases, boundary cases) distilled from legal experts' corrections. On a 249-question holdout, this multi-judge setup matched legal-expert majority vote at 98.8% accuracy with Cohen's κ=0.927.
Are the benchmark and code publicly available?
Partially. The base 2,100 questions come from the public AAIBench dataset on ModelScope, and the paper says results and expert feedback are in supplemental data. However, no public code repository or explicit release of the 203 self-built constitutional questions is stated, so full reproduction is not currently possible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 3 are embedded as garbled text in the extraction; per-model compliance/refusal values could not be read, so cross-model judgments rely on reported summaries and Tables 1-3.
- The extraction appears to be arXiv v1 with possible truncation; supplemental data, true figure resolutions, and any code-release statements outside the extracted text could not be verified.
- Some model/judge names (GPT-5.4-nano, Gemini-3.1-flash-lite, GLM-4.7) and 2026-dated citations could not be cross-checked against external sources per review constraints.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.