EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models
Yan Zhang, Ruien Li, Yaoyao Peng, Wanxin Ren, Yijia Zhang, Wusheng Zhang, Guangwen Yang
cs.AI
Quick verdict
Law-grounded compliance benchmark (4,435 QA, 24 LLMs) is well-designed, but an unspecified judge LLM and no error bars weaken its human-baseline claims.
Summary
The paper introduces EADC, a benchmark for evaluating whether large language models detect advanced and deep-level compliance risks — implicit/covert risks, multi-step logic-chain risks, and scenario/context-dependent risks — grounded in 13 Chinese and EU AI laws (including the EU AI Act, GDPR, and China's PIPL) via an AI compliance knowledge graph. QA pairs (4,435+) were generated by a compliance-specific LLM using skill-based strategies, then reviewed and corrected by AI legal experts in iterative rounds. Evaluation uses rubric-based must-include coverage and must-not adherence (weighted 0.6/0.4) scored by an LLM judge built on an unspecified open-source model; judge-human agreement on 36 responses is strong (Pearson 0.8152, quadratic weighted kappa 0.7937). Across 24 proprietary and open-source models, scores are low overall: GPT 5.5 leads at 79.34, GPT 5.4 at 78.50, and Claude Opus 4.8 at 72.91, versus a three-expert human baseline of 72.77 on a 2% stratified subset; the best open-source model (Intern-S2-Preview-35B) reaches 59.32. Fairness detection is weakest. Limitations: the judge model is unnamed and the same model class participates in data generation (circularity risk); no confidence intervals or significance tests support the claim that three models surpass the human baseline (Opus 4.8 by 0.14 points); the 0.6/0.4 weighting lacks justification; and no dataset or code link is provided despite a promised release.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing safety benchmarks (Do-Not-Answer, TrustLLM, SafetyBench) cover advanced/deep compliance risks poorly, at 25.77%-42.74% overall (Table 1).
- EADC provides 4,435+ expert-audited QA pairs spanning three deep-level risk dimensions and four indicators, grounded in 13 Chinese/EU AI laws via a compliance knowledge graph.
- 24 frontier LLMs score low on deep compliance detection; GPT 5.5 leads at 79.34 overall and the best open-source model (Intern-S2-Preview-35B) reaches only 59.32 (Table 3).
- GPT 5.5 (79.34), GPT 5.4 (78.50), and Claude Opus 4.8 (72.91) exceed the three-expert human baseline of 72.77.
- Compliance detection scales with model capacity, and legal-citation coverage in model outputs correlates positively with risk-detection ability (Appendix D, N=190).
Strengths
Statute-grounded taxonomy is genuinely novel
A knowledge graph links 13 Chinese/EU legal instruments (Appendix A: EU AI Act, GDPR, PIPL, Interim GenAI Measures, etc.) to risk types and detection rules, addressing the documented statute-anchoring gap of Do-Not-Answer/TrustLLM/SafetyBench (Table 1: overall deep-risk coverage only 25.77%-42.74%).
Unusually transparent judge protocol
Appendix E publishes the complete judge, strict-risk-discovery, and legal-instrument-matching prompts with item-level JSON output schemas; Appendix F case studies expose gold must-include/must-not items alongside verbatim GPT-5.5 responses, enabling real audit of item design.
Judge validation experiment included
Complete, balanced per-cell reporting
Honest disclosure of moderation interference
Weaknesses
Judge model is anonymous and entangled with data generation
No uncertainty quantification behind headline claims
Human baseline shows small-sample artifacts
Metric weighting and gap analysis lack validation
No artifact access and no contamination controls
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EADC (arXiv:2609.26175) is a benchmark paper from Tsinghua, UW-Madison, and UCASS arguing that existing LLM safety benchmarks test only explicit risks and are not grounded in AI statutes. The motivating analysis (Section 1, Table 1) reports that Do-Not-Answer, TrustLLM, and SafetyBench cover advanced/deep compliance risks sparsely — bias/discrimination at 0.75% (Do-Not-Answer) to 7.61% (SafetyBench), 25.77%-42.74% overall — with GPT 5.5 assisting the counting. EADC claims three contributions: a taxonomy of three deep-level risk dimensions (implicit/covert, logic relational reasoning, scenario/context) crossed with four indicators (bias & discrimination, fairness, privacy, values); a 4,435+ QA dataset distilled from an AI compliance knowledge graph over 13 Chinese and EU legal instruments with legal-expert review; and evidence that 24 frontier LLMs score low, with GPT 5.5 (79.34), GPT 5.4 (78.50), and Claude Opus 4.8 (72.91) exceeding a three-expert human baseline of 72.77 (Table 3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the EADC benchmark and what does it measure?
EADC (Evaluation of Advanced and Deep-level Compliance) is a benchmark of 4,435+ question-answer pairs testing whether LLMs detect hard-to-spot compliance risks — disguised/implicit requests, harms emerging from multi-step logical chains, and risks that depend on scenario or conversation context. Items are grounded in 13 Chinese and EU AI laws (e.g., the EU AI Act, GDPR, China's PIPL) via a compliance knowledge graph and were reviewed by AI legal experts.
Which LLM performs best on EADC?
Per the paper's Table 3, GPT 5.5 scores highest at 79.34 overall, followed by GPT 5.4 (78.50) and Claude Opus 4.8 (72.91), against a three-expert human baseline of 72.77 on a 2% sample subset. The best open-source model is Intern-S2-Preview-35B at 59.32. Fairness-related risks are the weakest area for nearly all models.
Is the EADC dataset or code publicly available?
The paper states the authors aim to publicly release EADC, but the extracted text contains no repository link, and the LLM judge used for scoring is not named. Until the dataset is posted and the judge is identified, the reported scores cannot be independently reproduced.
What laws and regulations does EADC cover?
Thirteen instruments: the EU AI Act, GDPR, the ePrivacy Directive, the Digital Services Act, China's Personal Information Protection Law, Cybersecurity Law, Data Security Law, the Interim Measures for Generative AI Services, algorithm-recommendation provisions, deep-synthesis provisions, humanized AI interaction measures, the New Generation AI Ethics Guidelines, and minor-protection cyberspace regulations — stated as 68% of the 19 instruments the authors examined.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-4 and the Appendix D coverage chart are extracted as garbled axis/label text only; visual layouts and Figure 1's pilot-study bars could not be analyzed.
- Supplementary model-version details are thin and per-item dataset samples are absent; item quality was judged from the four Appendix F case studies only.
- No repository URL appears in the text, so the promised dataset/code release is unverifiable from this document.
- arXiv v1 (22 Sep 2026); a few sections referenced as 'in the supplementary materials' (e.g., full model list details) appear only partially included.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.