Conformal Privacy Auditing: Calibrated Re-identification Attacks with Statistical Guarantees
Shuo Huang, Gholamreza Haffari, Xingliang Yuan, Ting Yu, Lizhen Qu
cs.CR · cs.CL
Quick verdict
Applies conformal prediction to LLM re-identification auditing, yielding finite-sample ambiguity certificates; sound work on standard theory.
Summary
The paper introduces Conformal Privacy Auditing (CPA), which recasts re-identification risk auditing of released text as conformal set construction. Under a declared threat configuration, an attacker — TF-IDF retrieval over candidate profiles or an LLM pipeline (Llama-3.1-8B, Qwen3-8B, GPT-5) — scores a finite candidate pool; an APS-style mass nonconformity score is computed on calibration documents, and the split-conformal quantile yields ambiguity sets guaranteed to contain the true identity with probability at least 1−α under exchangeability (Theorem 5.2), extended to open-world pools via Corollary 5.3. Certified set size is the certificate; its inverse is a leakage proxy. On TextWash, TAB, WikiBio, and the Blog Authorship corpus, empirical coverage sits near 0.95 across attacker knowledge levels; certified sets collapse sharply as auxiliary knowledge grows (TAB median |C|: 1268 → 1 from K1 to K3; TextWash famous: 120 → 90 with Llama clues), and drift stress tests show certificates do not transfer across configurations (Blog W10→W1 coverage 0.835), requiring recalibration. Limitations: the theory is a direct application of standard split conformal prediction; attackers are modest (TF-IDF plus clue-appending; GPT-5 clue rows are numerically identical to direct retrieval, suggesting the strongest model was never effectively exercised); closed-world candidate pools; no code release; and some coverage cells fall below target (Table 12, ncal=150: 0.927) without confidence intervals.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CPA converts any attacker that induces a distribution over a finite candidate pool into ambiguity sets with finite-sample marginal coverage at least 1−α under exchangeability, valid for both logit-access and sampling-only attackers (Theorem 5.2, Lemma 5.1).
- Certified identifiability is jointly determined by the release mechanism and the attacker's auxiliary knowledge, as shown by the TAB ladder from median set size 1268 (K1 metadata) to 1 (K3 case-text) in Table 1.
- The effect of LLM-assisted clue extraction is attacker-model- and context-dependent and becomes visible only through set-valued certificates (Tables 2 and 5).
- CPA certificates do not transfer across attacker or release drift and must be recalibrated, with coverage dropping to 0.835 under Blog W10→W1 drift (Table 9).
- Certified set size |Cα| is an interpretable, calibrated ambiguity certificate that can drive concrete release policies.
Strengths
Disciplined threat-model framing
Section 3.3 plus Tables 7-8 force every certificate to declare candidate pool, query mode, LLM/web augmentation, split sizes and sampling budget, with an explicit recalibration protocol; drift experiments (Table 9) empirically justify why this matters.
Unusually thorough stress testing of its own assumptions
Section 7.1 and Appendix E ablate attacker/release drift (0.835 coverage under Blog W10→W1), top-K truncation with candidate-miss rates matching Corollary 5.3 (Table 10: miss 0.160, conditional 1.000, unconditional 0.840), score constructors (Table 11), calibration size (Table 12), sampling budget (Table 13: median |C| 120→78 at m=10) and split stability (0.951 ± 0.037).
Genuinely black-box interface for proprietary models
Actionable empirical findings for auditors
Weaknesses
Limited theoretical novelty
Modest attacker family; strongest model apparently inert
No uncertainty quantification on the headline statistic
Marginal, conditional, closed-world certificates
No released code or artifacts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CPA targets release-time auditing of identity linkage risk for published or de-identified text. Its three stated contributions are: (1) casting re-identification auditing as conformal ambiguity-set construction with finite-sample guarantees (Theorem 5.2; open-world Corollary 5.3); (2) an attacker-agnostic, black-box framework covering logit-access and sampling-only attackers, including proprietary APIs; (3) empirical evidence that calibrated coverage holds across benchmarks while certified identifiability shifts sharply with auxiliary knowledge, LLM augmentation, and release mechanisms. The scope is explicitly conditional: every certificate is valid only for one declared candidate pool, attacker pipeline, knowledge level, and calibration distribution (Section 3.3, Tables 7-8; Limitations).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Conformal Privacy Auditing (CPA)?
CPA is a calibration framework that turns any re-identification attacker (e.g., TF-IDF retrieval or an LLM-based linkage pipeline) into a statistically certified output: for each released document it emits a set of candidate identities guaranteed to contain the true identity with a user-chosen probability (e.g., 95%), as long as the attacker configuration and data distribution declared at calibration time hold. The size of that set is the certified leakage measure.
Does CPA protect text from re-identification?
No. CPA is an auditing tool, not a protection mechanism: it measures how much ambiguity an attacker of a declared capability would retain. If the real attacker is stronger than the declared configuration, or the candidate pool differs, the certificate no longer applies — the paper's own drift tests show coverage dropping to 0.835 in that case.
Can CPA audit proprietary LLM APIs that do not expose logits?
Yes. The sampling-only interface repeatedly queries the model with forced-choice decoding over the candidate pool (budget m per document), builds empirical frequencies, and calibrates on those. The paper proves this preserves the coverage guarantee (Lemma 5.1, Theorem 5.2) and shows larger sampling budgets shrink sets (median 120→78 at m=10) without invalidating them.
What data and attackers were used in the experiments?
Four public benchmarks with known ground-truth identities: TAB (1,268 court records), TextWash (1,202 anonymized person descriptions in famous/semi-famous/fictional categories), WikiBio (1,000 biographies vs 1,000 profiles), and the Blog Authorship corpus (author profiles built from 1, 3, or 10 held-out posts). Attackers were TF-IDF retrieval plus LLM clue/attribute extraction using Llama-3.1-8B-Instruct, Qwen3-8B, and GPT-5.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 are not analyzable from the extracted text; α-sweep conclusions rely only on the numbers quoted in prose.
- Some Table 5 cells are absent in the extraction (GPT-5/famous, Llama/semifamous); extraction loss versus paper omission cannot be distinguished.
- Review covers arXiv v1 text only; referenced supplementary jsonl artifacts and code could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.