Privacy Washing: Detecting Internal Contradictions in Privacy Policies
cs.CY · cs.CL
Quick verdict
Defines 'privacy washing' via an LLM pipeline flagging contradictions in 12–37% of policies; rigorously self-critical, but zero human validation limits trust.
Summary
The paper defines 'privacy washing' — a privacy policy whose reassuring commitments are undermined by practices documented elsewhere in the same document — and operationalizes it with a four-stage pipeline: three-LLM consensus statement extraction, metadata compatibility filtering plus NLI screening, majority-vote verification by a three-model judge panel, and thematic analysis. Applied to 123 policies collected in 2026 (OPPT) and 115 from 2015 (OPP-115), it confirms contradictions in 12.2% of OPPT companies (9.8% excluding legacy pairs) and 36.5% of OPP-115 companies, with third-party sharing contradictions the modal pattern in both primary runs. A stability re-run seven months later, using entirely different models (judges from providers absent from both corpora) and matched filters, reproduces OPPT prevalence (13.0% vs 12.2%), shows sub-threshold pairs confirm at comparable rates (raising prevalence to 20.3% and 40.9%), and demonstrates the third-party majority is panel-sensitive while category-pair recurrence is not. The paper is unusually candid: no verdict is validated against human expert judgment, so precision is unknown and prevalences are lower bounds; the primary runs used different filter configurations, so the cross-corpus gap is not an era effect; and 48% of OPPT confirmations rest on outvoting the most conservative judge. Artifacts are fully released, but until expert validation lands this is a candidate-generation tool, not a measurement instrument.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Privacy washing — within-policy commitment-versus-practice contradiction — is operationalizable as a reproducible four-stage pipeline with three-LLM consensus extraction and majority-vote judge verification.
- Panel-confirmed contradictions appear in 12.2% of OPPT companies (2026; 9.8% excluding legacy pairs) and 36.5% of OPP-115 companies (2015); these figures are lower bounds under aggressive filtering with unknown precision.
- The same small set of first-party and third-party category pairs recurs across the 11-year corpus gap, though the third-party-sharing majority specifically is panel-sensitive.
- A fully separated stability re-run reproduces OPPT prevalence under the original protocol (13.0% vs 12.2%) and shows pairs below the similarity threshold confirm at rates of the same order, raising full-corpus prevalence to 20.3% (OPPT) and 40.9% (OPP-115).
- Until blind expert validation is performed, the pipeline is a candidate-generation tool for expert scrutiny (e.g., under the FTC net-impression doctrine), not a measurement or enforcement instrument.
Strengths
Exemplary self-audit transparency
The paper documents its own failures in detail: the silent OPP-115 annotation-join failure (all 4,975 statements typed by extraction models alone, Section 4.2), the legacy-pair indictment of the qualifier-coverage filter (26.7% vs 9.5% judge-confirmation, Sections 4/9.2), threshold circularity (40.6% of confirmations in the 0.50–0.55 band, Section 4.3.2), and exploratory unadjusted statistics (Section 6.3). This lets readers weigh claims correctly.
Pre-analyzed stability re-run with fully separated configuration
Section 8 re-runs the pipeline with a new extraction panel and judges from providers absent from both corpora (DeepSeek-V4-Flash, GLM-5.3-Flash, Kimi-K3), matched filters, and no similarity threshold; predictions are stated in advance and largely borne out (OPPT prevalence 13.0% vs 12.2%; legacy-pair findings fail 'as the primary analysis predicted'). A reclassification audit corrects both runs (OPP-115 stability: 155→140 confirmations, 42.6%→40.9%).
Full artifact release enabling community validation
Honest negative and null results
Weaknesses
No human expert validation of any verdict
Recall evidence is alarming and never scaled up
Headline numbers are aggregation-criterion dependent
Known filter defect left un-ablated before publication
Single author with declared commercial interest
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper coins and computationally operationalizes **privacy washing**: within one privacy policy, a commitment (promise, reassurance, or stated limitation) is undermined by a practice documented elsewhere in the same document. Three contributions are claimed: an operational definition pinned to an explicit judge prompt (Appendix A), a four-stage detection pipeline, and a cross-corpus measurement over OPPT (123 companies, collected 2026) and OPP-115 (115 companies, 2015). Headline results: panel-confirmed contradictions in **12.2%** of OPPT companies (15/123; 9.8% excluding 30 'legacy pairs') and **36.5%** of OPP-115 companies (42/115); third-party-sharing contradictions are the modal pattern in both primary runs (18/32 and 55/79 confirmed cases, Sections 6.2–6.3). A stability re-run (August 2026) with a fully separated configuration reproduces OPPT prevalence under the original protocol (13.0% vs 12.2%), reveals that pairs below the 0.5 similarity threshold confirm at rates of the same order (full-corpus prevalence 20.3% and 40.9% after reclassification audit), and shows the third-party majority is panel-sensitive while category-pair recurrence is not. The author is explicit that precision against human judgment is unknown and positions the pipeline as candidate generation, 'not a measurement instrument'.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'privacy washing'?
As defined in this paper, privacy washing is when a privacy policy contains a reassuring commitment (e.g., 'we do not sell your personal information') that is undermined by a data practice described elsewhere in the same document. It is a pragmatic contradiction: both statements may be individually defensible, but together they create a misleading overall impression, analogous to greenwashing. The authors stress the term does not imply intentional deception.
How many companies have contradictory privacy policies according to this study?
An LLM judge panel confirmed at least one contradiction for 12.2% of 123 companies in a 2026 corpus (9.8% excluding a legacy-pair subset) and 36.5% of 115 companies in a 2015 corpus. These are unvalidated LLM findings with unknown precision and are likely undercounts: pairs below a similarity threshold were never judged in the primary runs, and removing that threshold raises the figures to about 20% and 41%.
How is this different from PolicyLint or PoliGraph?
Prior tools like PolicyLint and PoliGraph detect logical contradictions — direct negations within structured data-collection tuples ('we collect X' vs 'we do not collect X'). This paper targets pragmatic contradictions: a broad reassurance whose impression is contradicted by a specific disclosed practice, a pattern that tuple-negation methods cannot express and that requires distinguishing commitments from practices and pairing them directionally.
Can I trust the specific companies named in the paper?
Treat the named findings as unverified candidates, not established facts. No human expert has validated any verdict, companies were not notified before publication, and the author explicitly states the tool is for prioritizing expert review rather than enforcement. Every pair quotes or traces to the company's own public policy, and all intermediate outputs and judge verdicts are released on GitHub so anyone can check them.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with several marked sampling gaps; Sections 5, 7, 10, most of 6.1–6.2, part of 8.1–8.2, and Appendix B fall in gaps, so assessments of those parts rely on cross-references in visible text.
- Tables 1, 2, 6–8 and all figures are not visible in the excerpt (likely within gaps); numbers cited here come from the abstract, Sections 3, 4.2–4.3, 6.3–6.5, 8.3, 9, 11–13, and Tables 3–5 and 9.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.