What Makes Adversarial Examples Transfer Across Deepfake Detectors?
Rafael M. Mamede, Pedro C. Neto, Ana F. Sequeira
cs.CV · cs.AI · cs.CR · cs.LG
Quick verdict
Shows adversarial transfer between deepfake detectors hinges on source-target compatibility and is attack-dependent; large controlled study, credible, useful.
Summary
This paper reports a controlled factorial study of black-box adversarial transferability in deepfake detection. The authors train 60 detectors combining six backbones (ResNet34, Xception, EfficientNet-B4, DeiT-S, ViT-B/16, Swin-T), two pretraining regimes (ImageNet versus face recognition on BUPT-BalancedFace), and five DF40 training-data subsets built on FaceForensics++ (FS, FR, EFS, FE, ALL). Using a threshold-adjusted margin objective at epsilon=8/255, they perturb 2,000 manipulated test images with AutoAttack (APGD-CE, FAB, Square) and CW-EOT (100 iterations, 10 EOT samples), producing 240,000 adversarial images and 3,540 ordered source-target pairs per attack. Matched stratified contrasts with leave-one-detector-out jackknife inference and Holm correction show that sharing an exact backbone, architecture family, pretraining regime, or training data significantly increases transfer, and that the dominant factor depends on the attack: exact backbone under AA (22.34 pp), shared pretraining (21.25 pp) and training data (19.03 pp) under CW-EOT. Mean source-averaged ASR is only 7.21% (AA) and 19.52% (CW-EOT), whereas a strict multi-source oracle excluding same-backbone and superset-data surrogates reaches 64.48%, indicating that single-source evaluations understate vulnerability. Limitations include one training seed per configuration, a homogeneous author-trained detector pool with modest Celeb-DF generalization (mean 62.6% AUC), and code and data promised only after peer review.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Black-box adversarial transfer between deepfake detectors is significantly higher when source and target share an exact backbone, architecture family, pretraining regime, or training data (all four contrasts positive and Holm-significant under both attacks, Table 1).
- The dominant compatibility factor is attack-specific: exact backbone under AutoAttack (22.34 pp) versus shared pretraining (21.25 pp) and shared training data (19.03 pp) under CW-EOT, with all three between-attack differences significant.
- Single-source and source-averaged evaluations understate target vulnerability: mean source-averaged ASR is 7.21% (AA) and 19.52% (CW-EOT), while a strict multi-source oracle excluding same-backbone and superset-data surrogates reaches 64.48% mean ASR.
- No directional group asymmetry (e.g., Transformer-to-CNN versus reverse) survives Holm correction, so the transfer structure is source-target compatibility rather than a universal ordering between detector groups (Table 3).
Strengths
Factorial design that actually isolates factors
The 6x2x5 bank of 60 detectors (Section 3) yields 3,540 ordered black-box pairs per attack and lets backbone, architecture family, pretraining, and training data be varied independently — precisely the confounding the authors show limits Neekhara et al. (four DFDC pipelines) and the DUMB benchmark (five detectors), where architecture, preprocessing, and data changed jointly.
Statistical treatment well above field norm
Equally weighted stratified subclassification contrasts (Section 3.3), leave-one-detector-out jackknife SEs (G=60; G=48 for training-data contrasts), and Holm correction jointly across all 19 prespecified tests; between-attack comparisons are paired over common strata (Table 1), and Table 3 reports directional contrasts honestly as null after correction.
Attack-dependence is quantified, not just asserted
Oracle protocols expose a practically important gap
Unusually complete planned release
Weaknesses
Single training seed per configuration
Homogeneous, moderately weak detector pool
Transfer scored on untransformed images
Strict-oracle 'partial overlap' scenario is not realized in the pool
Nonstandard attack configuration limits comparability
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper studies black-box transfer of l-infinity-bounded adversarial examples (epsilon = 8/255) between deepfake detectors and asks which source-target relationships make transfer succeed. Its vehicle is a factorial detector bank: 60 detectors combining six backbones (CNNs: ResNet34, Xception, EfficientNet-B4; Transformers: DeiT-S, ViT-B/16, Swin-T), two pretraining regimes (ImageNet; face recognition on BUPT-BalancedFace with ElasticArcFace+), and five DF40 training subsets built on FaceForensics++ (FS, FR, EFS, FE, ALL). Two attacks — a custom AutoAttack and CW-EOT — perturb 2,000 manipulated test images, yielding 240,000 adversarial images and 3,540 ordered black-box source-target pairs per attack. The four headline claims: (1) sharing an exact backbone, architecture family, pretraining regime, or training data each significantly increases transfer (Table 1); (2) the dominant factor is attack-specific — exact backbone under AA (22.34 pp) versus shared pretraining (21.25 pp) and training data (19.03 pp) under CW-EOT; (3) single-source and source-averaged ASR (7.21% AA, 19.52% CW-EOT) understate vulnerability relative to a strict multi-source oracle (64.48%); (4) no directional group asymmetry survives Holm correction (Table 3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do adversarial attacks transfer between deepfake detectors?
Yes, but weakly on average. In this 60-detector study, attack success when averaging over non-matching surrogate sources is only 7.21% with AutoAttack and 19.52% with CW-EOT. Transfer rises sharply when the attacker's surrogate shares the target's backbone, pretraining, or training data, and a multi-source attacker drawing on 59 surrogates succeeds on 64.48% of images even after excluding close matches — so single-surrogate tests understate real vulnerability.
What is the strict oracle in this paper?
It is a best-case portfolio evaluation: for each target detector, the attacker may use perturbations generated by any of the other 59 surrogate detectors, except those sharing the target's exact backbone or trained on a superset of its data. Even under these restrictions the paper reports a 64.48% mean attack success rate, versus 7.21-19.52% for single averaged sources. It is an upper bound on what a well-equipped black-box attacker could achieve, not an expected success rate.
Which factor matters most for transferring adversarial deepfake examples?
It depends on the attack. With AutoAttack, sharing the exact backbone dominates (+22.34 percentage points); with CW-EOT, shared pretraining (+21.25 pp) and shared training data (+19.03 pp) dominate. All four tested compatibility factors — exact backbone, architecture family, pretraining regime, and training data — significantly increase transfer under both attacks, while direction (which group is source versus target) does not matter.
Is the code and dataset from this paper available?
The authors commit to releasing 240,000 adversarially perturbed images, complete pairwise transfer results, detector configurations, and evaluation code, but state this will happen only after the double-blind review concludes — so no artifact was verifiable at review time.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-6 (heatmaps and 60x60 transfer matrices) were extracted as captions only; visual cell values could not be inspected, so assessments of the figures rely on the prose descriptions in Sections 4.2-4.3 and Appendix E.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.