Fingerprinting Multimodal Large Language Models
Chao Huang, Meng Tong, Kejiang Chen
cs.CR · cs.AI
Quick verdict
Claims first MLLM fingerprinting study; AttnPrint results are strong, but DistillTrace's TPR@1%FPR of 0.33 limits real-world forensic use.
Summary
The paper presents what the authors describe as the first fingerprinting study for multimodal large language models (MLLMs), targeting two infringement scenarios: derivative models built by modifying open-source checkpoints, and students distilled from teacher models. AttnPrint (white-box) extracts text-to-visual attention distributions, applies a DFT low-pass filter retaining ρ=0.2 of frequency components, converts back via IDFT, and summarizes mean per-layer low-frequency attention energy into an L-dimensional fingerprint; Hungarian layer alignment plus Pearson correlation yields a similarity score. DistillTrace (black-box) tests whether a candidate teacher assigns higher confidence (average log-odds of token logprobs) to suspect outputs than to outputs from b reference models, using a one-sided Wilcoxon signed-rank test. Evaluation covers 154 model instances across 19 architectures against seven LLM fingerprinting baselines. AttnPrint reaches AUC 0.9937 and TPR@1%FPR 0.9636 overall, with AUC 0.997-1.000 across fine-tuning, PEFT, quantization, merging and pruning, and the lowest similarity on four of five shared-backbone cross-family pairs. DistillTrace achieves AUC 0.8507 and TPR@1%FPR 0.3281 (200 queries, four references), degrading to 0.70-0.79 AUC under system prompts, RAG, or sampling changes. Main limitations: modest distillation sensitivity at strict false-positive rates, undisclosed threshold calibration and pair construction, reliance on white-box access or task-relevant queries, and no code release stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- AttnPrint's low-frequency cross-modal attention-energy fingerprints detect derivative MLLMs with AUC 0.9937 and TPR@1%FPR 0.9636 across 154 instances (Table 1).
- Text-centric LLM fingerprinting baselines conflate MLLMs sharing the same language backbone, whereas AttnPrint separates four of five cross-family pairs (Table 3).
- AttnPrint remains robust to five parameter-altering modifications (FT, PEFT, QZ, MM, PR) with AUC 0.997-1.000 (Table 2).
- DistillTrace provides statistical evidence of distillation without watermarking, reaching AUC 0.8507 with 200 queries and four reference models, while all seven baselines are near-random (Table 4).
- This is the first model fingerprinting study for multimodal large language models.
Strengths
MLLM-specific fingerprint signal is validated on the right hard cases
Table 3 shows AttnPrint attains the lowest similarity on four of five cross-family pairs sharing a language backbone (e.g., 0.2190 for LLaVA-v1.6-Mistral-7B/Idefics2-8B), while PDF, TensorGuard, HuRef and the black-box baselines assign 0.71-0.999 to these independently trained models — directly demonstrating the shared-backbone confound the paper targets.
Reference-calibrated hypothesis testing is a principled distillation detector
DistillTrace (Section 4.2) calibrates teacher confidence against b reference models to control for sample difficulty, then applies a one-sided Wilcoxon signed-rank test with explicitly stated hypotheses (Eq. 21); all seven baselines are near-random on distilled models (AUC 0.4942-0.5135, Table 4) while DistillTrace reaches 0.8507.
Strong low-FPR operating performance for derivative detection
Evaluation breadth above area norms, with practical ablations
Honest scoping of assumptions
Weaknesses
DistillTrace is weak at the operating point that matters forensically
Decision-rule and evaluation-protocol opacity for AttnPrint
Fingerprint collapses to L scalars with unquantified input sensitivity
Statistical assumptions asserted but unchecked
No code release; key evidence lives in supplementary
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This MM '26 paper targets MLLM copyright protection under two threat models (Section 3): attackers modifying an open-source victim model (fine-tuning, PEFT, quantization, merging, pruning) to disguise a derivative, and attackers distilling a teacher (open or API-served) into a student. It proposes AttnPrint for white-box derivative detection and DistillTrace for black-box distillation attribution, claiming the first MLLM fingerprinting study, state-of-the-art results over seven LLM-era baselines, robustness to five parameter-altering modifications, and distillation evidence under three parameter-independent deployment changes.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AttnPrint and how does it identify derivative multimodal models?
AttnPrint is a white-box fingerprinting method for multimodal LLMs. It records how text tokens attend to visual tokens inside a suspect model, applies a Fourier-transform low-pass filter that keeps 20% of frequency components, and summarizes the average low-frequency attention energy per layer into a compact fingerprint. Fingerprints of two models are compared after aligning layers with the Hungarian algorithm, using Pearson correlation: high correlation indicates a derivative relationship.
How does DistillTrace detect knowledge distillation without access to model weights?
DistillTrace works only through APIs. It checks whether a candidate teacher model assigns higher confidence (the average log-odds of returned token log-probabilities) to outputs generated by the suspect model than to outputs from independent reference models on the same queries. A one-sided Wilcoxon signed-rank test over the per-query differences decides whether the suspect aligns with the teacher significantly better than independent models, providing statistical evidence of distillation.
How accurate is this MLLM fingerprinting method?
On 154 model instances across 19 architectures, AttnPrint reports AUC 0.9937 and a true-positive rate of 0.9636 at a 1% false-positive rate for detecting modified derivatives, with AUC 0.997-1.000 across fine-tuning, PEFT, quantization, merging, and pruning. Distillation detection is harder: DistillTrace reaches AUC 0.8507 but detects only 0.3281 of distilled models at a 1% false-positive rate, degrading further under system-prompt, RAG, or sampling changes.
Is code available for this paper?
The extracted paper text does not mention a public code release. Model IDs, pruning and distillation settings, and several ablations are deferred to supplementary material, so reproducing results requires re-implementing the attention-extraction and hypothesis-testing pipelines on public Hugging Face checkpoints.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Supplementary material (model IDs, pruning/distillation settings, the ρ ablation, per-attention-type analyses) is not included; several judgments rely on the main text's summaries of it.
- Figures 1-3 (framework, t-SNE, ablation curves) are only partially recoverable from extracted text; ablation values quoted come from axis labels embedded in the text.
- Number extraction may contain noise — Table 3's SEF entry '0.94901' is an apparent typo, so exact digits should be checked against the PDF.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.