Multi-modal Knowledge Preserving Adapter for Embedding Backward Compatibility
Jaeseok Byun, Gukyeong Kwon, Han-Kai Hsu, Meher Gitika Karumuri, Zhikang Zhang, Hao Yang, Davide Modolo
cs.CV
Quick verdict
First adapter-only backward-compatible training for MLLM embeddings; broad, consistent gains over adapter baselines, but statistical rigor and external comparisons are thin.
Summary
The paper introduces MKP-Adapter, the first backward-compatible training (BCT) method for multi-modal embedding models that updates only a lightweight adapter rather than the backbone. Targeting the costly 'backfilling' of vector databases after model upgrades, it learns a mapping h from new (GME/UniME 7B MLLM) to old (CLIP ViT-L/14 or E5-V) embedding space using solely pre-extracted embeddings. Beyond an L2 projection loss, the authors add multi-level preservation regularizers - cycle-consistency reconstruction (point), pairwise cosine-distance matching (distance), and RKD-style triplet angle matching (angle) - plus focal re-weighting of hard samples. On MS-COCO, Flickr30K, Urban1K, MMEB (36 datasets), ViDoRe-V1, and three video benchmarks across three upgrade scenarios, MKP-Adapter consistently beats L2-, EC-style, and contrastive adapter baselines and typically surpasses the old model itself (e.g., CLIP-to-UniME MMEB average 49.56 vs 40.29 for the old model). A control using h(x_new) for both query and database stays within 0.2 points of the new model, supporting the preservation hypothesis. Training takes 1-2 hours on 8 A100s with 0.01% inference overhead. Limitations: all quantitative baselines are self-constructed, no significance testing (a single 5-seed std <0.2% check), mixed focal-loss ablation effects, and no stated code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MKP-Adapter is the first adapter-only backward-compatible training approach for MLLMs, requiring no backbone updates and no access to model weights.
- The multi-level preservation losses (point/distance/angle) with focal re-weighting prevent projection to the old space from sacrificing the new model's embedding quality.
- Projected new-query embeddings against an old-model database outperform the old model itself across image-text, MMEB, visual document, and video retrieval in three upgrade scenarios.
- The adapter is trained solely on pre-extracted embeddings in 1-2 hours on 8 A100 GPUs and adds only ~0.01% inference latency relative to the backbone forward pass.
Strengths
Practical adapter-only setting with strong efficiency story
Training operates purely on pre-extracted embeddings (Sec. 3.1), so no backbone forward/backward passes, no weight access (works for API models), and no regression risk to the frozen new model. Reported cost: 1-2 hours on 8 A100s for a 558k-pair dataset and +0.25 ms per batch (0.01% of a 7B backbone forward pass) at inference (Sec. 5.4, Sec. D).
Evaluation breadth across scenarios, tasks, and scales
Three upgrade scenarios (CLIP ViT-L/14 to GME-7B, CLIP to UniME-7B, E5-V to UniME-7B) evaluated on I2T/T2I retrieval (COCO, Flickr30K, Urban1K), MMEB (36 datasets, Precision@1), ViDoRe-V1 (nDCG@5), and MSR-VTT/MSVD/VATEX (Tab. 1-3), plus R@10 (Tab. S14) and a 40k-image full COCO-val large-scale retrieval test (Tab. S15).
Direct experimental test of the core preservation hypothesis
Thorough ablations and an honest XBT appendix
Weaknesses
All quantitative baselines are author-constructed adapters
Thin statistical reporting
Focal re-weighting benefit is inconsistent across configurations
Operational scope of BCT not discussed
No code, adapter weights, or embeddings released
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is backward compatible training (BCT) for embedding models?
BCT constrains a new embedding model so its vectors can be searched directly against a database built with the old model, avoiding a full re-embedding ('backfilling') of millions of stored vectors after every model upgrade. This paper's variant instead freezes both models and trains only a small adapter that projects new-model query embeddings into the old model's embedding space.
Do I need to re-index my vector database when upgrading to an MLLM-based embedder?
According to the paper, no: you can keep database vectors from the old model (e.g., CLIP) and pass new queries through the trained MKP-Adapter before search. Reported Recall@1 with projected queries typically matches or exceeds the old model's own performance across image-text, visual document, and video retrieval benchmarks, though the database must keep using the old model for newly added content.
How expensive is MKP-Adapter to train and run?
Training uses only pre-extracted embeddings (no backbone gradients): about 4-5 hours of one-time feature extraction plus 1-2 hours of adapter training on 8 A100 GPUs for a roughly 1M-sample dataset. At inference the 2-layer MLP adds about 0.25 ms per batch of 32, roughly 0.01% of a 7B MLLM forward pass.
Does MKP-Adapter work with proprietary or API-only embedding models?
In principle yes: because it operates purely on input and output embeddings, it never needs access to model weights, unlike LoRA-based compatibility methods that must modify the backbone. The paper motivates this setting but evaluates on open models (CLIP, E5-V, GME, UniME); the old model's embedding dimension must be known and query-time embedding access is required.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are not analyzable from the extracted text; conclusions about them rely on the corresponding tables, which are present and appear complete.
- Review is based on the extracted text of v1 including the full supplementary material; equations are flattened by extraction but their content is recoverable from the surrounding text.
- Per-modality batch composition ratios and some implementation details are deferred to supplementary sections whose extracted content is partial, which slightly limits reproducibility assessment.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.