Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis
Minsik Choi, Geewook Kim, Young Geun Kim
cs.LG
Quick verdict
Sink Strength, a cheap base-LLM scalar, predicts post-VL text loss (Spearman ρ=0.97, n=6) via attention-sink corruption; promising but correlational.
Summary
The paper studies why converting a pretrained LLM into a vision-language model (VLM) erodes text capability, showing the damage concentrates on format-sensitive tasks graded by strict output parsers (IFEval, EQ-Bench, GSM8K-CoT, GPQA-Diamond-CoT). The authors attribute this to attention-sink corruption: VL fine-tuning perturbs the early sink position that anchors attention, and a backbone's ability to preserve its sink predicts how much capability survives. They introduce Sink Strength S, a scalar measured on the base LLM alone via ~15 inference-only forward passes (~8 seconds on one GPU), defined as the median per-head log-odds of attention at the sink position over the last ~10 layers. Across six VLM-LLM pairs, S tracks post-VL IFEval loss with Spearman ρ=0.971 (leave-one-pair-out error 2.54 pt vs 4.40 for a constant baseline) and generalizes across four tasks (ρ≥0.82) and a 17-pair panel (ρ=0.88). A perturbation bound shows per-head QK-RMSNorm removes input-magnitude sensitivity from sink corruption while layerwise normalization does not, matching Molmo2-O's -18.7-pt collapse. Negative controls show post-pretraining QK-RMSNorm injection and seven weight-merging settings fail to prevent or recover the loss. Limitations: small sample (n=6 headline), S nearly co-clusters with QK-norm architecture, comparisons use released reference LLMs rather than true pre-VL checkpoints, and no causal disabling experiment is performed.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- VL fine-tuning erodes text capability concentrated on format-sensitive tasks, with double-digit drops on no-QK-norm backbones (e.g., Qwen2.5-VL: -9.6 IFEval, -14.3 GSM8K-CoT) while MMLU/BoolQ stay within ±2.3 pt.
- Sink Strength S, computed on the base LLM alone in seconds, ranks post-VL degradation (Spearman ρ=0.971 on IFEval across six pairs; ρ≥0.82 on all four format-sensitive tasks; ρ=0.88 on a 17-pair panel).
- Per-head QK-RMSNorm removes raw-input-magnitude sensitivity from the sink-gap perturbation bound (Theorem 1), while layerwise normalization compresses weak heads (Remark 1), explaining Molmo2-O's -18.7-pt IFEval collapse despite a healthy aggregate sink.
- Post-pretraining QK-RMSNorm injection and seven off-the-shelf weight-merging settings fail to prevent or recover the loss, narrowing interventions toward head-selective training-time protection.
Strengths
Genuinely cheap, pre-training diagnostic
S is inference-only on the base LLM (~8 s per 7B on one A6000, §3.2), robust to calibration size N∈[10,100] with the margin sign stable across all 108 (pair,N,seed) cells (App. D.1), and requires no VL run — a practically useful screening tool.
Layered confound controls
Text-only endpoint matches the reference on IFEval (Δ 0.00 vs VL -9.6, Table 3); a matched 3B Tülu trajectory isolates modality (S falls +0.41→+0.20 under VL but holds +0.58–0.65 under text-only, Table 14); a norm-matched random-ΔW control shows direction matters (-61.4 vs -9.6 pt, App. D.4); within-vendor contrasts are explicitly framed as 'one alternative explanation ruled out', not proof.
Non-obvious per-head vs layerwise dissociation
Honest negative results and self-auditing
Weaknesses
Small, architecture-collinear sample
S adds little beyond the architecture flag in the headline panel
Reference-LLM identity confound
Correlational core; no disabling ablation
Off-headline magnitude prediction and mixed baselines
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a practical failure: adapting a pretrained LLM into a VLM degrades text capability, concentrated on format-sensitive tasks — IFEval, EQ-Bench, GSM8K-CoT, GPQA-Diamond-CoT — where strict parsers grade surface output. Across five headline pairs plus one control, IFEval gaps run from -1.8 (InternVL3.5) through -9.6 (Qwen2.5-VL) to -18.7 (Molmo2-O), while MMLU/BoolQ stay within ±2.3 pt (Table 6) and coding even improves (MBPP mean +7.3, Table 8). Three claims: (1) the loss is attention-sink corruption; (2) Sink Strength S, a base-LLM scalar, predicts it before any VL training; (3) per-head (not layerwise) QK-RMSNorm removes input-magnitude sensitivity from the corruption bound. Two negative controls — QK-norm injection and weight merging — close off the cheap-remedy space.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Sink Strength in this paper?
Sink Strength S is a single number measuring how strongly a base LLM's attention concentrates on its 'attention sink' — the early token position that absorbs a large share of attention probability. It is computed as the median, over the last ~10 layers and all heads, of the log-odds of attention mass at the sink position, using about 15 short calibration prompts and only inference forward passes on the base model (about 8 seconds for a 7B model on one GPU). Higher S predicted less text-capability loss after vision-language fine-tuning.
Why do VLMs lose text ability after multimodal fine-tuning?
According to this paper, vision-language fine-tuning perturbs the query/key projections in a way that collapses the per-head attention sink that stabilizes surface-token tracking. The damage shows up mostly on tasks graded by strict output-format parsers (IFEval, EQ-Bench, GSM8K-CoT, GPQA-Diamond-CoT), reaching double-digit drops (e.g., -9.6 IFEval for Qwen2.5-VL, -18.7 for Molmo2-O), while knowledge benchmarks like MMLU stay nearly intact.
Does QK-RMSNorm protect an LLM's text ability in VLMs?
Backbones with native per-head QK-RMSNorm (Qwen3, InternVL3.5) preserved their sinks and lost the least text capability (IFEval -1.8 to -5.4), while backbones without it lost more (-7.9 to -9.6) and layerwise QK-RMSNorm (Olmo-3/Molmo2-O) lost the most (-18.7). However, injecting QK-RMSNorm after pretraining failed badly (IFEval 11 vs 59.9), so the protection appears to require co-adaptation during pretraining, not mere presence of the module. The evidence is correlational — no disabling ablation was run.
Can model merging recover the text capability a VLM lost?
In this paper, no. Seven off-the-shelf weight-merging settings (task arithmetic, uniform averaging, a 95/5 mix, TSV, STAR, TIES, DARE) were applied to pull two VLMs' language backbones back toward their reference LLMs. Only the two mildest mixes avoided making things worse, and none recovered the lost IFEval performance.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Numeric labels embedded in Figures 2-5 were extracted as flat, partly garbled figure text; every figure-sourced value quoted in this review (e.g., IFEval deltas in Figure 2, S values in Figure 3) was cross-checked against Tables 1-14 and is consistent with them.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.