Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs
Xiaofu Chen, Stella Frank, Yova Kementchedjhieva
cs.CV · cs.AI
Quick verdict
Claims canonical color stays linearly decodable in vision encoders after grayscale removal; VLM post-training reshapes it. Controls solid, dataset small.
Summary
This paper asks whether vision encoders encode canonical object color — the color tied to an object category rather than visible in the image — as linearly accessible knowledge. The authors assemble a 708-class dataset with ten basic color labels (extending Visual-CounterFact with Wikidata P462 entries) and probe frozen encoders — CLIP, SigLIP, DINOv2, ViT-MAE, Swin-V2 — with L2-regularized logistic regression, removing visible chroma via grayscale conversion plus per-image histogram equalization. Best-layer grayscale accuracy reaches roughly 40–44% against a 13.98% majority baseline, versus about 76–82% under RGB; Visual-CounterFact validation indicates the residual signal tracks canonical rather than counterfactual surface color. DINOv2 accuracy rises monotonically from Small to giant scale, and color decodability correlates with object recognition across layers (Spearman 0.60–0.90). Extending to VLMs, matched pre-/post-VLM vision towers show post-training can sharply reduce tower decodability — object recognition drops 38.8–57.4 points for CLIP→Molmo and CLIP→mPLUG-Owl — while information reappears at the visual interface and in decoder visual tokens; PaliGemma largely preserves its SigLIP tower. Limitations include roughly five images per class, Wikidata labels biased toward color-diagnostic objects, acknowledged residual texture/luminance confounds, five-fold statistics without multiple-comparison correction, and no stated code or data release.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Canonical color remains linearly decodable from histogram-equalized grayscale images across five vision encoders, at 40.2–43.8% best-layer accuracy versus a 13.98% majority baseline (Table 4).
- Visual-CounterFact validation shows the residual grayscale signal aligns with canonical object color rather than the actual (counterfactual) surface color (Sec 4.2).
- Color decodability is associated with, but not fully determined by, object-class recognition: layer-wise Spearman correlations run 0.601–0.895 and predicted-object-conditional color accuracy reaches 55–65% (Figs 3–4).
- VLM post-training can reduce vision-tower decodability (object recognition −38.8pp for Molmo, −57.4pp for mPLUG-Owl) while the information reappears at the visual interface and in decoder visual-token states, consistent with reformatting rather than removal (Table 5, Figs 5–6); PaliGemma's tower is largely preserved.
- Canonical-color decodability increases monotonically with DINOv2 scale from Small to giant (Table 1, ρ=1.00).
Strengths
Validated input-side controls
Grayscale + per-image histogram equalization is validated rather than assumed: on Visual-CounterFact the actual-color probe collapses to its majority baseline while canonical-color probes on both original and recolored images stay above baseline with matching layer-wise trends (Sec 4.2, Fig 1).
Leakage-aware probing protocol
Object-class-level 5-fold cross-validation with class-level probability averaging (Sec 4.1, A.3) tests generalization to unseen concepts, not merely unseen images of known classes — the right question for category-level canonical color.
Object-conditional color metric
Breadth across supervision signals and scale
Full-stack localization with honest statistics
Weaknesses
Thin, structurally biased dataset
Conceded residual object-mediated confound
Confounded VLM pair comparisons
Fragile fold-level statistics
Curated asset not released
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether frozen vision encoders encode canonical color — category-typical color knowledge (bananas→yellow) rather than pixels in a given image — as linearly accessible information, and whether VLM post-training relocates this information across the vision–language stack. Five claims anchor the work: (1) canonical color stays linearly decodable from histogram-equalized grayscale inputs for all five encoders, best-layer accuracy 40.2–43.8% against a 13.98% majority baseline (Table 4) versus roughly 76–82% under RGB; (2) Visual-CounterFact validation shows the residual signal tracks canonical, not counterfactual surface color (Sec 4.2, Fig 1); (3) color decodability is associated with but not fully determined by object-class recognition — layer-wise Spearman 0.601–0.895 (Fig 3) and predicted-object-conditional color accuracy of 55–65% (Fig 4); (4) matched pre-/post-VLM towers show post-training can slash tower decodability (object recognition −38.8pp Molmo, −57.4pp mPLUG-Owl, Table 5) while information reappears at the visual interface (Fig 6) and in decoder visual-token readouts (Fig 5); (5) decodability rises monotonically with DINOv2 scale (Table 1, ρ=1.00). Claims are consistently hedged to 'linear accessibility', which is appropriate.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do vision encoders know that bananas are yellow even in black-and-white images?
This paper trains linear probes on frozen vision encoders and finds canonical object color stays decodable from histogram-equalized grayscale images at roughly 40–44% accuracy (10 colors, 13.98% majority baseline) across CLIP, SigLIP, DINOv2, ViT-MAE and Swin-V2. Visual-CounterFact checks show the signal tracks canonical rather than visible surface color, supporting the claim that encoders carry category-level color knowledge in a linearly accessible form.
What is canonical color and how is it different from actual color?
Canonical color is the typical color of an object category, stored as conceptual knowledge (bananas are yellow), whereas actual color is the surface color in a specific image (a blue strawberry is actually blue). The paper separates the two by converting images to grayscale plus histogram equalization and by probing recolored counterfactual images where the actual surface color is largely unrecoverable but the canonical label remains decodable.
What happens to visual information inside a VLM after post-training?
Probing matched pre-/post-VLM vision towers, the authors find CLIP-based VLMs (Molmo, mPLUG-Owl) lose much linearly decodable object and color information in the tower — object recognition drops 38.8 and 57.4 points respectively — but the information reappears at the visual interface (e.g., the Q-Former) and in decoder visual-token states, a pattern the authors call reformatting rather than removal. PaliGemma's SigLIP tower is largely preserved.
Is the paper's dataset or code publicly available?
The paper describes dataset construction in detail (708 object classes, ten canonical colors, Wikidata P462 labels, about five images per class, Pokémon capped at 50) and lists Hugging Face repositories for all models used, but does not state a public release of its own dataset or probing code in the provided text.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels are extracted as scattered axis numbers and labels, so layer-wise curve values are taken only from Appendix Tables 4–5 and in-text figures; exact per-layer values for Figures 1–6 could not be read from the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.