Do Vision Model See Like the Brain? A Comparison Across EEG Encoding Model
Shashank Baghel, Kshitij Dwivedi, Dinesh Singh, Sanjeev Nara
cs.CV · cs.AI · cs.HC
Quick verdict
Claims transformers sustain deep-layer EEG fit where CNNs fade; well-controlled design but no significance tests; mechanism claim clashes with MoCo-v1 data.
Summary
The paper asks whether vision transformers match human vision like CNNs do, and at which network depth the two families diverge. Using the public THINGS-EEG2 dataset (10 participants; 16,540 training and 200 test images; 17 posterior EEG channels; −200 to 800 ms), the authors fit layer-wise encoding models — polynomial-kernel PCA to 3,000 components followed by OLS regression — for six CNNs (AlexNet, VGG-16, ResNet-50, CORnet-S, MoCo-v1, CLIP-RN50) and two vision transformers (CLIP-ViT-B/32, DINOv2-ViT-B/14), scoring Pearson correlation between predicted and held-out EEG at every layer or block. CNNs peak early (AlexNet maxpool2, r=0.654) and weaken with depth; transformers sustain late correspondence only when the CLS token plus all patch tokens are preserved (r=0.640 and 0.656), while pooled or CLS-only readouts stall near r≈0.48. Whole-model scores still rank AlexNet first (r=0.677). MoCo-v1 and ResNet-50 are nearly identical (0.673 vs 0.670), so training objective appears to matter less than architecture and readout structure. Limitations: the headline divergence lacks significance tests and multiple-comparison control; no supervised-classification ViT is tested; token comparisons confound information content with input dimensionality; and no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CNNs correspond best to EEG at their earliest layers (peaks r=0.636–0.654 within 60–150 ms, Table 1) and decline toward deeper layers, while transformers sustain deep-block correspondence into the late (>200 ms) post-stimulus response.
- The transformer advantage depends on token readout: CLS+all-patches representations reach r=0.640 (CLIP-ViT-B/32) and 0.656 (DINOv2-ViT-B/14) versus ≈0.48 for CLS-only or pooled token representations.
- Controlled comparisons indicate architecture and representation structure, not training objective, drive the divergence: MoCo-v1 and ResNet-50 are nearly identical (0.673 vs 0.670), and CLIP-ViT converges toward CLIP-RN50 only once patch tokens are preserved.
- All models share an occipital-dominant channel topography, so architecture-family differences reflect signal strength and temporal persistence rather than distinct brain regions.
Strengths
Matched layer-wise pipeline across 8 models
One KernelPCA+OLS pipeline, fit per subject, applied to every layer of all models (§3.3, Figs 3–8), using EEG's millisecond resolution to ask when rather than whether the architecture families diverge — a gap §2.4 correctly identifies relative to fMRI-based comparisons (Oota 2022; Conwell 2024).
Genuine two-factor controls
MoCo-v1 vs ResNet-50 isolates training objective under a matched ResNet-50 backbone (0.673 vs 0.670, Table 2); CLIP-RN50 vs CLIP-ViT-B/32 isolates architecture under a matched contrastive objective; CORnet-S's cortical layer labels add a reference whose IT layer still collapses (§4.1).
Token-readout analysis is the standout result
Methodological transparency
Self-limiting topography interpretation
Weaknesses
No inferential statistics behind headline claims
Incomplete architecture×objective factorial
Readout dimensionality confound
Mechanistic attribution contradicts own control
Metric and reporting ambiguities
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks where, along network depth, CNN and vision-transformer representations diverge in correspondence with human EEG, using one matched pipeline on the public THINGS-EEG2 dataset (10 participants; 16,540 training images ×4 repetitions; 200 test images ×80 repetitions; 17 posterior channels; −200 to 800 ms at 10 ms resolution). Eight frozen models are compared: six CNN/backbone models (AlexNet, VGG-16, ResNet-50, CORnet-S, MoCo-v1, CLIP-RN50) and two ViTs (CLIP-ViT-B/32, DINOv2-ViT-B/14). Central claims: (1) CNNs peak early (Table 1: AlexNet maxpool2 r=0.654; range 0.636–0.654) and decline with depth, while transformers sustain late (>200 ms) correspondence in deep blocks; (2) this advantage depends on readout — CLS+all-patches reaches r=0.640/0.656 versus ≈0.48 for CLS-only or pooled tokens; (3) controlled comparisons (MoCo-v1 vs ResNet-50: 0.673 vs 0.670; CLIP-RN50 vs CLIP-ViT) indicate architecture and readout structure, not training objective, drive the divergence; (4) all models share an occipital-dominant topography (O1/Oz/O2), so family differences concern strength and duration, not region.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What dataset was used and how large is it?
The public THINGS-EEG2 dataset: EEG recordings from 10 participants viewing everyday object images — 16,540 training images (4 presentations each) and 200 test images (80 presentations each). This study analyzes 17 posterior channels over the −200 to 800 ms window around stimulus onset, at 10 ms resolution.
Do transformers beat CNNs at predicting the brain?
Not on peak accuracy: AlexNet tops the whole-model ranking (r=0.677) and the best transformer configuration (DINOv2 keeping all patch tokens) reaches r=0.656. The transformer advantage is specifically that its deepest blocks keep predicting the late (>200 ms) brain response where CNN layers fade — and only when all patch tokens, not just the CLS token, are fed to the encoding model.
Why do CLS-token transformer representations perform so poorly (r≈0.48)?
A CLS token (or pooled patch tokens) collapses the spatial layout that CNN feature maps still contain. The paper shows that concatenating the CLS token with all individual patch tokens raises performance to r≈0.64–0.66, so the deficit reflects the readout choice rather than missing information inside the transformer.
Is code or data available for this study?
The dataset is public (THINGS-EEG2, Gifford et al. 2022) and the pipeline is fully specified (Kernel PCA to 3,000 components plus OLS regression). However, the paper text contains no code-release statement, so exact reproduction currently requires reimplementation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are not analyzable from the extracted text; all curve and heatmap values cited here come from tables or in-text numbers.
- The abstract is truncated mid-sentence in the provided text, and the extraction may omit sections or supplementary material.
- No data/code availability statement is visible in the extraction; the dataset's public status is inherited from Gifford et al. (2022).
- Feature dimensionalities (e.g., 768-d blocks, ~38,400-d all-patches) are inferred from standard public model architectures, not stated in the paper.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.