Matched-Input Estimates Differ in Sign Across Architectures: Auditing EEG Foundation Models on Motor Imagery
cs.LG · q-bio.NC
Quick verdict
Leakage-controlled audit: two EEG foundation models trail all supervised decoders on BCI IV-2a; the headline sign-flip rests on non-significant n=9 estimates.
Summary
The paper audits two pretrained EEG foundation models, LaBraM-base and CBraMod, against three supervised motor-imagery decoders under a validation-locked protocol in which every tunable choice is made from training-session data only, enforced by a machine-checked provenance field that rejects artifacts whose held-out labels touched selection. On four-class BCI Competition IV-2a (nine subjects, cross-session), every supervised comparator outperforms every foundation-model configuration: ShallowConvNet reaches 0.6962 accuracy versus 0.4796 for validation-selected CBraMod fine-tuning, with paired Wilcoxon comparisons hitting the exact-test floor at n=9. A matched-input control retrains the supervised architectures on the broadband arrays the foundation models consume; point estimates change sign across architectures (broadband helps ATCNet by 0.078, hurts EEG Conformer by 0.088), though no term survives Benjamini-Hochberg correction, so the authors frame this descriptively rather than as a formal interaction. The four-class deficit does not reproduce on two-class BNCI2014-004, where fine-tuned CBraMod (0.8085) is competitive. Validation-fitted temperature scaling restores foundation-model ECE to the supervised range despite lower accuracy. Limitations include only two of four attempted foundation-model implementations succeeding, n=9 inference, single seeds for most configurations, and a three-electrode second dataset that confounds class count and montage.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under a validation-locked cross-session protocol on four-class BCI IV-2a, every supervised comparator outperforms every foundation-model configuration, with best FM fine-tuned CBraMod at 0.4796 versus 0.5305-0.6962 for supervised arms.
- Matched-input accuracy differences change sign across the three supervised architectures (broadband helps ATCNet by 0.078, hurts EEG Conformer by 0.088), though no term is significant after BH correction.
- The four-class foundation-model deficit does not reproduce on two-class BNCI2014-004, where fine-tuned CBraMod reaches 0.8085 and is competitive.
- A single validation-fitted temperature returns foundation-model ECE (0.103-0.124) to the supervised range (0.101-0.143) despite substantially lower four-class accuracy.
Strengths
Structurally enforced leakage guard
Section III-A: every artifact row carries an eval_labels_used_for_selection field; the table-building code refuses rows where it is missing or true, and probability files are rechecked before calibration metrics. The guarantee lives in the artifact format, not author discipline — stronger than typical BCI comparisons.
Honest statistical self-limitation
Section V-C and Discussion explicitly decline a formal architecture-by-input interaction because none of the three matched-input terms survives Benjamini-Hochberg correction (pBH = 0.073, 0.304, 0.176 at n=9); the paper also distinguishes non-detection from equality and states the exact Wilcoxon floor (0.0039).
Redundant negative controls
Reproducibility infrastructure
Practically useful fine-tuning finding
Weaknesses
Title overstates statistical status of the headline
Narrow foundation-model coverage
Matched-input term is confounded and unreseeded
Second dataset cannot isolate the cause
Standard baselines absent
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper audits two pretrained EEG foundation models (LaBraM-base 5.8M, CBraMod 5.0M) against three supervised motor-imagery decoders (ShallowConvNet, ATCNet, EEG Conformer) under a validation-locked, cross-session protocol on BCI Competition IV-2a, with a replication attempt on BNCI2014-004. Five claims are advanced: (i) every supervised comparator beats every foundation-model configuration on four-class motor imagery; (ii) matched-input estimates of the input-pipeline effect change sign across comparator architectures; (iii) the four-class deficit does not reproduce on the two-class dataset; (iv) random-initialization controls separate architecture difficulty from pretrained-weight value; (v) one validation-fitted temperature restores foundation-model calibration error to the supervised range. The authors are unusually disciplined about what they do not claim: the sign-flip headline is explicitly descriptive because no matched-input term survives multiple-comparison correction.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do EEG foundation models beat supervised models in this study?
No. On four-class BCI Competition IV-2a under a cross-session protocol, every supervised decoder tested (ShallowConvNet 0.6962, ATCNet 0.6562, EEG Conformer 0.5305) outperformed every foundation-model configuration, including fine-tuned CBraMod (0.4796). On two-class BNCI2014-004 the gap was not detected; fine-tuned CBraMod reached 0.8085 and was competitive.
What is a matched-input estimate in this paper?
The authors retrain supervised decoders on the same broadband 0.1-75 Hz, 200 Hz inputs the foundation models consume, then subtract that accuracy from their narrowband 8-30 Hz, 250 Hz baseline. The term bundles filtering, sampling rate, notching, scaling and validation-selected retuning, so it is not an isolated preprocessing effect.
Is the sign-flip finding statistically significant?
No. None of the three matched-input differences survives Benjamini-Hochberg correction at n=9 subjects (pBH 0.073, 0.304, 0.176), so the authors present the sign variation as a descriptive caution against single-architecture controls rather than a confirmed architecture-by-pipeline interaction.
Is code or data available for this paper?
The paper states that code, per-subject result artifacts, per-trial probability files, pinned library versions and a SHA256 manifest accompany the study, and that every table is regenerated by scripts rather than transcribed. No public repository URL is visible in the extracted text, so availability should be verified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is assessable only through its caption; no other figures appear in the extracted text.
- No repository URL appears in the extracted text; code availability is as claimed by the authors, not independently verified.
- The metadata abstract is truncated mid-sentence, but the embedded full text appears complete through the reference list.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.