iMINDBench: iEEG Multi-Institution Neural Decoding Benchmark
Geeling Chau, Saba Hashemi, Yonghyeon Gwon, Eshani Patel, Jan DeWitt, Christopher Wang, Andrii Zahorodnii, Sabera J Talukder, et al.
cs.LG
Quick verdict
Solid multi-institution iEEG decoding benchmark; preprocessing controls and scaling negatives are valuable, but headline rankings lack uncertainty estimates.
Summary
The paper introduces iMINDBench, a benchmark for intracranial EEG (iEEG) decoding that evaluates models on fifteen language, auditory, and visual tasks across three naturalistic movie-watching datasets from different institutions (Neuroprobe/Brain Treebank, BYD, Pippi). It standardizes two preprocessing tracks (Multi-STFT and Waveform), fixed two-fold within-session splits, and unit decodability subsets (Main/Challenge) defined by screening-decoder validation ROC-AUC above 0.60. On Main units, pretrained systems beat non-pretrained baselines within their track (PopT-v2 0.645 overall vs 0.631 for the best non-pretrained Multi-STFT model; BaRISTA 0.636 and DIVER-1 0.634 vs HTNet 0.614 on Waveform), yet Multi-STFT baselines remain competitive with pretrained waveform models, especially on smaller datasets. Preprocessing ablations show global versus per-window normalization shifts ROC-AUC by more than 0.10 in some settings, and omitting a 0.5 Hz high-pass filter costs 0.023. A scaling study using PopT-v2 finds within-session doublings gain ~0.025 AUC, while 3-25x more cross-subject or cross-institution data yields gains of at most 0.014 or even negative changes. Limitations: the scaling conclusion rests on a single model, only binary tasks are reported, pretraining corpora overlap a benchmark dataset, and headline tables lack uncertainty estimates.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Evaluated pretrained systems outperform non-pretrained baselines within their preprocessing tracks, but strong Multi-STFT spectral baselines remain competitive with pretrained waveform systems across institutional datasets (Table 1).
- Preprocessing choices such as normalization strategy and high-pass filtering shift baseline ROC-AUC by up to more than 0.10, so model contributions must be assessed under matched preprocessing (Figure 3).
- Within-session data scaling yields roughly 0.025 ROC-AUC per doubling, while 3-25x more supervised data from other subjects or institutions produces only small or task-dependent gains (Figure 4).
- Frozen pretrained representations need not beat direct spectral decoders: BrainBERT with a linear head (0.619) underperforms single-STFT MLP baselines (0.634) under the benchmark protocol (Appendix D).
Strengths
Fills a genuine evaluation gap
First resource to jointly control task, institution, and preprocessing for naturalistic iEEG decoding: existing multi-institution benchmarks (Omni-iEEG, Mayo/FNUSA) target epileptology, while naturalistic resources (Neuroprobe, Brain Treebank, AJILE12) are single-dataset (Section 2).
The preprocessing-confound argument is quantified, not just asserted
Global vs sample-wise normalization improves ROC-AUC by >0.10 in some settings; omitting the 0.5 Hz high-pass drops HTNet from 0.612 to 0.589 consistently across all three datasets; optimal single-STFT parameters vary by dataset and task (Figure 3a-c), empirically justifying the Multi-STFT flagship track.
Informative negative scaling result with task-level decomposition
Decodability subsets separate headroom from saturation
Unusually thorough reproducibility documentation
Weaknesses
The scaling headline rests on a single model
No uncertainty quantification on headline comparisons
Pretraining-exposure confound in the 'pretrained beats baselines' claim
Cross-dataset label construction and signal handling differ
Narrow evaluation scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is iMINDBench?
iMINDBench is a public benchmark for evaluating intracranial EEG (iEEG) neural decoding models. It standardizes fifteen language, auditory, and visual decoding tasks on 1-second neural windows across three movie-watching datasets collected at different institutions (Neuroprobe/Brain Treebank, BYD, and Pippi), with fixed evaluation splits and two standardized preprocessing pipelines (a spectral Multi-STFT track and a waveform track), so that model improvements can be compared fairly across institutions rather than reflecting preprocessing differences. A leaderboard is hosted at imindbench.github.io.
What are the main findings of the iMINDBench paper?
Three findings: (1) pretrained models (PopT-v2, BaRISTA, DIVER-1) beat simple from-scratch baselines when using the same preprocessing, but a strong spectral baseline (logistic regression or MLP on Multi-STFT features) remains competitive with pretrained waveform models, especially on smaller datasets; (2) preprocessing choices such as normalization and high-pass filtering can change ROC-AUC by more than 0.10, so model comparisons need matched preprocessing; (3) adding up to 25x more training data from other subjects or hospitals improved target-session decoding by at most about 0.014 ROC-AUC, meaning current models do not effectively exploit heterogeneous multi-institution data.
Which models were evaluated on iMINDBench?
From-scratch decoders (logistic regression, MLP, CNN, and HTNet) plus pretrained systems: PopT-v2 on the spectral (Multi-STFT) track, and BaRISTA and DIVER-1 on the waveform track. BrainBERT was additionally evaluated through a model-native single-STFT route in an appendix, where its frozen representations underperformed direct spectral decoders. MVPFormer and Brant were excluded pending validated adaptation of their longer input representations to the benchmark's 1-second decoding windows.
Is the iMINDBench code and data available?
The paper states that evaluation code, preprocessing configurations, and baseline implementations will be released in a public repository, that dataset preparation pipelines will be available through the TorchBrain/Brainsets integration (Neuro-Galaxy ecosystem), and that submissions will be handled via pull request to a leaderboard repository at imindbench.github.io with automated validation. The paper also specifies exact preprocessing parameters, split construction, and compute costs. Note that at the time of the preprint these releases were announced rather than independently verified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as captions in the extracted text; figure-level claims (scaling curves, screening-score distributions, electrode coverage maps) could not be visually verified.
- Code and leaderboard availability is promised in the future tense ('will be released') and could not be verified offline.
- Unit counts per table cell are not stated in the main text (appendix figure labels only), limiting assessment of variance behind the headline means.
- Appendix K's AutoResearch results use single-STFT PopT-v2 variants explicitly not comparable to the main Multi-STFT tables, so they were treated as a feasibility demonstration rather than a benchmark entry.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.