The 2026 PNPL Competition: Word Classification and Efficient Cross-Subject Generalisation in LibriBrain100
Francesco Mantegna, Gereon Elvers, Dulhan Jayalath, Gilad Landau, Tasha Kim, Miran Özdogan, Luisa Kurth, Teyun Kwon, et al.
cs.LG
Quick verdict
Competition paper for 2026 PNPL word classification on LibriBrain100; strong benchmark design and infrastructure, though only early reference baselines exist.
Summary
This competition-description paper launches the 2026 PNPL competition on LibriBrain100, an expanded MEG dataset combining ~80 hours from one deeply recorded subject with ~40 minutes each from 32 additional subjects, plus 8 unreleased zero-shot holdout subjects. The task advances from phoneme to word classification: predict which of 50 words a participant hears from 306-channel MEG windows. Two tracks run concurrently: Deep (within-subject, Subject 0) and Broad (cross-subject, with fine-tuning budgets of ~40/~20/~10 minutes). The primary metric is top-10 balanced accuracy (20% chance), complemented by top-1 accuracy, a Moses-50 vocabulary for comparison with invasive BCIs, and open-vocabulary mutual information (OVMI). Organizer-trained baselines (Table 1): the d'Ascoli model reaches 73.23% BAcc@10 within-subject, while cross-subject MEG-XL reaches 42.96% against d'Ascoli's 33.13% — quantifying that cross-subject generalisation, not within-subject scale, is the field's bottleneck. Infrastructure is complete: pip-installable pnpl library, Hugging Face data, public code/checkpoints, Colab tutorials, and a two-partition holdout against overfitting. Limitations: the paper reports no competition outcomes yet (winners announced December 2026), no variance estimates for baselines, and the OVMI metric is defined only in an unpublished manuscript.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Word classification over a fixed 50-word vocabulary is a well-posed, reproducible intermediate benchmark between phoneme classification and open-vocabulary non-invasive brain-to-text.
- LibriBrain100's combination of ~80-hour within-subject depth (Subject 0) and 32-subject breadth uniquely supports both Deep and Broad tracks, including zero-shot cross-subject holdout evaluation.
- Reference baselines establish that within-subject decoding is strong (d'Ascoli model, 73.23% BAcc@10) while cross-subject generalisation remains the bottleneck (MEG-XL 42.96%, d'Ascoli 33.13%, chance 20%).
- A triple evaluation protocol (competition-vocabulary BAcc@10, Moses 50, OVMI under a SUBTLEX-UK prior) makes results comparable across studies with different target vocabularies.
- The ~10-minute fine-tuning regime of the Broad track falls within a clinically feasible calibration window.
Strengths
Two-track design targets the field's actual bottleneck
The Broad track (Section 1.4) evaluates cross-subject generalisation under tiered fine-tuning budgets (~40/~20/~10 min) plus 8 zero-shot holdout subjects (33–40) with no released training data — the first competition to explicitly target the regime clinical deployment requires, contrasting with within-subject-only invasive systems (Moses 2021 through Card 2024).
Triple evaluation protocol addresses a documented comparability failure
Section 1.2 argues custom K-most-frequent vocabularies make cross-study word-decoding scores incomparable; the response (Section 1.5, Appendix B) is a dataset-tailored 50-word set, the invasive-field Moses 50 (enabling direct comparison with Moses et al. 2021 / Willett et al. 2023), and OVMI with a SUBTLEX-UK prior — all three reported on both vocabularies in Table 1.
Complete, verifiable artifact chain
Careful, honest metric treatment
Anti-overfitting holdout architecture
Weaknesses
No community results exist yet
Cross-subject absolute performance is modest and unexplained
Zero statistical uncertainty on Table 1
Vocabulary self-tension
Key comparability machinery is unverifiable
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a competition-description paper (in the lineage of Landau et al., 2025) announcing the 2026 PNPL competition on word classification from MEG. Its claims are primarily about design rather than new empirical results: (1) word classification on a fixed 50-word vocabulary is the right curricular step between phoneme classification and open-vocabulary brain-to-text; (2) the expanded LibriBrain100 dataset (~80 h for Subject 0; ~40 min each for 32 subjects; 8 zero-shot holdout subjects 33–40) uniquely supports both within-subject depth and cross-subject breadth; (3) the Broad track is the first competition target of cross-subject generalisation, down to a clinically motivated ~10-minute calibration; (4) a triple evaluation protocol (competition-vocabulary BAcc@10, Moses 50, OVMI) enables fair cross-study comparison. The only quantitative results are organizer-trained reference baselines (Table 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 2026 PNPL competition about?
It is a machine-learning competition where participants classify which of 50 words a person hears from non-invasive MEG brain recordings. It runs 15 July to 15 October 2026 with two tracks: Deep (best within-subject accuracy on a subject with ~80 hours of data) and Broad (cross-subject generalisation with only ~40/20/10 minutes of per-subject data, plus zero-shot holdout subjects). Winners are announced 1 December 2026.
What is the LibriBrain100 dataset?
An expansion of the LibriBrain MEG dataset: ~80 hours of recordings from one deeply sampled subject plus ~40 minutes from each of 32 additional subjects listening to naturalistic speech (Sherlock Holmes stories, TIMIT, podcasts). Eight further subjects are reserved as a zero-shot holdout. Data are on Hugging Face in HDF5 and raw FIF formats, loadable via the pip-installable pnpl library.
What is BAcc@10 and what score is random chance?
Top-10 balanced accuracy: the balanced fraction of cases where the true word appears among a model's top-10 predictions over a 50-word vocabulary. Uniform random guessing yields 20% in expectation (10/50). The paper's reference models reach 73.23% within-subject (Deep track) and 42.96% cross-subject (Broad track, MEG-XL).
How well does cross-subject MEG word decoding currently work?
In the paper's baselines, the self-supervised MEG-XL model fine-tuned on minutes of per-subject data reaches 42.96% top-10 balanced accuracy (6.20% top-1) versus a 20% chance level — well above chance but far below the 73.23% achieved with ~80 hours of within-subject data, confirming cross-subject generalisation as the field's main bottleneck.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 can only be assessed via their captions; the visual content itself is not recoverable from text extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.