RobustSeiz: An Open-Source Framework for Benchmarking the Robustness of EEG Seizure Detection Models
Mohammad Mohammadi, Alireza Zarei
cs.LG · eess.SP
Quick verdict
RobustSeiz: open-source robustness benchmark for EEG seizure detectors; sound protocol, but demo covers one model on one corpus with many shifts at baseline.
Summary
RobustSeiz is an open-source framework for stress-testing EEG seizure-detection models under controlled, clinically motivated perturbations before deployment. The authors standardize four public corpora (CHB-MIT, TUSZ, Siena, SeizeIT1) into BIDS-EEG (256 Hz, 10-20 common-average montage), then apply environment transforms (rereferencing, sampling-rate mismatch, channel dropout, ADC quantization, band-pass filtering, impedance noise, drift), noise transforms (AWGN, mains hum, impulses, notch), and white-box FGSM/PGD adversarial attacks, each swept over hyperparameter grids. A Dockerized, model-agnostic pipeline scores every run with SzCORE sample- and event-level metrics, Lead/Lag onset timing, and a Monte Carlo dropout agreement index, logging runs in a machine-readable registry. The demonstration uses SeizureTransformer on TUSZ only: the clean baseline yields event sensitivity 0.57, F1 0.62, and 16.7 false positives per 24 h; severe AWGN (0 dB) suppresses detections (sensitivity 0.36) rather than inflating alarms; adversarial decrease attacks collapse precision to 0.2 even at the smallest epsilon; O1/O2 or T7/T8 channel dropout reduces sensitivity to 0; and most environment transforms leave metrics at baseline. Main limitations: a single model on a single corpus, unexplained invariance to many transforms, and no statistical uncertainty attached to any robustness curve.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Held-out accuracy on clean data is insufficient for pre-deployment assessment of EEG seizure detectors; controlled, clinically motivated stress testing exposes failure modes (missed seizures, alarm flooding) that clean splits cannot.
- RobustSeiz standardizes four public corpora in BIDS-EEG and applies environment, noise, and adversarial transform grids outside detector adapters, scoring accuracy, Lead/Lag onset timing, and stratified MCD agreement under matched inference scope.
- On TUSZ with SeizureTransformer, severe AWGN suppresses detections (event sensitivity 0.36 at 0 dB) rather than inflating false alarms (1.5 FP/24h), adversarial decrease attacks collapse precision to about 0.2 even at epsilon=0.01, and O1/O2 or T7/T8 channel dropout reduces sensitivity to zero.
- Stratified MCD agreement shows false positives are stably wrong (mean phi 0.764) while reference-seizure samples average phi 0.152, quantifying decision instability per error type.
- The experiment registry, Model Card, and reproducibility checklist make robustness profiles machine-checkable and comparable across research groups.
Strengths
Clinically grounded, sweepable perturbation taxonomy
Table 2 defines environment, noise, and adversarial transforms each with a concrete clinical rationale (impedance-like low-frequency bands for poor electrode contact, notch/mains mismatch for filtering errors, ADC bit depth) and documents the hyperparameter grid, so each family yields a robustness curve rather than a single robust/not label.
Multi-dimensional reporting beyond accuracy
Each run reports SzCORE sample- and event-level sensitivity/precision/F1/FP-per-24h plus Lead/Lag onset timing on event-level true positives and a Monte Carlo dropout agreement index phi stratified by seizure/background/FP/FN (Table 3), adding timing and uncertainty axes that clean-accuracy protocols lack.
Honest interpretation of composition artifacts
Strong reproducibility infrastructure
Candid limitations section
Weaknesses
Demonstration is one model on one corpus
Most environment transforms are inert at displayed precision
No statistical treatment anywhere
Adversarial grid has no informative low-severity region
Baseline discrepancy understated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
RobustSeiz (arXiv 2609.04007, cs.LG/eess.SP) is a framework and benchmark paper. Its central claim is that held-out clean accuracy — the SzCORE standard it explicitly builds on (ref 27) — cannot establish deployment readiness for EEG seizure detectors, and that a standardized, reproducible stress-test protocol should complement it. Four claimed contributions: (1) BIDS-EEG standardization of CHB-MIT, TUSZ, Siena, and SeizeIT1 (256 Hz, 19-electrode common-average montage, with a documented CHB-MIT bipolar exception and zero-filling of missing channels); (2) a domain-guided taxonomy of environment, noise, and adversarial transforms with documented hyperparameter grids (Table 2); (3) three analysis stages — SzCORE accuracy, Lead/Lag onset timing, and stratified Monte Carlo dropout agreement phi — under full and research-subset operating modes; (4) a Dockerized GPU pipeline with a machine-readable experiment registry. The paper is candid that its validation is a demonstration: one detector (SeizureTransformer, ref 55) on one corpus (TUSZ).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RobustSeiz and what problem does it solve?
RobustSeiz is an open-source framework for stress-testing EEG seizure-detection models before deployment. Instead of only measuring accuracy on clean held-out data, it applies controlled, clinically motivated perturbations — hardware-like environment shifts, noise, and white-box adversarial attacks — to standardized BIDS-EEG datasets and reports how detection quality, seizure-onset timing, and predictive uncertainty degrade as each perturbation intensifies.
Which datasets and models does RobustSeiz support?
The release standardizes four public scalp-EEG corpora — CHB-MIT, TUSZ, Siena, and SeizeIT1 — into BIDS-EEG format (256 Hz, 10-20 montage). Any subject-independent detector can be plugged in as a Dockerized adapter without retraining. The published demonstration uses one model, SeizureTransformer, on TUSZ.
What did the RobustSeiz demonstration on TUSZ find?
With SeizureTransformer, the clean baseline was event sensitivity 0.57, F1 0.62, and 16.7 false positives per 24 hours. Severe white noise (SNR 0 dB) suppressed detections (sensitivity 0.36) rather than creating false alarms; adversarial decrease attacks collapsed precision to about 0.2 even at the smallest tested strength; dropping the occipital (O1/O2) or temporal (T7/T8) channel pairs reduced sensitivity to zero; and most environment transforms left metrics essentially unchanged.
Is RobustSeiz code and data available?
The paper states that the framework source code, Docker recipe, and experiment registry are available at github.com/iMohammad97/RobustSeiz. The four EEG corpora are public but must be obtained from their original distributors (PhysioNet, TUH, KU Leuven), some of which require registration or a data-use agreement.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Sampling gaps in the provided text skip part of the perturbation formalization, part of the Results/Discussion narrative around Tables 6-7, and the tail of Table 9; judgments about those portions rest on the surrounding text only.
- The claimed GitHub repository (code, registry contents) could not be inspected; availability statements are taken from the paper's Data availability section.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.