Ten Architectures, One Error: Shared Failure Modes in Hyperspectral Classification under Spatially Disjoint Evaluation
Ehsan Faghih, Fatemeh Ashrafi, Marguerite Moore, Zahra Saki
cs.CV · cs.LG
Quick verdict
Shows random splits inflate Salinas HSI accuracy: leakage-free re-eval of 10 architectures drops Macro-F1 0.147, shifts ranks up to 5 places. Solid; one scene.
Summary
The paper argues that random pixel-level splits inflate hyperspectral classification accuracy because spatially autocorrelated test pixels sit beside training pixels and patch-based models read across partition boundaries. On Salinas, 47.69% of randomly split test pixels are immediately adjacent to a training pixel. The authors introduce a leakage-free protocol: a class-aware contiguous partition with per-class buffers (Protocol B) contrasted with a size-matched random split (Protocol A); disjointness is measured (minimum Chebyshev distance 2 versus 1), and a pre-registered admissibility rule ties evaluation to receptive field, fixing radius 2 and a hash-verified common test subset of 39,191 pixels. Ten architectures spanning eight families receive identical budgets (90 Optuna trials, five seeds). At matched training size, mean Macro-F1 drops 0.147 (maximum 0.228, HybridSN); rankings shift up to five places (SpectralFormer falls from third to eighth; a tuned RBF-SVM places fourth). All ten models err on largely the same pixels (mean pairwise overlap 0.471; 1,156 pixels missed by every model on every seed), dominated by a one-way class 8-to-15 confusion between never-adjacent classes, attributed to spectral ambiguity. Main limitations: one scene, one frozen partition; the top two models run under forced 5x5 inputs; HyperSIGMA is excluded outright.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Random pixel splits on Salinas place 47.69% of test pixels within Chebyshev distance 1 of a training pixel, and removing this adjacency lowers mean Macro-F1 by 0.147 at matched training size (0.165 at full size).
- Architecture rankings do not survive leakage-free evaluation: shifts reach five places (SpectralFormer falls from third to eighth), and a tuned RBF-SVM ranks fourth, ahead of six of nine other models.
- All ten architectures misclassify largely the same pixels (mean pairwise overlap 0.471; 1,156 pixels wrong under every model on every seed), driven by a one-directional class 8→15 confusion between classes never within one patch, indicating spectral ambiguity rather than partition artifacts.
- Leakage-free splits impose a geometric receptive-field ceiling that can exclude architectures outright (HyperSIGMA, native radius 16 vs training-partition ceiling 12), so spatial separation and receptive field must be reported together.
Strengths
Measured, not assumed, disjointness
Sec 3.1 quantifies the split rather than asserting it: minimum Chebyshev distance 2 (Protocol B) vs 1 (Protocol A), 47.69% of Protocol A test pixels adjacent at k=1 vs 0.00% for B (Fig 1), and a label-free nearest-neighbor probe shows the 0.110 Macro-F1 protocol effect before any architecture is trained.
Pre-registered, model-aware admissibility
Sec 3.2 derives the patch-overlap criterion d >= 2r+1 and fixes the evaluation radius by a rule registered before training (>=50% test retention, >=30 px/class, >=50% training availability), yielding Reval=2 and a hash-verified 39,191-pixel common subset on which all ten models are scored on identical pixels.
Confounds separated by design
Statistical discipline rare in this literature
Original error-structure finding
Weaknesses
Single scene, single partition
Top-ranked models evaluated under constraint
Reimplementation fidelity risk
Spectral ambiguity claim is inferential
Equal budgets do not guarantee equal search spaces
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an evaluation-methodology study, not a new classifier. The authors claim that (1) random pixel splits on single-scene hyperspectral benchmarks inflate reported accuracy through spatial autocorrelation and through patch-based receptive fields that physically read across partition boundaries; (2) a leakage-free protocol tying spatial separation to receptive field changes benchmark conclusions — a 0.147 mean Macro-F1 drop at matched training size (0.165 at full size) with rank shifts up to five places; (3) all ten architectures misclassify largely the same pixels, pointing to spectral ambiguity (a one-way class 8→15 confusion) that none resolves; (4) leakage-free splits impose a geometric receptive-field ceiling that can exclude architectures outright (HyperSIGMA). Scope is honestly bounded to Salinas and one frozen partition (Sec 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is spatial leakage in hyperspectral image classification?
Because neighboring pixels of the same field are near-duplicates, randomly splitting pixels of one scene puts test pixels directly next to training pixels — 47.69% within one pixel on Salinas in this paper — and patch-based models even read pixels from the other partition through their receptive field. Reported accuracy then partly reflects spatial memorization rather than generalization.
How much does accuracy drop with a leakage-free split?
On Salinas, mean Macro-F1 across ten architectures falls by 0.147 when training size is matched (0.165 at full training size), with a maximum drop of 0.228 for HybridSN. Even a nearest-neighbor classifier on raw spectra, which uses no spatial information, drops 0.110.
Which model performs best on Salinas under leakage-free evaluation?
ConvNeXt2D leads at 0.848 Macro-F1, narrowly ahead of S2Mamba at 0.847, with ResNet1D third and a tuned RBF-SVM fourth; SpectralFormer falls from third under random splitting to eighth. Note the top two run under forced 5x5 inputs rather than their published larger windows.
Why could HyperSIGMA not be evaluated in this benchmark?
A leakage-free split of Salinas leaves a training partition that supports patches only up to radius 12 in principle (half its pixels only to radius 3), below HyperSIGMA's native radius of 16, and its published tokenization cannot accept the 5x5 window allowed by the evaluation radius. The authors describe this as a geometric limit of scene size, not a defect of the model.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels (Figs 1-3) are available only via their embedded captions; the visual plot content could not be independently verified.
- Supplementary material (radius-sweep curves, per-class tables, hashes) is referenced but not part of the provided text; claims resting on it could not be checked.
- Tables 1-2 were extracted as flattened text; cell values were cross-checked against the prose where possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.