Towards Deployable Underwater Vessel Classification
Abishek Soti, Thura Pyae Sone, Naqib Ibnul, Htoo Htet Aung, Henry Zhong, Gregory Cohen, Ying Xu
cs.SD · cs.LG
Quick verdict
Careful, honest benchmark study: recording-level splits cut macro F1 from 0.99 (ShipsEar) to 0.72 (DeepShip); thin statistics and no code release limit impact.
Summary
This empirical study evaluates compact models for underwater vessel classification across two public benchmarks. On ShipsEar's provided 5-second split, the authors' two-layer STFT CNN reaches 0.9918 macro F1 and a pooled RBF-SVM 0.9883, with an ablation (Table 1) showing mean–std temporal pooling lifts both model families. Because the processed ShipsEar clips lack source-recording provenance, the authors argue these near-ceiling scores cannot demonstrate generalization to unseen recordings, and reframe ShipsEar as a diagnostic benchmark. They therefore evaluate on DeepShip with recordings partitioned before segmentation (26,933/3,266/3,457 segments). Here a 157K-parameter four-layer CNN attains a test macro F1 of 0.7226 (Mel), while validation-based selection favors BM (0.6766 validation, 0.6694 test); six representations span 0.6336–0.7226 test F1. An 11.17M-parameter ResNet18 yields worse validation F1 (0.6110) under the matched protocol, and a best-effort reproduction of Chen et al.'s reported 95.13% accuracy stalls at 62.13%. Five-fold grouped cross-validation for a Mel RBF-SVM (0.6200±0.0217 segment-level) confirms the generalization gap is not split-specific. Grad-CAM shows class-dependent spectral relevance. Main limitations: the headline test score is the best of six representation results, without confidence intervals or significance tests; published comparisons use incompatible protocols; no code release is stated; and the failed reproduction cannot isolate causes.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Near-ceiling ShipsEar results (0.9918 macro F1) cannot be interpreted as recording-independent generalization because source-recording provenance is unrecoverable from the processed distribution.
- Under recording-level partitioning, a 157K-parameter compact CNN reaches 0.7226 test macro F1 on DeepShip, with performance varying 0.6336–0.7226 across six acoustic representations.
- An 11.17M-parameter ResNet18 provides no validation improvement over the compact CNN (0.6110 vs 0.6519) under the matched DeepShip protocol.
- Temporal mean–std statistical pooling improves performance and enables small classifiers (ShipsEar RBF-SVM 0.9641→0.9883, 2-layer CNN 0.8987→0.9698).
- Grad-CAM reveals class-dependent spectral relevance, indicating the STFT representation retains class-discriminative structure used by the model.
Strengths
Methodologically honest reframing of ShipsEar
§1 and §3.4 explicitly downgrade the near-ceiling results (0.9918 macro F1 two-layer CNN, Table 2) to 'diagnostic benchmark' status because source-recording provenance cannot be reconstructed — a rare, explicit acknowledgment that the standard ShipsEar split likely leaks recordings across partitions.
Sound recording-level protocol on DeepShip
§2.2 assigns recordings to train/val/test before segmentation (26,933/3,266/3,457 segments from 609 recordings); §3.5 states the manifest has no cross-partition recording overlap, and a 5-fold grouped CV (0.6200±0.0217 segment-level, train–val gap 0.3196) shows the gap is structural rather than split-specific.
Disciplined model selection and transparent failure reporting
Useful pooling ablation and embedded proof-of-concept
Weaknesses
Test-set exposure at the representation level
Protocol-mismatched comparisons inside result tables
No code, manifest, or hyperparameter search details
Auditory-inspired narrative undersupported by results
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This ICNS-affiliated study addresses a real deployment question: can compact classifiers recognize vessel type from passive sonar without the inflated scores that leaky evaluation protocols produce? Five central claims: (1) ShipsEar's provided split (§2.2; 1778/445 pre-segmented 5 s clips) cannot be verified as recording-disjoint, so the 0.9918 macro F1 two-layer STFT CNN and 0.9883 pooled RBF-SVM (Table 2) are diagnostic only; (2) under recording-level partitioning on DeepShip (26,933/3,266/3,457 segments from 609 recordings), a 157K-parameter four-layer CNN reaches 0.7226 test macro F1; (3) ResNet18 (11.17M params, ~71× larger) is no better under the matched protocol (validation F1 0.6110 vs 0.6519, §3.7); (4) temporal mean–std pooling is the key enabler for small classifiers (Table 1: RBF-SVM 0.9641→0.9883, CNN 0.8987→0.9698); (5) Grad-CAM (§3.6) shows class-dependent spectral evidence. The claims are mostly well-calibrated to the evidence — unusually so for this subfield.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why does this paper report 99% accuracy on ShipsEar but only about 72% F1 on DeepShip?
ShipsEar's provided fixed split cannot be verified to keep segments from the same original recording within a single partition, so its near-ceiling scores likely overestimate generalization to unseen recordings. The authors therefore evaluate on DeepShip with recordings separated before segmentation, which yields more honest but substantially lower scores (best test macro F1 0.7226).
What is the best model in this paper and how big is it?
A compact four-layer CNN with roughly 157,000 parameters reached 0.7226 test macro F1 on DeepShip with Mel spectrogram inputs. An 11.17M-parameter ResNet18 was no better under the same protocol, achieving lower validation macro F1 (0.6110 vs 0.6519), supporting the claim that capacity alone does not help.
Did the auditory-inspired representations like CARFAC help?
Not clearly. On DeepShip, the CARFAC inner-hair-cell (IHC) representation gave the lowest test macro F1 (0.6336), and although the basilar-membrane (BM) variant had the best validation F1 (0.6766), it did not lead on test (0.6694). Conventional Mel and STFT representations performed best overall.
Is code or data available for this paper?
The paper does not state any code release. Both benchmark datasets (ShipsEar and DeepShip) are publicly available, and the protocols are described in the text, but the DeepShip recording-level manifest, trained models, and implementation details such as hyperparameter search spaces are not provided.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 (example acoustic representations; Grad-CAM frequency profiles) are described in text but not analyzable from the extraction, so visual claims are taken on trust.
- Hyperparameter search spaces, seeds, augmentation policies, and per-class test counts are absent from the text, limiting judgments about variance and class imbalance.
- Published comparisons in Tables 2–3 rest on the authors' citations; the cited papers' protocols were not independently verified.
- The abstract's phrase 'Conventional Neural Networks' appears to be a typo for 'Convolutional', suggesting light proofreading; no substantive content was judged affected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.