A Comparative Evaluation of Pre-trained Convolutional Neural Networks for Melanoma Detection
Wagner Moreno Schmitz, Marco Antonio de Castro Barbosa, Thiago Magalhães Amaral, Dalcimar Casanova, Jefferson Tales Oliva
cs.CV · cs.AI
Quick verdict
Frozen CNN triplet embeddings + kNN for melanoma on 3 datasets; hypothesis-tested, yet the histopathology arm is confounded by precomputed embeddings.
Summary
This paper benchmarks five ImageNet-pretrained CNN backbones (ResNet50, MobileNet, VGG16, VGG19, InceptionV3) for binary melanoma-versus-benign classification on two dermoscopic datasets (HAM10000, ISIC 2018) and one histopathological dataset (CR-AI4SkIN). Backbones stay frozen; a 256→128 projection head is trained with triplet loss, and a k=5 k-NN classifies the L2-normalized embeddings. Dermoscopic sets are augmented to balance classes (3,000/class on HAM10000; 1,800/class on ISIC 2018) before a 70/15/15 split; CR-AI4SkIN uses its official 150/23/43 patient split. Accuracies span 71% (InceptionV3, ISIC 2018) to 84% (ResNet50, HAM10000), and 72% (VGG19) to 83% (ResNet50) on CR-AI4SkIN. Friedman tests are significant on all three datasets; Nemenyi post-hoc finds no significant pairs on HAM10000, only ResNet50×InceptionV3 on ISIC 2018 (p=0.017), and all pairs significant on CR-AI4SkIN. The authors conclude ResNet50 is the most stable architecture and that modality shifts accentuate architectural differences. Main limitations: the histopathology arm reuses dataset-provided precomputed embeddings rather than the paper's own pipeline, confounding the architecture comparison; augmentation precedes splitting with no patient-level separation for dermoscopic data, risking leakage; the statistical block structure is never specified; no fine-tuned baseline contextualizes the modest 71–84% accuracies; and no code is released.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under an identical frozen-backbone triplet-learning plus k-NN protocol, ResNet50 attains the highest accuracy on all three datasets (83.8% HAM10000, 76.8% ISIC 2018, 83.0% CR-AI4SkIN).
- Friedman tests detect significant accuracy differences among the five architectures on all three datasets (p=0.0077 to <0.0001).
- Nemenyi post-hoc tests show full pairwise separation only on the histopathological CR-AI4SkIN dataset, versus no significant pairs on HAM10000 and a single pair (ResNet50×InceptionV3, p=0.017) on ISIC 2018.
- Architecture rankings observed on dermoscopic images do not transfer to histopathological images, so model selection must jointly consider imaging modality and multiple datasets.
Strengths
Unified, fully documented training protocol for the dermoscopic arm
Section 3.4 and Table 2 specify every hyperparameter (frozen backbones, Dropout 0.5 → Dense-256 ReLU → BN → Dropout 0.3 → Dense-128, triplet margin 0.5, Adam 1e-4, batch 8, 30 epochs, early stopping 15, kNN k=5), directly addressing the protocol inconsistency the authors criticize in their own Table 1 review.
Formal significance testing with honest null reporting
Friedman and Nemenyi tests (Tables 5–8) are rare in this literature, and the authors transparently report that no pair is significant on HAM10000 (lowest p=0.487) rather than hiding the null behind the significant global test.
Patient-level separation for the histopathology dataset
Extensive, structured related-work baseline
Transparent disclosure of the CR-AI4SkIN shortcut
Weaknesses
CR-AI4SkIN comparison is confounded by foreign embeddings
Leakage risk from augmentation before splitting
Statistical tests are under-specified
No fine-tuned baseline to contextualize the accuracy ceiling
Conclusion overstates ResNet50's superiority
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv:2609.11550, cs.CV) asks which of five ImageNet-pretrained CNN backbones — ResNet50, MobileNet, VGG16, VGG19, InceptionV3 — best supports binary melanoma-vs-benign classification, and whether architecture rankings survive an imaging-modality change. Three public datasets are used: HAM10000 and ISIC 2018 Task 3 (dermoscopic) and CR-AI4SkIN (histopathological). The stated differentiator (Section 1) is an identical training protocol across all models, motivated by the inconsistent protocols criticized in the authors' own 27-study review (Table 1). Three contributions are claimed: a controlled cross-modality comparison, an analysis of modality-shift effects, and formal Friedman/Nemenyi significance testing.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which CNN performed best for melanoma detection in this study?
ResNet50 achieved the numerically highest accuracy on all three datasets — 83.8% on HAM10000, 76.8% on ISIC 2018, and 83.0% on CR-AI4SkIN. However, on HAM10000 no pairwise difference reached significance in the Nemenyi post-hoc test, and on ISIC 2018 only ResNet50 vs InceptionV3 was significant, so the statistical case for ResNet50 rests mainly on the histopathological dataset.
What accuracy did the models reach on skin cancer classification?
Binary melanoma-versus-benign accuracies ranged from 71% (InceptionV3 on ISIC 2018) to 84% (ResNet50 on HAM10000) for dermoscopic images, and from 72% (VGG19) to 83% (ResNet50) on the histopathological CR-AI4SkIN dataset. These are below the 95–99% often reported in the literature because the backbones were frozen and used as triplet-loss embedding extractors rather than fine-tuned end-to-end classifiers.
What datasets were used in this melanoma CNN comparison?
Three public datasets: HAM10000 and the ISIC 2018 Challenge Task 3 training set (both dermoscopic, 10,015 images each, reorganized into melanoma vs non-melanoma), and CR-AI4SkIN (histopathological patches from 215 patients with expert ground-truth labels). Download links are given in the paper's data availability section.
Is code available for this paper?
No. The paper states that all three datasets are publicly available and provides their URLs, but it does not reference any released code, training scripts, seeds, or split files, so exact reproduction depends on the hyperparameter table alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with two marked sampling gaps: parts of Table 3 (VGG19 and InceptionV3 cells beyond the 81.5% HAM10000 entry), Table 4 (per-class precision/recall), and segments of Sections 2–4 were skipped; conclusions drawing on those parts rely on the abstract and visible tables only.
- Figures 1–5 appear in the excerpt only as captions; their actual content could not be verified, so the architecture-figure descriptions were not assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.