Deep learning from the crowd Fundamentals of morphological galaxy classification
Luis Enrique Sucar, Carlos del Burgo, Jonathan Serrano-Pérez
astro-ph.GA · cs.CV · cs.LG
Quick verdict
GZ1 study shows all-layer retraining lifts crowd-trained CNN EM ~10 pts; thorough ablations but no baselines, no error bars, and crowd-majority ground truth.
Summary
The paper investigates how to train deep CNNs for galaxy morphological classification from crowdsourced Galaxy Zoo 1 (GZ1) labels, systematically varying five factors: training scope (last layer vs all layers of EfficientNetV2-XL), annotator agreement thresholds (agr 1 to 0.3), hierarchical post-processing via a Bayesian network (BCNN), curriculum-style training over exclusive agreement ranges with transfer learning, and model ensembles. Experiments use up to ~395k training images. Key findings: retraining all layers improves exact match (EM) by roughly 10 points over last-layer training (the paper's claim; the gap ranges from ~9 to ~20 points depending on threshold); EM exceeds 99% for agreement of 0.6 or more on range-matched test sets (peak 99.75 at agr of 0.7 or more); the hierarchy adds little on GZ1's shallow tree except in low-data regimes of roughly 100-1000 images; curriculum learning offers small gains for limited data (EM 0.83 vs 0.80 on the full test set); a five-model ensemble reaches EM 91.04% on the full test set versus 90.91% for the best single model. The authors also propose two hierarchical agreement measures (WHA, NMDWA). Limitations: ground truth is itself the crowd majority vote; no comparison with prior galaxy-CNN results; training from scratch omitted; no error bars, seeds, or significance tests; no code release and hyperparameters largely unspecified. Generalization beyond GZ1's six-class tree is asserted rather than demonstrated.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Retraining all layers of EfficientNetV2-XL substantially outperforms last-layer-only training on GZ1 crowd data, with the paper citing roughly a 10-point exact-match improvement (e.g., 93.14 vs 84.39 EM at agr of 0.3 or more, Tables A.2/A.3).
- With annotator agreement of 0.6 or more, all-layer models exceed 99% exact match on range-matched test sets, peaking at 99.75 (Table A.3).
- Bayesian hierarchical post-processing (BCNN) yields negligible gains on GZ1's shallow hierarchy except when training data are scarce, at roughly 100-1000 images (Figure 10).
- Curriculum learning over exclusive agreement ranges slightly outperforms cumulative training for limited data, at 0.83 vs 0.80 EM on the full test set (Section 7).
- An ensemble of the five best models improves all measures on the full test set, reaching EM 91.04%, accuracy 93.64%, and hF 94.65% (Table 4).
Strengths
Genuinely comprehensive single-testbed ablation
Five factors (Last vs All layers, agreement thresholds 1 to 0.3, BCNN hierarchy, curriculum-by-range, ensembles) are varied in one controlled design on GZ1, with full metric tables for every cell of the matrix in Appendix A (Tables A.2, A.3, A.5, A.6, A.7, A.8).
Quantifies the label-quality vs data-quantity tradeoff
Figures 5-6 and Tables A.1-A.3 show EM rising above 99% for agr of 0.6 or more (peak 99.75 at agr of 0.7 or more) and collapsing to 93.14 at agr of 0.3 or more, giving survey pipelines a concrete quality gate.
Honest negative results and failure analysis
Novel hierarchical agreement formalization
Operational transparency on compute
Weaknesses
Circular ground truth
No statistical machinery
Reproducibility gaps
Hierarchy claims rest on a trivial tree and an unexercised method
Extreme class imbalance is unaddressed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This A&A manuscript (arXiv:2609.06316) asks how a CNN should be trained when labels come from the crowd, using Galaxy Zoo 1 as the testbed. Five factors are ablated: training scope (last layer vs all layers of EfficientNetV2-XL, 208M parameters), annotator agreement thresholds (agr = 1 down to 0.3), hierarchical post-processing via a Bayesian network (BCNN), curriculum-style transfer over exclusive agreement ranges, and ensembles. The central claims are a roughly 10-point exact-match (EM) gain for all-layer retraining, above-99% EM at high agreement, hierarchy benefits confined to low-data regimes, a small curriculum gain, and a small ensemble gain. Section 3.2 additionally proposes two hierarchical agreement measures, WHA and NMDWA (Eqs. 3-7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How accurate is this galaxy classification model?
On test sets restricted to the same agreement range as training, models trained on high-agreement data (annotator agreement of 0.6 or more) exceed 99% exact match, peaking at 99.75. On the full test set, which includes hard low-agreement images, the best single model reaches about 90.9% exact match and a five-model ensemble 91.04%, always measured against the crowd majority label rather than expert labels.
What is Galaxy Zoo 1 and how is it used in this paper?
Galaxy Zoo 1 is a citizen-science dataset in which roughly 100,000 volunteers classified about 668,000 SDSS galaxies into six classes (elliptical, clockwise spiral, anticlockwise spiral, edge-on/other, star or don't know, merger), averaging 38 votes per image. This paper filters training images by inter-annotator agreement and uses the majority vote both to train CNNs and to evaluate them.
Does exploiting the galaxy classification hierarchy improve results?
On GZ1's shallow three-level hierarchy, Bayesian post-processing made almost no difference when training data were plentiful, but it helped in low-data regimes of roughly 100-1000 images. The authors expect larger benefits on deeper schemes such as Galaxy Zoo 2, which they leave to future work.
Is the code or dataset from this paper available?
The manuscript contains no code or data release statement. Galaxy Zoo 1 itself is publicly available, but the exact training configuration (hyperparameters, splits, per-range settings) is only partially documented, so exact replication would require reimplementation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (Figs. 2-14, B.1, B.2) and the confusion matrices (Tables 1-2) are rendered as images in the PDF; their cell values were not extractable, so figure-level statements rely on the surrounding narrative text.
- No code/data availability statement appears in the provided text; absence of a release could not be cross-checked against the arXiv landing page or ancillary files.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.