SomBench: Benchmark Dataset for Advancing Machine Learning in Lunar Science
Himanshu Patil, Gabby Nyirjesy, Rachel A. Slank, Vishal Gaur, Daniela Szwarcman, Paolo Fraccaro, Nikolaos Dionelis, Michael K. Barker, et al.
cs.CV · cs.LG
Quick verdict
SomBench claims the first unified, ML-ready 42.5 TB multimodal lunar dataset; design is careful and leakage-aware, but validation rests on two ImageNet baselines and one model-derived target.
Summary
SomBench is a benchmark dataset paper introducing a unified, ML-ready lunar corpus: 32 co-registered layers from ten instruments across four missions (LRO, Kaguya/SELENE, GRAIL, Lunar Prospector), spanning 1 m to 20 km/pixel in 90 Lunar Transverse Mercator zones plus two polar caps, totaling 42.5 TB. An image-anchored tiling pipeline (WAC-anchored 51.2 km tiles; NAC-anchored 512 m tiles) produces per-modality netCDF tiles with Parquet catalogs and zone-wise train/val/test splits designed to prevent spatial leakage. Four benchmarks cover crater detection (Robbins WAC tiles; hand-labeled NAC tiles with 57,611 annotations), irregular mare patch segmentation (130 tiles), and polar ice-prospectivity regression (162 patches). ImageNet-initialized ResNet-50 and SwinV2-B baselines (five seeds each) confirm learnability: WAC crater mAP 0.2420, NAC crater mAP 0.1552, IMP IoU 0.5555, ice-prospectivity R2 0.9808. Limitations: validation uses only two generic baselines with no self-supervised pretraining results despite the stated goal, the ice-prospectivity target is itself a model output, and several counts are internally inconsistent (e.g., 63,468 WAC images selected vs 54,080 processed per Table 4).
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No unified, co-registered, ML-ready lunar dataset previously existed, and SomBench fills this gap with 32 co-registered layers from ten instruments across four missions in a common spatial framework (42.5 TB).
- An image-anchored tiling pipeline with Lunar Transverse Mercator/Lunar Polar Stereographic zone-wise splits prevents spatial data leakage between pre-training and evaluation.
- Baseline experiments with ImageNet-initialized ResNet-50 and SwinV2-B confirm every released benchmark task is learnable from the distributed inputs.
- SomBench extends Moonstone by providing 32 channels and 42.5 TB — more than 150 times Moonstone's 281 GB — while spanning 1 m to 20 km/pixel.
Strengths
Leakage-safe, fully specified dataset engineering
The image-anchored pipeline (Sec. 2.4) specifies adaptive sliding windows, per-layer resolution snapping so all modalities share integer pixel counts, four-corner densified bounds transforms, 90% NaN gates with per-tile null fractions recorded in Parquet catalogs, and per-LTM/LPS zone splits (achieved 78.3/12.0/9.7% and 75.1/15.7/9.2%), which is a genuine safeguard against spatial leakage that most geospatial ML datasets omit.
Unusually candid quality accounting
Table 5 itemizes per-modality tile counts and average null percentages (e.g., MINIRF_CPR 41.39% null in the low-res track; polar-only layers present in <1% of tiles), and Section 5 explicitly instructs users to treat illumination geometry as a confounder and to deduplicate by ground footprint rather than tile count.
Four benchmarks spanning three science themes with complete task scaffolding
Built almost entirely on established community products
Weaknesses
Thin baseline suite relative to stated ambitions
Ice-prospectivity benchmark is nearly self-referential
Internal bookkeeping inconsistencies
Tile redundancy acknowledged but unquantified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SomBench (arXiv:2609.13277) is a dataset/benchmark paper positioning itself as the first unified, co-registered, ML-ready lunar dataset: 32 data layers from ten instruments on four missions (LRO, Kaguya/SELENE, GRAIL, Lunar Prospector) spanning 1 m to 20 km/pixel, covering ±82° latitude in 90 Lunar Transverse Mercator (LTM) zones plus two polar stereographic caps, totaling 42.5 TB. It explicitly benchmarks against Moonstone [18] (28 channels, 281 GB), claiming >150× the volume. Central claims: (i) no comparable unified resource existed; (ii) the image-anchored tiling pipeline with zone-wise splits is leakage-safe; (iii) ResNet-50 and SwinV2-B baselines confirm each benchmark task is learnable from the released inputs.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SomBench and what does it contain?
SomBench is a large, machine-learning-ready lunar science dataset described in arXiv:2609.13277. It co-registers 32 data layers from ten instruments on four missions (LRO, Kaguya/SELENE, GRAIL, Lunar Prospector) onto common lunar map grids (90 Lunar Transverse Mercator zones plus two polar stereographic caps), and packages four small task benchmarks (crater detection, irregular mare patch segmentation, polar ice-prospectivity regression) with fixed splits and ResNet-50/SwinV2-B baselines. The excerpt does not state a license.
How big is SomBench?
The pre-training corpus is 42.5 TB: 963,609 WAC-anchored tiles (51.2 km tiles at 100 m/pixel) and 1,000,113 NAC-anchored tiles (512 m tiles at 1 m/pixel). The four task benchmarks are much smaller: 1,000 WAC crater tiles, 408 hand-labeled NAC crater tiles, 130 irregular-mare-patch tiles, and 162 polar ice-prospectivity patches.
What results do the SomBench baselines report?
On the released benchmarks: WAC Robbins crater detection mAP (IoU 0.50:0.95) 0.2420 for SwinV2-B with full training data; NAC hand-labeled craters mAP 0.1552; irregular mare patch segmentation IoU 0.5555 (F1 0.7141); polar ice-prospectivity regression R2 0.9808 (RMSE 0.0377). All values are five-seed means from ResNet-50 and SwinV2-B baselines; the authors frame them as learnability checks, not state-of-the-art results.
Where can I download SomBench?
Per the paper, it is hosted on Hugging Face in the nasa-ibm-ai4science/sombench collection, distributed as compressed netCDF tiles indexed by Parquet catalogs, with a PyTorch Dataset class. Source data originate from NASA's Planetary Data System and affiliated archives.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with '[...sampling gap...]' breaks inside Methods (2.3–2.4), Section 3.3, and References; judgments relying on skipped content are avoided and none of the cited numbers come from unseen text.
- Table 5 extracted with apparent number-format corruption in the snapped-resolution column (e.g., '10003.92', '17066.67'); these are quoted verbatim, not treated as authors' errors beyond legibility.
- License, persistent identifiers (DOI), and explicit baseline-code release statements would fall in the gapped Data Records region and are not claimed either way.
- No network access: the Hugging Face repository and the counts/claims of Moonstone [18] and Coyan et al. [65] are reported as stated, not independently verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.