Infra-Bench CLS: A Global, Open-Source Benchmark for Critical Infrastructure Classification with Earth Observation Foundation Models
Justin Guthrie, Edward Oughton, Konrad Wessels, Matthew Rice, Isaac Corley
cs.CV · cs.LG
Quick verdict
Claims first global FM benchmark for critical infrastructure; rigorous and open, but weak OSM labels and protocol mismatches qualify rankings.
Summary
The paper introduces Infra-Bench CLS, a benchmark of 18,756 Sentinel-1 and Sentinel-2 tiles (600 m, 10 m resolution) covering facility-scale critical infrastructure across seven continents, labeled via OpenStreetMap tags for 13 classes in four sectors (energy, water, transportation, telecom); results are reported for 10 classes after excluding three with insufficient test support or inconsistent visual signatures. Seven foundation models (SatlasPretrain S1/S2, CROMA, Prithvi-EO-2.0, AlphaEarth Foundations, OlmoEarth v1.1-Base, DINOv3 ViT-L/16) are evaluated with linear probing and fine-tuning at 1.0× and 0.3× training data against supervised and random-feature ResNet-18 baselines, using spatially blocked splits (200 km) and three seeds. DINOv3 fine-tuned leads at 0.579 macro F1 versus 0.392 for the supervised baseline; at 0.3× data it retains 0.525, exceeding the baseline trained on full data, supporting the label-efficiency claim. Airports, train stations, and data centers score highest (0.78–0.85), while substation subtypes and other power classes lag (0.27–0.46). Key limitations, mostly self-acknowledged: OSM weak labels with only one regional validation showing severe substation label ambiguity; per-model input mismatches (normalization, bands, temporal collapsing) that confound cross-model rankings; possible memorization of the evaluated classes in SatlasPretrain/OlmoEarth pretraining; and an architecture-size confound against the 11M-parameter baseline.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Fine-tuned foundation models outperform a fully trained ResNet-18 on facility-scale CI classification: best macro F1 0.579 (DINOv3 ViT-L/16) versus 0.392, a 48% relative improvement (Table S1).
- Foundation models retain value under label scarcity: four fine-tuned FMs at 30% of training labels exceed the from-scratch baseline trained on 100% (DINOv3 0.525, SatlasPretrain S2 0.499, Prithvi 0.481, OlmoEarth 0.444 vs 0.392).
- Linear probing alone never matches supervised training from scratch for any evaluated FM (all LP below 0.392 macro F1), so pretraining value is unlocked only by fine-tuning.
- Visual distinctiveness at 10 m, not FM choice, is the binding constraint for hard classes: substation subtypes reach at most 0.388 (transmission) and 0.330 (distribution), attributed jointly to resolution limits and OSM label ambiguity.
Strengths
First global, cross-sector facility-scale CI benchmark, fully open
18,756 Sentinel-1/S2 tiles over seven continents, 13 classes in four sectors; Zenodo DOIs for data (ODbL, 10.5281/zenodo.22118892) and code (MIT, 10.5281/zenodo.22256967), fixed seed 42, complete hyperparameters in Tables S3–S4, even wall-clock times in S5–S6.
Spatially blocked splits guard leakage
200 km × 200 km blocks in Equal Earth (EPSG:8857), region-stratified 70/15/15; 46.5% of tiles reassigned versus random splitting (Section 2.2) — a leakage guard most EO benchmark evaluations skip.
Clean label-efficiency ablation design
Empirical label-noise characterization, rare in benchmarks
Candid, specific limitations section
Weaknesses
Input-protocol inequity undermines cross-model rankings
Label-noise ceiling never disentangled from model skill
Baseline comparison conflates pretraining with capacity
Statistical uncertainty under-reported for close rankings
Absolute utility undemonstrated; ontology coverage partial
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Infra-Bench CLS is presented as the first global, cross-sector benchmark for classifying facility-scale critical infrastructure (CI) with Earth-observation foundation models. The dataset spans 18,756 Sentinel-1 (VV/VH) and Sentinel-2 L2A tiles (600 m × 600 m at 10 m) centered on OpenStreetMap-tagged assets in four sectors and 13 classes, with a 1,000-asset cap per region×sector and OSM class proportions preserved; 10 classes are actually evaluated (wind farm and port terminal have n=3 test tiles; water works lacks a consistent 10 m signature). Four headline claims are made: (1) fine-tuned FMs beat a fully trained ResNet-18 (0.392 macro F1), led by DINOv3 ViT-L/16 at 0.579 (+48% relative); (2) FMs are label-efficient — DINOv3 at 0.3× labels (0.525) beats from-scratch at 1.0×; (3) linear probing never matches supervision for any FM; (4) performance is sector-structured (transportation > telecom > water > energy), with typed substation classes capped at 0.388/0.330. For the most part, the claims match the tabulated evidence.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Infra-Bench CLS?
A public benchmark of 18,756 Sentinel-1 and Sentinel-2 satellite image tiles (600 m at 10 m resolution) centered on facility-scale critical infrastructure assets worldwide — substations, power plants, airports, train stations, data centers, and more — labeled from OpenStreetMap tags across 13 classes in four sectors (energy, water, transportation, telecom). Data and code are released on Zenodo, and the benchmark tests how well Earth-observation foundation models classify such assets.
Which foundation model performed best in the benchmark?
DINOv3 ViT-L/16 after full fine-tuning, reaching 0.579 macro F1 across the 10 reported classes. It beat every EO-specialized model (best: SatlasPretrain S2 at 0.559) and far exceeded a from-scratch ResNet-18 baseline at 0.392. Notably, DINOv3 was pretrained on natural web images, not satellite data.
Do foundation models need fewer labels than training from scratch?
According to the paper, yes: fine-tuned on just 30% of the training labels, DINOv3 still scored 0.525 macro F1, and three other fine-tuned models also beat the from-scratch baseline trained on 100% of labels (0.392). However, linear probing on frozen features never matched the supervised baseline for any model, so fine-tuning is where the value appears.
Why do substation and power-sector classes score so poorly?
Two compounding reasons: substation subtypes look nearly identical at 10 m resolution, and the OpenStreetMap weak labels are noisy — in a 376-asset manual check in Maine/New Hampshire, only 42% of substations had unambiguous transmission/distribution designations. Best F1 on typed substation classes was 0.388 (transmission) and 0.330 (distribution), and the paper argues higher-resolution imagery or authoritative labels are needed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure images (Figures 1–9, S1–S10) are not present in the extracted text; confusion-matrix patterns and training curves could only be assessed through the numeric tables (S1, S8–S9), not the plots themselves.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.