Exposing Blind Spots in Deep Imbalanced Regression Evaluation
Noah C. Puetz, Jens U. Brandt, Marc Hilbert, Elena Raponi, Elena Raponi, Thomas Bäck, Thomas Bartz-Beielstein
cs.LG · cs.AI
Quick verdict
Proposes bMASE + MuViS-DIR benchmark to close DIR evaluation gaps; multi-seed evidence solid, but single backbone and no significance tests weaken rankings.
Summary
The paper argues that Deep Imbalanced Regression (DIR) evaluation suffers from three blind spots: overreliance on image benchmarks, a many-/medium-/few-shot reporting protocol that is diagnostic but not decision-complete, and neglect of seed-level stability in tail regions. As remedies, the authors evaluate DIR on MuViS-DIR, nine multimodal time-series virtual-sensing regression tasks across six physical domains, and introduce bMASE, a scale-normalized balanced metric built on macro-averaged per-bin MAE with Freedman-Diaconis binning, aggregated across datasets via geometric mean. Six representative DIR methods (LDS, SQInv, Focal-L1, ConR, RnC, UVote) plus a vanilla ResNet1D baseline are trained for ten seeds each (630 runs). UVote achieves the best aggregate bMASE (0.1732 vs. 0.2163 vanilla), LDS is the strongest orthogonal method (0.1997), and RnC degrades performance (0.3314). Appendix D shows the balanced metrics are distribution-invariant (CV <= 0.005, Kendall tau = 1.00) where MAE, MSE, R2, GM, and SERA all drift. The stability analysis finds Spearman correlations of -0.57 to -0.95 between bin sample count and per-bin seed variability. Limitations include a single backbone architecture, hyperparameter selection at one seed on the aligned validation metric, test-set-dependent binning, absence of paired significance tests, and a deliberately selective method comparison.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- DIR evaluation is concentrated on image benchmarks: 18 of 19 surveyed methods evaluate primarily on AgeDB-DIR or IMDB-WIKI-DIR, and the three time-series evaluations share no datasets or public pipelines (Appendix C).
- The many-/medium-/few-shot protocol is diagnostic but not decision-complete; the proposed bMASE provides a single scale-invariant ranking criterion that is empirically distribution-invariant (Appendix D: CV <= 0.005, Kendall tau = 1.00).
- Existing DIR methods transfer unevenly to multimodal time-series: UVote improves GMean bMASE from 0.2163 to 0.1732, LDS is the best orthogonal method (0.1997), and RnC degrades performance (0.3314).
- The tail regions central to DIR are the least seed-stable: Spearman correlations between bin sample count and per-bin MAE standard deviation range from -0.57 to -0.95, and improved mean performance does not remove the trend (Section 4.3).
Strengths
The data blind spot is quantified, not anecdotal
Appendix C (Table 4) systematically maps 19 DIR methods to datasets, backbones, and code availability: 18 of 19 anchor on AgeDB-DIR or IMDB-WIKI-DIR; the three time-series usages (SHHS-DIR, TUAB, ECG-K-DIR) share no datasets and none released a training pipeline, with two arguably not continuous-target problems (21 and 96 discrete values).
Metric design is validated by controlled invariance experiments
Appendix D.1 sweeps four deterministic synthetic predictors over Gaussian/Laplace/Weibull targets from uniform to peaked: bMAE/bMASE hold per-model CV <= 0.005 with exactly one ranking (Kendall tau = 1.00), while MAE, MSE, R2, GM, and even SERA drift (SERA CV up to 0.515 on Weibull). Appendix D.2 cross-validates the binned estimator against a Nadaraya-Watson continuous one (1-5% bMAE agreement at Ntest >= 2000).
Multi-seed protocol with fully disclosed uncertainty across 630 runs
The tail-instability finding is novel and consequential
Strong reproducibility packaging
Weaknesses
All method comparisons rest on a single backbone
No inferential testing behind the method rankings
Selection protocol aligned with the evaluation metric at one seed
Metric definition depends on the test set
Stability analysis is confounded with documented label noise
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an evaluation/benchmark study of Deep Imbalanced Regression (DIR) and a companion to the authors' own survey [3]. It claims three blind spots: a Data blind spot (Appendix C, Table 4: 18 of 19 surveyed DIR methods evaluate primarily on AgeDB-DIR or IMDB-WIKI-DIR; the only three time-series usages share no datasets and release no training pipelines, and two use essentially discrete targets - 21 values for SHHS-DIR, integers 0-95 for TUAB); a Metric blind spot (many-/medium-/few-shot thresholds inherited from long-tailed classification are arbitrary in continuous spaces and yield no scalar ranking; Figure 3's synthetic predictors A-D flip ranks depending on which region is read); and a Stability blind spot (single-run reporting hides seed variance in exactly the sparse regions DIR targets). Remedies offered: the MuViS-DIR benchmark (9 tasks, 6 physical domains), the bMAE/bMASE metric family, and a 10-seed reevaluation of six DIR methods.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is bMASE in deep imbalanced regression?
bMASE (balanced Mean Absolute Scaled Error) divides a model's balanced MAE - a macro-average of per-bin MAE over equal-width target bins so every target region counts equally regardless of sample density - by the balanced MAE of the trivial median predictor. Values below 1 beat the trivial predictor, and because the ratio cancels each dataset's physical scale, bMASE values are comparable across datasets and aggregate via geometric mean into a single ranking number that many-/medium-/few-shot tables cannot provide.
Which DIR method performed best on the MuViS time-series benchmark?
UVote achieved the best aggregate score with a geometric-mean bMASE of 0.1732 versus 0.2163 for the vanilla ResNet1D baseline, winning on four of nine datasets. LDS was the strongest loss-based orthogonal method (0.1997, with the largest single gain on PM10), Focal-L1 and SQInv improved mildly, while the contrastive representation method RnC degraded performance (0.3314).
Why are many/medium/few-shot metrics criticized for imbalanced regression?
The paper argues the shot thresholds are inherited from long-tailed classification, are arbitrary for continuous targets, and can hide imbalance inside a region (e.g., PM2.5's few-shot zone concentrates samples at its lower end). Its Figure 3 shows four predictors whose ranking flips depending on which shot region you read, with no principled rule for aggregating the three numbers into one decision - the protocol is diagnostic but not decision-complete.
Is code and data available for this paper?
Yes. Code is public at github.com/noah-puetz/muvis-dir, built on the public MuViS multimodal virtual sensing benchmark. The appendix documents per-dataset backbone configurations, all selected method hyperparameters, the ten random seeds, and the full 630-run training protocol on a single H100 GPU.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (2-16) are only available as captions in the extracted text; visual claims such as per-bin curve shapes and the log-log linearity in Figure 6 are taken from captions and prose, not inspected.
- Bold/italic winner markings in Tables 2 and 13 were cross-checked numerically from the printed values rather than the table formatting.
- The abstract in the metadata is truncated mid-sentence, but the full text including Appendices A-G was present, so coverage is near-complete.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.