Beyond Performance Metrics: Uncertainty Mapping of Label Ambiguity in Fazekas Score Prediction
Susanne Schmid, Johanna Ospel, Richard Frayne, Roberto Souza
cs.CV
Quick verdict
Epistemic uncertainty mapped onto t-SNE features audits ambiguous Fazekas labels; the evidence is largely qualitative and the expert validation spans 12 cases.
Summary
The paper proposes 'uncertainty mapping' for periventricular Fazekas score prediction: epistemic uncertainty from Monte Carlo dropout (100 forward passes, mutual-information decomposition) is overlaid on t-SNE visualizations of the 256-dimensional penultimate features of a 3D VGG-like CNN trained on 424 Calgary Normative Study subjects (3T T1+FLAIR). Four losses - CCE, generalized cross-entropy, ordinal log-loss (OLL), and OLL with logit clipping - are compared under nested 5x4 cross-validation with fixed hyperparameters. Average MCC is modest (OLL clip 0.49±0.10; GCE 0.39; best single split 0.70), and unweighted Cohen's kappa (0.37-0.48) falls within reported human inter-rater agreement (0.34-0.42), supporting the thesis that much model 'error' reflects label variability. The maps reportedly show elevated uncertainty near ordinal class-boundary transitions plus isolated low-uncertainty misclassifications, while LRP heatmaps look similar across models. A targeted re-evaluation of 12 cases by an independent radiologist revised all 4 flagged 'label-inconsistency' cases (mean 1.75 points) versus 1 of 4 confident-agreement cases. Main limitations: conclusions rest on qualitative t-SNE inspection without a quantitative boundary-uncertainty metric; loss differences are small relative to split variance and untested statistically; expert validation covers 12 cases from one model and split; data are private and no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Conventional metrics alone do not capture model behavior for Fazekas prediction because performance varies across splits and losses (MCC up to 0.70, averages 0.39-0.49) while labels carry inter-rater ambiguity.
- Overlaying MC-dropout epistemic uncertainty on the t-SNE of penultimate features ('uncertainty mapping') separates ambiguous class-boundary cases from low-uncertainty, high-confidence misclassifications.
- Low-uncertainty misclassifications can flag inconsistent reference labels: an independent radiologist revised all 4 flagged cases (mean 1.75 Fazekas points) versus 1 of 4 confident-agreement cases.
- Loss choice shapes the uncertainty profile, with OLL + logit clipping producing the clearest class separation and uncertainty concentrated at ordinal boundaries.
- Model agreement with reference labels (unweighted kappa 0.37-0.48) lies within the reported human inter-rater range (0.34-0.42), implying much apparent model error reflects label variability.
Strengths
Transparent performance reporting
Table A.1 lists all 20 per-run MCC values for each loss, including the poorly performing GCE (mean 0.39), and the text openly discusses split-to-split variability rather than cherry-picking only the best configuration.
Clinically grounded error interpretation
Section 5 benchmarks the models' unweighted Cohen's kappa (0.37-0.48, Table 1) against published human inter-rater kappa for Fazekas scoring (0.34-0.42), carefully noting the models are not independent raters - a rare, honest reframing of 'error' as label variability.
Structured expert validation with contrast groups
Practical motivation for targeted dataset auditing
Thorough and candid limitations section
Weaknesses
Central claims are qualitative and unquantified
Circularity risk in label-error flagging
Underpowered and potentially unfair loss comparison
Only one uncertainty source, no ensemble baseline
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is uncertainty mapping in this paper?
It is a visualization framework that overlays epistemic uncertainty (estimated with Monte Carlo dropout over 100 forward passes) onto a t-SNE map of the model's penultimate-layer features, so that each test case's uncertainty can be read in relation to its position among the learned class clusters - typically high near class boundaries and low inside well-separated regions.
How accurately does the model predict Fazekas scores?
Modestly: across 20 nested cross-validation runs the best loss (ordinal log-loss with logit clipping) averaged MCC 0.49±0.10, with a best single split of 0.70; unweighted Cohen's kappa ranged 0.37-0.48, which is within the 0.34-0.42 agreement reported between human raters for Fazekas scoring.
Did expert review confirm that the flagged labels were actually wrong?
In a 12-case pilot, a radiologist who did not do the original grading revised all 4 cases flagged as confident model-label disagreement (average change of 1.75 Fazekas points, some by two points), revised 2 of 4 ambiguous-boundary cases, and 1 of 4 cases where model and label confidently agreed - supporting, but not yet proving, the flagging approach.
Is the code or dataset available?
No. The paper states no code release, and the Calgary Normative Study MRI data are not publicly available because the authors are not authorized to redistribute them; the study was approved by the University of Calgary ethics board (15-1285).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (t-SNE uncertainty maps, LRP heatmaps, FLAIR example images) could only be judged from captions; the qualitative claims they carry were not visually verifiable.
- Table 2's 'Reviewed' counts appear inconsistent with the listed case revisions in the extracted text, so exact reviewed-case numbers per group may differ from what is stated here.
- The abstract text was truncated mid-sentence in the source extract; the full abstract conclusions were reconstructed from the body and highlights.
- The referenced prior ISBI 2025 work [45] supplying the task/architecture baseline was not available for verification.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.