Reliable Virtual Sensing: A Multi-Domain Benchmark for Robustness Under Sensor Failures
Jens U. Brandt, Noah C. Puetz, Alexander Windmann, Marc Hilbert, Elena Raponi, Thomas B\"ack, Thomas Bartz-Beielstein
cs.LG · cs.AI
Quick verdict
MuViS-C: credible multi-domain benchmark showing all virtual sensors fail under corruption; trees most robust; solid but single-seed evidence.
Summary
The paper introduces MuViS-C, an open-source benchmark extending the MuViS virtual-sensing framework with systematic robustness evaluation under sensor failures. Building on established fault taxonomies, it defines ten severity-parameterized failure modes (bias, noise, scaling, drifts, outliers, trimming, hard fault) applied one channel at a time across a failure x channel x severity grid (20 severities, fixed scale k=3 on z-scored data). Robustness is quantified via mPC and rPC (adapted from vision corruption benchmarks) plus a new worst-case metric, scross, the earliest severity at which a model performs no better than a naive mean predictor. Across nine datasets from six domains, six architectures (XGBoost, CatBoost, ModernTCN, PatchTSMixer, xLSTM-Mixer, TST) are benchmarked, and three robustification strategies are probed on TST (PGD adversarial training, input sensor dropout, F2F self-supervised pretraining). Key results: every model drops below the naive predictor in at least one setting; gradient-boosted trees lead robustness ranks without any defense (XGBoost mean rank 2.3 on mPC); all three defenses lift TST into the leading rPC cluster at a nominal accuracy cost; rankings shift across datasets. When faults are detected and mean-imputed, ISensD nearly eliminates degradation (rPC 1.034). Main limitations: single-channel failures only, fixed k=3, one backbone for defenses, single training seed.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Every evaluated model degrades substantially under sensor corruption and becomes worse than a naive mean predictor in at least one failure/channel/severity combination.
- Gradient-boosted tree ensembles (XGBoost, CatBoost) achieve strong corruption robustness without dedicated defenses, with XGBoost the only model holding an exclusive top-cluster position on both mPC and rPC.
- All three robustification strategies (PGD, ISensD, F2F) move the fragile TST backbone into the leading rPC cluster, but each incurs a nominal clean-data penalty of +0.27 to +0.44 in normalized GM.
- Model rankings shift substantially across the nine datasets and ten failure modes, establishing multi-domain, multi-failure evaluation as a methodological requirement for virtual sensing.
- When a reliable failure detector enables mean imputation, training with input sensor dropout (ISensD) reduces average degradation to nearly zero (GM rPC 1.034, mean rank 1.0).
Strengths
Physically grounded, severity-parameterized failure catalogue
Ten modes (Table 1) distilled from established taxonomies (Balaban et al.; Jesus et al.), each extended into a continuous family Ds over severity s in [0,1], anchored at k=3 via the 3-sigma rule on z-scored data; atomic single-channel perturbation enables exhaustive failure x channel x severity sweeps with per-channel attribution (Sec. 3.2).
Complementary metric design with explicit cross-reading guidance
mPC (absolute error), rPC (degradation normalized by clean RMSE), and the new worst-case scross are defined so no single number hides the accuracy-robustness tradeoff (Sec. 3.3); nPC (Appendix E) adds a naive-predictor normalization, and Appendix F carefully relates all of them to mCE and the robustness score.
Transparent statistical protocol
Multi-domain evidence that rankings are not portable
Actionable deployment finding and reproducibility infrastructure
Weaknesses
Single training seed undermines ranking claims
scross is statistically unsupported yet presented as a ranked result
Asymmetric hyperparameter tuning favors the robustified variants
rPC can reward weak nominal models
Ecological validity limits acknowledged but consequential
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MuViS-C?
MuViS-C is an open-source benchmark for measuring how well learning-based virtual sensing models — models that estimate hard-to-measure quantities from other available sensors — hold up when input sensors fail. It implements ten physically motivated sensor failure modes (bias, noise, scaling, drift, outliers, trimming, hard fault) at twenty severity levels, applied one sensor channel at a time, across nine datasets from six application domains.
What did the MuViS-C benchmark find?
Three headline findings: (1) every tested model, including strong ones, degrades under sensor corruption and in at least one setting becomes worse than simply predicting the training-set mean; (2) gradient-boosted tree ensembles (XGBoost, CatBoost) are the most robust models without any special defense; (3) robustification training (adversarial training, input dropout, self-supervised pretraining) improves the fragile attention model but always costs some accuracy on clean data.
Which virtual sensing model should I choose for robustness to sensor failures?
There is no universal winner: model rankings shift across datasets and failure modes. As an all-around robustness baseline, XGBoost performed best in this benchmark. If your system can detect a failed sensor and replace its values with the training mean, training with random input-sensor dropout (ISensD) nearly eliminated degradation in the paper's experiments.
Is the MuViS-C code and data available?
Yes. The authors state the benchmark is open-source and extensible, released as a Python package with a Testbed API supporting custom models, datasets, failure modes, and metrics. Full reproduction (training all 81 model-dataset configurations plus the severity sweep) takes about 155 GPU-hours on a single NVIDIA H100.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (motivating example, severity curves, critical-difference diagrams) are not analyzable from the extracted text; assessments rest on tables and prose.
- Per-model hyperparameter values are said to live in the accompanying repository and could not be verified from the paper itself.
- This is a v1 preprint (arXiv:2609.18396, 16 Sep 2026) and has not, to the reviewer's knowledge, completed peer review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.