Cross-Dataset Transfer and Reliability of Explainable Artificial Intelligence for RhythmFormer Remote Photoplethysmography
Louis Chen, Torbjörn E. M. Nordling
cs.CV · cs.AI · eess.IV
Quick verdict
Quantifies rPPG XAI faithfulness over 9 conditions with statistical care; solid descriptive evidence, but single seed, single architecture, private data.
Summary
The authors quantify whether explainable-AI (XAI) attributions of a remote-photoplethysmography (rPPG) model transfer across datasets and track reliability. They train eight condition-specific RhythmFormer models on their private NCKU-rPPG dataset (77 participants; static recording at 40/200/700 lux, speaking, head rotation, cycling), reproduce a UBFC-rPPG evaluation, and score raw attention, attention rollout, attention flow, and Beyond Intuition by skin coverage and the perturbation-faithfulness SaCo coefficient. Beyond Intuition ranks best on both datasets (median coverage 0.789 and SaCo 0.837 on Static level 3; 0.826/0.917 on UBFC-rPPG). Yet within a single participant, neither measure correlates with clip-level heart-rate error, waveform correlation, or SNR: 186 of 252 coefficients fall below |ρ|=0.10 and 28 reach p<0.05 versus 13 expected by chance; pooling clips would spuriously yield 146 significant results, exposing participant confounding. Across the eight scenarios, only Beyond Intuition's coverage moves with the three performance measures (ρ=−0.43/+0.57/+0.43), while attention-only methods' SaCo runs opposite; Beyond Intuition collapses at 40 lux (median coverage 0.180, SaCo −0.178). Limitations include a single seed and final-epoch checkpoint per condition, one architecture, a spatial-resolution mismatch between methods in the SaCo evaluation, non-public NCKU-rPPG data, and unreleased code.
Review complete
Assessment at a glance
3
risk flags identified
● high 0
● medium 2
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Within one participant of one condition, neither skin coverage nor SaCo relates to clip-level heart-rate error, waveform correlation, or SNR (186 of 252 coefficients below |ρ|=0.10; 28 significant vs 13 expected by chance).
- Beyond Intuition attains the highest median skin coverage and SaCo on both Static level 3 (0.789/0.837) and UBFC-rPPG (0.826/0.917), but collapses specifically at 40 lux (median coverage 0.180, SaCo −0.178).
- Across the eight NCKU-rPPG scenarios, only Beyond Intuition's skin coverage co-moves with MAE, waveform correlation, and SNR (ρ=−0.43, +0.57, +0.43), while attention-only methods' SaCo moves opposite to each.
- Clip-pooled correlation analysis of this data measures participant differences rather than clip-level effects, yielding 146 nominally significant coefficients versus 28 within participants (Table 10).
Strengths
Exemplary treatment of nested data in statistics
All 252 clip-level correlations are computed within participants (Fisher-averaged Spearman, one-sample t over participants); Section 3.9 and Table 10 demonstrate concretely that clip pooling inflates significance (146 vs 28 significant; Static level 3 waveform-coverage flips from −0.41 pooled to −0.16..+0.13 within participants), and the between-participant medians (+0.20, −0.24, −0.28) correctly locate the effect at the participant level.
Near-audit-grade methodological documentation
Supplementary S1 fixes the conversion contract (pinned MTCNN/FaceXFormer commits, SHA-256 checkpoint digest, ≥0.99 encode-decode RGB correlation gate, ≥0.80 per-session detection gate); Table 20 validates per-recording synchronisation-lag fitting against a mismatched-pair null; Tables 19/21 decompose window-length error against a beat-to-beat reference; the SNR amplitude-vs-power convention is candidly disclosed (Section 3.6).
Honest scope and negative-result discipline
Controlled multi-condition design with verified separation
Weaknesses
Single seed, final-epoch checkpoint, no validation selection
Resolution mismatch confounds cross-method SaCo rankings
Independent verification currently impossible
Cycling references are weak and partially censored
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks two questions about explainability for RhythmFormer-based remote photoplethysmography (rPPG): do attribution explanations transfer between the private NCKU-rPPG dataset and the public UBFC-rPPG benchmark, and do quantitative XAI measures track model reliability? The claimed contributions are: (1) a quantitative (rather than visual-heatmap) comparison of raw attention, attention rollout, attention flow, and Beyond Intuition across eight recording scenarios plus a UBFC-rPPG reproduction, scored by skin coverage and the Salience-guided Faithfulness Coefficient (SaCo); (2) evidence that neither measure is a clip-level proxy for heart-rate error, waveform correlation, or SNR; (3) scenario-level evidence that only Beyond Intuition's coverage co-moves with performance while attention-only SaCo runs opposite. Claims are explicitly framed as descriptive; the abstract and Section 1 disclaim significance, causality, and cross-dataset generalisation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 3 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SaCo in this rPPG paper?
SaCo (Salience-guided Faithfulness Coefficient) measures how well an attribution map's ordering agrees with perturbation effects: spatial positions are ranked by salience, grouped into 8 groups, each group masked (set to zero) in turn, and the coefficient rewards cases where high-salience groups change the predicted waveform more than low-salience groups. It ranges from −1 to +1.
Did the XAI explanations actually predict when the rPPG model was accurate?
Not at the clip level. Within a single participant, neither skin coverage nor SaCo correlated with heart-rate error, waveform correlation, or SNR (186 of 252 coefficients had |ρ|<0.10; 28 significant vs 13 expected by chance). Only at the coarse scenario level did Beyond Intuition's coverage move with performance, and even that is descriptive and non-significant with eight points.
Is the NCKU-rPPG dataset publicly available?
No. At the time of writing it is available only on reasonable request to the corresponding author, subject to NCKU ethics approvals; a public release of consenting participants' recordings is planned for a later article. The UBFC-rPPG dataset used as the public reference is public, and the analysis code is likewise available only on request.
Why did Beyond Intuition fail in dim light?
At 40 lux its median skin coverage fell to 0.180 and its median SaCo to −0.178 on Static level 1 (117 of 200 clips below zero), meaning the model stopped concentrating attribution on facial skin, so the regions the map ranked highest no longer matched the regions the prediction depended on. Motion (speaking, rotation, cycling) degraded accuracy far more without producing such a collapse.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked '[...sampling gap...]' sections; parts of the Results narrative (around Sections 4.1-4.2), portions of the Discussion, parts of the reference list, and parts of Supplementary S6 were skipped — judgments touching those regions rest only on the visible text.
- Figures appear only as extracted caption and axis-label text; their visual content (e.g. Figures 3, 5, 9) could not be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.