TreeFI: Value-Aware Statistical Fault Injection for Deep Neural Networks
Noam Bires, Marcello Traiola, Angeliki Kritikakou, Elisa Fromont
cs.AR · cs.AI
Quick verdict
Value-stratified SFI cuts DNN bit-flip injection budgets up to 72.1x; rigorously validated against exhaustive FI on ResNet8 only, larger models unverified.
Summary
The paper tackles the cost of evaluating DNN reliability under floating-point bit flips. Exhaustive fault injection is intractable, and existing statistical fault injection (SFI) methods do not exploit the fact that a flip's impact depends strongly on the corrupted value. TreeFI partitions each layer's value distribution into intervals using one-dimensional regression trees trained on weighted histograms, maps each interval's mean log-magnitude of the value perturbation through a sigmoid into a risk score, allocates injections proportionally to interval frequency times sqrt(r(1-r)) in a Neyman-style stratified design, and aggregates interval-level failure rates weighted by empirical frequencies. On ResNet8/CIFAR-10, where exhaustive activation FI is feasible (9,437,184 injections), TreeFI-10x needs 34,934 injections (28 min versus 7 h 13 min for SFI) and achieves lower MAE than SFI and IFI near the most vulnerable bits. Across ResNet8, RepVGG-A0, and DeiT-Tiny/Small/Base, activation-injection reductions versus SFI span 5.9x-72.1x (average 44.9x); weight campaigns reduce 6.4x-17.9x with all MAEs below 0.05%. Limitations: exhaustive ground truth exists only for ResNet8 activations; the authors show that applying TreeFI's frequency reweighting to SFI results matches TreeFI estimates, so part of the accuracy gain comes from reweighting rather than the allocation; the budget rule for the stratified estimator is not derived; weight evaluations cover only 10 RepVGG and 6 DeiT-Tiny layers.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The impact of an FP32 single-bit flip on DNN behavior depends strongly on the corrupted value, so sampling organized by value intervals is more efficient than value-uniform SFI (Section 3.1, Fig. 2).
- TreeFI meets the same 1%-error/99%-confidence targets as SFI/IFI with up to 72.1× fewer activation injections (DeiT-Small vs SFI; average 44.9×) and 6.4×-17.9× fewer weight injections (Tables 1, 3).
- On ResNet8, where exhaustive activation FI is feasible, TreeFI-10× estimates are closer to exhaustive truth than SFI and IFI, especially around the most vulnerable layer-bit regions (Fig. 3).
- Wall-clock campaign savings track injection-count reductions almost exactly (e.g., RepVGG 15.9× vs SFI measured vs 17.5× theoretical; Section 4.1.3).
- Gains grow with model size, as shown by the DeiT family trend where baselines become impractical (Sections 4.1.2-4.1.3).
Strengths
Rare exhaustive ground-truth validation
Section 4.1.1 validates against a full exhaustive activation campaign on ResNet8/CIFAR-10 (9,437,184 injections over 5,000 inputs); TreeFI-10× needs only 34,934 injections (~270× fewer, 28 min vs 7 h 13 min for SFI) and reproduces the same most-vulnerable bits with lower MAE near L0-L2 and bit 30 (Fig. 3).
Large, internally consistent efficiency gains
Table 1/Fig. 4: activation reductions vs SFI from 5.9× (RepVGG-A0, γ=1) to 72.1× (DeiT-Small, γ=10), average 44.9× for TreeFI-10×; measured wall-clock savings track injection counts (RepVGG 15.9×/20.1× vs SFI/IFI; DeiT-Tiny first layer 20 min vs ~37-41 h).
Useful hyperparameter ablation with feasible-region analysis
Honest scoping and a genuine control experiment
Weaknesses
Single point of exhaustive ground truth
Budget guarantee asserted, not derived
Accuracy gains partially confounded with reweighting
Weight evaluation is partial and truth-free
Heuristic risk score standing in for stratum variance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
TreeFI (arXiv 2609.04912, cs.AR/cs.AI) is a method paper on statistical fault injection (SFI) for DNN reliability under FP32 single-bit flips in activations and weights. Its central claims: (1) bit-flip impact depends strongly on the corrupted value (Section 3.1, Fig. 2, 65,536 injections per panel); (2) stratifying injections over learned value intervals meets the same 1%-error/99%-confidence target with far fewer injections — up to 72.1× versus the one-shot SFI baseline [7] (DeiT-Small, Table 1), 44.9× average activation reduction, and 11.2× average over the executed weight campaigns; (3) on ResNet8, where exhaustive FI is feasible, TreeFI estimates are closer to exhaustive truth than SFI [7] and IFI [8] (Fig. 3). Scope is honestly bounded: single-bit independent faults, sampling efficiency rather than fault realism (Section 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does TreeFI solve?
It reduces how many hardware-fault injections are needed to estimate a deep neural network's failure rate under floating-point bit flips, at a fixed statistical confidence (99%) and error margin (1%). It does this by injecting more densely in value ranges where a bit flip actually matters, using regression trees to find those ranges.
How large are TreeFI's savings compared to existing methods?
Compared to the state-of-the-art one-shot statistical FI baseline (SFI), TreeFI uses 5.9× to 72.1× fewer activation-fault injections across five models (average 44.9× for the recommended configuration), and 6.4× to 17.9× fewer weight-fault injections. Runtime savings track these ratios, e.g., 28 minutes vs 7 h 13 min on ResNet8 activations.
Is TreeFI's accuracy actually verified?
Only on one benchmark: ResNet8 on CIFAR-10 activations, where a full exhaustive campaign (9,437,184 injections) provides ground truth and TreeFI stays within the 1% target with lower error than baselines near the most vulnerable bits. On larger models (RepVGG, DeiT), accuracy is argued from per-layer/per-bit trend agreement with the baselines, not from ground truth.
Is the TreeFI code available?
Yes. The paper's reproducibility section points to a public GitLab repository (gitlab.inria.fr/nbires/treefi) with installation instructions, ready-to-run activation and weight examples, and documentation for reproducing the paper's results.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal values (per-point MAE magnitudes in Fig. 3, failure-rate curves in Figs. 5-6, ablation points in Fig. 7, and the 15-seed variability numbers) are only partially recoverable from the extracted text, which preserves axis labels and in-text numeric claims but not plotted data.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.