Just add noise: Debiasing tree-based variable importance in mixed data
stat.ML · cs.LG · stat.ME
Quick verdict
First formal theory for CART's mixed-data importance bias plus a simple jitter fix; proofs idealized but experiments convincing — strong, near-ready work.
Summary
The paper gives the first asymptotic theory for the well-documented bias of impurity-based variable importance (MDI) in tree ensembles when data mix continuous and categorical predictors. Key insight: at a node with n samples a continuous predictor offers n−1 candidate splits while a K-level categorical offers at most 2^(K−1)−1. Under null models, Theorems 2.1–2.2 show CART splits continuously with probability tending to one, so categorical MDI vanishes; Theorem 2.4 shows that in Y=f(X)+g(Z)+ε, once the binary Z is split, the remaining noise variance σ² is absorbed into X's importance. The fix—jittering—adds Unif(−δ,δ) noise (δ=10⁻⁴) to integer-encoded categoricals as preprocessing, equalizing split counts. Simulations (n=200, 25+25 predictors, 100 trials) show jittered RF achieves the best ranking AUCs (up to 0.96 vs 0.80 for plain RF); combined with IPSS, jittering restores FDR control (plain RF's FDR plateaus at 0.8 and it never selects binary predictors). In TCGA bladder cancer data, jittering ranks pathologic stage first among 425 predictors (plain RF: median 36). Across seven public datasets, jittered RF selects almost no permuted synthetic nulls where RF selects up to 14.7. Limitations: theory excludes bootstrap resampling and feature subsampling, assumes min node size ≥ cn and fully grown trees, Theorem 2.4 requires Var(g(Z))>M*(X), and permutation importance is not benchmarked.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MDI bias in mixed data arises from a split-count asymmetry: continuous predictors offer n−1 candidate splits at a node versus at most 2^(K−1)−1 for a K-level categorical.
- Under null models, CART splits continuously with probability tending to one and categorical MDI vanishes asymptotically; in Y=f(X)+g(Z)+ε the noise variance σ² is absorbed into X's importance.
- Adding Unif(−δ,δ) noise (δ=10⁻⁴) to integer-encoded categoricals equalizes split opportunities, removing the bias without tuning and without harming predictive accuracy.
- Combined with IPSS, jittering restores FDR-controlled variable selection for mixed data, where plain RF never selects binary predictors and its empirical FDR plateaus at 0.8.
Strengths
First formal asymptotics for mixed-data MDI bias
Theorems 2.1–2.2 prove continuous splits dominate under nulls (root node and bounded-depth ensembles with node size ≥ cn), going beyond Louppe et al. (2013)'s totally randomized, all-categorical setting; proofs are complete and use standard empirical process tools (VC class, Donsker, Brownian bridge LIL in Lemmas S1.1–S1.3).
Remedy is a one-line preprocessing step
Jittering with δ=10⁻⁴ requires no tuning (Table S3: AUC flat within 0.006–0.016 as δ ranges over [10⁻⁶,10⁻¹]), leaves prediction error essentially unchanged (Figure S5: median error ratio 1.00 for RF, 0.99 for XGBoost on 33 PMLB datasets), and plugs into any existing software.
Large, multi-pronged empirical evidence
Honest scope statements
Weaknesses
Theory-practice gap
One-sided regime analysis
Missing permutation-importance baseline
Modest replication in null experiments
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This method paper (Li & Melikechi, Duke) targets the long-documented tendency of impurity-based variable importance (MDI) in CART and CART ensembles to favor continuous over categorical predictors. It claims three things: (1) a formal asymptotic account of the bias rooted in split-count asymmetry — at a node with n samples a continuous predictor offers n−1 candidate splits while a K-level categorical offers at most 2^(K−1)−1 (Section 2.1, Figure 1); (2) an exact allocation result in the additive model Y=f(X)+g(Z)+ε: once binary Z is split, the residual noise variance σ² is absorbed into X's MDI (Theorem 2.4), matching Figure 2's 2/3–1/3 convergence; (3) a minimal fix — jittering, adding Unif(−δ,δ) noise at δ=10⁻⁴ to integer-encoded categoricals — that restores fair rankings, FDR calibration under IPSS, and (Figure S5) unchanged predictive accuracy.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why do random forests favor continuous variables over categorical ones?
At a node with n samples, a continuous predictor offers n−1 candidate split points, while a categorical predictor with K levels offers at most 2^(K−1)−1. Under noise, CART therefore keeps choosing continuous splits with probability tending to one as n grows, so impurity-based importance (MDI) concentrates on continuous predictors — and once a binary variable is used, the remaining noise variance gets credited to continuous variables.
What is the 'just add noise' (jittering) fix?
Before fitting, add small uniform noise Unif(−δ, δ), e.g. δ=10⁻⁴, to integer-encoded categorical predictors. This makes them effectively continuous, so every predictor has a comparable number of candidate splits. It is a one-line preprocessing step, robust to the choice of δ, compatible with any tree software, and does not change predictions appreciably.
Does adding noise to categorical features hurt prediction accuracy?
On 51 Penn Machine Learning Benchmark datasets, the median ratio of jittered to unjittered test error was 1.00 for random forests and 0.99 for XGBoost — no systematic degradation (Figure S5). Datasets where both models were near-perfect (error below 0.05) were excluded from that summary.
Does the theory apply to real random forests with bootstrapping?
Not directly. The theorems cover CART and CART ensembles without observation resampling or feature subsampling, assume minimum node sizes proportional to n, and (for the non-null result) fully grown trees. The authors state this limitation and support practical relevance with experiments at n=200 using default scikit-learn settings.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 and S1–S5 are available only through captions; curve shapes and error bars could not be inspected directly.
- Math extraction shows minor ligature artifacts (subscripts, overlines), but all theorem statements and proofs are legible in full.
- Numbers in Tables S1–S3 were read from extracted text and not independently verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.