CRISP: Scalable Importance-Stratified Coresets for Imbalanced Tabular Learning
Hardhik Mohanty, Indrayana Rustandi, Mohamadreza Sheibani
cs.LG
Quick verdict
Score-stratified coreset for imbalanced GBDT training; Criteo evidence solid, but the proprietary headline, shifted settings and no tests temper it
Summary
CRISP is a linear-time coreset-selection method for highly imbalanced tabular GBDT training. It keeps every positive example and reduces only the negative class: a lightweight proxy LightGBM scores negatives, equal-count quantile strata preserve score coverage, each stratum receives a budget proportional to its mean score, and inverse-inclusion weights (clipped at 20) correct unequal sampling. On a ~25M-row proprietary payment-fraud dataset, CRISP retains 99.7% of full-data monthly Average Precision at 95% negative-class reduction (~1.70M rows, 6.8% of data), and the selected rows transfer to XGBoost (98.9%/98.2% retention). On public CriteoPrivateAds (~85M rows), CRISP attains the highest mean AP at every tested reduction rate (90–99.4%), with the lead over the best baseline growing from 0.0034 to 0.0074 AP. Sparkov results are mixed at lower rates. Limitations: the within-stratum sampling rule differs between production and public runs, FLAML zero-shot tuning lets target hyperparameters vary per method, results rest on three or five seeds without formal significance tests, the headline dataset is private with no code or artifacts released, and no end-to-end timing is reported.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CRISP retains 99.7% of full-data monthly Average Precision on a ~25M-row production fraud dataset at 95% negative-class reduction, training on ~1.70M of 25M rows (a 93.2% total-row reduction).
- On public CriteoPrivateAds, CRISP achieves the highest mean AP at every tested negative-reduction rate (90%–99.4%), with the lead over the strongest baseline growing from 0.0034 to 0.0074 AP.
- Importance-proportional budget allocation across quantile strata and inverse-propensity weighting are the main sources of the gain; removing strata (Q=1) or IPW each costs 1.8–3.8 points of AP retention at 95% reduction.
- Selection runs in linear time using a fixed-size proxy fit plus a constant number of distributed passes, and the selected rows transfer across two GBDT implementations (LightGBM and XGBoost).
- The deployed clipped-weight estimator is biased (Eq. 10), so CRISP is a task-aware subset selector rather than a strong coreset with universal approximation guarantees.
Strengths
Production-scale headline with honest accounting
At 95% negative-class reduction on a ~25M-row proprietary fraud dataset, CRISP retains 99.7% of full-data monthly AP training on ~1.70M rows (6.8%), and the retained rows transfer to XGBoost (98.9% monthly / 98.2% weekly, §5.6). Table 1 makes the retain-all-positives row-count semantics explicit instead of conflating negative and total reduction.
CriteoPrivateAds is the strongest externally checkable evidence
Five seeds, tiny standard deviations (10^-4–10^-3), and per-run Table 11 shows CRISP above every baseline for all five seeds at all five rates; margins over the best baseline grow from 0.0034 AP (90%) to 0.0074 (99.4%), matching the scarcity-allocation hypothesis in §4.3.
Ablations target each design decision
Estimator scope stated precisely
Negative results reported rather than buried
Weaknesses
The configuration changes across datasets
No isolation from target hyperparameter tuning
Statistical support is thin
Headline dataset cannot be independently verified
Baseline coverage and cost accounting are incomplete
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
CRISP: Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CRISP and how does it work?
CRISP (Coreset Reduction via Importance-Stratified Pruning) builds a weighted training subset for imbalanced tabular data. It keeps every positive example and reduces only negatives: a small proxy GBDT scores each negative row, rows are binned into equal-count quantile strata of that score, each stratum receives a share of the negative budget proportional to its mean score, sampling happens within strata, and selected rows get inverse-probability weights (capped at 20) so the training distribution stays close to the original.
How much accuracy does CRISP lose while shrinking the dataset?
On a ~25M-row proprietary payment-fraud dataset, at 95% negative-class reduction CRISP retains 99.7% of the full-data monthly Average Precision while training on ~1.70M rows (6.8% of the data, a 93.2% total-row reduction). On public CriteoPrivateAds it keeps 99.8% of full-data AP at 90% negative reduction (~9.1M rows) and 94.2% at 99.4% (~1.16M rows).
How is CRISP different from random undersampling or CCS?
Random undersampling removes easy and informative negatives at the same rate. CCS (Coverage-Centric Coreset Selection) also uses score strata but allocates the budget uniformly. CRISP instead allocates more budget to strata with higher mean proxy-model scores while still keeping nonzero inclusion probability in low-score strata, and corrects the resulting sampling bias with inverse-propensity weights — this matters most when the budget is very small.
Can I reproduce the results, and is code available?
The paper pledges no code or artifact release. The public datasets (CriteoPrivateAds, Sparkov) and all key hyperparameters (Q=10, α=β=1, wmax=20, proxy LightGBM depth 6/300 trees) are documented, so a reimplementation is possible; Sparkov at ~1.3M rows is CPU-feasible. The headline production-dataset result cannot be independently verified because the data is proprietary and absolute AP values are withheld.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 are recovered only as caption text; curve shapes, error-bar magnitudes, and the ablation suite layout could not be analyzed.
- Appendix E (XGBoost transfer table) and parts of Appendix F are present only as headers/references in the extracted text; per-run XGBoost numbers were taken from §5.6 prose.
- The production dataset, implementation, and absolute metrics are non-verifiable by construction; assessments of the headline result rely on internal consistency of Tables 4, 5, and 8.
- Extracted text appears otherwise complete (all main tables and appendices A–D legible), so the review's numerical checks are grounded in the primary tables.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.