Prevalence Determines Precision:Silent Contamination in Detector-Defined Datasets
Jia Huang, Yankai Wan, Yangjun Ou
cs.LG · cs.AI
Quick verdict
Claims dataset precision is set by pool prevalence via Bayes; futures case study shows +422% naive transfer error — sound and honest, but scope is narrow.
Summary
The paper argues that the precision of a dataset built by running a detector over a candidate pool is governed by true-positive prevalence π through Bayes' rule, not by detector quality alone. Using E-mini S&P 500 futures (2010–2026), the authors run one magnitude-threshold detector on three pools with recoverable ground truth (official CPI, non-farm payroll, and FOMC indices), obtaining phantom rates of 0.0%, 9.0%, and 81.7%. Naively transferring precision from the two high-π pools (0.955) to the low-π pool errs by +422% against the measured 0.183, while the Bayes formula holds within 3.3%. The contaminated response curve is an exact convex combination of true-event and phantom components (residual 1.1e-16; 473 phantoms vs 308 true events). Contamination direction is estimator-dependent: a denominator-free statistic is diluted, a terminally normalized one is inflated, and a per-item normalization with no finite expectation returns 0.40 versus 0.10 on identical events. A selection simulator confirms the mechanism. Limitations: one instrument and one deliberately unscaled detector family (FPR 0.361), official windows only from ~2013 onward, the selection-conditioned null matches phantom shape but not level, and the negative control has n=3.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Precision of a detector-defined dataset is governed by true-positive prevalence in the deployment pool via Bayes: one detector on three pools yields phantom rates 81.7%, 9.0%, and 0.0%, naive precision transfer errs by +422%, and the Bayes expression predicts all three within 3.3%.
- The contaminated response curve is an exact convex combination of a true-event and a phantom component (residual 1.1e-16), so contamination is a second signal with detector-inherited shape, not additive noise.
- The direction of contamination is estimator-dependent: a denominator-free statistic is diluted while a terminally normalized statistic is inflated, because the latter's scale is itself contaminated.
- Per-item normalization by a signed, mean-zero reference return produces an estimator with no finite expectation, returning 0.40 where the pooled estimator returns 0.10 on the same 335 events.
- Detector-defined datasets should report five quantities (π, TPR, FPR, c, ρ) and follow a seven-rule validation protocol.
Strengths
End-to-end measurement of a textbook mechanism with recoverable ground truth
Table 1 runs one detector on three pools (FOMC π=1.00, NFP π=0.82, CPI π=0.105) yielding phantom rates 0.0%/9.0%/81.7%; naive precision transfer predicts 0.955 vs measured 0.183 (+422%) while the Bayes expression holds within 3.3% (§5.1–5.2).
Exceptional transparency and self-audit
A pre-registered absolute claim is abandoned when the threshold is unmet (0.068 ≤ 0.15, §5.7), a failed experiment is retracted (App. B), six accidents in the authors' own pipeline are documented (App. A), and indices, labels, code, and a per-number traceability map are released.
Exact decomposition verified to machine precision on real data
Estimator-level findings with broad applicability beyond finance
Mechanism isolated by a clean simulation
Weaknesses
Headline real-data magnitudes depend on a deliberately broken detector
Selection-conditioned null reproduces phantom shape but not level
No uncertainty quantification anywhere
Narrow scope and a collapsed control
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses datasets built by running a detector, heuristic, or model over a candidate pool, where whatever is accepted becomes the label set — distant supervision, weak supervision, pseudo-labelling, and most event/anomaly benchmarks. Its claims: (1) the precision of such datasets is governed by true-positive prevalence π through elementary Bayes (Eq. 1), and precision does not transfer across pools while the pair (TPR, FPR) does; (2) an end-to-end real-data measurement of this mechanism with recoverable ground truth; (3) an exact mixture decomposition (Eq. 3) showing contamination is a second signal with detector-inherited shape, not additive noise; (4) the direction of contamination is estimator-dependent; (5) a per-item normalized estimator has no finite expectation (Prop. 1). A seven-rule reporting protocol and a six-accident appendix round it out. The authors explicitly disclaim novelty for the Bayes formula itself, locating the contribution in measurement and downstream consequences — an accurate self-assessment.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why does the same detector have different precision on different datasets?
Because precision depends on the fraction of true positives (prevalence) in the candidate pool the detector is applied to, via Bayes' rule. A detector with fixed TPR and FPR looks near-perfect where prevalence is high and near-useless where it is low. In this paper, one detector produced phantom rates of 0.0%, 9.0%, and 81.7% on three different pools of the same price series.
What does the +422% error reported in the paper refer to?
A practitioner who transfers the average precision measured on two high-prevalence pools (0.955) to a low-prevalence pool predicts 0.955 where the measured precision is 0.183 — an error of +422%. The Bayes formula using each pool's own prevalence instead predicts 0.177, within 3.3% of the measured value.
Is the 81.7% phantom rate just caused by a bug in the detector?
Largely yes, and the authors say so explicitly. The detector's statistic omitted a √120 time scaling (Appendix A, Accident 1), giving a false-positive rate of 0.361 under pure noise. The bug is retained deliberately as a realistic worst case; a correctly scaled detector is analyzed only in simulation (§5.6), which is why the real-data magnitudes should be read as an upper bound on contamination damage.
Can this study be reproduced?
Substantially, yes. The event indices, phantom labels, screens, curves, and simulator code are released with a per-number traceability map and a decision log. The raw futures prices cannot be redistributed under the vendor's licence, but exact request manifests (symbol, schema, window, checksum) allow reconstructing an identical dataset, and the simulator reproduces the core mechanism without any proprietary data.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction garbles figure axis labels and mathematical notation; every number cited in this review was verified against the body text, tables, or captions, which are intact.
- The provided text ends after Appendix E with the page-12 marker and no visible truncation; the abstract, all main sections, references, and Appendices A–E are present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.