The Accuracy Paradox: Empirical Diagnostic of Default Decision Thresholds in Multi-Label Enzyme Commission Prediction [With Code]
cs.LG · cs.AI · q-bio.QM
Quick verdict
Claims fixed τ=0.50 masks failures in imbalanced multi-label EC prediction; coherent diagnostic, but classifier unnamed and proposed fixes never tested.
Summary
This diagnostic study evaluates how a fixed decision threshold (τ=0.50) behaves under class imbalance in multi-label Enzyme Commission (EC) prediction. On 14,096 compounds spanning six EC classes, a baseline multi-label classifier achieves 77.16% mean accuracy while macro F1 (0.3976) and macro recall (0.3872) reveal severe failures. Majority classes over-predict: EC2 (80.09% active) reaches 100.00% recall with only one true negative, while EC1 shows 3,034 false positives. Minority classes decay: recall falls to 16.34% (EC3), 18.25% (EC4), and 10.55% (EC5), and EC6 collapses to zero positive predictions despite ROC-AUC of 0.5857. Mean ROC-AUC of 0.6451 indicates retained ranking ability, isolating the threshold as the failure point. Correlation analysis shows strong collinearity among Chi topological descriptors and near-zero linear correlation of Morgan fingerprint densities with targets; EC1-EC2 co-occurrence reaches 7,151 instances versus 245 for EC5-EC6. The authors recommend per-target threshold optimization and post-hoc conformal calibration but implement neither. Key limitations: the baseline classifier is never named, no train/test split or tuning protocol is described, and conclusions rest on one dataset and one unattributed model.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 19 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Applying a default τ=0.50 threshold to uncalibrated multi-label EC predictions induces an Accuracy Paradox: mean accuracy of 77.16% coexists with macro F1 of 0.3976 and macro recall of 0.3872 across N=14,096 compounds.
- Majority classes over-predict under the fixed threshold — EC2 (80.09% active) reaches 100.00% recall with TN=1 — while minority classes decay to total collapse (EC6: recall 0.00%, TP=0) despite retained ranking ability (ROC-AUC 0.5857).
- Topological connectivity descriptors (Chi family) are highly linearly redundant (r>0.90), whereas Morgan fingerprint densities show near-zero linear correlation (|r|<0.025) with all six EC targets.
- Target-specific threshold optimization combined with post-hoc conformal calibration (Mondrian CP) is presented as an essential safeguard for imbalanced biological multi-label pipelines — though it is recommended, not implemented, in this study.
Strengths
Real and operationally relevant problem
Default τ=0.50 thresholding under label imbalance is a genuine failure mode in applied annotation pipelines (Section 1), and the paper's target-by-target breakdown (Sections 3.3.1–3.3.2) makes the pathology concrete rather than anecdotal.
Failure modes are quantified at per-target granularity
EC2 trivial all-positive behavior (recall 100.00%, TN=1, FP=2,806) and EC6 total collapse (TP=0, FN=2,129) are precisely documented with confusion-matrix counts, and Section 3.3.2's recall decay (EC3 16.34%, EC4 18.25%, EC5 10.55%) tracks prevalence monotonically.
Decouples ranking power from threshold artifacts
Multi-label structure characterized, not assumed
Weaknesses
The classifier is never identified
Single dataset, single model, single threshold — sweeping conclusions
Proposed remedies never implemented
Citation hygiene and structural sloppiness
Overstated significance of weak ranking
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an empirical diagnostic study, not a method paper. The authors evaluate how a fixed decision threshold (τ=0.50) behaves under class imbalance in multi-label prediction of the six primary Enzyme Commission (EC) classes, on N=14,096 annotated compounds. Their central claims: (1) a 77.16% mean accuracy coexists with macro F1 0.3976 and macro recall 0.3872 (the "Accuracy Paradox"); (2) majority heads over-predict (EC2: recall 100.00% with TN=1) while minority heads decay (EC3–EC5 recall 16.34%/18.25%/10.55%) up to total collapse (EC6: TP=0, FN=2,129, despite ROC-AUC 0.5857); (3) topological descriptors are linearly redundant (Chi1–Chi2v r=0.971, Chi1–ExactMolWt r=0.960) while FpDensityMorgan features show |r|<0.025 with all targets; (4) per-target threshold optimization plus conformal calibration should be standard safeguards. Claims 1–3 are supported by the reported numbers; claim 4 is asserted but never demonstrated.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Accuracy Paradox in enzyme function prediction?
It is the situation where high overall accuracy conceals near-useless per-class performance. In this paper, a multi-label model predicting six Enzyme Commission (EC) classes scores 77.16% mean accuracy, but macro F1 is only 0.3976: EC2 is predicted positive for nearly everything (recall 100%, one true negative), while EC6 gets zero positive predictions (recall 0%). The default 0.50 decision threshold, combined with class imbalance, causes both failures.
Does the paper release code or data?
No public code or data link is provided. Despite '[With Code]' in the title, the Code Availability section only says the code can be requested from the corresponding author by email. The dataset's public source is not named in the text, and the classifier used is never identified.
How can recall be 0% when the model still has discriminative power for EC6?
Thresholding and ranking are separate steps. The model's ROC-AUC for EC6 is 0.5857, meaning its probability scores rank actives slightly above inactives. But if every predicted probability falls below the 0.50 cutoff, all 2,129 active compounds are labeled inactive — recall drops to 0% even though the ranking is not literally random (though 0.5857 is weak).
What fixes do the authors recommend?
Two post-processing steps: (1) optimize a separate decision threshold per EC class, e.g., by maximizing F-beta or precision-recall performance, instead of using 0.50 everywhere; and (2) add post-hoc probability calibration (Platt scaling) with class-conditional conformal prediction (Mondrian CP) to obtain per-class coverage guarantees. The paper recommends both but does not implement or benchmark either one.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The per-target results table referenced in Section 3.2 ('the table above') never appears in the extracted text; per-target numbers were taken from prose in Sections 3.3.1–3.3.2, which quote the same confusion-matrix counts and metrics.
- The Pearson correlation definition (Section 2.3) appears formula-less in the extracted text; cited values (r=0.971, |r|<0.025) come from Section 3.1 and the Conclusion.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.