PhenoBench: Mapping What a Deeply Phenotyped Human Cohort Can Tell Us
Gal Sapir, Alon Diament, Adva Wolf, Doron Yaya-Stupp, Dikla Gelbard Solodkin, Dana Azouri, Anat Etzion-Fuchs, Guy Lutsker, et al.
cs.LG · cs.AI
Quick verdict
PhenoBench turns a 13k-person deep-phenotyping cohort into 90 contract-bound tasks; execution is rigorous and honest, but headline gains are tiny and post hoc.
Summary
PhenoBench is an executable benchmark built on the Human Phenotype Project, a deeply phenotyped cohort with over 13,000 participants at the initial visit, that converts clinical, imaging, molecular, and wearable measurements into 90 clinically grounded tasks across 15 domains and 26 input modalities, each fixing target, population, timing, and allowed information with an explicit evaluation contract (split, metric, baseline, claim boundary). The authors demonstrate it three ways. First, across 160 matched regression comparisons on 52 tasks, six pretrained tabular foundation models (TabPFN 3.5, TabDPT, Google TabFM, TabICL, TabPFN v2, TabSwift) ranked above tuned XGBoost/CatBoost and ridge under a single-estimator, bounded-tuning protocol, but the mean advantage over ridge was only 0.0103 R² (95% task-bootstrap CI 0.0071–0.0136), positive in 43 of 52 tasks. Second, frozen Chronos-2 embeddings underperformed engineered features from the same CGM and sleep signals in 25 of 31 and 30 of 32 matched comparisons. Third, 14 language models evaluated on 40 tasks without cohort-specific fitting rarely surpassed ridge or logistic probes fitted on the same fields (8 of 304 phenotype-recovery comparisons), with Gemini 3.7 Flash leading at an 82.5% win rate on the 11 shared tasks. Key limitations: the cross-task synthesis is post hoc, tasks share participants and outcomes, the tabular protocol is narrow, and the LLM evaluation uses validation data, one run, and 150–480 participants per task.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PhenoBench provides 90 executable, versioned clinical tasks with evaluation contracts that make comparisons of measurements and models on the HPP cohort auditable and cumulative.
- Under a fixed single-estimator protocol with bounded tuning, six pretrained tabular foundation models ranked above the evaluated task-specific baselines, with a mean advantage over ridge of 0.0103 R² (95% task-bootstrap CI 0.0071–0.0136, positive in 43/52 tasks).
- Frozen zero-shot Chronos-2 embeddings underperformed engineered features from the same CGM and sleep signals (25/31 CGM and 30/32 exact-intersection WatchPAT comparisons), with gaps narrowing as training size grew.
- Without cohort-specific fitting, 14 language models made informative predictions on some of 40 tasks but rarely surpassed ridge or logistic probes fitted on the same fields (8/304 phenotype-recovery comparisons), showing task-specific capability gaps and shared calibration failures.
Strengths
Explicit, executable evaluation contracts
Every task fixes target, eligible population, timing, and allowed information; contracts pin split (stable participant-hash 70/15/15), metric, baseline, and claim boundary (§2.1, §3.1, Appendix E.4). Human-readable Task Cards document clinical rationale, and the leakage validator screens LLM evidence packets with forbidden-term scans (Appendix E.6).
Statistical discipline unusual for benchmark papers
The primary summary bootstraps whole tasks (5,000 draws, seed 20260828) to respect within-task dependence; pairwise claims require agreement of separately Holm-adjusted Wilcoxon and exact sign tests; Page's test is run over all 720 FM permutations and correctly labeled descriptive; η² decomposition (task identity 0.5421) and sensitivity analyses (equal-family merging of TabPFN versions) are published (§2.3, Appendix B.2–B.3).
Honest retention of negative and awkward results
Provenance and versioning rigor
Extensible submission workflow
Weaknesses
Tiny headline effect presented as the main quantitative result
Post hoc, non-preregistered cross-task synthesis on dependent cells
Protocol asymmetry may handicap classical baselines
LLM evaluation is descriptive and protocol-narrow
Commercial affiliation and self-governed benchmark
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PhenoBench (arXiv 2609.06080v2) packages the Human Phenotype Project (HPP) — a prospective cohort with over 13,000 initial-visit participants spanning clinical assays, CGM, WatchPAT sleep studies, ECG, DXA, retinal imaging, NMR metabolomics, and microbiome — into a versioned benchmark of 90 tasks across 15 clinical domains and 26 input modalities (§2.1, Figure 1). Each task fixes target, eligible population, timing, and allowed information; an evaluation contract pins split, metric, baseline, and claim boundary. Three demonstrations are reported: (1) 160 matched regression cells over 52 tasks comparing six pretrained tabular foundation models (TabPFN 3.5, TabDPT, Google TabFM, TabICL, TabPFN v2, TabSwift) against ridge, tuned XGBoost/CatBoost, and RealMLP; (2) same-signal comparisons of frozen Chronos-2 embeddings versus engineered CGM/sleep features; (3) PhenoBench-LLM, 14 language models across 40 tasks. Headline claims: pretrained tabular models rank above the evaluated baselines (mean +0.0103 R² over ridge, 95% task-bootstrap CI 0.0071–0.0136, positive in 43/52 tasks); engineered features beat generic time-series embeddings; LLMs without cohort-specific fitting rarely surpass fitted probes (8/304 phenotype-recovery comparisons), led by Gemini 3.7 Flash at 82.5% win rate on the 11 shared tasks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PhenoBench?
PhenoBench is an executable benchmark built on the Human Phenotype Project, a deeply phenotyped Israeli cohort with more than 13,000 participants at the initial visit. It defines 90 clinical tasks (15 domains, 26 input modalities), each with a fixed target, eligible population, timing, and evaluation contract specifying split, metric, baseline, and the boundary of what a result can claim.
Do pretrained tabular foundation models beat classical machine learning on clinical data, according to this paper?
They ranked higher in aggregate across 160 matched regression comparisons on 52 tasks, beating tuned XGBoost and CatBoost under a single-estimator protocol. But the mean advantage over ridge regression was only 0.0103 R² (95% CI 0.0071–0.0136), and the protocol constrained the tree baselines to a fixed 12-candidate tuning grid, so the result is protocol-specific and the practical gain is small.
Can large language models predict health measurements from patient data without being trained on it?
The paper evaluated 14 LLMs on 40 cohort tasks using fixed evidence packets (demographics, glucose-monitoring summaries, blood metabolomics). LLMs were informative on some tasks but rarely beat simple ridge or logistic models fitted on the same fields — only 8 of 304 phenotype-recovery comparisons — and showed calibration failures such as answering on the wrong measurement scale.
Is the Human Phenotype Project data used in PhenoBench public?
No. Individual-level participant data are de-identified and access-controlled. The public release includes the task catalog, Task Cards, synthetic input/output examples, and a container-submission path so external researchers can evaluate their models under the same contracts without receiving participant data.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: middle sections marked [...sampling gap...] were skipped, so some appendix and methods details between visible blocks were not inspected; judgments rely on the visible head/tail, Appendices B–G, and figure captions.
- Figures were assessed only through their embedded text descriptions and caption text; visual layout and plotted detail could not be inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.