WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data
Ji Soo Lee, Xilun Chen, Pierce Chuang, Ashish Shenoy, Jason Wei, Dohwan Ko, Hyunwoo J. Kim, Benoit Corda
cs.CL
Quick verdict
Solid dual-grounded benchmark (4,084 MCQs over 200 users' wearables) that separates 14 LLMs (19.6-72.9%); watch gold-label noise and QC-model overlap.
Summary
WearableQA is a benchmark of 4,084 ten-option multiple-choice questions built from the longitudinal wearable records of 200 real users (16 daily metrics, up to 500 days each), together with 17-biomarker blood panels and demographics. Questions span 16 types along two axes—data versus health reasoning and single- versus cross-signal reasoning. Gold answers come from a dual-grounding framework: relationships taken from peer-reviewed literature (11 anchor papers, each supported by at least two independent studies) or population patterns admitted only after statistical gates (effect size |ρ|≥0.5, bootstrap and leave-one-out robustness, cross-user null test with FDR≤0.20), then computed deterministically from each user's raw measurements. Evaluating 14 LLMs under chain-of-thought, accuracy ranges from 19.6% (Llama-3.2-3B) to 72.9% (Gemini-3.1-Pro) against a 10% chance level. Most models do better on health than data reasoning, and cross-signal questions remain hard; agentic Python access lifts GPT-5.4 by 20.1 points (51.2→71.3). Limitations include a possible gold-label noise floor (population grounding tolerates up to 20% false discoveries), thinly documented human verification, overlap between QC and evaluated models, and sparse detail on cohort provenance, consent, and de-identification.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- WearableQA is the first benchmark evaluating LLM reasoning over real users' longitudinal wearable records jointly with blood biomarkers and demographics, unlike prior synthetic or simulated settings.
- A dual-grounding framework (11 literature anchors plus statistically gated population patterns) yields deterministic, reproducible gold labels at scale.
- Current LLMs remain far from solving the benchmark: 19.6%-72.9% accuracy across 14 models against a 10% chance level, with data reasoning and cross-signal integration the dominant failure modes.
- Text serialization choice barely affects accuracy while agentic Python access lifts GPT-5.4 by 20.1 points (51.2→71.3), showing computation rather than input format is the bottleneck.
- Models over-rely on priors for familiar signal pairs — e.g., Gemini-2.5-Pro scores +51.5 points higher on definitional than empirical pairs of matched strength (Table 2).
Strengths
Deterministic, auditable gold labels
The discover-then-label pipeline (§3.4) expresses each answer as a computation graph over a primitive library (mean, cv, lagged_correlation, slope, rolling_mean); the program evaluates all ten options and items are kept only when the gold option uniquely satisfies it, making labels reproducible in principle.
Unusually strict statistical gates for pattern admission
§3.3 requires |ρ|≥0.5 with consistent sign in both window halves, a ≥0.15 winner margin, survival in ≥80% of 30 bootstrap resamples and under leave-one-out day removal, dead-band rejection, plus a cross-user null test bounding empirical FDR at 0.20.
Diagnostic value demonstrated, not just a leaderboard
Actionable tooling finding
Grounding ablation confirms data dependence
Weaknesses
Label-noise ceiling from the FDR gate
QC/eval model circularity
Clinical validity of health-reasoning items is by construction
Underspecified data provenance and ethics
Small-n strata behind a headline analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
WearableQA (arXiv 2609.05405; Meta, KAIST, Korea University) is a benchmark paper for LLM evaluation over real-world wearable health data. The authors claim four contributions (§1): (1) 4,084 ten-option MCQs grounded in 200 real users' longitudinal wearable records — 16 daily metrics across cardio-fitness, activity/energy, sleep and stress domains, up to 500 days per user — plus a 17-biomarker blood panel, demographics, and cohort-reference percentiles; (2) a dual-grounding construction framework (literature + population) with deterministic ground truth; (3) a diagnostic 2×2 taxonomy (data vs health reasoning × single vs cross-signal) spanning 16 question types (2.7k data / 1.4k health; 1.7k single / 2.4k cross-metric); and (4) a 14-model evaluation showing the benchmark discriminates capability (19.6%–72.9% vs a 10% chance floor) and diagnoses failure modes. The scope is benchmark construction and evaluation; no new modeling method is proposed.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the WearableQA benchmark?
WearableQA is a set of 4,084 ten-option multiple-choice questions built from the real longitudinal wearable records (up to 500 days of 16 daily metrics such as steps, resting heart rate, and sleep) of 200 users, combined with 17 blood biomarkers and demographics. It tests whether LLMs can both compute over noisy personal health data and interpret it physiologically.
Which models perform best on WearableQA?
Under chain-of-thought evaluation, Gemini-3.1-Pro leads at 72.9% accuracy, followed by Claude-Opus-4.6 at 60.2% and GPT-5.4 at 51.2%. The best open-weight model is Gemma-4-26B-A4B at 42.5%, and the weakest is Llama-3.2-3B at 19.6%, against a 10% random-guess baseline.
Is the WearableQA dataset publicly available?
The paper points to a GitHub repository (facebookresearch/WearableQA) and a Hugging Face dataset (facebook/WearableQA). The reviewed text does not describe the licensing terms or the consent and de-identification process behind the user data, so release scope should be checked at the source.
Why do LLMs struggle with wearable data according to this paper?
The breakdowns show models are weakest when they must compute answers directly from noisy longitudinal numbers (data reasoning, e.g., GPT-4o at 25.3%) and when integrating several signals. Giving models a Python tool raises GPT-5.4 from 51.2% to 71.3%, indicating that calculation rather than reading the data is the main bottleneck.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided extraction ends at Table 7 (p. 18) with no section following; any appendix content beyond §A.3 was not available for review.
- Values inside figures (Figs 1, 4-8) were read only from captions and axis labels; per-question-type numbers cited here come from Table 6.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.