What an odour descriptor corpus can and cannot measure: valence, attenuation, and the ceiling of the public record
Stylianos Kampakis, Fabio Rovai
cs.LG
Quick verdict
Audits four odour descriptor corpora: disagreement is criterion not construct, and valence—not more data—sets the ceiling. Rigorous and largely convincing.
Summary
The paper audits four public odour descriptor corpora (Dravnieks 1985, Keller & Vosshall 2016, Leffingwell, GoodScents) distributed via Pyrfume, asking whether a shared descriptor word measures the same construct across sources and how much a pooled corpus can support. The authors prove that conditioning on the molecule makes McNemar's test the exact conditional test of the corpus effect, machine-checked in Lean 4. Corpora disagree heterogeneously across descriptors (I²=80%, pooled OR 1.14) and non-uniformly with labelling breadth (z=17.2), so no single offset repairs pooling; median tetrachoric agreement 0.795 versus median κ 0.413 indicates disagreement about threshold (criterion) rather than construct. Measuring the ceiling against a human panel's own reliability, Morgan fingerprints plus the full RDKit descriptor block reach 32.9% of achievable, rising to 33.9% with every label from two merged corpora; one pleasantness rating per molecule reaches 54.6%, replicated at 57.1% with a 1985 instrument. Valence inferred from labels (ρ=0.457) yields only 16.6%, so it must be measured. Main limitations: the criterion rests on one panel dataset with a 64-molecule external replication, the trade corpora contain no human observers, and analysis code is not publicly released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Corpus disagreement on shared descriptor words is heterogeneous across descriptors (I²=80%) and non-uniform in labelling breadth (z=17.2), so neither a global nor a per-descriptor offset repairs pooling.
- The disagreement is criterion rather than construct: median tetrachoric agreement is 0.795 against a median κ of 0.413, and 36 of 109 estimable descriptors show large DIF on the ETS delta scale.
- Against a human panel's own reliability, Morgan fingerprints plus the full RDKit block reach 32.9% of achievable and adding every label from two merged corpora only 33.9%, with the gap insensitive to model capacity, encoding, molecules, and vocabulary.
- The missing variance is valence: one pleasantness rating per molecule reaches 54.6% of achievable (57.1% on an independent 1985 instrument), while valence recovered from descriptor labels reaches only 16.6%, so valence must be measured at source.
- Five raters already exceed structure plus the entire descriptor record (39.2% vs 33.6%), and the benefit saturates at fifteen to twenty raters.
Strengths
Exact conditional inference, formally verified
Section 2.2 proves that conditioning on the molecule-pair total removes molecule and molecule-by-descriptor nuisance effects, making McNemar's test exact; 20 theorems are machine-checked in Lean 4 against Mathlib with no sorry — unusual rigour for a data-audit paper.
Criterion/construct separation is quantified, not asserted
Section 3.2's saturated bivariate probit (exactly identified, three parameters for three free cells) yields median tetrachoric 0.795 (bootstrap 0.768–0.821) vs median κ 0.413; the authors preempt the rank-ordering objection by reporting ρ-vs-κ rank correlation of 0.865.
Honest self-correction and pre-registered confirmation
Ceiling claim triangulated against three failure modes
Actionable pricing of the remedy
Weaknesses
Analysis code not released
Single criterion dataset with a thin external replication
Share-of-achievable metric denominates everything on one reliability estimate
Trade-corpus independence unestablished
No downstream demonstration
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement audit of the public odour descriptor record, not a modelling paper despite the cs.LG tag. The authors take four corpora distributed via Pyrfume — Dravnieks (1985; 138 molecules, 146 descriptors), Keller & Vosshall (2016; 480 molecules, 20 slider items, 55 subjects), Leffingwell (3,487/113), and GoodScents (4,565/666) — pinned by SHA-256 manifest, and ask two questions in sequence: do shared descriptor words measure the same construct across sources, and what is the ceiling of what any pooled corpus can measure. Five headline claims: (1) corpus disagreement is heterogeneous across descriptors (I²=80%) and non-uniform in labelling breadth (z=17.2), so no single offset repairs pooling; (2) the disagreement is criterion (threshold) not construct — median tetrachoric 0.795 vs median κ 0.413; (3) against a human panel's own reliability, structure plus the full descriptor record reaches only about a third of achievable; (4) the missing variance is valence, which must be measured (54.6% for one rating) rather than inferred (16.6% from labels); (5) five raters already exceed the entire public record. The claims are ambitious but, on the presented data, largely carried.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
It audits four public odour descriptor corpora and finds that their disagreement is mostly about how readily a label is applied (criterion), not about what the label means (construct). More importantly, molecular structure plus all public descriptor labels explain only about a third of what a human panel can reliably measure, while a single pleasantness rating per molecule explains over half — so odour valence must be collected, not inferred from quality words.
Why can't odour descriptor labels be used to infer pleasantness?
The paper measures this directly: corpus labels predict panel pleasantness at ρ=0.457 out-of-fold, but a valence coordinate built from that prediction reaches only 16.6% of achievable, against 53.9% for measured valence on the same molecules and folds — an information gap that no modelling choice in the study closed.
How many human raters are needed to measure odour pleasantness well?
According to Section 3.7, five raters already outperform molecular structure plus every descriptor label in two merged corpora (39.2% vs 33.6% of achievable), and the benefit saturates: twenty to twenty-seven raters buy only about 1.7 more points, so fifteen to twenty raters is the practical sweet spot.
Is the paper's analysis code available?
No. The paper states raw corpus rows are not redistributed for licence reasons and that the analysis code and derived reports are not publicly released. Inputs are pinned via a SHA-256 manifest and an appendix claims sufficient detail to reimplement, including seven pitfalls that produced incorrect results during development.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The appendix (detailed procedures, including the seven development pitfalls) is referenced in the availability statement but is not part of the provided text.
- Figure graphics extract as interleaved text with captions (Figures 1–3), so visual layout is garbled; every number cited in this review appears in the body text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.