Embedding Models Measure in Peculiar Ways
cs.CL · cs.LG
Quick verdict
Embeddings weakly mirror physical measurements across 24 models; string similarity dominates. Sound triangulation, but bare strings and weak stats limit reach.
Summary
The paper asks whether text embedding models encode physical measurement relations — length in meters, mass in kilograms, volume in liters, time in seconds — as faithful metric spaces. Using cosine similarity over short measurement phrases (e.g., '1 meter' vs. '105 centimeters'), the authors evaluate 24 widely used embedders, from Sentence-Transformers baselines through BERT/ModernBERT encoders to LLM-derived models (Qwen3-Embedding 0.6B-8B, embeddinggemma-300m), across five value scales (local 0-10, medium 0-1000, log to 100,000, signed ±100, scientific) in digit and word form. They introduce PhysScore, a Kendall's tau / pairwise-accuracy benchmark over distance relations. All 24 models misalign badly: the best, multilingual-e5-large-instruct, reaches tau 53.23 and none exceeds 54; kilogram and the medium scale are hardest, and some model-scale cells invert the true ordering. Unit conversions (meter-centimeter, gallon-liter) are similarly fuzzy. A linear probe on embedding differences improves alignment only marginally (mean +3.0 tau; best +12.3 for e5-large-v2), rejecting pure miscalibration. Correlations with character/token Levenshtein versus true numeric distance show surface form dominates (embeddinggemma-300m: 51.6 vs. 3.3). Main limitations: bare out-of-context English strings only, no statistical significance testing, and a PhysScore inflated by trivially resolved diagonal pairs.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across all 24 tested embedding models, cosine similarity over measurement phrases only weakly reflects physical distance relations (best PhysScore Kendall's tau = 53.23 for multilingual-e5-large-instruct; no model exceeds 54).
- Model scale, family, architecture, embedding dimension, and release recency show no clear positive effect on measurement alignment (Sections 4, 6; e.g., Qwen3-0.6B outperforms Qwen3-8B).
- Embedding similarity of measurements is more strongly associated with character- and token-level string similarity than with true numerical distance (e.g., embeddinggemma-300m: 51.6 vs. 3.3 in Table 4).
- Linear-probe recalibration yields only marginal gains (mean +3.0 tau, max +12.33 for e5-large-v2) and no model passes 54, rejecting miscalibration as the explanation (Section 7.1).
Strengths
Objective ground-truth testbed
Physical quantities admit unique equivalence and distance ('1 meter' = '100 centimeters'), unlike aspect-dependent STS similarity (Section 1, citing Tversky 1977); this makes the idealized geometry (Figure 1A) genuinely well-defined.
Broad, current model coverage
24 models across four families (Appendix A.1): Sentence-Transformers classics, e5/bge/mxbai/granite contrastive encoders, ModernBERT-based granite-r2/DenseOn, and LLM-derived Qwen3-Embedding 0.6B/4B/8B, harrier-oss, embeddinggemma-300m — enabling architecture, scale, and recency comparisons.
Hypothesis-driven triangulation
Clean rejection of the miscalibration hypothesis
Reproducibility
Weaknesses
Ecological validity of bare strings
No statistical inference
PhysScore construction inflates scores
Missing baselines
Unexplained table inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv:2609.20821, cs.CL/cs.LG) poses a sharply scoped question: do contrastively trained text embedding models — which claim to measure semantic similarity — faithfully encode relations among physical measurements whose equivalence and distance are objectively defined ('1 meter' = '100 centimeters')? The authors advance three claims: (1) measurement alignment is weak across all 24 tested models; (2) stronger, larger, or newer models do not clearly improve alignment, and neither does dimensionality or model family; (3) embedding similarity of measurements tracks lexical string similarity rather than numerical semantics, and recalibrating the similarity function does not repair it. The claims are clearly stated and, within the narrow stimulus regime studied, mostly supported by the evidence.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do text embedding models understand physical units like meters and kilograms?
According to this study of 24 embedding models, only weakly. On the paper's PhysScore benchmark the best model (multilingual-e5-large-instruct) reached a Kendall's tau of 53.23 out of 100, and no model exceeded 54, meaning rankings of physical distances are far from faithful. Unit conversions such as '1 meter = 100 centimeters' are also poorly reflected in cosine similarity.
Why do embedding models fail at numbers and measurements?
The authors show that embedding similarity of number phrases tracks surface string similarity (character or token overlap) far more strongly than true numeric distance — embeddinggemma-300m correlates 51.6 with character-level Levenshtein but only 3.3 with numeric proximity. Models largely encode numbers as text strings, and re-weighting the similarity function (a linear probe) does not fix it.
Which embedding model is best at physical measurements?
In this paper's PhysScore benchmark, multilingual-e5-large-instruct ranked first with Kendall's tau 53.23 and pairwise accuracy 76.61, followed by LaBSE (tau 51.31) and e5-large-v2 (tau 47.35). Even the best is substantially misaligned with physical relationships, and larger models like Qwen3-Embedding-8B (tau 32.27) scored in the lower half.
Is the poor measurement alignment just a similarity calibration problem?
No. The paper trains a linear probe on embedding differences to re-weight dimensions and finds only small gains — an average of +3.0 Kendall tau points, with the largest single gain of +12.33 for e5-large-v2 (35.7 to 48.1) — and no model passes 54 tau, so miscalibration of cosine similarity does not explain the misalignment.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Heatmap and conversion figures (Figures 1-3 and appendix Figures 4-47) could not be visually inspected from the extracted text; qualitative pattern judgments rely on captions and the authors' descriptions.
- The Section 7.1 train/test split protocol and probe hyperparameters are not detailed in the extracted text, limiting assessment of Table 3.
- All numeric values were taken from the extracted tables, which appear complete; any detail conveyed only inside figure images (e.g., exact band artifacts) was not verifiable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.