Recovering Temporal and Geographic Signals from Language Model Embeddings
Esteban Feuerstein, Victoria Klimkowski, Juan Manuel Ortiz de Zarate, Federico Hernán Suaiter
cs.AI · cs.IR
Quick verdict
Black-box projection recovers time and geo order from embeddings (τ up to 0.87); suggestive, but no significance tests and topical-drift confound unresolved.
Summary
The paper asks whether off-the-shelf text embeddings encode temporal and geographic structure, and answers with a lightweight black-box technique: given a few seed pairs at opposite ends of a dimension, a 'd-ness' axis is formed from vector differences, and texts are ranked by cosine projection onto it — no probes, no training, no model internals. On U.S. State of the Union speeches (1988–2019), projection scores correlate strongly with delivery year (Kendall's τ up to 0.87 for voyage-lite-01 on Republican speeches, 0.79 overall; ada-002 trails at 0.59 overall), with K≥3 seed pairs needed for stability. A daily news corpus (May 2020–October 2025) shows the 'arrow of time' is distributed: chronologically sorted random seeds beat random controls, larger time windows sharpen the signal, and models interpolate within a local interval but fail to extrapolate. Geographically, landmark embeddings correlate with real coordinates (latitude τ=0.615, longitude τ=0.795 on WorldPlaces; one longitude arc reaching τ=0.848), hinting at circular longitude representation. Limitations: correlations come without significance tests or confidence intervals; creation time and focus time coincide in all datasets; topical drift is an admitted confound addressed only by 'preliminary' masking controls; several 'conclusive' experiments are asserted but not shown. Code and customized datasets are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Output embeddings of commercial language models encode a recoverable temporal axis: projection onto seed-pair-derived directions correlates with document dates (Kendall's τ up to 0.87 on Republican State of the Union speeches with voyage-lite-01).
- The 'arrow of time' is distributed across a corpus rather than an artifact of extremes — chronologically sorted random seeds yield higher correlations than random controls on a daily news corpus.
- The learned temporal representation is local: models interpolate within a seed-defined interval but fail to extrapolate beyond it.
- Place-name embeddings correlate with real latitude and longitude (τ=0.615/0.795 on WorldPlaces), and longitude may be represented circularly (one arc at τ=0.848).
- The method is fully black-box and model-agnostic: it requires only embeddings, with no access to weights, activations, probes, or training.
Strengths
Genuinely black-box, training-free protocol
§3.1 (Eqs. 1–3) defines the axis purely from K seed pairs and cosine projection on output embeddings, enabling evaluation of API-only models (ada-002, text-embedding-004, Voyage family) that probe-based work like Gurnee & Tegmark [16] cannot touch; the authors correctly scope their novelty as first application of d-ness scoring to temporal/geographic dimensions on text embeddings (§2).
Signal-vs-noise control design
§3.2's Random Control (seeds assigned to early/late regardless of date) and Random Sorted strategies, plus §3.3 chunking, actively test whether the temporal signal is distributed rather than an artifact of corpus extremes; the Random Sorted > Random Control contrast (Figs. 4–5) and the interpolation-yes/extrapolation-no finding (Fig. 6) map the signal's boundaries honestly.
Candid limitation discussion
Released code and datasets
Breadth across dimensions and models
Weaknesses
No inferential statistics anywhere
Central confound left unresolved
Unshown 'conclusive' evidence
Model identification inconsistencies
Tiny, ad hoc geographic evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.05721) tests whether **output embeddings** of commercial language models encode temporal and geographic structure, and answers without probes, fine-tuning, or model internals: a "d-ness" projection (§3.1, Eqs. 1–3) builds an axis d = Σ(p₁−p₂) over K seed pairs placed at opposite ends of the target dimension, then ranks any text by cosine projection onto d. This positions the work between the white-box probing of Gurnee & Tegmark [16] (space/time neurons in Llama-2) and supervised dating models like TempoBERT [41], with black-box applicability to API-only models as the selling point (§1–2). The authors correctly disclaim novelty of the projection idea itself (crediting Bolukbasi et al. [4], Waller & Anderson [45]); their claim is its first application to temporal/geographic dimensions on text embeddings. Central claims: (1) projection scores track document dates (τ up to 0.87); (2) the "arrow of time" is distributed, not an extremes artifact; (3) the temporal representation is local (interpolation yes, extrapolation no); (4) latitude/longitude are recoverable from place names and longitude may be circular.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do LLM embeddings encode time and space?
This paper provides evidence they do, at least coarsely. Using only off-the-shelf embeddings from API models (OpenAI ada-002, Google Gemini text-embedding-004, Voyage models), the authors project texts onto axes built from a few example pairs and recover document dates (Kendall's τ up to 0.87) and place coordinates (τ≈0.6–0.8). It complements earlier probe-based work (Gurnee & Tegmark, 2023) by requiring no access to model internals.
What is d-ness scoring?
A projection technique from word-embedding research: pick seed vector pairs that differ mainly on a target dimension (e.g., old vs. new texts, north vs. south places), sum their difference vectors into a direction, and score any text by cosine similarity to that direction. This yields a ranking along the dimension with no training, probes, or model access — only the embedding outputs.
What are the main limitations of this paper?
Correlations are reported without significance tests or confidence intervals; datasets conflate creation time with focus time; the recovered 'time' signal may partly reflect topical drift between eras, and the masking-control experiments addressing this are only described as preliminary with no numbers; several supporting experiments are asserted to be 'conclusive' but not shown; and the geographic evaluations rest on very small samples.
Is the code and data for this paper available?
Yes. The authors state that the method code and the customized datasets are published on GitHub at github.com/FedericoSu/BIICC-2024. Note the news corpus comes from an unnamed 'major newspaper' and may not be redistributable, and some tested API models (e.g., ada-002) have since been deprecated.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal text (axis labels, tick values, and point labels in Figs. 1, 2, 4, 5, 6, and 9) is garbled in the PDF extraction, so figure-level parameters (sample counts, axis ranges, per-city values) could not be verified; all numbers cited in this review come from the prose, which carries the headline results.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.