TEMPS: Temporal Sentence Embeddings for Temporal Information Retrieval
Mourad Hassani, Julien Romero, Amel Bouzeghoub, Christian Jacquelinet
cs.CL · cs.AI
Quick verdict
Proposes TTS task and TEMPS, a weakly-supervised Gaussian temporal branch for dense retrievers; gains are real on TimeQA/TS-Retriever but mixed on TempReason.
Summary
The paper formulates Temporal Textual Similarity (TTS) — scoring whether two anchor-date-conditioned texts express compatible times, independent of topic — and builds TEMPS, a ~27M-parameter temporal branch that attaches to frozen semantic retrievers. A pipeline grounds synthetic TimeML expressions and HeidelTime-tagged Wikipedia text to day-level intervals, moment-matches them into Gaussians, and uses the asymmetric Gaussian-KL measure (adopted, not new) as ranking supervision, yielding 60M tuples trained with a CoSENT-style loss. At inference, temporal and semantic scores fuse after per-query min-max normalization, with α calibrated per benchmark on a held-out validation split. Across TimeQA, TS-Retriever, and TempReason, TEMPS improves MRR for every backbone tested (e.g., MPNet on TimeQA 0.3008→0.5043), and on TS-Retriever it beats TSContriever at R@1 (25.39 vs 19.92) and P@1 (69.14 vs 58.63). TimeQA RAG with Qwen2.5-7B-Instruct improves EM for all retrievers. TempReason is mixed: Recall@5 drops for all backbones and a SUTime rule-based control wins on every metric there, which the authors report candidly. Key limitations: a single training seed (42), English-only evaluation, one RAG end-task, lossy single-Gaussian passage summaries, and per-benchmark α tuning.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- TTS, a task scoring temporal compatibility of anchored texts independent of topic, can be supervised at scale without manual labels via interval grounding and moment-matched Gaussian targets (60M tuples).
- Attaching TEMPS to frozen semantic retrievers improves MRR for every backbone tested on TimeQA, TempReason, and TS-Retriever (Table 1), though two of twelve MRR cells are not statistically significant.
- On TS-Retriever, E5base-v2+TEMPS surpasses the prior temporal SOTA TSContriever at R@1 (25.39 vs 19.92) and P@1 (69.14 vs 58.63) under a claimed protocol-matched comparison (Table 2).
- The learned temporal encoder, not rule-based grounding, drives gains on TimeQA and TS-Retriever (+SUTime control falls below baseline there), whereas the SUTime control wins every metric on TempReason.
- Retrieval gains transfer to downstream RAG: every +TEMPS retriever improves EM, F1, and containment on TimeQA with Qwen2.5-7B-Instruct (Table 3).
Strengths
Exemplary statistical transparency
Appendix C.7 reports paired Wilcoxon, paired t-test, and 10,000-resample bootstrap CIs for all 48 (benchmark × backbone × metric) cells with Bonferroni correction — 36 significant positive, 9 significant negative (all TempReason Recall/Precision), 3 non-significant — including failures of its own headline claim.
Clean rule-based control isolates the contribution
The +SUTime rows (§6.1, C.1) swap only the temporal representation, holding fusion, normalization, and α fixed; on TimeQA it falls 7.5–13.3% below the semantic baseline, proving the learned encoder — not any temporal signal — drives gains there.
Oracle study rules out trivial explanations
Strong, protocol-matched SOTA comparison
Practical and reproducible by design
Weaknesses
Headline claim is statistically thinner than stated
Single training seed
TempReason is a genuine weak spot
α is a per-benchmark operating point
One end-task, reused baselines, unclear module sharing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes two conceptual moves and one empirical program. Conceptually it defines **TTS (Temporal Textual Similarity)** — scoring whether two texts, each paired with an anchor date, express compatible times regardless of topic — as a temporal counterpart to STS. It then builds **TEMPS**, a ~27M-parameter temporal branch attachable to frozen dense retrievers. The empirical claims: (1) MRR improves for every backbone on TimeQA, TempReason, and TS-Retriever (Table 1); (2) on TS-Retriever, E5base-v2+TEMPS beats TSContriever at R@1 (25.39 vs 19.92) and P@1 (69.14 vs 58.63) under a claimed protocol-matched comparison (Table 2); (3) gains transfer to TimeQA RAG with Qwen2.5-7B-Instruct, where every +TEMPS retriever improves EM (Table 3); (4) the learned encoder, not rule-based grounding, drives gains on TimeQA/TS-Retriever. Unusually, the paper disclaims novelty for the Gaussian-KL score itself, locating its contribution in grounding and moment-matched supervision — a refreshingly precise novelty statement.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TEMPS and what problem does it solve?
TEMPS is a small (~27M-parameter) neural module that adds time-awareness to existing dense text retrievers. Standard retrievers match documents by topic but ignore when events happened, so they can return topically relevant but temporally wrong passages (e.g., the wrong WHO Director-General for a 2006–2017 query). TEMPS encodes a passage plus its anchor date as a Gaussian distribution over time and fuses a temporal similarity score with the usual semantic score.
Does TEMPS need manually labeled temporal training data?
No. Its supervision is generated automatically: temporal expressions from synthetic templates and HeidelTime-tagged Wikipedia are resolved to date intervals, converted to Gaussians by moment matching, and scored with an asymmetric Gaussian-KL formula. This produces about 60 million training tuples with no hand labeling, trained with a CoSENT-style ranking loss.
How much better is TEMPS than the previous state of the art (TSContriever)?
On the TS-Retriever benchmark, E5base-v2 + TEMPS reaches Recall@1 of 25.39 versus 19.92 for TSContriever, and Precision@1 of 69.14 versus 58.63, with higher recall up to R@100 (98.95 vs 94.57). Note the comparison reuses TSContriever's published numbers under a claimed-matched protocol rather than rerunning it.
Where does TEMPS not help?
On TempReason, TEMPS improves MRR but lowers Recall@5 for all four backbones (drops of 0.020–0.108), and a simple rule-based control (SUTime grounding combined with the same KL score and fusion) outperforms TEMPS on every metric on that benchmark. The authors attribute this to TempReason's regular query patterns, where regex-style matching already suffices.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–4 (architecture, data pipeline, α-sweep curves) are available only as extracted captions and prose; visual details could not be verified.
- Table 2's header alignment is garbled in the extraction; values were cross-checked against surrounding prose, but one or two column assignments could be off.
- Any supplementary material beyond the extracted text was not reviewed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.