VoiceLongMemEval: Do Assistants Remember How You Sounded?
Ramit Pahwa, Parivesh Priye, Apoorva Beedu
cs.AI · cs.LG
Quick verdict
VLME benchmarks paralinguistic long-term memory with a credible adversarial gate; striking affect gaps, but only oracle-regime eval and unreleased data.
Summary
The paper introduces VoiceLongMemEval (VLME), a benchmark testing whether conversational assistants retain how something was said—emotion labels, prosody, voice events—across multi-session histories, not merely what was said. Built on LongMemEval records, its 523 items comprise 202 adversarially gated taxonomy questions (three-stage gate using a Qwen2.5-72B adversary; no non-abstention core item is blind-solvable) plus 181 'nuanced' and 140 'indirect' LLM-generated questions that were not blind-attacked. Across eight models (Claude Opus 4.8, GPT-5.5, Sonnet 4.6, five open-weight systems), descriptive metadata lifts accuracy by +0.089 to +0.383 over blind (all p<0.001). Ablations show emotion labels are the strongest single cue, structured tags beat natural-language descriptions (Opus 0.757 vs 0.589), and chain-of-thought cannot substitute. A retrieval-time hint largely rescues ungated indirect items (Opus blind 0.169 to 0.557) but adds only +0.079 on gated items, isolating a controlled metadata contribution of +0.067. On Dia-synthesized audio, Qwen2-Audio (0.354) and Qwen2.5-Omni (0.412) exceed the blind baseline (0.325), while a Whisper-to-Opus cascade falls to 0.254. Main limitations: all evaluation sits in the ~10-15k-token oracle regime (the designed ~100k-token regime is untested); audio evaluation covers only two 7B models on evidence-only clips with partially human-verified emotion fidelity; code and data are promised only upon acceptance.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Current long-term conversational memory benchmarks test only what was said; VLME adds items whose answers are recoverable exclusively from paralinguistic metadata, verified by a three-stage adversarial gate.
- Text-track paralinguistic metadata improves accuracy by +0.089 to +0.383 over blind transcripts across eight models (all p<0.001), with the gap growing for stronger models.
- Emotion labels are the most informative cue and structured tags beat natural-language descriptions (Opus 0.757 vs 0.589); chain-of-thought prompting cannot substitute for missing metadata.
- A retrieval-time hint is largely interchangeable with annotation on ungated questions (blind 0.169→0.557 on indirect) but not on gated ones (+0.079), isolating a controlled metadata contribution of +0.067.
- Audio-native 7B models (0.354-0.412) extract delivery cues from speech better than a Whisper→frontier-LLM cascade (0.254) on identical clips, quantifying the ASR paralinguistic deficit.
Strengths
Adversarial gating is rigorous and empirically validated
Section 3.3's three-stage gate (G1 blind-unsolvable with a Qwen2.5-72B adversary and lexical-trap judge, G2 aware-solvable, G3 surface-clean) yields 0/175 non-abstention core items blind-solvable; Section 5.4 then shows gated 202Q items resist a hint attack (+0.079 blind lift) while ungated indirect items do not (+0.388), an elegant external validation of the gate design.
Controlled attribution of the metadata effect
Section 5.4's three controls (scrambled context 0.043, cross-judge 0.600, wrong-metadata+hint 0.564 vs descriptive+hint 0.631) isolate a net +0.067 metadata-content contribution, and the wrong-metadata ablation in Table 4 (Opus 0.228 vs blind 0.193) shows models genuinely read annotations rather than benefiting from their presence.
Breadth and robustness of evaluation
The cascade result is a practically valuable finding
Unusually honest limitations section
Weaknesses
The long-memory claim is not actually tested
61% of the corpus was never blind-attacked
Abstract's hinted-gap range is selectively reported
Audio channel fidelity is weakly verified
Internal numeric inconsistencies and LLM-judge reliance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
VoiceLongMemEval (VLME, arXiv 2609.00570, NeurIPS 2026 workshop submission) extends the long-term conversational memory line (LongMemEval [2], LongMemEval-V2 [4]) with a paralinguistic axis: every item's answer depends on emotion labels, prosody tuples, voice events, or acoustic delivery descriptions attached to user turns, never on the words alone. The corpus totals 523 items over 326 annotated evidence sessions: a 202-question adversarially gated taxonomy core (six types plus 27 abstention variants), plus LLM-generated 'nuanced' (181) and 'indirect' (140) families. Headline claims: (1) a pervasive affect gap of +0.089 to +0.383 across eight models (Table 2), (2) a cue hierarchy with emotion labels strongest and structured tags beating prose, and (3) a cascade deficit — Whisper→frontier-LLM losing to 7B audio-native models (Table 8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'affect gap' in VoiceLongMemEval?
It is the accuracy difference between answering with paralinguistic metadata in the transcript (the 'descriptive' render, e.g., stage directions like 'slow, low pitch, sighing') versus the plain transcript alone (the 'blind' render). Across the eight models tested it ranges from +0.089 (Gemma 3-12B) to +0.383 (Claude Opus 4.8) on the 202 core questions, showing models cannot recover how something was said from the words alone.
Is the VoiceLongMemEval dataset publicly available?
Not yet. The paper (arXiv 2609.00570, a NeurIPS 2026 workshop submission) states that code and dataset 'will be made available upon acceptance.' The corpus totals 523 questions over 326 annotated evidence sessions, including machine-generated (Dia TTS) audio for 104 clips.
Does VoiceLongMemEval actually test 100k-token contexts?
No. Although each item is assembled to sit in a ~100k-token history, all reported evaluations use roughly 10-15k-token contexts (5 distractor sessions in the main tables, 3-10 in the scaling study). The authors list the full ~100k-token regime as untested in their limitations.
Why does the Whisper-based pipeline lose to small audio models?
Whisper's transcription strips all vocal-delivery information — the authors found no voice events, fillers, or paralinguistic markers in any of the 104 clip transcripts — and introduces content errors. On identical synthesized clips, Whisper→Opus 4.8 scores 0.254 while the 7B audio-native models Qwen2-Audio and Qwen2.5-Omni score 0.354 and 0.412 respectively.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction flattens some table layouts (e.g., Tables 1 and 8), but all values remain legible; the text is complete through Appendix D with no truncation detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.