$τ$-Elicitation: Benchmarking multi-turn entity extraction in voice agents
cs.AI
Quick verdict
Voice agents fail exact entity capture (robust Pass3 0.14–0.41 vs text 1.0); well-designed, honestly analyzed benchmark, limited by single caller simulator.
Summary
The paper introduces τ-Elicitation, a 200-task benchmark that isolates exact entity collection (names, addresses, codes, dates, times, etc.) in voice agents, spanning 10 entity types with controlled difficulty, six caller realisms, and three audio/interaction environments. Each core task elicits one missing database field via a simulated callback; a matched text agent passes all 200 tasks, while four voice configurations (gpt-realtime-2 minimal/xhigh, gemini-3.1-flash-live, grok-voice) achieve robust Pass3 — success across regular, noise-heavy, and speech-heavy realizations — of only 14–41% agent-directed, rising to 37–54% with a scaffold prescribing spelling, read-back, correction, and confirmation, at 21–28 seconds extra per call. Scoring is deterministic database comparison, with no LLM judge in the reward path. Agents verify hard and unfamiliar entities more but not their weakest caller voice, and only 24–37% of verified errors are repaired. Caller realisms show no detectable success effect (Hájek confidence intervals spanning zero, Holm-corrected), though mispronunciation adds repair effort. A formative multi-field study shows errors compounding (0.53 task pass at three fields despite 0.75 field accuracy), with staged validation raising task Pass@1 from 0.64 to 0.82. Limitations: English only, proprietary APIs, one cooperative GPT-5.5-based caller simulator with vendor TTS, one run per main environment.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A matched text agent passes all 200 exact-entity tasks, while four voice configurations succeed robustly — across three prespecified environment realizations — on only 14–41% of them unaided and 37–54% with a prescribed verify-and-confirm scaffold.
- The scaffold raises robust Pass3 by 14–31 points at 21–28 seconds per call and narrows provider gaps without changing the rank order (Grok best, Holm-adjusted p ≤ .0032).
- Agents increase verification for hard and unfamiliar entities but not for their weakest caller voice, and only 24–37% of verified errors end repaired, locating the bottleneck in recovery rather than risk recognition.
- Six caller realisms show no detectable effect on exact success under Holm correction, though mispronunciation adds 24 points of spelling requests and 26 seconds of repair effort.
- Multi-entity calls compound errors (0.53 task pass at three fields despite 0.75 field accuracy), and staged per-field validation with retry raises task Pass@1 from 0.64 to 0.82.
Strengths
Clean capability isolation via a matched text control
A text agent passes all 200 tasks while the best voice configuration reaches only 66.5% Pass@1 and 40.5% Pass3 (Fig. 2), attributing failure to the spoken channel rather than task understanding; human review of 90 failures assigns 81 to the agent, dominated by transcription (42), not logic (16).
No LLM judge in the scoring path
The reward is exact string comparison of the final database value; the only judge (speech fidelity, Gemini 3.1 Pro) is diagnostics-only and validated on a frozen 60-utterance set with .80 precision / .75 recall / .77 F1 (Sec. 3.3, 5.3).
Robust Pass3 is a prespecified, non-averaged robustness metric
Statistically careful, honest about nulls
Actionable negative findings
Weaknesses
Entire caller stack is one simulator
Robust Pass3 rests on one run per environment
Scaffold is an unablated bundle
Timeout interaction with scaffold cost is unexamined
Verification-repair analysis is observational
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
τ-Elicitation (arXiv 2609.13602; Sierra + Mercor) targets one narrow, well-motivated capability: exact capture of a single critical value — name, address, identifier, date, time, code, email, phone, medication, coined name — during a voice call. Ten banks × 80 values form 200 core tasks, plus linked multi-field tasks. The central claim: end-to-end voice benchmarks obscure where capture fails, and this benchmark isolates it, finding (i) a matched text agent passes 200/200 while four voice configurations reach robust Pass3 of only 14–41% agent-directed (37–54% scaffolded, Fig. 2); (ii) verification is miscalibrated — agents check hard/unfamiliar entities but not their worst caller voice, and only 24–37% of verified errors are repaired; (iii) a prescribe-everything scaffold buys 14–31 Pass3 points for 21–28 s/call; (iv) caller realisms do not detectably move exact success (Fig. 4). Claims are properly scoped to English, three proprietary systems, one cooperative simulated caller.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is τ-Elicitation?
A 200-task benchmark for measuring whether voice AI agents can collect exact values (names, addresses, identifiers, dates, times, codes, emails, phone numbers, medications) in multi-turn spoken callbacks. The stored database string must match the gold value exactly, and scoring is deterministic comparison rather than an LLM judge.
How well do current voice agents perform on it?
A matched text agent passes all 200 tasks, but four voice configurations (GPT-Realtime-2 minimal and xhigh, Gemini 3.1 Flash Live, Grok Voice) succeed robustly — meaning across regular, noise-heavy, and speech-heavy conditions — on only 14–41% of tasks unaided, rising to 37–54% when forced to spell, read back, correct, and confirm.
What does robust Pass3 mean in this paper?
A task counts as Pass3 only if the agent succeeds on that same task in all three prespecified environment realizations: regular audio, noise-heavy (10 dB background noise, bursts, frame drops, muffling), and speech-heavy (interruptions, backchannels, restarts). It measures stability rather than averaging successes across runs.
What is the main bottleneck the paper identifies in voice agents?
Not noticing errors — agents do verify more for hard and unfamiliar entities — but completing recovery: only 24–37% of verified errors end up correct, agents do not increase verification for their weakest caller voice, and a prescribed verification protocol outperforms letting the agent choose its own strategy.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded numbers (Figs. 2–5) are interleaved into the running text by PDF extraction; every value quoted in this review was cross-checkable against the surrounding prose and tables, and none was unreadable.
- No truncation detected: the extraction runs complete through reference [21], including the ethics section and acknowledgments.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.