MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agents
Pritish Mishra, Ishaan Kumar, Akshat Mandoli, Sudarshan Kamath
cs.AI
Quick verdict
MTVA-Bench isolates the LM in cascaded voice agents with a rigorous asymmetric harness; well-designed but judge calibration and thin channel data limit trust.
Summary
MTVA-Bench evaluates the language model inside cascaded voice agents — the component between ASR and TTS — by simulating everything around it in text: an LLM caller improvising from a private brief, a channel layer injecting ASR-style garbles and VAD-style turn splits, and a mock backend that answers according to the arguments the model actually sent. The bundle spans 490 reviewed scenarios across 49 agents in seven languages, with median instruction documents around 30,000 characters. Each call is graded by three evaluators with disjoint views: a deterministic action checker, a scenario judge required to cite message indices and quote rules verbatim, and a task-blind conversation judge; task and conversation scores are equally weighted. In a seven-model study, six models chose correct tools within 6.4 points (82.3–88.7%) while overall scores spanned 24.4 points (73.9 down to 33.4), with the gap driven by argument values, call ordering, rule compliance, and speech around tool calls. The authors candidly flag the limits: the LLM judges lack blinded human calibration, each trace got one rollout and one judgment, only 53 scenarios carry channel damage (32–40 traces activated per run), the runs were non-official from a dirty working tree, and the scenario generator, gpt-5.6-sol, is itself the top-scoring candidate.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing evaluations either mix recognition and model errors (end-to-end voice benchmarks) or omit voice-channel obligations entirely (text benchmarks); MTVA-Bench evaluates exactly the LM between ASR and TTS.
- Six of seven evaluated models select the correct tool within 6.4 points of one another (82.3-88.7%), yet overall scores span 24.4 points, driven by argument values, action ordering, rule compliance, and speech around tool calls.
- Task and conversation performance fail independently: mid-table models are within half a point of each other on task score but spread 18 points on conversation.
- Code-enforced citation contracts anchor every LLM judgment to verifiable evidence, though the authors concede the judged half of the score remains an uncalibrated model estimate pending blinded human validation.
Strengths
Precise failure attribution by construction
Section 2.3 assigns garbled words to the channel and wrong arguments to the candidate, storing each separately in the trace; the benchmark never charges the model for audio-side errors it cannot see — a cleaner attribution target than end-to-end voice benchmarks.
Enforced information asymmetry across simulator, candidate, and graders
Table 1 separates caller knowledge from hidden backend state; the scenario judge receives checker facts with verdict labels stripped; the conversation judge is task-blind. Every LLM judgment must cite an existing trace index and quote violated rules verbatim, verified by code (Section 5).
The headline finding is genuinely diagnostic
Unusually honest meta-evaluation and limitations
Weaknesses
Half the score rests on uncalibrated LLM judges with unmeasured noise
The titular voice-channel conditions are severely underpowered
Generator–candidate circularity is never addressed
Reproducibility posture is weak
Author-set weights shape results without sensitivity analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MTVA-Bench targets a narrow, well-defined sliver of the voice-agent stack: the language model between ASR and TTS in a cascaded system. Section 1 argues that end-to-end voice benchmarks entangle recognition errors with model errors, while text-agent benchmarks drop the obligations a phone channel creates - garbled words, VAD-split messages, language/script discipline, TTS-speakable output. The harness simulates everything on the other side of the model's interface: an LLM caller improvising from a private brief, a channel layer applying authored ASR events or VAD splits, and a mock backend answering from ordered fixture queues keyed to the arguments the model actually sent. Central claims: (1) a precise evaluation target with clean failure attribution (Section 2.3); (2) a controlled 490-scenario, 49-agent, 7-language corpus with production-scale instructions (median 30,834 characters, 371 tools); (3) a three-evaluator split with code-verified evidence citations; (4) a seven-model study in which tool selection barely separates models while overall scores span 24.4 points.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MTVA-Bench?
A benchmark that evaluates the language model inside cascaded voice agents (the component between speech recognition and text-to-speech). It simulates the caller with an LLM, injects text-level ASR/VAD-style damage, and uses a mock backend, scoring 490 scenarios across 49 agents in 7 languages.
Which models performed best on MTVA-Bench?
In the paper's seven-model study, gpt-5.6-sol led with 73.9/100 overall, followed by electron at 69.8 and gemini-3.6-flash at 62.3; phonellm trailed at 33.4. Six of seven models were nearly equal at picking the right tool but differed by up to 24.4 points overall.
Does MTVA-Bench test audio quality or ASR accuracy?
No. Audio never reaches the evaluated model. The benchmark simulates recognition damage and turn splits as text and explicitly excludes pronunciation, noise, accent, and latency from scoring.
Are the LLM judges in MTVA-Bench reliable?
Partially validated: code verifies every cited message index exists and nine minimal-pair probes responded correctly, but the authors state there is no blinded human calibration yet, and each trace received only one judgment per judge in the reference study.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 (episode walkthrough, score and diagnostic panels) are available only as extracted caption/axis text; visual inspection was not possible.
- The metadata abstract is truncated mid-sentence, but the embedded full text contains the complete abstract; the review is based on the full text.
- No statement about public code or data release appears in the extracted text; codeAvailable=false reflects the absence of a release claim, not a stated refusal.
- All reported numbers are taken as printed; the runs are self-described as non-official from a dirty working tree, so values may shift on tagged reruns.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.