Voice of Reason: Reinforcement Learning for Spoken Math
Timoth\'ee Weisselberger, Edouard Graves, Alexandre D\'efossez
cs.CL
Quick verdict
RL with LLM-judge rewards lifts GLM-4-Voice to 74.8% GSM8K, a credible speech-native SOTA; most gain comes from SFT and some ablations are single-run.
Summary
The paper applies reinforcement learning with verifiable (LLM-judged) rewards to GLM-4-Voice, a 9B interleaved speech-text model, to improve spoken mathematical problem solving. After in-domain SFT on ~150k Orca-Math-derived question-answer pairs reformatted for speech synthesis, the authors train with a group-relative REINFORCE objective (GRPO-style, no PPO clipping) where a binary reward comes from an LLM judge reading the decoded text stream. Two technical contributions stand out: a temperature correction aligning the policy-gradient estimator with the stochastic sampling distribution (removing it collapses GSM8K accuracy from 65.5% to 12.3%), and audio-token merging that collapses audio-token probabilities into one abstract event, justified via Rao-Blackwellization. Main results: reasoning-free GSM8K rises 27.3% to 61.7% (SFT) to 65.5±1.1% (RL); combined with STITCH-style reasoning supervision, 74.8±1.1%, the best reported for speech-native models. ASR-based evaluation confirms gains transfer to audio; UTMOSv2 naturalness is stable. Limitations: most improvement comes from SFT rather than RL; OOD TriviaQA drops 7.2 points after full-data SFT; ablations beyond the default configuration are single runs; evaluation depends on GPT-4o as judge; a decoding-config mismatch (footnote 4) separates headline numbers from released checkpoints; cascaded baselines still reach 95.7%.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- RL with a binary LLM-judge reward improves GLM-4-Voice's GSM8K accuracy from 61.7% to 65.5±1.1% without any reasoning tokens, exceeding the previously published reasoning-token method STITCH (58.7%).
- Combining STITCH-style reasoning supervision with RL yields 74.8±1.1% GSM8K, the strongest reported result among speech-native models.
- Correcting the policy-gradient estimator for the sampling temperature is critical: without this correction, GSM8K accuracy collapses from 65.5% to 12.3%.
- Collapsing audio-token probabilities into a single abstract event yields an unbiased, lower-variance RL gradient estimator (Rao-Blackwellization under a value-invariance assumption).
- The accuracy gains are not explained by longer responses or longer reasoning traces, and speech naturalness (UTMOSv2 ~4.06-4.17) is preserved through RL.
Strengths
First RL application to math reasoning in speech-native models, with a new state of the art
Section 5/Table 1: reasoning-free model reaches 65.5±1.1% GSM8K (above STITCH's published 58.7% which uses reasoning tokens), and STITCH-style SFT+RL reaches 74.8±1.1%, the best among speech-native systems; ASR-based eval (72.0±1.9) confirms gains exist in actual speech, not just the decoded text stream.
Temperature correction is a striking, well-isolated finding
Section 3.3 motivates correcting the policy-gradient estimator for sampling at T=0.9 by dividing logits by T before log-softmax; Table 4 shows removing it collapses GSM8K from 65.5±1.1% to 12.3% — a practically important failure mode for on-policy RL at sub-1 temperatures.
Principled treatment of the interleaved vocabulary structure
Above-average evaluation hygiene
Artifact release
Weaknesses
RL's marginal contribution is modest and the abstract framing overstates it
Single-run ablations undermine fine-grained conclusions
Permissive reward model with limited validation
Out-of-domain regression from SFT is documented but unaddressed
Mismatch between headline numbers and released artifacts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What accuracy does Voice of Reason achieve on GSM8K?
The model reaches 65.5% (±1.1) on GSM8K without reasoning tokens, and 74.8% (±1.1) when combined with STITCH-style reasoning supervision, up from 27.3% for the base GLM-4-Voice model. Questions are spoken aloud; accuracy is measured from the decoded text stream and verified on ASR transcriptions of the generated speech (72.0% for the STITCH-style model).
Is the trained spoken math model publicly available?
Yes. The authors release HuggingFace checkpoints named glm-4-voice-of-reason-9b (reasoning-free) and glm-4-voice-of-reason-stitch-9b (with reasoning). Note the paper does not state that training code or the synthesized datasets are released, and the public checkpoints perform best with unconstrained decoding (70.3%/77.1%), slightly above the paper's top-k=50 headline numbers.
Does this speech model beat cascaded ASR + LLM + TTS systems on math?
No. A cascaded pipeline using a 31B text LLM reaches 95.7% on GSM8K and Qwen3-Omni reaches 94.6%, versus 74.8% for this speech-native model. The claimed advantage is not raw accuracy but staying within a single streaming speech model, which offers lower latency and access to paralinguistic information; the paper claims state of the art only among speech-native models.
What is the temperature correction and why does it matter so much?
During RL training, responses are sampled at temperature 0.9, so the actual sampling distribution differs from the model's raw distribution. The paper computes policy-gradient log-probabilities on temperature-scaled logits (logits divided by T before log-softmax) to keep the gradient estimator consistent with what was sampled. In their ablation, omitting this correction collapsed GSM8K accuracy from 65.5% to 12.3%, making it the single most important training detail in the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (pipeline illustration) is available only as caption text; its visual content could not be analyzed.
- Several cells (ASR accuracies for reasoning-free models, TriviaQA for some regimes) are single-seed point estimates without variance, as disclosed in Appendix C; this limits precision but is reflected in the review.
- The extracted text appears complete including Appendices A-D; no truncation was detected, so confidence is high.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.