Samsone: A Family of Open Small Audio Language Models for On-Device Inference
Piotr Masztalski, Michał K. Grzeszczyk, Olaf Sikorski
eess.AS · cs.AI
Quick verdict
Open 99M-356M audio-LM family claims size-class SOTA on MMAU (+15% vs Mellow); results are plausible but lack significance tests and contamination checks.
Summary
The paper introduces Samsone, a family of Small Audio Language Models (SALMs, <1B parameters) for on-device inference: Samsone-99M, -134M, and -356M, built from a Whisper-Tiny audio encoder, a Mellow-style nonlinear projector pooling each clip to 50 tokens, and a SmolLM2 decoder, compressed via vocabulary reduction (15,042 rare tokens removed, -8.7M parameters) and depth pruning (30 to 20 layers). All variants are trained single-stage for 100 epochs on ReasonAQA plus AudioSkillsXL (~9M QA pairs) with answer-option permutation, then exported with ExecuTorch. On MMAU-Test, Samsone-134M averages 61.33 vs. Mellow's 53.34 (~+15%) with 20% fewer parameters, and 37.57 vs. 27.50 on MMAU-Pro (~+36%), while remaining competitive with the 3B Audio Flamingo 2 on MMAU. Ablations (Table 4) show AST-encoder, GPT-2, or linear-projector swaps degrade performance. On a Galaxy S25 Ultra CPU, generation runs at 39-125 tokens/s. Training code, weights, mobile checkpoints, and an Android app are released. Main limitations: no significance testing or multi-seed variance, no audit of audio overlap between training corpora and evaluation benchmarks, no ablations of SEP tokens/pooling/data mix, AudioCaps captioning below Mellow (SPICE 14.4 vs. 17.8), and scaling-law claims resting on only three model sizes.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Samsone-134M establishes state-of-the-art among sub-1B audio language models on MMAU (61.33 test average vs. Mellow's 53.34) and MMAU-Pro (37.57 vs. 27.50).
- Even the 99M variant outperforms the 167M-parameter Mellow across MMAU domains while using roughly 40% fewer parameters.
- Vocabulary reduction and depth pruning provide a controllable footprint/capacity trade-off spanning 99M to 356M parameters.
- The family is competitive with multi-billion-parameter LALMs, beating LTU and GAMA and matching Audio Flamingo 2 on MMAU despite being 20x+ smaller.
- Real-time on-device inference is demonstrated on a Galaxy S25 Ultra CPU at 39-125 tokens per second without hardware-specific optimization.
Strengths
Consistent, size-class-leading benchmark results
Table 2: Samsone-134M averages 61.33 on MMAU-Test vs. Mellow v0s at 53.34, and even Samsone-99M (58.13) beats Mellow with ~40% fewer parameters; MMAU-Pro gains are large (37.57 vs. 27.50).
Unusually complete open-source release
Training code, server-side and ExecuTorch/XNNPACK mobile-optimized checkpoints, and an Android demo app are all released (contribution 3, footnote 1), with training restricted to public datasets.
Genuine on-device validation, not just parameter accounting
Attention to a subtle data confound
Internally consistent numbers
Weaknesses
No statistical rigor
No train/test contamination audit
Data-scale confound in the headline comparison
Scaling-law claims overreach
Ablation and deployment-evaluation gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Samsone (arXiv:2609.21666, Interspeech-format) introduces a family of Small Audio Language Models - defined here as under 1B parameters - targeting on-device inference: Samsone-99M, -134M, and -356M. Central claims: (1) Samsone-134M is SOTA for its size class on MMAU and MMAU-Pro (Table 2); (2) even the 99M variant beats the 167M Mellow; (3) the family is competitive with multi-billion LALMs; (4) real-time smartphone inference is demonstrated (Table 5); (5) three model sizes illuminate SALM scaling laws. Claims 1-4 are supported by the presented numbers; claim 5 is overreach.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Samsone?
Samsone is a family of three small audio-language models (99M, 134M, and 356M parameters) from Samsung R&D Institute Poland that answer questions about audio clips on smartphones. The 134M model reaches 61.33 average accuracy on the MMAU benchmark, above the previous best small model (Mellow, 53.34) and competitive with much larger models like the 3B-parameter Audio Flamingo 2.
How does Samsone compare to Mellow?
Samsone-134M scores 61.33 vs. Mellow's 53.34 on MMAU-Test (about +15% relative) with 20% fewer parameters, and 37.57 vs. 27.50 on MMAU-Pro (about +36%). Caveats: Samsone trains on roughly 9x more data (ReasonAQA plus AudioSkillsXL vs. ReasonAQA alone) and scores lower on AudioCaps captioning (SPICE 14.4 vs. 17.8).
Can Samsone run on a phone?
Yes. The authors released mobile-optimized ExecuTorch/XNNPACK checkpoints and an open-source Android app. On a Galaxy S25 Ultra CPU, models generate 39-125 tokens per second (99M fastest, 356M slowest) with audio encoding taking 0.7-1.1 seconds, though no NPU/GPU optimizations were implemented yet.
Is Samsone open source?
According to the paper, training code, model weights, mobile-optimized checkpoints, and an Android demo app are publicly released (github.com/SamsungLabs/samsone), and training uses only public datasets (ReasonAQA and AudioSkillsXL), which supports reproduction.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based on extracted text; Figure 1 (scatter plot) could not be independently verified beyond its caption.
- No supplementary material was available; per-domain MMAU details rely solely on Table 2.
- The public repository contents (code, weights, app) were not verifiable from the text alone.
- Short-format (Interspeech-style) paper; several implementation details are necessarily terse, limiting depth of methodology assessment.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.