SonicCaps: Large-Scale Diverse and Fine-Grained Captioning for Improved Audio-Retrieval
Zineb Lahrichi, Marc Ferras, Gaël Richard, Geoffroy Peeters
cs.SD · cs.CL · cs.MM · eess.AS
Quick verdict
Claims ~15M LLM-generated diverse audio captions boost CLAP retrieval; controlled ablations support it, but no significance tests and key evals are private.
Summary
The authors introduce SonicCaps, a dataset of ~15M captions for ~700k audio clips from FreeSound, AudioCaps, BBC Sound Effects, and AudioSet, generated by Qwen3-Omni conditioned on both audio and existing captions. A three-stage pipeline (fidelity-focused recaptioning, diversity-focused rephrasing, post-processing) yields four caption types averaging ~24 captions per clip. A pairwise human evaluation (25 participants, 60 clips) ranks SonicCaps main captions above AudioCaps, raw FreeSound, and WavCaps, with fewer captions lacking descriptive detail. Training CLAP models (RoBERTa + PaSST) with multi-caption sampling improves retrieval: the best variant Ours(9) reaches 79.5/91.3 T2A R@5/R@10 on AudioCaps-Val and 32.4/43.2 on a private commercial set, versus 64.7/75.8 and 28.7/41.6 for LAION-CLAP. Zero-shot classification improves on the disjoint FoleyBench (Ours(9) 25.6 R@5 vs 9.14 for LAION-CLAP), though LAION-CLAP retains an ESC-50 edge over single-caption variants. A model trained on main captions correlates better with human MOS (ρ=0.32 vs −0.07). Limitations: no statistical significance tests for retrieval or MOS claims, the commercial evaluation set and full audio are not released (captions plus identifiers only), missing comparison against comparable audio-conditioned recaptioning datasets such as AudioSetCaps, and the subjective evaluation rests on only 375 assessments.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Joint audio+text conditioning of Qwen3-Omni produces captions judged more faithful, descriptive and precise than existing datasets in a 25-participant pairwise study.
- Sampling from ~24 diverse captions per clip during CLAP training consistently improves audio-text retrieval and zero-shot classification at fixed audio scale.
- Caption diversity, not caption quality alone, is the primary driver of retrieval improvements.
- A CLAP trained on fidelity-focused main captions correlates far better with human MOS judgments (ρ=0.32 pairwise) than LAION-CLAP (ρ=−0.07).
Strengths
Cleanly controlled caption-composition ablations
Ours(1)–Ours(9) (Table VI) hold the ~700k audio corpus, RoBERTa-Large + PaSST architecture, and training setup fixed while varying only caption sampling mixtures, isolating caption diversity from audio scaling; T2A R@5 on AudioCaps-Val rises monotonically from 71.6 (single caption source) to 79.5 (Ours(9)).
Audio-conditioned generation fixes a known failure mode of text-only pipelines
Conditioning Qwen3-Omni on audio+caption raises unique-caption share on FreeSound from 80.4% (WavCaps) to 92.4% and cuts the most-frequent caption from 442 to 317 occurrences (Table III), directly evidencing the claimed diversity gain.
Structured human evaluation with an a priori error taxonomy
Honest reporting of negative and mixed results
Leakage-aware generalization testing
Weaknesses
No statistical significance testing anywhere
Key generalization evidence is not publicly verifiable
Missing baselines among contemporary audio-conditioned recaptioning datasets
Small and partially gapped subjective study
Unexplained zero-shot anomalies and unstated training budgets
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SonicCaps (arXiv 2609.02343, Sony CTC / Télécom Paris) is a dataset-plus-models paper targeting audio-text retrieval and CLAP-style contrastive pretraining. The authors claim: (1) joint audio+text-conditioned generation with Qwen3-Omni yields ~15M captions over ~700k clips that are more faithful and diverse than existing datasets (Sec. III); (2) sampling from the resulting ~24-captions-per-clip pool during CLAP training improves retrieval and zero-shot classification more than caption quality alone (Sec. V-A, Table VI); (3) main captions train a CLAP whose scores correlate with human MOS far better than LAION-CLAP (Sec. V-B, Table VII). Captions with audio identifiers and two checkpoints (SONICCLAPAR, SONICCLAPMOS) are released on Hugging Face.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SonicCaps dataset?
SonicCaps is a large-scale audio captioning dataset of about 15 million English captions for roughly 700,000 audio clips from FreeSound, AudioCaps, BBC Sound Effects and AudioSet. Captions were generated by the audio-language model Qwen3-Omni, conditioned on both the audio and existing captions, giving about 24 captions per clip in four styles: a main description, rephrased variants, short query-like rephrasings, and semantic tags. Only the captions and audio identifiers are released on Hugging Face, not the audio itself.
Does SonicCaps actually improve audio retrieval?
According to the paper, training CLAP models while sampling multiple caption types per clip improves text-to-audio retrieval: the best model reaches 79.5% R@5 on the AudioCaps validation set versus 64.7% for LAION-CLAP, and zero-shot classification on the disjoint FoleyBench benchmark rises to 25.6% R@5 versus 9.14%. Gains on a private commercial evaluation set are also reported. Caveats: no statistical significance tests are provided, and the commercial benchmark is not public.
Can I download the audio files of SonicCaps?
No. The authors release only the captions and the corresponding audio identifiers; users must obtain the audio from the original sources such as FreeSound or AudioSet. For the publicly released CLAP models, FreeSound content was additionally restricted to permissive licenses, reducing it to about 371k clips versus the full 515k used internally.
How was the quality of the generated captions verified?
Through a pairwise listening test with 25 participants on 60 audio clips, rated on completeness, correctness, plausibility and qualitativeness with a five-point MOS scale plus a checklist of error types. SonicCaps main captions scored highest — about 50% received no negative observation versus under 25% for other datasets — and a CLAP model trained on them correlates with human scores (Spearman ρ=0.32) far better than LAION-CLAP (−0.07).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded text (Figs 2–4) is extracted as scattered axis labels, so exact MOS values, CI widths, and perplexity-vs-R@5 points in the figures could not be read; judgments involving Fig 3 CI overlap rely on the paper's prose only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.