SsgCaps: A controlled dataset for the evaluation of sound scene generation algorithms
Modan Tailleur, Junwon Lee, Laurie M Heller, Mathieu Lagrange, Keunwoo Choi, Brian McFee, Keisuke Imoto, Yuki Okamoto
cs.GR · cs.AI
Quick verdict
Public Freesound-only rebuild of DCASE2024 Task7 eval set; FAD/KAD checks suggest equivalence to the closed set, but key evidence sits in figures, not numbers.
Summary
The paper introduces SsgCaps, a publicly released dataset of 310 human-engineered 4-second sound scenes (16-bit mono, 32 kHz) for evaluating prompt-based sound scene generation (SSG) systems. Prompts follow a fixed action-based typology ('Foreground with Background in the background') spanning six foreground categories (animal, vehicle, human, alarm, tool, entrance) and five background types, and were crafted with professional sound-design support for semantic unambiguity. SsgCaps rebuilds the unpublished DCASE 2024 Task 7 reference set (SsgCapsClosed) by replacing 24 privately owned clips with versions made exclusively from Freesound recordings, keeping prompts and splits unchanged (60 development / 250 evaluation clips; two background types withheld from development). The authors validate the open version in two ways: a 'virtual' perceptual re-analysis retaining ratings for 141 of 144 rated sounds, which preserves the system rankings of the original challenge, and FAD/KAD comparisons over 18 submitted systems, whose regressions against SsgCapsClosed lie near the diagonal. They conclude SsgCaps can substitute for the closed set in benchmarking. Limitations: headline equivalence claims are supported by figures without reported correlation coefficients, confidence intervals, or significance tests; a 360-vs-310 sample-count inconsistency appears in Section III-C; item-level equivalence of the 24 replaced clips is not directly assessed; and perceptual evidence covers only 6 systems while FAD/KAD cover 18.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SsgCaps publicly releases a Freesound-only rebuild of the DCASE 2024 Task 7 evaluation set, replacing 24 privately owned clips while keeping all 310 prompts and the dev/eval split unchanged.
- A virtual perceptual re-analysis retaining ratings for 141 of the 144 originally rated sounds preserves the challenge's system rankings.
- FAD and KAD computed against SsgCaps for 18 submitted systems closely match values computed against SsgCapsClosed, with regressions near the diagonal.
- SsgCaps can therefore be recommended as a public substitute for benchmarking sound scene generation systems.
Strengths
Explicit, acoustically motivated prompt typology
Section II-B documents exclusion rules (no adjectives/actions without audible consequences, e.g. 'a red bird is perching'), noun-specificity rules ('a human is clapping' when the sound cannot reveal the source), and discriminability-based source-action pairing (Table I) — a reusable methodology for other dataset builders.
Leakage-controlled, balanced dataset design
Each recording is used for exactly one prompt with no dev/eval sharing (Section III-B); two background types (no-background, birds) are withheld from the 60-clip development set (Table IV); Tables II-III document 10-12 background exemplars per foreground category across the 310 clips.
Dual-axis equivalence validation
Genuine open-release engineering with stated constraints
Weaknesses
Equivalence claims carry no statistics
Internal count inconsistency
No item-level validation of replaced clips
Evidence asymmetry across evaluations
Unaddressed metric caveats
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
SsgCaps — Preliminary Peer Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SsgCaps?
SsgCaps is a publicly available dataset of 310 human-engineered 4-second sound scenes (16-bit mono, 32 kHz), each paired with a short, structured, unambiguous text prompt (e.g., 'a small dog is barking in a crowd'). It is intended for evaluating text-to-audio / sound scene generation systems and is downloadable from Zenodo (record 15630417).
How does SsgCaps differ from the DCASE 2024 Task 7 dataset?
SsgCaps rebuilds the challenge's unpublished evaluation set (called SsgCapsClosed in the paper): the text prompts and the 60/250 development/evaluation split are unchanged, but 24 clips that used privately licensed audio were remade using only Freesound recordings, so the whole set can be openly distributed. Two background types are also excluded from the development split.
Do evaluation metrics computed with SsgCaps match the original closed dataset?
The authors report that Fréchet Audio Distance (FAD) and Kernel Audio Distance (KAD) computed over 18 challenge systems' outputs, and perceptual system rankings based on 141 of 144 retained ratings, differ only slightly between the open and closed versions. Detailed evidence is presented in the paper's figures; exact numbers are not given in the text.
Is SsgCaps free to use?
The paper states that SsgCaps contains only public-domain / Freesound audio and is openly available on Zenodo (record 15630417); for exact license terms you should check the Zenodo record itself, as the paper does not restate the license.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-4 carry all quantitative equivalence evidence; plot values could not be extracted from the text, so numeric claims could not be verified.
- Perceptual protocol details (rater count, inter-rater reliability) are delegated to reference [11] and were not available for verification.
- Equations were partly garbled by text extraction; the noted KAD notation slip (n used for both sets) may partly stem from extraction rather than the original.
- Zenodo record 15630417's license terms are not stated in the text and could not be checked offline.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.