TeleAntiFraud 2.0: A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection
Huiyuan Liu, Zhiming Ma, Yanxing Liu, Shun Zhang, Qifan Wang, Di Liu, Yifan Wang, Yuyang Deng, et al.
cs.SD · cs.CL
Quick verdict
Refreshable audio fraud benchmark with near-domain negatives; strong design and collapse-aware analysis, but synthetic-only data and thin human validation.
Summary
The paper introduces TeleAntiFraud 2.0, an audio benchmark for telecom-fraud detection organized as immutable monthly frozen snapshots (currently June/V1 and July/V2), each with 900 synthetic Chinese calls: 600 fraud and 300 near-domain lawful calls. A Mixed-Tree Generation Pipeline converts online fraud-case abstracts into scenario profiles, expands them into dialogue trees whose fraud and non-fraud sibling branches share context and diverge only at label-bearing actions, realizes paths via a six-agent role-play system, and renders TTS audio with provenance-rich manifests. Controlled text experiments show LR/SVM/RoBERTa achieve perfect Macro-F1 (1.000) against unrelated or ordinary negatives but 0.650-0.680 against siblings; TF-IDF+SVM separability is 0.93 (text) and 0.80 (ASR) versus 0.995 on TeleAntiFraud-28k. Full-set runs over roughly 16 direct-audio and ASR+LLM models expose frequent all-FRAUD collapse (fraud F1 0.800 equals the class-prior ceiling), out-of-contract error rates up to 900/900, and snapshot sensitivity. Main limitations: fully synthetic data from a single TTS engine, a single-annotator audio quality pilot, expert label agreement of 0.792 (packets 0.700-0.880), only two snapshots so far, and no fine-tuned specialist baselines to establish an achievable ceiling.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Mixed-tree construction with fraud/lawful sibling paths sharing scenario context suppresses topic-level shortcuts, dropping classifier Macro-F1 from 1.000 to 0.650-0.680.
- Monthly immutable frozen snapshots let the benchmark absorb newly observed scam patterns without invalidating or overwriting previously published evaluation sets.
- Many frontier audio and ASR+LLM systems collapse to all-FRAUD predictions on the snapshots (fraud F1 near the 0.800 prior ceiling), so fraud-class F1 alone is inadequate and collapse-aware reporting is required.
- The generation pipeline yields traceable, auditable labels, supported by path-level rationales and a ten-expert audit with 0.792 corrected-gold agreement.
Strengths
Near-domain sibling construction is empirically validated, not just asserted
Table 4 shows the same three classifiers (Bigram LR, SVM, RoBERTa) at Macro-F1 1.000 on unrelated and ordinary negatives but 0.680/0.673/0.650 (with 95% CIs) on mixed-tree siblings; Table 5 corroborates with TF-IDF+SVM separability of 0.93/0.80 on TeleAntiFraud 2.0 versus 0.995 on TeleAntiFraud-28k.
Unusually strong reproducibility infrastructure
Each frozen manifest binds audio, dialogue text, label with rationale, delivery tags, generation metadata, evaluation prompt, request parameters, raw response, parsed prediction, and timestamp; snapshots are immutable once frozen, so new case patterns are absorbed by later releases without overwriting prior evaluation sets.
Collapse-aware reporting rather than leaderboard inflation
Multi-layer construction audits with quantitative evidence
Breadth and transparency of model evaluation
Weaknesses
Fully synthetic data from a single TTS engine with skewed voice pool
Thin and partially opaque human validation
Error-scoring opacity and averaging that masks collapse
Refreshability claim under-evidenced and prior not prevalence-grounded
No fine-tuned specialist baselines
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TeleAntiFraud 2.0?
It is a benchmark for detecting telecom fraud from phone-call audio. It currently ships two immutable monthly snapshots (June and July), each with 900 synthetic Chinese calls: 600 fraudulent and 300 lawful near-domain calls that closely resemble fraud scenarios. Each snapshot includes audio, labels with rationales, evaluation prompts, and provenance records so results can be reproduced and audited.
Is a fraud F1 around 0.8 a good score on this benchmark?
No. The benchmark uses a fixed 2:1 fraud-to-non-fraud ratio, and the paper shows that a degenerate classifier predicting FRAUD for everything already achieves about 0.80 fraud F1. Many evaluated models show exactly this all-FRAUD collapse signature, so scores should be read together with balanced accuracy, class-conditional recall, and prediction distributions.
Is the data in TeleAntiFraud 2.0 real?
No. All dialogues are synthetically generated by a multi-agent pipeline grounded in real online fraud-case summaries, then rendered to speech with text-to-speech synthesis. The authors position it as a controlled diagnostic instrument, not a replacement for real call distributions.
Where can I get the TeleAntiFraud 2.0 code and data?
The paper links an anonymous.4open.science repository containing construction code, evaluation scripts, manifests, and documentation; each frozen snapshot ships with its audio, labels, rationales, and provenance records.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 and S1 are referenced but not analyzable from the extracted text, and Table S1 is redacted by design, so qualitative dialogue and audio quality could not be independently assessed.
- Some supplementary details (per-annotator counts, prompt texts, pro_train/val/test partition construction) may be truncated in the extraction.
- Public artifact availability at the anonymous repository was not verified (no network access); availability is as stated in the paper.
- Exact column semantics for error-flagged rows in Tables S2-S3 could not be fully reconciled from the text; the related judgment is framed as a reviewer question rather than a conclusion.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.