TamilEOT: A Dataset and Model for Semantic End-of-Turn Detection in Tamil Telephone Speech
cs.CL · cs.AI · cs.SD
Quick verdict
First open Tamil end-of-turn dataset (18,485 clips) and Smart Turn fine-tunes hitting 86.13%; notably honest about label noise and limits — credible work.
Summary
The paper releases TamilEOT, 18,485 labelled turn boundaries cut from 116 real Tamil narrowband telephone conversations (SPRING-INX), split by call, plus two audio-only end-of-turn detectors fine-tuned from Smart Turn v3 (Whisper-tiny and -base encoders). Because no open semantic EOT detector covers a South Indian language, zero-shot Smart Turn v3.2 scores 70.30% accuracy and 0.751 ROC-AUC on 4,168 held-out clips; the fine-tunes reach 83.71% (tiny int8, 8.7 MB) and 86.13% (base int8, 21 MB), running under 150 ms single-threaded on CPU. Methodologically, the authors measured their rule-derived labels against a blind human listening pass (197 clips) and found the negative class at 44.4% — below chance — because the rule answered a different question than the model is asked; an audio-LLM labeller measured at 97.5% human agreement replaced them for US$5.69, and a 97.1% label-noise ceiling is estimated. Of six measured training levers, only encoder capacity (+2.88) exceeded the 0.87-point run-to-run noise floor. A paired replay through the production VAD and streaming adapter costs 2.60 points, and 7.8% of boundaries are never surfaced. Limitations: a single corpus and domain, machine labels validated by one listener on only 197 clips, an uncontrolled live comparison (seven blocks, one speaker), and a heavy train-dev gap (98.99% vs 87.18%) with regularisation untried.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- TamilEOT is the first open dataset for semantic end-of-turn detection in Tamil and any South Indian language: 18,485 call-split-labelled boundaries from 116 real telephone conversations, CC BY 4.0.
- Fine-tuning Smart Turn v3 lifts held-out accuracy from 70.30% zero-shot to 83.71% (tiny int8) and 86.13% (base int8), ROC-AUC 0.751→0.921, at under 150 ms single-thread CPU latency.
- Rule-derived labels were only 44.4% accurate on the negative class because they answered a different question; a US$5.69 audio-LLM relabelling measured at 97.5% human agreement replaced them, with a 97.1% label-noise ceiling.
- Of six measured training levers, only encoder capacity (+2.88 points) exceeds the 0.87-point run-to-run noise floor established over three identical runs.
- Serving the same boundaries through the production VAD and streaming adapter costs 2.60 accuracy points, and 7.8% of boundaries are never surfaced to the model at all.
Strengths
First-in-language resource with leakage-safe splits
18,485 boundaries from 116 real narrowband Tamil calls (SPRING-INX), split per call (§3.4); the paper shows why this matters — SPRING-INX R2's own utterance-level splits put 511 of 545 recordings in both train and eval, letting a model recognise voices instead of prosody.
Labels validated by measurement, not assumption
A 197-clip blind listening pass found rule-derived labels 95.9% right on positives but 44.4% on negatives — below chance (§4.1) — diagnosed as a Q1/Q2 question mismatch (§4.2); seven audio LLMs were baked off (Table 3), and per-row label_pipeline/llm_verdict/dispute fields make the policy re-derivable without re-running anything (§4.3).
Noise-floor-aware negative results
Offline-served gap actually measured
Artefact-level reproducibility
Weaknesses
Thin, single-listener label validation
37.7% of the test set is witness-disputed
Single corpus, single domain
Overfitting acknowledged but unaddressed
Deployment evidence is weaker than the offline story
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets semantic end-of-turn (EOT) detection — deciding from audio alone whether a speaker has finished — for Tamil, which no open EOT detector covers (Smart Turn v3 lists 23 languages; its three Indic entries are all Indo-Aryan). Claims: (1) the first open Tamil, and South Indian, EOT dataset — 18,485 boundaries from 116 real narrowband telephone calls, CC BY 4.0, split by call; (2) two Smart Turn v3 fine-tunes (Whisper-tiny 8M / Whisper-base 20.3M encoders) lifting test accuracy from 70.30% zero-shot to 83.71% / 86.13% and ROC-AUC from 0.751 to 0.921, under 150 ms single-thread CPU; (3) a labelling protocol validated rather than assumed, after the original rule labels measured 44.4% on the negative class; (4) five of six training levers reported as negative results; (5) a measured offline-vs-served gap of 2.60 points with 7.8% of boundaries never surfaced. Scope is deliberately one corpus and one domain, and the Limitations section says so.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TamilEOT and what problem does it solve?
TamilEOT is an open dataset of 18,485 labelled turn boundaries cut from 116 real Tamil telephone calls, plus two small models that predict from audio alone whether a speaker has finished their turn. Voice agents use such detectors to decide when to start replying instead of waiting a fixed silence timeout, which reduces both interrupting the user and awkward dead air.
How accurate is the Tamil end-of-turn model?
On 4,168 held-out clips from 30 unseen calls, the released tiny model reaches 83.71% accuracy and the base model 86.13% (both int8 ONNX), up from 70.30% for zero-shot Smart Turn v3.2; ROC-AUC rises from 0.751 to 0.921. Both run in under 150 ms on a single laptop CPU core.
Is the dataset freely available and licensed for reuse?
Yes. The audio derives from the SPRING-INX Tamil R1 corpus (IIT Madras, CC BY 4.0) and the derived dataset carries the same licence on Hugging Face; code is BSD-2-Clause on GitHub, int8 ONNX weights are on Hugging Face, and a Pipecat/LiveKit inference package is on PyPI.
Why did the authors replace their original labels?
A blind human listening check found the rule-derived 'incomplete' labels matched humans only 44.4% of the time — below chance — because a pause inside a transcript segment is not the same as an unfinished utterance. An audio-LLM labeller agreeing with a human listener 97.5% of the time relabelled the whole pool for US$5.69, and the paper estimates a 97.1% label-noise ceiling.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The metadata abstract cuts off mid-sentence ('never surfaced'), but the embedded full text contains the complete abstract and runs through reference [8] with no visible truncation; this review relies on the full text.
- Table 1-3 were reflowed by PDF extraction (columns inline), but all values cited were legible; no data was treated as missing on that basis.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.