V\={a}kQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering
Bhavana Akkiraju, Ravi Sastry Kolluru, Sri Charan D, Srihari Bandarupalli, Santosh Kesiraju, Anil Vuppala
cs.CL · eess.AS
Quick verdict
First Telugu spoken factoid QA benchmark (2,001 pairs); valuable resource and judge study, but the same Gemini is both QA model and judge — self-bias risk.
Summary
The paper introduces VākQA, the first spoken question answering benchmark for Telugu: 2,001 factoid QA pairs extracted semi-automatically from Telugu quiz-style YouTube videos across six domains (Science 27%, General Knowledge 23%, Politics 16%, History 13%, Culture 12%, Geography 10%), totaling 2.53 hours of real speech with Telugu transcriptions and human-translated English versions. Five annotators verified transcript-audio alignment. The authors first validate automatic evaluation against human ratings on 400 (question, reference, candidate) triplets scored by five native speakers (Krippendorff's α=0.836 after removing 20 inconsistent items). Gemini-as-judge correlates best with humans (ρ=0.86, τ=0.77) versus Gemma-3 judges (ρ=0.57–0.81), EM/F1 (ρ≤0.49) and BLASER-2.0 (ρ=0.36). Benchmarks show Gemini-2.5-Flash leading in all configurations (oracle Telugu text 3.63/5 vs Gemma-27B 2.55), speech input costing 21.2% of questions relative to text, English translation degrading Culture most (3.76 to 2.42), and ASR→MT cascades compounding errors (M1–M4: 2.44–2.80). Main limitations: the strongest QA model and the selected judge are the same Gemini system, inviting self-preference bias; the human study is small and partly annotated by co-authors; α was computed only after outlier removal; quiz-style read speech limits phonetic diversity; and no significance tests or confidence intervals accompany the reported means.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- VākQA is the first benchmark for spoken factoid QA in Telugu, built from real human speech with bilingual transcriptions and human-verified reference answers.
- LLM-judge reliability for Telugu QA depends strongly on judge choice: Gemini-as-judge best approximates human ratings (ρ=0.86) while open-weight judges penalize correct answers differing in surface form from the reference.
- The proprietary Gemini model substantially outperforms all open-weight models across input configurations (oracle Telugu text: 3.63 vs 2.55 for the best open-weight model).
- Telugu phrasing preserves culturally specific scope lost in English translation, speech input introduces meaning-altering phonetic confusions, and ASR→MT cascade errors compound progressively (2.44–2.80 vs 3.63 oracle).
Strengths
First native-speech Telugu SQA resource
Section II builds 2,001 factoid QA pairs from real human quiz speech (not TTS like Spoken SQuAD or ViSQA), with human-verified transcript-audio alignment, bilingual transcriptions, and manual English translations across six curated domains.
Judge validation precedes benchmarking
Section III-IV scores 400 triplets with five native raters (α=0.836), then compares judges via Spearman ρ/Kendall τ plus bias and Bland-Altman limits of agreement (Fig 2), exposing that open-weight judges penalize correct Telugu surface-form variants (Table IV E2; 46.2% of scores drop when switching judge from Gemini to Gemma-12B).
Factor-isolated experimental design
Transparent, quality-controlled pipeline
Candid limitations section
Weaknesses
Judge-contender circularity
Small, partially conflicted, post-filtered human study
No inferential statistics on benchmark results
Narrow acoustic coverage of the spoken axis
Gaps and framing issues in the evaluation grid
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
VākQA positions itself as the first Telugu spoken factoid QA benchmark — 2,001 pairs, 2.53 hours of audio, six domains — plus a methodological study of whether LLM judges can score Telugu answers reliably. Three claims are advanced: judge choice strongly affects evaluation (Sec IV), Gemini-as-judge best matches humans, and input language, modality, and ASR→MT cascading each measurably degrade QA (Sec V). The dataset contribution is real and needed; the evaluation claims are supported within the paper's own framing but weakened by an unexamined circularity detailed below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the VākQA dataset?
VākQA is a benchmark of 2,001 factoid question-answer pairs in Telugu built from quiz-style YouTube videos across six domains (science, general knowledge, politics, history, culture, geography). It contains 2.53 hours of real human speech, Telugu transcriptions, and human-translated English versions, and is released publicly on Hugging Face.
Why not just use exact match or F1 to evaluate Telugu QA answers?
The paper shows these lexical metrics correlate poorly with human judgments (Spearman ρ=0.37 for EM, 0.49 for F1 on 400 human-rated answers) because a correct Telugu answer often differs in surface form from the reference — for example, listing the specific states hit by cyclones when the reference names the coastal region.
Which models were benchmarked and what did they score?
Gemini-2.5-Flash scored highest on the 1–5 judge scale in every configuration: 3.63 with oracle Telugu text, 3.52 with oracle English text, 3.28 with speech input, and 2.44–2.80 through ASR→MT cascades. The best open-weight model, Gemma-3-27B, scored 2.55 on oracle Telugu text; Gemma-3-12B scored 2.01, Sarvam-m 1.98, and smaller models roughly 1.5 or below.
Is the VākQA data publicly available?
Yes. The authors state the benchmark is publicly released at hf.co/datasets/Bhavanaakkiraju/VakQA, including audio, Telugu transcriptions, and English translations. Evaluation prompts and scripts are not explicitly linked in the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–4 (Bland-Altman plot and domain spider plots) cannot be analyzed from the extracted text; limit-of-agreement and domain values are taken from the prose.
- The judge prompts and evaluation code are not included in the extracted text, so prompt sensitivity cannot be assessed.
- Audio characteristics (speaker counts, noise, spontaneity) are inferred from the methodology description, not the audio itself.
- Minor numeric inconsistencies exist between Table V and Section V-A text (Gemma-4B std 1.12 vs 1.11).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.