Can Vision-Language Models Judge Olympic Diving? From Reasoning to Scores in Zero-Shot Action Quality Assessment
Henry O. Velesaca, David Freire-Obregon, Luigi Miranda, Abel Reyes-Angulo
cs.CV · cs.AI · cs.LG
Quick verdict
VLM reasoning + ensemble regression reaches Spearman 0.67 on AQA-7 diving, but test-set config selection, n=70, and missing SOTA baselines temper the claim.
Summary
This paper tests whether six open-source vision-language models (Qwen2.5-VL 3B/7B/32B, LLaVA-NeXT-Video-7B, MiniCPM-V 2.6, InternVL3-8B) can judge 10m platform diving zero-shot on the AQA-7 benchmark, using the official 300/70 train-test split. Three structured prompts elicit five-phase sub-scores (starting position, approach, take-off, flight, entry) plus free-text reasoning, parsed by regular expressions. Standalone, the models are weak: all Spearman correlations stay below 0.32 (best 0.3174, Qwen 32B with Prompt 1). The authors then vectorize the reasoning texts with TF-IDF plus 30-component truncated SVD, optionally adding sub-scores and degree of difficulty, and train random forest, gradient boosting, ridge, and SVR regressors in an exhaustive search over ensembles of one to four VLM configurations. The best four-model ensemble (InternVL P3, Qwen 3B P2, LLaVA P1, InternVL P2; GBR; text+scores) reaches Spearman 0.6664, Pearson 0.5223, MAE 9.306 — but R² only 0.2387. Text features generally beat the numerical sub-scores. Key limitations: no confidence intervals or significance tests on the 70-clip test set; the winning configuration appears picked by test-set performance from a large combinatorial search; no comparison with supervised AQA state of the art on the same split; Table 1's prompt grid is incomplete. Code is on GitHub.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Standalone zero-shot VLMs achieve only weak-to-moderate rank agreement with official diving scores (all Spearman below 0.32; best 0.3174, Qwen 32B P1).
- A supervised GBR ensemble over four VLM configurations' textual reasoning and sub-scores more than doubles this to Spearman 0.6664 on the AQA-7 10m platform test set.
- TF-IDF/SVD features derived from VLM reasoning text are generally more predictive than the numerical phase sub-scores the models output.
- Prompt design materially affects results, with Prompt 2 providing the best stability across model families.
- VLMs are promising as explainable assistive tools for semi-automated sports judging.
Strengths
Honest two-setting framing
Section 1 explicitly reserves 'zero-shot' for frozen VLM inference and labels the regression stage 'supervised calibration', avoiding the common mislabeling of post-hoc trained heads as zero-shot results.
Clean negative result for standalone VLMs
Table 1 reports all 15 configurations with six metrics, including unfavorable ones (Qwen 3B P1 Spearman −0.1305, R² −2.83), which is more transparent than typical VLM evaluations.
Systematic ensemble exploration
Interesting modality finding
Public code and official split
Weaknesses
No uncertainty quantification
Test-set configuration selection
Missing supervised baselines
Incomplete and under-specified setup
Absolute accuracy weak for the proposed use case
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether frozen, open-source VLMs can act as zero-shot judges for 10m platform diving on the AQA-7 subset (official 300/70 split, Section 4). Two settings are evaluated: (1) raw VLM outputs (phase sub-scores plus overall QoE), and (2) a supervised calibration layer regressing final scores from TF-IDF/SVD features of VLM reasoning text, optionally with numerical sub-scores and degree of difficulty (Section 3). Headline claims: standalone Spearman below 0.32 (best 0.3174, Qwen 32B P1, Table 1); a four-model GBR ensemble reaching 0.6664 (Table 1/2); text features beating numeric sub-scores; P2 prompting most stable. The abstract's 'strong potential as assistive tools' is plausible for ranking assistance but not for absolute scoring: the best configuration's R² is 0.2387 with 16.99% relative error.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can vision-language models actually judge Olympic diving?
Not reliably on their own, according to this study. Six open-source VLMs prompted zero-shot reached Spearman correlations below 0.32 with official 10m platform diving scores. Only after training a gradient-boosting regression ensemble on the models' written explanations did correlation reach about 0.67 on the AQA-7 test set — still with weak absolute accuracy (R² ≈ 0.24, ~17% relative error) and below what supervised deep-learning methods achieve.
What is the AQA-7 dataset?
A public benchmark introduced by Parmar and Morris (2019) containing Olympic sports video clips with official judge scores across seven disciplines (diving, gymnastic vault, skiing/snowboarding big air, synchronized diving, trampoline). This paper uses the Single Diving 10m Platform subset with the standard 300-training/70-test split.
Why does combining multiple VLMs improve the scores?
The paper treats each VLM's free-text explanation of the dive as a feature source: texts are converted to numeric vectors with TF-IDF and truncated SVD, and a gradient-boosting regressor trained on four models' outputs predicts the final score. Different model families apparently describe complementary execution details. However, with only 70 test clips and a large configuration search, the size of the gain (0.48 to 0.67) is not statistically tested.
Did the models' text explanations beat their numeric scores as features?
In the top configurations reported (Table 2), features built from the written reasoning generally yielded higher Spearman correlation than the phase sub-scores the models were explicitly asked to output as numbers — the paper's most interesting finding, though it is based on best-of-search comparisons rather than paired significance tests.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–6 are not analyzable from the extracted text; claims about the Pareto and radar charts (Sections 5.2, 5.4) are taken from the prose only.
- Table 2's extracted row order looks unsorted in places (e.g., size-1 rows list 0.4664 above 0.4792), suggesting extraction artifacts; numbers were cross-checked against the running text where possible.
- The actual wording of Prompts P1/P2/P3 is not included in the provided text (only the JSON schema in Figure 3), limiting assessment of prompt-level conclusions.
- Review is based on the paper text only; the public GitHub repository was not inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.