Efficient Reasoning Distillation: Small Video-Language Models via Synthetic CoT and Difficulty-Aware Fine-Tuning
Mantek Singh, Jeshwanth Challagundla, Siddharth Raina, Jasmin Jarsania
cs.LG · cs.AI · cs.CV
Quick verdict
Claims 2B VideoLM near 4B teacher via 900 CoT-distilled samples; promising but single-run, no variance, key CoT-order ablation unquantified.
Summary
This paper distills chain-of-thought reasoning into InternVL2-2B for video question answering. Using uncertainty-based selection (smallest top-2 prediction margin), the authors pick ~900 CinePile training questions on which the 2B model is least confident, have InternVL2-4B generate rationales capped at 20 tokens, and fine-tune with an 'Answer-then-Rationale' format (AdamW, lr 2e-5, 3 epochs; under two hours on a 4xA100 node). The best model reaches 38.34% average accuracy on CinePile versus 30.34% zero-shot, surpassing mPLUG-Owl3-8B (38.27%) and InternVL2-8B (32.28%) and narrowing the gap to InternVL2-4B (39.89%). A 26B teacher's longer rationales perform worse (37.41%), and random, length-matched CoT drops accuracy below baseline (28.00%), supporting the authors' capacity-mismatch and rationale-quality hypotheses. Transfer: 40.20% to 43.19% on ActivityNet-QA converted to multiple choice with GPT-4o-generated distractors, and 45.0% to 50.5% on an unspecified MLVU subset (teacher: 51.0%). The headline insight - rationales placed after the answer beat the standard order - rests on an unquantified ablation. Main limitations: single runs with no variance or significance testing; no 8B-teacher accuracy for the teacher-size claim; undefined MLVU subset; conflicting compute statements (single A100 in the abstract vs 4xA100 in Sec. 3.3); code not yet released.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Fine-tuning InternVL2-2B on ~900 top-2-margin-selected CinePile samples with 4B-teacher CoT rationales reaches 38.34% average accuracy, surpassing 8B-class models (mPLUG-Owl3-8B at 38.27%, InternVL2-8B at 32.28%) and approaching its 4B teacher (39.89%).
- Placing rationales after the answer substantially improves small-model reasoning versus standard rationale-first prompting, which the authors observed as a 'final-option bias' when using the conventional order.
- Top-2 margin sampling is the best selection strategy (+5.32 points over random without CoT), and a 4B teacher yields better students than a 26B teacher (38.34% vs 37.41%) due to a capacity mismatch with verbose rationales.
- Replacing real rationales with random length-matched text degrades accuracy below the zero-shot baseline (28.00% vs 30.34%), showing gains stem from rationale content rather than added tokens.
- The improvements transfer: 40.20% to 43.19% on GPT-4o-converted ActivityNet-QA and 45.0% to 50.5% on an unspecified MLVU subset, where the student reaches near-parity with the 4B teacher (51.0%).
Strengths
Clean component-ablation matrix
Table 2 crosses three selection strategies with and without CoT, isolating each component: Top-2 Gap beats Random by +5.32 points without CoT (36.34 vs 31.02), and CoT adds +2.0 on top of Top-2 Gap (38.34 vs 36.34), so the two ingredients' contributions are separately identifiable.
Genuine CoT-quality control
Table 4 replaces rationales with random length-matched text and accuracy falls to 28.00%, below the 30.34% zero-shot baseline - ruling out 'just more tokens' explanations and showing noisy supervision actively hurts, which strengthens the rationale-content interpretation.
Teacher-size evidence for capacity mismatch
Cheap, transferable recipe checked on multiple benchmarks
Weaknesses
Headline CoT-ordering claim is never quantified
No statistical rigor anywhere
Teacher-size conclusion outruns the evidence
Evaluation protocol gaps block verification
Internal inconsistencies and citation sloppiness
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets sample-efficient reasoning distillation for compact video-language models (VLMs). The recipe: (1) select ~900 CinePile training questions where InternVL2-2B is most uncertain, using the smallest gap between top-2 answer probabilities ('Top-2 Gap'); (2) generate <=20-token rationales with InternVL2-4B; (3) fine-tune the 2B student with an 'Answer-then-Rationale' target format. Central claims (Sec. 1, abstract): 38.34% average accuracy on CinePile vs 30.34% zero-shot; surpasses mPLUG-Owl3-8B (38.27%) and InternVL2-8B (32.28%); closes to 1.55 points of its 4B teacher (39.89%); transfers zero-shot to ActivityNet-QA (43.19%) and an MLVU subset (50.5% vs base 45.0%, teacher 51.0%). A second, more provocative claim: rationale-after-answer ordering beats standard CoT ordering in compact models. Note the 'outperforms VLMs up to 4x larger' wording holds only against InternVL2-8B; LongVA-7B (41.04%) still leads the student by 2.7 points.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'Answer-then-Rationale' mean and why does it matter?
The model is fine-tuned so the answer comes first and a short 1-2 sentence explanation follows, instead of the standard chain-of-thought order (reason first, then answer). The authors report the standard order degraded their 2B model's accuracy (an 'Option E bias' toward the last choice) while answer-first improved results - but they never publish the competing variant's accuracy, so the effect size is unknown from the paper.
How much data and compute does the method need?
About 900 selected CinePile questions (roughly 0.3% of the ~305k training pairs) with rationales of at most 20 tokens generated by InternVL2-4B, fine-tuned for 3 epochs. Training takes under two hours, though the paper is inconsistent about hardware: the abstract says a single A100, Section 3.3 says a 4xA100 node.
Does the 2B model really beat 8B models?
On CinePile it scores 38.34% average accuracy versus 38.27% for mPLUG-Owl3-8B (a 0.07-point margin from a single run, no variance reported) and 32.28% for InternVL2-8B. It remains below LongVA-7B (41.04%) and its own 4B teacher (39.89%), so 'beats 4x larger models' holds only for InternVL2-8B.
Is code or data available?
No. The paper states code will be released 'upon publication', but nothing is available yet. All ingredients are public (InternVL2 checkpoints, CinePile, ActivityNet-QA, MLVU), so the pipeline is reimplementable, though the MLVU evaluation subset and some training details are unspecified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 are referenced but not analyzable from the extracted text; sample-efficiency curve values (e.g., 34.85% at N=300, 37.81% at N=600) are taken from prose descriptions.
- Supplementary material (qualitative examples, ActivityNet distractor example) was not included in the extracted text.
- Evaluation split sizes are never stated in the text, so variance plausibility judgments are approximate.
- Minor OCR artifacts exist in the references; author-affiliation numbering (1,3,4,2) appears scrambled in the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.