Cite or Decline: A Strict Course-Grounded Chatbot for STEM Lecture Videos
S M Masrur Ahmed, Jaspal Subhlok
cs.CL · cs.CY · cs.IR
Quick verdict
Honest, well-instrumented deployment study of a course-isolated, citation-first lecture-video RAG chatbot; gains modest, grounding audit is LLM-only.
Summary
This paper reports a two-phase, semester-long deployment of a retrieval-augmented chatbot on the VideoPoints lecture-video platform at the University of Houston. Phase 1 (Fall 2025) tested LLM-generated chapter summaries with 128 survey respondents; Phase 2 (Spring 2026, 93 days) deployed a course-grounded chatbot that retrieves only from the active course — dense (all-MiniLM-L6-v2) plus BM25 plus a chapter-summary soft prior — and answers via Gemini-2.5 flash-lite with exactly seven clickable timestamped citations. Across 833 messages in 233 active sessions, citation coverage was 70.5% (95% CI 67.3–73.5), none of 4,109 citation events crossed a course boundary, and a weighted LLM-assisted audit of 224 unique messages estimated 65.0% full citation support and 61.9% appropriate refusals. On the real-world split of EduVidQA (269 questions), the full design raised video hit from 0.684 to 0.747 over dense-only retrieval (+6.3 points, p=0.0115); removing course isolation cost 17.8 points, the largest factor, while effect sizes were small (dz≈0.14–0.16). Surveys of 22 self-reported users rated citations highest (4.55/5) and practice-question generation the strongest unmet need (4.59/5). Main limitations: a single institution with 78.8% of messages from one course; session-level rather than student-level analysis; no learning-outcome measurement; fully automated labeling and auditing without human validation; and no code or data release.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A course-isolated, citation-first RAG chatbot over lecture videos is deployable in real coursework, achieving 70.5% citation coverage (95% CI 67.3–73.5) over 833 production messages with zero cross-course citations among 4,109 citation events.
- A weighted LLM-assisted audit estimates that 65.0% of cited non-refusal answers are fully supported and 61.9% of refusals are appropriate, demonstrating that citation presence is distinct from citation support.
- On EduVidQA's real-world split (269 questions), the full design improves video hit from 0.684 to 0.747 over dense-only retrieval (+6.3 points, p=0.0115), and removing course isolation costs 17.8 points — more than any scoring change.
- Students value timestamped citations most (4.55/5 among 22 self-reported users) and most strongly request practice-question generation (4.59/5), a task the retrieval-only design refuses about 78.7% of the time.
- Effective deployment depends on course isolation, supported citations, and alignment with student study practices — explicitly scoped to usage, perception, and grounding coverage, not learning outcomes.
Strengths
Decomposes outcomes most deployments conflate
Separates citation presence from citation support and refusal appropriateness (Table 4), empty sessions from active use (447 of 680 sessions excluded, Table 2), and total traffic from concentration (78.8% of messages from one course) — an evidence structure most production RAG papers never attempt.
Transparent weighted audit design with disclosed blind spots
A.4.1 spells out strata, weights (9.13/5.05), census of small cells, and judge configuration (temp 0.0, 1,106/1,107 parses); A.4.2 honestly discloses that the judge's course inventory built from cited passages biases answerable-but-missed estimates toward zero, and no answerable-but-missed population estimate is claimed.
Controlled benchmark with genuine reproducibility hygiene
Unusually candid limitations and label-quality reporting
Solid privacy and ethics handling
Weaknesses
Course-isolation headline is true by construction
Every quality judgment is machine-vs-machine
Deployed configuration not identifiable from the tables
Perception claims rest on 22 self-selected users
No code, prompts, or artifacts released; segmentation sensitivity untested
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a deployment/field study, not an architecture paper — the authors say so explicitly ('The contribution is not a new RAG architecture'). The system runs on the VideoPoints lecture-video platform at the University of Houston: Phase 1 (Fall 2025) generated chapter summaries (Gemini-2.5 flash-lite segmentation, GPT-4.1 Mini summaries; 128 survey respondents); Phase 2 (Spring 2026, Feb 23–May 26, 93 days) deployed a chatbot that hard-filters retrieval to the active course, fuses dense all-MiniLM-L6-v2 similarity, BM25, and a bullet-level chapter-summary prior (weights 1.0/1.25/0.4, per-query min–max normalized; Eq. 1, A.2.1), and generates with Gemini-2.5 flash-lite under a timestamped-grounding prompt with a fixed seven-citation output.
Central claims: (1) production grounding behavior — 70.5% citation coverage (Wilson CI 67.3–73.5) over 833 messages, 0/4,109 cross-course citations; (2) a weighted LLM audit estimating 65.0% full citation support and 61.9% appropriate refusals; (3) a four-arm EduVidQA real-world-split evaluation (269 questions) where the full design beats dense-only by +6.3 points of video hit (0.684→0.747, p=0.0115) and course isolation dominates (−17.8 points when removed); (4) perception — citations 4.55/5 among 22 self-reported users, practice-question generation the strongest unmet need (4.59/5). Claims are explicitly scoped to usage, perception, and grounding coverage, not learning outcomes (§4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Cite or Decline lecture-video chatbot?
It is a retrieval-augmented chatbot deployed on the VideoPoints platform at the University of Houston that answers student questions using only the active course's lecture transcripts and slide text. It combines dense sentence embeddings (all-MiniLM-L6-v2), BM25, and a chapter-summary relevance prior, generates answers with Gemini-2.5 flash-lite, and always returns seven clickable citations with video timestamps. When no course evidence matches, it declines to answer rather than guessing.
How often were the chatbot's answers actually supported by the cited lecture segments?
In production, 70.5% of 833 student messages received citations. A weighted LLM-assisted audit of 224 unique messages estimated that 65.0% of cited, non-refusal answers were fully supported (95% CI 55.0–75.0) and 86.3% fully or partially supported. These are automated diagnostic estimates; the paper reports no human-validated factuality rate, and at least one grounded-but-incorrect answer was confirmed.
Why does course isolation matter for a course chatbot?
The chatbot retrieves only from the course the student is in, which instructors required. In the paper's controlled EduVidQA evaluation (269 questions), removing course isolation cut video hit from 0.747 to 0.569 (−17.8 points) — a bigger drop than changing any scoring component — because unrestricted retrieval often confidently surfaces plausible-looking evidence from the wrong course.
Did the VideoPoints chatbot improve student learning?
No learning outcomes were measured. The study covers usage logs (833 messages, concentrated around midterms and finals), citation and grounding behavior, and survey perceptions from 22 self-reported users, who rated citations 4.55/5 and ease of use 4.64/5. The strongest unmet request was practice-question generation (4.59/5), which the retrieval-only design usually refused.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–5 appear only as captions in the extraction; their image content could not be inspected, so interface-level descriptions were taken on trust.
- Inline math is extraction-garbled (e.g., Eq. 1 and the DCG@5 summation render loosely), but all numeric tables parsed cleanly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.