Analyzing Public Discourse on Urbanism: Topic Clustering, Sentiment Analysis and Retrieval-Augmented Generation using YouTube Comments
Jakob Morales, Monica Hegde, Fayeq Jeelani Syed, Fayeq Jeelani Syed
cs.CL · cs.AI
Quick verdict
Urbanist YouTube RAG pipeline with honest negative results, but eval rests on 20 queries, 300 labels, one non-blind rater; findings useful, evidence thin.
Summary
The authors build a pipeline and chatbot that turn YouTube transcripts and comments about urbanism into queryable, per-city sentiment profiles: 218,177 raw records from 743 videos are length-filtered to 169,548, topic-relevance-filtered to 93,651 (55.2%), and chunked into 22,788 retrieval units across 309 US/Canada cities resolved by spaCy NER plus fuzzy gazetteer matching. BERTopic topics and six-aspect keyword labels annotate each chunk; Llama 3.2 generates grounded answers. The evaluation centers on transferable measurement lessons: a Twitter-tuned RoBERTa beats VADER by 12.6 macro-F1 points (0.589 vs 0.464; McNemar p=0.0001) on 300 hand-labeled comments, yet both models collapse on the neutral class, and the two annotators themselves reach only kappa=0.53. Dense FAISS retrieval beats TF-IDF (P@5 0.790 vs 0.560 over 20 queries); a single non-blind rater's chunk-level check puts dense P@10 at 0.94 versus 0.660 under video-level proxies. BERTScore proves uninformative under summary-vs-comment length mismatch, and ROUGE-1 groundedness (mean rate 23.3%, ranging 10-35% across three generation runs) is argued to be a paraphrase floor rather than a hallucination rate. Main limitations, mostly self-reported: no corpus-side geoparsing audit (London conflated with its UK namesake in Table 1), an un-evaluated relevance filter discarding 44.8% of data, no ablations, no code or data release, and groundedness instability partly attributable to a template fallback.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A Twitter-tuned RoBERTa classifier outperforms VADER by 12.6 macro-F1 points (0.589 vs 0.464, McNemar p=0.0001) on urbanist YouTube comments, yet both models collapse on the neutral class, which annotators themselves also find ambiguous (kappa=0.53).
- Dense FAISS retrieval beats a TF-IDF baseline at every cutoff on 20 held-out queries (P@5 0.790 vs 0.560), and video-level relevance proxies substantially understate chunk-level precision (0.660 vs 0.94 under a single-rater human check of the dense system only).
- BERTScore is uninformative when comparing a multi-sentence generated summary against a single short comment (scores ~0.83-0.86 regardless of relevance), and ROUGE-1-based groundedness (mean rate 23.3%, range 10-35% across runs) is a paraphrase-driven lower bound rather than a hallucination rate.
- The pipeline turns 218,177 raw YouTube records into 22,788 annotated retrieval chunks over 309 North American cities, yielding per-city, per-topic sentiment profiles queryable through a grounded RAG chatbot.
Strengths
Rare candor: Section 8 pre-empts nearly every objection a reviewer would raise
The limitations section self-reports the single non-blind rater behind the 0.94 chunk-precision figure (5.5), the unseparated template-fallback contamination of groundedness (5.7), the un-evaluated relevance filter discarding 44.8% of data (3.2), the unaudited corpus-side geoparsing including the knowingly conflated 'London' row of Table 1, and the missing ablations. Claims are consistently scoped to what the data supports.
Transferable evaluation findings with concrete measurements
BERTScore flatness (~0.83-0.86 regardless of relevance) under summary-vs-short-comment length mismatch; sentence-embedding cosine separating relevant (~0.65-0.75) from irrelevant (-0.10 to 0.25) pairs; video-level proxy understating chunk-level precision (0.660 vs 0.94). These are directly useful to anyone building RAG over short user-generated text.
Statistically careful sentiment comparison
Well-motivated annotation rule and thorough ethics treatment
Weaknesses
Evaluation base is very small and partially biased
Groundedness estimate is unstable and contaminated
Silent pipeline components with corpus-wide consequences
Missing internal baselines and ablations
No code, data, or evaluation-set release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes three claims of increasing generality: (1) a corpus-plus-system contribution - 218,177 raw YouTube records reduced to 22,788 annotated retrieval chunks across 309 North American cities, powering per-city sentiment profiles and a grounded chatbot (Contributions 1-2); (2) an annotation rule separating topic sentiment from creator sentiment (Contribution 3); (3) three transferable evaluation findings for RAG over short user-generated text (Contribution 4). The framing is appropriately modest: the geoparsing component is explicitly "deliberately lightweight rather than novel" (3.3), and the claimed contribution is measurement, not architecture.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper do?
It builds a pipeline and chatbot that turn YouTube transcripts and comments about urbanism (walkability, transit, cycling, housing density, safety) into per-city sentiment profiles. About 218,000 raw records are filtered and chunked into 22,788 retrieval units covering 309 US and Canadian cities, and a Llama 3.2-based system answers questions like 'Is Chicago walkable?' using retrieved comments as evidence.
What are the paper's main findings?
Three measurement lessons: (1) a Twitter-tuned RoBERTa sentiment model beats VADER by 12.6 macro-F1 points on 300 hand-labeled comments, but both fail on neutral comments, which even human annotators disagree on (kappa=0.53); (2) dense retrieval beats TF-IDF (P@5 0.790 vs 0.560), and judging relevance at the video level badly understates true chunk-level precision (0.660 vs 0.94); (3) BERTScore is unusable for checking whether a generated summary is grounded in a single short comment, and ROUGE-1-based groundedness is a conservative lower bound, not a hallucination rate.
Is the dataset or code available?
No public release is stated in the paper. The authors say any future release would consist of video and comment identifiers plus derived annotations (city, aspect labels, sentiment) rather than comment text, so other researchers could rehydrate the corpus via the YouTube API, but no repository, validation labels, or evaluation queries are currently published.
Can these city sentiment profiles be treated as public opinion?
No. The authors explicitly warn that the corpus comes from urbanist-focused channels, so it reflects a self-selected audience already engaged with urban planning, not city residents in general; the profiles describe online discourse only and should not inform policy decisions without independent, representative data.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (system architecture) cannot be assessed from text extraction; stage-level claims rely on the Methods prose.
- The extraction appears complete (abstract through references and Appendix A present), but table formatting was reconstructed from linearized text, so individual cell values were cross-checked against the prose where stated.
- Per-run, per-query groundedness details beyond Table 6 are unavailable, limiting judgment on run-to-run variance structure.
- arXiv listing (2609.22705, dated Sep 2026) is treated as given; no external verification was performed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.