Do LLMs Know Your Neighborhood? Auditing LLM Priors for Neighborhood-Level Mobility Prediction and Structural Alignment
Saad Mohammad Abrar, Eesha Kurella, Arnav Dadarya, Naman Awasthi, Kazi Tasnim Zinat, Vanessa Frias-Martinez
cs.LG · cs.CY
Quick verdict
Zero-shot LLMs partly predict CBG mobility (0.435 best vs 0.580 supervised) via invariant, biased priors; sound audit, but 2021 pandemic data limits validity.
Summary
This empirical audit evaluates whether zero-shot LLMs can infer aggregate, neighborhood-level mobility outcomes from sociodemographic and built-environment context. Using anonymized Cuebiq traces from 2021 (~977k devices, 8,756 Census Block Groups across Atlanta, Los Angeles, Miami, and San Francisco metros), the authors construct eight CBG-level outcomes (point-, trajectory-, and temporal-level) and prompt ten LLMs, at temperature 0, to predict tertile classes given ACS-2019 and EPA-SLD predictors; supervised baselines (multinomial logit, random forest, boosting) are trained on the same features. The best supervised model averages 0.580 accuracy versus 0.435 for the best LLM (chance 0.333); spatial-extent outcomes are most predictable, entropy outcomes weakest, Gemma3-27B leads most tasks, and Claude-Sonnet-4.5 the temporal one. A second contribution, directional alignment analysis, compares OLS coefficient signs with LLM-implied predictor pushes via one-sided Jonckheere–Terpstra tests over quintile buckets. LLM priors prove largely invariant across outcomes and cities; Black population share is uniquely mapped to lower mobility (25/25 clear cases), while Asian and Hispanic/Latino shares receive positive priors despite mostly negative empirical coefficients. Main limitations: ground truth comes from the pandemic year 2021, the RQ2 system prompt itself prescribes a per-predictor Facilitator/Restrictor logic that may induce the reported invariance, only accuracy is reported, and neither code nor the restricted Cuebiq data are available.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Zero-shot LLMs partially recover CBG-level mobility from sociodemographic and built-environment context but stay below supervised baselines (best LLM 0.435 vs best baseline 0.580 average three-class accuracy; chance 0.333).
- Spatial-extent outcomes (convex hull diameter, ellipse area, radius of gyration) are the most predictable from context and show the largest LLM-baseline gaps, while entropy outcomes are weakest for both model classes.
- LLM-implied predictor directions are largely invariant across mobility outcomes and metropolitan areas (9/10 predictors receive a single fixed direction among clear JT trends).
- Gemma-3-27B uniquely assigns Black population share a mobility-reducing directional prior (25/25 clear cases negative) while giving Asian and Hispanic/Latino shares positive priors despite mostly negative empirical OLS coefficients.
Strengths
Genuinely novel problem framing
Shifts LLM-mobility evaluation from individual trajectories/next-POI (Section 2.1) to aggregate CBG-level outcomes, a real gap given that planners need aggregate demand, not single-device traces (Section 1).
Original directional-alignment methodology
Section 3.4 introduces OLS-oriented LLM pushes (Eq. 18) tested with two one-sided Jonckheere-Terpstra tests across quintile buckets with BH correction per CBSA-outcome panel — a reusable auditing instrument beyond accuracy.
Large, carefully filtered real-world evaluation grid
Quantified, socially consequential bias finding
Honest capability framing
Weaknesses
Pandemic-era ground truth never discussed
RQ2 prompt scaffolding risks manufacturing the headline finding
No significance testing for accuracy claims
OLS sign as gold standard is fragile and underspecified
Reproducibility and manuscript polish
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether zero-shot LLMs can recover **aggregate, neighborhood-level** mobility outcomes from contextual urban features, and whether their implied context-mobility relationships match empirical ones (RQ1/RQ2, Section 1). This deliberately repositions the LLM-mobility literature, which has overwhelmingly evaluated individual trajectories, next-POI, or synthetic diaries (Section 2.1). The claims are measured: LLMs "partially recover" aggregate patterns (0.435 vs 0.580 supervised average accuracy, chance 0.333), rely on coarse outcome- and city-invariant priors, and exhibit asymmetric treatment of protected-group predictors (Abstract; RQ2 Findings 1-3). The headline finding - Gemma-3-27B uniquely assigns Black population share a negative mobility prior in 25/25 clear JT cases while assigning positive priors to Asian and Hispanic/Latino shares despite negative OLS coefficients in 27/28 and 25/28 relationships respectively - is socially consequential and framed as an audit result, not a capability claim.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can LLMs predict neighborhood-level mobility without any training data?
Only partially. In this study the best zero-shot LLM (Gemma3-27B) reached about 0.435 three-class accuracy across eight mobility outcomes in four U.S. metro areas - above the 0.333 chance level, but clearly below the 0.580 achieved by supervised models trained on local labels.
What is 'directional alignment' in this paper?
It is an audit that checks whether the direction of influence an LLM claims for each predictor (e.g., does higher income push mobility up or down?) matches the sign of empirical OLS regression coefficients, evaluated with Jonckheere-Terpstra trend tests across predictor quintile buckets.
Did the study find demographic bias in LLM mobility predictions?
Yes, an asymmetric pattern: the best model applied a uniquely consistent mobility-reducing prior to Black population share (25 of 25 clear cases negative), while assigning positive priors to Asian and Hispanic/Latino population shares even where the empirical coefficients were mostly negative.
Is the code or data from this paper publicly available?
The paper states no code release, and the Cuebiq mobility data are access-restricted through an audited cleanroom program, so fully independent replication is currently difficult.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure interiors (Fig. 1 pipeline, Fig. 3 accuracy bars, Fig. 4 OLS/JT heatmaps, Fig. 5 example, Fig. 9 distributions) are image content not present in the extracted text; assessments of them rest on their captions plus numeric values quoted in body text and Tables 2-4.
- The extracted text never enumerates the full contextual-predictor list; only nine predictors are named in the Section 4.3 narrative (median income; Black, Asian, Hispanic/Latino, young-adult shares; density, connectivity, transit access, land-use mix), so the tenth cannot be confirmed from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.