CALM: A Calibrated LLM Choice Network Framework for Activity-Based Traveler Simulation
cs.LG · cs.AI
Quick verdict
Claims a reproducible LLM+calibrated-choice traveler simulator; holdout calibration is strong (JSD 0.156→0.004) but live-LLM tests are small, no code stated.
Summary
The paper introduces CALM, a hybrid activity-based traveler simulation framework that couples an optional LLM activity planner (gpt-4o-mini) with a calibrated multinomial-logit choice core, a shared 13-zone network, memory and habit state, typed feasibility checks, and deterministic offline replay from frozen prompt-response pairs. Using the 2024 NYC Citywide Mobility Survey, 110,691 seven-mode trips from 3,368 respondents are split by respondent hash into 78,487 training and 32,204 holdout trips. Alternative-specific constants fitted on training data only reduce mean holdout mode Jensen-Shannon divergence from 0.15599 to 0.00394 across ten seeds. A matched live-LLM ablation over 300 traveler-days per arm shows trade-offs: the choice-only arm best matches mode shares (JSD 0.00111), while LLM arms cut within-traveler switching (0.698 to 0.324 in the full hybrid) at some cost in aggregate fit, with first-pass feasibility above 0.98 and final feasibility 1.0. Rain, delay, fare, and parking ladders produce the prespecified monotone responses. Key limitations: the live-LLM experiment is small and reported without variance estimates; the network is a coarse 13-zone abstraction; only one LLM and one prompt version are tested; stress tests validate direction, not magnitude; and no public code or frozen-response release is stated, so the reproducibility claim rests on protocol description.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Training-only, survey-weighted ASC calibration transfers to a person-disjoint holdout, cutting mean holdout mode JSD from 0.15599 to 0.00394 across ten seeds.
- A matched live-LLM ablation shows LLM planners reduce within-traveler switching (to 0.32444 with full hybrid) and improve trip-time fit while degrading aggregate mode fit and preserving final feasibility of 1.0.
- Frozen prompt-response pairs (614 unique) enable deterministic offline replay of all 900 live traveler-days without additional model calls.
- Controlled rain/delay/fare/parking ladders produce the prespecified monotone exposed-mode-share responses, including a sharp mock-planner rain threshold.
Strengths
Methodologically sound person-disjoint calibration anchor
Training-only simulated-method-of-moments ASC fitting within predeclared [-5,5] bounds, frozen before evaluation on 32,204 holdout trips from 986 unseen respondents; mean JSD drops 0.15599 to 0.00394 across ten seeds with bootstrap CI [-0.16577, -0.14057] (Sec. V-A, Table III).
Matched ablation design with honest trade-off reporting
Purpose and OD held fixed across choice-only / LLM / LLM+memory / full-hybrid arms, isolating module effects; the paper honestly reports that any LLM involvement worsens mode JSD (0.00111 to 0.035-0.045) while improving switching and trip-time fit (Table IV).
Reproducibility engineering rare in LLM-simulation work
Task-aligned evaluation protocol with complementary supervised reference
Explicitly scoped limitations
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CALM (single-author, independent researcher) is a framework plus evaluation-protocol paper: an optional gpt-4o-mini activity planner is embedded in a closed traveler-day loop - planner, typed feasibility, stochastic MNL choice, shared 13-zone network, memory/habit - and each generative component is scored against an empirical anchor built from the 2024 NYC Citywide Mobility Survey (dataset wgnh-qwsg; 110,691 seven-mode trips, 3,368 respondents; 5,229 'other' trips, 1.97% weighted, excluded). Four claims are made: (1) training-only survey-weighted ASC calibration transfers to a person-disjoint holdout, cutting mean mode JSD from 0.15599 to 0.00394 over ten seeds (Sec. V-A, Table III); (2) a matched live-LLM ablation quantifies planner/memory trade-offs (Sec. V-B, Table IV); (3) frozen prompt-response pairs deterministically replay all 900 live traveler-days from 614 cached pairs; (4) rain/delay/fare/parking ladders yield prespecified monotone responses (Sec. V-C, Table V). The scope is deliberately narrow - matched purpose-OD contexts, a 13-zone network, synthetic demographics - and Sec. VII says so honestly.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CALM in traveler simulation?
CALM is a framework that couples an optional LLM activity planner with a statistically calibrated multinomial-logit mode-choice core, a shared zone network, memory/habit state, typed feasibility checks, and deterministic replay from frozen prompt-response pairs, evaluated on the 2024 NYC Citywide Mobility Survey.
How much does calibration improve fit in CALM?
Fitting alternative-specific constants on training respondents only reduces mean holdout mode Jensen-Shannon divergence from 0.15599 to 0.00394 across ten simulation seeds (person-disjoint split: 78,487 training vs 32,204 holdout trips from the 2024 NYC CMS).
Does the LLM planner beat the statistical choice model?
No - LLM arms improve behavioral persistence (switching 0.471 to 0.324 with memory) and trip-time fit but worsen mode-share fit (JSD 0.035-0.045 vs 0.00111 for the choice-only arm); the paper frames this as a trade-off, not a win.
Is CALM's code available?
The paper describes a reproducibility protocol (prompt hashes, frozen responses, deterministic replay) but does not state a public code or artifact release; the underlying NYC CMS dataset is public on NYC Open Data (dataset wgnh-qwsg).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 3 contribute only axis/label text in the extraction; quantitative assessment of E3 relies on Table IV values.
- Figure 3's radar axis values are not recoverable from the linearized figure text; only table values were used for E3 claims.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.