What Users Think of Generative AI: A Cross-Platform NLP Analysis of Trust and Friction in App Store Reviews
Md Jafrin Hossain, Umme Nusrat Jahan, Shouvaggo Sharif Shammo
cs.CL · cs.HC · cs.IR
Quick verdict
NLP mining of 17,012 GenAI app reviews via validated BERTopic+RoBERTa; friction and polarization findings credible; self-selection biases warrant caution.
Summary
The paper analyzes 17,012 English app-store reviews of six generative-AI apps (ChatGPT, Gemini, Microsoft Copilot, Claude, DeepSeek, Perplexity), collected May 2026 from Google Play and Apple RSS feeds. It combines BERTopic topic modeling (all-MiniLM-L6-v2 embeddings, 24 retained topics, 73.1% coverage) with RoBERTa sentiment classification, chi-square/Kruskal-Wallis/Mann-Whitney tests with Bonferroni correction, and a multinomial logistic regression (app, topic cluster, platform, length, month). Validation uses human coding of a stratified 300-review sample. Negativity concentrates on ads (91%), authentication (89%), server reliability (83%) and pricing (73%); Claude shows the highest negativity (47.7%) with an enthusiastic core, and an exploratory DeepSeek geopolitical-trust topic emerges. A proposed Trust Friction Score (TFS) decomposes friction; its correlation with negativity (rho=0.886, n=6) is honestly labeled illustrative. Strengths include layered validation and candid bias quantification. Limitations: weak neutral-class sentiment performance, no released corpus or code, unmodeled self-selection across reviewer populations, and internal inconsistencies.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 2
● low 4
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Negative sentiment in GenAI app reviews concentrates on specific friction topics: advertising (91% negative), authentication (89%), server reliability (83%), and subscription pricing (73%).
- Sentiment distributions differ significantly across the six applications (chi-square(10)=561.34, p<.001, V=0.128), with Claude highest in negativity (47.7%) and Copilot lowest (25.3%), robust to unequal corpus sizes in 1,000 balanced subsamples.
- Claude exhibits verified polarization, pairing the highest negative-sentiment rate with an enthusiastic core user base (T08 positive rate 78%; 32.3% one-star vs 40.2% five-star).
- After controlling for topic, platform, length, and month, Claude retains elevated negative-sentiment odds (OR 1.56, 95% CI 1.37-1.79) relative to ChatGPT, indicating an app-level effect beyond topic composition.
- An exploratory subset of DeepSeek reviews expresses geopolitical and data-privacy distrust tied to its Chinese origin (T13, n=253, 47% negative), and a proposed Trust Friction Score decomposes per-app friction into actionable sub-dimensions.
Strengths
Three-layer validation and effect-size reporting
Human sentiment coding of a stratified 300-review sample (inter-coder kappa 0.894; RoBERTa accuracy 75.3%, macro-F1 0.725), a full-corpus star-rating proxy check (accuracy 79.3%), plus effect sizes (Cramer's V, eta-squared, rank-biserial r) reported alongside every test (Secs. 3.6-3.7, Tables 5-6). Negative-class precision 0.895 underpins the paper's headline friction findings.
Candid quantification of the length-filter selection bias
Appendix C / Table 12 shows the 10-word filter dropped 34,530 of 52,080 reviews that were markedly more positive (mean 4.41 vs 3.54 stars; 76.6% vs 50.8% five-star), and the authors explicitly re-read absolute negativity as upper bounds (Sec. 6). Balanced-subsampling (1,000 draws at n=812/app) and platform stratification (Table 10) support Claude's #1 negativity rank in 99.2% of draws.
Honest treatment of TFS and exploratory claims
Negative methodological result on BERTopic is valuable
TFS ablation isolates the Claude story cleanly
Weaknesses
Cross-app comparison does not separate who reviews from what is reviewed
Neutral-class sentiment behavior could distort cross-app rankings
Internal inconsistencies across sections
No code, corpus, or codebook availability statement
Data provenance vs claimed coverage is under-documented
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper mines 17,012 English-language reviews of six GenAI apps (ChatGPT, Gemini, Copilot, Claude, DeepSeek, Perplexity; Table 1) collected May 2026 from Google Play (48,000 raw) and Apple RSS (6,748, three apps only), spanning Sep 2025-May 2026. It answers three RQs: dominant concern themes, cross-app/platform sentiment differences, and which trust/usability factors drive negativity. Headline results: friction topics dominate negativity (T16 ads 91%, T05 authentication 89%, T14 server reliability 83%, T12 pricing 73%); per-app negativity runs Copilot 25.3% < ChatGPT 30.9% < DeepSeek 31.8% < Gemini 38.7% < Perplexity 42.3% < Claude 47.7% (Table 3); DeepSeek-only T13 (n=253, 1.5%) surfaces language/geopolitical trust; a prevalence-weighted Trust Friction Score (TFS) ranks Claude first (14.04%). Claims are mostly well-hedged: the TFS-negativity correlation (rho=0.886, n=6) is explicitly illustrative, and T13 is framed as indicative, not prevalence.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this study find about user complaints in ChatGPT, Gemini, Copilot, Claude, DeepSeek and Perplexity app reviews?
Analyzing 17,012 English app-store reviews with BERTopic topic modeling and RoBERTa sentiment analysis, the study finds negative sentiment concentrated on advertising (91% negative), sign-in/authentication problems (89%), server reliability (83%), and subscription pricing (73%). It also identifies an exploratory geopolitical-trust theme among some DeepSeek reviews and proposes a 'Trust Friction Score' that decomposes each app's friction into sub-dimensions such as authentication, pricing, and advertising.
Why does Claude have the most negative reviews in this study if users praise its quality?
Claude shows the highest negative-sentiment share (47.7%) and the lowest average star rating (3.19), driven mainly by phone-number verification friction (TFSauth 8.28 of its composite 14.04), message limits, and subscription requirements. At the same time, app-specific Claude feedback is 78% positive and ratings are strongly split (32.3% one-star, 40.2% five-star), which the authors read as a polarized user base: enthusiasts who value model quality versus mainstream users blocked by access friction.
Is this app-review study trustworthy?
It is unusually careful for review-mining work: human coding of a stratified 300-review sample validates both components (sentiment accuracy 75.3%, macro-F1 0.725; topic agreement kappa 0.241, honestly reported as weak for abstract constructs), robustness checks cover balanced subsampling and platform stratification, and the length-filter bias is quantified. Caveats: the sentiment model's neutral class is weak, the corpus and code are not released, and cross-app differences may partly reflect different reviewer populations rather than product quality.
How much data did the length filter discard, and does it bias the results?
Of 52,080 English-language reviews, 34,530 (66%) were under 10 words and removed. Discarded reviews were far more positive than retained ones (mean 4.41 vs 3.54 stars; 76.6% vs 50.8% five-star). The authors therefore state that absolute negativity percentages are upper bounds relative to the full reviewer population, while arguing cross-app comparisons remain valid because they are computed within the same filtered frame.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text extraction is fragmented in places: the bimodality-coefficient formula is truncated, the Esteban-Ray index definition is missing, and early parts of Sec. 4.4 (thematic validation) are omitted, so those details were judged from surrounding prose.
- All figures (heatmaps, word clouds, confusion matrices) are referenced but not visually analyzable from the extracted text.
- Some discussion sections (parts of Secs. 4.4, 5.1-5.3) were elided in extraction; inconsistency judgments rely on the visible passages.
- arXiv id 2609.19151 and 2026 dates are taken as given from the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.