"If I Had to Buy Just ONE: Galaxy S26 Ultra": Auditing AI-Generated Product Recommendations
Lucas G. Uberti-Bona Marin, Thales Bertaglia, Giovanni Astante, Bram Rijsbosch, Gijs van Dijck, Anikó Hannák, Gerasimos Spanakis, Konrad Kollnig
cs.CY · cs.CL
Quick verdict
A rigorous consumer-side audit of 1,536 responses showing AI product advice is unstable and API-based audits miss what consumers see, though causality and generalizability are limited.
Summary
This audit paper examines how ChatGPT, Gemini, and Google AI Overviews answer real product-recommendation queries. The authors curated ConsumerQ, 2,528 commercial-advice queries drawn from LMSYS-Chat-1M and WildChat-1M, and manually filtered 117 physical-product questions, submitting each three times through consumer interfaces, matching APIs, and Google Search from Dutch residential proxies (1,755 observations; 1,536 responses; Sept 4-5, 2026). Framing differs sharply: ChatGPT states a first-person product preference in 79% of product-recommending responses versus 7% for Gemini and 2% for AI Overviews; repeated queries change recommended products (mean Jaccard 0.178 for ChatGPT). Sources diverge: ChatGPT and Gemini share 5.4% of displayed domains per query (no shared domain in 76.7% of comparisons). Crucially, APIs do not proxy interfaces: domain overlap is 12.0% (ChatGPT) and 14.8% (Gemini), with 60.9% of ChatGPT pairs sharing nothing. Limitations: only 117 narrow queries and three repetitions; AI Overviews recovered in just 134 of 351 searches; interface-API gaps may reflect unverified model/configuration differences rather than access method; source-type classification was never validated against humans; no code or data release is mentioned.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Consumer-facing AI systems differ sharply in recommendation decisiveness: ChatGPT states a first-person product preference in 79% of product-recommending responses versus 7% for Gemini and 2% for AI Overviews.
- Recommendations are non-deterministic: repeated identical queries change the recommended products (mean Jaccard overlap of 0.178 for ChatGPT across repetitions), so a single response cannot characterize a system.
- Displayed sources diverge across systems: for the same query, ChatGPT and Gemini share only 5.4% of displayed domains on average, with no shared domain in 76.7% of comparisons.
- Provider APIs do not proxy consumer interfaces: interface-API mean domain overlap is 12.0% for ChatGPT and 14.8% for Gemini, with composition differing across source types and exposed layers.
- Independent audits of AI-mediated commercial advice must therefore incorporate repeated observations, consumer-facing conditions, and the specific source layer observed.
Strengths
Directly tests a methodological assumption underlying most LLM audits
RQ3 pairs each interface request with an API request submitted a median 0.17s later (IQR 0.16-0.19) and finds only 12.0%/14.8% mean domain overlap, with 60.9%/43.4% of pairs sharing no domain - evidence that API-based audit designs may not describe consumer-facing systems at all.
Queries grounded in real user behavior, not templates
ConsumerQ (2,528 queries) is curated from LMSYS-Chat-1M and WildChat-1M with a documented screening pipeline: filter precision 89.0%, three-annotator agreement alpha=0.65 (89.3% unanimous), and Table 2 reports LLM-vs-gold agreement for every downstream annotation task.
Careful, transparent collection design
Unusually candid limitations section
Weaknesses
Thin, fragmented effective samples
Interface-API gaps cannot be attributed to access method
Source-type taxonomy is exploratory yet supports a headline finding
Little inferential statistics for the central comparisons
Narrow scope sits uneasily with the title and framing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does ChatGPT recommend specific products more than Gemini?
In this audit of 1,536 responses to real shopping questions (September 2026), ChatGPT expressed a first-person product preference ('my pick is...') in 79% of product-recommending answers, versus 7% for Google Gemini and 2% for Google's AI Overviews. ChatGPT also labeled a product 'best' in 73% of responses versus 43-48% for the Google systems.
Can researchers audit ChatGPT or Gemini through their APIs instead of the website?
According to this paper, no - not for commercial advice. For identical queries submitted a fraction of a second apart, the ChatGPT API and its consumer website shared on average only 12.0% of displayed source domains (14.8% for Gemini), and 60.9% of ChatGPT pairs shared no domain at all. API-based audits characterize the API condition, not what consumers actually see.
Are AI chatbots consistent when asked the same product question repeatedly?
Only partially. Repeating the same query three times produced different recommended products: mean Jaccard overlap between repeats was 0.421 for AI Overviews, 0.287 for Gemini, and just 0.178 for ChatGPT. Displayed sources also kept changing - ChatGPT's distinct observed domains per question grew from 2.65 after one request to 5.56 after three.
How was this study conducted?
The authors curated 2,528 real commercial-advice queries from public chat logs (LMSYS-Chat-1M, WildChat-1M), filtered 117 physical-product questions, and submitted each three times to ChatGPT, Gemini, and Google AI Overviews - via both consumer websites and developer APIs - using rotating Dutch residential proxies in September 2026, yielding 1,536 analyzable responses.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-7 are reduced to inline numeric series in the extracted text; confidence intervals and visual patterns could not be verified from the rendering.
- Table 1 and Table 2 survive extraction, but the Query-and-advice-type codebook (Appendix B.2) shows arrow-artifacts from markdown conversion; wording was reconstructed in context.
- Whether a code or dataset release accompanies the paper (beyond the public source corpora) is not stated anywhere in the provided text.
- Review is based on the provided full text only; no external verification of collection dates, model names, or cited regulatory facts was performed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.