Evaluating Personal Information Output from Conversational Interactions in Generative AI Systems
cs.CL · cs.AI · cs.CY
Quick verdict
Exploratory n=15 study: ChatGPT infers personal traits from chat memory with little model-variant difference; underpowered, inconsistent stats, no code.
Summary
This exploratory pilot study examines how much personal information ChatGPT (paid tier; GPT-5.2 Instant and GPT-5.2 Thinking) outputs about its users from ongoing conversational memory. Fifteen Japanese paid-tier users (mean age 36.6) ran a fixed prompt asking the model to profile 18 attributes across seven categories, distinguishing Fact (explicitly stated), Inference, and a model-assigned 1-6 Confidence score; participants then rated Fact and Inference accuracy on a 1-6 Likert scale, and 'Not applicable' responses were counted as nulls. High-score (5-6) proportions were similar across models (Fact 77.88% vs 76.02%; Inference 71.34% vs 72%; Confidence 60.25% vs 60.44%), and Wilcoxon tests found no significant between-model differences. In GPT-5.2 Thinking, null outputs were significantly more frequent for Fact than Inference (chi-square(1)=8.45, p<.01), suggesting conservatism on explicit factual claims, while Psychological/Cognitive, Residual, and Holistic Profile attributes were always inferred. Behavioral and Linguistic attributes scored highest (languages used 92.86% in Fact), whereas Core Personal attributes drew the most withholding (place of origin: 60% Fact nulls). Model Confidence correlated moderately with perceived Inference accuracy (rho=0.41, p<.01). Key limitations: only 15 participants from a single language and culture, subjective single-rater validation without ground truth, undisclosed memory mechanics, Wilcoxon effect sizes inconsistent with reported p-values, and no code or data release.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GPT-5.2 Instant and Thinking show no statistically significant differences in personal-information output scores or null rates, suggesting model design has limited impact under these conditions.
- GPT-5.2 Thinking withholds Fact outputs significantly more often than Inference outputs (χ²(1)=8.45, p<.01), indicating conservatism on explicit factual claims.
- Core Personal attributes are withheld most, while Psychological/Cognitive, Residual, and Holistic Profile attributes are always inferred even without explicit factual support.
- Model-assigned Confidence correlates moderately with human-perceived Inference accuracy (ρ=0.41, p<.01) but diverges at intermediate accuracy levels.
- Weekly usage frequency shows no systematic association with output accuracy or null tendencies.
Strengths
Ecologically valid setting
Real paid-tier users' own live ChatGPT conversation memory (Section II-A), complementing the curated-corpus inference literature it builds on (Staab et al., ICLR 2024; Yukhymenko et al., NeurIPS 2024; Peters & Matz, PNAS Nexus 2024).
Original null-output analysis
The explicit 'Not applicable' protocol enables the paper's most novel results: withholding concentrates in Core Personal (place of origin: 60% Fact nulls, Table 1) while Psychological/Cognitive, Residual, and Holistic Profile have zero Inference nulls, and null occurrence is uncorrelated with score levels (Section III-D: ρ=.03, p=.90 for Inference; ρ=−.18, p=.49 for Fact).
Disciplined multiplicity control
Reproducible prompt and taxonomy
Weaknesses
Statistically inconsistent effect sizes
Clustered data analyzed as independent
Equivalence overclaim from null results
Subjective single-rater validation
Undefined denominators and no release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This IIAI-AAI 2026 preprint asks a privacy question with an unusually direct design: how much personal information does a production generative AI system output about its own users, drawing only on accumulated conversation memory? Fifteen Japanese paid-tier ChatGPT users ran a fixed elicitation prompt (Section II-B, quoted verbatim — an asset) asking the model to profile 18 attributes across seven categories (Core Personal, Behavioral, Linguistic, Professional, Psychological & Cognitive, Residual, Holistic Profile), separating Fact (explicitly stated), Inference, and a model-assigned 1–6 Confidence score. Participants then self-rated Fact/Inference accuracy on a 1–6 Likert scale; 'Not applicable' counted as a null output. The paper claims: (1) no meaningful difference between GPT-5.2 Instant and Thinking; (2) Fact output is more conservative than Inference; (3) category-level asymmetries (Core Personal withheld; Behavioral/Linguistic accurate; Psychological/Holistic always inferred); (4) Confidence moderately tracks perceived accuracy; (5) usage frequency is irrelevant.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can ChatGPT reveal your personal information from chat memory?
In this pilot with 15 paid-tier users, GPT-5.2 models reproduced explicitly stated facts with high perceived accuracy (77.88% and 76.02% high scores) and always offered psychological and holistic inferences with no 'Not applicable' responses, though they withheld more often on identity attributes such as place of origin (60% nulls in Fact) and name.
Is GPT-5.2 Thinking more privacy-conscious than GPT-5.2 Instant?
This study found no statistically significant differences between the two variants in personal-information output scores or null rates; Thinking produced slightly more nulls (e.g., 27.41% vs 22.96% for Fact), but the difference was not significant with 15 participants.
Which personal attributes do LLMs infer most easily in conversation?
Behavioral and linguistic attributes scored highest in this study — languages used reached 92.86% high-accuracy ratings in Fact outputs and interests 85.71% — while core identity attributes (place of origin, family composition, name) drew the most withholding and lower perceived accuracy.
Can you trust a model's self-reported confidence about its inferences on you?
Only moderately: the model's 1–6 confidence correlated with users' rated inference accuracy (ρ=0.41, p<.01), but at intermediate accuracy levels confidence varied widely, so it cannot substitute for actual accuracy.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are images; their exact cell values are available only to the extent quoted in the prose and Table 1 (which is fully present in the text).
- A minor PDF-extraction artifact renders the Table 1 label 'Holistic Profile' as 'A Holistic' and inserts spacing in headings (e.g., 'R ESULTS'); treated as formatting noise, not content loss.
- The provided text runs complete from the title through reference [7]; no end truncation was detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.