ChatGPT Images 2.5 in the Wild: A Launch-Period Dataset and Detector Evaluation
Dennis Ng, Xingyu Shen, Ankit Raj, Kidus Zewde, Tommy Duong, Yuchen Zhou, Yuxin Zhang, Neo Tiangratanakul, et al.
cs.CV · cs.AI
Quick verdict
A transparent launch-window AI-image dataset with candid attribution limits; honest dataset paper, but detector results are weak and cross-cohort contrasts exploratory.
Summary
Scam.ai authors assemble and release a launch-period collection for ChatGPT Images 2.5: 3,478 standalone images from 2,440 posts across 8 sources, all posted within 51.1 hours of the announcement, gathered via nine collection routes. Three attribution tiers (caption regex, caption plus creation-language classifier, NightCafe host records) are kept separate, and the paper candidly reports that attribution is unverified: model audits agree with caption claims only 64.0% of the time, and all 1,007 inspected API-generated files carry a '2.0' C2PA version string. Six frozen detectors, calibrated to flag 5% of reference photographs, flag 3.7-56.4% of the collection, 42-81 percentage points below their GenImage recall, while held-out artwork false-positive rates span 1.5-96.5%. The exploratory X-only April-versus-September comparison finds higher September flag rates for Effort (+6.1 pp) and suggestively for DoU (+3.6 pp), surviving Bonferroni adjustment and a held-out April check. Attribution, content and processing confounds, and NightCafe's dominance (81% of images in the final window), keep all causal interpretation off-limits. The contribution is the documented dataset and evidence fields, not detector findings.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Version attribution of online images is genuinely ambiguous after a silent generator upgrade: caption claims, host records and inspected C2PA fields are separate, individually insufficient evidence, and the inspected generator-version field reads '2.0' even for 2.5 endpoints.
- At fixed photograph-calibrated thresholds, all six frozen detectors flag 3.7-56.4% of the launch-period collection, 42-81 percentage points below their GenImage recall, while held-out artwork FPR spans 1.5-96.5% — so a high collection flag rate alone does not establish better detection.
- An exploratory X-only comparison finds higher September flag rates for Effort (+6.1 pp) and, suggestively, DoU (+3.6 pp) versus the April GPT-Image-2 release, surviving Bonferroni adjustment and a held-out April check, but identifying collection differences, not a generator-upgrade effect.
- Source composition shapes all pooled observations: NightCafe supplies 39.0% of images, 77.0% of CLIP-assigned fantasy scenes and 81% of images posted in the final observation window.
Strengths
Fully documented, auditable collection protocol
Section 3 and Appendices B-G expose the launch-anchor snowflake timestamp, three attribution tiers (gate_tier: 371 official-X classifier images, 1,749 caption-regex-only, 1,358 NightCafe host-recorded), the full funnel (10,163 candidates → 7,911 attributed → 6,356 deduplicated → 3,478 retained), exact regexes (Appendix F), hash-based dedup with replayable audit, and per-item source/endpoint/timestamp fields in the release.
Self-identified negative result on C2PA version fields
Section 3.6/Appendix K: all 1,007 API-ledger files — including both gpt-image-2.5-flare and -sunburst endpoints — carry generator-version '2.0', and only 31/66 Pixiv files contain C2PA markers (byte scan, no signature validation). This is a genuinely useful finding against automated version attribution.
Harness validated before interpretation
Unusually careful uncertainty handling
Weaknesses
Generator attribution is unverifiable at the pixel level
The 42-81 pp recall gap is confessedly unexplained
The headline cross-cohort contrast is exploratory and confounded
Severe source imbalance and instrument confounding
Audit evidence is thin for some components
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a dataset-and-evaluation paper from Scam.ai covering a frozen launch-period collection for ChatGPT Images 2.5 (released 8 September 2026 via codenamed endpoints flare and sunburst): 3,478 standalone images from 2,440 posts across 8 sources, posted within 51.1 hours of the announcement, plus 12 separately labelled videos excluded from image statistics. Four claims are made: (1) documentation of version-attribution evidence across three tiers; (2) source-composition analysis; (3) a fixed-threshold evaluation of six frozen detectors; (4) an exploratory April-vs-September contrast. The paper repeatedly disclaims causal identification, defining "in the wild" as retrieval under stated admission rules and stating the collection is not a census (Section 1). This discipline is the paper's defining feature: nearly every number arrives pre-qualified.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ChatGPT Images 2.5 launch-period dataset?
A released collection of 3,478 standalone AI-generated images from 2,440 online posts across 8 platforms (X/Twitter, NightCafe, Xiaohongshu, note.com, Pixiv, Weibo, Instagram, Reddit), all posted within about 51.1 hours of the 8 September 2026 announcement of ChatGPT Images 2.5, plus 12 separately labelled videos. Each image carries attribution-tier, source, language and timestamp metadata.
Can detectors reliably tell if an image was made by ChatGPT Images 2.5?
Not from this data. The paper shows attribution is unverified — caption claims agreed with an independent model audit only 64.0% of the time, and OpenAI's own C2PA generator-version field read '2.0' in all 1,007 API-generated files inspected, including 2.5 endpoints. Six published detectors flagged between 3.7% and 56.4% of the collection at fixed thresholds, far below their benchmark recall.
Why do detector flag rates differ so much from benchmark results like GenImage?
Because the collection differs from benchmarks in four stacked ways: some images may be misattributed, social platforms recompress images, the content is heavily non-photographic (much anime/fantasy art), and some detectors were trained on benchmark-like data. The paper reports the 42-81 percentage-point gap without assigning it to any single cause.
Is ChatGPT Images 2.5 easier or harder to detect than the previous model?
The paper cannot say causally. At fixed thresholds, two detectors (Effort, DoU) flagged the September X-only sample more often than an April GPT-Image-2 sample, but the comparison is explicitly exploratory, the cohorts differ in admission rules and content, and confounds could explain the difference.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Appendices A-O are present in the extraction, but figures (subject distributions, timing histograms, funnel diagram) are referenced only via captions; numeric values cited come from the text and tables.
- Companion paper [19] (forgery tasks) is cited as 'in preparation' and not reviewed.
- The dataset URL (scam.ai/research) is stated but not verified as live; release completeness is assumed.
- Dates, model names (gpt-5.4-mini, gpt-5.5) and citations reflect the paper's internal 2026 timeline as printed in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.