SlopShape: Identifying AI-Generated Commercial Web Content
cs.CL
Quick verdict
Claims structure alone detects AI blog posts at 98.0 F1, robust to rewording; rigorous replication, but year and brief confounds limit interpretation.
Summary
The paper asks whether AI-generated commercial blog content can be detected from structural signatures — how information is ordered, evidenced, and voiced — rather than word choice. It replicates StoryScope's fiction-study pipeline on 2,250 pre-ChatGPT company blog posts (268 domains, Wayback snapshots 2008-2022) against 11,250 AI mirrors from five frontier models written from reverse-engineered briefs. An LLM-discovered 214-feature instrument (187 structural features), scored by an LLM and validated by human annotation (human-model kappa 0.946), detects AI posts at 98.0 macro-F1 on domain-disjoint held-out data — and stays at 98.1 after every AI test post is reworded by its own model (73% of 13-gram sequences replaced). Six-source attribution reaches 79.3% accuracy versus a 16.7% chance rate; human posts occupy rarer structural configurations (mean rarity percentile 0.838 vs 0.435, d=1.83). Ten core features alone reach 93.5 F1. Key limitations: publication year is inherently confounded with authorship (a year-prediction check mitigates but a same-period comparison remains open); AI mirrors condition on lossy briefs; the attack tested is self-rewording only, not commercial humanizer tools; and the sole author operates a commercial GEO product, with company-affiliated annotators.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Structural features alone (187) detect single-pass AI-generated commercial blog posts at 98.0 macro-F1 on domain-disjoint held-out companies, and detection is unchanged (98.1) when every AI test post is reworded by its own model.
- The structural signal attributes authorship: 79.3% six-source accuracy against a 16.7% chance rate, with human posts near-perfectly separated (per-class F1 0.966).
- Human posts occupy statistically rarer structural configurations (mean rarity percentile 0.838 vs 0.435, Cohen's d = 1.83; rarest 1%: 149 human vs 4 AI), while the five AI models crowd common configurations.
- Every effect replicates StoryScope's fiction findings in direction with larger magnitude, via an LLM-run pipeline whose validity is anchored by human gold annotation (human-model kappa 0.946).
Strengths
Replication discipline with a declared deviation register
A 16-item register (Tables 2/E1) defends each departure from StoryScope; splits are domain-disjoint (198/32/38 domains), stricter than the original's random prompt split; feature discovery ran on a 100-prompt set excluded from all classifier splits, with a sensitivity check for the 11 domains it shares with the test split (headline 98.2 excluding them).
Instrument validity exceeds the replicated study
Repeatability Krippendorff alpha 0.891 across five scoring runs; gold-annotation acceptance thresholds fixed before annotation and cleared at human-human kappa 0.928 (original 0.739) and human-model 0.946 (original 0.839, Table 9); a separate style-boundary audit endorsed the structure/style split (annotator kappas 0.765/0.936).
Rewording attack verified, not asserted
Confound battery and negative controls
Artifact transparency, including auditing the original
Weaknesses
Inherent publication-year confound
Brief lossiness weakens the characterization claim
Conflict of interest with internally sourced validation
Attack coverage lags the motivating threat
Corpus scale and skew
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a single-author domain-transfer replication of StoryScope (Russell et al., 2026a), moving structural AI-text detection from fiction to commercial B2B blog posts. The claims, carefully scoped in Sections 1 and 9, are: (1) 187 structural features alone detect single-pass AI blog posts at 98.0 macro-F1 on held-out companies, unchanged under self-rewording (98.1, Section 5.4); (2) six-source attribution reaches 79.3% accuracy versus a 16.7% chance rate (Table 5); (3) human posts occupy rarer structural space (rarity percentile 0.838 vs 0.435, d=1.83, Table 7); (4) all effects replicate StoryScope in direction with larger magnitude. Scope is explicitly limited to single-pass generation from five named models and self-rewording attacks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can AI-generated blog posts still be detected after paraphrasing or rewording?
According to this study, yes. Structural features describing how information is organized, evidenced and voiced detected AI-written commercial blog posts at 98.1 macro-F1 even after each AI test post was rewritten by the model that generated it, a rewrite that replaced about 73% of 13-word sequences. Word-level detectors, by contrast, are documented in the literature to degrade sharply under paraphrasing.
How accurate is the SlopShape detector?
The paper reports 98.0 macro-F1 using 187 structural features alone on company domains never seen during training, and 98.1 using all 214 features. A six-way classifier assigning each post to the human source or one of five AI models is correct 79.3% of the time versus a 16.7% chance rate.
What are the main limitations of this study?
It covers only single-pass generation from five named AI models; human posts predate ChatGPT, so publication era is confounded with authorship; AI mirrors were written from reverse-engineered briefs that may lack business context; only self-rewording was tested, not commercial humanizer tools; and the sole author operates a commercial GEO product in this space.
Is the code or data available?
Partly. The sampling pipeline, the full 214-feature instrument, all prompts, analysis code and the aggregate artifacts behind every figure are public on GitHub. Per-post feature answers, briefs and AI mirrors are available to researchers on request under a non-commercial agreement; the copyrighted human posts themselves are not redistributed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (confusion matrix, SHAP ranking, LDA projection, length box plots) survive only as flattened text in the extraction; assessments rely on captions and in-text numbers.
- Several appendices (B, D, E, F, H) appear condensed in the extraction; per-item details such as full deviation defenses and rarity tail rows were only partially visible.
- Forward-dated references (StoryScope 2026, Anthropic watermark 2026) could not be verified within this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.