Detecting Authorship in Political Texts with Inductive Stylometry
Gennadii Iakovlev, Levente Littvay
cs.CL
Quick verdict
Applies character-trigram UMAP and Burrows' Delta to six political corpora; validated results are strong, but validation lacks formal benchmarks or nulls.
Summary
This paper stress-tests inductive stylometry for uncovering hidden authorship in political communication, combining character 3-gram features, UMAP projection, and Burrows' Delta across six English and Hungarian corpora. In US Congressional Research Service reports, analyst fingerprints are near-disjoint under full format control (silhouette +0.47) and remain so when all formats are pooled (+0.41), while topic labels score −0.31 in the same coordinates; a portability analysis shows reports resemble the same author's work on other topics (0.83) more than other analysts' work on the same topic (0.72). Hungarian ombudsman reports reveal visible rapporteur islands. For tweets, device separation is mostly formatting (silhouette 0.21 with formatting intact, 0.08 after removing URLs and hashtags), and 50-tweet chunks separate almost perfectly, yet device is judged a noisy authorship proxy. The 23 campaign speeches separate teleprompter from improvised delivery without overlap, but neither speech corpus yields individual-speechwriter clusters, and Orbán's speeches show no stable clusters, which the authors read as the method's boundary. Limitations: clusters are assessed via UMAP silhouettes and t-tests rather than formal supervised attribution benchmarks or permutation nulls, and replication materials are only promised on acceptance.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Character 3-gram UMAP and Burrows' Delta recover near-disjoint analyst fingerprints in CRS American Law reports, and the signal survives pooling all document formats (silhouette +0.47 controlled, +0.41 pooled) while topic labels fail in the same coordinates (−0.31).
- Tweet device separation is mostly formatting (silhouette 0.21 intact → 0.08 without URLs and hashtags), making device a noisy proxy for authorship despite near-perfect separation of 50-tweet chunk profiles.
- Stylometry separates teleprompter from improvised delivery in 23 campaign speeches without overlap, but individual speechwriters cannot be resolved within scripted corpora.
- Frequency-based stylometry fails on Orbán's Hungarian speeches (no stable clusters), which the authors attribute to agglutinative morphology, oral delivery, and institutional editing acting together.
Strengths
Nested controlled design in CRS cases
Use Case 1 holds topic, audience, register, and format constant (167 RS short reports), then Use Case 2 pools all 1,109 formats; silhouette only drops +0.47 → +0.41, cleanly isolating format as a confound (§§3–4).
Systematic confound controls
Topic-label control scores −0.31 silhouette vs +0.41 for authors in identical coordinates (Fig. A5); Table A1 portability test (942 reports, 36 analysts) shows author effect +0.13 vs topic effect +0.02, with nearest-neighbour attribution at 59% author vs 16% topic.
Cross-linguistic stress test
Unusually candid negative results
Weaknesses
Silhouette computed on UMAP output with no null
UMAP-vs-t-SNE comparison is anecdotal
No supervised classification benchmark
Internal tension in the validation logic
LLM-based corpus construction with thin validation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper applies a classical stylometric toolbox — character 3-gram matrices projected with UMAP (cosine distance), plus Burrows' Delta on frequent words — to six political corpora spanning English and Hungarian, written and oral modes, and text lengths from tweets to long reports. Six labeled or hypothesized cases: CRS American Law reports in a format-controlled short-report slice (n=167) and pooled (n=1,109; 282 analysts); Hungarian ombudsman reports (n=886, 75 rapporteurs); 4,000 device-labeled Trump tweets; 23 teleprompter-coded Trump speeches; and 466 Orbán speeches with no labels. The central claim is conditional: frequency-based stylometry recovers authorship traces depending on authorial signal strength and institutional editing, with the Orbán corpus as the boundary case.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How does inductive stylometry detect hidden authors in political texts?
It combines character 3-gram frequencies (spelling, punctuation, affix habits), UMAP dimensionality reduction for 2-D visualization, and Burrows' Delta on frequent function words. Because it uses no labeled training data, clusters emerge inductively; where known labels exist (agency analysts, device, teleprompter), the clusters can be validated against them.
Did the method successfully identify who wrote Trump's tweets?
Partially. Android-vs-iPhone separation was largely driven by formatting — silhouette fell from 0.21 with formatting intact to 0.08 after removing URL and hashtag characters — and 50-tweet chunks separated almost perfectly. The paper concludes device is a noisy proxy capturing production register rather than a clean two-author split.
Can stylometry identify individual speechwriters?
No, not in this paper. Neither Trump's scripted campaign speeches nor Orbán's speeches yielded clusters attributable to individual writers; the method separated scripted from improvised delivery instead. The authors attribute this to collaborative editing, oral delivery overwriting drafters' habits, and limited text per writer.
Is the paper's code and data available?
Not yet. The data availability statement promises that all corpora and R analysis code will be deposited in the Political Analysis Dataverse upon acceptance. The source corpora (EveryCRSReport, Trump Twitter Archive, kormany.hu via Wayback Machine) are themselves publicly accessible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extraction is text-only: all assessments of figures (UMAP plots, silhouette panels) rest on the authors' textual descriptions and captions, not visual inspection of the plots themselves.
- Replication code and data are stated as pending Dataverse deposit upon acceptance, so reproducibility claims rely on the pipeline parameters described in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.