The Illinois Social Attitudes Aggregate Corpus (ISAAC): An Open Tool and Reproducible Pipeline for Analyzing Social Group Discourse at Scale
Babak Hemmatian, Sarah Hadjarab, Jessica Chen, Benedek Kurdi
cs.CL · cs.SI
Quick verdict
527M-post Reddit corpus (2007–23), six social-group distinctions, <10% audited noise; validation solid, but filtering recall and label ground truth unreported.
Summary
The authors introduce ISAAC, a public corpus of 527,060,919 Reddit submissions and comments (2007–2023) filtered for relevance to six social group distinctions: race, sexuality, age, ability, body weight, and skin tone. A multi-stage, human-audited filtering pipeline kept irrelevant content below a 10% threshold in stratified samples, per distinction and post type. Each post carries Reddit metadata, a hierarchical user-location estimate (US state or world region), and algorithmic labels: binary moralization (MFRC-trained classifier), sentiment from three models (VADER, TextBlob, Stanza), six basic emotions from three models, and clause-level linguistic generalization features. Access is via a no-code website, SQL playground, Python package, HuggingFace, and direct download under a Data Use Agreement. Convergent validity evidence includes alignment with Google Trends, event-locked volume spikes (e.g., +219% for race in June 2020; Obergefell +46% in 2015), nine state marriage-equality votes where voting states exceeded non-voting medians 9/9 (sign test p=.004; geometric mean elevation 1.78, 95% CI 1.39–2.34), and correlation of sexuality sentiment with Gallup support (r=.71), though year-over-year increments did not co-move (r=-.08). Limitations: filtering recall (false negatives) is not reported, state-level location confidence is miscalibrated (ECE≈.52) and state shares deviate from Census (e.g., NJ 0.48×), semantic labels lack human validation on ISAAC text itself, and the generalization training corpus remains unpublished.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ISAAC provides 527,060,919 Reddit posts (2007–2023) relevant to six social group distinctions with audited irrelevance below 10% for every distinction and post type.
- A single unified pipeline supplies metadata, hierarchical user-location estimates, and moralization, sentiment, emotion, and generalization labels, enabling cross-category, longitudinal, and spatial analyses at scale.
- Corpus volume and sentiment indicators demonstrate convergent validity against Google Trends, event timing, state ballot outcomes, and Gallup national survey data.
- The fully public, modular pipeline eliminates research fragmentation and can be extended to new platforms, languages, and social categories.
Strengths
Unified infrastructure at unmatched scale
527,060,919 posts across 2007–2023 for six social group distinctions under one pipeline; Table 1 shows no comparator (Pushshift/Arctic Shift, GoEmotions, MFRC/MFTC, RedditBias, COHA, TweetsKB) jointly offers naturalistic text, longitudinal span, multi-category coverage, and rich labels.
End-to-end audited precision benchmark
Stratified human audits (up to 500 comments + 100 submissions per distinction-year-month) keep residual irrelevance below the 10% target for every distinction and post type — directly addressing the unreported-false-positive problem the introduction documents.
Multi-model label design treating disagreement as signal
Four-pronged convergent validation
Candid disclosure of weak spots
Weaknesses
Filtering recall unreported
Semantic labels lack human validation on ISAAC text
State-tier location labels are weak
Convergent evidence is aggregate-level and post hoc
Key assets gated or unpublished
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ISAAC is a resource paper: it introduces a 527,060,919-post corpus of Reddit submissions and comments (2007–2023) pre-filtered for relevance to six social group distinctions — race (White/Black), sexuality (straight/gay), age (young/old), ability (abled/disabled), weight (thin/fat), and skin tone (light/dark) — and ships it with a unified labeling stack: Reddit metadata, a hierarchical user-location estimate, binary moralization, sentiment from three tools (VADER, TextBlob, Stanza), six basic emotions from three models, and clause-level linguistic generalization. The framing claims are (i) that no existing resource jointly satisfies naturalistic text, longitudinal span, multiple curated group distinctions, a multi-label framework, convergent validation, and multi-format access (the Table 1 comparison), and (ii) that the audited sub-10% irrelevance benchmark is unmatched among comparable corpora. Both are plausible against the Table 1 landscape (Pushshift/Arctic Shift, GoEmotions, MFRC/MFTC, RedditBias, COHA, TweetsKB), though the abstract's 'eliminates research fragmentation' is aspirational: a resource reduces fragmentation only to the extent it is adopted.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ISAAC corpus?
ISAAC (Illinois Social Attitudes Aggregate Corpus) is a free, public dataset of over 527 million English Reddit posts and comments from 2007–2023, filtered for relevance to six social group topics (race, sexuality, age, ability/disability, body weight, skin tone). Each post is annotated with moralization, sentiment from three tools, six basic emotions from three models, clause-level linguistic generalization features, and an estimated user location (US state or world region).
How was ISAAC validated?
The authors had human coders audit stratified samples so irrelevant content stayed under 10%, and showed that corpus indicators track Google Trends, spike around major events (e.g., race discourse +219% in June 2020, Obergefell ruling +46% in 2015), rise in voting states around nine same-sex-marriage ballots (9/9 cases, sign test p=.004, mean elevation 1.78×), and correlate with Gallup marriage-equality support (r=.71), though year-to-year changes did not align (r=-.08).
How can researchers access ISAAC?
Through the project website (point-and-click downloads plus an SQL playground), the isaac-data Python package on PyPI, HuggingFace datasets, or scripted direct download. Non-commercial academic use requires accepting a Data Use Agreement that bans user re-identification, generative-AI training, and surveillance uses.
How accurate are ISAAC's location labels?
A manual audit put US-vs-non-US classification at 95.7% and state-level at 90.2%, but the shipped confidence scores are only well calibrated at coarse levels; at the US-state tier they are miscalibrated (expected calibration error about .52), state shares deviate from Census figures, and users needing state precision should treat those labels cautiously.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text extraction truncated several sections (filtering pipeline details, Table 2 variable list, Tables 4–5, Appendix C validation details, ethics Appendix A); assessment of those components relies on in-text summaries.
- Figures (2, 6–9, D1–D3) are available only via captions and surrounding text; plotted series and effect magnitudes beyond reported numbers could not be inspected.
- Held-out accuracy/F1 values for the in-house classifiers are referenced to tables and appendices not present in the extract, so 'at or near state-of-the-art' claims could not be independently checked.
- The 49%–74% moralization range (volume-weighted text) versus Table 6's 60.2%–72.2% (year-equal weighting) could not be fully reconciled from the extract.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.