Quantifying Organizational Environmental Action from Web Data and Large Language Models
Quinn Reynolds, Daniel Shore, Vianey Leos Barajas, Tanhum Yoreh, Meredith Franklin
cs.CL · cs.CY · cs.IR
Quick verdict
LLM pipeline measures environmental action across 2,657 synagogue websites; solid framework, but single-annotator validation and inconsistent counts weaken it.
Summary
The paper presents a scalable framework for measuring organizational environmental action from websites, applied to 4,964 U.S. Jewish congregations, of which 2,657 had active websites yielding a crawled corpus of roughly 154,000 pages (the abstract and Results disagree: 154,454 vs 158,454). Three detection pipelines were compared against 1,200 text chunks annotated by a single expert: keyword retrieval plus LLM classification (Gemini 3.1 Flash-Lite) reached κ=0.26, semantic vector retrieval (MiniLM-L6-v2, threshold 0.25) κ=0.42, and direct LLM classification κ=0.40, with retrieval recalls of 0.94 and 0.87 for the filtered approaches. Direct classification covered the most congregations (1,398, or 52.6%, with at least one action) at $153.57 versus $28.77–$85.80 for retrieval pipelines. Downstream analyses found action concentrated in Spirituality & Worship and Community categories (together 80% of actions), significantly lower counts for Orthodox and Chabad congregations (Kruskal–Wallis H(8)=413.8, p<0.001), and fewer explicitly framed actions in Republican-won congressional districts (p=0.01). Main limitations: validation relies on one annotator with no confidence intervals, confusion matrices, or human–human reliability; the denominational check used only 20 items (κ=1.00); several counts are internally inconsistent (1,398 vs 1,353 active congregations); and chunk-level validation does not quantify congregation-level aggregation error.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Direct LLM classification (Gemini 3.1 Flash-Lite) maximizes congregation-level coverage (1,398/2,657 = 52.6% with at least one action) because keyword and vector retrieval lose 6–13% of relevant content upstream (recalls 0.94 and 0.87).
- Against 1,200 expert-annotated chunks, agreement was κ=0.26 for keyword+LLM, κ=0.42 for vector+LLM (raw 97.6%), and κ=0.40 for direct LLM (raw 96.7%), at costs of $28.77, $85.80, and $153.57 respectively.
- Documented environmental action is concentrated in Spirituality & Worship (77% of active congregations) and Community (62%), which together account for 80% of all identified actions.
- Documented action counts differ by denomination (Kruskal–Wallis H(8)=413.8, p<0.001, n=2,584), with Orthodox synagogues significantly lower than all others and Chabad lower than most.
- Explicitly framed environmental actions are less frequent in Republican-won congressional districts (p=0.01) while implicit action counts are nearly identical (1.72 vs 1.71).
Strengths
Retrieval-vs-classifier error decomposition
Table 2 separates retrieval recall (0.94 keyword, 0.87 vector) from classification agreement (κ), and the EV-charger anecdote in §3.2 ('Charge Your Car While You Charge Your Soul') concretely shows content that no threshold-based filter could recover — an analysis most LLM-extraction papers omit.
Prespecified, power-based validation sizing
§2.3.5 sizes the 1,200-chunk validation set via 200,000 Monte Carlo simulations (H0 κ=0.20 vs Ha κ=0.40, unequal positive rates 1%/4%, 80% power) — unusually rigorous for NLP pipeline validation.
Multi-source national database with spatial sampling
Fixed reanalyzable corpus and full cost transparency
Cautious downstream interpretation
Weaknesses
Single-annotator gold standard without uncertainty
Internal numeric inconsistencies
Appendix prompt fidelity
Congregation-level construct never validated
Debatable analytic choices without sensitivity analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This methods-plus-empirical paper builds a pipeline that turns organizational website content into structured measures of environmental action, demonstrated on 4,964 U.S. Jewish congregations (§2.1). Of 3,917 congregations with identified websites, 2,657 had crawlable active sites, yielding a fixed archived corpus (abstract: 154,454 pages; §3.1: 158,454 — the discrepancy is unresolved). The central methodological claim is that pre-retrieval filtering trades compute for irrecoverable information loss: against 1,200 expert-annotated chunks (§2.3.5), keyword+LLM scored κ=0.26, vector+LLM κ=0.42 with retrieval recall 0.87, and direct LLM κ=0.40 (Table 2). The empirical claims are that documented environmental action is widespread (1,398 congregations, 52.6%), concentrated in Spirituality & Worship and Community (80% of actions), denominationally patterned (Orthodox/Chabad lower; H(8)=413.8), and politically patterned for explicitly framed actions (§3.4–3.6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How many U.S. Jewish congregations show environmental action on their websites?
Among 2,657 congregations with crawlable active websites, direct LLM classification found at least one environmental action at 1,398 of them (52.6%). Counts were strongly right-skewed: median 1 action per congregation, mean 5.1, with action concentrated in Spirituality & Worship and Community categories.
Which detection approach worked best — keyword search, semantic retrieval, or direct LLM classification?
On a 1,200-chunk expert-validated sample, semantic vector retrieval agreed most with the human reviewer (κ=0.42) but missed about 13% of relevant content (recall 0.87). Direct LLM classification scored κ=0.40, detected actions at 261 more congregations, and cost $153.57 versus $85.80; the authors chose it for maximum coverage.
Are Orthodox synagogues less environmentally active than other Jewish denominations?
In this dataset, documented (website-reported) environmental action counts were significantly lower for Orthodox synagogues than all other denominations, and Chabad lower than most (Kruskal–Wallis H(8)=413.8, p<0.001). This measures website documentation, not actual environmental impact, and the association is not causal.
Is the code or data for this study available?
The paper links a GitHub repository (Reyno256/Quantifying-Environmental-Action), publishes all classification prompts in its appendices, names the model (Gemini 3.1 Flash-Lite), and states that the full webpage corpus was archived (retrieved through June 1, 2026) to allow reanalysis with alternative methods.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–6 cannot be analyzed from extracted text; numeric claims are drawn from body text and Tables 1–4 only.
- Abstract and §3.1 disagree on corpus size (154,454 vs 158,454 webpages); both are cited as reported.
- Validation sampling frame and confusion matrices are not included in the extracted text, limiting assessment of per-class error.
- Taxonomy Table 4 extraction is truncated near the 'Other' category; the full 76-type list was only partially verifiable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.