Reproducibility is not construct validity: LLM measurement of institutionally situated communication
Veronika Batzdorfer (KIT), Carlo Romano Marcello Alessandro Santagiustina (ALMAnaCH, m\'edialab, Sciences Po)
cs.AI · cs.CL · cs.CY · q-fin.RM
Quick verdict
Strong evidence that highly reproducible LLM annotation (ICC>.994) is not construct validity; core dissociation convincing, spatial/robustness claims fragile.
Summary
The paper argues that reproducibility of LLM annotation should not be conflated with construct validity. Using the EU Commission's AI Act consultation, the authors annotate 348 linked consultation submissions with Qwen3.5-397B and validate against survey responses from the same stakeholders. Annotation is nearly perfectly reproducible (ICCs > 0.994 across five runs), yet correlations with target survey items reach only 0.029–0.176 (Lin's CCC < 0.08). Decomposing text–survey divergence by stakeholder type, business associations show the largest positive divergence (mean +1.00, N=38) and public authorities a negative one (−0.70); country-level divergence exhibits spatial clustering (Moran's I = 0.347, p = 0.036), which, however, does not survive multiple-comparison correction (FDR p = 0.054). Despite divergence, the survey-reported safety-concern–explainability association remains robust (coefficients roughly 1.0–1.4 across DML specifications), though it is non-significant in the medium-divergence tercile and drops to 0.58 after common-support trimming. The paper's core contribution—demonstrating that reproducibility cannot substitute construct validation—is convincing, but scope is one policy domain and one model, code and data are available only upon request, and the abstract's spatial-clustering headline should be read cautiously.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across five independent runs, LLM annotations of consultation texts are almost perfectly reproducible (ICC(C,1) 0.994–0.996), yet converge only weakly with same-stakeholder survey measures (r = 0.029–0.176; Lin's CCC < 0.08).
- Text–survey divergence is systematically patterned by stakeholder type: business associations show the largest positive divergence (mean +1.00, N=38) while public authorities show a negative one (−0.70, N=15).
- Country-level divergence exhibits positive spatial autocorrelation (Moran's I = 0.347, p = 0.036), although this does not survive multiple-testing correction (FDR p = 0.054).
- Despite diverging from text measures, survey-reported safety concern remains strongly associated with explainability support (coefficients ≈ 0.8–1.4 across DML specifications, per-unit 1.477).
- Reproducibility of LLM annotation therefore cannot substitute for construct validation, and text measures should be read as institutionally situated expression rather than survey-equivalent proxies.
Strengths
Direct two-channel linkage design
Linking 348 consultation submissions to survey responses from the same stakeholders (§3.1) enables a genuine test of construct correspondence across communication settings — the rare design where reproducibility and validity can be formally dissociated.
Well-structured validation framework
§3.3's four diagnostics (construct correspondence, convergent/discriminant validity, systematic divergence, downstream robustness) go beyond the usual inter-rater-agreement check criticized in §2.1.
Comprehensive agreement statistics
Transparent self-skeptical prompt analysis
Reproducibility side is solidly established
Weaknesses
Unresolved consultation identity mismatch
High-dimensional DML on a small sample
Abstract headline does not survive multiplicity
Data availability undermines replication
Agreement results plausibly reflect scale artifacts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a clean, consequential question: does high inter-run reproducibility of LLM annotation imply construct validity? Using 857 submissions to the European Commission's AI consultation (20-02-2020–27-04-2021), of which 348 link free text to structured survey items from the same stakeholder (§3.1), the authors annotate safety_concern, rights_concern and explainability_trust (0–5) with Qwen3.5-397B-A17B. The central dissociation is convincingly demonstrated: ICC(C,1) = 0.994–0.996 across five runs and leave-one-run-out r > 0.999 (§A.7), yet convergent correlations with survey items are 0.029–0.176 and Lin's CCC < 0.08 (Table 1). A standardized divergence index G = Text − Survey (Eq. 1) is then patterned across stakeholder types and countries, and survey-only DML analyses test downstream robustness.
One identity problem must be resolved: the title, abstract and Implications say "AI Act consultation", while §3.1 and footnote 4 describe the 2020 White Paper consultation — a different Commission exercise that predates the AI Act proposal. For a paper about mislabeled constructs, this mislabel is embarrassing and must be fixed.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper 'Reproducibility is not construct validity' actually show?
It shows that a large language model (Qwen3.5-397B) can annotate EU AI consultation texts almost identically across repeated runs (intraclass correlations above 0.994) while the resulting scores correlate only weakly (r = 0.029–0.176) with what the same organizations reported in a structured survey. In other words, a reproducible LLM measure is not automatically a valid proxy for the intended construct.
Why do LLM text measures and survey responses diverge for business associations?
The authors find business associations show the largest positive text–survey divergence (mean +1.00, N=38): they emphasize AI risks more strongly in public consultation texts than in their survey answers. They interpret this as institutionally situated public communication — multiple audiences, reputational incentives, and advocacy strategies — while explicitly cautioning that the design cannot prove strategic intent.
Is the country-level clustering result statistically reliable?
Only partially. Divergence scores across 23 European countries show positive spatial autocorrelation (Moran's I = 0.347, p = 0.036, 9,999 permutations), but as the paper's own Figure 8 reports, this does not survive correction for multiple testing (FDR-adjusted p = 0.054; Bonferroni p = 0.108). The companion result for public rhetoric (I = 0.446, p = 0.009) is the more robust pattern.
Can I reproduce the paper's analysis with the released data and code?
Not directly. The paper states that data and code are available 'upon request to the authors', not in a public repository. The methods (DML, Moran's I, agreement statistics) are standard and re-implementable, but the linked text–survey corpus is the authors' unique asset, so independent replication currently requires their cooperation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–4, 6–8 and their exact point estimates cannot be fully verified from extracted text; numbers are taken from captions, tables, and in-text reporting.
- Sections A.4 and A.10–A.11 appear as empty headers in the extraction, so any supplementary tables/figures there were not reviewable.
- The paper reports ICC(C,1) ranges (§3.2) and per-construct values (§A.7) inconsistently; the review cites each figure to its specific location.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.