Beyond the Illusion of Power: Calibrating Quasi-Experiments in Observational IS
stat.ME · cs.AI · cs.LG
Quick verdict
A 9,837-cell Monte Carlo decomposition of DiD/IV power gaps in IS panels; rigorous and unusually self-critical, though simplified estimators temper claims.
Summary
This paper asks how much of the planned-versus-achieved power gap in observational IS research survives a competent, cluster-robust, AR(1)-aware power calculation. Through a Monte Carlo grid of 9,837 parameter conditions (~9.8 million datasets, 1,000 replications per cell), it decomposes the gap for TWFE DiD and 2SLS IV under an IS-calibrated parameter space. Central findings: serial-correlation losses are fully recoverable by an oracle AR(1)-aware calculator, but only partially when ρ is estimated from five pre-periods (naive plug-in yields ρ̂ ≈ −0.12 against a true 0.3; a Nickell-type correction recovers about half). Exogenous attrition costs 8–11 power points at N ∈ [200, 1000]; outcome-dependent attrition instead induces ≈ −0.05 SD bias. Conditioning on pretrend-test passage inflates conditional Type I error to 0.249 at 0.10 SD/period violations. For IV under fixed first-stage F, power never exceeds ≈0.19 at β = 0.1 and exclusion bias grows as √N, while weak-instrument bias attenuates. Empirical anchors (job-training DiD, Card IV) show 22–55 point nominal-achieved gaps. The paper is notably self-critical: it flags its simplified Sun–Abraham/Callaway–Sant'Anna implementations as imperfectly validated (correlation 0.47 at T = 24; T = 16 unvalidated) and bounds anchor coarseness only for the IV case. Main limitations: a simple AR(1) DGP, judgment-based effect-size calibration, and a DiD anchor whose T = 3 lies below the grid's minimum.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Serial-correlation power losses in DiD are fully recoverable by an AR(1)-aware calculator when ρ is known, but only partially when ρ is estimated from five pre-periods (naive plug-in yields ρ̂ ≈ −0.12 against a true 0.3).
- Exogenous panel attrition costs ≈8–11 power points at N ∈ [200, 1000] and is captured by no closed-form formula; outcome-dependent attrition instead induces ≈ −0.05 SD bias that no power calculation repairs.
- Conditioning DiD interpretation on passing the pretrend test inflates conditional Type I error to 0.249 (vs 0.05 nominal) at 0.10 SD/period pretrend violations.
- Under a fixed first-stage F, larger N neither raises IV power (≤≈0.19 at β = 0.1) nor cures exclusion bias (plim bias grows as √N); with a fixed instrument, F grows with N, so identification rests on instrument strength rather than sample size.
- Simplified Callaway–Sant'Anna analytic intervals undercover in staggered designs (mean coverage 0.906; a third of cells below 0.90), so the reference multiplier-bootstrap inference should be preferred.
Strengths
Unified joint benchmark across DiD and IV
One DGP covers TWFE DiD (6,912 cells) and 2SLS IV (2,880 cells) with consistent metrics — power, bias, RMSE, CI coverage — so interactions between serial correlation, attrition, staggered timing, and pretesting are quantified rather than studied in isolation (Sections 3–5).
Rare self-correction and validation transparency
Section 4.1 explicitly shows its Figure 1(b) estimator ordering is a benchmark-mismatch artifact, validates against fixest::sunab and reference Callaway–Sant'Anna on a ten-cell subset, and Section 4.4 discloses its own simplified C&S undercoverage (mean 0.906, a third of cells < 0.90) rather than burying it.
Honest oracle framing with a realistic estimated-ρ follow-up
Empirical anchors with explicit coarseness bounding
Practical calibration and deployable tooling
Weaknesses
Main grid runs on simplified estimator reimplementations with incomplete validation
DiD empirical anchor maps below the grid's support
The IS-realism claim is judgment-based, not meta-analytic
Deliberately simple DGP bounds external validity
The fixed-instrument IV case is analytic, not simulated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This methodological paper asks a well-posed question: once a competent analyst abandons i.i.d. power calculators for a cluster-robust, AR(1)-aware one, how much of the planned-versus-achieved power gap in observational IS panel designs survives, and which frictions remain invisible to any closed-form formula? The scope covers TWFE difference-in-differences and 2SLS instrumental variables under one data-generating process, evaluated over 9,837 parameter conditions (6,912 DiD + 45 pretrend + 2,880 IV cells; ~9.8 million datasets at 1,000 replications per cell). The headline claims are precise and falsifiable: serial-correlation losses are recoverable when ρ is known; exogenous attrition costs ≈8–11 pp at N ∈ [200, 1000]; pretrend-gated inference inflates conditional Type I error to 0.249 vs the 0.05 nominal level; and, under fixed first-stage F, neither IV power (≤≈0.19 at β = 0.1, even at F = 100 and N = 10,000) nor exclusion bias (which grows as √N) improves with sample size.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper on power analysis in DiD and IV designs?
It decomposes the gap between planned and achieved statistical power in observational panel studies using ~9.8 million simulated datasets. Serial correlation is the recoverable part: an AR(1)-aware calculator closes it when the autocorrelation ρ is known. What no closed-form formula captures: exogenous panel attrition (costing about 8–11 percentage points of power at sample sizes of a few hundred to a thousand), staggered treatment timing, and conditioning on passing a parallel-trends pretest, which inflates conditional Type I error to 0.249 versus the nominal 0.05 in their simulation.
Does increasing sample size fix weak instruments or exclusion violations in IV?
Not under the paper's fixed-first-stage-F normalization: with F held constant, power never exceeded about 0.19 even at N = 10,000, and bias from a direct instrument-to-outcome channel (an exclusion violation) grew roughly with the square root of N. However, with a fixed instrument coefficient, more data raises the first-stage F and does increase power — so the actionable lever is instrument strength, not sample size.
Is the paper's code and simulation grid publicly available?
Yes. The paper links a GitHub repository (spandan1305/is-power-lookup) containing a lookup tool over the full 9,837-cell simulation grid. The pipeline uses per-cell hash-derived seeds in Python; the authors report a single design cell reproduces in minutes on commodity hardware and the full grid took about four days on a 14-core laptop.
How reliable are the paper's estimator comparisons between TWFE, Sun–Abraham, and Callaway–Sant'Anna?
The authors themselves caution against ranking estimators from their headline figure: the apparent ordering was an artifact of measuring against the base effect rather than the cohort-share ATT. Against the correct estimand, reference implementations are approximately unbiased while TWFE attenuates by about 0.02 SD. But the paper's own simplified implementations track the references imperfectly (correlation 0.47 at T = 24; T = 16 unvalidated), so applied readers should rely on reference packages, as the paper recommends.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels are available only as textual descriptions and embedded axis labels; exact plotted values for some panels (e.g., Fig. 1a averaging over T) were read from captions and inline numbers rather than the images.
- Supplementary material (48-cell attrition grid details, worked coverage example of 0.876, lookup tool implementation) is referenced but not included in the extracted text.
- Text extraction is locally garbled: Table 1 interrupts the IV grid description in Section 3.1, and Figure 1/3 axis labels are interleaved with body text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.