GPS-Bench: A Governance Policy Benchmark for Automating Policy Analysis
Linh Le, Melanie Bui, My Chiffon Nguyen, Zachary Schlosser, David Williams-King
cs.AI · cs.CY
Quick verdict
Claims first evidence-grounded policy simulation benchmark; leakage discipline exemplary, but headline tables not in excerpt and passage largely prospective.
Summary
GPS-Bench is a benchmark that evaluates LLM-based governance policy simulation against the historical record rather than plausibility. The authors assemble a corpus of 1,233 governance instruments from 350 publisher domains with field-level provenance, reconstruct policy-specific actors from dated public records (34 roster roles, a 15-action vocabulary), and score three linked targets: legislative passage, affected-actor identification, and actor-level impact direction. A human-annotated Gold pool of 679 cases serves as evaluation, while 4,235 LLM-labelled Silver records are restricted to training; a strict temporal split (train before 2024, test from 2024) and outcome-scrubbed contexts guard against contamination. Joint reasoning, independent actor agents, collaborative revision, and multi-round communication all read the same grounded state and emit the same schema, alongside a LoRA fine-tuned open 7B model. The headline findings: fine-tuning on the grounded record gives the strongest actor-level impact prediction; decomposition does not beat it, and communication contributes inspectable mechanism rather than accuracy — in the export-control case study, none of 270 agent pitches moved a stance. Coalitions are checked against dated commitments. Main limitations: passage prediction is largely deferred to prospective evaluation (only three contamination-controlled negatives), the corpus is English- and US-heavy, and per-role sample support remains thin.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Actors reconstructed from dated public records as evidence objects with provenance allow governance simulation to be scored against documented behaviour rather than plausibility.
- Weight-level fine-tuning (LoRA on an open 7B model) on the grounded record achieves the strongest actor-level impact prediction, surpassing all in-context inference modes including multi-agent decomposition.
- Multi-agent decomposition contributes inspectable mechanism — pitches, coalitions, stance revisions — rather than predictive accuracy; communication does not improve forecasts (0 of 270 pitches moved a stance in the export-control case).
- Coalition formation respects the US–China bloc boundary as an instruction-level prior (none of 77 committed coalitions crosses it; 37 of 72 when instructed), not as an emergent finding.
- Leakage is quantified rather than hidden: strict temporal splits, outcome scrubbing, Silver-never-as-test-labels, and prospective deferral of passage scoring.
Strengths
Controlled same-state comparison design
Every inference mode (joint, independent, collaborative single-revision, multi-round communication) reads the identical observed graph G≤t and emits the same output schema, so performance differences are attributable to inference architecture alone rather than to differing evidence — a cleaner design than most agent-simulation papers.
Unusually candid leakage discipline
Outcome tells are scrubbed from context; the paper admits its own constructed bill summary 'leaks the outcome for most US bills' and therefore scores passage with a title-only prompt; the pilot multi-agent passage score of 0.959 is explicitly flagged 'contaminated'; the fully contamination-controlled passage arm is excluded from primary results because only 3 resolved post-cutoff negatives exist, with 68 pending measures registered for prospective scoring after the 119th Congress adjourns (3 January 2027).
Strict Gold/Silver label taxonomy
Negative results reported, not buried
Provenance engineering throughout
Weaknesses
Passage task is largely unevaluated today
No uncertainty quantification on headline comparisons
Corpus skew and jurisdictional confound
Model-prior leakage into a provenance-centric artifact
Presentation errors undercut the provenance thesis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
GPS-Bench (arXiv 2609.03553) proposes the first evidence-grounded benchmark for governance policy simulation: instead of judging LLM agent behaviour by plausibility, it scores predictions along the policy → actor → impact chain against the historical record. The corpus holds 1,233 governance instruments from 350 publisher domains, curated URL-first with SHA-256 provenance; 431 instruments carry 641 independently published expert analyses from 441 publishers. Actors come from a 34-role roster reconstructed from dated records (roll-call history, SEC EDGAR, lobbying disclosures); actions use a 15-way vocabulary; impacts are typed through 12 mechanisms and 10 world dimensions (402 typed impact nodes, 214 graded causal maps). Three targets are scored: legislative passage (balanced accuracy), affected-actor relevance (macro-F1), and actor-level impact direction (macro-F1). Central claims: (1) weight-level fine-tuning on the grounded record beats all in-context inference modes on actor-level targets; (2) multi-agent decomposition adds inspectable mechanism, not accuracy; (3) leakage is quantified rather than hidden.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is GPS-Bench?
GPS-Bench is a benchmark for testing whether LLM-based simulations of governance and policy processes match reality. It reconstructs policy stakeholders from dated public records (bills, lobbying disclosures, corporate filings) and scores predictions of which actors a policy affects, what they did, and whether they were helped or harmed, using 679 human-verified Gold cases from instruments dated 2024 or later.
What did the GPS-Bench authors find?
Three main findings: fine-tuning a model on the evidence-grounded record gave the strongest actor-level impact predictions; multi-agent decomposition (separate actor agents) did not beat it; and communication between agents added inspectable mechanism rather than accuracy — in the chip export-control case study, none of 270 agent-to-agent lobbying pitches changed any actor's position.
Is the benchmark free of data contamination from LLM training data?
The authors take extensive measures — a strict temporal split (train before 2024, test from 2024), scrubbing outcome hints from the context, and title-only prompts for passage prediction — but they acknowledge that pretraining leakage cannot be fully ruled out, and the passage task is mostly deferred to prospective evaluation after the 119th Congress adjourns on 3 January 2027.
Who built GPS-Bench and where does its data come from?
The benchmark comes from Lida Safety and ERA. Its corpus contains 1,233 governance instruments (laws, regulations, bills) from 350 publisher domains, each with full text and provenance, plus 641 independently published expert analyses linked to 431 instruments; 71 instruments are non-English, of which 57 were machine-translated.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The primary aggregate results tables (the consolidated inference-mode comparison referenced as Table A20 and Table 4 for LoRA vs few-shot) fall inside sampling gaps of the provided excerpt, so headline macro-F1 numbers could not be verified.
- The Results narrative and Figure 2's contents are only partially present; coalition and case-study numbers cited in this review come from the case-study and appendix portions included in the excerpt.
- No code or data availability statement appears in the provided excerpt; a release statement may exist in skipped sections.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.