Toward Collective-Centric Evaluation of Preference Inference for Participatory Democracy
Pierre-Antoine Lequeu, Salim Hafid, Paul Lerner, Nazanin Shafiabadi, Laurène Cave, David Mas, Jean-Philippe Cointet, Benjamin Piwowarski, et al.
cs.SI · cs.AI
Quick verdict
基准测试表明，准确率无法捕捉投票推断中集体层面的扭曲；新指标和包含 100 万票的多语言数据集有力地支持了这一观点。
Summary
本文认为，对于参与式协商平台中的偏好推断（PI）——即预测未观测投票的任务——仅凭个体预测准确率进行评估是不够的，因为模型在得分良好的同时，仍可能扭曲集体偏好格局。作者沿着两个维度（提案层面的认可模式；参与者层面的社区和社会人口群体）将“偏好格局”形式化，并引入了多项指标：认可率命中率（Approval Rate Hit Ratio，ARHR，Wilson-CI 命中测试）、共识扭曲度（Consensus Distortion，CD，最大群体级认可率偏移），以及基于签名二部图的社区稳定性（Community Stability，CS，即观测图与推断图聚类间的 ARI），外加 Min B-Acc 和 Equalized Odds Ratio。他们在 14 次协商中基准测试了五个系列（CF、FM/STUMP、NN、SBGCL、LLM）的九种 PI 模型变体，并贡献了四个新的 Make.org 数据集（约 9 万参与者、100 万票、22 种语言；仅 eurhope 就有 5.4 万参与者、89.2 万票）。关键结果：平衡准确率与集体指标之间的相关性接近于零（R²=0.05–0.14）；准确率最高的 SBGCL 在格局保留方面表现滞后；STUMP 变体提供了最佳权衡；且在任何数据集上，都没有模型的 ARHR 超过 0.95。局限性包括依赖随机留出投票作为基准真值（忽略了现实平台中的非随机缺失）、设定的规范阈值缺乏依据、单次划分评估且缺乏方差估计，以及关键细节推迟至补充材料中说明。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Balanced accuracy is essentially uncorrelated with collective-centric preservation metrics (R² = 0.05 for ARHR, 0.14 for CS, 0.13 for CD), so per-vote accuracy alone is insufficient for evaluating PI.
- No evaluated PI model produces statistically plausible approval rates (ARHR > 0.95) on any dataset, and the best average Consensus Distortion (21.7 ×10⁻², FM) remains too high for democratic processes.
- Models with the highest predictive accuracy (SBGCL) can distort the preference landscape more than lower-accuracy alternatives; STUMP variants offer the best accuracy–preservation trade-off.
- Limitations of current PI are structural rather than sparsity-driven: ARHR, EOR, and CS stay flat as observed votes per participant grow from 1 to the maximum.
- The released Make.org corpus (four consultations, ~90k participants, 1M votes, 22 languages) is the largest multilingual PI resource of its kind, with eurhope the largest public consultation dataset to date.
Strengths
Genuinely novel evaluation target
Reframes PI evaluation from per-pair accuracy to preservation of collective structure (§3: ARHR, Consensus Distortion, Community Stability); the Fig. 1 example (75% accuracy yet approval rate dropping 80%→40%) and the group-swap counterexample in Related Work (equal group error ≠ faithful representation) make a compelling case that conventional accuracy and even group-fairness parity are insufficient.
Largest multilingual corpus for the task
Four new Make.org consultations bring the benchmark to ~90k participants, 1M votes, 22 languages; eurhope alone has 892,273 votes, 54,421 participants, 4,366 propositions (Table 1) — claimed 8x participants and 2x propositions of the largest prior benchmark. Code and data are released (GitHub + HuggingFace Democratic Commons).
Broad, honest model coverage
Quantified decoupling of accuracy from preservation
Weaknesses
Ground truth assumes random missingness
Normative thresholds asserted, not derived
Community Stability interpretability unclear
Thin statistical reporting
Limited LLM evidence and an Eq. 4 ambiguity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses Preference Inference (PI): predicting missing entries of the sparse participant–proposition voting matrices produced by deliberation platforms (Polis, Remesh, Make.org). Its central claim is that user-centric accuracy is insufficient because PI reshapes the collective 'landscape of preferences' — proposition-level approval patterns and participant-level group structure — and equally accurate models can distort it differently. Three contributions: (1) a metric suite (ARHR, Consensus Distortion, Community Stability, plus Min B-Acc and Equalized Odds Ratio); (2) the largest multilingual PI corpus — four new Make.org consultations, with eurhope alone at 892,273 votes, 54,421 participants, 4,366 propositions, 22 languages (Table 1); (3) an evaluation of nine model variants across five families on 14 consultations showing accuracy–preservation decoupling (R² = 0.05–0.14). Scoping is honest: the authors propose an evaluation framework, not a new PI model, and the Limitations section explicitly disclaims that the framework legitimizes PI as a substitute for participation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Preference Inference in participatory democracy platforms?
It is the task of predicting how a participant would vote on propositions they never saw, filling in the sparse voting matrix that platforms like Polis and Remesh use to identify opinion groups and summarize deliberations. This paper shows that these inferred votes can reshape apparent consensus and group representation even when the predictions are individually accurate.
Why is accuracy not enough to evaluate vote-prediction models?
The paper shows balanced accuracy is nearly uncorrelated (R² = 0.05–0.14) with whether the completed voting matrix preserves proposition-level approval rates, group-level consensus patterns, and participant communities. A model can score well per-vote while shifting a proposition's approval rate from 80% to 40% (the paper's Figure 1 example) or erasing differences between socio-demographic groups.
What data did the authors release?
Four new consultations from the Make.org platform: a German taxation debate (steuer-debate), French healthcare (gc-sante), foreign digital interference (ingerence), and Future of Europe (eurhope, 892k votes, 54k participants, 22 languages), available via the Democratic Commons collection on HuggingFace, with code at github.com/LequeuISIR/PrefInferenceEval. Combined with public Polis and Remesh data (14 consultations total), the benchmark spans roughly 90k new-participant votes and 1M votes.
Which preference-inference model performed best?
No model dominated. SBGCL, a graph neural network, had the highest balanced accuracy but lagged on preserving collective structure; STUMP variants (factorization models initialized with pretrained proposition embeddings) offered the best trade-off; collaborative filtering showed high accuracy but the worst distortion metrics. No model reached statistically plausible approval rates (ARHR > 0.95) on any dataset.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Supplementary material (per-dataset breakdowns, hyperparameter grids, LLM prompt, per-language Eurhope analysis, Spearman correlations) is referenced throughout but is not part of the provided text, so those details could not be assessed.
- Axis-tick values in Figures 2–4 are garbled by PDF extraction; all figure-derived numbers cited in this review come from the legible surrounding prose (R² values, CD 21.7, CS 22.9, the ARHR 0.95 bound).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.