Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment
Deepak Pandita, Christopher M. Homan
cs.CL
Quick verdict
Proposal argues disaggregated human labels improve reproducible ML evaluation and pluralistic alignment; evidence is simulation-based, validation pending.
Summary
This thesis proposal argues that human label disagreement, conventionally averaged away via plurality voting, is itself a threat to reproducible ML evaluation and to pluralistic AI alignment. It organizes work around six research questions in two pillars. For evaluation, the authors extend the VET p-value simulator: a human-in-the-loop procedure estimates the items (N) and responses-per-item (K) needed for reliable two-model comparison (RQ1), and a Bayesian Dirichlet-categorical extension optimizes fixed annotation budgets (RQ2) over five public multi-annotation datasets. Simulations indicate that, under a fixed budget, increasing K often beats increasing N, that the trade-off is metric-dependent (total variation needs the smallest budget, roughly N×K≤1000 with K>10), and that the common 3–5-labels-per-item practice is statistically underpowered. Preliminary alignment results include a rater-cohesion study on vicarious offense annotations (Independents most cohesive; Republicans least internally cohesive) and ProRefine, an inference-time prompt-refinement loop driven by LLM textual feedback, which outperforms TextGrad in 11 of 15 model-dataset cases, with the largest gains for small models. Future work targets rater-aware hierarchical simulators (RQ3) and perspective-specific feedback models with bandit prompt selection (RQ6). Main limitations: conclusions rest on parametric simulations whose distributions were fitted partly by visual inspection, item independence assumptions, no external validation yet, and ProRefine's added latency and lack of convergence guarantees.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Aggregating human labels via plurality voting obscures minority perspectives and contributes to the reproducibility crisis in ML evaluation.
- Under a fixed annotation budget, increasing responses per item (K) is often more effective than increasing items (N), and the optimal trade-off is metric-dependent (Table 1: TV achieves p<0.05 at N×K≤1000 with K>10).
- Current evaluation practice of 3–5 labels per item is statistically insufficient: even with 1,000 items, at least 25 raters are needed for MAE significance (§3.2).
- Rater cohesion differs by political group — Independents are most cohesive and Republicans least internally cohesive — with implications for rater recruitment (§4.2, Tables 2–3).
- ProRefine's inference-time textual-feedback loop improves multi-step reasoning, outperforming TextGrad in 11 of 15 model-dataset cases with the largest gains on small models (Table 4).
Strengths
Actionable, quantified budget guidance
Table 1 (RQ2) shows the TV metric reaches p<0.05 at N×K≤1000 with K>10 across datasets, and that the N-vs-K trade-off depends more on the metric than the data source — directly usable advice for evaluation design.
Honest, quantified critique of current practice
RQ1 results show 25,000–50,000 annotations per test set is rarely seen and that even 1,000 items need ≥25 raters for MAE significance, substantiating the claim that 3–5-label practice (Snow et al. 2008) is underpowered.
Statistical care in the simulator studies
ProRefine is simple, label-free, and helps small models most
Unusually candid Limitations section
Weaknesses
Simulator-conditional conclusions
Judge circularity in the ProRefine evaluation
Unexplained AQUARAT regressions
No multiple-comparison control
No compute parity reporting vs TextGrad
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a PhD thesis proposal (Rochester Institute of Technology; cs.CL) consolidating a research program around one thesis: human label disagreement, currently collapsed by plurality voting, is an underappreciated cause of irreproducible ML evaluation and an obstacle to pluralistic alignment. Six research questions split into two pillars — perspective-aware reproducible evaluation (RQ1–RQ3) and human-centered alignment (RQ4–RQ6). Roughly half the document reports completed, mostly published work (Homan et al. 2026 EACL Findings on N/K sizing; Pandita et al. 2026a AAAI on categorical budget optimization; Pandita et al. 2024 EMNLP Findings on rater cohesion; Pandita et al. 2026b AAAI student abstract on ProRefine); the rest sketches future work. Central claims: (1) the 3–5-labels-per-item convention is statistically underpowered — even 1,000 items need ≥25 raters for MAE significance (§3.2 RQ1); (2) under a fixed N×K budget, raising K often beats raising N, metric-dependently — TV achieves p<0.05 at N×K≤1000 with K>10 (Table 1); (3) rater cohesion varies by political group (Tables 2–3); (4) ProRefine beats TextGrad in 11/15 cases (Table 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the core argument of this thesis proposal?
That human disagreement is signal, not noise: plurality-vote aggregation hides minority perspectives, leaves evaluations statistically underpowered and hard to reproduce, and flattens AI alignment to one value system. The proposal builds statistical tools to size evaluations around the full response distribution and methods to align LLMs to multiple perspectives.
What did the preliminary experiments find?
Simulations on public multi-annotation datasets suggest that under a fixed annotation budget, collecting more responses per item (K>10) is often more effective than collecting more items; the optimum depends on the metric (total variation needed the smallest budget, about N×K≤1000). The common practice of 3–5 labels per item is underpowered — even 1,000 items need roughly 25+ raters for significance with MAE. Separately, ProRefine, a prompt-refinement loop using LLM textual feedback, beat TextGrad in 11 of 15 model–dataset cases, with the largest gains on small models.
What is ProRefine and how does it work?
ProRefine is an inference-time prompt optimization method: a task LLM generates partial output, a feedback LLM critiques it, and an optimizer LLM rewrites the prompt; the loop repeats (10 tokens generated per step, up to 25 steps, stopping at end-of-sequence). It requires no fine-tuning and no ground-truth labels, and improved accuracy on five multi-step reasoning benchmarks, especially for 1B–3B models.
Is code or data available for this work?
All datasets used are public (e.g., Measuring Hate Speech, DICES 350, VOICED, GSM8K, SVAMP), and the underlying VET toolkit is open source on GitHub (google-research/vet). The proposal itself does not state a code release for its own simulator extensions or for ProRefine.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Tables 2–3 use bold text and orange/cyan/green color coding to mark significance (per their captions); this formatting is lost in plain-text extraction, so per-cell significance claims could not be independently verified.
- Appendix headings A.2 (Rater Cohesion) and A.3 (ProRefine) appear in the extraction with no dedicated body text, though all Algorithms 1–5 and Tables 2–4 are present — likely a layout artifact rather than missing content.
- Figures 1, 5, and 6 are images unavailable in the text; their trends are taken from the prose descriptions and captions only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.