Aggregate Disambiguation Systems
José María Lago, Albert Castellana, Edgars Nemše
stat.ME · cs.AI
Quick verdict
Careful exact finite-sample theory for LLM panel reproducibility; rigorous and honestly scoped, but the 50-case LLM pilot is thin and underpowered.
Summary
The paper formalizes aggregate disambiguation systems (ADSs): mechanisms in which a finite panel of evaluator episodes casts binary votes on a frozen candidate resolution, aggregated to reproduce the threshold decision of an explicitly declared evaluator population — a reproducibility target, not semantic truth. The authors separate three sampling regimes — frozen finite censuses (exact hypergeometric error), probabilistic populations (exact binomial error), and growing censuses (needing convergence envelopes) — and show their guarantees are not interchangeable. The core contribution is a nested finite-sample certificate: marginal Clopper–Pearson intervals per generator column, a Tonelli–Markov allowance for interval failures, and exact one-sided binomial inversion at the generator layer, yielding simultaneous lower bounds on population coverage over a predeclared panel-size grid while permitting arbitrary shared-row dependence. Adversarial sections derive exact worst-case certificates for non-adaptive census contamination and a targeted-contamination sensitivity profile; simulations verify coverage and expose power limits. The empirical component is a hash-pinned LLM pilot (40 evaluator rows × 50 manually stratified units, 2,000 cells): at panel size 47, 45/50 units certify (lower coverage 0.8775 under declared weights) and 47/50 units are census-stable at K≤7; nine of thirteen construction-underdetermined units also certify. Limitations are candidly stated: no semantic adjudication, manual stratification precludes population claims, only the N=1 aggregation path is exercised, and raw provider text remains unpublished.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Panel reproducibility relative to a declared evaluator population is a distinct estimand, separable from frozen-census agreement and from semantic truth.
- A nested certificate of marginal exact binomial intervals plus exact generator-layer binomial inversion yields simultaneous lower bounds on population coverage over a predeclared panel-size grid, valid under arbitrary shared-row dependence (Theorem 2.1/E.7).
- The same vote matrix yields non-interchangeable conclusions under census versus population reference objects (pilot: 47/50 census-stable at K≤7 versus zero superpopulation certificates).
- Non-adaptive contamination admits exact worst-case hypergeometric certificates, and targeted contamination is at least as damaging as an equal fixed Byzantine share, with 50% frontier γH = α/(2(1−α)) (Section F.5).
Strengths
Operational regime separation
Table 1 pins three reference objects to their exact endpoint laws (hypergeometric / binomial / envelope-conditioned), and Section 4.2 makes the distinction concrete: the same 40×50 matrix yields 47/50 exact census-stability results at K≤7 but zero superpopulation certificates there — a genuinely instructive demonstration that census and population guarantees must not be conflated.
Dependence-tolerant exact certificate
Theorem 2.1/E.7 combines marginal Clopper–Pearson intervals, a Tonelli–Markov charge (ξE) for interval failures, and exact one-sided binomial inversion ℓA(s; ηG/|K|); no cross-column independence is assumed, so shared evaluator rows — the realistic failure mode for LLM panels — are covered by construction.
Exemplary scope discipline
Reproducibility artifacts
Weaknesses
Pilot is a mechanics check, not evidence
Load-bearing premise is untestable from released data
Modest mathematical novelty
Deployment relevance gap and partial auditability
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper introduces aggregate disambiguation systems (ADSs): a finite panel of protocol-following evaluator episodes casts binary votes on a frozen candidate resolution, and aggregation targets the threshold decision of an explicitly declared evaluator population rather than semantic truth (Section 2.1). Three claims are central: (1) fixed finite censuses, probabilistic populations, and growing censuses obey different endpoint laws (hypergeometric, binomial, envelope-conditioned — Table 1) and their guarantees are not interchangeable; (2) a nested finite-sample certificate lower-bounds, simultaneously over a predeclared panel grid K, the generator mass of resolutions that a fresh panel will decide reproducibly, valid under arbitrary shared-row dependence (Theorem 2.1, E.7); (3) non-adaptive contamination admits exact worst-case certificates and a workload sensitivity profile (Theorems F.3, F.5). Institutional disclosure: the work comes from GenLayer Labs, whose nominal panel ladder supplies the application grid — disclosed, with repeated disclaimers against deployed-network security readings.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is an aggregate disambiguation system (ADS)?
A decision mechanism in which several independent evaluator episodes (for example, LLM calls) each cast one binary vote on whether a frozen candidate solution should be accepted, and a fixed rule aggregates the panel's votes. The paper's question is whether a finite panel reproduces the accept/reject decision that an explicitly declared population of evaluators would make — a matter of reproducibility, not of whether the answer is semantically correct.
Does this paper show that LLM judge panels are reliable?
No. It supplies exact statistical machinery for certifying how often a finite panel would reproduce a declared evaluator population's decision. The small real-data pilot (40 evaluator rows × 50 hand-built cases) mostly verifies the mechanics: 45/50 cases certified at panel size 47, but cases were manually chosen, no truth labels were adjudicated, and the authors state the results apply only to those fixed cases.
Why do the census and population analyses give different answers on the same data?
Because they answer different questions. Conditioning on the 40 realized votes per case, exact hypergeometric calculations show 47/50 cases are stable at panel size 7 or less; treating the rows as an i.i.d. sample from a declared population, zero cases can be certified at that size because 40 rows carry too little information. The paper's central point is that these two reference objects must never be conflated.
Is the code and data available?
Partially, yes. A companion release (v1.0.0) provides a sanitized 2,000-cell binary ledger, the realized evaluator design, per-item CSVs, dependency locks, deterministic simulation code, and hash manifests; the ledger suffices to recompute the vote-based results offline. Raw provider text and routing traces are not published, so the underlying LLM calls cannot be independently re-audited.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: marked gaps skip mid-paper passages (parts of Section 3.2–3.3, Appendix B definitions, the E.4/F.5 proofs, and parts of S4); judgments about those specific passages rest on the surrounding visible text only.
- Figures 1–7 and S1 are referenced but not embedded in the extracted text, so figure content was not independently verified; all numbers cited here come from the visible text and tables.
- Certificate proofs in the skipped E.4/F.5 middle segments were assessed via the surrounding statement, proof sketch, and visible proof fragments, not line-by-line.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.