Robust Conformal Consensus: Multi-Agent LLM-as-a-Judge Interval Evaluation with Conformal Prediction
cs.LG · cs.AI
Quick verdict
Multi-LLM conformal consensus intervals for LLM-as-a-Judge with meta-calibration; sound marginal guarantee but thin novelty and unspecified judge weights.
Summary
This paper addresses uncertainty quantification for rating-based LLM-as-a-Judge evaluation. Three LLM judges (GPT-4o mini, DeepSeek-R1-Distill-Qwen-32B, Qwen2.5-72B-Instruct) rate SummEval summaries via G-Eval and ROSCOE reasoning steps via SocREval, and nine conformal prediction methods—CQR, asymmetric CQR, CHR, LVD, Boosted CQR, Boosted LCP, R2CCP, Ordinal APS, and Ordinal Risk Control—turn each judge's logits and ratings into prediction intervals. The proposed Weighted Conformal Consensus averages interval endpoints with reliability weights and adds a meta-calibration step on a held-out 10% split, which restores a marginal 1−α coverage guarantee (Theorem 1) through the standard split-conformal quantile argument. Across 30 seeds, CQR and asymmetric CQR reach roughly 90% coverage with the widest intervals (up to about 3.9 on ROSCOE), while CHR and boosted variants are narrower but under-cover. An integer boundary adjustment raises coverage—for example, LVD on e-SNLI with Qwen2.5 improves from 85.96% to 95.53%—and multi-agent aggregation is marginally more stable (e.g., 74.0% versus 72.65% for CHR on CosmosQA). Key limitations: the concordance weights are never formally specified or ablated against simple aggregation, only seed means are reported without variance, and the Limitations section contains unrelated boilerplate about knowledge graphs.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Applying conformal prediction separately to multiple LLM judges and aggregating their intervals yields more stable uncertainty estimates than any single judge.
- Weighted Conformal Consensus with a meta-calibration step restores a marginal 1-alpha coverage guarantee for the aggregated interval (Theorem 1).
- Boundary adjustment to integer Likert labels systematically improves empirical coverage (e.g., LVD on e-SNLI with Qwen2.5 from 85.96% to 95.53%).
- Interval quality depends strongly on both the conformal method and the judge: CQR/asymmetric CQR reach ~90% coverage with the widest intervals while CHR and boosted variants under-cover.
Strengths
Broad, systematic empirical sweep
Nine conformal methods (CQR, Asym CQR, CHR, LVD, Boosted CQR/LCP, R2CCP, Ordinal APS/RC) across two judge frameworks (G-Eval on SummEval, SocREval on ROSCOE), three judges plus a pooled Multi-Agent setting, 30 seeds; Tables 1-2 explicitly mark the <85%, 85-90%, and >=90% coverage bands, making the coverage-width trade-off directly readable.
Clean two-level calibration design
The 40/10/50 split keeps the meta-calibration set D_meta untouched by per-judge calibration (Section 4, Data Splits); Theorem 1 correctly treats the aggregation operator as a fixed predictor and applies the standard quantile lemma, so the marginal coverage statement is valid given its assumptions.
Honest reporting of the coverage-efficiency trade-off
Reasonable related-work positioning
Weaknesses
The core method component is unspecified
Statistical evidence too thin for the stability claim
Section 4.2 title overclaims and ordinal results are unexplained
Boundary-adjustment variant ambiguity
Serious polish/QC red flags
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.06367) proposes Weighted Conformal Consensus (WCC) for uncertainty quantification in rating-based LLM-as-a-Judge evaluation. Its claims: (1) per-judge conformal intervals aggregated across multiple LLM judges are more stable than single-judge intervals; (2) weighted endpoint averaging plus a meta-calibration step restores a marginal 1−α coverage guarantee (Theorem 1); (3) boundary adjustment to integer Likert labels systematically improves coverage; (4) interval quality depends strongly on the conformal method and the judge. Scope: SummEval (G-Eval) and ROSCOE (SocREval), three judges (GPT-4o mini, DeepSeek-R1-Distill-Qwen-32B, Qwen2.5-72B-Instruct), nine CP methods, 30 seeds.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Weighted Conformal Consensus (WCC) in LLM-as-a-Judge evaluation?
It is a proposed method in which several LLM judges each convert their 1-5 rating of a text into a conformal prediction interval; the interval endpoints are then averaged using reliability weights per judge, and a final 'meta-calibration' step on held-out data widens the merged interval so that it contains the true human rating at least 1−α of the time (the paper targets 90% coverage, α=0.1).
Does the paper's method come with a mathematical guarantee?
Yes, a marginal coverage guarantee (Theorem 1): if the meta-calibration examples are exchangeable with the test example and the judge weights are fixed in advance, the final interval contains the true rating with probability at least 1−α. The proof is the standard split-conformal quantile argument applied to the aggregated interval; it is correct but assumes the weights never depend on the meta-calibration data.
Which datasets, LLM judges, and conformal methods were used?
SummEval with G-Eval prompts and ROSCOE with SocREval prompts; judges were GPT-4o mini, DeepSeek-R1-Distill-Qwen-32B, and Qwen2.5-72B-Instruct. Nine conformal methods were tested: CQR, asymmetric CQR, CHR, LVD, Boosted CQR, Boosted LCP, R2CCP, Ordinal APS, and Ordinal Risk Control, over 30 random data splits.
Is code or data available for this paper?
The paper does not announce a code release; it states it builds on open-source datasets (SummEval, ROSCOE are public). The judge weighting rule is not specified in enough detail to re-implement from the text alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text ends at the reference list; the paper states prompts are 'provided in Appendix', which lies beyond the extracted text, so prompt wording could not be verified.
- Tables are extracted as flattened run-on rows; cell-to-column attribution relies on row-group ordering, so individual width/coverage pairs could be misread.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.