Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation
Zeyu He, Zhuqian Zhou, Kirk Vanacore, Rene F. Kizilcec, Ting-Hao 'Kenneth' Huang
cs.CL · cs.AI · cs.HC · cs.LG
Quick verdict
Claims a few hours of expert rationale-labeling beats a 6-month codebook (64.9% vs 57.8% accuracy); strong single-project evidence, generality unproven.
Summary
The paper proposes a human-AI workflow in which six LLMs annotate unlabeled text with an early codebook, and cross-model disagreement (normalized vote entropy) identifies ambiguous utterances and labels toward which limited expert attention should be directed. In a six-month tutoring-transcript annotation project (29 tutoring moves plus a None class; two PhD-level education experts), the authors compared three feedback workflows: Codebook Verifying (editing LLM-proposed revisions), Question Answering (answering six LLM-generated clarification questions per label), and Rationale Labeling (labeling selected disagreement cases with written rationales). Codebooks produced by each workflow, plus the initial and final expert codebooks, were evaluated by six LLMs on a held-out set of 4,032 expert-consensus utterances, aggregated with eight Crowd-Kit methods. Rationale Labeling performed best (64.9% accuracy, 0.658 weighted F1), beating the final expert codebook (57.8%, 0.583) across all eight aggregation methods; the best Question Answering variant reached 60.5%. Rationale Labeling required roughly 2.8 expert-hours versus an estimated ~300 hours for the expert process. Limitations include a single project with only two experts, possible carryover between workflows in fixed order, multi-label utterances collapsed to the first label, and no public data or code release.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Cross-LLM disagreement (normalized vote entropy) is a usable signal of annotation difficulty at both utterance and label level (Fig 7; per-tier results in §5.1).
- A codebook revised via Rationale Labeling reaches 64.9% accuracy / 0.658 weighted F1 on 4,032 held-out utterances, significantly beating the six-month expert Final Codebook (57.8% / 0.583) across all eight aggregation methods.
- The best Question Answering variant (Ambiguity Pattern + Within/Cross-Label QA) also beats the Final Codebook (60.5% / 0.613), while Codebook Verifying adds little over LLM-only revision.
- A few hours (~2.8h) of strategically targeted expert effort can substitute for months (~300 hours estimated) of untargeted expert codebook iteration in LLM annotation performance.
Strengths
Rare longitudinal evaluation asset
An independently motivated six-month project (§4.1) preserved codebook snapshots and 4,032 held-out expert-consensus utterances, enabling a genuine head-to-head of a few-hour AI-assisted revision against the 6-month expert Final Codebook (57.8%) — a baseline most papers cannot construct.
Honest negative and preliminary results
The authors report that Codebook Verifying added little over LLM-only revision (§5.2), that APO (51.2%) and SPO (48.7%) fell below the unrevised Initial Codebook (55.0%) on the 586-utterance preliminary set (App. A.2), and that model identity dominated expert personas (Fig 8, κ 0.68–0.90 within-model vs 0.48–0.64 cross-model) — with appropriately hedged interpretation.
Careful effort accounting and multi-method evaluation
Exemplary methodological transparency
Weaknesses
Statistical unit of inference is questionable
Workflow comparison is confounded
Single project, two experts, modest absolute performance
No code or data release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.26926v1, submitted to ACM) asks how to spend scarce expert time when revising annotation codebooks for LLM annotators. Six LLMs from six providers independently label an unlabeled corpus with an early codebook; normalized vote entropy (§3.2, Eq. 1–2) flags utterances and labels where models disagree, and that disagreement routes expert effort into three feedback interfaces: Codebook Verifying (CV — editing LLM-proposed revisions), Question Answering (QA — answering six generated clarification questions per label), and Rationale Labeling (RL — labeling selected contested utterances with rationales). Central claims: (1) cross-LLM disagreement is a usable difficulty signal; (2) RL-revised codebooks beat a codebook two experts refined over six months (64.9% vs 57.8% accuracy; weighted F1 0.658 vs 0.583) with ~2.8 expert-hours; (3) the best QA variant (Ambiguity Pattern + Within/Cross-Label QA) also beats it (60.5%), while CV adds little over LLM-only revision.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper 'Experts Rise Where LLMs Disagree' show?
It shows that when several LLMs annotate the same text with an early codebook, the cases where they disagree flag genuine ambiguities. Directing a few hours of expert attention to those cases — especially having experts label about 120 contested utterances with written rationales — let an LLM rewrite the codebook so six LLM annotators reached 64.9% accuracy on held-out data, beating a codebook that two experts spent six months refining (57.8%).
Which expert feedback method worked best: Codebook Verifying, Question Answering, or Rationale Labeling?
Rationale Labeling performed best (64.9% accuracy), followed by the best Question Answering variant (60.5%), while Codebook Verifying — manually editing AI-proposed revisions — added little over letting the LLM revise alone. All are compared against the six-month expert codebook at 57.8% accuracy.
Is the code or data for this paper publicly available?
No. The paper documents all prompts and pinned model versions in detail, but the tutoring transcripts are private (shared under provider agreements under IRB approval), and no public code or data release is stated, so reported numbers cannot be independently recomputed.
What are the main limitations of the study?
It evaluates one annotation project with two experts; workflows ran in a fixed order with possible learning carryover; multi-label utterances (6.1%) were collapsed to the first label; evaluation gold labels come from the same expert process that produced the compared Final Codebook; and the data is not public.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extracted text omits several passages (parts of §3.3.2, §4.3.1, the body of the main results Table 1, and a section referenced as 'Section 9'); headline numbers were verified from prose and figure captions, not the table itself.
- Figures 5–8 and 22 are not analyzable from text, so exact confidence intervals, per-condition table values, and t-statistics could not be checked.
- This is an arXiv preprint (2609.26926v1, Sep 2026) of a manuscript submitted to ACM; it has not yet completed peer review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.