CALICO: A Human-Centered, Codebook-Aligned System for Annotation
Boqin Yuan, Xiaoyi Gu, Fiona Li, Chang Wan, Angel Hsing-Chi Hwang, Jieyu Zhao
cs.HC · cs.CL
Quick verdict
CALICO makes codebook prompts versioned and optimizable; +13.0/+7.4 pp held-out gains but one small domain, 3 seeds, no significance tests, no user study.
Summary
CALICO is a web-based annotation workflow that treats LLM prompts as editable, versioned, and optimizable artifacts derived from expert codebooks. A parsing agent converts uploaded codebooks (PDF, DOCX, XLSX, etc.) into a validated schema; a prompt generator produces per-dimension prompts; experts refine prompts via natural-language feedback; and label-supervised optimizers — GEPA, MIPROv2, OPRO, and the authors' reflection-based ReflectAgent — improve prompts from labeled examples, with stratified splits, validation-based rollback, and a held-out test scored once. Evaluated on two AI-companion chatbot codebooks labeled independently by two coders (who disagree substantially, e.g., 34.8% raw agreement on Intimacy), ReflectAgent improves mean held-out accuracy by +13.0 and +7.4 points over a codebook-initialized zero-shot prompt across four Self-Disclosure dimensions. In backend comparisons on Coder A, MIPROv2 (73.9) and GEPA (73.8) edge out ReflectAgent (71.5). A coder-specificity analysis on Intimacy (n=35) shows tuned prompts align with the tuning coder (+23.8/+30.5 pp diagonal advantage), and gains replicate across four annotation models (+20.8 to +34.6 pp). Main limitations: a single domain with roughly 330–340 items per coder, three seeds without significance testing, no user study despite the human-centered framing, and the Depth dimension regresses for Coder A (−2.3 pp).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Treating prompts as editable, versioned, optimizable artifacts within a codebook-aligned workflow makes LLM-assisted annotation more reliable and auditable than fixed-prompt practice.
- ReflectAgent improves mean held-out agreement by +13.0 pp (Coder A) and +7.4 pp (Coder B) over a codebook-initialized zero-shot prompt across four Self-Disclosure dimensions (Table 1).
- Optimized prompts capture coder-specific interpretations, shown by a diagonal cross-coder pattern on Intimacy (Table 3: 75.2/41.0 vs 51.4/71.4).
- CALICO supports multiple optimization backends (GEPA, MIPROv2, OPRO, ReflectAgent) under one interface; MIPROv2 and GEPA are most accurate while ReflectAgent yields the most inspectable artifact (Table 2).
Strengths
Honest optimizer comparison
Table 2 reports that the in-house ReflectAgent (mean 71.5) loses to MIPROv2 (73.9) and GEPA (73.8) on Coder A's Self-Disclosure dimensions, with OPRO nearly flat (61.5 vs 58.5 zero-shot) — the paper sells inspectability, not SOTA.
Principled treatment of annotator disagreement
Rather than adjudicating into a fake gold set (raw agreement 67.4% on Level, 34.8% on Intimacy), §4.1 evaluates against each coder separately; the Table 3 cross-coder diagonal (+23.8/+30.5 pp matched-coder advantage) is a genuinely novel analysis.
Above-average methodological hygiene
Full system release
Weaknesses
Tiny, single-domain evaluation
No inferential statistics
Human-centered claim unvalidated
Metric and baseline gaps
Reporting inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CALICO (arXiv 2609.14726) is a system paper: a web workflow that turns expert codebooks into editable, versioned, optimizable LLM prompts. Four modules — codebook parsing agent (§3.1), per-dimension prompt generator (§3.2), natural-language feedback compiled into project-scoped calibration memory with diff-approved versioning (§3.3), and an optimizer layer wrapping GEPA, MIPROv2, OPRO and the authors' ReflectAgent (§3.4). Claims: (RQ1) label-supervised refinement beats a codebook-initialized zero-shot annotator — +13.0/+7.4 pp mean over four Self-Disclosure dimensions for Coders A/B (Table 1); (RQ2) optimized prompts encode coder-specific interpretations (Table 3 diagonal). The abstract's headline numbers match Table 1, but they are means over dimensions whose individual effects range from −2.3 pp (Depth, Coder A) to +29.4 pp (Confession, Coder A).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CALICO?
CALICO is a web application that converts expert annotation codebooks into editable, versioned LLM prompts, lets domain experts inspect draft annotations and correct the prompt via natural-language feedback, and runs prompt optimizers (GEPA, MIPROv2, OPRO, and the authors' ReflectAgent) when labeled examples exist. It is released under the Apache 2.0 license with a live demo.
How much does CALICO improve annotation accuracy?
In its case study on AI-companion chatbot codebooks, the ReflectAgent optimizer raised mean held-out accuracy over a zero-shot prompt by +13.0 points for one coder and +7.4 for another across four self-disclosure dimensions. Per-dimension effects varied widely, from +29.4 points down to −2.3 points, and gains were measured over three runs without statistical significance tests.
What is ReflectAgent?
ReflectAgent is the paper's own prompt optimizer. Each round it labels training examples with the current prompt, extracts reusable error-pattern rules into a human-readable rule library, folds accepted rules into the prompt, and rolls back updates that hurt validation performance. Unlike GEPA or MIPROv2, its output is an inspectable, editable set of labeling rules.
Does CALICO generalize beyond chatbot conversation analysis?
The authors state they do not claim cross-domain generalization. The evaluation covers two codebooks from a single AI-companion conversation project, with roughly 330-340 items labeled per coder, and no user study was conducted, so usability and transfer to other domains remain untested.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 appear only as captions in the extracted text; interface screenshots and the workflow diagram could not be visually assessed.
- Table 4 (macro-F1) is single-seed and Table 5 (model robustness) covers only Coder A's Intimacy dimension, so some robustness evidence is narrow.
- Dataset availability is not stated in the extracted text; exact-number reproduction may depend on non-public conversational data and coder labels.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.