Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Tasks
Xiangyu Wang, Jin Wu, Xiaoyu Li, Chanjin Zheng, Yifeng Zhou
cs.CL
Quick verdict
CreaEval decouples judging into evidence analysis plus scoring, lifting QWK to 0.64 on CGPST; but the 200-sample dataset is the authors' own benchmark.
Summary
The paper introduces CreaEval, a training-free framework for automated evaluation of complex multi-step creativity tasks. Targeting CGPST — six interdependent steps scored on ~20 subjective dimensions — it splits LLM-as-a-Judge into two phases: a Structure-of-Thought LLM (qwen3.6-plus) that converts each step's raw response into structured evidence with a rule-based cross-step memory, and a Judge-LLM that scores from evidence and rubrics alone, never seeing raw responses. On CGPST (200 samples, two human raters, inter-rater reliability 0.84), CreaEval reaches average QWK 0.64 versus 0.47 for fine-tuned Qwen3.5-9B and roughly 0.2-0.25 for training-free baselines (CoT, ToT, GoT, TaT, SaMer); on the simpler AUT and TTCW tasks it averages 0.707, with the strongest prior average 0.581 (reference-based on TTCW). Ablations attribute the gains to memory, evidence, decoupling, and step-wise extraction (QWK falls to 0.49, 0.24, 0.21, and 0.01 without them), and an evidence-conditioned SFT (0.632) nearly matches the training-free pipeline. The authors report reduced leniency and verbosity bias and lower inter-judge variance. Main limitations: the only multi-step benchmark originates from the authors' own prior work (TeamLLM), the sample is small, significance tests lack multiple-comparison correction and confidence intervals, and error propagation from evidence extraction is never quantified. Code and prompts are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Decoupling LLM-as-a-Judge into Memory-augmented Analysis and Evidence-based Judging raises average human-LLM agreement on CGPST to QWK 0.64, exceeding supervised SFT (0.47) by 0.17.
- Scoring from structured evidence alone — without access to raw responses — is the decisive factor; removing evidence or decoupling drops QWK to 0.24 and 0.21 respectively.
- The framework generalizes to simple creativity tasks: on AUT and TTCW it averages QWK 0.707, above the best prior average of 0.581 (reference-based on TTCW).
- The decoupled design mitigates leniency and verbosity bias (only CreaEval reduces length-score correlation, -5.81% vs Direct Score) and lowers inter-judge variance across four Judge-LLMs.
Strengths
Thorough component isolation via ablations
Table 4 quantifies each design decision with large, ordered drops: w/o Memory QWK 0.49 (23% down), w/o Evidence 0.24 (63%), w/o Decoupled 0.21 (67%), w/o Step-wise 0.01 (98%), directly supporting the mechanism story rather than only the aggregate result.
Appendix G (SFT_Evidence) is the most informative experiment
Fine-tuning on evidence-score pairs reaches QWK 0.632 vs vanilla SFT 0.4665 and nearly matches training-free CreaEval (0.6388), showing the evidence representation — not two-LLM orchestration — carries most of the value; the paper states this honestly.
Robustness across judge and analyzer models
Bias analysis goes beyond aggregate agreement
High implementation transparency
Weaknesses
Benchmark monoculture with in-group provenance
Thin statistical reporting
Evidence faithfulness never validated
Overhead and weak supervised reference
Presentation errors undercut the headline number
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CreaEval (arXiv 2609.03432) targets automated evaluation of Contextually-Grounded and Procedurally-Structured Tasks (CGPST): six interdependent steps — challenge identification, problem selection, solution generation, criteria selection, ranking, action plan — each scored on multiple subjective dimensions with ranges up to 0-16 and 1-10 (Table 1). The central claim: conventional LLM-as-a-Judge couples analysis with scoring and inherits leniency/verbosity bias, so the authors decouple it into **Memory-augmented Analysis** (a SoT-LLM turning each step's response into structured evidence with a rule-based cross-step memory, Eq. 1) and **Evidence-based Judging** (a Judge-LLM scoring from evidence and rubrics alone, Eq. 2). Headline results: average QWK 0.64 on CGPST (+0.17 over supervised SFT) and a claimed 22.74% average improvement over second-best baselines across CGPST, AUT and TTCW.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CreaEval and how does it work?
CreaEval is a training-free framework that uses LLMs to automatically score creative responses. It splits evaluation into two phases: an 'analysis' LLM converts each step of a multi-step creative response into structured evidence while carrying a memory summary across steps, and a separate 'judge' LLM assigns rubric-based scores using only that evidence — the judge never sees the raw responses. The paper argues this decoupling reduces leniency bias (over-generous scores) and verbosity bias (favoring longer answers).
How much better is CreaEval than standard LLM-as-a-Judge methods?
On the CGPST benchmark (200 six-step responses scored by two human raters), CreaEval reaches an average quadratic-weighted kappa of 0.64 with human scores, versus roughly 0.2-0.25 for direct scoring and CoT/ToT/GoT-style prompting, and 0.47 for a fine-tuned 9B model. On the simpler AUT and TTCW creativity tasks it averages 0.707, above the strongest prior average of 0.581 (a reference-based method on TTCW).
Does CreaEval require any training or fine-tuning?
No. The published pipeline is entirely prompt-based and model-agnostic: the authors run it with four different commercial LLMs as judges with nearly identical results. They also show that fine-tuning a model on evidence-score pairs (QWK 0.632) approaches the training-free pipeline (0.6388), which suggests the structured evidence representation carries most of the improvement.
What data was CreaEval evaluated on, and is it public?
The main benchmark is CGPST, built on the 50-year-old Future Problem Solving Program International competition: 10 future scenarios, 20 complete six-step responses each, scored on about 20 subjective dimensions by two human raters (inter-rater reliability 0.84). CGPST was introduced in the authors' own related 2026 preprint (TeamLLM). The generalizability tests use the public AUT (Alternate Uses Task) and TTCW (Torrance Test of Creative Writing) benchmarks.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with two '[...sampling gap...]' regions: most Step-1–3 rows of PCC Table 12 and parts of temperature Table 11 were skipped, so those specific cells were not audited.
- Figures (1-7, heatmaps, violin plots) appear only as text fragments in the extraction; visual judgments (e.g., heatmap balance in §6.2) rest on captions and stated numbers, not the images.
- The metric type of the reported 0.84 human inter-rater reliability is not stated anywhere in the visible text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.