Scored vs. Generated Readouts in Behavioral Language Models: An Empirical Study of Elicitation Format
Touchapon Kraisingkorn, Krittin Pachtrachai, Wachiravit Modecrua
cs.AI
Quick verdict
Controlled study: scored beats rationale-then-answer ranking in 12/13 retail LLM cells; credible, but mechanism is correlational and cells share checkpoints.
Summary
This empirical study tests whether two ways of querying a fine-tuned behavioral language model — reading P(YES) from answer-token probabilities (scored) versus letting the model write a short rationale before answering (generated) — are interchangeable for ranking real customer outcomes. Across 13 checkpoint-domain cells spanning four retail tasks (SEA grocery coupons, the public DMBGN voucher benchmark, assortment-change natural experiments, and the public Dunnhumby US grocery dataset) and five checkpoints from an untuned Qwen3.5-27B base to a frontier reasoning API, the scored readout ranks outcomes better in 12 of 13 cells (two-sided sign test, p≈.003), by 1.5–14.5 AUC points, with paired bootstrap confidence intervals excluding zero on newly measured cells. The gap grows with task-specific supervision and train/serve format mismatch, from −2.2 for the untuned base to +13.7 for rationale-format SFT. Per-row analysis of roughly 9,000 rationales links the gap to abandoning the dominant predictive feature and template convergence, while probability saturation does not track it; an ordering ablation (decision-first .736 vs reasoning-first .649) is the one causal manipulation. A third readout, verbalizing a probability before any verdict, recovers calibration (Brier 0.47→0.15) with ranking within noise of scoring, but only for outcome rates seen in training. Limitations: cells share checkpoints and domains, the mechanism evidence is correlational with per-row data on a subset of cells, historical cells lack confidence intervals, and tabular baselines still outperform all language-model arms.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- With checkpoint and prompt content fixed, the scored readout ranks real retail outcomes better than the rationale-then-answer generated readout in 12 of 13 cells (two-sided sign test p≈.003; 1.5–14.5 AUC points).
- The gap scales with task-specific supervision and train/serve format mismatch, from −2.2 points for an untuned base to +13.7 for rationale-format supervision.
- Anchor abandonment (e.g., Spearman .716→.305 on SEA coupon) and template convergence (up to 33% top-3 opening share) track the gap, while probability saturation does not.
- A verbalized-probability readout recovers calibration (Brier 0.47→0.15) with ranking within noise of scoring, but only for outcome rates present in training and only when the scored head is miscalibrated.
Strengths
Genuinely controlled readout isolation
§3.4 fixes system/user prompt content across arms; GENERATED appends the identical SCORED cue after </think> so the arm differs only by P(YES|own rationale). The ordering ablation (§4.1: decision-first .736 vs reasoning-first .649 on voucher; .841→.775 on the full test set) adds the paper's one causal manipulation.
Dose–response structure rather than a binary win
Table 1 orders the gap with supervision format and mismatch: −2.2 (untuned Base) to +13.7 (Rationale-SFT), with InDomain-SFT at +3.0 while posting the best in-house scored AUC (.808) — a falsifiable pattern, not just a direction.
Reproducible core on public instruments
Honest scope and negative-result reporting
Retrofittable diagnostics backed by a negative control
Weaknesses
The 13 cells are not independent draws
Smallest gaps carry no uncertainty quantification
Mechanism evidence is correlational and under-covered
Practical relevance of the fixed quantity is conditional
VERBAL-P evidence base is narrow
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, operational question: for language models fine-tuned as retail "behavioral simulators," does reading a decision probability from answer-token scores (SCORED) versus generating a 2–4 sentence rationale and then reading P(YES|rationale) (GENERATED) change ranking accuracy against real outcomes? Four contributions are claimed (§1): (1) SCORED wins 12 of 13 cells by 1.5–14.5 AUC points (Table 1, two-sided sign test p≈.003); (2) the gap scales with supervision dose and train/serve format mismatch (−2.2 untuned base → +13.7 rationale-SFT); (3) anchor abandonment and template convergence track the gap while probability saturation does not; (4) a VERBAL-P readout recovers calibration (Brier 0.47→0.15, Table 2) under two scope conditions. Scope is honestly bounded: retail binary outcomes only, ranking as the target metric, and — per §3.5 — no claim against tabular superiority, since GBDT baselines (.838 leaves-AUC; .866 profile+session voucher) beat every LM arm where fittable.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the difference between scored and generated readouts in LLM behavioral prediction?
A scored readout reads the model's token probability for YES/NO directly from the logits; a generated readout lets the model write a short rationale first and then reads the probability conditioned on that text. This paper holds the model checkpoint and prompt content fixed and finds the scored readout ranks real retail outcomes better in 12 of 13 tested cells, by 1.5–14.5 AUC points.
Does chain-of-thought reasoning before answering hurt prediction accuracy?
In this paper's retail prediction setting, yes for ranking: asking the model to reason before answering lowered AUC in 12 of 13 model-domain cells versus direct token scoring. The largest gaps (+9.3 to +13.7 points) appeared in models trained on rationale-format data, and the gap tracked rationales abandoning the dominant predictive feature and becoming formulaic — not probability saturation.
Is the code and data for this paper available?
Per the paper's reproducibility statement, two of the four evaluation domains are fully public (the Dunnhumby Complete Journey US grocery dataset and the DMBGN voucher benchmark), and the authors release checkpoints, prompts, evaluation and analysis code, and per-row dumps for those. The two Southeast-Asian domains use proprietary data and release protocols only.
What is the verbalized probability readout and when does it help?
It asks the model to state a 0–100 likelihood number before giving any verdict. In the paper it recovered calibration (Brier score improving from 0.47 to 0.15 on the main comparison) with ranking accuracy within noise of token scoring. However, it only works for outcome rates the model saw in training, and it is worse than direct scoring when the model's scored head is already well calibrated.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Appendices A–D appear in the extraction only as content outlines; the underlying prompt templates, training tables and protocol ledger are not in the provided text, so appendix-level details could not be verified.
- Figure 1 panels A–C are referenced but no images are present in the extraction; figure-dependent claims were assessed from the surrounding text descriptions only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.