Likelihood Ranking doesn't Scale Like Prompting in LLMs
Alessandro Bondielli, Lucia Passaro, Davide Bacciu, Alessandro Lenci
cs.CL
Quick verdict
In 95 LLMs, statement-likelihood accuracy stays flat with scale while prompted accuracy soars; a careful, large study, but family clustering tempers certainty.
Summary
The paper asks whether likelihood-based and prompting-based LLM evaluation probe the same capability. The authors contrast two protocols on 10 multiple-choice QA datasets (~88,910 questions): prompted answering (APX: greedy generation, regex-parsed letter answers) and declarative-statement likelihood ranking (APS: selecting the option rewritten as an affirmative statement with lowest perplexity). Statements were generated with gpt-oss-20b and validated on a stratified 1% sample (BLEU mean 0.69, BERTScore F1 mean 0.68, 99% judged semantically equivalent by two LLM judges with 0.98 raw agreement, 95% sharing one surface template). Across 95 open-weights decoder-only models (0.1B–104B, base and instruct variants; MoE and reasoning models excluded), AIC-selected scaling fits show APS improving slowly and staying bounded while APX rises sharply; instruction tuning shifts the APX-over-APS break-even to smaller models. A control (Appendix B, 51 models) shows standard option-likelihood MCQA scales like APX (slope difference p=0.58; ρ=0.5444), arguing declarative ranking measures a distinct signal. The authors acknowledge limits: MCQA-only scope, family imbalance across scale, open weights only, no training-token control, and residual surface-form sensitivity of APS. Reviewer concerns add family pseudo-replication and the absence of a formal statistical test for the main divergence in the main text.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Declarative-statement likelihood accuracy (APS) improves slowly and stays bounded across model scale, while prompted accuracy (APX) rises sharply (§3, Fig. 2).
- Instruction tuning shifts the prompting break-even point — where APX surpasses APS — toward smaller model sizes (§3).
- Standard likelihood-based MCQA scales like APX, sharing the task-conditioned answer-selection interface, whereas declarative ranking probes a distinct signal (Appendix B).
- Prompted gains conflate distributional support with a learned answer-selection policy; instruction tuning primarily optimizes that interface rather than the underlying distribution (§4).
Strengths
Unusual breadth of evidence
95 open-weights decoder-only models across ~15 families (0.1B–104B, Table 2) evaluated on 10 MCQA datasets totaling 88,910 questions (Table 1); the divergence claim rests on scale, not a handful of models.
A control that does real work
Appendix B (51 models) shows standard option-likelihood MCQA scales with APX (ρ=0.5444, p=3.0×10⁻⁵; slope difference p=0.58), so flat scaling is specific to declarative ranking rather than likelihood scoring in general — this is the paper's sharpest differentiator.
Validated statement-generation pipeline
Careful prompting hygiene
Weaknesses
Model-family pseudo-replication
Headline divergence lacks formal statistics
APS validity rests on a 1% validation sample
Aggregation unspecified
No code release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper compares two LLM evaluation protocols on identical data: prompted answering over multiple-choice questions (APX — Accuracy on Prompt Execution) and likelihood ranking of declarative statements (APS — Accuracy on Perplexity Score), where each option is rewritten as an affirmative statement and the lowest-perplexity statement is selected. The corpus spans 10 HuggingFace MCQA datasets totaling 88,910 questions (Table 1: MMLU-Pro 10,632 items averaging 9.45 options; TruthfulQA only 714). Evaluation covers 95 open-weights decoder-only models from ~0.1B to 104B parameters across ~15 families (Table 2), base and instruct variants, excluding MoE and reasoning models. Three central claims: (1) APS scales weakly and stays bounded while APX improves sharply (§3, Fig. 2); (2) instruction tuning shifts the prompting break-even point toward smaller models; (3) standard likelihood-based MCQA scales like APX, so the flat-scaling signal is specific to declarative ranking, not likelihood scoring in general (Appendix B).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the difference between APS and standard likelihood-based MCQA scoring?
Standard likelihood MCQA scores each answer option as a continuation while the model still sees the question and the full option list, so the task interface remains. APS instead compares the perplexity of full declarative statements (e.g., 'The capital of France is Paris') generated for each option, removing that selection context. The paper's control shows standard likelihood MCQA scales with prompting, while declarative-statement ranking stays nearly flat with model scale.
Did bigger models get better under likelihood ranking?
Only slowly. Across 95 open-weights models from 0.1B to 104B parameters, accuracy from ranking declarative statements by perplexity improved gradually and stayed bounded, whereas prompted answering improved sharply with scale, and instruction-tuned models crossed above the likelihood-ranking level at smaller model sizes.
How were the declarative statements generated and checked?
gpt-oss-20b rewrote each question-option pair into an affirmative statement, and items with a wrong number of statements were filtered. On a stratified 1% sample, two LLM judges deemed 99% of statements semantically equivalent (raw agreement 0.98), 95% of items shared one surface template, and BLEU/BERTScore medians were about 0.73–0.75.
Are the code and data available?
The authors state the constructed dataset (original QA pairs plus generated statements) will be released on HuggingFace upon acceptance of the paper. No code repository is mentioned in the paper itself.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–10 are images; their quantitative content (exact Figure 2 curves, Figure 3 error rates, per-dataset plots) could not be analyzed from the extracted text.
- The cross-dataset aggregation behind the headline curves is not specified in the text, so curve-level claims rest on the authors' descriptions.
- Per-model accuracies are not in the text; family-level checks rely on the Table 2 inventory.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.