From Token Probabilities to Semantic Constraints: Towards Declarative Probabilistic Evaluation of Language Models
Kyle Richardson, Cullen Anderson, Pranav Balakrishnan, Takuto Ban, Daksha Ladia, Ankita Gupta, Marisa Hudspeth
cs.CL
Quick verdict
ModelLog recasts LM evaluation as weighted model counting over token constraints; an elegant frame, but headline failures are sensitive to calibration choice.
Summary
The paper introduces ModelLog, a declarative probabilistic framework for pre-training evaluation. Token-level prediction events are encoded as Boolean variables, semantic expectations (implication, negation consistency, mutual exclusivity) as logical constraints, and evaluation as weighted model counting (WMC): the probability mass a model's calibrated token probabilities place on satisfying assignments. A nucleus-entropy calibration rescales raw token probabilities toward Bernoulli 'validity' probabilities, and a rho metric normalizes satisfaction against an uninformed uniform baseline. Formally, next-token likelihood is recovered as a maximally informative conjunctive special case (Props. 1-2), and constraint informativeness I(F) = -log2 WMC0 is shown to amplify semantic-loss gradients (Props. 4-5). Empirically, PROBCT — five constraint subsets generated from WordNet/YAGO and LLM-judge-filtered — is run on twelve base models (Gemma-3, Llama-3, Qwen3, 270M-14B). Many models fail to beat chance, especially on negation consistency (CAcc roughly 16-23%) and spatial exclusivity (roughly 9-31%); transitivity and forward implication show inverse scaling (e.g., Gemma-3 forward implication falls from 79.7 at 270M to 45.2 at 12B), traced to premise/conclusion probability shifts. Limitations include single-token events under a factorized independence assumption, synthetic KG-derived tasks, no training experiments, and — per this review — strong sensitivity of the spatial-exclusivity results to the calibration choice.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ModelLog turns pre-training evaluation into probabilistic satisfaction of declarative constraints over token-prediction events, computed exactly via weighted model counting against an uninformed uniform baseline.
- Next-token likelihood is the special case of a maximally informative conjunctive formula, and constraint informativeness appears as an exponential (2^I(F)) factor in semantic-loss gradients, linking evaluation and learning through shared semantics.
- Current base models (Gemma-3, Llama-3, Qwen3, 270M-14B) frequently fail to beat the uninformed baseline on PROBCT consistency constraints, most severely on negation consistency (CAcc ~16-23%) and spatial exclusivity (~9-31%).
- Consistency on transitivity and forward implication inverse-scales with model size across all three families, driven by a shift toward higher premise and lower conclusion probabilities.
Strengths
Clean formal unification of evaluation and learning
Prop. 1 recovers next-token likelihood as satisfaction of a conjunctive formula; Props. 2-5 derive informativeness ordering and the gradient identity ∇ℓsl = -A(F;θ)·2^I(F)·∇WMC, giving an interpretable decomposition of semantic-loss learning signals.
Principled metric design against chance satisfaction
The ρ metric (Eq. 5) and CAcc (Eq. 6) normalize WMC against the uniform uninformed baseline WMC0(F), directly addressing the confound that weaker formulas (P1∨P2) are trivially more probable than stronger ones (§3.1).
Mechanistic account of inverse scaling
Calibration robustness ablation
Released code and data
Weaknesses
Headline failure on spatial exclusivity is calibration-dependent
No inferential statistics
Internal numeric inconsistency between Tables 2 and 6
Added diagnostic value over simple baselines is asserted, not measured
Thin validation of synthetic data
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ModelLog (arXiv 2609.13520, cs.CL) is a primarily methodological paper proposing that pre-training evaluation be recast as declarative probabilistic inference. Its claims: (1) evaluation targets are Boolean constraints over token-prediction events, scored by weighted model counting (WMC, Eq. 1); (2) raw token probabilities must be calibrated into Bernoulli 'validity' probabilities, done via a nucleus-entropy offset (Eq. 4); (3) metrics ρ (Eq. 5) and CAcc (Eq. 6) normalize satisfaction against the uninformed uniform baseline WMC0; (4) likelihood training is recovered as a maximally informative conjunctive minterm (Props. 1-2), and constraint informativeness I(F) = −log₂WMC0(F) appears as an exponential factor in semantic-loss gradients (Props. 4-5); (5) the PROBCT suite shows base models systematically fail consistency constraints, with inverse scaling on transitivity and forward implication. The authors honestly scope claims to single-token events and diagnostic (not training) use.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ModelLog framework?
ModelLog is a method for evaluating language models by encoding desired token-level behaviors as logical constraints (e.g., 'if a model accepts X then it should accept its paraphrase') and computing, via weighted model counting, how much probability mass the model's calibrated token probabilities place on assignments that satisfy those constraints. Scores are normalized against a random-guess baseline.
What is PROBCT?
PROBCT (Probabilistic Consistency Tests) is the diagnostic benchmark introduced with ModelLog: five task subsets (transitivity, forward implication, negation consistency, entity mutual exclusivity, spatial exclusivity) automatically generated from WordNet and YAGO knowledge graphs, LLM-filtered for quality, and used to test whether models' token probabilities jointly satisfy these logical constraints.
Do larger language models fail consistency tests according to this paper?
The paper reports an inverse-scaling pattern: on transitivity and forward-implication constraints, larger Gemma-3, Llama-3, and Qwen3 base models satisfy the constraints less often than smaller ones (e.g., Gemma-3 forward implication drops from 79.7 to 45.2 from 270M to 12B parameters), because larger models assign relatively higher probability to specific premises and lower probability to general conclusions.
Is the code and data for ModelLog available?
Yes. The paper states that code and data are available at github.com/cullena20/ModelLog, and the evaluation requires only next-token probability queries on public model checkpoints plus exact weighted model counting over small formulas.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 3 and Figures 4-6 histogram tick labels are interleaved in the PDF extraction, so distribution shapes (bin masses) were assessed from captions, aggregate CAcc labels, and tables rather than from bin geometry; all numeric tables extracted cleanly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.