FLEET: From Logits Entropy to Enhanced Trajectories in Text Generation
Oleksii Streltsov, Oleksandra Vitko
cs.LG · cs.AI · cs.CL
Quick verdict
FLEET adds memory-guided deterministic search over repeated sampling; LiveCodeBench gains (59.9%→66.2% Pass@32) look promising but lack ablations and stats.
Summary
The paper introduces FLEET, a decoding-time framework that replaces memoryless temperature sampling with memory-augmented search. FLEET triggers only at high-entropy/high-varentropy states (detected via Logit Lens projections from an intermediate layer), maps hidden states to discrete search states through a Vector Disjoint Set Union keyed on cosine similarity, and applies logit penalties to previously explored, suboptimal tokens using pUCT statistics; decoding is greedy. Thresholds, layer, and resampling parameters come from a single calibration pass (GMM over entropy-varentropy, composite layer scoring). On GSM8K and a 222-problem post-cutoff 'easy' LiveCodeBench subset, with Llama 3.2-3B under equal candidate budgets (n=32), ground-truth-guided FLEET beats Bayesian-tuned temperature sampling: Pass@32 rises from 59.9% to 66.2% on LiveCodeBench and from 97.3% to 97.8% on near-saturated GSM8K. With a Skywork-Reward-V2 reward model selecting the final answer, FLEET wins on LiveCodeBench (25.2% vs 19.4%) but loses on GSM8K (83.6% vs 85.4%). The 3x speedup claimed in the abstract has no supporting table in the text. Main limitations: a single model, single runs without significance tests or error bars, no ablation isolating the memory mechanism from entropy-gated greedy decoding, a prefix-based metric applied only to FLEET, and acknowledged incompatibility with reflective reasoning models.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- FLEET matches temperature-sampling accuracy with a 3x speedup (abstract claim; no supporting table appears in the provided text).
- Ground-truth-guided FLEET raises LiveCodeBench Pass@32 from 59.9% to 66.2% at equal candidate budget (n=32) on Llama 3.2-3B.
- The authors claim accuracy is always higher than repeated sampling under the same budget on the evaluated tasks, though ORM-selected GSM8K (Table 4.3) contradicts the unqualified reading.
- VectorDSU clusters hidden states online such that cosine similarity above a calibrated threshold τdsu bounds the KL divergence between projected next-token distributions (Eq. 14).
- All principal hyperparameters derive from a single calibration pass rather than iterative Bayesian/grid search.
Strengths
Well-motivated problem framing
Section 1 articulates limitation (a) — sample inefficiency at high-entropy branching points — precisely, and reward-aware memory at decoding time is a coherent response; the pUCT adaptation to vocabulary-scale action spaces (Sec. 3.1) is reasoned, including the aggregated 'exploration action' over unvisited top-k tokens.
Unusually principled calibration pipeline
Sec. 3.3 derives layer, thresholds, Tresample, and τdsu from one pass: composite layer score (Eq. 22), GMM clustering of the entropy–varentropy joint distribution (Fig 3.1b), Tresample at the crossover where exploratory tail mass exceeds top-1 mass (Fig 3.1c), and τdsu from a μ+2σ cosine–KL bound with ε=0.05 (Fig 3.1d) — contrasting with ad hoc T/k/p tuning.
Honest reporting of negative and boundary results
Baseline fairness effort
Weaknesses
No ablation isolating the memory mechanism
3x speedup claim unsupported
Single model, single runs, no statistical testing
Metric asymmetry between methods
Calibration-to-experiment inconsistency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FLEET (Secs. 1–3) replaces memoryless temperature sampling with a memory-augmented, deterministic search over completions. The pipeline: (i) trigger only at states whose entropy H and varentropy V (computed from Logit-Lens projections of an intermediate layer, Eqs. 8–11) jointly exceed thresholds; (ii) map triggered hidden states to discrete search states via VectorDSU, an online cosine-similarity clustering structure (Eq. 15); (iii) maintain per-cluster action/successor/reward statistics and penalize explored suboptimal tokens via a pUCT rule (Eqs. 17–20); (iv) decode greedily. Central claims: equivalent accuracy to repeated sampling with 3x speedup; LiveCodeBench Pass@32 59.9%→66.2% at equal budget; hyperparameters from a single calibration pass (Sec. 3.3). Code is stated to be at github.com/Alexiush/fleet.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FLEET and how does it differ from temperature sampling?
FLEET is a decoding-time search method for LLMs. Instead of drawing independent samples at a fixed temperature, it detects high-uncertainty token states (via entropy and varentropy of intermediate-layer logits), stores past generation attempts in a vector-clustered memory (VectorDSU), and penalizes tokens that previously led to low-reward completions. In the paper's configuration it decodes greedily and is deterministic, while temperature sampling is stochastic and memoryless.
How much does FLEET improve accuracy?
On Llama 3.2-3B with 32 generations per problem: LiveCodeBench (a 222-problem post-cutoff 'easy' subset) Pass@32 rises from 59.9% to 66.2% under ground-truth guidance; GSM8K improves marginally from 97.27% to 97.80%. With a reward model (Skywork-Reward-V2) selecting the final answer, FLEET wins on LiveCodeBench (25.2% vs 19.4%) but loses on GSM8K (83.6% vs 85.4%).
Is the 3x speedup claim backed by experiments?
The abstract claims same accuracy with 3x speedup, but the paper text contains no timing, wall-clock, or cost table supporting it; the claim appears to refer to needing fewer generations for equal accuracy. A full cost picture should also include reward-model evaluations and memory bookkeeping.
Does FLEET work with reasoning models like o1/R1-style chain-of-thought models?
The authors explicitly caution against it: FLEET currently struggles with 'reflective' internal states central to thinking-style models, where deterministic (greedy) decoding is actively detrimental. They also report occasional gibberish degradation of structurally less important tokens.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3.1, 4.1 and 4.2 (calibration diagnostics and scaling curves) are referenced but not analyzable from the extracted text.
- The linked repository's contents (checkpoints, exact configs, filtering rule) are asserted by the paper but not independently verified here.
- Minor text-extraction artifacts are possible; internal consistency of reported numbers was cross-checked where visible (e.g., 147/222 = 66.2%, 133/222 = 59.9%).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.