Entropy Can Flow, or It Can Guide. Be Entropy. LEDFlow: Introducing Entropy-guided Generation Order into Uniform Discrete Flow

Summary

The paper addresses a failure mode of uniform discrete flow matching: because every position is revised at each step, late denoiser errors can overwrite correct intermediate predictions — on Sudoku, 9.4% of generated cells are correct mid-sampling but wrong at the endpoint, and 46% of token changes occur after t=0.75. The authors introduce selective absorption, an inference-time indicator that zeroes outgoing transition rates of chosen positions while leaving the uniform-flow velocity of active positions unchanged, thereby imposing a generation order. LEDFlow is a training-free policy that absorbs the lowest-predictive-entropy positions first under a cosine budget, fixing their argmax values. Theoretically, a KL decomposition splits absorption error into joint-dependence and conditional-prediction terms; under an entropy-error regularity assumption, lowest-entropy selection minimizes an upper bound on the conditional term (Theorem 1), and a second result shows global-lookahead decision error grows with the lookahead window under an imperfect denoiser (Lemma 1). Empirically, LEDFlow reaches 0.845 Nikoli Sudoku accuracy, the best average rank (1.56) across nine reasoning tasks, the best decode-time GenEval score (0.7814 vs 0.7536 native), and modest gains on all six multimodal-understanding benchmarks, at roughly native runtime. Limitations: the theory is purely local; entropy ordering is not statistically separable from margin or max-probability ordering; the strongest Sudoku results use absorption-aware retrained denoisers; and the GenEval margin over corrected samplers is within benchmark resolution.