GUT: Quantifying and Optimizing the Reasoning Uncertainty of LLMs via Graph Complexity
Shuang Liang, Xin-Yu Hu, Xiang-Jun Ou, Shao-Qun Zhang
cs.AI
Quick verdict
Graph-complexity UQ (DAG+NLI merge) with RL optimization; strong 45-baseline gains but Qwen3-only models and possible test-set tuning need scrutiny.
Summary
The paper proposes GUT, a two-module framework for reasoning uncertainty in LLMs. GUT-Q samples K chain-of-thought solutions, decomposes them into steps, and builds a directed acyclic graph whose semantically equivalent steps are merged via bidirectional NLI entailment (DeBERTa). Token-level statistics (entropy, negative max probability, etc.) are aggregated into node uncertainties, and three graph-complexity estimators — uncertainty-weighted width (GUT-Q-W), height (GUT-Q-H), and uncertainty propagation through a small FNN (GUT-Q-UP) — quantify uncertainty. GUT-O fine-tunes models with GRPO using the negative uncertainty proxy as reward. Experiments cover four Qwen3 scales (0.6B–8B) and five datasets (GSM8K, MATH-500, AMC2022-2024, FOLIO, MMLU-Pro) against 45 UQ methods, reporting PRR/AUROC/AUPRC over 1000 bootstraps. GUT-Q-UP reportedly beats the best contenders by averages of 11.79% PRR, 13.33% AUROC, and 9.66% AUPRC (e.g., AUROC 86.61 vs Topo-UQ 64.77 on GSM8K, Qwen3-4B); GUT-O reduces mean uncertainty ~13.95% and adds ~1.93% accuracy. Limitations: evaluation is restricted to the Qwen3 family; hyperparameters (K, T, F) are recommended per model/dataset based on UQ performance without a stated tuning/eval split; many strong baselines show inverted AUROC well below 50% that deserves verification; and GUT-O trades accuracy against GRPO at larger scales (64.80 vs 66.60 at 8B). No code release is stated in the provided text.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Modeling sampled reasoning chains as a DAG with NLI-based node merging captures reasoning-branch uncertainty that sequence-level UQ methods miss.
- GUT-Q (especially the UP variant) outperforms 45 UQ contenders by an average of 11.79% PRR, 13.33% AUROC, and 9.66% AUPRC across 4 Qwen3 scales and 5 datasets in selective generation.
- GUT-O, which uses the negative uncertainty proxy as the GRPO reward, reduces mean reasoning uncertainty by an average of 13.95% while improving accuracy by an average of 1.93%.
- GUT-Q's runtime cost is comparable to SE, Eig-E, and Topo-UQ (Appendix B.4, Table 8, Figure 4).
Strengths
Unusually broad baseline coverage
45 UQ contenders across five categories (information-based, diversity-based, reflexive, reasoning-enhanced, graph-based; Table 7), with CoT-UQ and TopoUQ from official repos and the rest via LM-Polygraph; full per-scale tables 9-12.
Statistical reporting discipline
All headline metrics (PRR/AUROC/AUPRC) are bootstrap means±std over 1000 resamples (Section 5.1), plus sensitivity analyses over K, T, F and three NLI merging criteria (Figures 2, 6-9).
Runtime transparency
Careful graph construction
Weaknesses
Single model family
Hyperparameter selection on the reported task
Inverted baseline scores left unexplained
GUT-O accuracy trade-off vs GRPO understated
Coverage overclaim
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
GUT tackles two coupled problems: quantifying reasoning uncertainty in LLMs (GUT-Q) and reducing it (GUT-O). Its central claims are: (1) a DAG over sampled reasoning chains, with semantically equivalent steps merged by bidirectional NLI entailment, captures branch-level uncertainty that sequence-based UQ misses; (2) graph complexity approximates reasoning-space complexity, and GUT-Q-UP beats 45 UQ contenders by averages of 11.79% PRR, 13.33% AUROC, and 9.66% AUPRC across four Qwen3 scales (0.6B-8B) and five datasets; (3) GUT-O — GRPO with the negative uncertainty proxy as reward — cuts mean uncertainty by 13.95% on average while adding 1.93% accuracy.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the GUT method for LLM reasoning uncertainty?
GUT is a two-module framework. GUT-Q samples multiple chain-of-thought solutions per question, splits them into reasoning steps, builds a directed acyclic graph, merges semantically equivalent steps using an NLI model's bidirectional entailment judgments, computes per-node uncertainty from token-level statistics, and scores overall uncertainty with three graph-complexity measures (width, height, uncertainty propagation). GUT-O then fine-tunes the model with GRPO reinforcement learning, using negative uncertainty as the reward, to make reasoning less uncertain.
How much does GUT improve over existing uncertainty quantification methods?
The authors report that GUT-Q-UP beats the best of 45 compared UQ methods by averages of 11.79% in PRR, 13.33% in AUROC, and 9.66% in AUPRC, evaluated on four Qwen3 model scales (0.6B-8B) across five datasets (GSM8K, MATH-500, AMC2022-2024, FOLIO, MMLU-Pro) with 1000 bootstrap resamples. For example, on GSM8K with Qwen3-4B, GUT-Q-UP reaches AUROC 86.61 versus 64.77 for the strongest baseline (Topo-UQ).
Does GUT improve model accuracy, or only uncertainty?
Both, but with a trade-off. GUT-O reduces mean reasoning uncertainty by about 13.95% and improves accuracy by about 1.93% on average versus the untuned models. However, plain GRPO trained with accuracy reward achieves higher accuracy than GUT-O at the 4B and 8B scales (e.g., 66.60 vs 64.80 at 8B on MATH-500), so the uncertainty reward trades some accuracy for much lower uncertainty.
Is code or data available for the GUT paper?
The provided text states no code release for GUT itself. The paper's evaluation uses public resources: Qwen3 models, five public datasets, DeBERTa as the NLI merger, and contender implementations from LM-Polygraph plus the official CoT-UQ and TopoUQ repositories, so a re-implementation from the paper's pseudocode, prompt templates, and hyperparameter appendix is feasible but nontrivial.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The excerpt is a sampled head+tail extraction: the middle of the paper was skipped, including Section 4 (GUT-O formulation) and parts of the Section 5 experimental setup; judgments about GUT-O internals rely on the visible Section 5.2 text and Table 3.
- All figures (1-10) appear only as caption text; their visual content could not be assessed.
- Parts of Appendices B.5/C (full per-scale tables 9, 12-14) fall inside sampling gaps, so per-scale verification of the 'consistent outperformance' claim was possible only for the visible table blocks.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.