Complex Problem Solving in Large Language Models: A Statistical Control Survey and Diagnostic Framework
Jiazhang Cai, Tao Wang, Ruidong Zhang, Siyuan Li, Terry Ma, Luyang Fang, Haoran Lu, Huimin Cheng, et al.
stat.ML · cs.LG
Quick verdict
调查将 LLM 复杂问题解决重构为序列估计与控制；条理清晰且坦诚，但其诊断假设仍未经测试。
Summary
本综述将 LLM 的复杂问题解决（CPS）重构为对潜在解状态的序列估计与决策问题，而非单纯追求更强的推理能力。控制器维持对未观察到的解轨迹的信念——通过动作条件化的贝叶斯滤波器（公式 2）进行形式化——并在 commit、verify、branch、roll back 或 abstain 之间进行选择，以最小化预期损失。文献围绕五个控制组件进行了重组：显式状态表示、转换结构化、验证与约束执行、搜索与回滚以及不确定性管理，此外还有两个交叉维度（多智能体分布和训练时内化）。由此，作者推导出了名为“问题-控制契合度”（problem–control fit）的诊断假设：当干预措施针对观察到的失败所涉及的误差或不确定性组件时最为有效，其中误差被分解为系统性、随机性和不可约部分，不确定性被分解为认知性和偶然性部分。表 3 重读了已发表的数字——自洽性（self-consistency）将 PaLM-540B 在 GSM8K 上的准确率从 56.5 提升至 74.4，在 GSM-HARD 上程序辅助提示（program-aided prompting）达到 61.2，而思维链（chain-of-thought）为 23.1——同时承认此类比较无法从因果关系上识别这些组件。主要局限在于：该诊断明确是一个假设，未经任何受控的等预算实验测试，且其中的数学部分是对文本推理状态的公认理想化。
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Persistent LLM CPS failures — early-step error amplification, prompt brittleness, inability to revise commitments — are better explained as process-control failures than as deficits of knowledge or expressive capacity (§1, §2.1–2.2).
- CPS can be formalized as sequential estimation and decision over a latent solution state via an action-conditioned Bayes filter with an expected-loss decision layer (§2.3, Eq. 2–3).
- Existing methods implement one or more of five control components (explicit state, transition structuring, validation/constraints, search/rollback, uncertainty management) within a single estimation-and-decision loop, with multi-agent distribution and training-time internalization as cross-cutting dimensions (§3, §4, Table 2).
- Interventions are most effective when matched to the error or uncertainty component implicated by the observed failure ('problem–control fit'); mismatched effort — e.g., added sampling against a shared systematic error — leaves the failure unchanged (§2.4, Table 1, Fig. 3).
- Outcome-only metrics collapse the error structure and cannot localize failure to a loop component; evaluation should additionally probe state sufficiency, transition validity, calibration, and recovery (§5).
Strengths
Coherent unifying formalization
A single estimation-and-decision loop (Eq. 2 Bayes filter + Eq. 3 decision layer) organizes five method components (§3.1–3.5) and two cross-cutting dimensions (§4), letting methods, evaluation (§5), and open problems (§6) share one set of axes rather than a technique catalogue.
Explicit falsifiability
§2.5 states a concrete, testable prediction — at fixed budget, effort on the failure-implicated component should beat equal effort elsewhere — and names what would count as evidence against the diagnostic (uniform benefit of added reasoning). Rare in surveys.
Unusually honest treatment of quoted numbers
Engagement with the strongest counter-view
Multi-component method mapping
Weaknesses
Diagnostic hypothesis is untested
No operationalization of component diagnosis
Survey methodology unreported
Decorative formalism risk
Thin quantitative engagement
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a conceptual survey (arXiv 2609.20973, stat.ML; 38 authors across UGA Statistics/Computing, Mount Sinai, CMU, Harvard, Boston University, MD Anderson, Stanford, UT Arlington, Arizona) that recasts complex problem solving (CPS) in LLMs as sequential estimation and decision over a latent solution state. A controller holds a belief over an unobserved solution trajectory (action-conditioned Bayes filter, Eq. 2) and a decision layer commits, verifies, branches, rolls back, or abstains to minimize expected loss (Eq. 3). The literature is re-sorted into five components — explicit state representation (§3.1), transition structuring (§3.2), search and rollback (§3.3), validation and constraint enforcement (§3.4), uncertainty management (§3.5) — plus two cross-cutting dimensions (§4: multi-agent distribution; training-time internalization via SFT, RLHF/DPO/GRPO). The headline deliverable is a diagnostic hypothesis, 'problem–control fit' (§2.4, Table 1): decompose trajectory error into systematic, stochastic, and irreducible parts (Eq. 4) and uncertainty into epistemic and aleatoric parts (Eq. 6), then match interventions to the component implicated by the observed failure; its failure mode is 'control mismatch' (Fig. 3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'control mismatch' in this paper?
It is the failure that occurs when an intervention targets the wrong component of the problem-solving loop — for example, extra sampling reduces stochastic (sample-dependent) errors but cannot fix a systematic error shared across samples, such as a recurring arithmetic slip, so accuracy stays flat despite more compute.
Is this survey empirical? Does it include new experiments?
No. It proposes a conceptual/statistical framework and reinterprets published results (e.g., self-consistency on GSM8K, program-aided prompting on GSM-HARD) through it. It reports no new experiments, datasets, or code, and its authors explicitly call the diagnostic a hypothesis inviting future testing.
What are the five control components the survey proposes?
Explicit state representation, transition structuring, validation and constraint enforcement, search and rollback, and uncertainty management — plus two cross-cutting dimensions: distributing control across multiple agents and internalizing control via training (SFT, RLHF/DPO/GRPO-style methods).
Does the paper provide code or datasets?
No. No code, data, or artifacts accompany the survey; the mathematical objects (belief, transition kernel, uncertainty summaries) are analytical scaffolds for reading the literature, not implemented systems.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extracted text is truncated: §3.1, portions of §3.4–3.5, the contents of Table 1 and Table 5, and Appendix A were not visible, so coverage and some section-specific judgments could not be audited.
- Figures 1–4 and 7 are known only from their captions; visual content was not analyzable.
- The task metadata lists 8 authors while the byline shows 38; this review follows the byline.
- All quantitative evidence is quoted from third-party papers and was not independently verified against the originals.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.