DiagEvo: Diagnosis-Guided Self-Evolution via Hierarchical Error Memory
Xincheng Wei, Yifan Ding, Yoshua Li, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Wenjian Ding, Yao Zhang
cs.AI
Quick verdict
Claims solver failure history can guide label-free self-play; broad ablations convince, but 1.1–1.3 pt gains over DARC lack statistical testing.
Summary
DiagEvo is a self-play framework that steers language-model self-evolution using the solver's own failure history instead of external task resources. A lightweight diagnostician LLM contrasts failed solver trajectories with pseudo-label-consistent ones, extracts transferable error causes, and stores them in a hierarchical memory that groups causes under skill nodes and tracks Active/Mastered states via solver self-consistency. The challenger mixes cause-targeted generation, weighted by recurrence counts, with free exploration under a frequency-driven schedule, and can stitch an Active cause with a Mastered sibling into composite questions. Double-confidence filtering keeps training questions only when the majority answer's vote share lies in [0.25, 0.75] and is at least 1.6 times the runner-up's. Across Qwen3-4B/8B and OctoThinker-8B on nine benchmarks, the default 4B diagnostician gives the best overall scores: 57.4% on Qwen3-8B (72.3% math average, +4.5 over R-Zero, +1.1 over DARC) and +1.3 over DARC on both other solvers. Ablations credit mixed generation, memory-state updates, and double-confidence filtering; oracle agreement of pseudo-labels rises from 65% to 75%. Limitations: pseudo-labels rest on self-consistency, so shared errors can persist; the round-5 checkpoint was chosen post hoc after rounds 6–7 declined; margins over DARC are small and untested statistically; some baselines are imported from the DARC paper; no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Curriculum direction for self-play can be derived endogenously from the solver's own failure history, removing dependence on external task resources such as human examples, corpora, or difficulty labels.
- With the default 4B diagnostician, DiagEvo achieves the best overall score among compared methods on all three solvers (e.g., 57.4% vs DARC's 56.3% on Qwen3-8B).
- Hierarchical error-cause memory with Active/Mastered states, mixed generation, and double-confidence filtering each contribute measurably (math average 72.3 vs 69.4–70.9 in ablations).
- Diagnosis and memory maintenance add only 5.7% wall-clock overhead over an R-Zero-style loop.
Strengths
Broad, well-controlled evaluation
Table 1 spans 3 solvers × 9 benchmarks × 3 diagnostician scales with mean±SD over three runs; diagnostician scale adds only ~1.0–1.2 math points (e.g., 68.3→69.3 on Qwen3-4B from 4B to 235B-A22B), supporting the claim that gains are not bought with a bigger judge.
Baseline provenance is validated, not assumed
Appendix F (Table 7) reproduces base models within 0.1 points and R-Zero within 0.5 points of DARC's reported aggregates, with per-benchmark deviations <1.0, giving the mixed-provenance Table 1 a defensible footing.
Ablations isolate every component
Unusual mechanism transparency
Cross-domain transfer is demonstrated
Weaknesses
No statistical testing behind headline margins
Post-hoc checkpoint selection
Self-referential quality signals
Four of six baselines are second-hand
No code or data release stated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
DiagEvo (arXiv 2609.00768, cs.AI) is a method paper on label-free self-play for LLM self-evolution. The central claim: the curriculum signal needed to keep generated questions at the solver's competence boundary can be extracted endogenously from the solver's failure history, removing guided methods' (R-Few, SPICE, DARC) dependence on external task resources. Section 1 states three supporting claims: (1) recurring error causes diagnosed from failed trajectories form a dynamic curriculum signal; (2) a hierarchical memory with Active/Mastered states plus double-confidence filtering improves over an R-Zero-style loop; (3) with a 4B diagnostician, DiagEvo achieves the best overall score among compared methods on all three solvers.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DiagEvo and how does it work?
DiagEvo is a self-play training framework for language models. A 'challenger' model generates math questions, a 'solver' model answers them, and a small frozen 'diagnostician' model compares the solver's failed attempts against pseudo-label-consistent ones to extract transferable error causes. These are stored in a hierarchical memory (grouped under skill nodes, marked Active or Mastered by self-consistency) and steer the next round of question generation toward unresolved weaknesses, while a double-confidence vote filter removes unreliable question–answer pairs before training.
Does DiagEvo need external data or human labels?
No. Its training loop uses only information produced during self-play: the solver's own trajectories and majority-vote pseudo-labels. The diagnostician and embedding models are frozen general-purpose models. This is its claimed distinction from guided methods like R-Few (human examples), SPICE (document corpus), and DARC (difficulty labels plus a privileged teacher).
How much better is DiagEvo than R-Zero or DARC?
On Qwen3-8B with the default 4B diagnostician: 72.3% average over five math benchmarks (+4.5 over R-Zero) and 38.8% over four general-reasoning benchmarks (+2.6 over R-Zero), for 57.4% overall (+1.1 over DARC, which uses external resources). On Qwen3-4B and OctoThinker-8B it leads DARC by 1.3 points overall. Gains are ablation-backed but not statistically tested.
What are DiagEvo's main limitations?
Pseudo-labels rest on solver self-consistency, so a shared systematic error can be reinforced (about 75% of retained labels agree with a GPT-5 oracle at best); the round-5 checkpoint was identified after observing decline in rounds 6–7; the curriculum is math-only; baseline scores are partly imported from the DARC paper; and no code release is stated.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal data (curves in Figures 1 and 3–5, case-study panels in Figure 6) is only partially recoverable from captions and body text; all quantitative judgments here rely on numbers reported in §4.4 and Appendices G–H.
- Inline math is garbled by PDF extraction in places (e.g., Eq. 1 renders as '1 1 + zt/k'); meaning was reconstructed from surrounding prose and is unambiguous in context.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.