SurveyAgent-HKA: A multi-agent framework for scientific survey generation with LLMs and human knowledge augmentation
Tong Bao, Mir Tafseer Nayeem, Yi Zhao, Davood Rafiei, Chengzhi Zhang
cs.CL · cs.DL · cs.IR
Quick verdict
Multi-agent survey generation with human/peer-review knowledge tops baselines on citation and structure, but absolute citation F1 (~17–19%) remains low.
Summary
SurveyAgent-HKA is a multi-agent LLM framework for end-to-end scientific survey generation that injects human-derived knowledge at every stage. Papers are retrieved from Semantic Scholar plus arXiv/PubMed, expanded by co-citation analysis, and clustered (K-Means, K=6) to build an initial outline refined with few-shot outlines from published surveys. A second, section-level retrieval pass is re-ranked using LLM-judged relevance plus citation, author-h-index, and venue-h-index signals; full-text PDFs are parsed (MinerU2) into knowledge cards, sections are drafted in parallel, and the draft is revised using a checklist distilled from peer reviews of 480 published surveys. On 20 CS and 10 medical topics it reports the best citation F1 (16.94/19.39), heading match (H1 38.43/30.22), and key-point recall (53.85/57.07) versus AutoSurvey, SurveyX, SurveyForge, and DeepResearch, ranking second to DeepResearch only on ROUGE-L/overlap; Wilcoxon tests confirm most gains, with some non-significant comparisons versus SurveyX and SurveyGen. Ablations credit peer-review-guided revision with the largest content gain (KPR −10.25 CS), and a resource-controlled variant keeps most advantages. Limitations: absolute citation fidelity is low (~17–19% F1; ~80% of human references missed), only 30 topics are evaluated, medicine few-shot outlines share the SurveyGen evaluation corpus (leakage risk), the text contradicts Table 7's Relevance column, and code is promised but not yet released.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A multi-agent pipeline augmented with human-derived knowledge (published survey outlines + real peer-review checklists) outperforms mainstream baselines in citation quality, structural consistency, and content quality on both CS and medicine survey generation (Table 7).
- SurveyAgent-HKA is the first framework to use real review feedback from published surveys to guide automatic survey revision.
- Multi-source retrieval with co-citation expansion and multi-signal re-ranking materially improves citation F1 (Table 14: −3.10 F1 CS without expansion, −4.83 CS without re-ranking).
- The gains are not merely a resource effect: a single-source, abstract-only controlled variant still beats most baselines (Table 22, e.g., CS KPR 50.67 vs SurveyForge 48.19).
- The system is practical: ~24 minutes and roughly $1.25–$1.64 per survey (Table 13).
Strengths
Genuine human-knowledge innovation with measurable contribution
Real peer-review comments from 480 published surveys (Table 5: OpenReview ICLR/EMNLP, F1000, PLOS One) are distilled into domain checklists guiding revision; the ablation shows the largest single content-quality effect (KPR 53.85→43.60 CS, 57.07→48.59 Med, Table 16).
Proper resource-controlled baseline
Appendix A.6/Table 22 reruns the system restricted to arXiv-only (CS) / S2ORC-only (Med) abstracts: controlled KPR 50.67 (CS) still beats SurveyForge's 48.19, showing the human-knowledge components add value beyond richer retrieval, though controlled citation F1 (13.61) falls just below SurveyForge (13.97).
Above-average statistical and judge hygiene
Cross-domain, template-aware design
Weaknesses
Low absolute citation fidelity undercuts the 'well-grounded survey' framing
Possible train–test leakage in structural evaluation
Peer-review ablation is confounded
Internal numeric inconsistencies
Small evaluation scale with in-sample parameter tuning
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SurveyAgent-HKA (arXiv:2609.05938, 38 pp., formatted for an Elsevier journal) targets end-to-end automatic scientific survey generation. Three claims: (1) a multi-agent pipeline that augments LLM generation with human-derived knowledge — outlines of published surveys and real peer-review comments — beats mainstream baselines (Naive-RAG, AutoSurvey, SurveyX, SurveyGen, SurveyForge, DeepResearch) on citation quality, structural consistency, and content quality; (2) it is the first system to use real review feedback from published surveys to guide automatic revision; (3) it is practical (~24 min, $1.25–$1.64 per survey, Table 13). Evaluation covers 20 CS topics (AutoSurvey's set) and 10 medicine topics (SurveyGen's set), with human-written surveys as ground truth.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SurveyAgent-HKA?
A multi-agent LLM framework that automatically writes scientific survey papers. It retrieves literature from multiple databases (Semantic Scholar, arXiv, PubMed), expands candidates via co-citation analysis, builds and refines an outline using outlines of published human-written surveys, re-ranks papers by relevance and impact signals, extracts knowledge from full-text PDFs, drafts sections in parallel, and finally revises the draft using a checklist distilled from real peer-review comments on ~480 published surveys.
How well does SurveyAgent-HKA perform compared with other survey-generation systems?
On 20 computer-science and 10 medical survey topics it reports the best citation F1 (16.94% CS / 19.39% medicine), best heading match with human outlines, and best key-point recall (53.85% / 57.07%) against baselines including AutoSurvey, SurveyX, SurveyForge, SurveyGen, Naive-RAG, and DeepResearch, at about 24 minutes and $1.25–$1.64 per survey. Note that absolute reference coverage is still modest: only ~17–19% F1 against human reference lists, and it trails DeepResearch on ROUGE-L/content overlap.
Is the code or dataset for SurveyAgent-HKA available?
The preprint states that code and data will be released at github.com/tongbao96/code-for-SurveyAgent-HKA after review; as of this preprint version they are not yet publicly available. Prompts are said to be included in supplementary material, and the system depends on commercial LLM APIs (GPT-4o-mini, Claude-3.5-Haiku, GPT-4.1-mini) plus Semantic Scholar/OpenAlex/arXiv/PubMed access.
What is new about using peer reviews in automatic survey writing?
According to the authors, it is the first framework to use real reviewer comments on published survey papers (from OpenReview, F1000 Research, and PLOS One) to guide automatic revision: common reviewer concerns are aggregated into a domain-specific checklist that an LLM reviewer agent applies to the draft. The ablation reports the largest content-quality drop (key-point recall −10.25 in CS) when this component is removed, though the ablation does not separate the human checklist from the effect of an extra revision pass.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Sampled excerpt: mid-paper text marked [...sampling gap...] was skipped, including the aggregate pairwise human-evaluation results table (region between §5.3 and §5.6); win/tie/loss percentages could not be verified, only the later validation numbers (Table 12).
- Table 6 (parameter summary) is only partially visible in the extraction; sensitivity ranges are cited but the full grid results were not verifiable.
- Figures (e.g., Fig. 3–7, 10–12) appear only as captions in the extracted text; plotted values could not be checked, so distribution and user-study claims rely on the prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.