RAFT: A Stateful Retrieval-Augmented Framework for Troubleshooting Agents
Mingxuan Zhang, Xiaowen Wang, Anupma Sharan, Zhengyi Chen, Chenyu Diana Zhang, Shanshan Yang, Chittibabu Pacharu
cs.AI
Quick verdict
RAFT's stateful entry-level retrieval beats RAG/GraphRAG baselines; credible stats, but gains could reflect LLM distillation and main evidence is synthetic.
Summary
RAFT is a retrieval-augmented framework for enterprise troubleshooting agents that treats each closed support case as a directed chain of LLM-extracted timeline entries rather than a static document. Retrieval scores individual entries (hybrid semantic plus BM25 via reciprocal rank fusion), promotes top entries to their parent cases under a token budget, and returns each matched case's full trajectory anchored at the matched state; an optional case-level graph links cases by root cause and resolution. Because public multi-stage support data is scarce, the authors build an 826-case synthetic benchmark from Microsoft Learn Windows Server documentation plus a 30-group Apache Jira duplicate evaluation set. On the synthetic benchmark RAFT reaches 84.2-88.8% Case Hit versus 67.3-76.9% for vanilla RAG and lower for HippoRAG2 and Fast-GraphRAG, with clustered-bootstrap 95% CIs supporting significant Case Hit gains at all three progress points; coverage gains are smaller, and late-stage coverage is not significant. On Jira, RAFT leads vanilla RAG by +10.5 to +17.3 points, presented as directional evidence only. Main limitations: the primary benchmark is synthetic and LLM-generated, the Jira set is small, coverage metrics rely on an unvalidated LLM judge, and no baseline retrieves the same distilled case representations without the stateful structure, so the contribution of timeline modeling alone remains unisolated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Entry-level retrieval over LLM-extracted case timelines yields higher Case Hit than vanilla RAG, HippoRAG2, and Fast-GraphRAG at every progress stage (Table 2), with clustered-bootstrap-significant gains over the strongest baseline, vanilla RAG (Table 5).
- The position of the triggering match within a case deepens with query progress (9.1% to 54.0% entry depth, Table 3), supporting the state-aware matching hypothesis.
- The advantage transfers directionally to real Apache Jira duplicate data, with +10.5 to +17.3 pp Case Hit over vanilla RAG (Table 4).
- The case-level graph is an optional, marginal addition (overall Case Hit changes of ±0.3 pp, Appendix C.5), not the source of RAFT's retrieval gains.
- Letting the agent compose its own queries and filters further improves Case Hit to 0.910-0.919 (Table 9).
Strengths
Evaluation protocol directly targets state-awareness
Prefix queries at 0/30/60% of case turns (Section 5.3) plus Table 3's matched-entry depth moving 9.1% -> 20.0% -> 54.0% provide direct, interpretable evidence that retrieval tracks the investigation stage rather than re-matching initial symptoms.
Above-average statistical honesty
Paired, issue-group-clustered bootstrap 95% CIs (Table 5); late-stage coverage claims explicitly declined where CIs cross zero ([−0.07,+3.89] and [−1.18,+2.32]); Jira framed as directional; graph contribution honestly reported as marginal (overall Case Hit +0.22/−0.28/−0.06 pp, Appendix C.5) despite being a listed contribution.
Real artifacts released
Useful secondary analyses
Weaknesses
Missing control isolates neither distillation nor structure
Synthetic ground truth aligns with RAFT's own extracted fields
Thin real-data evidence
Coverage gains are small and judge-dependent
Motivating features untested, indexing costs unquantified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
RAFT (Microsoft; footnoted as accepted to EMNLP 2026 Industry Track) targets retrieval for enterprise troubleshooting: given an active, partially investigated case, surface similar closed cases and actionable guidance. Claimed contributions are (1) a stateful representation — each closed case distilled by an LLM worker-reviewer workflow into a directed chain of timeline entries (symptom, hypotheses, root cause, resolution), retrieved at entry level with the parent-case trajectory returned anchored at the matched state; (2) an optional case-level graph linked by root-cause/resolution similarity (RRF of semantic + BM25); (3) a public 826-case synthetic benchmark from Microsoft Learn Windows Server documentation and a 30-group Apache Jira duplicate set. The central claim: entry-level stateful retrieval beats vanilla RAG and two GraphRAG baselines (HippoRAG2, Fast-GraphRAG) at 0/30/60% case progress, with statistically significant Case Hit gains over the strongest baseline. The work deliberately scopes to the retrieval layer, not end-to-end agent outcomes.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RAFT and how is it different from standard RAG?
RAFT (Retrieval-Augmented Framework for Troubleshooting Agents) is a Microsoft retrieval framework for support agents. Standard RAG treats each closed support ticket as a static document; RAFT first distills each case with an LLM into a chain of timeline entries (symptom, hypotheses, root cause, resolution), retrieves at the entry level so an in-progress case matches the corresponding intermediate state of past cases, and returns the matched case's full trajectory anchored at that state.
How much better is RAFT than vanilla RAG and GraphRAG?
On the authors' 826-case synthetic benchmark, RAFT's Case Hit is 84.2-88.8% across three stages of case progress versus 67.3-76.9% for vanilla RAG, 65.0-71.1% for HippoRAG2 and 42.1-58.3% for Fast-GraphRAG. Clustered bootstrap 95% CIs confirm significant gains over vanilla RAG; on 30 real Apache Jira duplicate groups the lead is +10.5 to +17.3 points, reported as directional evidence only.
Is the code and data available?
Yes. The paper states that the benchmark, implementation, and the Apache Jira evaluation set are released at github.com/microsoft/RAFT. Note that the pipeline depends on commercial APIs (GPT-5.x models for indexing and evaluation, text-embedding-3-large for embeddings), so exact replication incurs API costs.
Does the case-level graph in RAFT help much?
Not much in their experiments. Appendix C.5 shows budget-matched graph expansion changes overall Case Hit by only +0.22, −0.28 and −0.06 percentage points at 0/30/60% progress; the authors present the graph as an optional evidence-diversification mechanism, not the source of RAFT's gains.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 appear only as caption text in the extraction; visual layout could not be verified.
- Appendix prompt texts for extraction and querying are referenced but not included in the extracted text.
- The extraction appears complete through Appendix D, but any supplementary material beyond the embedded text was unavailable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.