Learning from Failures: Heterogeneous Graph Memory for Small Language Model Tool-Using Agents
Jiaxing Li, Lei Song, Rui Dong, Youyong Kong
cs.LG
Quick verdict
FRESH organizes agent successes/failures as heterogeneous-graph memory with retrieval guidance plus a risky-write gate; gains look real but unproven: no significance tests, and the gate ablation confounds.
Summary
FRESH is an external-memory framework for frozen small/mid-size LLM tool-using agents. Offline, successful and failed trajectories are mined into a typed heterogeneous graph spanning tasks, tools, actions, observations, errors, preconditions, and repairs; a trained heterogeneous GNN retriever (pairwise BPR ranking over feedback-derived positives/negatives) reranks sparsely recalled subgraphs. At test time, retrieved nodes are compressed into Do/Avoid/Check/Repair guidance, and a policy/precondition gate validates risky write actions before execution. On τ-bench (20 Airline, 40 Retail held-out tasks, four runs each) with Llama-3.2-1B, Qwen3-8B, and Gemma-4-26B-A4B, relative improvements over the no-memory baseline are +10.8%, +38.9%, and +53.2% average; AppWorld PR is best on both splits (0.268 Test-N, 0.146 Test-C). Ablations support each component, but removing the policy gate sharply lowers Airline performance, suggesting policy-domain gains may stem largely from the gate rather than graph memory. The experience miner and gate implementations are underspecified, and no statistical significance tests or confidence intervals are reported despite small test sets, so some relative gains may be noise. No code release is mentioned, limiting reproducibility.
Review complete
Assessment at a glance
5
risk flags identified
● high 3
● medium 2
● low 0
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Organizing both successes and failures into a typed heterogeneous experience graph with relation-aware retrieval improves tool-use reliability for frozen small/mid-size language models.
- A GNN retriever trained with feedback-derived BPR supervision ranks genuinely useful evidence (verified actions, preconditions, repairs) above failed actions and policy violations.
- A lightweight policy/precondition gate that validates risky writes before execution contributes substantially to performance in policy-constrained domains such as tau-bench Airline.
- FRESH consistently improves results over no-memory, prompting, retrieval, trajectory, and experience-memory baselines across three backbones (+10.8% avg for Gemma-4-26B-A4B, +38.9% for Llama-3.2-1B, +53.2% for Qwen3-8B on tau-bench; best AppWorld PR of 0.268/0.146).
Strengths
Well-motivated problem formulation
Action-level credit assignment inside failed trajectories (Appendix A: GetBooking kept as verified, premature CancelBooking as failure) addresses a real gap in flat/trajectory memory baselines that retrieve failed actions as exemplars.
Leakage-aware evaluation protocol
Both the experience graph and HG-Retriever are frozen before held-out evaluation (Algorithm 1, line 10), and all baselines share the same trajectory pool, budgets, decoding, and prompts (Appendix, 'Details on Baselines').
Breadth and consistency of results
Informative ablation design
Weaknesses
Zero statistical rigor
Gate-vs-memory confound
Core components underspecified
Numerous bookkeeping errors
Ambiguous metric definition
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FRESH (Failure-aware Retrieval over Experience-Structured Heterogeneous graphs) targets structural tool-use errors of small/mid-size language models (missing reads, premature writes, repeated calls, precondition violations). It stores successes *and* failures as a typed heterogeneous graph (tasks, tools, actions, observations, errors, preconditions, repairs; relations `requires`/`guards`/`fails_because`/`fixed_by`), retrieves subgraphs via sparse recall + a trained HGNN reranker, compresses them into Do/Avoid/Check/Repair sheets, and interposes a policy/precondition gate (Allow/Revise/Block/Transfer, Eq. 12) before risky writes. Four claims: failure-aware graph memory helps frozen SLMs; feedback-supervised BPR ranking ranks useful evidence above failures; the gate matters; and consistent gains over seven baselines on τ-bench and AppWorld.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FRESH?
FRESH (Failure-aware Retrieval over Experience-Structured Heterogeneous graphs) is a framework that lets a frozen small or mid-size language-model agent reuse past experience. Instead of flat notes or whole trajectories, it stores successes and failures in a heterogeneous graph that records what succeeded, what failed, why it failed, how it was repaired, and which preconditions an action requires. A trained graph retriever turns that graph into compact Do/Avoid/Check/Repair guidance for each new task, and a safety gate checks risky state-changing actions before they execute.
Does FRESH fine-tune the language model?
No. The executor model stays frozen throughout. Only the external experience graph and a trained graph-neural-network retriever are learned; test-time improvements come from retrieved guidance plus a policy/precondition gate, not from parameter updates.
How does the policy/precondition gate work?
Before a state-changing or policy-sensitive action runs, the gate checks the action's required preconditions against live tool observations and domain policies and outputs one of four decisions: Allow, Revise (fix a missing read or argument), Block (unsafe), or Transfer (escalate to a human/system). The paper does not describe its concrete implementation, which is one of this review's main criticisms.
How large are FRESH's improvements, and are they reliable?
On tau-bench with 20 Airline and 40 Retail held-out tasks (four runs each), the paper reports average relative gains of +10.8% (Gemma-4-26B-A4B), +38.9% (Llama-3.2-1B), and +53.2% (Qwen3-8B) over a no-memory agent, plus the best AppWorld pass rates (0.268 Test-N, 0.146 Test-C). No significance tests or confidence intervals are reported, and on such small test sets some relative gains could be noise; a missing ablation also means the gate itself may account for much of the policy-domain gain.
Can I reproduce FRESH?
Partially. The benchmarks are public (tau-bench, AppWorld), but the paper leaves the experience miner and the policy gate unspecified and mentions no code or prompt release, so an independent exact reproduction is currently not possible without author artifacts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are only available as embedded text labels, so radar/bubble plots (Figures 4-5, 8) could not be visually assessed; Figure 3's ablation values were read from axis labels.
- The main text cites an Appendix B for implementation details that is absent from the extracted text; specification criticisms apply to the reviewed version only.
- The full text appears otherwise complete (including appendix prompts and Tables 4-7), but extraction artifacts cannot be fully ruled out for the GRACE row and caption mismatches.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.