Making Prospective Memory SLM-Shaped: Typed Intention Stores for Small-Model Agents
cs.AI
Quick verdict
Training-free typed intention store lifts PM-Bench Set-F1 to 82.9% on DeepSeek-Chat; big SLM gains, but one benchmark, no ablations, tiny samples.
Summary
The paper proposes the Prospective Intention Store (PIS), a training-free agentic scaffold for prospective memory (PM): executing deferred intentions when a future cue fires. PIS types each intention as a triple (trigger φ, action α, status σ), keeps lifecycle logic (Form/Revise/Filter/Decide operators, eligibility board, done-marking) in deterministic code, and leaves only scoped semantic work — slot filling, patch decisions, cue judging, due-set selection — to the model. On the PM-Bench synthetic week, PIS reaches 82.9% Set-F1 with DeepSeek-Chat, above the published best scaffold (65.1%), and lifts Gemma-E2B from 4.2% without a store (≤6.6% under six retrospective memory baselines) to 66.2%; Qwen3.5-4B+PIS reaches 70.1% (best retrospective row 54.4%) and Qwen3-8B 57.2%. Secondary metrics (update miss, cross-day miss, false alarms per step) and a cost analysis (Table 3) are also reported. Main limitations: evaluation on a single synthetic benchmark; no operator ablation isolating which component drives the gain (explicitly deferred to future work); secondary miss metrics show visible small-sample granularity (multiples of 1/7 and 1/9) with no variance or significance testing; retrospective baselines are pattern adapters rather than full upstream systems, and no untyped-store control exists; update miss on Gemma remains high (77.8%); code is promised only upon acceptance.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Prospective memory in agents is schema-constrained state tracking rather than open-ended reasoning, so small models can execute it when the action space is typed (§1, §3.1).
- PIS, a training-free typed intention store with Form–Revise–Filter–Decide operators, sets the best published PM-Bench result: 82.9% Set-F1 with DeepSeek-Chat vs the prior best scaffold of 65.1% (§4.2).
- The same frozen scaffold lifts small models above the published large-model scaffold: Gemma-E2B from 4.2%/≤6.6% to 66.2%, Qwen3.5-4B to 70.1% (§4.2–4.3).
- Retrospective memory systems fail to beat even the no-store single baseline on PM-Bench (46.5–58.3% vs 67.7% on DeepSeek-Chat), indicating the gap is architectural, not just scale (§4.2).
Strengths
Clean problem decomposition with a typed object
§3 casts PM as schema-constrained state tracking: intentions typed as (φ, α, σ) with deterministic lifecycle code (Form/Revise/Filter/Decide, Algorithm 1) and only scoped grounding on the model — a well-motivated factorization consistent with the TinyAgent/Octopus evidence the authors cite.
Large, consistent gains on every backbone
Tables 1–2: PIS is best on all four backbones — DeepSeek-Chat 82.9% vs 67.7% no-store and the published best 65.1%; Gemma-E2B 66.2% vs ≤6.6% across all retrospective rows (A-Mem 0.0%); Qwen3.5-4B 70.1% (best retrospective 54.4%); Qwen3-8B 57.2%.
Training-free attribution is honest and informative
PM-specific failure metrics beyond aggregate F1
Genuine novelty in the SLM×PM intersection
Weaknesses
No operator ablation despite promising one
Tiny-sample secondary metrics with no statistics
Single benchmark and uneven baseline depth
Residual Revise failure on the smallest backbone is under-foregrounded
Editorial inconsistency in baseline count
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses prospective memory (PM) for language agents — executing a deferred intention at the right future cue while other work continues. Against PM-Bench's best published scaffold of 65.1% Set-F1 (Liu and Gabriel, 2026), the authors claim: (1) PM is schema-constrained state tracking rather than open-ended reasoning; (2) a training-free typed store, PIS, attains 82.9% Set-F1 with DeepSeek-Chat — a new SOTA on this benchmark; (3) the same frozen scaffold lets small models surpass the published large-model scaffold (Gemma-E2B 66.2%, Qwen3.5-4B 70.1%, vs at most 54.4% for the retrospective baselines tested). The framing — first SLM study of PM — is plausible given the cited landscape, though nearly all baselines are 2025–2026 arXiv preprints in a fast-moving area.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is prospective memory in LLM agents?
It is the ability to carry out a deferred intention — e.g., "remind me when the portal opens" or "take the medicine at 21:00" — at the right future cue or time while other work continues. It differs from retrospective memory (retrieving past notes). On the PM-Bench benchmark, the best previously published scaffold reportedly reached only 65.1% Set-F1 before this paper.
What is the Prospective Intention Store (PIS)?
A training-free scaffold that stores deferred intentions as typed records (trigger condition, action, status) and runs lifecycle logic — Form, Revise, Filter, Decide — in deterministic code, using the LLM/SLM only for narrow tasks like slot filling, patch decisions, cue judgment, and due-set selection. No fine-tuning or distillation of the model is required.
How much does PIS improve small models on PM-Bench?
Per the paper's tables: Gemma-E2B improves from 4.2% Set-F1 without a store (at most 6.6% with the retrospective memory baselines tested) to 66.2% with PIS; Qwen3.5-4B reaches 70.1% and Qwen3-8B 57.2%, with DeepSeek-Chat at 82.9% — all above the 65.1% published large-model scaffold, on this benchmark.
Is code or data available for this paper?
Not yet at submission time: the paper states the method and code will be released "upon acceptance". The underlying PM-Bench benchmark is publicly available from the prior benchmark paper (Liu and Gabriel, 2026), so independent replication of the scaffold would require reimplementing it from Algorithm 1.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's embedded diagram text extracts out of order and partially garbled; the operator wiring was reconstructed from the figure caption and §3 prose rather than the diagram itself.
- No prompt templates, judge definitions, or hyperparameter details appear anywhere in the provided text, so implementation-level fidelity cannot be assessed; this is an absence in the paper text as extracted, not a truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.