Beyond Context Windows: Persistent Discovery Context for Data-Centric Agents
cs.AI · cs.IR
Quick verdict
Memory-augmented retrieval shows solid gains on three tiny registries, but task-family design partly bakes in the benefit; interference honestly reported.
Summary
The paper proposes persistent discovery context: a memory layer for data-centric agents that stores prior intent-to-object mappings from successful discovery and reuses them to boost object retrieval. Given a query, registry retrieval scores candidate tables and attributes from natural-language metadata, and the top-K similar discovery memories add a bonus (weight α) to objects appearing in those memories. On three small public registries (Berka finance, 6 tables; NYPD collisions, 3; Northwind, 8) with 15 task families and 125 held-out tasks, memory augmentation raises F1@5 over registry-only search in every domain (finance 0.396→0.499 with TF-IDF; +0.051 to +0.167 with MiniLM embeddings), and memory-only retrieval can win in lexically sparse domains. Robustness checks include a neural retriever, BM25, a naive-injection baseline, an α ablation (peak 0.25), leave-one-out seed rotation, and Claude-Haiku-generated memories, which still beat registry-only search in two of three domains. The authors candidly report a reproducible interference failure: wrong-family memories drop F1@5 below the no-memory baseline (Northwind 0.000 vs 0.444). Main limitations: the family structure makes held-out tasks share gold object sets with seeds by construction, so part of the gain is designed-in; registries are tiny (3–8 tables, five memories per domain); tasks and annotations are author-constructed without inter-annotator agreement; no significance tests are reported; and no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Persisting and reusing intent-to-object discovery mappings consistently improves F1@5 over registry-only retrieval across three structured domains and both lexical and neural retrievers.
- In lexically sparse domains, memory-only retrieval can outperform metadata-based retrieval (NYC collisions, TF-IDF: 0.412 vs 0.299).
- Discovery memory remains effective when initialized with LLM-generated memories, improving over registry-only search in two of three domains (Northwind being the exception).
- A reproducible interference failure exists: retrieving a semantically similar but incorrect-family memory degrades retrieval below the no-memory baseline (Table 2).
Strengths
Unusually candid negative results
The interference failure is quantified, not just mentioned: Table 2 shows wrong-family memories dropping F1@5 below the S1 baseline in all three domains (Finance 0.222 vs 0.444; NYC 0.200 vs 0.400; Northwind 0.000 vs 0.444). Appendix D names three families with negative S1→S3 deltas (regional economics −0.102, customer demographics −0.077, product catalog −0.074), and Appendix C discloses that Haiku-generated memories fail in Northwind (0.435 vs S1 0.446).
Breadth of robustness checks rare for a workshop paper
Beyond Table 1: neural retriever (MiniLM) variants, BM25 (Appendix H, Table 6), a naive-injection baseline S4 (Appendix F, Table 4 — S3 beats S4 on MRR and F1@5 everywhere, e.g., Finance 0.816 vs 0.682 MRR), an α ablation (Appendix E, peak at 0.25 for both retrievers), and leave-one-out seed rotation (Appendix G, Table 5, 56/51/36 trials) with consistent orderings.
Simple, fully specified method on public data
Clear positioning in the agent-memory landscape
Weaknesses
Benchmark design partially guarantees the headline result
Scale is far from realistic deployments
No statistical inference anywhere
Author-constructed ground truth with circularity risk
No artifact release; headline uses a suboptimal setting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a workshop paper (Context Beyond the Window @ COLM 2026) proposing *persistent discovery context*: a registry-level memory that stores past intent→object mappings and reuses them via a linear score boost (Eq. 1: registry score r(q,o) plus α·Σ sim(q,m.intent)·1[o∈m.objects]). Four claims are staked: (1) consistent F1@5 gains over registry-only retrieval across three structured domains and both retriever families; (2) effectiveness survives LLM-generated seed memories; (3) a reproducible interference failure mode; (4) memory-only retrieval wins in lexically sparse domains. The scope is deliberately narrow — retrieval metrics only, not SQL generation or execution — and Section 4 is upfront that this is a smaller artifact than trajectory- or skill-based agent memories.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is persistent discovery context?
It is a lightweight memory layer for AI agents that work with structured data. It stores records of past successful discovery steps — a task description (intent) mapped to the database tables and attributes that proved useful — and, during future retrieval over a data registry, adds a similarity-weighted bonus to objects that appeared in those stored records, on top of the normal metadata search score.
How much does the memory layer improve retrieval?
On three small public datasets (a Czech banking dataset, NYPD collision records, and the Northwind sample database), F1@5 rose from registry-only 0.396/0.299/0.446 to 0.499/0.482/0.528 with TF-IDF (finance/NYC/Northwind), with smaller but still positive gains (+0.051 to +0.167) when using neural sentence embeddings as the base retriever.
What is the main failure mode of this approach?
Memory interference. If a stored memory from the wrong task family is retrieved because its wording resembles the current query, it promotes the wrong objects and retrieval quality falls below a plain no-memory search — for example F1@5 of 0.000 versus 0.444 in the Northwind experiment. The authors flag interference-aware memory selection as future work.
Is code or data available for this paper?
The paper states no code or task-set release. The three underlying datasets are publicly available, but the 140 tasks, gold annotations, and natural-language registry descriptions were constructed by the authors and are not published, so exact replication of the reported numbers is not currently possible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 appear only as captions in the extracted text; their plotted values (per-family bars, α-sweep curves) are partially described in surrounding prose but the graphics themselves could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.