DolphinBench: Mapping the Pareto Frontier of Agent Memory
Soumil Rathi, Deshraj Yadav, Taranjeet Singh
cs.CL · cs.AI
Quick verdict
Mem0 团队的智能体记忆基准测试：包含动作任务、oracle 验证、成本+延迟指标。设计合理，但厂商自身系统胜出——仍需复现验证。
Summary
DolphinBench 是一个通过任务完成度而非问答来评估 LLM 智能体长期记忆的基准测试。作者构建了三个合成知识型员工角色（初创公司 CEO、基础设施工程师、产品经理），每个角色包含跨越数年、约 50 万 token 的模拟用户消息，并定义了每个角色 200 个依赖记忆的动作任务（共 600 个），这些任务在模拟的日历、电子邮件和 GitHub 等应用中执行。每个测试都经过认证：运行 GPT-5.6-Luna 两次，需通过相关的 oracle 历史记录测试；运行两次不包含该历史记录，需失败。提交的内容必须报告准确率、总成本和中位延迟。在三组 harness-model 对中，最佳配置（使用 GPT-5.6-Luna 和 Mem0 的 Hermes）以 96.21 美元的总成本完成了 70.67% 的任务；记忆系统在不同 harness 下的排名有所不同，在 Claude Code 上 Honcho 领先（35.83%）。由于作者是 Mem0 团队，且 Mem0 在三组中的两组中名列前茅，因此独立复现至关重要。文中指出的局限性包括合成历史、仅有三个角色、模拟应用、1-4 步的短任务以及批量数据摄取；尽管依赖 LLM 裁判，但尚未报告显著性检验或裁判验证。
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- QA-format memory benchmarks leak a retrieval signal — the question itself announces which fact is needed — so memory should be tested through action-based task completion instead.
- Every DolphinBench test is certified solvable: an agent with oracle history must pass twice and must fail twice without it, eliminating answer-key and unanswerable-question defects.
- Requiring total cost and median latency alongside accuracy is necessary because accuracy alone lets systems buy scores with unsustainable compute.
- No existing memory benchmark combines action-based testing, cost+latency reporting, and per-test verification (Table 1).
- The best evaluated configuration (Hermes + GPT-5.6-Luna + Mem0) completes 70.67% of tasks, and memory-system rankings are not stable across harnesses.
Strengths
Action format removes the QA retrieval cue
§1 and the §3 worked example: the instruction 'order one small latte for pickup' never mentions the cafe, so the agent must recall Morgan's Blue Bottle preference unprompted — the task succeeds only if the system 'knows to remember'.
Per-test dual verification attacks a documented failure mode
§3.1 Test Verification requires 2× pass with oracle history and 2× fail without (Figure 2), directly addressing the LoCoMo 6.4% wrong-answer-key finding and LongMemEval ground-truth issues cited in §2.3 — most benchmarks never establish a solvability ceiling.
Cost+latency reporting exposes real Pareto tradeoffs
Rank instability is an honest, useful finding
Above-average transparency
Weaknesses
Vendor self-evaluation conflict of interest
No uncertainty quantification anywhere
LLM-judge reliance despite the paper's own criticism of judges
Confounded and incomplete evaluation grid
Verification circularity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
DolphinBench (arXiv 2609.24971v2, cs.CL) is a benchmark paper from the Mem0 team arguing that agent memory should be evaluated through task completion, not conversational QA. It claims three properties no prior benchmark combines (Table 1): action-based tasks that strip the retrieval cue a question provides; mandatory cost and latency reporting alongside accuracy; and per-test solvability certification via paired agent runs. Headline results (§4, Table 2): across three harness–model pairs, the best configuration (Hermes + GPT-5.6-Luna + Mem0) completes 70.67% of 600 tasks at $96.21 total cost, memory-system rankings flip across harnesses (Mem0 leads both Hermes groups; Honcho leads Claude Code + Sonnet 5 at 35.83%), and the most expensive system (Supermemory, $497.70 with MiniMax M3) is neither most accurate (20.00%) nor fastest.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DolphinBench?
DolphinBench is a benchmark for evaluating the long-term memory of LLM agents through task completion rather than question answering. It provides three simulated knowledge-worker personas with roughly 500k tokens of message history each, and 600 tasks (200 per persona) that can only be completed correctly by recalling facts from that history inside simulated apps like calendar, email, and GitHub. Every result must also report cost and latency, and every task was verified to be solvable with the relevant history and unsolvable without it.
Which memory system performed best on DolphinBench?
It depends on the harness and model. With Hermes + GPT-5.6-Luna, Mem0 scored highest at 70.67% accuracy ($96.21 total). With Hermes + MiniMax M3, Mem0 again led (47.83%), just ahead of Honcho (47.33%). With Claude Code + Claude Sonnet 5, Honcho led at 35.83% ahead of Mem0 (32.33%). Numbers are not directly comparable across harness groups, and no configuration exceeded 71% accuracy.
Who created DolphinBench, and is there a conflict of interest?
It was created by Soumil Rathi, Deshraj Yadav, and Taranjeet Singh, who are affiliated with Mem0 — and two of them are authors of the Mem0 memory system paper. Mem0 ranks first in two of the three evaluated harness groups, so independent third-party replications are important before treating the rankings as settled.
Is the DolphinBench dataset and code publicly available?
Yes. The paper states the dataset and evaluation code are available at dolphinbench.ai, and the authors commit to releasing configurations, cost calculations, and full traces of every run (tool calls, arguments, and per-check verdicts) alongside the dataset. Note that re-running the full benchmark requires paid access to frontier models and third-party memory services.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are available only as captions in the extracted text; pipeline diagrams could not be inspected directly.
- No appendices appear in the extraction — judge prompts, per-persona score breakdowns, and per-check examples are not visible, so grading-strength judgments rely on §3.2 prose.
- The dolphinbench.ai repository and released traces could not be verified in this offline review; 'codeAvailable' reflects the paper's claim only.
- Per-persona accuracy is not broken out in the extracted text, so persona-level difficulty effects cannot be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.