Measure Before You Manage: Evaluating Agent Working Memory in Coding Agents
Le Chen, Zishen Wan, Baixi Sun, Xiaolong Ma, Chih-Hsuan Yang, Feng Yan, Sheng Di, Franck Cappello, et al.
cs.AI
Quick verdict
A candid measurement study of coding-agent memory: its four-level evaluation lesson is valuable, but policy evidence is thin and the archive stays unreleased.
Summary
This NeurIPS workshop paper asks how semantic heterogeneity in coding-agent working memory should shape management and evaluation. From 55 archived SWE-bench-Lite trajectories (1,350 in-context objects, eight repositories), it shows tool outputs supply 55.5% of pooled content volume but 40.2% of retention-weighted cost, while artifacts contribute 28.3% versus 38.9%, with type-specific compression ratios of 0.150 for artifacts against 0.673 for tool outputs. Two case studies follow. A hand-specified object-aware compression policy beat FIFO by 1.633 fewer repeated tool calls on 15 calibration tasks (Holm-adjusted p=0.0146), but not on eight held-out tasks (-0.500, two nonzero pairs); in a six-arm retrieval study, only eight of 24 blocks completed, and no contrast survived correction. The authors then demonstrate that nominal token budgets conceal unequal delivered context (no capped-arm pair matched within 10% on all eight tasks; FIFO-GA gap up to 18.7%) and unequal management work (285 importance calls, 169 summary calls), and a GB10 serving replay shows an uncapped arm producing 37,883-token prompts against a 32,768-token limit. They distill a four-level evaluation framework: stored state, delivered context, management work, and task/process outcome. Severe limitations are candidly disclosed: tiny conditional samples, one run per arm, an unpinned served model, defective lifecycle signals, and no public data or code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Semantically distinct working-memory object types (tool outputs, artifacts, instructions, agent state) exhibit distinct size, retention, and compression behavior across 55 archived coding-agent trajectories.
- An object-aware compression policy's calibration gain (-1.633 repeated calls vs FIFO on 15 SymPy tasks, Holm-adjusted p=0.0146) does not confirm on eight held-out tasks.
- Equal nominal token budgets do not imply equal delivered context (no capped-arm pair within 10% on all eight complete retrieval tasks) or equal management work (285 importance calls for GA, 169 summary calls for OA).
- A serving replay on an NVIDIA GB10 shows an uncapped arm producing a 37,883-token prompt against a frozen 32,768-token limit — a feasibility boundary invisible in nominal budgets.
- Reliable evaluation of agent working memory requires reporting at four levels: stored state, delivered context, management work, and task/process outcome.
Strengths
Measurement honesty as a first-class contribution
Conflicts are disclosed rather than smoothed: two irreconcilable dollar estimates ($213.52 vs $235.99, Appendix E.3), narrative-vs-cell median discrepancies kept visible (Table E.1), and the bootstrap-CI-vs-test divergence explained as discreteness rather than equivalence; checklist questions 4, 5, and 8 are answered No.
The framework is backed by measurements, not rhetoric
The delivery diagnostic (Table E.2: no capped pair within 10% on all eight tasks; FIFO-UC and LRU-D-UC gaps 88.8%) and the auxiliary-call ledger (Table 3: GA 285 importance calls, OA 169 summary calls, GA-FIFO wall time +67.45 s) operationalize Levels 2-3 with numbers, not slogans.
Statistical care appropriate to small samples
Mechanism diagnostics with hard evidence
Quantitatively useful heterogeneity characterization
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (NeurIPS 2026 AgenticOS workshop; arXiv 2608.31057) makes three deliberately scoped claims: (1) a typed-object characterization of semantic heterogeneity in coding-agent working memory across 55 archived trajectories (Section 3); (2) two case studies of semantically informed management — a hand-specified object-aware compression policy (OA, Section 4.1) and an adapted Generative-Agents retrieval policy (GA, Section 4.2); (3) a four-level evaluation framework separating stored state, delivered context, management work, and task/process outcome (Section 6, Table 4). The abstract promises no winning policy; the contribution is measurement discipline. The paper largely delivers what it claims, but the empirical base is thin and partly rests on instrumentation the authors themselves show to be defective.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did the object-aware compression policy actually work?
Only on its 15 SymPy calibration tasks: -1.633 repeated tool calls vs FIFO, Holm-adjusted p=0.0146. On the 8 held-out tasks the contrast shrank to -0.500 with just two nonzero pairs (p=0.50), and in the six-arm retrieval study no comparison survived correction — the paper's own headline is that the gain did not transfer.
What are the four levels of the paper's evaluation framework?
Stored state (typed objects, sizes, residency), delivered context (actual tokens an arm receives versus its nominal budget label), management work (auxiliary LLM calls, embeddings, wall time), and task/process outcome (with valid repair evaluation kept distinct from process metrics like repeated tool calls).
Is the paper's data or code publicly available?
No. Appendix F locates evidence in an internal working archive with an audit script and input hashes, but the authors explicitly promise neither an anonymous public release nor byte-identical trajectory regeneration, and the served model revision behind the recorded alias is unpinned.
Why is it called 'Measure Before You Manage'?
Because nominal token budgets proved misleading in their own experiments: arms sharing a cap delivered context differing by up to 88.8%, policies cost unequal auxiliary work (285 importance calls, 169 summary calls), and an uncapped arm produced 37,883-token prompts against a real 32,768-token serving limit — none visible from budget labels alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's values arrive flattened into a raw digit stream in the PDF extraction; every figure number cited here was cross-checked against the caption and the agreeing Appendix B.1/B.2 tables, so no cited value is guessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.