MemRiskBench: Trace-Aware Risk-Preserving Evaluation for Long-Horizon LLM Agents

Summary

The paper introduces MemRiskBench, a 120-episode scripted benchmark that evaluates safety-relevant memory failures of long-horizon LLM agents across a five-category taxonomy: stale memory reliance, conflicting-memory misresolution, cross-scope leakage, revoked-memory reuse, and constraint decay, plus a documented but unscored sixth category. Scoring is fully deterministic, drawing on text, tool-call, memory-ledger, and constraint checks over logged traces, with no LLM-as-judge on the pass/fail path. Five quantized local models (1.5B-7B parameters; three Qwen2.5 variants, Phi-3.5-mini, Llama-3.2-3B) produced 600 traces. A second contribution is a coverage-constrained greedy subset selector on 12 trace-derived features that, at 20% of episodes, preserves the full-benchmark model ranking (Spearman rho=0.975), risk-type coverage (1.0), and high-risk-model detection (1.0), cutting compute fivefold. Notable findings include severe cross-scope leakage failures for Qwen2.5-1.5B (1 of 24 episodes, versus 10 of 24 for Phi-3.5-mini) and an inverted constraint-obedience pattern (the 1.5B model passes all 24). Main limitations: only small local models, three of five from one family; a quantization confound on the 7B (Q3 vs Q4); headline model-risk contrasts that do not survive Holm correction; a memory-baseline experiment reported without numbers; and the selector's dependence on a fixed reference model set.