mzCache: On-Device LLM Memory Management under Multitasking
Hongseung Yu, Minsung Kim, Jongseok Park, Kyunghan Lee
cs.OS · cs.DC · cs.LG
Quick verdict
Claims 2.1–5.5× TTFT cuts for on-device LLM inference under multitasking via hybrid swap; solid design, but baselines are self-built and devices Qualcomm-only.
Summary
The paper tackles Time-to-First-Token (TTFT) degradation in on-device mobile LLM inference when multitasking memory pressure evicts model weights and KV cache (TTFT rises from 0.8 s to 16 s; zRAM compresses KV cache by only 0.2–9.3%). mzCache, built on llama.cpp in ~6k lines of C/C++, partitions weights per layer and KV cache into 256-token chunks held in unified-memory shared buffers via OpenCL shared virtual memory, enabling partial eviction without kernel modification. Evicted KV chunks are distributed between in-memory compressed space (8-bit quantization or CacheGen) and storage reads using an offline-profiled load-balancing threshold, while backward-out/forward-in ordering lets GPU prefill overlap CPU-side restoration through a custom OpenCL attention kernel. On Galaxy S25+ and OnePlus 12 with Qwen3-0.6B and EXAONE-4.0-1.2B at 8k–32k contexts, mzCache cuts TTFT 2.1–5.5× versus a self-implemented storage-backed partial-offload baseline, and 2.5–3.0×/9.2–25.9× versus OS Paging (zRAM, CPU inference) at full eviction. A 45-minute real-app multitasking test shows the 25pp-eviction configuration surviving all ten pressure rounds while OS Paging is killed each round. Limitations: only two Qualcomm devices, two sub-3B models, accuracy checked only via 428 TriviaQA F1 pairs, and a coarse eviction-amount heuristic (15pp config killed in 4/10 rounds).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- OS memory management (paging, zRAM, LMK) is inadequate for on-device LLM memory under multitasking, degrading TTFT by 18–20× under full flash-swap eviction and ultimately killing the process.
- Partitioning weights per layer and KV cache into 256-token chunks held as unified-memory shared buffers enables partial eviction with concurrent CPU restoration and GPU inference without inter-processor copies.
- Hybrid swap with a profiled balance threshold T = N_total_W·⌊t_W^r/t_KV^d⌋ keeps the in-memory decompression and storage-read restore paths balanced from any eviction state.
- Backward-out/forward-in eviction-restoration ordering preserves early layers so GPU prefill overlaps CPU restoration, maximizing overlap regardless of eviction amount.
- mzCache achieves 2.1–5.5× lower TTFT than storage-backed partial offload and survives real multitasking pressure (10/10 rounds at 25pp eviction) where OS paging is LMK-killed every round.
Strengths
Strongly motivated problem framing with quantitative evidence
§3 grounds the problem empirically: a Perfetto trace shows the LMK killing the LLM process (Fig 2), zRAM saves only 0.2–9.3% of KV cache space across five models (Table 2), storage reads account for 59–61% of TTFT under full reload (Fig 4), and full eviction inflates TTFT 18–20× (flash) / 5–6× (zRAM) vs in-memory (Fig 3).
Principled, analyzable core design
Hybrid swap derives threshold T = N_total_W·⌊t_W^r/t_KV^d⌋ (§4.3) so both restore paths finish in equal time wherever eviction stops; the four-stage policy (KVonly→KVandW→Wonly→CompKV) and its balance-preservation argument are clean, and Fig 12 confirms near-equal path latencies empirically.
Thorough mechanism-level ablations
Realistic deployment validation with honest negative results
Candid limitations discussion
Weaknesses
Narrow hardware and model coverage
Headline speedup is against a self-built baseline
Thin accuracy evaluation and unsupported idempotency claim
Eviction-amount policy is ad hoc and partially fails
No statistical treatment of measurements
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a mobile-systems method paper (MobiCom '26, arXiv 2609.01338) targeting Time-to-First-Token (TTFT) degradation of on-device LLM inference when multitasking memory pressure evicts model weights and the KV cache. The authors first establish the problem (§1–§3): under Android paging, full eviction inflates TTFT 18–20× with flash swap and 5–6× with zRAM versus the all-in-memory case (Fig 3); zRAM's general-purpose compressors achieve only 0.2–9.3% space savings on KV cache (Table 2), so pressure persists and the Low-Memory Killer terminates the LLM process (Fig 2). Central claims: (1) OS mechanisms handle neither pageable nor page-locked LLM memory well; (2) a restoration-oriented design — fine-grained shared buffers, hybrid swap with a profiled balance threshold, and backward-out/forward-in ordering — enables responsive inference from any eviction state; (3) measured gains of 2.1–5.5× TTFT versus storage-backed partial offload on commercial smartphones.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does mzCache solve?
When you switch apps on a phone, Android may evict an on-device LLM's weights and KV cache; returning to the LLM then takes seconds (TTFT rose from 0.8 s to 16 s in the paper) because data must be re-read from slow storage or recomputed, and severe pressure kills the process entirely. mzCache manages that memory itself: it evicts only what is needed under pressure and restores it fast enough that inference starts immediately.
How much faster is mzCache than existing approaches?
On Galaxy S25+ and OnePlus 12 with two small models (Qwen3-0.6B, EXAONE-4.0-1.2B) at 8k–32k context, it cut time-to-first-token by 2.1–5.5× versus a storage-backed partial-offload baseline, and by 2.5–3.0× (S25+) / 9.2–25.9× (OnePlus 12) versus Android's default paging with zRAM under full eviction.
Does mzCache require root access or kernel modifications?
No. It is built on llama.cpp as an Android app using OpenCL shared virtual memory for CPU-GPU shared buffers, with Android's onTrimMemory callback for pressure detection; the authors note a system-level deployment could alternatively use vmpressure/PSI kernel signals.
Is mzCache code available?
The paper does not state a public code release. The implementation is ~6k lines of C/C++ on llama.cpp plus a 0.6k-line OpenCL kernel, evaluated on specific Qualcomm-based phones with per-device offline profiling, which limits third-party reproduction.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures appear only as extracted captions/annotations; exact per-point TTFT values (e.g., Fig 9 curves) are taken from the authors' stated summary numbers, not from readable plot data.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.