JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
cs.AI · cs.PF
Quick verdict
Claims 6.93× context gain on a 24 GiB laptop via 4-bit KV + JIT state management; credible systems work, but long-context quality and ablations are thin.
Summary
JustFit is an MLX-based serving runtime that runs a 27B hybrid model (16 full-attention + 48 Gated DeltaNet layers, fixed MXFP4 weights) beyond 200K retained positions on a 24 GiB Apple M4 Pro laptop. Three mechanisms cooperate: KVExec (TurboQuant 4-bit packed KV with page addressing, fused dequant/inverse-Hadamard reconstruction, explicit evaluation boundaries), PhaseSwap (leasing the 644-MiB LM head, vision tower, and MTP drafter only during required phases), and StateTrans (admission control, MTP-to-AR conversion, reference-counted page release at safe boundaries). With repetitive prompts, greedy decoding, and EOS suppression under a 21,000-MiB phys_footprint guard, three fresh-process runs complete 196,608 input plus 16,384 output tokens (212,992 positions, 6.93× the mlx-vlm baseline's 30,720) at 4.99 tokens/s decode and 20,975 MiB median peak; two concurrent requests retain 229,376 aggregate positions. The final 32K+6K configuration reaches 11.54 tokens/s at 16,374 MiB, and AIME 2026 yields 29/30 (paged TQ4) versus 28/30 (uniform INT8). Main limitations: development configurations conflate mechanisms with no PhaseSwap on/off ablation; segmented KV is faster (13.30 tokens/s) than the final paged system; no long-context comprehension evaluation exists (the 200K tests are synthetic stress runs); the AIME comparison is confounded by different output ceilings and a guard-recovery protocol; the benchmark companion is unpublished and historical environment metadata is incomplete.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- JustFit completes 196,608 input plus 16,384 output tokens in three independent fresh-process runs on a 24 GiB M4 Pro, raising completed single-request context from the mlx-vlm baseline's 30,720 to 212,992 positions (6.93×).
- TurboQuant-based 4-bit packed KV with page-native fused reconstruction cuts KV storage 3.94× versus FP16 (16,640 vs 65,536 bytes per position) without requantizing model weights.
- Component leasing and state-preserving transitions let concurrent requests share the expanded pool, retaining 229,376 aggregate positions across two requests and 204,800 across four.
- The integrated runtime sustains 11.54 tokens/s at a fixed 32K+6K workload (16,374 MiB peak) and answers 29/30 AIME 2026 problems, which the authors offer as evidence of reasoning over compressed state.
Strengths
Repeatable headline result with tight variance
Three independent fresh-process 192K+16K runs produce identical output-token sequences with PP 68.304–68.326 tok/s, peaks 20,960–20,977 MiB, and passing postflight/page-reuse checks (Tables 1 and 7) — unusually careful for a single-laptop capacity claim.
Transparent memory accounting
Section 2 separates logical live-set terms (Eq. 1) from OS-level phys_footprint sampling every 0.25 s, and derives the 16,640 B/position TQ4 payload vs 65,536 B FP16 (3.94× reduction including norms), so the capacity arithmetic is checkable rather than asserted.
Candid limitations section
Mechanisms are concretely engineered and priced
Capacity is shown as shared, not singleton-only
Weaknesses
No long-context quality evaluation
Conflated development study, no matched ablation
Confounded, statistically fragile AIME comparison
Single weak baseline, no alternative-KV comparison
Reproducibility gaps acknowledged but unresolved
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can a 27B model really run with 200K context on a 24 GB MacBook?
According to this paper, requests do complete: with 4-bit quantized KV storage, page-based memory management, and on-demand loading of large components (LM head, vision tower, draft model), three test runs finished 196,608 input plus 16,384 output tokens within a 21,000-MiB memory guard on an M4 Pro. Important caveats: the test prompts were repetitive filler text, decode speed at that scale is only about 5 tokens/s, cold prefill takes roughly 48 minutes, and the peak memory came within 23 MiB of the guard — so this demonstrates a capacity boundary, not a comfortable daily setting.
Does 4-bit KV cache quantization hurt answer quality?
The paper's only quality check is AIME 2026 (30 math problems, one random seed): the 4-bit paged-KV runtime answered 29/30 correctly versus 28/30 for a uniform 8-bit comparison. However, the two arms used different output-token ceilings and the 8-bit arm had to restart on fresh servers after a memory-guard event, so the one-problem difference is not meaningful. No long-context comprehension benchmarks (such as needle-in-a-haystack) were run, so quality when actually using 100K+ tokens of context is unverified.
Is the JustFit code available and can I reproduce the results?
Partially. The author's public mlx-vlm fork hosts the production branch plus two pinned source snapshots, and the paper ships manifests with request hashes and plot data. But the benchmark companion was described as still being prepared at publication time, per-run environment metadata (macOS, Python, MLX versions) is missing for some historical runs, and the original checkpoint's byte identity is unverified. The authors state that exact reproduction of the throughput and memory-margin numbers is not currently possible.
How is this different from just using a more aggressively quantized model like 3-bit weights?
JustFit is complementary rather than competing: it keeps the model's 4-bit weights fixed and instead compresses the KV cache (TurboQuant 4-bit), reconstructs attention operands just-in-time per layer, and leases large components only when a running request needs them. Weight quantization reduces static storage; JustFit expands how much conversation context and how many concurrent requests fit around those weights. The paper shows the approaches compose by running the heavily compressed Bonsai 27B model with its paged 4-bit KV, scoring 26/30 on AIME 2026 from a MacBook Air.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are available only through their captions in the extracted text; the visual tradeoff curves and schematic details could not be independently analyzed.
- Several key comparators (DwarfStar, Open-TQ-Metal, Bonsai 27B, vllm-mlx behavior, Qwen3.8-27B model-card numbers) are 2026 web or preprint references whose contents could not be verified from the text.
- The evidence archive, run manifests, and source snapshots referenced in Appendix C.2 are external artifacts and could not be inspected; review is based solely on the manuscript.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.