Lngram v2: Latent N-Gram Memory with Interpretable Discrete Representations
Yunao Zheng, Bin Wen, Xiaojie Wang, Kaiyu Jiang, Xuanyu Zheng, Changyi Liu, Hongyi Fu, Jianxiong Wang, et al.
cs.CL
Quick verdict
Lngram v2 decouples memory capacity from backbone width; results are statistically careful but gains are modest (+0.68 avg on 2B) and code release is unstated.
Summary
Lngram v2 is a conditional-memory module for Transformers: hidden states are discretized into multiple route codes, which form exact n-gram addresses into memory tables read out with grouped-query attention and a zero-value Sink; the hard discrete addressing is trained with counterfactual surrogate gradients. Compared with Lngram v1, it decouples route count, memory dimension, and backbone width: in a 0.6B MoE comparison, the R=16 variant cuts total module parameters by 82.6% and per-token activated parameters by 95.2% while slightly improving validation loss (2.8584 vs 2.8609). Continuing training of Keye2B for about 5B tokens raises a ten-benchmark average from 47.25±0.15 to 47.93±0.18 over three seeds, above a parameter-matched Sparse FFN (45.86) and PKM (47.24); on Keye30B the average rises from 78.23 to 79.71 (95% CI [0.86, 2.11]), though only AI2D and MMBench-EN survive Holm correction. Inference overhead for the r64 configuration is about 6.1% prefill and 8.0% decode. The paper also reports that discrete IDs alone retain 65.77%–84.27% of the excess semantic readout of continuous hidden states. Limitations include small absolute gains, single-run 30B and baseline comparisons, an interpretability analysis confined to two layers of one model, and no stated code release.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Decoupling route count, memory dimension, and backbone width lets memory capacity scale independently, cutting Lngram's total parameters by 82.6% and per-token activated parameters by 95.2% vs v1 (R=16, 0.6B MoE) with slightly better validation loss.
- Under matched data and compute, Lngram v2 improves Keye2B's ten-benchmark average from 47.25±0.15 to 47.93±0.18 (three seeds) and Keye30B's from 78.23 to 79.71 (95% CI [0.86, 2.11]).
- Parameter-matched Sparse FFN (45.86) and PKM (47.24) controls show the gains are not explained by added parameters or sparse capacity.
- Counterfactual surrogate gradients are necessary to train the hard discrete addressing: direct STE fails (validation loss 7.3349 vs 2.8585).
- Discrete IDs alone retain 65.77%–84.27% of the excess semantic readout capability of continuous hidden states, with statistically reproducible code-concept associations.
Strengths
Statistical rigor well above field norm
Paired per-sample McNemar, sign-flip permutation, 200k-replicate bootstrap CIs, and Holm/BH corrections on Keye30B (Table 11); three-seed mean±SD on Keye2B (Table 2); 500 document-level permutation tests for the ID semantics (p = 1/501, Sec 3.2).
Parameter-matched controls rule out capacity confound
Top-1 Sparse SwiGLU branch with 155.845M total / 20.972M active parameters vs Lngram's 155.6M / 20.9M, in the same layers (2, 14), scores 45.86 vs Lngram's 47.93 — cleanly isolating the addressing mechanism from raw capacity.
Honest failure and necessity analyses
Complete efficiency accounting including unfavorable rows
Demonstrated at 30B with deployment-level detail
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper proposes Lngram v2, a conditional-memory branch for Transformer decoders that offloads local pattern storage from dense computation. Hidden states are projected to R routes x M bits, threshold-discretized into symbols; consecutive symbols form exact 2/3-gram addresses into memory tables, read out by grouped-query attention with a fixed zero-value Sink, followed by a depthwise causal convolution injected as a pre-attention residual. Hard addressing is trained with counterfactual surrogate gradients (exact O(K) and one-bit O(M) variants, Appendix A). Four central claims: (1) decoupling route count, memory dimension, and backbone width cuts module parameters by 82.6% and per-token activated parameters by 95.2% versus v1 while slightly improving validation loss (Table 4: 2.8584 vs 2.8609); (2) consistent benchmark gains at 2B and 30B; (3) gains are not explained by extra parameters; (4) discrete IDs preserve enough semantics to serve as a queryable analysis interface.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Lngram v2?
A conditional-memory module for Transformers: hidden states are discretized into multiple route codes, which form exact n-gram addresses into a memory table, and the retrieved entries are read out with grouped-query attention plus a zero-value Sink. Hard discrete addressing is trained with counterfactual surrogate gradients, so training and inference share the same hard lookup path.
How much smaller is Lngram v2 than Lngram v1?
On a 0.6B MoE testbed at R=16, Lngram v2 cuts the module's total parameters by 82.6% and per-token activated parameters by 95.2% versus v1, while achieving a slightly lower validation loss (2.8584 vs 2.8609). This comparison was run at 0.6B scale, single seed.
Does Lngram v2 work on large models?
The authors report gains on the 30B-parameter Keye VLM: the ten-benchmark average rises from 78.23 to 79.71 (95% CI [0.86, 2.11]), with inference overhead of about 6.1% prefill and 8.0% decode for the r64 configuration. After multiple-comparison correction, only AI2D and MMBench-EN remain individually significant.
Is code for Lngram v2 available?
The paper does not mention a code release. It gives detailed hyperparameters (Appendix E) and the v1-vs-v2 comparison uses public FineWeb-Edu data, but the main Keye2B/Keye30B backbones are proprietary models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are images absent from the text extraction; loss-curve evidence (Figs. 2-3, Appendix A.5) was judged from the stated numbers only, not the plots.
- Paper metadata lists 8 authors, but the extracted byline lists 15 (adds Tianke Zhang, Haonan Fan, Yingxin Li, Jiankang Chen, Xu Wang, Tingting Gao, Han Li).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.