Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
Jinli Hu, Ross M. Clarke, Yichuan Zhang, Jos\'e Miguel Hern\'andez-Lobato
cs.AI · cs.LG
Quick verdict
Generates LLM experts from live data with an online Bayesian belief over the code; categorical-router results look strong, but the Bayesian core is untested.
Summary
The paper proposes an Infinite-Parameter LLM: instead of storing a Mixture-of-Experts bank, a compact encoder reads run-time data (facts, instructions) into a latent code that generates a low-rank LoRA-style delta over a frozen base FFN, and a Bayesian belief over that code is updated online across a session. The authors position this against stored-bank MoE, one-shot weight generators (Text-to-LoRA, SHINE), and prompt-side in-context learning/retrieval, arguing the coupling of generated weights plus an evolving belief is unclaimed in prior work. The implementation reuses SHINE (Qwen3-8B base, memory tokens, M2P network, r=8) and instantiates the belief as a categorical distribution over pre-materialized per-passage codes, adding only a trained activation-based selector; the continuous-Gaussian amortized filter that embodies the core methodological novelty is deferred to future work. Experiments (n=150) show compiled weights beat in-context prompting on long, noisy, multi-hop evidence (48.0 vs 33.6 F1 on MS MARCO) but lose on short clean evidence (SQuAD 85.3 vs 51.8); the trained selector beats dense retrieval by 8-12 top-1 points across four datasets; and an accumulated categorical belief routes better than memoryless routers and concat-history retrieval on authored conversations. Main limitations: the headline Bayesian mechanism is unvalidated, no matched-budget retrieval-plus-prompt end-to-end baseline exists, no confidence intervals or significance tests are reported, conversations are synthetic, and training details are withheld.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Run-time evidence compiled into generated weights is usable by a frozen LLM and beats in-context prompting when evidence is long, noisy, or multi-hop (48.0 vs 33.6 F1 on MS MARCO v2.1), while the prompt wins on short clean evidence (SQuAD).
- A trained categorical selector over per-passage materialized codes routes questions to the answer-bearing code 8-12 top-1 points better than dense retrieval across four QA datasets (Table 5).
- A persistent categorical belief accumulated across turns routes better than memoryless routers and growing concat-history retrieval, at flat per-turn cost (Section 4.4, Figure 6).
- The architecture realizes an unbounded set of reachable effective weights from a fixed stored footprint, with 'infinite parameters' defined narrowly and explicitly not claiming unbounded knowledge (Section 3.4).
- The full coupling - generated weights plus an online-updated Bayesian belief over the code - is unclaimed in prior work, with MoEGen treated as concurrent (Section 2.7).
Strengths
Honest weights-vs-prompt evaluation that reports where the method loses
Table 4 shows in-context prompting wins on SQuAD (85.3 vs 51.8 F1), and the paper frames the result as a crossover along evidence difficulty rather than claiming weights always win; Limitations (§5) restates this boundary explicitly.
The dilution study is a genuinely informative analysis
Figure 5 varies pool size (8-64 passages) at two encoder budgets (1300/3000 tokens), separating code saturation (oracle falls 51.6 to 46.9 F1 at 0% truncation) from truncation (realistic collapses to 27.8 F1 at 1300 tokens) and burial; this directly motivates one bounded read per code plus a selector.
Routing evaluation includes an oracle upper bound and a near-random zero-shot control
Clean end-to-end pool-size sweep demonstrating the structural advantage of selection
Unusually thorough related-work positioning
Weaknesses
The claimed differentiating mechanism is not the evaluated system
Missing matched-budget end-to-end prompt baseline
No uncertainty quantification anywhere
Synthetic, author-constructed multi-turn evaluation
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does the Infinite-Parameter LLM really have infinite parameters?
No. Every stored component - the frozen base model, the compact generator, and the small selector - is finite and fixed. The term refers narrowly to the unbounded set of effective weights the model can realize: because codes are generated from whatever live data is supplied rather than enumerated from a stored bank, each input can materialize a new low-rank expert over the shared base. The paper explicitly states that knowledge capacity remains bounded by the resident parameters, so this is not a claim about storing more knowledge.
How does this approach compare to RAG or putting documents in the prompt?
On the paper's QA benchmarks (n=150 groups), compiling evidence into generated weights beats putting all evidence in the prompt when evidence is long or noisy - 48.0 vs 33.6 F1 on MS MARCO with 10 passages - but loses on short clean evidence (SQuAD: 85.3 for the prompt vs 51.8 for weights). The paper does not report an end-to-end comparison against a retrieval-then-prompt pipeline at matched token budget, which reviewers flag as a missing baseline.
Is the Bayesian belief in the experiments actually Bayesian?
Only in a limited sense. The evaluated system carries a categorical distribution over pre-computed codes, updated by recursive categorical Bayes with a trained selector. The richer continuous-Gaussian belief with an amortized recursive filter, calibrated posterior precision, and uncertainty-gated updates - the part the paper presents as its main methodological contribution - is deferred to future work and is not tested in any experiment.
Is code or data available for this paper?
The paper text contains no code or data release statement, and Section 4.1 says full training details are outside the paper's scope. The weight generator is reused frozen from the SHINE checkpoint and only a small categorical selector is trained, but its training data and objective are not specified, so independent reproduction is currently difficult.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 5 and 6 are accessible only through their captions; their numeric content could not be verified, and the Section 4.4 accumulation magnitudes are figure-only.
- The references list shows OCR corruption (e.g., the Dohare, Krishnan, and Ritter entries are mangled), indicating minor extraction loss, though body tables appear intact.
- Any appendices, training appendix, or code-availability statement may exist outside the extracted text; reproducibility judgments are based on the provided body.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.