arXiv paper2026-09-22· machine learning (LLM fine-tuning)· method paper

Reasoning-Preserving Fine-Tuning of Post-RL LLMs with Null-Basis LoRA

Wenzhi Fang, Nicholas Tzou, Lazar Valkov, Srinivas Chappidi

cs.AI

Quick verdict

Solid method paper: freezing a null basis of reasoning activations into LoRA preserves post-RL reasoning during SFT, but evidence is small-scale, single-seed.

Summary

Reinforcement-learning post-training gives LLMs strong reasoning, but subsequent supervised fine-tuning can erase it. Building on the observation that reasoning activations occupy a low-dimensional subspace — leaving roughly 29-45% approximate null-space capacity at a 99% spectral-energy threshold — the authors propose Null-Basis LoRA (NB-LoRA). A frozen orthonormal basis, estimated offline from hidden states of a few hundred reasoning rollouts, reparameterizes the LoRA update as ΔW = BCN_s^⊤, so layer outputs on reasoning states are preserved by construction while B and C train with standard optimizers — no replay, no per-step gradient projection. On Qwen2.5-3B and 1.5B models post-trained with GRPO on MATH-lighteval and GSM8K, NB-LoRA matches vanilla LoRA on target-task adaptation (average 0.846 vs 0.847 across eight commonsense tasks plus tool use) while keeping reasoning essentially at its pre-fine-tuning level (63.2% vs 63.8% on the 3B model; 80.0% vs 80.1% on the 1.5B model), and retention generalizes to held-out math benchmarks from a single anchor corpus. Training is about 6.3x faster than the full-model null-space projection baseline NSCL. Main limitations: only two small Qwen models, single-run results without seed variance or significance tests, an approximate rather than exact null space, no stated code release, and several prose-versus-table numeric inconsistencies.

Review complete

Assessment at a glance

machine learning (LLM fine-tuning)

risk flags identified

● high 0

● medium 2

● low 3

Reproducibility readinessmedium

Code & data availablenot stated

Review confidencehigh

Deep research viableCode-reproducibleNeeds GPU

5 strengths · 5 weaknesses · 5 author questions

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

Reasoning hidden states of post-RL LLMs concentrate in a low-dimensional subspace, leaving a 29-45% approximate null space (99% energy threshold) that is stable when estimated from roughly 64-512 reasoning examples.
Reparameterizing the LoRA update as ΔW = BCN_s^⊤ through a frozen null basis preserves layer outputs on reasoning states by construction while training with standard optimizers and no per-step projection.
NB-LoRA matches vanilla LoRA target-task accuracy (0.846 vs 0.847 average over nine tasks) while retaining post-RL reasoning near pre-SFT levels (63.2% vs 63.8% on Qwen2.5-3B; 80.0% vs 80.1% on 1.5B).
Preservation generalizes to held-out reasoning benchmarks not used in constraint construction (Table 3: 67.1% vs 67.7% post-RL average on 3B from a single math anchor corpus).
NB-LoRA is about 6.3x faster and uses 2.1x less peak CUDA memory than full-model null-space gradient projection (NSCL) at 3B scale (Figure 4).

Strengths

Constraint enforced by reparameterization, not optimization
ΔW = BCN_s^⊤ (Eqs. 4-5) satisfies HΔW^⊤ = 0 for any B, C, removing per-step gradient projection; measured 6.3x runtime and 2.1x memory savings over NSCL on Qwen2.5-3B (Figure 4: 627.7s vs 3,924.9s; 13.1M vs 1,811.9M trainable parameters).
Careful feasibility analysis before committing to the design
Section 3.3 quantifies eigenvalue decay (Figure 2), approximate null-space sizes (Table 1: 34.3% QKV, 45.1% Down, 29.2% Up at 99% energy), and estimation stability via principal angles (Figure 3: similarity stabilizes beyond ~512 anchor examples).
Near-perfect retention with matched adaptation, plus cross-benchmark generalization
Thorough ablations and a sequential multi-task check
Practical recipe with full SFT hyperparameters and prompts

Weaknesses

Narrow empirical scope
Statistical reporting is thin
Numeric inconsistencies between prose and tables
Exact-preservation framing overstates the approximate construction
RL-stage reproducibility gap and anchor-data assumption

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

The paper addresses the RL-then-SFT setting: an LLM post-trained with RL (GRPO) to reason is subsequently adapted by supervised fine-tuning, and the RL-shaped reasoning collapses — under vanilla LoRA, Qwen2.5-3B falls from 63.8% on MATH-lighteval to single digits on several tasks (Table 2). The authors claim that (i) reasoning activations occupy a low-dimensional subspace, leaving a 29-45% approximate null space at a 99% spectral-energy threshold (Section 3.3, Table 1); (ii) reparameterizing the LoRA update as ΔW = BCN_s^⊤ through a frozen null basis preserves layer outputs on reasoning states by construction (Section 3.4); (iii) the method matches LoRA's adaptation, retains reasoning near pre-SFT levels, generalizes to held-out benchmarks, and is far cheaper than full-model gradient projection. Claims are scoped to two models: Qwen2.5-3B (GRPO on MATH-lighteval) and Qwen2.5-1.5B (GRPO on GSM8K).

Methodology Assessment

The core design is clean. Per adapted linear layer, collect input hidden states H from reasoning rollouts, eigendecompose G = H^⊤H (an n×n problem), keep eigenvectors outside the top 99% spectral energy as the frozen basis N_s, and train only B ∈ R^{m×r} and C ∈ R^{r×k} with Adam. Because Q/K/V share inputs, one basis serves three projections (Section 3.2). The feasibility study is the strongest analytical section: spectra decay by orders of magnitude within a few hundred indices (Figure 2); null dimensions average 34.3% (QKV), 45.1% (Down), 29.2% (Up) (Table 1); and

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 5 strengths & 5 weaknesses, analyzed in detail
5 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is NB-LoRA and how does it differ from standard LoRA?

NB-LoRA is a fine-tuning method for language models that were trained with reinforcement learning to reason. Standard LoRA learns an unrestricted low-rank weight update. NB-LoRA first estimates, from a few hundred reasoning examples, a frozen basis of directions that reasoning computations do not use (an approximate null space of the hidden states), then constrains every LoRA update to act only through that basis. Outputs on reasoning inputs therefore stay essentially unchanged while the model learns a new task, using ordinary optimizers with no replay buffer and no gradient projection during training.

Why do RL-tuned reasoning models forget after supervised fine-tuning?

RL post-training only nudges the base model slightly, so its reasoning behavior is a delicate output-distribution shape rather than deeply stored knowledge. A subsequent unconstrained fine-tune can drift far from that distribution: in this paper, vanilla LoRA collapsed a Qwen2.5-3B math-reasoning model from 63.8% MATH accuracy to single digits on several tasks, whereas the proposed constrained method kept it at about 63.2%.

How much reasoning data is needed to build the null basis?

Very little. A principal-angle analysis in the paper shows the estimated null space stabilizes beyond roughly 512 reasoning examples, and ablations show 64 or more examples are enough for reliable retention; only 32 examples showed degradation. The basis is computed once, offline, via an eigendecomposition of an n-by-n matrix, and one basis is shared across the Q, K and V projections that share the same input.

Does preserving reasoning slow down training or hurt new-task performance?

Not in the reported experiments: NB-LoRA matched vanilla LoRA's target-task accuracy (0.846 vs 0.847 average over nine tasks) and trained about 6.3 times faster with 2.1 times less GPU memory than NSCL, a full-model gradient-projection alternative, on a 3B model. The caveat is that these are single-run results on two small Qwen models, with no seed variance reported.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Figures 2, 3 and 5 appear only as axis labels and caption text; exact spectra, similarity curves and ablation curves cannot be independently verified from the extraction.
Figure 4 values were reconstructed from embedded numeric labels in the text rather than from the rendered graphic.
Several citations are dated 2026 (e.g., Ma et al. ReLIFT at ICLR 2026, Mu & Klabjan at ICML 2026); novelty relative to the very latest null-space PEFT literature could not be independently confirmed.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper