Regularized Least Squares Training of Quadratic Neural Networks with Applications to System Identification
Luis Rodrigues, Zachary Yetman Van Egmond, Mohammad R. Amiri Fard
cs.LG · cs.SY · eess.SY
Quick verdict
Closed-form least-squares training of quadratic NNs with provable cost lower bounds; coherent theory, but validation rests on a single small example.
Summary
The paper extends the authors' prior least-squares training of two-layer quadratic neural networks (QNNs) to include a 1-norm regularization term. Building on the SDP-lifting convex reformulation of Bartan and Pilanci (Lemma 2.1, ref. [4]), the authors drop the positive-semidefiniteness (LMI) constraints and show (Theorems 3.3-3.4) that the resulting nuclear-norm-regularized problems yield lower bounds on the convex training cost; the bound is tight for beta=0, where the least-squares solution equals the global optimum. For beta>0, Theorem 4.1 gives closed-form Tikhonov-type solutions whose evaluation yields lower bounds (38), (40) and an upper bound (39) on the regularized cost, and the relaxation is connected to nuclear norm minimization (Remark 3.6). An analytical sensitivity of the weights to output errors, delta-theta = -(H^T H)^{-1} H^T e_y, is also provided. The method is demonstrated on the DaISy flexible-robot-arm dataset (N=1024 samples, 122 training points, 2 regressor inputs), where the beta=0 least-squares solution coincides with the convex program solution (Fig. 1) and the bounds track the optimal cost across beta values (Fig. 2). Main limitations: validation is a single small example with training-objective comparisons only (no held-out prediction error), the claimed computational-time advantage over backpropagation is never measured, for beta>0 the method bounds the cost rather than returning optimal network weights, and scalability to the claimed deep-QNN setting via monomial lifting is untested.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- For beta=0 with a>0, c>0, b^2-4ac>=0, the least-squares solution theta* = (H^T H)^{-1} H^T y is the global optimum of the convex QNN training problem, making the induced lower bound tight.
- For beta>0, closed-form regularized least-squares solutions (37) yield valid lower bounds (38), (40) and an upper bound (39) on the optimal value of the regularized training problem.
- The relaxed bound computation is equivalent to nuclear norm minimization (Theorems 3.3-3.4, Remark 3.6), connecting QNN training to low-rank regularization.
- The sensitivity of the trained weights to output errors has the closed form delta-theta = -(H^T H)^{-1} H^T e_y.
- Via monomial lifting of the inputs (ref. [6]), this constitutes the first least-squares training approach for deep neural networks that treats hidden-layer weights as decision variables.
Strengths
Genuinely closed-form training with optimality guarantee
For beta=0 and activation conditions a>0, c>0, b^2-4ac>=0, Section IV-A gives theta* = (H^T H)^{-1} H^T y (eq. 29), and Theorem 3.3 proves this equals the global optimum of the convex reformulation (4); Fig. 1 reports the least-squares and SDP curves coinciding on the robot-arm data, an unusually strong guarantee for neural network training.
Careful, layered bound construction
Section III builds the lower bound in verifiable steps: Lemma 3.1 constructs structured PSD decompositions, Lemma 3.2 (Schur complement, from [8]) yields trace/Frobenius inequalities, and Theorems 3.3-3.4 chain these into nuclear-norm bounds; validity conditions (a>0, c>0, b^2-4ac>=0) are stated explicitly rather than hidden.
Principled bridge to nuclear norm minimization
Honest reporting of bound behavior
Closed-form sensitivity expression
Weaknesses
Minimal empirical validation
Headline efficiency claim is unmeasured
Regularized case yields bounds, not usable optimal weights
Deep-network claim is empirically unsupported
Novelty of the sensitivity result is overstated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can quadratic neural networks be trained in closed form without backpropagation?
According to this paper, yes for the unregularized case: a two-layer quadratic neural network with a quadratic activation and a 2-norm loss reduces to ordinary least squares on lifted quadratic features, with weights theta* = (H^T H)^{-1} H^T y. The authors prove this equals the global optimum of the convex (SDP) reformulation when the activation satisfies a>0, c>0, b^2-4ac>=0, avoiding the local minima of iterative training.
What data is used in the paper's system identification example?
The DaISy (Database for Identification of Systems, KU Leuven) flexible robot arm benchmark: the input is the measured reaction torque and the output is the arm's acceleration under a periodic sine-sweep excitation. The dataset has 1024 samples, of which only the first 122 are used to fit a one-step autoregressive model z_{i+1} = f(z_i, u_i).
Is code available for this paper?
The paper does not mention any code release or repository. However, the core training method is a closed-form least squares formula and the dataset is publicly downloadable from the DaISy website, so the beta=0 experiments are reproducible from the paper's equations alone; the convex-program comparison would require an SDP solver whose settings are not specified.
How does the method handle regularization, and what are its limits?
With a positive regularization coefficient beta the problem is no longer exactly solvable in closed form. The paper instead provides provable lower bounds (and one upper bound) on the optimal training cost via Tikhonov-regularized least squares, plus an unguaranteed approximation used in practice. The lower bound is tight only at beta=0, so for regularized training the method quantifies the cost but does not return certified optimal network weights.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 could not be analyzed; quantitative bound gaps and fit quality are judged from captions and the text alone.
- PDF-to-text extraction garbled several equations (notably (4), (12), (39), (45)); constant factors in the bounds of Theorem 4.1 could not be fully re-derived from the extracted text.
- Review is based on arXiv v1 (2609.17654, 15 Sep 2026); any appendix or supplementary material was not included in the extraction.
- Section V omits the neuron count M, SDP solver, and hardware used for the reference convex solution, which limits assessment of the computational-comparison claim.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.