PAC-Bayesian Bounds for Learning Partially Observed Stochastic Linear Time-Invariant State-Space Systems with Inputs and Sub-Gaussian Noise
Mihaly Petreczky, Mohamad Al Ahdab, John Leth
cs.LG · math.OC
Quick verdict
Rigorous PAC-Bayesian bounds for LTI state-space learning with sub-Gaussian noise; rates are conservative and constants data-dependent, but claims look sound.
Summary
This paper derives PAC-Bayesian generalization bounds for discrete-time, partially observed stochastic linear time-invariant (LTI) state-space systems with inputs, where data are generated by a stationary innovation-form process driven by i.i.d. sub-Gaussian noise. Predictors form a uniformly bounded, Schur-parametrized class of stable LTI systems, and accuracy is measured by steady-state prediction error under Lipschitz or quadratic losses. The main result (Theorem 3.1, Corollary 3.2) states that, with probability at least 1-2δ over a single trajectory of length N, the KL-regularized average generalization gap for any posterior is bounded by the empirical error plus a complexity term of order O((ln N)^r/√N), with r = 0, 1, 3/2 for bounded noise, Lipschitz and quadratic losses respectively; constants depend on ℓ1 norms and mixing coefficients of the Markov parameters of predictor and generator, and on the noise's sub-Gaussian norm. Proofs combine an infinite-past empirical-error decomposition, concentration inequalities for weakly dependent processes, noise truncation, and a high-probability reduction of quadratic loss to a truncated Lipschitz loss. Corollary 3.4 further yields empirical, oracle, parameter-estimation and H2 error bounds of order O((ln N)^(r+1)/√N) for MEP, MAP and single-draw learners from a Gibbs posterior, covering PEM, maximum-likelihood and Bayesian identification. Limitations: rates are more conservative than algorithm-specific finite-sample analyses; constants involve unknown data-generator quantities; parameter bounds require strong convexity, realizability and persistence of excitation; no numerical experiments appear in the provided text.
Review complete
Assessment at a glance
3
risk flags identified
● high 0
● medium 2
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- With probability at least 1−2δ over a single trajectory of length N, for any posterior over stable LTI predictors the KL-regularized average generalization gap is O((ln N)^r/√N), with r = 0, 1, 3/2 for bounded noise, Lipschitz loss and quadratic loss respectively (Theorem 3.1, Corollary 3.2).
- The same framework yields empirical, oracle, parameter-estimation and H2 error bounds of order O((ln N)^(r+1)/√N) for MEP, MAP and single-draw learners computed from a Gibbs posterior (Corollary 3.4, Table 1).
- These bounds apply to a broad class of identification algorithms — regularized PEM, maximum likelihood, Bayesian MCMC, regularized FIR — because all are instances of Gibbs-posterior learning (Remarks D.5-D.8).
- The results strictly improve prior dynamical PAC-Bayes bounds: unbounded sub-Gaussian noise versus bounded in [8, 11], ln(1/δ) confidence versus O(1/δ) in [12], and refined constants even for bounded signals (Remark D.11).
Strengths
Strict extension of prior PAC-Bayes results for dynamical systems
Relative to Eringis et al. [8, 11] (bounded noise, no inputs/partial-observation generality) and the Rényi-divergence bound [12] (O(1/δ) confidence, quadratic loss only), Theorem 3.1 handles unbounded sub-Gaussian noise, inputs, quadratic loss, and achieves ln(1/δ) confidence dependence; Remark D.11 shows concrete constant refinements over [11, 8] even in the bounded case.
Algorithm-agnostic coverage of standard identification practice
Remarks D.5-D.8 instantiate the Gibbs posterior for regularized FIR estimation (via the ridge form of [34]), Gaussian maximum likelihood in innovation form ([4, Ch. 7]), Bayesian MCMC identification [31], and penalized PEM, so the single Corollary 3.4 (MEP/MAP/single draw) delivers bounds for all of them, not one estimator at a time.
System-theoretic, interpretable constants
Complete pipeline from generalization gap to system distance
Weaknesses
Constants depend on unknown data-generator quantities
Conservative rates, possibly proof-induced
Stacked structural assumptions for the strongest results
No numerical validation in the provided text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv:2609.08740, cs.LG/math.OC) develops PAC-Bayesian generalization theory for a widely used model class: discrete-time, partially observed stochastic LTI state-space systems with inputs, learned from a single stationary trajectory of length N whose innovation noise is i.i.d. sub-Gaussian (Assumption 2.2). Predictors are stable LTI systems under a uniformly bounded Schur-type parametrization (Assumption 2.1: ∥Âθ^k∥2 ≤ Mγ^k, matrix norms ≤ C). The headline claim (Theorem 3.1, Corollary 3.2): with probability ≥ 1−2δ, for every posterior ρ over Θ, the KL-regularized average generalization gap is bounded by the empirical error plus a complexity term of order O((ln N)^r/√N), with r = 0 (bounded noise), r = 1 (Lipschitz loss), r = 3/2 (quadratic loss). Corollary 3.4 upgrades this to empirical, oracle, parameter-estimation and H2 bounds of order O((ln N)^(r+1)/√N) for three learner types — MEP, MAP, single draw — taken from a Gibbs posterior. The exact statement of Theorem 3.1 (bound (8)) falls in a sampling gap of the provided text; its content is reconstructed from Corollaries 3.2/3.4, the constants (9), and the proof-intuition paragraph, so confidence is medium.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 3 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper prove?
That for LTI state-space predictors learned from a single noisy (sub-Gaussian) trajectory of length N, the KL-regularized gap between true and empirical prediction error is, with probability at least 1−2δ, of order (ln N)^r/√N — r = 0 for bounded noise, 1 for Lipschitz loss, 3/2 for quadratic loss. Learners derived from a Gibbs posterior (mean, mode, or one random draw) additionally get oracle, parameter-estimation and H2 error bounds of order (ln N)^(r+1)/√N.
Which system identification algorithms do the bounds cover?
Any algorithm interpretable as the mean (MEP), the mode (MAP), or a random draw from a Gibbs posterior proportional to exp(−λ·empirical error) times a prior — including regularized prediction-error methods, maximum likelihood for Gaussian innovation-form state-space models, Bayesian MCMC identification, and regularized FIR estimation.
Does this give PAC-Bayesian bounds for RNNs?
Not directly. The authors argue LTI systems are a subclass of recurrent neural networks, so the techniques could be a first step toward RNN bounds; but no RNN theorem is proven here, and earlier RNN-flavored work (Eringis et al.) required bounded signals, which this paper removes for the LTI case.
Is there code or experimental data for this paper?
None appear in the provided text; the paper is purely theoretical. Its constants also depend on unknown properties of the data-generating system (stability constants and noise level), so the results are guarantees rather than directly computable numbers.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The complete statement of Theorem 3.1 (bound (8)) falls inside a sampling gap of the provided text; its content is reconstructed from Corollaries 3.2 and 3.4, the constant definitions (9), and the proof-intuition paragraph.
- Parts of Appendix D fall inside sampling gaps — notably the end of the MAP derivation in D.2 and part of D.3 (parameter-error proof); confidence-level claims for MAP rely on the visible statement of Corollary 3.4 rather than the full proof chain.
- Mathematical notation is heavily mangled by PDF extraction (norms, subscripts, radicals), so constant-by-constant verification of the appendix chains was not possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.