Stable invariant models via Koopman spectra.

Area of Science:

• Machine Learning
• Deep Learning
• Dynamical Systems

Background:

• Weight-tied models, including deep equilibrium models (DEQs), are gaining traction in neural network development.
• DEQs leverage weight-tying to represent infinitely deep networks but require iterative solutions and assume convergence to a fixed point.

Purpose of the Study:

• Introduce the stable invariant model (SIM) as a novel class of deep models.
• Extend the dynamics of DEQs to converge to more general invariant sets, not just fixed points.
• Provide a new perspective on stable dynamics in deep learning.

Main Methods:

• Represent model dynamics using the spectra of Koopman and Perron-Frobenius operators.
• Derive two variants of stable invariant models (SIMs) based on this spectral representation.
• Develop an implementation of SIMs trainable like standard feedforward models.

Main Results:

• The spectral analysis of Koopman and Perron-Frobenius operators provides insights into stable dynamics, approximating DEQs.
• Two distinct SIM variants were derived and implemented.
• Empirical experiments demonstrated that SIMs achieve performance comparable or superior to DEQs across various learning tasks.

Conclusions:

• Stable invariant models (SIMs) present a promising extension of deep equilibrium models (DEQs).
• The spectral approach offers a novel way to analyze and design stable deep learning dynamics.
• SIMs show competitive or improved performance, suggesting their potential for future deep learning applications.