Shallow Univariate ReLU Networks as Splines: Initialization, Loss Surface, Hessian, and Gradient Flow Dynamics

Area of Science:

• Machine Learning
• Artificial Intelligence
• Neural Network Theory

Background:

• Understanding neural network (NN) learning dynamics is challenging due to parameter-function opacity and inherent symmetries.
• Permutation and weight scale symmetries create redundant degrees of freedom, obscuring the relationship between NN parameters and their output functions.

Purpose of the Study:

• To simplify the analysis of neural network learning dynamics and inductive bias.
• To develop a more transparent and intuitive understanding of neural network parameterization and function representation.

Main Methods:

• Reparameterized Rectified Linear Unit (ReLU) neural networks as continuous piecewise linear splines by taking a quotient with respect to the weight scale symmetry group.
• Studied learning dynamics in shallow univariate ReLU networks using this spline representation.
• Analyzed the structure of the loss surface, including critical points, fixed points, and the Hessian spectrum.

Main Results:

• The spline representation provides a clear view of the loss surface structure and learning dynamics in shallow univariate ReLU networks.
• Standard weight initializations lead to flat initial functions, which, along with overparametrization and initial weight scale, dictate implicit regularization.
• Initialization scale critically controls implicit regularization, consistent with kernel-based arguments.

Conclusions:

• Removing the weight scale symmetry simplifies proofs and provides new insights into neural network behavior.
• The quotiented spline-based approach offers a transparent and intuitive framework for understanding neural networks.
• This approach is expected to extend naturally to multivariate and deep networks, serving as a foundational tool for future research.