A global gradient-noise covariance expression for stationary real Gaussian inputs

Area of Science:

• Artificial Intelligence
• Machine Learning
• Signal Processing

Background:

• Supervised parameter adaptation in artificial neural networks commonly uses the least-mean-square (LMS) algorithm, a form of gradient descent.
• This research focuses on neural models that are linear with respect to their adaptable parameters.

Purpose of the Study:

• To derive an expression for gradient-noise covariance in linear neural networks with correlated Gaussian inputs.
• To develop a recursive expression for the weight-error correlation matrix.
• To compare findings with the complex LMS algorithm.

Main Methods:

• Derivation of gradient-noise covariance under assumptions of real, stationary, Gaussian, and partially correlated input samples.
• Analysis of the relationship between gradient correlation, input correlation, and gradient-noise covariance.
• Derivation of a recursive expression for the weight-error correlation matrix using the gradient-noise covariance.

Main Results:

• An expression for gradient-noise covariance is derived, relating input and gradient correlation matrices.
• It is explained why gradient noise correlates maximally with the steepest principal axis and minimally with the smallest curvature.
• A recursive expression for the weight-error correlation matrix is obtained.

Conclusions:

• The study provides a deeper understanding of gradient noise characteristics in linear neural networks.
• The derived expressions offer insights into the behavior of adaptive algorithms and parameter adaptation.
• The findings facilitate comparisons with existing algorithms like the complex LMS algorithm.