Effect of nonlinear transformations on correlation between weighted sums in multilayer perceptrons

Area of Science:

• Artificial Intelligence
• Machine Learning
• Neural Networks

Background:

• Nonlinear transformations pose significant challenges in analyzing multilayer perceptron (MLP) properties.
• Understanding information processing within MLPs is crucial for advancing deep learning.

Purpose of the Study:

• To investigate the effect of nonlinear transformations on the correlations of random variables in MLPs.
• To demonstrate how sigmoidal transformations impact information redundancy in hidden neurons.

Main Methods:

• Mathematical proof demonstrating the decrease in correlation coefficients for jointly Gaussian random variables under nonlinear transformations.
• Approximation of continuous nonlinear transformations using piecewise linear functions.
• Analysis of weighted sums to hidden neurons as asymptotically jointly Gaussian random variables.

Main Results:

• Continuous nonlinear transformations, approximated by piecewise linear functions, decrease the correlation coefficient between jointly Gaussian random variables.
• Sigmoidal transformations, approximated piecewise linearly, reduce correlations among weighted sums to hidden neurons.
• This reduction in correlation implies a decrease in information redundancy.

Conclusions:

• Sigmoidal transformations, when used as activation functions in MLPs, effectively reduce information redundancy among hidden neurons.
• The findings provide a theoretical basis for understanding the information processing capabilities of MLPs.
• This work contributes to the analysis of deep learning models by addressing the complexities of nonlinear activations.