Sensitivity - Local index to control chaoticity or gradient globally.

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Machine Learning

Background:

• Neural networks (NNs) often suffer from chaotic dynamics and vanishing gradients, hindering training and performance.
• Controlling information transmission within neurons is crucial for stable and effective network operation.
• Recurrent neural networks (RNNs) and deep feedforward networks face specific challenges with gradient propagation.

Purpose of the Study:

• To introduce a local index, "sensitivity," for neuron-level control of chaoticity and gradients.
• To propose a learning method, "sensitivity adjustment learning" (SAL), to adjust this index.
• To demonstrate SAL's efficacy in controlling network dynamics and mitigating gradient problems.

Main Methods:

• Defined "sensitivity" as the gradient magnitude of a neuron's output with respect to its inputs.
• Developed SAL to adjust the time average of sensitivity to 1.0 in each neuron.
• Applied SAL to RNNs and deep feedforward NNs, evaluating its impact on chaoticity and gradient vanishing.

Main Results:

• SAL effectively controls chaoticity in RNNs by moderating information transmission.
• SAL resolves the vanishing gradient problem in deep NNs and RNNs during backpropagation through time (BPTT).
• Log-sensitivity correlated with the maximum Lyapunov exponent in RNNs, and SAL improved learning performance over manual spectral radius tuning.

Conclusions:

• SAL provides a robust, local mechanism for global control of neural network dynamics and learning.
• The sensitivity index and SAL offer a novel approach to enhance the stability and trainability of deep and recurrent neural networks.
• SAL demonstrates significant advantages in preventing gradient loss and improving learning performance in complex NN architectures.