Using random weights to train multilayer networks of hard-limiting units

Area of Science:

• Artificial Intelligence
• Machine Learning
• Neural Networks

Background:

• Conventional backpropagation algorithm is unsuitable for training neural networks with hard-limiting units due to non-differentiable activation functions.
• The probability of a hard-limiting unit's output is a continuously differentiable function of network weights when weights are random variables with smooth distributions.

Purpose of the Study:

• To present a novel gradient descent algorithm for training multilayer feedforward networks with hard-limiting units.
• To address the limitations of standard backpropagation in scenarios involving non-differentiable activation functions.

Main Methods:

• Development of a backpropagation-like algorithm by leveraging the differentiable nature of hard-limiting unit probabilities.
• Estimation of gradients for the hard-limiting case, introducing a computational overhead compared to standard backpropagation.
• Analysis of upper bounds for the gradient estimation penalty.

Main Results:

• The proposed algorithm demonstrates a computational framework similar to standard backpropagation.
• Performance evaluation through two examples indicates comparable results between the new algorithm for hard-limiting units and conventional backpropagation for sigmoidal units.
• Quantification of the additional computational expense associated with gradient estimation.

Conclusions:

• The developed gradient descent algorithm effectively trains multilayer feedforward networks with hard-limiting units.
• The algorithm offers a viable alternative to standard backpropagation when dealing with non-differentiable activation functions.
• The study provides insights into the performance and computational trade-offs of training hard-limiting neural networks.