Learning in the Machine: Random Backpropagation and the Deep Learning Channel.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Backpropagation is a standard algorithm for training neural networks.
• Random Backpropagation (RBP) replaces transposed forward matrices with random matrices for weight updates.
• RBP removes the need for symmetric weights in physical neural systems.

Purpose of the Study:

• To understand Random Backpropagation (RBP) by connecting it to local learning and learning channels.
• To derive and analyze novel RBP variants, including adaptive and sparse versions.
• To investigate the impact of activation function derivatives and bit precision on RBP performance.

Main Methods:

• Derivation of RBP variants (SRPB, ARBP, sparse RBP, ASRBP) based on local learning principles.
• Computational complexity analysis of the derived RBP variants.
• Empirical evaluation using MNIST and CIFAR-10 benchmark datasets.

Main Results:

• RBP variants exhibit robust performance, closely approximating standard backpropagation.
• The inclusion of activation function derivatives is crucial for effective learning.
• Analysis of low-bit precision for error communication in learning channels.

Conclusions:

• RBP and its variants offer a viable and potentially more hardware-friendly approach to neural network training.
• Mathematical proofs confirm convergence properties for linear and non-linear network structures under specific conditions.
• Further research into learning channel properties and hardware implementations is warranted.