Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing Its Gradient Estimator Bias

Area of Science:

• Computational neuroscience
• Machine learning
• Deep learning

Background:

• Equilibrium Propagation (EP) is a biologically-inspired algorithm for training recurrent neural networks using local learning rules.
• EP offers theoretical guarantees and is a promising approach for learning-capable neuromorphic systems.
• Standard EP implementations struggle with complex visual tasks due to limitations in gradient estimation.

Purpose of the Study:

• To identify and rectify the bias in Equilibrium Propagation's gradient estimate that hinders its performance on challenging visual tasks.
• To enhance Equilibrium Propagation's scalability and effectiveness for training deep convolutional neural networks.
• To generalize Equilibrium Propagation for use with cross-entropy loss functions.

Main Methods:

• Investigated the bias in EP's gradient estimate arising from finite nudging.
• Implemented symmetric nudging (positive and negative nudges) to reduce this bias.
• Generalized EP to handle cross-entropy loss, moving beyond squared error.

Main Results:

• Correcting the gradient bias enabled training of deep convolutional neural networks.
• Symmetric nudging significantly reduced the gradient bias.
• Achieved 11.7% test error on CIFAR-10, a substantial improvement over standard EP (86% error).
• Attained 13.2% test error on an architecture with unidirectional connections.

Conclusions:

• The identified bias in EP's gradient estimate was the key limitation for complex tasks.
• Symmetric nudging and cross-entropy generalization make EP a viable method for deep neuromorphic systems.
• EP is presented as a compelling, biologically-plausible alternative for computing error gradients in deep learning.