Effective methods and framework for energy-based local learning of deep neural networks.

Area of Science:

• Neuroscience
• Artificial Intelligence
• Machine Learning

Background:

• Artificial neural networks (ANNs) often use biologically implausible backpropagation for training.
• Energy-based models (EBMs) offer a brain-inspired alternative, learning by minimizing energy functions.
• Predictive coding (PC) is an EBM that uses prediction errors but struggles with gradient issues in deep networks.

Purpose of the Study:

• To address gradient explosion and vanishing problems in deep predictive coding networks.
• To improve the training efficiency and stability of brain-inspired learning models.
• To develop a competitive alternative to backpropagation for ANNs.

Main Methods:

• Introduced bidirectional energy to stabilize prediction errors and mitigate gradient explosion.
• Implemented skip connections to resolve gradient vanishing issues.
• Developed a layer-adaptive learning rate (LALR) for enhanced training efficiency.
• Created a Jax-based framework for efficient EBM training.

Main Results:

• Achieved high accuracies: 99.22% (MNIST), 93.78% (CIFAR-10), 83.96% (CIFAR-100), and 73.35% (Tiny ImageNet).
• Performance is comparable to identical networks trained with backpropagation.
• Reduced training time by 50% compared to PyTorch using the new Jax framework.

Conclusions:

• The proposed bidirectional energy-based model effectively overcomes PC's limitations in deep networks.
• This brain-inspired approach offers a viable and efficient alternative to backpropagation.
• The developed framework accelerates EBM training, demonstrating practical advantages.