Low-variance Forward Gradients using Direct Feedback Alignment and momentum.

Area of Science:

• Computational Neuroscience
• Machine Learning
• Deep Learning

Background:

• Supervised learning in deep neural networks typically relies on error backpropagation.
• The sequential nature of backpropagation hinders scalability and compatibility with low-power neuromorphic hardware.
• Existing local learning alternatives, such as forward-mode automatic differentiation, exhibit high variance, impacting convergence in large networks.

Purpose of the Study:

• To develop a novel, local learning algorithm for deep neural networks that overcomes the limitations of backpropagation.
• To address the high variance issue observed in existing forward-mode gradient techniques.
• To enable efficient online learning algorithms suitable for neuromorphic systems.

Main Methods:

• Proposed the Forward Direct Feedback Alignment (FDFA) algorithm.
• Combined Activity-Perturbed Forward Gradients with Direct Feedback Alignment (DFA).
• Incorporated momentum to enhance learning dynamics.

Main Results:

• Demonstrated theoretically and empirically that FDFA achieves lower variance compared to forward gradient techniques.
• Showcased faster convergence rates for FDFA.
• Achieved improved performance relative to other local alternatives to backpropagation.

Conclusions:

• FDFA offers a promising local learning alternative to backpropagation for deep neural networks.
• The algorithm's reduced variance and improved performance make it suitable for efficient online learning.
• Opens new avenues for developing neuromorphic-compatible learning algorithms.