Reverse Differentiation via Predictive Coding.

Area of Science:

• Neuroscience and Artificial Intelligence (AI)
• Computational Neuroscience
• Deep Learning

Background:

• Deep learning utilizes artificial neural networks inspired by brain structures.
• Backpropagation (BP) is a key algorithm in neural networks but lacks biological plausibility.
• Biologically plausible learning methods, like predictive coding (PC), are being explored as alternatives.

Purpose of the Study:

• To address the lack of biologically plausible algorithms that exactly replicate backpropagation (BP) on complex neural network models.
• To generalize predictive coding (PC) and zero-divergence inference learning (Z-IL) for exact BP replication.
• To bridge the gap between neuroscience and deep learning research.

Main Methods:

• Generalizing predictive coding (PC) and zero-divergence inference learning (Z-IL) by defining them on computational graphs.
• Demonstrating the ability of the generalized method to perform exact reverse differentiation.
• Developing a novel algorithm equivalent to BP for parameter updates in any neural network.

Main Results:

• The proposed generalized PC/Z-IL algorithm exactly replicates the weight updates of backpropagation (BP) on any neural network.
• This represents the first biologically plausible algorithm to achieve exact equivalence with BP for complex models.
• The method provides a novel local and parallel implementation of BP.

Conclusions:

• The developed algorithm offers a biologically plausible alternative to backpropagation (BP) for deep learning.
• This work facilitates a stronger connection between artificial intelligence and neuroscience research.
• The findings pave the way for more brain-inspired and efficient AI systems.