Associated Learning: Decomposing End-to-End Backpropagation Based on Autoencoders and Target Propagation

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Backpropagation (BP) is fundamental to deep learning but suffers from inefficiencies due to backward locking, hindering parallel processing.
• The sequential nature of BP limits the simultaneous updating of weights across different network layers.

Purpose of the Study:

• Introduce Associated Learning (AL), a novel deep learning structure designed to overcome BP's limitations.
• Enable independent and simultaneous parameter learning in different network layers to enhance training throughput.

Main Methods:

• Modularize deep networks into smaller components, each with a local, mutually independent objective.
• Implement a pipeline structure for parallel processing of layer weight updates.
• Compare the training time complexity of AL with Backpropagation and Stochastic Gradient Descent (SGD).

Main Results:

• Associated Learning (AL) reduces training time complexity from O(N*L) to O(N+L), where N is the number of training instances and L is the number of hidden layers.
• AL achieves predictive accuracies comparable to traditional BP methods, despite most parameters not directly interacting with the target variable.
• Demonstrate the scalability and predictive power of the AL method through experimental validation.

Conclusions:

• Associated Learning (AL) presents a scalable and effective alternative to Backpropagation for training deep learning models.
• Further research into AL's hyperparameter settings and network designs is warranted to optimize its performance.
• The AL framework offers potential for inspiring new deep learning architectures and efficient training methodologies.