MAN++: Scaling Momentum Auxiliary Network for Supervised Local Learning in Vision Tasks

Area of Science:

• Deep Learning
• Computer Vision
• Machine Learning

Background:

• End-to-end backpropagation is the standard deep learning training method but has limitations like high memory usage and poor biological plausibility.
• Supervised local learning offers an alternative by training network blocks independently, but faces accuracy challenges due to isolated gradients.
• Existing methods struggle to bridge the accuracy gap between local and end-to-end training paradigms.

Purpose of the Study:

• To introduce Momentum Auxiliary Network++ (MAN++), a novel framework designed to improve supervised local learning.
• To enable scalable training of deep neural networks with reduced computational overhead.
• To achieve accuracy on par with end-to-end backpropagation while mitigating its drawbacks.

Main Methods:

• MAN++ implements a lightweight parameter-space transfer between adjacent network blocks using exponential moving averages (EMA) of parameters.
• A learnable scaling bias is introduced to address feature mismatches and stabilize the EMA parameter transfer.
• The framework was evaluated across diverse vision tasks including image classification, object detection, and semantic segmentation on multiple architectures.

Main Results:

• MAN++ successfully bridges the accuracy gap often seen in supervised local learning, achieving results comparable to end-to-end backpropagation.
• The proposed method significantly reduces GPU memory consumption compared to traditional end-to-end training.
• Experiments demonstrated the framework's effectiveness and scalability across various deep learning architectures and computer vision tasks.

Conclusions:

• MAN++ presents a practical and effective alternative to conventional backpropagation for training deep neural networks.
• The framework offers a scalable solution for supervised local learning, enhancing efficiency without sacrificing accuracy.
• MAN++ provides valuable insights into improving gradient propagation and contextual information transfer in localized deep learning training.