Deep representation learning using layer-wise VICReg losses.

Area of Science:

• Machine Learning
• Deep Learning
• Computer Vision

Background:

• Deep Neural Networks (DNNs) often struggle with vanishing gradients and initialization sensitivity.
• Training DNNs typically requires large amounts of annotated data, which can be scarce.
• Backpropagation, the standard training method, involves one forward and one backward pass.

Purpose of the Study:

• To present a novel layer-wise training procedure for neural networks.
• To address challenges in training DNNs, particularly with limited annotated data.
• To enhance feature representation learning and classification accuracy.

Main Methods:

• A layer-wise training procedure minimizing Variance-Invariance-Covariance Regularization (VICReg) loss at each layer.
• Utilizing two forward passes with original and augmented data instead of backpropagation.
• Employing a pyramidal network architecture for effective feature extraction.
• Optimizing weights for variance, invariance, and covariance terms for semantic information capture.

Main Results:

• The procedure progressively constructs compact and informative feature spaces.
• Improved classification accuracy on MNIST (7%), EMNIST (16%), Fashion MNIST (1%), and CIFAR-100 (7%) compared to baseline models.
• Learned representations were assessed using clustering quality metrics and few-shot classification tasks.

Conclusions:

• The proposed VICReg layer-wise training enhances DNN performance, especially in low-data regimes.
• This method offers a viable alternative to backpropagation, mitigating common training issues.
• The approach effectively learns robust and informative representations for downstream tasks.