Training sparse convolutional deep predictive coding networks with attention

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Deep predictive coding networks (DPCNs) are explored for unsupervised feature learning.
• Self-supervised learning (SSL) methods leverage data structure without explicit labels.
• Attention mechanisms enhance model focus on relevant information.

Purpose of the Study:

• To propose a novel training methodology for sparse convolutional deep predictive coding networks with attention (DPCN-SCA).
• To adapt DPCN architecture for visual memory and improved feature interpretability.
• To evaluate the performance of DPCN-SCA on benchmark datasets.

Main Methods:

• Modified traditional DPCN equations to incorporate a top-down flow, inspired by autoencoders.
• Employed a bidirectional architecture and attention mechanisms within a sparse convolutional framework.
• Utilized benchmark datasets to validate the DPCN-SCA model, similar in depth to AlexNet.

Main Results:

• DPCN-SCA demonstrated feature extraction patterns similar to supervised CNNs, focusing on details in early layers and contours in deeper layers.
• Achieved over a 20% absolute gain in classification accuracy compared to unsupervised sparse coding baselines at 1-5% sparsity levels.
• Introduced a new visualization technique to observe attention patterns in deep layers by projecting activations back to the input space.

Conclusions:

• The proposed DPCN-SCA training methodology is effective for unsupervised feature learning and classification.
• The adapted DPCN architecture offers benefits for visual memory and interpretability.
• DPCN-SCA significantly advances unsupervised sparse coding performance, particularly under high sparsity constraints.