Information-Maximized Soft Variable Discretization for Self-Supervised Image Representation Learning

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Self-supervised learning (SSL) is vital for vision foundation models, leveraging unannotated data for enhanced downstream tasks.
• Existing SSL methods often require complex contrastive learning strategies.
• Developing efficient and interpretable image representation learning techniques is an ongoing challenge.

Purpose of the Study:

• Introduce Information-Maximized Soft Variable Discretization (IMSVD), a novel SSL approach for image representation learning.
• Develop an information-theoretic objective function for learning transform-invariant, non-trivial, and redundancy-minimized features.
• Provide a non-contrastive SSL method that statistically matches contrastive learning performance.

Main Methods:

• IMSVD employs soft discretization of latent variables to estimate probability distributions within training batches.
• An information-theoretic objective guides the learning process using information measures.
• A joint-cross entropy loss function is derived to minimize feature redundancy.

Main Results:

• IMSVD demonstrates effectiveness across various downstream tasks, improving both accuracy and efficiency.
• The method achieves performance comparable to contrastive learning approaches despite being non-contrastive.
• Variable-level explainability is offered by the embedding features optimized through IMSVD.

Conclusions:

• IMSVD presents a novel and effective self-supervised learning method for image representation.
• The approach offers advantages in feature redundancy reduction, efficiency, and explainability.
• IMSVD shows potential for adaptation to other machine learning paradigms.