Discrete Infomax Codes for Supervised Representation Learning

Area of Science:

• Machine Learning
• Computer Vision
• Information Theory

Background:

• High-dimensional data, such as images, requires effective encoders for compact and informative representations.
• Learning such encoders is crucial for subsequent data processing tasks.

Purpose of the Study:

• To present a novel model, Discrete Infomax Codes (DIMCO), for learning discrete representations of data.
• To maximize mutual information between learned codes and class labels while encouraging code independence.

Main Methods:

• Training a probabilistic encoder to generate k-way d-dimensional codes.
• Maximizing mutual information between codes and ground-truth labels.
• Applying regularization to promote statistical independence within codewords.

Main Results:

• Demonstrated that the infomax principle unifies existing loss functions like cross-entropy.
• Showed that shorter codes learned by DIMCO reduce overfitting in few-shot classification.
• Validated the implicit task-level regularization effect of DIMCO through experiments.

Conclusions:

• DIMCO learns efficient codes in terms of memory and retrieval time compared to existing methods.
• The model provides a principled approach to representation learning for high-dimensional data.