Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural network success often requires extensive labeled data, which is costly and time-consuming to acquire.
• Limited labeled data presents a significant challenge in many real-world machine learning applications.
• Representation learning aims to derive meaningful features from data, crucial for model performance.

Purpose of the Study:

• To review recent advancements in unsupervised and semi-supervised representation learning methods.
• To categorize and illustrate the interplay between various models in this domain.
• To identify future research directions for unifying learning paradigms and improving representation quality.

Main Methods:

• Reviewing principles of learning transformation-equivariant, disentangled, self-supervised, and semi-supervised representations.
• Categorizing a wide spectrum of models, including autoencoders and generative adversarial networks (GANs).
• Analyzing the integration of unsupervised and semi-supervised learning approaches.

Main Results:

• Unsupervised and semi-supervised generative models leverage unlabeled data for enhanced representations.
• Foundational principles like transformation equivariance underpin recent progress.
• Existing models like autoencoders and GANs are expanded by exploring unlabeled data distributions.

Conclusions:

• Intrinsic connections between unsupervised and semi-supervised learning are identified.
• Future work should bridge the gap between unsupervised equivariance and supervised invariance.
• Unifying unsupervised pretraining with supervised finetuning and exploring self-supervised regularization are key future directions.