Universum-Inspired Supervised Contrastive Learning | JoVE Visualize

Area of Science:

Computer Science
Machine Learning
Artificial Intelligence

Background:

Mixup is a data augmentation technique that synthesizes new samples via linear interpolation, enhancing model robustness and generalization.
Universum Learning utilizes out-of-class samples to aid target tasks, offering a novel perspective for data augmentation.
Supervised contrastive learning typically requires large batch sizes to effectively learn representations using hard negatives.

Purpose of the Study:

To investigate Mixup's potential for generating in-domain universum samples (samples not belonging to any target class).
To leverage Mixup-induced universum samples as high-quality hard negatives within supervised contrastive learning.
To propose Universum-inspired supervised Contrastive learning (UniCon) and its unsupervised variant (Un-Uni) for improved deep model training.

Main Methods:

The study proposes UniCon, which integrates Mixup to generate universum negatives, pushing them apart from anchor samples of target classes.
The method is extended to an unsupervised setting, resulting in the Unsupervised Universum-inspired contrastive model (Un-Uni).
The effectiveness of learned representations is evaluated using a linear classifier on various datasets.

Main Results:

UniCon achieves state-of-the-art performance on multiple datasets, demonstrating significant improvements over existing methods.
On CIFAR-100, UniCon reaches 81.7% top-1 accuracy, outperforming prior work by 5.2% with a substantially smaller batch size (256 vs. 1024).
Un-Uni also shows superior performance compared to state-of-the-art methods on CIFAR-100.

Conclusions:

Mixup-induced universum samples are effective hard negatives in supervised contrastive learning, reducing the need for large batch sizes.
UniCon and Un-Uni offer novel approaches to data generation and contrastive learning, achieving superior performance.
The proposed methods advance data augmentation strategies and contrastive learning frameworks for deep model training.