Self-supervised pre-training with contrastive and masked autoencoder methods for dealing with small datasets in deep...

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Computer Vision

Background:

• Deep learning models require large annotated datasets for medical imaging tasks, which are often scarce due to annotation complexity and data access limitations.
• Self-supervised learning (SSL) offers a solution by pre-training models on unannotated data, reducing the need for extensive manual annotations.
• Contrastive learning is a popular SSL method in medical imaging, but masked autoencoders are emerging as a promising alternative.

Purpose of the Study:

• To compare the performance of a state-of-the-art masked autoencoder approach (SparK) against contrastive learning methods for self-supervised pre-training on medical images.
• To evaluate the robustness of these SSL methods when fine-tuning on small, annotated datasets for computed tomography (CT) classification tasks.
• To determine the most effective pre-training strategy for medical imaging tasks with limited annotated data.

Main Methods:

• Pre-training convolutional neural networks (CNNs) on a large, unannotated CT image dataset using both contrastive learning and the SparK masked autoencoder approach.
• Fine-tuning the pre-trained models on multiple CT classification tasks using progressively smaller annotated datasets.
• Analyzing the impact of reduced training dataset size on the performance of each pre-training method.

Main Results:

• The SparK masked autoencoder approach demonstrated superior robustness compared to contrastive learning methods when fine-tuning on smaller annotated datasets.
• Performance degradation due to dataset size reduction was less pronounced with SparK than with contrastive learning.
• The choice of self-supervised pre-training method significantly influences model performance under data-scarce conditions.

Conclusions:

• The SparK masked autoencoder is a highly effective self-supervised pre-training method for medical imaging tasks, particularly when dealing with limited annotated data.
• SparK offers a more robust solution than contrastive learning for fine-tuning deep learning models in data-limited medical imaging scenarios.
• The findings support the adoption of SparK for developing reliable deep learning applications in medical imaging where annotated datasets are challenging to acquire.