A Distillation Approach to Transformer-Based Medical Image Classification with Limited Data

Area of Science:

• Computer Science
• Artificial Intelligence
• Medical Imaging

Background:

• Transformer deep learning models offer flexibility but struggle with small datasets in image classification.
• Distillation techniques show promise for improving transformer performance on limited data.
• The specific impact of distillation on transformer-based classification accuracy requires further investigation.

Purpose of the Study:

• To investigate the effect of distillation techniques on the classification performance of transformer deep learning models using limited data.
• To compare the performance of transformer models with and without distillation on a brain MRI dataset.
• To analyze the impact of distillation on accuracy and training time.

Main Methods:

• Utilized transformer models ViTx32 and ViTx16 (without distillation) and DeiT and BeiT (with distillation).
• Trained and tested models on a four-class brain MRI image dataset.
• Evaluated classification accuracy and training times for each architecture.

Main Results:

• DeiT and BeiT models with distillation achieved performance gains of 2.2% and 1% over ViTx16, respectively.
• Distillation techniques improved the detection of non-patient individuals by approximately 4%.
• Comparative analysis of training times for all investigated architectures was conducted.

Conclusions:

• Distillation techniques substantially enhance classification accuracy in transformer deep learning models when applied to limited datasets.
• Transformer models incorporating distillation are recommended for medical imaging and other fields utilizing flexible models with scarce data.
• The findings support the broader application of distillation in data-limited deep learning scenarios.