Analyzing Transfer Learning of Vision Transformers for Interpreting Chest Radiography

Area of Science:

• Artificial Intelligence
• Medical Imaging Analysis
• Deep Learning

Background:

• Deep learning models require large datasets, which are often scarce in medical imaging.
• Transfer learning, using models pre-trained on natural images, is a common workaround.
• Transformers, successful in NLP and image classification, offer powerful transfer learning capabilities.

Purpose of the Study:

• To investigate the effectiveness of pre-trained natural image transformers for medical image analysis.
• To compare the performance of vision transformers against traditional convolutional neural networks (CNNs) in medical imaging.

Main Methods:

• Utilized pre-trained vision transformers for transfer learning on medical imaging datasets (CheXpert, pediatric pneumonia).
• Employed standard CNN models (VGGNet, ResNet) as baseline comparisons.
• Analyzed model representations and performance metrics to evaluate transfer learning efficacy.

Main Results:

• Transfer learning using pre-trained vision transformers yielded superior results compared to pre-trained CNNs.
• Vision transformers demonstrated enhanced transferability to medical imaging tasks.
• The study identified improved performance metrics when employing transformers.

Conclusions:

• Pre-trained natural image transformers exhibit greater transfer ability in medical imaging than traditional CNNs.
• Vision transformers represent a promising approach for improving deep learning performance with limited medical datasets.
• Further research into transformer architectures for medical AI is warranted.