TransMed: Transformers Advance Multi-Modal Medical Image Classification.

Area of Science:

• Medical Image Analysis
• Artificial Intelligence
• Computer Vision

Background:

• Convolutional Neural Networks (CNNs) excel at local feature extraction in medical imaging but struggle with long-range dependencies.
• Transformers show promise in computer vision but require large datasets, a limitation in medical imaging.
• Multi-modal medical images possess crucial long-range dependencies that can improve deep learning model performance.

Purpose of the Study:

• To develop a novel deep learning architecture for multi-modal medical image classification.
• To address the limitations of existing models in handling long-range dependencies in medical imaging.
• To leverage the strengths of both CNNs and transformers for improved medical image analysis.

Main Methods:

• Proposed TransMed, a hybrid architecture integrating CNNs and transformers.
• Utilized CNNs for efficient low-level feature extraction.
• Employed transformers to establish long-range dependencies between image modalities.

Main Results:

• Achieved significant accuracy improvements of 10.1% and 1.9% on parotid gland tumor and knee injury classification datasets, respectively.
• Outperformed state-of-the-art CNN-based models in multi-modal medical image classification.
• Demonstrated the effectiveness of the hybrid CNN-transformer approach on limited medical imaging datasets.

Conclusions:

• TransMed offers a promising solution for multi-modal medical image classification by effectively combining CNN and transformer advantages.
• The proposed model shows potential for broad application across various medical image analysis tasks.
• This work represents the first application of transformers to multi-modal medical image classification.