O-Net: A Novel Framework With Deep Fusion of CNN and Transformer for Simultaneous Segmentation and Classification

Area of Science:

• Deep learning applications in medicine
• Medical image analysis and segmentation
• Artificial intelligence in healthcare

Background:

• Convolutional Neural Networks (CNNs), particularly the U-Net framework, are benchmarks for medical image segmentation but struggle with global context.
• Transformers excel at capturing global information but are less adept at local feature extraction compared to CNNs.

Purpose of the Study:

• To introduce a novel network, O-Net, that integrates CNNs and transformers to leverage both local and global information for enhanced medical image segmentation and classification.
• To improve the accuracy and effectiveness of medical image analysis by combining complementary strengths of different deep learning architectures.

Main Methods:

• The proposed O-Net framework utilizes a hybrid encoder combining CNNs and Swin Transformers to extract both local and global contextual features.
• In the decoder, results from Swin Transformer and CNN blocks are fused for final segmentation output.
• A classification network is simultaneously trained using the encoder weights from the segmentation network.

Main Results:

• The O-Net demonstrated superior segmentation performance compared to state-of-the-art approaches on the synapse multi-organ CT and ISIC 2017 datasets.
• Improved segmentation accuracy positively impacted the performance of the simultaneously trained classification task.
• The hybrid approach effectively utilizes both global and local information for medical image analysis.

Conclusions:

• The O-Net framework offers a significant advancement in medical image segmentation and classification by effectively integrating CNN and transformer architectures.
• This novel approach provides a robust solution for learning comprehensive features, leading to improved diagnostic accuracy in medical imaging.
• The study highlights the potential of hybrid deep learning models for complex medical image analysis tasks.