Related Experiment Video
Updated: Jan 10, 2026
Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography
Published on: November 30, 2022
Unsupervised SAM-guided mixture-of-multimodal-experts fusion network for medical image diagnosis
Jing Li1, Yixuan Wu1, Xiaorou Zheng1
1Guangdong Provincial Key Laboratory of Multimodal Big Data Intelligent Analysis, South China University of Technology, Guangzhou, China; School of Computer Science and Engineering, South China University of Technology, Guangzhou, China.
This study introduces an unsupervised method for cancer diagnosis using medical images, improving lesion localization and multimodal data fusion without costly manual annotations. The new approach enhances diagnostic accuracy and personalization for better patient outcomes.
Area of Science:
- Medical imaging analysis
- Artificial intelligence in healthcare
- Computational oncology
Background:
- Accurate cancer diagnosis requires precise lesion localization and multimodal data integration.
- Current methods face challenges with expensive pixel-level annotations and rigid data fusion strategies.
- Patient-specific variations in imaging modalities are often overlooked.
Purpose of the Study:
- To develop an unsupervised framework for cancer diagnosis using medical images.
- To enable precise lesion localization without manual segmentation labels.
- To implement adaptive multimodal data fusion tailored to individual patient characteristics.
Main Methods:
- Proposed an Unsupervised SAM-guided Mixture-of-Multimodal-Experts Fusion Network (UnSAM-MoME).
- Utilized a dual cross-validation segmentation network to generate prompts for the Segment Anything Model (SAM) for unsupervised lesion localization.
- Developed a Mixture-of-Multimodal-Experts (MoME) module for adaptive fusion of image and metadata features.
Main Results:
- UnSAM-MoME achieved state-of-the-art performance on skin and breast cancer datasets.
- Demonstrated significant improvements in diagnostic accuracy, precision, and generalizability.
- Ablation studies confirmed the effectiveness of individual network modules.
Conclusions:
- The UnSAM-MoME framework offers a scalable and personalized approach to cancer diagnosis.
- Unsupervised lesion localization reduces reliance on costly manual annotations.
- Adaptive multimodal fusion enhances diagnostic performance by considering patient-specific data contributions.
