MICDnet: Multimodal information processing networks for Crohn's disease diagnosis

Area of Science:

• Gastroenterology and Medical Imaging Analysis

Background:

• Crohn's disease (CD) is a chronic inflammatory condition with rising global incidence and unknown causes.
• Accurate diagnosis requires integrating diverse patient data, including colonoscopy, pathology images, and clinical records, which is time-consuming and resource-intensive.
• Current diagnostic systems struggle to effectively process multimodal data for CD.

Purpose of the Study:

• To develop and evaluate a multimodal information diagnosis network (MICDnet) for enhanced Crohn's disease diagnosis.
• To integrate colonoscopy images, pathology images, and clinical text data for improved diagnostic accuracy.

Main Methods:

• Developed MICDnet, a network designed for multimodal feature extraction and fusion.
• Preprocessed individual data modalities (colonoscopy, pathology, text) using separate encoders.
• Integrated features through multimodal fusion for final CD classification and diagnosis.
• Created a dataset of 136 patients with comprehensive multimodal data.

Main Results:

• MICDnet demonstrated superior diagnostic performance compared to other models.
• Multimodal data integration significantly improved the diagnostic accuracy for Crohn's disease.
• The developed dataset facilitated effective training and validation of the MICDnet model.

Conclusions:

• Multimodal data analysis using MICDnet offers a promising approach for improving Crohn's disease diagnosis.
• The MICDnet system can enhance diagnostic accuracy and efficiency in clinical settings.
• Further research and validation on larger datasets are warranted.