A recurrent multimodal sparse transformer framework for gastrointestinal disease classification.

Area of Science:

• Gastroenterology and Artificial Intelligence
• Medical Image Analysis
• Natural Language Processing in Medicine

Background:

• Accurate gastrointestinal (GI) disease diagnosis is crucial but challenged by data inconsistencies.
• Existing methods struggle with modality imbalance and feature redundancy in text and endoscopic images.
• Heterogeneous data integration for GI disease classification requires advanced multimodal fusion strategies.

Purpose of the Study:

• To develop a novel recurrent multimodal principal gradient K-proximal sparse transformer (RMP-GKPS-transformer) framework.
• To enhance the accuracy and interpretability of GI disease classification using integrated clinical text and wireless capsule endoscopy (WCE) images.
• To address limitations of existing diagnostic frameworks in handling multimodal data.

Main Methods:

• Integrated clinical text and WCE images using Bio-RoBERTa for text features and a graph vision spatial channel attention transformer for image features.
• Employed cross-attention mechanisms for modality alignment, principal component analysis (PCA) for dimensionality reduction, and gradient boosting machines (GBMs) for semantic conflict resolution.
• Utilized an ensemble classifier including random forest KNN, proximal policy optimization (PPO), and a sparse radial basis function (RBF) kernel.

Main Results:

• Achieved 99.82% accuracy and a 98.7% Dice coefficient on publicly available datasets.
• Demonstrated significantly lower execution time compared to state-of-the-art methods.
• Successfully aligned and leveraged multimodal data for precise classification of six GI diseases.

Conclusions:

• The RMP-GKPS-transformer framework offers a scalable and interpretable solution for GI disease classification.
• The study highlights the potential of multimodal data fusion for improved clinical decision-making in gastroenterology.
• The proposed method effectively overcomes modality imbalance and feature redundancy in diagnostic data.