Related Experiment Videos
Multimodal Fusion of Endoscopic and Histopathological Images for Lesion Detection Using Hybrid Deep Learning
Premananda Sahu1, Salil Bharany2, Jaibir Singh3
1Department of Computer Science and Engineering, Lovely Professional University, Phagwara, Punjab, India.
Current Medical Imaging
|June 2, 2026
Summary
A new deep learning model, SHF-Net, accurately classifies gastrointestinal lesions by integrating spatial and histological data. This advanced system improves diagnostic reliability and aids in early disease detection.
Area of Science:
- Artificial Intelligence in Medicine
- Medical Imaging Analysis
- Computational Pathology
Background:
- Accurate classification of gastrointestinal (GI) endoscopic lesions is crucial for early diagnosis but challenged by subtle appearances and interobserver variability.
- Automated diagnostic systems are needed to improve accuracy and reliability by integrating diverse image features.
Purpose of the Study:
- To develop a novel deep learning framework, the Spatio-Histological Fusion Network (SHF-Net), for enhanced GI lesion detection and classification.
- To integrate multiple feature-extraction approaches, including Convolutional Neural Networks (CNNs), Vision Transformers (ViTs), and Graph Neural Networks (GNNs), for comprehensive analysis.
Main Methods:
- SHF-Net was developed, fusing spatial features from CNNs, long-range dependencies from ViTs, and histopathological information from GNNs using a multi-channel attention mechanism.
- The model was trained on the HyperKvasir dataset (111,079 images) across five lesion categories, utilizing stain normalization, Generative Adversarial Networks (GANs) for data augmentation, and a combination of cross-entropy and topological loss functions.
Main Results:
- SHF-Net achieved high classification performance with 98.84% accuracy, 98.57% precision, 98.21% recall, and 98.46% F1-score across five GI lesion categories.
- Attention heatmaps demonstrated that the model focused on clinically significant regions in both endoscopic and histopathological images, indicating interpretability and clinical relevance.
Conclusions:
- The integration of CNNs, ViTs, and GNNs in SHF-Net effectively captures complementary features, addressing challenges of subtle lesions and interobserver variability.
- SHF-Net's multi-modal design, high accuracy, and interpretability offer a significant advancement for automated GI diagnosis and potential real-time clinical deployment.