An Attention-Enhanced ViT-HLNN Hybrid Ensemble Framework for Multi-Class Gastrointestinal Disease Classification

Gastrointestinal (GI) diseases such as polyps, esophagitis, and ulcerative colitis pose significant diagnostic challenges due to subtle visual patterns. Early, accurate, and scalable diagnostic tools are essential for improving outcomes and streamlining care. In this study, we propose a hybrid deep learning framework for multi-class GI disease classification that integrates a Vision Transformer (ViT) with a Hierarchical Long-Short-Term-Memory Neural Network (HLNN), further enhanced through ensemble learning using Bagging and Stacking. This hybrid architecture captures both global dependencies and local spatial-temporal features in endoscopic imagery, enabling precise multi-class classification. The model was trained and validated on the Kvasir-v2 dataset, comprising over 8000 labelled endoscopic images across eight GI disease classes. The proposed method achieved up to 99.99% accuracy on the test set and 99.96% mean accuracy across tenfold cross-validation, with macro F1-scores exceeding 0.999, and Cohen's kappa of 0.994, indicating excellent agreement. Test set performance was evaluated on a fixed, class-stratified held-out test set (100 images per class) and reported as the mean across multiple training runs with different random seeds, ensuring statistical robustness. In comparative analysis, we demonstrated that the proposed hybrid model outperforms established baselines, including ViT-only, MobileNet-V2, EfficientNet-B0, ResNet-101, and NASNet-Large. Controlled ablation studies validate the individual and combined contributions of HLNN and ensemble strategies. Robustness was further confirmed under Gaussian noise, motion blur, and illumination shifts, with accuracy consistently above 96%. Zero-shot external evaluation on the independent GastroVision dataset and class-holdout testing on the held-out Polyps class further demonstrate generalization beyond the Kvasir-v2 training distribution. Robustness of Grad-CAM explanations was validated through a sanity check, correlation r = 0.91, and focused attention entropy 0.32, confirming stability under perturbations. These results highlight the potential of the proposed framework for robust and interpretable GI image classification.