A hybrid approach of vision transformers and CNNs for detection of ulcerative colitis

Ulcerative Colitis is an Inflammatory Bowel disease caused by a variety of factors that lead to a serious impact on the quality of life of the patients if left untreated. Due to complexities in the identification procedures of this disease, the treatment timeline and quality can be severely affected, leading to further consequences for the sufferer. The difficulties in identification are due to high patients to healthcare professionals ratio. Researchers have proposed variety of machine/deep learning methods for automated detection of ulcerative colitis, however, several challenges exists including class imbalance problem, comprehensive feature extraction and accurate classification. We propose a novel method for accurate detection of ulcerative colitis with augmentation techniques to overcome class imbalance issue, a comprehensive feature vector extraction using custom architecture of Vision Transformer (ViT) and accurate classification using customized Convolutional Neural Network (CNN). We used the TMC-UCM and LIMUC datasets in this research for training and testing of proposed method and achieved accuracy of 90% with AUC-ROC scores of 0.91, 0.81, 0.94, and 0.94 for the endoscopic classes of Mayo 0, Mayo 1, Mayo 2, and Mayo 3 respectively. We have compared the proposed method with existing state of the art methods and conclude that the proposed method outperforms the existing methods.