A triple pronged approach for ulcerative colitis severity classification using multimodal, meta, and transformer...

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Gastroenterology

Background:

• Ulcerative colitis (UC) requires precise severity classification for effective treatment.
• Current methods may lack the accuracy needed for optimal therapeutic decisions.

Purpose of the Study:

• To develop and evaluate deep learning models for classifying ulcerative colitis severity.
• To explore the efficacy of Vision Transformers (ViT), multimodal learning, and few-shot meta-learning for this task.

Main Methods:

• Utilized a triple-pronged deep learning approach: multimodal inference, few-shot meta-learning, and ViT-based ensembling.
• Evaluated various ViT architectures (Swin-Base) and ensembles, alongside multimodal pre-trained frameworks (CLIP, BLIP, FLAVA) and few-shot methods (Matching Networks).
• Employed SHapley Additive exPlanations (SHAP) for model interpretability.

Main Results:

• A Swin-Base ViT model achieved 90% accuracy; a diverse ViT ensemble reached 93% accuracy.
• Multimodal frameworks with conventional ML yielded 83% accuracy.
• Few-shot meta-learning attained 83% accuracy in a 5-shot setting.

Conclusions:

• Deep learning, particularly ViT ensembles and multimodal strategies, shows significant potential for accurate UC severity classification.
• Model interpretability through SHAP is crucial for building clinical trust in AI-driven medical image analysis.
• These advanced techniques offer robust solutions for enhancing diagnostic precision in UC management.