An exploratory study on integrating radiomics with vision transformers for enhancing medical imaging classification...

Area of Science:

• Artificial Intelligence
• Medical Image Analysis
• Computer Vision

Background:

• Deep learning (DL) models, particularly Vision Transformers (ViTs), show promise in medical image analysis.
• However, ViTs require extensive data and lack inductive biases, limiting their use in medical imaging.
• Radiomics offers interpretable features but struggles with scalability and DL integration.

Purpose of the Study:

• To develop and evaluate a Radiomics-Embedded Vision Transformer (RE-ViT) framework.
• To integrate radiomics with patch-wise ViT embeddings for enhanced feature representation.
• To improve medical image classification across heterogeneous datasets.

Main Methods:

• A unified RE-ViT framework was designed, combining handcrafted radiomic features with data-driven visual embeddings.
• Input images were divided into patches; radiomic features and standard patch embeddings were extracted and combined.
• The combined embeddings were processed by a ViT encoder, with a learnable token for final classification. Evaluation used three datasets (BUSI, ChestXray2017, Retinal OCT) with 10-fold cross-validation.

Main Results:

• RE-ViT demonstrated robust and consistent classification performance across all tested medical imaging datasets.
• Achieved high accuracy, AUC, sensitivity, and specificity in BUSI, ChestXray2017, and Retinal OCT datasets.
• RE-ViT matched or outperformed CNN and hybrid models; ablation studies confirmed the importance of both radiomics and learned embeddings.

Conclusions:

• The developed RE-ViT effectively integrates radiomics and ViT for medical image classification.
• The framework shows potential for advancing transformer-based medical image analysis tasks.
• Attention maps indicated modality-specific feature utilization and improved localization of relevant regions.