A spatial-spectral vision transformer model for head and neck cancer detection with hyperspectral, RGB, and...

The potential benefit of employing hyperspectral imaging (HSI) over the normal color images, i.e., RGB, has been explored to improve cancer detection. In this study, we evaluate the efficacy of deep learning models in detecting head and neck squamous cell carcinoma (SCC) in histological images using HSI. We used 77 whole histologic slides from 51 patients to train DinoV2 vision transformer model and a ResNet-152 model on RGB, HSI, and HSI-synthesized RGB images, respectively. A spatial-spectral vision transformer (SST) model with spectral attention was also introduced for comparison and evaluation. Our study resulted in three major findings. First, we found that the SST model trained on HSI performed the best over other models with 79% accuracy, 74.37% specificity, and 78.92% sensitivity. Second, models trained using RGB data suffered from severe imbalance between sensitivity and specificity. Finally, the DinoV2 model trained on HSI was found to have significantly higher (10%) sensitivity compared to its RGB alternative. The proposed method of using the spatial-spectral vision transformer model shows the advantage of hyperspectral image in terms of sensitivity, accuracy, and computation for detecting squamous cell carcinoma in histologic slides.