Automated classification of oral cancer lesions: Vision transformers vs radiomics

Early diagnosis is paramount in the effective management of oral cancer, offering numerous benefits including improved treatment outcomes, reduced morbidity and mortality, preservation of function and appearance, cost-effectiveness, and enhanced quality of life for patients. Transformer-based models, increasingly used in medical image analysis, are the focus of our study. We aim to compare a vision transformer (ViT) classification method with a fully automated radiomics approach. This involves using object detection and segmentation algorithms to effectively classify oral lesions in both cancer and control cases. A combined approach is also presented.

The analysis included 322 patients with oral lesions, comprising 120 cancer cases and 202 controls, with standard JPG images. Pretrained transformer-based algorithms like DEtection TRansformer (DETR), Segment Anything (SAM), and Vision Transformers (ViT) were used to explore different pipelines for lesion classification. For the ViT approach, images were inputted in three configurations: the entire image, a bounding box around the lesion, and a lesion delineation. The radiomics approach involved pipelines with bounding boxes and lesion segmentations. Additionally, a ViT-Radiomics combined approach was proposed, using ViT attention maps as radiomics masks. To validate the models, a five-fold cross validation was used.

The combined ViT-radiomics model demonstrated superior performance for small training sets, achieving specificity = 0.97 ± 0.04, sensitivity = 0.96 ± 0.05, and accuracy = 0.97 ± 0.02 for the 100 % of the training set. When analyzed independently, the ViT approach using the entire image achieved the best results with specificity = 0.99 ± 0.02, sensitivity = 0.96 ± 0.05, and accuracy = 0.96 ± 0.02. Following closely was the pipeline using automatically obtained segmentations, while the one with bounding boxes had the least favourable outcomes. In the radiomics approach, the most effective classifier used the attention masks from the ViT classifier (derived from the entire image), achieving specificity = 0.97 ± 0.05, sensitivity = 0.95 ± 0.08, and accuracy = 0.94 ± 0.03. Manual segmentations yielded the best results for both approaches, indicating potential for performance enhancement through improved lesion segmentation.

The ViT classification surpassed the radiomics-based approach yet combining ViT with radiomics yielded similar results. However, the attention maps generated by ViT tend to associate oral lesions in cancer patients with regions distant from the lesions in control patients. For tasks requiring the examination and comparison of features within cancer and control oral lesions, utilizing the radiomics approach with an automatic lesion segmentation algorithm is recommended.