Hybrid Vision Transformer-CNN Architecture with Optimized Feature Selection for Skin Cancer Classification

Background/Objectives: Melanoma is a life-threatening skin cancer characterized by aggressive progression and high metastatic potential, making early diagnosis essential for improving patient survival and treatment outcomes. However, accurate automated skin lesion classification remains challenging due to variations in lesion appearance, illumination, image quality, and the presence of artifacts. This study proposes a unified framework for robust multi-class skin cancer classification by integrating preprocessing, lesion segmentation, feature extraction, optimization, and classification within a single end-to-end architecture. Methods: The proposed framework employs an iterative hair artifact removal strategy based on the fusion of Frangi vesselness filtering, Gabor texture filtering, morphological refinement, and Telea inpainting to preserve lesion integrity. A novel dermoscopic lesion segmentation network (CutisNet) is introduced to accurately delineate lesion boundaries. Hybrid representation learning combines deep features extracted using MobileNetV2 with uniquely selected handcrafted descriptors to capture complementary texture, structural, and contextual information. Gray Wolf Optimization is utilized for feature fusion and refinement, while a hybrid GNN-CNN classifier performs robust multi-class skin lesion classification. Results: Extensive experiments conducted on multiple benchmark dermoscopic datasets demonstrate the effectiveness and generalization capability of the proposed framework. The proposed model consistently outperformed existing state-of-the-art methods, achieving a maximum classification accuracy of 96.7% on the PH² dataset while maintaining competitive performance across other benchmark datasets. Conclusions: The proposed framework effectively integrates novel preprocessing, segmentation, hybrid feature representation, feature optimization, and classification strategies to improve the robustness and accuracy of automated skin cancer classification. These results demonstrate its potential to support reliable computer-aided diagnosis and assist clinicians in the early detection of skin cancer.