Foundation-Model-Driven Skin Lesion Segmentation and Classification Using SAM-Adapters and Vision Transformers | JoVE Visualize

Area of Science:

Artificial Intelligence in Medical Imaging
Computational Dermatology
Machine Learning for Healthcare

Background:

Automated skin cancer evaluation faces challenges in segmenting and classifying dermoscopic images due to lesion variability, low contrast, and artifacts.
Foundation models like Segment Anything Model (SAM) show generalization potential but need domain-specific adaptation for medical imaging.
Vision Transformers (ViTs) offer robust identification but lack spatial priors crucial for lesion analysis.

Purpose of the Study:

To develop an integrated foundation-model-based framework for enhanced dermoscopic image analysis.
To improve the accuracy and reliability of automated skin lesion segmentation and classification.
To create a lesion-centric automated analysis system for potential clinical decision support in skin cancer detection.

Main Methods:

Utilized SAM-Adapter-fine-tuning for precise lesion segmentation, keeping the SAM encoder frozen and fine-tuning lightweight adapters for skin surface adaptation.
Employed a Vision Transformer (ViT)-based classifier incorporating lesion-specific cropping derived from segmentation and cross-attention fusion.
Integrated segmentation priors with patch-embeddings for lesion-centric reasoning and trained the pipeline using a joint multi-task approach on ISIC 2018, HAM10000, and PH2 datasets.

Main Results:

The proposed SAM-ViT framework achieved state-of-the-art performance, outperforming existing methods in both segmentation and classification.
Achieved a 94.27% Dice score for segmentation and 95.88% accuracy for classification on the ISIC 2018 dataset.
Demonstrated high performance on other datasets with a 95.62% Dice score on PH2 and 96.37% accuracy on HAM10000, with statistically significant improvements (p<0.01).

Conclusions:

Combining foundation model segmentation with transformer-based classification significantly enhances lesion boundary quality and diagnostic accuracy.
The SAM-ViT framework provides a robust, generalizable, and lesion-centric approach to automated dermoscopic analysis.
This represents a promising step towards clinically deployable decision-support systems for skin cancer detection, with future work focusing on model compression and real-world clinical validation.