Multiclass Classification of Pigmented Skin Lesions Using a Multimodal Large Language Model

Pigmented skin lesions span benign to malignant entities that often appear similar on standard clinical photographs, complicating accurate diagnosis without specialized imaging. Recently, multimodal large language models (MMLLMs) have attracted attention as image-based diagnostic aids and hold promise as decision-support tools in resource-limited settings where dermoscopy may be unavailable.

This study aimed to determine whether a fine-tuned MMLLM can accurately classify eight common pigmented skin conditions using only clinical photographs, thereby providing a non-dermoscopic diagnostic support tool.

We fine-tuned InstructBLIP-flan-t5-xl (Salesforce, San Francisco, CA) using Hugging Face's Seq2SeqTrainer (Hugging Face Inc., New York City, NY) on a curated dataset of 979 manually cropped regions of interest depicting one of eight lesion types (acquired dermal melanocytosis, basal cell carcinoma, ephelis, malignant melanoma, melasma, nevus, seborrheic keratosis, or solar lentigo). Images were split 80% for training and 20% for validation. During training, lesion labels were masked to encourage learning of visual-text correlations. Model performance was evaluated by macro-average sensitivity, specificity, F1 score, and area under the receiver operating characteristic area under the curve (ROC AUC) for each class.

On the validation set, the model achieved a macro-average sensitivity of 86.0%, specificity of 98.2%, and F1 score of 0.86. ROC AUC exceeded 0.95 for six of eight classes. Malignant melanoma showed the highest performance (sensitivity 94%, ROC AUC 0.98), while nevus exhibited the lowest sensitivity (78%, ROC AUC 0.89).

Fine-tuned MMLLMs can accurately classify common pigmented skin lesions from clinical photographs alone, enabling rapid diagnostic support in environments lacking dermoscopy. Future work should expand dataset diversity, undertake multicenter validation, and assess real-world clinical utility to confirm broader applicability.