GPT Fusion: Integrating Convolutional Neural Networks with GPT-5.5 for Melanoma Diagnosis

Malignant melanoma (MM) is the most aggressive form of metastatic skin cancer, for which early detection is critical to reducing morbidity and mortality while improving patient outcomes. Recent advances in multimodal large language models (LLMs) have demonstrated considerable potential for melanoma detection and clinical decision support. However, their diagnostic accuracy remains inferior to that of specialized convolutional neural networks (CNNs). Whether LLMs can effectively integrate predictions from specialized CNN models to improve melanoma diagnosis has not been systematically investigated.

We developed a hybrid diagnostic approach, termed GPT Fusion, in which GPT-5.5 serves as a reasoning-based fusion framework rather than an image classifier. Instead of directly analyzing lesion images, GPT-5.5 received structured outputs from two independently developed CNN models: a multimodal ResNet-50 model trained on the MILK10K dataset for multiclass skin lesion classification and the first-place 90-model SIIM-ISIC ensemble optimized for melanoma detection. GPT-5.5 integrated these complementary predictions to generate the final diagnosis. The framework was evaluated using the publicly available Derm7pt dataset. Performance was assessed for melanoma prediction (melanoma vs. non-melanoma), malignancy prediction (malignant vs. benign lesions), primary diagnosis, and top-3 differential diagnosis.

GPT Fusion substantially outperformed image-based GPT-5.5 across all evaluation tasks. For melanoma prediction, accuracy improved from 63.3% to 86.9%, while ROC AUC increased from 0.746 to 0.923 and PR AUC from 0.545 to 0.853. For malignancy prediction, GPT Fusion not only exceeded the performance of image-based GPT-5.5 but also surpassed the specialized SIIM-ISIC ensemble in sensitivity (83.9%), F1 score (73.6%), ROC AUC (0.894), and PR AUC (0.823), while maintaining comparable overall accuracy. GPT Fusion further improved primary diagnosis accuracy and achieved the highest top-3 differential diagnosis accuracy (89.0%).

These findings demonstrate that LLMs can provide greater clinical value as reasoning-based integrators of specialized AI systems than as standalone image classifiers. By combining the complementary strengths of specialized CNN models, GPT-5.5 substantially improved diagnostic performance while providing interpretable reasoning. This hybrid strategy offers a promising direction for developing more accurate, transparent, and clinically useful AI systems for melanoma diagnosis.