Performance Evaluation of ChatGPT-4o in Dermatological Diagnoses Across Fitzpatrick Skin Types

Area of Science:

• Dermatology
• Artificial Intelligence
• Medical Diagnostics

Background:

• Multimodal AI models like ChatGPT-4o offer potential for patient self-assessment.
• Accessibility of AI tools before seeking medical care is increasing.

Purpose of the Study:

• To evaluate the diagnostic performance of ChatGPT-4o for dermatologic conditions.
• To assess AI performance across different Fitzpatrick skin types (FSTs).

Main Methods:

• Utilized 324 biopsy-confirmed dermatologic images.
• Assessed ChatGPT-4o's differential diagnoses and malignancy assessments against histopathologic ground truths.
• Compared model performance across lighter (FSTs I-II) and darker (FSTs III-VI) skin tones.

Main Results:

• ChatGPT-4o demonstrated significantly lower sensitivity, specificity, and accuracy for melanoma detection in darker skin tones (FSTs III-VI).
• Performance disparities were observed when comparing results between lighter and darker skin types.
• The AI model's diagnostic capabilities varied notably based on skin pigmentation.

Conclusions:

• AI diagnostic tools may exhibit performance bias against underrepresented populations, particularly in dermatology.
• The study underscores the critical need for inclusive and diverse dermatologic datasets to train AI models.
• Ensuring equitable AI performance across all skin types is essential for patient safety and effective healthcare.