Closing the AI generalisation gap by adjusting for dermatology condition distribution differences across clinical...

Area of Science:

• Medical Artificial Intelligence
• Dermatology AI
• Clinical AI Generalization

Background:

• Generalizing artificial intelligence (AI) models to new clinical settings presents significant challenges.
• This study investigates the robustness and generalization capabilities of a dermatology AI model across different image types.

Purpose of the Study:

• To assess the generalization of a dermatology AI model from telemedicine cases to in-clinic settings, including patient-submitted (PAT) and clinician-taken (CLIN) photographs.
• To understand the factors influencing AI and dermatologist performance in new clinical environments.

Main Methods:

• A retrospective cohort study of 2500 unseen cases (PAT and CLIN) from 22 clinics (Nov 2015-Jan 2021).
• Primary outcome: top-3 accuracy (AI and dermatologists) compared to a reference diagnosis.
• Analysis of demographic factors and condition categories associated with AI errors; resampling to match AI development dataset distributions.

Main Results:

• AI performance was similar for CLIN (74% top-3 accuracy) and PAT (71%) images; dermatologists performed better on PAT (87%) vs. CLIN (79%).
• Demographic factors did not correlate with errors; specific condition categories were linked to AI errors.
• Resampling improved AI accuracy (CLIN 84%, PAT 79%) and adjusted dermatologist accuracy (CLIN 77%, PAT 89%).
• Fine-tuning strategies, including end-to-end and classification layer retraining, achieved high accuracy (83-86%) without resampling.

Conclusions:

• AI models can be efficiently adapted to new settings through recalibration or targeted data acquisition for rare conditions and retraining.
• Retraining the final classification layer or end-to-end fine-tuning demonstrates comparable performance.
• These findings suggest practical strategies for deploying AI in diverse clinical environments.