Evaluating the generalizability of deep learning image classification algorithms to detect middle ear disease using...

Area of Science:

• Medical Imaging
• Artificial Intelligence in Healthcare
• Otolaryngology

Background:

• Deep learning algorithms are increasingly used for medical image analysis.
• Assessing the generalizability of these AI models across different datasets is crucial for clinical adoption.
• Middle ear disease diagnosis relies on interpreting otoscopic images.

Purpose of the Study:

• To evaluate the generalizability of artificial intelligence (AI) algorithms for identifying middle ear disease using otoscopic images.
• To compare the internal performance of AI models with their external performance on independent datasets.

Main Methods:

• Collected 1842 otoscopic images from three diverse international sources (Turkey, Chile, USA).
• Developed deep learning models to assess diagnostic performance using area under the curve (AUC) for internal and external validation.
• Conducted a pooled assessment combining all cohorts with fivefold cross-validation.

Main Results:

• AI algorithms achieved high internal performance (mean AUC: 0.95).
• External performance was significantly reduced (mean AUC: 0.76) compared to internal testing.
• A pooled assessment combining cohorts demonstrated substantial performance (AUC: 0.96).

Conclusions:

• While AI-otoscopy algorithms perform well internally, their generalizability to external, unseen data is limited.
• Significant performance drop in external validation highlights the need for improved data augmentation and pre-processing techniques.
• Further research is essential to develop robust and clinically applicable AI algorithms for middle ear disease detection.