Assessing generalizability of an AI-based visual test for cervical cancer screening

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Oncology

Background:

• Artificial intelligence (AI) models face challenges in clinical translation, primarily due to a lack of generalizability.
• Generalizability refers to a model's ability to perform well on data dissimilar to its training set.

Purpose of the Study:

• To assess the generalizability and repeatability of a developed AI classifier for cervical images on external data.
• To investigate the impact of data heterogeneity (image capture device and geography) on AI model performance.

Main Methods:

• Developed an AI pipeline using a multi-heterogeneous dataset (9,462 women, 17,013 images) for cervical image classification.
• Evaluated classifier performance and repeatability on external data, considering device and geographic variations.
• Assessed out-of-the-box inference and retraining strategies with external data.

Main Results:

• Device-level heterogeneity impacted model performance more than geography-level heterogeneity.
• The AI classifier maintained strong performance on images from new geographies without retraining.
• Incremental retraining with new device images improved performance, reaching a saturation point.

Conclusions:

• Optimized retraining approaches are necessary to address data heterogeneity for effective AI deployment in new clinical settings.
• AI models for cervical cancer detection show promise but require careful validation and adaptation strategies.
• Repeatability of the AI classifier remained robust despite data heterogeneity.