A framework for evaluating clinical artificial intelligence systems without ground-truth annotations.

Area of Science:

• Medical Artificial Intelligence
• Clinical AI Validation
• Algorithmic Bias Assessment

Background:

• Clinical AI systems are typically validated on withheld data, which may not reflect real-world data distribution.
• Estimating AI performance in the wild is challenging due to distribution shifts and lack of ground-truth annotations.

Purpose of the Study:

• Introduce SUDO, a novel framework for evaluating AI systems on data encountered in real-world clinical settings.
• Demonstrate SUDO's utility in assessing AI performance, selecting models, and identifying algorithmic bias without ground-truth data.

Main Methods:

• Developed and applied the SUDO framework to evaluate AI systems across diverse clinical data types.
• Experiments included AI systems for dermatology images, histopathology patches, and clinical notes.
• SUDO assesses AI performance and bias on 'in-the-wild' data.

Main Results:

• SUDO effectively identifies unreliable AI predictions in real-world clinical data.
• The framework aids in selecting appropriate AI models for deployment.
• SUDO enables the assessment of algorithmic bias in AI systems operating on unseen data.

Conclusions:

• SUDO provides a robust method for evaluating clinical AI systems on data in the wild.
• The framework supports the development of trustworthy and ethical AI in medicine.
• SUDO addresses key limitations in current AI validation practices for clinical applications.