Challenges in the Postmarket Surveillance of Clinical Prediction Models

Area of Science:

• Clinical Informatics
• Artificial Intelligence in Medicine
• Health Services Research

Background:

• Predictive artificial intelligence (AI) models are crucial for clinical workflows, including prognostication and decision support.
• Postdeployment performance degradation due to dataset shifts is a significant challenge for AI models.
• Current regulatory guidelines mandate continuous monitoring, but practical strategies for AI performance assessment are lacking.

Purpose of the Study:

• To address the challenges in assessing the postdeployment performance of predictive AI models.
• To highlight the issue of confounding medical interventions biasing AI performance assessment.
• To explore potential solutions and the need for reliable validation methods for AI in healthcare.

Main Methods:

• Discussion of postdeployment assessment challenges, focusing on confounding medical interventions.
• Review of proposed solutions such as withholding model outputs, outcome monitoring, and including interventions in models.
• Exploration of advanced causal modeling as a potential validation method.

Main Results:

• Confounding medical interventions can bias AI performance assessment, falsely indicating model decay.
• Existing solutions have ethical or practical limitations, hindering effective postdeployment evaluation.
• Lack of effective monitoring leads to potentially harmful, unevaluated AI models.

Conclusions:

• Advanced causal modeling offers a promising approach for reliable postdeployment validation of AI models.
• Evidence-based decisions considering causal pathways are essential for AI model updates in dynamic clinical environments.
• Developing and validating effective postdeployment monitoring methods is critical to ensure sustained AI utility and prevent patient harm.