Challenges in the Postmarket Surveillance of Clinical Prediction Models

Predictive artificial intelligence (AI) models enhance clinical workflows with applications such as prognostication and decision support, yet suffer from postdeployment performance challenges due to dataset shifts. Regulatory guidelines emphasize the need for continuous monitoring, but actionable strategies are lacking. A significant issue is postdeployment assessment of predictive AI models due to confounding medical interventions where effective interventions modify outcomes, introducing bias into performance assessment. This can falsely suggest model decay, leading to unwarranted updates or decommissioning, harming clinical outcomes. Proposed solutions include withholding model outputs, monitoring outcomes as surrogates, or including clinician interventions in models, each with ethical or practical limitations. The lack of effective solutions for this problem can lead to an abundance of models that cannot be later evaluated, tuned, or withdrawn if they become ineffective, leading to patient harm. Advanced causal modeling to assess counterfactual outcomes may offer a reliable validation method. Until effective methods for postdeployment monitoring of predictive models are developed and validated, decisions on model updates should consider the causal pathways and be evidence based, ensuring the sustained utility of AI models in dynamic clinical environments.