Evaluating the impact of prediction models: lessons learned, challenges, and recommendations
Teus H Kappen1, Wilton A van Klei1, Leo van Wolfswinkel1
1Division of Anesthesiology, Intensive Care and Emergency Medicine, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Mail stop F.06.149, 3508 GA Utrecht, The Netherlands.
Abstract:
An important aim of clinical prediction models is to positively impact clinical decision making and subsequent patient outcomes. The impact on clinical decision making and patient outcome can be quantified in prospective comparative-ideally cluster-randomized-studies, known as 'impact studies'. However, such impact studies often require a lot of time and resources, especially when they are (cluster-)randomized studies. Before envisioning such large-scale randomized impact study, it is important to ensure a reasonable chance that the use of the prediction model by the targeted healthcare professionals and patients will indeed have a positive effect on both decision making and subsequent outcomes. We recently performed two differently designed, prospective impact studies on a clinical prediction model to be used in surgical patients. Both studies taught us new valuable lessons on several aspects of prediction model impact studies, and which considerations may guide researchers in their decision to conduct a prospective comparative impact study. We provide considerations on how to prepare a prediction model for implementation in practice, how to present the model predictions, and how to choose the proper design for a prediction model impact study.
Related Concept Videos
02:27Predicting Molecular Geometry
01:19Impact of Groups on Groups
01:29Self-Evaluation Maintenance Model
01:03Prediction Intervals
However, the point estimate is most likely not the exact value of the population parameter, but close to it. After calculating point estimates, we construct interval estimates, called confidence intervals or prediction intervals. This prediction interval comprises a range of values unlike the point estimate and is a better predictor of the observed sample value, y.
01:30Impact
When particles with different initial velocities collide, they induce deformation by applying equal and opposite impulses. At the point of maximum deformation, the particles move together with...
01:14Avoidance Learning and Learned Helplessness
Avoidance learning occurs when an organism learns that a specific behavior can prevent an unpleasant outcome. For example, a student who receives a bad grade may start studying harder to avoid future poor grades. This behavior persists even when the negative outcome is no longer present. Avoidance learning is powerful because it maintains behavior in the absence of the...