Improving visual communication of discriminative accuracy for predictive models: the probability threshold plot.

Area of Science:

• Machine Learning
• Medical Informatics
• Biostatistics

Background:

• Predictive models are crucial in healthcare decision-making.
• Assessing discriminative accuracy of predictive models is essential.
• Existing methods like ROC curves may not fully capture model performance across all probability thresholds.

Purpose of the Study:

• To introduce the probability threshold plot (PTP) as a novel visual display.
• To transparently communicate a predictive model's discriminative accuracy.
• To evaluate the PTP's utility in understanding model performance across predicted probability ranges.

Main Methods:

• Replicated a validated machine learning model for predicting antihyperglycemic medication cessation.
• Applied the probability threshold plot (PTP) to visualize model performance.
• Compared PTP characteristics with receiver operating characteristic (ROC) curves.

Main Results:

• Analysis included 18,887 patients.
• Models showed similar ROC curves and Area Under the Curve (AUC) values (0.672 vs. 0.673).
• PTPs revealed significant differences in sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) across probability thresholds, despite similar ROC performance.

Conclusions:

• The PTP enhances the visual display of predictive model discriminative accuracy.
• PTPs offer a more nuanced understanding of model performance compared to ROC curves.
• This improved visualization can aid in the practical application of predictive models for medical decision-making.