Classifier calibration using splined empirical probabilities in clinical risk prediction

Area of Science:

• Machine Learning
• Statistical Modeling
• Data Science

Background:

• Supervised machine learning (ML) applications primarily focus on classification and ranking.
• Accurate probability estimation is crucial for many ML applications but remains a challenge.
• Existing methods for converting ML rankings into probability estimates have limitations.

Purpose of the Study:

• To present a novel method, splined empirical probabilities, for deriving accurate probability estimates from ML outputs.
• To compare the performance of splined empirical probabilities against existing calibration methods.
• To demonstrate the method's utility on simulated and real-world healthcare data.

Main Methods:

• Developed a new probability estimation technique based on the receiver operating characteristic (ROC) curve.
• Utilized splined empirical probabilities, a cumulative approach, integrated with ROC analysis.
• Evaluated performance using metrics like Hosmer-Lemeshow, Kullback-Leibler divergence, and cumulative distribution function differences.

Main Results:

• Splined empirical probabilities demonstrated favorable performance compared to the standard pool adjacent violators algorithm for isotonic regression.
• The method showed effectiveness across various measures of probability estimate quality.
• Successful application on both simulated and real healthcare datasets was observed.

Conclusions:

• Splined empirical probabilities offer a valuable and easily integrated alternative for accurate probability estimation in ML.
• The method provides a robust approach for enhancing ML model calibration.
• This technique holds significant potential for applications requiring precise probability assessments, particularly in healthcare.