Confidence scores for prediction models.

Area of Science:

• Medical Statistics
• Machine Learning in Healthcare
• Biostatistics

Background:

• Comparing prediction models is crucial in medical statistics, often using independent validation data.
• When only one dataset is available, repeated bootstraps can compare rival modeling strategies.
• Similar overall performance of rival strategies complicates model selection.

Purpose of the Study:

• To investigate the variability of prediction models when applied to different training sets.
• To develop a confidence score for assessing prediction model stability and performance.
• To provide a method for distinguishing between prediction models with similar predictive accuracy.

Main Methods:

• Investigated prediction model variability using the same modeling strategy on different training sets.
• Estimated a confidence score for each modeling strategy via repeated bootstraps.
• Derived a new decomposition of the expected Brier score.
• Calculated population average confidence scores.

Main Results:

• A novel confidence score was developed to quantify prediction model variability.
• The proposed method allows for distinguishing between rival prediction models with similar performance.
• Population average confidence scores were estimated.
• A new decomposition of the expected Brier score was obtained.

Conclusions:

• Confidence scores can differentiate prediction models with similar prediction performances.
• Subject-level confidence scores offer supplementary information for patient-level medical decisions.
• The methods are applicable to cancer studies, including those with high-dimensional predictor spaces.