Evaluating risk prediction models: the Predictiveness curve and its geometric summaries

Area of Science:

• Biostatistics
• Epidemiology
• Medical Informatics

Background:

• Risk prediction models are crucial in healthcare but often evaluated using discrimination measures like ROC/AUC.
• Existing measures offer limited insight into population-level risk distribution.
• A need exists for comprehensive evaluation frameworks beyond discrimination.

Purpose of the Study:

• To extend the predictiveness curve for risk prediction models.
• To derive novel geometric summaries (Pietra, Gini, scaled Brier) for model evaluation.
• To establish a population-oriented framework for transparent, purpose-specific model assessment.

Main Methods:

• Extension of the predictiveness curve and characterization of its geometry.
• Derivation of three geometric summaries: Pietra index, Gini index, and scaled Brier score.
• Implementation of a three-step post-processing procedure: cross-validation, isotonic recalibration, and bootstrap averaging.

Main Results:

• The Pietra, Gini, and scaled Brier indices capture complementary performance aspects.
• These indices support purpose-specific model selection (e.g., Pietra for gray zone, Gini for separation, scaled Brier for certainty).
• Application to lung cancer data demonstrated broad risk stratification with the post-processed model.

Conclusions:

• The predictiveness curve and its geometric summaries offer a population-oriented evaluation framework.
• This approach enhances transparency and supports purpose-specific choices in risk model development.
• The methods provide a more nuanced understanding of risk distribution than traditional discrimination measures.