Deriving the expected utility of a predictive model when the utilities are uncertain

Area of Science:

• Medical Informatics
• Decision Analysis
• Machine Learning

Background:

• Predictive models are crucial for clinical decision-making, often built from large clinical databases.
• Traditional evaluation methods like area under the ROC curve (AUC) are decision-independent.
• Uncertainty exists regarding the precise application of predictive models in clinical practice.

Purpose of the Study:

• To investigate techniques for deriving the expected utility of predictive models under uncertainty about their specific utilities.
• To provide a decision-theoretic framework for evaluating models when their exact use case is unknown.
• To complement existing decision-independent performance measures.

Main Methods:

• Exploration of decision theory principles for model evaluation.
• Development of methods to calculate expected utility considering utility uncertainty.
• Application of the developed approach to evaluate diagnostic models for coronary artery disease.

Main Results:

• Demonstrated a method for evaluating predictive models based on expected utility under uncertain conditions.
• Provided a practical example of applying this technique to compare two coronary artery disease diagnostic models.
• Showcased the utility of decision-theoretic evaluation beyond standard classification metrics.

Conclusions:

• Expected utility under uncertainty offers a valuable, complementary approach to model evaluation in clinical settings.
• This method enhances the assessment of predictive models by accounting for practical deployment variability.
• The findings support more robust model selection for clinical decision support systems.