Applicability Area: A novel utility-based approach for evaluating predictive models, beyond discrimination.

Area of Science:

• * Medical Informatics
• * Machine Learning Evaluation
• * Clinical Decision Support

Background:

• * Evaluating machine learning models for clinical practice requires assessing their real-world utility beyond discriminatory power.
• * Current methods often focus on metrics like Area Under the Receiver Operating Characteristic Curve (AUROC), which do not fully capture clinical decision-making complexities.

Purpose of the Study:

• * To introduce and evaluate the Applicability Area (ApAr), a novel decision-analytic, utility-based approach for assessing predictive model performance.
• * To demonstrate how ApAr can provide a more comprehensive evaluation of a model's clinical value compared to existing metrics.

Main Methods:

• * Development of the Applicability Area (ApAr) metric, which quantifies the range of prior probabilities and test cutoffs where a predictive model offers positive utility.
• * Validation of ApAr using simulated datasets and three published medical datasets.
• * Comparison of ApAr rankings with traditional AUROC metric analysis.

Main Results:

• * The ApAr metric provides additional value in evaluating predictive models, complementing traditional AUROC analysis.
• * In a diabetes dataset example, the top-ranked model by ApAr was significantly lower ranked by AUROC (23rd), highlighting differing model assessments.
• * Larger ApAr values indicate a broader range of clinical applicability for the predictive model.

Conclusions:

• * The Applicability Area (ApAr) offers a superior, utility-based framework for evaluating the clinical value of predictive models.
• * ApAr assists decision-makers in determining if a model's utility aligns with local clinical contexts and patient populations.
• * This approach facilitates more informed adoption and implementation of machine learning models in healthcare settings.