ROC and AUC with a Binary Predictor: a Potentially Misleading Metric

Area of Science:

• Statistics
• Machine Learning
• Data Science

Background:

• Receiver Operating Characteristic (ROC) curves are vital for evaluating binary classification model performance.
• Area Under the Curve (AUC) summarizes ROC curve performance but can be sensitive to predictor types.
• Binary predictors present unique challenges for ROC curve interpretation due to limited thresholds.

Purpose of the Study:

• To investigate the impact of interpolation methods on AUC calculations for binary predictors.
• To compare AUC results obtained from different statistical software packages.
• To provide recommendations for accurate reporting of model performance metrics.

Main Methods:

• Analysis of ROC curves with a focus on binary predictors.
• Comparison of linear interpolation versus step function interpolation.
• Evaluation of AUC calculation implementations in R, Python, Stata, and SAS.

Main Results:

• Linear interpolation, commonly used in software, can significantly alter AUC values for binary predictors.
• Different software implementations yield varying AUC results due to interpolation differences.
• The step function interpolator offers a more conservative and potentially less misleading AUC estimate.

Conclusions:

• The choice of interpolation method critically affects AUC interpretation for binary predictors.
• Users should be aware of and report the interpolation method used in AUC calculations.
• The step function (pessimistic) approach is recommended for more robust AUC estimation.