Assessing classifiers from two independent data sets using ROC analysis: a nonparametric approach

Area of Science:

• Machine Learning
• Statistical Modeling
• Data Science

Background:

• Binary classification performance is often evaluated using the Area Under the ROC Curve (AUC).
• Estimating the uncertainty associated with AUC is crucial for reliable model assessment.
• Existing methods may require distributional assumptions or large datasets.

Purpose of the Study:

• To develop a nonparametric method for estimating conditional AUC and its variance.
• To derive a closed-form expression for the variance of the AUC estimator.
• To provide a framework for understanding sources of uncertainty in AUC estimation.

Main Methods:

• Utilized U-statistics for nonparametric estimation.
• Derived a closed-form expression for the variance of the AUC estimator.
• Applied methods to binary classification tasks with distinct training and testing sets.

Main Results:

• Successfully estimated conditional AUC, its mean, and variance.
• Derived a novel closed-form expression for AUC estimator variance.
• Identified key components contributing to the uncertainty in AUC estimates.
• Demonstrated the utility of the estimators through simulation results.

Conclusions:

• The proposed nonparametric U-statistics approach offers a robust way to assess AUC and its uncertainty.
• The derived variance expression enhances understanding of AUC estimation reliability.
• This method is valuable for binary classification when data distributions are unknown.