A Novel Information Complexity Approach to Score Receiver Operating Characteristic (ROC) Curve Modeling

Area of Science:

• Statistics
• Machine Learning
• Biostatistics

Background:

• Traditional performance metrics for classification models have limitations, as no single metric captures all aspects of performance.
• Information criteria offer a quantitative approach to model selection, balancing complexity and goodness of fit.

Purpose of the Study:

• Introduce and develop a novel Information Complexity-Receiver Operating Characteristic (ICOMP-ROC) criterion for evaluating ROC curve models.
• Compare the performance of the ICOMP-ROC criterion against traditional metrics in both simulated and real-world datasets.

Main Methods:

• Construct and derive the Universal ROC (UROC) for sixteen Bi-distributional ROC models, minimizing the ICOMP-ROC criterion.
• Conduct large-scale Monte Carlo simulations using Normal-Normal and Weibull-Gamma pairs as pseudo-true ROC models.
• Apply the ICOMP-ROC criterion and traditional metrics to high-dimensional Magnetic Resonance Imaging (MRI) and Wisconsin Breast Cancer (WBC) datasets using machine learning algorithms and genetic algorithms for feature selection.

Main Results:

• The ICOMP-ROC criterion demonstrated a remarkable recovery rate in simulations, indicating its effectiveness in model selection.
• Numerical results showed the consistency and reliability of the ICOMP-ROC criterion over traditional metrics, especially for complex and high-dimensional datasets.
• The study identified the best fitting Bi-distributional ROC models and classification algorithms using the proposed ICOMP-ROC criterion.

Conclusions:

• The ICOMP-ROC criterion is a versatile and robust approach for ROC curve modeling and model selection.
• This novel criterion effectively addresses the limitations of traditional performance metrics in diverse and complex data scenarios.
• The ICOMP-ROC criterion provides a superior method for choosing optimal models and algorithms in statistical and machine learning applications.