Statistical Inference and Power Analysis for Comparative F ₁ and F _β Scores under Correlated Classifier Pairs

Area of Science:

• Biomedical Informatics
• Machine Learning in Healthcare
• Statistical Modeling

Background:

• Machine learning and AI are increasingly vital in healthcare, necessitating robust performance evaluation.
• F1 and F-beta scores are standard metrics for imbalanced biomedical data.
• Existing frameworks like psF1 assume classifier independence, which is often violated in practice.

Purpose of the Study:

• To develop a statistical framework, psF1pair, for accurate inference and power analysis of correlated classifier pairs.
• To address the overestimation of standard error and loss of statistical power caused by ignoring classifier dependency.
• To provide a tool for improved study design and evaluation of machine learning models in biomedical research.

Main Methods:

• Development of the psF1pair statistical framework accounting for correlations between classifier pairs.
• Extensive simulation studies to demonstrate the performance and validity of psF1pair.
• Application of psF1pair to a real-world medical imaging classification system.

Main Results:

• psF1pair accurately accounts for correlations between classifiers, unlike previous methods.
• Higher correlation between classifiers leads to narrower confidence intervals and increased statistical power.
• The framework demonstrated superior performance in simulation studies and a real-world case.

Conclusions:

• psF1pair offers a statistically sound approach for evaluating and designing studies involving paired classifiers in biomedical research.
• The framework enhances statistical power and accuracy, crucial for reliable machine learning applications in healthcare.
• A freely available R package facilitates the implementation of psF1pair for broader use in the research community.