A Unified Framework for Statistical Inference and Power Analysis of Single and Comparative F β $$ {F}_{\beta } $$...

Area of Science:

• Biomedical data science
• Statistical methodology
• Machine learning evaluation

Background:

• Machine learning and AI are crucial for medical diagnostics and clinical decisions.
• F1 and F-beta scores are standard metrics for evaluating classification model performance.
• Limited statistical inference and power analysis exist for F1/F-beta scores.

Purpose of the Study:

• To introduce psF1, a unified framework for statistical inference and power analysis of F1 and F-beta scores.
• To enable interval estimation, hypothesis testing, and sample size calculations for these metrics.
• To facilitate principled evaluation and comparison of machine learning classifiers in biomedical applications.

Main Methods:

• Development of psF1, a comprehensive framework for F1 and F-beta score analysis.
• Utilizing exact probability distributions and large-sample approximations for statistical validity.
• Extensive simulations to assess accuracy and robustness across various scenarios.

Main Results:

• psF1 provides accurate and robust statistical inference for F1 and F-beta scores.
• The framework supports both single and comparative score analyses.
• Simulations confirm psF1's reliability across diverse sensitivity, precision, and sample size conditions.

Conclusions:

• psF1 offers a principled approach to evaluating machine learning classifiers using F1/F-beta scores.
• The framework enhances uncertainty quantification and informs sample size planning.
• psF1 is a valuable tool for reliable biomedical classification model assessment.