Credible Intervals for Precision and Recall Based on a K-Fold Cross-Validated Beta Distribution

Area of Science:

• Machine Learning
• Statistical Inference
• Information Retrieval

Background:

• Precision and recall are key metrics for evaluating machine learning algorithms, particularly in information retrieval.
• Traditional confidence intervals for precision and recall, often based on K-fold cross-validated t-distributions, can yield unreliable results due to low confidence levels.
• There is a need for robust credible intervals that provide high confidence and short lengths for accurate performance assessment.

Purpose of the Study:

• To propose novel posterior credible intervals for precision and recall using K-fold cross-validated beta distributions.
• To address the limitations of existing confidence intervals that exhibit inadequate confidence degrees and potentially liberal inference.
• To offer improved methods for reliable statistical inference of machine learning model performance.

Main Methods:

• Developed two types of credible intervals based on K-fold cross-validated beta posterior distributions.
• Method 1: Inferred a single beta posterior distribution from all K confusion matrices.
• Method 2: Averaged K individual beta posterior distributions, each derived from a single confusion matrix.

Main Results:

• The first proposed credible interval consistently achieved confidence degrees exceeding 95% in experiments.
• Both proposed credible intervals demonstrated shorter interval lengths compared to corrected K-fold cross-validated t-distribution intervals.
• The new credible intervals outperformed traditional t-distribution-based intervals in terms of confidence degree and interval length rankings across 27 experimental cases.

Conclusions:

• The proposed K-fold cross-validated beta distribution credible intervals offer a more reliable approach for precision and recall inference.
• The first credible interval method is particularly recommended for its high confidence levels and reliability.
• These novel methods provide a superior alternative to existing techniques for robust machine learning performance evaluation.