The Effect of Class Imbalance on Precision-Recall Curves

Area of Science:

• Machine Learning
• Data Science
• Classification Algorithms

Background:

• Classifier performance evaluation is crucial in machine learning.
• Understanding the impact of data imbalance on classifier metrics is essential.
• Existing literature may not fully explore the effect of test set composition on precision.

Purpose of the Study:

• To analyze the dependency of binary classifier precision on the ratio of positive to negative cases (r).
• To investigate how true and false-positive rates influence classifier precision in imbalanced datasets.
• To predict the behavior of precision-recall curves and other performance measures with varying 'r'.

Main Methods:

• Mathematical analysis of classifier precision formula.
• Derivation of relationships between precision, 'r', true-positive rate, and false-positive rate.
• Exploration of the impact on F-beta score and precision/recall gain measures.

Main Results:

• A clear mathematical relationship is established between classifier precision and the ratio 'r'.
• The study predicts how precision-recall curves shift as the proportion of positive cases changes.
• Formulas are derived to show the variation of F-beta and gain measures with 'r'.

Conclusions:

• The precision of a binary classifier is demonstrably sensitive to the ratio of positive to negative instances.
• This work provides a predictive framework for classifier performance under different data distributions.
• The findings offer valuable insights for model selection and evaluation in imbalanced classification tasks.