Sampling Strategies to Evaluate the Performance of Unknown Predictors.

Area of Science:

• Machine Learning
• Statistical Modeling

Background:

• Evaluating classification models with imbalanced datasets presents challenges.
• Standard sampling methods may not accurately estimate performance metrics.

Purpose of the Study:

• To develop optimal sampling strategies for selecting labeled examples.
• To enable robust evaluation of diverse classification models on imbalanced data.
• To accurately estimate contingency table-derived evaluation measures.

Main Methods:

• Theoretical analysis of sensitivity, specificity, and precision.
• Investigation of sampling strategies tailored to data skewness.

Main Results:

• Sampling strategies must adapt to the degree of class imbalance.
• Proposed methods improve the accuracy of performance metric estimation.
• Effective sampling is crucial for reliable model evaluation.

Conclusions:

• Adapted sampling strategies are essential for accurate performance evaluation in imbalanced classification.
• The findings guide the selection of labeled data for model assessment.