Estimating classification accuracy in positive-unlabeled learning: characterization and correction strategies.

Area of Science:

• Biomedical research
• Machine learning
• Data science

Background:

• Accurate performance estimation is crucial for machine learning classifiers in biomedical research.
• Traditional methods fail when training data violate statistical assumptions, particularly in open-world settings.
• Positive-unlabeled (PU) learning, a semi-supervised approach, handles datasets where only positive examples are labeled.

Purpose of the Study:

• To investigate the accuracy of performance estimation in positive-unlabeled learning within the biomedical domain.
• To identify factors influencing the inaccuracy of performance estimates in PU learning.
• To develop and validate correction methods for improving performance estimation in PU learning.

Main Methods:

• Analysis of performance estimation quality in PU learning scenarios.
• Identification of key parameters affecting estimate accuracy: unlabeled positive fraction and labeled negative mislabeling rate.
• Development of correction methods for four common performance measures.
• Theoretical and empirical validation of the proposed correction techniques.

Main Results:

• Performance estimates in PU learning can be significantly inaccurate.
• Inaccuracy is dependent on the proportion of positive examples in unlabeled data and mislabeled negatives in labeled data.
• Correction methods effectively recover true classification performance.
• Accurate estimates of class priors and label noise are sufficient for performance recovery.

Conclusions:

• Standard performance estimation methods are unreliable for PU learning in biomedical applications.
• The proposed correction methods, utilizing prior knowledge and noise estimation, enhance the reliability of performance evaluation.
• This work improves best practices for machine learning model assessment in challenging data settings.