Bayesian Estimate of Mean Proper Scores for Diversity-Enhanced Active Learning

Area of Science:

• Machine Learning
• Computer Vision
• Natural Language Processing

Background:

• Active learning (AL) effectiveness hinges on acquisition function sampling efficiency.
• Expected Loss Reduction (ELR) estimates Bayesian reduction in classification error.
• General costs can be incorporated within the ELR framework.

Purpose of the Study:

• Propose Bayesian Estimate of Mean Proper Scores (BEMPS) to estimate increases in strictly proper scores.
• Develop a batch AL algorithm promoting diversity in expected score changes.
• Combine deep ensembles, dynamic validation sets, and Monte Carlo Dropout for high-performance classifiers.

Main Methods:

• Implemented BEMPS for estimating score increases within the ELR framework.
• Developed a batch AL algorithm with diversity encouragement.
• Utilized deep ensembles with dynamic validation and Monte Carlo Dropout for classifier enhancement.
• Conducted extensive experiments on text and image datasets.

Main Results:

• BEMPS with mean square error and log probability demonstrated robust acquisition functions.
• Classifiers produced by BEMPS were well-calibrated and outperformed existing methods.
• Qualitative analyses using data maps and t-SNE plots supported BEMPS's advantages.

Conclusions:

• BEMPS offers a powerful and versatile approach for active learning acquisition functions.
• The proposed methods lead to improved classifier performance and calibration.
• BEMPS provides a strong alternative to existing acquisition functions in active learning.