A Probabilistic Method to Predict Classifier Accuracy on Larger Datasets given Small Pilot Data.

Area of Science:

• Machine Learning
• Computational Statistics

Background:

• Classifier development often begins with limited data, with plans for future expansion.
• Accurate prediction of performance gains from increased dataset size is essential for resource allocation and project planning.

Purpose of the Study:

• To propose a novel method for probabilistic extrapolation of classifier performance metrics as dataset size grows.
• To address the limitations of existing deterministic extrapolation methods by incorporating uncertainty assessment.

Main Methods:

• Development of a Gaussian process model to predict performance metrics (e.g., accuracy) as a function of dataset size.
• Evaluation of the model's performance using error, likelihood, and coverage metrics across six diverse datasets.

Main Results:

• The proposed Gaussian process model provides reliable probabilistic extrapolations of classifier performance.
• The model effectively quantifies the uncertainty associated with accuracy predictions at different dataset sizes.
• Empirical evaluation across six datasets demonstrates the model's robustness and generalizability.

Conclusions:

• Gaussian process modeling offers a superior approach to extrapolating classifier performance compared to traditional methods.
• Incorporating uncertainty in performance predictions is critical for practitioners managing data-intensive machine learning projects.
• The open-source nature of this approach allows for broad applicability across various classification tasks and data modalities.