Predicting sample size required for classification performance.

Area of Science:

• Machine Learning
• Data Science
• Computational Biology

Background:

• Supervised learning requires annotated data, which is scarce and costly.
• Accurate estimation of required annotated sample size is needed for both passive and active learning strategies.

Purpose of the Study:

• To develop and evaluate a novel method for predicting the necessary sample size for supervised machine learning models.
• To improve the efficiency of model development by accurately estimating annotation needs.

Main Methods:

• An inverse power law model was fitted to learning curves using nonlinear weighted least squares optimization.
• The fitted model predicted classifier performance and confidence intervals for larger sample sizes.
• The method was evaluated on clinical text and waveform classification tasks, comparing weighted and un-weighted fitting.

Main Results:

• The weighted fitting method accurately predicted model performance across various datasets and sampling strategies.
• Between 80 to 560 annotated samples were sufficient to achieve low error rates (MSE < 0.01).
• The weighted fitting approach demonstrated statistically significant improvement over the un-weighted method (p < 0.05).

Conclusions:

• A simple, effective algorithm for sample size prediction in supervised machine learning was developed.
• The weighted fitting algorithm offers superior performance compared to un-weighted methods.
• This tool aids researchers in determining optimal annotation sample sizes, enhancing resource allocation.