Adaptive sampling methods facilitate the determination of reliable dataset sizes for evidence-based modeling

Area of Science:

• Machine Learning
• Statistical Modeling
• Data Science

Background:

• Assessing model reliability on unseen data is crucial for trustworthy predictions.
• Determining adequate sample size prevents significant conclusion variations across datasets.
• The law of large numbers suggests model accuracy converges with sufficient data, assuming no data drift.

Purpose of the Study:

• To systematically examine the relationship between data size and model accuracy gains.
• To develop a heuristic method for investigating convergence speed relative to sample size.
• To ensure reliable conclusions about data requirements by minimizing sampling deviations.

Main Methods:

• Utilized sampling methods to estimate the convergence speed concerning data sample size.
• Implemented an automated approach to determine the optimal number of repetitions for stabilizing results.
• Reduced sampling deviations below a predefined threshold to ensure result consistency.

Main Results:

• Established a method to analyze the speed of model convergence with increasing data size.
• Demonstrated how to stabilize convergence speed results across multiple sampling runs.
• Quantified the reduction in sampling deviations to ensure reliable data requirement conclusions.

Conclusions:

• The presented heuristic method reliably determines sufficient data size for model stability.
• Automated repetition determination enhances the trustworthiness of conclusions regarding data needs.
• This approach ensures predictions remain reliable on unseen data and model conclusions are robust.