Toward Generalizable Machine Learning Models in Speech, Language, and Hearing Sciences: Estimating Sample Size and...

Area of Science:

• Speech, Language, and Hearing Sciences
• Computational Linguistics
• Biomedical Data Science

Background:

• Machine learning (ML) studies in speech, language, and hearing sciences often use single data splitting for cross-validation.
• This approach can lead to biased results and an overestimation of model accuracy.
• Robust data splitting is crucial for reliable ML model development in these fields.

Purpose of the Study:

• To provide quantitative evidence promoting nested k-fold cross-validation over single data splitting for ML studies.
• To present methods and MATLAB code for power analysis in ML study design.
• To improve the reliability and validity of ML applications in speech, language, and hearing research.

Main Methods:

• Compared four cross-validation methods: single holdout, 10-fold, train-validation-test, and nested 10-fold.
• Utilized real-world clinical data and Monte Carlo simulations to assess ML outcomes.
• Quantified interactions between cross-validation, feature discriminative power, feature space dimensionality, model dimensionality, and sample size.

Main Results:

• Single holdout cross-validation yielded low statistical power and confidence, inflating accuracy estimates.
• Nested 10-fold cross-validation demonstrated superior statistical confidence and power, providing unbiased accuracy.
• Nested k-fold cross-validation can reduce required sample size by up to 50% compared to single holdout.
• Statistical confidence was up to four times higher with nested k-fold cross-validation.

Conclusions:

• Nested k-fold cross-validation is essential for unbiased and robust ML studies in speech, language, and hearing sciences.
• Implementing nested k-fold cross-validation ensures more reliable and accurate ML model performance.
• The study provides tools (MATLAB code, lookup tables) to aid researchers in sample size estimation for ML studies.