Machine learning algorithm validation with a limited sample size.

Area of Science:

• Machine learning
• Biostatistics
• Neuroscience

Background:

• High-dimensional datasets with few samples are common in human participant studies.
• Small sample sizes can lead to biased machine learning performance estimates.
• Previous studies show small sample sizes correlate with higher classification accuracy in autism prediction.

Purpose of the Study:

• Investigate bias in machine learning performance estimates due to small sample sizes.
• Evaluate the impact of different validation methods on bias.
• Identify robust methodologies for small dataset analysis.

Main Methods:

• Simulations were used to assess bias in machine learning validation methods.
• K-fold Cross-Validation (CV), Nested CV, and train/test split were compared.
• The influence of feature selection, data dimensionality, and hyper-parameter space was explored.

Main Results:

• K-fold CV yields biased performance estimates, even with up to 1000 samples.
• Nested CV and train/test split offer unbiased estimates irrespective of sample size.
• Feature selection on pooled data significantly increases bias compared to parameter tuning.

Conclusions:

• Standard K-fold CV is unreliable for small, high-dimensional datasets.
• Nested CV and train/test splits are recommended for robust machine learning validation.
• Careful consideration of validation and feature selection methods is crucial for accurate results.