Machine Learning Model Validation for Early Stage Studies with Small Sample Sizes.

Area of Science:

• Biomedical research
• Machine learning
• Data science

Background:

• Small datasets are prevalent in early-stage biomedical studies due to high costs and difficulties in human subject sample collection.
• Validating machine learning models is challenging with limited data, impacting their reliability and interpretation.

Purpose of the Study:

• To develop a protocol for estimating and interpreting early-stage machine learning model performance on small biomedical datasets.
• To identify optimal feature selection techniques, validation frameworks, and learning curve fitting strategies for small sample sizes.

Main Methods:

• Examined various feature selection techniques, validation frameworks (including nested cross-validation), and learning curve fitting.
• Utilized small simulated datasets with known underlying discriminability to assess methods.
• Evaluated the accuracy of feature selection and model performance estimation.

Main Results:

• Nested cross-validation best reflected feature discriminability but often failed to identify relevant features in small datasets.
• Filter-based feature selection methods were recommended to reduce overfitting to noise.
• Statistical exploration of datasets is crucial for estimating discriminability and problem feasibility.

Conclusions:

• Recommend filter-based feature selection, statistical exploration for discriminability assessment, and learning curves with nested cross-validation for performance forecasting.
• This protocol guides researchers in applying machine learning to small-scale pilot studies effectively.