Minimum Sample Size Requirements for Machine Learning: A Study on Diabetic Neuropathy Prediction.

Area of Science:

• Medical Informatics
• Computational Biology
• Clinical Prediction Models

Background:

• Diabetic neuropathy (DN) detection is challenging due to asymptomatic progression.
• Machine learning (ML) shows promise for identifying at-risk patients.
• Existing ML studies often lack generalizability due to small sample sizes.

Purpose of the Study:

• To determine minimum sample size requirements for ML models in DN prediction.
• To evaluate the impact of feature dimensionality on model performance.
• To provide guidance for clinical ML studies, especially in data-limited settings.

Main Methods:

• Utilized a large population-based dataset (n=77,724) of individuals with diabetes.
• Generated balanced subsets with varying sample sizes (100-25,000) and feature counts (3-46).
• Trained random forest models and assessed performance using ROC AUC and PR AUC metrics.

Main Results:

• Models trained on ≤500 samples exhibited significant overfitting and poor generalization.
• At n=1,000, performance was comparable to the reference model with limited features (3), but overfitting increased with more features (≥20).
• Stable performance without overfitting was achieved with sample sizes ≥3,000.

Conclusions:

• Approximately 3,000 samples are necessary for reliable DN prediction using random forest models.
• Constraining feature dimensionality is critical when working with smaller cohorts.
• Findings offer practical guidance on data sufficiency and model design for clinical ML applications.