Sample Size Requirements for Popular Classification Algorithms in Tabular Clinical Data: Empirical Study.

Area of Science:

• Machine Learning
• Biostatistics
• Clinical Informatics

Background:

• Classification algorithm performance plateaus, necessitating optimal sample size determination.
• Balancing performance gains with computational costs is essential for efficient research.

Purpose of the Study:

• To determine optimal sample sizes for binary classification algorithms.
• To investigate the relationship between sample size and dataset characteristics across various algorithms.

Main Methods:

• Assessed 4 algorithms (XGBoost, Random Forest, Logistic Regression, Neural Networks) on 16 diverse datasets.
• Calculated cross-validated Area Under the Curve (AUC) at increasing sample sizes to fit learning curves.
• Quantified relationships between dataset characteristics and required sample sizes using regression models.

Main Results:

• Median sample sizes to reach AUC stability varied: XGBoost (9960), Random Forest (3404), Logistic Regression (696), Neural Networks (12,298).
• Increased class balance and dataset complexity generally decreased or increased sample size requirements, respectively.
• Minority class proportion was a key predictor across all algorithms; dataset AUC and nonlinearity were important for XGBoost, Random Forest, and Neural Networks.

Conclusions:

• Optimal sample sizes for classification algorithms are method- and dataset-dependent.
• Dataset characteristics, such as class balance and feature complexity, can predict and potentially be leveraged to optimize sample size selection for research studies.