Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints.

Area of Science:

• Biostatistics
• Machine Learning in Healthcare
• Predictive Modeling

Background:

• Classical statistical methods are widely used for binary outcome prediction.
• Modern machine learning techniques may offer improved predictive accuracy.
• The data requirements (
• data hungriness
• ) of different modeling techniques are not well understood.

Purpose of the Study:

• To compare the predictive performance of modern and classical modeling techniques.
• To evaluate how effective sample size influences model performance.
• To define and assess
• data hungriness
• across various algorithms.

Main Methods:

• Simulation studies using three clinical cohorts (head and neck cancer, traumatic brain injury, minor head injury).
• Comparison of logistic regression (LR), classification and regression trees (CART), support vector machines (SVM), neural nets (NN), and random forests (RF).
• Generation of artificial datasets with varying sample sizes and application of models to assess Area Under the ROC Curve (AUC) and optimism.

Main Results:

• Logistic regression achieved stable AUC at 20-50 events per variable, followed by CART, SVM, NN, and RF.
• Optimism decreased with increasing sample size across all techniques.
• Random forests, SVM, and NN models exhibited instability and high optimism even with over 200 events per variable.

Conclusions:

• Modern techniques (SVM, NN, RF) require significantly more data (over 10x events per variable) for stable performance compared to classical methods (LR).
• Classical techniques like logistic regression are more data-efficient for achieving reliable predictions.
• Advanced modeling techniques should be reserved for medical prediction problems with very large available datasets.