Multivariate modeling of complications with data driven variable selection: guarding against overfitting and effects...

Area of Science:

• Medical Statistics
• Radiotherapy Oncology
• Machine Learning in Healthcare

Background:

• Multivariate modeling of radiotherapy complications often uses data-driven variable selection.
• Assessing the true predictive power of these models is crucial for clinical application.

Purpose of the Study:

• To quantify the risk of overfitting in data-driven modeling methods using bootstrapping.
• To estimate the minimum data size required for reliable predictive models in radiotherapy.

Main Methods:

• Simulated clinical data sets (50-1000 patients) mimicking radiotherapy head and neck cancer patient data.
• Logistic regression with bootstrapping and forward variable selection for complication modeling.
• Cross-validation with large independent datasets to determine true predictive power.

Main Results:

• Bootstrapping selected variables close to optimal but with spread; predictive power did not significantly differ from AIC/BIC.
• Severe overfitting observed in small datasets, particularly with few events.
• Over half of potential predictive power is achieved with ~200 samples; predictive power plateaus thereafter.

Conclusions:

• Bootstrapping is accurate for sufficiently large datasets, guarding against overfitting except in low-event scenarios.
• A minimum of ~200 patients and >32 events is recommended for high predictive power.
• Predictive power rapidly declines with fewer samples; benefits level off with larger datasets.