Assessing model accuracy using random data split: a simulation study

Area of Science:

• Biostatistics
• Machine Learning in Healthcare
• Clinical Trial Methodology

Background:

• Randomization is a cornerstone in clinical studies to mitigate bias.
• Assessing model generalizability often involves splitting data into training and testing sets.
• The effectiveness of random data splitting for evaluating model generalizability requires scrutiny.

Purpose of the Study:

• To investigate the limitations of random data splitting in assessing model generalizability.
• To evaluate how random splits impact model performance metrics under different data conditions.
• To determine optimal strategies for testing model generalizability.

Main Methods:

• Conducted simulation studies using three distinct datasets (binary and continuous endpoints).
• Employed large sample sizes (n=10,000) for each simulation.
• Repeated random data splits (training/testing) 1,000 times per scenario to compare model performance.

Main Results:

• Random splits showed similar model performance (true/false positive fractions, mean-squared errors) between training and testing sets.
• Significant discrepancies emerged when time drift effects were present in the data.
• Model performance on randomly split data does not reliably indicate generalizability in the presence of temporal changes.

Conclusions:

• Random data splitting is insufficient for robustly assessing model generalizability, particularly with time-varying data.
• Evaluating model accuracy requires testing on data distinct from the training set.
• Independent, separate studies are recommended for validating model generalizability.