Practical experiences on the necessity of external validation.

Area of Science:

• Clinical Epidemiology
• Biostatistics
• Health Services Research

Background:

• Prognostic model validity is crucial for clinical application.
• Generalizability, including temporal and geographic transportability, is key.
• Stroke patient functional independence prediction is an important clinical challenge.

Purpose of the Study:

• To evaluate the prognostic performance of models predicting functional independence in stroke patients.
• To assess temporal transportability (performance over time) and geographic transportability (performance across different centers).
• To compare the effectiveness of internal validation techniques, including leave-one-center-out cross-validation (CV), for estimating model generalizability.

Main Methods:

• Developed prognostic models (logistic regression, support vector machines, random forests) using a training dataset of stroke patients.
• Employed tenfold cross-validation (CV) and leave-one-center-out CV for performance estimation.
• Validated models on independent datasets from a later time point (temporal) and different medical centers (geographic).

Main Results:

• Classical internal validation accurately predicted model performance for temporal validation (later time point).
• All tested approaches struggled to predict geographic transportability (performance across different clinics).
• Leave-one-center-out CV provided more accurate estimates of transportability than classical CV.

Conclusions:

• External validation in diverse clinical settings is essential before implementing prognostic models in practice.
• Temporal validation alone is insufficient to guarantee a model's performance in different geographic locations.
• Prognostic models require rigorous geographic validation to ensure reliable clinical decision-making.