Influence of Continuous Predictor Modelling Methods on Prediction Stability in Clinical Prediction Model Development:...

Prediction stability is increasingly recognised as important for reliable clinical prediction model development, but the effect of continuous predictor modelling choices remains unclear. This study examined how these choices influence prediction stability.

From a real clinical dataset of 19,418 emergency department patients we created five sample size scenarios (n = 437 to 8739), or 0.5 to 10 times the minimum required sample size. Seven modelling approaches were compared: median dichotomisation (DIC), tertiles (TER), linear (LIN), quadratic (QUA), restricted cubic splines (RCS), multivariable fractional polynomials (MFP), and extreme gradient boosting (XGB). For each scenario, 100 independent development samples were drawn from the full cohort to quantify Monte Carlo uncertainty, and within each, 200 bootstrap resamples served for internal validation and for assessing stability. For each individual, the mean absolute prediction error (MAPE) was the absolute difference between bootstrap-model and development-model predicted probabilities, averaged over the 200 resamples. A model met the stability criterion when at least 90% of individuals had MAPE ≤5%. Results were summarised across the 100 samples with Monte Carlo standard errors.

Prediction stability increased with sample size but differed by method. At n = 437, no method met the criterion in any development sample. LIN met it in all 100 samples from the base size (n = 874) onwards, and DIC in 79. QUA and RCS required approximately twice that size, MFP and XGB four times, and TER ten times. XGB ranked highest on optimism-corrected AUC at every size, but that value was greatest in the smallest samples; under out-of-bag correction XGB led only at the two largest. Its calibration slope reached 1.157 at n = 8739.

In this empirical investigation, continuous predictor modelling methods appeared to influence prediction stability. LIN was consistently stable from the base sample size onwards, whereas more flexible approaches (eg, QUA, RCS, and MFP) and XGB generally required larger samples. These findings suggest that simpler methods may provide more stable predictions in smaller datasets, although borderline assessments of stability should be interpreted with caution.