Developing a Clinical Prediction Score: Comparing Prediction Accuracy of Integer Scores to Statistical Regression...

Area of Science:

• Biostatistics
• Clinical Prediction Modeling
• Health Informatics

Background:

• Statistical regression models are often simplified into integer scores or risk classifications for ease of use.
• This simplification process can lead to a loss of valuable information and reduced prediction accuracy.
• The impact of such simplification on model performance requires thorough investigation.

Purpose of the Study:

• To investigate the impact of simplifying regression models into integer scores on prediction accuracy.
• To compare the performance of continuous regression models versus simplified integer-score models.
• To assess the effects of dichotomization and risk stratification on model discrimination and calibration.

Main Methods:

• A simulation study with logistic regression models and continuous covariates was conducted.
• Simulated data sets (n=1000) were generated with target Area Under the Curve (AUC) values of 0.7, 0.8, or 0.9.
• Continuous variables were dichotomized, and models were refit to create integer scores and risk classifications. Performance was evaluated using AUC and Index of Prediction Accuracy (IPA).
• An external clinical data set was used to validate findings.

Main Results:

• Logistic regression models using continuous covariates consistently outperformed simplified integer-score models.
• Simplification to integer scores resulted in an average decrease of 0.057-0.094 in AUC and 6.2%-17.5% in IPA in simulations.
• The largest performance decrease occurred during the dichotomization step, which also increased model optimism.
• External validation on a clinical data set showed a decrease of 0.06 in AUC and 13% in IPA when converting to an integer score.

Conclusions:

• Converting regression models to integer scores and risk classification systems considerably decreases model performance.
• Retaining continuous variables in regression models is recommended for maximizing prediction accuracy.
• Researchers should use unaltered regression models for individual patient predictions and ensure proper validation of the intended clinical model.