Assessing the Adequacy of a Prediction Model

Many models claim to predict outcomes with good accuracy. However, not many seem to be adopted in practice. This could be because most of them do not have sufficient predictive accuracy. We analyzed 20 recently published papers on prediction models and found that most use inadequate measures to assess predictive performance. These measures primarily include the area under the ROC curve (C-index) that measures discrimination and not predictivity, that too accepting a relatively low value, and using aggregate concordance for assessing predictive accuracy instead of individual-based agreement between the observed and predicted values. Some use arbitrary scores in their models, consider only binary outcomes where multiple categories could be more useful, misinterpret P values, ignore future dynamics, use inappropriate validation settings, and do not fully consider the process of the outcomes. We give details of all these inadequacies and suggest remedies so that models with adequate predictive performance can be developed.