Comparative effectiveness of total population versus disease-specific neural network models in predicting medical...

The objective of this research was to compare the accuracy of two types of neural networks in identifying individuals at risk for high medical costs for three chronic conditions. Two neural network models-a population model and three disease-specific models-were compared regarding effectiveness predicting high costs. Subjects included 33,908 health plan members with diabetes, 19,264 with asthma, and 2,605 with cardiac conditions. For model development/ testing, only members with 24 months of continuous enrollment were included. Models were developed to predict probability of high costs in 2000 (top 15% of distribution) based on 1999 claims factors. After validation, models were applied to 2000 claims factors to predict probability of high 2001 costs. Each member received two scores-population model score applied to cohort and disease model score. Receiver Operating Characteristic (ROC) curves compared sensitivity, specificity, and total performance of population model and three disease models. Diabetes-specific model accuracy, C = 0.786 (95%CI = 0.779-0.794), was greater than that of population model applied to diabetic cohort, C = 0.767 (0.759-0.775). Asthma-specific model accuracy, C = 0.835 (0.825-0.844), was no different from that of population model applied to asthma cohort, C = 0.844 (0.835-0.853). Cardiac-specific model accuracy, C = 0.651 (0.620-0.683), was lower than that of population model applied to cardiac cohort, C = 0.726 (0.697-0.756). The population model predictive power, compared to the disease model predictive power, varied by disease; in general, the larger the cohort, the greater the advantage in predictive power of the disease model compared to the population model. Given these findings, disease management program staff should test multiple approaches before implementing predictive models.