Behind the Curtain: Comparing Predictive Models Performance in 2 Publicly Insured Populations

Area of Science:

• Health Services Research
• Health Informatics
• Predictive Analytics

Background:

• Predictive models are increasingly used in healthcare to forecast service utilization and patient outcomes.
• Understanding the adaptability and contextual functioning of these models across diverse populations is crucial but less explored.
• This study investigates the internal mechanisms of a large-scale predictive model in two distinct demographic groups.

Purpose of the Study:

• To examine the adaptability of a predictive model for avoidable hospitalizations across different populations.
• To elucidate the inner workings and performance variations of a health predictive model in distinct demographic contexts.
• To emphasize the importance of population-specific model training and adaptation.

Main Methods:

• Compared the performance and functioning of an avoidable hospitalization predictive model in Medicaid and Medicare enrollees.
• Assessed risk score characteristics, predictive accuracy, and key risk factors for both populations in March 2022.
• Evaluated an "unadapted" model by applying Medicare coefficients to the Medicaid population to gauge adaptation impact.

Main Results:

• The predictive model demonstrated successful adaptation and strong performance in both Medicaid and Medicare populations.
• Significant variations were observed in the primary risk factors and their weightings between the two populations.
• An unadapted model applied to the Medicaid population showed substantially poorer performance compared to the adapted model.

Conclusions:

• Predictive models require "peeking behind the curtain" to understand their function in different populations.
• Risk prediction is not a "one size fits all" solution; models must be tailored to the target population.
• Optimal performance of predictive models necessitates adaptation and training on the specific population for which they are intended.