Developing and testing a model to predict outcomes of organizational change

Area of Science:

• Healthcare Management
• Decision Science
• Predictive Analytics

Background:

• Healthcare improvement projects often face challenges in predicting success.
• Subjective probability estimates can be valuable in complex decision-making.
• Existing models may not fully capture the nuances of healthcare change initiatives.

Purpose of the Study:

• To evaluate a Bayesian model that utilizes subjective probability estimates for predicting the success or failure of healthcare improvement projects.
• To assess the model's predictive accuracy using real-world project data.

Main Methods:

• Developed a Bayesian model incorporating subjective likelihood ratios and prior odds from a panel of experts.
• Validated the model using retrospective data from 221 healthcare improvement projects across North America and Europe (1996-2000).
• Employed logistic regression and Receiver Operating Characteristic (ROC) analysis to assess model performance.

Main Results:

• The Bayesian model demonstrated high predictive performance, with significant logistic regression chi-square statistics (p < 0.001).
• Areas under the ROC curve exceeded 0.84, indicating strong discriminative ability for predicting project success.
• The model effectively predicted outcomes across three different definitions of success.

Conclusions:

• A Bayesian model leveraging subjective expert probability estimates proved effective in forecasting the success of healthcare improvement projects.
• Further prospective studies are recommended to confirm these findings.
• The potential of this model as an intervention to improve project outcomes warrants investigation.