Predicting non-emergency healthcare use in Australia using machine learning on longitudinal household data

Area of Science:

• Health Economics
• Machine Learning in Healthcare
• Public Health Policy

Background:

• Rising global healthcare expenditure challenges public financing sustainability.
• Identifying at-risk populations is crucial for targeted healthcare interventions.
• Predicting healthcare use is essential for effective resource allocation.

Purpose of the Study:

• To apply and compare machine learning (ML) methods with logistic regression for predicting non-emergency healthcare use.
• To identify key socio-economic and health-related predictors of healthcare utilization.
• To evaluate the predictive performance and calibration of different ML models.

Main Methods:

• Utilized three waves of the HILDA survey with a health module, analyzing 47,899 observations and 741 variables.
• Applied Random Forest, Gradient Boosting Decision Trees, Extreme Gradient Boosting, and multilayer perceptron neural networks.
• Evaluated models using accuracy, sensitivity, specificity, AUC, PPV, NPV, MCC, and Brier score; employed LIME and SHAP for interpretability.

Main Results:

• Socio-economic factors (age, status, insurance) and health variables (prior contact, designated doctor) significantly predicted healthcare use.
• Gradient Boosting Decision Trees demonstrated superior prediction performance over logistic regression across all waves.
• ML models achieved higher AUC (0.68-0.76) and PPV (75-77%) compared to logistic regression (AUC 0.69, PPV 71%).

Conclusions:

• Machine learning techniques provide robust estimates for predicting future healthcare use (primary and inpatient elective care).
• Identified predictors can inform targeted policies to manage healthcare expenditure.
• The study highlights the value of ML on large, longitudinal data for healthcare resource allocation and priority setting.