Improving Models to Predict Care Utilization Using Machine Learning: Retrospective Observational Study

Area of Science:

• Health Informatics
• Machine Learning in Healthcare
• Clinical Risk Prediction

Background:

• Artificial intelligence (AI) and machine learning (ML) are increasingly used in healthcare for risk estimation.
• Legacy systems like the Johns Hopkins Adjusted Clinical Group (ACG) System, developed over years of clinical expertise, are now being evaluated with novel statistical approaches.
• The ACG System is a widely adopted method for categorizing clinical risk factors and is suitable for ML techniques.

Purpose of the Study:

• To evaluate the performance of the ACG System using machine learning algorithms.
• To compare ML model performance against traditional logistic regression using AUROC and F1 optimization strategies.
• To assess the potential for integrating ML into ACG-related clinical workflows.

Main Methods:

• A retrospective observational study was conducted using 2019 data from 350,463 patients at the Johns Hopkins Health System.
• XGBoost, random forest, and elastic net models were tuned using cross-validation to optimize AUROC and F1 scores for predicting hospitalization and total cost.
• Performance was compared against logistic regression, with detailed analysis of sensitivity, positive predictive value, and F-beta statistics.

Main Results:

• XGBoost demonstrated statistically significant, though small, improvements in cross-validated AUROC and F1-scores compared to logistic regression.
• Logistic regression achieved average AUROC values of 0.886 (cost) and 0.841 (hospitalization), while XGBoost achieved 0.891 and 0.849, respectively.
• For F1 optimization, XGBoost exceeded logistic regression for cost (0.411 vs. 0.367) but not for hospitalization (0.328 vs. 0.341).

Conclusions:

• Logistic regression is highly effective for clinical risk prediction tasks, particularly when model efficiency and interpretability are paramount.
• ML models offer diverse performance trade-offs, making them suitable for specific clinical use cases.
• Healthcare systems must align model selection and calibration with specific needs and expectations to ensure optimal performance and utility.