Explainable AI for mental health emergency returns: integrating large language models with predictive modeling

Emergency department (ED) returns for mental health (MH) conditions pose a substantial burden on healthcare systems. Traditional machine learning (ML) models have shown promise in predicting ED returns but often fall short in clinical interpretability. This study evaluates whether integrating Large Language Models (LLMs) with ML methods can enhance both predictive performance and interpretability.

We conducted a retrospective analysis of 42 464 ED visits involving 27 904 unique MH patients at an academic medical center in the deep south (2018-2022). Two primary outcomes were assessed: (1) 30-day ED return prediction, and (2) interpretability of model outputs. We employed LLaMA 3 (8B) for few-shot classification of chief complaints and social determinants of health (SDoH) features, including tobacco, alcohol, substance use, exercise, and home environment. These LLM-derived features were integrated into an XGBoost model. Model explanation was enhanced using a novel LLM-SHAP hybrid framework.

The LLM-based chief complaint classifier achieved 0.882 accuracy, 0.95 precision, 0.88 recall, and 0.86 F1-score, outperforming traditional models. SDoH classifiers reached F1-scores ranging from 0.67 to 0.96 across different categories. Incorporating LLM-extracted features into XGBoost improved AUC from 0.74 to 0.76 and AUC-PR from 0.58 to 0.61. The LLM-SHAP framework provided enhanced contextualized explanations for risk predictions, improving interpretability at both patient and population levels.

Integrating LLMs with traditional ML models modestly improved predictive performance but substantially enhanced interpretability. This hybrid approach offers a promising path toward more actionable and explainable clinical decision support tools in emergency psychiatry.