Large Language Models Versus Expert Clinicians in Crisis Prediction Among Telemental Health Patients: Comparative...

Area of Science:

• Artificial Intelligence in Healthcare
• Mental Health Technology
• Clinical Decision Support Systems

Background:

• Large language models (LLMs) are advancing rapidly for language tasks.
• Limited research exists on LLMs for mental health crisis prediction.
• AI tools can analyze provider-patient interactions but their role in crisis prediction is underexplored.

Purpose of the Study:

• To evaluate GPT-4's performance in predicting mental health crises.
• To compare AI prediction accuracy against clinicians using patient intake data.
• To assess LLM capabilities in identifying patients at risk for suicidal ideation with a plan.

Main Methods:

• Utilized deidentified patient intake data from a telemental health platform.
• Included patients with and without suicidal ideation (SI) and SI with a plan.
• Compared GPT-4's predictions to those of 6 senior clinicians using chief complaint and suicide attempt history.

Main Results:

• Clinicians had higher average precision than GPT-4 for predicting SI with plan at intake (0.7 vs 0.6).
• GPT-4 showed higher sensitivity than clinicians when using chief complaint alone (0.62 vs 0.53).
• Adding suicide attempt history improved clinician performance but decreased GPT-4 precision; both improved in predicting future SI with plan.

Conclusions:

• GPT-4 performance approached clinician levels on some metrics with simple prompts.
• Further research and bias safety checks are required before clinical piloting.
• LLMs show potential for augmenting high-risk patient identification and improving timely care delivery.