Assessing Generative AI Chatbots for Alcohol Misuse Support: A Longitudinal Simulation Study

Area of Science:

• Digital Health
• Artificial Intelligence in Healthcare
• Mental Health Technology

Background:

• Large language model (LLM) chatbots are increasingly utilized for behavioral health support.
• Rigorous evaluations of their advice on alcohol misuse are limited.

Purpose of the Study:

• To evaluate the performance of publicly available chatbots in responding to alcohol misuse scenarios.
• To assess chatbot interactions across domains including empathy, information quality, and usefulness.

Main Methods:

• Seven chatbots (general and behavioral health-focused) were tested using simulated longitudinal interactions over seven days.
• Clinicians rated transcripts based on empathy, information quality, usefulness, responsiveness, and scope awareness.
• Secondary assessments included stigmatizing language and validation versus challenging the user.

Main Results:

• Empathy was the highest-rated domain (mean score 4.6/5), while quality of information was the lowest (mean 2.7/5).
• Performance varied significantly across chatbots, with no difference between general and behavioral health types.
• All chatbots provided some inappropriate, overstated, or inaccurate guidance, despite avoiding stigmatizing language.

Conclusions:

• Chatbots demonstrate strengths in empathy but require significant improvement in the quality and accuracy of information provided for alcohol misuse support.
• Users and clinicians should be aware of current chatbot limitations in this domain.
• Further development is needed to enhance the reliability of AI-driven behavioral health tools.