Can Large Language Models Address Problem Gambling? Expert Insights from Gambling Treatment Professionals

Large Language Models (LLMs) have transformed information retrieval for humans. People are increasingly turning to general-purpose LLM-based chatbots to find answers to questions across numerous domains, including advice on sensitive topics such as mental health and addiction. In this study, we present the first inquiry into how LLMs respond to prompts related to problem gambling, specifically exploring how experienced gambling treatment professionals interpret and reflect on these responses. We used the Problem Gambling Severity Index to develop nine prompts related to different aspects of gambling behavior. These prompts were submitted to two LLMs, GPT-4o (via ChatGPT) and Llama 3.1 405b (via Meta AI), and their responses were evaluated via an online survey distributed to human experts (experienced gambling treatment professionals). Twenty-three experts participated, representing over 17,000 hours of problem gambling treatment experience. They provided their own responses to the prompts and selected their preferred (blinded) LLM response, along with contextual feedback, which was used for qualitative analysis. Llama was slightly preferred over GPT, receiving more votes for 7 out of the 9 prompts. Thematic analysis revealed that experts identified strengths and weaknesses in LLM responses, highlighting issues such as encouragement of continued gambling, overly verbose messaging, and language that could be easily misconstrued. These findings offer a novel perspective by capturing how experienced gambling treatment professionals perceive LLM responses in the context of problem gambling, providing insights to inform future efforts to align these tools with appropriate guardrails and safety standards for use in gambling harm interventions.