Chatbot Responses to Frequently Asked Questions About Cannabis and Its Use for Cancer Symptoms

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Cannabis Research

Background:

• Public use of chatbots is rising, yet their efficacy in addressing complex health topics like cannabis remains unclear.
• Cannabis is increasingly explored for managing cancer-related symptoms.

Purpose of the Study:

• To assess the accuracy, comprehensiveness, and readability of responses from ChatGPT, Google Gemini, and Microsoft Co-Pilot concerning cannabis and its use in oncology.
• To evaluate the reliability of references provided by these AI models.

Main Methods:

• Questions about cannabis were formulated based on CDC and ASCO guidelines.
• Six physician experts scored chatbot responses for accuracy and comprehensiveness (0-10 scale).
• Reference reliability and readability (Flesch Kincaid, Flesch Reading Ease) were also assessed.

Main Results:

• ChatGPT and Gemini demonstrated higher comprehensiveness scores than Co-Pilot.
• While accuracy scores were generally high, Co-Pilot significantly underperformed compared to ChatGPT.
• Inaccurate statements, omissions regarding adverse effects/drug interactions, low reference reliability (Gemini), and poor readability were noted across all chatbots.

Conclusions:

• AI chatbots offer generally high accuracy and comprehensiveness for cannabis-related health queries.
• However, chatbots may provide misleading or inaccurate information and lack crucial details on safety and interactions.
• Users must critically evaluate chatbot-generated health information, especially regarding cannabis.