Chatbot Responses to Frequently Asked Questions About Cannabis and Its Use for Cancer Symptoms

Chatbots are increasingly used by the public, but their performance in answering questions about complex health topics, such as cannabis, is unknown.

To evaluate responses of three popular chatbots regarding cannabis and its use for cancer-related symptoms.

We asked ChatGPT, Google Gemini, and Microsoft Co-Pilot to answer questions about cannabis derived from the Centers for Disease Control website and American Society of Clinical Oncology guidelines regarding cannabis. Responses were collected on February 6, 2025. Six physicians with expertise in this field scored responses for accuracy and comprehensiveness (0-10 scale). Reliability of references was scored separately (0-10 scale). Readability was assessed using Flesch-Kincaid Grade Level, Flesch Reading Ease scores.

Mean accuracy scores (SD) for ChatGPT, Gemini, and Co-Pilot were 9.0 (1.8), 8.8 (2.3), and 8.3 (2.3), respectively. Co-Pilot significantly underperformed in accuracy compared to ChatGPT (mean difference -0.62, 95% CI: -1.11, 0.14; P = 0.008). Mean comprehensiveness scores (SD) for ChatGPT, Gemini, and Co-Pilot were 8.1 (2.2), 8.5 (2.2), and 7.2 (2.4), respectively. ChatGPT and Gemini performed better than Co-Pilot in comprehensiveness (mean difference Co-Pilot vs. ChatGPT: -0.88 [95% CI: 1.34, -0.42; P < 0.001]; mean difference Co-Pilot vs. Gemini: -1.28 [95% CI: -1.74, -0.82; P < 0.001]). Inaccurate or misleading statements regarding cannabis formulations and symptom benefits were identified, with missing information on adverse effects and drug interactions. Gemini had the lowest reliability (4.1). Readability among all chatbots was poor.

Despite overall high accuracy and comprehensiveness scores, chatbots made some misleading, inaccurate statements or missed information. For now, their answers should be interpreted with caution.