AI-Generated Content in Cancer Symptom Management: A Comparative Analysis Between ChatGPT and NCCN

Artificial intelligence-driven tools, like ChatGPT, are prevalent sources for online health information. Limited research has explored the congruity between AI-generated content and professional treatment guidelines. This study seeks to compare recommendations for cancer-related symptoms generated from ChatGPT with guidelines from the National Comprehensive Cancer Network (NCCN).

We extracted treatment recommendations for nine symptoms from NCCN, separated into four full Supportive Care sections and five subsections of the Palliative Care webpage. We entered "How can I reduce my cancer-related [symptom]" into ChatGPT- 3.5 for these same symptoms and extracted its recommendations. A comparative content analysis focused on recommendations for medications, consultations, and non-pharmacological strategies. We compared word count and Flesch-Kincaid Grade Level (FKGL) readability for each NCCN and ChatGPT section.

The mean percent agreement between NCCN and ChatGPT recommendations was 37.3% (range 16.7%-81.8%). NCCN offered more specific medication recommendations. ChatGPT did recommend medications in the constipation and diarrhea sections that were not recommended by NCCN. Significant differences in word count (P=0.03) and FKGL (P<0.01) were found for NCCN Supportive Care webpages, with ChatGPT having lower word count and reading level. In the NCCN Palliative Care webpage subsections, there was no significant difference in word count (P=0.076), but FKGL was significantly lower with ChatGPT (P<0.01).

While ChatGPT provides concise, accessible supportive care advice, discrepancies with guidelines raise concerns for patient-facing symptom management recommendations. Future research should consider how AI can be used in conjunction with evidence-based guidelines to support cancer patients' supportive care needs.