Assessing ChatGPT's Capability in Addressing Thyroid Cancer Patient Queries: A Comprehensive Mixed-Methods Evaluation

Area of Science:

• Oncology
• Endocrinology
• Artificial Intelligence in Healthcare

Background:

• Patients with thyroid cancer frequently have unmet information needs.
• Online resources, including generative AI like ChatGPT, are increasingly used by patients for health information.

Purpose of the Study:

• To evaluate the quality and reliability of ChatGPT's responses to thyroid cancer-related queries.

Main Methods:

• Twenty thyroid cancer questions were posed to ChatGPT.
• Responses were assessed by eight thyroid cancer experts (endocrinologists and endocrine surgeons).
• Evaluations included accuracy, completeness, and satisfaction using a 7-point Likert scale and qualitative analysis of comments.

Main Results:

• Only 57% of responses were deemed accurate, 56% complete, and 52% satisfactory.
• Qualitative analysis of 198 comments revealed frequent criticism.
• Key issues included overemphasis on diet/iodine and inaccurate/incomplete information on surgery and radioactive iodine therapy risks.

Conclusions:

• ChatGPT's current performance is insufficient for unsupervised patient use regarding thyroid cancer information.
• The AI tool requires significant improvement in accuracy and completeness to be a reliable resource for thyroid cancer patients.