Evaluating ChatGPT Responses on Thyroid Nodules for Patient Education

Area of Science:

• Artificial Intelligence in Medicine
• Patient Education
• Endocrinology

Background:

• Artificial intelligence (AI) chatbots like ChatGPT are rapidly gaining popularity.
• Patients increasingly use online resources for health information.
• The quality of AI-generated patient education materials requires evaluation.

Purpose of the Study:

• To assess the accuracy and readability of ChatGPT responses on thyroid nodules.
• To compare the impact of different prompting strategies on response quality.
• To evaluate the veracity of references provided by ChatGPT.

Main Methods:

• ChatGPT was queried with 30 identical questions on thyroid nodules using four different prompt types.
• Responses were scored for accuracy and analyzed for Flesch-Kincaid grade level.
• Reference veracity and accuracy were assessed.

Main Results:

• Overall, 69.2% of responses were at least correct, with no significant difference based on prompt type.
• Eighth-grade level prompting resulted in the lowest mean grade level (13.43).
• References were generated for all queries when requested, with 87.5% being legitimate citations and 72.5% accurately reflecting the source content.

Conclusions:

• ChatGPT generally provides appropriate information on thyroid nodules.
• Response readability often exceeds recommended levels for patient education.
• AI hallucinations and high reading levels currently limit ChatGPT's utility as a patient educational tool.