Assessing the Quality and Reliability of ChatGPT's Responses to Radiotherapy-Related Patient Queries: Comparative...

Area of Science:

• Oncology
• Artificial Intelligence in Medicine
• Medical Informatics

Background:

• Patients increasingly use online resources for cancer information, but accuracy and readability are often lacking.
• Artificial intelligence (AI) chatbots like ChatGPT offer potential for accessing medical information, including radiotherapy details.
• The quality and reliability of AI-generated radiotherapy information for patients remain unclear, posing risks of misinformation.

Purpose of the Study:

• To evaluate the quality and reliability of ChatGPT's responses to common radiotherapy questions.
• To compare the performance of two ChatGPT versions, GPT-3.5 and GPT-4, in providing radiotherapy information.

Main Methods:

• 40 common radiotherapy questions were posed to GPT-3.5 and GPT-4.
• Radiotherapy experts rated response quality using the General Quality Score (GQS).
• Response consistency (cosine similarity), readability (Flesch scores), and statistical significance were analyzed.

Main Results:

• GPT-4 outperformed GPT-3.5, achieving higher GQS and fewer low-quality ratings.
• Both versions showed high response similarity (median cosine similarity 0.81).
• Readability scores indicated college-level text, challenging for the general public.

Conclusions:

• GPT-4 demonstrates superior capability in answering radiotherapy queries compared to GPT-3.5.
• Both AI models provide information that is difficult for the general public to understand.
• ChatGPT shows promise for patient radiotherapy information but requires strategies to improve readability and mitigate misinformation risks.