Assessing ChatGPT for Clinical Decision-Making in Radiation Oncology, With Open-Ended Questions and Images

Area of Science:

• Artificial Intelligence in Medicine
• Radiation Oncology
• Medical Imaging Analysis

Background:

• Large language models (LLMs) are increasingly explored for clinical decision support.
• Assessing the accuracy and practicality of LLMs in specialized medical fields is crucial.
• Radiation oncology presents unique challenges due to complex data and treatment protocols.

Purpose of the Study:

• To evaluate the correctness and clinical utility of ChatGPT-4 and ChatGPT-4O responses to radiation oncology inquiries.
• To assess ChatGPT-4O's capability in staging nasopharyngeal carcinoma (NPC) using magnetic resonance (MR) images.

Main Methods:

• 164 open-ended clinical questions across four domains (Clinical_G, Clinical_C, Nursing, Technology) were posed to ChatGPT-4 and ChatGPT-4O.
• Answers were graded for practicality and correctness.
• ChatGPT-4O analyzed MR images of 20 NPC patients for T-stage determination.

Main Results:

• Both ChatGPT-4 and ChatGPT-4O demonstrated high correctness rates (GPT-4: 89.6%, GPT-4O: 98.8%) across all domains.
• Practicality varied by domain, with higher scores in Nursing and Clinical_G.
• ChatGPT-4O inaccurately staged all NPC cases from MR images, assigning T4 to all cases.

Conclusions:

• ChatGPT can serve as a safe clinical decision-support tool in radiation oncology, given its high accuracy in answering inquiries.
• Clinical practicality requires careful consideration, especially in complex clinical scenarios and technology-related domains.
• ChatGPT-4O is not yet suitable for interpreting diagnostic images for cancer staging.