Exploring Capabilities of Large Language Models such as ChatGPT in Radiation Oncology

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing Applications
• Radiation Oncology Knowledge Assessment

Background:

• Advancements in machine learning and NLP have enabled sophisticated large language models (LLMs).
• Conversational LLMs, such as ChatGPT, demonstrate significant capabilities across various domains, including specialized medical fields.
• The potential application of LLMs in assessing medical knowledge, specifically in radiation therapy, warrants investigation.

Purpose of the Study:

• To explore the capabilities of ChatGPT in answering questions related to radiation therapy.
• To evaluate the accuracy and quality of LLM-generated responses in a specialized medical context.

Main Methods:

• A set of multiple-choice and open-ended questions covering clinical, physics, and biology aspects of radiation oncology was developed.
• ChatGPT was prompted with these questions, and its responses were collected.
• Multiple-choice answers were assessed for correctness, while open-ended answers were evaluated by radiation oncologists for correctness and usefulness on a Likert scale.

Main Results:

• ChatGPT provided valid answers for 94.3% of multiple-choice questions, with an overall correct answer rate of 60.61% (varying by subspecialty).
• For open-ended questions, 12 out of 25 responses were rated as acceptable, good, or very good by all evaluators.
• Evaluators found ChatGPT's responses to be "very good" in 28-29.3% and "good" in 28-29.3% of cases regarding correctness and helpfulness.

Conclusions:

• ChatGPT can generate satisfactory responses to numerous radiation therapy questions, indicating its potential utility.
• The current inability of LLMs to consistently provide accurate medical information makes their direct use for medical queries problematic.
• Future improvements in LLMs are expected to increase their impact on clinical practice, including radiation oncology.