ChatGPT's role in the rapidly evolving hematologic cancer landscape

Area of Science:

• Oncology
• Artificial Intelligence
• Medical Informatics

Background:

• Patients increasingly seek online information about diseases and treatments.
• Chatbots are emerging as potential tools for delivering health information.
• Validating AI accuracy is crucial for integrating it into cancer care.

Purpose of the Study:

• To assess the accuracy and usefulness of ChatGPT 3.5 for patient-level hematologic malignancy information.
• To contribute to understanding AI's role and limitations in oncology patient support.
• To identify areas for chatbot improvement in cancer care communication.

Main Methods:

• A curated set of questions on hematologic malignancies was developed with expert input.
• ChatGPT 3.5 was queried, and responses were evaluated by hematology-oncology physicians.
• Physician ratings focused on accuracy and patient usefulness, using a 1-5 Likert scale.

Main Results:

• General queries scored higher (3.38) than novel therapies (3.06).
• No questions received scores above 4, with most responses rated neutrally (3.0-3.8).
• ChatGPT demonstrated limitations in providing current, specific information, particularly for acute myeloid leukemia and novel treatments.

Conclusions:

• ChatGPT 3.5 exhibits deficiencies in delivering up-to-date and specific information for complex oncology queries.
• The AI's performance suggests limitations due to reliance on static datasets, impacting its ability to reflect rapidly evolving medical knowledge.
• Further development is needed to enhance chatbot accuracy and utility for cancer patients and caregivers.