Assessing the Suitability of Artificial Intelligence-Based Chatbots as Counseling Agents for Patients with Brain...

The internet, particularly social media, has become a popular resource for learning about health and investigating one's own health conditions. The development of artificial intelligence (AI) chatbots has been fueled by the increasing availability of digital health data and advances in natural language processing techniques. While these chatbots are more accessible than before, they sometimes fail to provide accurate information.

We used representative chatbots currently available (Chat Generative Pretrained Transformer-3.5, Bing Chat, and Google Bard) to answer questions commonly asked by brain tumor patients. The simulated situations with questions were made and selected by the brain tumor committee. These questions are commonly asked by brain tumor patients. The goal of the study was introduced to each chatbot, the situation was explained, and questions were asked. All responses were collected without modification. The answers were shown to the committee members, and they were asked to judge the responses while blinded to the type of chatbot.

There was no significant difference in accuracy and communication ability among the 3 groups (P = 0.253, 0.090, respectively). For empathy, Bing Chat and Google Bard were superior to Chat Generative Pretrained Transformer (P = 0.004, 0.002, respectively). The purpose of this study was not to assess or verify the relative superiority of each chatbot. Instead, the aim was to identify the shortcomings and changes needed if AI chatbots are to be used for patient medical purposes.

AI-based chatbots are a convenient way for patients and the general public to access medical information. Under such circumstances, medical professionals must ensure that the information provided to chatbot users is accurate and safe.