Performance of Three Conversational Artificial Intelligence Agents in Defining End-of-Life Care Terms

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Palliative Care Research

Background:

• Conversational AI, or chatbots, represent a significant technological advancement with potential applications in end-of-life care.
• Despite their growing influence, AI chatbots remain understudied within the specific context of end-of-life care.

Purpose of the Study:

• To evaluate the accuracy, comprehensiveness, and credibility of leading AI chatbots (ChatGPT, Bard, Bing) in defining key end-of-life care terminology.
• To assess the readability of AI-generated content related to end-of-life care.

Main Methods:

• Six physicians evaluated chatbot responses to queries on "terminally ill," "end of life," "transitions of care," and "actively dying."
• Outputs were scored on accuracy, comprehensiveness, and credibility (0-10 scale).
• Readability was assessed using Flesch-Kincaid Grade Level and Flesch Reading Ease (FRE).

Main Results:

• AI chatbots demonstrated high accuracy (mean scores 7.5-9.0) and good comprehensiveness (mean scores 6.5-8.5).
• Credibility scores were notably low (mean 3.0).
• Readability was challenging, with a mean Flesch Reading Ease of 41.7 and a mean grade level of 14.1.

Conclusions:

• While AI chatbots offer promising accuracy in defining end-of-life care terms, their low credibility and readability underscore the need for critical evaluation.
• Clinician oversight is essential to mitigate the risk of misinformation when using AI-generated content in end-of-life care settings.