Evaluating Healthcare Provider and Artificial Intelligence Chatbot Responses to Patient Messages from a Health System...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Communication

Background:

• Integrating AI chatbots into healthcare necessitates patient-centered evaluation.
• The CREATE TRUST framework offers a novel method for assessing clinical responses.

Purpose of the Study:

• To compare the quality of responses from healthcare providers (HCPs) and AI chatbots (GPT-4, Mixtral) using the CREATE TRUST framework.
• To evaluate clinical substance and communication style in AI-generated versus human-authored healthcare messages.

Main Methods:

• A cross-sectional study involving 189 real-world clinical messages from cancer patients.
• Anonymized responses from HCPs, GPT-4, and Mixtral were blindly evaluated by oncologists using the CREATE TRUST framework.
• Evaluators rated responses on attributes like Correctness, Empathy, Authenticity, and Safety, and provided preference rankings.

Main Results:

• No significant difference in overall CREATE TRUST scores between HCPs, GPT-4, and Mixtral.
• HCPs performed significantly better on Authenticity and Tailoring; chatbots scored higher on Empathy and being Referenced.
• HCP responses showed greater variability, with a higher proportion of both high- and low-quality outputs.

Conclusions:

• HCPs and AI chatbots demonstrate comparable overall response quality but possess distinct strengths.
• AI chatbots can enhance human-authored messages, improving specific communication aspects and mitigating lower-quality responses.
• A synergistic workflow integrating AI and HCPs holds potential for optimizing patient communication in healthcare.