What is Artificial Intelligence (AI) "Empathy"? A Study Comparing ChatGPT and Physician Responses on an Online Forum

Artificial intelligence (AI) chatbots may be an asset to patient-provider communication, but not enough is known about how patients respond and how chatbots answer patients' questions.

How perceptions of empathy, quality, trust, liking, and goodness vary by both the actual and perceived source of responses to patient questions (chatbot vs. actual physician). We also coded and compared key verbal elements in chatbot and physician responses.

This cross-sectional experimental study used chatbot and physician responses from Ayers et al. (2023) in a 2 (actual source: chatbot vs. physician) × 2 (perceived source: chatbot vs. physician) factorial design.

U.S.-based, English-speaking participants were recruited online (N = 1454).

Participants rated responses on empathy, quality, trust, liking, and goodness. Verbal content of the chatbot and physician responses was independently coded by trained research assistants to identify elements contributing to higher empathy ratings by participants.

Replicating Ayers et al. (2023), participants rated chatbot responses as more empathic than physician responses (Cohen's d = 0.56, p < 0.001). Chatbot responses received higher empathy ratings than physician responses regardless of what participants were told about authorship (η_p² = 0.60, p < 0.001). Empathy ratings were higher when participants thought the response was physician-authored, whether it was or not (η_p² = 0.17, p < 0.001). Participant ratings of quality, trust, liking, and goodness followed the same pattern as empathy. Chatbot responses contained more coder-rated validation, reassurance, and non-judgmental language and were less rushed and more structured than physician responses (Cohen's d = 0.32 to 1.82, p's < 0.01).

AI-generated responses, with human oversight, could enhance computer-mediated clinical communication, although patient awareness of AI contributions may reduce perceptions of empathy. Identification of the specific verbal elements in AI-generated responses could augment communication and increase perceptions of empathic care.