A Sentiment-Based Comparison of AI- and Physician-Generated Empathic Statements in Palliative Care

Empathic communication promotes trust in patient-provider relationships. As healthcare integrates artificial intelligence (AI) into patient communication, we have yet to understand how these models' communication compares to that of physicians.

Our primary objectives were to examine patient preferences for AI-generated vs. palliative care physician-generated empathic statements addressing fear and anxiety around cancer treatment, and to analyze associations between linguistic features and patient preferences.

We conducted a secondary analysis of the PALL-AI trial, a randomized controlled survey comparing cancer patients' preferences of AI- to physician-generated empathic statements. Physicians and AI were provided the same prompt with a maximum sentence length. Patient preferences for each statement were measured in blinded surveys. We analyzed sentiment of the statements using the Valence Aware Dictionary and Sentiment Reasoner (VADER) and the National Research Council Canada (NRC) Emotion Lexicon. We evaluated associations between sentiment scores and patient preferences using Spearman's correlation coefficients.

A total of 105 patients completed blinded surveys, preferring the AI-generated statement 72.4% of the time. VADER sentiment analysis showed all three AI statements displayed positive sentiment, while all three physician statements displayed negative sentiment. Controlling for statement length, AI statements used twice as many positive words as human statements. However, they contained a similar number of negative words. Of the eight NRC emotions, "trust" and "joy" demonstrated the strongest correlations with patient preference.

Patients preferred AI-generated statements around cancer care over those from palliative care physicians when standardized for prompt and statement length. Analysis shows AI-generated statements contain more positive language which may be the factor driving patient preference toward AI.