Effectiveness of Communication Competence in AI Conversational Agents for Health: Systematic Review and Meta-Analysis

With advancements in artificial intelligence and large language models, researchers and designers have increasingly focused on enhancing the conversational capacity of health-related conversational agents (CAs). Communication competence, a key concept in interpersonal communication influencing relational and health outcomes, has been extended to human-machine communication to emphasize the CAs' ability to demonstrate appropriate communicative behaviors in managing relationships with humans.

This review aims to summarize the operationalization of communication competence in health CAs and assess its impact on 4 primary outcomes: users' evaluations of CA, use of CA, psychological outcomes, and health outcomes.

A systematic literature search was conducted in 7 databases (ACM Digital Library, APA PsycInfo, Communication and Mass Media Complete, ProQuest Dissertations & Theses, Scopus, Web of Science Core Collection, and PubMed). Studies were included if they adopted experimental designs to manipulate CAs' communication competence in health-related conversations, recruited human participants, and reported at least 1 relevant outcome. Risk of bias was assessed using the revised Cochrane risk-of-bias tool. The systematic review summarized commonly used communication competence strategies. Three-level random-effects meta-analytic models were used to estimate pooled effect sizes for 4 primary outcomes. Moderator analyses were conducted to assess whether effect sizes varied across publication year, participants' average age, type of interaction with CAs, health topics, and publication outlet.

Of the 8309 identified papers, 31 independent experimental studies were included in the systematic review. Eleven strategies were identified to enhance CAs' communication competence: empathetic response, contingency, humor, small talk, emotional expressiveness, self-disclosure, personalization, social etiquette, explanation, open-ended questions, and partnership. Of the 31 studies, 25 met the criteria for meta-analysis, which involved 4525 participants with a mean age of 29.7 (SD 9.2) years. The meta-analytic findings showed that communication competence has a significant small-to-medium effect on users' evaluations of CAs (Hedges g=0.45, 95% CI 0.24-0.66) and psychological outcomes (Hedges g=0.49, 95% CI 0.19-0.78). The effect sizes on the use of CA (Hedges g=0.11, 95% CI -0.05 to 0.26) and health outcomes (Hedges g=0.18, 95% CI -0.13 to 0.50) are not significant. Moderator analyses showed that the effects remain stable across participants' age, type of interaction, and health topics.

This review highlights communication competence as a critical component in the design of health care CAs, particularly in improving users' evaluations and psychological outcomes. However, the limited number of studies examining health outcomes restricts the robustness of its effectiveness on this outcome. Future research is encouraged to directly evaluate the effects on tangible health outcomes.