Metrics Used for the Evaluation of Chatbots Providing Cancer Genetic Risk Assessment and Education: Systematic Review

Chatbots have recently emerged as an alternative approach for delivering cancer risk assessment and genetic counseling. Understanding the metrics used to describe the user-chatbot experience highlights the strengths and weaknesses of chatbot-assisted health care applications, ensuring safe and reliable medical care. While research supports chatbots in cancer genetic risk assessment and counseling, the evaluation measures remain inconsistent and unsystematic.

This systematic review analyzes the metrics used to evaluate chatbot platforms providing cancer genetic risk assessment and pretest and posttest genetic education. We examine these measures to identify potential limitations and inform a more systematic evaluative approach.

A comprehensive search was conducted using PubMed, Web of Science, and Engineering Village. Articles were screened and analyzed using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) framework. Study and chatbot characteristics were documented, along with variables affecting metric use. Metrics evaluating the user-chatbot experience were extracted, categorized into domains, and organized within the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework to identify assessment gaps and insights regarding application and effectiveness. Risk of bias was assessed using 5 distinct evaluation tools.

This database search retrieved 692 citations, with 14 articles meeting the inclusion criteria. The studies varied in study objective, methodologies, research settings, chatbot functionalities, and participants' characteristics. A total of 136 measures were extracted and categorized into 16 groups. The number of individual metrics used in each study varied from 3 to 18 (median of 8.5). Measurement groups were organized into 5 domains-user experience, knowledge acquisition, outcomes and behaviors, emotional response, and technical performance-with user experience measures being the most common. Emotional response and technical performance were the least used. Knowledge acquisition measures ranked third and appeared in half of the final study pool. While metrics covered all 5 RE-AIM framework domains, they were unevenly distributed. Risk of bias assessment exposed several study limitations, including small sample size, self-selection bias, and potentially inflated engagement metrics.

This review highlights critical gaps and variability in metrics used to evaluate automated cancer genetic risk assessment and education. Studies most often measured user experience and patient outcomes and behaviors; however, despite its central role in informed consent, knowledge was assessed less consistently and was only moderately ranked. Expanding research efforts and standardizing educational metrics could improve chatbot effectiveness and better support patient decision-making. Important gaps remain in measures of knowledge, emotional response, technical performance, and long-term outcomes, emphasizing the need for increased evaluation in these areas. Using frameworks like RE-AIM can promote comprehensive measurement and a safer and more equitable implementation of novel cancer genetic counseling approaches. Future studies should aim to standardize outcome measures, strengthen missing data methods, and transparently report recruitment and analyses to improve the validity of findings.