Development and Usability Assessment of a Health Education Conversational Agent for Patients With Gastric Cancer:...

To provide patients with gastric cancer with adequate health education information for effective overall management is crucial, while traditional manners exposed certain challenges. Conversational agents have increasingly been adopted for health care use to provide innovative solutions for patient education.

This study aimed to develop a health education embodied conversational agent to focus on gastric cancer disease using an action research approach and test its accuracy, usability, and user experience among patients and other related stakeholders.

The AI-guided conversational agent was developed based on the OpenMEDLab 2.0 foundation model and the Retrieval-Augmented Generation (RAG) architecture. A 4-phase action research approach was adopted to implement this system at a gastric cancer center in China. Diagnose and plan phase used participatory observation and in-depth interviews to explore current health education models and patients' needs for health education. Act and implement phase was used to develop and deploy the conversational agent. Evaluate phase comprised 3 rounds of alpha testing to assess accuracy and RAG knowledge hit rate, and 1 round of beta testing to assess the usability and relevance. Reflect phase conducted in-depth interviews to gain insights into users' experiences. Data collection was conducted from September 2023 through April 2025. Participants include patients, clinical nurses, nursing managers, surgeons, clinical psychologists, and dietitians. Thematic analysis and multiple-group chi-square tests were performed, respectively, for qualitative and quantitative data. A 2-sided P value of <.05 was considered statistically significant.

A total of 44 patients, 13 nurses, 3 nursing managers, 2 surgeons, 1 clinical psychologist, and 1 dietitian were recruited during the study procedure. Favorable outcomes in terms of accuracy and usability were achieved. The accuracy of the agent in 3 rounds was 67% (37/55), 71% (44/62), and 82% (31/38), respectively; RAG knowledge hit rates reached 86% (47/55), 98% (61/62), and 100% (38/38). Significant differences (P<.01) in RAG knowledge hit rates were observed. The mean chatbot usability questionnaire score was 91.9 (SD 3.6), while the mean content relevance score was 3.75 (SD 0.9). Three themes of user experiences were identified: perceived usefulness, ease of use, and intention to use, revealing potential in reducing staff workload and reinforcing patient education.

This study provided insights into how the action research approach can inform the development and usability assessment of a gastric cancer health education conversational agent, also illustrating the value of RAG technology. Additional assessments and improvements are warranted to confirm the effectiveness and safety.