Effectiveness, Usability, and Acceptability of ChatGPT With Retrieval-Augmented Generation (SIV-ChatGPT) in...

Older adults are more vulnerable to severe consequences caused by seasonal influenza. Although seasonal influenza vaccination (SIV) is effective and free vaccines are available, the SIV uptake rate remained inadequate among people aged 65 years or older in Hong Kong, China. There was a lack of studies evaluating ChatGPT in promoting vaccination uptake among older adults.

This study aimed to evaluate the effectiveness of ChatGPT with retrieval-augmented generation in increasing SIV uptake among older adults over a 3-month study period in Hong Kong, China. Participants in an ongoing observational cohort study conducted in the same period served as the comparison group.

A quasi-experimental study was conducted between November 2024 and April 2025. Participants were (1) aged ≥65 years, (2) possessed a Hong Kong ID, (3) able to speak and comprehend Cantonese, (4) smartphone users, and (5) had no SIV uptake for the approaching flu season. Those with a diagnosis of cognitive impairment or dementia, blindness or deafness, or known contraindications to the SIV were excluded. Participants were recruited through random telephone calls. There were 45 and 55 participants in the SIV-ChatGPT group and the comparison group, respectively. All participants completed follow-up surveys at T1 (1 month after the baseline survey, for the SIV-ChatGPT group only) and T2 (3 months after the baseline survey, for both groups). Participants in the SIV-ChatGPT group gained access to SIV-ChatGPT in the format of a web-based app after completion of the baseline survey. They could use SIV-ChatGPT repeatedly throughout a 1-month intervention period and were free to spend as much time as they wanted with SIV-ChatGPT. Intention-to-treat analysis was used for outcome analyses.

At T2, the SIV uptake rate was higher in the SIV-ChatGPT group than the comparison group (15/45, 33% vs 8/55, 14.3%; adjusted odds ratio 2.72, 95% CI 1.01-7.35, P=.048). All participants were able to provide receipts to validate their SIV uptake. In the SIV-ChatGPT group, 40.5% (15/37) of participants who used SIV-ChatGPT at least once reported a SIV uptake at T2, which was significantly higher than nonusers (0/8, 0%; P=.04). Among the 37 SIV-ChatGPT users, the mean score of the System Usability Scale was 67.1 (SD 14.9). Levels of subjective behavioral and cognitive engagement with SIV-ChatGPT were relatively high, while the affective engagement was moderate.

SIV-ChatGPT was feasible and acceptable and demonstrated preliminary effectiveness in increasing SIV uptake among people aged 65 years or older. This study also provided implications to improve the performance of SIV-ChatGPT. A full-powered randomized controlled trial should be considered to evaluate its efficacy.

ClinicalTrials.gov NCT06679647; https://clinicaltrials.gov/study/NCT06679647.