Evaluation of the feasibility of artificial intelligence models as patient information advisors for human...

Artificial intelligence chatbots are increasingly used to disseminate health information. The aim of this study was to evaluate the accuracy, reliability, quality, and readability of responses generated by four artificial intelligence chatbots regarding human papillomavirus vaccination.

Frequently asked questions about human papillomavirus vaccination were identified using a Google search tool, and these questions were posed to the ChatGPT-3.5, Gemini, Copilot artificial intelligence, and ChatGPT-4 models. Responses were assessed for accuracy (five-point Likert scale), reliability (modified DISCERN scale), quality (Global Quality Scale), and readability (Flesch Reading Ease Score). Interobserver agreement was evaluated by the intraclass correlation coefficient. The results were evaluated at a significance level of p<0.05. SPSS 23.0 package programme was used in the analyses.

There were significant differences between chatbots in terms of accuracy (p=0.001), reliability (p<0.001), and quality (p<0.001), but no significant difference in readability (p=0.497). ChatGPT-4 demonstrated the highest accuracy and quality, while Copilot artificial intelligence demonstrated superior reliability. All models produced responses that were moderately difficult to read. The intraclass correlation coefficient values for inter-rater reliability ranged from 0.034 to 0.512.

Artificial intelligence chatbots show promising potential for use as patient information counsellors regarding human papillomavirus vaccination. However, improvements in readability and consistent evidence-based content generation are required before widespread clinical application.