Vaccination hesitancy: agreement between WHO and ChatGPT-4.0 or Gemini Advanced

An increasing number of individuals use online Artificial Intelligence (AI) - based chatbots to retrieve information on health-related topics. This study aims to evaluate the accuracy in answering vaccine-related answers of the currently most commonly used, advanced chatbots - ChatGPT-4.0 and Google Gemini Advanced.

We compared the answers provided by the World Health Organization (WHO) to 38 open questions on vaccination myths and misconception, with the answers created by ChatGPT-4.0 and Gemini Advanced. Responses were considered as "appropriate", if the information provided was coherent and not in contrast to current WHO recommendations or to drug regulatory indications.

The rate of agreement between WHO answers and Chat-GPT-4.0 or Gemini Advanced was very high, as both provided 36 (94.7%) appropriate responses. The few discrepancies between WHO and AI-chatbots answers could not be considered "harmful", and both chatbots often invited the user to check reliable sources, such as CDC or the WHO websites, or to contact a local healthcare professional. In their current versions, both AI-chatbots may already be powerful instrument to support the traditional communication tools in primary prevention, with the potential to improve health literacy, medication adherence, and vaccine hesitancy and concerns. Given the rapid evolution of AI-based systems, further studies are strongly needed to monitor their accuracy and reliability over time.