Assessment of vaccine information accuracy across large language models

Use of Large Language Models (LLMs) as a health information source is on the rise. As the amount of people willing to self-diagnose with the tools' help is increasing, verifying the correctness of generated responses becomes extremely important.

Four LLMs (ChatGPT, Copilot, DeepSeek, Gemini) were chosen for the analysis. All models were asked 90 multiple-choice questions on the various topics surrounding vaccination, sourced from a government information website. Additionally, the LLMs were asked two open-ended questions regarding the vaccine information sources, as well as advantages and disadvantages of universal vaccination. Answers were analysed quantitatively and qualitatively. Data was collected in March 2026.

Biggest proportion of correct answers was achieved by Gemini (0.98), smallest one - by ChatGPT (0.86). Models' results were found to be non-homogeneous (p = 0.0049). While answering open-ended questions, all LLMs were able to cite various, appropriate vaccine information sources. For advantages of vaccination all models pointed to herd immunity. Answers regarding disadvantages to vaccination all included the possibility of adverse reactions. The answers were judged to be factually correct, yet lacking in detail.

Tested LLMs were able to identify correct option when presented with multiple choices and to prepare correct, yet vague information when asked an open-ended question. This points to a limited potential of LLMs serving as a vaccine information source, as the generated content, while factually correct, was not enough to be usable on its own.