Evaluating large language models on multilingual vaccine knowledge: a benchmark study.

Area of Science:

• Artificial Intelligence
• Public Health
• Vaccinology

Background:

• Large language models (LLMs) are increasingly utilized for accessing health information, including vaccine details.
• The factual accuracy of LLMs for vaccine knowledge across different languages and specific vaccines is not well-established.
• Ensuring reliable vaccine information from LLMs is critical for both clinicians and the public.

Purpose of the Study:

• To evaluate the factual accuracy of 13 large language models (LLMs) on vaccine-related knowledge.
• To assess LLM performance across multiple languages (English, Spanish, Chinese) and diverse vaccine types.
• To compare the impact of different prompting strategies (zero-shot, few-shot, chain-of-thought) on LLM accuracy.

Main Methods:

• Utilized VaxEval, a benchmark comprising 1886 multiple-choice questions on 14 vaccines, translated into English, Spanish, and Chinese.
• Performed quality control on all questions, verifying answers against authoritative sources and peer-reviewed literature.
• Employed mixed-effect logistic regression to analyze factors influencing answer correctness, including model group, prompting strategy, language, and vaccine type.

Main Results:

• Mean accuracy across all models was 86.0% in English, 83.7% in Spanish, and 80.0% in Chinese.
• Newer flagship LLMs demonstrated higher accuracy than earlier models (OR 1.57).
• Few-shot prompting improved correctness (OR 1.17), while chain-of-thought prompting reduced it (OR 0.79).

Conclusions:

• LLMs exhibit notable accuracy in vaccine knowledge but with performance variations.
• Model advancements and specific prompting techniques significantly influence accuracy.
• Rigorous evaluation and targeted refinements are essential before deploying LLMs for vaccine communication.