Ensuring Accuracy and Equity in Vaccination Information From ChatGPT and CDC: Mixed-Methods Cross-Language Evaluation

Area of Science:

• Digital Health
• Health Informatics
• Public Health

Background:

• Large language models (LLMs) are increasingly used for health information, presenting opportunities to overcome traditional access barriers.
• However, the quality of LLM-generated health information is inconsistent, with limited research on non-English language responses.
• This study focuses on health equity by evaluating vaccination information quality across languages.

Purpose of the Study:

• To assess the quality and language equity of vaccination information from ChatGPT and the Centers for Disease Control and Prevention (CDC) in English and Spanish.
• To highlight the necessity of cross-language evaluations for equitable health information access.

Main Methods:

• Comparative analysis of ChatGPT and CDC responses to vaccination queries in English and Spanish.
• Quantitative and qualitative assessments of accuracy, readability (Flesch-Kincaid grade level), and understandability (NIH PEMAT).

Main Results:

• Both ChatGPT and CDC provided highly accurate and understandable responses (scores >95%).
• Readability scores, particularly in English, often exceeded recommended levels (e.g., ChatGPT English avg. grade level 12.84 vs. recommended 6).
• CDC responses showed better readability than ChatGPT; some Spanish responses had unnatural phrasing due to direct translation.

Conclusions:

• LLMs like ChatGPT show promise for health information but require enhancements in readability and linguistic equity.
• Default LLM interactions, often by users without advanced skills, significantly influence health perceptions.
• Ensuring default responses are accurate, understandable, and equitable is vital for public health and informed decision-making across communities.