Evaluating ChatGPT's responses to vaccine-related questions: the impact of question framing on content and quality

Vaccine hesitancy, fueled by mistrust, fear, and misinformation, remains a major public health challenge. Generative artificial intelligence tools such as ChatGPT have emerged as new information sources, particularly for younger users. It is essential that these tools provide medically accurate content, especially when responding to negatively framed vaccine questions. This study aims to examine how ChatGPT responds to vaccine-related questions, focusing on questions with negative or skeptical framing.

This anonymous survey targeted board-certified pediatric infectious disease specialists in Japan to evaluate ChatGPT's responses to vaccine-related questions. Using ChatGPT-4o, 20 question pairs (supportive vs. critical framing) were generated, and responses in Japanese were obtained. Participants were randomly assigned to assess half of the responses via Google Forms, rating Clarity, Appropriateness, Ambiguity, and Length on a five-point Likert scale, with optional free-text comments. Quantitative comparisons between supportive and critical questioning used the Mann-Whitney U test; qualitative feedback was thematically analyzed.

Twenty of 22 invited specialists (90.9%) completed the survey. Median [25th-75th percentile] scores for supportive vs. critical questioning were: Clarity, 4 [3-4] vs. 4 [3-4]; Appropriateness, 3 [3-4] vs. 3 [3-4]; Ambiguity, 4 [3-4] vs. 4 [3-4]; Length, 3 [3-4] vs. 3 [3-4]. No significant differences were found for any item. Among 81 free-text comments, the most frequent concerns were "bias toward COVID-19 vaccines" (n=38), "insufficient explanation" (n=19), and "potentially misleading expressions" (n=9). Examples included overemphasis on COVID-19 in unrelated contexts and problematic phrasing regarding human papillomavirus vaccine adverse events.

ChatGPT maintained comparable quality in Japanese responses to both supportive and critical vaccine questions, suggesting resilience to negative framing. However, expert reviewers identified thematic biases, occasional inadequacy of detail, and linguistic issues that could mislead lay readers. These findings underscore the need for continued human oversight, refinement of Japanese-language outputs, and algorithmic adjustments to reduce topical bias.