A comparative analysis of CDC and AI-generated health information using computer-aided text analysis

AI-generated content is easy to access. Members of the public use it as an alternative or to supplement official sources, such as the Centers for Disease Control and Prevention (CDC). However, the quality and reliability of AI-generated health information is questionable. This study aims to understand how AI-generated health information differs from that provided by the CDC, particularly in terms of sentiment, readability, and overall quality. Language expectancy theory serves as a framework and offers insights into how people's expectations of message content from different sources can influence perceived credibility and persuasiveness of such information.

Computer-aided text analysis was used to analyze 20 text entries from the CDC and 20 entries generated by ChatGPT 3.5. Content analysis utilizing human coders was used to assess the quality of information.

ChatGPT used more negative sentiments, particularly words associated with anger, sadness, and disgust. The CDC's health messages were significantly easier to read than those generated by ChatGPT. Furthermore, ChatGPT's responses required a higher reading grade level. In terms of quality, the CDC's information was a little higher quality than that of ChatGPT, with significant differences in DISCERN scores.

Public health professionals need to educate the general public about the complexity and quality of AI-generated health information. Health literacy programs should address topics about quality and readability of AI-generated content. Other recommendations for using AI-generated health information are provided.