Evaluating the Quality of Health Information: Comparison of Human and Artificial Intelligence

Over half of all Americans seek health-related information online, yet the quality of this digital content remains largely unregulated and variable. The DISCERN score, a validated 15-item instrument, offers a structured method to assess the reliability of written health information. While expert-assigned DISCERN scores have been widely applied across various disease states, whether artificial intelligence (AI) can automate this evaluation remains unknown. Specifically, it is unclear whether AI-generated DISCERN scores align with those assigned by human experts. Our study seeks to investigate this gap in knowledge by examining the correlation between AI-generated and human-assigned DISCERN scores for TikTok videos on Irritable Bowel Syndrome (IBS).

A set of 100 TikTok videos on IBS previously scored using DISCERN by two physicians was chosen. Sixty-nine videos contained transcribable spoken audio, which was processed using a free online transcription tool. The remaining videos either featured songs or music that were not suitable for transcription or were deleted or were not publicly available. The audio transcripts were prefixed with an identical prompt and submitted to two common AI models-ChatGPT 4.0 and Microsoft Copilot for-DISCERN score evaluation. The average DISCERN score for each transcript was compared between the AI models and with the mean of the DISCERN score given by the human reviewers using Pearson correlation (r) and Kruskal Wallis test.

There was a significant correlation between human and AI-generated DISCERN scores (r = 0.60-0.65). When categorized by the background of the content creators-medical (N = 26) versus non-medical (N = 43), the correlation was significant only for content made by non-medical content creators (r = 0.69-0.75, p < 0.001). Correlation between ChatGPT and Copilot DISCERN scores was stronger for videos by non-medical content creators (r = 0.66) than those by medical content creators (r = 0.43). On linear regression, ChatGPT's DISCERN scores explained 55.6% of the variation in human DISCERN scores for videos by non-medical creators, compared to 8.9% for videos by medical creators. For Copilot, the corresponding values were 47.2% and 9.3%.

AI models demonstrated moderate alignment with human-assigned DISCERN scores for IBS-related TikTok videos, but only when content was produced by non-medical creators. The weaker correlation for content produced by those with a medical background suggests limitations in current AI models' ability to interpret nuanced or technical health information. These findings highlight the need for further validation across broader topics, languages, platforms, and reviewer pools. If refined, AI-generated DISCERN scoring could serve as a scalable tool to help users assess the reliability of health information on social media and curb misinformation.