AI Chatbots as Sources of STD Information: A Study on Reliability and Readability

Artificial intelligence (AI) chatbots are increasingly used for medical inquiries, including sensitive topics like sexually transmitted diseases (STDs). However, concerns remain regarding the reliability and readability of the information they provide. This study aimed to assess the reliability and readability of AI chatbots in providing information on STDs. The key objectives were to determine (1) the reliability of STD-related information provided by AI chatbots, and (2) whether the readability of this information meets the recommended standarts for patient education materials.

Eleven relevant STD-related search queries were identified using Google Trends and entered into four AI chatbots: ChatGPT, Gemini, Perplexity, and Copilot. The reliability of the responses was evaluated using established tools, including DISCERN, EQIP, JAMA, and GQS. Readability was assessed using six widely recognized metrics, such as the Flesch-Kincaid Grade Level and the Gunning Fog Index. The performance of chatbots was statistically compared in terms of reliability and readability.

The analysis revealed significant differences in reliability across the AI chatbots. Perplexity and Copilot consistently outperformed ChatGPT and Gemini in DISCERN and EQIP scores, suggesting that these two chatbots provided more reliable information. However, results showed that none of the chatbots achieved the 6th-grade readability standard. All the chatbots generated information that was too complex for the general public, especially for individuals with lower health literacy levels.

While Perplexity and Copilot showed better reliability in providing STD-related information, none of the chatbots met the recommended readability benchmarks. These findings highlight the need for future improvements in both the accuracy and accessibility of AI-generated health information, ensuring it can be easily understood by a broader audience.