AI Chatbots as Sources of STD Information: A Study on Reliability and Readability

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Public Health Communication

Background:

• AI chatbots are increasingly used for medical inquiries, including sensitive topics like sexually transmitted diseases (STDs).
• Concerns exist regarding the reliability and readability of AI-generated health information.
• Assessing AI chatbot performance for STD information is crucial for patient safety and education.

Purpose of the Study:

• To evaluate the reliability of information on STDs provided by AI chatbots.
• To determine if AI chatbot-generated STD information meets recommended readability standards for patient education.
• To compare the performance of different AI chatbots in terms of information accuracy and accessibility.

Main Methods:

• Eleven STD-related queries were input into four AI chatbots (ChatGPT, Gemini, Perplexity, Copilot).
• Reliability was assessed using DISCERN, EQIP, JAMA, and GQS.
• Readability was measured using Flesch-Kincaid Grade Level and Gunning Fog Index, among others.
• Statistical comparisons were made for chatbot performance.

Main Results:

• Perplexity and Copilot demonstrated higher reliability scores (DISCERN, EQIP) compared to ChatGPT and Gemini.
• No AI chatbot met the recommended 6th-grade readability standard.
• All chatbots provided information that was too complex for the general public, particularly those with lower health literacy.

Conclusions:

• While Perplexity and Copilot offer more reliable STD information, current AI chatbots fail to meet essential readability benchmarks.
• Improvements are needed in AI-generated health content to enhance both accuracy and accessibility.
• Ensuring AI health information is easily understandable is vital for effective public health communication.