The digital dialogue on premature ejaculation: evaluating the efficacy of artificial intelligence-driven responses

Area of Science:

• Artificial Intelligence in Healthcare
• Digital Health Information Quality
• Medical Natural Language Processing

Background:

• The proliferation of AI chatbots presents new avenues for health information dissemination.
• Premature ejaculation (PME) is a common sexual health concern requiring reliable information.
• Evaluating the quality of AI-generated health content is crucial for patient safety and trust.

Purpose of the Study:

• To assess and compare the quality and comprehensibility of responses from ChatGPT, Gemini, and Llama regarding premature ejaculation (PME).
• To determine which AI chatbot provides the most accurate, clear, and trustworthy information on PME.

Main Methods:

• 25 frequently asked questions (FAQs) on PME were sourced using Google Trends and Semrush.
• Responses from ChatGPT, Gemini, and Llama were analyzed using Flesch Reading Ease (FRES), Flesch-Kincaid Grade Level (FKGL), modified DISCERN (mDISCERN), and Ensuring Quality Information for Patients (EQIP) scores.
• Statistical analysis was performed to compare readability and quality metrics across the chatbots.

Main Results:

• Readability scores (FRES, FKGL) were comparable across all three AI models.
• Llama demonstrated the highest mean EQIP scores (72.2 ± 1.1), followed by Gemini (67.6 ± 4.5) and ChatGPT (63.1 ± 4.9).
• Llama and Gemini achieved higher median mDISCERN scores (3) compared to ChatGPT (2), indicating superior information quality.

Conclusions:

• AI chatbots exhibit similar readability levels when discussing premature ejaculation.
• Llama and Gemini offer superior quality and trustworthiness in their responses on PME compared to ChatGPT, based on EQIP and mDISCERN evaluations.
• Further research is needed to optimize AI-generated health information for accuracy and patient comprehension.