The digital dialogue on premature ejaculation: evaluating the efficacy of artificial intelligence-driven responses

This study investigated the quality and comprehensibility of responses generated by three prominent artificial intelligence-powered chatbots (ChatGPT, Gemini, and Llama) when queried about premature ejaculation (PME).

A set of 25 frequently asked questions (FAQs) were identified on the basis of Google Trends and Semrush platforms. Each chatbot was prompted with these questions and their responses were analyzed via a comprehensive set of metrics. Readability was assessed via the Flesch Reading Ease (FRES) and Flesch-Kincaid Grade Level (FKGL) scores. Quality and reliability were evaluated via the modified DISCERN (mDISCERN) and Ensuring Quality Information for Patients (EQIP) scores, which assess the clarity, comprehensiveness, and trustworthiness of health information.

Readability scores, as assessed by FRES and FKGL, did not significantly differ across the three chatbots. In terms of quality, the mean EQIP scores were significantly different between the models, with Llama (72.2 ± 1.1) achieving the highest scores, followed by Gemini (67.6 ± 4.5) and ChatGPT (63.1 ± 4.9) (P < 0.001). The median (interquartile range) mDISCERN scores were 2 (1) for ChatGPT, 3 (0) for Gemini, and 3 (1) for Llama (P < 0.001), indicating a significant difference in the quality of information provided by the different models.

The three chatbots demonstrated statistically similar results in terms of readability. Llama achieved the highest EQIP score among them. Additionally, both Llama and Gemini outperformed ChatGPT in terms of mDISCERN scores.