Frequently asked questions on erectile dysfunction: evaluating artificial intelligence answers with expert mentorship

Area of Science:

• Urology
• Artificial Intelligence in Medicine
• Medical Informatics

Background:

• Erectile dysfunction (ED) is a common condition affecting many individuals.
• Patients frequently seek information online regarding ED causes, diagnosis, and treatments.
• The accuracy of AI-generated responses to medical queries is a growing concern.

Purpose of the Study:

• To evaluate the accuracy and quality of AI-generated responses to frequently asked questions (FAQs) about erectile dysfunction.
• To compare the performance of ChatGPT 3.5, ChatGPT 4, and BARD in providing reliable ED information.

Main Methods:

• A cross-sectional analysis of 56 ED-related questions across nine categories was conducted.
• Responses from ChatGPT 3.5, ChatGPT 4, and BARD were assessed by urology experts.
• Evaluation used F1 scores and global quality scores (GQS) for accuracy, relevance, and comprehensibility.

Main Results:

• ChatGPT 3.5 and ChatGPT 4 demonstrated significantly higher GQS than BARD in several categories, including causes, treatment options, protective measures, and herbal agents.
• F1 scores for ChatGPT models indicated alignment with medical guidelines in areas like causes, diagnosis, and treatment options.
• No significant difference in answer quality was found between ChatGPT 3.5 and ChatGPT 4, but both surpassed BARD.

Conclusions:

• Current AI models, particularly ChatGPT 3.5 and 4, offer a reliable source of information for erectile dysfunction FAQs.
• Expert evaluation confirms the potential of AI in disseminating accurate medical knowledge.
• Ongoing enhancement and validation of AI-generated medical content are essential to ensure patient safety and information integrity.