Can ChatGPT-5 educate the public about vasectomy?: a Google Trends-based expert panel assessment

Area of Science:

• Urology
• Artificial Intelligence in Healthcare
• Medical Information Dissemination

Background:

• Large language models (LLMs) like ChatGPT-5 are gaining public attention for health information.
• The effectiveness of ChatGPT-5 for sensitive urological topics, specifically vasectomy, remains unevaluated.
• Public interest in vasectomy is significant, as indicated by Google Trends data.

Purpose of the Study:

• To evaluate the accuracy, completeness, and public suitability of ChatGPT-5's responses to common vasectomy questions.
• To assess expert perceptions of ChatGPT-5's vasectomy-related information.
• To analyze the reliability of expert evaluations of AI-generated health content.

Main Methods:

• Eight medical experts (urologists, public health specialists, OB-GYN, fertility nurse) independently reviewed ChatGPT-5 responses.
• Responses to ten high-frequency vasectomy questions were rated on a 6-point Likert scale (accuracy, completeness, clarity, tone, usefulness, recommendability).
• Statistical analysis included descriptive statistics, Kruskal-Wallis tests, and intraclass correlation coefficients (ICC) for inter-rater reliability.

Main Results:

• Mean ratings ranged from 3.75 to 4.04 across domains.
• Clarity and tone received the highest scores, while medical accuracy and comprehensiveness showed greater variability.
• No significant differences were found between expert subgroups (p > 0.05).
• Inter-rater reliability was very low (ICC = -0.01), indicating substantial disagreement among experts.

Conclusions:

• ChatGPT-5 responses on vasectomy were generally clear and tonally appropriate for informational purposes.
• Significant variability in expert ratings highlights the need for cautious interpretation of AI-generated health information.
• AI outputs can be supportive educational tools but require expert oversight and adaptation for specific audiences.