Quality and Usability of Prostate Cancer Information Generated by Artificial Intelligence Chatbots: A Comparative...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Patient Education

Background:

• Patients increasingly use AI chatbots for health information, including prostate cancer details.
• Concerns exist regarding the quality, usability, and clinical relevance of AI-generated health content.
• This study evaluates patient-directed prostate cancer information from popular AI chatbots.

Purpose of the Study:

• To comparatively evaluate the quality, understandability, and actionability of prostate cancer information generated by AI chatbots.
• To assess the readability and clinical relevance of AI-generated patient-directed health content.

Main Methods:

• Standardized prostate cancer prompts were used across five AI chatbots (ChatGPT, Gemini, Claude, Copilot, Perplexity).
• Responses were assessed using DISCERN for quality and PEMAT-P for understandability/actionability by blinded reviewers.
• Readability was measured by Flesch-Kincaid scores, with inter-rater reliability assessed.

Main Results:

• Overall information quality was moderate (median DISCERN 56.5), with ChatGPT and Copilot scoring higher.
• Understandability was high (median PEMAT-P 91.7%), but actionability was poor (median PEMAT-P 0%).
• Readability indicated moderate complexity (median Flesch-Kincaid 50.4), with a median word count of 666.

Conclusions:

• AI chatbots provide understandable but moderately high-quality prostate cancer information lacking actionable guidance.
• Significant limitations include evidence transparency and practical patient support across evaluated platforms.
• Future AI development must integrate evidence-based resources and decision-support tools for enhanced prostate cancer education.