Related Experiment Video
Updated: Jan 13, 2026
06:28
E-Patient Counseling Trial E-PACO: Computer Based Education versus Nurse Counseling for Patients to Prepare for Colonoscopy
Published on: August 1, 2019
8.8K
Readability of Chatbot Responses in Prostate Cancer and Urological Care: Objective Metrics Versus Patient Perceptions
Lasse Maywald1, Lisa Nguyen2, Jana Theres Winterstein3,4
1Department of Urology, University Medical Center Mannheim, 68167 Mannheim, Germany.
Current Oncology (Toronto, Ont.)
|October 28, 2025
Summary
Large language models (LLMs) offer potential for patient education, but their readability in urooncology needs assessment. While objectively difficult to read, patients perceived chatbot responses as highly understandable, highlighting a gap between measured and perceived comprehension.
Area of Science:
- Urooncology
- Medical Informatics
- Health Literacy
Background:
- Low health literacy (12% of adults) necessitates improved readability of patient information.
- Large language models (LLMs) show promise for enhancing medical information accessibility.
- Existing evidence on LLM readability for patients is mixed, requiring clinical validation.
Purpose of the Study:
- To evaluate the measured and perceived readability of chatbot responses in speech-based interactions with urological patients.
- To assess the clarity, technical language, and explainability of GPT-4 chatbot outputs.
- To analyze associations between objective readability metrics and patient perception.
Main Methods:
- Urological patients engaged in unscripted conversations with a GPT-4 based chatbot.
- Transcripts were analyzed using Flesch-Reading-Ease (FRE), Lesbarkeitsindex (LIX), and Wiener-Sachtextformel (WSF) readability indices.
- Perceived readability was assessed via a survey on technical language, clarity, and explainability.
Main Results:
- 231 conversations were analyzed, focusing on prostate cancer, robotic-assisted prostatectomy, and follow-up.
- Objective readability scores indicated difficult text (FRE 43.1, LIX 52.8, WSF 11.2).
- Patients highly rated perceived readability (83-90%) for technical language, clarity, and explainability, with no correlation to objective scores.
Conclusions:
- Chatbot responses in urooncology were objectively difficult to read, exceeding health literacy recommendations.
- Despite objective difficulty, patients perceived the information as clear and understandable.
- A discrepancy exists between measured linguistic complexity and perceived comprehensibility, suggesting other factors influence patient understanding.
