Optimizing ChatGPT Applying prompt engineering to frequently asked questions in urologic oncology

Patients rely on online searches for patient education materials (PEMs). PEMs are recommended to be written at or below a sixth-grade reading level but are regularly written at a college reading level. Using prompt engineering, we assess the information, misinformation, and readability of ChatGPT responses to urologic oncology questions.

Forty-five questions relating to prostate, bladder, and kidney cancer were presented to ChatGPT (version 4o, OpenAI). Quality of health information was assessed using DISCERN (1 [low] to 5 [high]). Understandability and actionability were assessed using PEMAT-P (0 [low] to 100% [high]). Misinformation was scored from 1 [no misinformation] to 5 [high misinformation]. Grade and reading level were calculated using the Flesch-Kincaid scale [5 (easy) to 16 (difficult), and 100-90 (5th grade level) to 10-0 (professional level), respectively]. Prompt engineering was then applied to responses and evaluated.

ChatGPT answers are highly accurate but at too advanced a reading level and lacked explanations of benefits, risks, visual aids, actionability, and citations. Using prompt engineering, DISCERN (3.42-4.47, p<0.0001), PEMAT-P understandability (88.4-95.5%, p<0.0001), actionability (25.6-84.2%, p<0.0001), grade reading level (10.5-5.3, p<0.0001), and reading level (42 [college level] to 71.7 [7th grade], p<0.0001), all significantly improved. Misinformation did not change significantly.

Using prompt engineering, ChatGPT provides highly accurate and understandable PEMs at a patient-appropriate reading level and provides concrete resources for patient action. Urologists should understand prompt engineering and be involved in the development of artificial chatbots to optimize results.