Quality and readability of chatbot responses to patient questions: A systematic cross-sectional meta-synthesis

Introduction: Patients increasingly use chatbots to obtain medical information, a trend that has provoked both optimism and pessimism. Numerous studies have evaluated the quality and readability of these outputs. This study synthesizes these findings through a cross-sectional meta-synthesis. Methods: We identified studies that evaluated responses using the DISCERN instrument, designed to assess the quality of written material. Additionally, we only included studies that also evaluated readability. We recorded the chatbot used, DISCERN scores, the number of words in each question, the number of questions asked, the number of DISCERN evaluators, the readability of responses, and the year the study was conducted. We also assessed the influence of each publication's journal ranking using the Journal Citation Indicator. Results: We identified 42 studies that conducted 86 tests. Chatbot response readability decreased as response quality increased. Forty-nine tests produced responses ranked "good" or better, and only 10 scored below college-level readability. We significantly increased readability by adding the phrase "write responses at sixth-grade reading level" to prompts that previously produced post-graduate reading level responses in published studies. Discussion: Variable quality and poor readability of chatbot responses reinforce pessimism about their utility. Nevertheless, appropriate "prompt engineering" provides scope to enhance response quality and readability.