An analysis of ChatGPT 4.0 and Google Gemini: readability, accuracy, and quality as patient education tools for...

The aim of this study is to assess the performance of chatbot-generated information for a complex injury, concussion, compared to two less complex injuries, sprains and fractures.

A cross-sectional study design was implemented. Queries on concussions, sprains, and fractures were developed and inputted into ChatGPT Version 4.0 and Google Gemini (Gemini). Responses were graded on accuracy, readability, quality, understandability, and actionability using guideline-based Likert scales, Flesch-Kincaid Grade Level, the DISCERN instrument, and the Patient Education Materials Assessment Tool for Printable Materials (PEMAT) instrument. T-tests, Mann-Whitney U Tests and One-way ANOVAs compared continuous variables, and Chi-square tests were used for categorical variables.

Only three out of 180 responses had misinformation, all of which related to concussions. Concussion-related queries (mean: 10.80) had statistically greater reading grade level responses compared to sprain (mean: 8.13; P<0.0001) and fracture (mean: 8.29; P<0.0001) queries. Median cumulative DISCERN scores significantly differed between chatbots (V4.0 median = 38.00, Gemini median = 44.50; P<0.0001), with no significant difference between injury type. There were no significant PEMAT understandability or actionability differences by injury type.

Our results show that responses to complex injuries have significantly higher reading grade levels with minor accuracy issues, but no differences in understandability and actionability. With better reading level, quality, and actionability, ChatGPT and Gemini chatbots could become conventional options for general information on injuries because they have minimal misinformation and high understandability.