Chatbots versus retina specialists in answering real-world retina questions

Chatbots powered by large language models have shown promising results when addressing medical queries, thus transforming access to medical information. Although these chatbots produce detailed and accurate responses, it is unclear how they perform when handling real, unedited patient questions, particularly in non-English languages. This study aimed to assess the readability, accuracy, and comprehensiveness of responses to retinal disease queries provided by four chatbots (ChatGPT 4.0, ConsensusGPT, Gemini, and Claude 3) compared to responses from retina specialists.

In this cross-sectional, comparative, and blinded study, twenty unedited questions about retinal diseases were randomly selected from a popular online video channel in Portuguese. The questions were submitted to the four selected chatbots and retina specialists with fellowship training. Two independent retinal experts evaluated the responses using standardized Likert scales for accuracy and completeness. Readability was assessed using the Flesch Reading Ease Score and the Flesch-Kincaid Grade Level tests. Additional metrics, including word count and response generation time, were analyzed. Data were compared among groups using non-parametric statistical tests, including the Kruskal-Wallis test with Dunn's pairwise comparisons and chi-squared tests, with a two-sided p-value threshold of 0.05 for statistical significance.

Retinal specialists and the Gemini chatbot produced responses with higher readability, indicating lower educational levels were needed for comprehension. In contrast, ChatGPT 4.0, ConsensusGPT, and Claude 3 delivered more detailed and accurate answers but required a higher reading level. ChatGPT 4.0 and ConsensusGPT achieved superior quality and comprehensiveness ratings compared to human experts and the other chatbots. Additionally, all AI systems generated responses significantly faster than the human specialists. Evaluators could correctly distinguish between human-generated and AI-generated responses in most cases.

Artificial intelligence chatbots demonstrate considerable promise for rapidly disseminating accurate medical information directly to the end-user, not only in English. However, optimizing the simplicity of their language is essential to ensure that detailed responses remain accessible to a broad audience. Future research should aim to replicate our findings with larger datasets of questions with the goal of refining these systems to balance comprehensive content with user-friendly language.