Comparison of Responses from ChatGPT-4, Google Gemini, and Google Search to Common Patient Questions About Ankle...

Ankle sprains are among the most common injuries treated by foot and ankle specialists. Despite increasing online health information-seeking behavior, existing information on ankle sprains has been criticized by physicians for poor quality and readability. ChatGPT-4 and Google Gemini are artificial intelligence chatbots known for their accessible interfaces and versatility, making them popular for patient self-education. This study evaluated the accuracy and readability of answers to frequently asked questions (FAQs) on ankle sprains from ChatGPT-4, Google Gemini, and Google Search.

ChatGPT-4, Gemini, and Google Search each generated 10 FAQs and answers about ankle sprains, with questions classified based on the Rothwell Classification System and answers graded using the Flesch-Kincaid Grade Level, Flesch-Kincaid Reading Ease Score, and Gunning Fog Score. ChatGPT-4 and Gemini were then asked to answer the 10 questions generated by Google Search. Two fellowship-trained foot and ankle surgeons and a current orthopaedic foot and ankle fellow independently graded responses for accuracy and comprehensiveness using a 4-point scale.

Only 10% of FAQs generated by ChatGPT-4, Gemini, and Google Search overlapped. When answering the same 10 FAQs from Google Search, ChatGPT-4 and Gemini provided significantly longer responses than Google Search, with mean differences of 353.1 ± 42.8 words ( P < 0.0001) and 87 ± 35 words ( P < 0.0001), respectively. ChatGPT-4's answers were more difficult to read based on the Flesch-Kincaid Grade Level (3.2 ± 1.7, P = 0.0012) and Flesch-Kincaid Reading Ease Score (22.2 ± 9.3, P < 0.0001). Surgeons ranked ChatGPT-4's answers as the most comprehensive and accurate and Google Search's answers as the least.

This study suggests that ChatGPT-4 and Google Gemini provide longer, more complex, yet more accurate and comprehensive answers than Google Search, highlighting their potential as patient education tools for orthopaedic conditions such as ankle sprains.