Related Experiment Video
Updated: May 20, 2025
Metacarpal Small Incision for Carpal Tunnel Syndrome
Published on: April 5, 2024
GPT-4 as a Source of Patient Information for Carpal Tunnel Surgery: A Comparative Analysis Against Google Web Search
Paul G Mastrokostas1, Aaron B Lavi, Bruce B Zhang
1From the Department of Orthopaedic Surgery, SUNY Downstate Health Sciences University, Brooklyn, NY (Mastrokostas, Lavi, Zhang, Mastrokostas, Liu, Connors, and Hashem), and the Department of Orthopaedic Surgery, Warren Alpert Medical School of Brown University, Providence, RI (Connors).
Introduction:
Carpal tunnel surgery (CTS) accounts for approximately 577,000 surgeries in the United States annually. This high frequency raises concerns over the dissemination of medical information through artificial intelligence chatbots, Google, and healthcare professionals. The objectives of this study are to determine whether GPT-4 and Google differ in (1) the type of questions asked, (2) the readability of responses, and (3) the accuracy of numerical responses for the top 10 most frequently asked questions (FAQs) about CTS.
Methods:
A Google search was conducted to identify the top 10 FAQs related to CTS, which were then queried in GPT-4. Responses were categorized using the Rothwell classification system and evaluated for readability using Flesch Reading Ease and Flesch-Kincaid grade level scores. Statistical analyses included Cohen kappa coefficients for interobserver reliability and Student t -tests for comparing response characteristics. Statistical significance was set at the 0.05 level.
Results:
This study found that 70% of Google's FAQs were fact based, predominantly focusing on technical details (40%) and specific activities (40%). GPT-4's FAQs were mainly factual (50%), with technical details (40%) being the most queried topic. Complete agreement in interobserver reliability was observed. Google's answers were more readable than GPT-4's, with a Flesch Reading Ease score of 56.40 vs. 34.19 ( P = 0.001) and a Flesch-Kincaid grade level of 9.93 vs. 12.85 ( P = 0.007). Google responses were shorter, with an average word count of 91.50 compared with GPT-4's 162.90 ( P = 0.013). For numerical responses to FAQs, GPT-4 and Google differed in nine out of 10 questions, with GPT-4 often providing broader time frames.
Conclusion:
GPT-4 offers a more detailed and technically oriented approach to addressing patient queries about CTS when compared with Google. This suggests that GPT-4 can offer detailed insights where patients seek more in-depth information, enhancing the quality of healthcare education.
Level Of Evidence:
NA.
