Related Experiment Video
Updated: May 29, 2025
Measuring 3D In-vivo Shoulder Kinematics using Biplanar Videoradiography
Published on: March 12, 2021
ChatGPT and Gemini Are Not Consistently Concordant With the 2020 American Academy of Orthopaedic Surgeons Clinical
Michael Megafu1, Omar Guerrero2, Avanish Yendluri3
1Department of Orthopedic Surgery, University of Connecticut, Farmington, Connecticut, U.S.A..
Purpose:
To evaluate the accuracy of suggestions given by ChatGPT and Gemini (previously known as "Bard"), 2 widely used publicly available large language models, to evaluate the management of rotator cuff injuries.
Methods:
The 2020 American Academy of Orthopaedic Surgeons (AAOS) Clinical Practice Guidelines (CPGs) were the basis for determining recommended and non-recommended treatments in this study. ChatGPT and Gemini were queried on 16 treatments based on these guidelines examining rotator cuff interventions. The responses were categorized as "concordant" or "discordant" with the AAOS CPGs. The Cohen κ coefficient was calculated to assess inter-rater reliability.
Results:
ChatGPT and Gemini showed concordance with the AAOS CPGs for 13 of the 16 treatments queried (81%) and 12 of the 16 treatments queried (75%), respectively. ChatGPT provided discordant responses with the AAOS CPGs for 3 treatments (19%), whereas Gemini provided discordant responses for 4 treatments (25%). Assessment of inter-rater reliability showed a Cohen κ coefficient of 0.98, signifying agreement between the raters in classifying the responses of ChatGPT and Gemini to the AAOS CPGs as being concordant or discordant.
Conclusions:
ChatGPT and Gemini do not consistently provide responses that align with the AAOS CPGs.
Clinical Relevance:
This study provides evidence that cautions patients not to rely solely on artificial intelligence for recommendations about rotator cuff injuries.
