Related Experiment Videos
Does ChatGPT provide safe and reliable patient information related to hip and knee arthroplasty?
Joëlle Deschênes-Bilodeau1, Peter Staunton2, Charles Desgagné2
1From McGill University, Montréal, Que. (Deschênes-Bilodeau, Desgagné); the Department of Orthopedics, McGill University, Montréal, Que. (Staunton); the Department of Orthopedic Surgery, Jewish General Hospital, Montréal, Que. (Antoniou). joelle.deschenes-bilodeau@mail.mcgill.ca.
Background:
Patients have been resorting to online content as their main source of medical knowledge, notably on orthopedic interventions. However, Web-based research findings present a reverse correlation between quality of content and popularity. We sought to evaluate whether ChatGPT could provide an alternative and safe source of medical information for arthroplasty patients.
Methods:
We gave 5 commonly Googled questions related to hip and knee arthroplasty to board-certified arthroplasty surgeons, fellows, and orthopedic surgery residents, as well as to ChatGPT 4.0. We anonymized all answers, which were then analyzed by an independent board-certified arthroplasty surgeon. We scored the answers for accuracy of content (6-point Likert scale) and completeness (3-point Likert scale), then compared the performance of all groups.
Results:
Among human responders, the mean accuracy grade was 75% (standard deviation [SD] 17%), with a mean completeness grade of 69% (SD 22%). The fellows represented the strongest subgroup, with 75% of their answers scored above 5/6 for accuracy (mean 82%, SD 16%). We found that all human-generated answers had a statistically significant correlation between accuracy and completeness. ChatGPT had a mean accuracy grade of 93% (SD 12%), with a mean completeness of 93% (SD 14%).
Conclusion:
ChatGPT appears to be a safe tool for patients to access general arthroplasty information online. It outperformed all human responders on both accuracy and completeness of answers. The strongest human responder group was the arthroplasty fellows. Further work is required to clarify the tool's performance against other easily accessible online patient information sources.
