Related Experiment Videos
A focused PubMed artificial intelligence search program creates better literature reviews than general purpose
James S MacLeod1, Michael S Lee1, Hayden Baker2,3
1Department of Orthopaedic Surgery, Medical College of Wisconsin, Milwaukee, WI, USA.
Background:
As the use of artificial intelligence (AI) expands into orthopedic surgery, the potential to use AI for review and summation of literature leaves researchers and clinicians questioning the quality of AI-driven literature reviews. This study aims to compare AI-generated literature reviews to those done by orthopedic surgery trainees.
Methods:
ChatGPT-4o and The-Literature, a directed AI platform, were asked to produce a literature review on 5 shoulder and elbow-related research topics. The primary output produced by ChatGPT-4, The-Literature, a fourth-year medical student, a third-year orthopedic surgery resident, and an orthopedic surgery sports medicine fellow were blinded then graded on a 1-5 Likert Scale for evidence and analysis, conclusion, references, and overall quality by 2 fellowship-trained orthopedic shoulder and elbow surgeons.
Results:
The average overall Likert ratings were 2.90 ± 0.88 for the medical student, 3.90 ± 0.74 for the resident, 3.50 ± 0.53 for the fellow, 2.30 ± 1.16 for ChatGPT-4, and 3.40 ± 0.84 for The-Literature. The resident and fellow reviews were significantly better than ChatGPT-4 (P = .0004, P = .0209), while the medical student review was not (P = .5092). None of the trainee reviews were significantly better than The-Literature. The-Literature's reviews were significantly better than ChatGPT-4 (P = .0412). The majority (58.8%) of ChatGPT-4's references were fabricated.
Conclusion:
A limited and directed AI search program was comparable to orthopedic surgery trainees and performed better than an open AI large language model. ChatGPT-4 had poor overall utility stemming mainly from its tendency to fabricate references.