Evaluating the Efficacy and Efficiency of GPT-5 for Automated Title and Abstract Screening in Orthopedic Surgery...

To analyze the efficacy and efficiency of current large language models (LLMs), specifically GPT-5 in screening titles and abstracts for three review topics within different subspecialties in orthopedics.

Python scripts were developed to call on the GPT-5 model via OpenAIs application programming interface (API). Two human reviewers simultaneously performed screening based on the same inclusion and exclusion criteria. Performance metrics such as specificity, sensitivity, accuracy, positive predictive value (PPV), negative predictive values (NPV), and F1 scores for GPT-5 were calculated based on a gold-standard inclusion and exclusion list developed by a third human adjudicator. Efficiency metrics included total cost and time to completion for each task. The number of titles and abstracts to screen ranged between 668 and 1,131 amongst the three review topics. All performance metrics were above 92.3% amongst all three topics, with sensitivities ranging from 94.1%-100%. Time to completion ranged between 38.5-174.3 minutes. Cost ranged from $1.32-$3.73USD GPT-5 demonstrated exceptional accuracy, sensitivity, specificity, PPV, NPV, and F1 scores in automating title and abstract screening for three orthopedic systematic review topics in three different subspecialties. Results are similar to previous studies investigating the role of AI for screening, specifically increased accuracy and time-to-completion relative to humans. The average rate of screening ranged from 6.5-17.4 abstracts per minute and the average price ranged from $0.002-$0.0036USD per abstract, suggesting a high degree of efficiency compared to current standards.