Supporting Literature Reviews: A Comparison Between Human and Generative Artificial Intelligence Screening for a...

Scoping reviews are used to map the literature within a field or discipline, summarize existing evidence, and identify gaps in knowledge. Conducting a scoping review is often labor-intensive, requiring significant human resources. Artificial intelligence (AI) tools may offer efficiencies in stages of the review process; however, their accuracy and impact on rigor remain uncertain. This study used a retrospective cross-sectional agreement design to compare title and abstract screening decisions made by ChatGPT 3.5 with decisions made by human reviewers. Of the 3154 articles initially retrieved, 3148 were screened by both the research team and ChatGPT 3.5, with 6 articles excluded due to incomplete data or upload errors. During title and abstract screening, the human research team excluded 2661 articles (84.5%), whereas ChatGPT 3.5 excluded 1533 articles (48.7%). Our findings suggest that although AI-assisted screening may reduce time by filtering out a portion of irrelevant studies early in the process, these efficiencies must be balanced against the depth of understanding gained through review among the human team. Furthermore, the dialogue and consensus-building among research team members may be diminished when AI tools are used. This reduction in scholarly engagement may limit opportunities for critical appraisal, learning, and deeper comprehension of the evidence.