A comparative study of screening performance between abstrackr and GPT models: Systematic review and contextual...

Area of Science:

• Bibliometrics
• Information Science
• Artificial Intelligence in Research

Background:

• Systematic reviews and rapid reviews are crucial for evidence synthesis.
• Literature screening is a major challenge in systematic reviews due to increasing publication volume.
• Automated tools aim to improve the efficiency of evidence synthesis.

Purpose of the Study:

• To compare the performance of Abstrackr and Generative Pre-trained Transformer (GPT) models in literature screening for systematic reviews.
• To evaluate the effectiveness of GPT-3.5 and GPT-4 against Abstrackr using key performance metrics.
• To identify optimal strategies for literature screening in evidence synthesis.

Main Methods:

• A systematic review methodology was employed.
• Searches were conducted in PubMed, Cochrane Library, and Web of Science.
• Performance metrics including recall, precision, specificity, and F1 score were analyzed.

Main Results:

• GPT models showed superior precision (0.51 vs. 0.21), specificity (0.84 vs. 0.71), and F1 score (0.52 vs. 0.31) compared to Abstrackr.
• GPT models demonstrated higher overall efficiency and better screening balance.
• GPT models were particularly effective in reducing false positives during fine-screening.

Conclusions:

• Abstrackr is suitable for initial screening phases, while GPT models excel in fine-screening.
• Hybrid systems combining Abstrackr and GPT models could enhance systematic review efficiency and accuracy.
• Future research should focus on developing integrated screening tools for different stages of systematic reviews.