GPT-3.5 Turbo and GPT-4 Turbo in Title and Abstract Screening for Systematic Reviews

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Systematic Reviews

Background:

• Systematic reviews require efficient literature screening.
• Large language models (LLMs) are increasingly used in research.
• Comparing different LLM versions for specific tasks is crucial.

Purpose of the Study:

• To compare the performance of GPT-4 Turbo and GPT-3.5 Turbo in citation screening for systematic reviews.
• To evaluate specificity, sensitivity, and processing time of both LLM versions.

Main Methods:

• Citation screening was performed using both GPT-4 Turbo and GPT-3.5 Turbo.
• Performance metrics included specificity, sensitivity, and processing time for a set of studies.

Main Results:

• GPT-4 Turbo demonstrated superior specificity (0.98) compared to GPT-3.5 Turbo (0.51).
• Sensitivity was comparable between GPT-4 Turbo (0.85) and GPT-3.5 Turbo (0.83).
• GPT-3.5 Turbo processed studies faster (0.9 min) than GPT-4 Turbo (1.6 min) for 100 studies.

Conclusions:

• GPT-4 Turbo's higher specificity makes it potentially more suitable for citation screening in systematic reviews.
• LLMs offer significant potential for optimizing literature selection processes.
• Further research into LLM applications in evidence synthesis is warranted.