Optimal large language models to screen citations for systematic reviews.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Health Services Research

Background:

• Large language models (LLMs) show promise for automating systematic review citation screening.
• The comparative efficiency of different LLMs for this task is not well-established.
• Systematic reviews are crucial for evidence-based medicine but are time-consuming.

Purpose of the Study:

• To compare the accuracy, efficiency, cost, and consistency of four LLMs (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, Llama 3.3 70B) in citation screening.
• To evaluate the impact of an ensemble approach using multiple LLMs on screening performance.
• To assess the potential of LLMs to streamline systematic review workflows.

Main Methods:

• Four LLMs were used to screen citations for clinical questions from the Japanese Clinical Practice Guidelines for Sepsis and Septic Shock 2024.
• Sensitivity, specificity, screening time, cost, and consistency were calculated for each LLM.
• A post hoc analysis examined the effect of combining LLM outputs.

Main Results:

• GPT-4o and Llama 3.3 70B demonstrated high specificity but lower sensitivity.
• Gemini 1.5 Pro and Claude 3.5 Sonnet achieved higher sensitivity with lower specificity.
• GPT-4o was fastest, Llama 3.3 70B was most cost-effective, and consistency was similar across models.
• An ensemble approach improved sensitivity but increased false positives.

Conclusions:

• Individual LLMs offer distinct strengths for citation screening, saving time and reducing workload.
• While LLMs streamline screening, managing false positives remains a challenge.
• Combining LLMs can enhance sensitivity, highlighting their potential to optimize systematic review processes.