High-performance automated abstract screening with large language model ensembles.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Research
• Systematic Review Methodology

Background:

• Systematic reviews are crucial for evidence synthesis but involve labor-intensive abstract screening.
• Automating abstract screening can address the significant workload and time constraints in systematic reviews.

Purpose of the Study:

• To validate the accuracy of large language models (LLMs) for automating abstract screening in systematic reviews.
• To compare LLM performance against human screening and evaluate ensemble methods.

Main Methods:

• Six LLMs (GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o, Llama 3 70B, Gemini 1.5 Pro, Claude Sonnet 3.5) were tested on 23 Cochrane Library systematic reviews.
• Optimal prompting strategies were identified on a development dataset (n=800) and validated on a larger dataset (n=119,695).
• Performance metrics included sensitivity, precision, and balanced accuracy; ensemble methods (LLM-human, LLM-LLM) were also assessed.

Main Results:

• LLMs outperformed human screening on a development dataset in sensitivity, precision, and balanced accuracy.
• On a larger dataset, LLMs maintained high sensitivity but showed diminished precision due to class imbalance.
• LLM-human and LLM-LLM ensembles achieved perfect sensitivity and significant workload reductions (37.55%-99.11%).

Conclusions:

• Automated abstract screening using LLMs can reduce workload and maintain or improve quality in systematic reviews.
• Domain-specific validation is crucial due to performance variations across different reviews.
• LLM-human ensembles provide a viable method for efficient screening with continued human oversight.