Large language models show promising performance for some systematic review tasks but call for cautious...

Area of Science:

• Biomedical informatics
• Artificial intelligence in healthcare
• Systematic review methodology

Background:

• The increasing volume of biomedical literature poses challenges for conducting systematic reviews.
• Automation of systematic review processes is needed to improve efficiency.

Purpose of the Study:

• To evaluate the performance of large language models (LLMs) in automating systematic review and meta-analysis steps.
• To assess LLM capabilities across various stages of the systematic review workflow.

Main Methods:

• A systematic review of studies assessing LLM performance in systematic reviews.
• Searched PubMed, Embase, Cochrane Library, and preprint platforms up to January 14, 2025.
• Extracted data and assessed risk of bias, analyzing performance metrics like positive (PPA) and negative percent agreement (NPA).

Main Results:

• Included 63 studies with 148 LLM performance assessments, primarily on GPT models.
• LLMs demonstrated high performance in Title/Abstract (PPA 0.92, NPA 0.89) and Full-Text screening (PPA 0.93, NPA 0.92).
• Data extraction (median accuracy 0.95) and Risk of Bias assessment (median accuracy 0.62) showed variable but promising results, with newer LLMs outperforming older ones.

Conclusions:

• Large language models show significant potential for automating repetitive tasks in systematic reviews, especially screening.
• Integration of LLMs requires careful implementation and appropriate safeguards for reliable use in evidence synthesis.