Streamlining systematic reviews with large language models using prompt engineering and retrieval augmented...

Area of Science:

• Medical Informatics
• Evidence-Based Medicine
• Artificial Intelligence in Research

Background:

• Systematic reviews (SRs) are crucial for evidence-based guidelines but involve time-intensive literature screening.
• Large Language Models (LLMs) offer potential to accelerate SR processes.

Purpose of the Study:

• To compare the efficiency of a commercial tool (Rayyan) and an in-house LLM-based system for automating SR literature screening.
• To evaluate the performance metrics of both automated systems against manual screening.

Main Methods:

• A completed SR on Vitamin D and falls (14,439 articles) was used for comparison.
• Rayyan was trained on 2,000 articles, categorizing the rest.
• An LLM system utilized prompt engineering for title/abstract screening and Retrieval-Augmented Generation (RAG) for full-text screening.

Main Results:

• The LLM system achieved a 99.5% article exclusion rate (AER) and 100% negative predictive value (NPV), reducing manual screening time by 95.5% (25.5 hours total).
• Rayyan, with a threshold of 'likely to exclude,' achieved 0% FNR and 50.7% AER, but increased screening time to 81.3 hours.
• The LLM system successfully identified all relevant articles while significantly decreasing overall screening effort.

Conclusions:

• LLM-based systems substantially enhance SR efficiency compared to manual screening and commercial tools like Rayyan.
• The LLM approach maintains a low false negative rate, ensuring comprehensive inclusion of relevant studies.