Do it faster with PICOS: Generative AI-Assisted systematic review screening.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Research

Background:

• Systematic reviews (SRs) are time-intensive, particularly the screening phase.
• Large Language Models (LLMs) show promise for expediting screening.
• The utility of LLM-generated structured PICOS summaries for reviewer assistance is unexplored.

Purpose of the Study:

• To evaluate the impact of LLM-generated structured PICOS summaries on screening speed and accuracy.
• To assess the performance of an open-source LLM (Mistral-Nemo-Instruct-2407) in creating these summaries.

Main Methods:

• Four neurology trainees, divided by experience, screened 1,003 articles.
• Two reviewers received titles, abstracts, and LLM-generated PICOS summaries.
• Two reviewers received only titles and abstracts; screening times and metrics were recorded.

Main Results:

• PICOS-assisted reviewers screened 75% faster and achieved 100% sensitivity.
• Assisted reviewers demonstrated higher accuracy (99.9%), specificity (99.9), and F1 scores (98.0%).
• A less experienced reviewer with PICOS assistance outperformed an experienced reviewer without assistance.

Conclusions:

• LLM-generated PICOS summaries improve screening efficiency and accuracy.
• AI assistance empowers less experienced reviewers, potentially leveling the playing field.
• Further research should explore broader applications and integration into automated systems.