A question-answering framework for automated abstract screening using large language models

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Evidence Synthesis

Background:

• Systematic reviews (SRs) require rigorous abstract screening, a process that is often time-consuming and resource-intensive.
• Current methods for abstract screening can be inefficient, leading to delays in evidence synthesis.

Purpose of the Study:

• To develop and validate a novel framework for abstract screening in SRs using the zero-shot capabilities of large language models (LLMs).
• To transform abstract screening into a question-answering (QA) task, leveraging LLMs to align abstracts with SR selection criteria.

Main Methods:

• LLMs were employed to prioritize candidate studies by framing abstract screening as a QA task, where selection criteria act as questions.
• The framework involved breaking down criteria into questions, prompting LLMs, scoring answers, and combining responses for inclusion/exclusion decisions.
• Validation was conducted on the CLEF eHealth 2019 Task 2 benchmark, utilizing GPT-3.5 and comparing against traditional and fine-tuned BERT-family models across 31 datasets.

Main Results:

• The proposed LLM-based QA framework demonstrated a significant advantage over traditional information retrieval and fine-tuned BERT models in prioritizing studies.
• Performance improvements were achieved by re-ranking LLM answers based on semantic alignment between abstracts and selection criteria.
• The framework showed consistent effectiveness across diverse SR categories and proved viable with different LLMs.

Conclusions:

• LLMs are highly effective in prioritizing candidate studies for abstract screening within SRs using the developed QA framework.
• Leveraging selection criteria as queries significantly enhances the performance of automated abstract screening.
• The study underscores the potential of LLMs to streamline and improve the efficiency of evidence synthesis processes.