Collaborative Large Language Models for Automated Data Extraction in Living Systematic Reviews.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Systematic Review Methodology

Background:

• Data extraction is a significant bottleneck in conducting living systematic reviews (LSRs).
• Current methods are labor-intensive, hindering the timely synthesis of evidence.
• There is a need for efficient, automated approaches to data extraction.

Purpose of the Study:

• To develop and evaluate a generalizable, automated data extraction workflow using large language models (LLMs).
• To simulate a two-reviewer process for data extraction to enhance accuracy and reliability.
• To assess the performance of LLMs in extracting data for LSRs.

Main Methods:

• A dataset from a published LSR, comprising 10 clinical trials and 22 publications, was utilized.
• Two LLMs, GPT-4-turbo and Claude-3-Opus, were employed for data extraction.
• A cross-critique mechanism was implemented for discordant LLM responses, followed by accuracy evaluation against a gold standard.

Main Results:

• In the prompt development set, 96% of LLM responses were concordant with 0.99 accuracy.
• In the held-out test set, 87% of responses were concordant with 0.94 accuracy.
• Cross-critique resolved 51% of discordant responses, increasing their accuracy to 0.76.

Conclusions:

• LLM-driven data extraction in a simulated two-reviewer workflow demonstrates reasonable performance for LSRs.
• Concordant LLM responses are generally accurate, and cross-critique enhances accuracy for discordant ones.
• This automated approach facilitates the creation of truly 'living' systematic reviews.