AI-driven evidence synthesis: data extraction of randomized controlled trials with large language models

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Evidence Synthesis Methodology

Background:

• Large language models (LLMs) offer potential for improving evidence synthesis efficiency, especially in data extraction.
• Current LLM prompts for data extraction are limited, often focusing on standard items and not accommodating diverse research needs.
• Systematic reviews and evidence synthesis require accurate and efficient data extraction from studies like randomized controlled trials (RCTs).

Purpose of the Study:

• To develop and evaluate structured prompts for LLMs to extract data from randomized controlled trials (RCTs).
• To assess the feasibility and accuracy of using LLMs with structured prompts in evidence synthesis.
• To determine the efficiency of LLM-driven data extraction for systematic review methodology.

Main Methods:

• Developed comprehensive structured prompts with 58 items across six Cochrane Handbook domains for LLM data extraction.
• Utilized Claude (Claude-2) as the LLM platform for prompt testing.
• Tested the structured prompts on 10 randomly selected RCTs from published Cochrane reviews.

Main Results:

• Achieved a high overall correct data extraction rate of 94.77% (95% CI: 93.66% to 95.73%).
• Demonstrated strong domain-specific accuracy, ranging from 77.97% to 100%.
• Showcased remarkable efficiency, with an average extraction time of only 88 seconds per RCT.

Conclusions:

• LLMs, when guided by structured prompts, are feasible and valuable tools for enhancing data extraction in evidence synthesis.
• This approach represents a significant advancement in systematic review methodology, improving accuracy and efficiency.
• The findings support the integration of LLM-powered tools into the evidence synthesis workflow for more robust research.