Artificial Intelligence-Assisted Data Extraction With a Large Language Model: A Study Within Reviews

Area of Science:

• Health Services Research
• Artificial Intelligence in Healthcare
• Systematic Review Methodology

Background:

• Data extraction is crucial for evidence synthesis but is often time-consuming and prone to errors.
• Large Language Models (LLMs) present a novel AI solution for data extraction, notably without requiring pre-labeled training data.

Purpose of the Study:

• To compare the efficiency and accuracy of AI-assisted data extraction using LLMs against traditional human-only methods.
• To evaluate the performance of LLM-based data extraction within real-world systematic review workflows.

Main Methods:

• A prospective, parallel-group comparison (Study Within A Review - SWAR) was conducted across 6 ongoing systematic reviews.
• LLMs (Claude versions 2.1, 3.0 Opus, 3.5 Sonnet) performed initial data extraction, followed by human reviewer verification.
• Key metrics included concordance, time on task, accuracy, sensitivity, positive predictive value, and error analysis compared to a reference standard.

Main Results:

• The AI-assisted approach demonstrated higher accuracy (91.0%) than human-only extraction (89.0%) and reduced extraction time by a median of 41 minutes per study.
• Concordance between methods was 77.2%, with AI-assisted extraction showing higher sensitivity (89.4%) and positive predictive value (99.2%).
• Missed data items were the most frequent error type for both methods, with AI-assisted extraction having slightly lower error rates (9.0% vs. 11.0%).

Conclusions:

• AI-assisted data extraction using LLMs provides a viable and more efficient alternative to traditional human-only methods in evidence synthesis.
• The findings suggest that LLMs can significantly streamline the systematic review process, potentially improving resource allocation and review speed.