Related Experiment Video
Updated: Jan 12, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Artificial Intelligence-Assisted Data Extraction With a Large Language Model: A Study Within Reviews
Gerald Gartlehner1, Shannon Kugley2, Karen Crotty2
1Social, Statistical, and Environmental Sciences, RTI International, Research Triangle Park, North Carolina, and Department for Evidence-based Medicine and Evaluation, University for Continuing Education Krems, Krems, Austria (G.G.).
Background:
Data extraction is a critical but error-prone and labor-intensive task in evidence synthesis. Unlike other artificial intelligence (AI) technologies, large language models (LLMs) do not require labeled training data for data extraction.
Objective:
To compare an AI-assisted versus a traditional, human-only data extraction process.
Design:
Study within reviews (SWAR) using a prospective, parallel-group comparison with blinded data adjudicators.
Setting:
Workflow validation within 6 ongoing systematic reviews of interventions under real-world conditions.
Intervention:
Initial data extraction using an LLM (Claude, versions 2.1, 3.0 Opus, and 3.5 Sonnet) verified by a human reviewer.
Measurements:
Concordance, time on task, accuracy, sensitivity, positive predictive value, and error analysis.
Results:
The 6 systematic reviews in the SWAR yielded 9341 data elements from 63 studies. Concordance between the 2 methods was 77.2% (95% CI, 76.3% to 78.0%). Compared with the reference standard, the AI-assisted approach had an accuracy of 91.0% (CI, 90.4% to 91.6%) and the human-only approach an accuracy of 89.0% (CI, 88.3% to 89.6%). Sensitivities were 89.4% (CI, 88.6% to 90.1%) and 86.5% (CI, 85.7% to 87.3%), respectively, with positive predictive values of 99.2% (CI, 99.0% to 99.4%) and 98.9% (CI, 98.6% to 99.1%). Incorrect data were extracted in 9.0% (CI, 8.4% to 9.6%) of AI-assisted cases and 11.0% (CI, 10.4% to 11.7%) of human-only cases, with corresponding proportions of major errors of 2.5% (CI, 2.2% to 2.8%) versus 2.7% (CI, 2.4% to 3.1%). Missed data items were the most frequent error type in both approaches. The AI-assisted method reduced data extraction time by a median of 41 minutes per study.
Limitations:
Assessing concordance and classifying errors required subjective judgment. Consistently tracking time on task was challenging.
Conclusion:
Data extraction assisted by AI may offer a viable, more efficient alternative to human-only methods.
Primary Funding Source:
Agency for Healthcare Research and Quality and RTI International.
Related Concept Videos
00:56Extraction: Advanced Methods
01:16Non-equilibrium in the Cell