Performance of two large language models for data extraction in evidence synthesis

Area of Science:

• Artificial Intelligence in Scientific Research
• Biomedical Informatics
• Evidence Synthesis Methodologies

Background:

• Accurate data extraction is crucial for reliable evidence synthesis.
• Large language models (LLMs) present potential for automating data extraction.
• Uncertainty exists regarding the optimal LLM for evidence synthesis tasks.

Purpose of the Study:

• To compare the performance of two widely available LLMs, Claude 2 and GPT-4, for data extraction in evidence synthesis.
• To evaluate the accuracy of LLM-driven data extraction from full-text scientific articles.

Main Methods:

• Two LLMs (Claude 2, GPT-4) were used to extract pre-specified data elements from 10 published articles.
• Full study PDFs were processed using browser versions of the LLMs.
• GPT-4 required a third-party plugin for PDF parsing, while Claude 2 did not.
• Accuracy was assessed by comparing LLM outputs to previously extracted data.

Main Results:

• Claude 2 achieved high accuracy (96.3%) in data extraction.
• GPT-4, with its PDF parsing plugin, showed lower accuracy (68.8%), with most errors attributed to the plugin.
• Both LLMs accurately identified missing data elements and handled un-reported information.
• When provided with selected text, Claude 2 and GPT-4 achieved 98.7% and 100% accuracy, respectively.

Conclusions:

• LLMs have significant potential to enhance data extraction efficiency in evidence synthesis.
• Accurate PDF parsing is a critical factor influencing LLM performance.
• Human oversight remains essential for validating LLM-generated extractions.