Evaluating data extraction error by a large language model from randomised controlled trials: a large-scale empirical...

Area of Science:

• Medical informatics
• Artificial intelligence in healthcare
• Clinical trial methodology

Background:

• Large language models (LLMs) offer potential for automating data extraction from clinical research.
• Evaluating the accuracy of LLMs in extracting data from randomized controlled trials (RCTs) is essential.

Purpose of the Study:

• To assess the data extraction accuracy of Claude 3.5 Sonnet on RCTs.
• To identify common error types and influencing factors in LLM-based data extraction.

Main Methods:

• An empirical study compared Claude 3.5 Sonnet's extractions against a human-verified dataset of 664 RCTs.
• Data extraction focused on basic trial information and adverse outcomes.
• Error rates were calculated and analyzed by error type and trial reporting quality (CONSORT adherence).

Main Results:

• The overall data extraction error rate for Claude 3.5 Sonnet was 6.6%.
• Misallocation (57.1%) and omitted data (23.2%) were the most frequent error types.
• Higher adherence to Consolidated Standards of Reporting Trials (CONSORT) guidelines correlated with lower LLM extraction errors.

Conclusions:

• Claude 3.5 Sonnet demonstrates a relatively low error rate for RCT data extraction.
• LLM applications in evidence synthesis require rigorous human oversight and detailed checking of outputs.