Accuracy of large language models in data extraction from randomized controlled trials in sleep medicine: A...

This proof-of-concept study examined the performance of two prominent large language model (LLM)-based AI tools, ChatGPT 4o and Claude 3.5, in extracting data for four specific tasks: group size, event count, mean value, and standard deviation. Utilizing an established database that analyzed data extraction errors in systematic reviews on sleep medicine, we tested the ability of both AI tools to extract data from 648 randomized controlled trials (RCTs) using single- and multiple-sentences prompting approaches. The accuracy of the extracted data was compared to error-corrected metadata, with an overall accuracy reaching up to 71.5 % (95 % CI: 69.3 %, 73.7 %) for Claude and 69.1 % (95 % CI: 66.8 %, 71.3 %) for ChatGPT. Claude demonstrated superior performance over ChatGPT across all tasks, with the largest accuracy difference of up to 12.7 % (OR = 1.70, 95 % CI: 1.38, 2.10). The single-sentence prompt led to lower accuracy compared to the multiple-sentences prompts, with the largest percentage difference being -11.0 % (OR = 0.64, 95 % CI: 0.52, 0.78). Both AI tools achieved strong performance in extracting group size data. These findings underscore the potential of AI tools like Claude, especially when combined with effective prompting strategies like multiple-sentences prompts, to assist data extraction in sleep medicine research.