Accuracy of LLMs to retrieve numeric data for meta-analysis in dentistry

Area of Science:

• Dental research
• Evidence synthesis
• Artificial intelligence in healthcare

Background:

• Systematic reviews and meta-analyses (SRMA) are crucial for evidence-based dentistry.
• Data extraction in SRMA is time-consuming and prone to errors.
• Large language models (LLMs) offer potential for automating SRMA data extraction.

Purpose of the Study:

• To evaluate the accuracy of four LLMs (DeepSeek v3 R1, Claude 3.5 Sonnet, ChatGPT-4o, Gemini 2.0-flash) in extracting primary numeric outcomes from dental research.
• To compare the performance of different LLMs in data extraction for SRMA.

Main Methods:

• LLMs were queried via APIs using default settings and a SMART-format prompt.
• Data extraction accuracy was assessed at sub-outcome, outcome, and study levels.
• Errors were categorized as hallucinations, missed data, or omissions.

Main Results:

• Overall extraction accuracy was high at the sub-outcome level.
• Gemini 2.0-flash performed significantly worse than other models (p < 0.01).
• Claude 3.5 Sonnet and DeepSeek-v3 R1 demonstrated superior accuracy and lower omission rates in full-text extraction.

Conclusions:

• LLMs show significant potential for data extraction in dental SRMA but have limitations.
• Accuracy varies between models, and cost does not correlate with performance.
• Standardized outcome reporting and accurate, lower-cost LLMs can improve evidence synthesis efficiency.