Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models

Area of Science:

• Medical Informatics
• Natural Language Processing
• Clinical Trials

Background:

• Meta-analyses are crucial for robust treatment effectiveness estimates, synthesizing findings from multiple randomized controlled trials (RCTs).
• Current meta-analysis requires laborious manual data extraction from individual trial reports, limiting efficiency and scalability.
• Automating this data extraction using language technologies could enable on-demand meta-analyses.

Purpose of the Study:

• To evaluate the capability of modern large language models (LLMs) in reliably extracting numerical findings from clinical trial reports for meta-analysis.
• To assess LLM performance in zero-shot conditional extraction of numerical results linked to interventions, comparators, and outcomes.

Main Methods:

• Development and release of a granular evaluation dataset of clinical trial reports with annotated numerical findings.
• Evaluation of seven large language models (LLMs) using a zero-shot approach on the annotated dataset.
• Focus on extracting numerical results for interventions, comparators, and outcomes from trial reports.

Main Results:

• Massive LLMs demonstrate near-capability for fully automatic meta-analysis, particularly for dichotomous outcomes like mortality.
• LLMs exhibit poor performance when outcome measures are complex and require inferential reasoning to tally results.
• Performance limitations persist even for LLMs trained on biomedical texts.

Conclusions:

• Large language models (LLMs) are approaching the goal of fully automatic meta-analysis of randomized controlled trials (RCTs).
• Current LLMs face significant limitations in extracting and synthesizing complex numerical data from trial reports.
• Further advancements in LLMs are needed to overcome challenges in inferential data processing for comprehensive meta-analysis.