Evidence triangulator: using large language models to extract and synthesize causal evidence across study designs

Area of Science:

• Health research
• Biomedical interventions
• Behavioral interventions

Background:

• Conflicting health guidance on diet and behavior complicates evidence-based decision-making.
• Automated methods for evidence triangulation across diverse study designs are needed to balance biases and establish causality.
• Current approaches lack scalability for synthesizing complex scientific literature.

Purpose of the Study:

• To evaluate the performance of large language models (LLMs) in extracting ontological and methodological information from scientific literature.
• To automate the process of evidence triangulation for health research.
• To assess the utility of LLMs in identifying exposure-outcome relationships and their statistical significance.

Main Methods:

• A two-step information extraction approach using LLMs was developed.
• The first step focused on extracting exposure-outcome concepts.
• The second step involved relation extraction, including direction of effect and statistical significance.

Main Results:

• The two-step LLM extraction method outperformed a one-step approach.
• High performance was achieved in identifying the direction of effect (F1=0.86) and statistical significance (F1=0.96).
• Analysis of salt intake and blood pressure revealed a strong excitatory effect of salt on blood pressure (942 studies).

Conclusions:

• LLMs can effectively automate evidence triangulation by extracting crucial data from scientific literature.
• This automated approach complements traditional meta-analyses by integrating diverse study designs.
• The method enables rapid, dynamic assessment of scientific controversies and supports evidence-based health strategies.