Related Experiment Video
Updated: Jan 16, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Data Extractions Using a Large Language Model (Elicit) and Human Reviewers in Randomized Controlled Trials: A
Joleen Bianchi1,2, Julian Hirt1,3,4, Magdalena Vogt1
1Department of Health Eastern Switzerland University of Applied Sciences St. Gallen Switzerland.
Aim:
We aimed at comparing data extractions from randomized controlled trials by using Elicit and human reviewers.
Background:
Elicit is an artificial intelligence tool which may automate specific steps in conducting systematic reviews. However, the tool's performance and accuracy have not been independently assessed.
Methods:
For comparison, we sampled 20 randomized controlled trials of which data were extracted manually from a human reviewer. We assessed the variables study objectives, sample characteristics and size, study design, interventions, outcome measured, and intervention effects and classified the results into "more," "equal to," "partially equal," and "deviating" extractions. STROBE checklist was used to report the study.
Results:
We analysed 20 randomized controlled trials from 11 countries. The studies covered diverse healthcare topics. Across all seven variables, Elicit extracted "more" data in 29.3% of cases, "equal" in 20.7%, "partially equal" in 45.7%, and "deviating" in 4.3%. Elicit provided "more" information for the variable study design (100%) and sample characteristics (45%). In contrast, for more nuanced variables, such as "intervention effects," Elicit's extractions were less detailed, with 95% rated as "partially equal."
Conclusions:
Elicit was capable of extracting data partly correct for our predefined variables. Variables like "intervention effect" or "intervention" may require a human reviewer to complete the data extraction. Our results suggest that verification by human reviewers is necessary to ensure that all relevant information is captured completely and correctly by Elicit.
Implications:
Systematic reviews are labor-intensive. Data extraction process may be facilitated by artificial intelligence tools. Use of Elicit may require a human reviewer to double-check the extracted data.
More Related Videos
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
06:48Lexical Decision Task for Studying Written Word Recognition in Adults with and without Dementia or Mild Cognitive Impairment
Published on: June 25, 2019
Related Concept Videos
Group Design
Bioequivalence Experimental Study Designs: Repeated Measures, Cross-Over, Carry-Over, and Latin Square Designs
Bioequivalence Experimental Study Designs: Completely Randomized and Randomized Block Designs
Data Collection by Experiments
An example of the experimental method is a public...
Randomized Experiments
Simple randomization
Simple...
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...