Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers
These videos have been matched automatically. Contact us if they are not relevant.
These videos have been matched automatically. Contact us if they are not relevant.
View abstract on PubMed
Summary
This summary is machine-generated.Large language models (LLMs) and artificial intelligence (AI) tools show high performance in systematic review data extraction. AI-assisted extraction can replace a second human reviewer, improving efficiency.
Area Of Science
- Medical Informatics
- Artificial Intelligence in Healthcare
- Evidence Synthesis
Background
- Systematic reviews are crucial for evidence-based medicine but are resource-intensive.
- Automating data extraction using AI and LLMs offers a potential solution to reduce time and cost.
- No comprehensive workflow has been validated for AI in diverse systematic review types.
Purpose Of The Study
- To assess the efficacy of Elicit and ChatGPT in extracting data from journal articles.
- To evaluate AI tools as a substitute for one human data extractor in systematic reviews.
Main Methods
- Compared human-extracted data from 30 articles across three systematic reviews with AI-extracted data.
- Elicit and ChatGPT extracted population characteristics, study design, and review-specific variables.
- Calculated performance metrics (precision, recall, F1-score) against human double-extraction as the gold standard, followed by error analysis.
Main Results
- Elicit achieved 92% precision, recall, and F1-score; ChatGPT achieved 91%, 89%, and 90%, respectively.
- AI recall was high for study design (Elicit: 100%, ChatGPT: 90%) and population characteristics (Elicit: 100%, ChatGPT: 97%).
- Elicit and ChatGPT showed similar performance, with minor confabulations (4% of data points).
Conclusions
- AI tools demonstrate comparable performance to human reviewers for data extraction, especially for standardized variables.
- An AI-assisted extraction workflow, replacing the second human extractor, is proposed.
- Human reviewers can focus on reconciling AI-human discrepancies, optimizing the systematic review process.
These videos have been matched automatically. Contact us if they are not relevant.
These videos have been matched automatically. Contact us if they are not relevant.
Related Concept Videos
Metal ions can be separated from one another by complexation with organic ligands–the chelating agent– to form uncharged chelates. Here, the chelating agent must contain hydrophobic groups and behave as a weak acid, losing a proton to bind with the metal. Since most organic ligands used in this process are insoluble or undergo oxidation in the aqueous phase, the chelating agent is initially added to the organic phase and extracted into the aqueous phase. The metal-ligand complex is...
An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...
Multiple comparison test, abbreviated as MCT, is a post hoc analysis generally performed after comparing multiple samples with one or more tests. An MCT will help identify a significantly different sample among multiple samples or a factor among multiple factors.
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...