Artificial Intelligence and Machine Learning to Improve Evidence Synthesis Production Efficiency: An Observational...

Area of Science:

• Health Services Research
• Information Science

Background:

• Evidence syntheses are vital but time-consuming and resource-intensive.
• Artificial intelligence and machine learning (AI/ML) offer potential efficiency improvements in review processes.
• The impact of AI/ML on the entirety of evidence synthesis production remains largely unknown.

Purpose of the Study:

• To analyze the impact of AI/ML tools on resource use and completion time for evidence syntheses.
• To compare reviews using AI/ML with those that did not, based on internal recommendations.

Main Methods:

• Prespecified analyses of healthcare- or welfare-related reviews commissioned between August 2020 and January 2023.
• Compared reviews using AI/ML tools (ranking, classification, clustering, bibliometrics) versus those without.
• Accounted for nonrandomized assignment and censored outcomes; researchers and statisticians were blinded.

Main Results:

• AI/ML tools were used in 69% of the 39 reviewed studies.
• Reviews using AI/ML reportedly used more resources (667 vs. 291 person-hours) and were completed slightly faster (27.6 vs. 28.2 weeks).
• These differences in resource use and completion time were not statistically significant.

Conclusions:

• The association between AI/ML use and efficiency in evidence synthesis remains uncertain.
• Larger, multicenter studies or meta-analyses may be required to definitively assess the benefits of AI/ML tools.
• Further investigation is needed to determine if AI/ML tools can meaningfully reduce the time and resources required for evidence synthesis production.