Enhancing Evidence Synthesis Efficiency: Leveraging Large Language Models and Agentic Workflows for Optimized...

Area of Science:

• Public Health
• Epidemiology
• Artificial Intelligence

Background:

• Global public health events necessitate accessible, up-to-date evidence curated through sustainable processes.
• The Global Repository of Epidemiological Parameters (grEPI) initiative explores AI-assisted pipelines for evidence screening.
• Advancements in large language models (LLMs) are leveraged for efficient evidence curation.

Purpose of the Study:

• To evaluate the performance of the GREP-Agent, an agentic-AI assisted pipeline, for screening epidemiological evidence.
• To assess the impact of LLMs and human feedback on the accuracy and efficiency of evidence screening.
• To measure the effectiveness of the GREP-Agent framework in a public health context.

Main Methods:

• The GREP-Agent was tested on 2000 citations from a measles systematic review.
• Four LLMs (GPT4o, GPT4o-mini, Llama3.1, Phi4) were integrated into the GREP-Agent framework.
• Performance was measured using accuracy, precision, recall, and F1-score, with human feedback for fine-tuning.

Main Results:

• Each phase of the GREP-Agent system demonstrated incremental accuracy improvements across different LLMs.
• GREP-Agent increased sensitivity to 84.2%-88.9% post-fine-tuning and 86.4%-95.3% with workload reduction strategies.
• Clarity of screening questions and threshold settings significantly impacted performance.

Conclusions:

• The GREP-Agent shows potential for enhancing evidence synthesis efficiency in public health.
• Adaptable human-in-the-loop AI systems are crucial for supporting public health responses.
• Further development should prioritize human-centric approaches in AI-driven literature screening.