Evaluating GPT Models for Automated Literature Screening in Wastewater-Based Epidemiology

Area of Science:

• Environmental Science
• Public Health
• Computer Science

Background:

• Wastewater-based epidemiology (WBE) requires efficient literature synthesis for public health.
• Meta-analysis is crucial for WBE but hindered by manual literature screening.
• Automated screening methods are needed to accelerate evidence-based public health responses.

Purpose of the Study:

• To evaluate GPT-3, GPT-3.5, and GPT-4 models for automated screening of WBE publications for meta-analysis.
• To assess the performance of AI models in identifying original data and relevant study details.
• To determine the cost-effectiveness and limitations of AI-assisted literature screening in WBE.

Main Methods:

• Utilized GPT-3, GPT-3.5, and GPT-4 models to screen WBE literature abstracts.
• Evaluated model performance based on precision and recall for identifying original data.
• Assessed accuracy in detecting studies reporting sampling locations.
• Analyzed the impact of different model formulations on output quality.

Main Results:

• GPT-4 demonstrated high accuracy (Precision=0.96, Recall=1.00) in differentiating papers with original data.
• AI screening exceeded manual screening quality standards (Recall=0.95) at a low cost (<$0.01/paper).
• GPT models showed lower accuracy in identifying relevant sampling locations, indicating a need for human oversight.
• Certain AI model choices yielded nonsensical results, emphasizing the importance of robustness.

Conclusions:

• AI-assisted literature screening, particularly with GPT-4, can significantly accelerate meta-analysis in WBE.
• Human intervention remains essential for specific data points like sampling locations and ensuring AI reliability.
• AI offers a cost-effective complementary tool to enhance the speed and efficiency of research synthesis in public health.