Evaluating AI Performance in Systematic Literature Reviews for HEOR: A Case Study

Area of Science:

• Health economics and outcomes research
• Evidence synthesis
• Artificial intelligence applications

Background:

• Systematic literature reviews (SLRs) are crucial for evidence generation in health economics and outcomes research.
• Traditional SLR workflows are time-consuming and labor-intensive.
• Artificial intelligence (AI) is being explored to enhance evidence synthesis efficiency.

Purpose of the Study:

• To evaluate AI performance in screening and data extraction for SLRs.
• To provide recommendations for the appropriate use of AI in literature reviews.

Main Methods:

• An AI-assisted SLR was conducted, mirroring a human-only SLR.
• AI tools were assessed for title/abstract screening, full-text screening, and data extraction.
• AI performance was benchmarked against traditional SLR methods, measuring accuracy, recall, and precision.

Main Results:

• AI demonstrated high accuracy and recall in title/abstract screening but low precision, leading to false positives.
• AI identified only a subset of studies during full-text screening compared to the traditional SLR.
• Mean data extraction accuracy was 72.93%, with no instances of AI hallucinating data.

Conclusions:

• AI tools require further refinement for optimal performance in SLR tasks, particularly in precision and full-text screening.
• AI can assist in evidence synthesis, but human oversight remains essential.
• Careful evaluation and recommendations are needed for integrating AI into literature review processes.