A Machine Learning and Large Language Model Tool for Systematic Literature Reviews of Health Economic Evidence: A...

Area of Science:

• Health Economics
• Artificial Intelligence in Research
• Systematic Literature Review Methodology

Background:

• Artificial intelligence (AI) is increasingly integrated into systematic literature review (SLR) software, showing promise for efficiency gains.
• AI tools demonstrate acceptable performance in SLRs of clinical trial publications.
• Economic evaluation publications present unique challenges due to inconsistencies in content, terminology, and structure.

Purpose of the Study:

• To evaluate the efficiency and accuracy of AI-assisted tools in conducting SLRs of economic evaluations.
• To benchmark AI performance against a manual SLR for economic evaluations of chronic rhinosinusitis with nasal polyps.

Main Methods:

• Replication of a prior manual SLR using AI-powered tools (Robot Screener and Smart Screener) within Nested Knowledge software.
• Machine learning-based inclusion prediction model and large language model-based criteria screener were employed.
• Performance was compared against the original human-conducted SLR.

Main Results:

• AI search retrieved 51% of relevant articles from the original SLR.
• Title/abstract screening accuracy exceeded 95%, while full-text screening accuracy fell below 80%.
• Data extraction was reliable for general characteristics but less accurate for complex modeling details and assumptions.

Conclusions:

• AI tools improve efficiency in SLRs of economic evaluations, particularly for title/abstract screening and general data extraction.
• Full-text screening and interpretation of nuanced modeling choices require further development for AI.
• Current AI reliability for economic evaluations is lower than that observed for clinical trial SLRs.