Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and...

Area of Science:

• Health Informatics
• Evidence-Based Medicine
• Artificial Intelligence in Healthcare

Background:

• Systematic reviews (SRs) are crucial for evidence-based medicine.
• Improving SR speed is essential for timely medical advancements.
• Machine learning (ML) tools offer potential for semi-automating citation screening to enhance efficiency.

Purpose of the Study:

• To compare the performance of two ML citation screening tools, Abstrackr and EPPI-Reviewer.
• To evaluate the screening prioritization functionality of these tools.
• To assess the potential reduction in screening burden offered by each tool.

Main Methods:

• Nine completed evidence reports were used for comparison.
• Abstrackr and EPPI-Reviewer were trained and tested on citation screening tasks.
• Performance was assessed by the proportion of citations screened to identify all relevant studies.

Main Results:

• Both tools demonstrated potential reductions in screening burden for large reports (4-60%).
• Performance varied across different report types and topics; EPPI-Reviewer generally outperformed Abstrackr for smaller reports.
• EPPI-Reviewer showed an advantage in identifying articles for full-text review, while Abstrackr was better for identifying final included articles in some cases.

Conclusions:

• Abstrackr and EPPI-Reviewer show promise for increasing efficiency in systematic reviews.
• Prioritization accuracy varied, highlighting the need for careful tool selection based on report characteristics.
• ML-powered screening prioritization offers efficiency gains while retaining human oversight.