Prioritising references for systematic reviews with RobotAnalyst: A user study.

Area of Science:

• Health Informatics
• Information Science
• Machine Learning Applications

Background:

• Systematic reviews and guideline development require extensive reference screening, a time-consuming process.
• Machine learning software can reduce human effort by prioritizing large reference collections for efficient screening.

Purpose of the Study:

• To describe and evaluate RobotAnalyst, a web-based system using text-mining and machine learning for reference organization and prioritization.
• To assess the effectiveness of RobotAnalyst in reducing the workload for systematic reviews.

Main Methods:

• RobotAnalyst employs text-mining and machine learning to organize references by content and prioritize them using a dynamic relevancy model.
• Evaluated on 22 reference collections (43,610 decisions), assessing the reduction in screened references needed to identify 95% of relevant studies.

Main Results:

• The number of references screened to find 95% of relevant inclusions was reduced for 19 out of 22 collections.
• Active prioritization in RobotAnalyst showed significant gains over random sampling for all reviews, unlike non-prioritized screening.
• Descriptive clustering was found to be more coherent and understandable by users than topic modeling.

Conclusions:

• RobotAnalyst provides empirical evidence of accelerating the identification of relevant studies in systematic reviews.
• The study highlights the need for user awareness of complacency and the implementation of stopping criteria to maximize work-saving benefits.