Evaluation of text mining to reduce screening workload for injury-focused systematic reviews

Text mining to support screening in large-scale systematic reviews has been recommended; however, their suitability for reviews in injury research is not known. We examined the performance of text mining in supporting the second reviewer in a systematic review examining associations between fault attribution and health and work-related outcomes after transport injury.

Citations were independently screened in Abstrackr in full (reviewer 1; 10 559 citations), and until no more citations were predicted to be relevant (reviewer 2; 1809 citations, 17.1%). All potentially relevant full-text articles were assessed by reviewer 1 (555 articles). Reviewer 2 used text mining (Wordstat, QDA Miner) to reduce assessment to full-text articles containing ≥1 fault-related exposure term (367 articles, 66.1%).

Abstrackr offered excellent workload savings: 82.7% of citations did not require screening by reviewer 2, and total screening time was reduced by 36.6% compared with traditional dual screening of all citations. Abstrackr predictions had high specificity (83.7%), and low false negatives (0.3%), but overestimated citation relevance, probably due to the complexity of the review with multiple outcomes and high imbalance of relevant to irrelevant records, giving low sensitivity (29.7%) and precision (14.5%). Text mining of full-text articles reduced the number needing to be screened by 33.9%, and reduced total full-text screening time by 38.7% compared with traditional dual screening.

Overall, text mining offered important benefits to systematic review workflow, but should not replace full screening by one reviewer, especially for complex reviews examining multiple health or injury outcomes.

CRD42018084123.