Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers

Area of Science:

• Biomedical Informatics
• Systematic Review Methodology
• Machine Learning Applications

Background:

• Systematic reviews often include nonrandomized studies.
• Screening citations for full-text review is time-consuming.
• Automated methods could improve efficiency.

Purpose of the Study:

• Evaluate machine learning (ML) classifiers for identifying nonrandomized studies.
• Assess the impact of optimization on classifier performance.
• Determine the potential reduction in screening workload.

Main Methods:

• Utilized an open-source data-mining suite for citation classification.
• Trained and tested ML classifiers (k-nearest neighbor, naïve Bayes, evolutionary support vector machine) on biomedical citations.
• Compared performance across different feature sets (bag of words, n-grams) and citation portions (titles, abstracts, full citations).
• Investigated optimization techniques including manual thresholding and cross-validation.

Main Results:

• No classifier achieved sufficient recall without optimization in Phase I.
• Optimization significantly improved performance for evolutionary support vector machine and complement naïve Bayes classifiers in Phase II.
• Evolutionary support vector machine and complement naïve Bayes classifiers reduced the initial retrieval set by 46% and 35%, respectively.
• Generalization performance varied among classifiers.

Conclusions:

• Machine learning classifiers are effective tools for identifying nonrandomized studies for systematic reviews.
• Optimization is crucial for enhancing classifier performance.
• Classifier generalizability differs, but substantial reductions in screening workload are achievable.