Related Experiment Videos
Towards automating the initial screening phase of a systematic review
Tanja Bekhuis1, Dina Demner-Fushman
1Center for Dental Informatics, School of Dental Medicine, Department of Biomedical Informatics, School of Medicine, University of Pittsburgh, PA, USA.
Studies in Health Technology and Informatics
|September 16, 2010
Summary
Supervised machine learning, like EvoSVM, shows promise in reducing systematic review workload by assisting with initial citation screening. While recall is high, precision needs improvement for practical application in evidence-based medicine.
Area of Science:
- Medical Informatics
- Computational Biology
- Evidence-Based Medicine
Background:
- Systematic reviews are crucial for evidence-based medicine, but initial citation screening is labor-intensive.
- Current methods involve manual review of numerous citations by multiple team members.
- Automating parts of this process could significantly reduce workload and improve efficiency.
Purpose of the Study:
- To investigate the efficacy of supervised machine learning (ML) methods in reducing the workload of systematic review authors during the initial screening phase.
- To extend previous ML research by including observational studies for rare conditions.
- To evaluate the performance of different ML classifiers for citation screening.
Main Methods:
- Utilized annotated citations from a Cochrane systematic review for training and testing ML models.
- Extracted features from citation titles, abstracts, and metadata.
- Trained and optimized several classifiers, including EvoSVM, using 10-fold cross-validation.
- Evaluated classifier performance based on recall and precision metrics.
Main Results:
- In training, EvoSVM achieved 100% recall with 48% (Epanechnikov kernel) and 41% (radial kernel) precision.
- In testing, EvoSVM performance decreased, with 77% recall and 26%-37% precision.
- Near-perfect recall is identified as a critical performance measure for this task.
Conclusions:
- Supervised ML methods, particularly EvoSVM, demonstrate potential for reducing systematic review workload.
- The achieved recall is essential, but precision requires further optimization for practical implementation.
- ML tools may become valuable aids for systematic review teams under specific conditions.