Machine learning enables automated screening for systematic reviews and meta-analysis in urology

Area of Science:

• Urology
• Medical Informatics
• Machine Learning

Background:

• Meta-analyses (MA) are crucial in evidence-based medicine.
• Screening studies for MA can be time-consuming and prone to errors.
• Class imbalance in screening data presents a significant challenge.

Purpose of the Study:

• To investigate and implement semiautomated screening for MA in urology.
• To address the challenge of class imbalance during the screening process.
• To evaluate the performance of different machine learning algorithms and imbalance handling techniques.

Main Methods:

• Trained machine learning (ML) algorithms (Random Forest, Logistic Regression, SVM) on screening data from three urology MAs.
• Implemented various methods to handle class imbalance: sampling (up/down), weighting, cost-sensitive learning, and thresholding.
• Optimized models for sensitivity and evaluated metrics including specificity, ROC curves, missed studies, and work saved.

Main Results:

• Downsampling yielded optimal results during model training across algorithms.
• The weighting approach demonstrated superior performance on the final test dataset.
• Thresholding improved results over the standard 0.5 threshold, with optimized models achieving zero missed relevant studies in most cases.

Conclusions:

• A holistic methodology integrating presented methods and advanced text preprocessing is needed for practical implementation.
• Further investigation into cost-sensitive learning approaches for MA screening is warranted.
• No universal sample size recommendation can be made due to result heterogeneity.