Training sample selection: Impact on screening automation in diagnostic test accuracy reviews

Area of Science:

• Medical Informatics
• Bibliometrics

Background:

• Systematic reviews require extensive article screening, a time-consuming manual process.
• Computerized tools for screening exist but often need large datasets for effective model training.
• Developing accurate predictive models for study inclusion is crucial for efficient screening.

Purpose of the Study:

• To present a novel approach for selecting training data for predictive models in systematic reviews.
• To compare the performance of models trained on data selected by cosine similarity versus established methods.
• To assess the impact of training set size and topic similarity on model performance.

Main Methods:

• Utilized a dataset of 50 Cochrane diagnostic test accuracy reviews.
• Employed cosine similarity to select relevant training data from similar reviews.
• Developed prediction models using selected data and compared performance against models trained on all available data.

Main Results:

• Prediction models performed best when trained on a larger number of reviews.
• Model performance was enhanced for target reviews with topics similar to other reviews in the dataset.
• The cosine similarity approach showed potential for reducing computational costs and screening time.

Conclusions:

• The proposed cosine similarity method offers an efficient way to select training data for systematic review screening models.
• Optimizing training data selection can significantly improve the performance and efficiency of computer-assisted review processes.
• This approach may lead to reduced computational burden and faster completion of systematic reviews.