Machine learning to optimize literature screening in medical guideline development

Area of Science:

• Medical Informatics
• Evidence-Based Medicine
• Machine Learning

Background:

• The exponential growth of medical evidence necessitates efficient methods for literature selection in guideline development.
• Current manual processes for evidence selection are labor-intensive and can hinder the timely updating of medical guidelines.

Purpose of the Study:

• To evaluate the performance and feasibility of active learning (AL) in supporting the selection of relevant publications for medical guideline development.
• To investigate the impact of noisy labels on the effectiveness of AL in this context.

Main Methods:

• A mixed-methods design was employed, evaluating manual literature selection by clinicians across 14 searches.
• Simulations compared random reading with AL-based screening prioritization.
• Performance metrics included Work Saved over Sampling at 95% recall (WSS@95) and percentage Relevant Records Found at 10% reading (RRF@10).
• Average Time to Discovery (ATD) was used to identify potentially noisy labels, with accuracy discussed in a reflective dialogue.

Main Results:

• Manual title-abstract selection by clinicians showed moderate inter-rater reliability (mean Kappa = 0.50).
• Active learning demonstrated improved efficiency, with WSS@95 reaching up to 75.76% for full-text inclusion, significantly outperforming manual methods.
• The performance of AL deteriorated with increased label noise, highlighting the impact of human rater accuracy.

Conclusions:

• Active learning tools show promise in accelerating literature screening for medical guidelines.
• The efficacy of AL is critically dependent on the quality of human input; "noisy labels make noisy machine learning."
• Guideline developers and researchers must ensure accurate labeling to maximize the benefits of AL tools.