Study filters for non-randomized studies of interventions consistently lacked sensitivity upon external validation

Area of Science:

• Medical informatics
• Health sciences research
• Systematic review methodology

Background:

• Limited evidence exists on effective search strategies for non-randomized studies (NRS) in bibliographic databases.
• Challenges include identifying non-randomized controlled trials (NRCTs) using randomized controlled trial (RCT) filters and the availability/performance of filters for other NRS types.
• The study aimed to identify, validate, and evaluate NRS filters in MEDLINE.

Purpose of the Study:

• To identify and validate existing non-randomized study (NRS) filters in MEDLINE.
• To evaluate the performance of established randomized controlled trial (RCT) filters when applied to NRS.
• To assess the feasibility of restricting search strategies for specific NRS types.

Main Methods:

• Retrospective analysis of study filters using MEDLINE citations from Cochrane reviews.
• Identification of 14 NRS filters published since 2001.
• Generation of reference and test sets from 2890 NRS citations across 12 study types.

Main Results:

• Identified 14 NRS filters, with search yields ranging from 660,000 to 9.5 million hits in MEDLINE.
• Most filters covered multiple study types.
• No single filter achieved the target sensitivity (≥ 92%) across all NRS types evaluated.

Conclusions:

• Current non-randomized study (NRS) filters demonstrate insufficient performance for practical application in systematic reviews.
• Development of novel search strategies, potentially combining new NRS filters with other techniques, is required.
• Addressing the unique challenges of searching for NRS is crucial for improving evidence synthesis.