Errors in search strategies were identified by type and frequency

Errors in the electronic search strategy of a systematic review may undermine the integrity of the evidence base used in the review. We studied the frequency and types of errors in reviews published by the Cochrane Collaboration.

Data sources were MEDLINE searches from reviews in the Cochrane Library, Issue 3, 2002. To be eligible, systematic reviews must have been of randomized or quasi-randomized controlled trials, reported included and excluded studies, and used one or more sections of the Cochrane Collaboration's Highly Sensitive Search Strategy. MEDLINE search strategies not reported in enough detail to be assessed or that were duplicates of a search strategy already assessed for the study were excluded. Two librarians assessed eligibility and scored the eligible electronic search strategies for 11 possible errors. Dual review with consensus was used.

Of 105 MEDLINE search strategies examined, 63 were assessed; 31 were excluded because they were inadequately reported, and 11 were duplicates of assessed search strategies. Most (90.5%) of the assessed search strategies contained > or =1 errors (median 2, interquartile range [IQR] 1.0-3.0). Errors that could potentially lower recall of relevant studies were found in 82.5% (median 1, IQR 1.0-2.0) and inconsequential errors (to the evidence base) were found in 60.3% (median 1, IQR 0.0-1.0) of the search strategies. The most common search errors were missed MeSH terms (44.4%), unwarranted explosion of MeSH terms (38.1%), and irrelevant MeSH or free text terms (28.6%). Missed spelling variants, combining MeSH and free text terms in the same line, and failure to tailor the search strategy for other databases occurred with equal frequency (20.6%). Logical operator error occurred in 19.0% of searches.

When the MEDLINE search strategy used in a systematic review is reported in enough detail to allow assessment, errors are commonly revealed. Additional peer review steps are needed to ensure search quality and freedom from errors.