Handsearching versus electronic searching to identify reports of randomized trials

Systematic reviewers need to decide how best to reduce bias in identifying studies for their review. Even when journals are indexed in electronic databases, it can still be difficult to identify all relevant studies reported in these journals. Over 1700 journals have been or are being handsearched within The Cochrane Collaboration to identify reports of controlled trials in order to help address these problems.

To review systematically empirical studies, which have compared the results of handsearching with the results of searching one or more electronic databases to identify reports of randomized trials.

Studies were sought from The Cochrane Methodology Register (The Cochrane Library, Issue 2, 2002), MEDLINE (1966 to Week 1 July 2002), EMBASE (1980 to Week 25 2002), AMED (1985 to June 2002), BIOSIS (1985 to June 2002), CINAHL (1982 to June 2002), LISA (1969 to July 2002) and PsycINFO (1972 to May 2002). Researchers who may have carried out relevant studies were contacted.

A research study was considered eligible for this review if it compared handsearching with searching one or more electronic databases to identify reports of randomized trials.

The main outcome measure was the number of reports of randomized trials identified by handsearching as compared to electronic searching. Data were extracted on the electronic database searched, the complexity of electronic search strategy used, the characteristics of the journal reports identified, and the type of trial report identified.

Thirty-four studies were included. Handsearching identified between 92% to 100% of the total number of reports of randomized trials found in the various comparisons in this review. Searching MEDLINE retrieved 55%, EMBASE 49% and PyscINFO 67%. The retrieval rate of the electronic database varied depending on the complexity of the search. The Cochrane Highly Sensitive Search Strategy (HSSS) identified 80% of the total number of reports of randomized trials found, searches categorised as 'complex' (including the Cochrane HSSS) found 65% and 'simple' found 42%. The retrieval rate for an electronic search was higher when the search was restricted to English language journals; 62% versus 39% for journals published in languages other than English. When the search was restricted to full reports of randomized trials, the retrieval rate for an electronic search improved: a complex search strategy (including the Cochrane HSSS) retrieved 82% of the total number of such reports of randomized trials.

Handsearching still has a valuable role to play in identifying reports of randomized trials for inclusion in systematic reviews of health care interventions, particularly in identifying trials reported as abstracts, letters and those published in languages other than English, together with all reports published in journals not indexed in electronic databases. However, where time and resources are limited, searching an electronic database using a complex search (or the Cochrane HSSS) will identify the majority of trials published as full reports in English language journals, provided, of course, that the relevant journals have been indexed in the database.