Do authors of review articles use systematic methods to identify, assess and synthesize information?

Review articles are an important source of summary information for practising clinicians to assist them in remaining current with the rapidly expanding medical literature. Consequently it is essential that these be of the highest quality. In this study we evaluate, according to published criteria, the methodological quality of review articles (R) including meta-analysis (MA) appearing in a major cancer journal, Journal of Clinical Oncology (JCO), 1983-1995.

A hand-search of JCO was performed, from the first issue January 1983 through December 1995, to identify R, defined as publications that describe and comment on studies relevant to a specific topic or clinical intervention. Only those dealing with aspects of treatment of human cancer were considered further. Methodological quality was first assessed using 8 criteria proposed by Mulrow, rated independently by two medical oncologists as: specified, unclear or not specified. MA, including studies of dose intensity, were further analyzed according to 23 more detailed criteria proposed by Sacks et al. and rated as adequate, partial or no/unknown compliance.

Of 176 review articles, 122 dealt with aspects of treatment of cancer. Compliance with four of Mulrow's eight criteria was generally good, in that 99% clearly stated a purpose, all attempted qualitative synthesis of data. 95% presented a summary and 76% considered future directions. However, in the 106 qualitative reviews (QR), authors rarely gave information on methods of data identification (11.3%), data selection (10.4%) and assessment of validity (8.4%). Structured abstracts seemed to improve the focus and clarity of QR and there was a minor improvement in deficient areas in the later time cohort (1990-1995). Based on 'adequate' compliance with each of the 23 criteria identified by Sacks et al., six dose intensity studies scored 7-12, seven literature data MA scored 10-15 and three individual patient data MA scored 16-18. The highest scores were in the sections relating to prospective design, combinability and statistical analysis. Factors relating to control of bias, sensitivity analysis and application of results were addressed less consistently.

With the exception of MA, the majority of authors contributing reviews to a major cancer journal, JCO, did not use systematic methods to identify, assess and synthesize information. Initiatives such as the Cochrane Collaboration Cancer Network can support and educate clinicians who wish to perform systematic reviews, but quality of reviews would also improve if author, editors and readers systematically applied any of the sets of criteria now available in the literature.