Development of a case report review instrument

Case reports are valued components of the medical literature. The assessment of case reports by editors of medical journals and peer reviewers is largely subjective. The purpose of this study was to develop a reliable instrument to evaluate the quality of written case reports. Instrument development involved review of the literature and the materials provided to peer reviewers who review manuscripts, communications with journal editors and discussions of the study team. After multiple amendments, the instrument was pilot tested on both published and unpublished case reports. Further revisions resulted in the final 11-item tool. Four independent reviewers evaluated 28 case reports in their original submission format that had been submitted to five medical journals. The reviewers were blinded to the specific journal that the manuscripts had been submitted and to whether the case reports had been accepted for publication. Inter-rater reliability was assessed using multirater kappa. Inter-rater reliability ranged from 0.03 to 0.90. The four variables with the highest agreement between raters were (i) rationale for writing the case report; (ii) implications of the case report; (iii) adequacy of the literature review; and (iv) overall impression about whether to accept or reject the manuscript (kappas of 0.67, 0.67, 0.90 and 0.67, respectively). Six of the instrument's first 10 variables were highly correlated with the reviewers' decision about whether to accept or reject the case report for publication (item 11) (all p < 0.001). No correlation existed between the reviewers' decision to accept or reject the manuscript and the actual decision that had been made by the various journals. The case report review instrument is the first such tool for objectively evaluating case reports and appears to have reasonable reliability. Medical journals may wish to incorporate the use of this instrument into the decision making about a case report's suitability for publication.