Variation between experienced observers in the interpretation of accident and emergency radiographs

Skill mix and role extension initiatives have highlighted the difficulty of establishing quality standards for the accuracy of plain film reporting. An acceptable performance might be one which is indistinguishable from that of a group of experienced consultant radiologists. In order to assess the feasibility of setting such a standard, the variation between experienced observers must first be established. This study examines the variation found between three observers with the three major types of plain film examination. 402 plain film examinations (205 skeletal, 100 chest and 97 abdominal) performed on accident and emergency patients were reported retrospectively and independently by three experienced radiologists. The clinical data supplied on the request cards were available to the readers. Each examination was categorized by each reader as being normal, as showing significant abnormality relevant to the current clinical problem, or as showing insignificant or irrelevant abnormality. Concordance between all three readers was found in 51%, 61% and 74% of abdominal, chest and skeletal radiographs, respectively. Weighted kappa values confirmed that the level of agreement between pairs of observers was higher with skeletal radiographs (kappa w = 0.76-0.77) than with chest (kappa w = 0.63-0.68), or abdominal (kappa w = 0.50-0.78) examinations. However, the frequency of major disagreements (at least one reader reporting "normal" and one reporting "relevant abnormality") was similar for abdominal (11%), chest (12%) and skeletal (10%) radiographs. When the reports were reclassified into only two groups--either significantly abnormal or not--pairs of observers disagreed on 9-10% of skeletal, 11-19% of chest and 8-18% of abdominal cases. The average incidence of errors per observer was estimated to be in the range 3-6%. The magnitude of interobserver variation in plain film reporting is considerable, and must be taken into account when designing assessment techniques and setting quality standards for this activity.