The kappa statistic was representative of empirically observed inter-rater agreement for physical findings

To determine empirically chance agreement between different clinicians evaluating physical examination findings in children with acute abdominal pain.

Cross-sectional study of children age 3 to 18 years treated in a pediatric emergency department for acute abdominal pain. Three different examiners were provided the same historic information and asked to predict, independently and prior to examining the patient, the presence or absence of seven different clinical findings. Agreement between pairs of observers on these predicted findings was determined, and was defined as observed chance agreement. Actual examination findings were also recorded, and expected agreement due to chance was determined from the kappa statistic calculation.

There were 68 pair of observations between two pediatric examiners, and 46 pair between pediatric and surgical examiners. Observed and expected chance agreement were very similar for six of the seven clinical findings. Agreement beyond chance for the actual exam findings was generally poor, with kappa less than 0.5 for all but one finding.

Expected chance agreement, as calculated from the kappa statistic, is a reasonable reflection of empirically observed chance agreement between clinicians.