A test of medical problem-solving scored by nurses and doctors: the handicap of expertise

There is evidence that nurses fulfil the requirements of objective judgement better than doctors. Simulation of Initial Medical Problem-Solving (SIMP), a paper-and-pencil test for the assessment of medical problem-solving, consists of case histories, followed by an open-ended question. The scoring of open-ended questions is time-consuming and adds subjective bias to measurement error. In order to reduce scoring error, answers on SIMP are scored by means of scoring models in the form of check-lists with descriptions of elements of a correct answer. The reliability of the scoring was analysed in a study, with six nurses rating 500 answers. The overall interrater reliability was high, expressed by an intra-class correlation of 0.83. Selection of raters, and improvement of the scoring models, could increase the interrater reliability even further. In addition to the scoring by the nurses part of the material was scored again by two experienced doctors. The reliability of the scoring method on the whole was confirmed. Nevertheless, some evidence was found of misinterpretation of the scoring models by the nurses. Analysis at the item level revealed several instances in which both doctors agreed on a score for an element in an answer and all the nurses agreed on the opposite score. On the other hand, however, the two doctors were less consistent between themselves than the nurses. The disagreement between the doctors seems to be a consequence of differences in their own medical judgement of the case in question. The impact of the mistakes that are made by the nurses is much smaller than the loss of reliability caused by the inconsistency among the doctors.