Variability in doctors' problem-solving as measured by open-ended written patient simulations

The uncertain validity of written simulations could be due to the difficulty in setting criteria for optimal performance. Usually criteria are set by definition of a limited number of 'correct answers' by a panel of experts reached through an open discussion. This is an artificial situation which entails mutual influence and forces the participants to respond to the necessity to reach a consensus. In the present report we describe an attempt to set 'correct answers' by the independent performance of 15 board-certified internists on four written simulations. There was a marked variability in responses due to legitimate differences in approach, to obvious errors in interpretation of the provided data and to possible differences between the expert behaviour in a real life and in a simulated setting. We believe that the criteria for acceptable performance on written clinical simulations should be determined by independent experts, rather than by a group consensus. Students who receive after the examination a compiled list of options selected by experts in response to the same questions may obtain a more realistic insight into the complexity of clinical problem-solving.