A situational judgement test of professional behaviour: development and validation

To describe the development of a single best answer multiple choice situational judgement test (SJT) to assess medical students' ability to judge professional behaviour; to present results of an expert validation of the SJT; to describe experiences with two different validation formats.

Based on educational objectives and work situations concerning professional behaviour, a SJT with 17 vignettes and 35 questions was developed. Best answers were developed according to available evidence. Forty-four experts validated the answers using either a rank order or a rating scale.

For sixteen questions in the rating group, thirteen questions in the ranking group and twelve questions in both groups more than two thirds of the experts agreed on a best answer. In the ranking group some experts found it difficult to commit themselves to a single best answer as instructed. In the rating group some experts did not mark any of the proposed answer options as adequate.

The definition of the construct of and experts for professional behaviour remain a challenge. Both validation scales have advantages and disadvantages. The two thirds criterion is somewhat arbitrary and lower or higher agreement may be acceptable or necessary depending on the purpose of the assessment.