Relative or Absolute Standards in Assessing Medical Knowledge Using Progress Tests

Norm-referenced pass/fail decisions are quite common in achievement testing in health sciences education. The use of relative standards has the advantage of correcting for variations in test-difficulty. However, relative standards also show some serious drawbacks, and the use of an absolute and fixed standard is regularly preferred. The current study investigates the consequences of the use of an absolute instead of a relative standard. The performance of the developed standard setting procedure was investigated by using actual progress test scores obtained at the Maastricht medical school in an episode of eight years. When the absolute instead of the relative standard was used 6% of the decisions changed: 2.6% of the outcomes changed from fail to pass, and 3.4% from pass to fail. The failure rate, which was approximately constant when using the relative standard, varied from 2% to 47% for different tests when an absolute standard was used. It is concluded that an absolute standard is precarious because of the variations in difficulty of tests.