Related Experiment Videos
Sequential testing in the assessment of clinical skills
A M Muijtjens1, F H van Vollenhoven, S J van Luijk
1Department of Educational Development and Research, Faculty of Medicine, University of Maastricht, The Netherlands. a.muijtjens@educ.unimaas.nl
Purpose:
Educators who use standardized-patient-based (SP-based) tests may save resources by using sequential testing. In this approach, students take a short screening test; only those who fail take a second test. This study investigated whether sequential testing increases efficiency with only a minor decrease of validity.
Method:
In 1994-95, first- through fourth-year (Group 1) and sixth-year (Group 2) medical students at the University of Maastricht took SP-based tests. Each test took two days. In a simulation experiment based on the data from those tests, the authors considered the first day as the screening test and the second day as the second test. They investigated efficiency and validity as a function of the cutoff score of the screening test. They developed and evaluated a new method to determine the optimum cutoff score of the screening test, a method based on minimization of the loss represented by the (weighted) numbers of false positives and negatives in the screening test.
Results:
The negative predictive value (probability that a student would fail the complete test if he or she had failed the screening test) was low (<60%), while the positive predictive value was high (>96%). Accordingly, stringent pass/fail cutoff scores in the screening test (75% for Group 1 and 80% for Group 2) produced optimum results. Using those cutoff values, only 26% and 11% of 'the students would have had to take the complete test to get a "true" score, while only 0.2% and 0.0% of the students who passed the screening test went on to fail the complete test (false positives).
Conclusions:
In a sequential SP-based test, the pass/fail cutoff score of the screening test should be stringent. This can considerably reduce testing time (30% to 40%), while keeping the percentage of false positives at an acceptably low level of less than 0.2%. As an alternative to receiver operator characteristic analysis, minimization of the loss function was found to be an appropriate method to determine the optimum cutoff value of the screening test.