Progress testing with short answer questions

The concept of progress testing was developed in the 1970s. Significant features of progress tests are that the content is not linked to any specific course or unit, and that it reflects the final objectives of the curriculum as a whole. The questions are taken from a broad domain and cover a range of disciplines. Furthermore, the test is taken repeatedly over a period of time, to monitor students' progress. Known progress tests all use closed format questions. In 2002-2003 the University Medical Center Utrecht initiated a progress test with short answer questions. The test consists of 40 cases, each with a clinical and a basic science short-answer key feature question. This differs from other progress tests that use close format items, but also in the philosophy of mastery level testing and in the deliberate linking of basic science concepts to clinical case vignettes. The first four executions of the test show high internal consistencies (Cronbach's alpha 0.85 to 0.87) and satisfactory item parameters. The effort of marking answers is reasonable, the effort of writing case vignettes with short-answer items is less than writing MC-items if similar test reliabilities are to be achieved.