Open-book tests to complement assessment-programmes: analysis of open and closed-book tests

Today's health sciences educational programmes have to deal with a growing and changing amount of knowledge. It is becoming increasingly important for students to be able to use and manage knowledge. We suggest incorporating open-book tests in assessment programmes to meet these changes. This view on the use of open-book tests is discussed and the influence on test quality is examined. To cope with the growing amount of medical knowledge, we have divided the body of knowledge into core knowledge, which students must know without need for references, and backup knowledge, which students need to understand and use properly with the help of references if so desired. As a result, all tests consist of a subtest for reproduction and understanding of core knowledge (a closed-book test) and a subtest for the ability to understand and manage backup knowledge (an open-book test). Statistical data from 14 such double-subtest exams for first and second-year students were analyzed for two cohorts (N = 435 and N = 449) with multilevel analysis, in accordance with generalizability theory. The reliability of the open and closed-book sections of the separate tests varied between 0.712 and 0.850. The open-book items reduce reliability somewhat. The estimated disattenuated correlation was 0.960 and 0.937 for cohorts 1 and 2 respectively. It is concluded that the use of open-book items with closed-book items slightly decreases test reliability but the overall index is acceptable. In addition, open and closed-book sections are strongly positively related. Therefore, open-book tests could be helpful in complementing today's assessment programmes.