Assuring the quality of high-stakes undergraduate assessments of clinical competence

In the UK, and in many Commonwealth countries, a university degree is accepted by registration bodies as an indication of competence to practice as a PRHO or intern. Concerns have been raised that the quality of university examinations may not always be sufficient for such high-stakes decision-making. Assessments of clinical competence are subject to many potential sources of error. The search for standardization, and high validity and reliability, demands the identification and reduction of measurement errors and biases due to poor test design or variation in test items, judges, patients or examination procedures. Generalizability and other research studies have identified where the likely sources of error might arise and have been taken into account in the development of published guidelines on international best practice, which institutions should strive to follow. The purpose of this paper is to describe the development of the integrated final-year assessment of clinical competence at the University of Sheffield. The aim was to introduce a range of strategies to ensure the examination met the best practice guidelines. These included blueprinting the assessment to achieve a high degree of content validity; lengthening the examination by adding a written component to the OSCE component to ensure an adequate level of reliability; providing training and feedback for examiners and simulated patients; paying attention to item development; and providing statistical information to assist the examination committee in standard setting and decision-making. This evidence-based approach should be readily achievable by all medical schools.