Audit and summative assessment: system development and testing

Registrars in general practice have to submit an audit project as one of four parts of summative assessment. A criterion-referenced marking schedule has been developed in the West of Scotland, consisting of five independent criteria all of which have to be judged above minimum competence to pass. A system was developed to test the instrument using a marking exercise which calculated the sensitivity and specificity of the assessment process, for different combinations of assessors. One hundred and two registrar audit projects were then assessed by three independent assessors. Ninety-two (90%) passed and 10 projects (10%) were referred back to the registrar as being below minimum competence. After resubmission six projects (6%) passed, two projects (2%) were still below minimum competence, and two (2%) were not resubmitted. A referral process for assessing the audit projects of general practice registrars has been developed to maximize the opportunity of finding a project below minimum competence.