Consultation letters as a method for assessing in-training performance in a department of medicine

This study examined the feasibility of using outpatient consultation letters as a means of assessing the clinical performance of clerks and house staff in a large teaching hospital. A total of 222 assessments were obtained based on samples of consultation problems from two clinical services. The method involved a two-stage rating scale based on the performance areas of Data Base, Problem Formulation, Diagnostic Tests, Management and Overall Assessment. The technique was investigated in terms of the objectivity, reliability, validity and efficiency of the performance ratings. Product-moment coefficients were used to assess inter- and intrarater agreement in scoring the consultation letters. These analyses identified a consistent difficulty in assessing the category of Data Base. For the remaining performance categories, interrater coefficients in the range .19-.76 and intrarater coefficients in the range .23-.96 were obtained. Other preliminary findings indicated good potential for a reliable and valid assessment. The method is technically feasible, and is based on a novel and reasonably unobtrusive approach to assessing trainee performance in a clinical setting.