Closing the loop: assessing the effectiveness of psychiatric competency measures

During the past several years there has been a growing emphasis on competency assessment in graduate education in all specialties, including psychiatry. Methods of assessment are continuously being devised and refined.

To outline principles and methods of assessing the validity of competency measures and their relevance to trainees.

The authors reviewed the literature relevant to competency assessment and quality improvement in the fields of elementary and secondary education, adult learning, undergraduate medical education, and postgraduate medical training. Three means were used to evaluate the assessment program: 1) The authors surveyed graduates and credentialing agencies for feedback on the relevance of the measures they previously developed for current and envisioned future practice of psychiatry; 2) They measured the completion rates of competency measures by residents in the first 3 years of a program, measuring general psychiatric competencies; and 3) They surveyed residents and faculty regarding satisfaction with the existing process.

Despite small numbers, the results indicated that measures devised in 1998 were relevant to the practice of psychiatry in 2001 and envisioned by practitioners for the subsequent 5 to 10 years. Resident completion rates of competency requirements indicated that the structure of the measures was appropriately paced to parallel the learning expectations of the training program. Participant satisfaction with the process of developing and implementing the measures was high.

While devising and implementing competency assessments is critical, the process should not stop with the application of the measures. Ongoing evaluation of these measures for their continued suitability is essential. Training programs can readily determine the effectiveness of competency programs and improve them as necessary.