Assessment of psychotherapy skills: the problem of interrater agreement

Thirteen experienced supervising psychiatrists independently rated the psychotherapy interviews of six residents, using a detailed assessment instrument. Although interrater agreement was significantly greater than would be expected by chance, the extent of agreement was uniformly low. Agreement was significantly poorer for judgments about technique skills than about communication skills and for second-year than for first-year residents. Other factors, such as the faculty rank and training of the rater, did not covary significantly with observed ratings. The findings suggest that rating methods which purport to assess psychotherapy skills should undergo careful methodological evaluation and testing before they are accepted as valid assessment techniques.