Can two psychotherapy process measures be dependably rated simultaneously? A generalizability study

Observer ratings in psychotherapy are a common way of collecting information in psychotherapy research. However, human observers are imperfect instruments, and their ratings may be subject to variability from several sources. One source of variability can be raters' assessing more than 1 instrument at a time. The purpose of this research is to investigate whether it is possible to have raters assess 2 different psychotherapy process measures simultaneously and still produce dependable scores. Two studies were designed. The first compared scores assessed by raters who rated either 1 instrument or 2 instruments simultaneously. The second compared scores of raters who assessed 2 instruments simultaneously and scores of expert raters who assessed 1 instrument. The results show that variability in scores is largely due to differences across the object of measurement (e.g., patients). Small variability was reported for raters, raters' interaction with patients, and whether the rater assessed 1 instrument or 2 instruments simultaneously. The results are promising for the quality of observer ratings of psychotherapy process and for the feasibility of future psychotherapy process research. (PsycINFO Database Record (c) 2012 APA, all rights reserved).