A novel approach to assess inter-rater reliability in the use of the Overt Aggression Scale-Modified

The ability of raters to apply measures of efficacy accurately and reproducibly in clinical trials of psychotropic medications is vital to the validity of study data. In a large, multi-center trial using the Overt Aggression Scale-Modified (OAS-M), inter-rater reliability was evaluated using standardized patients (SP) in a live 'mock' interview. Thirty raters experienced in the OAS-M each interviewed two actors trained to portray patients with intermittent explosive disorder, borderline and antisocial personality disorder, and/or post-traumatic stress disorder. Inter-rater reliability of OAS-M scores was evaluated using the intra-class correlation coefficient (ICC). The joint reliability of the raters in the study with an expert rater was also assessed by an ICC. The ICC was 0.96 between the raters, and 0.98 between the raters and expert rater. Results suggest that SP can be utilized in rater assessment and that raters administering the OAS-M in this model of rater reliability assessment demonstrate a high level of consistency and reliability. The method described here may be useful in future assessments of inter-rater reliability.