Maintaining reliability in a long-term psychiatric study: an ongoing inter-rater reliability monitoring program using...

The Longitudinal Interval Follow-up Evaluation (LIFE), has been shown to be a valid and reliable instrument for characterizing the week-by-week course of anxiety disorders examined retrospectively over the period of 1 year. Due to the chronic nature of these disorders, there is a need for reliable, valid instruments for measuring course over periods of several years if we are to learn more about the natural history of these disorders. This paper describes a rater-monitoring program designed to ensure long-term inter-rater reliability and prevent "rater drift". In this program, clinical interviewers score taped interviews and are required to maintain a median intra-class correlation coefficient (ICC) of at least 0.80 with the other raters. Raters also assess tapes from previous years, to ensure that they are using the same diagnostic criteria as earlier generations of interviewers. A reliability study was conducted to compare psychiatric status ratings (PSRs) collected using biweekly telephone interviews with the semi-annual interviews. The ICCs for panic, agoraphobia, social phobia, and generalized anxiety disorder were very good to excellent. Another reliability study examined the PSRs of subjects who had been previously lost to follow-up. ICCs for panic, agoraphobia, generalized anxiety disorder and depression were good to excellent. These results show that the LIFE, when used in conjunction with an intensive training and rater monitoring system, is a reliable instrument for use in longitudinal studies of the course of anxiety disorders.