Two-unit reliability analysis of questionnaires used in a regulatory system

An interjudge reliability test was conducted to evaluate the questionnaires used in the surveillance of residential care institutions. Because the reliability test was carried out as part of the routine surveillance program and not as part of a controlled experiment, it was subject to deviations from the optimal reliability test model. However, this nonpure design provided an opportunity to not only examine the reliability of the items in the surveillance tool, but also to gain a better understanding of the use of a reliability test in an "imperfect" field setting. Two different surveyor teams administered the 257 questions on the questionnaires to a representative sample of 32 institutions on two separate occasions. In order to explain the variance in the reliability scores, a multivariate analysis was conducted for two units of analysis: the surveillance questions and the institutions. Based on the results of the reliability test, changes were introduced to improve the questionnaires and their administration.