Two-unit reliability analysis of questionnaires used in a regulatory system

Area of Science:

• Health Services Research
• Quality Improvement
• Survey Methodology

Background:

• Surveillance of residential care institutions relies on questionnaires.
• Ensuring the reliability of these tools is crucial for accurate data.
• Real-world application may deviate from ideal testing conditions.

Purpose of the Study:

• To evaluate the interjudge reliability of questionnaires used in residential care institution surveillance.
• To understand the application of reliability testing in a field setting.
• To identify areas for improving surveillance tools and administration.

Main Methods:

• An interjudge reliability test was conducted within a routine surveillance program.
• Two surveyor teams administered 257 questions to 32 institutions on two occasions.
• Multivariate analysis examined variance in reliability scores at question and institution levels.

Main Results:

• The study identified specific questionnaire items and institutional factors influencing reliability.
• Deviations from optimal testing models were observed but provided practical insights.
• The non-pure design allowed for examination of reliability in an "imperfect" field setting.

Conclusions:

• The reliability test provided valuable data for enhancing surveillance questionnaires.
• Adjustments were made to improve questionnaire content and administration procedures.
• The study demonstrated the utility of reliability testing in practical, non-experimental settings.