Reliability of instruments in a cooperative, multisite study: employment intervention demonstration program.

Area of Science:

• Psychiatry
• Psychometrics
• Mental Health Research

Background:

• Assessing the reliability of psychometric instruments is crucial for valid research in severe mental illness.
• Multisite studies require robust measures to ensure data quality across diverse settings.
• Previous reliability data may not fully represent individuals with severe mental illness.

Purpose of the Study:

• To evaluate the interrater reliability, internal consistency, and test-retest reliability of widely used instruments in a severe mental illness population.
• To identify potential psychometric weaknesses of specific scales within this clinical group.
• To inform best practices for data collection in multisite studies involving severe mental illness.

Main Methods:

• Examined reliability of the Positive and Negative Syndrome Scale (PANSS), Rosenberg Self-Esteem Scale, Medical Outcomes Study Short Form-36 (SF-36), and Quality of Life Interview.
• Data collected from 202 participants with severe mental illness in a multisite vocational study.
• Calculated intraclass correlation coefficients (ICCs) and coefficient alphas to assess reliability.

Main Results:

• Most scales demonstrated good reliability (ICCs and alphas > .70).
• The SF-36 showed lower temporal stability, particularly the Social Functioning subscale (ICC = .55).
• Test-retest reliability was notably lower for less educated participants and ethnic minorities.

Conclusions:

• While many instruments are reliable for severe mental illness populations, careful monitoring of psychometric properties is essential.
• The SF-36 may require cautious interpretation due to its temporal instability in this group.
• Future multisite studies should consider potential demographic influences on scale reliability and implement strategies for mitigation.