Using reliability generalization methods to explore measurement error: an illustration using the MMPI-2 PSY-5 scales

Area of Science:

• Psychological assessment
• Psychometrics
• Meta-analysis

Background:

• Test score reliability is often assumed to be constant.
• However, reliability can vary significantly across different samples.
• Reliability generalization (RG) provides a method to systematically study this variation.

Purpose of the Study:

• To demonstrate the application of RG analysis to the MMPI-2 Personality Psychopathology 5 (PP5) scales.
• To examine the sample dependency of score reliability for the PP5 scales.
• To identify factors influencing score reliability variations.

Main Methods:

• A meta-analytic reliability generalization (RG) study was conducted.
• Sixty-three reliability coefficients for each of the MMPI-2 PP5 scales were collected and analyzed.
• Statistical analyses examined mean reliability differences and relationships with sample characteristics.

Main Results:

• Significant variability in reliability coefficients across samples was observed, supporting the sample-dependent nature of reliability.
• Reliability differed significantly across the five PP5 scales, with Negative Emotionality being most reliable and Aggression/Disconstraint least reliable.
• Score reliability was lower in nonclinical settings compared to clinical settings for some scales; sample sex composition and size were not significant predictors.

Conclusions:

• The study underscores the critical need for researchers to compute reliability estimates specific to their own research samples.
• RG analysis offers valuable insights into the psychometric properties of personality tests and informs their appropriate use.
• Findings highlight the importance of considering sample characteristics, such as clinical vs. nonclinical settings, when interpreting test score reliability.