Erroneous generalization-Exploring random error variance in reliability generalizations of psychological measurements

Area of Science:

• Psychometrics
• Educational Measurement
• Psychological Assessment

Background:

• Reliability generalization (RG) studies often link score reliability heterogeneity to measurement quality differences.
• Classical test theory posits reliability depends on true score and error score variance.
• Current RG approaches may inadequately differentiate these variance components.

Purpose of the Study:

• To propose and evaluate Boot-Err, a method for explicitly modeling error score variance in RG studies.
• To demonstrate that score reliability can vary independently of measurement quality.
• To enhance inferences about random measurement error and assess measurement consistency across administrations.

Main Methods:

• Utilizing bootstrapping and meta-analytic techniques to model error score variance.
• Conducting a comprehensive simulation scheme to test the Boot-Err approach.
• Applying the method to openly available direct replication data.

Main Results:

• Score reliability can fluctuate while measurement quality remains stable.
• Explicitly modeling error score variance improves inferences regarding random measurement error.
• The method allows for assessing the degree of measurement quality consistency across different sites.

Conclusions:

• Distinguishing true score variance from error score variance is crucial for accurate RG.
• Boot-Err offers a more nuanced approach to evaluating measurement quality in RG studies.
• This method enhances the validity of reliability generalization findings.