Beyond the Spearman-Brown: a structural approach to maximal reliability

Area of Science:

• Psychometrics
• Statistical Modeling
• Reliability Theory

Background:

• Traditional reliability theory relies on the assumption of parallel parts, limiting its application.
• Existing methods often treat items, raters, or judges as interchangeable entities.

Purpose of the Study:

• To recast reliability theory within a structural equation modeling (SEM) framework.
• To develop a method for determining unique reliability estimates for individual components (items, raters, judges).
• To assess the maximal reliability of weighted composites and perform inferential tests on composite reliability.

Main Methods:

• Utilizing a structural equation framework to model reliability.
• Calculating unique reliability estimates for individual components.
• Aggregating unique estimates to determine composite reliability.
• Generalizing procedures from single-factor to multifactor applications.

Main Results:

• Demonstrated that items, raters, or judges do not need to be treated as equivalent.
• Established procedures for determining unique reliability estimates for each component.
• Showcased the assessment of maximal reliability for weighted composites.
• Extended the methodology to multifactor applications.

Conclusions:

• A structural approach offers a more flexible and nuanced method for reliability determination.
• This framework moves beyond the limitations of the parallel parts assumption.
• The approach allows for inferential testing of composite reliability.
• Researchers should consider the implications for the true score hypothesis.