Reliability for multilevel data: A correlation approach

Area of Science:

• Psychometrics
• Multilevel Modeling
• Generalizability Theory

Background:

• Reliability of measurement instruments is crucial in psychology and medicine.
• Existing reliability coefficients are limited for nested or multilevel data.
• Recent generalizability theory approaches by Schönbrodt et al. (2022) and ten Hove et al. (2022) show inconsistencies for cluster-level reliability.

Purpose of the Study:

• To propose an alternative approach for defining reliability coefficients in multilevel data.
• To address inconsistencies in current methods for quantifying reliability in nested designs.
• To compare the proposed method with existing approaches.

Main Methods:

• Developing a novel approach based on the expected correlation between repeated measurements.
• Analyzing common nested data structures: (a) raters crossed with persons/clusters, persons nested in clusters; (b) raters nested in persons/clusters; (c) persons nested in clusters, crossed with raters/days.
• Comparing the proposed method with Schönbrodt et al. (2022) and ten Hove et al. (2022).

Main Results:

• The proposed method offers a consistent definition of reliability coefficients for multilevel data.
• Identified and explained differences between the proposed approach and existing methods.
• Demonstrated the application across various common nested data structures.

Conclusions:

• The proposed approach provides a more accurate and consistent way to assess reliability in multilevel settings.
• Highlights the need for refined methods in psychometrics for complex data structures.
• Offers a valuable alternative for researchers dealing with nested data in psychology and medicine.