The effect of measurement unreliability on sleep and respiratory variables

Unreliability associated with scoring sleep variables is a potentially problematic issue in clinical and research studies. When scoring unreliability is unrecognized, it can contribute to the following: increase variability in the measures of interest, decrease a study's ability to detect important relationships, attenuate correlation coefficients, and increase clinical trial costs.

This paper first models the relationship between scoring variability and reliability in commonly studied sleep variables. The paper then models the relationship between unreliability and sample size requirements and statistical power. Standard methods are used to model reliability using the intraclass correlation coefficient.

The analysis shows that when scoring unreliability is minimized (i.e., scoring reliability is maximized), correlation coefficients are more robust, sample size requirements are reduced, statistical power is increased, and clinical trial costs are reduced.

When scoring unreliability is recognized, research studies can compensate by increasing the number of research subjects studied; however, it is at the cost of increasing the costs of research and exposing greater numbers of subjects to possible study risks. An effective solution is to implement rigorous initial and ongoing training efforts to maintain high inter-rater and intra-rater reliability coefficients.