Grand rounds in methodology: four key things to know about the reliability of measurement

Reliability is the most reported measurement characteristic used as evidence of the quality of a measurement. However, researchers often miss opportunities to design studies in a way that allows reliability to be calculated and report a calculation that is not germane to the purpose of the measurement.Reliability is a number that quantifies the ability of a measurement to distinguish between the members of a population with respect to a measured quantity. It is a simple function of the signal and noise in a measurement. In this paper, I review four key attributes. First, reliability has a single technical definition, but there are many statistics that purport to quantify reliability that do not fit this definition or obscure their relationship to it. This undermines the fundamental simplicity of the concept and its useful implications. Second, researchers sometimes do not appreciate that the relevant calculation of reliability changes with the purpose and conditions of measurement and then report the wrong number. Third, reliability is a summary measure with several components that may be as or more relevant to report than reliability. Fourth, reliability is specific to a population, for example, a patient satisfaction score that is highly reliable in one population could have abysmal reliability in a different population when using the same survey instrument.Reliability is an important part of evaluating and improving the measurements that form the foundations of scientific research in healthcare quality and safety. Understanding these four key attributes of reliability will improve the description and use of healthcare quality and safety measurements.