Limitations of the Inter-Unit Reliability: A Set of Practical Examples

Area of Science:

• Health Services Research
• Quality Improvement
• Biostatistics

Background:

• Healthcare quality measures are vital for evaluating provider performance.
• Assessing measure reliability is crucial before clinical implementation.
• Inter-Unit Reliability (IUR) is a common statistic for assessing reliability.

Purpose of the Study:

• To illustrate the practical implications of a limitation in the Inter-Unit Reliability (IUR) statistic.
• To demonstrate how certain measure development practices can affect IUR values.
• To highlight potential negative consequences of using arbitrary IUR thresholds.

Main Methods:

• Illustrative examples to demonstrate the practical implications of IUR limitations.
• Analysis of how risk adjustment and exclusion of unstable values impact IUR.
• Discussion of the consequences of discarding measures based on IUR thresholds.

Main Results:

• The Inter-Unit Reliability (IUR) may include variation unrelated to actual healthcare quality.
• Best practices like risk adjustment and exclusion of unstable data can decrease the sample IUR.
• This suggests that seemingly low IUR values do not always indicate poor reliability.

Conclusions:

• The IUR statistic has a severe limitation where between-provider variation may not reflect quality of care.
• Standard measure development practices can inadvertently reduce IUR.
• Discarding measures solely based on low IUR values may lead to the loss of useful quality indicators.