MEASUREMENT: ACCOUNTING FOR RELIABILITY IN PERFORMANCE ESTIMATES.

Area of Science:

• Healthcare Management
• Statistical Methods in Medicine
• Physician Performance Evaluation

Background:

• Physician leaders struggle to differentiate true performance signals from random error in performance measurements.
• This uncertainty hinders confident decision-making for performance improvement.
• Random error, often due to small sample sizes, impacts the reliability of physician performance data.

Purpose of the Study:

• To explore the feasibility and impact of incorporating reliability adjustment into physician performance measurement.
• To enhance the confidence of physician leaders in performance data for improvement actions.
• To reduce the influence of random error on the interpretation of physician performance.

Main Methods:

• Application of reliability adjustment techniques to physician-level performance data.
• Utilizing Bayesian methods for reliability adjustment (case study explored a subset).
• Using provider group average as a basis for shrinkage (one of several potential methods).

Main Results:

• Incorporating reliability adjustment is feasible and can significantly reduce the appearance of random variation.
• The approach helps distinguish true performance signals more effectively than traditional methods.
• Results suggest a notable decrease in "real" signals compared to traditional approaches.

Conclusions:

• Reliability adjustment is a valuable tool for physician leaders seeking to improve performance measurement systems.
• It increases confidence in performance data, enabling more effective improvement strategies.
• Further research is needed on advanced methods and specialty-specific applications.