Performance profiling in primary care: does the choice of statistical model matter?

Area of Science:

• Health Services Research
• Biostatistics
• Health Informatics

Background:

• Provider profiling is vital for healthcare improvement efforts.
• Accurate profiles necessitate appropriate statistical models to reflect true provider performance.
• Sophisticated models exist but can be complex for providers to understand.

Purpose of the Study:

• To evaluate how different statistical models influence primary care provider performance profiles.
• To assess the impact of model choice on quality and resource use measures.

Main Methods:

• Compared ordinary least squares, generalized linear models, and multilevel models.
• Ranked 4396 general practitioners on 6 quality and 5 resource use measures using administrative data (2006-2008).
• Assessed model impact via weighted kappa, outlier designation agreement, and changes in rankings.

Main Results:

• Overall agreement between models was high (kappa ≈ 0.85), but outlier designation agreement varied, often below 80%.
• Rankings showed more consistency for process measures than for outcomes or expenses.
• Annual rankings per model showed low agreement across all models.

Conclusions:

• Statistical model choice influences provider rankings, though differences may be minor.
• Many performance measures appear significantly influenced by chance, irrespective of the model.
• Emphasizes the need for careful consideration of both statistical models and performance measures in profiling.