A Comprehensive Evaluation of Statistical Reliability in ACS NSQIP Profiling Models

Area of Science:

• Surgical Quality Improvement
• Health Services Research
• Biostatistics

Background:

• The American College of Surgeons' National Surgical Quality Improvement Program (ACS NSQIP) provides hospital quality assessments.
• The January 2013 Semiannual Report included 137 risk-adjusted profiling models.

Purpose of the Study:

• To evaluate the statistical reliability of hospital profiling models within the ACS NSQIP.
• To determine if these models are suitable for assessing surgical quality and hospital performance.

Main Methods:

• Calculated median reliability for each of the 137 ACS NSQIP models.
• Determined the percentage of hospitals achieving acceptable reliability (≥0.40) for each model.
• Computed average median reliability across models with shared outcomes.

Main Results:

• Median reliability varied significantly across models, ranging from 0.91 to 0.005.
• Reliability generally correlated with larger sample sizes and higher event rates.
• 72% of 'Essentials' models met a median reliability of 0.40+, compared to 29% of 'Procedure-Targeted' models.
• 98% of 'All Cases Morbidity' models achieved acceptable reliability, while 0% of 'Procedure-Targeted Pancreatectomy Mortality' models did.

Conclusions:

• A substantial portion of ACS NSQIP 'Essentials' models demonstrate adequate statistical reliability for assessing surgical quality.
• 'Procedure-Targeted' models show evolving statistical reliability with promising initial results.
• The findings advocate for broader consideration of statistical reliability in professional performance assessments.