American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator Accuracy When...

Area of Science:

• Surgical outcomes research
• Machine learning in healthcare
• Health informatics

Background:

• Current risk calculators (RCs) use only the principal Current Procedural Terminology (CPT) code due to technical limitations.
• This study explores using advanced machine learning (ML) to incorporate multiple CPT codes for improved risk adjustment.

Purpose of the Study:

• To evaluate if incorporating multiple CPT codes enhances the accuracy of the ACS NSQIP risk calculator.
• To compare the performance of ML algorithms using principal CPT code versus multiple CPT codes.

Main Methods:

• Utilized ACS-NSQIP data from over 5 million patients (2016-2020).
• Assessed predictive accuracy for 13 outcomes using extreme gradient boosting (XGB) with principal CPT code and CatBoost (CATB) with up to 21 CPT codes.
• Evaluated discrimination and calibration on training and validation datasets.

Main Results:

• No consistent discrimination advantage for CATB over XGB.
• XGB showed slightly better calibration overall, but CATB performed better when multiple CPT codes were present.
• CATB demonstrated significantly more accurate estimates for patients with 4 or more CPT codes.

Conclusions:

• The current ACS NSQIP RC using XGB and principal CPT code is viable for routine assessment.
• An advanced RC utilizing the CATB algorithm with multiple CPT codes may offer more precise risk estimates, particularly for complex cases.