Evaluation of an automated safety surveillance system using risk adjusted sequential probability ratio testing

Area of Science:

• Healthcare Quality Improvement
• Medical Device Surveillance
• Patient Outcome Analysis

Background:

• Automated surveillance tools can enhance quality improvement and medical product safety.
• Assessing hospital performance using patient outcomes with these tools is underexplored.
• This study compares an automated risk-adjusted sequential probability ratio testing (RA-SPRT) method against public mortality reports for cardiac surgery.

Purpose of the Study:

• To evaluate the effectiveness of an automated RA-SPRT tool in identifying hospital outliers for 30-day mortality after isolated coronary artery bypass graft (CABG) surgery.
• To compare the performance of the automated RA-SPRT method with existing state public reporting data.
• To assess the potential for automated systems to facilitate earlier quality improvement interventions.

Main Methods:

• Retrospective analysis of 23,020 isolated adult CABG admissions in Massachusetts hospitals (2002-2007).
• Implementation of the RA-SPRT method in an automated tool to detect hospital outliers annually.
• Defined an event as a higher-than-expected hospital mortality rate; used a type I error rate of 0.05 and type II error rate of 0.10.

Main Results:

• The RA-SPRT method analyzed 83 hospital-year observations.
• It detected 6 adverse events across three hospitals, compared to 5 events in two hospitals identified by state reports.
• Achieved 100% sensitivity and 98.8% specificity in identifying mortality outliers.

Conclusions:

• The automated RA-SPRT method demonstrated strong performance, accurately identifying all true outliers with a low false positive rate.
• This automated system can provide timely, confidential alerts to hospitals before public disclosure.
• Early notification enables proactive quality improvement initiatives, potentially improving patient outcomes.