Inter-Rater Reliability for the Amputation Endpoint in the National Vascular Quality Initiative

Area of Science:

• Vascular Surgery
• Health Services Research
• Quality Improvement

Background:

• National quality reporting for peripheral artery disease (PAD) revascularization requires validated endpoints.
• Assessing the accuracy of data in quality registries is essential for reliable national reporting.

Purpose of the Study:

• To evaluate the interrater reliability of major amputation at one year using the Vascular Quality Initiative (VQI) and VQI-Vision registries against electronic health record (EHR) data.
• To identify discrepancies in amputation data reporting between registry abstraction and EHR review.

Main Methods:

• Retrospective analysis of revascularization procedures (2010-2017) from VQI and VQI-VISION registries.
• Major amputation data abstracted from VQI and derived from Current Procedural Terminology codes for VQI-VISION.
• Protocolized adjudication of amputation endpoints via EHR review, with statistical analysis using paired tests, Spearman's ρ, and Cohen's κ.

Main Results:

• Significant underreporting of major amputations was found: 64.1% missing in VQI and 32.3% in VQI-VISION compared to EHR review.
• Interrater reliability (Cohen's κ) for major amputation was moderate (0.48 for VQI, 0.74 for VQI-VISION).
• Sensitivity for detecting major amputations was low (35.9% for VQI, 67.7% for VQI-VISION).

Conclusions:

• Current data collection methods in VQI and VQI-VISION incompletely capture major amputation events at one year post-revascularization.
• Substantial data gaps necessitate significant reform in data collection systems for national quality reporting of PAD outcomes.
• Improved data accuracy is critical for effective quality assessment and patient care in PAD management.