Inter-Rater Reliability for the Amputation Endpoint in the National Vascular Quality Initiative

National quality reporting efforts after revascularization for peripheral artery disease (PAD) are ongoing. Validation of endpoints are necessary in national quality registries.

This study sought to examine the interrater reliability for the endpoint of major amputation at 1 year in the Vascular Quality Initiative (VQI) registry and the Medicare-linked Vascular Quality Initiative registry (VQI-VISION) against electronic health record (EHR) review.

Surgical or endovascular revascularization procedures between January 1, 2010, and December 31, 2017, in the VQI registry and VQI-VISION for 2 academic health systems were queried. Major amputation data were abstracted by trained data collectors for the VQI and derived from Current Procedural Terminology codes for VQI-VISION. Cases underwent protocolized adjudication for the endpoint of major amputation by EHR review. Paired tests were used to evaluate the sensitivity and specificity. Spearman's ρ and Cohen's κ were used to evaluate interrater reliability.

Amputation endpoints for 1,936 revascularizations were examined. Compared with major amputation data in EHR review, the sensitivity for the VQI registry was 35.9% and the specificity was 99.4% (ρ = 0.53; κ = 0.48). For VQI-VISION, sensitivity was 67.7% and specificity was 98.9% (ρ = 0.75; κ = 0.74). For any amputation in VQI data, sensitivity was 35.3% and specificity was 99.3% (ρ = 0.53; κ = 0.46), and for VQI-VISION, they were 71.6% and 97.7%, respectively (ρ = 0.75; κ = 0.74).

Almost two-thirds of the amputations in the VQI registry and one-third of amputations in VQI-VISION were missing at 1 year compared against adjudicated EHR review. In preparing for national reporting systems for major amputation tracking, data collection system reform is needed.