Pulmonary embolism risk assessment screening tools: the interrater reliability of their criteria

Diagnostic evaluation for suspected pulmonary embolism (PE) is challenging. Dimerized plasmin fragment D (D-dimer) assays are increasingly used but have been validated only in "low-risk" patients. The accurate interpretation and application of risk assessment criteria are critical to the appropriate use of D-dimer. We sought to determine the interrater agreement of attending and third-year resident emergency medicine physicians in the specific elements of the Canadian and the Charlotte risk stratification tools and their clinical application.

We prospectively enrolled a convenience sample of patients presenting to an urban university emergency department with suspected PE. Standardized data collection sheets were used by an attending physician and a third-year resident physician to determine the presence or absence of risk factors included in published PE prediction instruments. Each physician was blinded to the other's results and the patients' D-dimer result. Interrater agreement was measured using kappa statistics (with 95% confidence intervals).

Two hundred seventy-one patients were screened. The kappa scores for each risk criterion were as follows: previous deep vein thrombosis, 0.90 (95% confidence interval, 0.83-0.97); malignancy, 0.87 (0.76-0.97); deep vein thrombosis symptoms, 0.54 (0.39-0.70); immobilization, 0.41 (0.26-0.57); unexplained hypoxia, 0.58 (0.42-0.74); tachycardia, 0.94 (0.89-0.98); hemoptysis, 0.76 (0.51-1.0); and PE more likely than another diagnosis, 0.50 (0.36-0.64).

Interrater agreement was only fair for several important risk criteria. Small differences in determining pretest probability can lead to significant variability in risk assessment and how, or whether, the diagnosis of PE is evaluated. This study raises questions about the reliability and applicability of published PE screening criteria in clinical settings.