Development and external validation of a text-matching algorithm to identify patients with lower extremity deep...

Venous ultrasound studies evaluating for deep venous thrombosis (DVT) are often only available as unstructured radiology reports. To identify the presence or absence of DVT in a cohort of patients often requires manual review. Existing automated methods (i.e., natural language processing) for identifying the presence of DVT from free text are computationally complex and difficult to integrate into the electronic health record (EHR) and electronic data warehouses as structured data. We sought to develop and validate a simple regular expression algorithm-based approach in structured query language (SQL) to extract the presence or absence of DVT from unstructured ultrasound (US) reports.

We extracted the radiology reports for venous US obtained from the ED at five academic medical centers to create the Regular Expression Aided Determination of Deep Venous Thrombosis (READ-DVT) algorithm. All reports were reviewed for the presence or absence of DVT by trained adjudicators. We split data from one medical center into test and training sets and used the data from four remaining centers for external validation. We created the algorithm through an iterative process within the training set and applied these results to the test and validation sets. We then evaluated performance, and based on these results, re-iterated and conducted multiple rounds of testing and validation including validation on new, unseen data when indicated. For each data set we calculated the sensitivity, specificity, positive predictive value, negative predictive value, and F1 score.

We extracted 9249 radiology reports, of which 1398 (15.1%) were adjudicated positive for DVT. Our algorithm achieved a sensitivity of 93.4%, a specificity of 99.8%, and an F1 score of 0.958 on the test set. On three external validation sets our algorithm achieved sensitivities ranging from 93.7% to 100.0%, specificities ranging from 98.9% to 99.5%, and F1 scores ranging from 0.991 to 0.957. Notably, on our fourth external validation set, our algorithm underperformed with sensitivity of 83.8%, specificity of 99.4%, and F1 score of 0.901. By using these data to iteratively update READ-DVT we found a simple addition to our algorithm vastly improved the model function with no change in performance in the original test, training, and initial three validation sets. We created a new external validation set from the same medical center as our fourth external validation set and achieved sensitivity of 98.0%, specificity of 99.5%, and F1 score of 0.980.

READ-DVT is a robust, multi-center validated method for identifying the presence or absence of DVT in radiology reports. The algorithm performed similarly to the published results of more complex natural language processing tools designed for the same task. In combination with similar validated tools for pulmonary embolism, READ-DVT may enable accurate automated coding of venous thromboembolism.