Validity of Routinely Reported Rutherford Scores Reported by Clinicians as Part of Daily Clinical Practice

Routinely reported structured data from the electronic health record (EHR) are frequently used for secondary purposes. However, it is unknown how valid routinely reported data are for reuse. This study aimed to assess the validity of routinely reported Rutherford scores by clinicians as an indicator for the validity of structured data in the EHR. This observational study compared clinician-reported Rutherford scores with medical record review Rutherford scores for all visits at the vascular surgery department between April 1, 2016 and December 31, 2018. Free-text fields with clinical information for all visits were extracted for the assignment of the medical record review Rutherford score, after which the agreement with the clinician-reported Rutherford score was assessed using Fleiss' Kappa. A total of 6,633 visits were included for medical record review. Substantial agreement was shown between clinician-reported Rutherford scores and medical record review Rutherford scores for the left ( k  = 0.62, confidence interval [CI]: 0.60-0.63) and right leg ( k  = 0.62, CI: 0.60-0.64). This increased to the almost perfect agreement for left ( k  = 0.84, CI: 0.82-0.86) and right leg ( k  = 0.85, CI: 0.83-0.87), when excluding missing clinician-reported Rutherford scores. Expert's judgment was rarely required to be the deciding factor (11 out of 6,633). Substantial agreement between clinician-reported Rutherford scores and medical record review Rutherford scores was found, which could be an indicator for the validity of routinely reported data. Depending on its purpose, the secondary use of routinely collected Rutherford scores is a viable option.