Prospective Blinded Evaluation of Patient-Physician Agreement Using the Summary Outcome Determination (SOD) Score

To determine whether the Summary Outcome Determination (SOD) score demonstrates a high level of physician-patient agreement in a prospective setting with multiple raters.

For this study, 100 patients who were being evaluated at various intervals following shoulder or elbow surgery were prospectively enrolled from May 30, 2017, through August 31, 2017. The patients' attending physicians and a member of their team (physician assistant, resident, fellow, medical student) assigned categorical and numerical SOD scores while blinded to the scores given by each other. All scores were analyzed among raters, assessing internal consistencies, agreement, and reliability.

The mean follow-up (interval between surgery and completion of the survey) was 31 months, with a range of 1 to 220 months. The intraclass correlation coefficient for patient and physician numerical scores was excellent at 0.82. The weighted κ value for categorical scores was 0.64. Bland-Altman analysis revealed low average discrepancy at 0.6 with a 95% CI of -3.3 to 4.5. The Cronbach α value was 0.94, indicating strong internal validity. The categorical physician-patient agreement occurred within one category 96% of the time.

This study found that the SOD score has strong agreement with excellent intraclass correlation coefficient and weighted κ values, indicating substantial agreement, reproducibility (shown by low average error), and strong internal validity. With promising results in the prospective setting, the SOD score was found to be an easy to use outcome measure with reliable agreement between patient and physician. This score has potential to be a metric revealing the "value" of a specific surgical intervention.