Poor quality patient reported outcome measures bias effect estimates in orthopaedic randomized studies

The objective was to assess the potential for biased treatment effects associated with patient-reported outcome measures (PROMs) of varying psychometric quality in randomized clinical trials (RCTs) for rotator cuff disease (RCD).

We searched for RCTs published in the past 5 years (January 2011 to December 2016) in the top five 2015 impact factor orthopedic journals. We accepted RCTs including human participants with RCD, published in English, and using PROMs specific to RCD. We extracted data on study design, sample size, risk of bias for RCTs, quality of PROM used, estimates of effect, and associated measures of variance. PROMs were given numerical ratings of psychometric quality from a prior publication. Continuous measures of effect were transformed by dividing the effect estimate by the standard deviation. Multilevel linear regression analyses were performed to determine whether PROM quality was associated with the magnitude of effect.

Overall, we included 72 RCTs reporting 174 separate outcomes. Mean sample size was 66.8 (95% CI 62.30 to 71.27), mean risk of bias score across all studies was 7.00/10 (95% CI 6.72 to 7.29), psychometric quality summary scores ranged from -2 to 10, and the standardized mean effect estimate was 0.47 (95% CI -0.17 to 1.11). Regression revealed that higher-quality PROMs had smaller estimates of effect (β = -0.32; 95% CI -0.51 to -0.13; P = 0.001). We also found that a longer follow-up period predicted slightly increased effect estimates (β = 0.08; 95% CI 0.02 to 0.13; P = 0.007).

PROMs with poor or unknown psychometric properties overestimate treatment effects in clinical research of RCD by 68.4% (β -0.32/standardized mean effect 0.47). To our knowledge, this is the first empirical evidence that variations in the quality of PROMs bias treatment effect estimates. Researchers and clinicians using data from PROMs must be cautious to explore the quality of that measure so as to not mislead decision-making resulting from biased outcomes.