Measurement Under the Microscope: High Variability and Limited Construct Validity in Emergency Department...

We evaluate variability and construct validity in commercially generated patient-experience survey data in a large sample of US emergency departments (EDs).

We used Press Ganey patient-experience data from a national emergency medicine group from 2012 to 2015 across 42 facilities and 242 physicians. We estimated variability as month-to-month changes in percentile scores and through intraclass correlations. Construct validity was assessed with linear regression analysis for monthly facility- and physician-level percentile scores.

A total of 1,758 facility-months and 10,328 physician-months of data were included. Across facility-months, 40.8% had greater than 10 points of percentile change, 14.7% changed greater than 20 points, and 4.4% changed greater than 30. Across physician-months, 31.9% changed greater than 20 points, 21.5% changed greater than 30, and 13.6% changed greater than 40. Intraclass correlation estimates demonstrated similar variability; however, this was reduced as data were aggregated over fixed time increments. For facility-level construct validity, several facility factors predicted higher scores: teaching status; more older, male, and discharged patients without Medicaid insurance; lower patient volume; less requirement for physician night coverage; and shorter lengths of stay for discharged patients. For physician-level construct validity, younger physician age, participating in satisfaction training, increasing relative value units per visit, more commercially insured patients, higher computed tomography or magnetic resonance imaging use, working during less crowded times, and fewer night shifts predicted higher scores.

In this sample, both physician- and facility-level patient-experience data varied greatly month to month, with physician variability being considerably higher. Facility-level scores have greater construct validity than physician-level ones. Optimizing data gathering may reduce variability in ED patient-experience data and better inform decisionmaking, quality measurement, and pay for performance.