Threats to Reliability and Validity With Resident Wellness Surveying Efforts

Residency programs and the Accreditation Council for Graduate Medical Education (ACGME) use survey data for the purpose of program evaluation. A priority for many programs is to improve resident wellness, often relying on self-reported surveys to drive interventions.

We tested for result differences on wellness surveys collected through varying survey methodology and identified potential causes for differences.

Aggregated results on the resident wellness scale for a single institution were compared when collected electronically through the ACGME Resident Survey immediately following the program evaluation survey for accreditation purposes and anonymously through an internal survey aimed at program improvement.

Across 18 residency programs, 293 of 404 (73%) residents responded to the internal survey, and 383 of 398 residents (96%) responded to the 2018 ACGME survey. There was a significant difference (P < .001, Cohen's d = 1.22) between the composite wellness score from our internal survey (3.69 ± 0.34) compared to its measurement through the ACGME (4.08 ± 0.30), indicating reports of more positive wellness on the national accreditation survey. ACGME results were also statistically more favorable for all 10 individual scale items compared to the internal results.

Potential causes for differences in wellness scores between internal and ACGME collected surveys include poor test-retest reliability, nonresponse bias, coaching responses, social desirability bias, different modes for data collection, and differences in survey response options. Triangulation of data through multiple methodologies and tools may be one approach to accurately gauge resident wellness.