Developing Resident-Sensitive Quality Measures for Internal Medicine

The ultimate goal of residency education is to train physicians to deliver high-quality patient care. However, residents rarely receive data-driven feedback because resident-level quality measures are lacking.

To develop and evaluate resident-sensitive quality measures (RSQMs) using electronic health record data to inform graduate medical education.

This cohort study used call schedules linking senior internal medicine residents to patient admissions during overnight internal medicine call shifts at 5 teaching hospitals in Canada from July 1, 2010, through December 31, 2019. Using clinical practice guidelines, 7 RSQMs related to pneumonia or general care for all internal medicine admissions were developed. To support interpretation, a care variation framework was applied that categorized measures as low value (not recommended), discretionary (context dependent), or evidence based (recommended) to enable comparisons between observed and expected variation. The data were analyzed between March 1, 2024, and February 23, 2026.

The low-value care RSQM measured potentially inappropriate red cell transfusions (all admissions). Discretionary RSQMs measured use of antibiotics, imaging, or blood work for either pneumonia or all admissions. The evidence-based care RSQM measured ordering of first-line antibiotics for pneumonia. Resident-level variation was assessed using descriptive statistics, including the median proportion of eligible admissions with each RSQM performed and interquartile range.

The cohort included 132 291 patient admissions (median [IQR] age, 70 [55-83] years; 50.6% male) linked to 793 residents. Residents had a median (IQR) of 187 (89-228) admissions, including a median (IQR) of 18 (10-24) admissions for pneumonia. Potentially inappropriate red cell transfusions occurred in a low proportion of encounters, with little variation (median, 0%; IQR, 0%-0%). Discretionary measures, including use of second-line antibiotics, advanced imaging, chest computed tomography, and serum protein electrophoresis, varied across residents. For pneumonia admissions (n = 13 470), the RSQM for first-line antibiotic use in pneumonia was sensitive to the time windows for included orders, ranging from 22% (3027 admissions) to 76% (10 205 admissions), depending on the cutoffs applied.

This cohort study outlined an approach to developing and evaluating RSQMs using readily available electronic health record data to evaluate internal medicine residents' quality of care. Although the RSQMs showed potential, their use for inpatient internal medicine may be more appropriate at the program level due to unresolved concerns regarding attribution and statistical reliability.