Reliability and utility of anesthesiology entrustable professional activities assessed with a mobile web application

A prior study developed and validated 14 Entrustable Professional Activities (EPA) for anesthesia residents for anesthesiology residency programs to use as the basis for workplace assessment.

The goals of this study were to: 1) measure the reliability of seven of those 14 EPAs using entrustment data collected via a mobile web application (Web App) and 2) examine implementation and utility of the EPA Web App.

Web App design prioritized a simple and efficient user interface, and incorporated revisions based on feedback. Assessments were completed during the 2020-2021 academic year. Generalizability theory was used to examine the distribution of EPA global scores. Qualitative interview data of Web App usage by participants focused on domains of the Consolidated Framework for Implementation Research.

The Web App was used in a total of 1116 observations in 31 residents. Two EPAs were effective in discriminating differences in resident performance, whereas two were not. With 24 observations, one of the EPAs had a G coefficient of 0.68 and phi coefficient of 0.66 approaching a level suitable for formative feedback, whereas the other EPAs did not reach the 0.7 threshold (for moderate reliability in formative assessment, not summative use) even with 24 observations. Several constructs were elicited from the interview data in the domains of intervention characteristics, individual characteristics, outer setting, and inner setting.

The utility of the Web App for EPA assessment was deemed positive for formative, low-stakes assessment. EPAs may not all necessarily be able to discriminate differences in resident performance. However, even a single EPA assessment may still be useful as one component of competency-based medical education via programmatic assessment.