Not All EPAs Are Created Equal: Fixing Sampling Bias With Utility Modeling

Entrustable professional activities (EPAs) are foundational for understanding resident progress towards practice readiness. Unfortunately, when EPAs were initiated manually, EPA assessment completion has been uneven, creating biases from assessment variability across individuals, specialties, and institutions. Therefore, we introduce EPA assessment utility modeling, which can retrospectively correct for and prospectively avoid these biases by informing each attending of the usefulness of each EPA assessment opportunity and highlighting when EPA assessments are most needed.

We performed a longitudinal analysis of general surgery EPA assessments using an EHR-integrable medical-education platform across 37 institutions. EPA assessment counts were fitted with power law curves to measure skewing. Raw EPA assessment ratings, combined with historical case logs and OR schedules, were analyzed with the platform's large-scale Bayesian network model to quantify each EPA assessment's impact on entrustment learning curves. Lastly, we used Monte Carlo simulations to develop an assessment utility score, as an intuitive label for the predicted benefit of each EPA assessment opportunity, in order to prompt faculty members to complete the most highly useful assessments.

From 6/2023 to 5/2025, 444 faculty assessed 532 residents with 17,245 EPA assessments. EPA assessment counts showed substantial skewing across several factors. By EPA type, 52.8% of EPA assessments were of the top 4 (22.2%) types (power law α = 0.27, 2 p ≈ 0). By faculty, 33.5% of EPA assessments were from the most active 15 (4.3%) faculty members (α = 0.15, 2 p ≈ 0). By faculty specialty, 31.0% were from the most active 2 (9.5%) specialties (α = 0.24, 2 p ≈ 0). By resident, 20.1% were received by the 20 (4.5%) most assessed residents (α = 0.21, 2 p ≈ 0).

EPA assessments were heavily skewed with sampling biases, misrepresenting entrustment levels. To fix these biases and provide a data-driven approach to CBE measurement, we propose an assessment utility framework to optimize EPA assessment timing, assessor, and prioritization.