Does direct observation influence the quality of workplace-based assessment documentation?

Area of Science:

• Medical Education
• Assessment in Healthcare
• Competency-Based Medical Education (CBME)

Background:

• Direct observation is crucial for workplace-based assessment (WBA) in CBME.
• Challenges exist in implementing direct observation effectively.
• The impact of observation type on WBA documentation quality is unknown.

Purpose of the Study:

• To determine if observation type (direct vs. indirect) influences the quality of entrustable professional activity (EPA) assessment documentation.
• To compare the quality of written feedback associated with direct and indirect observations.

Main Methods:

• Four raters assessed 244 EPA assessments using the Quality of Assessment for Learning (QuAL) instrument.
• EPA assessments were divided into direct (n=122) and indirect (n=122) observation groups.
• Analysis of variance (ANOVA) compared mean QuAL scores; generalizability analysis assessed QuAL reliability.

Main Results:

• No statistically significant difference in mean QuAL scores was found between direct and indirect observation groups (p=0.17).
• The QuAL instrument demonstrated good reliability for EPA assessments (0.84).

Conclusions:

• Observation type does not appear to affect the quality of EPA assessment documentation.
• Further investigation is needed to understand the practical differences between direct and indirect observation.
• Implications for competence committees evaluating trainee progression require consideration.